noamaanMulla-03/transformers-issues huggingface dataset:
VisualBertTokenizer
Author: mszsorondo
Pull request: no

### Feature request

VisualBert takes 2 main inputs: tokenized text and tokenized images. 
The text tokenization can already be handled by the BertTokenizer, but the visual tokenization still has no support, and is no trivial task. This visual tokens are built with embeddings derived from a set of regions, each one corresponding to the region of a detected object of the image from an object detector.
Here's a more detailed description of those embeddings from the [paper](https://arxiv.org/pdf/1908.03557.pdf):

    Each embedding in F is computed by summing three embeddings:

    f_o, a visual feature representation of the bounding region of f, computed by a convolutional neural network.
    f_s, a segment embedding indicates it is an image embedding as opposed to a text embedding.
    f_p, a position embedding, which is used when alignments between words and bounding regions are provided as part of the input, and set to the sum of the position embeddings corresponding to the aligned words.

As a tip, remember that some VisualBert checkpoints handle different visual embedding dimensions. You can use the [examples from the model docs](https://github.com/huggingface/transformers/blob/main/docs/source/en/model_doc/visual_bert.mdx) as a guide.
Also note that, given that the embedding depends of an object detector, this should be an explicit parameter of the visual tokenizer since different detectors will perform differently.

### Motivation

Building a visual embedding is conceptually simple, but implementing it is a tedious task, and there is no standard way to handle this directly with Transformers.

### Your contribution

This issue arised while building the ``` DummyVisualBertInputGenerator ``` as a requisite for exporting the model to ONNX in Optimum. This is still in progress.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
load_module will be removed in Python 3.12
Author: Rocketknight1
Pull request: no

Several of our scripts call `spec.loader.load_module()`. Although this mostly affects standalone utils scripts, it's also called at the top level of `processing_utils.py`, which will be executed by all `Processor` classes.

`load_module()` has been deprecated for a while and will be fully deleted in Python 3.12, which is entering beta soon. We need to replace this code or the library will not be usable in Py3.12.

I can investigate and try to find a suitable replacement when I have time, but if anyone is more familiar with that code and can think of an obvious replacement that achieves the same goal, let me know!

cc @sgugger

Comments:
Thanks for pointing this out! Will have a look.
Update deprecated load_module
Author: sgugger
Pull request: yes

# What does this PR do?

This PR updates the uses of `load_module` (which is going to be dropped in Python 3.12) to a non-deprecated API (this should work starting Python 3.5, so all good for Transformers).

Fixes #21645 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Ernie-M
Author: KnutJaegersberg
Pull request: no
Labels: name: New model

### Model description

Ernie-M looks pretty good in multilingual benchmarks, beating XLM-Roberta. 
Paddlepaddle recently added ernie-m to the huggingface repo, we can use it with paddlenlp.transformers. 
Would be nice to have the model supported in huggingface transformer as well. 

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

https://huggingface.co/PaddlePaddle/ernie-m-base
https://huggingface.co/PaddlePaddle/ernie-m-large

Comments:
https://github.com/PaddlePaddle/ERNIE/blob/ernie-kit-open-v1.0/erniekit/modules/ernie.py

has more implementation details.

Hi, @shermansiu is there any pytorch/tf implementation of this model?

None that I'm aware of.

Anyways, the author of [ERNIE-Pytorch](https://github.com/nghuyong/ERNIE-Pytorch) ported over a few other Ernie models to Huggingface. I'm sure it could be adapted for this. And the PaddlePaddle syntax is quite similar to that of PyTorch, so I'm sure it should be relatively easy, though it'll probably take some time.

@shermansiu Thanks for the resources!
I am currently trying to port the model to huggingface(pytorch), (done till Embedding Layer with acceptable tolerance of 1e-3)

Hi @KnutJaegersberg, Ernie-M is implemented!
`tf2` BERT checkpoint to `pytorch_model.bin` (with MLM head)
Author: Iulian277
Pull request: no

### System Info

- `transformers` version: 4.25.1
- Platform: Windows-10-10.0.19044-SP0
- Python version: 3.9.5
- Huggingface_hub version: 0.11.1
- PyTorch version (GPU?): 1.13.0+cpu (False)
- Tensorflow version (GPU?): 2.10.0 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: False
- Using distributed or parallel set-up in script?: False

### Who can help?

@ArthurZucker @younesbelkada @gante 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I am trying to convert a `tf2` BERT checkpoint to the `pytorch_model.bin` format in order to upload it on the Huggingface hub. I know that there are 2 scripts for doing this type of conversion (one for [tf1](https://github.com/huggingface/transformers/blob/main/src/transformers/models/bert/convert_bert_original_tf_checkpoint_to_pytorch.py) checkpoints and one for [tf2](https://github.com/huggingface/transformers/blob/main/src/transformers/models/bert/convert_bert_original_tf2_checkpoint_to_pytorch.py) checkpoints).

The problem is that my BERT checkpoint is in `tf2` format and I want to convert it to a pytorch model (encoder + heads).

I mention that I've followed this [guide](https://github.com/tensorflow/models/blob/master/official/nlp/docs/train.md#pre-train-a-bert-from-scratch) and used the [official ](https://github.com/tensorflow/models/tree/master/official/nlp) tensorflow scripts for obtaining the checkpoints.

### Expected behavior

I would like to export a `tf2` BERT checkpoint to a `pytorch` model, exporting the MLM head alongside the encoder.

Comments:
Please use the [forums](https://discuss.huggingface.co/) for such questions as we keep issues for bugs and feature requests only.

Note that we do not provide conversion scripts to convert checkpoints obtained with other libraries, the ones exposed are to convert the checkpoints from their original implementation to the Hugging Face format. If you train a Hugging Face Tensorflow model, you'll then seamlessly be able to convert it to PyTorch/Flax.
[WIP] Move X-MOD models to facebook organization
Author: jvamvas
Pull request: yes

# What does this PR do?

As discussed in https://github.com/huggingface/transformers/pull/20939, the new models https://huggingface.co/jvamvas/xmod-base etc. should be moved to the [facebook](https://huggingface.co/facebook) organization.

This PR changes the hardcoded model names in the code.

Next steps:
- [ ] Someone please add me to the facebook org
- [ ] I move the models
- [ ] PR can be merged
- [ ] I leave the facebook org

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case. https://github.com/huggingface/transformers/pull/20939
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

@ArthurZucker


Comments:
_The documentation is not available anymore as the PR was closed or merged._
Update document of WhisperDecoderLayer
Author: ling0322
Pull request: yes

# What does this PR do?

Fix the document of input `hidden_states` and `encoder_hidden_states` in `WhisperDecoderLayer.forward`
According to the document of `WhisperDecoder` the shape should be `(batch, seq_len, embed_dim)` not `(seq_len, batch, embed_dim)`


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@sanchit-gandhi

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Perfect, thanks again!
Can the transformer models run without any local storage at all?
Author: varnlp
Pull request: no

### Feature request

We have a use case where we'd like to download transformer models from our S3 or other storage location directly into memory (without saving it in local storage), finetune the model and save the final model directly to the remote storage through an API.

We're wondering if this use case of not using local storage at all is possible using the current library?

### Motivation

Our usecase requires minimizing local storage usage.

### Your contribution

We are trying to figure out if this feature is already supported

Comments:
This is indeed to supported by the library.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[Trainer] Optimize the use of datasets.IterableDataset in distributed setup
Author: lhoestq
Pull request: no

Right now the Trainer uses `IterableDatasetShard` to skip examples on each node and avoid ending up with duplicate data.
This is not efficient for vision or audio tasks since we waste I/O and CPU time reading and decoding files that are not used.

We consider implementing an optimized sharding for distributed training directly in `datasets`. Right now a `datasets.IterableDataset` is already a `torch.utils.data.IterableDataset` that automatically takes care of distributing the necessary input shards to subprocesses in single node (since `datasets` 2.3.0).

The idea would be to also take into account the rank and world size to distribute the input shards. Maybe distributing the. `datasets.IterableDataset` across nodes should be asked explicitly by the user.

cc @sgugger WDYT ? Do you have other ideas in mind to optimize the use of `datasets.IterableDataset` for distributed training in pytorch ?

Comments:
It seems that `webdataset` also does this when you use `wds.split_by_node` and `wds.split_by_worker`, and is used to train clip or diffusion models at scale.

That would be awesome! I don't think I need more than an API to tell the iterable dataset that it should take care of the data on process i over n.

Maybe one thing that might be needed is the total length (when available) in some kind of attribute, since the length of the itreable dataset would then be smaller (and the total length is not going to be just a round multiple of the number of processes).

Opened a PR here, introducing `datasets.distributed.split_dataset_by_node`: https://github.com/huggingface/datasets/pull/5369

feel free to play with it and share your feedbacks :)

That'll be for when I'm back from vacation ;-)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Re-ping me in one month GitHub bot ;-)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[WhisperModel] fix bug in reshaping labels
Author: jonatasgrosman
Pull request: yes

Currently, in the Whisper model's forward pass, the target `labels` are reshaped using the `view` method before being passed into the loss function:

https://github.com/huggingface/transformers/blob/1567bef3b35c51b7a3cc6b4edf243b208279155d/src/transformers/models/whisper/modeling_whisper.py#L1214

The view method requires the Torch Tensor to be contiguous, and certain operations are commonly performed on the labels that might cause them not to be contiguous.

So using the `view` can cause problems during model training. This issue has already been fixed on another model by the @sanchit-gandhi in a [PR](https://github.com/huggingface/transformers/pull/16748), and I'm just replicating the same solution (using `reshape()` instead of `view()`) here for the Whisper model.


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Also cc @ArthurZucker
[bloom] gradient_checkpointing fix
Author: stas00
Pull request: yes

The  `BloomBlock.forward`'s args signature is:

https://github.com/huggingface/transformers/blob/9d1116e9951686f937d17697820117636bfc05a5/src/transformers/models/bloom/modeling_bloom.py#L417-L425

but when it's called in `gradient_checkpointing` code this is used:

https://github.com/huggingface/transformers/blob/9d1116e9951686f937d17697820117636bfc05a5/src/transformers/models/bloom/modeling_bloom.py#L772-L778

so unless I'm mistaken `head_mask` is passed as `layer_past`.

This PR re-injects the missing `layer_past` arg.

I see that there are tests that test the overall feature, but I haven't looked closely at what they test. Perhaps it happens that `head_mask` isn't being used, so it happens to work w/ it.

@younesbelkada, could you please check if this is an omission or it wasn't passed for a specific reason? 

Comments:
Add OPT resources to the transformers documentation
Author: alissadb
Pull request: yes

# What does this PR do?

Fixes #20055 (partially)

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).

## Who can review?
@stevhliu 
Thanks in advance, if I miss anything please let me know :) 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks a lot! I think when I saved the file in my editor it automatically changed the formatting. But anyway, I've reverted the formatting changes 🙂  

Thanks again for the changes, everything looks great! Pinging @sgugger for a final review, and then we can merge 😄
[CLAP] Add CLAP to the library
Author: ArthurZucker
Pull request: yes
Labels: name: New model

# What does this PR do?
Adds CLAP to the HF library cc @younesbelkada

Comments:
Now we should create a new task : `zero-shot audio classification`! 


_The documentation is not available anymore as the PR was closed or merged._

Ok! Looking very good, last thing is the `configuration` docstring. @younesbelkada will finish this 😉 

Models are here (with nice model cards hehe):
- https://huggingface.co/ybelkada/clap-htsat-fused
- https://huggingface.co/ybelkada/clap-htsat-unfused
Will transfer them on `laion-ai` once we got an approval :)

1. Some renaming has to happen to normalise the parameters around mel extraction. Will do this and everything should look good 😉 
2. Add the dependency on torchvision as the np implementation received a big NO 😢 

Found a few discrepancies with the various variables in the config that are not used. Will finish asap should makes things clearer. 

Ok I broke the history
NeoX underperforming on A100
Author: mrseeker
Pull request: no

### System Info

- `transformers` version: 4.25.1
- Platform: Linux-5.15.0-50-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.13
- Huggingface_hub version: 0.11.1
- PyTorch version (GPU?): 1.12.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help?

When running EleutherAI/pythia-12b-deduped on A100, generation slows down from 15tok/s to 3tok/s when sending 1024 tokens as input. This behaviour does not occur using A6000 with the same setup (14.04 tok/s, 1024 tok input). Ruled out "machine" issue as this happens on multiple servers on multiple locations (runpod, vast.ai, google...)

Does any of you have an idea what the root cause of this issue could be? Running pure python with no accelerate or anything that could "interfere" with the speed of HF. I am running a standard KoboldAI application.

Tagging: @ArthurZucker @younesbelkada @StellaAthena 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

1. Clone https://github.com/henk717/KoboldAI + install dependencies.
2. Start the KoboldAI using model EleutherAI/pythia-12b-deduped
3. Let it generate 100 tokens until it reaches 1024 tokens.
On A100, this would slowly start to slow down token generation. On A6000, this has no effect.

### Expected behavior

Expected behaviour would be that the token generation would be same, regardless of amount inserted.

Comments:
Are you using a 40 GB or 80 GB A100? A 40 GB A100 has slightly less VRAM than an A6000 (48 GB), and it’s possible that you’re tripping a failsafe such as CPU-offload to avoid an OOM error.

12 billion params * 3 Bytes per parameter = 36 GB of VRAM. My general rule of thumb is to add 20% overhead space for a full context length of 2048, which would push the model over the 40 GB limit.

One way to test this theory would be to try running it with LLM.int8. Another would be to carefully monitor GPU and CPU usage during inference.

This might be the issue indeed. It's automatically offloading to CPU but not generating any overflow warnings, making it difficult to pinpoint where the issue would lie. Going for the 80Gb did indeed solve the issue.
[`bnb`] Introducing `BitsAndBytesConfig`
Author: younesbelkada
Pull request: yes

# What does this PR do?

This PR introduces a new feature termed as `BitsandbytesConfig` - enabling users to play more flexibly with `transformers` + `bitsandbytes` API.
This is also a first step of one of the major refactor we are planning to support possibly more quantization features

This PR also addresses: https://github.com/huggingface/transformers/pull/20281#issuecomment-1409894311 

With this PR it will be also possible to apply advanced usecases for `bnb` models such as offloading parameters in cpu & gpu to run some part in int8 and the other part in cpu (but in `fp32`)

Draft for now, will update the docs and fix the CI tests after a first pass 💪 

One of my comment being that we should keep `load_in_8bit` argument as it is for now (as I think this argument is quite powerful) and progressively entirely replace it with the config in the future if more quantization methods will be supported on `bitsandbytes`.

cc @sgugger 

Comments:
@younesbelkada great job again. :)
I really like the refactored API which supports passing a whole config object containing all relevant BnB options in one place.
I am curious are the calculations during inference on the offloaded parameters (in fp32) to CPU and disk actually executed on the CPU or do they get transferred to one of the GPUs for calculations during every pass?

_The documentation is not available anymore as the PR was closed or merged._

This PR is now ready for review! Would love to have a round of review @sgugger 

Thanks for the extensive review! I should have addressed the comments now 💪 

Thanks ! Exaclty! Looking forward to it!
[`CLAP`] Fix few broken things
Author: younesbelkada
Pull request: yes

# What does this PR do?

This PR fixes the forward pass that was broken in the `main` branch of `transformers` for `ClapModel`. To reproduce:

```python
from datasets import load_dataset
from transformers import ClapModel, ClapProcessor

dataset = load_dataset('ashraq/esc50')
input_text = ["Sound of a dog", "Sound of vaccum cleaner"]
audio_sample = dataset["train"]["audio"][-1]['array']

model_id = "ybelkada/clap-htsat-unfused"

processor = ClapProcessor.from_pretrained(model_id)
model = ClapModel.from_pretrained(model_id)

input_text = processor.tokenizer(input_text, return_tensors="pt", padding=True)
input_sample = processor.feature_extractor(audio_sample, return_tensors="pt")

out = model(input_ids=input_text.input_ids, attention_mask=input_text.attention_mask, input_features=input_sample.input_features, is_longer=input_sample.is_longer)
print(out.logits_per_audio.softmax(dim=-1)[0])
```

This PR fixes also few other nits that were missing during the bad rebase

This PR also fixes the doctest and the failing slow tests

cc @ArthurZucker @sgugger 

Comments:
There are also 2 more lines that got erased, shared that with you on slack 😉 

_The documentation is not available anymore as the PR was closed or merged._

Local doctests and tests pass: 
```
============================================================ 119 passed, 39 skipped, 38 warnings in 75.88s (0:01:15) ============================================================
```