noamaanMulla-03/transformers-issues huggingface dataset:
Make use of KenLM's pypi version
Author: patrickvonplaten
Pull request: no

### Feature request

Use PyPi version of KenLM instead of installing archieve file from Github.

### Motivation

Now that kenlm is on pypi: https://github.com/kpu/kenlm/issues/363#issuecomment-1280050903 let's use it :-) 


### Your contribution

@sanchit-gandhi would you give this a try as an exercise to see how we deal with soft dependencies in Transformers maybe? :-) 

I think we can now update the `setup.py` file to include the pypi version of `kenlm` and remove all those `pip install https://github.com/kpu/kenlm/archive/master.zip` statements  e.g. in https://github.com/huggingface/transformers/blob/c7edde1a692012eda23bc2b837588557b97ad729/docker/transformers-all-latest-gpu/Dockerfile#L44

I'd also advocate to add a `is_kenlm_available` flag to give a nice error message and update warnings / errors here: https://github.com/huggingface/transformers/blob/c7edde1a692012eda23bc2b837588557b97ad729/src/transformers/pipelines/__init__.py#L839

Comments:
Also cc @ydshieh and @sgugger here

And let's not forget the [circleCI config](https://github.com/huggingface/transformers/blob/c7edde1a692012eda23bc2b837588557b97ad729/.circleci/create_circleci_config.py#L130) as well ;-)

@patrickvonplaten I would like to pick up this.
Update `max_diff` in `test_save_load_fast_init_to_base`
Author: ydshieh
Pull request: yes

# What does this PR do?

This test seems flaky. After looking a bit deeper, I am not sure if we should expect to get the same (or very close) weights with/without `_fast_init` for the deleted key.
https://github.com/huggingface/transformers/blob/9ecb13d63a9524478656b2233e6fb4e9f15d3fbf/tests/test_modeling_common.py#L343
https://github.com/huggingface/transformers/blob/9ecb13d63a9524478656b2233e6fb4e9f15d3fbf/tests/test_modeling_common.py#L400

My intuition is that **the values for that deleted key could be different with 2 different init. methods.**

I also change the way of `max_diff` being calculated.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thank you @patrickvonplaten! Although I still have some doubt:

Assumption we have:
> now this initialization should be the same for both the fast and slow init method

What we do
> _init_weights weights is overwritten by a deterministic self._mock_init_weights

But `_mock_init_weights` is only defined in the testing module, and basically it just does `data.fill_(3)`.
So the assumption is only True in our own testing (which uses `_mock_init_weights`). This won't be the case when we want to load the model outside the testing. So I am not very sure the purpose of this testing.

But good for me if we don't want to touch it. We probably need to add some common about the flakyness for some tests though.
 

@sgugger Could you check if the change in the way `max_diff` being calculated worth the merge 🙏 ? Thanks
Add `flan-t5` documentation page
Author: younesbelkada
Pull request: yes

# What does this PR do?

This PR adds `FLAN-T5` on the documentation page - following the same approach for `t5-v1.1`: https://huggingface.co/docs/transformers/model_doc/t5v1.1 

cc @sgugger @ydshieh @ArthurZucker 


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thank you for the feedback @sgugger ! I should have addressed the comments now
Allow flax subfolder
Author: patrickvonplaten
Pull request: yes

First, I'm sorry about this long list of commits :sweat_smile: - I have my fork set up correctly now so this shouldn't happen again.
Second, this change would be very useful for this PR in `diffusers` so that clip can be loaded from a subfolder: https://github.com/huggingface/diffusers/pull/880#discussion_r1004209900

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Support Roberta on `accelerate`
Author: younesbelkada
Pull request: yes

# What does this PR do?

This PR adds `Roberta` model family support with `accelerate` ! This aims to support `int8` quantization for these models.
Before merging I have notice few nits that I need to fix and discuss! I am unsure whether these fixes should be here or in `accelerate`.

1- If the model has the attribute `_keys_to_ignore_on_save`, it seems that it does not get properly initialized by `accelerate` (but I might be missing something here). AFAIK all models that has `accelerate` support for now, has at most the attribute `_keys_to_ignore_on_load_missing` but not `_keys_to_ignore_on_save`. Therefore [when the base model gets saved in the `accelerate` test](https://github.com/younesbelkada/transformers/blob/aefea27619b43cbb3e518ea972009e525676dc8d/tests/models/roberta/test_modeling_roberta.py#L587), these parameters does not get saved in the state_dict. I had to come up with a modification in the `_load_pretrained_model` function to randomly initialize these parameters since they're ignored by the `_load_state_dict_into_meta_model`. This post processing trick happens [here](https://github.com/younesbelkada/transformers/blob/aefea27619b43cbb3e518ea972009e525676dc8d/src/transformers/modeling_utils.py#L2598). 
2- Therefore I had to change the `accelerate` tests. Since the parameters that are assigned in the `_keys_to_ignore_on_save` are initialized randomly, I propose to check the logits compatibility between the base model and the accelerate model only for the attention outputs and not the `lm_head` output. These modifications happens [here](https://github.com/younesbelkada/transformers/blob/aefea27619b43cbb3e518ea972009e525676dc8d/tests/models/roberta/test_modeling_roberta.py#L569) - maybe this modification could happen in the super class?
3- Last nit: in the `accelerate` tests, it is better to not override the variable `inputs_dict` since inside the main loop, we can switch from a `xxxForMultipleChoice` to a `xxxForQuestionAnswering` model. Therefore, this variable does not get modified by the class function `_prepare_for_class` since it [gets modified only if the model is a `MODEL_FOR_MULTIPLE_CHOICE` model.](https://github.com/younesbelkada/transformers/blob/aefea27619b43cbb3e518ea972009e525676dc8d/tests/test_modeling_common.py#L165-L173) and not reset to the correct `inputs_dict` afterwards.

Need also to fix the slow test for `lilt` model where I am having `ValueError: embeddings.position_ids doesn't have any device set.`

cc @sgugger  


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Closing in favor of #19906
DeprecationWarning from Pillow (with Pillow ≥ 9.1.0)
Author: tasercake
Pull request: no

### System Info

`transformers`: 4.22.2
`pillow`: 9.2.0
Python 3.9.9

### Who can help?

@NielsRogge @amyeroberts (tagged based on changes to `image_utils.py` in #18520, but the issue seems to span most of the repo)

## Reproduction

[Pillow 9.1.0 deprecated a bunch of constants](https://pillow.readthedocs.io/en/stable/releasenotes/9.1.0.html#deprecations) such as `PIL.Image.BILINEAR`, leading to the following warning when importing the CLIP model

Note: I ran python with warnings enabled (`python -W always`)

```
>>> from transformers import CLIPFeatureExtractor

.../transformers/image_utils.py:239: DeprecationWarning: BILINEAR is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BILINEAR instead.
  def resize(self, image, size, resample=PIL.Image.BILINEAR, default_to_square=True, max_size=None):
```

Caused by [this line in `image_utils.py`](https://github.com/huggingface/transformers/blob/bbd150e92f84db72e7507d0c3ce69474b2948839/src/transformers/image_utils.py#L364) (though there's other instances where deprecated PIL constants are used)

These constants are pending removal in Pillow 10 (July 2023).

## Action required

Noticed that transformers doesn't currently enforce a Pillow version constraint in [setup.py](https://github.com/huggingface/transformers/blob/main/setup.py), so I've opened this issue to check if any action is required – **either enforce Pillow < 10, or migrate to using the new Pillow constants**

---

Additional info: discovered this warning when importing https://github.com/huggingface/diffusers – simply running `import diffusers` on a fresh install (version 0.4.1) triggers this warning for me.


Comments:
cc @amyeroberts @alaradirik 

Closing as this has been resolved in #19654
`accelerate` support for `RoBERTa` family
Author: younesbelkada
Pull request: yes

# What does this PR do?

This PR adds `accelerate` support for:
- `RoBERTa`
- `data2vec_text`
- `Lilt`
- `Luke`
- `XLM-RoBERTa`
- `CamemBERT`
- `LongFormer`
This way, any of the models above can be loaded in 8bit using `load_in_8bit=True`.

Since these models copy the same `xxxLMHead` from `RoBERTa` I had to change the copied modules too - happy also to break down this PR into several smaller PRs, 

This PR also fixes a small bug on `accelerate` tests where the variable `input_dict` is overriden by `xxForMultipleChoice` models. 

Can also confirm all slow tests pass (single + multiple GPUs)

cc @sgugger @ydshieh 


Comments:
_The documentation is not available anymore as the PR was closed or merged._
there is no log and processbar when running trainer.train()
Author: louisyeaaah
Pull request: no

### System Info

When I open a .ipynb in vscode or datashell, there is no log and processbar when running trainer.train(), but in jupyter notebook, it shows.
<img width="770" alt="截屏2022-10-26 11 55 46" src="https://user-images.githubusercontent.com/87161948/197910485-5d16a652-48a4-4c83-b110-b9ad7a95046b.png">


<img width="1138" alt="截屏2022-10-26 12 06 12" src="https://user-images.githubusercontent.com/87161948/197910740-3f7ba219-f27c-4a76-82ac-827109d9257e.png">

### Who can help?

@sgugger 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

`from transformers import AutoModelForSequenceClassification

model = AutoModelForSequenceClassification.from_pretrained('bert-base-cased',
                                                           num_labels=2)

print(sum([i.nelement() for i in model.parameters()]) / 10000)
import numpy as np
from datasets import load_metric
from transformers.trainer_utils import EvalPrediction
metric = load_metric('accuracy')

def compute_metrics(eval_pred):
    logits, labels = eval_pred
    logits = logits.argmax(axis=1)
    return metric.compute(predictions=logits, references=labels)


eval_pred = EvalPrediction(
    predictions=np.array([[0, 1], [2, 3], [4, 5], [6, 7]]),
    label_ids=np.array([1, 1, 1, 1]),
)

compute_metrics(eval_pred)
from transformers import TrainingArguments, Trainer

args = TrainingArguments(output_dir='./output_dir', evaluation_strategy='epoch')
args.num_train_epochs = 1
args.learning_rate = 1e-4
args.weight_decay = 1e-2
args.per_device_eval_batch_size = 32
args.per_device_train_batch_size = 16

trainer = Trainer(
    model=model,
    args=args,
    train_dataset=dataset_train,
    eval_dataset=dataset_test,
    compute_metrics=compute_metrics,
)
# train
trainer.train()`

### Expected behavior

I hope it will output totally the same in all of the three IDEs

Comments:
Vscode or datashell do not support widgets, as far as I know, so we can't show the same progress bars there as in a notebook.

thanks
TypeError: ('Keyword argument not understood:', 'ignore_mismatched_sizes')
Author: Xappuccino
Pull request: no

### System Info

transormers~=4.21.0
tensorflow~=2.8.2
python~=3.7.3
when I modify type_vocab_size in config.json in bert-base-chinese, and pass ignore_mismatched_sizes param to from_pretrianed function like below:
`self.bert = TFBertModel.from_pretrained(pretrain_path, ignore_mismatched_sizes=True)
`
Then I got this error `TypeError: ('Keyword argument not understood:', 'ignore_mismatched_sizes')`

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

transormers~=4.21.0
tensorflow~=2.8.2
python~=3.7.3
modify type_vocab_size in config.json in bert-base-chinese, and pass ignore_mismatched_sizes param to from_pretrianed function like below:
`self.bert = TFBertModel.from_pretrained(pretrain_path, ignore_mismatched_sizes=True)
`

### Expected behavior

load bert-base-chinese checkpoint successfully

Comments:
Can you try to upgrade your version of Transformers and try again? If it still fails, could you please post the full traceback?

I have solved my problem, it was due to the old version transformers code I saved in my project directory that invalidated the transformers version upgraded through pip install~, thx for your replay!
Add checkpoint links in a few config classes
Author: ydshieh
Pull request: yes

# What does this PR do?

Add checkpoint links in the following config classes:

- CLIPConfig
- GroupViTConfig
- OwlViTConfig
- XCLIPConfig

A necessary condition to make the tiny model creation work (PR #19901) for those models.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Generate: contrastive search uses existing abstractions and conventions
Author: gante
Pull request: yes

# What does this PR do?
This PR updates contrastive search to follow existing abstractions conventions in other generation functions. It consists of several tiny changes, with the reasoning for each change in the PR comments below.

This is part of the effort to make converting to TF easier. All slow tests pass (`RUN_SLOW=1 py.test tests/generation/test_generation_utils.py -k contrastive -vv`)

Comments:
_The documentation is not available anymore as the PR was closed or merged._
data_collator.py:131: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow
Author: FlorinAndrei
Pull request: no

### System Info

- `transformers` version: 4.23.1
- Platform: Linux-5.15.0-52-generic-x86_64-with-glibc2.35
- Python version: 3.10.6
- Huggingface_hub version: 0.10.1
- PyTorch version (GPU?): 1.12.1+cu116 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help?

@sgugger 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

https://colab.research.google.com/drive/1gK9iXBiIEmmt2OMq6wmxwcpslkchHJh-?usp=sharing

See the output of `trainer.train()`

### Expected behavior

The warning should not occur.

Comments:
Thanks for the report, the PR linked above should fix it.
Fix warning when collating list of numpy arrays
Author: sgugger
Pull request: yes

# What does this PR do?

As reported in #19822 there is a warning issued by PyTorch when we try to batch the result of a feature extractor when `return_tensors="pt"` is not activated. This PR fixes it by stacking the list of NumPy array in a big array first.

Fixes #19822

Comments:
_The documentation is not available anymore as the PR was closed or merged._
t5 model's decoder do not use EncDecAttention.key & value in text generation task
Author: CaffreyR
Pull request: no

### System Info

- `transformers` version: 4.20.1
- Platform: macOS-12.4-arm64-arm-64bit
- Python version: 3.9.10
- Huggingface_hub version: 0.8.1
- PyTorch version (GPU?): 1.13.0.dev20220709 (False)
- Tensorflow version (GPU?): 2.8.0 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@patrickvonplaten, @Narsil, @gante

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I use `transformers.T5ForConditionalGeneration` to do inference in text generation task(NaturalQuestions). We all know that the longer the answer is the more times the `decoder` goes `forward`. But when I tried to log the forward in each layer. I found that the model goes `encoder` once, `A type decoder` once,  `B type decoder` many times based on the answer length.

### Expected behavior

`A type decoder` use EncDecAttention. q,k,v,o

![image](https://user-images.githubusercontent.com/84232793/197341985-bf12427d-2323-475c-865a-e3a0a61d7bde.png)


But `B type decoder` only use EncDecAttention. q,o
![image](https://user-images.githubusercontent.com/84232793/197341988-d9466cfa-c217-4cc2-ae73-296b50d4c77c.png)

Many thanks!! :)

Comments:
Hi @CaffreyR 👋 Our models, when used for generation purposes, have a cache that stores repeated computations (key and value of the attentions) turned on by default. If you'd wish to see the full (redundant) computations being executed, try using the model with `use_cache=False`.

Great! Thanks for your help!
Remove embarrassing debug print() in save_pretrained
Author: Rocketknight1
Pull request: yes

@sgugger spotted this one, sorry about that! (cc @gante)

Comments:
_The documentation is not available anymore as the PR was closed or merged._

This is already part of #19900 which is awaiting your review 😛 

If it is, you haven't pushed it!

Oh! Then let's merge this.
v4.22.1 ErnieForMaskedLM Bug
Author: wzjj98
Pull request: no
Labels: name: bug

### System Info

![Xnip2022-09-17_22-50-00](https://user-images.githubusercontent.com/50035364/190864836-f71c7f8c-8acf-45b0-90d2-e1d5ac8556e2.jpg)


@LysandreJik

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

from transformers import ErnieForMaskedLM

### Expected behavior

Failed to import transformers.models.ernie

Comments:
Hi @wzjj98, could you please share a bit more information about your PC setup and the script you're calling?

I can confirm it shouldn't be a problem to import `ErnieForMaskedLM` with `transformers==4.22.1`

My GPU NVIDIA GeForce GTX 3090
Processor:Intel(R) Xeon(R) Platinum 8255C CPU @ 2.50GHz


![Xnip2022-09-18_14-58-11](https://user-images.githubusercontent.com/50035364/190889830-83662ecc-0861-4b7f-8cda-6bbaa94f531d.jpg)
![Xnip2022-09-18_14-59-01](https://user-images.githubusercontent.com/50035364/190889832-ffcb86f0-d941-41d1-ad1f-a9982769c4f3.jpg)
![Xnip2022-09-18_14-59-12](https://user-images.githubusercontent.com/50035364/190889835-4ac3e831-c96b-4488-b60d-566f09c01083.jpg)



This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Add `accelerate` support for M2M100
Author: younesbelkada
Pull request: yes

# What does this PR do?

This PR adds `accelerate` support to `M2M100`, therefore this enables loading NLLB models in 8-bit using `load_in_8bit=True`.

This might contain a breaking change but I am not sure. 
When initializing the model in the meta device using `accelerate` the module `self.shared` is intialized and set to the correct device using `set_tensor_to_device` thrice - since it is shared by 3 modules (base model, encoder, decoder) - so it somehow ends up being on the `meta` device. 
Therefore manually assigning a new module with the weights that correspond to the weights of the `shared` module should do the trick. But I am wondering if this is a breaking change since the `shared` module of the Encoder & Decoder won't be "shared" anymore. It should not be a problem at inference time, but can be problematic when training the model.

cc @sgugger 

Also I know T5 also supports `accelerate` and uses `shared` embeddings. The only difference I see from both implementations are the `_keys_to_ignore_on_load_missing` that contains the `shared` weights for `T5` and doesn't contain the shared weights for M2M100

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Add type hints to TFPegasusModel
Author: EdAbati
Pull request: yes


# What does this PR do?

This adds type annotations to `TFPegasusModel` and `TFPegasusForConditionalGeneration` as part of #16059


## Who can review?

@Rocketknight1

Thanks :)


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Looks perfect now. Thank you!
Safetensors tf
Author: sgugger
Pull request: yes

# What does this PR do?

This PR continues to explore loading models using `safetensors` by adding support for TensorFlow model. It adds support for:
- saving model using `safetensors` with the same API as PyTorch models
- loading models with a safetensors file in TensorFlow-format
- loading models with a safetensors file in PyTorch-format

Follow-up PRs will add the support for sharded checkpoints in TensorFlow as well as loading in PyTorch a safetensors file in TensorFlow-format

**Note:** All tests failures are due to the new release of safetensors being broken, not this PR :-)

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Let inputs of fast tokenizers be tuples as well as lists
Author: sgugger
Pull request: yes

# What does this PR do?

Fixes #19882
Not sure if this was an oversight when introducing fast tokenizers or if there is a real reason for not accepting tuples as well as list here (which is the case everywhere else from a quick search). We'll see if the CI picks something failing but it looks like it fixes the issue.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Add `accelerate` support for BART-like models
Author: younesbelkada
Pull request: yes

# What does this PR do?

This PR adds `accelerate` support for BART-like models, so that these models can be loaded in 8bit using `load_in_8bit=True`.
Follows the same logic as https://github.com/huggingface/transformers/pull/19912 regarding shared embeddings

Do not merge before https://github.com/huggingface/accelerate/pull/792 gets merged! 

cc @sgugger 



Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks a lot! Merging since https://github.com/huggingface/accelerate/pull/792 has been merged 🟢