noamaanMulla-03/transformers-issues huggingface dataset:
Fix gradient_checkpointing backward compatibility
Author: sgugger
Pull request: yes

# What does this PR do?

This supercedes #14405 and fixes #14388 by going at the root of the problem. When the code for backward compatibility is executed in the main init, the submodules of the model have not been created yet, so there is nothing to do. That code needs to be executed in some kind of `post_init`.

We currently don't have a `post_init` in our models, and for another operation that is very similar (`init_weights`, which needs ot be executed at the end of the init), we have a call to that method at the end of the init of every model. The good fix will thus be to replace that call to `init_weights` to a call to `post_init` (which will call `init_weights` internally). This will be a big PR that touches every model, so will implement this for the end of the week.

For a quick fix since we need to do a patch release because of the BC problem, this PR uses a forward pre hook (executed before the forward method) that removes itself. So the code is executed just before the first forward (not as clean as in a post init but the next best thing).

Comments:
This broke HF/deepspeed integration with pt-1.8 or pt-1.9 - works fine with pt-1.10. found with git bisecting and reported by @jeffra, as their CI broke with our master.

```
RUN_SLOW=1 pyt tests/deepspeed/test_deepspeed.py::TestDeepSpeedWithLauncher::test_clm_1_zero3 -sv
```


```
E           Traceback (most recent call last):
E             File "/mnt/nvme1/code/huggingface/transformers-master/examples/pytorch/language-modeling/run_clm.py", line 524, in <module>
E               main()
E             File "/mnt/nvme1/code/huggingface/transformers-master/examples/pytorch/language-modeling/run_clm.py", line 472, in main
E               train_result = trainer.train(resume_from_checkpoint=checkpoint)
E             File "/mnt/nvme1/code/huggingface/transformers-master/src/transformers/trainer.py", line 1316, in train
E               tr_loss_step = self.training_step(model, inputs)
E             File "/mnt/nvme1/code/huggingface/transformers-master/src/transformers/trainer.py", line 1849, in training_step
E               loss = self.compute_loss(model, inputs)
E             File "/mnt/nvme1/code/huggingface/transformers-master/src/transformers/trainer.py", line 1881, in compute_loss
E               outputs = model(**inputs)
E             File "/home/stas/anaconda3/envs/py38-pt19/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
E               return forward_call(*input, **kwargs)
E             File "/mnt/nvme1/code/github/00optimize/deepspeed/deepspeed/runtime/engine.py", line 1580, in forward
E               loss = self.module(*inputs, **kwargs)
E             File "/home/stas/anaconda3/envs/py38-pt19/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1057, in _call_impl
E               for hook in itertools.chain(
E           RuntimeError: OrderedDict mutated during iteration
```
Improve semantic segmentation models
Author: NielsRogge
Pull request: yes

# What does this PR do?

* add SegFormer documentation (including a figure)
* fix padding (fixes #14332) for SegFormer
* add attribute to the configuration of SegFormer and BEiT called `semantic_ignore_index`, which defaults to 255. The loss functions of semantic segmentation models typically use 255 instead of -100. The reason for this is that some datasets include a 0 in the annotated segmentation maps to indicate "background", however it can be that background is not included in any of the labels of the dataset. e.g. ADE20k has 150 labels, but "background" is not included. Therefore, one reduces the labels of all segmentation maps by 1 value, and replaces the 0 by 255 as shown [here](https://github.com/open-mmlab/mmsegmentation/blob/441be4e435127868a0c72a4e0e6b87662a4c415b/mmseg/datasets/pipelines/loading.py#L140-L145). It's only after that that images get resized using PIL. However, if we replace values by -100, PIL can't read these images, and will thrown an error.
* add option to pass `segmentation_maps` to `BeitFeatureExtractor`, and add corresponding tests.

To do:
`SegformerFeatureExtractor` currently includes the `align`, `do_random_crop` and `do_pad` arguments at initialization, however I wonder whether it's maybe better to remove those, and only include the bare minimum in the feature extractors (similar to `ViTFeatureExtractor`) to get started (i.e. resizing, center cropping, normalizing). Things like random cropping and padding is maybe already a bit too much and makes the feature extractors more complex. It's also not easy to determine good default values for this feature extractor; should it randomly crop + pad by default, or not? 
- [x] remove random cropping and padding from `SegformerFeatureExtractor`, if one agrees on this.
- [x] make tests of SegformerFeatureExtractor and BeitFeatureExtractor consistent.
- [ ] update the `preprocessor_config.json` of the semantic segmentation models on the hub.

Comments:
force_bos_token_to_be_generated is depricated, should be replaced by forced_bos_token_id in BART documentation
Author: kforkaran
Pull request: no
Labels: name: Good First Documentation Issue

Replace force_bos_token_to_be_generated in the following file

https://github.com/huggingface/transformers/blob/29dfb2dbb10cdba6327ff287db56b182c1db29b1/docs/source/model_doc/bart.rst

Comments:
Thanks for spotting. Feel free to open a PR to fix this!

Closing, as it's fixed by #14434
AttributeError: 'NoneType' object has no attribute 'encode_plus' with XLNet tonkenizer
Author: kutayoncuyilmaz
Pull request: no

Hi everyone,

I am using XLNet tokenizer , however I am receiving this error.

```
**from transformers import XLNetTokenizer, XLNetModel

tokenizer = XLNetTokenizer.from_pretrained('xlnet-base-cased')

input_txt = "India is my country. All Indians are my brothers and sisters"
encodings = tokenizer.encode_plus(input_txt, add_special_tokens=True, max_length=16, return_tensors='pt', return_token_type_ids=False, return_attention_mask=True, pad_to_max_length=False)**
```

---------------------------------------------------------------------------
```
AttributeError                            Traceback (most recent call last)
<ipython-input-64-4b204650a45a> in <module>()
      1 input_txt = "India is my country. All Indians are my brothers and sisters"
----> 2 encodings = tokenizer.encode_plus(input_txt, add_special_tokens=True, max_length=16, return_tensors='pt', return_token_type_ids=False, return_attention_mask=True, pad_to_max_length=False)

AttributeError: 'NoneType' object has no attribute 'encode_plus'
```

Comments:
Hey @kutayoncuyilmaz, could you share your software versions as asked in the template? Thanks

You can do so by running `transformers-cli env` in your environment.

Thank you. I solved the problem
Recover Deleted XNLI Instructions
Author: Helw150
Pull request: yes

# What does this PR do?

This reintroduces the instructions and task description of XNLI to the examples documentation for text classification. These seem to have been inadvertently deleted during the examples re-org since they followed the instructions for Tensorflow instructions for `run_glue.py`. You can see the deletion [here](https://github.com/huggingface/transformers/pull/11350/files#L206).

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

## Before submitting
- [x] This PR fixes a typo or improves the docs.


## Who can review?

The reorg was done by @sgugger, so he can best confirm that the deletion of these was inadvertent.

Comments:
Mismatch in pre-processed wikitext-103 corpus and using pre-trained tokenizer for TransfoXLLMHeadModel
Author: Akhila-Yerukola
Pull request: no

In `examples/run_transfo_xl.py`, the pre-processed wikitext-103 corpus is loaded using:

`corpus` = TransfoXLCorpus.from_pretrained(args.model_name)
`
Example of pre-processed batch converted to tokens:

> ['<eos>', '=', 'Homarus', 'gammarus', '=', '<eos>', '<eos>', 'Homarus', 'gammarus', ',', 'known', 'as', 'the', 'European', 'lobster', 'or', 'common', 'lobster', ',', 'is', 'a', 'species', 'of', 'clawed', 'lobster', 'from', 'the', 'eastern', 'Atlantic', 'Ocean', ',', 'Mediterranean', 'Sea', 'and', 'parts', 'of', 'the', 'Black', 'Sea', '.', 'It', 'is', 'closely', 'related', 'to', 'the', 'American', 'lobster', ',', 'H.', 'americanus', '.', 'It', 'may', 'grow', 'to', 'a', 'length', 'of', '60']

Evaluating the TransfoXLLMHeadModel model on this corpus gives a ppl of ~18.

However when I use the pre-trained `TransfoXLTokenizer` for wikitext-103 using:
`tokenizer = TransfoXLTokenizer.from_pretrained('transfo-xl-wt103')`,
there is a mismatch in the tokenizations. 

Example of using pre-trained tokenizer to tokenize wikitext-103:

> ['<eos>', '=', 'Homarus', 'gammarus', '=', '<eos>', '<eos>', 'Homarus', 'gammarus', ',', 'known', 'as', 'the', 'European', 'lobster', 'or', 'common', 'lobster', ',', 'is', 'a', 'species', 'of', 'clawed', 'lobster', 'from', 'the', 'eastern', 'Atlantic', 'Ocean', ',', 'Mediterranean', 'Sea', 'and', 'parts', 'of', 'the', 'Black', 'Sea', '.', 'It', 'is', 'closely', 'related', 'to', 'the', 'American', 'lobster', ',', 'H', '.', 'americanus', '.', 'It', 'may', 'grow', 'to', 'a', 'length', 'of']

Here, `H.` is being split, whereas the pre-processed version has it as a single token.
Evaluating the TransfoXLLMHeadModel model on this version of the corpus gives a ppl of ~29. 

Could you please help me understand why there is a mismatch?


Comments:
You are right, thanks for catching this.

This behavior is inherited from the BERT Tokenizer but the TransformerXL Tokenizer should behave differently ([this line](https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/pytorch_pretrained_bert/tokenization_transfo_xl.py#L316) which split on punctuation is not present in the original tokenizer of Transformer XL [here](https://github.com/kimiyoung/transformer-xl/blob/master/tf/vocabulary.py#L25-L42)).

I'll check there no other differences, add a test on this and fix this in the next release.

@thomwolf Hello thomwolf, first of all very ty for your library. I'm reproducing transformer-xl benchmark performance using hugginface Tokenizer, but I think there is still mismatch. I just want to ask you if its solved.
i would very much appreciated if you reply this :)
(EncoderDecoderModel) Why decoder_start_token_id are different between train and generation?
Author: hansd410
Pull request: no

As source code given below from [EncoderDecoderModel](https://huggingface.co/transformers/model_doc/encoderdecoder.html?highlight=decoder_start_token#transformers.EncoderDecoderModel.forward), **decoder_start_token_id** are different between train and generation.

```
>>> # training
>>> model.config.decoder_start_token_id = tokenizer.cls_token_id
```

```
>>> # generation
>>> generated = model.generate(input_ids, decoder_start_token_id=model.config.decoder.pad_token_id)
```

Why they are not same?

Comments:
Which language is available for EncoderDecoderModel pre-trained model?
Author: hansd410
Pull request: no
Labels: name: Good First Documentation Issue

Hi, I have a question.

From the example code of [EncoderDecoderModel](https://huggingface.co/transformers/model_doc/encoderdecoder.html#transformers.EncoderDecoderModel.forward), pre-trained model 'bert-base-uncased' looks trained by EN-FR sentence pairs. Is pre-trained model for decoder is trained in French? How can I load pre-trained model in EN-EN pair? Am I confusing?

```
from transformers import EncoderDecoderModel, BertTokenizer
import torch

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = EncoderDecoderModel.from_encoder_decoder_pretrained('bert-base-uncased', 'bert-base-uncased') # initialize Bert2Bert from pre-trained checkpoints

# training
model.config.decoder_start_token_id = tokenizer.cls_token_id
model.config.pad_token_id = tokenizer.pad_token_id
model.config.vocab_size = model.config.decoder.vocab_size

input_ids = tokenizer("Hello, my dog is cute", return_tensors="pt").input_ids
labels = tokenizer("Salut, mon chien est mignon", return_tensors="pt").input_ids
outputs = model(input_ids=input_ids, labels=input_ids)
loss, logits = outputs.loss, outputs.logits

# save and load from pretrained
model.save_pretrained("bert2bert")
model = EncoderDecoderModel.from_pretrained("bert2bert")

# generation
generated = model.generate(input_ids, decoder_start_token_id=model.config.decoder.pad_token_id)
```

Comments:
Hi,

Not sure what you mean, `bert-base-uncased` is the English pretrained BERT checkpoint. So in the code example above, we initialize the weights of the encoder and the decoder with the weights of BERT, and the weights of the cross-attention layers of the decoder are randomly initialized. One should fine-tune this warm-started model on an English downstream dataset, like summarization.

Thank you for reply @NielsRogge .
Then I want to ask, why the decoder output is French in example code?

`labels = tokenizer("Salut, mon chien est mignon", return_tensors="pt").input_ids`

Oh I get your confusion. 

The `EncoderDecoderModel` framework is meant to be fine-tuned on text-to-text datasets, such as machine translation. Of course, if you want to fine-tune an `EncoderDecoderModel` to perform translation from English to French, it makes sense to warm-start the decoder with a pre-trained French checkpoint, e.g.:

`model = EncoderDecoderModel.from_encoder_decoder_pretrained('bert-base-uncased', 'camembert-base') 
`
I'll update the code example. Also note that you then should use CamemBERT's tokenizer to create the labels.

@hansd410 do you mind opening a PR to fix this is in the docs?

@NielsRogge Not at all. Please do so.
Add a post init method to all models
Author: sgugger
Pull request: yes

# What does this PR do?

This PR introduces the proper fix for #14388 by introducing a new `post_init` method to each model, which replaces the current `init_weights()` call. The method can execute any code that requires the model to be properly initialized, such as the `init_weights()` or the gradient checkpointing BC fix (and more if need to in the future).
 

Comments:
Incosistent vocab sizes in t5-base model & tokenizer
Author: alex4321
Pull request: no

## Environment info

- `transformers` version: 4.11.3
- Platform: Linux-5.4.104+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.12
- PyTorch version (GPU?): 1.9.0+cu111 (False)
- Tensorflow version (GPU?): 2.6.0 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

## Information

I'm using `t5-base` model (yet my test shows the same result for `t5-small`).

To implement some knowledge-distillation task and "classical" sequence-to-sequence tasks - I'm trying to use `batch_size x seq_length x vocab_size` array as labels for both kinds of tasks (soft labels in KD case, one-hot hard labels I'm seq2seq case).

So I need to convert tokenizer output from `batch_size x seq_length` to `batch_size x seq_length x vocab_size` one-hot array to pass it to my custom loss later.
Yet I found out that I can't just use `T5Tokenizer(...).vocab_size` to build one-hot matrix - `T5ForConditionalGeneration(...).config.vocab_size` have different value.
So when I'm trying to build a one-hot vector based on tokenizer vocab size - I'm getting dimension mismatch errors.

The next code shows me different vocabulary sizes when I'm trying to access vocab size through `T5Tokenizer` and related `T5ForConditionalGeneration` config:
```
from transformers import T5ForConditionalGeneration, T5Tokenizer

T5_MODEL = "t5-base"
print("tokenizer", T5Tokenizer.from_pretrained(T5_MODEL).vocab_size)
print("model", T5ForConditionalGeneration.from_pretrained(T5_MODEL).config.vocab_size)
```
```
tokenizer 32100
model 32128
```

## Expected behavior

I expected the tokenizer & model to have the same vocabulary size.

Comments:
cc @patrickvonplaten @patil-suraj 

Duplicate of https://github.com/huggingface/transformers/issues/4875 I think

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
[Bert, et al] fix early device assignment
Author: stas00
Pull request: yes

As flagged by @cbalioglu we are doing device placement in a sub-module's `__init__` in some models, which is a wrong place to do that. 

https://github.com/huggingface/transformers/blob/83ef8bcac2f6ce00a3c6256a4ba747c8802480f6/src/transformers/models/bert/modeling_bert.py#L185

It appears that it originally was in `forward` and then was moved to `__init__` without removing the device placement. Here is the use of the same in `forward`

https://github.com/huggingface/transformers/blob/83ef8bcac2f6ce00a3c6256a4ba747c8802480f6/src/transformers/models/luke/modeling_luke.py#L248

So this PR fixes 7 models where this happened.

@sgugger, should we add another check to our quality control to catch any `device=` in `__init__`?

@sgugger, @LysandreJik 

Comments:
Add ImageGPT
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR adds [ImageGPT](https://openai.com/blog/image-gpt/), "Generative Pre-training from Pixels", by OpenAI. ImageGPT is to GPT2 what ViT is to BERT.

OpenAI released 3 variants (small, medium and large) more than a year ago. Models are on the hub: https://huggingface.co/models?other=imagegpt

It directly fits into the existing GPT-2 model (with some minor changes: "quick gelu" activation function, different layernorm, no tied embeddings). The cool thing is that you can just use the `generate()` method to generate pixel values.

Here's a [Colab notebook](https://colab.research.google.com/drive/1u8dmI4uAvZ5oO-E01S6gh5ThWjiPCKXO?usp=sharing) for both conditional and unconditional image generation.
Update: [new notebook](https://colab.research.google.com/drive/1AHtycNtck6qxjxI5UhqaZTc1POH2U1we?usp=sharing), with `ImageGPTFeatureExtractor`.

Big thanks go to https://github.com/openai/image-gpt/issues/7, who made it very easy for me to understand and contribute the model.

## To do:

- [x] Write tests for `ImageGPTFeatureExtractor`.

Comments:
I'm trying to use the notebook you linked, but I'm receiving an import error:

ImportError: cannot import name 'ImageGPTForCausalLM' from 'transformers' (/usr/local/lib/python3.7/dist-packages/transformers/__init__.py)

Hi @NielsRogge! I just realized the [original Image GPT architecture](https://github.com/openai/image-gpt/blob/c6af2ebf57e2460c71fefa53cd9054b060cf716d/src/model.py#L36) uses a root mean square instead of a standard deviation in its layer normalization so I believe it should look like [this](https://github.com/apeguero1/image-gpt/commit/6ad5120fa3538843c9a020a1f43833b13f0aef06). Looks like there's a [paper](https://arxiv.org/abs/1910.07467) on it too haha :D

Interestingly, the image generation produces only subtle visible differences when using the same random seed but better to stick with the original tensorflow implementation I guess?

Thanks, I've updated it.
Fix TFViT
Author: NielsRogge
Pull request: yes

# What does this PR do?

Fix the code example of `TFViTForImageClassification` + fix the integration test.

cc @ydshieh 

Comments:
LGTM, thanks! 



I open an issue #14403 for the following.
_________________________________

Hi, I found that there is something wrong (not introduced in this PR).

`model.save_pretrained("tmp", saved_model=True)` will give an error `AttributeError: 'ViTConfig' object has no attribute 'items'`.
(I found this when running `test_saved_model_creation_extended` in TF ViT test script)

Maybe it's better to open a new issue for this.

**Update**
Not specific to `TFViT`, even occurs for `TFBert`
```
from transformers import TFBertModel
model = TFBertModel.from_pretrained("bert-base-uncased")

# failed
for x in model.config.items():
    print(x)

# failed
model.save_pretrained("tmp", saved_model=True)
```
This probably suggests that `test_saved_model_creation_extended` (@tooslow) in `test_modeling_tf_common.py` will fail for all TF models.


Full example:
```
from transformers import TFViTForImageClassification
model = TFViTForImageClassification.from_pretrained('google/vit-base-patch16-224')
model.save_pretrained("tmp", saved_model=True)
```
Add GitPython to quality tools
Author: LysandreJik
Pull request: yes

Adds gitpython to the `quality` command of the setup, as otherwise the quality tools cannot be used.

cc https://github.com/huggingface/transformers/pull/14379

Comments:
Should we thus remove this line in the circleCI config?
```
            - run: pip install isort GitPython
```

It's there twice (code quality and repo consistency jobs).

Correct, just pushed the update. Thanks!