noamaanMulla-03/transformers-issues huggingface dataset:
Wrong model is used in example, should be character instead of subword model
Author: jsteggink
Pull request: yes

# What does this PR do?

A canine.rst fix.
In the original Google repo for CANINE there was mixup in the model names in the README.md, which was fixed 2 weeks ago. Since this transformer model was created before, it probably resulted in wrong use in this example in canine.rst.

s = subword, c = character

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

## Who can review?

@sgugger

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Comments:
It's not easy to make a file the styler is happy with, haha.
Too Many kernels and embeddings were randomly initialized when loading Hugging Face GPT-2 Model
Author: vivekvkashyap
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:
- Platform:colab
- Jax version (CPU):0.2.13
- Using GPU in script?:No
- Using distributed or parallel set-up in script?:No

### Who can help
@patrickvonplaten @patil-suraj 

Models:
Used hugging face GPT2 model for multiple choice task.


Examples:
```python
self.gpt2=FlaxGPT2Model(config=self.config, dtype=self.dtype)
```


## Information

Model I am using GPT2:

The problem arises when using:
*loading the model

The tasks I am working on is:
* mutliple choice
* dataset:COSMOS

## To reproduce

https://colab.research.google.com/drive/1uTwJ1X1WTxOTDSduKqoUTg3oizehPFJB?usp=sharing

While executing this command:
```python
model = FlaxGPT2ForMultipleChoice.from_pretrained('gpt2')
```
## Expected behavior

To run the following code without any warnings and randomly initialized kernels.


Comments:
Hi, this is because you are using a different base model prefix `self.gpt2` to load the base model. To be able to add any heads and still be able to load the base model weights the pre-trained class expects the base model to have the same prefix.

For gpt2 it is `self.transformer`, changing `self.gpt2` to `self.transformer` should fix this.

Also, try to avoid posting screen-shots, it's usually better for us, if you post the warning/stack-trace as text. Thanks!
Add option to load a pretrained model with mismatched shapes
Author: sgugger
Pull request: yes

# What does this PR do?

Sometimes, users want to load a checkpoint for a given task with a new head for the same task but different shapes. For instance, they may want to use a checkpoint that does text classification on 2 labels to initialize a model that does text classification on 5 labels. 

This PR enables that by adding a new argument to the `from_pretrained` method of `PreTrainedModel`, `TFPreTrainedModel` and `FlaxPreTrainedModel` named `ignore_mismatched_sizes`. When set to True, this argument will ignore the weights from the checkpoint that do not have the same shape as the ones inside the model and leave the randomly initialized weights.

Comments:
[Deepspeed] adapt multiple models, add zero_to_fp32 tests
Author: stas00
Pull request: yes

Massive fixing and testing of different models under Deepspeed, followed up by a recovery of full fp32 weights and various other related tweaks.

zero_to_fp32 isn't currently being tested on deepspeed side (as I didn't write any tests), so testing it here and while at it starting to include more and more models instead of waiting for users to discover that this or that doesn't work.

This PR:

- [x] adds multiple model tests which include a short training and zero_to_fp32.py fp32 weights recovery from the shell
- [x] docs update to the new `zero_to_fp32.py` syntax and new API
- [x] adds a test for `load_state_dict_from_zero_checkpoint` (added in deepspeed)
- [x] fixes multiple models to work with deepspeed
- [x] removed some deepspeed craft from `modeling_wav2vec2.py` as it's no longer needed

TODO:

 Dependencies to resolve:

- [x] https://github.com/microsoft/DeepSpeed/pull/1181
- [x] https://github.com/microsoft/DeepSpeed/pull/1202 - which will also fix https://github.com/huggingface/transformers/issues/12403 marian has the same issue (the test is already here). 
- [x] new deepspeed release after the above PRs are merged (probably 0.4.3)

Fixes: https://github.com/huggingface/transformers/issues/12403

@sgugger, @LysandreJik 


Comments:
Add timeout to CI.
Author: LysandreJik
Pull request: yes

Adds a global timeout to 60 seconds for non-slow tests, and a global timeout of 5 minutes for slow tests.

These can be adjusted later on, but it prevents the two hanging suites right now and is important to merge to get feedback on the current coverage.

I've re-enabled the `-v` option on `pytest` as this was instrumental in discovering the failing test, and would have saved me a lot of time had it been activated by default. 

I'm also removing the `pytest-sugar` dependency, because even if a nice QOL improvement, it was detrimental to the discoverability of the hanging test.

Comments:
Replace NotebookProgressReporter by ProgressReporter in Ray Tune run
Author: krfricke
Pull request: yes

# What does this PR do?

This PR replaces the local trainer NotebookProgressReporter callback by a ProgressReporter. Generally we cannot guarantee correct display of IPython renderables in remote processes (e.g. because they get redirected into files), so it's better to replace these by text based reporters.

Otherwise, these are produced: `<IPython.core.display.HTML object>`

See also https://github.com/ray-project/ray/issues/16197

<!-- Remove if not applicable -->

Fixes https://github.com/ray-project/ray/issues/16197


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@sgugger 



Comments:
Has this fix been packaged with the latest release? I see that the fix was merged into the master branch 18 days ago and the latest release (v4.8.2) was 13 days ago but then I don't see the issue mentioned in the patch release notes. 

I assume it hasn't since I am still getting the same output.
Refactored code to improve performance
Author: AllStars101-sudo
Pull request: yes

 # What does this PR do?
 Refactors several segments of code in the `scripts`,`src`,`tests`,`utils` and `setup.py` and increases performance by a bit, using compression methods and newer practices.
 
 No new functions or methods/models were added; therefore no documentation changes were required.
 
 ## Before submitting
 * [ ]  This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
 * [x]  Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
   Pull Request section?
 * [ ]  Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
   to it if that's the case.
 * [ ]  Did you make sure to update the documentation with your changes? Here are the
   [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
   [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
 * [ ]  Did you write any new necessary tests?



Comments:
[Deepspeed] non-native optimizers are mostly ok with zero-offload
Author: stas00
Pull request: yes

As noticed in https://github.com/huggingface/transformers/issues/11044#issuecomment-870742459 most non-DS optimizers should work with zero-offload as long as they have cpu+gpu implementation (except LAMB).

So this PR relaxes the earlier incorrectly imposed restriction.

@sgugger 



Comments:
Translate README.md to Simplified Chinese
Author: JetRunner
Pull request: yes

This is part of the Hugging Face document translation project. I appreciate it if anyone (from Hong Kong / Taiwan) could help verify the traditional Chinese version (which is still WIP).

Comments:
The Chinese translations looks good to me!

LGTM.

@JetRunner I can help with the translation of Traditional Chinese. Would you like me submitting a PR or any kinds of assistance?

> @JetRunner I can help with the translation of Traditional Chinese. Would you like me submitting a PR or any kinds of assistance?

Sure! Please do so. I recommend you convert the simplified Chinese version to traditional Chinese with a software and then polish it (e.g., replace `软件` with `軟體`) - in this way we can keep the two versions consistent (which is desirable for future maintenance).

Thanks a lot for your help! @qqaatw 

@JetRunner No problem, I'll work on this soon. Once the PR is opened, I'll ping you for reviewing.
Provide mask_time_indices to `_mask_hidden_states` to avoid double masking
Author: mfuntowicz
Pull request: yes

The current behavior for training Hubert masks randomly some "spans" of time according to a `mask_time_indices`  which can be provided or not to the `forward(..., mask_time_indices=Optional[torch.Tensor])` .

When providing this value (_interesting to mask the loss over non masked spans_), the mask was applied outside of the `_mask_hidden_states(...)` function. 

Then, a new mask `_mask_hidden_states(...)` inside was generated potentially masking again some others tokens independently from what was provided through `mask_time_indices`.

This PR provides a fix by ensuring we only mask spans inside `_mask_hidden_states(...)` and correctly apply the masking operating one time.

Comments:
Thanks a lot for fixing this! Can you also make the fix to `modeling_wav2vec2.py`? Think the same error is there

and for both tf_hubert and tf_wav2vec2, we need to do the change as well I think 

Thanks a lot!
How to figure out which pretrained tokenizers support emojis?
Author: Prashant446
Pull request: no

Hi, I am working on a dataset with emojis. I found that the BERT tokenizer doesn't support emojis, and we have to manually add them and train their embeddings (#7648). But the RoBERTa tokenizer seems to identify emojis on tokenization as the following code:
```Python3
tokenizer = AutoTokenizer.from_pretrained('distilroberta-base', use_fast=True, normalization=True)
tokenizer.encode('I 🙁 hate ☹️ to 😣 see 😖 this 😫 fail 😩 , 🥺 pls 😢 help 😭 me 😤')
```
outputs: 
``` 
[ 0, 100, 8103, 27, 10172, 4157, 42699, 9253, 12605, 7, 17841, 2469, 192, 17841, 25448, 42, 17841, 4958,
 5998, 17841, 15375, 2156, 8103, 8210, 3070, 2968, 29, 17841, 7258, 244, 17841, 12410, 162, 17841, 10470, 2]
 ```
None of this represents the "\<unk\>" token so, all of them should be trained embeddings. Is that right? Also, why are there weird characters in most of the words in the vocab of RoBERTa model like - 'ĸ', 'Ġthis', 'ĠðŁĺ', '«', 'Ġfail', 'ĠðŁĺ', etc.?

Comments:
you can try this:
tokenizer.decode(tokenizer.encode('I 🙁 hate ☹️ to 😣 see 😖 this 😫 fail 😩 , 🥺 pls 😢 help 😭 me 😤'))

If the tokenizer successfully decodes back to origin emojis then yes! Your tokenizer can encode emojis.
In this case, you use distillroberta tokenizer which use BPE (Radford et al. 2019) method, hence your tokenizer can encode emojis.

![Selection_467](https://user-images.githubusercontent.com/42698038/125319165-81b07200-e308-11eb-947f-4169b1d8fa97.png)



I already found this hack somewhere, but thanks anyway!
Support for pointer-generator architectures.
Author: AmirAktify
Pull request: no

# 🚀 Feature request

Is there interest in adding pointer generator architecture support to huggingface? These are currently supported in [fairseq](https://github.com/pytorch/fairseq/blob/master/examples/pointer_generator/README.md), and in general should not be terrible to add for most encoder-decoder seq2seq tasks and modeks.

## Motivation

Pointer-generator architectures generally give SOTA results for extractive summarization, as well as for semantic parsing. (See for instance this paper [https://arxiv.org/pdf/2001.11458.pdf](url)).

## Your contribution
If there is interest but not bandwidth from huggingface members, I could try to add pointer-generator support for a specific architecture such as the T5 and see how hard it would be to port over fairseq's implementation for instance.

Note: apologies also if I've missed where huggingface supports it already.


Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
After loading fine-tuned model from local and use it for prediction, it continue training from scratch again!
Author: Fushier
Pull request: no

transformers: 4.6.1
torch: 1.3
gpu: k40m * 2
datasets: msra_ner
model: hfl/chinese-bert-wwm

I'm fine-tuning a model for token classification, and after training, I save the model:
`trainer.save_model('./model')
trainer.save_metrics('./model')`

Now I load the saved model : 
`
tokenizer = AutoTokenizer.from_pretrained("./model")
config = transformers.AutoConfig.from_pretrained("./model")
model = AutoModelForTokenClassification.from_pretrained("./model", config=config)
args = TrainingArguments(
    output_dir='./results'
)

trainer = Trainer(
    model,
    args,
    data_collator=data_collator,
    tokenizer=tokenizer,
)
`

And then predict with test dataset:
`
predictions, labels, metrics = trainer.predict(tokenized_datasets)
`

But in terminal, I get:
Some weights of the model checkpoint at hfl/chinese-bert-wwm were not used when initializing BertForTokenClassification: ['cls.predictions.transform.dense.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.dense.bias', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.bias', 'cls.predictions.decoder.weight']
- This IS expected if you are initializing BertForTokenClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BertForTokenClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of BertForTokenClassification were not initialized from the model checkpoint at hfl/chinese-bert-wwm and are newly initialized: ['classifier.bias', 'classifier.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
  0%|                                                                                                                                 | 0/28130 [00:00<?, ?it/s]

It seems like that the model starts training again?

@sgugger 


Comments:
No, this means you are loading a model that hasn't been fine-tuned. The model present in in your `./model` directory seems to have a pretraining head which is discarded, and a new token classification head is instantiated.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Only test the files impacted by changes in the diff
Author: sgugger
Pull request: yes

# What does this PR do?

This PR  adds some utilities to only run the tests that are impacted by the diff in a PR, to have the CI run faster, save on CI costs and avoid hanging tests. For now the first stage of deployment only concerns PRs, the jobs run at each push (either on circle CI or GitHub actions) still run all the tests.

To make this work, the new utility `tests_fetcher.py` works in three stages:
1. It analyzes the diff and to grab the added/deleted/modified files
2. It builds a internal map that contains for each module all the other modules that depend on it (recursively). For instance `trainer` depends on `trainer_utils` so that map says `trainer_utils` impacts `trainer`. It is recursive, so since `trainer_utils` depends on `file_utils`, the map says `file_utils` impacts `trainer_utils` and `trainer`.
3. It maps all the impacted files to their corresponding test files. Note that some files in the library may not have direct test files (for instance a model configuration file has no direct test) but we the impacted files computed above, a model configuration files impacts the corresponding modeling file, so changing a model configuration will run the tests of that model.

The result is then run for each of the tests in circle CI. Note that for some tests (like text_examples and test_custom_tokenizer) we just check that there is at least some tests to run (so no trivial diff) but still run all the tests like before. In all the jobs, the output of the `tests_fetcher.py` is saved as an artifact for future debugging.

Comments:
What happens if one only changes a file like `src/transformers/models/bert/__init__.py`? The only files that are impacted by this is `src/transformers/__init__` -> does this mean no tests are run? What if someone introduces a typo in this file that doesn't allow anymore to import `BertModel`? 

Ah great point, I was supposed to add the direct_deps when the file is an init (so that if you change the bert init, it runs the model and tokenizers tests) but forgot! Will make a PR this afternoon!
Flax - Loading pretrained model overwrites weights of different shapes
Author: borisdayma
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: master
- Platform: Ubuntu
- Python version: 3.9

### Who can help
@patil-suraj @sgugger 
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @Rocketknight1

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

Model hub:

- for issues with a model report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

## Information

Model I am using (Bert, XLNet ...): Custom FlaxBart

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [X] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Create a custom model by subclassing - just change output shape (lm_head & final_logits_bias)
2. use `CustomModel.from_pretrained('facebook/bart-large-c')
3. check `model.params['final_logits_bias'].shape`, it will come from the pretrained model

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

The shape of weights should be checked prior to be overwritten.

Right now my approach is:
* load pre trained model
* init custom model from config
* update manually the weights needed


Comments:
This should be fixed by the work in #12664 

Closing because it was fixed