noamaanMulla-03/transformers-issues huggingface dataset:
Difference between bart-large and bart-large-cnn vocabulary
Author: swethmandava
Pull request: no

Trying to finetune from pretrained bart checkpoint as follows:

```
config = BartConfig(**json.load(open(args.config_path, "r"))) #pointing to bart-large-cnn/config.json
model = BartForConditionalGeneration.from_pretrained('facebook/bart-large', config=config) #use pretrained bart model's weights
tokenizer = BartTokenizer.from_pretrained('facebook/bart-large-cnn')
```

but since facebook/bart-large and facebook/bart-large-cnn have different vocab size, it fails. What's the reason behind different vocab sizes? How can I use pretrained bart for finetuning - should I modify bart-large-cnn's config to use the same vocab size as bart-large?

@sshleifer 

Comments:
The reason is the mask token, see https://github.com/huggingface/transformers/issues/3108.
You could try to use the resize_token_embeddings method, but even easier would be to pass the config changes you want to init
```python
BartForConditionalGeneration.from_pretrained('bart-large', num_beams=4, min_length=56, max_length=142, length_penalty=142, ...)
```
[testing] remove USE_CUDA
Author: stas00
Pull request: yes

As discussed at https://github.com/huggingface/transformers/issues/6349 this PR removes `USE_CUDA`.

I don't think I needed to add `CUDA_VISIBLE_DEVICES=""` to `.circleci` config files since those CIs have no gpus anyway.

@LysandreJik, @sgugger, @sshleifer 

fixes: #6349

Comments:
[CIs] report slow tests add --durations=0 to some pytest jobs
Author: stas00
Pull request: yes

It appears that over the last 10 days or so the CIs have gone 4-5 times slower. From 2-3 minutes to 10-11 minutes for `run_tests_torch`.

This PR adds a report, generated by `pytest`, that lists the slowest tests so that we could quickly detect such regressions in our test suite's performance.

As suggested by @sshleifer, pytest can do the diagnostics using `--durations=` flag. I'm not sure what the best number to set it to - let's try with 50:

I propose adding this to:
* `run_tests_torch_and_tf` job - as it runs all non-slow tests in "real-time"
* all scheduled jobs that run slow tests

**edit**: Several fixes have been merged since this slowdown was found, so we are much better off and there is an effort to perform a major clean up here https://github.com/huggingface/transformers/pull/7895 so support this effort let's start with reporting the runtime of all tests, that is `--durations=0` (`pytest` currently doesn't have an option to report only tests slower than a certain run time) and once the clean up has been done, then we dial down to `--durations=50` to only monitor any new outliers.

@sshleifer, @sgugger, @LysandreJik 

Comments:
Trainer with Iterable Dataset
Author: j-rossi-nl
Pull request: yes

# What does this PR do?

Fixes #5990 
Follows #5995 (closed because stale).
Merges all commits from master.

* accomodate iterable dataset without predefined length
* set it as 1 use case: provide max_steps, and NO num_epochs
* Is a merge of master and PR 5995

- [ ] This PR fixes a typo or improves the docs (you can dimiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests), 
      Pull Request section?
- [X] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [X] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [X] Did you write any new necessary tests? 


## Who can review?
@sgugger 


Comments:
Now investigating this FAILED test.
So far, it is PASSED on 3 different environments I have tried.
`FAILED tests/test_tokenization_fast.py::SentencePieceFastTokenizerTest::test_add_special_tokens` from `run_tests_torch_and_tf`

@sgugger 

CircleCI `run_tests_torch_and_tf` is FAILED, because 
```
FAILED tests/test_tokenization_fast.py::SentencePieceFastTokenizerTest::test_add_special_tokens
==== 1 failed, 2964 passed, 493 skipped, 386 warnings in 925.76s (0:15:25) =====
```
On 2 different environments, I have a PASSED.
I reproduced the CI run step by step, and it is still PASSED:
```
conda create -n trans_test_tf_torch python=3.6 
conda activate trans_test_tf_torch
git clone git@github.com:huggingface/transformers.git
cd transformers
git fetch origin pull/7858/head:pull/7858
git pull origin pull/7858 
git checkout pull/7858
pip install --upgrade pip      
pip install git+https://github.com/huggingface/datasets 
pip install .[sklearn,tf-cpu,torch,testing]     
pip install codecov pytest-cov 
```

Result:
`python -m pytest -n auto --dist=loadfile -s -v tests/test_tokenization_fast.py --cov`
`91 passed, 11 warnings in 1851.05s (0:30:51)`

For all the tests:
`python -m pytest -n auto --dist=loadfile -s -v ./tests/ --cov `
`2965 passed, 493 skipped, 385 warnings in 3124.66s (0:52:04)`

Thanks for your contribution!
Modelling Encoder-Decoder | Error :- `decoder_config` used before intialisation
Author: ayubSubhaniya
Pull request: yes

Getting error when sending `decoder_config` as a parameter while initializing an encoder-decoder model from pretrained. 

# What does this PR do?
fixes "UnboundLocalError: local variable 'decoder_config' referenced before assignment"

## Who can review?
@patrickvonplaten @sgugger

Comments:
@patrickvonplaten @sgugger, please review

Great catch @ayubSubhaniya !
[EncoderDecoder] Fix Typo
Author: patrickvonplaten
Pull request: yes

# What does this PR do?

Remove dead code


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @sshleifer
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->


Comments:
Raise error when using AMP on non-CUDA device
Author: BramVanroy
Pull request: yes

The trainer currently implements native AMP in such a way that a GradScaler is always used. AFAIK, and as supported by [this post](https://github.com/pytorch/pytorch/issues/36169#issuecomment-611261781) and [this report on the forums](https://discuss.huggingface.co/t/training-gpt2-on-cpus/1603), this will only work on CUDA devices. Therefore, an error should probably be thrown when a user tries to use `--fp16` alongside a non-CUDA device.

This PR is currently very brief because I am uncertain about a few things:

- I am not sure if the current implementation of `--fp16` in the trainer works with TPUs;
- I am not sure about differences in behaviour between using `apex` and native AMP in this respect;
- I am not entirely sure whether `_post_init` is the right place to do an argument sanity check. Particularly because I am now calling `self.device` which will set the device through `_set_devices`, even though you may (?) want to delay that as long as possible until the arguments are used in the trainer.

cc @sgugger 

Comments:
> I think the postinit is a great place for this sanity check, so the error comes fast to the user, and I don't think our current FP16 implementations (Apex and Amp) work with TPUs. I can check the latter on Monday and report back here.

If you'd rather be more forgiving, let me know. In that case we can do a logging.warning(f"AMP only supported when using CUDA. Turning off AMP because not using a CUDA device ({self.device.type})") and subsequently automatically turning off --fp16. 

Ok, just tested on TPUs and FP16 does not work on them indeed, though native amp fails graciously (testing on TPUs with PyTorch < 1.6 is suicide so I did not try APEX). So if you fix the styling issue, I think this is good to be merged!


It has to be run by the user, I can force-push the styling on your branch if you don't have the setup to run it easily.

> It has to be run by the user, I can force-push the styling on your branch if you don't have the setup to run it easily.

Thanks but it should work fine now. Updated local dependencies and all that and reformatted.

Thanks for your contribution!
[wip] [pegasus] encode s/\r?\n/<n>/g + test
Author: stas00
Pull request: yes

As discussed in https://github.com/huggingface/transformers/issues/7743

* [x] On encode: `s/\r?\n/<n>/g` + test
* [ ] On decode: `s/<n>/\n/g` - not sure what to do since `run_eval.py` can't work with multiline outputs as it uses readline to get each record and inserting `\n` would break the records if they are now multiline.

@sshleifer 

Fixes: #7743

Comments:
I have some concerns defaulting this logic to `True`, given that some `sshleifer/` models like `sshleifer/pegasus-cnn-ft-v2/`, `distill-pegasus-cnn-16-4/` were trained without newlines in the dataset.  I'm sorry I forgot to mention these until now.

I would merge this if the logic to do the regex were defaulted to False and we checked that `PegasusFastTokenizer` did the same thing.
I would also be fine abandoning, since you have already solved the Pegasus replication issue with your `build/` scripts, and we haven't had anybody external ask for this yet.

Sure, we can just leave it unmerged and if someone reports an issue get back to sorting this out. It was a trivial change, so I have no problem with closing this. It's your call, @sshleifer.

will reopen if needed, thx for being flexible @stas00
`decoder_config` variable not defined in EncoderDecoderModel.from_encoder_decoder_pretrained
Author: jsilter
Pull request: no

https://github.com/huggingface/transformers/blob/890e790e16084e58a1ecb9329c98ec3e76c45994/src/transformers/modeling_encoder_decoder.py#L330

Using this function results in an error:

`UnboundLocalError: local variable 'decoder_config' referenced before assignment`

Suggest changing `decoder_config.add_cross_attention` to `kwargs_decoder["config"].add_cross_attention`



Comments:
@jsilter - great catch! I agree 100% with your suggestion! Do you want to open a PR to fix it? :-) 

https://github.com/huggingface/transformers/pull/7903 this is already fixed yesterday @jsilter
Fix issue #7781
Author: jsilter
Pull request: yes

`decoder_config` may not be defined at this line, use `kwargs_decoder["config"]` instead. This is also consistent with the rest of the function.

Fixes # 7781
https://github.com/huggingface/transformers/issues/7781

## Who can review?

Anybody can review, though @patrickvonplaten gave an informal approval in the issue discussion.

Comments:
Hey @jsilter - sorry this fix was already pushed yesterday in #7903. Thanks for spotting the bug!!!
AttributeError: module 'tensorflow_core.keras.activations' has no attribute 'swish'
Author: Mandule
Pull request: no

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     -->

## Details
<!-- Description of your issue -->

tensorflow 2.0.0
after "pip install transformers", i import transformers, meet this error.

<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**:

Comments:
> # ❓ Questions & Help
> ## Details
> tensorflow 2.0.0
> after "pip install transformers", i import transformers, meet this error.
> 
> **A link to original question on the forum/Stack Overflow**:

兄弟，升级tf到2.3

Hi, you should install TensorFlow 2.3 as mentioned by @wmathor
xlm-mlm-17-1280 & xlm-mlm-100-1280 include languages?
Author: wmathor
Pull request: no

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
Where is the list showing what languages are supported by each of these two models?

Comments:
Hi, you can find this data [here](https://github.com/facebookresearch/XLM#the-17-and-100-languages).
[EncoderDecoder] google/roberta2roberta_L-24_wikisplit
Author: patrickvonplaten
Pull request: no

It seems like `google/roberta2roberta_L-24_wikisplit` should be pre-processed differently than originally thought:

```python
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tokenizer = AutoTokenizer.from_pretrained("google/roberta2roberta_L-24_wikisplit")
model = AutoModelForSeq2SeqLM.from_pretrained("google/roberta2roberta_L-24_wikisplit")

long_sentence = """Due to the hurricane, Lobsterfest has been canceled, making Bob very happy about it and he decides to open Bob 's Burgers for customers who were planning on going to Lobsterfest."""

input_ids = tokenizer(long_sentence, return_tensors="pt").input_ids
output_ids = model.generate(input_ids)[0]
print(tokenizer.decode(output_ids, skip_special_tokens=True))
# should output
# Due Due hurricane, Lobsterfest has been canceled, making Bob very happy about it. He decides to open B
# ob's Burgers for customers who were planning on going to Lobsterfest.com.
```

yields a weird word duplication bug, whereas:

```python
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tokenizer = AutoTokenizer.from_pretrained("google/roberta2roberta_L-24_wikisplit")
model = AutoModelForSeq2SeqLM.from_pretrained("google/roberta2roberta_L-24_wikisplit")

long_sentence = """Due to the hurricane, Lobsterfest has been canceled, making Bob very happy about it and he decides to open Bob 's Burgers for customers who were planning on going to Lobsterfest."""

input_ids = tokenizer(tokenizer.bos_token + long_sentence + tokenizer.eos_token, return_tensors="pt").input_ids
output_ids = model.generate(input_ids)[0]
print(tokenizer.decode(output_ids, skip_special_tokens=True))
# should output
# Due to the hurricane, Lobsterfest has been canceled, making Bob very happy about it. He decides to open Bob's Burgers for customers who were planning on going to Lobsterfest.  
```

yields good results.
Notice the difference of 
`input_ids = tokenizer(tokenizer.bos_token + long_sentence + tokenizer.eos_token, return_tensors="pt").input_ids` between code sample 1 and 2.

@patrickvonplaten 

Wait for author's (@shashiongithub) answer before changing code example.


Comments:
Confirmed by author that BOS and EOS have to be added -> fixed in https://github.com/huggingface/transformers/commit/0724c0f3a2d302246d0bd0b7d2f721fa902dee1b.
labels and decoder_input_ids to Glossary
Author: LysandreJik
Pull request: yes

Completes the glossary with entries for `labels` and `decoder_input_ids`.

Closes https://github.com/huggingface/transformers/issues/7865

Pinging @sshleifer and @patrickvonplaten for advice regarding the `decoder_input_ids`, @sgugger for docs.

Comments:
[ProphetNet] add model summary
Author: patrickvonplaten
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @sshleifer
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->


Comments:
labels and decoder_input_ids
Author: AI678
Pull request: no

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     -->

## Details
<!-- Description of your issue -->
what is the difference between labels  and decoder_input_ids for EncoderDecoderModel in the text summarization task?
<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**:

Comments:
I also get confused by the notation and would appreciate some clarifications.
Thank you.

@AI678 I think I got it. Do not take my words as absolute truth since I am new to the subject.
If you are talking about a full Transformer architecture (e.g. BART, T5, PEGASUS), the labels are the token ids to which you compare the logits generated by the Decoder in order to compute the cross-entropy loss. This should be the only input necessary during training or fine tuning.
On the other hand, the decoder_input_ids have the exact same structure of the labels, but are shifted one position to the right in order to add the \<start of sentence\> token. The decoder_input_ids are then passed to the decoder (along with the mask, which masks all unseen tokens!!). The first input of the decoder will be the \<start of sentence\> token so that it starts generating the sentence.

Hope this help, waiting for somebody to acknowledge my answer.

Hi, the glossary has been completed with these terms, you can check it out [here](https://huggingface.co/transformers/glossary.html), let me know if it's still not clear enough.