noamaanMulla-03/transformers-issues huggingface dataset:
Doctest Integration
Author: will-rice
Pull request: no

# 🚀 Feature request

It would be nice to add doctest to ensure new additions have working examples in the docstrings and that existing examples do not become outdated with API changes. The feature is already in [pytest](https://docs.pytest.org/en/6.2.x/doctest.html) so adding it should be straightforward.

## Motivation

This will reduce the number of PRs for typos/errors in docstring examples which should allow reviewers to focus on other PRs.

## Your contribution

I wouldn't mind working on this if everyone thinks it is a good idea.

Comments:
Hi @will-rice! Most of our tests are setup to work with doctest, and we had doctest coverage a few months back. Unfortunately, as the number of model grows, the issue is less of a documentation issue and more of an infrastructure issue :) We're working on setting them back up as we speak.

ok awesome! I'll go ahead and close this then.
Fix MBart failing test
Author: LysandreJik
Pull request: yes

Fixes the failing test by adjusting the expected sentence. The sentence turns from
```
[...] will only worsen the violence and misery of millions.
```
to
```
[...] only make violence and misery worse for millions of people.
```

which seems grammatically correct still. @patrickvonplaten please let me know if you think this is a real issue or not.

Comments:
Thanks for fixing this test - this was actually failing since a long time and no-one fixed it
Fix led torchscript
Author: LysandreJik
Pull request: yes

LED can't run on `torchscript`:

```
Failure
Traceback (most recent call last):
  File "/home/xxx/transformers/tests/test_modeling_common.py", line 538, in _create_and_check_torchscript
    traced_model = torch.jit.trace(
  File "/home/xxx/transformers/.env/lib/python3.8/site-packages/torch/jit/_trace.py", line 735, in trace
    return trace_module(
  File "/home/xxx/transformers/.env/lib/python3.8/site-packages/torch/jit/_trace.py", line 952, in trace_module
    module._c._create_method_from_trace(
RuntimeError: 0INTERNAL ASSERT FAILED at "/pytorch/torch/csrc/jit/ir/alias_analysis.cpp":532, please report a bug to PyTorch. We don't have an op for aten::constant_pad_nd but it isn't a special case.  Argument types: Tensor, int[], bool, 

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/xxx/transformers/tests/test_modeling_led.py", line 284, in test_torchscript
    self._create_and_check_torchscript(config, inputs_dict)
  File "/home/xxx/transformers/tests/test_modeling_common.py", line 545, in _create_and_check_torchscript
    self.fail("Couldn't trace module.")
AssertionError: Couldn't trace module.
```

Comments:
[doc] parallelism - when to use which mode
Author: stas00
Pull request: no
Labels: name: Documentation

name: Performance

# 🚀 Feature request

Was asked to expand https://huggingface.co/transformers/master/parallelism.html to include recommendations on which mode to use when.

Comments:
@BramVanroy, please have a look if this addresses your question and I will add it to the doc. It of course assumes that https://huggingface.co/transformers/master/parallelism.html has been read (hence the abbreviations).

If more information is needed please don't hesitate to say what you feel is missing and how things can be improved. Thank you.

I just wasn't sure about single node / multi-gpu as I haven't played much with PP/TP on a single node.

------------
## Which Strategy To Use When

Here is a very rough outlook at which parallelism strategy to use when. The first on the list is typically faster.

**⇨ Single GPU**

* Model fits onto a single GPU:

    1. Normal use

* Model doesn't fit onto a single GPU:

    1. ZeRO + Offload CPU and optionally NVMe


**⇨ Single Node / Multi-GPU**

* Model fits onto a single GPU:

    1. DDP - Distributed DP
    2. ZeRO - may or may not be faster depending on the situation and configuration used

* Model doesn't fit onto a single GPU:

    1. ZeRO
    2. TP
    3. PP

    (not sure which one will be faster here - haven't done enough experiments)

**⇨ Multi-Node / Multi-GPU**

* When you have fast inter-node connectivity:

    1. ZeRO - as it requires close to no modifications to the model
    2. PP+TP+DP - less communications, but requires massive changes to the model

* when you have slow inter-node connectivity:

    1. DP+PP+TP+ZeRO


This is already very useful for most people I think! I personally haven't even tried anything else but regular training and single node-muliti gpu DDP, but I can see how this small overview helps users. It makes it easier for them to "choose what to do".

Thanks!
Use MSELoss with single class label in (M)BartForSequenceClassification
Author: calpt
Pull request: yes

# What does this PR do?

Similar to `BertForSequenceClassification`, `(M)BartForSequenceClassification` now uses a regression loss in case `num_labels` equals 1 (as already documented for both model classes). E.g. required when running the GLUE script for STS-B with these model classes.

## Before submitting
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?

## Who can review?

@patrickvonplaten @patil-suraj 

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Comments:
Fix minor docstring typos.
Author: qqaatw
Pull request: yes

# What does this PR do?

Fix minor docstring typos in #12664

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@sgugger 



Comments:
Translate README.md to Traditional Chinese
Author: qqaatw
Pull request: yes

# What does this PR do?

1. Add README_zh-hant.md and links to direct users to each README.
2. Some of terms in the file can be found at [National Academy for Educational Research](https://terms.naer.edu.tw/), an official website providing bilingual translations between English and Traditional Chinese.

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

cc @JetRunner 


Comments:
I'll ask my friends who are also native in Traditional Chinese, to help double check the terms in the files. So, we can ensure the accuracy of the translation.

@JetRunner We have had the files checked. It is ready to merge if there is no other mistake.

Cool! I'll give it a look then we are ready to merge.
Not able to load the custom model after training in Hugging Face
Author: vivekvkashyap
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:
- Platform: colab

### Who can help

 @patil-suraj @patrickvonplaten 

Models:
Used FlaxGPT2Module as base class and built FlaxGPT2ForMutlipleChoice.After training is done when loading the saved weights in hugging face hub there seems to be an error.
Model hub:

-Hugging Face Hub:https://huggingface.co/Vivek/gpt2-common-sense-reasoning

Error messgae:  unpack(b) received extra data.

## Information

Model I am using (Bert, XLNet ...):FlaxGPT2ForMultipleChoice(custom model)

The problem arises when using: 
1.when i try to load the weights of   saved weights in hugging face hub there seems to be an error.

The tasks I am working on is:
* Dataset: COSMOS QA

Expected Behaviour :To able to load the weights and the configuration without any error

Colab Notebook:https://colab.research.google.com/drive/1C-M1GLMk7jiomXIpngbLZvCNvEJMfJ1K?usp=sharing

## To reproduce
``` python
new_model=FlaxGPT2ForMultipleChoice.from_pretrained('/content/gpt2-common-sense-reasoning',
                 input_shape=(1,4,1), config=config)```



Comments:
Add a custom timeout for log replica test
Author: LysandreJik
Pull request: yes

Add a custom timeout for log replica test. Let's keep these outliers to a minimum.

Comments:
Hmm, it's really slow - clocked `1m16.735s` on my machine. 

Let me see first if it can be made faster.

It's like 4 tests in one - so it adds up - I guess I could just split it in several sub-tests.

What do you guys prefer here? We can also make it @slow - which will shave off ~80sec - it doesn't need to run all the time at all.

No strong opinion on my side, do what you think is best!

oh, but these are multi-gpu tests so they are @slow already as they only run on our machine only

@LysandreJik, does this impact the push workflow? or just scheduled one?

If so I'd also `@slow` all the fairscale/apex tests, as these definitely don't need to run often at all.

Here we go: https://github.com/huggingface/transformers/pull/12710 - reworked 1 to 4 subtests, shouldn't run longer than the timeout now.

merged the alternative solution, closing this then

The multi GPU tests are run every time there is a commit on `master`, so it's not only slow tests. We have fast and slow GPU & multi-GPU tests.

Thanks a lot for splitting that test, way better solution.

ok, so should we put fairscale and apex tests to @slow then? These are hardly ever used by anyone, so would be a waste to spend $$ and time on those.

We can, but it's not a super high priority, they seem to run quickly enough
add intel-tensorflow-avx512 to the candidates
Author: zzhou612
Pull request: yes

# What does this PR do?

`intel-tensorflow-avx512` PyPI package is not in the candidates list of transformers.

This PR adds support for detecting `intel-tensorflow-avx512`.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. I think @sgugger could be the right person to review this PR since he has reviewed a similar PR before.

Comments:
Change create_model_card to use best eval_results when args.load_best_model_at_end==True
Author: nadahlberg
Pull request: no

# 🚀 Feature request

When I use Trainer.push_to_hub, the model card that gets generated uses my most recent checkpoint's eval_results.  However, if I include load_best_model_at_end=True in my TrainingArguments, then the model that is being pushed can often be from an earlier checkpoint (right?).  The suggestion is to change create_model_card (and parse_log_history) to use the best loss when appropriate.

## Motivation

So that the reported evaluation results on the model card match the correct checkpoint of the model pushed to the hub.

## Contribution

Happy to submit a PR, with some guidance on where to make the changes.  Would it make sense to pass the trainer as an optional argument to parse_log_history and then use the training args load_best_model_at_end, metric_for_best_model, and greater_is_better to return the best loss as the eval_results if appropriate?


Comments:
cc @sgugger 

I don't think there is an easy way to get that loss and metrics. However, note that if you run a last `trainer.evaluate()` (as is done in all the example scripts) the loss and metrics reported will be the ones from the final model, so I would suggest doing this.

@sgugger that makes sense.  Thanks!
Replace specific tokenizer in log message by AutoTokenizer
Author: SaulLu
Pull request: yes

# What does this PR do?

As mentioned by @sgugger in PR #12619, I propose with this PR to harmonize the messages in the logs to encourage users to use `AutoTokenizer`.

I've checked that all these tokenizers appear in `TOKENIZER_MAPPING` int the `tokenization_auto.py` file.
<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. @sgugger since this PR is based on one of your comments, I think you would be interested. 

@europeanplaice, I'm just tagging you for your information (I didn't include the change you proposed in your PR).

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Comments:
Thank you for accepting my proposal!
What should be the label of sub-word units in Token Classification with Bert
Author: ereday
Pull request: no

Hi, 

I'm trying to use BERT for a token-level tagging problem such as NER in German.

This is what I've done so far for input preparation:

```
from pytorch_pretrained_bert.tokenization import BertTokenizer, WordpieceTokenizer

tokenizer = BertTokenizer.from_pretrained("bert-base-multilingual-cased", do_lower_case=False)

sentences=  ["Bis 2013 steigen die Mittel aus dem EU-Budget auf rund 120 Millionen Euro ."]
labels = [["O","O","O","O","O","O","O","B-ORGpart","O","O","O","O","B-OTH","O"]]
tokens = tokenizer.tokenize(sentences[0])
```

When I check the tokens I see that there are now 18 tokens instead of 14 (as expected) because of the sub-word units. 
```
>>> tokens
['Bis', '2013', 'st', '##eig', '##en', 'die', 'Mittel', 'aus', 'dem', 'EU', '##-', '##B', '##ud', '##get', 'auf', 'rund', '120', 'Millionen', 'Euro', '.']
```
My question is that how should I modify the labels array. Should I label each sub-word unit with the label of the original word or should I do something else ? As second question, which one of the examples in the resository can be used as an example code for this purpose ? `run_classifier.py` ? `run_squad.py `? 

**UPDATE**

OK, according to the paper it should be handled as follows (From Section 4.3 of BERT paper):

> To make this compatible with WordPiece tokenization, we feed each CoNLL-tokenized
> input word into our WordPiece tokenizer and use the hidden state corresponding to the first
> sub-token as input to the classifier.  Where no prediction is made for X. Since
> the WordPiece tokenization boundaries are a known part of the input, this is done for both
> training and test.

Then, for the above example , the correct input output pair is :
```
['Bis', '2013', 'st', '##eig', '##en', 'die', 'Mittel', 'aus', 'dem', 'EU', '##-', '##B', '##ud', '##get', 'auf', 'rund', '120', 'Millionen', 'Euro', '.']
['O', 'O', 'O', 'X', 'X', 'O', 'O', 'O', 'O', 'B-ORGpart', 'X', 'X', 'X', 'X', 'O', 'O', 'O', 'O', 'B-OTH', 'O']
```

Then my question is evolved to " How the sub-tokens  could be masked during training & testing ?"

Comments:
I have a similar problem. I labeled the tokens as "X" and then got an error relating to NUM_LABELS. BERT appears to have thought the X was a third label, and I only specified there to be two labels.

You do not need to introduce an additional tag. This is explained here:

https://github.com/huggingface/pytorch-pretrained-BERT/issues/64#issuecomment-443703063

Yes, I've left #64 open to discuss all these questions. Feel free to read the discussion there and ask questions if needed. Closing this issue.

@ereday AFAIK 
To answer your question "How the sub-tokens could be masked during training & testing"
There is no need of masking. The sub-word token_ids (except for the first) are not fed to the BERT model.
Please tell me if i am wrong.
Add early stopping args to TrainingArguments
Author: albertvillanova
Pull request: yes

# What does this PR do?

While working in the collaborative training project, I added early stopping args to `TrainingArguments`.

Feel free to close this PR if you consider it is not pertinent.

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@sgugger 

Comments:
I reused the script:
- https://github.com/huggingface/transformers/blob/master/examples/pytorch/token-classification/run_ner.py

but allowing early stopping: [`EarlyStoppingCallback`](https://github.com/huggingface/transformers/blob/master/src/transformers/trainer_callback.py#L505)

Therefore, I had to add those args to `TrainingArguments` to allow end users of my script pass those parameters to EarlyStoppingCallback.

I thought that it might be useful for end users to have those args in TrainingArguments so that they can use early stopping in their trainer the same way I did for my script.

We don't have any `TrainingArguments` that are not used anywhere. Users are already complaining this class has too many, so if we had some they have to do something.
If they go with an update in the example scripts, then all the example scripts should be updated in the PR :-) 

I could also add early stopping to some of the example scripts... 😅 I may do it this weekend though...

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Question: Masked Loss for LukeForEntitySpanClassification
Author: zhenbangt
Pull request: no

In transformers.LukeForEntitySpanClassification, the loss is calculated from labels of shape `(batch_size, entity_length)` and in range `[0, ..., config.num_labels - 1]`. I did not see in the source code you masked out the loss for padded tokens, nor did you make a special label for padded tokens. So how do you deal with the loss of padded sequences?

Many thanks!

Comments:
Normally, in PyTorch, you have to set labels for padding tokens equal to -100, as -100 is the default ignore index that loss functions use.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.