noamaanMulla-03/transformers-issues huggingface dataset:
Clarify definition of seed argument in TrainingArguments
Author: lewtun
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Clarifies the definition of the `seed` argument in `TrainingArguments` to:

* Explain what "initialisation" refers to
* How to ensure reproducibility across runs


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

Link to discussion on the HF forum: https://discuss.huggingface.co/t/fixing-the-random-seed-in-the-trainer-does-not-produce-the-same-results-across-runs/3442?u=lewtun

## Who can review?

@sgugger 


Comments:
> Thanks for flagging the doc was incorrect! You changes are not entirely correct either so I made some suggestions.

Thanks for fixing my tweaks - I like the changes so committed them 😃 

Sorry for missing that last suggestion of yours - should be ready to go now!

Yes, thanks a lot!
TFBart lables consider both pad token and -100
Author: kiyoungkim1
Pull request: yes

For #9770,

1. ```TFBartModels``` use -100 as a masking token for ```decoder_input_ids``` and ```compute_loss``` like other models(```T5```).
2. For legacy, all the  ```padding token``` in ```labels``` are replace by ```-100`` token.

Below examples show the same result for ```labels``` with ```-100 token``` or ```padding token```, where previously ```Nan``` are shown in the latter case(#9770). 

```
import tensorflow as tf
from transformers import BartTokenizer, TFBartForConditionalGeneration

tokenizer = BartTokenizer.from_pretrained("facebook/bart-base")
model = TFBartForConditionalGeneration.from_pretrained("facebook/bart-base")

inputs = tokenizer("My dog is <mask>", return_tensors='tf', truncation=True, max_length=16, padding="max_length")
labels_ids = tokenizer("My dog is cute", return_tensors='tf', truncation=True, max_length=16, padding="max_length").input_ids

## labels padding_token = 1
loss = model(inputs, labels=labels_ids)[0]
print(labels_ids)
print(loss)

## labels padding_token = -100
labels_ids = tf.where(
    labels_ids == 1, tf.fill(tf.shape(labels_ids), tf.constant(-100, dtype='int32')), labels_ids
)

loss = model(inputs, labels=labels_ids)[0]
print(labels_ids)
print(loss)
```

```
tf.Tensor(
[[    0  2387  2335    16 11962     2     1     1     1     1     1     1
      1     1     1     1]], shape=(1, 16), dtype=int32)
tf.Tensor(
[2.2291888e-05 4.8874615e-05 3.7192607e-05 7.9230859e-04 6.1941862e+00
 1.1058818e+00], shape=(6,), dtype=float32)
tf.Tensor(
[[    0  2387  2335    16 11962     2  -100  -100  -100  -100  -100  -100
   -100  -100  -100  -100]], shape=(1, 16), dtype=int32)
tf.Tensor(
[2.2291888e-05 4.8874615e-05 3.7192607e-05 7.9230859e-04 6.1941862e+00
 1.1058818e+00], shape=(6,), dtype=float32)
```

TFBart gives the same result with both -100 and padding token.
However, ```Bart(pytorch) with -100 token in labels```, ```Bart with padding token in labels``` and ```TFBart(tensorflow with -100 or padding token)``` gives three different results. This is noticed but not treated in this PR.

@patrickvonplaten
@jplu 
@patil-suraj 


Comments:
@patrickvonplaten 
I merged upstream to the branch!

You have an error in code quality, could you run `make style` and `make quality` to check it out? Thanks.
ValueError: You have to specify either decoder_inputs or decoder_inputs_embeds
Author: rk0033
Pull request: no

## Environment info
- `transformers` version:4.0.0
- Platform:Google Colab
- Python version:3
- Tensorflow version (GPU?):2.3.0
- Using GPU in script?:No

### Who can help
@patrickvonplaten
@patil-suraj
@jplu

## Information

I refer to the URL below and want to run the fine-tuning on mT5.
https://huggingface.co/transformers/training.html
https://huggingface.co/transformers/model_doc/mt5.html

Model I am using (mT5):

```
from transformers import MT5Model, T5Tokenizer, TFMT5Model
model = TFMT5Model.from_pretrained("google/mt5-base")
tokenizer = T5Tokenizer.from_pretrained("google/mt5-base")
```

```
from transformers import BertTokenizer, glue_convert_examples_to_features
import tensorflow as tf
import tensorflow_datasets as tfds

data = tfds.load('glue/mrpc')
train_dataset = glue_convert_examples_to_features(data['train'], tokenizer, max_length=128, task='mrpc')
train_dataset = train_dataset.shuffle(100).batch(32).repeat(2)
```

```
optimizer = tf.keras.optimizers.Adam(learning_rate=3e-5)
loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
model.compile(optimizer=optimizer, loss=loss)
model.fit(train_dataset, epochs=2, steps_per_epoch=115)
```

the output produced :
```
ValueError                                Traceback (most recent call last)
<ipython-input-9-650f77977ac3> in <module>()
      3 loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
      4 model.compile(optimizer=optimizer, loss=loss)
----> 5 model.fit(train_dataset, epochs=2, steps_per_epoch=115)
      6 # model.fit({"inputs": train_dataset},epochs=2, steps_per_epoch=115)
      7 # model.fit(train_dataset)

10 frames
/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/func_graph.py in wrapper(*args, **kwargs)
    971           except Exception as e:  # pylint:disable=broad-except
    972             if hasattr(e, "ag_error_metadata"):
--> 973               raise e.ag_error_metadata.to_exception(e)
    974             else:
    975               raise

ValueError: in user code:

    /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py:806 train_function  *
        return step_function(self, iterator)
    /usr/local/lib/python3.6/dist-packages/transformers/models/t5/modeling_tf_t5.py:1094 call  *
        decoder_outputs = self.decoder(
    /usr/local/lib/python3.6/dist-packages/transformers/models/t5/modeling_tf_t5.py:642 call  *
        raise ValueError(f"You have to specify either {err_msg_prefix}inputs or {err_msg_prefix}inputs_embeds")

    ValueError: You have to specify either decoder_inputs or decoder_inputs_embeds

```

Comments:
analogs to #8923

this one is probably safe to close - probably your link was satisfactory, @patrickvonplaten
Pre-trained BART performance on XSum lower than expected
Author: morningmoni
Pull request: no
Labels: name: wontfix

Greetings,

I am trying to reproduce BART's results on xsum using 'bart-large-xsum' and modified `examples/summarization/bart/evaluate_cnn.py` (max_length=60, min_length=10, beam=6, lenpen=1)  but got lower ROUGE scores than reported. 

I first obtained comparable results on CNNDM using 'bart-large-cnndm'  and the dataset on s3:


CNNDM | R-1 | R-2 | R-L
-- | -- | -- | --
BART (Lewis et al., 2019) | 44.16 | 21.28 | 40.9
BART (ours) | 44.32 | 21.12 | 41.13

I then obtained the raw xsum dataset from the original authors and saved them to test.source and test.target (cased) as for CNNDM. Then I ran evaluate_cnn.py with the new parameters above. Is there anything that I am missing? Thank you!

XSum | R-1 | R-2 | R-L
-- | -- | -- | --
BART (Lewis et al., 2019) | 45.14 | 22.27 | 37.25
BART (ours) | 44.7 | 21.04 | 35.64












     


Comments:
I'm having the [exact same issue](https://github.com/pytorch/fairseq/issues/1971), with the official BART code on fairseq.

The author is currently looking into it.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


I downloaded data from [here](https://s3.amazonaws.com/datasets.huggingface.co/summarization/xsum.tar.gz) and was able to get 45.37 / 22.30 / 37.19 using facebook/bart-large-xsum model


> I downloaded data from [here](https://s3.amazonaws.com/datasets.huggingface.co/summarization/xsum.tar.gz) and was able to get 45.37 / 22.30 / 37.19 using facebook/bart-large-xsum model

Hi @swethmandava , this dataset seems to have different train/valid/test split from the original dataset. Can you reproduce the scores with the original dataset?
Fit chinese wwm to new datasets
Author: wlhgtc
Pull request: yes

Sorry for my later update. 
I make my code(especially in chinese mlm_wwm) fit the newest code.
Here are the changes:
1. add `chinese_ref` key to avoid miss ref inf.
2. fix the type bug in `data_collator.py`
3. re-add `run_chinese_ref.py` cause it could run with the newest version code (4.2.2).
4. update readme

Comments:
@sgugger  @LysandreJik 
Could you help me review these code ?

> Hi there! Thanks for updating your example. We have now created a `research_projects` project for the examples not directly maintained by the core team, and I think the `run_mlm_wwm` script and the chine_ref file could all go there in a new folder. Would you mind adjusting your PR in that direction?

Sure,  maybe move `run_chinese_ref.py` to  `research_projects` folder and leave `run_mlm_wwm.py` in where it was would be better ? And I don't know which folder is better ?
The two files are independent, we could move it to anywhere.

The `run_mlm_wwm` file is not maintained by us directly and it only works for BERT-models, compared to the other examples, so I think it can all go together there. You can create a new folder named `mlm_wwm` (since it's not just Chinese) for instance and have the specific requirements in the `requirements.txt` file there?

> The `run_mlm_wwm` file is not maintained by us directly and it only works for BERT-models, compared to the other examples, so I think it can all go together there. You can create a new folder named `mlm_wwm` (since it's not just Chinese) for instance and have the specific requirements in the `requirements.txt` file there?

done!

Last thing is to run `make style` to make sure the files are properly formatted, let me know if you have any issue doing this!

> Last thing is to run `make style` to make sure the files are properly formatted, let me know if you have any issue doing this!

yeah, seem my previous PR also failed in format :(
I got error as follow:
```
#!/bin/bash -eo pipefail
black --check examples tests src utils
would reformat /home/circleci/transformers/examples/research_projects/mlm_wwm/run_chinese_ref.py
would reformat /home/circleci/transformers/src/transformers/trainer.py
Oh no! 💥 💔 💥
2 files would be reformatted, 706 files would be left unchanged.

Exited with code exit status 1
```
But I formate my code.
![image](https://user-images.githubusercontent.com/16603773/106291319-713e9a80-6286-11eb-902b-a9240135ac15.png)

Maybe you could help me do this part ?

@sgugger My pleasure. Maybe you could help me fix the formate error :(
My python version `3.9.1` black `20.8b1`, why I got diff result in CI.
run_seq2seq.py doesn't work after enabling sortish sampler
Author: shamanez
Pull request: no

It gives an error saying **AttributeError: 'Dataset' object has no attribute 'make_sortish_sampler'**. Seems like something is wrong with the pipeline or versions. I install both transformers and datasets from the sources. 

[Exact line](https://github.com/huggingface/transformers/blob/master/src/transformers/trainer_seq2seq.py#L46)
```


    self.train_dataset.make_sortish_sampler(
AttributeError: 'Dataset' object has no attribute 'make_sortish_sampler'



```













## Environment info

- `transformers` version: 4.3.0.dev0
- Platform: Linux-5.8.18-050818-generic-x86_64-with-glibc2.29
- Python version: 3.8.5
- PyTorch version (GPU?): 1.7.1+cu110 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help
@LysandreJik  @patrickvonplaten




Comments:
Remove subclass for sortish sampler
Author: sgugger
Pull request: yes

# What does this PR do?

When putting the sortish sampler in the main `Trainer`, I forgot to remove the overide in `Seq2SeqTrainer` which led to an issue (see #9900). This in turn makes the old `finetune_trainer` script fail because its datasets don't have the right entries (the texts are processed during the data collation) so it requires reverting the changes in that script to use back the old `Seq2SeqTrainer`  (which is fine since that script will soon go to legacy).

Fixes #9900

Comments:
Add new model docs
Author: patrickvonplaten
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

This PR adds more information on how to add a model to Transformers docs.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

## UPDATE

The `model_doc/add_new_model.rst` is now finished for a first merge IMO. It would be amazing if @LysandreJik @sgugger you could review the file real quick again - I tried to add all of your suggestions. Also, I added a diagram showing the model design of Transformers - which was not reviewed yet. Note that I did not add a clear design for Tokenizers since it takes a lot of time to do so and I want to iteratively improve this step-by-step explanation. The first model, for which I'd like to mentor someone from the community would also be BigBird which does not need a new tokenizer.

In addition, I would be extremely grateful if @stas00 @abhishekkrthakur @patil-suraj @stefan-it @NielsRogge you have 10 minutes review the `model_doc/add_model.rst` file for possible improvements since you guys just recently added a new model. Your feedback would be especially useful since you might have a much more "unbiased" view what is difficult/easy when adding a model.

Comments:
[seq2seq] fix logger format for non-main process
Author: stas00
Pull request: yes

Currently, in `finetune_trainer.py` non-main process doesn't have any formatting at all, so we end up with:

```
[WARNING|modeling_t5.py:1645] 2021-01-30 20:01:37,246 >> [p0] got MPU
[WARNING|modeling_t5.py:1646] 2021-01-30 20:01:37,246 >> [p0] DP group [0]
[p1] got MPU
[p1] DP group [1]
```

as you can see the 2nd process in DDP misses formatting in logger.

this PR fixes it.

I looked in the take-over version `run_seq2seq.py` if it needed to be fixed too and it doesn't have these function calls, not sure why.  They appear to be needed, unless they get called elsewhere.

@sgugger, @patil-suraj 


Comments:
@LysandreJik knows better for the centralized logging system so I'll defer to him.
ConvBERT: minor fixes for conversion script
Author: stefan-it
Pull request: yes

Hi,

the conversion script for ConvBERT throws the following error message when using it:

```bash
Traceback (most recent call last):
  File "convert_convbert_original_tf1_checkpoint_to_pytorch.py", line 19, in <module>
    from ...utils import logging
ImportError: attempted relative import with no known parent package
```

I fixed that error, as well as using the correct name for the configuration file argument.

Additionally, I just found that the configuration files from the [YituTech](https://huggingface.co/YituTech) organization for ConvBERT from aren't correct, because they use:

```json
"model_type": "conv_bert",
```

instead of:

```json
"model_type": "convbert",
```

(This currently results in a `KeyError: 'conv_bert'` error).

Comments:
I hate this forking/syncing stuff with GitHub 🙈

Preparing a clean PR now...
Does Sortish Sampler work with multiple GPUs in seq2seq?
Author: shamanez
Pull request: no

I am referring to the training script of run_seq2seq.py.  I am exactly referring to  [this line in seq2seq_trainer.py](https://github.com/huggingface/transformers/blob/1420b5ff675ccdc3296c6776b339a08a22d2e941/src/transformers/trainer_seq2seq.py#L48). So when should I enable **distributed** parameters and how should I do it?

Comments:
Currently `SortishSampler` will only work with the `finetune_trainer.py` scripts. It will be supported in `run_seq2seq.py` in the soon. And to answer your question, yes it works with multiple GPUs, and you won't need to enable distributer parameter if the training is launched on multi GPUs using `torch.distributed.launch`, it'll be enabled automatically. 

thanks a lot.
trainer_seq2seq.py Question
Author: Ansper11
Pull request: no

Hi, does trainer_seq2seq.py in transformers/src support multi GPU training? Thank you

Comments:
Hi @caincdiy 

All the example scripts using `Trainer` or it's a subclass use `python -m torch.distributed.launch` to launch multi GPU training.  See https://github.com/huggingface/transformers/tree/master/examples#distributed-training-and-mixed-precision

Also the [forum](https://discuss.huggingface.co/) is the best place to ask such questions :)

Oh sorry. thank you very much for your help
Fix 9918
Author: sgugger
Pull request: yes

# What does this PR do?

This PR addresses the problem shown in #9918 by:
- adding the documentation of the `encode` method to the `PreTrainedTokenizer` and `PreTrainedTokenizerFast` (it is in all their subclasses already)
- adding the "What are input IDs" link where missing in some models docstrings.

In passing, I uncovered a failure of the doc styling script on DPR, so this PR also fixes that.

Fixes #9918

Comments:
Bump numpy
Author: sgugger
Pull request: yes

# What does this PR do?

As pointed out on the [forums](https://discuss.huggingface.co/t/typeerror-full-like-got-an-unexpected-keyword-argument-shape/2981), the method `np.full_like` used in the evaluation of the `Trainer` with the argument `shape=` does not work for all versions of numpy. According to the [numpy documentation]() it was introduced in version 1.17 only, so this PR bumps the setup to that version.

If for some reason we don't want to have a minimum version of numpy, I can try to find another way to do the same thing in `Trainer`.

Comments:
Use compute_loss in prediction_step
Author: sgugger
Pull request: yes

# What does this PR do?

As requested in #9915, this PR uses `compute_loss` in the `prediction_step` method of `Trainer`, so it properly computes losses when the user have customized the way to do that. It does require a new argument to `compute_loss` to return the outputs on top of the loss for the prediction loop, so users that want to use this feature will have to tweak their subclass a little bit, but there is no breaking change.

Comments:
Problem while initializing custom model with added tokens 
Author: yeounyi
Pull request: no

Hi, I'm trying to train my model with new special token 'name', but it keeps throwing size mismatch error. 
I think the problem is that my model has pretrained models inside initialization.

Model Structure
```
class MaskedLM(RobertaPreTrainedModel):
    def __init__(self, config):
        super().__init__(config=config)
        self.roberta = RobertaModel(config)
        self.lm_head = RobertaLMHead(config)
        self.refinement_num = 5
        self.mask_id = 50264
        self.init_weights()

    def forward(...):

```

After resizing my model, embedding size changed from 50265 to 50266. 

```
MaskedLM(
  (roberta): RobertaModel(
    (embeddings): RobertaEmbeddings(
      (word_embeddings): Embedding(50266, 768)
      (position_embeddings): Embedding(514, 768, padding_idx=1)
      (token_type_embeddings): Embedding(1, 768)
      (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
      (dropout): Dropout(p=0.1, inplace=False)

```

But the problem is out_features size in lm_head remains the same. (50265)

```
  (lm_head): RobertaLMHead(
    (dense): Linear(in_features=768, out_features=768, bias=True)
    (layer_norm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
    (decoder): Linear(in_features=768, out_features=50265, bias=True)
  )

```

Is there any way that I can both load the pretrained weights and add one new token? 

-------------

Setting Tokenizer 
```
tokenizer = RobertaTokenizer.from_pretrained('roberta-base')
tokenizer.add_tokens('<name>')
```

Setting Model
```
model = MaskedLM.from_pretrained('roberta-base')
model.resize_token_embeddings(len(tokenizer))
```


Comments:
ConvBERT: minor fixes for conversion script
Author: stefan-it
Pull request: yes

Hi,

the conversion script for ConvBERT throws the following error message when using it:

```bash
Traceback (most recent call last):
  File "convert_convbert_original_tf1_checkpoint_to_pytorch.py", line 19, in <module>
    from ...utils import logging
ImportError: attempted relative import with no known parent package
```

I fixed that error, as well as using the correct name for the configuration file argument.

Additionally, I just found that the configuration files from the [YituTech](https://huggingface.co/YituTech) organization for ConvBERT from aren't correct, because they use:

```json
"model_type": "conv_bert",
```

instead of:

```json
"model_type": "convbert",
```

(This currently results in a `KeyError: 'conv_bert'` error).

Comments:
Pinging @abhishekkrthakur and @sgugger :hugs: 

Weird that relative imports failed. Anyways, thanks for the PR. The model_type in hub has been fixed.