noamaanMulla-03/transformers-issues huggingface dataset:
TFTrainingArguments
Author: tangzhy
Pull request: no

## Environment info
- `transformers` version: 4.0.1
- Platform: linux
- Python version: 3.7
- PyTorch version (GPU?): 
- Tensorflow version (GPU?): 2.3.1
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: yes

@sgugger @jplu @stefan-it 

## Information

Model I am using (Bert, XLNet ...): Bert

The problem arises when using:
 [ ] the official example scripts: (give details below)
 [x] my own modified scripts: (give details below)

The tasks I am working on is:
 [ ] an official GLUE/SQUaD task: (give the name)
 [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. specifiy training args.
2. run trainer
3. raise Exception where `evaluation_strategy` in training_args becomes `evaluate_strategy`

```python
training_args = TFTrainingArguments(
    output_dir="/root/Data/marco-passage-ranking/results",
    overwrite_output_dir=True, 
    do_train=True, 
    do_eval=True,
    do_predict=False, 
    evaluation_strategy="no",
    eval_steps=1000,
    
    per_device_train_batch_size=8,  # batch size per device during training
    per_device_eval_batch_size=8,   # batch size for evaluation
    
    learning_rate=1e-6, 
    
    max_steps=400000,
    warmup_steps=40000,   
    
    logging_dir="./tmp/log", 
    logging_steps=1000, 
    save_steps=1000,
    
    fp16=False, 
    
#     eval_steps=1000, 
    xla =False
)

trainer = TFTrainer(
    model=model,                        
    args=training_args,             
    train_dataset=train_ds.take(100000),
    eval_dataset=dev_ds.take(10000), 
    compute_metrics=compute_metrics,
)


trainer.train()

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-19-25eb465360cc> in <module>
      7 )
      8 
----> 9 trainer.train()

~/Softwares/anaconda3/envs/tf2.0/lib/python3.7/site-packages/transformers/trainer_tf.py in train(self)
    562                     if (
    563                         self.args.eval_steps > 0
--> 564                         and self.args.evaluate_strategy == EvaluationStrategy.STEPS
    565                         and self.global_step % self.args.eval_steps == 0
    566                     ):

AttributeError: 'TFTrainingArguments' object has no attribute 'evaluate_strategy'
```


I think this might be a bug where the inconsistency of eval_strategy name raises Exception. Any advice?

Comments:
Oh this is a typo, do you want to open a PR to fix it?
BART cannot accept -100 as ignored label
Author: Huanghongru
Pull request: no

## Environment info

- `transformers` version: 4.0.1
- Platform: Linux
- Python version: 3.7.9
- PyTorch version (GPU?): 1.6.0
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help

 Bart: @patrickvonplaten

## Information

I'm using ``BartForConditionalGeneration`` to do some natural language generation tasks. By the [doc](https://huggingface.co/transformers/model_doc/bart.html#bartforconditionalgeneration) I should be able to set -100 for some tokens to ignore. However, it would raise an out of index error.

## To reproduce

```python
from transformers import BartForConditionalGeneration, AutoTokenizer

b = BartForConditionalGeneration.from_pretrained("facebook/bart-base")
t = AutoTokenizer.from_pretrained("facebook/bart-base")

s1 = "hello hello hello hello world"
inputs = t(s1, return_tensors="pt")
label = inputs["input_ids"].clone()
label[0, 2:3] = -100

outputs = b(**inputs, labels=label)
```

Then it raise the following error:

> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File "/home/hongru/.conda/envs/commonsense/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
>     result = self.forward(*input, **kwargs)
>   File "/home/hongru/.conda/envs/commonsense/lib/python3.7/site-packages/transformers/models/bart/modeling_bart.py", line 1032, in forward
>     return_dict=return_dict,
>   File "/home/hongru/.conda/envs/commonsense/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
>     result = self.forward(*input, **kwargs)
>   File "/home/hongru/.conda/envs/commonsense/lib/python3.7/site-packages/transformers/models/bart/modeling_bart.py", line 915, in forward
>     return_dict=return_dict,
>   File "/home/hongru/.conda/envs/commonsense/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
>     result = self.forward(*input, **kwargs)
>   File "/home/hongru/.conda/envs/commonsense/lib/python3.7/site-packages/transformers/models/bart/modeling_bart.py", line 564, in forward
>     x = self.embed_tokens(input_ids) * self.embed_scale
>   File "/home/hongru/.conda/envs/commonsense/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
>     result = self.forward(*input, **kwargs)
>   File "/home/hongru/.conda/envs/commonsense/lib/python3.7/site-packages/torch/nn/modules/sparse.py", line 126, in forward
>     self.norm_type, self.scale_grad_by_freq, self.sparse)
>   File "/home/hongru/.conda/envs/commonsense/lib/python3.7/site-packages/torch/nn/functional.py", line 1814, in embedding
>     return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
> IndexError: index out of range in self

Without giving -100 in the label, it can return the output correctly.

## Expected behavior

Should return the output correctly.

Comments:
Trainer: support iterable datasets for evaluation 
Author: marcelgwerder
Pull request: no

The trainer seems to support passing iterable datasets as the `train_dataset`  (see #5829) but misses to support the same for the `eval_dataset`. I have implemented an iterable dataset for training and now I cannot use the same implementation for evaluation. This doesn't make much sense as evaluation could easily be done using the iterable dataset. 

Currently the evaluation fails with the following exception:
```
ValueError: DataLoader with IterableDataset: expected unspecified sampler option, but got sampler=<torch.utils.data.sampler.SequentialSampler object at 0x7f53900c50d0>
```

I suspect the below method should return None in case of an iterable dataset:

https://github.com/huggingface/transformers/blob/ef2d4cd4457a344b633173c14ca7789f18f75b59/src/transformers/trainer.py#L402-L408

Just like how it is handled for the `train_dataset`:

https://github.com/huggingface/transformers/blob/ef2d4cd4457a344b633173c14ca7789f18f75b59/src/transformers/trainer.py#L380-L400


Comments:
This is more complex than this as the `Trainer` needs to knows in advance the number of elements in your evaluation dataset, so you need to implement a `__len__` method in your evaluation dataset to have `Trainer` work on it.

An iterable dataset might make sense for training since you want to yield "infinite" examples and stop at a certain step, it doesn't really makes sense for evaluation where, by definition, you have a finite number of samples.

I see, but that also means that you'd have to implement two different datasets for the same data, which is a bit annoying. Why does the trainer need to know the length of the eval dataset?

This is for the distributed evaluation to work: we need to initialize the containers for the logits and predictions to the right size and fill them with the data returned by each node.

You iterable dataset must be finite, so you can just wrap it like this before sending it to `Trainer`:
```
class FromIterableDataset:
    def __init__(self, iterable_dataset):
        self.dataset = list(iterable_dataset)

    def __getitem__(self, i):
        return self.dataset[i]

    def __len__(self):
        return len(self.dataset)
```

Yeah I guess that works for most cases. Although It might make sense to catch the case where one implements `__len__` in an iterable dataset, which might be reasonable depending on how the data is stored. Currently you're first told to implement `__len__` and then the code just fails with above exception. It would probably be better if there was a more meaningful exception regarding the use of iterable datasets for evaluation.

Anyways, this seems to be an edge case, thanks for your help!
[TF Bart] Refactor TFBart
Author: patrickvonplaten
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Mirror of #8900 for TFBart.

The same improvements are done for Bart except adding torchscript functionality (as it does not exist in tf bart).

- [x] Keep dims consistent within the model -> no switching around between time x batch_size and batch_size x time. We can just stick to batch_size x time throughout the whole forward pass just like other models do too.
- [x] Clean the Attention layer: Replace dict cache by past_key_values tuple (consistency with other models and stateless which is better IMO). Break up complicated if-else cascade and remove unnecessary parameters.
- [x] Correct error with past_key_values/decoder_input_ids/use_cache
- [x] Add input_embeds to Bart
- [x] (very subjectively) better naming
- [x] Check that all slow tests are passing
- [x] Update docstring and final design change check
- [x] Refactor Bart tests
- [x] should solve https://github.com/huggingface/transformers/issues/9048
- [x] Check no speed regression

Comments:
No speed regression on GPU brutasse in graph mode. PR is ready for review IMO.

> Awesome!! Thanks for taking care of this part!!
> 
> Should we merge #9063 before or after this one?

Let's merge after your PR. I'll take the merge conflicts from you :-)
Also this way I can play around a bit with the new not-existing-cast-bool functionality, yaaay!
Fix Bart Shift
Author: patrickvonplaten
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Previous PR #9134 was still WIP and accidentally merged to quickly -> sorry for the many commits.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @patrickvonplaten
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSMT: @stas00
 -->


Comments:
Not able to train RoBERTa language model from scratch
Author: shubhamk0027
Pull request: no

I tried training RoBERTa language model from scratch using 

<code> !python ./run_mlm.py 
    --model_name_or_path roberta-base 
    --train_file './data_lm.txt' 
    --do_train 
    --line_by_line 
    --num_train_epochs 3
    --output_dir ./roberta
</code>

Due to the limits on the colab storage, I deleted all the checkpoints generated during the training process. So finally my model directory has the following files - 
1. config.json
2. merges.txt
3. pytorch_model.bin
4. special_tokens_map.json
5. tokenizer.config
6. vocab.json

But after loading my trained model using RobertaTokenizer, RobertaForSequenceClassification and fine-tuning it  for my classification task, i am receiving almost same accuracy as by loading and fine-tuning the readily available 'roberta-base'. 

Also, when i try loading it as
`model = RobertaModel.from_pretrained('./roberta)
`

 I get the warning-

<code>
Some weights of RobertaModel were not initialized from the model checkpoint at ./roberta/ and are newly initialized: ['roberta.pooler.dense.weight', 'roberta.pooler.dense.bias']

You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
</code>

So, my question is, Is there something wrong in the training procedure of the Language Model on my dataset and the loading process? Or the checkpoints that i deleted were important?  
 

Comments:
The `run_mlm` script trains a `RobertaForMaskedLM` model, which does not have a pooler layer. That's why you get this warning when using this pretrained model to initialize a `RobertaModel`.

Thanks @sgugger! Now I get where the problem was. 
Moreover, I found some good tutorials on it.  Here are the links for those who need it.
https://zablo.net/blog/post/training-roberta-from-scratch-the-missing-guide-polish-language-model/
https://colab.research.google.com/github/huggingface/blog/blob/master/notebooks/01_how_to_train.ipynb
[Examples] Add automatic dataset splitting in language-modeling examples
Author: TevenLeScao
Pull request: yes

# What does this PR do?

Currently, language-modeling examples support passing a HF-datasets dataset as training data. However, this dataset needs to have a `train` and `validation` split, which is not the case for many language-modeling datasets, which are just unstructured text. The updated scripts automatically partition the `train` split to create a `validation` split if it doesn't exist already, and adds `validation_split_percentage` argument to control the split ratio, set to 5% by default.

Comments:
Ah, the commit from #9127 seems to have snuck its way in there. Should I remove it?

If you can do it easily, that would be best!

> If you can do it easily, that would be best!

I've tried for a bit but I think I just made things worse ! If that's OK I'll leave it there and I'll fix things at merge time.
Add possibility to switch between APEX and AMP in Trainer
Author: sgugger
Pull request: yes

# What does this PR do?

When PyTorch >= 1.6 is installed, Trainer is always using native AMP right now. This PR adds the option to switch between AMP and APEX, which can be useful:
- because of the memory leak in AMP fixed (fixed in 1.7.1 but present in 1.6)
- to benchmark APEX vs. AMP

It also simplifies a little bit the internal of Trainer with those.

Comments:
This PR also removes `_use_ddp_no_sync` since presumably `transformers` no longer supports pytorch < 1.2

There was one nit that you agreed with but didn't integrate - but I'm fine if it remains as merged - just a potential for divergence down the road...

Oh, which one did I miss?

https://github.com/huggingface/transformers/pull/9137#discussion_r543638157


Argh, one sec - I see what happened - that wasn't what I meant - sorry for not being clear. `choices` is crucial here - since you don't validate the user-provided values - this is error-prone. 

I tried to suggest not repeating the options in the help comment - please let's have `choices` back, and duplicate them if you prefer the help to have the explicit repetition - thanks.

Fixed directly on master in [this commit](https://github.com/huggingface/transformers/commit/51adb97cd644a5840d971868d18c1d436fd6ff5d).

That's perfect. Thank you, @sgugger!
Added sampler'set_epoch when use distributed training
Author: graykode
Pull request: yes
Labels: name: wontfix

`run_squad.py` file is independent of `Trainer Class`(https://github.com/huggingface/transformers/issues/4398). Therefore, there is no method related to `set_epoch` in distributed training.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Moved feature generation into getitem to save ram
Author: mariusjohan
Pull request: yes
Labels: name: wontfix

# What does this PR do?
I had this issue that my Colab Notebook would run out of memory when creating the feature in the SQuAD dataset, so I basically just moved it to the getitem section. I also added a function to force create the features immediatly incase somehow for some reason would want the features created in the beginning

Fixes # (issue)


## Before submitting
- I submitted an issue, but never got any feedback.
- I did not update the documentation due to the fact that it doesn't exist
- I did not write tests either since I couldn't get the tests to work out on my machine. However I did test it while training the model. However I could easily write tests if you believe it's necessary. 


## Who can review?
@patrickvonplaten, maybe you can take a look at the pull request?


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Clarify use of TrainingArguments.disable_tqdm in Jupyter Notebooks  
Author: lewtun
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Closes #8831 and adds some minor tweaks / improvements to the `TrainingArguments` classes.


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).


## Who can review?

@sgugger 

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @patrickvonplaten
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSMT: @stas00
 -->


Comments:
Thanks for the suggestions! I've included them so should be good to go :)

Thanks a lot!
adapting trainer.py for multiple optimizers 
Author: rabeehkarimimahabadi
Pull request: no

Hi
I was wondering if there can be an easy way to adapt trainer.py for multiple optimizers, where each optimizer is responsible for updating a part of model. thanks 

Comments:
Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discusss.huggingface.co) instead?

Thanks!