noamaanMulla-03/transformers-issues huggingface dataset:
TypeError: add_() takes 1 positional argument but 2 were given
Author: mqliu7
Pull request: no

# 🐛 Bug

## Information

I was trying to reproduce the GLUE fine-tuning example (https://huggingface.co/transformers/examples.html#fine-tuning-example) when I ran into this error:

```
  File "~/anaconda3/lib/python3.7/site-packages/transformers/optimization.py", line 155, in step
    exp_avg.mul_(beta1).add_(grad, 1.0 - beta1)
TypeError: add_() takes 1 positional argument but 2 were given
```

The problem arises when using:
* [x] the official example scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: (give the name)


## To reproduce

Steps to reproduce the behavior:
Follow the official guide example here:  https://huggingface.co/transformers/examples.html#fine-tuning-example

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 2.9.1
- Platform: Linux 5.4.0-29-generic x86_64 Ubuntu 20.04 LTS
- Python version: 3.7.6
- PyTorch version (GPU?): 1.5.0 (Yes)
- Tensorflow version (GPU?): 
- Using GPU in script?: Yes, 1
- Using distributed or parallel set-up in script?: 


Comments:
This was fixed on master on Friday, can you try pulling from master again?
[T5 Conf] rename docstring to acuatly argument names
Author: patrickvonplaten
Pull request: yes

As mentioned in issue: #4139, the docstring names of the T5 Config are confusing since those names cannot be used to set the arguments. 

This PR renames the arguments in the docstring and adds an explanation that those arguments can also be accessed via their properties. 

To not break backward compatibility, renaming the docstring is better than renaming the actual variables IMO. 

Comments:
feat(wandb): display logger
Author: borisdayma
Pull request: yes

Logger info when `wandb` not installed was set to `info` which does not display by default.
It has been changed to `warning`.

Comments:
It don't think it should really be a warning if you don't use wandb.

The root issue here is probably discoverability of wandb for users who don't know it? Then it would probably be better solved in documentation.

We will start some documentation on `Trainer`/`TFTrainer` in the coming weeks (cc @LysandreJik) we'll mention wandb there. (if you want to help with this let us know)

Makes sense, let me know when it's started and I can help writing the section related to wandb.
[Communtiy notebooks] Fine-tuning / Training 
Author: patrickvonplaten
Pull request: yes

Proposal of how notebooks for training could be added.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4442?src=pr&el=h1) Report
> Merging [#4442](https://codecov.io/gh/huggingface/transformers/pull/4442?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/d9ece8233d584cdc2eeae5165dd3329328fae328&el=desc) will **increase** coverage by `0.01%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4442/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4442?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #4442      +/-   ##
==========================================
+ Coverage   78.14%   78.16%   +0.01%     
==========================================
  Files         120      120              
  Lines       20087    20087              
==========================================
+ Hits        15697    15701       +4     
+ Misses       4390     4386       -4     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4442?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4442/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `90.51% <0.00%> (+0.12%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4442/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `92.93% <0.00%> (+0.32%)` | :arrow_up: |
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4442/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `73.85% <0.00%> (+0.41%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4442?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4442?src=pr&el=footer). Last update [d9ece82...8ae2c5c](https://codecov.io/gh/huggingface/transformers/pull/4442?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Actually closing this, I think community notebooks should only be added in a single place.
[T5 fp16] Fix fp16 in T5
Author: patrickvonplaten
Pull request: yes

This PR fixes the issue: #4287. 

- A test for T5 is added.
- The function self.invert_attention_mask included a if statement now so that no errors will occur when using the function in `fp16` mode.

Comments:
> Bart doesn't use this method yet, but LGTM!

Yeah, I noticed that as well - it's Bert that is using it.
[Community notebooks] General notebooks
Author: patrickvonplaten
Pull request: yes

A proposal how we could link community notebooks.

I'm using the awesome notebook of @patil-suraj (`nlp` + `Trainer` + `transformers` :-)) as an example of how community models can be added. 

@patil-suraj - could you maybe review the PR and see whether it's ok for you?

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4441?src=pr&el=h1) Report
> Merging [#4441](https://codecov.io/gh/huggingface/transformers/pull/4441?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/590adb130be8e99eb638bb22136dda537b2da71d&el=desc) will **increase** coverage by `0.00%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4441/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4441?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #4441   +/-   ##
=======================================
  Coverage   78.14%   78.15%           
=======================================
  Files         120      120           
  Lines       20087    20087           
=======================================
+ Hits        15697    15698    +1     
+ Misses       4390     4389    -1     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4441?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4441/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `92.77% <0.00%> (+0.16%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4441?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4441?src=pr&el=footer). Last update [590adb1...df8c3a1](https://codecov.io/gh/huggingface/transformers/pull/4441?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


@patrickvonplaten This sounds good! Having a separate table for community models makes sense.
run_ner.py example
Author: jmamou
Pull request: no
Labels: name: Ex: Named Entity Recognition

# ❓ Questions & Help

## Details
It seem that examples/run_ner.py has been removed from github repo. Is there any issue with this code?
Thanks

Comments:
Are you looking for this? https://github.com/huggingface/transformers/blob/master/examples/ner/run_ner.py

On a (somewhat) related note: the code might still be there but on huggingface's webpage the following link doesn't point to anything: https://huggingface.co/transformers/examples.html#named-entity-recognition (there is no more an example of how to call the script)

the example has been moved to https://github.com/huggingface/transformers/tree/master/examples/ner


Thx Jonathan

I found it here: https://github.com/huggingface/transformers/blob/master/examples/token-classification/run_ner.py

The NER has been renamed to Token Classification.
Update README.md (model_card)
Author: sy-wada
Pull request: yes
Labels: name: model card

- add a citation.
- modify the table of the BLUE benchmark.

The table of the first version was not displayed correctly on https://huggingface.co/seiya/oubiobert-base-uncased.
Could you please confirm that this fix will allow you to display it correctly?

Comments:
Yes, looks good now
Added model cards for Romanian BERT models
Author: dumitrescustefan
Pull request: yes
Labels: name: model card

Added model card for ``dumitrescustefan/bert-base-romanian-cased-v1`` and ``dumitrescustefan/bert-base-romanian-uncased-v1``

Comments:
Awesome, thanks for sharing

https://huggingface.co/dumitrescustefan/bert-base-romanian-cased-v1

I've added a filter for 🇷🇴 here: https://huggingface.co/models?filter=romanian
Reformer training error
Author: Brock007
Pull request: no

# 🐛 Bug

When training a Reformer model from scratch, I got the following error:

**TypeError: forward() got an unexpected keyword argument 'masked_lm_labels'**

## Information

I am trying to train a Reformer model from scratch on English documents. My data is one document per line:

The problem arises when using:
* [ x ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ x ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Split my training documents on '.' to create a corpus of sentences. Use Google SentencePiece script to train a tokenization model.
2. Use run_language_modeling.py with --mlm --tokenizer_name=path/to/pretrained_SP_tokenizer to train the model.
3. My config.json looks like this:

```
{
  "architectures": [
    "ReformerModelWithLMHead"
  ],
  "model_type": "reformer",
  "vocab_size": 32000
}
```

File "/Users/a9dvzzz/.virtualenvs/cf-mlc/lib/python3.7/site-packages/transformers/trainer.py", line 506, in _training_step
    outputs = model(**inputs)
  File "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
TypeError: forward() got an unexpected keyword argument 'masked_lm_labels'

## Expected behavior

The training process produces a saved Reformer model.

## Environment info
- `transformers` version: 2.9.1
- Platform: Darwin-19.4.0-x86_64-i386-64bit
- Python version: 3.7.4
- PyTorch version (GPU?): 1.3.0 (False)
- Tensorflow version (GPU?): 2.0.0 (False)
- Using GPU in script?: no, but it doesn't seem to matter, both failed.
- Using distributed or parallel set-up in script?: no



Comments:
Reformer does not support `mlm` training at the moment. Please make sure you use `lm` training :-) 

@patrickvonplaten thanks! Is there a plan to support mlm in the near future? 

I assume I can just remove the mlm flag to do lm training, right? How can I tell the script to pad the input sequences to a certain length as reformer requires the sequence length to be a multiple of least common multiple chunk_length? Thanks!

Yes, there are plans to add a `MaskedLM` version for Reformer. I will release a notebook this week (probs on Friday) on how to train the Reformer :-)
AttributeError: 'LambdaLR' object has no attribute 'get_last_lr'
Author: wasiahmad
Pull request: no

I am trying to run the summarization example using BART and getting the following error.

```
tqdm_dict = {"loss": "{:.3f}".format(avg_loss), "lr": self.lr_scheduler.get_last_lr()[-1]}
AttributeError: 'LambdaLR' object has no attribute 'get_last_lr'
```

The error occurred at this [line](https://github.com/huggingface/transformers/blob/master/examples/transformer_base.py#L101). I am using PyTorch 1.3.


Comments:
I encountered the same problem, have you solved it？

Upgrading to PyTorch 1.4 would solve that issue :)

Current `master` branch recommends that:

https://github.com/huggingface/transformers/blob/4e817ff41885063e08bb3bcd63e5adfd835b9911/setup.py#L70

Thank you so much, I have solved it.

`get_last_lr` is introduced in pytorch 1.4.0 . Maybe you need to upgrade your pytorch.

If you really are stuck with PyTorch <= 1.3 please feel free to open a PR to fix this with backward compatibility

I upgraded pytorch to v1.5.0, but run_glue.py with Bert model failed. So, you may have to set it to 1.4.0
Trainer is missing sampler.set_epoch for distributed mode
Author: shaoyent
Pull request: no

# 🐛 Bug

## Information

`train_dataloader.sampler.set_epoch(epoch)` is missing before the start of each epoch in [trainer.py](https://github.com/huggingface/transformers/blob/3e0f06210646a440509efa718b30d18322d6a830/src/transformers/trainer.py#L406).

According to [here](https://pytorch.org/docs/stable/data.html#torch.utils.data.distributed.DistributedSampler) :

> In distributed mode, calling the `set_epoch` method is needed to make shuffling work; each process will use the same random seed otherwise.


Comments:
Good spot! I think @julien-c's latest PR (https://github.com/huggingface/transformers/pull/4243) for distributed eval will also take care of this!
FileNotFoundError when running distributed Trainer
Author: shaoyent
Pull request: no

# 🐛 Bug

## Information

I'm running the language modeling example in distributed mode and am getting FileNotFoundError in the Trainer.

The error :
```
Traceback (most recent call last):
  File "run_language_modeling.py", line 292, in <module>
    main()
  File "run_language_modeling.py", line 257, in main
    trainer.train(model_path=model_path)
  File "/home/shaoyent/anaconda3/envs/bert/lib/python3.7/site-packages/transformers/trainer.py", line 451, in train
    torch.save(optimizer.state_dict(), os.path.join(output_dir, "optimizer.pt"))
  File "/home/shaoyent/anaconda3/envs/bert/lib/python3.7/site-packages/torch/serialization.py", line 327, in save
    with _open_file_like(f, 'wb') as opened_file:
  File "/home/shaoyent/anaconda3/envs/bert/lib/python3.7/site-packages/torch/serialization.py", line 212, in _open_file_like
    return _open_file(name_or_buffer, mode)
  File "/home/shaoyent/anaconda3/envs/bert/lib/python3.7/site-packages/torch/serialization.py", line 193, in __init__
    super(_open_file, self).__init__(open(name, mode))
FileNotFoundError: [Errno 2] No such file or directory: '/home/shaoyent/transformers/examples/language-modeling/output/checkpoint-500/optimizer.pt'
```

The error seems to be the Trainer saving checkpoints in each [local master](https://github.com/huggingface/transformers/blob/b290c32e1617fa74f44ccd8b83365fe764437be9/src/transformers/trainer.py#L413) while the directory is created by [world master](https://github.com/huggingface/transformers/blob/b290c32e1617fa74f44ccd8b83365fe764437be9/src/transformers/trainer.py#L448) causing a race condition. 

I think the suggested method is to handle save checkpoints in world master, since weights should be synchronized after backprop. 


Comments:
Yes, I think you're right. Relaunching training from a checkpoint would however require the user to copy the weights to other machines, right?

> Yes, I think you're right. Relaunching training from a checkpoint would however require the user to copy the weights to other machines, right?

Yes, loading from checkpoint can be done on all nodes.


So I think we should save checkpoints and final models on each **local master**.

Do you agree?

Would there be a difference in checkpoints between each local master ?
If they're identical I'd think it be best to reduce write overhead and save only once (i.e world master)

No difference, but the issue is that you won't be able to restart training from a checkpoint (unless you copy it around) if we only save to world master

Ah, I see, thanks for the clarification. 
I'm assuming a shared file system so it's all saving and loading from the same file. 
I guess for other cluster configurations that makes sense, but care would have to be taken to avoid write/mkdir conflicts from multiple processes.

I see. Thinking about it more, yes, I think it like it if we only save from world_master. (even if it adds one step for some users). Thanks for your insight.
Distributed eval: SequentialDistributedSampler + gather all results
Author: julien-c
Pull request: yes

As discussed in #3944

In our [**`Trainer`**](https://github.com/huggingface/transformers/blob/master/src/transformers/trainer.py), in torch.distributed mode, eval/predict are currently running on a single node. This PR attempts to fix this.

This relies on:
- a custom `DistributedSampler` named SequentialDistributedSampler that makes it easier to collate inference results at the end of the loop
- some tricky code to gather numpy arrays back to a single node (or to all of them). 

I might be missing something simpler here. For instance in `pytorch/xla` there seems to be a [built-in function to do this](https://github.com/huggingface/transformers/blob/7b75aa9fa55bee577e2c7403301ed31103125a35/src/transformers/trainer.py#L656-L659). 

Please let me know if that's the case 😁

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4243?src=pr&el=h1) Report
> Merging [#4243](https://codecov.io/gh/huggingface/transformers/pull/4243?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/2d05480174c736399c212b3d60b8774fbbf2b6c2&el=desc) will **decrease** coverage by `0.13%`.
> The diff coverage is `32.05%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4243/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4243?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #4243      +/-   ##
==========================================
- Coverage   78.15%   78.02%   -0.14%     
==========================================
  Files         120      120              
  Lines       20053    20105      +52     
==========================================
+ Hits        15673    15686      +13     
- Misses       4380     4419      +39     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4243?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/trainer.py](https://codecov.io/gh/huggingface/transformers/pull/4243/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90cmFpbmVyLnB5) | `39.67% <32.05%> (-1.78%)` | :arrow_down: |
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4243/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `90.48% <0.00%> (-0.13%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4243?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4243?src=pr&el=footer). Last update [2d05480...3e32aea](https://codecov.io/gh/huggingface/transformers/pull/4243?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Map optimizer to correct device after loading from checkpoint.
Author: shaoyent
Pull request: yes

Loading from `optimizer.pt` causes `optimizer` to be mapped to the same device as the saved `optimizer.pt`. In most cases it's `cuda:0`(saved by local master), which puts all optimizers on 
gpu0, causing OOM more easily in multi-gpu training.

Might fix issues like [#3730](https://github.com/huggingface/transformers/issues/3730). 

Comments:
Thank you!
Which model should I use for machine translation?
Author: maurus56
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

I’m interested in training a model for translating articles from Spanish to English. There is too little information (Tutorials) about MT, should I use BERT, XLM or any other one? Also could you explain how to train the proposed model feeding the data, and output the predicted translation.
 And is there a way to use XLNet so when translating chapters of a book it can remember the context of the previous ones and better translate?

There is even a model by Microsoft ([MASS](https://github.com/microsoft/MASS)) that looks simple to use, would you recommend it?


Comments:
Hi I recommend using XLM from Facebook for MT currently: https://github.com/facebookresearch/XLM
We may add some models for MT in the mid-term though.

[MASS](https://arxiv.org/pdf/1905.02450.pdf) reports higher BLEU-scores than [XLM](https://arxiv.org/abs/1901.07291) which is good in pretraining an encoder, but lacks in the training description of the decoder. So we could try to extend the XML-R #1769 encoder with MASS.

In which languages and domains are you interested?

I'm looking for translation mainly of Spanish and Chinese to English, mainly books and articles so maintaining an overall consistency of the terms and words is crucial.

Just a single way translation is enough and also it should be possible to further train the model on the already translated works.

Please  help me on how to use xmlr for summarization?  Also if there is any example based on xmlr.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.