noamaanMulla-03/transformers-issues huggingface dataset:
RAG finetuning - unexpected keyword argument 'early_stop_callback'
Author: ioannist
Pull request: no

## Environment info
transformers version: 3.3.1
Platform: Ubuntu
Python version:3.6.12
PyTorch version (GPU: yes): 1.6.0
Using GPU in script?: 1 gpu
Using distributed or parallel set-up in script?: no

### Who can help
@patrickvonplaten @sgugger
 
## Information
model name: facebook/rag-token-base

The problem arises when using:
* [x ] the official example scripts: (give details below)

The tasks I am working on is:
* [x ] my own task or dataset: (give details below)

## To reproduce
Call finetune ona  rag model
`python examples/rag/finetune.py --data_dir=$(pwd)/examples/rag/ioannis-data --output_dir $(pwd)/examples/rag/ioannis-output --model_name_or_path=facebook/rag-token-base --model_type rag_sequence --fp16 --gpus 1`

```
Traceback (most recent call last):
  File "examples/rag/finetune.py", line 469, in <module>
    main(args)
  File "examples/rag/finetune.py", line 442, in main
    logger=logger,
  File "/home/ioannis/Desktop/transformers/examples/lightning_base.py", line 379, in generic_train
    **train_params,
  File "/home/ioannis/anaconda3/envs/transformers/lib/python3.6/site-packages/pytorch_lightning/trainer/properties.py", line 122, in from_argparse_args
    return argparse_utils.from_argparse_args(cls, args, **kwargs)
  File "/home/ioannis/anaconda3/envs/transformers/lib/python3.6/site-packages/pytorch_lightning/utilities/argparse_utils.py", line 50, in from_argparse_args
    return cls(**trainer_kwargs)
  File "/home/ioannis/anaconda3/envs/transformers/lib/python3.6/site-packages/pytorch_lightning/trainer/connectors/env_vars_connector.py", line 41, in overwrite_by_env_vars
    return fn(self, **kwargs)
TypeError: __init__() got an unexpected keyword argument 'early_stop_callback'
```

I noticed a variable named early_stopping_callback in finetune.py. A typo perhaps?


Comments:
If you don't need early stopping just comment out `early_stop_callback=early_stopping_callback` on line 379 of `/home/ioannis/Desktop/transformers/examples/lightning_base.py`. You should be able to run your script.

I think lightning may have changed their api
You can also just uninstall your pytorch lightning and do `pip install pytorch_lightning==0.9.0` and script should work

Awesome! Installing 0.9.0 worked.

I manually installed pytorch_lightning and gitpython as they were not included in the transformers installation and the rag requirements file.
LXMERT imports
Author: ecekt
Pull request: no

# ❓ Questions & Help
Hello, I was very happy to see that LXMERT is being integrated into this library and I wanted to try it out. I got a KeyError in configuration_auto.py, as in CONFIG_MAPPING there was no 'lxmert'. Then, I re-installed transformers from source. The KeyError went away, but this time I encountered issues in modeling_auto.py as below:

```model = AutoModelWithLMHead.from_pretrained("unc-nlp/lxmert-base-uncased")```

/transformers/src/transformers/modeling_auto.py", line 841, in from_pretrained
    raise ValueError(

ValueError: Unrecognized configuration class <class 'transformers.configuration_lxmert.LxmertConfig'> for this kind of AutoModel: AutoModelWithLMHead.
Model type should be one of T5Config, DistilBertConfig, AlbertConfig, CamembertConfig, XLMRobertaConfig, MarianConfig, BartConfig, LongformerConfig, RobertaConfig, BertConfig, OpenAIGPTConfig, GPT2Config, MobileBertConfig, TransfoXLConfig, XLNetConfig, FlaubertConfig, XLMConfig, CTRLConfig, ElectraConfig, EncoderDecoderConfig, ReformerConfig.
`

In the example [here](https://huggingface.co/unc-nlp/lxmert-base-uncased), AutoModelWithLMHead is imported. However, because of the error above, I also tried LxmertPreTrainedModel, which led to some other error concerning the initialization of the weights. Importing LxmertModel instead seems to work out without errors. Would that be the correct way if I want to extract features from a pretrained model?

I would appreciate any help regarding this issue!

Best regards,
Ece

Comments:
Hey @ecekt, 

Thanks for your issue!
Yes, you are correct ```"unc-nlp/lxmert-base-uncased"``` should only be imported with the `AutoModel` or `LxmertModel` class.

@julien-c - the config looks correct I'm not sure why it states `AutoModelWithLMHeadModel` in the examlpe here: https://huggingface.co/unc-nlp/lxmert-base-uncased

Hi @patrickvonplaten, thank you for the reply! I will explore its use with the LxmertModel import then.

Best.

This is now fixed on https://huggingface.co/unc-nlp/lxmert-base-uncased, thanks for the heads up
[model cards] yaml specs and related questions
Author: stas00
Pull request: no

Is it possible to write a spec for how model cards should declare languages when it's more than one? 

I sorted it out:
```
language: 
- ru
- en
```
could we add this somewhere?

I tried this https://github.com/huggingface/model_card/pull/2 - definitely easier to mimic, but perhaps not ideal.

------------

The array of datasets is unclear too. https://huggingface.co/allenai/wmt16-en-de-dist-12-1
I added:
```
- datasets: http://www.statmt.org/wmt16/ ([test-set](http://matrix.statmt.org/test_sets/newstest2016.tgz?1504722372)) 
```
but it rendered them as one entry - how can I make those into proper dataset links?

I think it'd have helped a lot with having the template demo a few entries and let the user modify those. 

I found a hint doing:

```
grep -r -A2 datasets: model_cards
```

edit: ok, now I know it expects just a label for a dataset that is already in the transformers list., and not the urls to the datasets.

probably a guide could say, pick one more dataset entries from https://huggingface.co/datasets?

--------------

Do we have a tool that validates model_card's yaml section? This can probably be automated, right?
```
make card-check
```
?

-------------------------
Why at https://huggingface.co/allenai/wmt19-de-en-6-6-big
it added a bunch of tags that are wrong and weren't in the yaml spec? 
`lm-head` and `masked-lm` shouldn't be there

--------------------------
Also how can I tell the models site that the demo should be of a translation type and not masking?
somehow it picked up the correct demo type for https://huggingface.co/allenai/wmt16-en-de-dist-12-1
but for https://huggingface.co/facebook/wmt19-en-ru it wants to do fill-mask

----

Related thread: https://discuss.huggingface.co/t/tips-for-debugging-model-cards/814

---

@julien-c 


Comments:
Addressed in https://github.com/huggingface/model_card/pull/3 so closing this one.
bert-base-uncased have weird result on Squad 2.0 
Author: f422661
Pull request: no

I followed the example to fine-tuning BERT on SQuAD2.0:
https://huggingface.co/transformers/examples.html#fine-tuning-bert-on-squad1-0

I run the code as follow:
```
python /content/drive/My\ Drive/squad2/run_squad.py \
  --model_type bert \
  --model_name_or_path bert-base-uncased \
  --do_train \
  --do_eval \
  --do_lower_case \
  --train_file /content/drive/My\ Drive/squad2/train-v2.0.json \
  --predict_file /content/drive/My\ Drive/squad2/dev-v2.0.json \
  --per_gpu_train_batch_size 12 \
  --learning_rate 3e-5 \
  --num_train_epochs 2.0 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --output_dir /content/drive/My\ Drive/squad2_model/
```
However, I got weird results as follow:

```
Results: {'exact': 40.722648024930514, 'f1': 44.3783712849203, 'total': 11873, 'HasAns_exact': 81.5620782726046, 'HasAns_f1': 88.88400847939587, 'HasAns_total': 5928, 'NoAns_exact': 0.0, 'NoAns_f1': 0.0, 'NoAns_total': 5945, 'best_exact': 50.11370336056599, 'best_exact_thresh': 0.0, 'best_f1': 50.11370336056599, 'best_f1_thresh': 0.0}
```
'NoAns_exact' and 'NoAns_f1' are zero.

Do I miss anything when running the example code?







Comments:
Hi, running the exact same command but specifying you're using the version 2 with `--version_2_with_negative` gives me the following results:

```
01/29/2020 13:20:35 - INFO - __main__ -   Results: {'exact': 73.29234397372188, 'f1': 76.50792180947842, 'total': 11873, 'HasAns_exact': 71.94669365721997, 'HasAns_f1': 78.38707079013807, 'HasAns_total': 5928, 'NoAns_exact': 74.63414634146342, 'NoAns_f1': 74.63414634146342, 'NoAns_total': 5945, 'best_exact': 73.29234397372188, 'best_exact_thresh': 0.0, 'best_f1': 76.50792180947839, 'best_f1_thresh': 0.0}
```

Here are the exact arguments I used:
```
--model_type=bert --model_name_or_path=bert-base-uncased --do_train --do_eval --do_lower_case --version_2_with_negative --train_file=../../datasets/squad-v2.0/train-v2.0.json --predict_file=../../datasets/squad-v2.0/dev-v2.0.json --per_gpu_train_batch_size=12 --learning_rate=3e-5 --num_train_epochs=2.0 --max_seq_length=384 --doc_stride=128 --save_steps=10000 --output_dir=output_pt --overwrite_output_dir
```

@LysandreJik thanks for your reply.
I will try again.

My result is the same as yours. Maybe we not use the argument `--version_2_with_negative` ?
fix wandb/comet problems
Author: stas00
Pull request: yes

As discussed in https://github.com/huggingface/transformers/issues/7821

* handle the case where `comet_ml` is installed but not configured
* fix error in wandb code:
```
>               combined_dict = {**model.config.to_dict(), **combined_dict}
E               AttributeError: 'NoneType' object has no attribute 'to_dict'
```

Fixes: #7821

@sgugger 

Comments:
Thanks for this! W're working on a refactor on the comet_ml SDK to move some of the complications from here over to there.

@dsblank, also while you're at it, would it be possible to mend these in both projects. Getting these w/ py38 in the test suite. Thank you!

```
/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/comet_ml/utils.py:30
  /home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/comet_ml/utils.py:30: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.9 it will stop working
    from collections import Mapping, defaultdict

/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/comet_ml/monkey_patching.py:19
  /home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/comet_ml/monkey_patching.py:19: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
    import imp

/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/wandb/util.py:35
  /home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/wandb/util.py:35: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.9 it will stop working
    from collections import namedtuple, Mapping, Sequence

/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/wandb/vendor/graphql-core-1.1/graphql/type/directives.py:55
  /home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/wandb/vendor/graphql-core-1.1/graphql/type/directives.py:55: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.9 it will stop working
    assert isinstance(locations, collections.Iterable), 'Must provide locations for directive.'
```

@stas00 Sure, I'll take a look. I know some of these are caused by our support of Python 2.7, but perhaps there is a way to hide those.

Much appreciated, @dsblank!
[seq2seq] get_git_info fails gracefully
Author: stas00
Pull request: yes

It looks like gitpython is broken when one tries to use it from `git checkout hash`, see: https://github.com/gitpython-developers/GitPython/issues/633

While debugging/bisecting we need to be able to step through different commits and for the code to still work. 

Currently if I do:
```
git checkout fb94b8f1e16eb
```
and run an examples test, like `examples/seq2seq/test_seq2seq_examples_multi_gpu.py` (soon to be merged) it fails with:
```
ERR:   File "./examples/seq2seq/distillation.py", line 504, in <module>
ERR:     distill_main(args)
ERR:   File "./examples/seq2seq/distillation.py", line 494, in distill_main
ERR:     model = create_module(args)
ERR:   File "./examples/seq2seq/distillation.py", line 411, in create_module
ERR:     model = module_cls(args)
ERR:   File "/mnt/nvme1/code/huggingface/transformers-multigpu/examples/seq2seq/finetune.py", line 58, in __init__
ERR:     save_git_info(self.hparams.output_dir)
ERR:   File "/mnt/nvme1/code/huggingface/transformers-multigpu/examples/seq2seq/utils.py", line 355, in save_git_info
ERR:     repo_infos = get_git_info()
ERR:   File "/mnt/nvme1/code/huggingface/transformers-multigpu/examples/seq2seq/utils.py", line 374, in get_git_info
ERR:     "repo_branch": str(repo.active_branch),
ERR:   File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/git/repo/base.py", line 705, in active_branch
ERR:     return self.head.reference
ERR:   File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/git/refs/symbolic.py", line 272, in _get_reference
ERR:     raise TypeError("%s is a detached symbolic reference as it points to %r" % (self, sha))
ERR: TypeError: HEAD is a detached symbolic reference as it points to 'fb94b8f1e16eb21d166174f52e3e49e669ef0ac4'
```

The odd leading `ERR` string is just a replay of stderr from the sub-process - please ignore this nuance.

This PR provides a workaround.

@sshleifer


Comments:
Thanks for returning the same keys!
Two slow deberta test failures
Author: sshleifer
Pull request: no
Labels: name: Tests

https://github.com/huggingface/transformers/runs/1204063498?check_suite_focus=true

```
FAILED tests/test_modeling_deberta.py::DebertaModelIntegrationTest::test_inference_classification_head
FAILED tests/test_tokenization_deberta.py::DebertaTokenizationTest::test_torch_encode_plus_sent_to_model
```

@LysandreJik I think ?

Comments:
Fix DeBERTa integration tests
Author: LysandreJik
Pull request: yes

Fix the two DeBERTa integration tests.

- The classification test doesn't actually make sense, since there is no classification head saved in `microsoft/deberta-base`
- Fix to the way the attention was initialized according to the pos type.

closes https://github.com/huggingface/transformers/issues/7565
closes https://github.com/huggingface/transformers/pull/7645

Comments:
Small fixes to HP search
Author: sgugger
Pull request: yes

# What does this PR do?

HP search has two small bugs in `Trainer` right now:
- it pops things form the metrics dict, which are then improperly logged
- it doesn't pop out the `total_flos`, so the hp search tries to maximize the flos (which I guess is @TevenLeScao master plan to conquer the universe ;-) )

Comments:
Fix missing reference titles in retrieval evaluation of RAG
Author: lhoestq
Pull request: yes

There's currently a bug in the retrieval evaluation script of RAG where for each sample, the first reference title is discarded.
Moreover only `k` reference titles were taken into account while we need all of them to compute the Precision @ k.

Because of this bug the index provided by the RAG team (hnsw M=128 + SQ8 with the "inner product to L2" trick) only had 35% of Precision @ k. With this fix it is back to 70%, which is consistent with what they had internally as far as I know.

Apparently the original parsing script used to add the question before the titles, that's why it was discarding the first entry.

cc @ola13 

Comments:
Great!
Transformer-XL not working with DistributedDataParallel
Author: RafaelWO
Pull request: no
Labels: name: wontfix

# 🐛 Bug

## Information
I'm using the pretrained `TransfoXLLMHeadModel` for finetuning but I get an error when trying to use it with pytorch's `DistributedDataParallel`:
> self.reducer.prepare_for_backward(list(_find_tensors(output)))
RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one. This error indicates that your module has parameters that were not used in producing loss. You can enable unused parameter detection by (1) passing the keyword argument `find_unused_parameters=True` to `torch.nn.parallel.DistributedDataParallel`; (2) making sure all `forward` function outputs participate in calculating loss. If you already have done the above two steps, then the distributed data parallel module wasn't able to locate the output tensors in the return value of your module's `forward` function. Please include the loss function and the structure of the return value of `forward` of your module when reporting this issue (e.g. list, dict, iterable). (prepare_for_backward at /opt/conda/conda-bld/pytorch_1579022034529/work/torch/csrc/distributed/c10d/reducer.cpp:514)

I'm using the pytorch distributed launch utility.  
I've already gone through similar issues regarding this topic, but none of the solutions solved my problem. A guess from me is, that this is caused by the returned `mems` and `DistributedDataParallel` could think that they require a gradient?

Model I am using: Transformer-XL

Language I am using the model on: English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)


## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 2.11.0
- Platform: Linux-4.15.0-108-generic-x86_64-with-debian-buster-sid
- Python version: 3.6.10
- PyTorch version (GPU?): 1.4.0 (True)
- Tensorflow version (GPU?): 2.1.0 (True)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Yes `torch.distributed.launch`


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


I have the same kind of problem with transformers 3.3.1 and a BertModel, but the error is now an inplace operation on torch.cuda.LongTensor problem, something related to EmbeddingBackward.

Could you please open a new issue with the full stacktrace + the information related to your environment?

Done as issue #7848.
configuration_utils: fix handling of `id2labels`
Author: Lodifice
Pull request: yes

# What does this PR do?

The parameter `id2labels` of class `PretrainedConfig` is documented as `List[str]`, so enumerate() should be used rather
than dict.items() in the constructor.

Since a lot of code (including test code) passes `id2labels` as a dict, enumerate() is only used if it is a list indeed.

Comments:
Hi, as you can see in [this example NER configuration file](https://s3.amazonaws.com/models.huggingface.co/bert/dslim/bert-base-NER/config.json), the `id2label` and `label2id` are actually dictionaries. 

Instead of doing the change you propose, changing the documentation to reflect that would be better. Thank you!

Wow, seems like someone else already fixed the documentation in the mean time.
Remove masked_lm_labels from returned dictionary in DataCollatorForNextSentencePrediction
Author: vblagoje
Pull request: yes

# What does this PR do?

This PR removes ```masked_lm_labels``` from dictionary returned in DataCollatorForNextSentencePrediction ```__call__``` method. I noticed the warning while BERT pre-training. I removed the lines (452,453) and the warning was finally gone. See discussion George (@gmihaila) and I already had in the [merged PR](https://github.com/huggingface/transformers/pull/7595)

<!-- Remove if not applicable -->


## Before submitting
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests), 
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case. **Discussed on forum but on previously merged [PR](https://github.com/huggingface/transformers/pull/7595) **
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation). **Not needed**
- [x] Did you write any new necessary tests? **Not needed**


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

@LysandreJik, @sgugger, and @gmihaila can review


Comments:
Trucated Outputs while finetuning 'bart-base' on XSUM [Summarization Task]
Author: yashgupta-7
Pull request: no

Hey!
I am trying to finetune bart-base model on the XSUM using standard commands. In the test-generations.txt file, the outputs I am getting after a few epochs (2-3) are truncated arbitrarily. Here is the exact command I am using:
`./finetune.sh --data_dir $XSUM_DIR --train_batch_size=8 --eval_batch_size=8 --output_dir=xsum_results --num_train_epochs 1 --model_name_or_path facebook/bart-base`
xsum_results is the directory I created and I am running this inside the examples/seq2seq directory. 

I referred to these issues but could not find anything that could help me:
https://github.com/huggingface/transformers/issues/5656
https://github.com/huggingface/transformers/issues/6502

Some examples of the outputs I am getting:
German carmaker Daimler has reported a rise in sales of its cars and trucks in
Angelina Jolie has been honoured at a film festival in Bosnia, where she was the
Cuba's President Raul Castro has said he will introduce a series of reforms to the country
People who are suicidal should be given more help to stop them from jumping, a charity has

My pip freeze:
`absl-py==0.10.0
cachetools==4.1.1
certifi==2020.6.20
chardet==3.0.4
click==7.1.2
dill==0.3.2
filelock==3.0.12
future==0.18.2
gitdb==4.0.5
GitPython==3.1.8
google-auth==1.21.3
google-auth-oauthlib==0.4.1
grpcio==1.32.0
idna==2.10
joblib==0.16.0
Markdown==3.2.2
nlp==0.4.0
nltk==3.5
numpy==1.19.2
oauthlib==3.1.0
packaging==20.4
pandas==1.1.2
Pillow==7.2.0
portalocker==2.0.0
protobuf==3.13.0
pyarrow==1.0.1
pyasn1==0.4.8
pyasn1-modules==0.2.8
pyparsing==2.4.7
python-dateutil==2.8.1
pytorch-lightning==0.9.0
pytz==2020.1
PyYAML==5.3.1
regex==2020.9.27
requests==2.24.0
requests-oauthlib==1.3.0
rouge-score==0.0.4
rsa==4.6
sacrebleu==1.4.14
sacremoses==0.0.43
sentencepiece==0.1.91
six==1.15.0
smmap==3.0.4
tensorboard==2.2.0
tensorboard-plugin-wit==1.7.0
tokenizers==0.8.1rc2
torch==1.6.0+cu101
torchvision==0.7.0+cu101
tqdm==4.49.0
transformers @ git+https://github.com/yashgupta-7/transformers@9e68d075a4100906509170498480823e7e61874a
urllib3==1.25.10
Werkzeug==1.0.1
xxhash==2.0.0
zipp==3.2.0`

Here is the recommended setting for XSUM.
`--max_target_length=60 --val_max_target_length=60 --test_max_target_length=100`
But the default values are all higher than these, so this cannot be a problem in my opinion.

It would be great if someone can point me to the potential problems that may be the reason. I am looking forward to fine-tuning on a custom dataset and really want the standard XSUM to get working!

Comments:
Try adding `decoder_start_token_id=2` to `best_tfmr/config.json` and let me know if that changes anything!

I just did the above-mentioned change and decoded (run_eval.py) without training any further. Still, the issue persists. 

Ok, you could try re-training/training more with new training code. We can't reproduce this on `master`

okay, thanks!