noamaanMulla-03/transformers-issues huggingface dataset:
Fixed training for TF XLM
Author: tlkh
Pull request: yes

This PR fixes `model.fit()` training for TF XLM model, and tested in a script similar to `run_tf_glue.py`. It also is tested and works with AMP and tf.distribute for mixed precision and multi-GPU training.

This changes some Python `assert` statements to `tf.debugging.assert_equal` both in `TFXLMMainLayer.call()` and `gen_mask()`

Otherwise, errors encountered:
* `TypeError: You are attempting to use Python control flow in a layer that was not declared to be dynamic. Pass 'dynamic=True' to the class constructor.`
* `OperatorNotAllowedInGraphError: using a 'tf.Tensor' as a Python 'bool' is not allowed in Graph execution. Use Eager execution or decorate this function with @tf.function.`

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1668?src=pr&el=h1) Report
> Merging [#1668](https://codecov.io/gh/huggingface/transformers/pull/1668?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/079bfb32fba4f2b39d344ca7af88d79a3ff27c7c?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `75%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1668/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1668?src=pr&el=tree)

```diff
@@          Coverage Diff           @@
##           master   #1668   +/-   ##
======================================
  Coverage    85.9%   85.9%           
======================================
  Files          91      91           
  Lines       13653   13653           
======================================
  Hits        11728   11728           
  Misses       1925    1925
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/1668?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [transformers/modeling\_tf\_xlm.py](https://codecov.io/gh/huggingface/transformers/pull/1668/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX3RmX3hsbS5weQ==) | `90.39% <75%> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1668?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1668?src=pr&el=footer). Last update [079bfb3...842f3bf](https://codecov.io/gh/huggingface/transformers/pull/1668?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Awesome, thanks a lot @tlkh
Error raised by "tmp_eval_loss += tmp_eval_loss.item()" when using multi-gpu
Author: focox
Pull request: yes

fixed the bug raised by "tmp_eval_loss += tmp_eval_loss.item()" when parallelly using multi-gpu.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1608?src=pr&el=h1) Report
> Merging [#1608](https://codecov.io/gh/huggingface/transformers/pull/1608?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/ef1b8b2ae5ad1057154a126879f7eb8de685f862?src=pr&el=desc) will **increase** coverage by `0.02%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1608/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1608?src=pr&el=tree)

```diff
@@            Coverage Diff            @@
##           master   #1608      +/-   ##
=========================================
+ Coverage   86.17%   86.2%   +0.02%     
=========================================
  Files          91      91              
  Lines       13595   13595              
=========================================
+ Hits        11715   11719       +4     
+ Misses       1880    1876       -4
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/1608?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/1608/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL2ZpbGVfdXRpbHMucHk=) | `76.37% <0%> (+2.19%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1608?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1608?src=pr&el=footer). Last update [ef1b8b2...bd847ce](https://codecov.io/gh/huggingface/transformers/pull/1608?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Thanks @focox!
Missing a line in examples/distillation/README.md
Author: jianwolf
Pull request: no

In How to train Distil* -> B, in both of the training commands, you should add `--alpha_clm 0.0 \`, otherwise an assertion error will be triggered (https://github.com/huggingface/transformers/blob/079bfb32fba4f2b39d344ca7af88d79a3ff27c7c/examples/distillation/train.py#L49).

Comments:
Oh yes indeed. Let me correct it. Thank you for pointing that out @jianwolf!
Versioning in documentation
Author: LysandreJik
Pull request: yes

Several versions of the documentation can now be accessed:

`huggingface.co/transformers` for the master release
`huggingface.co/transformers/v2.1.1` for the 2.1.1 official release and so on.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1604?src=pr&el=h1) Report
> Merging [#1604](https://codecov.io/gh/huggingface/transformers/pull/1604?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/ef1b8b2ae5ad1057154a126879f7eb8de685f862?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1604/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1604?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #1604      +/-   ##
==========================================
- Coverage   86.17%   86.16%   -0.01%     
==========================================
  Files          91       91              
  Lines       13595    13593       -2     
==========================================
- Hits        11715    11713       -2     
  Misses       1880     1880
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/1604?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [transformers/tokenization\_ctrl.py](https://codecov.io/gh/huggingface/transformers/pull/1604/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL3Rva2VuaXphdGlvbl9jdHJsLnB5) | `96.03% <0%> (-0.08%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1604?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1604?src=pr&el=footer). Last update [ef1b8b2...6e85bcc](https://codecov.io/gh/huggingface/transformers/pull/1604?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Ready to merge

Awesome!
run_tf_glue works with all tasks
Author: LysandreJik
Pull request: yes

Slightly changed the logic of the DataProcessor so that it can handle GLUE data coming from the `tensorflow_datasets` package.

Updated the script so that all tasks are now available, with regression tasks (STS-B) as well as all classification tasks.

Should update the import into PyTorch that currently tests if two sentences are paraphrases of each other; as the glue script handles all GLUE tasks it should be a different test.

Comments:
[FIX] fix run_generation.py to work with batch_size > 1
Author: mataney
Pull request: yes

I expended the `top_k_top_p_filtering` function, and by that the`run_generation.py` script to work with num_samples > 1.

This can be expended by scattering the sorted tensors.
First pull request in this repository, so let me know if I need to do anything else :)

Cheers, Matan.

Comments:
@thomwolf
I created this PR to deal with the `top p` generations.
Should I have opened an issue first to check if it is needed?
Should I deal with the conflicts?

Cheers.

Hi @mataney, thanks.

This was rebased, fixed by https://github.com/huggingface/transformers/commit/f96ce1c24151349251880c95e9a9fb144b62367c, and merged to master by 2a5663c28043dc6d2746e69f0fb89e0d5872c63d.

Check that everything looks good on your side if you can.
Wrong Roberta special tokens in releases on GitHub
Author: fhamborg
Pull request: no

## 🐛 Bug

Model I am using (Bert, XLNet....): Roberta

Language I am using the model on (English, Chinese....): Potentially wrong on any language

The problem arise when using the official example scripts: see https://github.com/huggingface/transformers/releases/tag/1.1.0

In the section `Tokenizer sequence pair handling` the special tokens for Roberta are wrong if I'm not mistaken. The example reads:
```
[CLS] SEQUENCE_0 [SEP] [SEP] SEQUENCE_1 [SEP]
```

whereas Roberta's actual representation for a sequence pair including special tokens should be (also following transformer's official documentation, cf. https://huggingface.co/transformers/model_doc/roberta.html):
```

 SEQUENCE_0 
 
 SEQUENCE_1 
 
```
Note the 
 or 
 instead of [SEP]. I am not sure about the [CLS], though, but I think for Roberta it should not be there.



Comments:
You are right the release is wrong, it should be `
 SEQUENCE_0 

 SEQUENCE_1 
`. I just updated it; thank you!
Unpickling errors when running examples
Author: Khev
Pull request: no

## ❓ Questions & Help

Hi there, when I run the examples

```
%run run_generation.py \
    --model_type=gpt2 \
    --model_name_or_path=gpt2
```

I keep getting the following errors:

```
---------------------------------------------------------------------------
UnpicklingError                           Traceback (most recent call last)
~/research/transformers/examples/run_generation.py in <module>
    258 
    259 if __name__ == '__main__':
--> 260     main()

~/research/transformers/examples/run_generation.py in main()
    186     model_class, tokenizer_class = MODEL_CLASSES[args.model_type]
    187     tokenizer = tokenizer_class.from_pretrained(args.model_name_or_path)
--> 188     model = model_class.from_pretrained(args.model_name_or_path)
    189     model.to(args.device)
    190     model.eval()

~/anaconda3/lib/python3.7/site-packages/transformers/modeling_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    343 
    344         if state_dict is None and not from_tf:
--> 345             state_dict = torch.load(resolved_archive_file, map_location='cpu')
    346 
    347         missing_keys = []

~/anaconda3/lib/python3.7/site-packages/torch/serialization.py in load(f, map_location, pickle_module, **pickle_load_args)
    385         f = f.open('rb')
    386     try:
--> 387         return _load(f, map_location, pickle_module, **pickle_load_args)
    388     finally:
    389         if new_fd:

~/anaconda3/lib/python3.7/site-packages/torch/serialization.py in _load(f, map_location, pickle_module, **pickle_load_args)
    562             f.seek(0)
    563 
--> 564     magic_number = pickle_module.load(f, **pickle_load_args)
    565     if magic_number != MAGIC_NUMBER:
    566         raise RuntimeError("Invalid magic number; corrupt file?")

UnpicklingError: invalid load key, '<'.

```


Comments:
Closing this in favor of #1684
Missing RobertaForMultipleChoice
Author: malmaud
Pull request: no
Labels: name: wontfix

Hi,
It seems like a `RobertaForMultipleChoice` class should exist to parallel `BertForMultipleChoice`. Or was there a particular reason it was elided? 

Comments:
Hi @malmaud, no particular reason. But it's also super easy to just implement your own classifier on top of the model (and then you have full control)

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
evaluate bert on Senteval dataset
Author: ghost
Pull request: no
Labels: name: wontfix

Hi
I would like to evaluate bert on senteval datasets, with Senteval, I am not sure how to do it,
Do you provide any evaluation toolkit to evaluate the trained models?
thanks

Comments:
This should help you: https://medium.com/dsnet/running-pytorch-transformers-on-custom-datasets-717fd9e10fe2
I did it for IMDB dataset which you should be able to customize for any other dataset.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Can I load a CTRL model that was fine-tuned using the Salesforce code?
Author: orenmelamud
Pull request: no

## ❓ Questions & Help

I have a custom CTRL model that I trained using the Salesforce TF code and I was hoping that I could convert it into the transformers format and load it there. Any advice?


Comments:
cc @keskarnitish :)

Sure, I'll get on this soon. I'll push it to https://github.com/salesforce/ctrl and link here once I'm done. 

Thanks  @keskarnitish! That would be great!

Added in https://github.com/salesforce/ctrl/commit/a0d0b4d2f38ae55a1396dfad4d6bff7cc9435c2d , see updated `README.md` for usage. 

That is awesome @keskarnitish! We'll add it to our doc here as well.
ALbert Model implementation is finished on squad qa task. but some format is different with huggingface.(specify on albert qa task)
Author: pohanchi
Pull request: no

# 🌟New model addition

## Model description

<!-- Important information -->
Hey, seems like your community creates the same one branch for Albert, actually, I also reimplement the code and transfer from tf-hub to PyTorch by referencing your other preview convert_tf_xxx.py code. and also reproduce the author's performance on squad 1.1 for the albert-base model.

The reproduce is show below
Squad 1.1 EM: 79.98, F1 score: 87.98
The paper performance:
Squad 1.1 EM: 82.3,  F1 score: 89.30

## Open Source status

* [x] the model implementation is available: (give details)
* [x] the model weights are available: (give details)

## Additional context

<!-- Add any other context about the problem here. -->
The code is below https://github.com/pohanchi/huggingface_albert.
I just want to ask that I reference your code module from other old model, so what should i do to set my repo public safety.  



Comments:
Fix https://github.com/huggingface/transformers/issues/1673
Author: cregouby
Pull request: yes


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1679?src=pr&el=h1) Report
> Merging [#1679](https://codecov.io/gh/huggingface/transformers/pull/1679?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/fa735208c96c18283b8d2f3fcbfc3157bbd12b1e?src=pr&el=desc) will **increase** coverage by `0.05%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1679/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1679?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #1679      +/-   ##
==========================================
+ Coverage   85.08%   85.14%   +0.05%     
==========================================
  Files          94       94              
  Lines       13920    13920              
==========================================
+ Hits        11844    11852       +8     
+ Misses       2076     2068       -8
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/1679?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/1679/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX3V0aWxzLnB5) | `92.44% <ø> (+1.45%)` | :arrow_up: |
| [transformers/tests/modeling\_common\_test.py](https://codecov.io/gh/huggingface/transformers/pull/1679/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL3Rlc3RzL21vZGVsaW5nX2NvbW1vbl90ZXN0LnB5) | `76.49% <0%> (+0.59%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1679?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1679?src=pr&el=footer). Last update [fa73520...ac29353](https://codecov.io/gh/huggingface/transformers/pull/1679?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


This looks good to me!

Yes indeed, thanks!
modifying config
Author: ghost
Pull request: no
Labels: name: wontfix

Hi
I need to add more variables to the config file, while using pretrained models. I could not figure this out how to add parameters to config file, could you provide me please with examples?
very much appreciated!

Comments:
Hi! What kind of values are you trying to add?
The configuration file is a simple python object, so you can handle it just as you would any Python object:

```
config = GPT2Config.from_pretrained("gpt2")
config.values = [1, 2]

print(config.values)
# [1, 2]
```

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Best practices for combining large pretrained models with smaller models?
Author: dchang56
Pull request: no
Labels: name: wontfix

Hello,

If I were to try to combine a large model (like BERT) with a smaller model (some variation of fully connected, convolutional network with significantly less params and pretraining) by jointly training them and concatenating their outputs for a final classifier, what would be some things I should consider?

For example, should they have different optimizers and learning rates? Should I try to keep the number of params in the smaller model relatively small? What would be some good ways of fusing the output of BERT and the output of the small model besides concatenating? 

I'd really appreciate any insight from anyone who's tried something like this or have thought about it. Thank you!

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


@dchang56 Any updates? I am looking to do this as well. 

Hi! I see that you're also doing scientific/medical NLP :)
I sent you an email at your gmail address.
Unable to read pre-trained model using BertModel.from_pretrained 
Author: rshah1990
Pull request: no
Labels: name: wontfix

I am using pre-trained BERT model kept at AWS s3 bucket. When i am trying to read the model using BertModel.from_pretrained  It return NONE object. Things are working offline when i download a folder on the same location as my code resides. 

Comments:
Hi, you are trying to download one of your own models kept on a personal AWS s3 bucket or one of our models? What string do you pass to the `from_pretrained` method?

Thank you for your response, 
I am trying to use my own model kept on my personal AWS s3 bucket. 
The string to from_pretrained method is the path of folder which contains three files config , vocab and model. I have also tried to zip all files (.tar.gz) and use that file rather then folder but it also didn't work.
Please let me know in case you need more information.



Firstly, have you checked that the model in the bucket is reachable? 

Secondly, what is the name of the config file?

Yes its reachable, I can read vocab.txt file .
Name of config file is bert_config.json

I do have the same (similar) problem, except that I use the pretrained models (e.g. bert-base-uncased"). The script repeatedly downloads the vocab, json and model file and often fails to load the model. Everything is works if I do it on the local machine. 

I also tried to download the files locally and unsuccessfully loaded the model directly. (tokenizer failed, model failed, BertConfig worked). Maybe I do something wrong here - example code would help

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Can we get a 1.1.1 release so that AutoRoberta is included?
Author: matt-gardner
Pull request: no
Labels: name: wontfix

See issue title.  I'm about to open a PR to add roberta to allennlp; it'd be nice to have a released version to depend on, instead of a github commit.

Comments:
Yes! @LysandreJik 

Sounds good, we'll release one soon. Probably around the end of the week or early next week.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Using pretrained XLNET for long sentences
Author: aviclu
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help
Is it possible to feed the pretrained large XLNET model with sentences of length of more than 512 tokens?
If no, is there any model which supports that?

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Why do I get 13 hidden layers?
Author: acmilannesta
Pull request: no

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Hi,

When I try to type
```
model = TFRobertaForSequenceClassification.from_pretrained('roberta-base', output_hidden_states=True)
hidden_layers = model([input])

```

The length of hidden_layers is 13. But base model should have 12 layers, right?
And if I want to get the last hidden layer, should I use "hidden_layers[-1]"?

Thanks in advance!

Comments:
The first element is the embedding output.

Yes, use `hidden_layers[-1]`.
Efficient data loading functionality
Author: shubhamagarwal92
Pull request: no
Labels: name: wontfix

## 🚀 Feature

Efficient data loader for huge dataset with lazy loading! 

## Motivation

<!-- Please outline the motivation for the proposal. Is your feature request related to a problem? e.g., I'm always frustrated when [...]. If this is related to another GitHub issue, please link here too. -->

I am working with a huge dataset consisting of 120m examples (~40G raw text) in a single csv file. I tried to follow the [run_glue](https://github.com/huggingface/pytorch-transformers/blob/master/examples/run_glue.py) distributed training example, however this is too slow as it first creates all the examples and cache it [here](https://github.com/huggingface/pytorch-transformers/blob/master/examples/run_glue.py#L251). Basically only first process in distributed training process the dataset and others just use the cache. 

Is there any data loader (or a working example) that would be efficient for training the model on such a huge dataset? 

## Additional context

<!-- Add any other context or screenshots about the feature request here. -->

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


@thomwolf Is there an efficient implementation for this? Could you re-open the issue please. 

CSV generally has slower load times. How about benchmarking with pickle, parquet, and feather? Pytorch's dataloader can handle multiple files and multiple lines per file 
https://discuss.pytorch.org/t/dataloaders-multiple-files-and-multiple-rows-per-column-with-lazy-evaluation/11769