noamaanMulla-03/transformers-issues huggingface dataset:
Add XLM-RoBERTa model for TF2
Author: jplu
Pull request: yes

Hello,

I have implemented the XLM-RoBERTa model handling for Tensorflow 2.

Comments:
There is a little incompatibility between isort and black apparently https://github.com/psf/black/issues/251

# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2454?src=pr&el=h1) Report
> Merging [#2454](https://codecov.io/gh/huggingface/transformers/pull/2454?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/9d87eafd118739a4c121d69d7cff425264f01e1c?src=pr&el=desc) will **increase** coverage by `0.6%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2454/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2454?src=pr&el=tree)

```diff
@@            Coverage Diff            @@
##           master    #2454     +/-   ##
=========================================
+ Coverage   74.51%   75.11%   +0.6%     
=========================================
  Files          87       88      +1     
  Lines       14920    14945     +25     
=========================================
+ Hits        11117    11226    +109     
+ Misses       3803     3719     -84
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2454?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/\_\_init\_\_.py](https://codecov.io/gh/huggingface/transformers/pull/2454/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9fX2luaXRfXy5weQ==) | `98.8% <100%> (+0.01%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_xlm\_roberta.py](https://codecov.io/gh/huggingface/transformers/pull/2454/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl94bG1fcm9iZXJ0YS5weQ==) | `100% <100%> (ø)` | |
| [src/transformers/data/processors/squad.py](https://codecov.io/gh/huggingface/transformers/pull/2454/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9kYXRhL3Byb2Nlc3NvcnMvc3F1YWQucHk=) | `56.72% <0%> (+27.54%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2454?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2454?src=pr&el=footer). Last update [9d87eaf...bb1aa06](https://codecov.io/gh/huggingface/transformers/pull/2454?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


does it work even if xlm-roberta-large is pretrained pytorch model? i mean do we need to convert pytorch model to tensorflow?

@jplu I took the liberty of updating the documentation to the new format directly on your fork. Thank you for your contribution, this is awesome!
Issue with my profile on the upload/share models webpage
Author: jplu
Pull request: no

Hello,

The URL of my [profile](https://huggingface.co/jplu) on the upload/share models webpage looks like if there was a model called `jplu`. Any idea why?

Thanks.

Comments:
Should be fixed.

Nice! Thanks :)

By the way, you should also add a README.md to the same `pretrained_model` folders so that it's displayed on the model pages (see [this one](https://huggingface.co/dbmdz/bert-base-german-cased) for instance)

I'll document this feature better today.



Should `jplu/camembert-base` be `jplu/tf-camembert-base`?

The name should be ok now, it was an issue in the naming.

I will work on a README file, this is a super cool feature!!

(note that the READMEs will be in this repo in the future – that way we can collaborate on them/link them together/etc)

(see https://github.com/huggingface/transformers/issues/2520#issuecomment-579009439 if you haven't already)

Thanks a lot @jplu!
Correct PyTorch distributed training command in examples/README.md
Author: jarednielsen
Pull request: yes

Running the command currently detailed in the documentation yields 
`58.4/70.3 EM/F1` with Ubuntu 18.04 and torch 1.4.0, not `86.9/93.1` as promised. It also looks wrong because we're using a cased model with `--do_lower_case`. 

Switching it to match the PyTorch distributed training example given in the main README gives me the approximately-correct `86.68/93.03` results.

Comments:
That's cool, thanks @jarednielsen !
Can I use HF XLNet to make a Model that Predicts Backwards?
Author: Fredrum
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
I would like to create a model that lets me generate multi sentence text sequences but backwards, given the tail end of some input text.

Could I do that using the HF XLNet framework?

I am new to this stuff so if this is possible would you be able to please give me some general pointers on how to go go about doing this?

Grateful for any advice!

Cheers
Fred


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Best weights/models after fine-tuning gpt2
Author: sb1992
Pull request: no

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
I am fine-tuning gpt2 on a new dataset and it checkpoints after every 50 iterations and saves the model.  It stretches my local storage to extreme and would want to delete redundant models. So my queries are following:

1) Is it possible to only save the best weights (which gave the lowest perplexity/loss on evaluation data)?
2) When we run_generation.py and passed the directory of our fine-tuned model, which model weights are actually used for generation? ( as there are so many checkpoint folders with model weights).
3) And hence, related to above 2 questions how does the model consider "the best" model/weights from the fine-tuned model directory we pass as arguments? and can we just mention the chekcpoint folder as well in run_generation.py?

Thanks


Comments:
Hi! Indeed saving checkpoints after every 50 iterations is quite a lot, therefore we've upped this value to 500 yesterday in 335dd5e. Concerning your questions:

1. You can use the `--evaluate_during_training` flag to evaluate the model every `--logging_step`, or you can use the `--evaluate_all_checkpoints` to evaluate all the checkpoints at the end. There is no feature to save only the best model, but you could easily do it by modifying the script to save only when the evaluation yields better results than the previous one.
2. For the `run_generation.py` script, pass it the folder which contains the weights you want to use for generation. The folder must contain a `pytorch_model.bin`, a `config.json`, as well as a tokenizer object.
3. Yes, just mention the checkpoint folder in `run_generation.py`. 

Thank you. My bad i did not pay full attention to the arguments present in the generation script. That answers my queries.
Hardware requirements for BERT QA inference
Author: tothniki
Pull request: no

Hi,

I am using **bert-large-uncased-whole-word-masking-finetuned-squad** model for QA inference.

I used my laptop's CPU to build the pipeline and try it out. 

Now I want to deploy it, and so I would like to know what is the minimum hardware requirement?(If I use the same settings as in your usage example script)?

I am more interested in the **minimum size of the GPU**. 
Of course I don't want it to be too slow. Is there any studies on this matter, or measurement?

Thanks!

Comments:
@LysandreJik wrote an article about benchmarking transformers (focusing on inference) that might be of interest to you.

https://medium.com/huggingface/benchmarking-transformers-pytorch-and-tensorflow-e2917fb891c2

Thanks for the reply!
I checked the benchmarks, there you tested with 16GB GPU and bert-base model. I suppose then I have to try different, smaller size of GPUs, and figure out which can handle my task.
It is really informative though, nice work.
Loading pretrained RobertaForSequenceClassification fails, size missmatch error
Author: jlealtrujillo
Pull request: no

## 🐛 Bug

<!-- Important information -->

Model I am using `RobertaForSequenceClassification` and when I tried to load `'roberta-base'` model using this code on Google Colab:

```from transformers import RobertaForSequenceClassification, RobertaConfig
config = RobertaConfig()
model = RobertaForSequenceClassification.from_pretrained(
    "roberta-base", config = config)
model
```

I get the following error:
```
RuntimeError: Error(s) in loading state_dict for RobertaForSequenceClassification:
	size mismatch for roberta.embeddings.word_embeddings.weight: copying a param with shape torch.Size([50265, 768]) from checkpoint, the shape in current model is torch.Size([30522, 768]).
	size mismatch for roberta.embeddings.position_embeddings.weight: copying a param with shape torch.Size([514, 768]) from checkpoint, the shape in current model is torch.Size([512, 768]).
	size mismatch for roberta.embeddings.token_type_embeddings.weight: copying a param with shape torch.Size([1, 768]) from checkpoint, the shape in current model is torch.Size([2, 768]).
```

Maybe related to #1340 

## Environment

* Google Colab Platform Linux-4.14.137+-x86_64-with-Ubuntu-18.04-bionic
Python 3.6.8 (default, Oct  7 2019, 12:59:55) 
[GCC 8.3.0]
PyTorch 1.3.0+cu100
Transformers 2.1.1



Comments:
Hi! You're initializing RoBERTa with a blank configuration, which results in a very BERT-like configuration. BERT has different attributes than RoBERTa (different vocabulary size, positional embeddings size etc) so this indeed results in an error.

To instantiate RoBERTa you can simply do:

```py
model = RobertaForSequenceClassification.from_pretrained("roberta-base")
```

If you wish to have a configuration file so that you can change attributes like outputting the hidden states, you could do it like this:

```py
config = RobertaConfig.from_pretrained("roberta-base", output_hidden_states=True)
model = RobertaForSequenceClassification.from_pretrained("roberta-base", config=config)
```

Hi @LysandreJik ,

Thanks a lot for the clarification, this is indeed much clearer.  I tried the code again and it is working.
Add layerdrop to Flaubert
Author: formiel
Pull request: yes

This PR adds `layerdrop` to Flaubert.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2686?src=pr&el=h1) Report
> Merging [#2686](https://codecov.io/gh/huggingface/transformers/pull/2686?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/df27648bd942d59481a13842904f8cb500136e31?src=pr&el=desc) will **decrease** coverage by `0.01%`.
> The diff coverage is `16.66%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2686/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2686?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #2686      +/-   ##
==========================================
- Coverage    74.1%   74.09%   -0.02%     
==========================================
  Files          92       92              
  Lines       15168    15172       +4     
==========================================
+ Hits        11240    11241       +1     
- Misses       3928     3931       +3
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2686?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/configuration\_flaubert.py](https://codecov.io/gh/huggingface/transformers/pull/2686/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9jb25maWd1cmF0aW9uX2ZsYXViZXJ0LnB5) | `75% <ø> (ø)` | :arrow_up: |
| [src/transformers/modeling\_flaubert.py](https://codecov.io/gh/huggingface/transformers/pull/2686/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19mbGF1YmVydC5weQ==) | `29.19% <16.66%> (-0.13%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2686?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2686?src=pr&el=footer). Last update [df27648...15f8b5d](https://codecov.io/gh/huggingface/transformers/pull/2686?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Great, thanks!

Thanks so much for the quick merge, @LysandreJik!
Exact and F1 score do not increase when fine-tunes XLM on the SQuAD dataset
Author: ZhengWeiH
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

I am trying to fine-tune XLM on the SQuAD dataset.
The command is as following:
[CUDA_VISIBLE_DEVICES=0 python run_squad.py --model_type xlm --model_name_or_path xlm-mlm-tlm-xnli15-1024 --do_train --do_eval --train_file $SQUAD_DIR/train-v1.1.json --predict_file $SQUAD_DIR/dev-v1.1.json --per_gpu_train_batch_size 12 --learning_rate 3e-5 --num_train_epochs 100 --max_seq_length 384 --doc_stride 128 --eval_all_checkpoints --save_steps 50 --evaluate_during_training --output_dir /home/weihua/Sqad/transformers/xlm_out/ ]
![WeChat Image_20191112113346](https://user-images.githubusercontent.com/43492059/68640277-2da52700-0542-11ea-981b-c748e918e1d8.png)

All the parameters I set are in accordance with the example in the pytorch-transformers document. But Exact and F1 scores have barely increased. And the loss of each epoch also drops very slowly. Each epoch drops by about 0.01.
![68477603-d6b10080-0268-11ea-947f-995b157da8d3](https://user-images.githubusercontent.com/43492059/68640296-3f86ca00-0542-11ea-93b3-db1bd64539b2.png)


Is there a problem with my parameter settings? Or do I need to adjust some parts of the model when I use XLM?


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


duplicate #1799
Typo on markdown link in README.md
Author: arnaudmiribel
Pull request: yes


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2698?src=pr&el=h1) Report
> Merging [#2698](https://codecov.io/gh/huggingface/transformers/pull/2698?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/0aa40e9569a71306036de3a217eed55521699604?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2698/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2698?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #2698   +/-   ##
=======================================
  Coverage   74.24%   74.24%           
=======================================
  Files          92       92           
  Lines       15215    15215           
=======================================
  Hits        11297    11297           
  Misses       3918     3918
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2698?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2698?src=pr&el=footer). Last update [0aa40e9...9a11e6f](https://codecov.io/gh/huggingface/transformers/pull/2698?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Great, thanks!
get_linear_schedule_with_warmup method can't be found in optimization.py
Author: ghost
Pull request: no

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): BERT

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

My transformer was downloaded from anaconda cloud by using command "conda install -c conda-forge transformers", when I tried to use AdamW as what you show in the example, I found there is no **get_linear_schedule_with_warmup** in the **transformers.optimization.py** file for me to create a scheduler.

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1.
2.
3.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

## Environment

* OS:
* Python version:
* PyTorch version:
* `transformers` version (or branch):
* Using GPU ?
* Distributed or parallel setup ?
* Any other relevant information:


Comments:
There clearly is:

https://github.com/huggingface/transformers/blob/0aa40e9569a71306036de3a217eed55521699604/src/transformers/optimization.py#L47-L59

Please fill out the complete template - it's there for a reason. If you had shown us which version you're working with, we could probably tell you that your version is too late, or at least dig further. Now we can't help at all, it's just guess work.

Give code, full error trace, and your PyTorch/Tensorflow version.

I thought my transformers was the lastest version, but I found it's not when I checked it on the anaconda cloud "https://anaconda.org/conda-forge/transformers". The problem has been solved after reinstalling, Thanks for your reply :)

In the future, please fill out the complete template. Please close this question if you don't have any more questions.

OK
XLNet is getting slower when enabling mems
Author: makcedward
Pull request: no
Labels: name: wontfix

Per API doc, using mems help to reduce inference time. However, I noticed that more time is needed when increasing mem_len. Do I misunderstand the usage of mem_len parameter? 
Here is the testing code

```
import time
import torch
from transformers import XLNetTokenizer, XLNetLMHeadModel

tokenizer = XLNetTokenizer.from_pretrained('xlnet-base-cased')

text = """
    A horrible, messy split second presents
    itself to the heart-shaped version as Scott is moved. The upcoming movie benefits at 
    the mental cost of ages 14 to 12. Nothing substantial is happened for almost 48 days. 
    When that happens, we lose our heart. <eod> The quick brown fox jumps over the lazy dog. <mask>
"""

mems = None

input_ids = tokenizer.encode(text)
input_ids = torch.tensor(input_ids).unsqueeze(0)

epoch = 20

for men_len in [0, 16, 32, 64, 128]:
    model = XLNetLMHeadModel.from_pretrained('xlnet-base-cased', mem_len=men_len)

    start_dt = time.monotonic()
    for i in range(epoch):
        outputs = model(input_ids=input_ids, mems=mems)
        if men_len > 0:
            mems = outputs[1]
    end_dt = time.monotonic()

    print('Average Duration for men_len {}: {}'.format(men_len, round(end_dt-start_dt, 2)))
```

Output is 
```
Average Duration for men_len 0: 2.49
Average Duration for men_len 16: 2.62
Average Duration for men_len 32: 2.67
Average Duration for men_len 64: 2.81
Average Duration for men_len 128: 3.28
```

Comments:
When using `past` or `mems` values, you should be careful not to give the model the input ids which have already been computed. We've recently [added a documentation section ](https://huggingface.co/transformers/quickstart.html#using-the-past)detailing the use of `past`, which is similar to the way `mems` should be used. 

Please notice we're only feeding the model the tokens for which the attention values have not be computed yet; which is only the last token in the case of sequential decoding.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.