noamaanMulla-03/transformers-issues huggingface dataset:
Reproducing squad score with TFXLMRoberta?
Author: nchocho
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

Hello all, 

first of all, thanks for the library that is very helpful. There have been several discussions regarding XLM-Roberta and questions answering (#3732  #3694 ). On my side, I added a TFXLMRobertaForQuestionAnswering but never reproduced a decent squad score (I was always below 50% f1). The base LM I was using was xlm-roberta-base converted to tf, or the community ones (jplu), I tried with type_vocab_size=1 or type_vocab_size=2 (in order to use segment_ids as for Bert, I did it by overwriting the create_token_type_ids_from_sequences in the tokenizer). This did not really change anything. I am using a AdamWeightDecay as for Bert finetuning and I start to believe that there is no problem with my code since I saw yesterday the PR #3812 which is basically the same code as mine. For info, I am not only training on squad, I am training on squad + mlqa. Using the exact same approach, I got very good score with Bert multilingual. Since I am a bit stuck I was wondering if someone here (from huggingface or not) managed to properly train XLMRoberta with tensorflow and get good squad results. If so, it would be super helpful for me if you can share the parameters that was used (learning rate, number of epochs, etc). 

** 

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
[BART] test_generate_fp16 fails after PR#3140
Author: sshleifer
Pull request: no

passes after `git checkout d6de6423` (commit preceding #3140)
Traceback:
```
                 unfinished_sents.mul_((~eos_in_sents).long())

            # stop when there is a 
 in each sentence, or if we exceed the maximul length
>           if unfinished_sents.max() == 0:
E           RuntimeError: cuda runtime error (716) : misaligned address at /pytorch/aten/src/THC/THCReduceAll.cuh:327

src/transformers/modeling_utils.py:992: RuntimeError
```
@thomwolf @patrickvonplaten 

Comments:
Interesting - will investigate. Probably something with the `device` of `unfinished_sents` then!

@julien-c 

Caused by previously unexposed kwargs changing behavior:
passing
```
do_sample=False, early_stopping=True
```
in the unit test fixes them.

I'm getting this error @sshleifer 

https://github.com/huggingface/transformers/issues/5221

I have early stopping = False, but do_sample = True
[pl_examples] revert deletion of optimizer_step
Author: sshleifer
Pull request: yes

using default `optimizer_step` has at least 2 issues:
The default version...
1) doesn't call `lr_scheduler.step()`
2) does call `self.trainer.scaler.step(optimizer)`

I haven't diagnosed which of these is the main culprit of the issue I was seeing (very high loss, not going down). This fixes that issue. 

I suspect it is mostly the latter: @williamFalcon 


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5227?src=pr&el=h1) Report
> Merging [#5227](https://codecov.io/gh/huggingface/transformers/pull/5227?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/c01480bba3b2f0bd8516679476235f4701c21b3b&el=desc) will **increase** coverage by `0.06%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5227/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5227?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #5227      +/-   ##
==========================================
+ Coverage   77.98%   78.05%   +0.06%     
==========================================
  Files         138      138              
  Lines       23839    23839              
==========================================
+ Hits        18592    18608      +16     
+ Misses       5247     5231      -16     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5227?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5227/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `76.42% <0.00%> (-0.39%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5227/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `85.86% <0.00%> (-0.30%)` | :arrow_down: |
| [src/transformers/trainer.py](https://codecov.io/gh/huggingface/transformers/pull/5227/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90cmFpbmVyLnB5) | `39.33% <0.00%> (-0.24%)` | :arrow_down: |
| [src/transformers/data/processors/squad.py](https://codecov.io/gh/huggingface/transformers/pull/5227/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9kYXRhL3Byb2Nlc3NvcnMvc3F1YWQucHk=) | `35.03% <0.00%> (+6.36%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5227?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5227?src=pr&el=footer). Last update [c01480b...51bb9b3](https://codecov.io/gh/huggingface/transformers/pull/5227?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Only put tensors on a device
Author: sgugger
Pull request: yes

Fix Trainer when users have inputs containing non-tensor values.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5223?src=pr&el=h1) Report
> Merging [#5223](https://codecov.io/gh/huggingface/transformers/pull/5223?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/c01480bba3b2f0bd8516679476235f4701c21b3b&el=desc) will **decrease** coverage by `0.03%`.
> The diff coverage is `60.00%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5223/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5223?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #5223      +/-   ##
==========================================
- Coverage   77.98%   77.95%   -0.04%     
==========================================
  Files         138      138              
  Lines       23839    23841       +2     
==========================================
- Hits        18592    18586       -6     
- Misses       5247     5255       +8     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5223?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/trainer.py](https://codecov.io/gh/huggingface/transformers/pull/5223/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90cmFpbmVyLnB5) | `39.62% <60.00%> (+0.04%)` | :arrow_up: |
| [src/transformers/modeling\_openai.py](https://codecov.io/gh/huggingface/transformers/pull/5223/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19vcGVuYWkucHk=) | `79.51% <0.00%> (-1.39%)` | :arrow_down: |
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5223/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `76.42% <0.00%> (-0.39%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5223/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `85.86% <0.00%> (-0.30%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5223?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5223?src=pr&el=footer). Last update [c01480b...10ff478](https://codecov.io/gh/huggingface/transformers/pull/5223?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Create README.md
Author: ahotrod
Pull request: yes
Labels: name: model card

electra_large_discriminator_squad2_512 Question Answering LM

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5217?src=pr&el=h1) Report
> Merging [#5217](https://codecov.io/gh/huggingface/transformers/pull/5217?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/b28b53713161a6299c757c32f7179a2cb2d8cbd7&el=desc) will **increase** coverage by `0.02%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5217/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5217?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #5217      +/-   ##
==========================================
+ Coverage   77.96%   77.98%   +0.02%     
==========================================
  Files         138      138              
  Lines       23838    23838              
==========================================
+ Hits        18585    18590       +5     
+ Misses       5253     5248       -5     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5217?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5217/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `86.15% <0.00%> (+0.14%)` | :arrow_up: |
| [src/transformers/modeling\_openai.py](https://codecov.io/gh/huggingface/transformers/pull/5217/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19vcGVuYWkucHk=) | `80.90% <0.00%> (+1.38%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5217?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5217?src=pr&el=footer). Last update [b28b537...6f847e8](https://codecov.io/gh/huggingface/transformers/pull/5217?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Thanks! [model page](https://huggingface.co/ahotrod/electra_large_discriminator_squad2_512)
Update README.md
Author: aodiniz
Pull request: yes
Labels: name: model card

Fix/add information in README.md

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5199?src=pr&el=h1) Report
> Merging [#5199](https://codecov.io/gh/huggingface/transformers/pull/5199?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/a34fb91d541bdb235a6c9fa96ecf11d51426ac84&el=desc) will **decrease** coverage by `0.89%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5199/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5199?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #5199      +/-   ##
==========================================
- Coverage   77.99%   77.10%   -0.90%     
==========================================
  Files         138      138              
  Lines       23786    23786              
==========================================
- Hits        18553    18340     -213     
- Misses       5233     5446     +213     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5199?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_openai.py](https://codecov.io/gh/huggingface/transformers/pull/5199/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9vcGVuYWkucHk=) | `19.92% <0.00%> (-75.00%)` | :arrow_down: |
| [src/transformers/modeling\_openai.py](https://codecov.io/gh/huggingface/transformers/pull/5199/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19vcGVuYWkucHk=) | `79.51% <0.00%> (-1.39%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5199/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `85.71% <0.00%> (-0.30%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5199?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5199?src=pr&el=footer). Last update [a34fb91...e374d82](https://codecov.io/gh/huggingface/transformers/pull/5199?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Create README.md
Author: flozi00
Pull request: yes
Labels: name: model card


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5108?src=pr&el=h1) Report
> Merging [#5108](https://codecov.io/gh/huggingface/transformers/pull/5108?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/355954ffca798bb81d9db8886e30ce10f11e8a40&el=desc) will **decrease** coverage by `0.08%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5108/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5108?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #5108      +/-   ##
==========================================
- Coverage   77.28%   77.20%   -0.09%     
==========================================
  Files         133      133              
  Lines       22134    22134              
==========================================
- Hits        17107    17088      -19     
- Misses       5027     5046      +19     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5108?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_xlnet.py](https://codecov.io/gh/huggingface/transformers/pull/5108/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ194bG5ldC5weQ==) | `76.35% <0.00%> (-2.30%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5108/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `86.49% <0.00%> (-0.94%)` | :arrow_down: |
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5108/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `75.00% <0.00%> (-0.41%)` | :arrow_down: |
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5108/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `91.06% <0.00%> (+0.12%)` | :arrow_up: |
| [src/transformers/trainer.py](https://codecov.io/gh/huggingface/transformers/pull/5108/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90cmFpbmVyLnB5) | `39.62% <0.00%> (+0.23%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5108?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5108?src=pr&el=footer). Last update [355954f...6330b4e](https://codecov.io/gh/huggingface/transformers/pull/5108?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
using BERT as pretraining with custom classifier
Author: ghost
Pull request: no
Labels: name: wontfix

Hi
I need to pretrain the bert on one dataset and finetune it then on other datasets, so basically
removing classifier from first part and substitute it with a new one with the specific number of 
labels, currently with current codes, it will be error to do it when loading pretrained model,
could you please assist me how I can do this?
thanks
Best
Julia 

    model = model_class.from_pretrained(args.model_name_or_path, from_tf=bool('.ckpt' in args.model_name_or_path), config=config)
  File "julia/libs/anaconda3/envs/transformers/lib/python3.5/site-packages/pytorch_transformers/modeling_utils.py", line 461, in from_pretrained
    model.__class__.__name__, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for BertRUBIForSequenceClassification:
	size mismatch for classifier.weight: copying a param with shape torch.Size([174, 768]) from checkpoint, the shape in current model is torch.Size([3, 768]).
	size mismatch for classifier.bias: copying a param with shape torch.Size([174]) from checkpoint, the shape in current model is torch.Size([3]).


Comments:
Hi, it seems you have saved a model with a classification head of dimension `174 x 768`. You're then trying to load this model with a different classification head of dimension `3 x 768`, is that correct?

If you are trying to save/load the model without the classification head, you can simply save the BertModel **without the classification head**, and then load it from here.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


@LysandreJik : I have the same problem, I trained the ner model and now want to fine tune on other datasets.
`Error(s) in loading state_dict for XLMRobertaForTokenClassification:
	size mismatch for classifier.weight: copying a param with shape torch.Size([9, 768]) from checkpoint, the shape in current model is torch.Size([24, 768]).
	size mismatch for classifier.bias: copying a param with shape torch.Size([9]) from checkpoint, the shape in current model is torch.Size([24])`

Can you please let me know how to "If you are trying to save/load the model without the classification head, you can simply save the BertModel without the classification head, and then load it from here."
getting random results when running run_glue
Author: rabeehk
Pull request: no
Labels: name: wontfix

Hi
I am running run_glue.py script on RTE dataset with BERT base model, on different gpus and I am getting very random results, like changing a lot by changing the gpu. I am using python 3.6 and transformer version 2.5.0. I tried with two gpu types like 
Kepler , GTX1080ti and P40, ...
such randomness really affects the benchmarking and appreciate your help. thanks
I have a deadline and appreciate your prompt response.
thanks.
Best
Rabeeh

Comments:
Hi
Any comment on this?  I also tested with these versions, run_glue with BERT gets fully random results. Like this I cannot run experiments, could you please have a look?
 
python                    3.6.9                h265db76_0  
pytorch                   1.2.0           py3.6_cuda10.0.130_cudnn7.6.2_0    pytorch
torchvision               0.4.0                py36_cu100    pytorch
transformers              2.5.0                     <pip>

thanks


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
BART(base) - Finetune Is this a bug ? Or I am doing something wrong?
Author: ShoubhikBanerjee
Pull request: no

## I have finetuned my dataset using latest piece of code from master, by cloning and the building. 

Now when I try to load my finetuned model as shown in the examples of "summarization" (previous commit).

`from transformers import BartTokenizer, BartForConditionalGeneration, BartConfig
model = BartForConditionalGeneration.from_pretrained('FinetuneOutput/best_tfmr')
tokenizer = BartTokenizer.from_pretrained('FinetuneOutput/best_tfmr')`

I got the following error : 

`TypeError: special token mask_token has to be either str or AddedTokenFast but got: <class 'dict'>`

Am I missing something, or it is a BUG again ? 


Comments:
Can you post the full error message?

Are you asking for this or more??
```bash
TypeError                                 Traceback (most recent call last)

<ipython-input-13-31ed8b67b601> in <module>()
      2 # see ``examples/summarization/bart/run_eval.py`` for a longer example
      3 model = BartForConditionalGeneration.from_pretrained('FinetuneOutput/best_tfmr')
----> 4 tokenizer = BartTokenizer.from_pretrained('FinetuneOutput/best_tfmr/')
      5 model.eval()

5 frames

/usr/local/lib/python3.6/dist-packages/transformers/tokenization_utils.py in __init__(self, **kwargs)
    511                 else:
    512                     raise TypeError(
--> 513                         "special token {} has to be either str or AddedTokenFast but got: {}".format(key, type(value))
    514                     )
    515 

TypeError: special token mask_token has to be either str or AddedTokenFast but got: <class 'dict'>
```

I can't reproduce this on master.

Can you work with a stable release version or do you need features that are on master only?

I am working from your master branch. 

After commit no #5227 ,  its not working, previously it was working fine.

There was also an example to load from "best_tfmr". But now its missing from  [this](https://github.com/huggingface/transformers/tree/master/examples/summarization)  readme

Ok, we are fixing some issues related to tokenizer serialization here: https://github.com/huggingface/transformers/pull/5056

Maybe it will solve your problem as well. Should be merged pretty soon.

Thanx a lot Sir.

After "factory resetting" colab, and build the project from the source again, solved the issue.

I am really sorry for the inconvenience and appreciate the time you gave me. 

Thank a lot. From next time on wards I will keep in mind to "factory reset" colab.  :)