noamaanMulla-03/transformers-issues huggingface dataset:
Strange bug when Finetuning own pretrained model (with an even stranger solution)
Author: aditya-malte
Pull request: no
Labels: name: wontfix

name: Should Fix

# 🐛 Bug

## Information
Roberta

Language I am using the model on (English, Chinese ...): Latin script(migh have a mix of languages)

The problem arises when using:
run_glue on model obtained from run_language_modeling


The tasks I am working on is:
Sequence Classification(single)

Steps to reproduce the behavior:

1. Train model using run_language_modeling
2. Use trained model in run_glue script

Error:
File "run_glue.py", line 148, in train
    optimizer.load_state_dict(torch.load(os.path.join(args.model_name_or_path, "optimizer.pt")))
  File "/usr/local/lib/python3.6/dist-packages/torch/optim/optimizer.py", line 116, in load_state_dict
    raise ValueError("loaded state dict contains a parameter group "
ValueError: loaded state dict contains a parameter group that doesn't match the size of optimizer's group 
Note: I searched what might cause this error(freezing some layers and passing an incorrect params_group). But I have not done anything like that, so this error should not occur. 

## Quick Hack/Solution:
There is a strange solution simply by deleting optimizer.pt and setting number of epochs to an arbitrarily large number. Not setting epochs to a very high number causes the script to proceed directly to evaluation and not do any training.

## Environment info
Google Colab
Tokenizers 0.5
Transformers 2.5
GPU:P4

Comments:
Hi! This is an interesting use-case, I think the error stems from the `run_glue` script trying to re-use the different attributes the `run_language_modeling` script had saved.

That includes:
- the optimizer state
- the scheduler state
- the current global step, which is inferred from the name

Your patch works because 
1) the optimizer state shouldn't be kept across different trainings. Deleting the optimizer file makes sense.
2) The script believes you're already at a very high global step, as inferred from the name of your file. Setting a very high number of epochs means a very high number of steps to complete the training, hence some remaining steps.

We should work to fix the issue, but for now I would recommend deleting the files you don't need (`optimizer.pt` and `scheduler.pt`), and rename your folder containing your model/config/tokenizer files so that it doesn't end with a number.

Maybe we could raise a warning after pretraining is over. Ideally, this should be handled by the script itself, and such deletion etc. should not be required 

Yes, I was also stuck on this issue. @LysandreJik , kudos to your hack.

Stuck in the same issue too. Thanks for your suggestion @LysandreJik 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Zero shot multilingual BERT
Author: paulthemagno
Pull request: no
Labels: name: wontfix

❓ Questions & Help

I have a doubt about the usage of multilingual BERT.

I did a domain adaptation on the language model **[`BERT-Base, Multilingual Cased`](https://storage.googleapis.com/bert_models/2018_11_23/multi_cased_L-12_H-768_A-12.zip)** with a dataset of a slightly different kind of text. This dataset is unbalanced on English language but contains also other languages as Italian.

Then I did a finetuning on an Italian dataset on NER.

Has this kind of training a name, like **zero shot** classification (beacuse I adapted on a multilingual dataset but unbalanced on English and then we finetune on Italian)?
Or maybe the 0 shot would be in the case in which I finetuned with an Italian dataset and then I evaluated on a corpus of another language?

Thanks

Comments:
I think what you did there is probably best described as transfer learning.

> Or maybe the 0 shot would be in the case in which I finetuned with an Italian dataset and then I evaluated on a corpus of another language?

Yeah, this is closer to the way the term "zero-shot" is being used in the field right now. Here's a [recent example](https://arxiv.org/abs/1812.10464). 

I'll also note that the way I've seen "zero shot learning" used traditionally was pretty narrow: it meant training a classifier on one set of labels and then evaluating on a different set of labels on in-domain data. Recently, especially in NLP, it's often been used more broadly to mean "do a task that the model wasn't explicitly trained on without additional fine tuning", e.g. in the GPT-2 paper.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Memory leakage with bert-large-uncased-whole-word-masking-finetuned-squad
Author: ashispapu
Pull request: no

# 🐛 Bug

## Information
I am using 'bert-large-uncased-whole-word-masking-finetuned-squad'  and observed an memory leak  during the inference. Below the code snippet to reproduce it

tokenizer = AutoTokenizer.from_pretrained("bert-large-uncased-whole-word-masking-finetuned-squad")
model = AutoModelForQuestionAnswering.from_pretrained("bert-large-uncased-whole-word-masking-finetuned-squad")

device = 'cuda'
question = 'what is your question'
answer = 'This is the  answer'

 def tokenize_qa(tokenizer,question, answer_context):
    input_ids =  tokenizer.encode(question, answer_context, max_length=512)
    tokens =  tokenizer.convert_ids_to_tokens(input_ids)
    sep_index = input_ids.index( tokenizer.sep_token_id)
    num_seg_a = sep_index + 1
    num_seg_b = len(input_ids) - num_seg_a
    segment_ids = [0] * num_seg_a + [1] * num_seg_b
    assert len(segment_ids) == len(input_ids)
    return input_ids, segment_ids, tokens

###################### Inference Code  ##########################
input_ids, segment_ids, tokens= tokenize_qa(tokenizer,question,answer_context)
start_scores, end_scores = model(torch.tensor([input_ids], device= device),token_type_ids=torch.tensor([segment_ids], device= device))

When i run the inference code multiple times it accumulates large  gpu memory and finally runs OOM. Please check let me know if anything wrong here.

Thank you.

Language I am using the model : English,
     
- `transformers` version: '2.11.0'
- Platform: Ubuntu 18.04
- Python version:3.6.9
- PyTorch version (GPU): 1.4.0


Comments:
Hi, @ashispapu, you might wanna wrap the inference code with
`with torch.no_grad():`

Thanks for the response. Should not it be grad disabled by default like other models  in transformer during inference.

I don't think so, how would the model know that you are doing training or inference. You can use `pipeline` for question answering, that takes care of such things
BERT and cross entropy
Author: alshahrani2030
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

How can I feed probability of classes to BERT as label? For example, in sentiment analysis
let say we have this sentence "I like to sleep" instead of 0 or 1 I want to label it as 0.6 negative and 0.4 positive.

Thank you in advance

Comments:
Yes , you can use a MSEloss

thank you for replying me.
Can I use cross entropy Loss, since BERT use it, and replacing the one hot vector by the weight of each class?

SO, What are the labeles of classes?

1 for positive and 0 for negative

I think it may work well ,and you may use NLLloss, it will be seem as a regression problem,
You can try mse loss ,I think mse loss will have better proformence

Why "seen as a regression problem" the output still 1(positive) or 0(negative). The reason of multiplying by weight is to help the model to generalized better and avoiding overconfident .  

![CrossEntropy](https://user-images.githubusercontent.com/55197626/72849700-44b87180-3c75-11ea-933d-b619cfaf9598.PNG)
I am try to use like the in the figure (https://pytorch.org/docs/master/nn.html#crossentropyloss) and I am not sure what is the best way to pass the weight tensor to the model in order CrossEntropyLoss to use it.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


I would suggest adding a class_weight parameter to `BertForSequenceClassification`. This should be an easy fix 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Ability to pickle/unpickle BatchEncoding pickle (reimport)
Author: mfuntowicz
Pull request: yes

Overrides the methods get_state() & set_state() to (respectively) export the content of the underlying data dictionnary and - if defined - the content of encodings.

Unittests added to covert the serialization & deserialization of all the exported properties.

Reimported from #4515

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5039?src=pr&el=h1) Report
> Merging [#5039](https://codecov.io/gh/huggingface/transformers/pull/5039?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/36434220fc807c5015bc8f0f1e50ab21f7d34914&el=desc) will **increase** coverage by `0.00%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5039/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5039?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #5039   +/-   ##
=======================================
  Coverage   77.36%   77.37%           
=======================================
  Files         130      130           
  Lines       21989    21998    +9     
=======================================
+ Hits        17012    17021    +9     
  Misses       4977     4977           
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5039?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/tokenization\_utils\_base.py](https://codecov.io/gh/huggingface/transformers/pull/5039/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fdXRpbHNfYmFzZS5weQ==) | `91.69% <100.00%> (+0.13%)` | :arrow_up: |
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5039/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `75.00% <0.00%> (-0.41%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5039/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `87.42% <0.00%> (+0.15%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5039?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5039?src=pr&el=footer). Last update [3643422...5e40fe4](https://codecov.io/gh/huggingface/transformers/pull/5039?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
refactor(wandb): consolidate import
Author: borisdayma
Pull request: yes

This PR consolidates the import logic of wandb as suggested [here](https://github.com/huggingface/transformers/pull/4946#discussion_r440070708)

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5044?src=pr&el=h1) Report
> Merging [#5044](https://codecov.io/gh/huggingface/transformers/pull/5044?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/f9f8a5312e92541ff9a5f483fc4907ec87da876e&el=desc) will **increase** coverage by `0.01%`.
> The diff coverage is `64.28%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5044/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5044?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #5044      +/-   ##
==========================================
+ Coverage   77.39%   77.40%   +0.01%     
==========================================
  Files         130      130              
  Lines       22018    22014       -4     
==========================================
  Hits        17041    17041              
+ Misses       4977     4973       -4     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5044?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/trainer\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5044/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90cmFpbmVyX3V0aWxzLnB5) | `78.26% <50.00%> (-21.74%)` | :arrow_down: |
| [src/transformers/trainer.py](https://codecov.io/gh/huggingface/transformers/pull/5044/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90cmFpbmVyLnB5) | `39.52% <100.00%> (-0.06%)` | :arrow_down: |
| [src/transformers/trainer\_tf.py](https://codecov.io/gh/huggingface/transformers/pull/5044/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90cmFpbmVyX3RmLnB5) | `18.69% <100.00%> (-0.30%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5044?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5044?src=pr&el=footer). Last update [f9f8a53...47b9975](https://codecov.io/gh/huggingface/transformers/pull/5044?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Thanks!
Add reference to NLP (package) dataset
Author: mrm8488
Pull request: yes
Labels: name: model card


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5029?src=pr&el=h1) Report
> Merging [#5029](https://codecov.io/gh/huggingface/transformers/pull/5029?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/bbad4c6989d489097f42bbe38001a3f8ca1c5c11&el=desc) will **decrease** coverage by `0.00%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5029/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5029?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #5029      +/-   ##
==========================================
- Coverage   77.19%   77.18%   -0.01%     
==========================================
  Files         128      128              
  Lines       21877    21877              
==========================================
- Hits        16888    16886       -2     
- Misses       4989     4991       +2     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5029?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5029/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `73.38% <0.00%> (-0.41%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5029/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `87.26% <0.00%> (-0.16%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5029?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5029?src=pr&el=footer). Last update [bbad4c6...0df885f](https://codecov.io/gh/huggingface/transformers/pull/5029?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


I forgot to add the meta tag:
datasets:
- squad_v2

Sorry

added the metadata, @mrm8488
Chatbot QnA feature for given text corpus
Author: shashankMadan-designEsthetics
Pull request: no

Is there a way we can have a chatbot specifically which can answer questions on a given text corpus?
Is there any transformer model which we can train for this and how?

Comments:
You might want to check this, It has a very nice example. [https://huggingface.co/transformers/model_doc/bert.html#bertforquestionanswering](https://huggingface.co/transformers/model_doc/bert.html#bertforquestionanswering) 

> You might want to check this, It has a very nice example. https://huggingface.co/transformers/model_doc/bert.html#bertforquestionanswering

Well, I've known this solution for quite some time, this give outputs of start and end logits.
But i'd like to know if there is any implementation like if i just feed in just a corpus of text say a article, essay and so on and ask questions it would give some relevant outout
[How to] Carefully designing the head of a Transformer model?
Author: innat
Pull request: no

# ❓ Questions & Help

While using the pre-trained any transformer model, what are the main things we normally should consider while designing the head? Simply like,

```python
distill = transformer.distilbert.... <----- slicing the first position
x  = Dense(n, activation = ' ')(distill) <-------- simple classifer head
``` 
Is it really necessary to design an additional head? (I'm using `tensorflow` backend.)

Comments:
Hight CPU and low GPU on XLNet
Author: laiviet
Pull request: no
Labels: name: wontfix

## 🐛 Bug

I am running Bert, GPT, GPT2, XLNET. I got very high CPU usage (e.g. 16 cores) with  XLNet while the others (Bert, GPT, GPT2) dont.

For BERT, GPT, GPT2: CPU 1 cores, 100%GPU 
For XLNet: CPU 16 cores, 50 to 60% GPU 


Is there any hidden implementation which requires CPU?

<!-- Important information -->

Model I am using (XLNet....): XLNET

Language I am using the model on: English

The problem arise when using:
* [ ] the official example scripts: (give details)
* [ ] my own modified scripts: (give details)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details)

## To Reproduce

Steps to reproduce the behavior:

1.
2.
3.

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->

## Environment

* OS: Ubuntu 18.04
* Python version: 3.6
* PyTorch version: 1.0
* PyTorch Transformers version (or branch): pytorch_transformers 1.2.0
* Using GPU ? RTX 2080 TI
* Distributed of parallel setup ? No
* Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->


Comments:
I also meet the above problem with XLNet. The GPU usage is very low and unstable, but the CPU usage is very high. The running speed is very low.

Are there any ops running on CPU rather than GPU in your XLNet implementation? How to improve the GPU usage and speed up the running speed ? Thanks!

Environment:
Pytorch: 1.1.0
GPU: V100, 16G
pytorch_transformers: 1.2.0
OS: centos 7.6
Python: 3.6

Does it occur while training or predicting? Are you sure your gpu is available (to PyTorch or TensorFlow)? What do logs say? 

It is likely that this is caused by the tokenisation rather than the model training. Tokenisation typically happens on the CPU, the token_ids are then transferred to the GPU and from then on out the training happens on the GPU. 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


The same problem. How to solve it?
How can I load the finetuned BART model to memory?
Author: tomaszgarbus
Pull request: no

I have finetuned a `facebook/bart-large` model following the example here: https://github.com/huggingface/transformers/blob/master/examples/summarization/finetune.py

As an output I got a `checkpointcheckpoint_ckpt_epoch_0.ckpt` file. How can I create a BartForConditionalGeneration instance with updated weights?

Comments:
Duplicate of #4144 

Thanks!
[cleanup] Hoist ModelTester objects to top level
Author: aretius
Pull request: yes
Labels: name: cleanup

Fixes #4902 
@sshleifer 
All tests seem to pass. Please review!


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4939?src=pr&el=h1) Report
> Merging [#4939](https://codecov.io/gh/huggingface/transformers/pull/4939?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/f9f8a5312e92541ff9a5f483fc4907ec87da876e&el=desc) will **increase** coverage by `0.00%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4939/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4939?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #4939   +/-   ##
=======================================
  Coverage   77.39%   77.40%           
=======================================
  Files         130      130           
  Lines       22018    22018           
=======================================
+ Hits        17041    17042    +1     
+ Misses       4977     4976    -1     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4939?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4939/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `87.26% <0.00%> (-0.16%)` | :arrow_down: |
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4939/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `90.73% <0.00%> (+0.11%)` | :arrow_up: |
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4939/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `75.40% <0.00%> (+0.40%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4939?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4939?src=pr&el=footer). Last update [f9f8a53...16a4a1e](https://codecov.io/gh/huggingface/transformers/pull/4939?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


I think we can investigate code reuse between `ModelTester`s in a separate PR, given the enormity of this one :)

I like the changes! I only checked the files superficially. I trust @aretius and @sshleifer that no tests functionality was changed of got lost :-)

Thanks, @sshleifer :)
Would be glad to pick up code reuse in a separate PR 
Yes @patrickvonplaten no test functionality was lost!

Merge conflict resolved @LysandreJik
Perform evaluation on HANS with Trainer (like GLUE example)
Author: prajjwal1
Pull request: no
Labels: name: Good First Issue

Current [HANS](https://github.com/huggingface/transformers/tree/master/examples/adversarial) evaluation implementation is carried out in the old way. It'd be good to do it in the same manner as other examples are implemented now with the Trainer class. 


Comments:
Maybe @sgugger would be interested in taking a look at this

There is an evaluation issue that I faced with HANS evaluation. Issue [here](https://github.com/huggingface/transformers/issues/4766). I tried to run in the exact same manner as listed in `examples/adversarial`. But the results obtained seem skewed for some reason. If @sgugger can point some possible reason out, I can work on it and send a PR. Thanks.