noamaanMulla-03/transformers-issues huggingface dataset:
Fine-tune distilbert-base-uncased under run_glue
Author: YosiMass
Pull request: no

## ❓ Questions & Help

I am a bit confused on how to fine-tune distilbert. I see three options

1. Fine-tune bert for the task and then use distillation.distiller 
2. Fine-tune bert for the task and then use distillation.train 
3. Fine-tune distilbert-base-uncased directly for the task using run_glue.py

I tried the 3rd option under run_glue.py as follows

add distilbert to MODEL_CLASSES 
```MODEL_CLASSES = {
    'bert': (BertConfig, BertForSequenceClassification, BertTokenizer),
    'distilbert': (DistilBertConfig, DistilBertForSequenceClassification, DistilBertTokenizer),
    'xlnet': (XLNetConfig, XLNetForSequenceClassification, XLNetTokenizer),
    'xlm': (XLMConfig, XLMForSequenceClassification, XLMTokenizer),
    'roberta': (RobertaConfig, RobertaForSequenceClassification, RobertaTokenizer),
}
```
and add a flag `--model_type=distilbert`

Which of the three methods above should be used?

Comments:
Hello @YosiMass,
The simplest/more direct way to do transfer learning is indeed the 3rd solution.
If you use `run_glue.py`, the modification you made is correct. You also have to be careful since DistilBERT doesn't take `token_type_embeddings` as input --> [here](https://github.com/huggingface/pytorch-transformers/blob/master/examples/run_glue.py#L131) and [here](https://github.com/huggingface/pytorch-transformers/blob/master/examples/run_glue.py#L221).
I'll add these modifications in a few days directly to these scripts so that it's seamless to use DistilBERT with run_squad or run_glue.

Thanks @VictorSanh.  

Yes, I handled the missing token_types as follows. In run_glue.train and run_glue.evaluate I changed From  
```            
inputs = {'input_ids':      batch[0],
                'attention_mask': batch[1],
                'token_type_ids': batch[2] if args.model_type in ['bert', 'xlnet'] else None,  # XLM and RoBERTa don't use segment_ids
                'labels':         batch[3]}
```

To
```
if args.model_type not in ['distilbert']:
     inputs = {'input_ids':      batch[0],
                     'attention_mask': batch[1],
                     'token_type_ids': batch[2] if args.model_type in ['bert', 'xlnet'] else None,  # XLM and RoBERTa don't use segment_ids
                     'labels':         batch[3]}
else:
      inputs = {'input_ids':      batch[0],
                      'attention_mask': batch[1],
                      'labels':         batch[3]}
```

> handled the missing token_types as follows. In run_glue.train and run_glue.evaluate I changed From

It looks good to me!
Simple LM finetuning falls with RunTime Error: CUDA out of memory
Author: MNCTTY
Pull request: no
Labels: name: wontfix

I tried to run simple_lm_finetuning.py on my own data with multi lingual uncased model, and the script breaks down with error 'CUDA out of memory'. Can anyone say what should I do in this situation?
I've already decreased batch size from 32 to 2, but even then I get this error.
![image](https://user-images.githubusercontent.com/37251686/60744362-3ecf9d00-9f7e-11e9-8c3c-7abacce122e6.png)

Offtop: does anybody know, if I can use my own pretrained model in this script instead of one of listed ones?

Comments:
How much memory does your GPU have. You can check this by running `nvidia-smi`.

I also has same phenomena. Also, the learning time become slower and much GPU consumption occur, both of which I think is natural, regarding parameters BERT has.

The substitutional way is that, no fine-tuning and dump. 
I mean, feed your sequence to Bert and dump your layers. In the training process of your task, simply load sequence with your dumped result of Bert.

Try `--reduce_memory `, it worked on mine with base multilingual uncased BERT with batch size= 4 on my single 2080Ti.


```
python3 finetune_on_pregenerated.py 
--pregenerated_data training/ 
--bert_model bert-base-multilingual-uncased 
--do_lower_case 
--output_dir finetuned_lm/ 
--epochs 3 
--reduce_memory 
--train_batch_size 4
```

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Redundant sep_token_extra option for RoBERTa Fine-tuning
Author: todpole3
Pull request: no

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): RoBERTa

Language I am using the model on (English, Chinese....): English

## Context

I was reading the code on RoBERTa fine-tuning and noticed the [`sep_token_extra` option](https://github.com/huggingface/pytorch-transformers/search?q=sep_token_extra&unscoped_q=sep_token_extra), which looks like a misinterpretation of a sentence from the original paper.

The current implementation [added an extra `[SEP]` to the RoBERTa input compared to BERT](https://github.com/huggingface/pytorch-transformers/blob/d8923270e6c497862f990a3c72e40cc1ddd01d4e/examples/utils_glue.py#L453), which seems wrong. Check out:

1. The [Facebook language model format](https://github.com/pytorch/fairseq/blob/e75cff5f2c1d62f12dc911e0bf420025eb1a4e33/fairseq/data/legacy/masked_lm_dataset.py#L193)
2. A related [Twitter discussion](https://twitter.com/VictoriaLinML/status/1175596109009321986)

The tasks I am working on is:
Fine-tuning RoBERTa on downstream tasks

## Code Sample
https://github.com/huggingface/pytorch-transformers/search?q=sep_token_extra&unscoped_q=sep_token_extra

Comments:
Myle Ott from Facebook commented on the Twitter thread (https://twitter.com/myleott/status/1175750596630056961) confirming that there is an extra separator being used, so there should be details I did not understand well.

I will revisit this issue when I understand it better.

The `sep_token_extra` param is deprecated, as we have simpler ways to do this now (thanks to @LysandreJik). Closing this for now, feel free to re-open if needed.
fix link
Author: FeiWang96
Pull request: yes


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1362?src=pr&el=h1) Report
> Merging [#1362](https://codecov.io/gh/huggingface/transformers/pull/1362?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/a6a6d9e6382961dc92a1a08d1bab05a52dc815f9?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1362/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1362?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #1362   +/-   ##
=======================================
  Coverage   84.69%   84.69%           
=======================================
  Files          84       84           
  Lines       12596    12596           
=======================================
  Hits        10668    10668           
  Misses       1928     1928
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1362?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1362?src=pr&el=footer). Last update [a6a6d9e...60f7916](https://codecov.io/gh/huggingface/transformers/pull/1362?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


👍
fix padding_idx of RoBERTa model
Author: ikuyamada
Pull request: yes

The padding index of the pretrained RoBERTa model is 1, and 0 is assigned to `
` token. The padding index of the current RoBERTa model is set to be 0, therefore `
` is treated as padding.
This PR aims to fix this problem.

Comments:
LGTM but let's have @julien-c or @LysandreJik confirm

lgtm too

It would be appreciated if you review this PR! @LysandreJik 

@ikuyamada Out of curiosity, in which cases did you need to specify this `padding_idx`? It shouldn't have any impact on the inference so are you training a model from scratch? (with @LysandreJik)

Merged in https://github.com/huggingface/transformers/commit/a6a6d9e6382961dc92a1a08d1bab05a52dc815f9

big drawback is that we initialize the embeddings multiple times.

@ikuyamada do you have an idea to improve this?

Thank you for merging this PR!

> Out of curiosity, in which cases did you need to specify this padding_idx? It shouldn't have any impact on the inference so are you training a model from scratch?

In my understanding, the embedding corresponding to `padding_idx` is not updated while training (pre-training or fine-tuning). Because the embedding of the token `
` may play some roles for computing contextualized embeddings for other tokens, and the output embedding of the `
` token is used for computing a feature vector for some fine-tuning tasks, I think the embedding should be updated while training.

> big drawback is that we initialize the embeddings multiple times.
> @ikuyamada do you have an idea to improve this?

We can avoid this by removing [the constructor call of the `BertEmbeddings`](https://github.com/huggingface/transformers/blob/master/transformers/modeling_roberta.py#L44) and simply initialize the `token_type_embeddings`, `LayerNorm`, and `dropout` in the constructor of the `RobertaEmbeddings`.
If you prefer this implementation, I will create a PR again! :)
@julien-c 

> In my understanding, the embedding corresponding to padding_idx is not updated while training (pre-training or fine-tuning). Because the embedding of the token `
` may play some roles for computing contextualized embeddings for other tokens, and the output embedding of the `
` token is used for computing a feature vector for some fine-tuning tasks, I think the embedding should be updated while training.

Yes you are correct.

> If you prefer this implementation, I will create a PR again! :)

That would be great, thank you.
AdamW does not have args warmup and t_total
Author: sophiapeng0426
Pull request: no
Labels: name: wontfix

In finetune_on_pregenerated.py, below code throw error cause AdamW does not have those two arguments. This can be fixed by comment out those two columns but not sure if that means warmup will be not effective after that?

optimizer = AdamW(optimizer_grouped_parameters,
                             lr=args.learning_rate)
                            #warmup=args.warmup_proportion,
                           # t_total=num_train_optimization_steps)


Comments:
Yes this example should have been updated now by #797.

Regarding `AdamW` and the schedule, details, and examples for the conversion are indicated in the migration section of the readme: https://github.com/huggingface/pytorch-transformers#Migrating-from-pytorch-pretrained-bert-to-pytorch-transformers

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
adaptive softmax in transformer-xl
Author: lepus2
Pull request: no
Labels: name: wontfix

I guess there is some incomplete part for adaptive softmax in [modeling_transfo_xl.py](https://github.com/huggingface/pytorch-transformers/blob/master/pytorch_transformers/modeling_transfo_xl.py)
Actually, it is impossible to build model not using adaptive softmax even though `TransfoXLConfig` has `adaptive` parameter.
I can see that if `sample_softmax` is larger than -1, model uses sample softmax other than adaptive softmax, which seems to be the case of not using adaptive softmax.
However, in case of `sample_softmax` > -1 and `tie_weight`=True, there is a problem on [this line](https://github.com/huggingface/pytorch-transformers/blob/master/pytorch_transformers/modeling_transfo_xl.py#L1317)<code><pre>self.out_layer.weight = self.transformer.word_emb.weight</pre></code> because the model always use `AdaptiveEmbedding` as `word_emb`, which has no `weight` property.

Presumably we need some code for not using adaptive softmax and but using standard softmax and usual `nn.Embedding` as a word embedding.
Then we can tie weights between standard `nn.Embedding` and `nn.Linear` when we doesn't need adaptive softmax.

Can you consider about this problem? Thanks.

Comments:
Yes. At the moment, this library is designed for loading pretrained models mostly and no one has open-sourced a Transformer-XL pretrained model using something else than adaptive softmax so I have not spent time adding these options.

Happy to welcome PR though.

The main thing of interest here, if you want to give it a try would be to add bias to all clusters in PyTorch official [`AdaptiveLogSoftmaxWithLoss`](https://pytorch.org/docs/stable/nn.html?highlight=adaptivelogsoftmaxwithloss#torch.nn.AdaptiveLogSoftmaxWithLoss) module so we could just use the official implementation without maintaining ours.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Detaching Variables
Author: rishibommasani
Pull request: no
Labels: name: wontfix

Something I noticed in transitioning from Pretrained-BERT to Transformers is that for the purposes of using BERT as a feature extractor/probing the pretrained representations, I need to detach variables whereas I previously didn't. I am not sure if this is noted somewhere (I didn't see it in the section in the docs about transitioning) but found it be highly relevant to prevent unnecessary memory usage. 

Comments:
Maybe you are not using `with torch.grad()` when calling the model for inference?

I've added that in the readme example (it used to be mentioned there indeed).

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Get the different result at BertModel
Author: jackhsu2005
Pull request: no
Labels: name: wontfix

At old version pytorch-pretrained-bert :
I used the BertModel to fine-tuned, loss will decrease.

But I used the New version BertModel to use the same data to finetune, but loss won't decrease.


> optimzer 
>I have tried different optimzer AdamW ,BertAdam. 
 
> learning rate
>0.1 0.01 0.001 ... 0.0000000001
 
> Batch Size
>1

> input I put **input_ids** and **token_type_ids**
> At old version return _,CLS,I take CLS
> At new version return [seq_len,1,768],[seq_len,768],I take [0,1,:] or [0,:] loss won't decrease.

I don't know what the detail I miss in the new version?
 

Comments:
Have you read in detail the [migration guide](https://github.com/huggingface/pytorch-transformers#migrating-from-pytorch-pretrained-bert-to-pytorch-transformers) of the readme?

There is also a new `run_glue` example which is an updated version of the previous `run_classifier` and that you can use as a starting point for designing your own fine-tuning scripts.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Test a fine-tuned BERT-QA model
Author: wasiahmad
Pull request: no
Labels: name: Help wanted

I have fine-tuned a BERT-QA model on SQuAD and it produced a `pytorch_model.bin` file. Now, I want to load this fine-tuned model and evaluate on SQuAD. How can I do that? I am using the `run_squad.py` script.

Comments:
I noticed the following snippet in the code. (which I have edited to solve my problem)

    if args.do_train and (args.local_rank == -1 or torch.distributed.get_rank() == 0):
        # Save a trained model, configuration and tokenizer
        model_to_save = model.module if hasattr(model, 'module') else model  # Only save the model it-self

        # If we save using the predefined names, we can load using `from_pretrained`
        output_model_file = os.path.join(args.output_dir, WEIGHTS_NAME)
        output_config_file = os.path.join(args.output_dir, CONFIG_NAME)

        torch.save(model_to_save.state_dict(), output_model_file)
        model_to_save.config.to_json_file(output_config_file)
        tokenizer.save_vocabulary(args.output_dir)

        # Load a trained model and vocabulary that you have fine-tuned
        model = BertForQuestionAnswering.from_pretrained(args.output_dir)
        tokenizer = BertTokenizer.from_pretrained(args.output_dir, do_lower_case=args.do_lower_case)
    else:
        model = BertForQuestionAnswering.from_pretrained(args.bert_model)


So, if we want to load the fine-tuned model only for prediction, need to load it from `args.output_dir`. But the current code loads from `args.bert_model` when we use `squad.py` only for prediction.

@wasiahmad  tokenizer  is not needed at prediction time?

Thanks
Mahesh

need help in understanding how to get the model trained with SQuAD + my dataset. Once trained, how to use it for actual prediction.

model : BERT Question Answering


@Swathygsb 
https://github.com/kamalkraj/BERT-SQuAD
inference on bert-squad model

> @Swathygsb
> https://github.com/kamalkraj/BERT-SQuAD
> inference on bert-squad model

thx for your sharing, and there is inference on bert-squad model by tensorflow?
3Q~
fix redundant initializations of Embeddings in RobertaEmbeddings
Author: ikuyamada
Pull request: yes

Based on the discussion with @julien-c in #1258, this PR fixes the issue of redundant multiple initializations of the embeddings in the constructor of `RobertaEmbeddings` by removing the constructor call of its parent class (i.e., `BertEmbeddings`) and creating `token_type_embeddings`, `LayerNorm`, and `dropout` in the constructor.

Comments:
Sorry, I will fix this
XLMForMaskedLM
Author: Shuailong
Pull request: no
Labels: name: wontfix

Hi, I am currently training a BERT model using facebook XLM framework. I use the script in this repo to convert XLM format to PyTorch format. Is it possible to implement an `XLMForMaskedLM` which is just like `BertForMaskedLM` but use XLM trained BERT instead?

Comments:
You can do that using `XLNetLMHeadModel` and custom masks as shown in the [`run_generation` example](https://github.com/huggingface/pytorch-transformers/blob/master/examples/run_generation.py#L115-L121).

But note that XLNet is rather bad on short text input completions as I discussed in https://github.com/huggingface/pytorch-transformers/issues/846#issuecomment-514228565

> #846 (comment)

Thanks for your reply! However, I mean the `XLM` version of BERT, instead of XLNet. Is it also convenient to do that for XLM?

Thanks!

Oh, right! You can just use `XLMWithLMHeadModel` with an input sequence containing XLM masked token `tokenizer.mask_token` (which is `<special1>` for XLM).

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
XLNet bidirectional input pipeline requires batch size at least 2
Author: langfield
Pull request: no
Labels: name: wontfix

This may not be a true bug since it's mentioned in the paper that 

> each of the forward and backward directions takes half of the batch size

but when using the bidirectional input pipeline, any call to `XLNetModel.forward()` will raise an error of the form 

```
RuntimeError: shape '[x, y, z]' is invalid for input of size 0
```

if the **batch size of the `input_ids` passed is less than 2**. This is because it halves (integer div) `bsz` in accordance with the above quote in the following block:

```
if self.bi_data:
            fwd_pos_seq = torch.arange(beg, end, -1.0, dtype=torch.float)
            bwd_pos_seq = torch.arange(-beg, -end, 1.0, dtype=torch.float)

            if self.clamp_len > 0:
                fwd_pos_seq = fwd_pos_seq.clamp(-self.clamp_len, self.clamp_len)
                bwd_pos_seq = bwd_pos_seq.clamp(-self.clamp_len, self.clamp_len)

            if bsz is not None:
                fwd_pos_emb = self.positional_embedding(fwd_pos_seq, inv_freq, bsz//2)
                bwd_pos_emb = self.positional_embedding(bwd_pos_seq, inv_freq, bsz//2)
```

The result is a batch size of `0`, which obviously wreaks havoc later on. It's only relevant if people are trying to run a MWE with really small input as I was for testing, but maybe an assert statement somewhere near the top of the `XLNetModel.forward()` function is a good idea, conditional on `bi_data` being `True`. 

More generally, a shape mismatch is caused for the same reason if `bi_data` is `True` and `bsz` is any positive odd integer. 

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Best loss
Author: jasonmusespresso
Pull request: no

I am building a classifier by adopting codes from `run_glue.py`. There is plenty of optimization in training and tuning hyperparameters. Could anyone explain the difference between loss, tr_loss and logging_loss in these parts?
https://github.com/huggingface/pytorch-transformers/blob/a2d4950f5c909f7bb4ea7c06afa6cdecde7e8750/examples/run_glue.py#L120
https://github.com/huggingface/pytorch-transformers/blob/a2d4950f5c909f7bb4ea7c06afa6cdecde7e8750/examples/run_glue.py#L134
Thank you.

Comments:
run_openai_gpt.py issues with Adamw
Author: rabeehk
Pull request: no
Labels: name: wontfix

Hi
Adamw in this script has parameters not existing anymore, ...
Thanks for updates in advance.

Comments:
Yes, so this should be fixed by (your own :) PR #845 Thanks again!

thanks :)
Best regards,
Rabeeh

On Tue, Jul 23, 2019 at 3:30 PM Thomas Wolf <notifications@github.com>
wrote:

> Yes, so this should be fixed by (your own :) PR #845
> <https://github.com/huggingface/pytorch-transformers/pull/845> Thanks
> again!
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/pytorch-transformers/issues/837?email_source=notifications&email_token=ABP4ZCAGTX6DNAYMBCHNORTQA4BYFA5CNFSM4IFGMGK2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2TDV6Y#issuecomment-514210555>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABP4ZCGDZR3TG2SS2LXIBTDQA4BYFANCNFSM4IFGMGKQ>
> .
>


Thanks @rabeehk! #845 with flat learning rate gives a good result of 87.2% on ROCStories. Here are the args (the defaults in the file work well).

```
python run_openai_gpt.py  \
  --model_name openai-gpt  \
  --do_train  \
  --do_eval  \
  --train_dataset "./ROCStories/cloze_test_val__spring2016 - cloze_test_ALL_val.csv"  \
  --eval_dataset "./ROCStories/cloze_test_test__spring2016 - cloze_test_ALL_test.csv"  \
  --train_batch_size 8  \
  --eval_batch_size 16  \
  --num_train_epochs  3
```

@prrao87  great :) 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Is there any plan for Roberta in SQuAD?
Author: kugwzk
Pull request: no

## 🚀 Feature

<!-- A clear and concise description of the feature proposal. Please provide a link to the paper and code in case they exist. -->
Hello, thx for the RoBERTa implementation. But I want to know is there any plan for the RoBERTa in SQuAD, because it is complex. And I simple changed the run_squad code as the run_gule code, I got some bugs. And the fairseq doesn't give a official code, too. I really want to know how to use the RoBERTa in SQuAD use the transformers.
## Motivation

<!-- Please outline the motivation for the proposal. Is your feature request related to a problem? e.g., I'm always frustrated when [...]. If this is related to another GitHub issue, please link here too. -->

## Additional context

<!-- Add any other context or screenshots about the feature request here. -->

Comments: