noamaanMulla-03/transformers-issues huggingface dataset:
how to set the init learning rate when use bertAdam?
Author: mmxuan18
Pull request: no
Labels: name: wontfix

i set the BertAdam learning rate as the default value of args (3e-5), and i step in to the BertAdam step by step , and print lr_scheduled see that the acturly lr is very small over all the training process (between <0 ~ 1> * 3e-5), this cause the loss decrease very slow, when i set the init learning rate as 0.1, the loss decrease much more fast, so what's the right way to set the learning rate for BertAdam param?

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
`bert-base-uncased` works for CoLA, `bert-large-uncased` always predicts one class
Author: rococode
Pull request: no
Labels: name: wontfix

I'm having an issue with CoLA where finetuning off of bert-large results in a model that only predicts one class. 

I make one change in configs to train the large model - I set `train_batch_size` to `16` for `bert-large-uncased`. 

These are the two training commands I use (missing do_lowercase, I know, but it's forced anyways):

```
python ./run_classifier.py \
    --task_name CoLA \
    --do_train \
    --data_dir ./data/cola/ \
    --bert_model bert-large-uncased \
    --output_dir ./out/cola-finetune-large-uncased/
```

```
python ./run_classifier.py \
    --task_name CoLA \
    --do_train \
    --data_dir ./data/cola/ \
    --bert_model bert-base-uncased \
    --output_dir ./out/cola-finetune-base-uncased/
```

---

Now here are the results I get with base vs large models:

Here are my results evaluating with plain bert-large-uncased
```
eval_loss = 0.6977422047745098
mcc = -0.05490997894843018
```

Here are the results with bert-base-uncased
```
eval_loss = 1.013142795273752
mcc = 0.02904813156816523
```

Now here are the results with fine-tuning on bert-base-uncased: 
```
eval_loss = 0.590644522203189
mcc = 0.5313406823271718
```

Pretty much reflects what the paper says, nice. But when I do the exact same process, training on bert-large-uncased and with a slightly smaller batch size (16 instead of 32) b/c of GPU memory limitations, I get these results:

```
eval_loss = 0.61876411058686
mcc = 0.0
```

Just to be clear, in these examples I only change the `bert_model` flag from `bert-large-uncased` to `bert-base-uncased` and change the batch size when training, no other changes at all.

I feel I must be doing something wrong. I'm using `run_classifier.py` from this repo. Any ideas what could be the problem?

I've read there's some instability with BERT-Large for small datasets such as CoLA, but surely it doesn't degenerate this much? If I understand correctly mcc = 0 indicates random guessing... Or is that actually the case, and I just need to run it more times and cross my fingers for a non-degenerate run?

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
How to set the token_type_ids in XLNet correctly?
Author: Dongfeng-He
Pull request: no

Hi, I am fine-tuning an XLNet, and I want to use type embedding to indicate different parts of a sequence. I am facing a difficulty that `“indices are selected in the vocabulary (unlike BERT which has a specific vocabulary for segment indices)”` which is the description of `token_type_ids` in the official document. Does that mean the type embedding and token embedding share the same vocabulary? In that case, how can I select the right indices for the types?  If I use `0` and `1` for types, is there a collision between types and the special tokens (like `UNK`)?

Thanks in advance!


Comments:
Hello! We have an example using `token_type_ids` in our `run_glue` script. You can look at how we build the features in the  [`utils_glue`, especially concerning the `segment_ids`](https://github.com/huggingface/pytorch-transformers/blob/master/examples/utils_glue.py#L456-L484) which are the `token_type_ids` that will be fed to the model. 

If I recall correctly the XLNet model has `0` for the first sequence `token_type_ids`, `1` for the second sequence, and `2` for the last (cls) token.


> Hello! We have an example using `token_type_ids` in our `run_glue` script. You can look at how we build the features in the [`utils_glue`, especially concerning the `segment_ids`](https://github.com/huggingface/pytorch-transformers/blob/master/examples/utils_glue.py#L456-L484) which are the `token_type_ids` that will be fed to the model.
> 
> If I recall correctly the XLNet model has `0` for the first sequence `token_type_ids`, `1` for the second sequence, and `2` for the last (cls) token.

Thank you for your explanation!

I still have a question, the code in `utils_glue` is for BERT. As far as I know, the token embeddings and type embeddings are selected from two embedding matrices in BERT, therefore, the type index `0` won't give you a `PAD` token embedding. In XLNet, the type indices are selected in the vocabulary, in which `0` index represents `UNK` token and `1` index represents `BOS` token. 

Do I misunderstand the meaning of `"indices are selected in the vocabulary"`, or we can freely use the `BOS` `EOP` `EOD` tokens for our type embeddings?


Oh, that's a typo in XLNet docstring that I thought we had corrected already.
Thanks for reminding us of that.

The type indices in XLNet are not selected in the vocabulary, they can be arbitrary.
In XLNet segment ids (what we call `token_type_ids in the repo) don't correspond to embeddings, they are just numbers and the only important thing is that they have to be different for tokens which belong to different segments, hence the flexibility in the exact values (XLNet is using relative segment difference with just two segment embeddings: 0 if the segment id of two tokens are the same, 1 if not). See [here](https://github.com/huggingface/pytorch-transformers/blob/master/pytorch_transformers/modeling_xlnet.py#L926-L928).
How to reduce embedding size from 768?
Author: kbulutozler
Pull request: no
Labels: name: wontfix

I am using simple_lm_finetuning.py to fine tune BERT. However I want to get smaller embeddings. Where can I change this?

Comments:
You need to retrain it with your embeddings replacing `BertModel.embeddings.word_embeddings` and the model size being your embeddings size.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Citing DistilBERT
Author: rishibommasani
Pull request: no

Currently, my understanding is citing the repo/codebase should be done by via a link (i.e. in the paper as a footnote) as there is no citation (i.e. in BibTeX style) yet. 
For citing DistilBERT (the released model and distillation approach), how should this be done? 

Comments:
Hello @rishibommasani 
Thank you for your question.
For citing DistilBERT, you're right, there is no formal write-up like an arXiv paper yet (it's definitely in our TODO stack). For the moment, I would recommend citing the blogpost as an URL.
Victor
Usual loss when pretraining?
Author: PedroUria
Pull request: no
Labels: name: wontfix

We are pretraining on our own corpus using the `pregenerate_training_data.py` and `finetune_on_pregenerated.py` scripts.

The input text to the first script follows the same format as `sample_text.txt` on the samples folder, and contains about 515000 lines of text. We run `finetune_on_pregenerated.py` for 60 epochs with the default learning rate (3e-5), 128 sequence length, 300 batch size using 8gpus. The loss got to 1.3. We used this model for a classification task and we didn't see any difference than using the original pretrained BERT. We also compared the weights of some attention layers and they are very similar.

Do you have an estimate as to what the loss should be in order to see improvements? We are also aware we should use 512 for some of the pretraining process because most of our input sequences are 512 long, but still we were expecting some kind of change.

Also, do you think it might be a problem with the small size of our corpus?


Comments:
@PedroUria 

We're in a similar boat as you are. In our case,  the problem was with accuracy though.

We used `BertForSequenceClassification` on  a multi-label classification task. We've actually written a similar version of `BertForSequenceClassification` as written in `models.py` of this repository changing `CrossEntropyLoss` with `BCELogitLoss`.

Initially we faced some issues with loading pre-trained weights, but once we did sucessfully and started training ( 8 label target ), even though our loss started off very small `0.24` in first step of an epoch and went down to `0.023` we never saw accuracy on training set more than `0.71` and validation `.81` in a specific batch. In comparision we have much smaller data set than yours , actually not even close : `350 rows training` and `120 rows` evaluation. So, it's likely that the poor results we're experiencing is because of the minimal data size.

We ran for only 4 epochs though, we also used `Cyclic_LR` for scheduling with `BertAdam` optimizer with same learning rate as yours. I did create an issue looking for answers. Hope they'll respond.



Are you finetuning on the data afterwards? If yes then I believe it is due to the very small dataset. Try to finetune on a larger dataset on a similiar task first and then finetune on your small dataset.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Question on the position embedding of DistilBERT
Author: gpengzhi
Pull request: no

## ❓ Questions & Help

To the best of my knowledge, sinusoidal position embedding is used in the training procedure of DistilBERT, which is computed by [create_sinusoidal_embeddings](https://github.com/huggingface/pytorch-transformers/blob/master/pytorch_transformers/modeling_distilbert.py#L52).

When I compute the position embedding by using `create_sinusoidal_embeddings` with `n_pos=512` and `dim=768`, I got the following position embedding tensor:

```python
Parameter containing:
tensor([[ 0.0000e+00,  1.0000e+00,  0.0000e+00,  ...,  1.0000e+00,
          0.0000e+00,  1.0000e+00],
        [ 8.4147e-01,  5.4030e-01,  8.2843e-01,  ...,  1.0000e+00,
          1.0243e-04,  1.0000e+00],
        [ 9.0930e-01, -4.1615e-01,  9.2799e-01,  ...,  1.0000e+00,
          2.0486e-04,  1.0000e+00],
        ...,
        [ 6.1950e-02,  9.9808e-01,  5.3551e-01,  ...,  9.9857e-01,
          5.2112e-02,  9.9864e-01],
        [ 8.7333e-01,  4.8714e-01,  9.9957e-01,  ...,  9.9857e-01,
          5.2214e-02,  9.9864e-01],
        [ 8.8177e-01, -4.7168e-01,  5.8419e-01,  ...,  9.9856e-01,
          5.2317e-02,  9.9863e-01]])
```

However, when I looked into the position embedding from the pre-trained DistilBERT checkpoint files (`distilbert-base-uncased` and `distilbert-base-uncased-distilled-squad`,), I got the position embedding tensor as follows:

```python
tensor([[ 1.7505e-02, -2.5631e-02, -3.6642e-02,  ...,  3.3437e-05,
          6.8312e-04,  1.5441e-02],
        [ 7.7580e-03,  2.2613e-03, -1.9444e-02,  ...,  2.8910e-02,
          2.9753e-02, -5.3247e-03],
        [-1.1287e-02, -1.9644e-03, -1.1573e-02,  ...,  1.4908e-02,
          1.8741e-02, -7.3140e-03],
        ...,
        [ 1.7418e-02,  3.4903e-03, -9.5621e-03,  ...,  2.9599e-03,
          4.3435e-04, -2.6949e-02],
        [ 2.1687e-02, -6.0216e-03,  1.4736e-02,  ..., -5.6118e-03,
         -1.2590e-02, -2.8085e-02],
        [ 2.6413e-03, -2.3298e-02,  5.4922e-03,  ...,  1.7537e-02,
          2.7550e-02, -7.7656e-02]])
```

I am wondering if I missed or misunderstood something in details. Why is there a difference between these two position embedding tensors? I think the sinusoidal position embedding should be unchanged during training.

Thanks a lot!



Comments:
Hello @gpengzhi 
You're right, the name is quite confusing: the second matrix of embeddings that you're showing is actually initialized from `bert-base-uncased` (compare with `bert = BertModel.from_pretrained('bert-base-uncased'); print(bert.embeddings.position_embeddings.weight)`). Once initialized, these position embeddings are frozen (both distillation or fine-tuning)
Victor
Unable to load DistilBertModel after training
Author: dalefwillis
Pull request: no

## ❓ Questions & Help


I'm following the example to train a DistilBert model from scratch from:  examples/distillation/README.md

I perform the training step:

```
python examples/distillation/train.py --dump_path ser_dir/sm_training_1 --data_file data/sm_bin_text.bert-base-uncased.pickle --token_counts data/sm_token_counts.bert-base-uncased.pickle --force --fp16
```

Which completes successfully, however the `dump_path` doesn't contain a `pytorch_model.bin` file so I cannot load in the model:

```py
from pytorch_transformers import DistilBertModel

DistilBertModel.from_pretrained("ser_dir/sm_training_1")
Model name 'ser_dir/sm_training_1' was not found in model name list (distilbert-base-uncased, distilbert-base-uncased-distilled-squad). We assumed 'ser_dir/sm_training_1/pytorch_model.bin' was a path or url but couldn't find any file associated to this path or url.
Traceback (most recent call last):
...
OSError: file ser_dir/sm_training_1/pytorch_model.bin not found
```

Content of serialization directory:

```sh
ls ser_dir/sm_training_1/
checkpoint.pth  config.json  git_log.json  log  model_epoch_0.pth  model_epoch_1.pth  model_epoch_2.pth  parameters.json
```

I also tried to load it as a regular `BertModel`. Is there something else I need to do to load in a basic .bin file then apply the checkpoint weights on top? I cannot find any tutorials or examples specifying the next steps on this process.

Thanks!

Comments:
Hi @dalefwillis,
You simply have to rename your last checkpoint (I guess in your case it's _"model_epoch_2.pth"_) to _"pytorch_model.bin"_ --> `mv model_epoch_2.pth pytorch_model.bin`.
I updated the training code so that the very last _"model_epoch_*.pth"_ checkpoint is also saved as _"pytorch_model.bin"_ so that you don't have to do this manip manually.

Thanks for the fast response! I just tried it and it works.
Get Attention Values for Pretrained Model
Author: Sparkier
Pull request: no
Labels: name: wontfix

When using BertModel.from_pretrained, I am not able to have it also return the attention layers. Why does that not word? Am I doing something wrong?

Comments:
You need to install the master version (not with pip or conda) : 
```
git clone https://github.com/huggingface/pytorch-pretrained-BERT.git
cd pytorch-pretrained-BERT
python setup.py install
```

Then you can use it like this : 
```
model = BertModel.from_pretrained('bert-base-uncased',
                                   output_attentions=True,
                                   keep_multihead_output=True)
model.eval()  # turn off dropout layers
attn = model(tokens)[0]
```

Tell me if I'm misinterpreting your problem

Thank you a lot for the help, I didn't expect this to only work on the current release. 

However, I think with this I found a problem in the BERT encoder module:

```
def forward(self, hidden_states, attention_mask, output_all_encoded_layers=True, head_mask=None):
        all_encoder_layers = []
        all_attentions = []
        for i, layer_module in enumerate(self.layer):
            hidden_states = layer_module(hidden_states, attention_mask, head_mask[i])
```
The forward function by default gets `None` for the `head_mask` parameter. Then, however, it indexes it, which causes an error. I think it would be nice to handle this case.

Hi. I want to do something similar but with the **BertForQuestionAnswering** model.

The BertModel is the general BERT model that is used to classify whether a sentence is the next sentence or not. I want to get the attention values for QuestionAnswering while I pass a new paragraph and a question as inputs. I want to use the **BertForQuestionAnswering** model (which is pretrained on SQuAD if I am not wrong) and get the self-attention values on the question words. Is it possible to achieve this in a similar way as mentioned above?

**NOTE:** I know the above method gives attention values of the pre-trained model. I want to get attention values of the model when I feed a new input question to the model. Something similar to what can be done using [BertViz](https://github.com/jessevig/bertviz) (although I do not want to visualize attention, just want to get the values).

Thanks.

Hi, this will be in the next release (release date sometime next week).
There will be attention/hidden-state output options for all the models.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Question regarding crossentropy loss function for BERTMaskedLM 
Author: chithangduong
Pull request: no
Labels: name: wontfix

How does BERT handle large number of classes to predict? The number of classes is essentially the vocabulary size which is  30522 for the BERT-base model. When BERT tries to predict a word using CrossEntropy loss, it needs to compute the softmax for a large number of classes. 

In shallow approach such as word2vec, negative sampling or hierarchical softmax is used. I wonder why it is not the case for BERT.
Thanks in advance.

Comments:
30k is ok for a softmax, it's not that much and that because Bert is using a sub-word (open-)vocabulary.

Full word (and closed-vocabulary) models like word2vec have to handle several 100k words hence the specific speed-ups. They are also older and the computation power available at the time was more constrained.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
how to finetuning with roberta-large
Author: zhaoguangxiang
Pull request: no

## 🐛 Bug

<!-- Important information -->

Model I am using (Roberta-large):

Language I am using the model on (English, ):

The problem arise when using:
* [ ] my own modified scripts: (give details)
export CUDA_VISIBLE_DEVICES=2
export GLUE_DIR=/home/zhaoguangxiang/bert/glue_data
DATA=MNLI
NUM_CLASSES=3
LR=1e-5
MAX_SENTENCES=32
TOTAL_NUM_UPDATES=123873
WARMUP_UPDATES=7432
for seed in 42
do
    python3 examples/run_glue.py \
    --model_type bert \
    --model_name_or_path https://s3.amazonaws.com/models.huggingface.co/bert/roberta-large-pytorch_model.bin \
    --task_name ${DATA} \
    --do_train \
    --do_eval \
    --do_lower_case \
    --data_dir $GLUE_DIR/${DATA} \
    --save_steps 10000 \
    --logging_steps 1000 \
    --max_seq_length 512 \
    --max_steps ${TOTAL_NUM_UPDATES}  --warmup_steps ${WARMUP_UPDATES} --learning_rate ${LR} \
    --per_gpu_eval_batch_size 32   \
    --per_gpu_train_batch_size 32   \
    --seed ${seed} \
    --output_dir checkpoint/roberta_${DATA}_output_seed${seed}/
done
The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)

## To Reproduce

Steps to reproduce the behavior:

1. run scripts
2. i will see 
![image](https://user-images.githubusercontent.com/17742385/64612608-cca88800-d406-11e9-82d8-3e45e02f8577.png)


<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior

load or download roeberta and s tart  training

## Environment

* PyTorch Transformers version (or branch): latest
* Using GPU ? yes
* Distributed of parallel setup ? no



Comments:
What do you mean by "latest" version of PyTorch tranformers.

Are you using a release or installing from source from master?

> What do you mean by "latest" version of PyTorch tranformers.
> Are you using a release or installing from source from master?
Thanks for your reply.
pytorch transformers version 1.1.0;
 downloading the code from master;
examples of finetuning roberta were not given in the docs.



this is not a bug,   since i write the wrong model_type, that should be 'roberta'
How to deal with oov tokens with pretrained models
Author: skurzhanskyi
Pull request: no

## ❓ Questions & Help

Can you, please, give an advice on how to handle out of vocabulary word? Just to use `[UNK]` token or there is a way to add this token to vocabulary and thus to train embedding for it?
Also, I noticed that oov words by default return multiple tokens. In my task (sequence tagging) I would like to save token-label correspondence.

Comments:
Hello! We have a method called `add_tokens` in our tokenizers that does just that. [Here's the relevant information](https://huggingface.co/pytorch-transformers/main_classes/tokenizer.html#pytorch_transformers.PreTrainedTokenizer.add_tokens) in the documentation.

Thanks a lot for your answer. That's exactly what I was looking for.
changes in evaluate function in run_lm_finetuning.py
Author: SKRohit
Pull request: yes

changed the return value of `evaluate` function from `results` to `result` and also removed unused empty dict `results`

Comments:
# [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1229?src=pr&el=h1) Report
> Merging [#1229](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1229?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/pytorch-transformers/commit/84d346b68707f3c43903b122baae76ae022ef420?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1229/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1229?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #1229   +/-   ##
=======================================
  Coverage   81.22%   81.22%           
=======================================
  Files          57       57           
  Lines        8027     8027           
=======================================
  Hits         6520     6520           
  Misses       1507     1507
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1229?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1229?src=pr&el=footer). Last update [84d346b...4b082bd](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1229?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Thanks!