noamaanMulla-03/transformers-issues huggingface dataset:
bug in init_bert_weights
Author: mjc14
Pull request: no

hi ,
    there is a bug  in init_bert_weights(). 
    the BERTLayerNorm  has twice init, the first init  is in the BERTLayerNorm module __init__(). the second init in init_bert_weights().
if you want to get pre-training model that is not from google model, the second init will lead to bad convergence  in my experiment 。 gamma is variance , beta is mean, there  are usually  1 and 0. the second init change it.
first: 
        self.gamma = nn.Parameter(torch.ones(config.hidden_size))
        self.beta = nn.Parameter(torch.zeros(config.hidden_size))
second:
            elif isinstance(module, BERTLayerNorm):
                 module.beta.data.normal_(mean=0.0, std=config.initializer_range)
                module.gamma.data.normal_(mean=0.0, std=config.initializer_range)


Comments:
Fixed, thanks
Using large model with fp16 enable causes the server down
Author: hguan6
Pull request: no

I am using a server with Ubuntu 16.04 and 4 TITAN X GPUs. The server runs the base model with no problems. But it cannot run the large model with 32-bit float point, so I enabled fp16, and the server went down.
(When I successfully ran the base model, it consumes 8G GPU memory for each of the 4 GPUS. )

Comments:
Could you give more informations such as the command that you are using to run the model and the batch size that you are using? Have you tried reducing it?

Hi, @hguan6 try to adjust the batch size and use gradient accumulation (see [this section](https://github.com/huggingface/pytorch-pretrained-BERT#training-large-models-introduction-tools-and-examples) in the readme and the `run_squad` and `run_classifier` examples) if needed.
Question about hidden layers from pretained model
Author: mvss80
Pull request: no

In the example shown to get hidden states https://github.com/huggingface/pytorch-pretrained-BERT#usage
I want to confirm - the final hidden layer corresponds to the last element of `encoded_layers`, right?

Comments:
Yes you are right. The first value returned is the output for `BertEncoder.forward`.

https://github.com/huggingface/pytorch-pretrained-BERT/blob/8da280ebbeca5ebd7561fd05af78c65df9161f92/pytorch_pretrained_bert/modeling.py#L623-L634
Cannot reproduce the result of run_squad 1.1
Author: hmt2014
Pull request: no

I train 5 epochs with learning rate 5e-5, but my evaluation result is {'exact_match': 32.04351939451277, 'f1': 36.53574674513405}.
What is the problem?

Comments:
I can reproduce the results, learning rate is 3e-5 , epoch is 2.0

by using fp16, the f1 is 90.8 

> by using fp16, the f1 is 90.8

So the key is set fp16 is True?

@hmt2014 could you give the exact command line that you use to train your model?

Yes, please use the command line example indicated [here](https://github.com/huggingface/pytorch-pretrained-BERT#squad) in the readme for SQuAD.
How to pretrain my own data with this pytorch code?
Author: Gpwner
Pull request: no

I wonder how to pretrain with my own data.

Comments:
Sir @Gpwner  i think you have to refer to do pretrain for google/bert repo( https://github.com/google-research/bert#pre-training-with-bert ) and then convert tensorflow model as pytorch.

A pre-training script is now included in `master` thanks to @tholor's PR #124 

> A pre-training script is now included in `master` thanks to @tholor's PR #124
Thanks ！！
Does it support Multiple GPU？Because the official script does not support Multiple GPU

It does (you can read more about it [here in the readme](https://github.com/huggingface/pytorch-pretrained-BERT#lm-fine-tuning))

> It does

great job，thanks ～

All thanks should go to @tholor :-)
Why not the mlm use the information of adjacent sentences?
Author: l126t
Pull request: no


I prepare two sentences for mlm predict the mask part:"Tom cant run fast. He [mask] his back a few years ago." The result of model (uncased base) is 'got'. That is meaningless. Obviously ,"hurt" is better.
I wander how to make mlm to use the information of adjacent sentences.

Comments:
The model is already using adjacent sentences to make its predictions, it just happens to be wrong in your case.

If you would like to make it choose from a specific list of words, you could use the code that I mentionned in #80. 

Thanks @rodgzilla !

What 's the mlm accuracy of pretrained model? I find  the scores of candidate in top 10 are very close，but most are not suitable. Is this the same prediction as Google's original project?
What 's the mlm accuracy of pretrained model? 
Author: l126t
Pull request: no

What 's the mlm accuracy of pretrained model? In my case, I find  the scores of candidate in top 10 are very close，but most are not suitable. Is this the same prediction as Google's original project?

_Originally posted by @l126t in https://github.com/huggingface/pytorch-pretrained-BERT/issues/155#issuecomment-452195676_

Comments:
Hi, we didn't evaluate this metric. If you do feel free to share the results.
Regarding the comparison between the Google and PyTorch implementations, please refere to the included Notebooks and the associated section of the readme.
Did you suport squad2.0
Author: zhaoguangxiang
Pull request: no

What is the command to reproduce the results of squad2.0 reported in the  BERT.  
Thanks~

Comments:
This is still not supported but should be soon. You can follow/try this PR by @abeljim here:

https://github.com/huggingface/pytorch-pretrained-BERT/pull/152

This is now on master
Weights not initialized from pretrained model
Author: lemonhu
Pull request: no

Thanks for your awesome work!

When I execute the following code for a named entity recognition tasks:
`model = BertForTokenClassification.from_pretrained("bert-base-uncased", num_labels=num_labels)`

Output the following information:

> Weights of BertForTokenClassification not initialized from pretrained model: ['classifier.weight', 'classifier.bias']
Weights from pretrained model not used in BertForTokenClassification: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias']

What puzzles me is that the parameters of the classifier are not initialized.

Comments:
Hi!

Those messages are correct, the pretrained weights that have been released by Google Brain are just the ones of the core network. They did not release task specific weights. To get a model that solves a specific classification task, you would have to train one yourself or get it from someone else.

@thomwolf There have been multiple issues about this specific behavior, maybe we should add some kind of text either as a print while loading the model or in the documentation. I would be happy to do it. What would you prefer?

Oh, I see, I will train the model with my own dataset, thank you for your answer.

Yes you are right @rodgzilla we should detail a bit the messages in [modeling.py](https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/pytorch_pretrained_bert/modeling.py#L545) to say that `These weights will be trained from scratch`.
All about the training speed in classification job 
Author: zhusleep
Pull request: no

I run the bert-base-uncased model with task 'mrpc' in ubuntu,nvidia p4000 8G. 
It's a classification problem, and I use the default demo data.
But the training speed is about 2 batch every second. Any problem?
I think it maybe too slow, but can not find why. I have another task with 1300000 data costs 6 hours per epoch.  

Comments:
Maybe try to use a bigger batch size or try fp16 training?
Please refer to the [detailed instructions in the readme](https://github.com/huggingface/pytorch-pretrained-BERT#examples).
Python 3.5 + Torch 1.0 does not work
Author: yuhui-zh15
Pull request: no

When running `run_lm_finetuning.py` to fine-tune language model with default settings (see command below),  sometimes I could run successfully, but sometimes I received different errors like `RuntimeError: The size of tensor a must match the size of tensor b at non-singleton dimension 1`, `RuntimeError: Creating MTGP constants failed. at /pytorch/aten/src/THC/THCTensorRandom.cu:35` or `RuntimeError: Dimension out of range (expected to be in range of [-1, 0], but got 1)`. This problem can be solved when updating `python3.5` to `python3.6`.

```
python run_lm_finetuning.py \
  --bert_model ~/bert/models/bert-base-uncased/ \
  --do_train  \
  --train_file ~/bert/codes/samples/sample_text.txt  \
  --output_dir ~/bert/exp/lm \
  --num_train_epochs 5.0  \
  --learning_rate 3e-5  \
  --train_batch_size 32  \
  --max_seq_length 128 \
  --on_memory
```

Comments:
Thank you @yuhui-zh15  sir. i will check.

This should be fixed on master now (thanks to #191 )
BertOnlyMLMHead is a duplicate of BertLMPredictionHead
Author: artemisart
Pull request: no

https://github.com/huggingface/pytorch-pretrained-BERT/blob/35becc6d84f620c3da48db460d6fb900f2451782/pytorch_pretrained_bert/modeling.py#L387-L394

I don't understand how it is useful to wrap the BertLMPredictionHead class like that, perhaps it was forgotten in some refactoring ? I can do a PR if you confirm me it can be replaced.

BertOnlyMLMHead is only used in BertForMaskedLM.

Comments:
That's an heritage of how I converted the TF code (by reproducing the scope architecture in TF with PyTorch classes). We can't really change that now without re-converting all the TF code.
If you want a more concise version of PyTorch BERT, you can check [pytorchic-bert](https://github.com/dhlee347/pytorchic-bert).
run_classifier.py doesn't save any configurations and I can't load the trained model. 
Author: anz2
Pull request: no


Comments:
I trained BertForSequenceClassification model with cola dataset mode for binary classification. It saved only eval_results.txt and pytorch_model.bin files. When I am loading model again like:
model = BertForSequenceClassification.from_pretrained('models/') 
it produces such error:
with open(json_file, "r", encoding='utf-8') as reader:
FileNotFoundError: [Errno 2] No such file or directory: 'models/bert_config.json'

I trained models using the command:
export GLUE_DIR=data_dir_path; python run_classifier.py --task_name cola --do_train --do_eval --data_dir $GLUE_DIR/ --bert_model bert-base-multilingual-cased --max_seq_length 128 --train_batch_size 16 --learning_rate 2e-5 --num_train_epochs 3.0 --output_dir models/

Do I have any error with training script? 
How can I produce such config.json file to load model successfully?

You can fetch the configuration from S3 like it's [done in the example](https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/examples/run_classifier.py#L560).

Alternatively, you can save the configuration with:
```python
with open('config.json', 'w') as f:
    f.write(model.config.to_json_string())
```
[Feature request] Add example of finetuning the pretrained models on custom corpus
Author: elyase
Pull request: no


Comments:
Hi I don't plan to add that in the near future but feel free to open a PR if you would like to share an additional example.

Necrobumping this for reference, as this is addressed in https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/examples/run_lm_finetuning.py
TODO statement on Question/Answering Model
Author: phatlast96
Pull request: no

Has this been confirmed?
https://github.com/huggingface/pytorch-pretrained-BERT/blob/647c98353090ee411e1ef9016b2a458becfe36f9/pytorch_pretrained_bert/modeling.py#L1084

Comments:
Not really, I've moved to something else since I don't expect this to change significantly the results.
I will remove the TODO.
Missing softmax in BertForQuestionAnswering after linear layer?
Author: jianyucai
Pull request: no

https://github.com/huggingface/pytorch-pretrained-BERT/blob/0a9d7c7edb20a3e82cfbb4b72515575543784823/pytorch_pretrained_bert/modeling.py#L1089-L1113

It seems there should be a softmax after the linear layer, or did I miss something?

Comments:
It depends on what you use as loss, as mentioned in the [documentation](https://pytorch.org/docs/stable/nn.html#crossentropyloss):

>This criterion combines nn.LogSoftmax() and nn.NLLLoss() in one single class.
Two to Three mask word prediction at the same sentence is very complex
Author: MuruganR96
Pull request: no

Two to Three mask word prediction at the same sentence also very complex.
how to get good accuracy?
if i have to pretrained bert model and own dataset with **masked_lm_prob=0.25** (https://github.com/google-research/bert#pre-training-with-bert),  what will happened?

Thanks.

Comments:
Hi @MuruganR96, from my experiments two to three mask word prediction doesn't seems to be possible with BERT.

thanks @thomwolf  sir
How can I get the confidence score for the classification task
Author: fenneccat
Pull request: no

In evaluation step, it seems it only shows the predicted label for the data instance.
How can I get the confidence score for each class?

Comments:
You can use `torch.nn.functional.softmax` on the `logits` that the model outputs here:

https://github.com/huggingface/pytorch-pretrained-BERT/blob/0a9d7c7edb20a3e82cfbb4b72515575543784823/examples/run_classifier.py#L589-L591

It will give you the confidence score for each class.
Add some new layers from BertModel and then 'grad' error occurs
Author: lhbrichard
Pull request: no

I wanna do the fine-tuning work by adding a textcnn on the base of BertModel. I write a new class and add two layers of conv (like a textcnn) basically on Embedding Layer. And then an error occurs, called "grad can be implicitly created only for scalar outputs" i search for the Internet and can't find a good solution to that, hope someone can solve it

Comments:
If you can share a (minimal) example reproducing the error, I can have a look.

I'm closing this. Feel free to re-open and share more information if you still have some issues.
how to add new vocabulary?
Author: hahmyg
Pull request: no

for specific task, it is required to add new vocabulary for tokenizer.
It is ok that re-training for those vocabulary for me :)

Is it possible to add new vocabulary for tokenizer? 


Comments:
Hi @hahmyg, please refer to the relevant section in the original implementation repository: https://github.com/google-research/bert#learning-a-new-wordpiece-vocabulary.
Freezing base transformer weights
Author: ZhaofengWu
Pull request: no

As I understand, say if I'm doing a classification task, then the transformer weights, along with the top classification layer weights, are both trainable (i.e. `requires_grad=True`), correct? If so, is there a way to freeze the transformer weights, but only train the top layer? Is that a good idea in general when I have a small dataset?

Comments:
Hi!

You can modify the trainable attributes as described in #95.

Thanks!
Never split some texts.
Author: WrRan
Pull request: yes

I have noticed bert tokenize texts by two steps:
1. punctuation: split text to tokens
2. wordpiece: split token to word_pieces

Some texts such as `"[UNK]"` are supposed to be left as they are. However, they become `["[", "UNK", "]"]` or something like this.

This PR is to solve the above problem.

Comments:
Please have a look. :)

Looks good indeed, thanks @WrRan!
Why is the output bias computed separately?
Author: yamrzou
Pull request: no

Hi !

Sorry if this is a dumb question, but I don't understand why is the bias [added separately](https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/pytorch_pretrained_bert/modeling.py#L379) to the decoder weights instead of using `self.decoder = nn.Linear(num_features, num_tokens, bias=True)`? Isn't it equivalent?

Comments:
The code section you linked follows the original TensorFlow code: https://github.com/google-research/bert/blob/f39e881b169b9d53bea03d2d341b31707a6c052b/run_pretraining.py#L257

Exactly.
ConnectionError returned if Internet network is not stable
Author: renjunxiang
Pull request: no

Hi,
although I have download BERT pretrained model,  "ConnectionError" returned if my Internet network is not very stable.
Function ```file_utils.cached_path``` needs stable internet. 
Is there any way to avoid checking for amazonaws before loading bert-embedding?

Comments:
I'm guessing you're using some of the classes defined in modeling.py, such as one of the Bert "pretrained models" (e.g. any of the models that inherit from `PreTrainedBertModel`)? On construction, each of these classes takes a `config` argument, where `config` is a BertConfig object (also defined in modeling.py). The BertConfig can either be created from a model at one of the links in `PRETRAINED_MODEL_ARCHIVE_MAP` or from a config file stored in a local directory. You just have to set the `pretrained_model_name` to a local directory containing a bert_config.json file and a pytorch_model.bin file rather than one of 'bert-base-uncased', 'bert-large-uncased' etc. Setting `pretrained_model_name` to one of the latter options will try to pull from the Amazon AWS repositories. So if you're running the run_classification.py script, you would set the 'bert-model' flag to the directory with your downloaded bert model if you don't want it to pull from AWS. One thing is if you've downloaded one of the original Google Bert models,  you'll need to convert  tf checkpoints to pytorch bin files. There's a script for this in the repository. You shouldn't need to worry about this if you've downloaded one of the models at the links in `PRETRAINED_MODEL_ARCHIVE_MAP` (defined at the top of modeling.py)

TLDR: Set `--bert-model` to the directory with your downloaded Bert model

Does that make any sense?

Yes! Set local directory in modeling.py and tokenization.py can solve my problem. Thank you so much!

Thanks @cmeister747 !
SQuAD output layer and the computation loss
Author: jianyucai
Pull request: no

Hi, I noticed that in the final linear layer of `BertForQuestionAnswering`, the loss is computed based on `start_logits` and `end_logits `. That means the positions of questions are also considered to compute loss. Maybe we should only care about the positions of context? e.g., by setting the question part of `start_logits` and `end_logits ` to `-inf`?

https://github.com/huggingface/pytorch-pretrained-BERT/blob/0a9d7c7edb20a3e82cfbb4b72515575543784823/pytorch_pretrained_bert/modeling.py#L1089-L1113


Comments:
Yes you can also do that.
Preprocessing necessary for lengthier text
Author: StalVars
Pull request: no

Hi, I tried to train squad model on a different dataset where I have lengthier questions/contexts. It gave memory error 

CUDA out of memory. Tried to allocate 4.50 MiB (GPU 5; 11.78 GiB total capacity;

This error seems to happen in pytorch when there are lengthier data points ( pytorch tells how much it tried to allocate as opposed to normal CUDA out of memory error).

tensorflow code for bert doesn't give this error as it trims the question/text lengths. I think you should include in this package as well.

Comments:
Ok. I see you included this. (max_sent_length, max_query_length)
I will debug my error.You probably can close this issue.
Cleaning `~/.pytorch_pretrained_bert`
Author: ZhaofengWu
Pull request: no

What is inside `~/.pytorch_pretrained_bert`? Is it just the downloaded pre-trained model weights? Is it safe to remove this directory?

Comments:
This folder contains the pretrained model weights as they have been trained by google and the vocabulary files for the tokenizer.

I would not remove it unless you are really tight on disk space, in this case I guess you could only keep the `.json` files with the vocabulary and load your finetuned model.

Yes it contains the weights, configuration and vocabulary files. You can remove it if you want. In that case the weights will be downloaded again the next time you initiate a BertModel.
Tokenization doesn't seem to match BERT paper
Author: JohnGiorgi
Pull request: no

In the [BERT paper](https://arxiv.org/abs/1810.04805) section 4.3 ("Named Entity Recognition") there is an example of some tokenized text:

```python
['Jim', 'Hen', '##son', 'was', 'a', 'puppet', '##eer']
```

However, when I take that sentence and try to tokenize it myself with `BertTokenizer` from this repo

```python
from pytorch_pretrained_bert import BertTokenizer

tokenizer = BertTokenizer.from_pretrained('bert-base-cased', do_lower_case=False)

text = "Jim Henson was a puppeteer"

tokenizer.tokenize(text) == ['Jim', 'He', '##nson', 'was', 'a', 'puppet', '##eer']
```

same thing happens if I pre-tokenize and just use `BertTokenizer.wordpiece_tokenizer.tokenize()`

```python
from itertools import chain
from pytorch_pretrained_bert import BertTokenizer

tokenizer = BertTokenizer.from_pretrained('bert-base-cased', do_lower_case=False)

text = "Jim Henson was a puppeteer".split()

list(chain.from_iterable([tokenizer.wordpiece_tokenizer.tokenize(token) for token in text])) == ['Jim', 'He', '##nson', 'was', 'a', 'puppet', '##eer']
```

Is there something I am misunderstanding / doing wrong or is this an actual bug? The BERT paper and this repo tokenize `"Henson"` as `['Hen', '##son']` and `['He', '##nson']` respectively.

Comments:
WordPiece tokenization depends on the particular BERT model: In general, one model, say, bert-based-cased will produce a different tokenization than another, say, bert-large-uncased.

If you try all models, one or more might produce the tokenization shown in the example in the paper.
It might also happen that none of them does, in which case the example was probably produced with an unpublished model.

A bug would be if the same model leads to different tokenizations in the pytorch and tensorflow version.

Hmm, I see. I didn't know that, thanks for pointing it out!

For what it's worth, `bert-base-multilingual-cased` is the only model (from those currently listed in the readme of this repo) that produces the tokenization shown in the example in the paper.
Accurate SQuAD answer start and end position
Author: cooelf
Pull request: yes

Previous version might miss some overlong answer start and end indices (which should be 0), and sometimes the start/end positions would be outside the model inputs.
Doc_start and doc_end are based on tokenized subword sequences while example.start_position and example.end_position are indices in original word-level, which are usually shorter than the subword sequences. 

Comments:
Fix learning rate problems in run_classifier.py
Author: matej-svejda
Pull request: yes

- Don't do warmup twice (in BertAdam and manually)
- Compute num_train_steps correctly for the case where gradient_accumulation_steps > 1. The current version might lead the the LR never leaving the warmup phase, depending on the value of gradient_accumulation_steps.

With these changes I get > 84% accuracy on MRPC, without them its around 77%.

Comments:
Thanks @matej-svejda.
So this problem was actually introduced by adding NVIDIA's fp16 optimizer (`FusedAdam`) to the examples. This optimizer is a simple Adam which doesn't incorporate a learning rate schedule so we had to add a manual learning rate schedule in the examples.
So a better solution is to keep the `warmup_linear` function but to only modify the learning rates when the fp16 optimiser is used (i.e. updating the weights only if `args.fp16==True`).
Also it would  be great to update the other examples similarly.
Do you want to do that in your PR?
I can also do that if you don't have the time. 

Sure, I can do that. Wanted to try out fp16 anyways :+1: 

@thomwolf Something like this?

Thanks @matej-svejda, I was a bit late on this PR. I've made a small commit to make the notation more explicit (removed `t_total` which was mainly a duplicate of `num_train_steps` and renamed `num_train_steps` in a more explicit `num_train_optimization_steps`).
Merging this now