noamaanMulla-03/transformers-issues huggingface dataset:
80min training time to fine-tune BERT-base on the SQuAD dataset instead of 24min?
Author: gqoew
Pull request: no

I just fine-tuned BERT-base on the SQuAD dataset with an AWS EC2 `p3.2xlarge` Deep Learning AMI with a single Tesla V100 16GB:

I used the config in your README:

```
export SQUAD_DIR=/path/to/SQUAD

python run_squad.py \
  --bert_model bert-base-uncased \
  --do_train \
  --do_predict \
  --do_lower_case \
  --train_file $SQUAD_DIR/train-v1.1.json \
  --predict_file $SQUAD_DIR/dev-v1.1.json \
  --train_batch_size 12 \
  --learning_rate 3e-5 \
  --num_train_epochs 2.0 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --output_dir /tmp/debug_squad/
```

It took 80min. According to your README:

> This example code fine-tunes BERT on the SQuAD dataset. It runs in 24 min (with BERT-base) or 68 min (with BERT-large) on a single tesla V100 16GB.

How to explain this difference? Is there any way to accelerate the training to 24min as well? Thanks


Comments:
You should use 16bit training (`--fp16` argument). You can use the dynamic loss scaling or tune the loss scale yourself if the results are not the best.

@thomwolf Thanks! I enabled 16bit training and it took about 20min/epoch. Is that what you experienced?

Sounds good.

@thomwolf 
May I know what is the expected EM & F1 score if users train for 2-3 epochs? I got 43 and 48 respectively.

You can have a look at the readme examples but it should be a lot higher, around 88-90.
Maybe your batch size is too small, look at the readme for more information.
Include a reference for LM finetuning
Author: Rocketknight1
Pull request: yes

@lopuhin recently made me aware of a published paper covering domain fine-tuning of BERT models, so I added a reference to the LM finetuning README.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/715?src=pr&el=h1) Report
> Merging [#715](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/715?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/commit/c304593d8fa93f25febe1458c63497a846749c89?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/715/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/715?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master     #715   +/-   ##
=======================================
  Coverage   62.27%   62.27%           
=======================================
  Files          18       18           
  Lines        3979     3979           
=======================================
  Hits         2478     2478           
  Misses       1501     1501
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/715?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/715?src=pr&el=footer). Last update [c304593...c7b2808](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/715?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Nice!
Correct a broken link on README
Author: changukshin
Pull request: yes

I've correct a broken link and its contexts on README.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/714?src=pr&el=h1) Report
> Merging [#714](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/714?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/commit/c304593d8fa93f25febe1458c63497a846749c89?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/714/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/714?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master     #714   +/-   ##
=======================================
  Coverage   62.27%   62.27%           
=======================================
  Files          18       18           
  Lines        3979     3979           
=======================================
  Hits         2478     2478           
  Misses       1501     1501
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/714?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/714?src=pr&el=footer). Last update [c304593...ada0d8f](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/714?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


👍
BERT multilingual for zero-shot classification
Author: ramild
Pull request: no
Labels: name: Help wanted

name: Discussion

name: wontfix

Hi!
I'm interested in solving a classification problem in which I train the model on one language and make the predictions for another one (zero-shot classification).

It is said in the README for the multilingual BERT model (https://github.com/google-research/bert/blob/master/multilingual.md) that:

> For tokenization, we use a 110k shared WordPiece vocabulary. The word counts are weighted the same way as the data, so low-resource languages are upweighted by some factor. We intentionally do not use any marker to denote the input language (so that zero-shot training can work).

But after finetuning the BERT-multilingual-uncased for one language dataset, it absolutely doesn't work for the texts in another languages. Predictions turn out to be inadequate: I tried multiple pairs `(text, the same text translated to another language)` and probability distributions over labels (after apply softmax) were wildly different.

Do you know what can be the cause of the problem? Should I somehow change the tokenization when applying the model to other languages (BPE embeddings are shared, so not sure about this one)? Or should I use multilingual-cased instead of multilingual-uncased (is it possible it can be the source of the problem)?

Comments:
UPD. I tried it with bert-multilingual-cased, but the results are still bad. A number of very simple (text, translated text) give very different probability distributions (the translated versions almost always fall into one major category).

Specifiically, **I fine-tune pre-trained bert-multilingual-cased on Russian text classification problem and then make a prediction using the model on an English text** (tried other languages -- nothing works).

Hi, my feeling is that this is still an open research problem.

[Here](https://twitter.com/nlpmattg/status/1091367511117881345) is a recent thread discussing the related problem of fine-tuning BERT on English SQuAD and trying to do QA in another language. Maybe you can get a pre-print from the RecitalAI guys if they haven't published it yet.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Difference between this repo and bert-as-service
Author: tcqiuyu
Pull request: no
Labels: name: Discussion

name: wontfix

Hi, 

I wondered if anybody knows the difference between the `BertModel` of this repo and [bert-as-service](https://github.com/hanxiao/bert-as-service).

1. I cannot get the same result between these two even if I use the same checkpoint. pytorch-pretrained-BERT yield a lower acc and slower convergence.

2. The memory usage of `BertModel` seems much higher than bert-as-service. With the same batch-size=32, max_seq_len=100, the bert-as-service will take about 8000MB but `BertModel` will cost more than 16000MB because I got an OOM issue.

Does any one knows the reason behind it? Is there any optimization done for bert-as-service? 

Comments:
Yes there are some optimizations. Please take a look here:

https://hanxiao.github.io/2019/01/02/Serving-Google-BERT-in-Production-using-Tensorflow-and-ZeroMQ/#engineering-building-a-scalable-service

Hi, there is no specific relation between the present repo (which provides PyTorch implementations of several transformer's models) and bert-as-a-service (which is built on top of BERT TensorFlow implementation AFAICT).

We want to keep the code simple in the present repo and don't provide specific optimizations out-of-the-box but if you want to add some of bert-as-a-service optimizations, here are the most straightforward ones:
- use `with torch.no_grad()` to avoid computing gradient during evaluation (should divide memory consumption by two)
- use fp16 code to get another factor of 2, probably the easiest is to use the amp wrapper in NVIDIA's apex library, see the details [here](https://github.com/NVIDIA/apex).

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
BERT does mask-answering or sequence prediction or both???
Author: bladedsupernova
Pull request: no
Labels: name: Discussion

name: wontfix

I'm working on a exciting project but need to know something fast.

I seen BERT adds a word at the end of your input, like sequence prediction, elongating your input text.

But I read that BERT has been trained at (and is a pro at) filling in the blank word mask, and in fact CANNOT do sequence prediction at the end of one's input (contradiction!!). When BERT adds a fill-in answer where the mask sits, it goes in context.

I want a tool that returns me a similar word in context (the mask ANSWER!!), but the colab tool I tried is not a mask/answer giver, ..it adds words at the endddd of my input lol!!!

Comments:
I'm not sure I understand your question.

But one thing BERT can do is mask-answering indeed (guessing a word in the middle of a sentence).

BERT is quite bad at doing sequence prediction at the end of an input because it's not trained on partial sentences.

When BERT fills-in a MASK, does it always use a answer it has seen or does it generalize and create never-seen Q-As? Ex. "The *dog* in the hat, by Dr. Souse the Cat"

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
How to obtain attention values for each layer 
Author: serenaklm
Pull request: no
Labels: name: Discussion

name: wontfix

Hi all, 

Please correct me if I am wrong. 

From my understanding, The encoded values for each layer (12 of them for base model) would be returned when we run our results through the pre-trained model. 
However, I would like to examine the self-attention values for each layer. Is there a way I can extract that out? 

Regards

Comments:
Not really.
You should build a new sub-class of `BertPreTrainedModel` which is identical to `BertModel`but send back self-attention values in addition to the hidden states.


I see. Thank you! 

Hi, 

Just to add on. If this is what I would be doing, would it be advisable to fine-tune the weights for the pretrained model? 

Regards

Probably.
It depends on what's your final use-case.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Using GPT2 to implement GLTR
Author: asad1996172
Pull request: no
Labels: name: Discussion

name: wontfix

How can we use this GPT2 model to create the basic functionality of the GLTR tool?
Like getting probabilities for each word in a sequence.?

Comments:
the allen institute tool may help you....it has probabilities of next 10 words for when adding a next word....i think the prorbablities are shown....may just be the 10 words but the git code is available i think so may be what you want.

The output of `GPT2LMHeadModel` are logits so you can just apply a softmax (or log-softmax for log probabilities) on them to get probabilities for each token.

Then if you want to get probabilities for words, you will need to multiply (or add if you used a log-softmax) the probabilities of the sub-words in each word.

Maybe @sebastianGehrmann you have some additional insights (or plan to release the code of GLTR, it's a great demo!)?

Thanks for pinging me @thomwolf. GLTR actually uses this amazing repo in the backend, so you could just use our code in conjunction. 

We are still working on the docu, but you can find all our code here: https://github.com/HendrikStrobelt/detecting-fake-text 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
layer_norm_eps
Author: suchithtuple
Pull request: no

In [modeling.py](https://github.com/huggingface/pytorch-pretrained-BERT/blob/c304593d8fa93f25febe1458c63497a846749c89/pytorch_pretrained_bert/modeling.py#L303) why `self.layer_norm_eps` is written even config don't have these parameter. Check [here](https://github.com/google-research/bert#pre-trained-models)

Or I am missing something.

Comments:
Because some people wanted to configure this: https://github.com/huggingface/pytorch-pretrained-BERT/pull/585

So I have to add `config.layer_norm_eps = 1e-12` if I am taking the config from the link above ?

You don't need to, it's the default value when instantiating a `BertConfig` class.

When I printed `dir(bert_config)` I am not able to see layer_norm_eps. 

I have figured out the problem. I have to initialize like this `BertConfig.from_json_file(config_file_path)` but rather I have done like this `BertConfig(config_file_path)`. My bad :disappointed: 

Thanks for the reply. Closing the issue.
Embedding and predictions in one forward pass
Author: lcswillems
Pull request: no

Is it possible to mix `BertModel` and `BertForMaskedLM`? i.e. is it possible to get the embedding and the predictions in one forward pass?

Comments:
Yes, just make your own PyTorch model taking inspiration from BertModel and BertForMaskedLM.
If you sub-class `BertPreTrainedModel`, you'll be able to load the pretrained weights using the `from_pretrained()` method

Okay, thank you :)
How convert pytorch to tf checkpoint?
Author: semsevens
Pull request: no
Labels: name: Help wanted

name: wontfix

How convert pytorch to tf checkpoint?

Comments:
I don't think such a conversion is currently implemented in this repository, but I have my own implementation here (if you're interested in adapting it for your use-case): https://github.com/nikitakit/self-attentive-parser/blob/8238e79e2089300db059eddff78229a09e254f70/export/export_bert.py#L94-L141

Thanks @nikitakit, do you think your scripts would make sense in the present repo as well or is it tided to your parsing application?

Closing this for now. Feel free to re-open.

Would be actually quite nice to have such a conversion script in order to serve pytorch models via [bert-as-service](https://github.com/hanxiao/bert-as-service)

Any progress on this? As @tholor says, would be nice for bert-as-service. @nikitakit is it possible to run your script for any finetuned pytorch model? If so, any tips/suggestions on how to do that?

Re-opening the issue.
I don't have time to work on that at the moment but I would be happy to welcome a PR if there is interest in this feature.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Any news on this ?
"Received 'killed' signal" during the circleci python3 build after submitting PR
Author: chrisyue1998
Pull request: no

I submitted a PR after modifying convert_gpt2_checkpoint_to_pytorch.py, and my build_py2 test passed, but I received a very vague error from build_py3 (as written in the title of this issue) that caused my build to fail. Does anyone have any ideas as to where the issue could be?

Edit: Attached image below
<img width="625" alt="Screen Shot 2019-06-19 at 3 48 25 PM" src="https://user-images.githubusercontent.com/39981081/59795692-c8f2e280-92a9-11e9-8aa1-77cb96c2db33.png">


Comments:
Yeah I've removed the memory-heavy tests

Thanks!
BERT Input size reduced to half in forward function 
Author: xinsu626
Pull request: no

I was trying to modify your BertForSequenceClassification class for long sequence classification. Like below:

```
class MyBertForSequenceClassification(BertPreTrainedModel):
    def __init__(self, config, num_labels=2, output_attentions=False):
        super(MyBertForSequenceClassification, self).__init__(config)
        self.output_attentions = output_attentions
        self.num_labels = num_labels
        self.bert = BertModel(config)
        self.dropout = nn.Dropout(config.hidden_dropout_prob)
        self.classifier = nn.Linear(config.hidden_size*20, num_labels)
        self.softmax = nn.Softmax()
        self.apply(self.init_bert_weights)

    def forward(self, input_ids, token_type_ids=None, attention_mask=None, labels=None, head_mask=None):
        print(input_ids.shape)  # half of actual passed size
        outputs = self.bert(input_ids, token_type_ids, attention_mask, output_all_encoded_layers=False)
        _, pooled_output = outputs
        pooled_output = self.dropout(pooled_output)
        flat_pooled_output = pooled_output.view(-1)
        print(flat_pooled_output.shape)
        logits = self.classifier(flat_pooled_output)
        logits = self.softmax(logits)
        return logits
```

I found that when I passed the input_ids tensor with dimensions (40, 128) into the model, the actual input_ids I got in the forward function was (20, 128). It always reduce my input to half of original size.  


Comments:
Maybe you have 2 GPUs?

@thomwolf Thanks a lot. I forgot I was running on two gpus.
Add method to directly load TF Checkpoints for Bert models
Author: chrisgzf
Pull request: yes

## Summary
In this PR, I changed some documentation, and added `from_tf_ckpt()` method to `BertPreTrainedModel`.

This method allows users to directly load TensorFlow checkpoints (e.g. `model.ckpt-XXXX` files) for a task specific Bert model like `BertForTokenClassification` or `BertForSequenceClassification`.

**For example:**
```python
model = BertForSequenceClassification.from_tf_ckpt("/path/to/bert/bert_config.json",
                                                   "/path/to/bert/model.ckpt-12000",
                                                   num_labels=num_labels)
```

## Why this is needed:
This functionality has been requested by a number of people, like #676, https://github.com/huggingface/pytorch-pretrained-BERT/issues/676#issuecomment-501778493, #580, https://github.com/huggingface/pytorch-pretrained-BERT/issues/580#issuecomment-497286535, https://github.com/huggingface/pytorch-pretrained-BERT/issues/438#issuecomment-479405364

Comments:
Hi, I'm not convinced we need this additional option, see my [comment](https://github.com/huggingface/pytorch-pretrained-BERT/issues/676#issuecomment-502252962) in the associated issue thread.

As mentioned in, https://github.com/huggingface/pytorch-pretrained-BERT/issues/676#issuecomment-506134506, I recognise that directly loading TF checkpoints is a rather niche use case and will close this PR.

I don't think it's a niche but the `from_tf` option was there for importing from tf files so I would rather fix it to work in all cases rather than have several ways to import from a tf checkpoint.

I'll have a look.
merge
Author: Eurus-Holmes
Pull request: yes


Comments:
# [Codecov](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/731?src=pr&el=h1) Report
> Merging [#731](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/731?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/commit/98dc30b21e3df6528d0dd17f0910ffea12bc0f33?src=pr&el=desc) will **increase** coverage by `0.05%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/731/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/731?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master     #731      +/-   ##
==========================================
+ Coverage   62.22%   62.27%   +0.05%     
==========================================
  Files          18       18              
  Lines        3979     3979              
==========================================
+ Hits         2476     2478       +2     
+ Misses       1503     1501       -2
```


| [Impacted Files](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/731?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pytorch\_pretrained\_bert/tokenization\_gpt2.py](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/731/diff?src=pr&el=tree#diff-cHl0b3JjaF9wcmV0cmFpbmVkX2JlcnQvdG9rZW5pemF0aW9uX2dwdDIucHk=) | `83.51% <0%> (+1.06%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/731?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/731?src=pr&el=footer). Last update [98dc30b...4633033](https://codecov.io/gh/huggingface/pytorch-pretrained-BERT/pull/731?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).