noamaanMulla-03/transformers-issues huggingface dataset:
Bert Uncased Large giving very low results with SQUAD v1.1 dataset
Author: sarthak221995
Pull request: no

**Configuration:**
- do_lower_case=True
- max_answer_length=30
- max_answer_length=30
- n_best_size=20
- verbose_logging=False
- bert_model="bert-large-uncased"
- max_seq_length=384
- doc_stride=128
- max_query_length=192
- local_rank=-1
- train_batch_size=12
- predict_batch_size=12
- num_train_epochs=2.0
- gradient_accumulation_steps=1
- fp16=True
- warmup_proportion=0.1
- learning_rate=3e-5

**Results:**
1. Getting very bad results with the given configuration.
2. Kindly let me know if there is an issue with the configuration.
3. Getting many repeated terms for different questions under the same context

[dev1.1_squad_best_results.txt](https://github.com/huggingface/pytorch-pretrained-BERT/files/2935422/dev1.1_squad_best_results.txt)
Sample Dev1.1 Squad(First 9 Questions from development dataset):
    "56be4db0acb8001400a502ec": "Levi's Stadium in the San Francisco Bay Area at Santa Clara,",
    "56be4db0acb8001400a502ed": "Levi's Stadium in the San Francisco Bay Area at Santa Clara,",
    "56be4db0acb8001400a502ee": "Levi's",
    "56be4db0acb8001400a502ef": "Levi's",
    "56be4db0acb8001400a502f0": "Levi's Stadium in the San Francisco Bay Area at Santa Clara,",
    "56be8e613aeaaa14008c90d1": "7",
    "56be8e613aeaaa14008c90d2": "Levi's",
    "56be8e613aeaaa14008c90d3": "7, 2016, at Levi's",
    "56bea9923aeaaa14008c91b9": "7",

Looking forward for the help.
Thanks

Comments:
Problems converting TF BioBERT model to PyTorch
Author: jwhite2a
Pull request: no

My goal is to convert and train on the [BioBERT pretrained checkpoints](https://github.com/naver/biobert-pretrained) in pytorch and train on the [SQuAD v2.0 Dataset](https://rajpurkar.github.io/SQuAD-explorer/).  

I have (seemingly) successfully transfered the checkpoint using the `./pytorch_pretrained_bert/convert_tf_checkpoint_to_pytorch.py` [script](https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/pytorch_pretrained_bert/convert_tf_checkpoint_to_pytorch.py). 

I loaded the converted checkpoint into the `run_squad.py` [example](https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/examples/run_squad.py). I also changed the tokenizer to use the vocab file found with the BioBERT model. At this point, I was able to train the model and observe the loss to decrease. 

My first issue appeared when trying to write the SQuAD predictions. `best_non_null_entry.start_logit` did not have a `start_logit` because the `best_non_null_entry` was `NoneType`. This error resembles the previous issue #207. I implemented the solution found and my code was able to run. 

My results from training have been the same or worse than a random model. Nearly all of the SQuAD predictions are the "empty" string text from the fix of #207. 

**I believe the original cause of the `NoneType` error for `best_non_null_entry` is the reason for the failure to predict anything.** 

Are there specs to obey when converting a TF pretrained BERT model?
What would cause the `NoneType` error for `best_non_null_entry`?
Any and all help is appreciated.

Comments:
I have solved my issue. All my code was correctly written. The error was an corrupted/improperly saved model.bin file.

I'm trying to convert BioBert to Pytorch also, so just wondering if you could share a bit more details on how you are doing the conversion. Thanks!

First, I downloaded the BioBERT TF checkpoints [here](https://github.com/naver/biobert-pretrained). Each model (i.e. biobert_pmc) should have three `.ckpt` files, a `vocab.txt` file, and a `bert_config.json` file. 

Initially, I tried to use the command line interface `pytorch_pretrained_bert convert_tf_checkpoint_to_pytorch` using the `bert_config.json` and `.ckpt` files seen above. I ran into `AttributeError: 'Parameter' object has no attribute 'BERTAdam'`. I followed the solution [here](https://github.com/dmis-lab/biobert/issues/2). 

To do this, I copied the `convert_tf_checkpoint_to_pytorch.py` file and the `load_tf_weights_in_bert` function found in `modeling.py`. I then added the two lines seen in the [solution above](https://github.com/dmis-lab/biobert/issues/2) in my own version of the function and file. 

Given correct file paths, this worked to convert all three BioBERT checkpoints into pytorch `.bin` files.

@jwhite2a Thank you! This worked for me.
Is the GPT-2 pretrained model language agnostic?
Author: AlyShmahell
Pull request: no

I'm trying to build a language model that trains on a Polish corpus. And I'm wondering if the GPT-2 pretrained model you present supports that, or if it's English only.

Thank You.

Comments:
Hi Aly, GPT-2 is pretrained on an English only corpus.

Hi @thomwolf , thank you for the clarification.
Add full language model fine-tuning
Author: Rocketknight1
Pull request: yes

These scripts add language model fine-tuning that closely mirrors the training process in the original BERT repo. The old fine-tuning example has been renamed `simple_lm_finetuning.py`. The key difference is the old script did not merge sentences when creating training examples, and so tended to create short training examples with lots of padding tokens - this is explained more fully in the README. The new scripts follow the BERT repo approach, which concatenates sentences from both documents to pack the training example up to `max_seq_len`.

All the scripts for LM fine-tuning have been moved to a subfolder of `examples/` with an included README to explain what LM fine-tuning is and how to use them.

Comments:
Also, build_py2 has failed because I deliberately did not include Py2 compatibility - it's only a few months from end-of-life now. If you really want me to, I can go back and include it, but we should be trying to let it go by now!

This is really great @Rocketknight1!
Thanks for taking the time to make a very clear and informative README.
I've fixed the formatting issues in the README, slightly re-worded it and added a link to it in the main README.
I'm merging it now.
Great job!
Why average the loss when training on multi-GPUs
Author: pkuyym
Pull request: no
Labels: name: Discussion

Could anyone help to explain the motivation for this operation?

https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/examples/run_classifier.py#L573-L574

Comments:
Multi-GPU loss returns a tuple of losses with one loss for each GPU.
We average them to get the full loss. See [this blog post](https://medium.com/huggingface/training-larger-batches-practical-tips-on-1-gpu-multi-gpu-distributed-setups-ec88c3e51255) for more details.
add BertTokenizer flag to skip basic tokenization
Author: john-hewitt
Pull request: yes

When tokenization is done before text hits this package (e.g., when tokenization is specified as part of the dataset) there exists a use case for skipping the `BasicTokenizer` step, going right to `WordpieceTokenizer`.

When one still wants to use the `BertTokenizer.from_pretrained` helper function, they have been able to do this (without claiming this is necessarily the best way) by

```
text = "[CLS]  `` Truly , pizza is delicious , '' said Mx. Caily."
bert_tokenizer = BertTokenizer.from_pretrained('bert-base-cased')
tokenized_text = bert_tokenizer.wordpiece_tokenizer.tokenize(text)
```

With this PR, we instead use
```
text = "[CLS]  `` Truly , pizza is delicious , '' said Mx. Caily."
bert_tokenizer = BertTokenizer.from_pretrained('bert-base-cased', do_basic_tokenize=False)
tokenized_text = bert_tokenizer.tokenize(text)
```

a flag for which I add documentation in the docstring and README, hopefully making it clear that this is possible. 


Comments:
Thanks for the PR (and documenting this), I added a note

Ok this is great, thanks @john-hewitt, thanks!
Multi GPU
Author: dirkgr
Pull request: yes

This is an incomplete proof-of-concept of how to run BERT across multiple GPUs. It will take advantage of multiple GPUs' memory, but not of their compute cores.

Do not merge

Comments:
Hi @dirkgr, thanks for this PR.

I think we will keep all the GPU/multi-GPU logic outside of the main library for now. It makes it easier to integrate the module in downstream libraries and integrating such modifications at the current stage would cause too many breaking changes for the users unfortunately.

So I'm closing this PR for now.

Oh, this was only meant for internal consumption.

That said, I thought a bit about how to make it general enough to properly integrate it. I thought the setting for multi-GPU could be part of the config, so it wouldn't affect normal operation. How would you want to do it?

I didn't get that much interest in this patch internally though, so this is not the most important project right now.
Added remaining GLUE tasks to 'run_classifier.py'
Author: ananyahjha93
Pull request: yes

Also added metrics used in the GLUE paper for each task.

Comments:
Hi @ananyahjha93,
Thanks for this PR.
Do you have some results from fine-tuning BERT on the other tasks?
Also, I think we should add some details on the available tasks in the readme as well.

@thomwolf I have added results on GLUE dev set in the README and details on how to run any GLUE task. But, I have also added a warning stating that all GLUE tasks have not been tested with half-precision training. With the new tasks and metrics being added, there should be one round of tests with half-precision training as well. Unfortunately, I do not have direct access to a V100 or any of the RTX cards in order to run that. 

Also, for some reason the previous pull request was passing each build_py2 test but now it is failing the 'OpenAIGPTTokenizationTest'. I only added changes to the README in my latest commit.

This looks great, thanks @ananyahjha93!
For sequence classification, is this model using the wrong token?
Author: CatalinVoss
Pull request: no

According to the BERT paper, we want to use the weights for the `[CLS]` token, which – as far as I understand – would be the first hidden output here, not the last?

i.e. shouldn't this be `encoded_layers[0]` below?

https://github.com/huggingface/pytorch-pretrained-BERT/blob/f7c9dc8c998395d2ad9edbf0fd6fa072f03cc667/pytorch_pretrained_bert/modeling.py#L715-L719

@thomwolf Let me know if I'm missing something and if you're concerned about this breaking anything else and I can PR.

Comments:
Never mind, I think I got confused. Is it that the pooler already takes care of that and encoded_layers represents the pooler output for each attention layer?

Hi Catalin, the content of the outputs (`encoded_layers` and `pooled_output`) is detailed in the readme [here](https://github.com/huggingface/pytorch-pretrained-BERT#1-bertmodel) and in the model docstring. `encoded_layers` contains the encoded-hidden-states at the end of each attention block (i.e. 12 full sequences for BERT-base, 24 for BERT-large), each encoded-hidden-state is a torch.FloatTensor of size [batch_size, sequence_length, hidden_size].

The Pooler take care of extracting the hidden state of the first token from the encoded hidden state of the full sequence (see [here](https://github.com/huggingface/pytorch-pretrained-BERT/blob/f7c9dc8c998395d2ad9edbf0fd6fa072f03cc667/pytorch_pretrained_bert/modeling.py#L413))
performance degraded when using paddings between queries and contexts.
Author: leonwyang
Pull request: no
Labels: name: Discussion

I just want to ask this here and see whether other people encountered the same situation.

I am doing modifications on the run_squad.py example.

So for the original training feature, the input ids are [cls]qqqqq[sep]cccccc000000. The attention mask is just something like 111111100000 where first k inputs were masked with 1 and 0 for the laters.

I tried to generate input ids look like [cls]qqqq[sep]0000[sep]cccccccc00000, the I can have fixed length of query and context with proper padding. Also, I changed the attention mask accordingly, namely, to have something like 11110001111100000.

However, when I trained the model on this new feature, the  score degraded from 76 to 44 for the bertforquestionanswering model. I am wondering if there is any catastrophic results by doing this kind of masking? Did anyone experienced with similar situations?

Comments:
Similar problem. My token ids is something like "[cls]qqqq[sep]0000cccccccc[sep]00000". Have you solve it? or is there anyone met the similar problem?

> Similar problem. My token ids is something like "[cls]qqqq[sep]0000cccccccc[sep]00000". Have you solve it? or is there anyone met the similar problem?

I felt like this is caused by the way how we pretrain BERT. BERT is pretrained on contiguous texts. The way we pad zeros in between broke such continuity so essentially we need to re-train the whole model. 

Since I was trying to separate query and context, the way I tackled this is just masking all query tokens to create the context and same for the query. 

> > Similar problem. My token ids is something like "[cls]qqqq[sep]0000cccccccc[sep]00000". Have you solve it? or is there anyone met the similar problem?
> 
> I felt like this is caused by the way how we pretrain BERT. BERT is pretrained on contiguous texts. The way we pad zeros in between broke such continuity so essentially we need to re-train the whole model.
> 
> Since I was trying to separate query and context, the way I tackled this is just masking all query tokens to create the context and same for the query.

It could be the same way as inputing the query and context separately if we mask the query or context part. Anyway, I will try it. Thanks.
BertForTokenClassification for NER, mask labels 
Author: alexyalunin
Pull request: no

I'm trying to do Named Entity Recognition with BertForTokenClassification.
Say I have 10 words with 10 labels, after WordPiece tokenization I get 15 tokens and I assign them labels, "X" for pieces of words like (##ing).
In the original paper https://arxiv.org/pdf/1810.04805.pdf in 4.3 section is said that we don't make predictions for such pieces or we can simply do not care about them and not take derivatives. 
Does this functionality currently exist in BertForTokenClassification or BertModel? 

Comments:
Sequence tagging is explained here: https://github.com/huggingface/pytorch-pretrained-BERT/issues/64#issuecomment-443703063

Yes, this is the relevant issue on this topic. I'll close this issue in favor of #64.
how to do the pre training the model form scratch?
Author: SeekPoint
Pull request: no

for example, use sample_text.txt

Comments:
```
python3 examples/lm_finetuning/simple_lm_finetuning.py 
--train_corpus sample_text.txt 
--bert_model bert-base-uncased 
--do_lower_case 
--output_dir finetuned_lm/
```
In addition, you can refer to #385 

Yes let's keep a single issue on this. Closing in favor of #385.
Fix cosine schedule
Author: lukovnikov
Pull request: yes

Fixing similar problem to #327 and #324 in cosine schedule.

Btw, do you think it would make sense to have [something like this](https://github.com/lukovnikov/pytorch-pretrained-BERT/blob/optim/pytorch_pretrained_bert/optimization.py) for both of your `optimization.py`'s?

Comments:
Hi @lukovnikov, yes I think the various schedules you have added to your fork are very nice!
Do you want to add them in this PR as well?
Otherwise, I'll merge it.

Merging it for now. Thanks @lukovnikov 

Hi, sorry, lost track of this, will make a new PR soon.
pytorch_pretrained_bert/convert_tf_checkpoint_to_pytorch.py  error
Author: antxiaojun
Pull request: no

attributeError: 'BertForPreTraining' object has no attribute 'global_step'

Comments:
Maybe some additional information could help me help you?

Initialize PyTorch weight ['cls', 'seq_relationship', 'output_weights']
Skipping cls/seq_relationship/output_weights/adam_m
Skipping cls/seq_relationship/output_weights/adam_v
Traceback (most recent call last):
  File "/home/tiandan.cxj/python/model_serving_python/lib/python3.5/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/tiandan.cxj/python/model_serving_python/lib/python3.5/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/tiandan.cxj/platform/pytorch_BERT/pytorch-pretrained-BERT/pytorch_pretrained_bert/__main__.py", line 19, in <module>
    convert_tf_checkpoint_to_pytorch(TF_CHECKPOINT, TF_CONFIG, PYTORCH_DUMP_OUTPUT)
  File "/home/tiandan.cxj/platform/pytorch_BERT/pytorch-pretrained-BERT/pytorch_pretrained_bert/convert_tf_checkpoint_to_pytorch.py", line 69, in convert_tf_checkpoint_to_pytorch
    pointer = getattr(pointer, l[0])
  File "/home/tiandan.cxj/python/model_serving_python/lib/python3.5/site-packages/torch/nn/modules/module.py", line 518, in __getattr__
    type(self).__name__, name))
AttributeError: 'BertForPreTraining' object has no attribute 'global_step'

Hum I will see if I can let people import any kind of TF model in PyTorch, that's a bit risky so it has to be done properly.
In the meantime you can add `global_step` in the list line 53 of `convert_tf_checkpoint_to_pytorch.py`

@thomwolf  sir, i am also same issue. it doen't resolve. how i am convert my finetuned pretrained model to pytorch?

```
export BERT_BASE_DIR=/home/dell/backup/NWP/bert-base-uncased/bert_tensorflow_e100

pytorch_pretrained_bert convert_tf_checkpoint_to_pytorch \
  $BERT_BASE_DIR/model.ckpt-100 \
  $BERT_BASE_DIR/bert_config.json \
  $BERT_BASE_DIR/pytorch_model.bin

```

```
Traceback (most recent call last):
  File "/home/dell/Downloads/Downloads/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/dell/Downloads/Downloads/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/dell/backup/bert_env/lib/python3.6/site-packages/pytorch_pretrained_bert/__main__.py", line 19, in <module>
    convert_tf_checkpoint_to_pytorch(TF_CHECKPOINT, TF_CONFIG, PYTORCH_DUMP_OUTPUT)
  File "/home/dell/backup/bert_env/lib/python3.6/site-packages/pytorch_pretrained_bert/convert_tf_checkpoint_to_pytorch.py", line 69, in convert_tf_checkpoint_to_pytorch
    pointer = getattr(pointer, l[0])
  File "/home/dell/backup/bert_env/lib/python3.6/site-packages/torch/nn/modules/module.py", line 535, in __getattr__
    type(self).__name__, name))
AttributeError: 'BertForPreTraining' object has no attribute 'global_step'
```
sir how to resolve this issue?
thanks.


thanks @thomwolf  sir. it was resolved.

I added the global_step to the skipping list in the modelling.py . Still facing the error. Am I missing something?
Cannot find Synthetic self-training in this repository.
Author: Dogy06
Pull request: no

The SQuAD leader board's (https://rajpurkar.github.io/SQuAD-explorer/) 3rd highest scored model uses 'synthetic self-training'.
There is a PDF explaining it: https://nlp.stanford.edu/seminar/details/jdevlin.pdf?fbclid=IwAR2TBFCJOeZ9cGhxB-z5cJJ17vHN4W25oWsjI8NqJoTEmlYIYEKG7oh4tlY but I have found no such model within this repository.
On the leader board website, it shows me a link to this repository.
Is the synthetic self-training in this repository? (If so, please show me the link to the code)
Is it private? Will it be released?
Thanks,
Dogy06

By the way: I checked both the Pytorch and Tensorflow version of the code., I posted the same issue here:https://github.com/google-research/bert/issues/532

Comments:
Hi, indeed there is no Synthetic self-training in this repository, and the SQuAD leaderboard website actually refers to the Tensorflow repository so I'll close this issue.

Will you be adding synthetic self training though?
modeling_openai.py bug report
Author: Jonbean
Pull request: no

line 651 has a potential bug

```        # Copy word and positional embeddings from the previous weights
        self.tokens_embed.weight.data[: self.config.vocab_size, :] = old_embed.weight.data[: self.config.vocab_size, :]
        self.tokens_embed.weight.data[-self.config.n_positions :, :] = old_embed.weight.data[-self.config.n_positions :, :]
```
isn't it supposed to be `self.positions_embed ` instead of `self.tokens_embed` ?

Comments:
Good catch, I'll fix this.
Compilation terminated
Author: hongkahjun
Pull request: no

Hi,

I cannot pip install the package. I have

>   regex_3/_regex.c:48:10: fatal error: Python.h: No such file or directory
   #include "Python.h"
            ^~~~~~~~~~
  compilation terminated.
  error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

Might have something to do with CI failure for the new merge.

Comments:
managed to install it by running 

> sudo apt-get install python3 python-dev
How to select a certain layer as token's representation?
Author: yexing99
Pull request: no
Labels: name: Discussion

My understanding from the paper that each token is represented by a 768-dim vector from the last hidden layer. Is it correct? If so, how can I get the second-to-the last layer's parameter as token representation? 

model = BertForTokenClassification.from_pretrained("bert-base-uncased", num_labels=len(tag2idx))
 list(model.named_parameters()) gives all weights but how can I know which parameters are from the second to the last layer?

for example, I have a word 'test', how can I get the BERT features from the second to the last hidden layer for the word 'test' in a given sentence?

Comments:
Hi @yexing99,
What you need is the `hidden_state`, not the `weights`of the model.
Don't use `model.named_parameters()` but just use the output of the model.
Here is an example: https://github.com/huggingface/pytorch-pretrained-BERT#bert
And more details here: https://github.com/huggingface/pytorch-pretrained-BERT#1-bertmodel
You should do somethings like this to get the BERT features from the second to the last hidden layer if the word `test` is for example the *fifth* word in your sentence (index 4 in the sentence):
```python
encoded_layers, _ = model(tokens_tensor, segments_tensors)
features = [encoded_layer[4] for encoded_layer in encoded_layers[1:]]
```
How to get vocab.txt and bert_config.json as output of fine tuning?
Author: search4mahesh
Pull request: no

Hi,
I am fine tuning bert on custom data.
As output I am getting only pytorch_model.bin but how to get updated vocab.txt and bert_config.json .

Please suggest.

Thanks
Mahesh

Comments:
You can get `pytorch_model.bin` and `config.json` just as indicated in the examples: https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/examples/run_classifier.py#L861-L866

The vocabulary stays the same, just load the tokenizer as you did for the training (`BertTokenizer.from_pretrained(...)`)

Thanks Thomas. I really appreciate it.
Our custom text might have some new words which may not be in original vocab.txt
Original vocab.txt contain some unused placeholders. (some 993) out of total 30522
Please let me know your thoughts on this.
Thanks
Mahesh

Then you should go have a look at https://github.com/huggingface/pytorch-pretrained-BERT/issues/463.
And I will close this issue in favor of #463 ;-)
Update README.md
Author: yaroslavvb
Pull request: yes

Fix for

```> > > > 04/09/2019 21:39:38 - INFO - __main__ -   device: cuda n_gpu: 1, distributed training: False, 16-bits training: False
Traceback (most recent call last):
  File "/home/ubuntu/pytorch-pretrained-BERT/examples/lm_finetuning/simple_lm_finetuning.py", line 642, in <module>
    main()
  File "/home/ubuntu/pytorch-pretrained-BERT/examples/lm_finetuning/simple_lm_finetuning.py", line 502, in main
    raise ValueError("Training is currently the only implemented execution option. Please set `do_train`.")
ValueError: Training is currently the only implemented execution option. Please set `do_train`.
```

Comments:
Thanks Yaroslav!