noamaanMulla-03/transformers-issues huggingface dataset:
fix sample_doc
Author: jeonsworld
Pull request: yes

If the value of rand_end is returned from the randint function, the value of sampled_doc_index that matches current_idx is returned from searchsorted.

example:
cumsum_max = {int64} 30
doc_cumsum = {ndarray} [ 5  7 11 19 30]
doc_lengths = {list} <class 'list'>: [5, 2, 4, 8, 11]
if current_idx  = 1,
rand_start = 7
rand_end = 35
sentence_index = randint(7, 35) % cumsum_max
if randint return 35, sentence_index becomes 5.
if sentence_index is 5, np.searchsorted returns 1 equal to current_index.

Comments:
Good catch, thanks!

Gah. I meant to use `randrange()`, but this fix is equivalent!
Fix tsv read error in Windows
Author: jiesutd
Pull request: yes

The initial version suffers from the error of `UnicodeDecodeError: 'charmap' codec can't decode byte 0x90 in position` when loading the `.tsv` filein **Windows** System, as indicated in https://github.com/huggingface/pytorch-pretrained-BERT/issues/52

It is solved by adding `encoding='utf-8'` when reading the `.tsv` file.


Comments:
Ok, thanks @jiesutd!
making unconditional generation work
Author: dhanajitb
Pull request: yes

The unconditional generation works now but if the seed is fixed, the sample is the same every time.
n_samples > 1 will give different samples though.
I am giving the start token as '
' for the unconditional generation.

Comments:
Hi thanks for the PR.
I think we still need to clean up the example a little more indeed.
These lines should be taken care off:
```python
while not args.unconditional:
   if not args.unconditional:
```
I will see if I can find time to refactor it next week or you can update your PR if you want to fix this too.

I have further cleaned the redundant lines and some conditions.

Great, thanks @dhanajitb!
Possible error in "pytorch-pretrained-BERT/examples/run_gpt2.py" unconditional
Author: brunokinder
Pull request: no

Hello,
First of all thanks for offering us those great NLP implementations.

I think there may be an error in the file pytorch-pretrained-BERT/examples/run_gpt2.py
![image](https://user-images.githubusercontent.com/1786870/55035530-94df8c80-4fee-11e9-90eb-bde5bcba7832.png)

The way it is implemented if we do unconditional=True nothing happens.

If you fix this line, don't forget to look at too
![image](https://user-images.githubusercontent.com/1786870/55035656-dd974580-4fee-11e9-85e4-b0aa050fa274.png)

Thank you.


Comments:
This should be fixed by #462.
Suggestion: add warning when using BertForSequenceClassification without special [CLS] token
Author: mboyanov
Pull request: no
Labels: name: Discussion

Thank you for the awesome package!

As I understand it right now, it is the user's responsibility to add the special `CLS` and `SEP` tokens. People who haven't read the paper might miss this detail. 
It would be nice to issue a warning in the tokenizer or the model itself if the input is missing these tokens. An alternative is to include it in a more prominent location in the docs.
I understand that the package needs to be flexible so multiple architectures and tokenization procedures can be implemented, so I'm unsure where would be the best place for such a notification.

In short, my suggestion:
# GIVEN
A text sequence without the `CLS` tag
# WHEN
`BertForSequenceClassification` is called
# THEN
A warning is logged


Comments:
I understand the issue but I'm not sure this would be very easy to implement as we would like to keep the model and tokenizer separated one from the other.

Do you have a solution in mind?

Otherwise, I'll guess people will have to continue to read the paper before using the model... 😉

I totally understand the need for separation between tokenizer and model. 
But, the specific `BertForSequenceClassification` does have the assumption and expects the input to be in a specific form. Even the docs mention "look at the preprocessing logic". 
I'd suggest extending the documentation for the model - sth like https://github.com/huggingface/pytorch-pretrained-BERT/pull/480
UnboundLocalError: local variable 'i' referenced before assignment when using fine_tuning code
Author: KavyaGujjala
Pull request: no

Hi @thomwolf 
I am using the lm_finetuning codes. Generated training data using generate_pretraining_data.py

When running finetune_on_pregenerated.py . I am getting this error.

logs

 python finetune_on_pregenerated.py --pregenerated_data training_1/ --bert_model bert-base-uncased --do_lower_case --output_dir finetuned_lm_1Msents_3epochs/ --epochs 3
Better speed can be achieved with apex installed from https://www.github.com/nvidia/apex.
2019-04-13 09:32:43,014: device: cuda n_gpu: 4, distributed training: False, 16-bits training: False
2019-04-13 09:32:43,361: loading vocabulary file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-vocab.txt from cache at /home/cloud/.pytorch_pretrained_bert/26bc1ad6c0ac742e9b52263248f6d0f00068293b33709fae12320c0e35ccfbbb.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084
2019-04-13 09:32:43,676: loading archive file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased.tar.gz from cache at /home/cloud/.pytorch_pretrained_bert/9c41111e2de84547a463fd39217199738d1e3deb72d4fec4399e6e241983c6f0.ae3cef932725ca7a30cdcb93fc6e09150a55e2a130ec7af63975a16c153ae2ba
2019-04-13 09:32:43,677: extracting archive file /home/cloud/.pytorch_pretrained_bert/9c41111e2de84547a463fd39217199738d1e3deb72d4fec4399e6e241983c6f0.ae3cef932725ca7a30cdcb93fc6e09150a55e2a130ec7af63975a16c153ae2ba to temp dir /tmp/tmprmsendyc
2019-04-13 09:32:48,700: Model config {
  "attention_probs_dropout_prob": 0.1,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "max_position_embeddings": 512,
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "type_vocab_size": 2,
  "vocab_size": 30522
}

2019-04-13 09:33:17,672: ***** Running training *****
2019-04-13 09:33:17,672:   Num examples = 0
2019-04-13 09:33:17,672:   Batch size = 32
2019-04-13 09:33:17,672:   Num steps = 0
2019-04-13 09:33:17,674: Loading training examples for epoch 0
Training examples: 0it [00:00, ?it/s]
Traceback (most recent call last):
  File "finetune_on_pregenerated.py", line 333, in <module>
    main()
  File "finetune_on_pregenerated.py", line 286, in main
    num_data_epochs=num_data_epochs)
  File "finetune_on_pregenerated.py", line 101, in __init__
    assert i == num_samples - 1  # Assert that the sample count metric was true
UnboundLocalError: local variable 'i' referenced before assignment

Comments:
I found out the issue. The text corpus I was using is just one document. So the code is for two or more documents only?

Yes only for multiple documents.
We have a test now to check that since #478 thanks to @Rocketknight1.
UnboundLocalError: local variable 'special_tokens_file' referenced before assignment
Author: yaroslavvb
Pull request: no

Happens during this
```enc = GPT2Tokenizer.from_pretrained('gpt2')```

```
  File "example_lambada_prediction_difference.py", line 23, in <module>
    enc = GPT2Tokenizer.from_pretrained(model_name)
  File "/bflm/pytorch-pretrained-BERT/pytorch_pretrained_bert/tokenization_gpt2.py", line 134, in from_pretrained
    if special_tokens_file and 'special_tokens' not in kwargs:
UnboundLocalError: local variable 'special_tokens_file' referenced before assignment
```
Looking at offending file, it looks like there's a path for which `special_tokens_file` is never initialized

https://github.com/huggingface/pytorch-pretrained-BERT/blob/3d78e226e68a5c5d0ef612132b601024c3534e38/pytorch_pretrained_bert/tokenization_gpt2.py#L134

Comments:
Yes, this should be fixed by #498.
fp16 overflow in GPT-2
Author: andrewPoulton
Pull request: no

When trying to train in mixed precision, after casting model weights to fp16 overflow is bound to occur since multiplication by 1e10 is used to mask the attention weights.

I noticed BERT multiplies by 1e4 (within fp16 range) instead, and the overflow problem doesn't occur and now it's training happily :)

I'm happy to make the various changes and PR if wanted?

Comments:
Hi @andrewPoulton, yes indeed we could update that for GPT-2, would be happy to get a PR.
Can you check the generations are identical for a few seeds (it should be)?

Yeah, sure - what generations do you mean?

Fixed with #495
Fine-Tuned BERT-base on Squad v1. 
Author: Maaarcocr
Pull request: no

I have fine-tuned the TF model on SQuAD v1 and I've made the weights available at: https://s3.eu-west-2.amazonaws.com/nlpfiles/squad_bert_base.tgz

I get 88.5 FM using these weights on SQuAD dev. (If I recall correctly I get roughly 82 EM). 

I think it may be beneficial to have these weights here, so that people could play with SQuAD and BERT without the need of fine-tuning, which requires a decent enough setup. Let me know what you think!

Comments:
Thanks for the details.
This PyTorch repo is starting to be used by a larger community so we would have to be a little more precise than just rough numbers if we want to include such pre-trained weights.
If you want to add your weights to the repo, you should convert the weights in the PyTorch repo model and get evaluation results on SQuAD with the PyTorch model so everybody has a clean knowledge of what they are using. Otherwise I think it's better that people do their own training and know what are the capabilities of the fine-tuned model they are using.
Feel free to come back and re-open the issue if this something you would like to do.


@thomwolf On SQuAD v1.1, BERT (single) scored 85.083 EM and 91.835 F1 as reported in their paper but when I fine-tuned BERT using `run_squad.py` I got {"exact_match": 81.0975, "f1": 88.7005}. Why there is a difference? What I am missing?
KeyError: in convert_tokens_to_ids()
Author: wasiahmad
Pull request: no
Labels: name: Need more information

In BertTokenizer's, [convert_tokens_to_ids](https://github.com/huggingface/pytorch-pretrained-BERT/blob/19666dcb3bee3e379f1458e295869957aac8590c/pytorch_pretrained_bert/tokenization.py#L117) function gives KeyError. So, I suggest to modify the **for loop** in the function as follows.

```
for token in tokens:
    ids.append(self.vocab.get(token, self.vocab['[UNK]']))
```



Comments:
Hi @wasiahmad,
This should actually already been taken care of by the WordPieceTokenizer ([here](https://github.com/huggingface/pytorch-pretrained-BERT/blob/19666dcb3bee3e379f1458e295869957aac8590c/pytorch_pretrained_bert/tokenization.py#L357)).
Do you have a simple example to share so I can try to reproduce the behavior?

Hi @thomwolf ,
 I alse found the same question when using BertTokenizer class. 
Code likes:
```
sent = ['hi', 'mary']
tokenizer = BertTokenizer.from_pretrained('bert-base-cased')
model_bert = BertModel.from_pretrained('bert-base-cased')
tokenizer.convert_tokens_to_ids(sent)
```

Hi @thomwolf 
I didn't record that token for which I encountered that error but that token was not an English word, it was more like a weird symbol which was probably able to bypass WordPieceTokenizer. I was using BERT as a feature extractor for MSMARCO v2 QA dataset when I encountered the error.

@congjianluo you should not tokenize the sentence your self but use Bert tokenizer.
Please follow the usage example in the [readme](https://github.com/huggingface/pytorch-pretrained-BERT#bert).
In your case:
```python
sent = "hi mary"
tokenizer = BertTokenizer.from_pretrained('bert-base-cased')
tokens = tokenizer.tokenize(sent)
tokens_ids = tokenizer.convert_tokens_to_ids(tokens)
```

@wasiahmad ok then I'm closing this issue.
Feel free to re-open it if you have a (reproductible) example of such issue.
Learning rate schedules improvement + extension
Author: lukovnikov
Pull request: yes

re: [PR#389](https://github.com/huggingface/pytorch-pretrained-BERT/pull/389)
- refactored learning rate schedules into objects
- added `WarmupCosineWithHardRestartsSchedule` for cosine schedule with hard restarts
- added `WarmupCosineWithWarmupRestartsSchedule` for cosine schedule with restarts where each restart uses the same warmup slope

Comments:
Sorry for the delay in reviewing this.
This is a great PR and it looks good to me.
Thanks for adding some tests also.
I agree with the (mostly cosmetic) comments from @marpaia.
Do you think you can fix them and then we can merge?

Fixed @marpaia 's comments.

Awesome @lukovnikov, I think it looks great! Thanks for making this already awesome library even more awesome 💯 

Thanks @lukovnikov!
Fix --reduce_memory in finetune_on_pregenerated
Author: Rocketknight1
Pull request: yes

On reviewing the code I realized the --reduce_memory code path in `finetune_on_pregenerated.py` had a bug, but also wasn't getting used because the relevant argument wasn't getting passed correctly. The bugs have been fixed and the argument is now passed correctly. Performance still seems good, so now it should be possible to train without loading the whole epoch of training data into memory.

Comments:
Good catch!
No GPT2 model
Author: lcswillems
Pull request: no
Labels: name: Need more information

I tried to load the `gpt2` model listed in the README.md, but I got this error:

```
Model name 'gpt2' was not found in model name list (bert-base-uncased, bert-large-uncased, bert-base-cased, bert-large-cased, bert-base-multilingual-uncased, bert-base-multilingual-cased, bert-base-chinese). We assumed 'gpt2' was a path or url but couldn't find any file associated to this path or url.
```

The code I used:
```
# Load pre-trained model tokenizer (vocabulary)
tokenizer = BertTokenizer.from_pretrained('gpt2')

# Load pre-trained model (weights)
model = BertModel.from_pretrained('gpt2')
_ = model.eval()
```

Comments:
Do you have a working internet connection?
We should probably improve the error messages here, 2 different error are bundled in this error (no internet connection and wrong model name)

Yes, I have an internet connection. I am able to download the other models.

Oh wait, you are mixing two models here.
GPT-2 and BERT are two different architectures.
If you want to use GPT-2 do:
```
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
model = GPT2Model.from_pretrained('gpt2')
```
An example of usage is [here in the readme](https://github.com/huggingface/pytorch-pretrained-BERT#openai-gpt-2)


Okay, thank you! Sorry, for this...
Init BertForTokenClassification from from_pretrained
Author: ailzhang
Pull request: no

```
model = BertForTokenClassification.from_pretrained('bert-base-uncased', 2)
```
will complain about missing positional arg for `num_labels`. 
The root cause is here the function signature should actually be 
https://github.com/huggingface/pytorch-pretrained-BERT/blob/19666dcb3bee3e379f1458e295869957aac8590c/pytorch_pretrained_bert/modeling.py#L522
```
    def from_pretrained(cls, pretrained_model_name_or_path,  *inputs, state_dict=None, cache_dir=None, from_tf=False, **kwargs):
``` 
But note that the signature above above is actually only supported in py3 not py2. See a similar workaround here: https://github.com/pytorch/pytorch/pull/19247/files#diff-bdb85c31edc2daaad6cdb68c0d19bafbR300

Comments:
actually this is related to my current work, I will send a fix along with my PR.
Fix training schedules in examples to match new API
Author: thomwolf
Pull request: yes

Re #445:
- update examples to work with the new optimizer API

Comments:
@lukovnikov do you want to give this PR a look and confirm it's fine?

Also, we should document a bit the new optimizer API in the README. Do you want to use this PR to copy a few docstring in the README (we currently don't have auto-generated doc)?

Hi. Sorry, forgot about the examples.
Did a couple fixes in my 'schedules_in_examples' branch (see PR #531).
However, I don't have the fp16 setup yet so wasn't able to run the examples to be completely sure.
Docs update is here: PR #533.

Got it.
Ok to merge this PR @lukovnikov?

With the fixes from #531, should be good.

Thanks!
fixed BertForMultipleChoice model init and forward pass
Author: dhpollack
Pull request: yes

the number of choices is not respected because you've hardcoded '1' into the classifier layer.  also `token_type_ids` and `attention_mask` will cause an error if `None` because `None` does not have a `view` method.

Comments:
Indeed, it looks better.
Do you want to have a look and confirm @rodgzilla?

@thomwolf any word on this?

Oh yes sorry. Looking at it and reading Alec Radford's paper on GPT (section 3.3) again, I think @rodgzilla was actually right in the original implementation.

So I guess we should close this PR.

I still would have been happy to get @rodgzilla input on that.

Oh sorry, we should still keep the `token_type_ids` and `attention_mask` `NoneType` fixes.
These ones are correct!
ImportError: cannot import name 'WEIGHTS_NAME' from 'pytorch_pretrained_bert.file_utils'
Author: lcswillems
Pull request: no

I just tried to run `run_squad.py` example and I got this error:

```
Traceback (most recent call last):
  File "run_squad.py", line 37, in <module>
    from pytorch_pretrained_bert.file_utils import PYTORCH_PRETRAINED_BERT_CACHE, WEIGHTS_NAME, CONFIG_NAME
ImportError: cannot import name 'WEIGHTS_NAME' from 'pytorch_pretrained_bert.file_utils' (/mnt/Data/miniconda3/lib/python3.7/site-packages/pytorch_pretrained_bert/file_utils.py)
```

Comments:
Same is happening for `run_classifier.py` 

Yes the examples currently require to install from source (see the section in the readme).
I'll release a new version tomorrow so the pip release will be in sync with `master` examples again.

Okay, thank you :)

Waiting for this; installing from source gives the error : `ImportError: cannot import name 'warmup_linear'`

> Waiting for this; installing from source gives the error : `ImportError: cannot import name 'warmup_linear'`

It is not actually using `warmup_linear` so you can safely remove that from the file 

Ok, I've just published and uploaded the new v0.6.2 release on pip which should fix this (among other things). Release notes are [here](https://github.com/huggingface/pytorch-pretrained-BERT/releases/tag/v0.6.2).
'NoneType' object with constructor
Author: leelaylay
Pull request: no

I run the code below and often get 'NoneType' object. (I usually run multiprocessing)
```python
model = BertModel.from_pretrained('bert-base-uncased')
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
```

Comments:
Me too.It is probably Network connection problem.

The network connection check has been relaxed in the now merged #500.
It will be included in the next PyPI release (probably next week).
In the meantime you can install from `master`.

@thomwolf  Thank you.

The new release is on pypi!
Can we use 'bert-base-uncased' to question_answer just for start, rather rather than run_squad pretraining?
Author: search4mahesh
Pull request: no

Hi, 
Can we use 'bert-base-uncased' to question_answer just for start, rather rather than run_squad pretraining?

model = BertForQuestionAnswering.from_pretrained('bert-base-uncased') 

Thanks
Mahesh

Comments:
Hi, no you need to fine tune the model on a question answering task like SQuAD before you can use it
Update example files so that tr_loss is not affected by args.gradient…
Author: Mathieu-Prouveur
Pull request: yes

Hi developpers!

Fix training loss value : 

* if gradient_accumulation_steps > 1 then the batch loss value(which is a mean) is scaled by a factor 1/args.gradient_accumulation_steps.

To compare it to evaluation loss it is thus necessary to scale it back by multiplying by args.gradient_accumulation_steps (as done in finetuning script)

Another way to fix this would be to replace the lines with tr_loss/nb_tr_steps by tr_loss/global_step. I thought you might want to consider this alternative

Comments:
Hi @Mathieu-Prouveur, thanks for that.
Indeed I think using `tr_loss/global_step` would be more easy to read.
Can you update this? 

Sure, I've just done the update 

Great, thanks!
Fix GPT2 crash on special quotes in Python 3
Author: AdamDanielKing
Pull request: yes

In Python 3 the line
https://github.com/huggingface/pytorch-pretrained-BERT/blob/b832d5bb8a6dfc5965015b828e577677eace601e/pytorch_pretrained_bert/tokenization_gpt2.py#L224
splits `token` into full characters, not UTF-8 bytes, so for example the right single quote ’ gives `ord('’') == 8217`. That causes a crash since it's a much larger key than any in `self.byte_encoder`. The official GPT2 repo uses `token.encode('utf-8')` but it doesn't work the same in Python 2. I've suggested a fix that uses `token.encode` only in Python 3.

Tested on Python 3.7 but not Python 2.

Thanks for this very useful repo!

Comments:
Thanks, this is closed now with #564
no to_json_file(file) in BERT
Author: seanie12
Pull request: no

Hi, https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/examples/run_squad.py#L1035
in the line 1035, I cannot use config.to_json_file(output_config_file) because there is no such function.
Instead I use
`file = model_to_save.config.to_json_string()`
`with open(file_path, "w") as f:`
`    f.write(file)`
Is it correct way to save config file?

Comments:
Are you using the latest release (0.6.2) ?

Yes I am.

Strange, `to_json_file` should be provided in 0.6.2 (cf code [here](https://github.com/huggingface/pytorch-pretrained-BERT/blob/e6cf62d49945e6277b5e4dc855f9186b3f789e35/pytorch_pretrained_bert/modeling.py#L222) and the associated test [here](https://github.com/huggingface/pytorch-pretrained-BERT/blob/68a889ee43916380f26a3c995e1638af41d75066/tests/modeling_test.py#L258))

Thanks :0) I'll check the version and code again.

After removing  and install the package, the problem solved
Fix bug in run_squad.py
Author: MottoX
Pull request: yes

I fired an issue in Google's repo, https://github.com/google-research/bert/issues/540#issue-428344784

After testing, I found it is indeed a bug.
We should not put these chunks with `start_position==0` and `end_position==0` into training set.

Thanks for code review.

Comments:
Hi @MottoX, sorry for the delay on reviewing this.
It seems to make sense to me.
What kind of testing are you referring to? Could you share a bit more about them?

Hi, @thomwolf 
I was trying to train a BERT-based model on NewsQA, a document-level QA dataset, using `run_squad.py`. I used 384 for max_seq_length and 128 for doc_stride. The model got an extremely bad result. I found there were numerous chunks with start_position and end_position as zeros.
Then I checked the implementation of `run_squad.py` as well as `BertForQuestionAnswering`, which is basically BERT + one fully connected layer. I suspected that such out-of-span chunks are harmful for model to converge. So I modified the script to drop all these out-of-span chunks during training and got an acceptable result. I am sorry that I do not remember the exact numbers before and after doing this.

I think the reason why the problem is not identified before is that context length (token-level) in SQuAD is not that long, which means the model can still easily converge anyway, with or without out-of-span chunks during training as they are really rare. However, when we have numerous out-of-span chunks in the training set, there will be a problem for model to converge.

I looked at the code again, and found it is quite confusing, since chunks with `start_position==0` and `end_position==0` are also used for the cases where we have examples that do not have answer (for version_2_with_negative). So the current fix seems not compatible with this. Now I am not exactly sure whether we need to train the model with those out-of-span chunks. However, from my last experience, there should be a problem when we have too many such out-of-span chunks in the training set and the model is biased to predicting the result on the first token (CLS), despite the fact that `write_predictions` filters out those invalid predictions.