noamaanMulla-03/transformers-issues huggingface dataset:
Fix to prevent crashing on assert len(tokens_b)>=1
Author: searchivarius
Pull request: yes

Thank you for the awesome library! One little issue that I have sometimes with somewhat "noisy" text is that Bert tokenizer fails to process some weird stuff. In such a case, one unexpectedly gets no tokens and the converter fails on assert with a message like this one:
```
Epoch:   0%|          | 0/1 [00:00<?, ?it/s]                      Traceback (most recent call last):
  File "../pytorch-transformers/examples/lm_finetuning/pregenerate_training_data.py", line 354, in <module>
    main()
  File "../pytorch-transformers/examples/lm_finetuning/pregenerate_training_data.py", line 350, in main
    create_training_file(docs, vocab_list, args, epoch)
  File "../pytorch-transformers/examples/lm_finetuning/pregenerate_training_data.py", line 276, in create_training_file
    whole_word_mask=args.do_whole_word_mask, vocab_list=vocab_list)
  File "../pytorch-transformers/examples/lm_finetuning/pregenerate_training_data.py", line 244, in create_instances_from_document   
    assert len(tokens_b) >= 1
AssertionError

```

Comments:
Ok, merging.
Note that we are in the process of deprecating these finetuning scripts and replacing them with the common `run_lm_finetuning.py` which handles several models.

Thank you @thomwolf sorry didn't realize this was deprecated.
Performing optimization on CPU
Author: ikuyamada
Pull request: no
Labels: name: wontfix

In README.md, it is explained that the optimization step was performed on CPU to train a SQuAD model. 

> perform the optimization step on CPU to store Adam's averages in RAM.

https://github.com/huggingface/pytorch-pretrained-BERT#fine-tuning-bert-large-on-gpus

Is it still supported in `run_squad.py`?
If I understand correctly, the current implementation always places the Adam's averages on GPUs if we train the model using GPUs.

Comments:
Oh yes you are right, we removed that, I'll update the readme

Thank you for your reply!
Because Adam's averages (i.e., `next_m` and `next_v`) are large, it is very helpful to support performing optimization step on CPU to reduce GPU memory.
Therefore, I would like to know how you implemented this.
Did you move the parameters to CPU (e.g., `grad.to(torch.device('cpu'))`) in `BertAdam.step` function?

@ikuyamada In the old repository, the model parameters on gpu obtained by forward and backward are copied to the optimizer that stores the model parameters on cpu, and the updated model parameters are returned to the model on gpu at each training step.
Check https://github.com/huggingface/pytorch-pretrained-BERT/blob/v0.3.0/examples/run_squad.py#L682

@thomwolf I have questions. Why did you decide to delete `optimize_on_cpu`? Does cpu and gpu parameter transfer affect training speed?


@ryonakamura thank you for the pointer!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
ModuleNotFoundError in distillation/scripts/binarized_data.py
Author: MatejUlcar
Pull request: no

Hello,

importing logger (still? I've seen previous issues, but this is the first time I'm running the code myself) throws the ModuleNotFoundError. 

```
Traceback (most recent call last):
  File "examples/distillation/scripts/binarized_data.py", line 25, in <module>
    from examples.distillation.utils import logger
ModuleNotFoundError: No module named 'examples.distillation'
```

Comments:
Hello @MatejUlcar 
Thanks for pointing that out. I fixed it once and for all by having a local logger (and not importing the global one) in commit 32e1332acf6fd1ad372b81c296d43be441d3b0b1.
Different performance between pip install vs. download zip code
Author: wyin-Salesforce
Pull request: no

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Hi guys,
I meet a weird  problem. Basically I am using BERT for sentence pair classification, such as MNLI, RTE tasks.

I installed the pytorch_transformers by "pip install pytorch-transformers" and 

> from pytorch_transformers.modeling_bert import BertForSequenceClassification

everything works fine, the performance on RTE can reach 70%.

Instead, I downloaded the zip code to local since I want to do some modification about the BertForSequenceClassification in my future projects. (let's say I renamed the unzipped folder into "my_pytorch_transformers"), and now import as:

> from my_pytorch_transformers.modeling_bert import BertForSequenceClassification

Now, everything was not the same; the performance was just 53%, and it showed difference from the beginning of iterations.

So, what's wrong here?  I thought maybe some initializations are different between the default installation and my local code?  But I always used the pretrained BERT "bert-large-uncased", ....I can not figure out where the difference comes from.

Thanks for any hints











Comments:
sorry I found the problem: the pip source code and the zip code downloaded are different, especially for the "BertForSequenceClassification" class
LSTM returns nan after using the pretrained BERT embedding as input 
Author: laifi
Pull request: no

Hello ,
 i'm using the pretrained Bert model (from pytorch-transformers) to get the contextual embedding of a written text ,  i summed the last 4 hidden layers outputs (i red that the concatenation of the last four layers usually produce the best results )
than i use a LSTM layer with attention to get the paragraph level embedding from the word embedding produced by the Bert model
the output should be a score range [-1,1]
I tried the rmsprop , adam ... optimizers with MSEloss and always after just  few batches iterations the lstm layer produces nan values .
 any suggestions will be greatly appreciated

Comments:
i found that the problem is related to the data
i will close this issue , or if anyone has the permission he could delete this :)
Using BERT as feature extractor
Author: shivanshpatel35
Pull request: no
Labels: name: wontfix

Following file extract_features.py, I use bert-large-uncased and it outputs 4 layer outputs for each token(word). Since I want to use it as feature extractor of entire sentence, which values should I use? Or  is there any other processing we should do(like concatenate output for last token)?

Comments:
i have same question 

How best to fine-tune or pool BERT is an open question in bertology :p 

["How to Fine-Tune BERT for Text Classification?"](https://arxiv.org/pdf/1905.05583.pdf) has a comprehensive overview. Look at table 3 specifically which found that taking the max of the last 4 layers achieves the best performance (in their testing configuration with some fine tuning), this would be my guess. 

You can also look at the approaches taken by [bert-as-a-service](https://github.com/hanxiao/bert-as-service#q-what-are-the-available-pooling-strategies) which are explained pretty well, in this repo they use MEAN as the default pooling strategy.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
randrange() error when running pregenerate_training_data.py code in lm_finetuning
Author: KavyaGujjala
Pull request: no
Labels: name: wontfix

Hi,

I am trying to run pregenerate_training_data.py code in lm_finetuning using a text file which has two documents ( each document has around 200 sentences )
I ran into this error:

```
Better speed can be achieved with apex installed from https://www.github.com/nvidia/apex.
Loading Dataset: 399 lines [00:00, 2214.07 lines/s]
Epoch:   0%|                                                                                                                                                                  | 0/3 [00:00<?, ?it/s]
Traceback (most recent call last):                                                                                                                                            | 0/1 [00:00<?, ?it/s]
  File "pregenerate_training_data.py", line 292, in <module>
    main()
  File "pregenerate_training_data.py", line 277, in main
    vocab_list=vocab_list)
  File "pregenerate_training_data.py", line 187, in create_instances_from_document
    random_document = doc_database.sample_doc(current_idx=doc_idx, sentence_weighted=True)
  File "pregenerate_training_data.py", line 52, in sample_doc
    sentence_index = randint(rand_start, rand_end-1) % self.cumsum_max
  File "/home/cloud/anaconda3/lib/python3.6/random.py", line 221, in randint
    return self.randrange(a, b+1)
  File "/home/cloud/anaconda3/lib/python3.6/random.py", line 199, in randrange
    raise ValueError("empty range for randrange() (%d,%d, %d)" % (istart, istop, width))
ValueError: empty range for randrange() (198,198, 0)
```

command line prompt looks like

`python pregenerate_training_data.py --train_corpus=./ack_belief_training_testing/ack_belief_all_categories_data2.txt --bert_model=bert-base-uncased --do_lower_case --output_dir=./ack_belief_training_testing/pytorch_gen_data/ack_belief_all_categories_data2_train_data_3epochs/ --epochs_to_generate=3`

What can be the issue? 
Can somebody help me through this?


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Invalid Syntax Error trying to run pregenerate_training_data.py
Author: MNCTTY
Pull request: no
Labels: name: wontfix

I'm getting such error, can't understand what's wrong
![image](https://user-images.githubusercontent.com/37251686/60671996-072cfc00-9e7d-11e9-9b73-b8b1f01652c9.png)

Plus, is it possible to further fine tune once fine tuned model, that appears after simple finetuning .py in corresponding folder (pytorch_model.bin) ?

Comments:
Your snippet is too short to see what type of error there is, can you extract a larger one?

Hi 
yes
![image](https://user-images.githubusercontent.com/37251686/60735756-30bd5480-9f5d-11e9-940c-d39c73cb4084.png)


Pull the latest changes from master and report if that helps.

it has been disappeared in pregenerate script, and appeared in finetuning on pregenerated data script
![image](https://user-images.githubusercontent.com/37251686/60804690-b708af00-a186-11e9-9e21-024e745c8bdf.png)


You are getting these errors because you are using Python 3.5 and the code is making use of f-strings which are introduced in Python 3.6. You could try using Python 3.6 or change the source code replacing f-string with str.format syntax.

@thomwolf The readme says repository supports Python 3.5+. Does that mean Python 3.5 is supported as well? If yes, I think we should change f-strings for the format syntax. 


Well, only the library code supports Python 3.5+, I don't check the examples which are mostly contributed by the community.
If you want to fix Python 3.5 support for the examples I'm happy to welcome a PR.


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
''bert-large-uncased-whole-word-masking-finetuned-squad' CAN'T be reached.
Author: sinboyxx
Pull request: no
Labels: name: wontfix

'bert-large-uncased-whole-word-masking-finetuned-squad' can't be reached from the addr in tokenization.py:

https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-uncased-whole-word-masking-finetuned-squad-vocab.txt


Comments:
also ran into this.  I think they forgot to upload the file/make it public. You can find the vocab file on the original google repo

https://github.com/google-research/bert

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Incorrect training loss scaling factor in examples/run_classifier.py?
Author: ethanjperez
Pull request: no
Labels: name: wontfix

In [examples/run_classifier.py](https://github.com/huggingface/pytorch-pretrained-BERT/commit/87b9ec3843f7f9a81253075f92c9e6537ecefe1c), the overall 'loss' is produce as 'tr_loss/global_step' (instead of 'tr_loss/nb_tr_steps'). Is this behavior correct? @mprouveur made the change in this [commit](https://github.com/huggingface/pytorch-pretrained-BERT/commit/87b9ec3843f7f9a81253075f92c9e6537ecefe1c).

I'm wondering as 'global_step' is never reset after a training epoch, while tr_loss is reset every training epoch. So even if 'tr_loss' remains constant, 'loss' will decrease over more training iterations, given the increasingly large denominator ('global_step').

If this is correct, maybe the 'nb_tr_steps' variable should be removed? It looks unused throughout the code currently.

At any rate, it's only a 2-line fix, and it only affects the logging behavior I believe.

Comments:
You are right Ethan.
I'm refactoring the examples which were a bit rotten, let's include this fix as well.

Great! Either way the examples are a great starting point :)

I'm also wondering if tensorboard is [only logging](https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/examples/run_classifier.py#L345) the training loss for the last forward pass for a batch (if several are required / when using gradient accumulation)? A fix would be to maintain a variable ```tr_batch_loss``` (similar to ```tr_loss```) for each full training batch (reset after each parameter update) and log that instead.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
run_generation.py  'encode' error for gpt2 and xlnet
Author: GenTxt
Pull request: no

Hello:

Been using nshepperd's tf repo and various excellent forks for fine-tuning and inference without issue.

Wanted to check out py-torch transformers and compare. First test is simple conditional sampling from the pytorch models:

python3 run_generation.py --model_type=xlnet --length=20 --model_name_or_path=models/xlnet-large-cased

Loads config.json and model weights then Model prompt >>> 

Enter anything: Hello there huggingface. What's up?

then ...

Traceback (most recent call last):
  File "run_generation.py", line 195, in <module>
    main()
  File "run_generation.py", line 175, in main
    context_tokens = tokenizer.encode(raw_text)
AttributeError: 'NoneType' object has no attribute 'encode'

Same error when switching to gpt2, in this case  gpt-large.

Is there a modification to the script required?  Is the terminal syntax incorrect?

Can't move on to fine-tuning until basic inference sorted out.

Tested both pytorch 1.2.0 and 1.1.0.  Same error.

    OS: ubuntu 18.04
    Python version: python 3.6.8
    PyTorch version: torch 1.2.0 tested and changed to 1.1.0 to match transformer
    PyTorch Transformers version (or branch): pytorch-transformers 1.2.0 tested and changed to 1.1.0 to match above
    Using GPU ? yes
    Distributed of parallel setup ? no

All help appreciated. Would like to test distributed fine-tuning but need the basic repo working first on local machine.

Cheers


Comments:
Closing own non-issue. Needed to download missing files listed in the four applicable tokenizer scripts. Working 100%. On to fine-tuning.
Implementation of 15% words masking in pretraining
Author: jianyucai
Pull request: no
Labels: name: wontfix

In the BERT paper, they randomly mask 15% words for pretraining, and that's exactly what they do in the TF version. 

https://github.com/google-research/bert/blob/0fce551b55caabcfba52c61e18f34b541aef186a/create_pretraining_data.py#L342

However, the implementation here is a little bit different, instead of randomly select 15% tokens, it assigns a probability of 15% to each token, that is, each token has a probability of 15% to be masked. That means, each time we might have less than or more than 15% tokens masked.

So, is it correct to mask tokens with the expectation of 0.15 rather than fixed 15%?

https://github.com/huggingface/pytorch-pretrained-BERT/blob/f9cde97b313c3218e1b29ea73a42414dfefadb40/examples/lm_finetuning/simple_lm_finetuning.py#L276-L301

Comments:
It should be fine. A bit of randomness in the pre-processing of the inputs is never bad when training a deep learning model.

> It should be fine. A bit of randomness in the pre-processing of the inputs is never bad when training a deep learning model.

I found the same problem that the implementation is different from tensorflow. But the key point is not the fixed 15% prob of all the token. If  we use the implementation of pytorch will produce two extreme case especially for short sentences like article title,usually 10-20 characters.
case 1. sentence with too much '[MASK]'
case 2. sentence with none '[MASK]'
both case1 and case2 would cause the drop of performance. case 1 make the model difficult to predict and case2 would not produce the loss.
Given a corpus with an average sentence length of 10. The implementation of tensorflow would generate 1 '[MASK]' for the sentences, but the implementation of pytorch would have :
0.85^10                      = 0.19 to generate 0 '[MASK]'
0.15 * 0.85^9 * 10      =0.34 to generate 1 '[MASK]'
0.15^2 * 0.85^8  * 45 =0.27 to generate 2 '[MASK]'
0.15^2 * 0.85^7 * 120 =0.12 to generate 3 '[MAKS]'
... 

If we roughly consider the sentence with 15% '[MASK]' is  appropriate, only 1/2 '[MASK]' is useful for training models. So only 0.34 + 0.27  = 0.61 training case is useful.

And we found it is this is a very serious problem for short text.





This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
[develop] Convenience args.{train/dev}_file arguments.
Author: kshitij12345
Pull request: yes

Adds Arguments
```
    --train_file any_train_file.tsv \
    --dev_file any_dev_file.tsv \
```
to use any file for training/dev in the pointed data directory.

Especially handy for evaluation.

Allows for
```
python run_glue.py \
    --model_type bert \
    --model_name_or_path bert-base-uncased \
    --task_name $TASK_NAME \
    --do_train \
    --do_eval \
    --train_file any_train_file.tsv \
    --dev_file any_dev_file.tsv \
    --do_lower_case \
    --data_dir $GLUE_DIR/$TASK_NAME \
    --max_seq_length 128 \
    --per_gpu_eval_batch_size=8   \
    --per_gpu_train_batch_size=8   \
    --learning_rate 2e-5 \
    --num_train_epochs 3.0 \
    --output_dir ./Data/Test/
```

Comments:
Why don't give the  full path of the train/dev file instead of  giving data_dir?

My thought was not the change much of the argument interface. If we support full path, then the `data_dir` will not be required and considered. So I wasn't sure if that change is the way to go. Sure, we can change it like that as well.

Personally, I do agree that full path for train/dev is more convenient while using.
Fail to run finetune_on_pregenerated.py
Author: allisonyw
Pull request: no
Labels: name: wontfix

Hi,

I am fine tuning BERT for my own data set.  Pregenerate training data was smooth but when I run finetune_on_pregenerated.py I got the following KeyError:

2019-07-11 22:53:04,151: ***** Running training *****
2019-07-11 22:53:04,151:   Num examples = 35832
2019-07-11 22:53:04,151:   Batch size = 32
2019-07-11 22:53:04,152:   Num steps = 1119
2019-07-11 22:53:04,156: Loading training examples for epoch 0
Training examples:   0%|          | 0/12078 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "finetune-hugging.py", line 348, in <module>
    main()
  File "finetune-hugging.py", line 297, in main
    num_data_epochs=num_data_epochs, reduce_memory=args.reduce_memory)
  File "finetune-hugging.py", line 105, in __init__
    features = convert_example_to_features(example, tokenizer, seq_len)
  File "finetune-hugging.py", line 43, in convert_example_to_features
    input_ids = tokenizer.convert_tokens_to_ids(tokens)
  File "/anaconda3/lib/python3.7/site-packages/pytorch_pretrained_bert/tokenization.py", line 121, in convert_tokens_to_ids
    ids.append(self.vocab[token])
KeyError: 'Ad'
Out[21]: 256

I could really use some help from you guys. Many Thanks!

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Implementation of 15% words masking would cause the drop of performance in short text 
Author: zhangsh950618
Pull request: no
Labels: name: wontfix

I found the same problem that the implementation is different from tensorflow. If we use the implementation of pytorch will produce two extreme case especially for short sentences like article title,usually 10-20 characters.
case 1. sentence with too much '[MASK]'
case 2. sentence with none '[MASK]'
both case1 and case2 would cause the drop of performance. case 1 make the model difficult to predict and case2 would not produce the loss.
Given a corpus with an average sentence length of 10. The implementation of tensorflow would generate 1 '[MASK]' for the sentences, but the implementation of pytorch would have :
0.85^10 = 0.19 to generate 0 '[MASK]'
0.15 * 0.85^9 * 10 =0.34 to generate 1 '[MASK]'
0.15^2 * 0.85^8 * 45 =0.27 to generate 2 '[MASK]'
0.15^2 * 0.85^7 * 120 =0.12 to generate 3 '[MAKS]'
...

If we roughly consider the sentence with 15% '[MASK]' is appropriate, only 1/2 '[MASK]' is useful for training models. So only 0.34 + 0.27 = 0.61 training case is useful.

And we found it is this is a very serious problem for short text.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Why you need DistilBertModel class?
Author: tomohideshibata
Pull request: no

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

You have `DistilBertModel`, `DistilBertForSequenceClassification`, etc. in `modeling_distilbert.py`. Why you need these classes? How about using `BertModel`, `BertForSequenceClassification`, etc.?

I found the weight names are different (e.g., `transformer.layer.0.attention.q_lin.weight`and `bert.encoder.layer.0.attention.self.query.weight`), but I think it's better to use the same weight names.

(A conversion script from distilbert weights to normal bert weights is useful to use `BertModel`.)

Thanks.

Comments:
This is a smaller model compared to the original, and is thus better suitable for usage on embedded devices / devices without large gpu's. Their blog posts explains this: https://medium.com/huggingface/distilbert-8cf3380435b5.

Hello @tomohideshibata,
True, we could use the same code base for BERT and DistilBERT.
For now, I prefer keeping them separate mainly for clarity since the architectures are slightly different:
- No token_type_embeddings in DistilBERT
- No Sequence Pooler in DistilBERT
Handling these two in BertModel would unnecessarily (slightly) complexify the code and I'd like to keep it clean.
Another caveat: I use torch's `nn.LayerNorm` in DistilBERT while BERT uses a custom `BertLayerNorm`. There might be slightly different edge cases (I have to check).
But overall, you can totally implement a single class for BERT and DistilBERT on your side. I would suggest to have a look at [this script](https://github.com/huggingface/pytorch-transformers/blob/master/examples/distillation/scripts/extract_for_distil.py) to have the mapping between the names.

@VictorSanh Thanks for your comments. I will try to make a conversion script from DistilBERT to BERT weights.
Issue 
Author: bmanishreddy
Pull request: no
Labels: name: wontfix


Comments:
@bmanishreddy Your issue contains no text, should it be closed?

Yeah .. my bad it can be closed 

No worries, please close it so it doesn't create clutter.  Thanks!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.