noamaanMulla-03/transformers-issues huggingface dataset:
run_squad script gets stuck
Author: samyam
Pull request: no

Hello, 

I am trying to run the squad fine tuning script, but it hangs after printing out a few predictions. I am attaching the log. Can you help take a look?

I am running the script on a machine with 8 M40s.

[bert_squad.log](https://github.com/huggingface/pytorch-pretrained-BERT/files/2634588/bert_squad.log)


Best,
Samyam

Comments:
Never mind, it just needed time to process the examples. It might be good to have the progress bar inside convert_examples_to_features.

Maybe try distributed training? I don't think PyTorch `DataParallel` will be very efficient on 8 GPUs due to the python GIL.

Thanks for the suggestion. I will try that. Currently, its showing me about 9 hours to fine tune bert-large on squad with batch size of 32 using DataParallel.

The performance improves quite a bit if a if I use a batch size of 256 with gradient accumulate, which makes sense as this reduces the frequency of communication of the gradients. A question I have is, does the learning rate adapt automatically to the batch size being used?  Have you tried larger batch sizes?
[Feature request ] Add support for the new cased version of the multilingual model
Author: elyase
Pull request: no

https://github.com/google-research/bert/commit/332a68723c34062b8f58e5fec3e430db4563320a

Comments:
Hi @elyase, this model is now added in the new release 0.3.0.
I also added the other new model by Google (`bert-large-cased`)
BERTConfigs in example usages in `modeling.py` are not OK (?)
Author: davidefiocco
Pull request: no

Hi!

In the `config` definition https://github.com/huggingface/pytorch-pretrained-BERT/blob/21f0196412115876da1c38652d22d1f7a14b36ff/pytorch_pretrained_bert/modeling.py#L848
in the Example usage of `BertForSequenceClassification` in `modeling.py`, there's things I don't understand:

- `vocab_size` in not an acceptable parameter name, by looking at the `BertConfig` class definition https://github.com/huggingface/pytorch-pretrained-BERT/blob/21f0196412115876da1c38652d22d1f7a14b36ff/pytorch_pretrained_bert/modeling.py#L70

- even by changing `vocab_size` into `vocab_size_or_config_json_file`, for the choice of the other params given in the example i.e.
        ```  
        vocab_size=32000, hidden_size=512, num_hidden_layers=8, num_attention_heads=6,     intermediate_size=1024
        ``` 
I get:
`ValueError: The hidden size (512) is not a multiple of the number of attention heads (6)`
 I think that something similar may be true for the other classes as well, `BertForQuestionAnswering`, `BertForNextSentencePrediction`, etc.

Am I missing something?


Comments:
Hi @davidefiocco, you are right, I updated the docstrings in the new release 0.3.0.
Specify a model from a specific directory for extract_features.py
Author: johann-petrak
Pull request: no

I have downloaded  the model and vocab files into a specific location, using their original file names, so my directory for bert-base-cased contains:
```
bert-base-cased-vocab.txt
bert_config.json
pytorch_model.bin
```

But when I try to specify the directory which contains these files for the `--bert_model` parameter of `extract_features.py` I get the following error:
```
ValueError: Can't find a vocabulary file at path <THEDIRECTORYPATHISPECIFIED> ...
```

When I specify a file that exists and is a proper file, the error messages seem to indicate that the program wants to untar and uncompress the files.

Is there no way to just specify a specific directory that contains the vocab, config, and model files?

Comments:
The last update broke this, but you can fix this in tokenization.py, you have to add this after `vocab_file = pretrained_model_name`:
```
if os.path.isdir(vocab_file):
    vocab_file = os.path.join(vocab_file, "vocab.txt")
```


Thank you, is it fair to assume that this will get accepted as an issue and fixed in a future update/release?

Yes :-) There is a new release planned for tonight that will fix this (among other things, basically all the other open issues).

Ok, this is now included in the new release 0.3.0 (by #73).
Third release
Author: thomwolf
Pull request: yes

This third release comprise the following updates:
- added the two new pre-trained model from Google: `bert-large-cased` and `bert-multilingual-cased`,
- added a model for token-level classification: `BertForTokenClassification`,
- added tests for every model class, with and without labels,
- fixed tokenizer loading function `BertTokenizer.from_pretrained()` when loading from a directory containing a pretrained model,
- fixed typos in model docstrings and completed the docstrings,
- improved examples (added `do_lower_case`arguments).


Comments:
Accuracy on classification task is lower than the official tensorflow version
Author: ejld
Pull request: no

Hi, I am running the same task with the same hyper parameters as the official Google Tensorflow implementation of BERT, however, I am getting around 1.5% lower accuracy.  Can you please give any hint about the possible cause?

Thanks！

Comments:
Hi!
Could it be different seeds?
See e.g. https://github.com/huggingface/pytorch-pretrained-BERT/issues/53#issuecomment-441565229

Hi @ejld, yes BERT has a large variance on many fine-tuning tasks (see also the discussion in #64).
You should try a bunch of different seeds (like 10 seeds for example) and compare the mean and standard deviation of the results.
Wrong signature in model call in run_classifier.py example (?)
Author: davidefiocco
Pull request: no

I think that
https://github.com/huggingface/pytorch-pretrained-BERT/blob/063be09b714bf4d2fbbc3de7f52c45b8bc6817eb/examples/run_classifier.py#L608

may well have a problem, as it's not consistent with
https://github.com/huggingface/pytorch-pretrained-BERT/blob/063be09b714bf4d2fbbc3de7f52c45b8bc6817eb/examples/run_classifier.py#L549
nor with
https://github.com/huggingface/pytorch-pretrained-BERT/blob/063be09b714bf4d2fbbc3de7f52c45b8bc6817eb/pytorch_pretrained_bert/modeling.py#L875

and this currently breaks the example.
One quick patch would be to replace that line with 

```
tmp_eval_loss = model(input_ids, segment_ids, input_mask, label_ids)
logits = model(input_ids, segment_ids, input_mask)
```

But I am not so sure, there are likely better ways.

Comments:
You are right, I also encountered this small error.

Thanks for noticing, fixed in #77.
TypeError: object of type 'WindowsPath' has no len()
Author: Deep1994
Pull request: no

Hi, when I run "tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')", the error "TypeError: object of type 'WindowsPath' has no len()" occurs, what is the problem? Thank you for your excellent code!

Comments:
Can you post a more detailed log?

I install your PyTorch pretrained bert with pip like "pip install pytorch-pretrained-bert", then I run the code in Usage section like:

`import torch`
`from pytorch_pretrained_bert import BertTokenizer, BertModel, BertForMaskedLM`

`# Load pre-trained model tokenizer (vocabulary)`
`tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')`

but there is an error occurs, the error information is: 

Traceback (most recent call last):

  File "<ipython-input-2-7725148c607d>", line 5, in <module>
    tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

  File "C:\Users\Deep\Anaconda3\lib\site-packages\pytorch_pretrained_bert\tokenization.py", line 117, in from_pretrained
    resolved_vocab_file = cached_path(vocab_file, cache_dir=cache_dir)

  File "C:\Users\Deep\Anaconda3\lib\site-packages\pytorch_pretrained_bert\file_utils.py", line 88, in cached_path
    return get_from_cache(url_or_filename, cache_dir)

  File "C:\Users\Deep\Anaconda3\lib\site-packages\pytorch_pretrained_bert\file_utils.py", line 169, in get_from_cache
    os.makedirs(cache_dir, exist_ok=True)

  File "C:\Users\Deep\Anaconda3\lib\os.py", line 226, in makedirs
    head, tail = path.split(name)

  File "C:\Users\Deep\Anaconda3\lib\ntpath.py", line 204, in split
    d, p = splitdrive(p)

  File "C:\Users\Deep\Anaconda3\lib\ntpath.py", line 139, in splitdrive
    if len(p) >= 2:

TypeError: object of type 'WindowsPath' has no len()

Strange error. I am only using standard library here. Maybe it has something to do with your installation of Conda. You can try to manually specify a cache directory for the package by either:
- setting the environment variable `PYTORCH_PRETRAINED_BERT_CACHE=XXX` to a directory `XXX` you created to store the downloaded models.
- sending the path to this directory to the tokenizer and model using the `cache_dir=XXX` arguments, for example: `tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', cache_dir=XXX)`

I follow your second instruction and change the code to:

`tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', cache_dir='C:/Users/Deep/Anaconda3/Lib/site-packages')`

It works! Thank you!
numpy.core._internal.AxisError: axis 1 is out of bounds for array of dimension 1
Author: A-Rain
Pull request: no

hello, when I am running run_classifier.py with MRPC dataset, there seems to be an mistake. the mistake is as following:
<img width="752" alt="default" src="https://user-images.githubusercontent.com/29532760/49360256-9de0e100-f713-11e8-9a5c-d9f2bc5331e6.PNG">
the mistake is happening when training is over and the model is for evaluating

```
with torch.no_grad():
                tmp_eval_loss, logits = model(input_ids, segment_ids, input_mask, label_ids)
```
here I found the size of logits is []

I'm using python3.5 and torch=0.4.1, I don't know how to fix it.



Comments:
Hi, just update the repo to the current master, this should have been fixed this weekend (re-open the issue of it's not).
AttributeError: 'tuple' object has no attribute 'backward'
Author: Qzsl123
Pull request: no

Traceback (most recent call last):                                                                                                                                    | 0/11 [00:00<?, ?it/s] 
  File "examples/run_classifier.py", line 637, in <module>
    main()
  File "examples/run_classifier.py", line 558, in main
    loss.backward()
AttributeError: 'tuple' object has no attribute 'backward'


Comments:
Looks like there was a code change which changed the forward method of the model involved here from returning a tensor to returning a tuple of tensors and the example hasn't been updated yet to reflect that change. There's probably a line in run_classifier.py like
```Python
loss = model(input...)
```
which now needs to be
```Python
loss, something_else = model(input...)
```

> Looks like there was a code change which changed the forward method of the model involved here from returning a tensor to returning a tuple of tensors and the example hasn't been updated yet to reflect that change. There's probably a line in run_classifier.py like
> 
> ```python
> loss = model(input...)
> ```
> 
> which now needs to be
> 
> ```python
> loss, something_else = model(input...)
> ```

You are right! Thx!
code in run_squad.py line 263
Author: xilinniao123
Pull request: no

 # Zero-pad up to the sequence length.
 while len(input_ids) < max_seq_length:
        input_ids.append(0)
        input_mask.append(0)
        segment_ids.append(0)
in segment_ids array,1 indicates token from passage and 0 indicate token form query.
when padding,why segment_ids filled with 0,which represents query

Comments:
![image](https://user-images.githubusercontent.com/11830865/49438135-61d37c00-f7f8-11e8-8b2a-a7222bd30f0e.png)


Hi, what is your question?

Strictly speaking, the zero-padding in segment_ids leads to ambiguous tensor entries, because 0 can mean both "first sentence" (or query in another task?) and "padding".

But in practice this isn't a problem because anything related to padding gets masked out later.
Squad dataset has multiple answers to a question.
Author: nischalhp
Pull request: no

https://github.com/huggingface/pytorch-pretrained-BERT/blob/3ba5470eb85464df62f324bea88e20da234c423f/examples/run_squad.py#L143

The confusing part here is that in line 146, only the first answer is considered, so I am wondering why is there a check for multiple answers before.
Also, SQuad dataset has multiple answers for the same question. Is this by design or am I fundamentally missing something?

Comments:
Hi,
In `train-v2.0.json`, there is only one answer for the question.
In `dev-v2.0.json` and hidden `test-v2.0.json`, there are several answers for a given question.
I think the code that you mentioned is designed for not mistakenly using `dev-v2.0.json` for training. If you are going to use your own data or other types of data that has multiple answers, you can simply comment out this part.

Best

Hello @ymcui ,

I did exactly that, thank you for confirming. Just wanted to be sure that there are no other implications. You are right, I have converted our dataset into SQuAD form and using that with the model. 

Regards,
Nischal
There is some problem in supporting continuously training
Author: ZacharyWaseda
Pull request: no

I change the run_classfifier.py in order to support continuously training. i save the model.state_dict() and the BertAdam optimizer.state_dict(), and I load them when start continuously training. However, After some epochs, the loss will increase little by little and finally end with a large loss value. I do not know the reason. Please help me.

Comments:
Hi @ZacharyWaseda, continuous training is an open-research problem. You should rather seek some solution in the papers/workshop/conference discussing researches in this field. This is not my personal field of expertise so I can only direct you to google and other search engine for more information.
bert-base-multilingual-cased - Text bigger than 512
Author: agemagician
Pull request: no

Hello,

I am trying to extract features from German text using bert-base-multilingual-cased. However, my text is bigger than 512 words.
Is there any way to use the pertained Bert for text greater than 512 words 

Comments:
Hello,

I do not think that it is possible out of the box. The article states the following:

> We use learned positional embeddings with supported sequence lengths up to 512 tokens.

The positional embeddings are therefore limited to 512 tokens. You may be able to add positional embeddings for position greater than 512 and learn them on your specific dataset but I don't know how efficient that would be.

Hi @agemagician, you cannot really use pretrained bert for text longer than 512 tokens per se but you can use the sliding window approach.

Check this issue of the original bert repo for more details: https://github.com/google-research/bert/issues/66
weights initialized two times
Author: friskit-china
Pull request: no

Hi,

I found that you initilized all weights twice:

The first one is in BertModel class:

https://github.com/huggingface/pytorch-pretrained-BERT/blob/3ba5470eb85464df62f324bea88e20da234c423f/pytorch_pretrained_bert/modeling.py#L586

And the second one is in classes of each tasks such as in BertForSequenceClassification class:

https://github.com/huggingface/pytorch-pretrained-BERT/blob/3ba5470eb85464df62f324bea88e20da234c423f/pytorch_pretrained_bert/modeling.py#L674

I think maybe you only need the second one?

Comments:
I think it required for both the places. Because both of them can be used individually. As it is mentioned in the README.md file, the model can be loaded with 7 classes. In fact if you check `BertForMaskedLM` and `BertForNextSentencePrediction` classes it also has the weights initialised.

Please correct me if I am wrong :)

You are right @Arjunsankarlal :-)
Picking max_sequence_length in run_classifier.py CoLA task
Author: artemlos
Pull request: no

Is there an upper bound for the max_sequence_length parameter when using run_classifier.py with CoLA task?

When I tested with the default max_sequence_length of 128, everything worked good, but once I changed it to something else, eg 1024, it started the training and failed on the first iteration with the error shown below:
````
Traceback (most recent call last):
  File "run_classifier.py", line 643, in <module>
    main()
  File "run_classifier.py", line 551, in main
    loss = model(input_ids, segment_ids, input_mask, label_ids)
  File "/jet/var/python/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/jet/var/python/lib/python3.6/site-packages/pytorch_pretrained_bert/modeling.py", line 868, in forward
    _, pooled_output = self.bert(input_ids, token_type_ids, attention_mask, output_all_encoded_layers=False)
  File "/jet/var/python/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/jet/var/python/lib/python3.6/site-packages/pytorch_pretrained_bert/modeling.py", line 609, in forward
    embedding_output = self.embeddings(input_ids, token_type_ids)
  File "/jet/var/python/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/jet/var/python/lib/python3.6/site-packages/pytorch_pretrained_bert/modeling.py", line 199, in forward
    embeddings = self.dropout(embeddings)
  File "/jet/var/python/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/jet/var/python/lib/python3.6/site-packages/torch/nn/modules/dropout.py", line 53, in forward
    return F.dropout(input, self.p, self.training, self.inplace)
  File "/jet/var/python/lib/python3.6/site-packages/torch/nn/functional.py", line 595, in dropout
    return _functions.dropout.Dropout.apply(input, p, training, inplace)
  File "/jet/var/python/lib/python3.6/site-packages/torch/nn/_functions/dropout.py", line 40, in forward
    ctx.noise.bernoulli_(1 - ctx.p).div_(1 - ctx.p)
RuntimeError: Creating MTGP constants failed. at /jet/tmp/build/aten/src/THC/THCTensorRandom.cu:34
```` 

The command I ran is 

```
python run_classifier.py \
  --task_name CoLA \
  --do_train \
  --do_eval \
  --do_lower_case \
  --data_dir $GLUE_DIR/Test/ \
  --bert_model bert-base-uncased \
  --max_seq_length 128 \
  --train_batch_size 32 \
  --learning_rate 2e-5 \
  --num_train_epochs 3.0 \
  --output_dir /tmp/BERT/test1
````

Comments:
As mentioned in #89, the maximum value of `max_sequence_length` is 512. 

@rodgzilla thanks!
run_classifier.py improvements
Author: rodgzilla
Pull request: yes

Hi !

This PR contains multiple improvements to the `run_classifier.py` file. 

The changes are:
- removing trailing whitespaces ([PEP 8](https://www.python.org/dev/peps/pep-0008/)),
- simplifying a bit the data processing code, in particular tensor formatting,
- fixing issue #83 by adapting the value of the `num_labels` argument of `BertForSequenceClassification.from_pretrained` to the dataset being used. 

Comments:
Neat!
Words after tokenization replaced with #
Author: nischalhp
Pull request: no

Hello,

When training the bert-base-multilingual-cased model for Question and Answering, I see that the tokens look like this : 

```tokens: [CLS] what is the ins ##ured _ name ? [SEP] versi ##cherung ##ss ##che ##in erg ##o hau ##srat ##versi ##cherung hr - sv 927 ##26 ##49 ##2 ```

Any idea why words are getting replaced with #? 

Here is the command I am using : 

```python run_squad.py --bert_model bert-base-multilingual-cased --do_train --do_predict --train_file dataset_train.json --predict_file dataset_predict.json --train_batch_size 12 --learning_rate 3e-5 --num_train_epochs 2.0 --max_seq_length 400 --doc_stride 20 --output_dir output_dir```

Comments:
Because it uses WordPiece tokenization, and will introduce the `#` token.
Check: https://github.com/google-research/bert#tokenization

@ymcui okay sweet, thank you. Will use the relevant one. 

@ymcui How do I change this ? or is not possible to do so?

1. If you are training completely from scratch, then it will be possible to use your own tokenizer.
2. However, if you are fine-tuning on the existing pre-trained BERT models, I think it will not be possible to change the tokenizer, as the pre-trained BERT models are trained using WordPiece tokenizer. 

@ymcui is right.

Since the purpose of the present repo is to supply pre-trained model basically you are stuck with WordPiece tokenization.

If you build a new model and train it from scratch, you can selected whatever tokenization you want :-)

@ymcui @thomwolf - Yes, that is quite a problem and thanks for getting back. Evaluating building something on our own now 🗡
What is the best dataset structure for BERT?
Author: wahlforss
Pull request: no

First I want to say thanks for setting up all this!

I am using BertForSequenceClassification and am wondering what the optimal way is to structure my sequences. 

Right now my sequences are blog post which could be upwards to 400 words long. 

Would it be better to split my blog posts in sentences and use the sentences as my sequences instead? 

Thanks!

Comments:
update: add from_state_dict for PreTrainedBertModel
Author: friskit-china
Pull request: yes

For restoring the training procedure. 
Now we can use torch.save to store their model and restore their model by e.g. `model = BertForSequenceClassification.from_state_dict('bert-large-uncased', state_dict=torch.load('xx.pth'))`

Comments:
Hi, I like the idea but I not a big fan of all the code duplication I'ld rather fuse the two loading functions in one.
Basically we can just add a `state_dict` argument to `from_pretrained` and add a check in `from_pretrained` to handle the case.

> Hi, I like the idea but I not a big fan of all the code duplication I'ld rather fuse the two loading functions in one.
> Basically we can just add a `state_dict` argument to `from_pretrained` and add a check in `from_pretrained` to handle the case.

Hi, thanks for your reply, I just tried to make sure that my code will not incorporate bugs so I added a new function instead of changing code in `from_pretrained`.
:)