noamaanMulla-03/transformers-issues huggingface dataset:
fix compatibility with python 3.5.2
Author: hzhwcmhf
Pull request: yes

When I run the following command on python 3.5.2

```
python3 extract_features.py --input_file input.txt --output_file output.txt --bert_model bert-base-uncased --do_lower_case
```

Get this error:

```
Traceback (most recent call last):
  File "extract_features.py", line 298, in <module>
    main()
  File "extract_features.py", line 231, in main
    tokenizer = BertTokenizer.from_pretrained(args.bert_model, do_lower_case=args.do_lower_case)
  File "/home/huangfei/.local/lib/python3.5/site-packages/pytorch_pretrained_bert/tokenization.py", line 117, in from_pretrained
    resolved_vocab_file = cached_path(vocab_file, cache_dir=cache_dir)
  File "/home/huangfei/.local/lib/python3.5/site-packages/pytorch_pretrained_bert/file_utils.py", line 88, in cached_path
    return get_from_cache(url_or_filename, cache_dir)
  File "/home/huangfei/.local/lib/python3.5/site-packages/pytorch_pretrained_bert/file_utils.py", line 169, in get_from_cache
    os.makedirs(cache_dir, exist_ok=True)
  File "/usr/lib/python3.5/os.py", line 226, in makedirs
    head, tail = path.split(name)
  File "/usr/lib/python3.5/posixpath.py", line 103, in split
    i = p.rfind(sep) + 1
AttributeError: 'PosixPath' object has no attribute 'rfind'
```

I find makedirs didn't support PosixPath in python3.5, so I make a change to fix this.

Comments:
Thanks, it could be nice to keep Python 3.5 compatibility indeed (see #110) but I think this will break (at least) the other examples (`run_squad` and `run_classifier`) which uses the Pathlib syntax `PATH / 'string'`.

I'm sorry for my previous stupid workaround, but now I modify some functions in ``file_utils.py``, just convert type for local variables. I think it won't affect the function behaviour.
However, I only test the ``extract_features.py`` on python3.5. So I'm not sure it prefectly solve the problems, but it should be unharmful.

I think this will break [this line](https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/examples/run_classifier.py#L484) at least.

I we want to merge this PR you should check that the three examples and the tests are running (at least).

Did you see my second commit (485adde742)? I think I have fixed the problem you mentioned.
Now I have tested all unittests and 3 examples, and they all work right on python 3.5.2.

Indeed, I missed this commit.
Ok, this solution makes sense, let's go for it!
Thanks!
Pretrained Tokenizer Loading Fails: 'PosixPath' object has no attribute 'rfind'
Author: decodyng
Pull request: no

I was trying to work through the toy tokenization example from the main README, and I hit an error on the step of loading in a pre-trained BERT tokenizer. 

```
~/bert_transfer$ python3 test_tokenizer.py                                                 
Traceback (most recent call last):
  File "test_tokenizer.py", line 10, in <module>
    tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
  File "/usr/local/lib/python3.5/dist-packages/pytorch_pretrained_bert/tokenization.py", line 117, in from_pretrained
    resolved_vocab_file = cached_path(vocab_file, cache_dir=cache_dir)
  File "/usr/local/lib/python3.5/dist-packages/pytorch_pretrained_bert/file_utils.py", line 88, in cached_path
    return get_from_cache(url_or_filename, cache_dir)
  File "/usr/local/lib/python3.5/dist-packages/pytorch_pretrained_bert/file_utils.py", line 169, in get_from_cache
    os.makedirs(cache_dir, exist_ok=True)
  File "/usr/lib/python3.5/os.py", line 226, in makedirs
    head, tail = path.split(name)
  File "/usr/lib/python3.5/posixpath.py", line 103, in split
    i = p.rfind(sep) + 1
AttributeError: 'PosixPath' object has no attribute 'rfind'

~/bert_transfer$ python3 --version
Python 3.5.2
```

Exact usage in script: 
```
from pytorch_pretrained_bert import BertTokenizer

test_sentence = "When PyTorch first launched in early 2017, it quickly became a popular choice among AI researchers, who found it ideal for rapid experimentation due to its flexible, dynamic programming environment and user-friendly interface"

if __name__ == "__main__":
    tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
```

 I am curious if you're able to replicate this error on python 3.5.2, since the repo states support for 3.5+.

Comments:
Oh you are right, the file caching utilities requires python 3.6.

I don't intend to maintain a lot of backward compatibility in terms of Python versions (I already surrendered maintaining a Python 2 version) so I will bump up the requirements to python 3.6.

If you are limited to python 3.5 and find a way around this, don't hesitate to share your solution with a PR though.

Ok @hzhwcmhf fixed this issue with #113 and we will be compatible with Python 3.5+ again in the coming release (today probably). Thanks @hzhwcmhf!
How to use pre-trained SQUAD model? 
Author: danyaljj
Pull request: no

After training squad, I have a model file in a local folder: 

```
-rw-rw-r--  1 khashab2 cs_danr 4.7M Nov 21 19:20 dev-v1.1.json
-rw-rw-r--  1 khashab2 cs_danr 3.4K Nov 29 22:52 evaluate-v1.1.py
drwxrwsr-x  2 khashab2 cs_danr   10 Nov 30 14:57 out2
-rw-rw-r--  1 khashab2 cs_danr  29M Nov 21 19:20 train-v1.1.json
-rw-rw-r--  1 khashab2 cs_danr 490M Nov 29 23:14 train-v1.1.json_bert-base-uncased_384_128_64
-rw-rw-r--  1 khashab2 cs_danr 490M Nov 30 15:05 train-v1.1.json_bert-large-uncased_384_128_64
```

I want to use this pre-trained model to make predictions. Is there any example that I can follow this? (if not any pointers?) I looked into the instructions and didn't find anything relevant on this. 

Comments:
Hi there are now examples on how you can save and reload the models in the examples (`run_classifier`, `run_squad` and `run_swag`)
Problem about convert TF model and pretraining
Author: zhezhaoa
Pull request: no

First of all, Thank you for this great job. I use the official tensorflow implementation to pretrain  on my corpus and then save the model. I want to convert this model to pytorch format and use it, but I got the error: 
Traceback (most recent call last):
  File "convert_tf_checkpoint_to_pytorch.py", line 105, in <module>
    convert()
  File "convert_tf_checkpoint_to_pytorch.py", line 86, in convert
    pointer = getattr(pointer, l[0])
AttributeError: 'Parameter' object has no attribute 'adam_m'

Could you give me some advice? Thank you very much.

It is great if you can release the pretrain code. I think it is useful even we cannot use TPU. Because we can fine-tune  above google's pertained model.

Comments:
Hi @zhezhaoa, I see, I will fix this in the next release.

For now you should be able to fix that by installing the repo from source (git clone the repo and `pip install -e .` and changing [line 53 of convert_tf_checkpoint_to_pytorch.py](https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/pytorch_pretrained_bert/convert_tf_checkpoint_to_pytorch.py#L53) from
```if name[-1] in ["adam_v", "adam_m"]:```
to
```if any(n in ["adam_v", "adam_m"] for n in name):```

Thank you very much! It could be great if you can provide pertaining code like the official TF  implementation.

Ok this loading issue is now fixed in master and the new 0.4.0 release.
How to run a saved model?
Author: wahlforss
Pull request: no

How can you run the model without training the model? If we already trained a model with run_classifer?

Comments:
It looks like @thomwolf is planning to illustrate this in the examples soon.
You find some useful code to do what you want to do in https://github.com/huggingface/pytorch-pretrained-BERT/pull/112/

Hi this is now included in the new release 0.4.0 and there are examples on how you can save and reload the models in the updated run_classifier, run_squad and run_swag.
High accuracy for CoLA task
Author: pfecht
Pull request: no

I try to reproduce the CoLA results from the BERT paper (BERTBase, Single GPU).

Running the following command

```
python run_classifier.py \
  --task_name cola \
  --do_train \
  --do_eval \
  --do_lower_case \
  --data_dir $GLUE_DIR/CoLA/ \
  --bert_model bert-base-uncased \
  --max_seq_length 128 \
  --train_batch_size 32 \
  --learning_rate 2e-5 \
  --num_train_epochs 3.0 \
  --output_dir $OUT_DIR/cola_output/
```

I get eval results of 

```
12/16/2018 12:31:34 - INFO - __main__ -   ***** Eval results *****
12/16/2018 12:31:34 - INFO - __main__ -     eval_accuracy = 0.8302972195589645
12/16/2018 12:31:34 - INFO - __main__ -     eval_loss = 0.5117322660925734
12/16/2018 12:31:34 - INFO - __main__ -     global_step = 804
12/16/2018 12:31:34 - INFO - __main__ -     loss = 0.17348005173644468
```

An accuracy of 0.83 would be fantastic, but compared to the 0.521 stated in the paper this doesn't seem very realistic.

Any suggestions what I'm doing wrong?


Comments:
The metric used for evaluation of CoLA in the GLUE benchmark is not accuracy but the https://en.wikipedia.org/wiki/Matthews_correlation_coefficient (see https://gluebenchmark.com/tasks).
Indeed authors report in https://arxiv.org/abs/1810.04805 0.521 for Matthews correlation with BERT-base.

Makes sense, looks like I missed that point. Thank you.
Fourth release
Author: thomwolf
Pull request: yes

New:
- 3-4 times speed-up in fp16 thanks to NVIDIA's work on apex
- SWAG (multiple-choice) model added + example fine-tuning on SWAG
- bump up to PyTorch 1.0
- backward compatibility to python 3.5
- load fine-tuned model with `from_pretrained`
- add examples on how to save and load fine-tuned models

Comments:
raises value error for bert tokenizer for long sequences
Author: patrick-s-h-lewis
Pull request: yes

addesses #125 

(all pre-trained bert models have a positional embedding matrix with 512 embeddings. Sequences longer than 512 tokens will cause indexing errors when you attempt to run a bert forward pass on them)

added a max_len arg to bert tokenizer. the function convert_tokens_to_indices will raise a value error if the inputted list of tokens is longer than max_len.

if no max_len is supplied, then no value error will be raised, however long the sequence is

Pre-trained bert models have max_len set to 512 at object construction time (in BertTokenizer.from_pretrained. It can be overridden by explicitly passing max_len to BertTokenizer.from_pretrained as a kwarg

if bert models with larger positional embedding matrices are released, it is possible to have different max_lens for different pretrained_models.

Comments:
Thanks @patrick-s-h-lewis, this is nice.

The max number of positional embeddings is also available in the pretrained models configuration files (as `max_position_embeddings`) but accessing this requires some change in the models stored on S3 (not storing them as tar.gz files) so I will take care of it in the next release.
BERT + CNN classifier doesn't work after migrating from 0.1.2 to 0.4.0
Author: jwang-lp
Pull request: no

I used BERT in a very simple sentence classification task:
in `__init__` I have
```python3
self.bert = BertModel(config)
self.cnn_classifier = CNNClassifier(self.config.hidden_size, intent_cls_num)
```
and in forward it's just
```python3
encoded_layers, _ = self.bert(input_ids, token_type_ids, attention_mask, output_all_encoded_layers=False)
confidence_score = self.cnn_classifier(encoded_layers)
masked_lm_loss = loss_fct(confidence_score, ground_truth_labels)
```
This code works perfectly when I use 0.1.2 version, but in 0.4.0, it:
- always predicting the most common class when have a large training set
- cannot even learn a dataset with only 4 samples (fed in as one batch); can learn a single sample though

Why are these problems happening in 0.4.0? The only change in my code is that I changed `weight_decay_rate` to `weight_decay`...

Comments:
I don't know...
If you can open-source a self contained example with data and code I can try to give it a deeper look.
Are you using `apex`? That's the main change in 0.4.0.

Hi Thomas! I've found the problem. I think It's because you modified your `from_pretrained` function and I'm still using a part of the `from_pretrained` function from version 0.1.2, which resulted in some compatibility issues. Thanks!
Is there a plan to have a FP16 for GPU so to have larger batch size or longer text documents support ?
Author: howardhsu
Pull request: no

Is there a plan to have an FP16 for GPU so to have a larger batch size or longer text documents support?

Comments:
Yes probably. I am testing fp16 right now. If it works well I will push it to the repo.

Ok I've added FP16 support (see updated readme)

Thanks for this quick updates.

I'm not able to work with FP16 for pytorch BERT code. Particularly for BertForSequenceClassification, which I tried and got the issue
**Runtime error: Expected scalar type object Half but got scalar type Float for argument #2 target**
when I enabled fp16.
Also when using 
`logits = logits.half()
labels = labels.half()`
then the epoch time also increased.
It's possible to avoid download the pretrained model?
Author: rxy1212
Pull request: no

When I  run this code `model = BertModel.from_pretrained('bert-base-uncased')` , it would download a big file and sometimes that's very slow. Now I have download the model from [https://github.com/google-research/bert](url). So, It's possible to avoid download the pretrained model when I use pytorch-pretrained-BERT at the first time?

Comments:
I just find the way.

@rxy1212  could you explain the method used 

@makkunda 
In `modeling.py`, you can find this codes
```
PRETRAINED_MODEL_ARCHIVE_MAP = {
    'bert-base-uncased': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased.tar.gz",
    'bert-large-uncased': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-uncased.tar.gz",
    'bert-base-cased': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-cased.tar.gz",
    'bert-large-cased': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-cased.tar.gz",
    'bert-base-multilingual-uncased': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-multilingual-uncased.tar.gz",
    'bert-base-multilingual-cased': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-multilingual-cased.tar.gz",
    'bert-base-chinese': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-chinese.tar.gz",
}
```

just download the model you need by the url and unzip it, then you will get `bert_config.json` and `pytorch_model.bin`. You can put them in a folder X.
Now, you can use `model = BertModel.from_pretrained('THE-PATH-OF-X')`
Does the final hidden state contains the <CLS> for Squad2.0
Author: SparkJiao
Pull request: no

Recently I'm modifying the `run_squad.py` to run on CoQA. In the implementation of TensorFlow from Google, they use the probability on the first token of a context segment, where is the location of `<CLS>` to as the that of the question is unanswerable. So I try to modified the `run_squad.py` in your implementation as this. But when I looked at the predictions, I have found that many questions answers are the first word of the context not the first token, <CLS>, so I wanna know if your implementation have removed the hidden state of start token and end token? Or there may be other problems ? Thank you a lot!

Comments:
I'm sorry that I have found a bug in my code. I have invalidly called a attribute of the `InputFeature` but it have run successfully. Now I have fixed it and re-run it. If I have more questions I will reopen this. Sorry to bother you!
Embeddings from BERT for original tokens
Author: nihalnayak
Pull request: no

I am trying out the `extract_features.py` example program. I noticed that a sentence gets split into tokens and the embeddings are generated. For example, if you had the sentence “Definitely not”, and the corresponding workpieces can be [“Def”, “##in”, “##ite”, “##ly”, “not”].  It then generates the embeddings for these tokens.

My question is how do I train an NER system on CoNLL dataset? 

I want to extract embeddings for original tokens for training an NER with a neural architecture. If you have come across any resource that gives a clear explanation on how to carry this out, post it here. 

Comments:
Hi, you should read the discussion in #64. I left this issue open for reference on these questions.
Don't hesitate to participate there.
BertForQuestionAnswering: Predicting span on the question?
Author: valsworthen
Pull request: no

Hello,

I have a question regarding the `BertForQuestionAnswering` implementation. If I am not mistaken, for this model the sequence should be of the form `Question tokens [SEP] Passage tokens`. Therefore, the embedded representation computed by `BertModel` returns the states of both the question and the passage (a tensor of length `passage + question + 1`).

If I am not mistaken, the span logits are then calculated for the whole sequence, i.e. **they can be calculated for the question** even if the answer is always in the passage (see [the model code](https://github.com/huggingface/pytorch-pretrained-BERT/blob/8da280ebbeca5ebd7561fd05af78c65df9161f92/pytorch_pretrained_bert/modeling.py#L1097) and the [squad script](https://github.com/huggingface/pytorch-pretrained-BERT/blob/8da280ebbeca5ebd7561fd05af78c65df9161f92/examples/run_squad.py#L899)). I wonder if this behavior is really desirable. Doesn't it confuse the model?

Thank you for your work!

Comments:
This is the original behavior from the TF implementation.
The predictions are filtered afterward (in `write_predictions`) so this is probably not a big issue.
Maybe try with another behavior and see if it improve upon the results?
Not able to use FP16 in pytorch-pretrained-BERT
Author: Ashish-Gupta03
Pull request: no

I'm not able to work with FP16 for pytorch BERT code. Particularly for BertForSequenceClassification, which I tried and got the issue
**Runtime error: Expected scalar type object Half but got scalar type Float for argument #2 target**
when I enabled fp16.
Also when using 
`logits = logits.half()
labels = labels.half()`
then the epoch time also increased.

_Originally posted by @Ashish-Gupta03 in https://github.com/huggingface/pytorch-pretrained-BERT/issue_comments#issuecomment-449096213_

Comments:
change in run_classifier.py
Author: PatriciaRodrigues1994
Pull request: yes

while running the dev set for multi-label classification (more than two), it gives an assertion error. Specifying the num_labels while creating the model again for eval testing solves this problem. Thus the only change is while running classification for multi label is in the starting dict of num labels and specifying the base classes in DataProcessor class that is chosen.

Comments:
Thanks! #141 already addressed this problem.
Allow do_eval to be used without do_train and to use the pretrained model in the output folder
Author: jaderabbit
Pull request: yes

If you wanted to use the pre-trained model to redo evaluation without training, it errors because the output directory already exists (the output directory that contains the pre-trained model that one might like to evaluate). 

Additionally, a couple of fields are not initialised if one does not train and only evaluates.


 

Comments:
Indeed!
LayerNorm initialization
Author: donglixp
Pull request: yes

The LayerNorm gamma and beta should be initialized by .fill_(1.0) and .zero_().

reference links:

https://github.com/tensorflow/tensorflow/blob/989e78c412a7e0f5361d4d7dfdfb230c8136e749/tensorflow/contrib/layers/python/layers/layers.py#L2298

https://github.com/tensorflow/tensorflow/blob/989e78c412a7e0f5361d4d7dfdfb230c8136e749/tensorflow/contrib/layers/python/layers/layers.py#L2308

Comments:
Yes, the weights are overwritten by the loading model. But if the code is used to pretrain a model from scratch, it might affect performance. (related issue: https://github.com/huggingface/pytorch-pretrained-BERT/issues/143 )

Great, thanks @donglixp !
_load_from_state_dict() takes 7 positional arguments but 8 were given
Author: guanlongtianzi
Pull request: no


Comments:
Full log of the error?

This is caused by pytorch version.
I found , In 0.4.0 version, _load_from_state_dict() only take 7 arguments, but In 0.4.1 and this code, we need feed 8 arguments.

```
module._load_from_state_dict(
                state_dict, prefix, local_metadata, True, missing_keys, unexpected_keys, error_msgs)
```

local_metadata should be removed in pytorch 0.4.0


Ok thanks @SummmerSnow !
Warning/Assert when embedding sequences longer than positional embedding size
Author: patrick-s-h-lewis
Pull request: no

Hi team, love the work.

Just a feature suggestion: when running on GPU (presumably the CPU too), BERT will break when you try to run on sentences longer than 512 tokens (on bert-base).

This is because the position embedding matrix size is only 512 (or whatever else it is for the other bert models)

Could the tokenizer have an assert/warning on it that doesn't allow you tokenize a sentence longer than the number of positional embeddings, so that you get a better error message than a bit scary (uncatchable) cuda error.




Comments:
Could do that indeed Patrick.
In particular when the tokenizer is loaded from one of Google pre-trained model.
If you have a working implementation feel free to do a PR.
Otherwise I will have a look at that when I start working on the next release.

Happy to do a PR :) will do today or tomorrow
lower accuracy on OMD(Obama-McCain Debate twitter sentiment dataset)
Author: AIRobotZhang
Pull request: no

I run the classification task with BERT pretrianed model, but while it's much lower than other methods on OMD dataset, which has 2 labels. The final accuracy result is only 62% on binary classification task!

Comments:
We need more informations on the parameters you use to run this training in order to understand what might be wrong.

> We need more informations on the parameters you use to run this training in order to understand what might be wrong.

THANK YOU! Because of limited mermory, the batch_size is 8 and epoch is 6, and the content is short, so set the max_length is 50, other parameters are default.

Try various values for the hyper-parameters and at least 10 different seed values.
Limited memory should not be a limitation when you use `gradient accumulation` as indicated in the  readme [here](https://github.com/huggingface/pytorch-pretrained-BERT#training-large-models-introduction-tools-and-examples) (see also how it is used in all the examples like `run_classifier`, `run_squad`...)
Not able to use FP16 in pytorch-pretrained-BERT. Getting error **Runtime error: Expected scalar type object Half but got scalar type Float for argument #2 target**
Author: Ashish-Gupta03
Pull request: no

I'm not able to work with FP16 for pytorch BERT code. Particularly for BertForSequenceClassification, which I tried and got the issue
**Runtime error: Expected scalar type object Half but got scalar type Float for argument #2 target**
when I enabled fp16.
Also when using 
`logits = logits.half()
labels = labels.half()`
then the epoch time also increased.

The training time without fp16 was 2.5 hrs per epoch after doing logits.half() and labels.half() the runtime per epoch shot up to 8hrs.

Comments:
Which kind of GPU are you using? `fp16` only works on recent GPU (better with Tesla and Volta series).

I experienced a similar issue with CUDA 9.1. Using 9.2 solved this for me. 

Yes, CUDA 10 is recommended for using fp16 with good performances.