noamaanMulla-03/transformers-issues huggingface dataset:
MRPC hyperparameters question
Author: ethanjperez
Pull request: no

When describing how you reproduced the MRPC results, you say:
"Our test ran on a few seeds with the original implementation hyper-parameters gave evaluation results between 82 and 87."
and you link to the SQuAD hyperparameters (https://github.com/google-research/bert#squad).

Is the link a mistake? Or did you use the SQuAD hyperparameters for tuning on MRPC? More generally, I'm wondering if there's a reason the MRPC dev set accuracy is slightly lower (in [82, 87] vs. [84, 88] reported by Google)

Comments:
Hi Ethan,
Thanks we used the MRPC hyper-parameters indeed, I corrected the README.
Regarding the dev set accuracy, I am not really surprised there is a slightly lower accuracy with the PyTorch version (even though the variance is high so it's hard to get something significant). That is something that is generally observed (see for example [the work of Remi Cadene](https://github.com/Cadene/pretrained-models.pytorch)) and we also experienced that with [our TF->PT port of the OpenAI GPT model](https://github.com/huggingface/pytorch-openai-transformer-lm).
My personal feeling is that there are slight differences in the way the backends of TensorFlow and PyTorch handle the operations and these differences make the pre-trained weights sub-optimal for PyTorch. 

Great, thanks for clarifying that. Regarding the slightly lower accuracy, that makes sense. Thanks for your help and for releasing this!

Maybe it would help to train the Tensorflow pre-trained weights for e.g. one epoch in PyTorch (using the MLM and next-sentence objective)? That may help transfer to other tasks, depending on what the issue is

Hi @ethanjperez, actually the weight initialization fix (`tf. truncated_normal_initializer(stddev=0.02)` was translated in `weight.data.normal_(0.02)` instead of `weight.data.normal_(mean=0.0, std=0.02)` fixed in 2a97fe22) has brought us back to the TensorFlow results on MRPC (between 84 and 88%).
I am closing this issue.

@thomwolf Great to hear - thanks for working to fix it!
activation function in BERTIntermediate
Author: lukovnikov
Pull request: no

BERTConfig is not used for `BERTIntermediate`'s activation function. `intermediate_act_fn` is always `gelu`. Is this normal?

https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/modeling.py#L240

Comments:
Yes, I hard coded that since the pre-trained models are all trained with gelu anyway.

ok. but since config is there anyway, isn't it cleaner to use it (to avoid errors for people using configs that use a different activation for some reason) ?

Yes we can, I'll change that in the coming first release (unless you would like to submit a PR which I would be happy to merge).

yeah let me clean up and I'll PR
Excluding AdamWeightDecayOptimizer internal variables from restoring
Author: donatasrep
Pull request: yes

I tried to use  convert_tf_checkpoint_to_pytorch.py script to convert my pretrained model, but in order to do so, I had to make some minor tweaks. I thought I would share in case you find it useful. 

Comments:
Is your pre-trained model a TensorFlow model?

Yes

Nice, thanks for that!
[Feature request] Port SQuAD 2.0 support
Author: elyase
Pull request: no

Recently the Google team added support for Squad 2.0:

https://github.com/google-research/bert/commit/60454702590a6c69bd45c5d4258c7e17b8a3e1da

Would be great to also have it available in the Pytorch version.

Comments:
Hi, I don't have plan for that in the near future but feel free to open a PR.
adding `no_cuda` flag
Author: rahular
Pull request: yes

The `--no_cuda` flag is missing from the flagset in `extract_features.py`. On running the current code, the following error occurs.

```
(py3.5) [rahul pytorch-pretrained-BERT]$ python extract_features.py \
>   --input_file=./input.txt \
>   --output_file=./output.jsonl \
>   --vocab_file=$BERT_BASE_DIR/vocab.txt \
>   --bert_config_file=$BERT_BASE_DIR/bert_config.json \
>   --init_checkpoint=$BERT_BASE_DIR/pytorch_model.bin \
>   --layers=-4 \
>   --max_seq_length=128 \
>   --batch_size=8
Traceback (most recent call last):
  File "extract_features.py", line 306, in <module>
    main()
  File "extract_features.py", line 223, in main
    device = torch.device("cuda" if torch.cuda.is_available() and not args.no_cuda else "cpu")
AttributeError: 'Namespace' object has no attribute 'no_cuda'
```

Comments:
Thanks, I've added that manually (the library organization has changed a bit with the first pip release).
include the output layer in the model using the pretrained weights
Author: fabiopetroni
Pull request: yes

This is to be able to load the final output layer (bert.output_layer) from the TensorFlow pre-trained model.
In particular, it is a fully connected layer that is used to map the final hidden layer to the vocabulary size, to then apply the softmax, as follows:

logits = bert.output_layer(sequence_output)
log_softmax = nn.LogSoftmax(dim=-1)
log_probs = log_softmax(logits)


Comments:
Thanks for that. I've ended up taking a more modular approach in the first pip release of the library.
ValueError while using --optimize_on_cpu
Author: rsanjaykamath
Pull request: no

> Traceback (most recent call last): | 1/87970 [00:00<8:35:35, 2.84it/s]
File "./run_squad.py", line 990, in <module>
main()
File "./run_squad.py", line 922, in main
is_nan = set_optimizer_params_grad(param_optimizer, model.named_parameters(), test_nan=True)
File "./run_squad.py", line 691, in set_optimizer_params_grad
if test_nan and torch.isnan(param_model.grad).sum() > 0:
File "/people/sanjay/anaconda2/envs/bert_pytorch/lib/python3.5/site-packages/torch/functional.py", line 289, in isnan
raise ValueError("The argument is not a tensor", str(tensor))
ValueError: ('The argument is not a tensor', 'None')

Command: 
CUDA_VISIBLE_DEVICES=0 python ./run_squad.py \
--vocab_file bert_large/uncased_L-24_H-1024_A-16/vocab.txt \
--bert_config_file bert_large/uncased_L-24_H-1024_A-16/bert_config.json \
--init_checkpoint bert_large/uncased_L-24_H-1024_A-16/pytorch_model.bin \
--do_lower_case \
--do_train \
--do_predict \
--train_file squad_dir/train-v1.1.json \
--predict_file squad_dir/dev-v1.1.json \
--learning_rate 3e-5 \
--num_train_epochs 2 \
--max_seq_length 384 \
--doc_stride 128 \
--output_dir outputs \
--train_batch_size 4 \
--gradient_accumulation_steps 2 \
--optimize_on_cpu 

Error while using --optimize_on_cpu only. 
Works fine without the argument. 

GPU: Nvidia GTX 1080Ti Single GPU.

PS: I can only fit in train_batch_size 4 on the memory of a single GPU.

Comments:
Thanks! I pushed a fix for that, you can try it again. You should be able to increase a bit the batch size.

By the way, the real batch size that is used on the gpu is `train_batch_size / gradient_accumulation_steps` so `2` in your case. I think you should be able to go to `3` with `--optimize_on_cpu`

The recommended batch_size to get good results (EM, F1) with BERT large on SQuaD is `24`. You can try the following possibilities to get to this batch_size:
- keeping the same 'real batch size' that you currently have but just a bigger batch_size `--train_batch_size 24 --gradient_accumulation_steps 12`
- trying a 'real batch size' of 3 with optimization on cpu `--train_batch_size 24 --gradient_accumulation_steps 8 --optimize_on_cpu`
- switching to fp16 (implies optimization on cpu): `--train_batch_size 24 --gradient_accumulation_steps 6 or 4 --fp16`

If your GPU supports fp16, the last solution should be the fastest, otherwise the second should be the fastest. The first solution should work out-of-the box and give better results (EM, F1) but you won't have any speed-up.

Should be fixed now. Don't hesitate to re-open an issue if needed. Thanks for the feedback!

Yes it works now! 

With 

> --train_batch_size 24 --gradient_accumulation_steps 8 --optimize_on_cpu

I get {"exact_match": 83.78429517502366, "f1": 90.75733469379139} which is pretty close.

Thanks for this amazing work!
issues with accents on convert_ids_to_tokens()
Author: perezjln
Pull request: no

Hello, the BertTokenizer seems loose accents when convert_ids_to_tokens() is used :

Example:
- original sentence: "great breakfasts in a nice furnished cafè, slightly bohemian."
- corresponding list of token produced : ['great', 'breakfast', '##s', 'in', 'a', 'nice', 'fur', '##nis', '##hed', 'cafe', ',', 'slightly', 'bohemia', '##n', '.']

Here the problem is in "cafe" that loses its accent. I'm using BertTokenizer.from_pretrained('Bert-base-multilingual') as the tokenizer, I also tried with "Bert-base-uncased" and experienced the same issue.

Thanks for this great work!

Comments:
This is expected behaviour and is how the multilingual and the uncased models were trained. From the [original repo](https://github.com/google-research/bert/blob/master/README.md):

> We are releasing the BERT-Base and BERT-Large models from the paper. Uncased means that the text has been lowercased before WordPiece tokenization, e.g., John Smith becomes john smith. The Uncased model also strips out any accent markers. 



Yes this is expected.
Typo in README
Author: weiyumou
Pull request: no

I think I spotted a typo in the README file under the Usage header. There is a piece of code that uses `BertTokenizer` and the typo is on this line:
`tokenized_text = "Who was Jim Henson ? Jim Henson was a puppeteer"`

I think `tokenized_text` should be replaced with `text`, since the next line is
`tokenized_text = tokenizer.tokenize(text)`

Comments:
Yes
Command-line interface Document Bug
Author: delldu
Pull request: no

There is a bug in README.md about Command-line interface:
`export BERT_BASE_DIR=chinese_L-12_H-768_A-12`

**Wrong:**
```
pytorch_pretrained_bert convert_tf_checkpoint_to_pytorch \
   --tf_checkpoint_path $BERT_BASE_DIR/bert_model.ckpt.index \
   --bert_config_file $BERT_BASE_DIR/bert_config.json \
   --pytorch_dump_path $BERT_BASE_DIR/pytorch_model.bin
```

**Right:**
```
pytorch_pretrained_bert convert_tf_checkpoint_to_pytorch \
  $BERT_BASE_DIR/bert_model.ckpt.index \
  $BERT_BASE_DIR/bert_config.json \
  $BERT_BASE_DIR/pytorch_model.bin
```


Comments:
Thanks!
[Bug report] Ineffective no_decay when using BERTAdam
Author: xiaoda99
Pull request: no

https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/examples/run_classifier.py#L505-L508

With this code, all parameters are decayed because the condition "parameter_name in no_decay" will never be satisfied.

I've made a PR #32 to fix it.

Comments:
You're right, thanks!
Fixed UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2
Author: weiyumou
Pull request: yes

I encountered `UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 3793: ordinal not in range(128)`  when running the starter example shown under the Usage section. It turned out to be related to the `load_vocab` function in `tokenization.py`. Forcing `open` to use encoding `utf8` solved this issue on my machine. 

Comments:
Thanks!
Issue of `bert_model` arg in `run_classify.py`
Author: llidev
Pull request: no

Hi,

I am trying to understand the `bert_model` arg in `run_classify.py`. In the file, I can see 
```
tokenizer = BertTokenizer.from_pretrained(args.bert_model)
```
where `bert_model` is expected to be the vocab text file of the model

However, I also see

```
model = BertForSequenceClassification.from_pretrained(args.bert_model, len(label_list))
```
where `bert_model` is expected to be a archive file containing the model checkpoint and config.

Please help to advice the correct use of `bert_model` if I have my pretrained model converted locally already.

Thanks!

Comments:
Hi, please read [this section](https://github.com/huggingface/pytorch-pretrained-BERT#loading-google-ais-pre-trained-weigths-and-pytorch-dump) of the readme.
grad is None in squad example
Author: vpegasus
Pull request: no

Hi, guys, I try the `run_squad` example with
```
Traceback (most recent call last):                                                                                                                                                                 | 0/7331 [00:00<?, ?it/s]
  File "examples/run_squad.py", line 973, in <module>
    main()
  File "examples/run_squad.py", line 904, in main
    param.grad.data = param.grad.data / args.loss_scale
AttributeError: 'NoneType' object has no attribute 'data'

```  
I find one of the param.grads is None, so the param.grad.data doesn't exist. 
by the way I down load the data by myself from the urls in this prject. my os is ubuntu 18.04, pytorch 0.41 gpu 1080t

anyone else encounters this situation?  
wanna help, please, thx in advance...

Comments:
Oh you're right. I've just fixed that. you can try to pull the current master and test again.

@thomwolf  it works, thanks
Multilingual Issue
Author: hahmyg
Pull request: no

Dear authors,
I have two questions.

First, how can I use multilingual pre-trained BERT in pytorch?
Is it all download model to $BERT_BASE_DIR?

Second is tokenization issue.
For Chinese and Japanese, tokenizer may works, however, for Korean, it shows different result that I expected

```
import torch
from pytorch_pretrained_bert import BertTokenizer, BertModel, BertForMaskedLM
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
text = "안녕하세요"
tokenized_text = tokenizer.tokenize(text)
print(tokenized_text)
```
` ['ᄋ', '##ᅡ', '##ᆫ', '##ᄂ', '##ᅧ', '##ᆼ', '##ᄒ', '##ᅡ', '##ᄉ', '##ᅦ', '##ᄋ', '##ᅭ']

The result is based on not 'character' but 'byte-based character'
May it comes from unicode issue. (I expect ['안녕', '##하세요'])


Comments:
Hi, you can use the multilingual model as [indicated in the readme](https://github.com/huggingface/pytorch-pretrained-BERT#loading-google-ais-pre-trained-weigths-and-pytorch-dump) with the commands:
```python
tokenizer = BertTokenizer.from_pretrained('bert-base-multilingual')
model = BertModel.from_pretrained('bert-base-multilingual')
````
This will load the multilingual vocabulary (which should contain korean) that your command was not loading.
UnicodeDecodeError: 'charmap' codec can't decode byte 0x90 in position 3920: character maps to <undefined>
Author: superchthonic
Pull request: no

Installed pytorch-pretrained-BERT from source, Python 3.7, Windows 10

When I run the following snippet:

import torch
from pytorch_pretrained_bert import BertTokenizer, BertModel, BertForMaskedLM

# Load pre-trained model tokenizer (vocabulary)
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

I get the following: 

---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
<ipython-input-2-7725148c607d> in <module>()
      3 
      4 # Load pre-trained model tokenizer (vocabulary)
----> 5 tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

~\Anaconda3\lib\site-packages\pytorch_pretrained_bert\tokenization.py in from_pretrained(cls, pretrained_model_name, do_lower_case)
    139                     vocab_file, resolved_vocab_file))
    140             # Instantiate tokenizer.
--> 141             tokenizer = cls(resolved_vocab_file, do_lower_case)
    142         except FileNotFoundError:
    143             logger.error(

~\Anaconda3\lib\site-packages\pytorch_pretrained_bert\tokenization.py in __init__(self, vocab_file, do_lower_case)
     93                 "Can't find a vocabulary file at path '{}'. To load the vocabulary from a Google pretrained "
     94                 "model use `tokenizer = BertTokenizer.from_pretrained(PRETRAINED_MODEL_NAME)`".format(vocab_file))
---> 95         self.vocab = load_vocab(vocab_file)
     96         self.ids_to_tokens = collections.OrderedDict(
     97             [(ids, tok) for tok, ids in self.vocab.items()])

~\Anaconda3\lib\site-packages\pytorch_pretrained_bert\tokenization.py in load_vocab(vocab_file)
     68     with open(vocab_file, "r", encoding="utf8") as reader:
     69         while True:
---> 70             token = convert_to_unicode(reader.readline())
     71             if not token:
     72                 break

~\Anaconda3\lib\encodings\cp1252.py in decode(self, input, final)
     21 class IncrementalDecoder(codecs.IncrementalDecoder):
     22     def decode(self, input, final=False):
---> 23         return codecs.charmap_decode(input,self.errors,decoding_table)[0]
     24 
     25 class StreamWriter(Codec,codecs.StreamWriter):

UnicodeDecodeError: 'charmap' codec can't decode byte 0x90 in position 3920: character maps to <undefined>


Comments:
I am facing the same problem.

Fixed it with "with open(vocab_file, "r"**, encoding="utf-8"**) as reader:" in line 68 of tokenization.py

Thanks, it's fixed on master and will be included in the next release.
Loss calculation error
Author: jwang-lp
Pull request: no

https://github.com/huggingface/pytorch-pretrained-BERT/blob/982339d82984466fde3b1466f657a03200aa2ffb/pytorch_pretrained_bert/modeling.py#L744

Got `ValueError: Expected target size (1, 30522), got torch.Size([1, 11])` at line 744 of `modeling.py`. I think the line should be changed to `masked_lm_loss = loss_fct(prediction_scores.view([-1, self.config.vocab_size]), masked_lm_labels.view([-1]))`.

Comments:
Hi Jian, can you give me a small (self-contained) example showing how to get this error?

Hi Thomas! I modified the code in your `README.md` for an example:

```python
from pytorch_pretrained_bert.modeling import BertForMaskedLM, BertConfig
from pytorch_pretrained_bert import BertTokenizer
import torch

model = BertForMaskedLM.from_pretrained('bert-base-uncased')

# Tokenized input
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
text = "Who was Jim Henson ? Jim Henson was a puppeteer"
tokenized_text = tokenizer.tokenize(text)

# Mask a token that we will try to predict back with `BertForMaskedLM`
masked_index = 6
tokenized_text[masked_index] = '[MASK]'

# Convert token to vocabulary indices
indexed_truths = tokenizer.convert_tokens_to_ids(tokenized_text)
indexed_tokens = tokenizer.convert_tokens_to_ids(tokenized_text)

# Convert inputs to PyTorch tensors
tokens_tensor = torch.tensor([indexed_tokens])
indexed_truths_tensor = torch.tensor([indexed_truths])

# Evaluate loss
model.eval()
masked_lm_logits_scores = model(tokens_tensor, masked_lm_labels=indexed_truths_tensor)
print(masked_lm_logits_scores)
```

Thank you, you are right, I fixed that on master. It will be in the next release.
example in BertForSequenceClassification() conflicts with the api 
Author: labixiaoK
Pull request: no

Hi, firstly, admire u for the great job.  but I encounter 2 problems when i use it:
**1**. `UnicodeDecodeError: 'gbk' codec can't decode byte 0x85 in position 4527: illegal multibyte sequence`,
same problem as ISSUE 52 when I excute the `BertTokenizer.from_pretrained('bert-base-uncased')`, but I successfully excute `BertForNextSentencePrediction.from_pretrained('bert-base-uncased')`, >.<
**2**. in the pytorch-pretrained-BERT/pytorch_pretrained_bert/modeling.py, 
line 761 -->     ```
   `token_type_ids`: an optional torch.LongTensor of shape [batch_size, sequence_length] **with the token
            types indices selected in [0, 1]**. Type 0 corresponds to a `sentence A` and type 1 corresponds to
            a `sentence B` token (see BERT paper for more details).
```
but in the following example,  in **line 784**-->     `token_type_ids = torch.LongTensor([[0, 0, 1], [0, **2**, 0]])`, why the '2' appears?  I am confused.  Otherwise, is the situation similar to '0, 1, 0 ' correct ? Or it should be similar to [000000111111] , that is continuous '0' and continuous '1' ?
ty.

Comments:
Hi,
(1) is solved on master. I will release a new release soon with the fixes on pip. In the mean time you can install from sources if you want.
I fixed the typo in the docstring you mention in (2), thanks, it should be a `1` instead of a `2`.
Missing options/arguments in run_squad.py for BERT Large
Author: avisil
Pull request: no

Thanks for the great code..However, the `run_squad.py` for BERT Large seems to not have the `vocab_file`  and `bert_config_file` (or other) options/arguments. Did you push the latest version?
Also, it is looking for a pytorch model file (a bin file). Does it need to be there?

I also had to add this line to the file to make BERT base to run on Squad 1.1:
`parser.add_argument('--do_lower_case', action="store_true", default=True, help="Lowercase the input")`

Comments:
Yes, the readme example was for an older version. I have updated them with the simplified parameters used in the current release. Thanks.
truncated normal initializer
Author: ruotianluo
Pull request: no

I have a reasonable truncated normal approximation. (Actually that is what tf does).
https://discuss.pytorch.org/t/implementing-truncated-normal-initializer/4778/16?u=ruotianluo


Comments:
We could try that. Not sure how important it is though. Did you try it?

Ok I think we will stick to the normal_initializer for now. Thanks for indicating this option!
Missing function convert_to_unicode in tokenization.py
Author: ptrichel
Pull request: no

The function _convert_to_unicode_ is not in tokenization.py but used to be there in v0.1.2. When fine tuning with run_classifier.py, you get an ImportError: cannot import name 'convert_to_unicode'.

https://github.com/huggingface/pytorch-pretrained-BERT/blob/ce37b8e4819142171b61558e64f7dcb0286e9937/examples/run_classifier.py#L33

Comments:
Fixed in master, thanks!
Multi-GPU training vs Distributed training
Author: llidev
Pull request: no

Hi,

I have a question about Multi-GPU vs Distributed training, probably unrelated to BERT itself.

I have a 4-GPU server, and was trying to run `run_classifier.py` in two ways:

(a) run single-node distributed training with 4 processes and minibatch of 32 each
(b) run Multi-GPU training with minibatch of 128, and all other hyperparams keep the same

Intuitively I believe a and b should yield the closed accuracy and training times. Below please find my observations:
1.  (a) runs ~20% faster than (b). 
2.  (b) yields a better final evaluation accuracy of ~4% than (a)

The first looks like reasonable since I guess the loss.mean() is done by CPU which may be slower than using NCCL directly?  However, I don't quite understand the second observation. Can you please give any hint or reference about the possible cause?

Thanks!


Comments:
Hi,

Thanks for the feedback, it's always interesting to compare the various possible ways to train the model indeed.

The most likely cause for (2) is that MRPC is a small dataset and the model shows a high variance in the results depending on the initialization of the weights for example (see the original BERT repo on that also). The distributed and multi-gpu setups probably do not use the random generators in the exact same order which lead to different initializations.

You can have an intuition of that by training with different seeds, you will see there is easily a 10% variation in the final accuracy...

If you can do that, a better way to compare the results would thus be to take something like 10 different seeds for each training condition and compare the mean and standard deviation of the results.

Thanks for your feedback!

After some investigations, it looks like `t_total` is not set properly for distributed training in BertAdam. The actual `t_total` per distributed worker should be divided by the worker count. 

I have included the following fix in my PR https://github.com/huggingface/pytorch-pretrained-BERT/pull/58

```
    t_total = num_train_steps
    if args.local_rank != -1:
        t_total = t_total // torch.distributed.get_world_size()
    optimizer = BertAdam(optimizer_grouped_parameters,
                         lr=args.learning_rate,
                         warmup=args.warmup_proportion,
                         t_total=t_total)
```