noamaanMulla-03/transformers-issues huggingface dataset:
fix squad v1 error (na_prob_file should be None)
Author: Liangtaiwan
Pull request: yes

When running squad v1, na_prob_file should be None.
Or there will be an error when evaluate on testing data.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/882?src=pr&el=h1) Report
> Merging [#882](https://codecov.io/gh/huggingface/pytorch-transformers/pull/882?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/pytorch-transformers/commit/067923d3267325f525f4e46f357360c191ba562e?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/pytorch-transformers/pull/882/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/pytorch-transformers/pull/882?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master     #882   +/-   ##
=======================================
  Coverage   79.03%   79.03%           
=======================================
  Files          34       34           
  Lines        6234     6234           
=======================================
  Hits         4927     4927           
  Misses       1307     1307
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/882?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/882?src=pr&el=footer). Last update [067923d...a7fce6d](https://codecov.io/gh/huggingface/pytorch-transformers/pull/882?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Thanks indeed, this should fix #891
BERT: run_squad.py falling over after eval
Author: jforbes14
Pull request: no

I'm having an issue fine-tuning BERT with run.squad.py, as it falls over at the end of the evaluation stage. I'm fine tuning on SQuAD v1.1. Has anyone else encountered the same issue, or is able to point out where I'm going wrong?

`python run_squad.py \
    --model_type bert \
    --model_name_or_path bert-base-uncased \
    --do_train \
    --do_eval \
    --do_lower_case \
    --overwrite_output_dir \
    --train_file $TRAIN_FILE \
    --predict_file $PREDICT_FILE \
    --learning_rate 2e-5 \
    --num_train_epochs 1.0 \
    --max_seq_length 384 \
    --per_gpu_eval_batch_size=12   \
    --per_gpu_train_batch_size=12   \
    --output_dir /content/SQuAD_for_bert/models/bert_base_uncased_finetuned_script/`

Which completes fine-tuning the model, and then fits the eval set but returns:

`Evaluating: 100% 257/257 [01:29<00:00,  2.86it/s]
07/22/2019 02:17:18 - INFO - utils_squad -   Writing predictions to: /content/SQuAD_for_bert/models/bert_base_uncased_finetuned_script/predictions_.json
07/22/2019 02:17:18 - INFO - utils_squad -   Writing nbest to: /content/SQuAD_for_bert/models/bert_base_uncased_finetuned_script/nbest_predictions_.json
Traceback (most recent call last):
  File "run_squad.py", line 521, in <module>
    main()
  File "run_squad.py", line 510, in main
    result = evaluate(args, model, tokenizer, prefix=global_step)
  File "run_squad.py", line 257, in evaluate
    results = evaluate_on_squad(evaluate_options)
  File "/content/SQuAD_for_bert/utils_squad_evaluate.py", line 291, in main
    with open(OPTS.na_prob_file) as f:
FileNotFoundError: [Errno 2] No such file or directory: '/content/SQuAD_for_bert/models/bert_base_uncased_finetuned_script/null_odds_.json'`

(Running this in Google Colab - in case that is of any relevance).

Any help would be greatly appreciated - thanks!

Comments:
Fixed in #882
Update docs for parameter rename
Author: rococode
Pull request: yes

small fix: OpenAIGPTLMHeadModel now accepts `labels` instead of `lm_labels`

 

Comments:
# [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/888?src=pr&el=h1) Report
> Merging [#888](https://codecov.io/gh/huggingface/pytorch-transformers/pull/888?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/pytorch-transformers/commit/067923d3267325f525f4e46f357360c191ba562e?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/pytorch-transformers/pull/888/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/pytorch-transformers/pull/888?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master     #888   +/-   ##
=======================================
  Coverage   79.03%   79.03%           
=======================================
  Files          34       34           
  Lines        6234     6234           
=======================================
  Hits         4927     4927           
  Misses       1307     1307
```


| [Impacted Files](https://codecov.io/gh/huggingface/pytorch-transformers/pull/888?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pytorch\_transformers/modeling\_openai.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/888/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvbW9kZWxpbmdfb3BlbmFpLnB5) | `74.76% <ø> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/888?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/888?src=pr&el=footer). Last update [067923d...66b15f7](https://codecov.io/gh/huggingface/pytorch-transformers/pull/888?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Yes, thanks @rococode!
fix a bug of saving added tokens
Author: askerlee
Pull request: yes

Refer to the code that loads `added_tokens.json`:
`added_tok_encoder = json.load(open(added_tokens_file, encoding="utf-8"))`
We can see that `added_tokens_encoder` should be saved in `added_tokens.json`. But the original code saved `added_tokens_decoder`. 


Comments:
# [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/895?src=pr&el=h1) Report
> Merging [#895](https://codecov.io/gh/huggingface/pytorch-transformers/pull/895?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/pytorch-transformers/commit/067923d3267325f525f4e46f357360c191ba562e?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `0%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/pytorch-transformers/pull/895/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/pytorch-transformers/pull/895?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master     #895   +/-   ##
=======================================
  Coverage   79.03%   79.03%           
=======================================
  Files          34       34           
  Lines        6234     6234           
=======================================
  Hits         4927     4927           
  Misses       1307     1307
```


| [Impacted Files](https://codecov.io/gh/huggingface/pytorch-transformers/pull/895?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pytorch\_transformers/tokenization\_utils.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/895/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvdG9rZW5pemF0aW9uX3V0aWxzLnB5) | `86.53% <0%> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/895?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/895?src=pr&el=footer). Last update [067923d...c9a7b29](https://codecov.io/gh/huggingface/pytorch-transformers/pull/895?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Thanks, that was fixed in #893
Fixed import to use torchscript flag.
Author: sukuya
Pull request: yes


Comments:
# [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/899?src=pr&el=h1) Report
> Merging [#899](https://codecov.io/gh/huggingface/pytorch-transformers/pull/899?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/pytorch-transformers/commit/067923d3267325f525f4e46f357360c191ba562e?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/pytorch-transformers/pull/899/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/pytorch-transformers/pull/899?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master     #899   +/-   ##
=======================================
  Coverage   79.03%   79.03%           
=======================================
  Files          34       34           
  Lines        6234     6234           
=======================================
  Hits         4927     4927           
  Misses       1307     1307
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/899?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/899?src=pr&el=footer). Last update [067923d...e1e2ab3](https://codecov.io/gh/huggingface/pytorch-transformers/pull/899?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Increased number of hidden states returned from transformers in latest release
Author: anlsh
Pull request: no

I noticed an (undocumented?) change in the latest release: namely that transformers now include the pre-encoder input vector in the list returned when `output_hidden_states` is True.

For example, the `hidden_states` output from `BertEncoder` now returns a length-13 list of tensors, whereas it used to return a length-12 list for each of Bert's 12 encoder modules

The change does seem to be intentional, as we have the following line in the tests

https://github.com/huggingface/pytorch-transformers/blob/067923d3267325f525f4e46f357360c191ba562e/pytorch_transformers/tests/modeling_common_test.py#L244

Personally I'm not affected, I just wanted to double-check that this was intended behavior

Comments:
Yes, this is the initial embedding layer (i.e. this is layers "0" through 12 or 24). There are a number of small changes that haven't been migrated into the documentation yet.
tokenization_gpt2.py - on python 2 you can use backports.functools_lru_cache package from pypi
Author: philip-bl
Pull request: no
Labels: name: wontfix

See https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/pytorch_pretrained_bert/tokenization_gpt2.py#L28. Instead of not using `lru_cache` you can use this package.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
error: the following arguments are required: --bert_model, --output_dir
Author: CaesarLuvAI
Pull request: no

the above error arose when i ran the run_squad.py in pycharm(i just copied and ran locally). so can anbody tell how to input these two parameters "--bert_model"," --output_dir" in the IDE?

Comments:
Check this site : https://github.com/huggingface/pytorch-pretrained-BERT and find your 2 Parameters "--bert_model"," --output_dir" . You'll find the Below example 👍 

Example 👍 💯 
 --bert_model : We can use BERT Models like :  bert-base-uncased, bert-base-cased, bert-large-uncased, bert-large-cased, etc. All BERT models Refer : [https://github.com/google-research/bert#pre-trained-models](url)

--output_dir : Output Directory in Local : C:/0.GITHUB_Desktop/pytorch-pretrained-BERT [https://github.com/huggingface/pytorch-pretrained-BERT](url)

--bert_model bert-base-uncased
--output_dir /tmp/mrpc_output/

-------------------------------------------------------------------------------------------------------------------

export GLUE_DIR=/path/to/glue

python run_classifier.py \
  --task_name MRPC \
  --do_train \
  --do_eval \
  --do_lower_case \
  --data_dir $GLUE_DIR/MRPC/ \
  --bert_model bert-base-uncased \
  --max_seq_length 128 \
  --train_batch_size 32 \
  --learning_rate 2e-5 \
  --num_train_epochs 3.0 \
  --output_dir /tmp/mrpc_output/

What do you do specifically with the above? I have downloaded the BERT model, placed it in the same directory and am still getting this error.
fix multi-gpu training bug when using fp16
Author: zijunsun
Pull request: yes

multi-gpu training (orch.nn.DataParallel) should  also be after apex fp16 initialization.

Comments:
Thanks, can you update `run_squad` similarly?

> Thanks, can you update `run_squad` similarly?

updated already.

# [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/896?src=pr&el=h1) Report
> Merging [#896](https://codecov.io/gh/huggingface/pytorch-transformers/pull/896?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/pytorch-transformers/commit/067923d3267325f525f4e46f357360c191ba562e?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/pytorch-transformers/pull/896/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/pytorch-transformers/pull/896?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master     #896   +/-   ##
=======================================
  Coverage   79.03%   79.03%           
=======================================
  Files          34       34           
  Lines        6234     6234           
=======================================
  Hits         4927     4927           
  Misses       1307     1307
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/896?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/896?src=pr&el=footer). Last update [067923d...f0aeb7a](https://codecov.io/gh/huggingface/pytorch-transformers/pull/896?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Thanks a lot!
Question on duplicated sentence
Author: youngminpark2559
Pull request: no
Labels: name: wontfix

Hi. I wonder whether they are unnecessary duplicated sentence or not.  

When I run in "test mode", similar sentences are called twice.
https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/examples/run_squad.py#L908
https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/examples/run_squad.py#L913

https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/examples/run_squad.py#L1042
https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/examples/run_squad.py#L1044

Thanks.

Comments:
Looks like a minor bug. However, it seems that simply removing this `else` statement may cause some problems according to the previous code logic.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
bug: it is broken to use tokenizer path
Author: zsk423200
Pull request: no

run run_glue.py with the parameter of tokenizer_name:
`--tokenizer_name=/path/bert-base-chinese-vocab.txt`
but get the error:
```
Traceback (most recent call last):
  File "run_glue.py", line 485, in <module>
    main()
  File "run_glue.py", line 418, in main
    tokenizer = tokenizer_class.from_pretrained(args.tokenizer_name if args.tokenizer_name else args.model_name_or_path, do_lower_case=args.do_lower_case)
  File "/opt/conda/lib/python3.6/site-packages/pytorch_transformers/tokenization_bert.py", line 200, in from_pretrained
    return super(BertTokenizer, cls)._from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/pytorch_transformers/tokenization_utils.py", line 234, in _from_pretrained
    special_tokens_map = json.load(open(special_tokens_map_file, encoding="utf-8"))
  File "/opt/conda/lib/python3.6/json/__init__.py", line 299, in load
    parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
  File "/opt/conda/lib/python3.6/json/__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "/opt/conda/lib/python3.6/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/opt/conda/lib/python3.6/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 2 (char 1)
```
i debug the variable of resolved_vocab_files, has the same value:
```
{'added_tokens_file': '/home/zhoushengkai/script/NLP/pytorch-transformers/pytorch_transformers/vocab_files/bert-base-chinese-vocab.txt', 'special_tokens_map_file': '/home/zhoushengkai/script/NLP/pytorch-transformers/pytorch_transformers/vocab_files/bert-base-chinese-vocab.txt', 'vocab_file': '/home/zhoushengkai/script/NLP/pytorch-transformers/pytorch_transformers/vocab_files/bert-base-chinese-vocab.txt'}
```


Comments:
Had the same issue when passing the exact path of the vocabulary file. Fixed it by just passing the name of the directory that contains the vocabulary file (in my case it was `vocab.txt`).

Good catch.

For non-BPE models with a single vocabulary file (Bert, XLNet, Transformer-XL) we can fix this workflow so you can provide a direct path.

Updating this.
Upgrade to new FP16
Author: bhavsarpratik
Pull request: no

The original FP16_Optimizer and the old “Amp” API are deprecated and subject to removal at any time. Should we consider moving to the new one?

https://nvidia.github.io/apex/amp.html#for-users-of-the-old-fp16-optimizer

Comments:
Just saw run_glue has the new one.
How to add new special token
Author: dchang56
Pull request: no

I noticed the never_split functionality is no longer used to keep track of special tokens to never split on. If I wanted to add a new special token like '[NEW]' so the tokenizer never splits it, how should I go about doing that? (I've already manually added it to vocab.txt by replacing an unused token with [NEW]. Now I just need to not split it)

Comments:
How did you add a new vocab.txt file? 

I actually figured it out. I manually replaced one of the unused tokens in the vocab file with [NEW] and added "additiona_special_tokens": "[NEW]" to the special_tokens.json file in the same directory as the vocab.txt file. It works, but I realized that adding new tokens without the ability to do further pretraining isn't all that useful, especially given small dataset size. I decided not to do it.
BERT tokenizer - set special tokens 
Author: adigoryl
Pull request: no
Labels: name: wontfix

Hi,

I was wondering whether the team could expand BERT so that fine-tuning with newly defined special tokens would be possible - just like the GPT allows.

@thomwolf Could you share your thought with me on that? 

Regards,
Adrian.

Comments:
Hi Adrian, BERT already has a few unused tokens that can be used similarly to the `special_tokens` of GPT/GPT-2.
For more details see https://github.com/google-research/bert/issues/9#issuecomment-434796704 and issue #405 for instance.

In case we use an unused special token from the vocabulary, is it enough to finetune a classification task or do we need to train an embedding from scratch? Did anyone already do this?

Two different and somehow related questions I had when looking into the implementation:

1) The Bert paper mentions a (learned) positional embedding. How is this implemented here? examples/extract_features/convert_examples_to_features() defines tokens (representation), input_type_ids (the difference between the first and second sequence) and an input_mask (distinguishing padding/real tokens) but no positional embedding. Is this done internally?

2) Can I use a special token as input_type_ids for Bert? In the classification example, only values of [0,1] are possible and I'm wondering what would happen if I would choose a special token instead? Is this possible with a pretrained embedding or do i need to retrain the whole embedding as a consequence?


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
from_pretrained
Author: steindor
Pull request: no
Labels: name: wontfix

I have a question regarding the from_pretrained method since I experienced a bit unexpected behaviour. It's regarding how to save a BERT classifier model as a whole.

I am experimenting with a classifier on top of BERT on the stack overflow question / tags dataset which contains 20 classes of 40.000 text samples. I trained a classifier on 80% of the data for 3 epochs and saved the model in the following and recommended way:

```
def save_model(model):
    # Save a trained model, configuration and tokenizer
    model_to_save = model.module if hasattr(model, 'module') else model  # Only save the model it-self

    # If we save using the predefined names, we can load using `from_pretrained`
    output_model_file = os.path.join("./", "pytorch_model.bin")
    output_config_file = os.path.join("./", "bert_config.json")

    torch.save(model_to_save.state_dict(), output_model_file)
    model_to_save.config.to_json_file(output_config_file)
    tokenizer.save_vocabulary("./")
```

This gave my a a model .bin file and a config file.

The training accuracy was around 90% after the last epoch on 32.000 training samples, leaving 8.000 samples for evaluation. I then instantiated a new BERT model with from_pretrained method with state_dict as False and ran the evaluation which surprisingly gave these results:

{'eval_loss': 9.04939697444439, 'eval_accuracy': 0.036875}

I ran through the from_pretrained method and saw that the .bin file is a PyTorch dump of a BertForPreTraining instance which I presume means that the classifier weights are not saved when saving this way?

Then it would probably be necessary to pass the state_dict parameter when loading a classifier model with from_pretrained? If so, is it necessary to pass two files (.bin and state_dict as model_state) or is there another way to do this? 

Comments:
Did you try to load the model following the best-practices indicated here: https://github.com/huggingface/pytorch-pretrained-BERT#serialization-best-practices

All but load the tokenizer from the vocab file. Think that would make a large difference?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.