noamaanMulla-03/transformers-issues huggingface dataset:
Tokenizer added special token attributes missing
Author: Mrpatekful
Pull request: no

It might not be a bug but I think it would be useful and more consitent behaviour if tokenizers could maintain the added special tokens as attributes after saving and loading a tokenizer. See the following example.

```python
if 'added_tokens.json' in os.listdir('.'):
    # loading the saved extended tokenizer
    # and trying to reach the added special token
    # through the attribute raises an error
    tokenizer = XLNetTokenizer.from_pretrained('.')
    print(tokenizer.custom_token)
else:
    # loading a base tokenizer and extending it with special
    # token which is added to the instance attributes
    tokenizer = XLNetTokenizer.from_pretrained('xlnet-base-cased')
    tokenizer.add_special_tokens({'custom_token': '<custom>'})
    # saving the extended tokenizer
    tokenizer.save_pretrained('.')
    print(tokenizer.custom_token)
```

**1st run result:**
```text
<custom>
```

**2nd run result:**
```text
Traceback (most recent call last):
  File "src/_test.py", line 19, in <module>
    main()
  File "src/_test.py", line 9, in main
    print(tokenizer.custom_token)
AttributeError: 'XLNetTokenizer' object has no attribute 'custom_token'
```

Comments:
The framework has been updated to store all additional special tokens in `additional_special_tokens` list and custom tokens are no longer available through class attributes.
`get_final_text` bug when dealing with chinese sentence
Author: alanwang93
Pull request: no
Labels: name: wontfix

Hi,

I set `max_answer_length` to `30`, but I still got really long answers, so I print the `tok_text`, `orig_text` and `final_text` in function `write_predictions`. 

```
tok_text: 权 健 公 司 可 能 涉 及 的 刑 事 罪 名 是 否 仅 仅 是 [UNK] 虚 假 广 告 罪
orig_text: 根据相关法律，权健公司可能涉及的刑事罪名是否仅仅是“虚假广告罪”“组织、领导传销活动罪”两个罪名，应该说仍有不少需要进一步深入调查的空间
final_text: 根据相关法律，权健公司可能涉及的刑事罪名是否仅仅是“虚假广告罪”“组织、领导传销活动罪”两个罪名，应该说仍有不少需要进一步深入调查的空间
```

From `tok_text` we can see that the answer should be `权健公司可能涉及的刑事罪名是否仅仅是“虚假广告罪`, however, `final_text` we got turned out to be much longer.


Comments:
Perhaps it's the problem of tokenizer...After stripping the lengths of two sequences changed, so `orig_text` is returned...

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Can BertForMaskedLM be used to predict out-of-vocabulary words?
Author: lcswillems
Pull request: no
Labels: name: wontfix

Hi,

I have this text:

```
[CLS] This is a picture of a boa.
```

And would like to have the predictions of the `BertForMaskedLM` model for the word `boa`, without masking this word.

However, when I tokenize the text to give it to the network, I get:

```
['[CLS]', 'This', 'is', 'a', 'picture', 'of', 'a', 'b', '##oa', '.']
```

And the network gives me predictions for `b` and `##oa`. But nothing relevant. Could I get predictions for `boa`?



Comments:
What do yuu get when you multiply the probabilities for words in these 2 places. Probability for b times probability for "##oa".
['[CLS]', 'This', 'is', 'a', 'picture', 'of', 'a', '[MASK]', '[MASK]', '.']
['[CLS]', 'This', 'is', 'a', 'picture', 'of', 'a', '[MASK]', '##oa', '.']

But there is also whole word masking model is realized by Google team, I hope it will be added here soon. It mask all the parts of these words at the same time, so it would give a better accuracy (1% for some tasks).
https://github.com/google-research/bert



Nothing interesting.

Moreover, I would not like to mask words. I just want to get predictions with the word in clear.

Well, another solution is to recreate bert with a large vocabulary. This would take days on TPU.


Okay thanks for the answer

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Minor bug in evaluation phase in example code for SQUAD
Author: aakanksha19
Pull request: no

In run_squad.py, the model saving code is outside the loop which performs training (if args.do_train - line 477).
Due to this, after finetuning a model with --do_train, if we only run --do_eval, the existing trained model gets overwritten before being loaded for testing.

**Simple fix:** Pushing model saving code inside the training loop will exhibit desired behavior during testing

Comments:
I fixed it in #923 weeks ago.
waiting merge

Thanks, I did not come across it earlier! I'll close this issue.
when I use bert-large-uncased to load bert,runtimeError occured,but base-uncase is ok
Author: HooFaya
Pull request: no
Labels: name: wontfix

using BertModel.from_pretrained( path of bert-large-uncased) caused error

RuntimeError: $ Torch: invalid memory size -- maybe an overflow? at ..\aten\src\TH\THGeneral.cpp:188

But using  BertModel.from_pretrained( path of bert-case-uncased)  can work


Comments:
Yes, probably overflow. Try a smaller batch size?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Update README after RoBERTa addition
Author: kriss-b
Pull request: yes


Comments:
# [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1046?src=pr&el=h1) Report
> Merging [#1046](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1046?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/pytorch-transformers/commit/d8923270e6c497862f990a3c72e40cc1ddd01d4e?src=pr&el=desc) will **increase** coverage by `0.05%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1046/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1046?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #1046      +/-   ##
==========================================
+ Coverage    79.6%   79.65%   +0.05%     
==========================================
  Files          42       42              
  Lines        6863     6863              
==========================================
+ Hits         5463     5467       +4     
+ Misses       1400     1396       -4
```


| [Impacted Files](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1046?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pytorch\_transformers/file\_utils.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1046/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvZmlsZV91dGlscy5weQ==) | `74.1% <0%> (+2.87%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1046?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1046?src=pr&el=footer). Last update [d892327...b97b7d9](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1046?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Thanks!
Don't save model without training (example/run_squad.py bug)
Author: Liangtaiwan
Pull request: yes

There is a mirror bug in run_squad.py.
The model should not be saved if only do_predict.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/923?src=pr&el=h1) Report
> Merging [#923](https://codecov.io/gh/huggingface/pytorch-transformers/pull/923?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/pytorch-transformers/commit/a7b4cfe9194bf93c7044a42c9f1281260ce6279e?src=pr&el=desc) will **decrease** coverage by `0.03%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/pytorch-transformers/pull/923/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/pytorch-transformers/pull/923?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master     #923      +/-   ##
==========================================
- Coverage   79.22%   79.19%   -0.04%     
==========================================
  Files          38       38              
  Lines        6406     6406              
==========================================
- Hits         5075     5073       -2     
- Misses       1331     1333       +2
```


| [Impacted Files](https://codecov.io/gh/huggingface/pytorch-transformers/pull/923?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pytorch\_transformers/tokenization\_bert.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/923/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvdG9rZW5pemF0aW9uX2JlcnQucHk=) | `94.17% <0%> (-0.98%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/923?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/923?src=pr&el=footer). Last update [a7b4cfe...40aa709](https://codecov.io/gh/huggingface/pytorch-transformers/pull/923?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Ok, could you fix that also in the `run_glue` example?

@thomwolf ```run_glue```  is correct.

@thomwolf I just solved conflicts

This looks good to me, thanks!
Training Transformer XL from scratch
Author: anshuman1992
Pull request: no
Labels: name: wontfix

Hello,

I'm trying to train a transformer XL model from scratch by combining the architecture code from this library and training code from the official paper repo. But this yields to NaNs during training, just wanted to clarify the recommended way to initialize a new model.

Im doing it by,
```
architecture = TransfoXLConfig().from_json_file(args.config_path)
 model = TransfoXLLMHeadModel(architecture)
```
Is there a bug in this?

Comments:
This looks good to me

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


@anshuman1992 could you share a code snippet/gist used for training TransformerXL model?


@anshuman1992 this will be great for me too
Add [CLS] and [SEP] tokens in Usage
Author: tomohideshibata
Pull request: no

Thank you for this great job.

In the Usage section, the `[CLS]` and `[SEP]` tokens should be added in the beginning and ending of  `tokenized_text`?

```
# Tokenized input
text = "Who was Jim Henson ? Jim Henson was a puppeteer"
tokenized_text = tokenizer.tokenize(text)
```

In the current example, if the first token is masked (this position should be reserved for `[CLS]`), the result will be strange.

Thanks.

Comments:
You are right, I'll fix the readme

So, just to clarify, I should add '[CLS]' and '[SEP]' to the beginning and end of each utterance respectively, and it's a bug in the examples that they dont do this?

@hughperkins did you get any clarification on this?
Issue using Roberta
Author: tuhinjubcse
Pull request: no

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): Roberta

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* [x] the official example scripts: (run_glue.py)
* [ ] my own modified scripts: (give details)

![image](https://user-images.githubusercontent.com/3104771/63219474-9ac23f80-c140-11e9-98d2-efe9c8547e05.png)


The tasks I am working on is:
* [x] an official GLUE/SQUaD task: (MNLI/MRPC) 
* [ ] my own task or dataset: (give details)

## To Reproduce

Steps to reproduce the behavior:

just tried running run_glue.py 
see image

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->

## Environment

* OS: Ubuntu
* Python version: 3.7.3
* PyTorch version: 1.1.0
* PyTorch Transformers version (or branch): 1.1.0
* Using GPU ? yes v100
* Distributed of parallel setup ?
* Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->

Comments:
Hello!

It seems you're having trouble accessing the file on our S3 bucket. Could it be your firewall?
If you paste the URL in your browser: https://s3.amazonaws.com/models.huggingface.co/bert/roberta-base-pytorch_model.bin

Can you download it or are you blocked?

Hi 
You can close it . I managed to change firewall settings
Pytorch-Bert: Why this command: pip install pytorch-pretrained-bert doesn't work for me
Author: abril4416
Pull request: no

I try to install pytorch-bert using the command: pip install pytorch-pretrained-bert
However this doesn't work for me.
And the feedback is below:

  Could not find a version that satisfies the requirement pytorch-pretrained-be
rt (from versions: )
No matching distribution found for pytorch-pretrained-bert

I also tried to update my pip, but in vain.
So how can I install bert? (Will the root of the issue be that I use python2.7?)


Comments:
Hi, most likely you'll have to switch to python 3.5 or newer!
https://pypi.org/project/pytorch-pretrained-bert/ (check for requirements in the page: `Requires: Python >=3.5.0`)  

Indeed

Need some help. I'm experiencing the same problem with Python 3.7.3

Error code:
ERROR: Could not find a version that satisfies the requirement torch>=0.4.1 (from pytorch-pretrained-bert==0.4.0) (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2)
ERROR: No matching distribution found for torch>=0.4.1 (from pytorch-pretrained-bert==0.4.0)



Try installing pytorch first following the official instruction on the pytorch website: https://pytorch.org/
Cannot inherit from BertPretrainedModel  anymore after migrating to pytorch-transformers
Author: ereday
Pull request: no

Hi, 

After I updated my environment today, I cannot run my old code anymore. I think I followed all the steps in migration section of README but still the following code gives me the `NameError: name 'BertPreTrainedModel' is not defined` error. To migrate latest version, I cloned the repository and run `pip install --editable .` command within the directory.

Here is the code: 

```python

from pytorch_transformers import *
class BertForMultiLabelSequenceClassification(BertPreTrainedModel):
    def __init__(self, config, num_labels=2):
        super(BertForMultiLabelSequenceClassification, self).__init__(config)
        self.num_labels = num_labels
        self.bert = BertModel("bert-base-multilingual-cased")       
        self.dropout = torch.nn.Dropout(config.hidden_dropout_prob)
        self.classifier = torch.nn.Linear(config.hidden_size, num_labels)
        self.apply(self.init_bert_weights)

    def forward(self, input_ids, token_type_ids=None, attention_mask=None, labels=None):
        _, pooled_output = self.bert(input_ids, token_type_ids, attention_mask, output_all_encoded_layers=False)
        pooled_output = outputs[-1]
        pooled_output = self.dropout(pooled_output)
        logits = self.classifier(pooled_output)
        return logits

args = {
    "train_size": -1,
    "val_size": -1,
    "bert_model": "bert-base-multilingual-cased",
    "do_lower_case":False,
    "max_seq_length": 100,
    "do_train": True,
    "do_eval": True,
    "train_batch_size": 32,
    "eval_batch_size": 32,
    "learning_rate": 3e-5,
    "num_train_epochs": 20,
    "warmup_proportion": 0.1,
    "no_cuda": False,
    "local_rank": -1,
    "seed": 42,
}

num_labels = 2
model = BertForMultiLabelSequenceClassification.from_pretrained(args['bert-model'],num_labels)
```

Comments:
You should do `from pytorch_transformers.modeling_bert import BertPreTrainedModel`

I'll add these to the main `__init__.py`

Thank you for the answer @thomwolf . It solved that error but now I'm getting another one (which wasn't there when I was using previous versions of the repository): `TypeError: unhashable type: 'BertConfig'` what could be wrong ?  

We need a full error log and more details.
docs: correct number of layers for various xlm models
Author: stefan-it
Pull request: yes

Hi,

during some NER experiments I found out, that the number of reported layers in the documentation is different compared to the model configuration for some XLM models.

This PR fixes the documentation :)

Comments:
# [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/984?src=pr&el=h1) Report
> Merging [#984](https://codecov.io/gh/huggingface/pytorch-transformers/pull/984?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/pytorch-transformers/commit/7729ef738161a0a182b172fcb7c351f6d2b9c50d?src=pr&el=desc) will **increase** coverage by `0.06%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/pytorch-transformers/pull/984/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/pytorch-transformers/pull/984?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master     #984      +/-   ##
==========================================
+ Coverage   79.16%   79.22%   +0.06%     
==========================================
  Files          38       38              
  Lines        6406     6406              
==========================================
+ Hits         5071     5075       +4     
+ Misses       1335     1331       -4
```


| [Impacted Files](https://codecov.io/gh/huggingface/pytorch-transformers/pull/984?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pytorch\_transformers/file\_utils.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/984/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvZmlsZV91dGlscy5weQ==) | `74.1% <0%> (+2.87%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/984?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/984?src=pr&el=footer). Last update [7729ef7...39f51cd](https://codecov.io/gh/huggingface/pytorch-transformers/pull/984?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Thanks Stefan, there were a few other typos in these models details indeed so I'll take care of this in another PR.
Swap of optimizer.step and scheduler.step for lm finetuning examples
Author: Morizeyao
Pull request: yes

Prior to PyTorch 1.1.0, the learning rate scheduler was expected to be called before the optimizer’s update; 1.1.0 changed this behavior in a BC-breaking way. If you use the learning rate scheduler (calling scheduler.step()) before the optimizer’s update (calling optimizer.step()), this will skip the first value of the learning rate schedule. If you are unable to reproduce results after upgrading to PyTorch 1.1.0, please check if you are calling scheduler.step() at the wrong time.

This is my first time very simple PR, please correct me if there's anything done wrong xD.

[link](https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate)

Comments:
# [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1056?src=pr&el=h1) Report
> Merging [#1056](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1056?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/pytorch-transformers/commit/c589862b783b94a8408b40c6dc9bf4a14b2ee391?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1056/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1056?src=pr&el=tree)

```diff
@@          Coverage Diff           @@
##           master   #1056   +/-   ##
======================================
  Coverage    79.6%   79.6%           
======================================
  Files          42      42           
  Lines        6863    6863           
======================================
  Hits         5463    5463           
  Misses       1400    1400
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1056?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1056?src=pr&el=footer). Last update [c589862...d86b49a](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1056?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Looks good to me, thanks @Morizeyao!
Correct truncation for RoBERTa in 2-input GLUE
Author: zphang
Pull request: yes

Extend the truncation fix to the two-input case.

(Example: currently throws if running MRPC with `max_seq_length=32`.)

Comments:
Looks good to me, thanks!

Actually, for single-sentence inputs, do we expect one or two terminating `
`s? Currently we will generate two, I think.

@LysandreJik we can now update the GLUE scripts to use the newly added option `add_special_tokens` (added to all the tokenizers), don't you think?

Indeed, we should use it. I'll add that soon.
Fix bug of multi-gpu training in lm finetuning
Author: FeiWang96
Pull request: yes

Current code will raise error when running multi-gpu training (n_gpu > 1 & local_rank = -1).
 


Comments:
# [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1040?src=pr&el=h1) Report
> Merging [#1040](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1040?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/pytorch-transformers/commit/9d0029e215f5ad0836d6be87458aab5142783af4?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1040/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1040?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #1040   +/-   ##
=======================================
  Coverage   79.55%   79.55%           
=======================================
  Files          42       42           
  Lines        6863     6863           
=======================================
  Hits         5460     5460           
  Misses       1403     1403
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1040?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1040?src=pr&el=footer). Last update [9d0029e...856a63d](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1040?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Make sense, thanks @FeiWang96!