noamaanMulla-03/transformers-issues huggingface dataset:
Updated typo on the link
Author: ejarkm
Pull request: yes

Updated documentation due to typo

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2221?src=pr&el=h1) Report
> Merging [#2221](https://codecov.io/gh/huggingface/transformers/pull/2221?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/8efc6dd544bf1a30d99d4b5abfc5e214699eab2b?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2221/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2221?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #2221   +/-   ##
=======================================
  Coverage   81.47%   81.47%           
=======================================
  Files         122      122           
  Lines       18344    18344           
=======================================
  Hits        14946    14946           
  Misses       3398     3398
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2221?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2221?src=pr&el=footer). Last update [8efc6dd...f9dbf62](https://codecov.io/gh/huggingface/transformers/pull/2221?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Great, nice catch!
Trouble loading Albert model
Author: jswift24
Pull request: no

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): Albert

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* [x] the official example scripts: (give details):
* [ ] my own modified scripts: (give details)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: (give the name) 
* [ ] my own task or dataset: (give details)

Using example from docs: https://huggingface.co/transformers/model_doc/albert.html

## To Reproduce
Trying to load the Albert model using the code below:
```
import tensorflow as tf
from transformers import AlbertTokenizer
tokenizer = AlbertTokenizer.from_pretrained('bert-base-uncased')
```
Getting the following error:
```
Traceback (most recent call last):

  File "<ipython-input-4-56254e5f4b51>", line 3, in <module>
    tokenizer = AlbertTokenizer.from_pretrained('bert-base-uncased')

  File "c:\users\admin\appdata\local\programs\python\python37\lib\site-packages\transformers\tokenization_utils.py", line 302, in from_pretrained
    return cls._from_pretrained(*inputs, **kwargs)

  File "c:\users\admin\appdata\local\programs\python\python37\lib\site-packages\transformers\tokenization_utils.py", line 437, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)

  File "c:\users\admin\appdata\local\programs\python\python37\lib\site-packages\transformers\tokenization_albert.py", line 90, in __init__
    self.sp_model.Load(vocab_file)

  File "c:\users\admin\appdata\local\programs\python\python37\lib\site-packages\sentencepiece.py", line 118, in Load
    return _sentencepiece.SentencePieceProcessor_Load(self, filename)

RuntimeError: Internal: C:\projects\sentencepiece\src\sentencepiece_processor.cc(73) [model_proto->ParseFromArray(serialized.data(), serialized.size())] 
```


Note, the same code works for bert instead of albert:
> from transformers import *
> tokenizer = BertTokenizer.from_pretrained('bert-base-cased')
> 

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior
Looking for albert model to load without errors
<!-- A clear and concise description of what you expected to happen. -->

## Environment

* OS: Windows 10
* Python version: 3.7
* PyTorch version: 1.3.1
* PyTorch Transformers version (or branch): pip install transformers (fresh install today 12/19/2019)
* Using GPU ?: Yes
* Distributed of parallel setup ?: N/A
* Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->


Comments:
Indeed, that's an error from the docs! I just fixed it with 33adab2. The doc should be updated now, thanks for raising this issue.

Wow, that was fast. Thank you!
weights not initialised in pre-trained Roberta
Author: btel
Pull request: no

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): Robert

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* [ ] the official example scripts: (give details)
* [X] my own modified scripts: see below

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: I am importing pre-trained models

## To Reproduce

When running the following code:

```
import logging
logging.basicConfig(level=logging.INFO)
from transformers import RobertaForMaskedLM
pt_m = RobertaForMaskedLM.from_pretrained('roberta-base')
```
 
I am getting the following messages in the log:

```
INFO:transformers.modeling_utils:loading weights file https://s3.amazonaws.com/models.huggingface.co/bert/roberta-base-pytorch_model.bin from cache at /home/bartosz/.cache/torch/transformers/228756ed15b6d200d7cb45aaef08c087e2706f54cb912863d2efe07c89584eb7.49b88ba7ec2c26a7558dda98ca3884c3b80fa31cf43a1b1f23aef3ff81ba344e
INFO:transformers.modeling_utils:Weights of RobertaForMaskedLM not initialized from pretrained model: ['lm_head.decoder.weight']
INFO:transformers.modeling_utils:Weights from pretrained model not used in RobertaForMaskedLM: ['lm_head.weight']
```

In particular, I am concerned that the weight of the LM head were not initialised from the stored values.

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior

The model should load and all weights should get initialized from the pre-trained model.

<!-- A clear and concise description of what you expected to happen. -->

## Environment

* OS: Linux
* Python version: 3.7
* PyTorch version: 1.2.1
* PyTorch Transformers version (or branch): 2.2.2
* Using GPU ? No
* Distributed of parallel setup ? No
* Any other relevant information:

## Additional context

If I comment out these two lines the messages disappear:

https://github.com/huggingface/transformers/blob/f061606277322a013ec2d96509d3077e865ae875/transformers/modeling_utils.py#L445-L446

<!-- Add any other context about the problem here. -->


Comments:
Thanks for the bug report. Fixed on `master` in 9a399ead253e27792cbf0ef386cc39f9b7084f8f by reverting the output of #1778.
Minor/basic text fixes
Author: aidankierans
Pull request: yes


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2229?src=pr&el=h1) Report
> Merging [#2229](https://codecov.io/gh/huggingface/transformers/pull/2229?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/a1f1dce0ae511ef7766c6b6a8f5ebf9118279e73?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2229/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2229?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #2229   +/-   ##
=======================================
  Coverage   81.47%   81.47%           
=======================================
  Files         122      122           
  Lines       18344    18344           
=======================================
  Hits        14946    14946           
  Misses       3398     3398
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2229?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [transformers/data/processors/squad.py](https://codecov.io/gh/huggingface/transformers/pull/2229/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL2RhdGEvcHJvY2Vzc29ycy9zcXVhZC5weQ==) | `14.18% <0%> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2229?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2229?src=pr&el=footer). Last update [a1f1dce...70dbca5](https://codecov.io/gh/huggingface/transformers/pull/2229?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Thanks!
How to load a different domain BERT-based pre-trained model?
Author: vr25
Pull request: no
Labels: name: wontfix

I am trying to load the pre-trained model at pred/FinBERT-Pre2K_128MSL-500K  [FinBERT](https://github.com/psnonis/FinBERT)  and trying to run the basic task of SST-2 (sentiment classification) using run_glue.py (https://huggingface.co/transformers/examples.html#glue).

But I run into the following error:
OSError: Model name '/data/ftm/xgb_regr/FinBERT/pred/FinBERT-Pre2K_128MSL-250K' was not found in model name list (bert-base-uncased, bert-large-uncased, bert-base-cased, bert-large-cased, bert-base-multilingual-uncased, bert-base-multilingual-cased, bert-base-chinese, bert-base-german-cased, bert-large-uncased-whole-word-masking, bert-large-cased-whole-word-masking, bert-large-uncased-whole-word-masking-finetuned-squad, bert-large-cased-whole-word-masking-finetuned-squad, bert-base-cased-finetuned-mrpc, bert-base-german-dbmdz-cased, bert-base-german-dbmdz-uncased). We assumed '/data/ftm/xgb_regr/FinBERT/pred/FinBERT-Pre2K_128MSL-250K' was a path or url to a configuration file named config.json or a directory containing such a file but couldn't find any such file at this path or url.

Also, since this seems to be trained using TF, I was wondering if I can use PyTorch to load it.

Thanks.

Comments:
You can use torch to load it, convert the weights using the helper files;
Not sure about your task, but for mine, i was using a BertModel with different pre-trained weights,
```
model = BertForSequenceClassification(MODEL_PATH, num_labels=len(np.unique(y_train_torch)))
```
(iirc from_tf is also a param to the function)

where `MODEL_PATH` is a directory that has 
- config.json.
- your model [checkpoint/bin file].
- a vocab file as well.

Thank you for your reply. 

The issue is a little different. All the 3 files: config.json, checkpoint, and vocab.txt are linked by a symbolic link in their repo. I am not sure how to get the actual files. Any suggestions for such a case?

Well if you are running the experiments yourself, you will be downloading them either ways, just make changes where ever needed?

(i haven't tried passing a symbolic link to this func so not sure myself but it should work imo as well)

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
[REVIEW] Removed duplicate XLMConfig, XLMForQuestionAnswering and XLMTokenizer in run_squad.py
Author: TheEdoardo93
Pull request: yes

Before this PR, in run_squad.py at lines 58-65 were the following:
```
MODEL_CLASSES = {
    'bert': (BertConfig, BertForQuestionAnswering, BertTokenizer),
    'xlnet': (XLNetConfig, XLNetForQuestionAnswering, XLNetTokenizer),
    'xlm': (XLMConfig, XLMForQuestionAnswering, XLMTokenizer),
    'distilbert': (DistilBertConfig, DistilBertForQuestionAnswering, DistilBertTokenizer),
    'albert': (AlbertConfig, AlbertForQuestionAnswering, AlbertTokenizer),
    'xlm': (XLMConfig, XLMForQuestionAnswering, XLMTokenizer)
}
```

After this PR, I've **removed** the last (key, value) pair in the `MODEL_CLASSES` dictionary because it contains _xlm_ as key **two** times.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2224?src=pr&el=h1) Report
> Merging [#2224](https://codecov.io/gh/huggingface/transformers/pull/2224?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/8efc6dd544bf1a30d99d4b5abfc5e214699eab2b?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2224/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2224?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #2224   +/-   ##
=======================================
  Coverage   81.47%   81.47%           
=======================================
  Files         122      122           
  Lines       18344    18344           
=======================================
  Hits        14946    14946           
  Misses       3398     3398
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2224?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2224?src=pr&el=footer). Last update [8efc6dd...e6a7670](https://codecov.io/gh/huggingface/transformers/pull/2224?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Great, thanks @TheEdoardo93 !
[ Structure of LM vocab trained from scratch ]
Author: simonefrancia
Pull request: no

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

Hi,
I am trying to create a BERT LM trained from scratch and I have a question about the tokenizer.
I have a big text corpus and I trained a tokenizer with SentencePiece with 32K as dimension of the vocabulary. Then I applied a transformation from SentencePiece notation to WordPiece notation in order to be compatible with BERT.
The result is that my dictionary has this structure:

----------------------------------------

MY DICTIONARY
```
0 [PAD]
1 [UNK]
2 [CLS]
3 [SEP]
4 [MASK]
5 ,
6 .
7 '
... 
.... [ ALL USED TOKENS ]
....
100 ##ndo
...
.... [ ALL USED TOKENS ]
....
31741 [ disgustat ]
31742 [ UNUSED TOKEN]
.......
.......  [ ALL UNUSED TOKENS ]
.......
31999 [ UNUSED TOKEN]
```
And all the unused tokens are at the end of the vocab, in my case from 31742 to 31999.

---------------------------

And this is the STANDARD VOCAB for BERT:


BERT cased_L-12_H-768_A-12 VOCABULARY

```
0  [PAD]
1 [unused1]
2  ........ 
3  ........ [ ALL UNUSED TOKENS ]
4  .......
...........
100 [UNK]
101 [CLS]
102 [SEP]
103 [MASK]
[unused100]
[unused101]
!
"
..... [ ALL USED TOKENS ]
```
----------------------------

My question is: the fact that the SPECIAL TOKENS in MY DICTIONARY are in different positions than the Standard BERT VOCABULARY , can be a problem? Do you think I should keep the same positions of BERT vocab also for my dictionary? Also for unused token I have the same doubts. 

( I saw that in the sentence piece training it's possibile to specify the exact positions of special tokens, but my question is if the position of special tokens will affect the LM training in some way )

Thank you






Comments:
I don't think it is a problem. 
Your model will learn the embeddings of the words in your own dictionary.

Actually Nothing will be unchanged if you changed dictionary position as well as you keeped the embedding weight just the same order with your dictionary. 

Thanks @karajan1001. 
I am not sure if I understood well the last sentence. 
You meant that I must not change the dictionary after I train the LM based on that? 
After that I train the SentencePiece tokenizer and the relative vocabulary is given, than the vocab is an input of Language Model training and it cannot be changed anymore.

> After that I train the SentencePiece tokenizer and the relative vocabulary is given, than the vocab is an input of Language Model training and it cannot be changed anymore.

I think so. The vocabulary tells the model which array to get with the input tokens.
Keep even the first of the special tokens intact while lowercasing
Author: dirkgr
Pull request: yes

This fixes #2220. The order of `all_special_tokens` is random, and the first one of these will get broken by the lowercasing. There are 5 special tokens, so you have a 1 in 5 chance of hitting the problem.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2232?src=pr&el=h1) Report
> Merging [#2232](https://codecov.io/gh/huggingface/transformers/pull/2232?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/a5a06a851e1da79138e53978aa079a093f243dde?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2232/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2232?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #2232   +/-   ##
=======================================
  Coverage   81.43%   81.43%           
=======================================
  Files         122      122           
  Lines       18338    18338           
=======================================
  Hits        14933    14933           
  Misses       3405     3405
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2232?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [transformers/tokenization\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/2232/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL3Rva2VuaXphdGlvbl91dGlscy5weQ==) | `90.49% <100%> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2232?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2232?src=pr&el=footer). Last update [a5a06a8...06b022d](https://codecov.io/gh/huggingface/transformers/pull/2232?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


> This fixes #2220. The order of `all_special_tokens` is random, and the first one of these will get broken by the lowercasing. There are 5 special tokens, so you have a 1 in 5 chance of hitting the problem.

You're right! Good job @dirkgr :-)

Great, thanks a lot for that @dirkgr. I've added a test in fb393ad9945f66b081f88b81b90a2974d81e9601 to make sure this doesn't happen again.
Main and train for CTRL model
Author: roholazandie
Pull request: no
Labels: name: wontfix

## 🚀 Feature

I have seen the CTRL model has been added to the repo but I don't see any script to run or train it. Is this going to be added soon?



Comments:
The CTRL model has been added to the [run_generation](https://github.com/huggingface/transformers/blob/master/examples/run_generation.py) script as of now. We will implement it in other scripts as time goes on, but as it has the same API as the other models hosted on our repo it the training script would be very similar to the current training scripts.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
:bug: #2120 in model.from_pretrained, PosixPath crashes at "albert" check
Author: mandubian
Pull request: yes

- `pretrained_model_name_or_path` is now stringified to allow the "albert" and "v2" checks with PosixPath (or any other path representation that isn't iterable).
- If `pretrained_model_name_or_path` is None, it gives string "None" which doesn't contain "albert" so it's OK.
- 2 x `str(pretrained_model_name_or_path)` doesn't impact performances as it's called only once per program and `and` operator will generally fail at left side.
- added a test to check non-regression

Comments:
Ok, I've checked the errors in CI. Those are linked to the fact that in older python version, PosixPath is not converted automatically to String and `os.path.isdir/isfile` crash because it expects a string or int. So my patch works perfectly in latest version of python (like 3.7) but not older (like 3.5) which is quite ugly.
Solutions are:
- anywhere there is `isdir/isfile`, force `str(path)` in call which is ugly but will work except if you want to manage other types than strings in path
- officially ask people to convert their path to strings when calling `from_pretrained` which is shame because it shows the impossibility to be backward compatible completely but it won't introduce `str()` everywhere in the code.

WDYT?


Hey @mandubian, thanks for offering that fix. I think that in the next release we'll remove this warning about albert models v2, which will solve the problem with PosixPaths.

@LysandreJik perfect! It will still fail with python 3.5 on `isdir/isfile` but can we do anything for that? I'm not sure... that's the history of Python ;)

The line was removed for version 2.3.0 so there's no need for that anymore. Thanks @mandubian :).
Predefined token classification 
Author: ghost
Pull request: no
Labels: name: wontfix

Hello, 
I am just wondering if "BertForTokenClassification"  can be modified to classify predefined tokens (just targeted tokens). E.g. in NER it identifies the entity and then classify it, but in my case I want to classify the targeted tokens only in a sentence with a predefined labels. I thought of adding the targeted token in the sentence to the end of the sentence to let the model knows this is the targeted token but I am sure it is not a good idea because I need to again add a label for the added token. (silly Q I know :)
Any idea on how to approach it? I feel that NER task "BertForTokenClassification" 
model can be modified to achieve it but do not know how. 
Thanks 

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Accuracy drop in finetuning roBERTa
Author: Arjunsankarlal
Pull request: no
Labels: name: wontfix

## ❓ How to achieve GLUE leaderboard acc for QQP task trained with roBERTa?

I am trying to finetune roberta-base model for Quora Question Pair task. In the [GLUE Leaderboard](url) the accuracy claimed is F1 / Accuracy is 74.3/90.2

I am training with the following command to fine tune the roberta-base model,

> CUDA_VISIBLE_DEVICES=0 python run_glue.py --data_dir /home/arjun/datasets/quora_roberta --model_type roberta --model_name_or_path /home/arjun/transformers/models/roberta --task_name qqp --output_dir /home/arjun/transformers/output_models/run-2/ --do_train --do_eval --do_lower_case --logging_steps 250 --save_steps 5000

All other params are default params.

I get the accuracy after 3 epochs as (taken from the eval_results.txt file),
acc = 0.6329982984448578
acc_and_f1 = 0.3164991492224289
f1 = 0.0

Also I ran evaluation for all the checkpoints stored, for every checkpoint I got the same accuracy which is quite confusing.

I'm I doing something wrong here? How to reproduce the accuracy mentioned in GLUE leaderboard?

Thanks in advance.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.