noamaanMulla-03/transformers-issues huggingface dataset:
Add CamemBERT models to modeling_auto
Author: Evpok
Pull request: yes

CamemBERT was in autoconfig but not in automodel, this PR aims to correct that. Also first PR here so please tell me if I missed some things :-)


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1902?src=pr&el=h1) Report
> Merging [#1902](https://codecov.io/gh/huggingface/transformers/pull/1902?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/e70cdf083ddb8bfe298d43e6d70d698a3a2f56d3?src=pr&el=desc) will **decrease** coverage by `0.03%`.
> The diff coverage is `14.28%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1902/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1902?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #1902      +/-   ##
==========================================
- Coverage   84.08%   84.04%   -0.04%     
==========================================
  Files          97       97              
  Lines       14316    14323       +7     
==========================================
+ Hits        12037    12038       +1     
- Misses       2279     2285       +6
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/1902?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [transformers/modeling\_auto.py](https://codecov.io/gh/huggingface/transformers/pull/1902/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX2F1dG8ucHk=) | `31.81% <14.28%> (-1.52%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1902?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1902?src=pr&el=footer). Last update [e70cdf0...75ef125](https://codecov.io/gh/huggingface/transformers/pull/1902?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Great, thanks @Evpok !
F1 socre is zero while loss is about 0.12xx when using run_ner.py to fine tuning bert model
Author: Sunnycheey
Pull request: no

## ❓ Questions & Help

I want to use run_ner.py to do the sequence labeling work, and I make a few class of labels(e.g., B-School_name), then I annotate a few data (about 100 annotations for test) from dataset. Finally, I use run_ner.py to do the fine tuning job, but the f1 score and precision are both zero (but the loss is not). Have I got something wrong?

Comments:
Hi @Sunnycheey I had the same issue when using my own data and labels. However, I fixed this once I ensured that the training data (train.txt) was in the correct format: WordToken/space/label, newline after every sentence (fullstop). 

While O
visiting O
Otonga B-Sc
primary I-Sc
he O
noticed O
an O
anomaly O
. O

The O
next O
sentence O
etc...




Thanks for your reply. 

Is the fullstop matters? I use spacy to get sentences from document, so there may not be fullstop in the end of sentence...

BTW, I have used training dataset to test the model trained, and the f1 score is 1 (not 0)...

I got it to work with fullstop and newline as above.
I use stanfordnlp to prepare the data, but it is probably fine to use SpaCy.  Make sure you have a separate folder for the output too.

@jensam It turns out if all the predict result is O (in BIO annotation scheme), the f1 score tends to be 0.

Thanks anyway.
Documentation error in GPT2Tokenizer
Author: CrafterKolyan
Pull request: no

Documentation page:
https://huggingface.co/transformers/model_doc/gpt2.html#transformers.GPT2Tokenizer
Code place:
https://github.com/huggingface/transformers/blob/d7d36181fdefdabadc53adf51bed4a2680f5880a/transformers/tokenization_gpt2.py#L112-L113

This phrase:
> Otherwise, this tokenizer encode and decode method will not conserve the absence of a space at the beginning of a string: tokenizer.decode(tokenizer.encode(“Hello”)) = ” Hello”

is **NOT** correct.

Actually:
> tokenizer.decode(tokenizer.encode(“Hello”)) = ”Hello”

Try this example:
```
from transformers import GPT2Tokenizer
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
print("'" + tokenizer.decode(tokenizer.encode("Hello")) + "'")
```
Output:
```
'Hello'
```

Comments:
According to the code (https://github.com/huggingface/transformers/blob/master/transformers/tokenization_gpt2.py#L183) the docstring is incomplete. 

The `add_prefix_space` parameter should be passed to `tokenizer.tokenize()` as well as `tokenizer.encode()` or `tokenizer.decode()`.

As well, the docstring should actually say:

```
Otherwise, this tokenizer encode and decode method will not conserve a space at the beginning of a string: tokenizer.decode(tokenizer.encode(“ Hello”)) = ”Hello”
```

According to #1380, the Roberta/GPT2 tokenizer expects sequences to start with a space.  Simply prepending a space to the input sequence doesn't give the same result because `GPT2Tokenizer` only overrides the internally-used `_tokenizer()` method and `PreTrainedTokenizer`'s `tokenize()` method (which is called by the user) does it's own preprocessing.
access to the vocabulary
Author: weiguowilliam
Pull request: no

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

Is there any way we can get access to the vocabulary in GPT2? Like a list: [subtoken1, subtoken2, ...subtoken 10000...]

Thank you in advance!

Comments:
You can obtain the **50.257 different tokens** with the following code:
```
import transformers
from transformers import GPT2Tokenizer

tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
vocab = list(tokenizer.encoder.keys())
assert(len(vocab) == tokenizer.vocab_size) # it returns True!
```

Close the issue if you've resolved your problem! ;)

> ##  Questions & Help
> Is there any way we can get access to the vocabulary in GPT2? Like a list: [subtoken1, subtoken2, ...subtoken 10000...]
> 
> Thank you in advance!

thank you！
run_finetuning resize token embeddings
Author: iedmrc
Pull request: yes

Please see #1848 

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1849?src=pr&el=h1) Report
> Merging [#1849](https://codecov.io/gh/huggingface/transformers/pull/1849?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/0477b307c7501ea76e01b03cb387a2312db752b3?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1849/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1849?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #1849   +/-   ##
=======================================
  Coverage   84.08%   84.08%           
=======================================
  Files          97       97           
  Lines       14316    14316           
=======================================
  Hits        12037    12037           
  Misses       2279     2279
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1849?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1849?src=pr&el=footer). Last update [0477b30...a76db2e](https://codecov.io/gh/huggingface/transformers/pull/1849?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Looks good to me, thanks for looking into it!

You are welcome and thanks @LysandreJik for interested. Do you mind approve and merge, if there is no problem?
Using GPT-2 XL
Author: IrvDelgado
Pull request: no

Hi, Im trying to use the pretrained gpt-xl
but I get the following error: 
OSError: Model name 'gpt2-xl' was not found in tokenizers model name list (gpt2, gpt2-medium, gpt2-large, distilgpt2). We assumed 'gpt2-xl' was a path or url to a directory containing vocabulary files named ['vocab.json', 'merges.txt'] but couldn't find such vocabulary files at this path or url.

Im following an example from the documentation. 

      7 # Load pre-trained model tokenizer (vocabulary)
---> 8 tokenizer = GPT2Tokenizer.from_pretrained('gpt2-xl')
      9 # Load pre-trained model (weights)
     10 model = GPT2LMHeadModel.from_pretrained('gpt2-xl')
Any Idea why?

Thank you. :)

Comments:
You don't have to use the PyPi version of this library, but **from the source code** with `pip install git+https://github.com/huggingface/transformers.git`. This is because, at the moment, GPT2 XL version is available only in a dedicated branch called *gpt2-xl* and not in the PyPi version.

My environment is the following:
- __Python__: 3.6.9
- __O.S.__: Linux-4.15.0-70-generic-x86_64-with-debian-buster-sid
- __Transformers__: 2.1.1 (installed from source)
- __Torch__: 1.3.1

After that, you're able to use OpenAI GPT-2 XL version as always, e.g.
```
import transformers
from transformers import GPT2Tokenizer
from transformers import GPT2Model

tokenizer = GPT2Tokenizer.from_pretrained('gpt2-xl')
model = GPT2Model.from_pretrained('gpt2-xl')
...
```

Please, close this issue!

> Hi, Im trying to use the pretrained gpt-xl
> but I get the following error:
> OSError: Model name 'gpt2-xl' was not found in tokenizers model name list (gpt2, gpt2-medium, gpt2-large, distilgpt2). We assumed 'gpt2-xl' was a path or url to a directory containing vocabulary files named ['vocab.json', 'merges.txt'] but couldn't find such vocabulary files at this path or url.
> 
> Im following an example from the documentation.
> 
> ```
>   7 # Load pre-trained model tokenizer (vocabulary)
> ```
> 
> ---> 8 tokenizer = GPT2Tokenizer.from_pretrained('gpt2-xl')
> 9 # Load pre-trained model (weights)
> 10 model = GPT2LMHeadModel.from_pretrained('gpt2-xl')
> Any Idea why?
> 
> Thank you. :)
Minor bug fixes on run_ner.py
Author: manansanghi
Pull request: yes

Adding a dictionary entry outside of initialization requires an '=' instead of ':'

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1918?src=pr&el=h1) Report
> Merging [#1918](https://codecov.io/gh/huggingface/transformers/pull/1918?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/26db31e0c09a8b5e1ca7a61c454b159eab9d86be?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1918/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1918?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #1918   +/-   ##
=======================================
  Coverage   84.04%   84.04%           
=======================================
  Files          97       97           
  Lines       14333    14333           
=======================================
  Hits        12046    12046           
  Misses       2287     2287
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1918?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1918?src=pr&el=footer). Last update [26db31e...17949e4](https://codecov.io/gh/huggingface/transformers/pull/1918?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


:+1:  sorry, that was a pasting mistake in https://github.com/huggingface/transformers/pull/1792 🙈
Need a Restore training mechenisim in run_lm_finetuning.py
Author: chuanmingliu
Pull request: no

## 🚀 Feature

<!-- A clear and concise description of the feature proposal. Please provide a link to the paper and code in case they exist. -->

## Motivation
When training run_lm_finetuning.py for a long time, a restore training feature should be added.

Otherwise, states of sheduler and optimizer are changed when restart.

For example, when it breaks at step checkpoint-30000, it will restart at step 0 with initial learning rate and other configs. This is really troublesome.

Thanks, please.

<!-- Please outline the motivation for the proposal. Is your feature request related to a problem? e.g., I'm always frustrated when [...]. If this is related to another GitHub issue, please link here too. -->

## Additional context

<!-- Add any other context or screenshots about the feature request here. -->


Comments:
If you want to resume training with the same learning rate, you can save the scheduler and optimizer and reload them when resuming training.

For example, you could save the current training state with:

```python

# Save the model and tokenizer
model.save_pretrained('./checkpoints/')
tokenizer.save_pretrained('./checkpoints/')

# Save the optimizer and scheduler
torch.save(optimizer.state_dict(), './checkpoints/optimizer.pt')
torch.save(scheduler.state_dict(), './checkpoints/scheduler.pt')
```

And resume training with:

```python
# Initialize model and tokenizer from checkpoints dir
model = BertModel.from_pretrained('./checkpoints/')
tokenizer = BertTokenizer.from_pretrained('./checkpoints/')

# Load optimizer and scheduler state
optimizer.load_state_dict(torch.load('./checkpoints/optimizer.pt'))
scheduler.load_state_dict(torch.load('./checkpoints/scheduler.pt'))
```

If you want more information, take a look at #839 and Pytorch's model serialization [tutorial](https://pytorch.org/tutorials/beginner/saving_loading_models.html)

If you want to resume training at the exact epoch and batch where you left off, like this [person](https://github.com/huggingface/transformers/issues/839#issuecomment-515129371), you could save the epoch and batch number as well and `continue` all iterations until you reach the correct batch

@bkkaggle Thanks for your reply, it really helps a lot!

Thank you!

@bkkaggle 
However, the reasons that I change to PyTorch (Transformers by huggingface) are easy to use and thousands more positive ones.

> Why not adding an universal functionality to smoothly support this feature, like TF checkpoint does?

I think that is a natural way to save checkpoint when training.

It sounds more troublesome to customize the checkpoint style by users themselves, considering the high-level encapsulation characteristic brought by the framework.
Can i train my own text corpus
Author: shashankMadan-designEsthetics
Pull request: no

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
I get the idea of using `from_pretrained` but Can i train my own text corpus and then get the weights? If So how?

Comments:
You can initialize the weights of your model as the ones of e.g. BERT, and after that you can fine-tune your model with _your own data_ (**transfer learning**). Please see [here](https://github.com/huggingface/transformers/blob/master/examples/run_lm_finetuning.py) for fine-tuning for a particular task.

Have i answered to you?

> ##  Questions & Help
> I get the idea of using `from_pretrained` but Can i train my own text corpus and then get the weights? If So how?

> You can initialize the weights of your model as the ones of e.g. BERT, and after that you can fine-tune your model with _your own data_ (**transfer learning**). Please see [here](https://github.com/huggingface/transformers/blob/master/examples/run_lm_finetuning.py) for fine-tuning for a particular task.
> 
> Have i answered to you?
> 
> > ## Questions & Help
> > I get the idea of using `from_pretrained` but Can i train my own text corpus and then get the weights? If So how?

Hey Thanks! I'll go through the source code you referred, also just wanted to confirm the same goes with gpt-2 model right?

Yeah, this particular script works with OpenAI GPT-2 too.
In general, the most part of the code is the same even changing the model chosen.

> > You can initialize the weights of your model as the ones of e.g. BERT, and after that you can fine-tune your model with _your own data_ (**transfer learning**). Please see [here](https://github.com/huggingface/transformers/blob/master/examples/run_lm_finetuning.py) for fine-tuning for a particular task.
> > Have i answered to you?
> > > ## Questions & Help
> > > I get the idea of using `from_pretrained` but Can i train my own text corpus and then get the weights? If So how?
> 
> Hey Thanks! I'll go through the source code you referred, also just wanted to confirm the same goes with gpt-2 model right?

Thanks for the quick response, i'll close this one then.
Fixed typo
Author: AveryLiu
Pull request: yes

Changed `emove` to `remove`

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1946?src=pr&el=h1) Report
> Merging [#1946](https://codecov.io/gh/huggingface/transformers/pull/1946?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/5d3b8daad2cc6287d30f03f8a96d0a1f7bc8d0dc?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1946/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1946?src=pr&el=tree)

```diff
@@          Coverage Diff           @@
##           master   #1946   +/-   ##
======================================
  Coverage      84%     84%           
======================================
  Files          97      97           
  Lines       14340   14340           
======================================
  Hits        12047   12047           
  Misses       2293    2293
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/1946?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [transformers/modeling\_bert.py](https://codecov.io/gh/huggingface/transformers/pull/1946/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX2JlcnQucHk=) | `87.77% <100%> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1946?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1946?src=pr&el=footer). Last update [5d3b8da...e1d116d](https://codecov.io/gh/huggingface/transformers/pull/1946?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
CUDA runtime error (59) : device-side assert triggered 
Author: iedmrc
Pull request: no

Hi,
When you run `run_lm_finetuning.py` with a custom tokenizer via `--tokenizer_name tokenizer` parameter, you need to resize model embedding according to the new tokenizer. Otherwise you are going to get `CUDA runtime error (59) : device-side assert triggered` or `Assertion 'srcIndex < srcSelectDimSize' failed`.

I fixed this issue and going to create a PR if it's okay?
Thanks!

Comments:
Indeed, this seems to be a problem. Thanks for the PR!

fixed at #1849
add_tokens() leading to wrong behavior
Author: amity137
Pull request: no

Hey folks, I am using GPT2Tokenizer and tried add_tokens() below. But it gives unintended behavior and this is because of #612 in tokenization_utils.py
                sub_text = sub_text.strip()
As you see, dollars is split because of space removal. Could someone look into it please.

```
t1 = GPT2Tokenizer.from_pretrained('gpt2')
t1.tokenize('my value in dignums dollars')
```
Output - ['my', 'Ġvalue', 'Ġin', 'Ġdign', 'ums', '**Ġdollars**']

```
t2 = GPT2Tokenizer.from_pretrained('gpt2')
t2.add_tokens(['dignums'])
t2.tokenize('my value in dignums dollars')
```
Output - ['my', 'Ġvalue', 'Ġin', 'dignums', **'d', 'oll', 'ars'**]

Comments:
I guess that's why add_prefix_space=True would be needed here.
Expected object of scalar type Byte but got scalar type Bool for argument #2 'mask'
Author: bigzhouj
Pull request: no

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
fine tuning bert  use wikitext2 

Comments:
in probability_matrix.masked_fill_(torch.tensor(special_tokens_mask, dtype=torch.bool), value=0.0)

Please, describe your environment, post the source code for reproducibility and the error.

> in probability_matrix.masked_fill_(torch.tensor(special_tokens_mask, dtype=torch.bool), value=0.0)

Please upgrade your Pytorch version to 1.2.0+.

You're probably passing in a boolean tensor (true or false) instead of a byte tensor (0 or 1) for your attention mask.

Try changing
```
probability_matrix.masked_fill_(torch.tensor(special_tokens_mask, dtype=torch.bool), value=0.0)
```

to
```
probability_matrix.masked_fill_(torch.tensor(special_tokens_mask, dtype=torch.uint8), value=0.0)
```
Download model too slow, is there any way
Author: bigzhouj
Pull request: no

in run_lm_finetuning.py  
transformers.file_utils -   https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-cased-pytorch_model.bin not found in cache or force_download set to True, downloading to ....

Comments:
If your model download is too slow and fails, you can manually download it from our S3 using your browser, wget or cURL as an alternative method.

You can then point to a directory that has both the model weights (xxx-pytorch_model.bin) and the configuration file (xxx-config.json) instead of the checkpoint name as the argument for `run_lm_finetuning.py`.

The models on  s3  are downloaded by **botocore**. And can be accelerated using a proxy.  Detailed information can be found on [](https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html ).
Because It only supports **http** proxy now, other form of proxies like socks5 need to be converted to a http form.

OSError: Couldn't reach server at 'https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-cased-config.json' to download pretrained model configuration file. 

Can you open this [ https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-cased-config.json](url) in your browser ?

It can be opened in a browser

in run_lm_finetuning.py,
    probability_matrix.masked_fill_(torch.tensor(special_tokens_mask, dtype=torch.bool), value=0.0).
Why does dtyped equal torch.bool?
I have a difficulty here:
     Expected object of scalar type Byte but got scalar type Bool for argument #2 'mask'

Tipically, when you say _masked_ *, you want to use boolean values (0 for absence and 1 for presence). In this particular case (rows n.144-151), you are sampling some tokens in the in each sequence for **masked** language modeling. For this reason, the _probability_matrix_ variable is being set to boolean values. In fact, the first argument of the _masked_fill()_ method is a boolean Torch tensor (i.e. the boolean vector). You can read more info in the PyTorch docs [here](https://pytorch.org/docs/stable/tensors.html).

For what concern your issue, post the code for reproducibility and version of TensorFlow, PyTorch, Transformers.

> in run_lm_finetuning.py,
> probability_matrix.masked_fill_(torch.tensor(special_tokens_mask, dtype=torch.bool), value=0.0).
> Why does dtyped equal torch.bool?
> I have a difficulty here:
> Expected object of scalar type Byte but got scalar type Bool for argument #2 'mask'

@bigzhouj this is probably due to a Pytorch version error. I believe `bool` was introduced in pytorch v1.2.0. What is your Pytorch version?