noamaanMulla-03/transformers-issues huggingface dataset:
Tokenization Incorrect
Author: dhirajmadan1
Pull request: no

The tokenizer is not working correctly for me for e.g. [CLS] is gettig broken '[' , 'cl ', '##s', ']'
In [1]: import torch 
   ...: from pytorch_pretrained_bert import BertTokenizer, BertModel, BertForMas
   ...: kedLM 
   ...:  
   ...: # Load pre-trained model tokenizer (vocabulary) 
   ...: tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')          
Better speed can be achieved with apex installed from https://www.github.com/nvidia/apex.

In [2]: text = "[CLS] Who was Jim Henson ? [SEP] Jim Henson was a puppeteer [SEP
   ...: ]" 
   ...: tokenized_text = tokenizer.tokenize(text)                               

In [3]: tokenized_text                                                          
Out[3]: 
['[',
 'cl',
 '##s',
 ']',
 'who',
 'was',
 'jim',
 'henson',
 '?',
 '[',
 'sep',
 ']',
 'jim',
 'henson',
 'was',
 'a',
 'puppet',
 '##eer',
 '[',
 'sep',
 ']']


Comments:
Though we are not facing the same issue……

I followed the instruction from the readme, the `tokenized_text` is expected by assertion to be:
 `['[CLS]', 'who', 'was', 'jim', 'henson', '?', '[SEP]', 'jim', '[MASK]', 'was', 'a', 'puppet', '##eer', '[SEP]']`. 

However, the actual `tokenized_text` is:
`['[CLS]', 'who', 'was', 'jim', 'henson', '?', '[MASK]', 'jim', 'henson', 'was', 'a', 'puppet', '##eer', '[SEP]']`.

I believe this is because of `masked_index = 6` in the example code from the readme. If you let it to `masked_index = 8`, everything is perfect.

I opened a PR to fix this minor bug.

I think the tokenizer issue has been resolved in the latest version (0.5.0). 

Yes!

I just encountered the same issue as @dhirajmadan1 with `transformers==2.2.1`. Is this expected somehow?

I am following the quickstart guide: https://huggingface.co/transformers/quickstart.html

```
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

# Run an example text through this:
text = "[CLS] Who was Jim Henson ? [SEP] Jim Henson was a puppeteer [SEP]"
tokenized_text = tokenizer.tokenize(text)

masked_index = 8
tokenized_text[masked_index] = '[MASK]'
predicted_tokenized_sentence = ['[CLS]', 'who', 'was', 'jim', 'henson', '?', '[SEP]', 'jim', '[MASK]', 'was', 'a', 'puppet', '##eer', '[SEP]']
        
```

What is your issue @yenicelik? Do you mind opening a new issue with your problem?

ah, my apologies: https://github.com/huggingface/transformers/issues/2047

apparently a PR is on the way!
Error when calculate tokens_id and Mask LM
Author: monanahe
Pull request: no
Labels: name: wontfix

## 🐛 Bug

<!-- Important information -->

Model I am using (DistilBert):

Language I am using the model on (English):

The problem arise when using: Distiller.prepare_batch( ) 
Error when token_ids is masked by mask LM matrix
*  the official example scripts:
_token_ids_real = token_ids[pred_mask] 
*  my own modified scripts: 
 _token_ids_real=torch.mul(token_ids, pred_mask)

The tasks I am working on is:
* [GLUE ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details)

## To Reproduce

Steps to reproduce the behavior:
1. pred_mask is matrix with 0,1. 
Operation token_ids[pred_mask] seems to make some same matrix, instead of masking token_ids

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->

## Environment

* OS: Win10
* Python version: 3.6
* PyTorch version: 1.1
* PyTorch Transformers version (or branch): 2.0/0
* Using GPU ? Yes
* Distributed of parallel setup ? No
* Any other relevant information: 

## Additional context

<!-- Add any other context about the problem here. -->

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
UnboundLocalError: local variable 'extended_attention_mask' referenced before assignment
Author: aclifton314
Pull request: no

## Finetuning Setup
* Model: roberta-base
* Language: english
* OS: Ubuntu 18.04.3
* Python version: 3.7.3
* PyTorch version: 1.3.1+cpu
* PyTorch Transformers version (or branch): 2.2.0
* Using GPU ? No
* Distributed of parallel setup ? No
* Script inputs:
```
python run_lm_finetuning.py \
--output_dir=$OUTPUT_DIR \
--model_type=roberta \
--model_name_or_path=roberta_base \
--do_train \
--train_data_file=$TRAIN_FILE \
--do_eval \
--eval_data_file=$TEST_FILE \
--mlm \
--no_cuda
```

## ❓ Questions & Help
I finetuned roberta on some domain specific data I have and was trying to follow the example in the Quick Tour section for getting the output, however I get the following error:
```python
Traceback (most recent call last):
  File "/path/to/code/roberta_compare.py", line 26, in <module>
    last_hidden_states = model(input_ids)
  File "/usr/local/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/anaconda3/lib/python3.7/site-packages/transformers/modeling_roberta.py", line 246, in forward
    inputs_embeds=inputs_embeds)
  File "/usr/local/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/anaconda3/lib/python3.7/site-packages/transformers/modeling_bert.py", line 688, in forward
    extended_attention_mask = extended_attention_mask.to(dtype=next(self.parameters()).dtype)  # fp16 compatibility
UnboundLocalError: local variable 'extended_attention_mask' referenced before assignment
```
Here is my code:
```python
import torch
from transformers import RobertaTokenizer, RobertaForMaskedLM

model_class = RobertaForMaskedLM
model = model_class.from_pretrained('/path/to/models/roberta_finetuned/model')
tokenizer_class = RobertaTokenizer
tokenizer = tokenizer_class.from_pretrained('/path/to/models/roberta_finetuned/model')
tmp = tokenizer.encode('A sentence to encode with roberta.', add_special_tokens=True)
input_ids = torch.tensor(tmp)

with torch.no_grad():
    last_hidden_states = model(input_ids)[0]
```

Any thoughts on what I might be messing up?

Comments:
You forgot to add the batch size.
You can either
- do `input_ids = tokenizer.encode('A sentence to encode with roberta.', add_special_tokens=True, return_tensors='pt')`
- or `input_ids = torch.tensor([tokenizer.encode('A sentence to encode with roberta.')])`

But for a specific reason, the current failure message is really not clear. Improving that in #2068

That fixes it! Thank you for the response! Closing the issue.
How can we view different versions of documentation?
Author: drydenb
Pull request: no

## ❓ Questions & Help

How can I select a specific version of transformers in the documentation located here: https://huggingface.co/transformers/index.html?


Comments:
We're in the process of building better versioned documentation with easier links to follow, but at the moment the different versions are accessible in the [README](https://github.com/huggingface/transformers#state-of-the-art-natural-language-processing-for-tensorflow-20-and-pytorch), right before the `installation` section.

Great, thank you!
Check the validity of download links
Author: rlouf
Pull request: yes

We would like to make sure regularly that every download link in the codebase works. The best way to do this is to check automatically with the CI; this also prevents us from merging code with broken links.

This PR adds a small script that:
- Lists all source code files
- Extract links with a regexp
- Perform HEAD requests to check the validity of each links
- Returns an error if at least one link is broken, along with the list of all broken links.

I also add a Circle CI workflow `repository-consistency` with a small machine that runs this script. It could be used to enforce things such as coding styles etc in the future.

Comments:
It works so well that the CI failed because of a broken link :)
TransfoXLCorpus requires pytorch to tokenize files
Author: tomweingarten
Pull request: no
Labels: name: wontfix

## 🐛 Bug

The current TransfoXLCorpus code requires pytorch, and fails if it is not installed.

Model I am using (Bert, XLNet....): Transformer-XL

Language I am using the model on (English, Chinese....): Other

The problem arise when using:
* [ X ] my own modified scripts: I'm using a very simple script to read in text files, see code below

The tasks I am working on is:
* [ X ] my own task or dataset: I am attempting to build a corpus from my own dataset of long text sentences.

## To Reproduce

Steps to reproduce the behavior:

corpus = TransfoXLCorpus(lower_case=True, delimiter=" ")
corpus.build_corpus(EXAMPLE_DIR, "text8")

Traceback (most recent call last):
  File "build_xl_corpus.py", line 26, in <module>
    corpus.build_corpus(EXAMPLE_DIR, "text8")
  File "/home/tom/.local/lib/python3.7/site-packages/transformers/tokenization_transfo_xl.py", line 521, in build_corpus
    os.path.join(path, 'train.txt'), ordered=True, add_eos=False)
  File "/home/tom/.local/lib/python3.7/site-packages/transformers/tokenization_transfo_xl.py", line 187, in encode_file
    encoded.append(self.convert_to_tensor(symbols))
  File "/home/tom/.local/lib/python3.7/site-packages/transformers/tokenization_transfo_xl.py", line 246, in convert_to_tensor
    return torch.LongTensor(self.convert_tokens_to_ids(symbols))
NameError: name 'torch' is not defined

## Expected behavior

I did not expect this behavior to require pytorch

## Environment

* OS: Ubuntu
* Python version:
* PyTorch version: None
* PyTorch Transformers version (or branch): 2.0.0
* Using GPU ? Yes
* Distributed of parallel setup ? No
* Any other relevant information:


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Finetuning distilbert-base-uncased
Author: aah39
Pull request: no

## ❓ Questions & Help
When trying to finetune distilbert-base-uncased on my own dataset  
I receive the following error message:
ERROR - pytorch_transformers.tokenization_utils -   Model name 'distilbert-base-uncased' was not found in model name list (bert-base-uncased,bert-large-uncased, bert-base-cased, bert-large-cased, bert-base-multilingual-uncased, bert-base-multilingual-cased, bert-base-chinese, bert-base-german-cased, bert-large-uncased-whole-word-masking, bert-large-cased-whole-word-masking, bert-large-uncased-whole-word-masking-finetuned-squad, bert-large-cased-whole-word-masking-finetuned-squad, bert-base-cased-finetuned-mrpc). 

How do I add distilbert-base-uncased to the model name list (I assume I will somehow have to modify PYTORCH-TRANSFORMERS_CACHE)




code used:
python simple_lm_finetuning.py \
--train_corpus dataset.txt \
--bert_model distilbert-base-uncased \
--do_lower_case \
--output_dir finetuned_lm/ \
--do_train

Comments:
Hi! Yes we're in the process of adding DistilBERT to the examples. Until then, you can simply edit the script to add it.

Please note that the `simple_lm_finetuning` script is now deprecated in favor of `run_lm_finetuning`.

Hi there @aah39 , I came across the same issue in run_glue.py when I tried to fine tune distilbert_base_uncased. Later I found the fix was easy: just change the model_type to be distilbert when running the script (I saw run_lm_finetuning has this input parameter as well)
After this change, when running the script, it will load the cache from DISTILBERT_PRETRAINED_CONFIG_ARCHIVE_MAP instead of BERT_PRETRAINED_CONFIG_ARCHIVE_MAP

I think this is not an issue anymore since it's been fixed in the https://github.com/huggingface/transformers/commit/88368c2a16d26bc2d00dc28f79196c81373d3a71
The generation script could fail when there's a double space in the prompt
Author: cloudygoose
Pull request: no

## 🚀 Feature

Hey, thanks for everything,
The generation script could fail when there's a double space in the prompt, e.g. " I  go to"
![image](https://user-images.githubusercontent.com/1544039/70262743-e4bc4900-1762-11ea-9041-9bee082a0054.png)

I know it's not important, but it would be good if the tokenize is more "robust"

Comments:
Hi! Could you specify the command you used to launch `run_generation` as well as the versions in your environment? Pyton, pytorch, transformers? Thanks.

`python scripts_htx/run_generation.py  --model_type ctrl --model_name ctrl  --repetition 1.2`
python=3.7.3
torch=1.3.0
transformers=2.2.1

But I guess this issue is not related to the versions....

This is actually the same as #1920 
Now fixed on master (will be in the next release).
Tried to import  TFBertForPreTraining in google colab
Author: mandavachetana
Pull request: no
Labels: name: wontfix

Tried to import TFBertForPreTraining and received an error

from transformers import BertTokenizer, TFBertForPreTraining
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-24-91f8709e090f> in <module>()
----> 1 from transformers import BertTokenizer, TFBertForPreTraining

ImportError: cannot import name 'TFBertForPreTraining'

---------------------------------------------------------------------------
NOTE: If your import is failing due to a missing package, you can
manually install dependencies using either !pip or !apt.

To view examples of installing some common dependencies, click the
"Open Examples" button below.
---------------------------------------------------------------------------

Comments:
Hey @mandavachetana its not just a google colab thing. Take a look here #1375 You need to make sure you are using tensorflow 2.0 and it should work.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
mask_tokens sometimes masks special tokens 
Author: kaushaltrivedi
Pull request: no
Labels: name: wontfix

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): RoBERTa

Language I am using the model on (English, Chinese....):

The problem arise when using:
* [ ] the official example scripts: (give details) run_lm_finetuning
* [ ] my own modified scripts: (give details)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details)

## To Reproduce

Steps to reproduce the behavior:

1.
2.
3.


the mask_token function in run_lm_finetuning script sometimes masks the special tokens.  This leads roberta model to throw a warning message: 
A sequence with no special tokens has been passed to the RoBERTa model. "
                           "This model requires special tokens in order to work. "
                           "Please specify add_special_tokens=True in your encoding


I would prevent the first and last tokens being masked by adding the below lines immediately after masked_indices are calculated.

masked_indices[:, 0] = False
masked_indices[:, -1] = False


<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->

## Environment

* OS:
* Python version:
* PyTorch version:
* PyTorch Transformers version (or branch):
* Using GPU ?
* Distributed of parallel setup ?
* Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->

Comments:
Hi, thank you for the bug report. Indeed, this does seem problematic. I'm looking into it.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
[cli] Uploads: add progress bar
Author: julien-c
Pull request: yes

see https://github.com/huggingface/transformers/pull/2044#discussion_r354057827 for context

There might be a more pythonic way (to do a "simple" method overriding) but I couldn't find it.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2078?src=pr&el=h1) Report
> Merging [#2078](https://codecov.io/gh/huggingface/transformers/pull/2078?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/35ff345fc9df9e777b27903f11fa213e4052595b?src=pr&el=desc) will **increase** coverage by `0.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2078/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2078?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #2078      +/-   ##
==========================================
+ Coverage   83.16%   83.18%   +0.01%     
==========================================
  Files         109      109              
  Lines       15858    15874      +16     
==========================================
+ Hits        13188    13204      +16     
  Misses       2670     2670
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2078?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [transformers/hf\_api.py](https://codecov.io/gh/huggingface/transformers/pull/2078/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL2hmX2FwaS5weQ==) | `97.5% <100%> (+0.62%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2078?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2078?src=pr&el=footer). Last update [35ff345...5543617](https://codecov.io/gh/huggingface/transformers/pull/2078?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Fixing camembert tokenization
Author: thomwolf
Pull request: yes

The original fairseq implmentation of Camembert has a bunch of duplicate tokens in the dictionary, in particular there are two `<unk>` tokens but only the index of the first `<unk>` should be used:
```
import torch
camembert = torch.hub.load('pytorch/fairseq', 'camembert.v0')
list(camembert.task.source_dictionary[i] for i in range(10))
>>> ['
', '<pad>', '
', '<unk>', '<unk>', '
', '
', ',', '▁de', '.']
```

This PR updates Camembert tokenizer to fix this behavior and as a consequence fixes #2019 and #2020

Comments:
Merging now to fix the xlnet test issue on master at the same time.

Also cc'ing @louismartin on this.

Thanks for fixing that.
This comes from a problem in fairseq where special tokens are added twice when using SentencePiece.
Cross-referencing the fairseq issue: [https://github.com/pytorch/fairseq/issues/1309](https://github.com/pytorch/fairseq/issues/1309)
Fix loading of question answering bert from tf weights.
Author: Talmaj
Pull request: yes
Labels: name: wontfix

I've got an attribute error when loading pretrained tf weights for question answering (bert) in `load_tf_weights_in_bert` at:
```
elif l[0] == 'squad':
     pointer = getattr(pointer, 'classifier')
```
since `BertForQuestionAnswering` does not have a 'classifier' attribute but `qa_outputs`. I've added a try except, which resolves the error.  

Comments:
# [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1197?src=pr&el=h1) Report
> Merging [#1197](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1197?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/pytorch-transformers/commit/89fd3450a61b5efd76d2524df2454e0a0e4ca070?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1197/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1197?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #1197   +/-   ##
=======================================
  Coverage   80.83%   80.83%           
=======================================
  Files          46       46           
  Lines        7878     7878           
=======================================
  Hits         6368     6368           
  Misses       1510     1510
```


| [Impacted Files](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1197?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pytorch\_transformers/modeling\_bert.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1197/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvbW9kZWxpbmdfYmVydC5weQ==) | `88.03% <100%> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1197?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1197?src=pr&el=footer). Last update [89fd345...d6fb182](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1197?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
[WIP] Refactor Tokenizers creation to support in-memory initialization
Author: mfuntowicz
Pull request: yes
Labels: name: wontfix

As pointed out in #916 currently tokenizers ask for the path from where they'll load the required vocabulary files.

This PR allows tokenizers to take their vocab from data living in-memory and cold-storage.

Implementations details:

- All tokenizer now have a specific ${TokenizerName}Vocab dataclass holding all the required information to run the model.
- All ${TokenizerName}Vocab dataclass provide a from_pretrained method in charge of reading necessary files 
- All tokenizer now take as first argument vocabs which has to ${TokenizerName}Vocab instance. 
- All model now have a static member vocab_class which points to the desired ${TokenizerName}Vocab data class 

- Some ${TokenizerName}Vocab.from_pretrained share loading routines and thus code is currently duplicated across all of them. It might be possible to refactor to use a generic method that handles such loading. 

- [x] Bert
- [x] Transformer XL
- [x] GPT
- [x] GPT-2
- [x] XLNet
- [x] XLM
- [x] RoBERTa
- [x] DistilBERT

Comments:
@thomwolf cc @honnibal Drafting this PR to have dedicated space for discussions.

@thomwolf @honnibal Can you have a look plz :) ?

Ok, I went through this PR and it looks nice.
Great job @mfuntowicz.
No problem for the slight code duplication in the tokenizer loading classes, as you've noticed, the repo's philosophy is rather pragmatic and we only add abstractions when they are needed for easier code maintenance and added functionalities.
Thanks for following the general organization of the repo as well.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Fix bug in run_openai_gpt.py file.
Author: liyucheng09
Pull request: yes
Labels: name: wontfix

Add a example for add special tokens to OpenAIGPTTokenizer, and resize the embedding layer of OpenAIGPTModel.

Comments:
run_gpt2.py file has been test on ROCStories corpus, it runs fine and return auccuracy of 76%, lower than GPT1.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.