noamaanMulla-03/transformers-issues huggingface dataset:
"This tokenizer does not make use of special tokens." warning
Author: weiguowilliam
Pull request: no

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

I just updated to the latest version transformers. Now when I use tokenizer to encode word, it always show the warning "This tokenizer does not make use of special tokens." 

Is there any way to hide that warning? Thank you.

Comments:
Same here. Is there any way to suppress this warning? I use `run_lm_finetuning.py` to finetune distilgpt2 and it outputs thousands "This tokenizer does not make use of special tokens.". It's so annoying :(

Here's how to suppress the warnings until this is fixed:

```py
import logging
logging.getLogger('transformers.tokenization_utils').setLevel(logging.ERROR)
```

> Here's how to suppress the warnings until this is fixed:
> 
> ```python
> import logging
> logging.getLogger('transformers.tokenization_utils').disabled = True
> ```

Thank you!

Is this fixed? If not, I think it should be open until it's been fixed.

This has been fixed on the master and in the latest release (2.2.1)

Hi, I use the latest release but I still have this problem.

@iedmrc 
I close it because the 'log' method works. I don't know whether it's a bug or not.

Hi @yeliu918, could you please show us what you obtain when running this script in your environment?

```py
from transformers import GPT2Tokenizer, __version__
print(__version__)

tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
print(tokenizer.encode("What does this output?"))
```

I am getting warning despite trying everything mentioned above....

```
import torch
from transformers import GPT2LMHeadModel, GPT2Tokenizer, __version__
import logging
tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
print(__version__) 
logging.getLogger('transformers.tokenization_utils').disabled = True
tokens_tensor = torch.tensor([tokenizer.encode("some example sentence")])
greedy_output = model.generate(tokens_tensor, max_length=60, num_beams=16)
```

Version 2.8.0

Setting `pad_token_id` to 50256 (first `eos_token_id`) to generate sequence
run_glue.py example doesn't work for distilbert models
Author: ereday
Pull request: no

# 🐛 Bug

## Information
Hi all, 

I am succesfully able to run the run_glue.py example with BERT, XLNet and other architectures. However, when I try distilbert I got the following error:
```

Traceback (most recent call last):
  File "run_glue.py", line 562, in <module>
    main()
  File "run_glue.py", line 510, in main
    train_dataset = load_and_cache_examples(args, args.task_name, tokenizer, evaluate=False)
  File "run_glue.py", line 373, in load_and_cache_examples
    all_token_type_ids = torch.tensor([f.token_type_ids for f in features], dtype=torch.long)
TypeError: an integer is required (got type NoneType)
```

Model I am using (Bert, XLNet ...):
distilbert (distilbert-base-cased)
Language I am using the model on (English, Chinese ...):
English
The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: GLUE/SST-2
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:
```
export GLUE_DIR=./glue/glue_data
export TASK_NAME=SST-2
CUDA_VISIBLE_DEVICES=2,3 python run_glue.py \
    --model_type DistilBERT \
    --model_name_or_path distilbert-base-cased \
    --task_name $TASK_NAME \
    --do_train \
    --do_eval \
    --data_dir $GLUE_DIR/$TASK_NAME \
    --max_seq_length 128 \
    --per_gpu_train_batch_size 32 \
    --learning_rate 2e-5 \
    --num_train_epochs 3.0 \
    --output_dir ./output/$TASK_NAME/
```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
```
     - `transformers` version: 2.8.0
- Platform: Linux-5.5.15-200.fc31.x86_64-x86_64-with-fedora-31-Thirty_One
- Python version: 3.6.5
- PyTorch version (GPU?): 1.4.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: YES
- Using distributed or parallel set-up in script?: NO
```


Comments:
I think I might have broken this on `master` when merging #3688 🤔

Hi @ereday could you please try from the `trainer` branch described in PR #3800?

Otherwise, hotfixing this in your code should be easy (just remove the `all_token_type_ids` line)
Fix bug in GLUE example for models that do not require token_type_ids
Author: douwekiela
Pull request: yes

If you try to run the `run_glue.py` example with e.g. roberta from a fresh install of the library, it errors out with the following error:

```
Traceback (most recent call last):
  File "examples/run_glue.py", line 564, in <module>
    main()
  File "examples/run_glue.py", line 512, in main
    train_dataset = load_and_cache_examples(args, args.task_name, tokenizer, evaluate=False)
  File "examples/run_glue.py", line 373, in load_and_cache_examples
    all_token_type_ids = torch.tensor([f.token_type_ids for f in features], dtype=torch.long)
TypeError: an integer is required (got type NoneType)
```

To reproduce, run e.g.

`python examples/run_glue.py --model_name_or_path roberta-base --task_name SST-2 --per_gpu_eval_batch_size=8 --per_gpu_train_batch_size=8 --data_dir ./glue_data/SST-2/ --output_dir ./blah --model_type roberta --do_train --do_eval --max_seq_length 128 --learning_rate 2e-5 --num_train_epochs 3.0`

The reason is obviously that roberta does not have segment ids so `token_type_ids` is set to null in the data loader, causing `torch.tensor` to freak out. There's probably a more elegant long-term solution for this, but it's easy to fix by just setting it to 0 instead of null for those models.

Comments:
Hi @douwekiela, thanks for the PR. This should be fixed soon in a more stable way in the soon-to-be-merged #3800 
Let us know if it works.

Should be fixed on master by #3800, please open a new issue if you encounter other problems.
ImportError: cannot import name 'MODEL_CLASSES' from 'run_glue' 
Author: ThomasSYT
Pull request: no

# 🐛 Bug

## Information
I try to run the latest versions of the examples and got the error message(I have installed the main README mentioned procedure from source):

Traceback (most recent call last):
  File "run_bertology.py", line 33, in <module>
    from run_glue import ALL_MODELS, MODEL_CLASSES, load_and_cache_examples, set_seed
ImportError: cannot import name 'MODEL_CLASSES' from 'run_glue' (/home/stud-yantao/Transformer/transformers/examples/run_glue.py)

Model I am using (Bert, XLNet ...): Bert

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
run_bertology.py


The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name) 
MNLI


## To reproduce

Steps to reproduce the behavior:

export TASK_NAME=mnli
python ./run_bertology.py --data_dir $GLUE_DIR/$TASK_NAME
--model_name bert-base-uncased
--task_name $TASK_NAME
--output_dir ./tmp/$TASK_NAME/
--try_masking

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

Traceback (most recent call last):
  File "run_bertology.py", line 33, in <module>
    from run_glue import ALL_MODELS, MODEL_CLASSES, load_and_cache_examples, set_seed
ImportError: cannot import name 'MODEL_CLASSES' from 'run_glue' (/home/stud-yantao/Transformer/transformers/examples/run_glue.py)


## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 2.8.0
- Platform:
- Python version:3.7
- PyTorch version (GPU?):
- Tensorflow version (GPU?):
- Using GPU in script?:yes
- Using distributed or parallel set-up in script?:


Comments:
Should be fixed on master, please share if it does/your results
Can not import DataCollatorForLanguageModeling
Author: parmarsuraj99
Pull request: no

# 🐛 Bug

## Information

Model I am using (ALBERT):

Language I am using the model on (Sanskrit, Hindi):

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. In Google Colab
2. ` !python /content/transformers/examples/run_language_modeling.py \
        --train_data_file /content/corpus/train/full.txt \
        --eval_data_file /content/corpus/valid/full_val.txt \
        --model_type albert-base-v2 \ `
3. This worked yesterdy, bbut the latest added `DataCollatorForLanguageModeling` can't be imported.

The error i am getting 

`2020-04-22 05:12:25.640328: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
Traceback (most recent call last):
  File "/content/transformers/examples/run_language_modeling.py", line 29, in <module>
    from transformers import (
ImportError: cannot import name 'DataCollatorForLanguageModeling'`

So, I checked if it can be imported directly. 

`from transformers import DataCollatorForLanguageModeling`
 ERROR
`from transformers import DataCollatorForLanguageModeling
ImportError: cannot import name 'DataCollatorForLanguageModeling' `
## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 2.8.0
- Platform: Colab
- Python version:  3.6.9
- PyTorch version (GPU?): 1.4.0
- Tensorflow version (GPU?): 2.2.0-rc3
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No


Comments:
I was installing from pip and it isn't updated yet. Building from source solved this.

I am trying to run this from source and still I am getting the same error! 
Please have a look at this [here](https://github.com/huggingface/transformers/issues/3896#issue-604681806)

It's because the pip package hasn't been updated. The script to train is changed fundamentally. so you can try building from scratch using 
`git clone https://github.com/huggingface/transformers
cd transformers
pip install .` 
or 
You can use old script of `run_language_modeling.py` from previous commits.
BART/T5 seq2seq example
Author: deepanwayx
Pull request: no
Labels: name: seq2seq

# 🚀 Feature request

Can we have a seq2seq example with training/fine-tuning and generation for BART/T5 models?

Comments:
We are hard at work on this! I'd estimate 6 weeks out.

Looking forward to this for the T5 model :)

@sshleifer any updates? 

The example doesn't seem to show training/fine-tuning, only evaluation of already fine-tuned models.

@sshleifer Hello, any updates for training/fine-tuning on text generation for T5 model ?

`summarization/bart/finetune.py` supports T5.
ImportError: cannot import name 'DataCollatorForLanguageModeling' in run_language_modeling.py
Author: ishaansharma
Pull request: no

# 🐛 Bug ImportError: cannot import name 'DataCollatorForLanguageModeling' in run_language_modeling.py
## Information

> Traceback (most recent call last):
>   File "transformers/examples/run_language_modeling.py", line 29, in <module>
>     from transformers import (
> ImportError: cannot import name 'DataCollatorForLanguageModeling'

Model I am using (Bert, XLNet ...): roberta, Trying to build the model from scratch using the tutorial. 

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [x] the official example scripts: (give details below)
transformers/examples/run_language_modeling.py

* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)
My own Data set

## To reproduce

Steps to reproduce the behavior:

Follow the tutorial [here](https://huggingface.co/blog/how-to-train)

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: transformers-2.8.0 sacremoses-0.0.41 sentencepiece-0.1.85 tokenizers-0.5.2
- Platform:
- Python version: 3.x
- PyTorch version (GPU?):1.4.0
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: None


Comments:
Duplicate of #3893 

Please install from source until we release a new version of the library on PyPI
Fast tokenizers don't properly tokenize special tokens
Author: dirkgr
Pull request: no
Labels: name: Core: Tokenization

Slow tokenizers:

```
>>> import transformers
>>> t_slow = transformers.AutoTokenizer.from_pretrained("roberta-base", use_fast=False)
>>> t_slow.encode_plus("A <mask> sentence.")
{'input_ids': [0, 83, 50264, 3645, 4, 2],
 'token_type_ids': [0, 0, 0, 0, 0, 0],
 'attention_mask': [1, 1, 1, 1, 1, 1]}

>>> t_slow.convert_ids_to_tokens([0, 83, 50264, 3645, 4, 2])
['
', 'ĠA', '<mask>', 'Ġsentence', '.', '
']
```

Fast tokenizers:
```
>>> import transformers
>>> t_fast = transformers.AutoTokenizer.from_pretrained("roberta-base", use_fast=True)
>>> t_fast.encode_plus("A <mask> sentence.")
{'input_ids': [0, 250, 1437, 50264, 3645, 4, 2],
 'token_type_ids': [0, 0, 0, 0, 0, 0, 0],
 'attention_mask': [1, 1, 1, 1, 1, 1, 1]}

>>> t_fast.convert_ids_to_tokens([0, 250, 1437, 50264, 3645, 4, 2])
['
', 'A', 'Ġ', '<mask>', 'Ġsentence', '.', '
'] 
```

Comments:
I think it's a duplicate of #2919. Btw, I should have told you when I saw you open the PR, [I opened a bunch of issues related to AllenNLP usage](https://github.com/huggingface/transformers/issues?utf8=%E2%9C%93&q=is%3Aissue+author%3Abryant1410). I think one that was closed it's not completely solved, but not sure.

It's not the same issue. This one is about special tokens in text form in the middle of your string. #2919 is about `[CLS]` and `[SEP]` being added to the beginning and end. Also, #2919 has already been fixed.

Ohh, :+1: 

This is now fixed on `master`:

```
>>> import transformers
>>> t_fast = transformers.AutoTokenizer.from_pretrained("roberta-base", use_fast=True)
>>> t_fast.encode_plus("A <mask> sentence.")
{'input_ids': [0, 83, 50264, 3645, 4, 2], 'attention_mask': [1, 1, 1, 1, 1, 1]}
>>> t_fast.convert_ids_to_tokens([0, 83, 50264, 3645, 4, 2])
['
', 'ĠA', '<mask>', 'Ġsentence', '.', '
']
```
Fast tokenizers fail when the input is just spaces
Author: dirkgr
Pull request: no
Labels: name: Core: Tokenization

Slow tokenizers:
```
>>> import transformers
>>> t = transformers.AutoTokenizer.from_pretrained("bert-base-cased", use_fast=False)
>>> t.encode_plus(" ", add_special_tokens=False)
{'input_ids': [], 'token_type_ids': []}
```

Fast tokenizers:
```
>>> import transformers
>>> t = transformers.AutoTokenizer.from_pretrained("bert-base-cased", use_fast=True, add_special_tokens=False)
>>> t.encode_plus(" ")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/dirkg/anaconda3/envs/allennlp/lib/python3.7/site-packages/transformers/tokenization_utils.py", line 1894, in encode_plus
    return {key: value[0] if isinstance(value[0], list) else value for key, value in batched_output.items()}
  File "/Users/dirkg/anaconda3/envs/allennlp/lib/python3.7/site-packages/transformers/tokenization_utils.py", line 1894, in <dictcomp>
    return {key: value[0] if isinstance(value[0], list) else value for key, value in batched_output.items()}
IndexError: list index out of range
```

The `add_special_tokens=False` bit is critical. Otherwise, there is no failure because the results aren't empty.

Comments:
This is now fixed on `master`:
```
>>> t = transformers.AutoTokenizer.from_pretrained("bert-base-cased", use_fast=True)
>>> t.encode_plus(" ", add_special_tokens=False)
{'input_ids': [], 'token_type_ids': [], 'attention_mask': []}
```
Also, `add_special_tokens` now works exactly the same for both slow and fast tokenizers. We give it with `encode`, `tokenize`, ... and not during initialization anymore.
Fast tokenizers ignore `add_special_tokens=False`
Author: bryant1410
Pull request: no
Labels: name: Core: Tokenization

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Bert

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

```python
from transformers import AutoTokenizer

pretrained_model_name = "bert-base-cased"

fast_tokenizer = AutoTokenizer.from_pretrained(pretrained_model_name)
slow_tokenizer = AutoTokenizer.from_pretrained(pretrained_model_name, use_fast=False)

text = "hello"

assert fast_tokenizer.encode(text, add_special_tokens=False) == slow_tokenizer.encode(text, add_special_tokens=False)
```

## Expected behavior

The fast tokenizers shouldn't add the special tokens if `add_special_tokens` is equal to `False`.

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 2.5.0
- Platform: Linux-4.15.0-76-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.4
- PyTorch version (GPU?): 1.4.0 (True)
- Tensorflow version (GPU?): 2.0.0 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

Comments:
This works now (but I can't close the issue).

I see now that it works with some changes:

```python
from transformers import AutoTokenizer

pretrained_model_name = "bert-base-cased"

fast_tokenizer = AutoTokenizer.from_pretrained(pretrained_model_name,
        add_special_tokens=False, use_fast=True)
slow_tokenizer = AutoTokenizer.from_pretrained(pretrained_model_name)

text = "hello"

assert fast_tokenizer.encode(text) == slow_tokenizer.encode(text, add_special_tokens=False)
```

However, I see `add_special_tokens` needs to be specified differently in the fast version (init) and in the slow version (encode). Can it be made more homogeneous? I'll leave this issue open for this because the fast version still ignores it in `encode` and there's this discrepancy (maybe the slow version can be changed then).

It also doesn't work for `roberta-base`.

In the last version, available on `master` for now, we actually changed this to match the slow version. So in all cases, `add_special_tokens` should be specified with `tokenize`, `encode` etc, and not during initialization.
Calculated offsets are wrong
Author: dirkgr
Pull request: no

This is on the latest `master` (from 2020-04-13):
```Python
import transformers
text = 'A, <mask> AllenNLP sentence.'
t = transformers.AutoTokenizer.from_pretrained("roberta-base", use_fast=True, add_special_tokens=True)
x2 = t.encode_plus(text, return_offsets_mapping=True)
print(repr(t.convert_ids_to_tokens(x2['input_ids']))
print(repr([text[start:end] for start, end in x2['offset_mapping']]))
```

This prints (with some manual alignment):
```
['
', 'ĠA', ',', '<mask>',   'ĠAllen', 'N', 'LP', 'Ġsentence', '.', '
']
['',    'A',  ',', ', <mask>', ' Alle',  'n', 'NL', 'P sentenc', 'e', '']
```

Comments:
It's probably normal since Roberta's tokenizers is a byte-level tokenizer which split words at byte-level (ie. possibly smaller than the character unit).

Cc @n1t0 

There is a bug indeed, the offsets shouldn't be shifted after the `<mask>` token. I should be able to fix this.
I'm not sure I'll be able to have the right offsets for the `<mask>` token though as this one is tricky.

This is now fixed on the latest `master`, with the output being
```
['
', 'ĠA', ',',  '<mask>', 'ĠAllen', 'N', 'LP', 'Ġsentence', '.', '
']
['',     'A', ',',  '<mask>',  'Allen', 'N', 'LP',  'sentence', '.', '']  
```
The spaces are not part of the offsets because the `trim_offsets` option is `True` by default.