REALM

Feature request

Request for a new feature

# 🌟 New model addition

REALM is from some of the authors of BERT (I like to think of it as the next BERT :) ) that have found a way to incorporate world knowledge (from Wikipedia) into the model.

They do this by having the concept of a retriever module that retrieves information from wikipedia articles.

<!-- Important information -->

## Open source status

Code not released at the moment but will probably be released by Google soon I'd imagine.

https://arxiv.org/abs/2002.08909


How to load BertforSequenceClassification  models weights into BertforTokenClassification model?

Usage

General questions about the library

Initially, I have a fine-tuned BERT base cased model using a text classification dataset and I have used BertforSequenceClassification class for this. Now I want to use this fine-tuned BERT model weights for Named Entity Recognition and I have to use BertforTokenClassification class for this. I'm unable to figure out how to load the fine-tuned BERT model weights into the new model created using BertforTokenClassification.  @thomwolf

Thanks in advance.......................


CUDA error: CUBLAS_STATUS_ALLOC_FAILED When running language modeling using bert-base-cased

Ex: LM (Finetuning)

Related to language modeling fine-tuning

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Bert

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: (give the name) Language modeling
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Create a shell file under example folder called run_lm.sh
2. bash run_lm.sh

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

```
export TRAIN_FILE=../../data/wikitext-2-raw/wiki.train.raw
export TEST_FILE=../../data/wikitext-2-raw/wiki.test.raw

python run_language_modeling.py \
    --output_dir=output \
    --model_type=bert \
    --model_name_or_path=bert-base-cased \
    --do_train \
    --train_data_file=$TRAIN_FILE \
    --do_eval \
    --eval_data_file=$TEST_FILE \
    --mlm
```

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

It would arouse:
```
RuntimeError: CUDA error: CUBLAS_STATUS_ALLOC_FAILED when calling `cublasCreate(handle)`
```

However, if you only change ```--model_name_or_path=bert-base-cased \``` to ```--model_name_or_path=bert-base-uncased \```, it will work well. Hence, I'm not sure what went wrong.

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.5.1
- Platform: Linux-5.3.0-28-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.6
- PyTorch version (GPU?): 1.4.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No


TypeError when using Feature Extraction Pipeline with XLM roberta

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): XLM-R

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. after importing the required libraries, run the following snipped of code:
```
xlmr_model = XLMRobertaForSequenceClassification.from_pretrained("xlm-roberta-base")
xlmr_tokenizer = XLMRobertaTokenizer.from_pretrained("xlm-roberta-base")
nlp = pipeline(task ="feature-extraction", model = xlmr_model, tokenizer=xlmr_tokenizer, framework="tf")
features = nlp('We are very happy to include pipeline into the transformers repository.')
print(features)
```
2. You should get the following error: result = self.forward(*input, **kwargs)
**TypeError: forward() got an unexpected keyword argument 'training'**

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
I would expect this code to run without this error.

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.6.1
- Platform: Ubuntu 16.04.6 LTS
- Python version: 3.6.1
- PyTorch version (GPU?): No GPU
- Tensorflow version (GPU?): No GPU
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


Finetuning GPT-2

It looks like there used to be a script `run_lm_finetuning.py` that has been replaced by `run_language_modeling.py`. It's unclear to me how to use this script to run finetuning. I want to finetune GPT-2 on a variety of downstream tasks, and would love some help!


[model_cards]: use MIT license for all dbmdz models

model card

Related to pretrained model cards

Hi,

this PR adds MIT license tag for all dbmdz models 🤗


cased -> uncased for example cmdline consistency


which iterator to use for different hugging face transformer models for solving multiple choice questions?

Discussion

Discussion on a topic (keep it focused or open a new issue though)

Help wanted

Extra attention is needed, help appreciated

Hello,

I have used Hugging Face GPT-2 models to do Natural Language Processing.
When I used the GPT-2 model, I used the ```BPTTIterator``` from TorchText to pre-process my text data, since GPT-2 essentially performs regular language modelling (next token prediction).

I am wondering, when I use GPT-2, BERT and XLNet for **multiple-choice solving**:

1. What type of iterator should I use for multiple-choice question solving? If there is no specific iterator that can accommodate multiple-choice question solving, how should I pre-process my text (multiple-choice questions) before feeding them into the different transformer models?

2. if hugging face transformer models (BERT, XLNet, GPT-2)  use special tokens to separate questions from multiple-choice options, what are those special tokens? I want to make the use of the pre-trained models rather than training a new model with new special tokens on my own.

3. Is the format of multiple-choice questions that can be processed with, for example, BERT different than the multiple-choice question format that can be processed by GPT-2? or can each of the transformer models process any type of multiple-choice questions? (for example, BERT could only be used to solve fill-in-the-blank multiple-choice questions, whereas multiple-choice question for GPT-2 does not have to have the fill-in-the-blank format?)

Thank you,


missing import in BartForConditionalGeneration example

Need more information

Further information is requested

Add

`from transformers import AutoTokenizer, AutoModelWithLMHead`

to Bart for Conditional Generation example in

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...):

Language I am using the model on (English, Chinese ...):

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1.
2.
3.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:
- Platform:
- Python version:
- PyTorch version (GPU?):
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?:


[bart-tiny-random] Put a 5MB model on S3 to allow faster examples test

- Vocab size is the same to avoid requiring a new tokenizer.
- this allows examples that parametrize `model_name`, like `evaluate_cnn.py`, to run much quicker: `summarization/bart/test_bart_examples.py` runs in 6 seconds vs 22 + download time previously.
- Would be happy to do this for more models.
- Will update `run_sum.py` if this idea is OK with people.
- this also makes local debugging easier. You can easily pull down a tiny model without making a config.


Create model card

model card

Related to pretrained model cards


setup.py succeeds, then can't import transformers

```bash
pip install -e .
```
works succesfully and makes `transformers.egg-info/`

Then,
```python
import transformers
```
fails with

```
ModuleNotFoundError: No module named 'transformers'
```

env
```
- `transformers` version: 2.6.0
- Platform: Darwin-19.0.0-x86_64-i386-64bit
- Python version: 3.7.5
- PyTorch version (GPU?): 1.4.0 (False)
- Tensorflow version (GPU?): 2.1.0 (False)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>
```


Fix circle ci flaky fail of wmt example

Weird bug which might be fixed by forcing bleu scorer.

Test pass locally, but seem to fail on circle ci.


Sphinx build for documentation fails when tensorflow is installed

## Information

I am trying to build the documentation in `docs/` using `sphinx-build` so that I can add it to [Dash](https://kapeli.com/dash) as one of the docsets. However, I get an assertion error with sphinx-build when I run `make html` in the `docs/` folder *only when tensorflow is installed*. The build works without tensorflow installed, but the tensorflow methods and classes are emtpy in the generated documentation - only the pytorch ones have documentation in the resulting HTML.

## To reproduce

Steps to reproduce the behavior:

1. Create a conda environment, and install pytorch, tensorflow and transformers using the official methods.
2. Run `pip install -e ".[docs]` in source directory, before running `make html` in `docs` folder. You will get the following error (full stacktrace given later):
```
Exception occurred:
  File "/Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/sphinx/util/docfields.py", line 260, in transform
    assert len(field) == 2
AssertionError
```
Full stacktrace:
```
# Sphinx version: 2.4.4
# Python version: 3.6.10 (CPython)
# Docutils version: 0.16 release
# Jinja2 version: 2.11.1
# Last messages:
#   reading sources... [ 13%] glossary
#
#   reading sources... [ 16%] index
#
#   reading sources... [ 18%] installation
#
#   reading sources... [ 21%] main_classes/configuration
#
#   reading sources... [ 24%] main_classes/model
#
# Loaded extensions:
#   sphinx.ext.mathjax (2.4.4) from /Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/sphinx/ext/mathjax.py
#   sphinxcontrib.applehelp (1.0.2) from /Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/sphinxcontrib/applehelp/__init__.py
#   sphinxcontrib.devhelp (1.0.2) from /Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/sphinxcontrib/devhelp/__init__.py
#   sphinxcontrib.htmlhelp (1.0.3) from /Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/sphinxcontrib/htmlhelp/__init__.py
#   sphinxcontrib.serializinghtml (1.1.4) from /Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/sphinxcontrib/serializinghtml/__init__.py
#   sphinxcontrib.qthelp (1.0.3) from /Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/sphinxcontrib/qthelp/__init__.py
#   alabaster (0.7.12) from /Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/alabaster/__init__.py
#   sphinx.ext.autodoc.type_comment (2.4.4) from /Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/sphinx/ext/autodoc/type_comment.py
#   sphinx.ext.autodoc (2.4.4) from /Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/sphinx/ext/autodoc/__init__.py
#   sphinx.ext.coverage (2.4.4) from /Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/sphinx/ext/coverage.py
#   sphinx.ext.napoleon (2.4.4) from /Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/sphinx/ext/napoleon/__init__.py
#   recommonmark (0.6.0) from /Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/recommonmark/__init__.py
#   sphinx.ext.viewcode (2.4.4) from /Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/sphinx/ext/viewcode.py
#   sphinx_markdown_tables (<module 'sphinx_markdown_tables.__version__' from '/Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/sphinx_markdown_tables/__version__.py'>) from /Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/sphinx_markdown_tables/__init__.py
Traceback (most recent call last):
  File "/Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/sphinx/cmd/build.py", line 276, in build_main
    app.build(args.force_all, filenames)
  File "/Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/sphinx/application.py", line 349, in build
    self.builder.build_update()
  File "/Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/sphinx/builders/__init__.py", line 299, in build_update
    len(to_build))
  File "/Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/sphinx/builders/__init__.py", line 311, in build
    updated_docnames = set(self.read())
  File "/Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/sphinx/builders/__init__.py", line 418, in read
    self._read_serial(docnames)
  File "/Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/sphinx/builders/__init__.py", line 439, in _read_serial
    self.read_doc(docname)
  File "/Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/sphinx/builders/__init__.py", line 479, in read_doc
    doctree = read_doc(self.app, self.env, self.env.doc2path(docname))
  File "/Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/sphinx/io.py", line 316, in read_doc
    pub.publish()
  File "/Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/docutils/core.py", line 218, in publish
    self.settings)
  File "/Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/sphinx/io.py", line 130, in read
    self.parse()
  File "/Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/docutils/readers/__init__.py", line 77, in parse
    self.parser.parse(self.input, document)
  File "/Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/sphinx/parsers.py", line 93, in parse
    self.statemachine.run(inputlines, document, inliner=self.inliner)
  File "/Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/docutils/parsers/rst/states.py", line 171, in run
    input_source=document['source'])
  File "/Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/docutils/statemachine.py", line 242, in run
    context, state, transitions)
  File "/Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/docutils/statemachine.py", line 459, in check_line
    return method(match, context, next_state)
  File "/Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/docutils/parsers/rst/states.py", line 2769, in underline
    self.section(title, source, style, lineno - 1, messages)
  File "/Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/docutils/parsers/rst/states.py", line 327, in section
    self.new_subsection(title, lineno, messages)
  File "/Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/docutils/parsers/rst/states.py", line 395, in new_subsection
    node=section_node, match_titles=True)
  File "/Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/docutils/parsers/rst/states.py", line 282, in nested_parse
    node=node, match_titles=match_titles)
  File "/Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/docutils/parsers/rst/states.py", line 196, in run
    results = StateMachineWS.run(self, input_lines, input_offset)
  File "/Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/docutils/statemachine.py", line 242, in run
    context, state, transitions)
  File "/Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/docutils/statemachine.py", line 459, in check_line
    return method(match, context, next_state)
  File "/Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/docutils/parsers/rst/states.py", line 2769, in underline
    self.section(title, source, style, lineno - 1, messages)
  File "/Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/docutils/parsers/rst/states.py", line 327, in section
    self.new_subsection(title, lineno, messages)
  File "/Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/docutils/parsers/rst/states.py", line 395, in new_subsection
    node=section_node, match_titles=True)
  File "/Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/docutils/parsers/rst/states.py", line 282, in nested_parse
    node=node, match_titles=match_titles)
  File "/Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/docutils/parsers/rst/states.py", line 196, in run
    results = StateMachineWS.run(self, input_lines, input_offset)
  File "/Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/docutils/statemachine.py", line 242, in run
    context, state, transitions)
  File "/Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/docutils/statemachine.py", line 459, in check_line
    return method(match, context, next_state)
  File "/Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/docutils/parsers/rst/states.py", line 2342, in explicit_markup
    nodelist, blank_finish = self.explicit_construct(match)
  File "/Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/docutils/parsers/rst/states.py", line 2354, in explicit_construct
    return method(self, expmatch)
  File "/Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/docutils/parsers/rst/states.py", line 2097, in directive
    directive_class, match, type_name, option_presets)
  File "/Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/docutils/parsers/rst/states.py", line 2146, in run_directive
    result = directive_instance.run()
  File "/Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/sphinx/ext/autodoc/directive.py", line 157, in run
    result = parse_generated_content(self.state, params.result, documenter)
  File "/Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/sphinx/ext/autodoc/directive.py", line 104, in parse_generated_content
    state.nested_parse(content, 0, node)
  File "/Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/docutils/parsers/rst/states.py", line 282, in nested_parse
    node=node, match_titles=match_titles)
  File "/Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/docutils/parsers/rst/states.py", line 196, in run
    results = StateMachineWS.run(self, input_lines, input_offset)
  File "/Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/docutils/statemachine.py", line 242, in run
    context, state, transitions)
  File "/Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/docutils/statemachine.py", line 459, in check_line
    return method(match, context, next_state)
  File "/Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/docutils/parsers/rst/states.py", line 2342, in explicit_markup
    nodelist, blank_finish = self.explicit_construct(match)
  File "/Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/docutils/parsers/rst/states.py", line 2354, in explicit_construct
    return method(self, expmatch)
  File "/Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/docutils/parsers/rst/states.py", line 2097, in directive
    directive_class, match, type_name, option_presets)
  File "/Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/docutils/parsers/rst/states.py", line 2146, in run_directive
    result = directive_instance.run()
  File "/Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/sphinx/domains/__init__.py", line 265, in run
    return super().run()
  File "/Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/sphinx/directives/__init__.py", line 195, in run
    self.state.nested_parse(self.content, self.content_offset, contentnode)
  File "/Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/docutils/parsers/rst/states.py", line 282, in nested_parse
    node=node, match_titles=match_titles)
  File "/Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/docutils/parsers/rst/states.py", line 196, in run
    results = StateMachineWS.run(self, input_lines, input_offset)
  File "/Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/docutils/statemachine.py", line 242, in run
    context, state, transitions)
  File "/Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/docutils/statemachine.py", line 459, in check_line
    return method(match, context, next_state)
  File "/Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/docutils/parsers/rst/states.py", line 2344, in explicit_markup
    self.explicit_list(blank_finish)
  File "/Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/docutils/parsers/rst/states.py", line 2374, in explicit_list
    match_titles=self.state_machine.match_titles)
  File "/Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/docutils/parsers/rst/states.py", line 319, in nested_list_parse
    node=node, match_titles=match_titles)
  File "/Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/docutils/parsers/rst/states.py", line 196, in run
    results = StateMachineWS.run(self, input_lines, input_offset)
  File "/Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/docutils/statemachine.py", line 242, in run
    context, state, transitions)
  File "/Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/docutils/statemachine.py", line 459, in check_line
    return method(match, context, next_state)
  File "/Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/docutils/parsers/rst/states.py", line 2647, in explicit_markup
    nodelist, blank_finish = self.explicit_construct(match)
  File "/Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/docutils/parsers/rst/states.py", line 2354, in explicit_construct
    return method(self, expmatch)
  File "/Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/docutils/parsers/rst/states.py", line 2097, in directive
    directive_class, match, type_name, option_presets)
  File "/Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/docutils/parsers/rst/states.py", line 2146, in run_directive
    result = directive_instance.run()
  File "/Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/sphinx/domains/__init__.py", line 265, in run
    return super().run()
  File "/Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/sphinx/directives/__init__.py", line 198, in run
    DocFieldTransformer(self).transform_all(contentnode)
  File "/Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/sphinx/util/docfields.py", line 248, in transform_all
    self.transform(child)
  File "/Users/venkat/opt/miniconda3/envs/tf-pt/lib/python3.6/site-packages/sphinx/util/docfields.py", line 260, in transform
    assert len(field) == 2
AssertionError
```
3. Uninstall tensorflow. Now when running `make html`, it does fininsh building, albeit with a bunch of warnings of the following form: `AttributeError: module 'transformers' has no attribute 'TFCamembertForMaskedLM'` -- for every `TFmethod`.

## Expected behavior

`make html` should build with no errors.

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.6.0
- Platform: macOS 10.15.4
- Python version: 3.6.10
- PyTorch version (GPU?): 1.4.0 (No)
- Tensorflow version (GPU?): 2.1.0 (No)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No
- sphinx version: 2.4.4


Tests for more examples

# Add tests for more of the examples/

We need to have more testing code for examples. This is particularly true with NER which recently had a tokenizer issue.

(Self-assigning)


Correct output shape for Bert NSP models in docs

In the docs, for Bert models that have the NSP head, the output shape for one of the params returned by the forward method, `seq_relationship_scores`, is incorrect. Fixing it.


Do I need to pad non-fixed examples or does run_language_modeling.py already takes care of that?

I use a custom TextDataset in [examples/run_language_modeling.py](https://github.com/huggingface/transformers/blob/master/examples/run_language_modeling.py). I can't figure if I need to pad the examples to `bucket_size` myself, or is it already been taken care of in [L221-L223](https://github.com/huggingface/transformers/blob/master/examples/run_language_modeling.py#L221-L223)?

Thanks!


Add option to choose T5 model size.

I believe the error mentioned in #3469 is due to the example tests loading T5-large in memory. One of the workers load that model which fills up the machine's memory, and other workers crash with an oom error.

This PR gives the option to choose the T5 model size, and changes the tests to only use the small model.


Add Colab with the evaluation procedure


add summarization and translation to notebook

Add summarization and translation to pipeline notebook


Added CovidBERT-NLI model card

Related to pretrained model cards


Finetuning FlauBERT with hugging face's Transformers : WARNING - __main__ -   Process rank: -1, device: cuda, n_gpu: 2, distributed training: False, 16-bits training: True

I am trying to use the script run_glue present in the transformers library to fintenue Flaubert for french data using huggingface's transformers and I am gettinh this error :

`

03/27/2020 11:54:48 - WARNING - __main__ -   Process rank: -1, device: cuda, n_gpu: 2, distributed training: False, 16-bits training: True
Traceback (most recent call last):
  File "/home/getalp/kelodjoe/transformers/examples/run_glue.py", line 693, in <module>
    main()
  File "/home/getalp/kelodjoe/transformers/examples/run_glue.py", line 613, in main
    config_class, model_class, tokenizer_class = MODEL_CLASSES[args.model_type]
KeyError: 'flaubert'

`

Could you help me find how to solve it ?


[WIP] General docs polish

Rebased from ##3461. So merge after this one.

- [x] Add Bart, T5 and MMBT to main docs page
- [x] Add MMBT docs
- [ ] Add MMBT pretrained info
- [ ] Polish MMBT docstring


[examples] fine-tuning `bert-base-finnish-(un)cased-v1` model for Named Entity Recognition

No new features added and no modifications to the existing code has been done. Just addition of scripts for the fine-tuning.


Inversion of a mask in newer pytorch versions

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): XLNet

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)
Finetuning on downstream task.
## To reproduce

Steps to reproduce the behavior:

1.Just run the XLNet Model with a newer pytorch version.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
```
RuntimeError: Subtraction, the `-` operator, with a bool tensor is not supported. If you are trying to invert a mask, use the `~` or `logical_not()` operator instead.
```

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.6.0
- Platform: Ubuntu
- Python version: 3.6
- PyTorch version (GPU?): 1.4.0
- Tensorflow version (GPU?):
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: parallel

## Additional Comments
This change was made in pytorch 1.2.0, check the release notes [here](https://github.com/pytorch/pytorch/releases/tag/v1.2.0).


Optimize tokenization (85-92% time reduction)

This optimization in tokenization reduces the tokenization time by ~ 85-92 %

Stats:

Before fix:
 8%|████████▉                                                                                | 119574/1486030 [01:03<12:09, 1874.22it/s]

After fix:
 98%|████████████████████████████████████  | 1459590/1486030 [01:59<00:02, 12244.82it/s]

Another stats:
When huge number of new tokens (~58K) are added, tokenization time bumps up to ~27-28 hours
With this fix the time drops to ~2.15 hours !


TFAlbertForMaskedLM Decoding Error

# 🐛 Bug

## Information

### Model
TFAlbertForMaskedLM <https://huggingface.co/transformers/model_doc/albert.html#tfalbertformaskedlm>

### Language
English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)
```python
import tensorflow as tf
from transformers import AlbertTokenizer, TFAlbertForMaskedLM

tokenizer = AlbertTokenizer.from_pretrained('albert-base-v2')
model = TFAlbertForMaskedLM.from_pretrained('albert-base-v2')
input_ids = tf.constant(tokenizer.encode("This is a test!"))[
    None, :]  # Batch size 1
outputs = model(input_ids)
prediction_scores = outputs[0]
outputTokens = tf.math.argmax(prediction_scores, axis=2)
outputTokens = tf.keras.backend.eval(outputTokens[0])
outputTokens = tokenizer.decode(outputTokens)
print(outputTokens)

```
The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Copy the above code
2. See output in terminal
```
time this is a test! your
```

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
Output
`This is a test!`
## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: How to install transformers-cli?
- Platform:  MacOS 10.15.2
- Python version: 3.6.6
- PyTorch version (GPU?): 1.4.0 CPU
- Tensorflow version (GPU?): 2.1.0 CPU
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


Update README.md

Related to pretrained model cards

Fix typo


CircleCI ExamplesTests::test_run_squad failing

Started happening at  https://github.com/huggingface/transformers/pull/3428
and has been happening consistently.

scroll all the way down for [traceback](https://circleci.com/gh/huggingface/transformers/26044?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link)


Issue in generating samples for text generation

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...):GPT2

```python
def generate_samples(args, model, prompt_text):
  """Generating sampling for the provided prompt using the provided model."""
  set_seed(args.seed)

  _, _, tokenizer_class = run_language_modeling.MODEL_CLASSES[args.model_type]
  tokenizer = tokenizer_class.from_pretrained(args.model_name_or_path, cache_dir=None)

  requires_preprocessing = args.model_type in run_generation.PREPROCESSING_FUNCTIONS.keys()
  encoded_prompt = tokenizer.encode(prompt_text, add_special_tokens=False, return_tensors="pt")
  encoded_prompt = encoded_prompt.to(args.device)
```

Error: run_language_modeling has no 'MODEL_CLASSES'

Language I am using the model on (English):


Model Cards: Fix grammar error


Docstring cannot be build anymore

# 🐛 Bug

## Information

Docstring

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. `pip install -e ".[docs]"` at `transformers` root folder
2. `cd docs`
3. `make html`

## Expected behavior

It should work, but an error message is displayed:

> /home/patrick/hugging_face/transformers/src/transformers/modeling_utils.py:docstring of transformers.PreTrainedModel.from_pretrained:23: WARNING: Unexpected indentation.
> /home/patrick/hugging_face/transformers/src/transformers/modeling_tf_utils.py:docstring of transformers.TFPreTrainedModel.from_pretrained:20: WARNING: Definition list ends without a blank line; unexpected unindent.
>
> Exception occurred:
>   File "/home/patrick/hugging_face/transformers_venv/lib/python3.6/site-packages/sphinx/util/docfields.py", line 260, in transform
>     assert len(field) == 2
> AssertionError
> The full traceback has been saved in /tmp/sphinx-err-mblqzztk.log, if you want to report the issue to the developers.
> Please also report this if it was a user error, so that a better error message can be provided next time.
> A bug report can be filed in the tracker at <https://github.com/sphinx-doc/sphinx/issues>. Thanks!
> Makefile:19: recipe for target 'html' failed

## Environment info

- `transformers` version: 2.6.0
- Platform: Linux-5.3.0-42-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.4.0+cpu (False)
- Tensorflow version (GPU?): 2.1.0 (False)
- Using GPU in script?:No
- Using distributed or parallel set-up in script?: No


Add link to 16 POS tags model

Related to pretrained model cards


Add text shown in example of usage

Related to pretrained model cards


Question Answering pipeline not working

This is error while running the question answering pipeline !

convert squad examples to features:   0%|          | 0/1 [00:00<?, ?it/s]

---------------------------------------------------------------------------

RemoteTraceback                           Traceback (most recent call last)

RemoteTraceback:
"""
Traceback (most recent call last):
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 44, in mapstar
    return list(map(*args))
  File "/usr/local/lib/python3.6/dist-packages/transformers/data/processors/squad.py", line 198, in squad_convert_example_to_features
    p_mask = np.array(span["token_type_ids"])
KeyError: 'token_type_ids'
"""

The above exception was the direct cause of the following exception:

KeyError                                  Traceback (most recent call last)

<ipython-input-3-3c4dd3618524> in <module>()
      2
      3 nlp_qa = pipeline('question-answering')
----> 4 nlp_qa(context='Hugging Face is a French company based in New-York.', question='Where is based Hugging Face ?')

8 frames

/usr/local/lib/python3.6/dist-packages/transformers/data/processors/squad.py in squad_convert_example_to_features()
    196         # p_mask: mask with 1 for token than cannot be in the answer (0 for token which can be in an answer)
    197         # Original TF implem also keep the classification token (set to 0) (not sure why...)
--> 198         p_mask = np.array(span["token_type_ids"])
    199
    200         p_mask = np.minimum(p_mask, 1)

KeyError: 'token_type_ids'


SyntaxError when fine-tuning ALBERT on NER

Should Fix

This has been identified as a bug and should be fixed.

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): ALBERT

Language I am using the model on (English, Chinese ...):

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

## To reproduce

Steps to reproduce the behavior:

Running on a GCP VM:

1.
```
python3 ${REPO_DIR}/src/transformers/convert_albert_original_tf_checkpoint_to_pytorch.py --tf_checkpoint_path ${BERT_CKPT_DIR} --albert_config_file ${BERT_CKPT_DIR}/config.json --pytorch_dump_path ${BERT_CKPT_DIR}/pytorch_model.bin
```
2.
```
python3 ${REPO_DIR}/examples/ner/run_ner.py --model_type albert --model_name_or_path ${BERT_CKPT_DIR} --do_train --do_eval --data_dir ${NER_DATA_DIR} --labels ${NER_DATA_DIR}/labels.txt --max_seq_length 128 --num_train_epochs 3 --per_gpu_train_batch_size 32 --output_dir ${NER_MODEL_CKPT_DIR} --seed 1 --do_predict --save_steps 750
```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->
```
Traceback (most recent call last):
  File "transformers/src/transformers/convert_albert_original_tf_checkpoint_to_pytorch.py", line 23, in <module>
    from transformers import AlbertConfig, AlbertForMaskedLM, load_tf_weights_in_albert
  File "<frozen importlib._bootstrap>", line 969, in _find_and_load
  File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 664, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 634, in _load_backward_compatible
  File "/home/manuto/.local/lib/python3.5/site-packages/transformers-2.6.0-py3.5.egg/transformers/__init__.py", line 23, in <module>
    from .benchmark_utils import (
  File "<frozen importlib._bootstrap>", line 969, in _find_and_load
  File "<frozen importlib._bootstrap>", line 954, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 896, in _find_spec
  File "<frozen importlib._bootstrap_external>", line 1147, in find_spec
  File "<frozen importlib._bootstrap_external>", line 1123, in _get_spec
  File "<frozen importlib._bootstrap_external>", line 1104, in _legacy_get_spec
  File "<frozen importlib._bootstrap>", line 444, in spec_from_loader
  File "<frozen importlib._bootstrap_external>", line 541, in spec_from_file_location
  File "/home/manuto/.local/lib/python3.5/site-packages/transformers-2.6.0-py3.5.egg/transformers/benchmark_utils.py", line 44
    filename: str
            ^
SyntaxError: invalid syntax
Exception ignored in: <function WeakValueDictionary.__init__.<locals>.remove at 0x7fc729d8a598>
Traceback (most recent call last):
  File "/usr/lib/python3.5/weakref.py", line 117, in remove
TypeError: 'NoneType' object is not callable
Traceback (most recent call last):
  File "transformers/examples/ner/run_ner.py", line 33, in <module>
    from transformers import (
  File "<frozen importlib._bootstrap>", line 969, in _find_and_load
  File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 664, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 634, in _load_backward_compatible
  File "/home/manuto/.local/lib/python3.5/site-packages/transformers-2.6.0-py3.5.egg/transformers/__init__.py", line 23, in <module>
  File "<frozen importlib._bootstrap>", line 969, in _find_and_load
  File "<frozen importlib._bootstrap>", line 954, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 896, in _find_spec
  File "<frozen importlib._bootstrap_external>", line 1147, in find_spec
  File "<frozen importlib._bootstrap_external>", line 1123, in _get_spec
  File "<frozen importlib._bootstrap_external>", line 1104, in _legacy_get_spec
  File "<frozen importlib._bootstrap>", line 444, in spec_from_loader
  File "<frozen importlib._bootstrap_external>", line 541, in spec_from_file_location
  File "/home/manuto/.local/lib/python3.5/site-packages/transformers-2.6.0-py3.5.egg/transformers/benchmark_utils.py", line 44
    filename: str
            ^
SyntaxError: invalid syntax
Exception ignored in: <function WeakValueDictionary.__init__.<locals>.remove at 0x7f1bd863ed08>
Traceback (most recent call last):
  File "/usr/lib/python3.5/weakref.py", line 117, in remove
TypeError: 'NoneType' object is not callable
```

## Expected behavior

Convert ALBERT TF checkpoint to PyTorch `model.bin` (works in 2.5.1 version of transformers) and fine-tune the model on NER (error in 2.5.1 mentioned [here](https://github.com/huggingface/transformers/issues/3412) which is the reason why I switched to 2.6.0).
<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.6.0 (installed with setup.py)
- Platform: Linux Ubuntu 18.04
- Python version: 3.5.3
- PyTorch version (GPU?): 1.4.0
- Tensorflow version (GPU?): 1.14.0
- Using GPU in script?:
- Using distributed or parallel set-up in script?:


Add T5 to docs

- [x] Copy past from bart docs
- [x] polish main docs page
- [x] improve docstring in `modeling_t5.py` and `modeling_tf_t5.py`


Error : forward() got an unexpected keyword argument 'inputs_embeds'

Hello,

I am trying to train a GPT2 from scratch using ```modeling_gpt2.py``` as a base.
I declared the model as follow :
```Python
config = GPT2Config(vocab_size = VELSIZE, n_positions = SEQLEN, n_embd = EMBEDSIZE, n_layer = NUMLAYER,n_ctx = SEQLEN, n_head = NUMHEAD)
model = GPT2Model(config)
```
I don't need to use the buid-in embeddings for my application and would like to pass my input tensor as-is, but trying ```model(inputs_embeds = test)```,  ```model.forward(inputs_embeds = test)```, ```model(input_ids=None,
      past=None,
      attention_mask=None,
      token_type_ids=None,
      position_ids=None,
      head_mask=None,
      inputs_embeds=test)``` or any other variant I can think of always results in the following error :
```Python
Traceback (most recent call last):

  File "<ipython-input-52-616a2eb9b3f4>", line 7, in <module>
    inputs_embeds=testx)

  File "C:\Users\cnelias\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)

TypeError: forward() got an unexpected keyword argument 'inputs_embeds'
```

Is this a bug or am I doing it wrong ?


[Docs] Add better explanation to check `docs` locally.


I am trying to use the script run_glue to fintenue Flaubert for french data using huggingface's transformers and I am gettinh this error :

`

03/27/2020 11:54:48 - WARNING - __main__ -   Process rank: -1, device: cuda, n_gpu: 2, distributed training: False, 16-bits training: True
Traceback (most recent call last):
  File "/home/getalp/kelodjoe/transformers/examples/run_glue.py", line 693, in <module>
    main()
  File "/home/getalp/kelodjoe/transformers/examples/run_glue.py", line 613, in main
    config_class, model_class, tokenizer_class = MODEL_CLASSES[args.model_type]
KeyError: 'flaubert'

`

Could you help me find how to solve it ?


ImportError: cannot import name 'TF_MODEL_FOR_TOKEN_CLASSIFICATION_MAPPING'

Wanted to use the ner demo, but after running:

 `!python3 ./ner/run_tf_ner.py --data_dir /data --model_type bert --labels .data/labels.txt --model_name_or_path bert-base-multilingual-cased --output_dir germeval-model --max_seq_length 128 --num_train_epochs 3 --per_device_train_batch_size 32 --save_steps 750 --seed 1 --do_train --do_eval --do_predict`

i encounter the ImportError

Any suggestions?


Fine-tuning with BertForSequenceClassification on custom dataset yields a model that outputs only the label with highest support in training set

Hello!
I have a custom dataset that I wish to fine tune BERT on for classification. The examples consist of 3 sequences each and the label set is {0, 1, ..., 9}. The training data has 570 examples, and validation has 150 examples.
The encoding of the input examples is as follows:
`tmp_enc = sequence_a + ' [SEP] ' + sequence_b + ' [SEP] ' + sequence_c`
 `enc = tokenizer.encode(tmp_enc, add_special_tokens=True)`
where
`tokenizer =  BertTokenizer.from_pretrained('bert-base-uncased')`

my training loop is:
```
optimizer = torch.optim.SGD(model.parameters(), lr=0.001)
scheduler = get_linear_schedule_with_warmup(optimizer, num_warmup_steps=0,
num_training_steps=5*len(X_train))

train_losses = []
valid_losses = []
for epoch in range(5):
    #train
    model.train();
    epoch_train_loss = 0
    for i in range(len(X_train)):
        model.zero_grad()
        loss, logits = model(input_ids=X_train[i], labels=y_train[i].unsqueeze(0))
        loss.backward()

        optimizer.step()
        scheduler.step()
        optimizer.zero_grad()
        nn.utils.clip_grad_norm_(model.parameters(), 1.0)
        epoch_train_loss += loss.item()
        if i % 100 == 0:
            print(f'example {i + 1}, loss: {loss.item()}')

    avg_loss = epoch_train_loss / len(X_train)
    train_losses.append(avg_loss)
    print(f'epoch {epoch + 1}, average train loss: {avg_loss}')

    #validation
    model.eval();
    epoch_valid_loss = 0
    for i in range(len(X_test)):
        with torch.no_grad():
            loss, logits = model(input_ids=X_test[i], labels=y_test[i].unsqueeze(0))
        epoch_valid_loss += loss.item()
    avg_loss = epoch_valid_loss / len(X_test)
    valid_losses.append(avg_loss)
    print(f'epoch {epoch + 1} done. average valid loss: {avg_loss}')
```
where
`model = BertForSequenceClassification.from_pretrained('bert-base-uncased', num_labels=10)`
and all layers but the classifier are frozen:
```
for param in model.bert.parameters():
    param.requires_grad = False
```
My issue is that loss is not decreasing and the model eventually always outputs a single label which is the one with highest support in the data. (validated by removing the examples with highest occurring label and getting similar results). Also, the logits for all examples, passed to the model after training are quite similar. e.g.:
```
logits: tensor([[ 1.5107, -0.0595,  0.3490, -0.8669, -0.8848, -0.8097,  0.2685,  0.7246,
         -0.3133,  0.4215]]), true label: 4
logits: tensor([[ 1.4187, -0.3009,  0.3776, -0.5615, -0.7881, -0.5849,  0.3391,  0.5756,
         -0.3861,  0.3639]]), true label: 6
logits: tensor([[ 1.3919, -0.4227,  0.3455, -0.4626, -0.7795, -0.5608,  0.2996,  0.5791,
         -0.4275,  0.3700]]), true label: 8
```
I have verified that a linear classifier can achieve better results (65% accuracy as opposed to 30%) on the embeddings from pytorch_transformers' BertModel.


Tokenizers: Start cleaning examples a little


NER pipeline usage examples

In the NER usage examples:
https://huggingface.co/transformers/usage.html#named-entity-recognition

Can you explain why the examples give only I- entities? For example:
('New', 'I-LOC'), ('York', 'I-LOC'), ('City', 'I-LOC')
('Hu', 'I-ORG'), ('##gging', 'I-ORG'), ('Face', 'I-ORG'), ('Inc', 'I-ORG')

Why are there no B-s? As in:
('New', 'B-LOC'), ('York', 'I-LOC'), ('City', 'I-LOC') etc


Create card for the model: GPT-2-finetuned-covid-bio-medrxiv

model card

Related to pretrained model cards


Write With Transformer returning a 502 on gpt2/xl model

When setting the model size to **gpt2/xl**, WwT gets stuck on loading the autocomplete.
Checking Chrome's console tells me
"Failed to load resource: the server responded with a status of 502 (Bad Gateway)"

Having a quick look through the older tickets, I saw that this has happened before. #2121


[examples] SummarizationDataset cleanup

- factor out redundant tokenization logic
-  For both articles and summaries, batches are "trimmed" such that no columns are full of `pad_token_id`.
- The max sizes are 1024 for source and 56 for target. This ensures that truncation never happens for summaries, and rarely happens for articles. These values are unchanged, but converted to command line arguments for ease of use.
- added small unittest (just for the dataset).
- Verified manually on GPU. Loss goes down, peak memory usage identical, speed improved by ~10\%.

Summary Statistics on summary lengths (in # tokens) for cnn_dm test data:
![image](https://user-images.githubusercontent.com/6045025/77860710-94b83500-71de-11ea-8d09-e952f3dd9e8c.png)


Create card for model GPT-2-finetuned-CORD19

Related to pretrained model cards


revert unpin isort commit

This PR reverts the change of
https://github.com/huggingface/transformers/commit/fbc5bf10cfe4d4ca81f8daacc148b0abd51dda5a

Using the unpinned version of `isort` makes black and `isort` disagree in some cases.
In this PR the unpinned version of `isort` leads to a falied code quality test:
https://github.com/huggingface/transformers/pull/3411


Failure to load checkpoints saved during distributed training

Core: Modeling

Internals of the library; Models.

Ex: LM (Pretraining)

Related to language modeling pre-training

# 🐛 Bug

## Information

Model I am using: Bert

Language I am using the model on: English, German, Swedish

The problem arises when using:
* [x] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Run [run_language_modeling.py](https://github.com/huggingface/transformers/blob/master/examples/run_language_modeling.py) using distributed training: e.g.
`python3 -m torch.distributed.launch --nproc_per_node=4 --nnodes=1 --node_rank=0 run_language_modeling.py` --output_dir output_dir [--args...]
2. When the training is over, try to load the final model: BertModel.from_pretrained('output_dir'): this works.
3. Then, try to load a checkpoint: e.g., `BertModel.from_pretrained('output_dir/checkpoint-1000')`: this gives a runtime error:
`
Traceback (most recent call last):
  File "/cluster/shared/nlpl/software/modules/in5550/202002/lib/python3.7/site-packages/transformers/modeling_utils.py", line 470, in from_pretrained
    state_dict = torch.load(resolved_archive_file, map_location="cpu")
  File "/cluster/shared/nlpl/software/modules/pytorch/1.4.0/lib/python3.7/site-packages/torch/serialization.py", line 529, in load
    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
  File "/cluster/shared/nlpl/software/modules/pytorch/1.4.0/lib/python3.7/site-packages/torch/serialization.py", line 709, in _legacy_load
    deserialized_objects[key]._set_from_file(f, offset, f_should_read_directly)
RuntimeError: storage has wrong size: expected 4434893008627221919 got 2359296
`

## Expected behavior
We should be able to load from the checkpoints also when the training is distributed.

## Suggested solution
Check `if args.local_rank == -1 or torch.distributed.get_rank() == 0` on line 370 (just like on line 736).

## Environment info
- `transformers` version: 2.5.0
- Platform: UNIX
- Python version: Python 3.5.3
- PyTorch version (GPU?): 1.4.0
- Tensorflow version (GPU?): No
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Distributed (1 node, 4 GPUs)


Save models after each epoch

Examples

Which is related to examples in general

# 🚀 Feature request
In the various training scripts in `examples`, would it be better to checkpoint the model at the end of each epoch, as well as every `save_steps` iterations as specified by the user?

## Motivation
I suppose for language modelling, saving the model after each epoch is not as important, but for anything supervised (and some other applications)  it seems  natural to want checkpoints after each epoch. There are plenty of examples in the literature where one wants to inspect models when they have seen each training example some specific number of times. I have been doing some experiments recently where this was the case (with `run_languagemodeling.py` actually) and found myself having to manually enter the number of iterations per epoch as `save_steps to get the desired checkpoints.

## Your contribution
Would be a simple change to the various training scripts. Happy to do a PR.


Special tokens to pre-trained BART model

Core: Tokenization

Internals of the library; Tokenization.

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
Is it possibile to add special tokens to the pre-trained BART model? My text has `<s>` as sequence separator for sentences. I would like that the encoder will handle it as a whole token, otherwise the model will break it in codes and learn like `<s` or `s>` etc. in the same we did  for other tokenizers like `GPT2Tokenizer`?

```python
tokenizer = GPT2Tokenizer.from_pretrained(args.out,
                                                    unk_token="<unk>",
                                                    bos_token="<s>",
                                                    eos_token="</s>",
                                                    pad_token = "<pad>",
                                                    additional_special_tokens=["<startoflyrics>",  "<endoflyrics>", "<nl>"])
```


run_lm_finetuning on multiple training files

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->

Hi,

I would like to fine-tune huggingface's pre-trained BERT model on a relatively big text data.
I split the data into multiple files (~10k files) in .raw format in the same folder.

First, I succeeded to run run_lm_finetuning.py on one of the raw files I generated.

Now, I would like to run that in my 10k raw files.
I realised that that train_data_file argument does not accept a folder name (only the path to a single raw file).
I believe that we can do a loop over the 10k train files and incrementally fine-tuning the model, but it does not look like a best solution for me...

Could you please tell me if it exists a simple way to achieve the above?

Thank you very much for your help.


Import error in example script `run_language_modeling.py`

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): RobertaForMaskedLM

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. pip install transformers
2. run `run_language_modeling.py`, which is the example script
3.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->
Error message:
>  Traceback (most recent call last):                                                                                File "run_language_modeling.py", line 42, in <module>                                                                                from transformers import (
ImportError: cannot import name 'MODEL_WITH_LM_HEAD_MAPPING'

## Expected behavior
The script should run..
<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.5.1
- Platform: Ubuntu 18.04
- Python version: 3.7
- PyTorch version (GPU?): 1.4.0
- Tensorflow version (GPU?): na
- Using GPU in script?: y
- Using distributed or parallel set-up in script?: n

## Note
The work around is to use
> from transformers.modeling_auto import MODEL_WITH_LM_HEAD_MAPPING
> from transformers.file_utils import WEIGHTS_NAME

Can you please update the example script? It is confusing ...


`run_language_modeling` fails with community model (BioClinicalBERT)

Core: Modeling

Internals of the library; Models.

# 🐛 Bug

The `run_language_modeling.py` fails when using community models.

## Information

Model I am using (Bert, XLNet ...): BioClinical_BERT

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. download wikitext-2 raw
2. run exactly the same
3.

This code command line yields the following error:
```
python run_language_modeling.py \
    --output_dir=output \
    --model_type=BioClinicalBERT \
    --model_name_or_path=emilyalsentzer/Bio_ClinicalBERT \
    --output_dir=/kaggle/working/model \
    --do_train \
    --line_by_line \
    --train_data_file=wikitext-2-raw/wiki.train.raw \
    --do_eval \
    --eval_data_file=wikitext-2-raw/wiki.valid.raw \
    --num_train_epochs=4 \
    --mlm
```

the following error will appear:
```
  File "run_language_modeling.py", line 782, in <module>
    main()
  File "run_language_modeling.py", line 732, in main
    global_step, tr_loss = train(args, train_dataset, model, tokenizer)
  File "run_language_modeling.py", line 333, in train
    outputs = model(inputs, masked_lm_labels=labels) if args.mlm else model(inputs, labels=labels)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/transformers/modeling_bert.py", line 987, in forward
    encoder_attention_mask=encoder_attention_mask,
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/transformers/modeling_bert.py", line 790, in forward
    encoder_attention_mask=encoder_extended_attention_mask,
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/transformers/modeling_bert.py", line 407, in forward
    hidden_states, attention_mask, head_mask[i], encoder_hidden_states, encoder_attention_mask
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/transformers/modeling_bert.py", line 368, in forward
    self_attention_outputs = self.attention(hidden_states, attention_mask, head_mask)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/transformers/modeling_bert.py", line 314, in forward
    hidden_states, attention_mask, head_mask, encoder_hidden_states, encoder_attention_mask
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/transformers/modeling_bert.py", line 234, in forward
    attention_scores = torch.matmul(query_layer, key_layer.transpose(-1, -2))
RuntimeError: cublas runtime error : the GPU program failed to execute at /opt/conda/conda-bld/pytorch_1579022034529/work/aten/src/THC/THCBlas.cu:368
```

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
transformers v2.6.0

- `transformers` version:
- Platform: Kaggle
- Python version: 3.7
- PyTorch version (GPU?): 1.4.0
- Tensorflow version (GPU?):
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No


can't import TFBertModel from transformers

this is the log when I imported the TFBertModel from transformers
```
from transformers import TFBertModel
ImportError: cannot import name 'TFBertModel' from 'transformers' (/home/cally/.local/lib/python3.7/site-packages/transformers/__init__.py)

```
my enviroment is transformers 2.4 linux python 3.7


Add support for the null answer in `QuestionAnsweringPipeline`


feat: config what's trainable in Bert layers

Make it possible to configure trainability for specific subparts of a `TFBertMainLayer`.

This is minimal and Bert-only. Really should be added in more model types, not just Bert, but it probably necessarily differs quite a bit between model types, so probably can't be done in a very “DRY” way.

See https://github.com/tensorflow/tensorflow/issues/37541 which makes this necessary: if we set `l.trainable = False` on a layer _after_ initializing it, then training and serialization proceeds without apparent problems but then deserialization will fail, because:

* the `trainable` attribute doesn't get persisted
* parameter values are deserialized and batch-assigned to model parameters _in order_ — with the implicit assumption that the ordering of parameters is the same as in the model before serialization
* that assumption doesn't hold if some layers were not trainable before serialization, because the ordering of parameters depends on the `trainable` attribute, which is `True` by default because it wasn't persisted


Force the return of token type IDs

close #3313
close #3227

These two issues happen because the `token_type_ids` are not generated by the `encode_plus` method when the SQuAD and multiple choice scripts expect it. The GLUE script was patched by #3240


Same probability from fine-tuning custom pre-trained LM

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Description of your issue -->

I have a dataset with QA sentence pairs. I pre-trained a bert model from scractch using run_language_modeling.py where I concatenated the QA pairs and trained the dataset line_by_line.

I tried to fine-tune the custom pre-trained model, but am getting the same probability output for different inputs. I also tried to reduce the learning rate, but the constant probability problem remains.

What could I be doing wrong?

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:


[Bug fix] Using loaded checkpoint with --do_predict (instead of random init)

Without this fix, I'm getting near-random validation performance for a trained model, and the validation performance differs per validation run. I think this happens since the `model` variable isn't set with the loaded checkpoint, so I'm using a randomly initialized model. Looking at the model activations, they differ each time I run evaluation (but they don't with this fix).


TFXLMRoberta impossible to load base and large model with pretrained weight ?

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...):
XLMRoberta
Language I am using the model on (English, Chinese ...):

The problem arises when using:
* [ x] the official example scripts: (give details below)

The tasks I am working on is:
my own task or dataset: (give details below)

## To reproduce

launch the command :
`model = TFXLMRobertaForSequenceClassification.from_pretrained(pretrained_model_name_or_path="xlm-roberta-large" )`
Steps to reproduce the behavior:

1. from transformers import*
2. launch the command above
3. error :

> TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType

## Expected behavior

model should be load.

## Environment info

- `transformers` version: 2.6.0
- Platform: colab

- Tensorflow version (GPU?):2.1 gpu
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no


Updated/added model cards

model card

Related to pretrained model cards

Trained four models, adding/updating model cards to make consistent!


How to detokenize a BertTokenizer output?

Core: Tokenization

Internals of the library; Tokenization.

For example, let's tokenize a sentece "why isn't Alex' text tokenizing":

    tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', do_lower_case=True)
    tokens = tokenizer.tokenize("why isn't Alex' text tokenizing")

We are getting the next output:

    ['why', 'isn', "'", 't', 'Alex', "'", 'text', 'token', '##izing']

I want to convert it to:

    why isn't Alex' text tokenizing

It seems the tokenizer doesn't do its job in the best way. If after tokenization we would have something like that:

    ['why', 'isn', "##'", '##t', 'Alex', "##'", 'text', 'token', '##izing']

It would be easy to convert:

    tokens = ['why', 'isn', "##'", '##t', 'Alex', "##'", 'text', 'token', '##izing']
    restored_text = [None] * len(tokens)
    tokens.extend(['#', '#'])
    for i in range(len(tokens )-2):
        if re.findall("#{2}", tokens[i+1]):
          restored_text [i] = tokens[i] + tokens[i+1].replace('##', '')
          if re.findall("#{2}", tokens [i+2]):
            restored_text[i] = restored_text[i] + tokens[i+2].replace('##', '')
        else:
          restored_text[i] = tokens[i]

    restored_text_without_masks = []
    for i in range(len(restored_text)):
        if not restored_text[i].startswith('#'):
            restored_text_without_masks.append(restored_text[i])


Extend config with task specific configs.

As discussed and proposed by @thomwolf in PR #3413, another step towards a combined tokenizer/model config is this PR. It extends the normal config with the following parameters:

```
{
....
prefix = "", # generic generation HP
max_length: 100,
length_penalty: 1.0,
task_specific_params: {
     "summarization": {  # task id (e.g. name of the pipeline?)
          max_length: 140,
          length_penalty: 2.0
      },
     "translation_en_to_de": {
          prefix: "translate English to German: "
          max_length: 160,
          length_penalty: 3.0
      },
    },
}
```

In terms of hierarchy for a task-specific generation it would go as follows:
1) Is the parameter provided as an argument to the `generate` method ? Yes use these. No - go to 2.
2) Is the parameter provided in the `task_specific_params dict` ?  Yes use these. No - go to 3.
3) Is the parameter provided in the default `config dict`? Yes use these. No - go to 4.
4) Is the parameter provided hard-coded in the model's config file? Yes use these. No - use the very default parameters of `PretrainedConfig`

These were our arguments in favor of this:

- This removes a lot of hard coded parameters in pipelines and examples
- Another step towards a combined tokenizer / model config
- A lot of weird if-else statements can be saved ("If task is en-de translation then do X" won't be necessary as the en-de specific parameters will override the default ones)

### TODO

If you guys are fine with this structure:

- [ ]  I will add the `task_specific_params` for Bart and T5s configs on S3
- [ ] clean up the examples and pipelines.
- [ ] rebase all the T5s PRs: #3428, #3419, #3413, #3411


how to use transformers to get all pretraining model names in transformers hub


Error ImportError: cannot import name 'MODEL_FOR_SEQUENCE_CLASSIFICATION_MAPPING' from 'transformers' (C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python37_64\lib\site-packages\transformers\__init__.py)

# 🐛 Bug

## Information

I am doing the tutorial ["Quick tour of the fine-tuning/usage scripts"](https://github.com/huggingface/transformers#quick-tour-of-the-fine-tuningusage-scripts)

I downloaded the Glue dataset.
When I try to run this command from pytorch
```
python ./examples/run_glue.py \
    --model_type bert \
    --model_name_or_path bert-base-uncased \
    --task_name MRPC \
    --do_train \
    --do_eval \
    --do_lower_case \
    --data_dir C:/Git/RemoteDGX/MRPC/glue_data/MRPC \
    --max_seq_length 128 \
    --per_gpu_eval_batch_size=8   \
    --per_gpu_train_batch_size=8   \
    --learning_rate 2e-5 \
    --num_train_epochs 3.0 \
    --output_dir /tmp/MRPC/

```
I am getting this error:
ImportError: cannot import name 'MODEL_FOR_SEQUENCE_CLASSIFICATION_MAPPING' from 'transformers' (C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python37_64\lib\site-packages\transformers\__init__.py)

:

## To reproduce

Steps to reproduce the behavior:

1.download the Glue database
2. execute the script run_glue.py

this is the stacktrace:
```
2020-03-25 14:09:19.698135: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll
Traceback (most recent call last):
  File "C:/Git/RemoteDGX/transformers/examples/run_glue.py", line 32, in <module>
    from transformers import (
ImportError: cannot import name 'MODEL_FOR_SEQUENCE_CLASSIFICATION_MAPPING' from 'transformers' (C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python37_64\lib\site-packages\transformers\__init__.py)
```

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:
- Platform: windows
- Python version: 3.7
- PyTorch version 1.4.0 without GPU:


Problem saving and/or loading fine-tuned model

# ❓ Questions & Help
I have finetuned bert-base-cased for multi-class classification with BertForSequenceClassification and got great results (about 88% accuracy on my dev data). However, when I save the finetuned model, load it and run the evaluation on the exact same dev data, I got awful results (about 0.17 accuracy).

At first glance, it seems that either I am wrongly saving the fine-tuned model OR wrongly loading it after training. Would it be possible that save_pretrained only save the weights of the BERT model without the ones of the classifier above ?  @patrickvonplaten

## Details
Here is how I save the fine-tuned model after training:
`model_to_save = model.module if hasattr(model, "module") else model # Take care of distributed/parallel training`
 `model_to_save.save_pretrained(args.output_dir)`
 `tokenizer.save_pretrained(args.output_dir)`

And here is how I load the fine-tuned model for running evaluation:
`model = BertForSequenceClassification.from_pretrained(
                         args.model_name_or_path,
                         num_labels = args.num_labels,
                         output_attentions = False,
                         output_hidden_states = False,
                         cache_dir = args.cache_dir)`
where `args.model_name_or_path` is the path of my .bin checkpoint, and args.num_labels stays unchanged during all process.

## Full code

```python
def train(args, model, tokenizer, dataset, tb_writer, categories):

    # Load training dataset.
    if args.do_eval and args.eval_filepath is None:
        print("No validation file given: splitting dataset to train/test datasets...\n")
        train_dataset, validation_dataset = split_data(dataset, args.test_percent, args.seed)
    else:
        train_dataset = dataset

    print("Creating training dataloader...\n")
    train_data, train_sampler, train_dataloader = create_dataloader(train_dataset, args.batch_size, training_data=True)

    # Setting up Optimizer & Learning Rate Scheduler.
    optimizer = AdamW(model.parameters(),
                  lr = args.learning_rate,
                  eps = args.adam_epsilon
                )
    total_steps = len(train_dataloader) * args.num_epochs
    scheduler = get_linear_schedule_with_warmup(optimizer,
                                                num_warmup_steps = 0, # Default value in run_glue.py
                                                num_training_steps = total_steps)
    # Init some useful variables.
    global_step = 0
    tr_loss, logging_loss = 0.0, 0.0

    # For each epoch...
    t = time.time()
    for epoch_i in range(0, args.num_epochs):

        # Perform one full pass over the training set.
        print('======== Epoch {:} / {:} ========'.format(epoch_i + 1, args.num_epochs))
        print('Training...')

        # Measure how long the training epoch takes.
        t0 = time.time()

        # Put the model into training mode. Don't be mislead--the call to
        # `train` just changes the *mode*, it doesn't *perform* the training.
        # `dropout` and `batchnorm` layers behave differently during training
        # vs. test (source: https://stackoverflow.com/questions/51433378/what-does-model-train-do-in-pytorch)
        model.train()

        # For each batch of training data...
        for step, batch in enumerate(train_dataloader):

            # Unpack this training batch from our dataloader.
            # As we unpack the batch, we'll also copy each tensor to the GPU using the `to` method.
            # `batch` contains three pytorch tensors:
            #   [0]: input ids
            #   [1]: attention masks
            #   [2]: labels
            b_input_ids = batch[0].to(args.device)
            b_input_mask = batch[1].to(args.device)
            b_labels = batch[2].to(args.device)

            # Always clear any previously calculated gradients before performing a backward pass.
            # PyTorch doesn't do this automatically because accumulating the gradients is "convenient while training RNNs".
            # (source: https://stackoverflow.com/questions/48001598/why-do-we-need-to-call-zero-grad-in-pytorch)
            model.zero_grad()

            # Perform a forward pass (evaluate the model on this training batch).
            # This will return the loss (rather than the model output) because we have provided the `labels`.
            # The documentation for this `model` function is here:
            # https://huggingface.co/transformers/v2.2.0/model_doc/bert.html#transformers.BertForSequenceClassification
            outputs = model(b_input_ids,
                        token_type_ids=None,
                        attention_mask=b_input_mask,
                        labels=b_labels)

            # The call to `model` always returns a tuple, so we need to pull the loss value out of the tuple.
            loss = outputs[0]

            if args.n_gpu > 1:
                loss = loss.mean()  # mean() to average on multi-gpu parallel training

            # Accumulate the training loss over all of the batches so that we can calculate the average loss at the end.
            # `loss` is a Tensor containing a single value; the `.item()` function just returns the Python value from the tensor.
            tr_loss += loss.item()

            # Perform a backward pass to calculate the gradients.
            loss.backward()

            # Clip the norm of the gradients to 1.0. This is to help prevent the "exploding gradients" problem.
            torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)

            # Update parameters and take a step using the computed gradient.
            # The optimizer dictates the "update rule"--how the parameters are modified based on their gradients, the learning rate, etc.
            optimizer.step()

            # Update the learning rate.
            scheduler.step()

            # Update global step.
            global_step += 1

            # Progress update every 'logging_steps' batches.
            if args.logging_steps > 0 and step != 0 and step % args.logging_steps == 0:

                # Calculate elapsed time in minutes.
                elapsed = format_time(time.time() - t0)

                # Compute average training loss over the last 'logging_steps'. Write it to Tensorboard.
                loss_scalar = (tr_loss - logging_loss) / args.logging_steps
                tb_writer.add_scalar('Train/Loss', loss_scalar, global_step)
                logging_loss = tr_loss

                # Print the log.
                print('  Batch {:>5,}  of  {:>5,}.    Elapsed: {:}.    Training loss: {:.2f}'.format(step, len(train_dataloader), elapsed, loss_scalar))

        print("  Training epoch took: {:}\n".format(format_time(time.time() - t0)))

        if args.do_eval and args.eval_filepath is None:
            print("Running Validation...")
            # After the completion of each training epoch, measure our performance on our validation set.
            t0 = time.time()
            result, df_wrong, df_right = evaluate(args, model, validation_dataset, categories)

            # Write results to tensorboard.
            tb_writer.add_scalar('Test/Accuracy', result[0], epoch_i + 1)
            tb_writer.add_scalar('Test/Recall', result[1], epoch_i + 1)
            tb_writer.add_scalar('Test/Precision', result[2], epoch_i + 1)
            tb_writer.add_scalar('Test/F1 score', result[3], epoch_i + 1)
            tb_writer.add_scalar('Test/MCC', result[4], epoch_i + 1)

            # Plot confusion matrix.
            plot_confusion_matrix(result[5], categories, args.output_dir)

            # Save dataframes of wrong and right predictions for further analysis.
            df_wrong.to_csv(os.path.join(args.output_dir, 'preds_wrong.csv'))
            df_right.to_csv(os.path.join(args.output_dir, 'preds_right.csv'))

            print("  Validation took: {:}\n".format(format_time(time.time() - t0)))

    print("Training complete!  Took: {}\n".format(format_time(time.time() - t)))

    print("Saving model to {}...\n.".format(args.output_dir))
    model_to_save = model.module if hasattr(model, 'module') else model  # Take care of distributed/parallel training
    model_to_save.save_pretrained(args.output_dir)
    tokenizer.save_pretrained(args.output_dir)
    torch.save(args, os.path.join(args.output_dir, 'training_args.bin'))  # Good practice: save your training arguments together with the trained model
    return

def evaluate(args, model, validation_dataset, categories):

    # Creating validation dataloader.
    validation_data, validation_sampler, validation_dataloader = create_dataloader(validation_dataset, args.batch_size, training_data=False)

    # Get validation sentences.
    validation_sentences = validation_dataset[3]

    # Tracking variables
    nb_eval_steps = 0
    preds = None
    out_label_ids = None

    # Put the model in evaluation mode--the dropout layers behave differently during evaluation.
    model.eval()

    # Evaluate data for one epoch
    for batch in validation_dataloader:

        # Add batch to GPU.
        b_input_ids, b_input_mask, b_labels = tuple(t.to(args.device) for t in batch)

        # Telling the model not to compute or store gradients, saving memory and speeding up validation
        with torch.no_grad():

            # Forward pass, calculate logit predictions.
            # This will return the logits rather than the loss because we have not provided labels.
            # token_type_ids is the same as the "segment ids", which differentiates sentence 1 and 2 in 2-sentence tasks.
            outputs = model(b_input_ids,
                            token_type_ids=None,
                            attention_mask=b_input_mask)

        # Get the "logits" output by the model. The "logits" are the output values prior to applying an activation function like the softmax.
        logits = outputs[0]

        # Move logits and labels to CPU and store them.
        if preds is None:
            preds = logits.detach().cpu().numpy()
            out_label_ids = b_labels.detach().cpu().numpy()
        else:
            preds = np.append(preds, logits.detach().cpu().numpy(), axis=0)
            out_label_ids = np.append(out_label_ids, b_labels.detach().cpu().numpy(), axis=0)

        # Track the number of batches
        nb_eval_steps += 1

    # Take the max predicitions.
    preds = np.argmax(preds, axis=1)

    # Report results.
    result = compute_metrics(preds, out_label_ids, categories)
    print("  * Accuracy: {0:.4f}".format(result[0]))
    print("  * Recall: {0:.4f}".format(result[1]))
    print("  * Precision: {0:.4f}".format(result[2]))
    print("  * F1 score: {0:.4f}".format(result[3]))
    print("  * MCC: {0:.4f}".format(result[4]))

    # Get wrong and right predictions.
    df_wrong, df_right = analyze_predictions(preds, out_label_ids, validation_sentences)

    return result, df_wrong, df_right

def main(args):

    # Create tensorboard summarywriter.
    tb_writer = SummaryWriter()

    # Create output dir if none mentioned.
    if args.output_dir is None:
        model_name = os.path.splitext(os.path.basename(args.model_name_or_path))[0]
        args.output_dir = "./output/" + model_name + '/'
    if not os.path.exists(args.output_dir):
        os.makedirs(args.output_dir)

    # Set the seed value all over the place to make this reproducible.
    set_seed(args.seed)

    print("\n========================================")
    print('               Load model                 ')
    print("========================================\n")
    print("Loading BertForSequenceClassification model...\n")
    model = BertForSequenceClassification.from_pretrained(
        args.model_name_or_path, # Use the 12-layer BERT model, with an cased vocab.
        num_labels = args.num_labels, # The number of output labels
        output_attentions = False, # Whether the model returns attentions weights.
        output_hidden_states = False, # Whether the model returns all hidden-states.
        cache_dir = args.cache_dir,
    )
    #model = BertForSequenceClassification.from_pretrained(args.model_name_or_path)
    print('Loading BertTokenizer...\n')
    tokenizer = BertTokenizer.from_pretrained(args.model_name_or_path, do_lower_case=False)

    print("Setting up CUDA & GPU...")
    if torch.cuda.is_available():
        if args.gpu_id:
            torch.cuda.set_device(args.gpu_id)
            args.n_gpu = 1
            print("-> GPU training available! As '--gpu_id' was set, only GPU {} {} will be used (no parallel training).\n".format(torch.cuda.get_device_name(args.gpu_id), args.gpu_id))
        else:
            args.n_gpu = torch.cuda.device_count()
            gpu_ids = list(range(0, args.n_gpu))
            if args.n_gpu > 1:
                model = torch.nn.DataParallel(model, device_ids=gpu_ids, output_device=gpu_ids[-1])
            print("-> GPU training available! Training will use GPU(s) {}\n".format(gpu_ids))
        args.device = torch.device("cuda")
    else:
        args.device = torch.device("cpu")
        args.n_gpu = 0
        print("-> No GPU available, using the CPU instead.\n")
    model.to(args.device)  # Tell pytorch to run the model on the device.

    print("\n========================================")
    print('               Processing data            ')
    print("========================================\n")
    df, categories = load_data(args)

    print("Tokenizing sentences...")
    tokenized = tokenize_sentences(tokenizer, df)
    attention_masks = create_masks(tokenized)

    dataset = (tokenized, df.Class_id.values, attention_masks, df.Sentence.values)
    if args.do_train:
        print("\n========================================")
        print('            Launching training            ')
        print("========================================\n")
        train(args, model, tokenizer, dataset, tb_writer, categories)

    elif args.do_eval and args.eval_filepath is not None:
        print("\n========================================")
        print('            Launching validation          ')
        print("========================================\n")
        result, df_wrong, df_right = evaluate(args, model, dataset, categories)

        # Save dataframes of wrong and right predictions for further analysis.
        df_wrong.to_csv(os.path.join(args.output_dir, 'wrong_preds.csv'))
        df_right.to_csv(os.path.join(args.output_dir, 'right_preds.csv'))
```


Confusion in understanding the output of BERTforTokenClassification class from Transformers library

It is the example given in the documentation of transformers pytorch library
```
from transformers import BertTokenizer, BertForTokenClassification
import torch

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertForTokenClassification.from_pretrained('bert-base-uncased')

input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze(0)  # Batch size 1
labels = torch.tensor([1] * input_ids.size(1)).unsqueeze(0)  # Batch size 1
outputs = model(input_ids, labels=labels)

loss, scores, hidden_states,attentions = outputs
```
Here hidden_states is a tuple of length 13 and contains hidden-states of the model at the output of each layer plus the initial embedding outputs. **I would like to know, whether hidden_states[0] or hidden_states[12] represent the final hidden state vectors**?

Thanks in advance  @thomwolf  @nreimers


Add wmt translation example

PR adds translation example for T5.
It uses the `sacrebleu` BLEU scorer.

I adapted the README.md a bit so that users are aware that models in official paper were attained with finetuned T5 @craffel


I want to create a tokenizer which the vocab file in my computer

I want to create a tokenizer in my computer, which the vocab file is below, my code as below
```
self.model_config = AutoConfig.from_pretrained(r'/home/cally/Awake/Code/bert/pre_model/bert_config.json')
self.tokenizer = AutoTokenizer.from_pretrained(r'/home/cally/Awake/Code/bert/pre_model/vocab.txt')
```
it appeared this error, what's wrong?, my enviroment is linux python 3.7 transformer 2.6
```
OSError: Couldn't reach server at '/home/cally/Awake/Code/bert/pre_model/vocab.txt' to download configuration file or configuration file is not a valid JSON file. Please check network or file content here: /home/cally/Awake/Code/bert/pre_model/vocab.txt.

```


I wrapped the code base into a flask container and tried to run it on a GPU and I am running into a GPU device copy issue. Looking for pointers.

I have verified that the model actually gets moved to the GPU:
`        self.model.to(torch.device("cuda" if torch.cuda.is_available() and not "store_true" else "cpu"))
        self.model.eval()  # TO HERE`

```
backend_1        |   File "/app/run_generation.py", line 285, in hook
backend_1        |     num_return_sequences=args.num_return_sequences,
backend_1        |   File "/usr/local/lib/python3.6/dist-packages/torch/autograd/grad_mode.py", line 49, in decorate_no_grad
backend_1        |     return func(*args, **kwargs)
backend_1        |   File "/app/transformers/modeling_utils.py", line 979, in generate
backend_1        |     attention_mask=attention_mask,
backend_1        |   File "/app/transformers/modeling_utils.py", line 1016, in _generate_no_beam_search
backend_1        |     outputs = self(**model_inputs)
backend_1        |   File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 541, in __call__
backend_1        |     result = self.forward(*input, **kwargs)
backend_1        |   File "/app/transformers/modeling_gpt2.py", line 599, in forward
backend_1        |     inputs_embeds=inputs_embeds,
backend_1        |   File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 541, in __call__
backend_1        |     result = self.forward(*input, **kwargs)
backend_1        |   File "/app/transformers/modeling_gpt2.py", line 465, in forward
backend_1        |     inputs_embeds = self.wte(input_ids)
backend_1        |   File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 541, in __call__
backend_1        |     result = self.forward(*input, **kwargs)
backend_1        |   File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/sparse.py", line 114, in forward
backend_1        |     self.norm_type, self.scale_grad_by_freq, self.sparse)
backend_1        |   File "/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py", line 1484, in embedding
backend_1        |     return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
backend_1        | RuntimeError: Expected object of device type cuda but got device type cpu for argument #1 'self' in call to _th_index_selec
```


Update model card huseinzol05/bert-base-bahasa-cased

model card

Related to pretrained model cards


Where is the code of Bart fine-tuning?Thanks


Experiment w/ dataclasses (including Py36)


[BART] add bart-large-xsum weights

- conversion script can now take a path, which is required since this model is not on `torch.hub`. Finetuning with fairseq and then converting to huggingface should work. I also cleaned it up a bit
- Config in S3 is already updated with author-recommended generation parameters:
`(num_beams=6, length_penalty=1., min_length=11, max_length=62`

Context:
These weights are from bart finetuned on the XSum abstractive summarization challenge, which encourages shorter (more abstractive) summaries. It achieves state of the art.

Discussion:
- I propose changing the SummarizationPipeline default to this model in a separate PR, since the summarizations are shorter (and high quality)!


Added BioBERT-NLI model card

Related to pretrained model cards


Reading files takes for ever in language modeling

# 🐛 Bug

## Information

Model I am using (GPT2):

Language I am using the model on (Music Midifiles tokens):

The problem arises when using:
* [ ] the official example scripts: (give details below)

The tasks I am working on is:
* language modeling, running the script run_langauge_modling.py

## To reproduce

Steps to reproduce the behavior:

```
python -m torch.distributed.launch \
--nproc_per_node 4 run_language_modeling.py \
 --train_data_file /nethome/abashir/data/train.txt \
--output_dir /data/users/abashir/model \
--model_type gpt2 --tokenizer_name /nethome/abashir/data/PianoAI \
  --do_train --line_by_line --learning_rate 1e-4 --num_train_epochs 5 \
 --save_total_limit 2 --save_steps 1000 --per_gpu_train_batch_size 8 \
 --seed 42 --overwrite_cache --block_size 128
```

The output freeze at this stage for more than a day. train file size is less than 1 GB:
`03/23/2020 19:43:11 - INFO - __main__ -   Creating features from dataset file at /nethome/abashir/data/train.txt`

## Expected behavior

Start the training right away after adding --line_by_line
## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.5.1
- Platform: Linux-4.4.0-45-generic-x86_64-with-debian-jessie-sid
- Python version: 3.7.6
- PyTorch version (GPU?): 1.4.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: yes


Adds translation pipeline

The API for translation is as follows:

```
en_fr_translation = pipeline("translation_en_to_fr")
en_fr_translation("How old are you?")
```

for English to French translation.

PR adds tests and gives an example in the docstring.
PR builds on #3413 and should be merged after this one.

Example:
![Screenshot from 2020-03-26 11-10-52](https://user-images.githubusercontent.com/23423619/77637172-899ea400-6f55-11ea-83a6-0a476fd33430.png)


Unused function in squad metrics

In the squad metrics file, I noticed an unused function `find_all_best_thresh_v2`

https://github.com/huggingface/transformers/blob/master/src/transformers/data/metrics/squad_metrics.py#L167

It looks to be pertaining to the squad v2.0 dataset type, which has 'impossible' questions for contexts.

It looks like it was meant to be used something like

```
    if no_answer_probs:
        if version_2_with_negative:
            find_all_best_thresh_v2(evaluation, preds, exact, f1, no_answer_probs, qas_id_to_has_answer)
        else:
             find_all_best_thresh(evaluation, preds, exact, f1, no_answer_probs, qas_id_to_has_answer)
```

here

https://github.com/huggingface/transformers/blob/master/src/transformers/data/metrics/squad_metrics.py#L236


Fix XLNet batch generation bug

When doing batch generation with XLNet, only the first element in the batch contains any predictions. This seems to be caused by the target_mapping being improperly initialized in prepare_inputs_for_generation.


XLNet model on S3 not set up correctly?

Hi all,

**I'm trying to initialize an XLNetForSequenceClassification model as follows:**

```
from transformers import XLNetForSequenceClassification, XLNetConfig
configuration = XLNetConfig()
model = XLNetForSequenceClassification.from_pretrained("xlnet-base-cased", config=configuration)
```

**However I'm getting an error:**

> RuntimeError: Error(s) in loading state_dict for XLNetForSequenceClassification:
> 	size mismatch for transformer.mask_emb: copying a param with shape torch.Size([1, 1, 768]) from checkpoint, the shape in current model is torch.Size([1, 1, 1024]).
> 	size mismatch for transformer.word_embedding.weight: copying a param with shape torch.Size([32000, 768]) from checkpoint, the shape in current model is torch.Size([32000, 1024]).
> 	size mismatch for transformer.layer.0.rel_attn.q: copying a param with shape torch.Size([768, 12, 64]) from checkpoint, the shape in current model is torch.Size([1024, 16, 64]).
>     ...

**I can get my code to run by specifying in my config:**

```
configuration.d_model = 768 # hidden size --> should be 1024
configuration.n_head = 12 # number of attention heads --> should be 16
configuration.d_inner = 3072 # FFN inner hidden size --> should be 4096
```

But I am somewhat confused: These adjustments to the model are not in line with XLNet as introduced in the [XLNet paper](https://arxiv.org/pdf/1906.08237.pdf) (page 13). Am I understanding something wrong here, or is the XLNet model on S3 not set up correctly?


Problem with running Transformer Notebook: How to train a language model

Hello,

in the page:

https://github.com/huggingface/transformers/blob/master/notebooks/README.md

I clicked "open in colab" on the notebook "How to train a language model".

When the last run cell was run:

%%time
!{cmd}

The following error was presented:

Traceback (most recent call last):
  File "run_language_modeling.py", line 40, in <module>
    from transformers import (
ImportError: cannot import name 'CONFIG_MAPPING'
CPU times: user 42.2 ms, sys: 18 ms, total: 60.2 ms
Wall time: 5.82 s

Any suggestion?

Thank you!


Add custom rules for sampling from GPT-2 Generator

Hi @patrickvonplaten,
I've read your [blogpost](https://huggingface.co/blog/how-to-generate) and it's really interesting. Thanks! I have a question for that.
We have recently trained a GPT-2 generator with HF on general text and it works well.

But to have better results on my custom task, I would like to add a custom function of sampling that is able to maximize a certain behaviour of the text I am generating.
I would like to add a custom sampling applied to top-k words that are choosen by GPT-2 in order to maximize , for example, the fact that in the text I am generating there must be the max number of vocals.
Could you help me to think about this solution?
Thanks


Add t5 to pipeline(task='summarization')

This PR:

- adds T5 to summarization piplines.
- adds warnings and better defaults to Bart/T5 summarization
- removes unnecessary assert in generate() function