noamaanMulla-03/transformers-issues huggingface dataset:
Reorganize documentation
Author: sgugger
Pull request: yes

This PR does two things:
- reorganize the doc topics in five sections
- complete the model list in the index

Also I cut all lines at 119 chars (like the code) otherwise it's not readable in visual studio code (and I imagine other viewers), removed the stars from authors since it wasn't pointing to anything (can add them back but we should explain what they mean in that case) and made all authors list comma-separated with one last 'and'.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5064?src=pr&el=h1) Report
> Merging [#5064](https://codecov.io/gh/huggingface/transformers/pull/5064?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/d5477baf7d87b9bdad386f2f317732b85277b06b&el=desc) will **decrease** coverage by `0.05%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5064/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5064?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #5064      +/-   ##
==========================================
- Coverage   77.41%   77.36%   -0.06%     
==========================================
  Files         130      130              
  Lines       22023    22023              
==========================================
- Hits        17050    17037      -13     
- Misses       4973     4986      +13     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5064?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_xlnet.py](https://codecov.io/gh/huggingface/transformers/pull/5064/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ194bG5ldC5weQ==) | `76.35% <0.00%> (-2.30%)` | :arrow_down: |
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5064/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `75.00% <0.00%> (-0.41%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5064/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `87.57% <0.00%> (+0.31%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5064?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5064?src=pr&el=footer). Last update [d5477ba...d7a3d5d](https://codecov.io/gh/huggingface/transformers/pull/5064?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
updated hans eval instructions
Author: prajjwal1
Pull request: yes

I've updated the info regarding how hans evaluation can be carried out. 

I've also renamed `run_hans.py` to `test_hans.py` to restore the previous file convention and also indicate that HANS only supports evaluation.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5083?src=pr&el=h1) Report
> Merging [#5083](https://codecov.io/gh/huggingface/transformers/pull/5083?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/7291ea0bff57a017e71b1ea8ec01ff19da298bf0&el=desc) will **decrease** coverage by `0.39%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5083/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5083?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #5083      +/-   ##
==========================================
- Coverage   77.24%   76.84%   -0.40%     
==========================================
  Files         133      133              
  Lines       22146    22146              
==========================================
- Hits        17107    17019      -88     
- Misses       5039     5127      +88     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5083?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_xlm.py](https://codecov.io/gh/huggingface/transformers/pull/5083/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl94bG0ucHk=) | `56.09% <0.00%> (-19.76%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5083/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `85.71% <0.00%> (-0.94%)` | :arrow_down: |
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5083/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `90.61% <0.00%> (-0.12%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5083?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5083?src=pr&el=footer). Last update [7291ea0...b485684](https://codecov.io/gh/huggingface/transformers/pull/5083?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Add header and fix command
Author: sgugger
Pull request: yes

This finishes to fix #4742

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5082?src=pr&el=h1) Report
> Merging [#5082](https://codecov.io/gh/huggingface/transformers/pull/5082?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/7291ea0bff57a017e71b1ea8ec01ff19da298bf0&el=desc) will **increase** coverage by `0.00%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5082/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5082?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #5082   +/-   ##
=======================================
  Coverage   77.24%   77.25%           
=======================================
  Files         133      133           
  Lines       22146    22146           
=======================================
+ Hits        17107    17108    +1     
+ Misses       5039     5038    -1     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5082?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5082/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `90.61% <0.00%> (-0.12%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5082/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `86.95% <0.00%> (+0.31%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5082?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5082?src=pr&el=footer). Last update [7291ea0...cc54022](https://codecov.io/gh/huggingface/transformers/pull/5082?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


LGTM
Make default_data_collator more flexible and deprecate old behavior
Author: sgugger
Pull request: yes

This PR does two things:

- avoid breaking changes when people had a custom `DataCollator` with a `collate_batch` method (there is still the breaking change with `DataCollator` not being a class anymore)
- makes `default_data_collator` more flexible by handling dicts on top of `InputExamples` classes coming from our examples

This fixes #5049

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5060?src=pr&el=h1) Report
> Merging [#5060](https://codecov.io/gh/huggingface/transformers/pull/5060?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/d5477baf7d87b9bdad386f2f317732b85277b06b&el=desc) will **increase** coverage by `0.01%`.
> The diff coverage is `84.21%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5060/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5060?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #5060      +/-   ##
==========================================
+ Coverage   77.41%   77.43%   +0.01%     
==========================================
  Files         130      130              
  Lines       22023    22029       +6     
==========================================
+ Hits        17050    17059       +9     
+ Misses       4973     4970       -3     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5060?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/trainer.py](https://codecov.io/gh/huggingface/transformers/pull/5060/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90cmFpbmVyLnB5) | `39.62% <50.00%> (+0.09%)` | :arrow_up: |
| [src/transformers/data/data\_collator.py](https://codecov.io/gh/huggingface/transformers/pull/5060/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9kYXRhL2RhdGFfY29sbGF0b3IucHk=) | `98.33% <93.33%> (+8.67%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5060?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5060?src=pr&el=footer). Last update [d5477ba...857975c](https://codecov.io/gh/huggingface/transformers/pull/5060?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Refactored on @julien-c suggestion and removed the little hack to handle tensors and lists of ints (which was cauding a 4x slowdown on my tests). I just use the first features to test if we have a tensor (then stack) or not (then use torch.tensor).

LGTM!
Why does the T5Tokenizer prepend and '_' to every token?
Author: abhisheknovoic
Pull request: no

I am using the T5Tokenizer as follows:

```
In [119]: tokenizer = T5Tokenizer.from_pretrained('t5-small')

In [120]: input_ids = tokenizer.encode_plus('I love my dog', return_tensors='pt')

In [121]: [tokenizer.convert_ids_to_tokens([ele]) for ele in input_ids['input_ids'][0]]
Out[121]: [['▁I'], ['▁love'], ['▁my'], ['▁dog']]
```
I even tried another one as follows:

```
In [129]: [tokenizer.convert_ids_to_tokens([ele]) for ele in input_ids['input_ids'][0]]
Out[129]: [['▁I'], ['▁love'], ['▁my'], ['▁school'], ['▁National'], ['x'], ['y'], ['z']]
```
Above, shouldn't 'x', 'y', and 'z' have # prepended to them  as they are part of the same word?

Why do I see an underscore before every token? Is this related to how it is sent inside T5?

Comments:
Hi, the T5 tokenizer is a SentencePiece tokenizer, and that's the way SentencePiece works. This underscore means that it's the start of a word. When it's not the start of a word, it's not prepended by anything.

I think you're thinking of the # symbol because you're used to the BertTokenizer (wordpiece). SentencePiece works a bit differently!
Update installation page and add contributing to the doc
Author: sgugger
Pull request: yes

This PR simplifies the installation page, adds the mention of TF/PT installation with one command only (for CPU) and adds a test that the installation was successful.

The tests mention is moved to CONTRIBUTING. The mention of the tokenization process for OpenAI GPT is moved to that model doc page.

It also adds the contributing guide to the documentation with a simlink (add to fix a few links to make it work).

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5084?src=pr&el=h1) Report
> Merging [#5084](https://codecov.io/gh/huggingface/transformers/pull/5084?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/7291ea0bff57a017e71b1ea8ec01ff19da298bf0&el=desc) will **increase** coverage by `0.02%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5084/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5084?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #5084      +/-   ##
==========================================
+ Coverage   77.24%   77.26%   +0.02%     
==========================================
  Files         133      133              
  Lines       22146    22146              
==========================================
+ Hits        17107    17112       +5     
+ Misses       5039     5034       -5     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5084?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5084/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `90.61% <0.00%> (-0.12%)` | :arrow_down: |
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5084/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `75.40% <0.00%> (+0.40%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5084/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `87.42% <0.00%> (+0.77%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5084?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5084?src=pr&el=footer). Last update [7291ea0...e9e1ea6](https://codecov.io/gh/huggingface/transformers/pull/5084?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
[MarianTokenizer] Switch to sacremoses for punc normalization
Author: sshleifer
Pull request: yes
Labels: name: marian

Attempt at fixing #4491 using @jpcorb20 's solution

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5092?src=pr&el=h1) Report
> Merging [#5092](https://codecov.io/gh/huggingface/transformers/pull/5092?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/20fa82898495f516b221115fc3ef9ec8ebf50b1e&el=desc) will **increase** coverage by `0.07%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5092/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5092?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #5092      +/-   ##
==========================================
+ Coverage   77.21%   77.29%   +0.07%     
==========================================
  Files         133      133              
  Lines       22134    22134              
==========================================
+ Hits        17091    17108      +17     
+ Misses       5043     5026      -17     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5092?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/tokenization\_marian.py](https://codecov.io/gh/huggingface/transformers/pull/5092/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fbWFyaWFuLnB5) | `93.75% <100.00%> (+0.89%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5092/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `87.42% <0.00%> (+0.15%)` | :arrow_up: |
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5092/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `75.40% <0.00%> (+0.40%)` | :arrow_up: |
| [src/transformers/modeling\_xlnet.py](https://codecov.io/gh/huggingface/transformers/pull/5092/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ194bG5ldC5weQ==) | `78.65% <0.00%> (+2.29%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5092?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5092?src=pr&el=footer). Last update [20fa828...e30eaf5](https://codecov.io/gh/huggingface/transformers/pull/5092?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
[style] add pandas to setup.cfg
Author: sshleifer
Pull request: yes

Need to add pandas to setup.cfg. Otherwise, for people that have pandas installed locally isort tries to change `eli5_utils.py` everytime.


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5093?src=pr&el=h1) Report
> Merging [#5093](https://codecov.io/gh/huggingface/transformers/pull/5093?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/90c833870c78bb3d5d807a9a3e6a40d24bf2302b&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5093/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5093?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #5093   +/-   ##
=======================================
  Coverage   77.28%   77.28%           
=======================================
  Files         133      133           
  Lines       22134    22134           
=======================================
  Hits        17107    17107           
  Misses       5027     5027           
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5093?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5093/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `87.26% <0.00%> (-0.16%)` | :arrow_down: |
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5093/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `75.40% <0.00%> (+0.40%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5093?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5093?src=pr&el=footer). Last update [90c8338...eae4841](https://codecov.io/gh/huggingface/transformers/pull/5093?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
model.multiple_choice_head( ) function for Hugging Face GPT2 models
Author: h56cho
Pull request: no

Hello,

I just have a quick question about the ```multiple_choice_head( )``` function for the Hugging Face GPT2 models.

when I execute the codes below, everything runs smoothly:

```python
# get pre-trained GPT2Model and GPT2DoubleHeadsModel
model_gpt2 = GPT2Model.from_pretrained('gpt2', output_hidden_states = True)
model_gpt2DoubleHeadsModel = GPT2DoubleHeadsModel.from_pretrained('gpt2')

# some parts of the codes are missing....

# trying to make a use of the gpt2DoubleHeadsModel.multiple_choice_head() function
hidden_states = model_gpt2(input_ids=input_ids, token_type_ids = token_type_ids)[2][1][:,:,:]
mc_logits = model_gpt2DoubleHeadsModel.multiple_choice_head(hidden_states).detach()
loss_fct = CrossEntropyLoss()
mc_loss = loss_fct(mc_logits.view(1,4), 1)
```
Here, I noticed that I do not need to specify the ```mc_token_ids``` when executing the line ```model_gpt2DoubleHeadsModel.multiple_choice_head(hidden_states)```.

My question is, does the ```multiple_choice_head()``` function from the line ```model_gpt2DoubleHeadsModel.multiple_choice_head(hidden_states)``` automatically take the 
**last token** of my input text sequence as the ```cls_token``` (classification token) that is used by the ```mc_head``` to predict answers for the multiple-choice questions?

Thank you,

Comments:
Method's description says this


        _mc_token_ids (:obj:`torch.LongTensor` of shape :obj:`(batch_size, num_choices)`, `optional`, default to index of the last token of the input)
            Index of the classification token in each input sequence.
            Selected in the range ``[0, input_ids.size(-1) - 1[``._
        
So, the default is the last token of the input.