noamaanMulla-03/transformers-issues huggingface dataset:
RobertaTokenizer different than fairseq for 'world'
Author: sshleifer
Pull request: no
Labels: name: Core: Tokenization

`pip install fairseq`
```
roberta = torch.hub.load('pytorch/fairseq', 'roberta.base')
rt = RobertaTokenizer.from_pretrained('roberta-base')
for ex in ['Hello world', ' Hello world', ' world', 'world', 'Hello', ' Hello']:
    print(f'{ex} fairseq: {roberta.encode(ex).tolist()}, Transformers: {rt.encode(ex,  add_prefix_space=True)}')

>>>

Hello world fairseq: [0, 31414, 232, 2], Transformers: [0, 20920, 232, 2]
 Hello world fairseq: [0, 20920, 232, 2], Transformers: [0, 20920, 232, 2]
 world fairseq: [0, 232, 2], Transformers: [0, 232, 2]
world fairseq: [0, 8331, 2], Transformers: [0, 232, 2]
Hello fairseq: [0, 31414, 2], Transformers: [0, 20920, 2]
 Hello fairseq: [0, 20920, 2], Transformers: [0, 20920, 2]
```
Notice that even the token "world" is different, but results always the same with leading spaces.
Is this related to @joeddav recent work?
h/t @pnpnpn for uncovering:)

Comments:
Reading more, pretty sure we only expect to have the same results as fairseq when the argument to fairseq starts with a space. Closing but would love verification/knowledge !

Yes, this comes from #2778, which changes the default behavior to automatically prepending a space when `add_special_tokens=True` for Roberta, since you want a space after `
`. Can be overriden with `add_prefix_space=False`. This does deviate from fairseq's encode fn, ~~but reflects the behavior of their `fill_mask` [which also prepends a space](https://github.com/pytorch/fairseq/blob/master/fairseq/models/roberta/hub_interface.py#L149).~~ Nvm, fairseq's `fill_mask` function doesn't prepend a space after all. They expect the user to know that they have to prepend a space to get correctly encoded sequences.
Create README.md
Author: BinWang28
Pull request: yes
Labels: name: model card


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2892?src=pr&el=h1) Report
> Merging [#2892](https://codecov.io/gh/huggingface/transformers/pull/2892?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/2ae98336d17fceea7506af9880b862b6252a38f6?src=pr&el=desc) will **decrease** coverage by `1.07%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2892/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2892?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #2892      +/-   ##
==========================================
- Coverage   75.06%   73.98%   -1.08%     
==========================================
  Files          94       94              
  Lines       15288    15288              
==========================================
- Hits        11476    11311     -165     
- Misses       3812     3977     +165
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2892?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_pytorch\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/2892/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9weXRvcmNoX3V0aWxzLnB5) | `8.72% <0%> (-81.21%)` | :arrow_down: |
| [src/transformers/modeling\_roberta.py](https://codecov.io/gh/huggingface/transformers/pull/2892/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19yb2JlcnRhLnB5) | `85.91% <0%> (-9.86%)` | :arrow_down: |
| [src/transformers/modeling\_xlnet.py](https://codecov.io/gh/huggingface/transformers/pull/2892/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ194bG5ldC5weQ==) | `70.86% <0%> (-2.3%)` | :arrow_down: |
| [src/transformers/modeling\_ctrl.py](https://codecov.io/gh/huggingface/transformers/pull/2892/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19jdHJsLnB5) | `92.07% <0%> (-2.21%)` | :arrow_down: |
| [src/transformers/modeling\_openai.py](https://codecov.io/gh/huggingface/transformers/pull/2892/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19vcGVuYWkucHk=) | `80.2% <0%> (-1.35%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2892?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2892?src=pr&el=footer). Last update [2ae9833...25c0467](https://codecov.io/gh/huggingface/transformers/pull/2892?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Thanks for sharing @BinWang28!

[model page](https://huggingface.co/binwang/xlnet-base-cased)
RobertaTokenizer runs slowly after add _tokens
Author: fatmelon
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
Hi, I use RobertaTokenizer like this:  
```python
tokenizer = RobertaTokenizer.from_pretrained(FLAGS.vocab_file, do_lower_case=FLAGS.do_lower_case)
tokenizer.add_tokens([x.strip() for x in open('add_tokens.txt').readlines()])
```  
There are about 200 words in `add_tokens.txt`. I tested it on 300 sample datasets, and 250% more time after using `add_tokens.txt`. Is there any way to optimize it？

Comments:
Hi, I've done a short study and I confirm the behavior you see.
I've proposed a simple PR attached that gives interesting results and quite important speed improvement in any case.
To be discussed!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Model I am using Roberta
Author: nrjvarshney
Pull request: no

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...):

Language I am using the model on (English, Chinese ...):

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1.
2.
3.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version:
- Platform:
- Python version:
- PyTorch version (GPU?):
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?:


Comments:
Override build_inputs_with_special_tokens for fast tokenizers
Author: mfuntowicz
Pull request: yes

Signed-off-by: Morgan Funtowicz <morgan@huggingface.co>

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2912?src=pr&el=h1) Report
> Merging [#2912](https://codecov.io/gh/huggingface/transformers/pull/2912?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/59c23ad9c931ac4fe719abeb3c3851df046ef3a6?src=pr&el=desc) will **increase** coverage by `0.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2912/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2912?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #2912      +/-   ##
==========================================
+ Coverage    75.3%   75.32%   +0.01%     
==========================================
  Files          94       94              
  Lines       15424    15438      +14     
==========================================
+ Hits        11615    11628      +13     
- Misses       3809     3810       +1
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2912?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/tokenization\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/2912/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fdXRpbHMucHk=) | `90.14% <100%> (+0.05%)` | :arrow_up: |
| [src/transformers/tokenization\_bert.py](https://codecov.io/gh/huggingface/transformers/pull/2912/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fYmVydC5weQ==) | `96.99% <100%> (+0.06%)` | :arrow_up: |
| [src/transformers/tokenization\_roberta.py](https://codecov.io/gh/huggingface/transformers/pull/2912/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fcm9iZXJ0YS5weQ==) | `100% <100%> (ø)` | :arrow_up: |
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/2912/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `61.15% <0%> (-0.18%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2912?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2912?src=pr&el=footer). Last update [59c23ad...3b7752b](https://codecov.io/gh/huggingface/transformers/pull/2912?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Fast Tokenizers save pretrained should return the list of generated file paths.
Author: mfuntowicz
Pull request: yes


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2918?src=pr&el=h1) Report
> Merging [#2918](https://codecov.io/gh/huggingface/transformers/pull/2918?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/2708b44ee9c151a2cdb84620d295c997af6fa7f0?src=pr&el=desc) will **increase** coverage by `0.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2918/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2918?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #2918      +/-   ##
==========================================
+ Coverage   75.33%   75.35%   +0.01%     
==========================================
  Files          94       94              
  Lines       15444    15444              
==========================================
+ Hits        11635    11638       +3     
+ Misses       3809     3806       -3
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2918?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/tokenization\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/2918/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fdXRpbHMucHk=) | `90.58% <100%> (+0.44%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2918?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2918?src=pr&el=footer). Last update [2708b44...c10fcae](https://codecov.io/gh/huggingface/transformers/pull/2918?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Attention values occasionally exceed 1 in BertModel
Author: Akella17
Pull request: no
Labels: name: wontfix

name: Core: Modeling

```Python
outputs = self.model(x, attention_mask = x_mask)  # Models outputs are now tuples
print(outputs[2][-1].max())
print((outputs[2][-1]>1).sum().item()) # Number of attention values > 1
print((outputs[2][-1]>-1).sum().item()) # Total number of attention values
```
```
tensor(1.0750, device='cuda:7', grad_fn=<MaxBackward1>)
1545
480000
```

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


@thomwolf Did someone looking into this issue?

No and I'm afraid we don't really have the bandwidth for that at the moment.


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


I think this is related to a feature of Dropout applied to the attention values. Dropout will scale up the values that are not zeroed out which causes the problem you described. See this [pytorch/issues/5752](https://github.com/pytorch/pytorch/issues/5752). 
Setting the model to eval mode should produce normal attention values.
What does ## mean in the bert vocab?
Author: NapsterLong
Pull request: no
Labels: name: Core: Tokenization

What does ## mean in the bert vocab?

some words are starts with ##, such as ##a ##m ##er ##h ，I don't quite understand。

Comments:
This question is better suited for [Stack Overflow](https://stackoverflow.com/). Please ask similar questions there in the future.

It indicates that the token is a subword unit, i.e. part of a larger word. For instance, the word "potatoes" might be tokenised as "po, ##ta, ##toes". If you want to learn more about this kind of tokenisation, I suggest you read up on byte-pair encoding and the like.
make RobertaForMaskedLM implementation identical to fairseq
Author: BramVanroy
Pull request: yes

closes https://github.com/huggingface/transformers/issues/1874

The implementation of RoBERTa in `transformers` differs from the original implementation in [fairseq](https://github.com/pytorch/fairseq/tree/master/fairseq/models/roberta), as results showed (cf. https://github.com/huggingface/transformers/issues/1874). I have documented my findings here https://github.com/huggingface/transformers/issues/1874#issuecomment-588359143 and made the corresponding changes accordingly in this PR.

Someone should check, however, that removing `get_output_embeddings()` does not have any adverse side-effects.

Comments:
Awesome. Could you check if the existing @slow tests break for Roberta, and add a new one that hardcodes the fairseq logits from your example and makes sure we also return them. Trying to avoid accidental breakage. Thanks again! 

@sshleifer Not sure how I would hardcode a tensor of size 1, 12, 50265. Can I just add a small pickled file to the test instead?

# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2913?src=pr&el=h1) Report
> Merging [#2913](https://codecov.io/gh/huggingface/transformers/pull/2913?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/59c23ad9c931ac4fe719abeb3c3851df046ef3a6?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `85.71%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2913/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2913?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #2913      +/-   ##
==========================================
- Coverage    75.3%   75.29%   -0.01%     
==========================================
  Files          94       94              
  Lines       15424    15424              
==========================================
- Hits        11615    11614       -1     
- Misses       3809     3810       +1
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2913?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_roberta.py](https://codecov.io/gh/huggingface/transformers/pull/2913/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19yb2JlcnRhLnB5) | `95.3% <85.71%> (-0.47%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2913?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2913?src=pr&el=footer). Last update [59c23ad...51514a4](https://codecov.io/gh/huggingface/transformers/pull/2913?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Sorry for the close, had to do some rebasing.
Fix spell: EsperBERTo, not EspertBERTo
Author: vochicong
Pull request: yes

This misspelling almost drove me crazy :)

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2931?src=pr&el=h1) Report
> Merging [#2931](https://codecov.io/gh/huggingface/transformers/pull/2931?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/d490b5d5003654f104af3abd0556e598335b5650?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2931/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2931?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #2931   +/-   ##
=======================================
  Coverage   75.35%   75.35%           
=======================================
  Files          94       94           
  Lines       15444    15444           
=======================================
  Hits        11638    11638           
  Misses       3806     3806
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2931?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2931?src=pr&el=footer). Last update [d490b5d...b5607ba](https://codecov.io/gh/huggingface/transformers/pull/2931?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


You're right, thanks for fixing!
Small run_squad nit: eliminate trailing "_" in "best_predictions_.json" when no prefix
Author: mfeblowitz
Pull request: no
Labels: name: wontfix

## 🐛 Bug

<!-- Important information -->
Prior convention for tf-based run_squad is to output best predictions in an nbest_predictions.json file. Now with the new convention of including a "prefix" in the generation of potentially many nbest files, in cases where there's not prefix, the name becomes nbest_predictions_.json (nothing after the "_"). Might be more backward compatible to include the rightmost "_" only when there's a prefix after it.

The same thing applies to predictions and null_odds, ...

Model I am using (Bert, XLNet....): albert

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* [X] the official example scripts: run_squad.py
* [ ] my own modified scripts: (give details)

The tasks I am working on is:
* [X] an official GLUE/SQUaD task: SQuAD2
* [ ] my own task or dataset: (give details)

## To Reproduce

Steps to reproduce the behavior:

1. look at output of a successful run of run_squad.py 
<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->


## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->

## Environment

* OS:
* Python version:
* PyTorch version:
* PyTorch Transformers version (or branch): v2.1.1, from source
* Using GPU ?
* Distributed of parallel setup ?
* Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


As it's only a cosmetic change and for the sake of not breaking backward compat over cosmetic issues I'm reluctant to change this.
Add syntax highlighting to the BibTeX in README
Author: bryant1410
Pull request: yes


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2914?src=pr&el=h1) Report
> Merging [#2914](https://codecov.io/gh/huggingface/transformers/pull/2914?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/e67676424191e5935362e5fe7e04b5c317d706a9?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2914/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2914?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #2914   +/-   ##
=======================================
  Coverage   75.32%   75.32%           
=======================================
  Files          94       94           
  Lines       15438    15438           
=======================================
  Hits        11629    11629           
  Misses       3809     3809
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2914?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2914?src=pr&el=footer). Last update [e676764...5574050](https://codecov.io/gh/huggingface/transformers/pull/2914?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
How to train a LM with a custom Dataset?
Author: ohmeow
Pull request: no
Labels: name: Ex: LM (Pretraining)

# ❓ Questions & Help

I'm attempting to build a LM following the tutorial here (https://huggingface.co/blog/how-to-train).

Unfortunately, it is incomplete.  It shows how to create a custom `Dataset` but not how to execute `run_language_modeling.py` so that it is used.  

**Any chance we can get the full script for training the LM, included how to specify our custom dataset?**



Comments:
This is in process of being addressed at huggingface/blog#3

(You'll need to tweak the code of `run_language_modeling.py`, this is not – yet – a code-free tutorial)