noamaanMulla-03/transformers-issues huggingface dataset:
Multiple token IDs for same token
Author: dakshvar22
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

I am using GPT2Tokenizer. I observed that some tokens are duplicated in the vocabulary with a space appended to them in the beginning. For example, there exist two separate tokens - `can` and `<space>can`. Both of them are mapped to different token IDs - `5171` and `460` respectively.

Although, both - 
```
can_token_id = tokenizer.encode(' can', add_special_tokens=False)
```
and
```
can_token_id = tokenizer.encode('can', add_special_tokens=False)
```
return can_token_id as `[5171]`

Is this an intended property of the tokenizer? Is there a mapping available between what token IDs are just a space appended version of other tokenIDs?

Comments:
Okay, I found that the symbol is not `<space>` but this `Ġ`
So there exist two words in the vocab - `can` and `Ġcan`
Why is this so?

Hi! This is indeed an intended property of the tokenizer. The GPT-2 tokenizer is a byte-level BPE that has a sufficient vocabulary size to make the distinction between tokens that are at the beginning of a sentence (not prepended by a space), and those that are in the middle of a sentence (prepended by a space). 

The tokens `the`  and ` the` are therefore encoded differently, however the tokenizer strips the spaces from the sequences it receives as input. You can suppress that behavior by setting the `add_prefix_space` flag to `True`:

```py
tokenizer.encode("the")
# [1169]

tokenizer.encode("the", add_prefix_space=True)
# [262]
```

Concerning your question regarding why the vocabulary displays `can` and `Ġcan`, you can actually see the `Ġcan` as being ` can` (notice the space). The GPT-2 tokenizer converts all spaces/control characters to other tokens. These spaces and control characters could have some unwanted behavior when using a BPE tokenizer (for example if it splits on whitespace).

The space token is switched to `Ġ`. You can see it being done using the `bytes_to_unicode` method in the [tokenization_gpt2.py](https://github.com/huggingface/transformers/blob/master/src/transformers/tokenization_gpt2.py#L63-L85) file.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Create README.md for ktrapeznikov/scibert_scivocab_uncased_squad_v2
Author: ktrapeznikov
Pull request: yes
Labels: name: model card


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3629?src=pr&el=h1) Report
> Merging [#3629](https://codecov.io/gh/huggingface/transformers/pull/3629?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/243e687be6cd701722cce050005a2181e78a08a8&el=desc) will **decrease** coverage by `0.01%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3629/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/3629?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #3629      +/-   ##
==========================================
- Coverage   78.30%   78.28%   -0.02%     
==========================================
  Files         104      104              
  Lines       17627    17627              
==========================================
- Hits        13802    13800       -2     
- Misses       3825     3827       +2     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3629?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3629/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `93.45% <0.00%> (-0.17%)` | :arrow_down: |
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3629/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `91.97% <0.00%> (-0.13%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3629?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3629?src=pr&el=footer). Last update [243e687...e1252ef](https://codecov.io/gh/huggingface/transformers/pull/3629?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Update German Bert model card
Author: Timoeller
Pull request: yes
Labels: name: model card

We changed the vocab to work with run_split_on_punc tokenization.
Now there are much less [UNK] punctuation tokens.

For more details see deepset-ai/FARM/issues/60

Comments:
@Timoeller @tholor LGTM and thanks for linking to the discussion on the FARM repo. 👍

cherry-picked only the relevant change in 4ab8ab4f50baf391612cbc78cfa3f09b7ad0c3ac
wrong parameters order in TFTransfoXLMainLayer _update_mems call
Author: dmytyar
Pull request: no

# 🐛 Bug

## Information

In file src/transformers/modeling_tf_transfo_xl.py present small typo in method parameters call.
In line 491 present method:
    def _update_mems(self, hids, mems, qlen, mlen)
And in line 610 on it's call:
    new_mems = self._update_mems(hids, mems, mlen, qlen)

As you can see qlen and mlen placed in wrong order. In result memory size can raise outside of specified value.

Comments:
Good catch @dmytyar :-)
AdamW requires torch>=1.2.0
Author: carter54
Pull request: no
Labels: name: wontfix

## 🐛 Bug

<!-- Important information -->
AdamW requires torch>=1.2.0, torch < 1.2.0 will cause an importError: cannot import name 'AdamW'
Model I am using (Bert, XLNet....):

Language I am using the model on (English, Chinese....):

The problem arise when using:
* [ ] the official example scripts: (give details)
* [ ] my own modified scripts: (give details)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details)

## To Reproduce

Steps to reproduce the behavior:

1.
2.
3.

<!-- If you have a code sample, error messages, stack traces, please provide it here as well. -->

## Expected behavior

<!-- A clear and concise description of what you expected to happen. -->

## Environment

* OS:
* Python version:
* PyTorch version:
* PyTorch Transformers version (or branch):
* Using GPU ?
* Distributed of parallel setup ?
* Any other relevant information:

## Additional context

<!-- Add any other context about the problem here. -->


Comments:
I don't think that's right. AdamW is implemented in transformers.optimization

https://github.com/huggingface/transformers/blob/82f6abd98aaa691ca0adfe21e85a17dc6f386497/transformers/optimization.py#L107

As far as I can see that does not require anything specific to torch 1.2. _However_, if you are trying to import [AdamW from torch ](https://pytorch.org/docs/stable/optim.html#torch.optim.AdamW), you may indeed be required to use torch 1.2.

I haven't compared the implementation in torch vs. transformers, but I'd go with torch's native implementation if you can and otherwise fallback to transformers' implementation.

sorry, I didn't show the details:
the error is from 29 line in transformers/examples/distillation/distiller.py
from torch.optim import AdamW
this AdamW is imported from torch.optim

does this really need torch >=1.2? I met this problem

my torch version is 1.1

No. Importing AdamW from transformers should work with earlier versions. If you're trying to import it directly from torch, then you'll need 1.2+.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Unstale. 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
How can u make sure that my transformer model should only one GPU, though the serve has multiple GPU cards.
Author: tiru1930
Pull request: no

I have transformer BERT model and I am trying to train on lambda server which has 8 GPU cards, How can u make sure that this model should use only once GPU out of 8, by default , it is using all GPUs. even after setting 
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")


Comments:
You should have used the template, because now we don't have enough information to help you: how are you running the script (torch launch utility? Which command?), which script are you using (your own (give details) or one of the example scripts)?

By default, PyTorch will only use one GPU unless you specify it to go DDP.

Use tags please. Read through this guide. https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks

@BramVanroy  it was my bad, I am in bit hurry so that  I was not able to provide my code path. I have just pushed my code to github. Please check below for path 
[https://github.com/tiru1930/bert_intent_classification](url)

In this PATH I use src/Train.py to train the model.



As you see, you are just wasting your own time and mine when you are _in a hurry_. In the future, take your time to write a good starting posts so that we _want_ to help you and _can_ help you quickly.

In your code, you are calling DataParallel on your mode, which will automatically run your model over multiple GPU's (but under a single process). Remove this line.

https://github.com/tiru1930/bert_intent_classification/blob/master/src/train.py#L80
[Generate, Test] Split generate test function into beam search, no beam search
Author: patrickvonplaten
Pull request: yes

- Clean the generate testing functions 
- Also should fix flaky behaviour of bad_word_tokens test (see #3367 and https://circleci.com/gh/huggingface/transformers/27997?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link) 



Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3601?src=pr&el=h1) Report
> Merging [#3601](https://codecov.io/gh/huggingface/transformers/pull/3601?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/f68d22850ced09bb194b30068ff94ca3409f0879&el=desc) will **decrease** coverage by `0.00%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3601/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/3601?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #3601      +/-   ##
==========================================
- Coverage   78.06%   78.05%   -0.01%     
==========================================
  Files         100      100              
  Lines       17134    17134              
==========================================
- Hits        13375    13374       -1     
- Misses       3759     3760       +1     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3601?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3601/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `93.28% <0.00%> (-0.17%)` | :arrow_down: |
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3601/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `92.10% <0.00%> (ø)` | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3601?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3601?src=pr&el=footer). Last update [f68d228...857e77e](https://codecov.io/gh/huggingface/transformers/pull/3601?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
BioMed Roberta-Base (AllenAI)
Author: kernelmachine
Pull request: yes
Labels: name: model card

This PR includes the model card for Biomed-roberta base, which @kyleclo recently uploaded to allenai's huggingface model repository.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3643?src=pr&el=h1) Report
> Merging [#3643](https://codecov.io/gh/huggingface/transformers/pull/3643?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/1789c7daf1b8013006b0aef6cb1b8f80573031c5&el=desc) will **increase** coverage by `0.94%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3643/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/3643?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #3643      +/-   ##
==========================================
+ Coverage   77.32%   78.26%   +0.94%     
==========================================
  Files         104      104              
  Lines       17628    17628              
==========================================
+ Hits        13630    13796     +166     
+ Misses       3998     3832     -166     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3643?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_openai.py](https://codecov.io/gh/huggingface/transformers/pull/3643/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19vcGVuYWkucHk=) | `81.54% <0.00%> (+1.34%)` | :arrow_up: |
| [src/transformers/modeling\_xlnet.py](https://codecov.io/gh/huggingface/transformers/pull/3643/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ194bG5ldC5weQ==) | `75.77% <0.00%> (+2.29%)` | :arrow_up: |
| [src/transformers/modeling\_ctrl.py](https://codecov.io/gh/huggingface/transformers/pull/3643/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19jdHJsLnB5) | `98.24% <0.00%> (+2.63%)` | :arrow_up: |
| [src/transformers/modeling\_roberta.py](https://codecov.io/gh/huggingface/transformers/pull/3643/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19yb2JlcnRhLnB5) | `95.71% <0.00%> (+10.00%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_pytorch\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3643/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9weXRvcmNoX3V0aWxzLnB5) | `89.93% <0.00%> (+81.20%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3643?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3643?src=pr&el=footer). Last update [1789c7d...1d980d5](https://codecov.io/gh/huggingface/transformers/pull/3643?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Slightly tweaked and merged! [**model page**](https://huggingface.co/allenai/biomed_roberta_base)
Add model card for BERTeus
Author: jjacampos
Pull request: yes
Labels: name: model card

This PR includes the model card for the BERTeus model which has been recently uploaded to the huggingface repository. 

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3649?src=pr&el=h1) Report
> Merging [#3649](https://codecov.io/gh/huggingface/transformers/pull/3649?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/2ee410560e45ae3c619dc1e0b0fc4d257c48e18a&el=desc) will **increase** coverage by `0.00%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3649/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/3649?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #3649   +/-   ##
=======================================
  Coverage   78.28%   78.29%           
=======================================
  Files         104      104           
  Lines       17628    17628           
=======================================
+ Hits        13800    13801    +1     
+ Misses       3828     3827    -1     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3649?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3649/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `93.28% <0.00%> (+0.16%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3649?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3649?src=pr&el=footer). Last update [2ee4105...5b63e6a](https://codecov.io/gh/huggingface/transformers/pull/3649?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Awesome, thanks for sharing – also cc @joeddav 

[**Model page**](https://huggingface.co/ixa-ehu/berteus-base-cased)
Create model card  for NLP4H/ms_bert
Author: MichalMalyska
Pull request: yes
Labels: name: model card


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3662?src=pr&el=h1) Report
> Merging [#3662](https://codecov.io/gh/huggingface/transformers/pull/3662?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/261c4ff4e297e919ba993e1214a805e988bc9e79&el=desc) will **decrease** coverage by `0.04%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3662/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/3662?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #3662      +/-   ##
==========================================
- Coverage   78.29%   78.25%   -0.05%     
==========================================
  Files         104      104              
  Lines       17628    17628              
==========================================
- Hits        13802    13794       -8     
- Misses       3826     3834       +8     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3662?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/tokenization\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3662/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fdXRpbHMucHk=) | `90.65% <0.00%> (-0.84%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3662/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `93.12% <0.00%> (-0.17%)` | :arrow_down: |
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3662/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `92.10% <0.00%> (-0.13%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3662?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3662?src=pr&el=footer). Last update [261c4ff...fccd0c7](https://codecov.io/gh/huggingface/transformers/pull/3662?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


@MichalMalyska This is super interesting, thanks for sharing

[**Model page**](https://huggingface.co/NLP4H/ms_bert)
Add model card
Author: mrm8488
Pull request: yes
Labels: name: model card


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3655?src=pr&el=h1) Report
> Merging [#3655](https://codecov.io/gh/huggingface/transformers/pull/3655?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/39a34cc375ed79d18888464289b83713fc20f7d4&el=desc) will **increase** coverage by `0.00%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3655/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/3655?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #3655   +/-   ##
=======================================
  Coverage   78.29%   78.29%           
=======================================
  Files         104      104           
  Lines       17628    17628           
=======================================
+ Hits        13801    13802    +1     
+ Misses       3827     3826    -1     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3655?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3655/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `93.28% <0.00%> (ø)` | |
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3655/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `92.23% <0.00%> (+0.12%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3655?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3655?src=pr&el=footer). Last update [39a34cc...0ee8e8c](https://codecov.io/gh/huggingface/transformers/pull/3655?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).