noamaanMulla-03/transformers-issues huggingface dataset:
Knowledge distillation from internal representation GPT2
Author: snaik2016
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

I am trying to implement above paper for the GPT2 model. The attention outputs are softmax probabilities as seen from modeling_gpt2.py line (152-162). If KLD loss is computed on these values from teacher and student I am getting negative value of KLD indicating that the inputs are not probability distributions.
The attention output is of  dimensions (bs, nh, sl, sl) nh=12 I am just flattening the output and computing kld loss.

Is my understanding correct.?

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**: 

Comments:
The attention outputs are indeed softmax probabilities and should all be > 0. Could you post a code snippet that reproduces negative attention outputs? Or a code snippet of what exactly you are doing with the attention outputs? 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Problem with using pretrained BertTokenizer for Korean
Author: alirezamshi-zz
Pull request: no
Labels: name: wontfix

name: Core: Tokenization

I have a corpus that contains Korean sentences. Here is the output of a berttokenizer for a token:

`tok = BertTokenizer.from_pretrained('bert-base-multilingual-uncased')`
`tok.convert_tokens_to_ids(tok.tokenize('잘해놨습니다'))`
`[44628, 14840, 97071, 97089, 97104, 13212, 79427]`
`tok = BertTokenizer.from_pretrained('bert-base-multilingual-cased')`
`tok.convert_tokens_to_ids(tok.tokenize('잘해놨습니다'))`
`[100]`
transformers version: 2.4.1

Totally, 'cased' tokenizer produces more 'unknown' token than 'uncased' one for Korean. 
Is it a bug? 

Comments:
Intuitively, I would say that this might not be a bug but a limitation of the size of the vocabulary. In the cased version, all data that the tokenizer is 'trained' on is cased, meaning that there are tokens in the vocabulary that only differs by case (e.g. `Be` and `be`). As a consequence, this may cause the vocabulary to be a lot bigger, leaving less room for other tokens. 

That is just my intuition and not based on any research.

You can try out KoBERT, though: https://github.com/SKTBrain/KoBERT

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Fast tokenizers can't `encode_plus` a list of ids; slow tokenizers can
Author: dirkgr
Pull request: no
Labels: name: wontfix

name: Core: Tokenization

With the slow tokenizers:
```
>>> import transformers
>>> t = transformers.AutoTokenizer.from_pretrained("bert-base-cased", use_fast=False)
>>> t.encode_plus([1000])
{'input_ids': [101, 1000, 102],
 'token_type_ids': [0, 0, 0],
 'attention_mask': [1, 1, 1]}
```

With the fast tokenizers:
```
>>> import transformers
>>> t = transformers.AutoTokenizer.from_pretrained("bert-base-cased", use_fast=True)
>>> t.encode_plus([1000])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/dirkg/anaconda3/envs/allennlp/lib/python3.7/site-packages/transformers/tokenization_utils.py", line 1889, in encode_plus
    **kwargs,
  File "/Users/dirkg/anaconda3/envs/allennlp/lib/python3.7/site-packages/transformers/tokenization_utils.py", line 1815, in batch_encode_plus
    tokens = self._tokenizer.encode(*batch_text_or_text_pairs[0])
  File "/Users/dirkg/anaconda3/envs/allennlp/lib/python3.7/site-packages/tokenizers/implementations/base_tokenizer.py", line 141, in encode
    return self._tokenizer.encode(sequence, pair)
TypeError
```


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
XLM-RoBERTa can't add new tokens.
Author: Pelellone
Pull request: no
Labels: name: wontfix

name: Core: Tokenization

# 🐛 Bug

Model I am using (Bert, XLNet ...): XLM-RoBERTa

## To reproduce
Steps to reproduce the behavior:
 
1. tokenizer = XLMRobertaTokenizer.from_pretrained('xlm-roberta-large', do_lower_case=True)
2. tokenizer.add_tokens(['<a1>', '</a1>', '<a2>', '</a2>'])
3. tokenizer.convert_tokens_to_ids('<a1>')

It always respond 1 as ids for the new tokens.
The problem seems connected to: 
if (
                token != self.unk_token
                and self.convert_tokens_to_ids(token) == self.convert_tokens_to_ids(self.unk_token)
                and token not in to_add_tokens
            ):

in the second condition. 
When it tries to get self.convert_tokens_to_ids(token) always returns 1 instead of 3. 
3 is the id for unk_token.
1 is the id for pad.

Comments:
Can you post a minimal verifiable example that we can just copy-and-paste to try? I guess you don't really try to add empty strings as tokens?

Hi @BramVanroy, no ofc, probably i've pasted the wrong snippet of code. 
If you try to expand the vocabulary of the tokenizer with the following code: 

```
tokenizer = XLMRobertaTokenizer.from_pretrained('xlm-roberta-large', do_lower_case=True)
tokenizer.add_tokens(['[A1]', '[/A1]', '[A2]', '[/A2]']) 
```

the size of tokenizer remains the same. 
Obviously if you try the same identical code with Bert or DistilBert (the ones i'm testing) all works fine. 
All seems connected with the second condition of if-else block: 

```
if (
   token != self.unk_token
   and self.convert_tokens_to_ids(token) == self.convert_tokens_to_ids(self.unk_token)
   and token not in to_add_tokens
):
```
This condition seems returns the wrong id for self.unk_token.
Removing this condition let me add the new tokens and extend the tokenizers.


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
How to add data to  pretrained model.
Author: ynebula
Pull request: no
Labels: name: wontfix

name: Usage

I want to add wiki data to pretrained model.

   i.e., kor wiki data is added to XLMRoberta Model weight.

I think that files in /transformers/templates/adding_a_new_model do

Is it right?

please let me know way to add data.



Comments:
Have a look at the language modeling example:

https://huggingface.co/transformers/examples.html#language-model-training

https://github.com/huggingface/transformers/blob/master/examples/run_language_modeling.py

Thank you for your answering.

I did that^^

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
'utf-8' codec can't decode byte 0x80 in position 229888: invalid start byte
Author: codehunk628
Pull request: no
Labels: name: Usage

os : google colab
from transformers import *
tokenizer = AutoTokenizer.from_pretrained('https://s3-us-west-2.amazonaws.com/ai2-s2-research/scibert/huggingface_pytorch/scibert_scivocab_uncased.tar')
![SCI_BERT](https://user-images.githubusercontent.com/51091425/81477029-d94ddc00-9232-11ea-90ca-827a0e0ecc80.PNG)



Comments:
I don't think you can load .tars directly. Simply use the name as described in [the model page](https://huggingface.co/allenai/scibert_scivocab_uncased), and it should work.

```python
tokenizer = AutoTokenizer.from_pretrained("allenai/scibert_scivocab_uncased")
```

Thank You @BramVanroy .
Its working.
Explanation of the results derived from fine tuning
Author: pentegroom
Pull request: no
Labels: name: wontfix

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->


Hi,

It would be super nice if you can add a visualization util that can show why the model inferred a particular result such that which words of the "sentencex" made it labelled as positive?

Thank you.

Comments:
@gofimofi You might want to look at https://github.com/jessevig/bertviz which is compatible with transformers

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Making past and mems variables have batch size as their first output dimension.
Author: patrickvonplaten
Pull request: no
Labels: name: wontfix

# 🚀 Feature request

And the moment, the variables **past** / **mems** have the shape: 
`(2, batch_size, num_heads, sequence_length, embed_size_per_head)` and `(mem_len, batch_size, embed_size)` , respectively.

meaning that the variable `batch_size` dim is not on first position. Change the variables structure to have `batch_size` on first position.

## Motivation

This might be confusing as all other variables have the `batch_size` dim no first position. 
Being certain that the first dimension is always the `batch_size` would be very helpful for the user. 
Normally `mems` and `past` variables are just used to speed up decoding, and not too much changed by the user (or even looked at) I think, but consistency should be good anyways.

## Your contribution

Changing this for GPT2/CTRL is very straightforward (changing three lines of code), but for xlnet and transfo-xl would probably take a slightly bigger change.



Comments:
Up for discussion @LysandreJik @thomwolf @julien-c 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
[Marian] documentation and AutoModel support
Author: sshleifer
Pull request: yes
Labels: name: marian

- Adds integration tests for en-fr, fr-en.
- Easier bulk conversion
- remove unused pretrained_model_archive_map constant.
- boilerplate to make AutoModelWithLMHead, AutoTokenizer, AutoConfig work.

### Metrics:

For [fr-en test set](https://object.pouta.csc.fi/OPUS-MT-models/fr-en/opus-2020-02-26.test.txt):
- BLEU score from posted translations: 57.4979
- BLEU score from `MarianMTModel.from_pretrained('Helsinki-NLP/opus-mt-fr-en')`: 57.4817
- no perf change for fp16
- can fit batch_size=512 on 16GB card in fp16
- speed: 89s for 5k examples = 56 examples/second


Comments:
### Screenshots of Documentation

![image](https://user-images.githubusercontent.com/6045025/81184705-b8f5f700-8f7e-11ea-896f-9571d542cb32.png)
![image](https://user-images.githubusercontent.com/6045025/81184741-c317f580-8f7e-11ea-96f2-a38c89224f7b.png)

[  x] Shows up in TOC:
![image](https://user-images.githubusercontent.com/6045025/81184776-cd39f400-8f7e-11ea-8420-cf5f34eac1d7.png)


The documentation looks great!

# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4152?src=pr&el=h1) Report
> Merging [#4152](https://codecov.io/gh/huggingface/transformers/pull/4152?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/a01a3fecb4fe203c086d3d7450b76bbcfa035725&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4152/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4152?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #4152   +/-   ##
=======================================
  Coverage   77.52%   77.52%           
=======================================
  Files         120      120           
  Lines       19932    19932           
=======================================
  Hits        15453    15453           
  Misses       4479     4479           
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4152?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4152?src=pr&el=footer). Last update [a01a3fe...a01a3fe](https://codecov.io/gh/huggingface/transformers/pull/4152?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
TPU Trainer's PerDeviceLoader has no len()
Author: caseysackett
Pull request: no

# 🐛 Bug

The new TPU Trainer fails when running on a single TPU core in a colab notebook since the XLA ParallelLoader's  PerDeviceLoader does not implement a length method.

## Information

Model I am using (Bert, XLNet ...): Bert

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

I used a custom script, but I would imagine it's not hard to reproduce using one of the example scripts. The error trace is below, with the relevant line in bold:

`Exception in device=TPU:0: object of type 'PerDeviceLoader' has no len()
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/torch_xla/distributed/xla_multiprocessing.py", line 119, in _start_fn
    fn(gindex, *args)
  File "<ipython-input-6-3aa4c6105066>", line 264, in main
    trainer.train()
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 286, in train
    **t_total = int(len(train_dataloader) // self.args.gradient_accumulation_steps *** self.args.num_train_epochs)
TypeError: object of type 'PerDeviceLoader' has no len()
An exception has occurred, use %tb to see the full traceback.`


## Expected behavior

Based on the conversation in the issue linked below, the PerDeviceLoader does not implement length, so the Trainer would need to be aware of that and get the length another way.

https://github.com/pytorch/xla/issues/1191

## Environment info

- `transformers` version: 2.9.0
- Platform: Linux-4.19.104+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.5.0a0+ab660ae (False)
- Tensorflow version (GPU?): 2.2.0-rc4 (False)
- Using GPU in script?: Using TPU
- Using distributed or parallel set-up in script?: XLA is used by the Trainer, but it's a single core TPU in a colab notebook


Comments:
@jysohn23 (and @dlibenzi) might want to chime in on this but I think this works in xla's nightly from https://github.com/pytorch/xla/pull/1991

(We might still want to improve things but if this is blocking you, please try using the nightly build of xla)

Yeah it's recently been added, so if you could select a newer version in the first cell of your colab notebook (`env-setup.py` script), it should do.

Thanks for the quick response. Confirmed that using the "nightly" build of XLA works, so closing this issue.
A better way to process extended_attention_mask in BertModel.forward()
Author: erikchwang
Pull request: no
Labels: name: Core: Modeling

In the `forward()` method of the `BertModel` (https://huggingface.co/transformers/_modules/transformers/modeling_bert.html#BertModel.forward), the `extended_attention_mask` is processed in the following way: 

> extended_attention_mask = (1.0 - extended_attention_mask) * -10000.0

I know this is to make an additive mask so that the unmasked positions will be unchanged (by adding 0) and the masked positions will be very small (by subtracting 10000). 

But I think it is better to achieve this goal by the following way: 

> extended_attention_mask = torch.log(extended_attention_mask)

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
BartTokenizer and 'bart-large-cnn' out of sync
Author: sshleifer
Pull request: no
Labels: name: wontfix

name: seq2seq

tok.encode('<mask>') -> 52064, but BartForMaskedLM.from_pretrained('bart-large-cnn') does not support a mask token.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
[Help] how to make a constrained text generation
Author: lexmen318
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
What I need is to make a constrained text generation via XLNet or GPT-2:

Input:  No one has the intention of building a wall.
Constraint: the output should include two pre-defined key words: 'No one' and 'construct'.
Expected output(e.g.): No one has the intention, a wall to construct.
(with a text length being predefined).

I found some reference like followings, 
https://awslabs.github.io/sockeye/inference.html#lexical-constraints

but it is too complecate to transfer. Could u give me some advice?

thx a log!


Comments:
Yes, I think the sockeye paper and code is the right place to start even if it may look complicated at first sight. Try to combine it with the `run_generation.py` example.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


> ## ❓ Questions & Help
> What I need is to make a constrained text generation via XLNet or GPT-2:
> 
> Input: No one has the intention of building a wall.
> Constraint: the output should include two pre-defined key words: 'No one' and 'construct'.
> Expected output(e.g.): No one has the intention, a wall to construct.
> (with a text length being predefined).
> 
> I found some reference like followings,
> https://awslabs.github.io/sockeye/inference.html#lexical-constraints
> 
> but it is too complecate to transfer. Could u give me some advice?
> 
> thx a log!

Hi have you successfully implemented this constrained generation method? Thanks a lot!
Fix TF optimization classes and apply the changes in the NER TF script
Author: jplu
Pull request: yes

This PR fix the AdamW and GradientAccumulator optimization classes for Tensorflow. Furthermore, we apply these new changes in the NER TF script.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3907?src=pr&el=h1) Report
> Merging [#3907](https://codecov.io/gh/huggingface/transformers/pull/3907?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/cb3c2212c79d7ff0a4a4e84c3db48371ecc1c15d&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3907/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/3907?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #3907   +/-   ##
=======================================
  Coverage   78.45%   78.45%           
=======================================
  Files         111      111           
  Lines       18521    18521           
=======================================
  Hits        14531    14531           
  Misses       3990     3990           
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3907?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3907?src=pr&el=footer). Last update [cb3c221...f75b213](https://codecov.io/gh/huggingface/transformers/pull/3907?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Add TF2 version of FlauBERT
Author: jplu
Pull request: yes

Hello,

I worked today to add the new FlauBERT model in TF2 version. Translated models are available in:
```
jplu/tf-flaubert-base-cased
jplu/tf-flaubert-large-cased
jplu/tf-flaubert-small-cased
jplu/tf-flaubert-base-uncased
```

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2700?src=pr&el=h1) Report
> :exclamation: No coverage uploaded for pull request base (`master@bae644c`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit).
> The diff coverage is `75%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2700/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2700?src=pr&el=tree)

```diff
@@            Coverage Diff            @@
##             master    #2700   +/-   ##
=========================================
  Coverage          ?   73.78%           
=========================================
  Files             ?       93           
  Lines             ?    15351           
  Branches          ?        0           
=========================================
  Hits              ?    11326           
  Misses            ?     4025           
  Partials          ?        0
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2700?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_tf\_flaubert.py](https://codecov.io/gh/huggingface/transformers/pull/2700/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9mbGF1YmVydC5weQ==) | `21.32% <75%> (ø)` | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2700?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2700?src=pr&el=footer). Last update [bae644c...a39b8de](https://codecov.io/gh/huggingface/transformers/pull/2700?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Did we mean to delete camembert imports @LysandreJik . That's why the tests on HEAD are breaking afaict

I think they were a duplicate