noamaanMulla-03/transformers-issues huggingface dataset:
Max encoding length + corresponding tests
Author: LysandreJik
Pull request: yes

The encoding function eases the encoding of sequences across tokenizers. The addition of the `head_mask` return further removes the pressure on the user to manually check the added special tokens. 

There is currently no easy method to truncate the encoded  sequences while keeping the special tokens intact. This PR aims to change this by providing a `max_length` flag to be passed to the encoding function. This flag works even when no special tokens are involved (e.g. for GPT-2).

The second sequence is truncated while the first stays intact. If the first sequence is longer than the specified maximum length, a warning is sent and no sequences are truncated.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1252?src=pr&el=h1) Report
> Merging [#1252](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1252?src=pr&el=desc) into [glue-example](https://codecov.io/gh/huggingface/pytorch-transformers/commit/5583711822f79d8b3b7e7ba2560748cc0cf5654f?src=pr&el=desc) will **increase** coverage by `0.08%`.
> The diff coverage is `90.9%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1252/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1252?src=pr&el=tree)

```diff
@@               Coverage Diff               @@
##           glue-example   #1252      +/-   ##
===============================================
+ Coverage         81.32%   81.4%   +0.08%     
===============================================
  Files                57      57              
  Lines              8074    8104      +30     
===============================================
+ Hits               6566    6597      +31     
+ Misses             1508    1507       -1
```


| [Impacted Files](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1252?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...h\_transformers/tests/tokenization\_tests\_commons.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1252/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvdGVzdHMvdG9rZW5pemF0aW9uX3Rlc3RzX2NvbW1vbnMucHk=) | `100% <100%> (ø)` | :arrow_up: |
| [pytorch\_transformers/tokenization\_utils.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1252/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvdG9rZW5pemF0aW9uX3V0aWxzLnB5) | `90.02% <80%> (+0.57%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1252?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1252?src=pr&el=footer). Last update [5583711...a804892](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1252?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Fix encode plus
Author: LysandreJik
Pull request: yes

Fixing the tensor creation in encode_plus

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2148?src=pr&el=h1) Report
> Merging [#2148](https://codecov.io/gh/huggingface/transformers/pull/2148?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/030faccb8d45be9bdd2b4b80ff26f36dc41f622a?src=pr&el=desc) will **decrease** coverage by `0.02%`.
> The diff coverage is `30.76%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2148/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2148?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #2148      +/-   ##
==========================================
- Coverage   80.07%   80.05%   -0.03%     
==========================================
  Files         112      112              
  Lines       16866    16868       +2     
==========================================
- Hits        13505    13503       -2     
- Misses       3361     3365       +4
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2148?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [transformers/tokenization\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/2148/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL3Rva2VuaXphdGlvbl91dGlscy5weQ==) | `90.22% <30.76%> (-0.8%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2148?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2148?src=pr&el=footer). Last update [030facc...3d57c51](https://codecov.io/gh/huggingface/transformers/pull/2148?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Great! Nice catch @LysandreJik
GLUE on TPU
Author: LysandreJik
Pull request: yes

This takes advantage of the pytorch 1.3 XLA implementation to fine-tune GLUE on a TPU.

MRPC fine-tuning in 3 epochs + evaluation takes a total of 6 minutes and 30 seconds.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1488?src=pr&el=h1) Report
> Merging [#1488](https://codecov.io/gh/huggingface/transformers/pull/1488?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/f382a8decda82062bb6911f05b646f404eacfdd4?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1488/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1488?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #1488   +/-   ##
=======================================
  Coverage   85.59%   85.59%           
=======================================
  Files          91       91           
  Lines       13526    13526           
=======================================
  Hits        11578    11578           
  Misses       1948     1948
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1488?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1488?src=pr&el=footer). Last update [f382a8d...639f4b7](https://codecov.io/gh/huggingface/transformers/pull/1488?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


LGTM!

It is possible to save/load when on TPU, just move them to CPU and save them:

https://github.com/pytorch/xla/blob/master/API_GUIDE.md#saving-and-loading-xla-tensors
Pytorch/TF2 determinism
Author: LysandreJik
Pull request: yes

Check to see if the models have the same results when in eval mode (pt) or when training=False (tf)

Comments:
# [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1327?src=pr&el=h1) Report
> Merging [#1327](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1327?src=pr&el=desc) into [tf2](https://codecov.io/gh/huggingface/pytorch-transformers/commit/128bdd4c3549e2a1401af87493ff6be467c79c14?src=pr&el=desc) will **increase** coverage by `0.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1327/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1327?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##              tf2    #1327      +/-   ##
==========================================
+ Coverage   85.99%   86.01%   +0.01%     
==========================================
  Files          79       79              
  Lines       12028    12041      +13     
==========================================
+ Hits        10344    10357      +13     
  Misses       1684     1684
```


| [Impacted Files](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1327?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pytorch\_transformers/tests/modeling\_common\_test.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1327/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvdGVzdHMvbW9kZWxpbmdfY29tbW9uX3Rlc3QucHk=) | `74.01% <100%> (+0.4%)` | :arrow_up: |
| [...orch\_transformers/tests/modeling\_tf\_common\_test.py](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1327/diff?src=pr&el=tree#diff-cHl0b3JjaF90cmFuc2Zvcm1lcnMvdGVzdHMvbW9kZWxpbmdfdGZfY29tbW9uX3Rlc3QucHk=) | `95% <100%> (+0.26%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1327?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1327?src=pr&el=footer). Last update [128bdd4...1761d20](https://codecov.io/gh/huggingface/pytorch-transformers/pull/1327?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Yes, good!
How to add traditional features for transformers?
Author: zysNLP
Pull request: no

## ❓ Questions & Help

I don't know whether models like Bert can capture some traditional features or not, such as the number of nouns in a sentence, the number of words in a sentence, etc. But I want to add these features to Bert or xlnet for training. Can anyone help me?


Comments:
What do you mean "for training"? Training what? I imagine that you wish to do this for a downstream task. In that case you can just use BERT for feature extraction and use its features in another system. I suppose one could also fine-tune the model during training where you somewhere inject other features into the model, but you'll have to think carefully where and how you optimize the model (one optimizer for finetuning and training newly injected parameters, multiple optimizers, and so on).

I'd say you could start with feature extraction and adding other features. 

As correctly said by @BramVanroy , you can see some examples of this approach [here](https://jalammar.github.io/a-visual-guide-to-using-bert-for-the-first-time/).

> ##  Questions & Help
> I don't know whether models like Bert can capture some traditional features or not, such as the number of nouns in a sentence, the number of words in a sentence, etc. But I want to add these features to Bert or xlnet for training. Can anyone help me?

Hi @TheEdoardo93 , I read the approach you told me. But I still have some confusing, since the model could output a vector of 768 for every sentence. Where I concat my feature like "the number of nouns in a sentence"? Just the end line of the vector and normalized all of them to 1? Thank you.

`BERT` gives you a feature vector of size 768 for each sentence. Besides this, you can add _N_ features to this vector in order to have at the end a vector of size 768 + _N_, in which the N features have been chosen by you, e.g. number of nouns in a sentence.
So, from 0 to 767 you have the features extracted with `BERT` model, and from 768 to 768+_N_-1 you have your **custom** features extracted by you.

N.B: remember that it's very important in Machine Learning to have the range for all features the same (or quite the same), in order to not give more importance to some rather than others. For this task, you can use e.g. StandardScaler or MinMaxScaler from [Scikit-learn](https://scikit-learn.org/stable/) or a custom scaler implemented by you. 


> Hi @TheEdoardo93 , I read the approach you told me. But I still have some confusing, since the model could output a vector of 768 for every sentence. Where I concat my feature like "the number of nouns in a sentence"? Just the end line of the vector and normalized all of them to 1? Thank you.

@TheEdoardo93 Thank you very much !
Should I always use bert as a teacher to distillation distilbert as a student?
Author: graykode
Pull request: no

## ❓ Questions & Help
Should I always use bert as a teacher to distill distilbert as a student?
Is it fine RoBERTa model as a teacher to distill [distilbert](https://github.com/huggingface/transformers/blob/master/transformers/modeling_distilbert.py)?
I assume roberta and distilbert use the same tokenizer and dataloader method.

Comments:
Hello @graykode 
You can use whichever teacher you want, however in the method we propose, you need to make sure that the vocabularies match (knowledge distillation loss is applied to the distributions over the vocabulary).
Victor

Thanks for your advice! I will close this issue
GPT2 attention mask and output masking
Author: anandhperumal
Pull request: no

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

I have couple of questions:
1. In the original gpt2 they didn't pad the sequence, so they didn't need a  attention mask, but in other cases where we our input sequence is small and we pad the input, don't we need a attention mask?
2. I have padded the labels in the left with -1, in the cost function how do I skip the padded elements in labels? and same for logits how do I skip the padded elements ? 

Comments:
Closing this because I found my answer

Hi, would you mind sharing what the answer you found is? Thank you so much!

Sorry for the delay. Gpt2 was trained as a CLM model with a fixed block size of data. So there was no need for attention mask. (That is what I understood).
Does run_lm_finetuning.py finetune the entire BERT / Xlnet architecture
Author: karandesaiii
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
1. When finetuning on data without a task, i.e. **unsupervised finetuning** by running the **run_lm_finetuning.py** script, does the code finetune all the weight layers of the model or just finetunes by adding an extra layer over the top of the architecture? 

2. The correct way to do this is to first finetune the top n layers for some epochs and then finetune all the layers. Is this how the finetuning takes place through run_lm_finetuning.py ? 
I think answer is no. 

I went through the run_lm_finetuning.py code and it seems that the entire model is getting finetuned from the start.

Still having doubt on this as I have little experience with pytorch.

Comments:
1) Yes, the entire model is fine-tuned.

2) We follow the fine-tuning that takes place in the BERT paper:
    > All of the parameters of BERT and W are fine-tuned jointly to maximize the log-probability of the correct label.

Thanks.


Why would _The correct way to do this is to first finetune the top n layers for some epochs and then finetune all the layers._ be the "one and only correct way to do this"? I've seen this being done to finetune VGG19 (IR), for different tasks, but I haven't see papers showing that one technique will result in better performance than others. Do you have a paper reference? As @LysandreJik indicates, the BERT paper indicates what the authors thing is the most efficient approach.

Follow up on this thread: in practice, if we have a small training set, would it make sense to only finetune the top n layers?
In that case, do we want to consider adding an option that controls the number of trainable layers?

You can also consider replacing the classification head first if you find a pre-trained model suiting your task..

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
corrected documentation for past tensor shape for ctrl and gpt2 model
Author: patrickvonplaten
Pull request: yes

fix issue #1904

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2077?src=pr&el=h1) Report
> Merging [#2077](https://codecov.io/gh/huggingface/transformers/pull/2077?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/35ff345fc9df9e777b27903f11fa213e4052595b?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2077/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2077?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #2077   +/-   ##
=======================================
  Coverage   83.16%   83.16%           
=======================================
  Files         109      109           
  Lines       15858    15858           
=======================================
  Hits        13188    13188           
  Misses       2670     2670
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2077?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [transformers/modeling\_ctrl.py](https://codecov.io/gh/huggingface/transformers/pull/2077/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX2N0cmwucHk=) | `96.47% <ø> (ø)` | :arrow_up: |
| [transformers/modeling\_tf\_ctrl.py](https://codecov.io/gh/huggingface/transformers/pull/2077/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX3RmX2N0cmwucHk=) | `97.86% <ø> (ø)` | :arrow_up: |
| [transformers/modeling\_gpt2.py](https://codecov.io/gh/huggingface/transformers/pull/2077/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX2dwdDIucHk=) | `84.44% <ø> (ø)` | :arrow_up: |
| [transformers/modeling\_tf\_gpt2.py](https://codecov.io/gh/huggingface/transformers/pull/2077/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX3RmX2dwdDIucHk=) | `94.75% <ø> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2077?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2077?src=pr&el=footer). Last update [35ff345...d0383e4](https://codecov.io/gh/huggingface/transformers/pull/2077?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


LGTM, merging!
[REVIEW] Updated comments in run_lm_finetuning.py
Author: TheEdoardo93
Pull request: yes

I've added DistilBERT and CamemBERT models in the description of the models that can be used for fine-tuning a LM model on a custom dataset.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2225?src=pr&el=h1) Report
> Merging [#2225](https://codecov.io/gh/huggingface/transformers/pull/2225?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/8efc6dd544bf1a30d99d4b5abfc5e214699eab2b?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2225/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2225?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #2225   +/-   ##
=======================================
  Coverage   81.47%   81.47%           
=======================================
  Files         122      122           
  Lines       18344    18344           
=======================================
  Hits        14946    14946           
  Misses       3398     3398
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2225?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2225?src=pr&el=footer). Last update [8efc6dd...035bfd9](https://codecov.io/gh/huggingface/transformers/pull/2225?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
corrected typo in example for t5 model input argument
Author: patrickvonplaten
Pull request: yes

For the T5Model the argument name of the input has to be specified explicitly since the forward function is defined as
`def forward(self, **kwargs):`
and can therefore only handle keyworded arguments such as `input_ids=inputs_ids`.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2218?src=pr&el=h1) Report
> Merging [#2218](https://codecov.io/gh/huggingface/transformers/pull/2218?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/8efc6dd544bf1a30d99d4b5abfc5e214699eab2b?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2218/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2218?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #2218   +/-   ##
=======================================
  Coverage   81.47%   81.47%           
=======================================
  Files         122      122           
  Lines       18344    18344           
=======================================
  Hits        14946    14946           
  Misses       3398     3398
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2218?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [transformers/modeling\_tf\_t5.py](https://codecov.io/gh/huggingface/transformers/pull/2218/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX3RmX3Q1LnB5) | `96.55% <ø> (ø)` | :arrow_up: |
| [transformers/modeling\_t5.py](https://codecov.io/gh/huggingface/transformers/pull/2218/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX3Q1LnB5) | `81.22% <ø> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2218?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2218?src=pr&el=footer). Last update [8efc6dd...e280aa8](https://codecov.io/gh/huggingface/transformers/pull/2218?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Indeed, nice catch!