noamaanMulla-03/transformers-issues huggingface dataset:
Repetition penalty work falsely in case the logit of the token is negativ
Author: patrickvonplaten
Pull request: no

## 🐛 Bug

<!-- Important information -->

Model I am using (LMHeadModels; distilgpt2 in this example but holds true for all LMHeadModels):

Language I am using the model on English:

The problem arise when using:
* [ ] the official example scripts: (give details)
* [x] my own modified scripts: (give details)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: language genaration

## To Reproduce

Run the following code:

```
input_sentence = 'The dog'
tokenizer = AutoTokenizer.from_pretrained('distilgpt2')
model = AutoModelWithLMHead.from_pretrained('distilgpt2')
input_ids = torch.tensor(tokenizer.encode(input_sentence)).unsqueeze(0)
outputs = model.generate(input_ids=input_ids, do_sample=True, bos_token_id=tokenizer.bos_token_id, eos_token_ids=tokenizer.eos_token_id, pad_token_id=tokenizer.pad_token_id, repetition_penalty=1.5)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

## Expected behavior
Output: 
`"The dog dog dog dog dog dog dog dog dog dog dog dog dog dog dog dog dog dog"`

In the output, the word dog is repeated multiple times. It can be noticed that the higher the `repetition_penalty`, the more likely already occurring words are to be repeated. Thus, the penalty achieves exactly the opposite of what it is supposed to do.

## Environment

* OS: Linux
* Python version: 3.6.8
* PyTorch version: 1.2.0
* PyTorch Transformers version (or branch): main branch v.2.3.0
* Using GPU ? No
* Distributed of parallel setup ? No
* Any other relevant information:

## Additional context

The reason for this behavior can be understood when looking at line https://github.com/huggingface/transformers/blob/81db12c3ba0c2067f43c4a63edf5e45f54161042/src/transformers/modeling_utils.py#L731 :

If the logit `next_token_logits[i, previous_tokens]` is < 0, then dividing by a number > 1 is actually going to increase the probability of sampling that token the next time instead of reducing it.


Comments:
Propsed fix in PR #2303
run_squad.py for SQuAD2.0 have bad f1 score
Author: WenTingTseng
Pull request: no

why I use run_squad.py for SQuAD2.0 have bad f1 score 43.638
The noanser_f1=0.0 it look like do not deal with not answerable situation
I don't have change anything
I just run like this

python3 run_squad.py \
  --model_type bert \
  --model_name_or_path bert-base-cased \
  --do_train \
  --do_eval \
  --do_lower_case \
  --train_file /share/nas165/Wendy/transformers/examples/tests_samples/SQUAD/train-v2.0.json \
  --predict_file /share/nas165/Wendy/transformers/examples/tests_samples/SQUAD/dev-v2.0.json \
  --per_gpu_train_batch_size 4 \
  --learning_rate 4e-5 \
  --num_train_epochs 2.0 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --output_dir /share/nas165/Wendy/transformers/examples/squad_debug_SQuAD_1213_bert/

Do anyone know how to do? Thanks a lot

Comments:
Can you please post the required version numbers? (Should be in the issue template)

the version numbers is 2.2.0

- Is it answering every question? for V2 you might want this flag passed
https://github.com/huggingface/transformers/blob/master/examples/run_squad.py#L409

- Your model is cased?
https://github.com/huggingface/transformers/blob/master/examples/run_squad.py#L428

--- 
Edit: I should say I haven't actually done this. I just know this flag exists. 

@WenTingTseng If you want to use SQuAD v2.0, you have to pass the `--version_2_with_negative` argument to `run_squad.py`, otherwise the model supposes every question has at least an answer. Without that flag, you are basically not "learning" the no-answerable part.

ok, problem is resolved
Thanks a lot
gpt-2 implement issue
Author: bigprince97
Pull request: no

Thanks for your good implementation some model in pytorch!
gpt-2 paper mentioned that they did few  modifications with original gpt, included "A modified initialization which accounts
for the accumulation on the residual path with model depth
is used. We scale the weights of residual layers at initialization by a factor of 1/
√
N where N is the number of
residual layers."
I assume that will help in training, it is crucial to reimplement the gpt2, so consider add this to repo? 


Comments:
Hi, this is for the initialization. We don't have any scripts that show how to pretrain GPT-2 (therefore no need for initialization), only scripts to fine-tune it from a checkpoint.

thanks for your reply
rename _has_sklearn to _sklearn_available
Author: jeongukjae
Pull request: yes
Labels: name: wontfix

Rename `_has_sklearn` to `_sklearn_available`, because variables that act like `_has_sklearn` in `transformers/file_utils.py` are named like `_{module}_available`

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Typo in serving.py
Author: kashif
Pull request: yes


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2324?src=pr&el=h1) Report
> Merging [#2324](https://codecov.io/gh/huggingface/transformers/pull/2324?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/aeef4823ab6099249679756182700e6800024c36?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2324/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2324?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #2324   +/-   ##
=======================================
  Coverage   73.49%   73.49%           
=======================================
  Files          87       87           
  Lines       14793    14793           
=======================================
  Hits        10872    10872           
  Misses       3921     3921
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2324?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/commands/serving.py](https://codecov.io/gh/huggingface/transformers/pull/2324/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9jb21tYW5kcy9zZXJ2aW5nLnB5) | `0% <ø> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2324?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2324?src=pr&el=footer). Last update [aeef482...7211541](https://codecov.io/gh/huggingface/transformers/pull/2324?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
How can I read my bert model by using transformers?
Author: Little-Frog-233
Pull request: no

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
when I use pytorch_pretrained_bert, i can read my model like this:

from pytorch_pretrained_bert import BertModel, BertTokenizer
tokenizer = BertTokenizer.from_pretrained(bert_vocab_path)
bert = BertModel.from_pretrained(bert_model_path)

when i use transformers, how can i do this?


Comments:
Not clearly sure what your question is but i guess u need to change
`from pytorch_pretrained_bert import BertModel, BertTokenizer` to `from transformers import BertModel, BertTokenizer` 
Download latest version if not-found-module error occurs...

thank u for your replay, I tryed&nbsp;from transformers import BertModel, BertTokenizer, but it reminds me that&nbsp;OSError: Model name 'model/chinese_L-12_H-768_A-12' was not found in model name list, does it means that i can't use my local model?




------------------&nbsp;原始邮件&nbsp;------------------
发件人:&nbsp;"shashankMadan-designEsthetics"<notifications@github.com&gt;;
发送时间:&nbsp;2019年12月26日(星期四) 晚上7:48
收件人:&nbsp;"huggingface/transformers"<transformers@noreply.github.com&gt;;
抄送:&nbsp;"晓辰"<1342468180@qq.com&gt;;"Author"<author@noreply.github.com&gt;;
主题:&nbsp;Re: [huggingface/transformers] How can I read my bert model by using transformers? (#2318)




Not clearly sure what your question is but i guess u need to change
 from pytorch_pretrained_bert import BertModel, BertTokenizer to from transformers import BertModel, BertTokenizer
 Download latest version if not-found-module error occurs...
 
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or unsubscribe.

Well You can and it should work. 
First Try using just `bert-uncased` to check if its working correctly or maybe with `BERT-Base, Chinese`. 
If it says not found it may be some error from your local url so check if its the right folder location and name.
Then if nothing works i guess you may need to finetune it.

i tryed and it works, but so slow, may because i'm in china, HAHA, thank you very much




------------------&nbsp;原始邮件&nbsp;------------------
发件人:&nbsp;"shashankMadan-designEsthetics"<notifications@github.com&gt;;
发送时间:&nbsp;2019年12月26日(星期四) 晚上8:17
收件人:&nbsp;"huggingface/transformers"<transformers@noreply.github.com&gt;;
抄送:&nbsp;"晓辰"<1342468180@qq.com&gt;;"Author"<author@noreply.github.com&gt;;
主题:&nbsp;Re: [huggingface/transformers] How can I read my bert model by using transformers? (#2318)




Well You can and it should work.
 First Try using just bert-uncased to check if its working correctly or maybe with BERT-Base, Chinese.
 If it says not found it may be some error from your local url so check if its the right folder location and name.
 Then if nothing works i guess you may need to finetune it.
 
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or unsubscribe.

Your welcome, Utilize cuda if u have gpus or try doing it on cloud. 
Do close the issue if it seems solved.
Add hint to install pytest-xdist
Author: patrickvonplaten
Pull request: yes

Just a small hint that pytest-xdist should be installed before running the make test step

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2315?src=pr&el=h1) Report
> Merging [#2315](https://codecov.io/gh/huggingface/transformers/pull/2315?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/aeef4823ab6099249679756182700e6800024c36?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2315/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2315?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #2315   +/-   ##
=======================================
  Coverage   73.49%   73.49%           
=======================================
  Files          87       87           
  Lines       14793    14793           
=======================================
  Hits        10872    10872           
  Misses       3921     3921
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2315?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2315?src=pr&el=footer). Last update [aeef482...4c48701](https://codecov.io/gh/huggingface/transformers/pull/2315?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


They're installed by `pip install -e .[dev]`. You don't have them because you modified that step. Let's discuss on #2316.
Is there a computation/speed advantage to batching inputs into `TransformerModel` to reduce its number calls
Author: Santosh-Gupta
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

For my particular application, I need to have several `output = TransformerModel(inputIDs)` calls per step, from different datasets. 

so

```
output1 = TransformerModel(inputIDs_dataset1)
output2 = TransformerModel(inputIDs_dataset2)
output3 = TransformerModel(inputIDs_dataset3)
```

Initially I preferred to have these calls separate, as each dataset has a different average and distribution of sequence length, so keeping these separated would decrease the number of paddings I need to do within each batch. 

On the other hand, I imagine that the TransformerModel objects have some optimizations which  would make it overall more computationally efficient just to concatenate all the datasets, and make only one call to the `TransformerModel`.

My intuition is towards the latter approach, but would hear takes from those who designed it. 

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Idea to improve DistilBERT
Author: tchaton
Pull request: no
Labels: name: wontfix

## 🚀 Feature

<!-- A clear and concise description of the feature proposal. Please provide a link to the paper and code in case they exist. -->

Based on https://medium.com/huggingface/distilbert-8cf3380435b5, you are using KL_loss to train the student from the teacher.

You can do something a little bit different there.

Loss = KL_loss(teacher, student when teacher is right) and CE(when the teacher is wrong).
Therefore, the student should converge to be better than the teacher by no learning from its mistake. 

Or even better, you could correct the teacher by artificially telling him the GT;
By replacing in the teacher prediction the GT class probability value by 1 and then renormalizing 
by the sum of probs (here: some of sort of artificial smooth).

## Motivation

<!-- Please outline the motivation for the proposal. Is your feature request related to a problem? e.g., I'm always frustrated when [...]. If this is related to another GitHub issue, please link here too. -->

Improve DistillBert performance.

## Additional context

<!-- Add any other context or screenshots about the feature request here. -->

Comments:
good idea.  i aggree with the second plan

If you give it a try, please tell me in the loop : thomas.chaton.ai@gmail.com

There is Albert now https://arxiv.org/abs/1909.11942 which seems to be even better. It isn't based on KD.

anthor solution is here. [https://github.com/intersun/PKD-for-BERT-Model-Compression](url). Here is a awesome about distillaiton:  [https://github.com/dkozlov/awesome-knowledge-distillation](url)

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Question answering for SQuAD with XLNet
Author: kayoyin
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

Dear huggingface,

Thank you very much for your great implementation of NLP architectures! I'm currently trying to train an XLNet model for question answering in French.

I studied your code to understand how question answering is done with XLNet, but I am struggling to follow how it works. Especially, I would like to understand the reasoning behind `PoolerStartLogits`, `PoolerStartLogits` and `PoolerAnswerClass`. 
I also don't quite understand how prediction of the answer indices works during inference time.

I know this is a lot of questions, I appreciate any help you can give me!

Thank you very much!


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Delete [dev] behind pip install -e .
Author: patrickvonplaten
Pull request: yes

I might be wrong here, but I think it should simply be 
```bash
$ pip install -e .
``` 
without the [dev] 
When executing 
```bash
$ pip install -e .[dev]
``` 
in my terminal I get the error: 
`no matches found: .[dev]`

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2316?src=pr&el=h1) Report
> Merging [#2316](https://codecov.io/gh/huggingface/transformers/pull/2316?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/aeef4823ab6099249679756182700e6800024c36?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2316/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2316?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #2316   +/-   ##
=======================================
  Coverage   73.49%   73.49%           
=======================================
  Files          87       87           
  Lines       14793    14793           
=======================================
  Hits        10872    10872           
  Misses       3921     3921
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2316?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2316?src=pr&el=footer). Last update [aeef482...73511e8](https://codecov.io/gh/huggingface/transformers/pull/2316?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


`[dev]` is there to install the development dependencies.

What shell are you using?

Does `pip install -e ".[dev]"` or `pip install -e .\[dev\]` work?





We're probably going to modify the syntax for shell scripts. When we do this, we should modify it throughout the repository, because there are a bunch of other instances of this.

I see!
I was using the zsh shell. 
`pip install -e ".[dev]"` and `pip install -e .\[dev\]` both work with zsh shell.

When switching to the bash shell 
`pip install -e .[dev]` works as well.

I'm using zsh as well, but I must have enabled an option that makes the unquoted syntax work.

I'm going to fix the instructions to prevent others from hitting the same problem.

Thanks for the report!
duplicated line for repeating_words_penalty_for_language_generation
Author: patrickvonplaten
Pull request: yes

length_penalty has a duplicated wrong documentation for language generation -> delete two lines

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2290?src=pr&el=h1) Report
> Merging [#2290](https://codecov.io/gh/huggingface/transformers/pull/2290?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/aeef4823ab6099249679756182700e6800024c36?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `0%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2290/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2290?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #2290      +/-   ##
==========================================
- Coverage   73.49%   73.48%   -0.01%     
==========================================
  Files          87       87              
  Lines       14793    14794       +1     
==========================================
  Hits        10872    10872              
- Misses       3921     3922       +1
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2290?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/2290/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `63.34% <0%> (-0.12%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2290?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2290?src=pr&el=footer). Last update [aeef482...0f6017b](https://codecov.io/gh/huggingface/transformers/pull/2290?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Yes, actually the doc is still not complete for this new feature.

We should add some examples and double-check all. Feel free to clean this up if you feel like it.

clean up documentation, add examples for documentation and rename some variables

checked example generation for openai-gpt, gpt2, xlnet and xlm in combination with #2289.


Also checked for ctrl

also checked for transfo-xl

Awesome, merging!
Typo in tokenization_utils.py
Author: adelevie
Pull request: yes

avoir -> avoid

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2286?src=pr&el=h1) Report
> Merging [#2286](https://codecov.io/gh/huggingface/transformers/pull/2286?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/23dad8447c8db53682abc3c53d1b90f85d222e4b?src=pr&el=desc) will **increase** coverage by `0.18%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2286/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2286?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #2286      +/-   ##
==========================================
+ Coverage   74.27%   74.45%   +0.18%     
==========================================
  Files          85       85              
  Lines       14610    14610              
==========================================
+ Hits        10851    10878      +27     
+ Misses       3759     3732      -27
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/2286?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/tokenization\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/2286/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90b2tlbml6YXRpb25fdXRpbHMucHk=) | `92.08% <ø> (+0.77%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/2286/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `93.12% <0%> (+1.58%)` | :arrow_up: |
| [src/transformers/pipelines.py](https://codecov.io/gh/huggingface/transformers/pull/2286/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9waXBlbGluZXMucHk=) | `68.31% <0%> (+2.32%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_auto.py](https://codecov.io/gh/huggingface/transformers/pull/2286/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl9hdXRvLnB5) | `32.14% <0%> (+7.14%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2286?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2286?src=pr&el=footer). Last update [23dad84...7cef764](https://codecov.io/gh/huggingface/transformers/pull/2286?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).