noamaanMulla-03/transformers-issues huggingface dataset:
BertForQuestionAnswering output to predict text
Author: qiyama
Pull request: no


<!-- A clear and concise description of the question. -->

In predict mode, BertForQuestionAnswering model output a tuple like below, how to get the text answer interactively
```
tensor([[ 0.4691,  0.3912, -0.3447,  0.9756,  0.7171,  0.3746,  0.5273,  0.3756,
          0.2083,  0.4130,  0.2145,  0.1327,  0.7265,  0.4678,  0.6294,  0.3284]])
```


Comments:
[FIX] fix repetition penalty in `examples/run_generation.py`
Author: leo-du
Pull request: yes

The repetition penalty in `examples/run_generation.py` is incorrectly implemented due to the following snippet.

```python
            for _ in set(generated):
                next_token_logits[_] /= repetition_penalty
```

`generated` is a tensor, and python built-in `set` does not compare tensors correctly, e.g.:

```python
>>> import torch
>>> set(torch.cat([torch.arange(2),torch.arange(3)]))
{tensor(0), tensor(1), tensor(1), tensor(0), tensor(2)}
```

This PR fixes this subtle error.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1551?src=pr&el=h1) Report
> Merging [#1551](https://codecov.io/gh/huggingface/transformers/pull/1551?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/c5441946112e68441b46866d114bf8d3c29b0c1d?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1551/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1551?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #1551   +/-   ##
=======================================
  Coverage   86.16%   86.16%           
=======================================
  Files          91       91           
  Lines       13593    13593           
=======================================
  Hits        11713    11713           
  Misses       1880     1880
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1551?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1551?src=pr&el=footer). Last update [c544194...4f05239](https://codecov.io/gh/huggingface/transformers/pull/1551?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Hello! That's great, thank you!
can somebody share an example of how to use GPT2 model for multiclass classification problem with fine tuning Language model ?
Author: p9anand
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help
I have huge text corpus without label and few data points with label.
can somebody guide on how to use GPT2 model for multi class classification problem with fine tuned Language model ? 

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
gpt2 fine tuning sources
Author: radiodee1
Pull request: no
Labels: name: wontfix

hi. I'm looking to fine tune the gpt2 model. I missed the part where that sort of fine tuning is taking place. Can someone point out where that code is (...or maybe where an example might be elsewhere on line)? 

Comments:
I encountered the same issue

Also looking for how to finetune the GPT2 model, thanks.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
run_classifier.py missing from examples dir
Author: XinCode
Pull request: no
Labels: name: wontfix

Hi, it seems that run_classifier is removed (or changed name?) from the examples dir. I am working on NER task with bert, can anyone suggest where I can find the sample/tutorial training/prediction code?

Comments:
It is now replaced by run_glue.py in the /examples folder

@ningjize Got it, thanks.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Intended Behaviour for Impossible (out-of-span) SQuAD Features
Author: Shayne13
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

Hello! We have a quick question regarding the featurization for BERT/XLNet Question Answering.

We noticed a confusing contradiction in your current `utils_squad` implementation: regardless of how the `version_2_with_negative` flag is set, you do not discard “impossible” features (chunks of a context). Instead of discarding them, you train on them but with the span start and end indices pointing to the [CLS] token. However, this comment in your code indicates that you *do* intend to discard such features (at least for SQuAD 1.1 we would assume): https://github.com/huggingface/pytorch-transformers/blob/a7b4cfe9194bf93c7044a42c9f1281260ce6279e/examples/utils_squad.py#L332-L333.

We noticed that this behavior is the same with the Google TensorFlow BERT repository, though we see no reference in their paper to training SQuAD 1.1 with impossible contexts. Should we assume for SQuAD 1.1 the `max_sequence_length` was just always longer that all SQuAD contexts, and thus no "impossible" features were produced?

Ultimately, we are wondering if this behavior is intentional or not for purely extractive QA (like SQuAD 1.1, as opposed to 2.0)? Are you aware of anyone using “impossible" inputs to train a model for extractive QA without an abstention objective?

Thank you for your time and insights!

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
inconsistent between class name (Pretrained vs PreTrained)
Author: boy2000-007man
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

https://github.com/huggingface/pytorch-transformers/blob/1b35d05d4b3c121a9740544aa6f884f1039780b1/pytorch_transformers/__init__.py#L37

I notice there are `Pre**t**rainedConfig`, `Pre**T**rainedModel` and `Pre**T**rainedTokenizer` have different naming case which is confusing.

Is this naming style expected? Or just typo?

thx

Comments:
Hi, it isn't really expected and I agree that it can be a bit confusing, but now that it's like that we'll probably keep is so as to not make a breaking change.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
puzzling issue regarding evaluation phase
Author: ghost
Pull request: no
Labels: name: wontfix

Hi
I observe that if you run the run_glue code on WNLI, and activate both do_train and do_eval once you get one accuracy, if you run_glue with only eval with the path to the trained model,
you get another accuracy. This is very puzzling, thanks for your help

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Running CTRL Model On Google Colab Environment
Author: yusufani
Pull request: no

## ❓ Questions & Help

As you know Google Colab environment has ****12 GB** ram** limit . When i want to run [run_generation.py](https://github.com/huggingface/transformers/blob/master/examples/run_generation.py) file , Colab automatically stops the process. How much ram does the CTRL Model need? Where can I learn that? Or is there another way to run it?


Comments:
[In the official repo,](https://github.com/salesforce/ctrl) you can find a 'lower_memory' branch. You can take a look there. As always, you can try to make the batch size and max sequence length smaller, too.

Thank you for your help

Please close this topic if you have no further questions.
Fail to load pre-trained tokens.
Author: xijiz
Pull request: no
Labels: name: wontfix

The PreTrainedTokenizer fails to load tokenizer files when I load tokenizer files from local tokenizer files.

The error is caused by code line 174 - 182 in [tokenization_utils.py](https://github.com/huggingface/pytorch-transformers/blob/master/pytorch_transformers/tokenization_utils.py). The code assumes that there are three tokenizer files: added_tokens.json, special_tokens_map.json, vocab.txt. However, these path of tokenizer files will be the same path if the given parameter "pretrained_model_name_or_path" is full path of "vocab.txt".

Comments:
Hi, how do you solve this problem? If we set `pretrained_model_name_or_path` as a path to vocab.txt, it still need the two files: added_tokens.json, special_tokens_map.json. Where can we get these files? 

> Hi, how do you solve this problem? If we set `pretrained_model_name_or_path` as a path to vocab.txt, it still need the two files: added_tokens.json, special_tokens_map.json. Where can we get these files?

You can ignore these files: added_tokens.json, special_tokens_map.json. All you need to do is to modify some code lines in the file: tokenization_utils.py. I have modified it in my forked repository as you can see [here](https://github.com/xijiz/pytorch-transformers/commit/31bc1ddf4f68ad790da9874a3623cf22d62dc186). 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Unseen Vocab
Author: siddsach
Pull request: no

Thank you so much for this well-documented and easy-to-understand implementation! I remember meeting you at WeCNLP and am so happy to see you push out usable implementations of the SOA in pytorch for the community!!!!!

I have a question: The convert_tokens_to_ids method in the BertTokenizer that provides input to the BertEncoder uses an OrderedDict for the vocab attribute, which throws an error (e.g. `KeyError: 'ketorolac'`) for any words not in the vocab. Can I create another vocab object that adds unseen words and use that in the tokenizer? Does the pretrained BertEncoder depend on the default id mapping? 

It seems to me that ideally in the long-term, this repo would incorporate character level embeddings to deal with unseen words, but idk if that is necessary for this use-case. 

Comments:
If you tokenize properly the input (tokenize before convert_tokens), it automatically 'fallbacks' to subword/character-level(-like) embedding.
You can add new words in the vocabulary but you'll have to train the corresponding embeddings.

Hi @siddsach,
Thanks for your kind words!
@artemisart is right, BPE progressively falls-back on character level embeddings for unseen words.

> If you tokenize properly the input (tokenize before convert_tokens), it automatically 'fallbacks' to subword/character-level(-like) embedding.
> You can add new words in the vocabulary but you'll have to train the corresponding embeddings.

Hi, what do you mean `tokenize properly the input (tokenize before convert_tokens)` ?
Can you refer a tokenization sample (before and after) or a sample code if any? thank you
Add special tokens to documentation for bert examples to resolve issue: #1561
Author: enzoampil
Pull request: yes

**Currently the BERT examples only show the strings encoded without the inclusion of special tokens (e.g. [CLS] and [SEP]) as illustrated below:**

```
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
sentence = "Hello there, General Kenobi!"
print(tokenizer.encode(sentence))
print(tokenizer.cls_token_id, tokenizer.sep_token_id)

# [7592, 2045, 1010, 2236, 6358, 16429, 2072, 999]
# 101 102
```

**In this pull request i set ```add_special_tokens=True``` in order to include special tokens in the documented examples as illustrated below:**
```
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
sentence = "Hello there, General Kenobi!"
print(tokenizer.encode(sentence, add_special_tokens=True))
print(tokenizer.cls_token_id, tokenizer.sep_token_id)
# [101, 7592, 2045, 1010, 2236, 6358, 16429, 2072, 999, 102]
# 101 102
```

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1584?src=pr&el=h1) Report
> Merging [#1584](https://codecov.io/gh/huggingface/transformers/pull/1584?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/82f6abd98aaa691ca0adfe21e85a17dc6f386497?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1584/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1584?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #1584   +/-   ##
=======================================
  Coverage   86.16%   86.16%           
=======================================
  Files          91       91           
  Lines       13593    13593           
=======================================
  Hits        11713    11713           
  Misses       1880     1880
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/1584?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [transformers/modeling\_bert.py](https://codecov.io/gh/huggingface/transformers/pull/1584/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL21vZGVsaW5nX2JlcnQucHk=) | `88.17% <ø> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1584?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1584?src=pr&el=footer). Last update [82f6abd...1972e0e](https://codecov.io/gh/huggingface/transformers/pull/1584?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
seq2seq with gpt2
Author: ghost
Pull request: no

Hi,
I want to have a seq2seq model from gpt2, if I change the script of "run_lm_finetuning.py" in a way that it gets a sequence, then make it a context ids, and let it generate another sequence, like "run_generation.py" code, then minimize the cross-entropy loss, does it this way, create a seq2seq model? I rgreatly appreciate your help. thanks a lot.

Comments:
Merging with #1506
How to make a new line when using gpt2 to generate lyrics?
Author: tine8899
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->

I use pre-trained gpt2 to generate lyrics (text generation). I can generate a long string of lyrics. While how to break the line, I try to add "\n" into it. But it seems that is not a good idea.

Thanks!

Comments:
Did you pre-train your model while keeping all the line returns or did you remove them? You can keep them during the training so that the model learns to predict them.

If you remove them during training and wish to apply them later on, I guess you can always just create the long string of lyrics and split them with line returns.

Actually, I remove them during. Do you think keeping all the line returns during training is a better way? I mean input the whole song (use encode(text)  method) instead of splitting each line into echo tokens -> ids .  

By the way, Can I input POS of text into the model? Is it necessary?

I think keeping the line returns during your training is a good idea. The model is very likely to learn their position and frequency.

You can input the `position_ids` in your forward, but it is not necessary. If no position information is provided, the model will create it on its own.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Fix hanging when loading pretrained models
Author: daemon
Pull request: yes

- Fix hanging when loading pretrained models from the cache without having internet access. This is a widespread issue on supercomputers whose internal compute nodes are firewalled.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1568?src=pr&el=h1) Report
> Merging [#1568](https://codecov.io/gh/huggingface/transformers/pull/1568?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/82f6abd98aaa691ca0adfe21e85a17dc6f386497?src=pr&el=desc) will **decrease** coverage by `0.02%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1568/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1568?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #1568      +/-   ##
==========================================
- Coverage   86.16%   86.14%   -0.03%     
==========================================
  Files          91       91              
  Lines       13593    13593              
==========================================
- Hits        11713    11710       -3     
- Misses       1880     1883       +3
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/1568?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/1568/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL2ZpbGVfdXRpbHMucHk=) | `74.17% <100%> (ø)` | :arrow_up: |
| [transformers/tests/modeling\_tf\_common\_test.py](https://codecov.io/gh/huggingface/transformers/pull/1568/diff?src=pr&el=tree#diff-dHJhbnNmb3JtZXJzL3Rlc3RzL21vZGVsaW5nX3RmX2NvbW1vbl90ZXN0LnB5) | `95.21% <0%> (-1.6%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1568?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1568?src=pr&el=footer). Last update [82f6abd...a2c8c8e](https://codecov.io/gh/huggingface/transformers/pull/1568?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Ok, LGTM thanks
Gradient norm clipping should be done right before calling the optimiser
Author: pminervini
Pull request: yes

Right now it's done after each step in the gradient accumulation. What do you think?

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1580?src=pr&el=h1) Report
> Merging [#1580](https://codecov.io/gh/huggingface/transformers/pull/1580?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/82f6abd98aaa691ca0adfe21e85a17dc6f386497?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/1580/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/1580?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #1580   +/-   ##
=======================================
  Coverage   86.16%   86.16%           
=======================================
  Files          91       91           
  Lines       13593    13593           
=======================================
  Hits        11713    11713           
  Misses       1880     1880
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/1580?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/1580?src=pr&el=footer). Last update [82f6abd...abd7110](https://codecov.io/gh/huggingface/transformers/pull/1580?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Oh yes, great, thanks, Pasquale. Would you mind fixing the `run_glue` and `run_ner` examples as well?

@thomwolf done! what's the best way to check this code before merging?

Thanks a lot!
It should be fine, we have continuous integration tests on `run_glue` and `run_squad` so if it passed at least the code run.
_tokenize() got an unexpected keyword argument 'add_prefix_space' in CTRL
Author: BramVanroy
Pull request: no

## 🐛 Bug

If you look at [the search results in this repo](https://github.com/huggingface/transformers/search?q=add_prefix_space) for `add_prefix_space`, you'll find gpt2, roberta, and ctrl all document that 

> `add_prefix_space`: Requires a space to start the input string => the encoding methods should be called with the --``add_prefix_space`` flag set to ``True``.

However, this attribute is only implemented in the GPT2Tokenizer. Since RobertaTokenizer subclasses GPT2Tokenizer, that is fine. However, CTRLTokenizer just subclasses the PretrainedTokenizer. As such, it does not have a `_tokenize()` method that accepts the `add_prefix_space` keyword. 

I would fix this in a PR, but I am not sure what the actual correct fix is: does CTRL need the added space, or not? And can it subclass GPT2's tokenizer, or should it implement its own `_tokenize(*, add_prefix_space)` method?




Comments:
Hi @BramVanroy, thanks for reporting this. There was an issue in the docstring. It does not use prefix spaces and it does not use a byte-level BPE like GPT-2 does. The docstring should be fixed now.
distilled gpt2 to be added to run_generation and run_lm_fintuning
Author: ghost
Pull request: no

Hi
I greatly appreciated also adding distilled GPT2 to the codes above, thanks

Comments:
Hi, DistilGPT-2 is considered to be a checkpoint of GPT-2 in our library (differently to DistilBERT). You can already use DistilGPT-2 for both of these scripts with the following:
```bash
python run_generation --model_type=gpt2 --model_name_or_path=distilgpt2
```
Make benchmark more flexible (TF or PT)
Author: BramVanroy
Pull request: no

I've been trying to run the benchmark, but I gave up after running into a trillion compatibility issues with tensorflow and bazel. To be fair, I just want to contribute and test all there is to test on PyTorch with 4x Tesla V100. It would be great if only the required modules are needed, and not all of them. So only try to import PyTorch or Tensorflow when the tester actually wants to test those frameworks.

Comments:
I believe a quick workaround is to just install the pre-built, CPU version of TensorFlow 2.0. If you won't be running the TF benchmarks, it wouldn't affect anything.

True, but still not quite flexible. Since the goal of the benchmark script is to, I believe, encourage the community to add there runtimes, it's a good to make this as easy-to-use as possible.

You're right that we shouldn't require to have both libraries installed in order to benchmark only one of them. I've updated the Benchmark code so that you can run it with only a single library installed.

That's great, Lysandre. Thanks for pushing out changes so quickly!