noamaanMulla-03/transformers-issues huggingface dataset:
XLNet: Incorrect segment id for CLS token
Author: ChristophAlt
Pull request: no
Labels: name: wontfix

Thanks for the great work!

It seems that the XLNetTokenizer assigns an incorrect segment id to the CLS token when a single sequence of token ids is provided. If token_ids_1 is None, all segment ids are '0', including the segment id of the CLS token. In my understanding, the segment ids should always differ.

```python
if token_ids_1 is None:
    return len(token_ids_0 + sep + cls) * [0]
return len(token_ids_0 + sep) * [0] + len(token_ids_1 + sep) * [1] + cls_segment_id
```
https://github.com/huggingface/transformers/blob/983c484fa2fcad307d37cb81f3e1125aa7b9dc37/src/transformers/tokenization_xlnet.py#L243

The original implementation assigns a different segment id to the CLS token in both cases (single sequence of tokens and pair of sequences): https://github.com/zihangdai/xlnet/blob/bbaa3a6fa0b3a2ee694e8cf66167434f9eca9660/classifier_utils.py#L109

Comments:
Indeed, this is an error ! Thanks for letting us know, it was patched with 088fa7b!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
How to run a batch of data through BERT model?
Author: yrf1
Pull request: no

## ❓ Questions & Help

<!-- A clear and concise description of the question. -->
I understand how to run **1 data** point of d words through a BERT model, but how can I run **n data** sequence of words through the BERT model?

Nvm solved this issue.

I can just pass sth like a 2xd data that looks like this:
tensor([[2182, 2003, 1996, 6251, 1045, 2215, 7861, 8270, 4667, 2015, 2005,
           1012],
          [2182, 2003, 1996, 6251, 1045, 2215, 7861, 8270, 4667, 2015, 2005,
           1012]], device='cuda:0')
through the BERT forward function


Comments:
Did you solve it? I have the same problem as you.
Fix examples/run_tf_ner.py label encoding error #2559 
Author: HuiyingLi
Pull request: yes
Labels: name: wontfix

This is an explanation and a proposed fix for #2559 
The code set `pad_token_label_id = 0`, and increase the total number of labels `num_labels = len(labels) + 1`, but made no change to the label list. Thus the first label in label list has the same index as pad_token_label_id.

Following instructions in README take GermEval 2014 as an example, for one sentence in test dataset the token `Aachen` is labeled as `B-LOC` (`B-LOC` is the first label in label list), yet because of the collision with pad_token_label_id, both pad tokens and `Aachen` are encoded as 0:
![image](https://user-images.githubusercontent.com/1331543/72657322-bb90fa00-3957-11ea-980d-33ac12e54545.png)
And the test_predictions.txt is also off by one:
```
1951 I-PERpart
bis I-PERpart
1953 I-PERpart
wurde I-PERpart
...
```

The fix adds a placeholder label `[PAD]` at position 0 when loading the datasets and all labels positions are shifted by 1. The resulting encoding for the same sample sentence:
![image](https://user-images.githubusercontent.com/1331543/72657627-5e974300-395b-11ea-905e-4a3028f4b5c4.png)

And the test_predictions.txt thus has correct index:
```
1951 O
bis O
1953 O
wurde O
...
```


Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2575?src=pr&el=h1) Report
> Merging [#2575](https://codecov.io/gh/huggingface/transformers/pull/2575?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/1a8e87be4e2a1b551175bd6f0f749f3d2289010f?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/2575/graphs/tree.svg?width=650&token=9qOlN6Hb1c&height=150&src=pr)](https://codecov.io/gh/huggingface/transformers/pull/2575?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #2575   +/-   ##
=======================================
  Coverage   74.53%   74.53%           
=======================================
  Files          87       87           
  Lines       14819    14819           
=======================================
  Hits        11046    11046           
  Misses       3773     3773
```



------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/2575?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/2575?src=pr&el=footer). Last update [1a8e87b...bd3fe2f](https://codecov.io/gh/huggingface/transformers/pull/2575?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
gpt2 - convert examples to features(tensorflow 2)
Author: yagelardan
Pull request: no

I'm trying to fine-tune GPT2 to generate shakespeare text. 
I have variable "train_examples", which is list of InputExamples:
```
>> print(train_examples)
 <__main__.InputExample at 0x7f55e0fafd68>,
 <__main__.InputExample at 0x7f55e0fafda0>,
 <__main__.InputExample at 0x7f55e0fafef0>,
 <__main__.InputExample at 0x7f55e0fafeb8>,
 <__main__.InputExample at 0x7f55e0f6aeb8>,
```
I created the examples using the following function:
```
    def _create_examples(self, lines, set_type):
        """Creates examples for the training and dev sets."""
        examples = []
        for (i, line) in enumerate(lines):
            guid = "%s-%s" % (set_type, i)
            #guid = i
            text_a = line[1]
            examples.append(
                InputExample(guid=guid, text_a=text_a))
        return examples

class InputExample(object):
    def __init__(self, guid, text_a):
        self.guid = guid
        self.text_a = text_a
```

As I understand I need to convert the type to 'features', before I call the fit function. But how can I convert the examples to features? I saw many cases for BERT, but I couldn't find for gpt2.

I tried:
```
from transformers import glue_convert_examples_to_features

input_train_tensor_data = glue_convert_examples_to_features(train_examples, gpt2_tokenizer, max_length=128, task='mrpc')
```

But got:
```
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-118-d04221b84b4f> in <module>()
      1 from transformers import glue_convert_examples_to_features
      2 
----> 3 input_train_tensor_data = glue_convert_examples_to_features(train_examples, gpt2_tokenizer, max_length=128, task='mrpc')

/usr/local/lib/python3.6/dist-packages/transformers/data/processors/glue.py in glue_convert_examples_to_features(examples, tokenizer, max_length, task, label_list, output_mode, pad_on_left, pad_token, pad_token_segment_id, mask_padding_with_zero)
    120 
    121         if output_mode == "classification":
--> 122             label = label_map[example.label]
    123         elif output_mode == "regression":
    124             label = float(example.label)

KeyError: None
```

Comments:
Hi @yagelardan ,

In order to fine-tune gpt2 you should be able to use this example [script](https://github.com/huggingface/transformers/blob/master/examples/run_language_modeling.py)

Also, you can refer to this issue https://github.com/huggingface/transformers/issues/1407 where people already trained gpt2 on different languages. 
And this issue https://github.com/huggingface/transformers/issues/2008 could help you out :-)
Issue in generating samples for text generation
Author: Ahanmr
Pull request: no

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...):GPT2

```python
def generate_samples(args, model, prompt_text):
  """Generating sampling for the provided prompt using the provided model."""
  set_seed(args.seed)

  _, _, tokenizer_class = run_language_modeling.MODEL_CLASSES[args.model_type]
  tokenizer = tokenizer_class.from_pretrained(args.model_name_or_path, cache_dir=None)

  requires_preprocessing = args.model_type in run_generation.PREPROCESSING_FUNCTIONS.keys()
  encoded_prompt = tokenizer.encode(prompt_text, add_special_tokens=False, return_tensors="pt")
  encoded_prompt = encoded_prompt.to(args.device)
```

Error: run_language_modeling has no 'MODEL_CLASSES'

Language I am using the model on (English):




Comments:
The file `run_language_modeling.py` does indeed not have a variable called `MODEL_CLASSES`. Can you explain what you are trying to do exactly?
run_ner.py huge discrepancy between eval and predict (or "dev" and "test" evaluation modes)
Author: MatejUlcar
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

I'm comparing two different pretrained Bert models on the NER task. One is a bert-base-multilingual-cased model, which works fine, consistently, as one would expect. The other is our own pretrained multilingual Bert, which is trained on fewer languages and has so far shown better results on those few languages. However, when running run_ner.py from here, it doesn't evaluate consistently. The `dev.txt` dataset is completely identical to the `test.txt`dataset, so I'd expect identical results, but I get
```
--do_eval
f1 = 0.7603833865814698
loss = 0.06039658671007991
precision = 0.7531645569620253
recall = 0.7677419354838709
```
and
```
--do_predict
f1 = 0.025925925925925925
loss = 0.41404916612165316
precision = 0.030434782608695653
recall = 0.02258064516129032
```
I also tried `--evaluate_during_training` flag and I get "solid" results already from very few steps and identical to `--do_eval` at the end of training, but `--do_predict` are always much worse. Surprisingly, though, this doesn't occur with bert-base-multilingual-cased model even if I save it to disk and point to that folder. Additionally if under the clause if args.predict I change the mode from "test" to "dev" in function evaluate() I get good results. I repeat, dev and test are identical files that differ only in name.

The problem occurs on any max sequence length, after deleting cache, etc. No errors are displayed during training or evaluation/prediction.
Please help if you have any ideas what might be going wrong.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
XLnet memory usage for long sequences
Author: AlaFalaki
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

Hello,
I have some questions regarding how the XLnet memory and output work in this implementation.

1. As it's been mentioned before, by default, XLnet doesn't use memory. So, how is this possible that it accepts long sequences as input (in other words, why there isn't any limit on the number of input tokens), unlike BERT, for example, that will only accept 512 tokens.

2. If I set the memory length to 512, and feed the XLnet with 512 tokens at a time (for a 1024 sequence length), also pass the memory in each step. (like the example code below) Will the final output of the network, includes all the information from the whole sequence?

```
tokenizer = XLNetTokenizer.from_pretrained('xlnet-large-cased')
model = XLNetModel.from_pretrained('xlnet-large-cased')
mems = None
for i in range(2):
    input_ids = torch.tensor(tokenizer.encode(text[i])).unsqueeze(0)
    outputs = model(input_ids, mems=mems)
    mems = outputs[1]
```

To be clear, I want to use XLnet for long text summarization. So I need to feed the XLnet output to a decoder part and need a fixed-length representation.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Can not upload BertTokenizer.from_pretrained() from an AWS S3 bucket
Author: bnicholl
Pull request: no
Labels: name: wontfix

## 📚 Migration

the model I am using is BertForSequenceClassification

The problem arises when I serialize my Bert model, and then upload to an AWS S3 bucket. Once my model is inside of S3, I can not import the model via BertTokenizer.from_pretrained()

For example, in order to save my model to S3, my code reads,

```
byte_obj = pickle.dumps(model)
s3_resource = boto3.resource('s3')
s3_resource.Object("s3 name", "bertClassifier").put(Body=byte_obj)
```

This saves the BertForSequenceClassification model. I can not use the  `model.save_pretrained("s3 name")`, as I get an error from AWS. I beleive in order to transfer files in AWS, one needs to first pickle the file.

When I want to re upload the model, I can not use 

`BertTokenizer.from_pretrained("s3 name")`
or
`BertForSequenceClassification.from_pretrained("s3 name")`

because the object I am trying to load is serialized via pickle module. Instead I upload the file this way.

```
session = boto3.session.Session()
s3client = session.client('s3')
response = s3client.get_object(Bucket='s3 name', Key='bertClassifier')

body_string = response['Body'].read()
bert_nn = pickle.loads(body_string)
```

This succesfully loads the BertForSequenceClassification model, but I have no way of loading BertTokenizer from this same pre trained model. Again, becuase I am not able to upload via the BertTokenizer.from_pretrained("s3 name") function. 

Is there a work around for this?







Comments:
Hi Ben,

Unless I misunderstand what you're trying to do, this is not really what `save_pretrained()` and `from_pretrained()` are made for.

`save_pretrained()` lets you save a tokenizer or a model _locally_, inside a local folder. (you can then upload those files to your own s3 bucket, or use the `transformers-cli` to upload to our bucket). 

`from_pretrained()` lets you re-spawn a model or tokenizer from either a local folder, a model shortcut (hardcoded in the library's code), or a community model identifier (which resolves to files on our S3 bucket)

Let me know if things are clearer


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Why GPT2 train loss and topK accuracy both decrease?
Author: lx-kika
Pull request: no

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
<!-- Description of your issue -->
Hi,

I use GPT2 LMHeadModel training from scratch. The training loss decreases, however, I test it on the same train dataset, I get top3 accuracy decreasing as well. Moreover, is it normal that my top3 accuracy goes down sharply like from 0.2 to 0.05 only one or two epochs? Because it seems to be stable when it converges. Did anyone meet the same problem?

![loss](https://user-images.githubusercontent.com/62126666/77870461-14441380-7274-11ea-9406-c8c56bb8143f.jpg)

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**: 

Comments:
I found a mistake that in GPT2LMHeadModel, the label is shifted, however, I shift it again when preparing the batch.
Finetuning T5 Model
Author: caffeinetoomuch
Pull request: no
Labels: name: Usage

Hi there. I am trying to fine tune T5, but I have noticed your documentation gives conflicting instructions.

In modeling_t5.py you say
> To match pre-training, T5 input sequence should be formatted with [CLS] and [SEP] tokens

but then in tokenization_t5.py both of those tokens are set to None and the only tokens that are defined are EOS, UNK, and PAD.
Additionally, the actual T5 implementation makes no mention of SEP and CLS as far as I can tell.
Given that, could you clarify how we should be formatting our training data for HuggingFace's T5 implementation? Thank you!

Comments:
Hi @jkangsta,

Thanks for posting your question here. The docstring was out of date and an in-detail description for T5 will be added here #3507 .
[T5] Add training documenation 
Author: patrickvonplaten
Pull request: yes

- Fixes T5 docstring regarding pretraining
- Add detailed description on how to process input and target for T5 training

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3507?src=pr&el=h1) Report
> Merging [#3507](https://codecov.io/gh/huggingface/transformers/pull/3507?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/601ac5b1dc1438f00d09696588f2deb0f045ae3b&el=desc) will **increase** coverage by `0.00%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3507/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/3507?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #3507   +/-   ##
=======================================
  Coverage   77.79%   77.80%           
=======================================
  Files         100      100           
  Lines       17051    17051           
=======================================
+ Hits        13265    13266    +1     
+ Misses       3786     3785    -1     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3507?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_bart.py](https://codecov.io/gh/huggingface/transformers/pull/3507/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ19iYXJ0LnB5) | `97.58% <ø> (ø)` | |
| [src/transformers/modeling\_t5.py](https://codecov.io/gh/huggingface/transformers/pull/3507/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190NS5weQ==) | `81.29% <ø> (ø)` | |
| [src/transformers/modeling\_tf\_t5.py](https://codecov.io/gh/huggingface/transformers/pull/3507/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl90NS5weQ==) | `94.98% <ø> (ø)` | |
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3507/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `91.94% <0.00%> (+0.13%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3507?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3507?src=pr&el=footer). Last update [601ac5b...ea7e6a8](https://codecov.io/gh/huggingface/transformers/pull/3507?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
[T5] fix lm lables in docstring
Author: patrickvonplaten
Pull request: yes

Add better explanation to T5 `lm_labels` dosctring.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3529?src=pr&el=h1) Report
> Merging [#3529](https://codecov.io/gh/huggingface/transformers/pull/3529?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/75ec6c9e3a7de6cc3e2920f3bb531e7c840b8ada&el=desc) will **increase** coverage by `0.01%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/3529/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/3529?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #3529      +/-   ##
==========================================
+ Coverage   77.80%   77.81%   +0.01%     
==========================================
  Files         100      100              
  Lines       17062    17062              
==========================================
+ Hits        13275    13277       +2     
+ Misses       3787     3785       -2     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/3529?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_t5.py](https://codecov.io/gh/huggingface/transformers/pull/3529/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190NS5weQ==) | `81.52% <ø> (ø)` | |
| [src/transformers/modeling\_tf\_t5.py](https://codecov.io/gh/huggingface/transformers/pull/3529/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl90NS5weQ==) | `94.98% <ø> (ø)` | |
| [src/transformers/modeling\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3529/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ191dGlscy5weQ==) | `91.94% <0.00%> (+0.13%)` | :arrow_up: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/3529/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `88.32% <0.00%> (+0.17%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/3529?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/3529?src=pr&el=footer). Last update [75ec6c9...8692264](https://codecov.io/gh/huggingface/transformers/pull/3529?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
summarization codes 
Author: rabeehk
Pull request: no
Labels: name: wontfix

Hi
I greatly appreciate to add also possibilities to train the summarization codes from scratch. I see only evaluation part in the codes. Does this also work for training?
thanks a lot for your response.
Kind regards
Rabeeh 

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
[Docs] examples/summarization/bart: Simplify CNN/DM preprocessing steps
Author: sshleifer
Pull request: yes

I added the preprocessed data in S3. 

Evidence that it is the correct size:
![image](https://user-images.githubusercontent.com/6045025/77852114-d29b6600-71aa-11ea-979c-1547007a0d28.png)

Comments:
Merging to unblock patrick.
Resizing embedding matrix before sending it to the optimizer.
Author: ngarneau
Pull request: yes

This bug will rise when you add new token types in the Tokenizer.
Since the resizing was done **after** passing the params to the optimizer, the wrong set of params for the embedding table were optimized.

Comments:
Since I just swapped some lines, I guess this code quality check got activated after this file came up in the repo..! 😬

Hi Nicolas,
You have to install the code style tools and run `make style` and `make quality` on your PR.
Check the contributing guide for the details: https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests

Shoot I'm sorry, totally forgot to check the contributing guidelines. Let me fix this real quick.