noamaanMulla-03/transformers-issues huggingface dataset:
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 1176: character maps to <undefined>
Author: neonrights
Pull request: no

## 🐛 Bug

<!-- Important information -->
```UnicodeDecodeError``` when using vocab file generated by ```GPT2Tokenizer```.  Specifically, I created an instance of the ```GPT2Tokenizer``` by calling ```from_pretrained('gpt2')``` then saved the vocab and merges file for that instance to a local directory.  When creating a new ```GPT2Tokenizer``` from the saved files I encounter a ```UnicodeDecodeError``` when reading from the vocab file.

Model I am using (Bert, XLNet....): GPT2 Tokenizer

Language I am using the model on (English, Chinese....): N/A

The problem arise when using:
* [x] the official example scripts: (give details)
* [ ] my own modified scripts: (give details)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details)

## To Reproduce

Steps to reproduce the behavior:

```python
pretrained_tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
vocab_file, merges_file = pretrained_tokenizer.save_vocabulary('.')
new_tokenizer = GPT2Tokenizer(vocab_file, merges_file) # <- UnicodeDecodeError occurs here
```

## Expected behavior

I expect ```new_tokenizer``` to initialize a tokenizer with the same behavior as ```pretrained_tokenizer```.

## Environment

* OS: Windows 10
* Python version: 3.6.8
* PyTorch version: 1.1.0
* PyTorch Transformers version (or branch): 1.1.0

## Additional context

seems likely to be a bug during encoding in ```save_vocabulary```

Comments:
Looking at [these lines](https://github.com/huggingface/pytorch-transformers/blob/07681b6b5859b630077b742b2f06d440869f17e3/pytorch_transformers/tokenization_gpt2.py#L108-L115), the issue seems to be the file is encoded in utf-8 but read using a different encoder.

Changing line 112 to `self.encoder = json.load(open(vocab_file, 'r', encoding='utf-8'))` should fix this issue.

Thanks this is fixed on master now with #1074

Is there a test for which encoder should be used?

Encountered same error and had the same doubt. Used 'iso-8859-1' as it suits me almost anytime. Worked just fine. @ChebonRunner
Fix typo in language-modeling README.md
Author: gpengzhi
Pull request: yes

# What does this PR do?

Fix the typo in `README.md` in the `language-modeling` folder.

<!-- Remove if not applicable -->

Fixes # (issue)


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @sshleifer
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->


Comments:
Validation data in `run_mlm.py` is the same as train data
Author: GuillemGSubies
Pull request: no

Inspecting the script I found the following:

https://github.com/huggingface/transformers/blob/cb966e640b8b9d0f6e9c06c1655d078a917e5196/examples/language-modeling/run_mlm.py#L204

Am I missing something? Otherwise, I could send a PR.

Comments:
Very good catch! Thanks for pointing it out, the PR mentioned above should fix this.
adding model cards for distilled models
Author: VictorSanh
Pull request: yes
Labels: name: model card

# Model cards for distilled models 

As discussed on Slack, a bunch of model cards for distilled models (at least the ones I contributed to).

cc @julien-c

I never know which tags are auto-generated, so please correct me if I did something useless!

Comments:
> I never know which tags are auto-generated, so please correct me if I did something useless!

looks good to me in terms of the tags 👍
Low ROUGE scores for BertSum
Author: loganlebanoff
Pull request: no
Labels: name: wontfix

name: seq2seq

Great work, very easy to pick up and play with. I downloaded the CNN/DM stories from the link provided and selected only the files that belong to the test set following See et al.'s dataset splits (https://github.com/abisee/cnn-dailymail/blob/master/url_lists/all_test.txt). Then I ran the model using the first command provided in the readme.

My question is, what is the expected ROUGE F1 scores on the test set? I expected something near what was presented in the paper for BertSumExtAbs, which is:
R-1: 0.4213
R-2: 0.1960
R-L: 0.3918
But the ROUGE scores I got were much lower:

****** ROUGE SCORES ******

** ROUGE 1
F1        >> 0.303
Precision >> 0.328
Recall    >> 0.288

** ROUGE 2
F1        >> 0.185
Precision >> 0.210
Recall    >> 0.172

** ROUGE L
F1        >> 0.335
Precision >> 0.356
Recall    >> 0.320



Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


I'm struggling with this also :( did you make any progress diagnosing the problem?

No, I did not...I decided to go with a different model

hello.. guys.. any answers to this? Why is there such a low score? I looked at the summaries, and they seem to be good, but i have no comparison benchmark. However, the rouge scores are much lower than paper. how so? 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


active it

@AI678 The BertSum is unfortunately not maintained anymore. If you're looking to do summarization, please check out the [seq2seq](https://github.com/huggingface/transformers/tree/master/examples/seq2seq) scripts.
Fix path to old run_language_modeling.py script
Author: mrm8488
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @sshleifer
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->


Comments:
Appropriate dataset format for language modeling example
Author: arccoxx
Pull request: no

# What is the most memory efficient way/best way to format your dataset file for language modeling?
## Details
I am running run_clm.py and can only get my dataset to work with the smallest GPT2 model. I would like to experiment with gpt2-xl ideally but would settle for large and xlnet. I am using distributed training on TPU assuming that this improves memory. I have saved my file in a .txt. It is roughly 20mb with 82,000 samples of mean length 256 std 250 and is line delimited (where each line signifies a sample).

Is this the correct approach? I notice the .raw files used in training, are these smaller? Is there a way to pretokenize?
I hesitate to ask this on stack as it is not a bug.
@sgugger sorry to bother if this is a common issue I'd love to hear more no worries if it will require much work I can focus on that. Just lost for resources.

Thank you all!

Comments:
General questions should be asked on the [forum](https://discuss.huggingface.co/) as we keep the issues for bugs.
`run_clm` doesn't use the approach of taking the different lines of a dataset (that's in `run_mlm` as it's usually done for masked language modeling) so you will have to tweak the example script to do this.

There is no way to pretokenzie, but the result of the tokenization will be cached, so it will only be run once on a given machine.

Thank you for the help! I won't make that mistake again my apologies. Editing the clm file now. New to open source software and excited to dig in and help please bear with me while I learn the ropes!

No worries, I'm just telling you for next time :-)
Good luck with your scripting, closing this issue for now.
Fix TF Trainer for TPU
Author: jplu
Pull request: yes

This PR try to fix the trainer for TensorFlow by updating the order of some steps:
1. Dataset preprocessing
2. Strategy creation
3. Model creation

Instead of

1. Strategy creation
2. Model creation
3. Dataset preprocessing

Fixes #6672 


Comments:
I have added a model_init function in the PyTorch Trainer to support hp-search. Is it possible to use this instead of changing the `args`? This would make a very big difference between the PT Trainer and TF Trainer.

OK, I will check this. Thanks.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Hello did you have any success fixing this? Can I help? I'm on a tight art student collegiate budget and tpu speed would be awesome if not necessary. I've spent like 20-30 hours on fixing the tpu issue myself and no luck. Any help getting run_clm.py on a tpu so I can quickly iterate would be awesome. But generally I'd love to move mainly to tpu but I'm not sure its there yet. New to open source really want to learn as much as possible. Can I help?

@arccoxx This PR should be closed because we have identified two different issues:

1. The first one don't come from Transformers but from TensorFlow. To make it short, TPU don't handle `tf.data.Dataset.from_generator`, Google is currently working on it and we have to wait they release the fix once they have one.
2. Currently you cannot train a LM from scratch with any TF model. We are currently working on this, and it will be possible in our next release.

So for your project the best solution would be to use the PyTorch version that works on TPU and you can train from scratch any LM model.

> @arccoxx This PR should be closed because we have identified two different issues:
> 
> 1. The first one don't come from Transformers but from TensorFlow. To make it short, TPU don't handle `tf.data.Dataset.from_generator`, Google is currently working on it and we have to wait they release the fix once they have one.
> 2. Currently you cannot train a LM from scratch with any TF model. We are currently working on this, and it will be possible in our next release.
> 
> So for your project the best solution would be to use the PyTorch version that works on TPU and you can train from scratch any LM model.

I was not able to get any pytorch version to run on xla. Is there any reference notebook that could be linked? I tried finetuning in native pytorch, running (pytorch) tuner, run_language_modeling with multiple transformers library versions 2.1.0-2.9.1, and run_clm with 3.4.0 all with no luck. Ive also tried building a pytorch lightning module and no luck. As the speedup would be that helpful (provided it can handle gpt2 medium) it would be awesome to figure out reduce these compatibility issues. My hope is to then use the tpu in a more complicated model that will use this fine tuned model. Any help would be super appreciated. Thank you!
Clean up data collators and datasets
Author: sgugger
Pull request: yes

# What does this PR do?

This PR cleans a bit the `DataCollatorForLanguageModeling` by:
- making sure it keeps additional labels (such a next sentence labels)
- use the `tokenizer.pad` function when possible, rewrite the function that was doing the padding when it's not (to handle padding on the left as well as the right side)
- take advantage of the `special_tokens_mask` that the tokenizer can return, to avoid unnecessary conversions tensor -> list -> tensor again

As a result it deprecates `DataCollatorForSOP` (which can be replaced by `DataCollatorForLanguageModeling`) and changes the elements of `TextDatasetForNextSentencePrediction` to work with `DataCollatorForLanguageModeling`. That dataset change renders `DataCollatorForNextSentencePrediction` unusable, so it's removed (breaking change, though this is relatively contained, users that were using it should just use `DataCollatorForLanguageModeling` instead).

Tests are adapted to check `DataCollatorForLanguageModeling` does work for all those tasks.

In passing, all text datasets get their deprecation warning as we are now encouraging users to move the datasets library.


Comments:
Make tokenizer.pad() also pad `labels`
Author: changjonathanc
Pull request: no

# 🚀 Feature request

Make tokenizer.pad() also pad `labels`

## Motivation

I tried to use this:
https://github.com/huggingface/transformers/blob/8065fea87007fbf7542fc060ff8ddd0b5df567da/src/transformers/data/data_collator.py#L69 

But since labels is not padded, the result cannot turn into a tensor. `ValueError: Unable to create tensor, you should probably activate truncation and/or padding with 'padding=True' 'truncation=True' to have batched tensors with the same length.`

It currently pads `input_ids, attention_mask, token_type_ids, special_tokens_mask`
It seems logical to me that `tokenizer.pad()` should also pad `'labels'`. 

## Your contribution

I have already created a PR #8116. It solves the problem above.

Comments:
Create README.md
Author: yfpeng
Pull request: yes
Labels: name: model card

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @sshleifer
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->


Comments:
Data collator for token classification
Author: sgugger
Pull request: yes

# What does this PR do?

This PR adds a `DataCollatorForTokenClassification`, very similar to `DataCollatorWithPadding` but whose job is to pad the labels to the same size as the inputs.
In passing, it adds tests of `DataCollatorWithPadding` and cleans all the tests of various data collators that were marked as slow because they required a pretrained tokenizer. For the unit testing, no real tokenizer is needed since we just need the pas/mask token.

Comments:
I just tried it, and noticed that it doesn't work if `features` is `List[Dict[str, torch.Tensor]]`,
because `tokenizer.pad()` will set `return_tensors` to `pt` if `input_ids` is `torch.Tensor` and `return_tensors` is `None`.

For example my dataset looked like this.
```python
def  __getitem__(self, i):                                                                  
       return {k: torch.tensor(v, dtype=torch.long) for k,v in self.examples[i].items()}                     
```
Changing to this solves the problem.
```python
def  __getitem__(self, i):                                                                  
       return self.examples[i]      
```

Maybe I should have always used this and leave it to collator to tensorize the features.

Will have look. In general, yes it's better to have your examples be the results of the tokenization (so `Dict[str, List[int]]`) and let the data collator handles the conversion to tensors.
Corrected tpu typo in examples readme
Author: GuillemGSubies
Pull request: yes

 Before submitting
- [X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [X] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

 documentation: @sgugger

**EDIT**: Tests fail but that is definitely not due to my change


Comments:
Confirming the failure is spurious. Thanks a lot for your fix!