noamaanMulla-03/transformers-issues huggingface dataset:
Raise errors that are not raised now
Author: harupy
Pull request: yes
Labels: name: wontfix

Raise errors that are not raised now.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5012?src=pr&el=h1) Report
> Merging [#5012](https://codecov.io/gh/huggingface/transformers/pull/5012?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/f7c93b3ceec341c3c794e9fc18939bc5d50b0fc2&el=desc) will **increase** coverage by `0.00%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/5012/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/5012?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master    #5012   +/-   ##
=======================================
  Coverage   77.26%   77.26%           
=======================================
  Files         128      128           
  Lines       21856    21856           
=======================================
+ Hits        16887    16888    +1     
+ Misses       4969     4968    -1     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/5012?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5012/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `73.38% <0.00%> (-0.41%)` | :arrow_down: |
| [src/transformers/modeling\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/5012/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl91dGlscy5weQ==) | `87.57% <0.00%> (+0.31%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/5012?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/5012?src=pr&el=footer). Last update [f7c93b3...d643ec8](https://codecov.io/gh/huggingface/transformers/pull/5012?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Pinging @VictorSanh here

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


I'll close this.
added model_card for model codeswitch-hineng-lid-lince and codeswitch-spaeng-lid-lince
Author: sagorbrur
Pull request: yes
Labels: name: model card

Hi,
Thank you so much for this beautiful, awesome repo. 
I added model card for model `codeswitch-hineng-lid-lince` and `codeswitch-spaeng-lid-lince`.
Please check and if possible please merge. 

thanks and regards
Sagor

Comments:
Looks great! Maybe also add a link to https://github.com/sagorbrur/codeswitch?

Hi @julien-c,
thank you so much. 
regards
Sagor
ProphetNet
Author: aretius
Pull request: no
Labels: name: wontfix

name: New model

# 🌟 New model addition
ProphetNet 

## Model description
ProphetNet introduces a novel self-supervised objective named future n-gram prediction and the proposed n stream self-attention mechanism. Instead of the optimization of one-step-ahead prediction in the traditional sequence-to-sequence model, the ProphetNet is optimized by n-step ahead prediction which predicts the next n tokens simultaneously based on previous context tokens at each time step. The future n-gram prediction explicitly encourages the model to plan for the future tokens and prevent overfitting on strong local correlations

<!-- Important information -->

## Open source status

* [X] the model implementation is available: (give details)
https://github.com/microsoft/ProphetNet
* [X] the model weights are available: (give details)
Weights for both small and large pre-trained dataset version of models are available
https://github.com/microsoft/ProphetNet#pre-trained-models
* [X] who are the authors: (mention them, if possible by @gh-username)
Yu Yan @yuyan2do , Weizhen Qi @weizhen
 

Comments:
@aretius Thank you for mentioning ProphetNet.  ProphetNet for huggingface is sheduled as you suggested.  

@qiweizhen this sounds great, I would love to give it a go. Any planned date for delivering this?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Error when loading Flaubert model 
Author: saharghannay
Pull request: no
Labels: name: wontfix

# 🐛 Bug

## Information
I am trying to run the  example run_ner.pl using the Flaubert model. 
But I got this error: 

Traceback (most recent call last):
  File "/py37/lib/python3.7/site-packages/torch/serialization.py", line 191, in _check_seekable
    f.seek(f.tell())
AttributeError: 'NoneType' object has no attribute 'seek'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/py37/lib/python3.7/site-packages/transformers/modeling_utils.py", line 516, in from_pretrained
    state_dict = torch.load(resolved_archive_file, map_location="cpu")
  File "/py37/lib/python3.7/site-packages/torch/serialization.py", line 387, in load
    return _load(f, map_location, pickle_module, **pickle_load_args)
  File "/py37/lib/python3.7/site-packages/torch/serialization.py", line 549, in _load
    _check_seekable(f)
  File "/py37/lib/python3.7/site-packages/torch/serialization.py", line 194, in _check_seekable
    raise_err_msg(["seek", "tell"], e)
  File "/py37/lib/python3.7/site-packages/torch/serialization.py", line 187, in raise_err_msg
    raise type(e)(msg)
AttributeError: 'NoneType' object has no attribute 'seek'. You can only torch.load from a file that is seekable. Please pre-load the data into a buffer like io.BytesIO and try to load from it instead.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/py37/lib/python3.7/site-packages/transformers/modeling_auto.py", line 1098, in from_pretrained
    return model_class.from_pretrained(pretrained_model_name_or_path, *model_args, config=config, **kwargs)
  File "/py37/lib/python3.7/site-packages/transformers/modeling_utils.py", line 519, in from_pretrained
    "Unable to load weights from pytorch checkpoint file. "
OSError: Unable to load weights from pytorch checkpoint file. If you tried to load a PyTorch model from a TF 2.0 checkpoint, please set from_tf=True. 


I got an other error when I added  from_tf=True. 

Do you have idea how can I solve this problem.




Comments:
Hi! Could you show what command you used to launch the script?

I was runing the token classification example : https://github.com/huggingface/transformers/blob/master/examples/token-classification/run.sh with flaubert-large-cased as model name. 

I tried to  use the downloaded  model from https://huggingface.co/flaubert models, but after 100 epochs the results are very bad, the model did not learn any thing. 
I Don't  understand what is the problem with  flaubert-large-cased model. 
Note that flaubert-base-cased model give  good results on NER taks.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
🐛 [TFTrainer] Wrong number of optimization steps
Author: astariul
Pull request: no
Labels: name: wontfix

# 🐛 Bug

It seems TFTrainer compute the wrong number of optimization steps :

https://github.com/huggingface/transformers/blob/e4aaa4580515446cd5a2972ab42fec0b95819c84/src/transformers/trainer_tf.py#L79

**It does not take into account the gradient accumulation**.

---

I believe this line should be changed to :

```
if self.args.dataloader_drop_last:
    approx = math.floor
else:
    approx = math.ceil
self.train_steps: int = approx(self.num_train_examples / (self.args.train_batch_size * self.args.gradient_accumulation_steps))
```

---

Also, on TPU `drop_remainder` should be called on gradient accumulation steps as well. 

https://github.com/huggingface/transformers/blob/e4aaa4580515446cd5a2972ab42fec0b95819c84/src/transformers/trainer_tf.py#L81-L86

Should be changed to :

```
ds = (
    self.train_dataset.cache()
    .shuffle(self.num_train_examples)
    .batch(self.args.train_batch_size * self.args.gradient_accumulation_steps, drop_remainder=self.args.dataloader_drop_last)
    .unbatch()
    .batch(self.args.train_batch_size)
    .prefetch(tf.data.experimental.AUTOTUNE)
)
```

---

@jplu Maybe to add in #5065 ?


Comments:
You are right, the number of optimization steps is already fixed in an another PR (#5051). And indeed the batch size for TPUs has to take into account the gradient accumulation, but I put this for later because we don't have yet a proper way to differenciate a run on TPU/CPU/GPU. Also the `unbatch` then `batch` is not needed. Same thing apply later when computing the loss in the training step.

Thanks for your answer @jplu !

Can I ask you clarification about this :

>Also the unbatch then batch is not needed. Same thing apply later when computing the loss in the training step.

We still need the data to be batched based on the batch_size_per_device, not the total batch size, right ?
Is the `floor` / `ceil` needed ?

Is there any other changes I should apply locally to make it work ?

I'm pretty sure that the TPUs as to be set for the full size of batches (including those with the accumulation).

| Is the floor / ceil needed ?
`floor` and `ceil` are also needed in order to be sure you get the proper approximation.

| Is there any other changes I should apply locally to make it work ?

There are certainly other changes to make, but I still have to figure out which one :smile: Don't hesitate to participate on the PR I have opened ^^

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
**Specifically for Pegasus-arxiv** -  PegasusForConditionalGeneration - Error in loading state dictionary
Author: suchig
Pull request: no

Please note that this specifically happens only for **pegasus-arxiv** (Reopening Issue 6609). Before you close, please let me know if this works for pegasus-arxiv. Because I am getting the below error even after taking a fresh git fetch of transformers and removing all checkpoints from cache.

## Environment info
- `transformers` version: 3.0.2
- Platform: Linux-5.3.0-1034-azure-x86_64-with-debian-buster-sid
- Python version: 3.7.7
- PyTorch version (GPU?): 1.6.0 (True)
- Tensorflow version (GPU?): N/A
- Using GPU in script?:
- Using distributed or parallel set-up in script?:

### Who can help
@sshleifer
## Information

Model I am using (Bert, XLNet ...): **google/pegasus-arxiv**

## To reproduce
```ruby
mname = "google/pegasus-arxiv"
model = PegasusForConditionalGeneration.from_pretrained(mname)
```

throws error as below
File "/anaconda/envs/py37_default/lib/python3.7/site-packages/transformers/modeling_utils.py", line 894, in from_pretrained
model.class.name, "\n\t".join(error_msgs)
RuntimeError: Error(s) in loading state_dict for PegasusForConditionalGeneration:
size mismatch for model.encoder.embed_positions.weight: copying a param with shape torch.Size([512, 1024]) from checkpoint, the shape in current model is torch.Size([1024, 1024]).
size mismatch for model.decoder.embed_positions.weight: copying a param with shape torch.Size([512, 1024]) from checkpoint, the shape in current model is torch.Size([1024, 1024]).

Comments:
works for me with
```
- Python version: 3.7.4
- PyTorch version (GPU?): 1.5.1 (True)
```
Try
```
mname = "google/pegasus-arxiv"
model = PegasusForConditionalGeneration.from_pretrained(mname, force_download=True)
```


> works for me with
> 
> ```
> - Python version: 3.7.4
> - PyTorch version (GPU?): 1.5.1 (True)
> ```
> 
> Try
> 
> ```
> mname = "google/pegasus-arxiv"
> model = PegasusForConditionalGeneration.from_pretrained(mname, force_download=True)
> ```

Ok. It works now. Even though I did not use force_download, it downloaded a fresh copy of checkpoint. This can be closed.
Colab session crashes on transformers
Author: ghost
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     
     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question 
     that you posted.
     -->

## Details
Hi everybody, 
Something really strange happens in Colab after the recent updates of the DataCollate class.
I don't know if the two things are correlated, however, after I install the following packages

!git clone https://github.com/huggingface/transformers.git
!pip install ./transformers
!pip install -U nlp

and then I try to load them

import nlp
from transformers import T5Tokenizer

The Colab instance crashes.
Please find below the log.

**WARNING:root:kernel 485c962d-efa0-4103-9c68-ed22abd8839f restarted**




Comments:
The same thing for me. Colab (using TPU) crashes when importing transformers package.
![image](https://user-images.githubusercontent.com/35801846/84882301-ee662800-b086-11ea-9a1c-ff07853af864.png)

![image](https://user-images.githubusercontent.com/35801846/84882177-c5459780-b086-11ea-8c47-4b06eb3e8249.png)


Same issue.

I'm trying to reproduce, but can't manage to make colab crash. @khalilRhouma, @amitness, did you have similar code to @antoniomastro1996? Would it be possible for you to show me the code you used?

@LysandreJik yes, of course you can follow this:
https://colab.research.google.com/drive/1jwXgtOXE8v8_qkiOCbjFQRFC5semK8T7?usp=sharing

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
What happened to https://huggingface.co/zero-shot/ ?
Author: PhilipMay
Pull request: no

Hi, this page was very interesting. 
https://huggingface.co/zero-shot/

It is down since yesterday.
What is up with it? Do you plan to bring it up again?

Comments:
We're back up, and should now be setup to auto-restart when it fails. Thanks for the heads up!

Awsome! Thanks.

Hi! It seems the page is down again.

Man, streamlit's killing me. Thanks, rebooting now.

Ouch, this means we users are indirectly killing you! Thanks a lot. 🙂 

This page is giving me a 502. reopen this issue, please

Back up, and fixed the issue with auto-relaunch. Thanks for the heads up.

Hi! Thanks for the great demo!
The page seems to be down again.

This time it was actually just me making some changes. Back up.

I am getting a timeout on it... so, still not available I think...
Tokenizer.encode documentation not correct
Author: lubok-dot
Pull request: no

# 🐛 Bug

## Information

Model I am using Bert (bert-base-german-cased):

Language I am using the model on German:

The problem arises when using:
* tokenizer of this model

The tasks I am working on is:
* encoding

## To reproduce

Steps to reproduce the behavior:

1. lang_model = 'bert-base-german-cased'
2. tokenizer = BertTokenizer.from_pretrained(lang_model)
3. test_sentence = 'Das war gut'
4. tokenizer.encode(test_sentence)
output: [3, 295, 185, 1522, 4]
5. tokenizer.convert_tokens_to_ids(tokenizer.tokenize(test_sentence))
output: [295, 185, 1522]

## Expected behavior

According to the documentation of the encoding method in

https://huggingface.co/transformers/main_classes/tokenizer.html 

these two outputs should be the same. The problem is, that the _encode_ method adds special tokens [CLS] and [SEP] at the beginning and the end. However, the transformation in line 5. does not. This is not a problem at all but one should consider correcting the online-documentation. That is, the hint 

**Same as doing self.convert_tokens_to_ids(self.tokenize(text))**

in the encode method is misleading. Instead, one could also add, that these two commands are the same if _add_special_tokens_ is set _False_ in the _encode_ method.

## Environment info
     
- `transformers` version: 2.8.0
- Platform: Windows 10
- Python version: 3.8.3
- PyTorch version (GPU?): 1.5.0+cpu
- Tensorflow version (GPU?): None
- Using GPU in script?:No
- Using distributed or parallel set-up in script?:No


Comments:
You're right! The documentation should be updated.

Done! Thanks for raising an issue :)

You are welcome. Thank you for the transformers package -- great work!

Hi, I was checking the documentation page however, could not find the documentation for encode and encode_plus. 

[Documentation Link](https://huggingface.co/transformers/main_classes/tokenizer.html#transformers.PreTrainedTokenizer.encode)

Can someone point me to the right documentation for these methods?

Hi, I think, in the newer version of the transformer package, the encode_plus method has been consumed by the __call__ method which yields (as batch_encode_plus, and encode_plus) a BatchEncoding object. The preamble of the Tokenizer documentation contains this information. 

@ank-shukla the response given by @lubok-dot is correct, the `__call__` method should be used instead of `encode` and `encode_plus` in newer versions. If you're on an older version or would still like to use said methods, please check an older version of the documentation, for example [v2.11.0](https://huggingface.co/transformers/v2.11.0/main_classes/tokenizer.html#transformers.PreTrainedTokenizer.encode_plus)

@lubok-dot @LysandreJik Thanks for the clarification. I am on the new version and the said methods encode and encode_plus still work as expected. I understand now that __call__ method can carry out the same process based on the parameters.
How to convert tokenizer output to train_dataset which is required by Trainer API 
Author: questpavan
Pull request: no

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     -->

## Details
<!-- Description of your issue -->

I tried doing tokenisation using documentation of huggingface transformers.

```
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained('bert-base-cased')
encoded_input = tokenizer(batch_of_sequences)
```

Pre Trained Tokenizer gives output of dictionary containing three keys which are

```
encoded_input = {
'input_ids': [[],[],[]],
'token_type_ids': [[],[],[]],
'attention_mask': [[],[],[]]
}

```

Trainer API requires input of Train & Eval Dataset of type `torch.utils.data.Dataset`.

How can we use this output to create training dataset required for Trainer API?

<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**:

https://stackoverflow.com/questions/63519373/how-to-convert-tokenizer-output-to-train-dataset-which-is-required-by-trainer-ap

Comments:
Hi @questpavan ,
This awesome [tutorial](https://huggingface.co/transformers/master/custom_datasets.html) walks you through how you can fine-tune transformer models using custom dataset. It also covers pre-processing and how to create dataset etc.

Thanks @patil-suraj. It helped a lot.
Don't reset the dataset type + plug for rm unused columns
Author: sgugger
Pull request: yes

This PR avoids resetting the dataset type when removing columns, and also introduces a filed in `TrainingArguments` to disable that behavior (in case the use wants to use some of those fields in an elaborate data collator).

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6683?src=pr&el=h1) Report
> Merging [#6683](https://codecov.io/gh/huggingface/transformers/pull/6683?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/1a779ad7ecb9e5215b6bd1cfa0153469d37e4274?el=desc) will **decrease** coverage by `0.04%`.
> The diff coverage is `14.28%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/6683/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/6683?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #6683      +/-   ##
==========================================
- Coverage   79.65%   79.60%   -0.05%     
==========================================
  Files         156      156              
  Lines       28250    28256       +6     
==========================================
- Hits        22503    22494       -9     
- Misses       5747     5762      +15     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/6683?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/trainer.py](https://codecov.io/gh/huggingface/transformers/pull/6683/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90cmFpbmVyLnB5) | `55.63% <0.00%> (-0.53%)` | :arrow_down: |
| [src/transformers/training\_args.py](https://codecov.io/gh/huggingface/transformers/pull/6683/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy90cmFpbmluZ19hcmdzLnB5) | `91.34% <100.00%> (+0.08%)` | :arrow_up: |
| [src/transformers/generation\_tf\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/6683/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9nZW5lcmF0aW9uX3RmX3V0aWxzLnB5) | `84.21% <0.00%> (-2.51%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/6683?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/6683?src=pr&el=footer). Last update [1a779ad...eea7e8d](https://codecov.io/gh/huggingface/transformers/pull/6683?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).