noamaanMulla-03/transformers-issues huggingface dataset:
FSMT Training scripts
Author: wangyong1122
Pull request: no
Labels: name: wontfix

Hi, thank you very much for your implementation and it helps a lot. I want to train the MT models from scratch. Could you provide some instructions about the training details for FSMT? Thanks a lot. @stas00 

Comments:
Thank you for the kind words, @wangyong1122.

The initial task was to port the weights so that it could be used for Translation as is, which is mostly done. I'm still working out some kinks under CUDA - I forgot to test with USE_CUDA=1 :( But I'm almost there.

I'd say, how about you start the training - like you'd any other `transformers` model that you already have the instructions for and I'm 100% sure you will encounter problems since that part wasn't worked on/tested at all. So when you run into problems please post the code so that I could reproduce the issues and slowly slowly we will sort it out. How does that sound?

It's my first model ported to `transformers` so things take much longer as I'm learning the nuances of this platform.

Thanks very much for your kind reply, @stas00.
It sounds great! If I run into problems, I will post the code.

To explain why I expect problems: `transformers` has been written with a single merged vocab in mind, whereas fairseq's transformer has 2 distinct vocabs of different size (for en-ru/ru-en, but merged for en-de/de-en). So I had to do all kinds of hacks to make things work - for example to overcome `self.vocab_size` usages in core functions, as here we have `self.src_vocab_size` and `self.tgt_vocab_size`. 

So for the initial task of making translation work - almost everything should be fine, as most of it is well tested. 

But for training from scratch and finetuning I expect problems - so we will solve them gradually as you uncover them. 

The best help you can provide is to code those problems in new tests. If you can't, that's ok too, this is just if you'd like an extra learning challenge. But in any case, please, make small simple reproducible examples - train on 3 lines of text that you can hardcode here, and not 3GB that need to be set up :)

Thanks very much for your details. @stas00.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
A confusion about mrc model
Author: Maybewuss
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

- I found all of pretrain models using for mrc use softmax to pred start and end probability. 

- If a paragraph has multi-answers, softmax not worked, otherwise, adding some postprocess to combine start and end probability to sort that we can choose n-best answer, but the numbert is  fixed for all paragaph.
- Squad 2.0 has unanswerable questions and i found  data processor define start index == end index == 0 to handle this situation.
- So my confusion is why not use sigmoid instead of softmax in the last layer, if i use sigmoid, i can deal with muti-answers or non-answer easily....


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Multitask pre-training setting
Author: ghost
Pull request: no
Labels: name: wontfix

# 🚀 Feature request

Hi everybody!

Is it possible to pre-train T5 model on a mixture of tasks?
Just to be more clear, let's assume I have created a dataset by following the example provided here:
https://huggingface.co/transformers/model_doc/t5.html

Specifically I'm referring to the _**Unsupervised denoising training**_ and _**Supervised training**_.
What I've done is to construct a huge dataset where each tasks is concatenated to the previous one.

Is this enough to experiment with the multitask pertaining setting?






Comments:
Hey @antoniomastro1996,

Sure you can train T5 on a mixture of tasks. If you frame each tasks as a text-to-text task and mix it with *unsupervised denoising training* training examples, you would have the pre-training setting used in the original T5 paper.

This might also help: https://github.com/huggingface/datasets/issues/217

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
TypeError: __init__() got an unexpected keyword argument 'gradient_checkpointing'
Author: xixiaoyao
Pull request: no
Labels: name: wontfix

## Environment info

     
- `transformers` version: 3.1.0
- Platform: Linux-3.10.0_3-0-0-17-x86_64-with-debian-buster-sid
- Python version: 3.7.7
- PyTorch version (GPU?): 1.6.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help
 
 Longformer/Reformer: @patrickvonplaten

 -->

## Information

Model I am using (Longformer):

The problem arises when using:
[ * ] the official example scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: SQuAD v1.1

## To reproduce

Steps to reproduce the behavior:

1. add gradient_checkpointing argument for AutoModelForQuestionAnswering in examples/question-answering/run_squad.py
2. run with longformer-base-4096

runtime errors as belows
```
File "run_squad.py", line 821, in <module>
    main()
  File "run_squad.py", line 739, in main
    gradient_checkpointing=True,
  File "/opt/conda/lib/python3.7/site-packages/transformers/modeling_utils.py", line 852, in from_pretrained
    model = cls(config, *model_args, **model_kwargs)
TypeError: __init__() got an unexpected keyword argument 'gradient_checkpointing'
```

even if you explicitly replace AutoModel with LongformerModel like following, error is the same.

![image](https://user-images.githubusercontent.com/24541791/92999722-d1da6e80-f555-11ea-9588-2269fcacee90.png)

When I run with python/ipython interactive mode, the model loaded success.

![image](https://user-images.githubusercontent.com/24541791/92999756-0e0dcf00-f556-11ea-8845-8a661dab1e08.png)

And I have ensured the python enviroment is the same during these two runs.

## Expected behavior

Longformer can be loaded with gradient checkpointing


Comments:
Hey @xixiaoyao,

Sorry I cannot reproduce the error. Both when running `run_squad.py` and this snippet (which is essentially the same as in `run_squad.py`):

```python
from transformers import AutoModelForQuestionAnswering
model = AutoModelForQuestionAnswering.from_pretrained("allenai/longformer-base-4096", gradient_checkpointing=True)
```

I do not get any error. Can you post a code snippet (at short as possible) that I could copy paste to reproduce the error? 

Thanks!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
When I updated my transformers to the latest, the previously trained model loaded with an error
Author: wulaoshi
Pull request: no
Labels: name: wontfix

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version:'3.1.0'
- Platform:
- Python version:
- PyTorch version (GPU?):
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?:

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.
 
 albert, bert, GPT2, XLM: @LysandreJik 
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao 
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @TevenLeScao
 blenderbot: @mariamabarham
 Bart: @sshleifer
 Marian: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao 
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->

## Information

Model I am using (Bert, XLNet ...):

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1.
2.
3.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->


Comments:
![image](https://user-images.githubusercontent.com/27938964/93694930-3f812e80-fb44-11ea-938e-6e36e51bf183.png)


Duplicate of https://github.com/huggingface/transformers/issues/6882 I think

> Duplicate of #6882 I think

Thanks. I'll look into it.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
KeyError with DPR reader in Question Answering pipeline
Author: antoniolanza1996
Pull request: no
Labels: name: wontfix

## Environment info
- `transformers` version: 3.0.2 (I have also tried with current `master`)
- Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic (with Google Colab)
- Python version: 3.6.9
- PyTorch version (GPU?): 1.6.0+cu101 (False)
- Tensorflow version (GPU?): 2.3.0 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
@LysandreJik

## Information

Model I am using: DPR reader (i.e. https://huggingface.co/facebook/dpr-reader-single-nq-base) to perform a Question Answering task.
## To reproduce

Steps to reproduce the behavior:

```from transformers import pipeline
model_name="facebook/dpr-reader-single-nq-base"
tokenizer_name="facebook/dpr-reader-single-nq-base"
my_pipeline = pipeline("question-answering", model=model_name, tokenizer=tokenizer_name)
```

Error message:
``` KeyError                                  Traceback (most recent call last)
<ipython-input-2-b3a2784cffcf> in <module>()
      4 model_name="facebook/dpr-reader-single-nq-base"
      5 tokenizer_name="facebook/dpr-reader-single-nq-base"
----> 6 my_pipeline = pipeline("question-answering", model=model_name, tokenizer=tokenizer_name)

2 frames
/usr/local/lib/python3.6/dist-packages/transformers/configuration_auto.py in from_pretrained(cls, pretrained_model_name_or_path, **kwargs)
    208 
    209         if "model_type" in config_dict:
--> 210             config_class = CONFIG_MAPPING[config_dict["model_type"]]
    211             return config_class.from_dict(config_dict, **kwargs)
    212         else:

KeyError: 'dpr'
```

Comments:
Hi @LysandreJik and @lhoestq ,
FYI, there is still this error on Transformers.
Have you planned to support DPR into pipeline API?

Thanks

Thanks for reporting !
The reader is currently not configured as a model for question answering indeed, but that's definitely something we'll change.
We'll also make it loadable as an an open-domain question answering Pipeline object.

Hi @lhoestq ,
I got it.
Is there any approximate release date?
I believe that open-domain QA Pipeline will be really useful, thank you. 



This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Add support for DeBERTa
Author: stefan-it
Pull request: no
Labels: name: wontfix

name: New model

# 🌟 New model addition

## Model description

DeBERTa (Decoding-enhanced BERT with disentangled attention) is a new model architecture:

>  In this paper we propose a new model architecture DeBERTa (Decoding-enhanced BERT with disentangled attention) that improves the BERT and RoBERTa models using two novel techniques. The first is the disentangled attention mechanism, where each word is represented using two vectors that encode its content and position, respectively, and the attention weights among words are computed using disentangled matrices on their contents and relative positions. Second, an enhanced mask decoder is used to replace the output softmax layer to predict the masked tokens for model pretraining. We show that these two techniques significantly improve the efficiency of model pre-training and performance of downstream tasks. 

The paper can be found [here](https://arxiv.org/abs/2006.03654).

## Open source status

* [x] the model implementation is available: [GitHub](https://github.com/microsoft/DeBERTa)
* [x] the model weights are available: [GitHub release](https://github.com/microsoft/DeBERTa/releases/tag/v0.1)
* [ ] who are the authors: @BigBird01


Comments:
Hello, Our code has just been released at [DeBERTa]( https://github.com/microsoft/DeBERTa)
Please take a try and your feedback will be good for our improvements, we also welcome the community to work together with us to improve it.
We are also glad to integrate DeBERTa into transformers.


PR Add deberta model #5929 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Unstale - very close to merge!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Support serialized tokenizer in AutoTokenizer
Author: djstrong
Pull request: no
Labels: name: wontfix

# 🚀 Feature request

Support loading serialized tokenizer (using `Tokenizer.from_file`) in `AutoTokenizer`.


## Motivation

Some standard models use different tokenizers, e.g. `SentencePieceBPETokenizer`. So, there would be no need for implementing new models in `transformers`.


Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
[Question] Pegasus tokenizer
Author: patrickvonplaten
Pull request: no

@sshleifer - sorry to ping you here on this. Would be amazing if you find some time to explain the Pegasus tokenizer a bit.

A couple of things I don't understand:

- In the official Pegasus Tokenizer and from reading the paper it seems that exactly 2 mask tokens are necessary.
    See https://github.com/google-research/pegasus/blob/master/pegasus/ops/pretrain_parsing_ops.cc#L66
   a) ID=2 seems to correspond to the sentence mask token, called `[MASK_1]` and 
   b) ID=3 seems to correspond to the word mask token, called `[MASK_2]`

=> Why don't we have `[MASK_1]` and `[MASK_2]` tokens in the tokenizer's special tokens? I would actually add them at the id's 2 and 3 instead of having `unk_2` and `unk_3` there. Wdyt?

- Why do we call the tokens unk_2 - unk_104 ? Why unk? And why aren't those part of the `special_tokens_map` - is this on purpose? 

- Why does Pegasus inherit from the Reformer Tokenizer -> I don't really see what they have in common...

Would be awesome if you could take 10min to reply :-) 

Comments:
No problem at all!

+ The inheritance is just for the purpose of not duplicating code.
+ You can change to whatever unk you would like, i wasn't at all careful about this stuff since i didn't try to replicate/test pre-training, just fine-tuning and generation.
+ Your changes sound like obvious low risk improvements. 
+ I don't know whether standard mask-filling will work for integration testing purposes, given the seq2seq pre-training objective .
PEGASUS do not have mask token
Author: ShichaoSun
Pull request: no

@mfuntowicz @patrickvonplaten 

Hi,

I am using PEGASUS  google/pegasus-large

I would like to fill the mask sentence of a document , i.e. try the pretraining task. But I don't find the mask_token. 

Steps to reproduce the behavior:

1. from transformers import PegasusTokenizer
2. tok = PegasusTokenizer.from_pretrained("google/pegasus-large")
3. tok.mask_token

output is "Using mask_token, but it is not set yet."




Comments:
Hey @ShichaoSun - thanks for the issue! 
 I agree with you that Pegasus should have some mask tokens defined and I'd set `tokenizer.mask_token` to Pegasus' MLM mask token => `[MASK_2]` and add an additional `mask_token_sent` ... 

I'm still waiting for some more insight on Pegasus of the Pegasus expert @sshleifer -> https://github.com/huggingface/transformers/issues/8689 .

I'll hope to get an answer there to be sure that adding the `[MASK_1]` and `[MASK_2]` tokens is the correct thing to do here!

Hi @patrickvonplaten ,
Really thanks for your reply and great job !

Is it possible to MASK several tokens using Pegasus?
Minor docs typo fixes
Author: guyrosin
Pull request: yes

# What does this PR do?
Just a few typo fixes in the docs.

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

## Who can review?
@sgugger

Comments:
Looks like you have a styling problem, could you run the command `make style` after doing a dev install with
```
pip install -e .[dev]
```
in the repo?

Oops! Done.
cannot run "examples/language-modeling/run_mlm.py"
Author: HenryPaik1
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version:
- Platform: cmd
- Python version: 3.7
- PyTorch version (GPU?): 1.6.0
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?:

### Who can help
 albert, bert, GPT2, XLM: @LysandreJik 

## Information

Model I am using (Bert)

The problem arises when using:
* [o] the official example scripts: (give details below)
* [] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [o] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:
```
python examples/language-modeling/run_mlm.py

>>
Traceback (most recent call last):
  File "examples/language-modeling/run_mlm.py", line 30, in <module>
    from datasets import load_dataset
ModuleNotFoundError: No module named 'datasets'
```


Comments:
You need to install datasets library: https://github.com/huggingface/datasets
```
pip install datasets
```

Thanks!
Migration guide from v3.x to v4.x
Author: LysandreJik
Pull request: yes

Additionally to the release notes, this completes the migration guide to showcase the expected breaking changes from v3.x to v4.x, and how to retrieve the original behavior.

Comments:
Will update the switch to fast tokenizers when a decision has been made @thomwolf @n1t0 

@sgugger please let me know if this is what you had in mind.
"BertForMaskedLM - pretrained model" cannot resize vocab output size
Author: HenryPaik1
Pull request: no


### Who can help
 albert, bert, GPT2, XLM: @LysandreJik 

## Information

Model I am using (Bert)

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [0] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [0] my own task or dataset: (give details below)

## To reproduce
I resized embedding dim, but output dim cannot change. Please kindly refer to the code below:
```
from transformers import BertTokenizer, BertForMaskedLM
from transformers import LineByLineTextDataset

model = BertForMaskedLM.from_pretrained('bert-base-uncased')

###### tokenizer
tokenizer_path = './data/transformer_tokenizer_add_entitymasking_token.pt/'
tokenizer = BertTokenizer.from_pretrained(tokenizer_path)
model.bert.resize_token_embeddings(len(tokenizer))
model.cls.predictions.decoder.out_features = len(tokenizer)

out[0].shape, label.shape
>>>
(torch.Size([2, 20, 30522]),  torch.Size([2, 20])) # 30522 should be len(tokenizer) 30544
```


Comments:
I solved it by following code:
```
with torch.no_grad():
    replace_linear = torch.nn.Linear(in_features=768, out_features=len(tokenizer))
    replace_linear.weight[:30522,:].copy_(model.cls.predictions.decoder.weight)
    model.cls.predictions.decoder = replace_linear
    model.cls.predictions.decoder = model.cls.predictions.decoder.requires_grad_(True)
```
Add T5 Encoder for Feature Extraction
Author: agemagician
Pull request: yes

# What does this PR do?

While using T5 for feature extraction, I found out that T5 encoder provides better features than T5 decoder. Hence, it makes sense to have T5 encoder only, which should reduce the memory and inference time by half, if feature extraction is needed rather than conditional generation. 

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

 T5: @patrickvonplaten
 tensorflow: @jplu


Comments:
I like it! 

Great, I am glad that you did like it.

Thanks @patrickvonplaten and @jplu for your feedback.

@patrickvonplaten :
I have adjusted all your code review, and also add it to the T5 documentation.
The only missing part is the tests, it will be great if you could add it.

@jplu :
I have removed the unnecessary parameters from the TF model.

Is there anything else needed from my side to merge the pull request ?

> Great, I am glad that you did like it.
> 
> Thanks @patrickvonplaten and @jplu for your feedback.
> 
> @patrickvonplaten :
> I have adjusted all your code review, and also add it to the T5 documentation.
> The only missing part is the tests, it will be great if you could add it.
> 
> @jplu :
> I have removed the unnecessary parameters from the TF model.
> 
> Is there anything else needed from my side to merge the pull request ?

I think that's great! I'll fix the tests and merge :-) 

## Update:

PR is ready for review IMO. Would be great if @LysandreJik @jplu and @sgugger you can take a look :-) 

> ## Update:
> PR is ready for review IMO. Would be great if @LysandreJik @jplu and @sgugger you can take a look :-)

Thanks a lot @patrickvonplaten ^_^

> Very clean implementation, thanks a lot @agemagician!

You are welcome.
I am glad that I could help making the library better, even with a small contribution.
I have to say without @patrickvonplaten help, I could not make it ^_^