noamaanMulla-03/transformers-issues huggingface dataset:
Reformer for questions answering(squad)
Author: bela572
Pull request: no

I want to use Reformer for Questions Answering.  I tried to use pretrained model 'google/reformer-crime-and-punishment'. I was using this example https://huggingface.co/transformers/custom_datasets.html#qa-squad just replaced with reformer. I get the exception related with pad, cls tokens, sequence length and so on, but that does not matter now, I first want to know: is it even possible to get good results(model to answer questions more than 50 percent accuracy for example? Because I see that you working on this model now and some functions maybe are not implemented yet or so. I saw issues like this: https://github.com/huggingface/transformers/issues/5436 and you wrote that you be very surprised if it will give any good results using reformer for q&a. (but it was year ago, so maybe it changed)
Thanks

Comments:
It hasn't changed yet. The problem is not code, the problem is the lack of a decent pre-trained model.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
add the chinese ref in place to tackle the memory issue
Author: orctom
Pull request: yes

# What does this PR do?

@wlhgtc 
It eats my 200+GB memory when adding the chinese refs, OOM killed, can't go further.
My train corpus has 17067704 lines (size: 1GB)

datasets 1.6.2 come with a new in-place add_column function, which saves the heavy coping.

Comments:
Both two ways work well on my own process, but `add_columns` is 4x faster than the original method.
![CleanShot 2021-05-13 at 10 19 22@2x](https://user-images.githubusercontent.com/16603773/118068382-da30c500-b3d4-11eb-8a74-f169593a482b.png)
LGTM!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
DeBERTa pretraining data preparation
Author: mansimane
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:
- Platform: 4.6.0.dev0
- Python version: 3.6
- PyTorch version (GPU?): 1.6
- Tensorflow version (GPU?):
- Using GPU in script?: Y
- Using distributed or parallel set-up in script?: Y

### Who can help
 --> @LysandreJik @BigBird01 




## Information

Model I am using (Bert, XLNet ...): DeBERTa

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: (give the name): MLM + SQUAD 1
* [ ] my own task or dataset: (give details below)



<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
I am pretraining DeBERTa Base from scratch on Wikipedia + Book Corpus dataset. After pretraining for 500K steps I observe SQUAD 1.1 score of 76 which is much less than Figure 1(b) in paper (although figure 1b reports squad 2.0 numbers) squad 1.1 numbers would be much better than that as it is easier task. Using same hyperparameters as reported in paper. I would like to confirm the preprocessing steps that authors took to prepare pretraining data. 

1. In section 4.4.1, authors report that they used Megatron code base to deduplicate the data.  The code provided performs deduplication based on urls. https://github.com/NVIDIA/Megatron-LM/tree/main/tools/openwebtext Was the deduplication performed on url -> document set or on shards of dataset? 
2. [This](https://github.com/NVIDIA/Megatron-LM/blob/main/tools/openwebtext/cleanup_dataset.py) codebase also cleans up the dataset based and removes non-english characters. Were these data cleanup steps performed on pretraining data? 
3. Is it possible to provide scripts used to generate pretraining data? 

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Mbart continue training with same training task on a specific language
Author: maroxtn
Pull request: no

Hello.

I am not sure if this is possible using the transformers library, but it isn't it would be nice to have.

Mbart was initially trained using span corruption and other training tasks on a corpus containing many languages. Since I am going to use it for specifically the Arabic language, I wish to solely finetune it on Arabic text with the same training tasks it had, only then at a third step to fine tune for specific text generation task.

I believe this is possible using fairseq, but having it here in the transformers library would better. Is that possible / useful per your opinion ?

Comments:
Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discuss.huggingface.co) instead?

Thanks!

@LysandreJik  Sorry, I will close the issue now. 
However this might qualify as a useful feature request, don't you think?
Pretraining for TFWav2Vec2
Author: will-rice
Pull request: no

# 🚀 Feature request

TFWav2Vec2 needs the pretraining implementation like the PyTorch [version](https://huggingface.co/transformers/master/model_doc/wav2vec2.html#wav2vec2forpretraining)

## Motivation

Users of the TensorFlow model will most likely want to be able to do pretraining just like with Pytorch. 

## Your contribution

I recently added the [tensorflow model](https://github.com/huggingface/transformers/pull/11617). So I would like to do this one as well. 


Comments:
[Flax] [WIP] allow loading head model with base model weights
Author: patil-suraj
Pull request: yes
Labels: name: Flax

# What does this PR do?
Allows loading flax head model with base model weights.

Right now it's not possible to load a flax head model using weights from the base model as weights `dict` of the base model does not contain the `base_model_prefix` key. To reproduce

```python
from transformers import BertConfig, FlaxBertModel, FlaxBertForSequenceClassification
config = BertConfig(hidden_size=64, intermediate_size=128, max_position_embeddings=128, num_attention_heads=8, num_hidden_layers=8)
base_model = FlaxBertModel(config)
base_model.save_pretrained("base")
head_model = FlaxBertForSequenceClassification.from_pretrained("base")
```

Comments:
Thanks for fixing this! Could we maybe also add some tests that make sure that loading/saving works correctly for all models?
[deepspeed] supporting `--adafactor` 
Author: stas00
Pull request: no
Labels: name: DeepSpeed

name: WIP

It was flagged that in this example https://github.com/huggingface/transformers/issues/11044 `--adafactor` is used, but Deepspeed doesn't get it passed since the DS config's optimizer overrides it. So need to sort it out.

Comments:
[DeepSpeed] don't ignore --adafactor
Author: stas00
Pull request: yes

This PR adds a small improvement that checks that if `--adafactor` is passed and the DS config has an optimizer section, then we assert. Before that it was silently ignored, which was misleading to the user.

Fixes: https://github.com/huggingface/transformers/issues/11749

@sgugger 

Comments:
[Flax] Add jax flax to env command
Author: patrickvonplaten
Pull request: yes

This PR adds jax/flax libs to the env command.

@lewtun - can you maybe try out running `transformers-cli env` after this command to see why you cannot import `FlaxBigBirdForMaskedLM`

Comments:
If most of the tests pass, then can I use the changes made by the PR in the time being to check whether I can atleast start my training?
[WIP] Fix cross attentions for TF T5
Author: stancld
Pull request: yes

This PR fixes cross attentions for TF T5 model. This includes adding a new input argument `cross_attn_head_mask` and also adding `cross_attentions` to the model's output.

<hr>

**Reviewers:** @patrickvonplaten @Rocketknight1 

Comments:
Hosted inference api keeps returning 400 error
Author: kevhahn97
Pull request: no

I'm not sure if it's okay to make issue with this topic, but I couldn't find a place to share my problem so I'm making an issue.

### Problem description
When I try to inference a public model (facebook/blenderbot-1B-distill), it keeps returning 400 error with message below, whether I tried it on model hub or through HTTP request.
`'We could not properly load your model with any of the classes {model_classes}, are you sure this model can be loaded with the specified task ?'`

I used this model normally a few days ago, but now it's not working. May I ask for a help? Any advice would be appreciated..

![image](https://user-images.githubusercontent.com/5675167/121697156-bd88c980-cb07-11eb-94f6-5b03222d3ae2.png)


Comments:
I found it resolved by fixing model config. Now it's working
[tests] reset report_to to none, avoid deprecation warning
Author: stas00
Pull request: yes

This PR fixes the warnings:
```
The default value for the training argument `--report_to` will change in v5 (from all installed integrations to none). 
In v5, you will need to use `--report_to all` to get the same behavior as now. You should start updating your code 
and make this info disappear :-).
```
by setting `report_to=[]`, which also makes the tests a tad faster by not doing any reporting, unless the test explicitly asks for it.

@sgugger 

Comments:
[tests] multiple improvements
Author: stas00
Pull request: yes

This PR:

1. splits the 2 groups of `TrainerIntegrationTest` tests into separate subclasses, since at the moment many run 2x training in `setUp` and then don't use the results - should make things a bit faster, and mainly this removes the weird unexpected logs when debugging tests.
2. introduce, use and document `require_torch_up_to_2_gpus` as we have a bunch of tests that can only run on up to 2 gpus and currently they weren't `skipped`, but hacked to return reporting `passed`!
3. fix `test_resume_training_with_randomness` to use `assertAlmostEqual` so that we get debug data when it fails - and also fix the comment to match the code that this test only works with 0 or 1 gpus - and use the marker.

@sgugger 

Comments:
Different Weights between google-bert (uncased_L-12_H-768_A-12) and Huggingface-bert (bert-base-uncased)
Author: Doragd
Pull request: no

Environment: torch==1.8.1+cu111  transformers==4.3.3

```python
# google-bert (uncased_L-12_H-768_A-12)
model = BertForPreTraining.from_pretrained('uncased_L-12_H-768_A-12/', from_tf=True)
print(model.bert.embeddings.word_embeddings.weight)
```
```python
# google-bert (uncased_L-12_H-768_A-12) output
Parameter containing:
tensor([[-0.0314, -0.0045,  0.0182,  ..., -0.0309,  0.0204, -0.0345],
        [-0.0295, -0.0486,  0.0746,  ..., -0.0363,  0.0262, -0.0108],
        [-0.0328, -0.0582, -0.0149,  ..., -0.0932,  0.0444,  0.0221],
        ...,
        [-0.0337, -0.0518, -0.0280,  ..., -0.0174,  0.0078, -0.0010],
        [-0.0022, -0.0297, -0.0167,  ..., -0.0472, -0.0006,  0.0128],
        [-0.0631, -0.0144, -0.0232,  ...,  0.0072, -0.0704, -0.0479]],
       requires_grad=True)
```

```python
# Huggingface-bert (bert-base-uncased)
model = BertModel.from_pretrained('bert-base-uncased')
print(model.embeddings.word_embeddings.weight)
```
```python
# Huggingface-bert (bert-base-uncased) output
Parameter containing:
tensor([[-0.0102, -0.0615, -0.0265,  ..., -0.0199, -0.0372, -0.0098],
        [-0.0117, -0.0600, -0.0323,  ..., -0.0168, -0.0401, -0.0107],
        [-0.0198, -0.0627, -0.0326,  ..., -0.0165, -0.0420, -0.0032],
        ...,
        [-0.0218, -0.0556, -0.0135,  ..., -0.0043, -0.0151, -0.0249],
        [-0.0462, -0.0565, -0.0019,  ...,  0.0157, -0.0139, -0.0095],
        [ 0.0015, -0.0821, -0.0160,  ..., -0.0081, -0.0475,  0.0753]],
       requires_grad=True)
```


**Why the output of `embeddings.word_embeddings.weight` is different?**



Comments:
When checking the  [model card]() of `google/bert_uncased_L-12_H-768_A-12`, it states the following:

> Note that the BERT-Base model in this release is included for completeness only; it was re-trained under the same regime as the original model.

Hence, `google/bert_uncased_L-12_H-768_A-12` is a retrained version of `bert-base-uncased`. So of course, the parameter values will differ.

@NielsRogge  Is the `google/bert_uncased_L-12_H-768_A-12` equal to `https://storage.googleapis.com/bert_models/2020_02_20/uncased_L-12_H-768_A-12.zip`
[Documentation] Example for LEDForConditionalGeneration does not work
Author: ionicsolutions
Pull request: no

The [documentation for LEDForConditionalGeneration](https://huggingface.co/transformers/model_doc/led.html#transformers.LEDForConditionalGeneration) appears to be incorrect. The same example is also used for [BartForConditionalGeneration](https://huggingface.co/transformers/model_doc/bart.html#bartforconditionalgeneration), where it works as intended. I believe that the example was just copied and not adapted, but perhaps I'm missing something? 

```python
from transformers import LEDTokenizer, LEDForConditionalGeneration
tokenizer = LEDTokenizer.from_pretrained('allenai/led-base-16384')
TXT = "My friends are <mask> but they eat too many carbs."

model = LEDForConditionalGeneration.from_pretrained('allenai/led-base-16384')
input_ids = tokenizer([TXT], return_tensors='pt')['input_ids']
logits = model(input_ids).logits
```
Here, the last step fails with `ValueError: You have to specify either decoder_input_ids or decoder_inputs_embeds`, which as far as I can tell is not a bug, but expected, as no `decoder_input_ids`/`embeds` (or `labels`) are provided. (BART [silently generates the `decoder_input_ids` from the `input_ids`](https://github.com/huggingface/transformers/blob/master/src/transformers/models/bart/modeling_bart.py#L1151), which LED does not.)

I believe the example should look like this:
```python
input_ids = tokenizer([TXT], return_tensors='pt')['input_ids']

prediction = model.generate(input_ids)[0]
print(tokenizer.decode(prediction, skip_special_tokens=True))
# My friends are good at eating healthy but they eat too many carbs.
```
This is also a nice demonstration that LED generates more than just one token for the masked parts of the sequence.

Tagging @patrickvonplaten who contributed the model and the example.

Comments:
Fix and improve documentation for LEDForConditionalGeneration
Author: ionicsolutions
Pull request: yes

# What does this PR do?

As reported in #12268, the example for text generation with LED does not work because it relies on an implementation detail of the BART model for which it was originally conceived. Further, the summarization example uses a checkpoint that was not finetuned on a summarization task, leading to the model just repeating the entire input.

This PR replaces both examples with versions that are fully functional and illustrate the respective task.

<!-- Remove if not applicable -->

Fixes #12268

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@sgugger @patrickvonplaten 

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Comments:
Thanks again!
Issue getting prediction_scores from TransfoXLHeadLM model when labels are provided
Author: RainIwakura
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.5.1
- Platform: Colab
- Python version: 3.8.1
- PyTorch version (GPU?): 1.8.1+cu101 (Yes), same bug reproduced on CPU side (windows 10)
- Tensorflow version (GPU?): 
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help
@patrickvonplaten

## Information

Model I am using (Bert, XLNet ...): TransfoXL

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)
Slightly modified example from TransfoXL docs (https://huggingface.co/transformers/model_doc/gpt2.html#gpt2lmheadmodel)

## To reproduce

Steps to reproduce the behavior:

```python
import torch
from transformers import TransfoXLTokenizer, TransfoXLLMHeadModel

tokenizer = TransfoXLTokenizer.from_pretrained('transfo-xl-wt103')
model = TransfoXLLMHeadModel.from_pretrained('transfo-xl-wt103')



with torch.no_grad():
	inputs = tokenizer("Hello, my dog is cute", return_tensors="pt",)
	print(inputs)
	outputs = model(inputs["input_ids"],return_dict=True,labels=inputs["input_ids"])

	print(outputs['prediction_scores'])
```

## Expected behavior
```outputs['prediction_scores']``` should return a torch.FloatTensor, not ```()```. In this example
```
tensor([[[ -4.4980,  -4.7363,  -3.8697,  ..., -18.4604, -20.6320, -15.2920],
         [ -4.0868,  -3.7895,  -2.9193,  ..., -19.4917, -20.0318, -15.7870],
         [ -4.4769,  -4.7728,  -1.5619,  ..., -21.3586, -22.2751, -18.7071],
         [ -6.1670,  -6.8841,  -0.6857,  ..., -21.4503, -22.3682, -19.5937],
         [ -7.3567,  -3.1381,  -2.7641,  ..., -18.3717, -20.6145, -17.4109],
         [ -7.1151,  -6.4929,  -0.9753,  ..., -21.8517, -21.9864, -20.3518]]])
```
is returned correctly when ```labels=None``` but not when ```labels=inputs["input_ids"]```. I've tested almost identical example in GPT2 and it did return (albeit unnormalized) logits regardless of whether labels are provided or not.

Comments:
Hey @RedneckedCrake,

I think I agree with you here! It should be a very easy fix simply by change this line: https://github.com/huggingface/transformers/blob/6ee1a4fd3e80feef8fe7dc65aabb4c5270524f8a/src/transformers/models/transfo_xl/modeling_transfo_xl.py#L1100. Would you like to give it a try to fix it?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
[trainer] 2 bug fixes and a rename
Author: stas00
Pull request: yes

Fixes a bug in https://github.com/huggingface/transformers/pull/12257 and https://github.com/huggingface/transformers/pull/12276 and renames the function in the latter and adds a docstring.

Also added an extended DDP test to tests `log_level_replica`.

Fixes: https://github.com/huggingface/transformers/issues/12310

@sgugger 


Comments:
[docs]  performance 
Author: stas00
Pull request: yes

This PR imports the doc I started working on some months back https://github.com/huggingface/transformers/issues/9824 after syncing the code to master's API.

Surely a lot more work can and will be done, this is just a starting baseline.

Fixes: https://github.com/huggingface/transformers/issues/9824

@sgugger 

Comments: