noamaanMulla-03/transformers-issues huggingface dataset:
[Example] Fixed finename for Saving null_odds in the evaluation stage in QA Examples 
Author: bhadreshpsavani
Pull request: yes

# What does this PR do?

Earlier file `eval_null_odds_eval.json` because of a typo in code now it will save it like `eval_null_odds.json` for squadv2 dataset Saving null_odds in the evaluation stage

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? I mentioned it in #10482
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@stas00 @sgugger

Comments:
Local Attention for GPT2
Author: leogao2
Pull request: no

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->

Our model uses local attention in some layers (i.e each position can only see the last k=256 tokens in every other layer). We would like to be able to specify this in the config on the model hub.

## Motivation

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->

Right now we can't integrate the 1.3B and 2.7B EleutherAI GPT models because local attention is not supported in transformers.



Comments:
@patil-suraj is working on implementing GPT Neo over at https://github.com/huggingface/transformers/pull/10848! The 1.3B and 2.7B should be loadable in that architecture once finalized.
added py7zr
Author: philschmid
Pull request: yes

# What does this PR do?

This PR adds `py7zr` to use `samsum` as a dataset. 

```python
[1,14]<stdout>:    use_auth_token=use_auth_token,
[1,14]<stdout>:  File "/opt/conda/lib/python3.6/site-packages/datasets/load.py", line 448, in prepare_module
[1,14]<stdout>:    f"To be able to use this {module_type}, you need to install the following dependencies"
[1,14]<stdout>:ImportError: To be able to use this dataset, you need to install the following dependencies['py7zr'] using 'pip install py7zr' for instance'
```

Comments:
Thanks a lot for adding this!
Reproducing DistilRoBERTa
Author: DavidHarrison
Pull request: no

I've been trying to retrain DistilRoBERTa from the information given [here](https://huggingface.co/distilroberta-base) along with the example code/documentation [here](https://github.com/huggingface/transformers/tree/master/examples/research_projects/distillation).

I'm a bit unclear on the exact configuration used to train the DistilRoBERTa model. I have been assuming it uses the same configuration as the DistilBERT model with minor changes, though some things, such as the loss coefficients are still a bit ambiguous.

**Would it be possible to share the exact command/configuration to train DistilRoBERTa?**

I've been able to replicate DistilRoBERTa to similar evaluation MLM perplexity but there still seems to be a small but statistically significant difference, I can share the full config if it's helpful.

Thank you!

Comments:
Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discusss.huggingface.co) instead?

You can ping @VictorSanh as he'll be the most helpful regarding distillation.

Thanks!

Apologies, I've posted to the forum [here](https://discuss.huggingface.co/t/reproducing-distilroberta/5217?u=davidharrison).

Thanks!
What is the score of trainer.predict()?
Author: Yuukp
Pull request: no

I want to know the meaning of output of trainer.predict().

example:
`PredictionOutput(predictions=array([[-2.2704859,  2.442343 ]], dtype=float32), label_ids=array([1]), metrics={'eval_loss': 0.008939245715737343, 'eval_runtime': 0.0215, 'eval_samples_per_second': 46.56})`

What is this score? -> predictions=array([[-2.2704859,  2.442343 ]]

I use it for Sequence Classification.


Comments:
These are the logits from your model, check:- https://github.com/huggingface/transformers/blob/8780caa388c7b2aa937454ed96bcdd3f097f851d/src/transformers/trainer.py#L1852

I see! Thank you so much!!
Please implement DUMA: Reading Comprehension with Transposition Thinking
Author: max-yue
Pull request: no
Labels: name: New model

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->
This one is on the race leaderborad top, will you guys consider implement this?

## Motivation

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->

## Your contribution

<!-- Is there any way that you could help, e.g. by submitting a PR?
     Make sure to read the CONTRIBUTING.MD readme:
     https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md -->


Comments:
Are the source code and the model weights avaiable?

https://arxiv.org/abs/2001.09415

> Are the source code and the model weights avaiable?
> 
> https://arxiv.org/abs/2001.09415

I do not have the source code and model weights.
Does the default weight_decay of 0.0 in transformers.AdamW make sense?
Author: julioasotodv
Pull request: no

# ❓ Questions & Help

## Details
Hi,

I tried to ask in SO before, but apparently the question seems to be irrelevant. Anyways, here it is:

In the [Docs](https://huggingface.co/transformers/main_classes/optimizer_schedules.html#transformers.AdamW) we can clearly see that the AdamW optimizer sets the default weight decay to 0.0.

Given that the whole purpose of AdamW is to decouple the weight decay regularization, is my understanding that the results anyone can get with AdamW and Adam if both are used with `weight_decay=0.0` (this is, without weight decay) should be exactly the same.

Therefore, shouldn't make more sense to have the default weight decay for AdamW > 0?

Thank you so much!

Comments:
Too bad you didn't get an answer on SO. I think you would multiply your chances of getting a good answer if you asked it over at https://discuss.huggingface.co!

Sure, I will ask there. Thank you!

I use weight decay and not use weight and surprisingly find that they are the same, why?
RAG with RAY workers keep repetitive copies  of knowledge base as  .nfs files until the process is done. 
Author: shamanez
Pull request: no


As mentioned in [this PR](https://github.com/huggingface/transformers/pull/10410), I update the **my_knowledge_dataset**  object all the time. I save the new my_knowledge_dataset in the same place by removing previously saved stuff. But still, I see there are always some hidden files left.  Please check the screenshot below.


![image](https://user-images.githubusercontent.com/16892570/110067455-a0da4880-7dd8-11eb-95cd-5b908c7a04af.png)
![image](https://user-images.githubusercontent.com/16892570/110067527-ccf5c980-7dd8-11eb-9328-7a361f554454.png)


I did some checks and found that these .nfs files being used by RAY. But my local KB is 30GB, so I do not want to add a .nfs file in every iteration. Is there a way to get over this?

@amogkam 
@lhoestq 


Comments:
I don't know what those .nfs are used for in Ray, is it safe to remove them @amogkam ?
Fixed some typos and removed legacy url
Author: WybeKoper
Pull request: yes

# What does this PR do?

Removed legacy url to colab notebook in examples/multiple-choice/README.md
Fixed some typos.



## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@sgugger 
@patil-suraj


Comments:
Thanks a lot for doing this!
Add more metadata to the user agent
Author: sgugger
Pull request: yes

# What does this PR do?

This PR adds a bit more metadata to the user agent to allow us to have more statistics on the usage, more precisely, it registers:
- the type of the file asked on the hub ("config", "tokenizer", "model" or "model_card")
- the framework used for the model ("pytorch", "tensorflow" or "flax"). Note that this is the framework actually used even in the case of a conversion (so if download the PyTorch checkpoint but use it to instantiate a Flax model, it will be "flax")
- for a tokenizer, whether it's fast or slow (like from the framework it checks the class used at the end, not the files downloaded)
- whether the Auto API was used or not
- if the instantiation came from a given pipeline or not
- if the instantiation came from the CI or not (by using a specific env variable)

There is no personal data collected but if a user wants to deactivate this behavior, the `DISABLE_TELEMETRY` env variable can be set to any truthy value and none of this will be shared.


Comments:
Enforce string-formatting with f-strings
Author: sgugger
Pull request: yes

# What does this PR do?

This PR removes any strings formatted with `.format` or `%` to use f-strings exclusively (unless there is a very good reason to use the other syntax, or the file is in a research_project/legacy folder).

The mix of three syntaxes does not make any sense and we all agree in the team that f-strings are more readable. Now that Python 3.5 is officially dead, there is no reason not to switch fully to f-strings.

cc @stas00  as we had a conversation about that.


Comments:
Merge trainers
Author: sgugger
Pull request: yes

# What does this PR do?

This PR merges the specific `SageMakerTrainer` into the main `Trainer` to make all the scripts work directly with model parallelism. In passing, a few internal breaking changes:
- `is_sagemaker_distributed_available` is renamed to `is_sagemaker_dp_enabled` since it's about data parallelism and not specifically distributed training, and it's True when the user has activated it, not when it's merely "available"
- in the ParallelMode enum, the case `SAGEMAKER_DISTRIBUTED` is renamed as well (but it wasn't used anywhere).

Both only concern the internals of the library and no public API is breaking.

Comments:
I tested run all our `sagemaker/tests` and a few additional `model_parallel` tests. ✅ 
I also tested everything with the upcoming `pytorch 1.7.1` image with the new `smp 1.3.0` (their model parallelism library)✅

What is still open for me is how does this behaves with `Seq2SeqTrainer`. Can users now use the `Seq2SeqTrainer` for model parallelism too? Data parallelism works already. Tested with `BART`

After we have merged the  `SageMakerTrainer` with the `Trainer` I would update the docs for sagemaker/model parallelism and the tests in tests/sagemaker.
Transformers QA Online Demo is not working
Author: gavishpoddar
Pull request: no


Transformers QA Online Demo is not working: https://huggingface.co/qa/


I am trying to recreate ELI5 but unable to find enough information @yjernite Can you Please help. 

Please let me know if I can help 

Thanks

Comments:
I re-launched the app.

The app itself doesn't have much information to help recreate the system though (and is not designed for heavy use :) )

I'd recommend reading through the blog post instead: https://yjernite.github.io/lfqa.html

Thank you so much @yjernite
BART : Cannot run trainer.evaluate() after trainer.train()
Author: Avditvs
Pull request: no

## Environment info

- `transformers` version: 4.4.2
- Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.10
- PyTorch version (GPU?): 1.8.1+cu101 (True)
- Tensorflow version (GPU?): 2.4.1 (True)
- Using GPU in script?: True
- Using distributed or parallel set-up in script?: False

### Who can help

trainer : @sgugger 
bart : @patrickvonplaten 


## Information

Model I am using : bart, barthez, mbart  
I am working on text summarization.

The problem arises when using my own modified script, inspired by the official Seq2Seq example. I am using the Seq2SeqTrainer class.  
I am unable to run trainer.evaluate(...) after trainer.train(...) as well as evaluating the model during training after x epochs or steps.  


## To reproduce

I you want to try it out, here is a link to [my notebook](https://colab.research.google.com/drive/1CqxxM0nOdJRpre_SwLOajl_s9hlKTT9e?usp=sharing)

Steps to reproduce the behavior:

1. Download model, tokenizer, and dataset from hub
2. Run trainer.evaluate(...) (works)
3. Run trainer.train(...) (runs fine)
4. Run trainer.evaluate(...) (returns the error below)

```
/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    887             result = self._slow_forward(*input, **kwargs)
    888         else:
--> 889             result = self.forward(*input, **kwargs)
    890         for hook in itertools.chain(
    891                 _global_forward_hooks.values(),

/usr/local/lib/python3.7/dist-packages/transformers/models/bart/modeling_bart.py in forward(self, input_ids, attention_mask, decoder_input_ids, decoder_attention_mask, head_mask, decoder_head_mask, encoder_outputs, past_key_values, inputs_embeds, decoder_inputs_embeds, use_cache, output_attentions, output_hidden_states, return_dict)
   1160         elif return_dict and not isinstance(encoder_outputs, BaseModelOutput):
   1161             encoder_outputs = BaseModelOutput(
-> 1162                 last_hidden_state=encoder_outputs[0],
   1163                 hidden_states=encoder_outputs[1] if len(encoder_outputs) > 1 else None,
   1164                 attentions=encoder_outputs[2] if len(encoder_outputs) > 2 else None,

KeyError: 0
```





Comments:
This is not an issue with Transformers but with using Apex with the "O3" opt-level. This changes your model during the training and results in the error you're seeing. The best is to re-instantiate a clean `Trainer` for evaluation after you're done with training.

Thank you for your reply, I just tried it and it actually only works with Apex "01" opt-level !
Tagged Model Version Not Working
Author: rkunani
Pull request: no

I am trying to download a specific version of the `roberta-large` model using the `revision` parameter of `from_pretrained()` as shown below:

```
from transformers import RobertaForSequenceClassification
model_type = "roberta-large"
v1 = "v3.5.0"
model = RobertaForSequenceClassification.from_pretrained(model_type, num_labels=2, revision=v1)
```

This code gives me the following 404 error:

```
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://huggingface.co/roberta-large/resolve/v3.5.0/config.json
```

Am I using the `revision` parameter incorrectly?

Extra notes:
- I am using transformers v4.1.1
- Running the same code above with `v1 = "main"` works just fine

Comments:
I think you're using it wrong indeed. As said in the [docs](https://huggingface.co/transformers/model_sharing.html#model-versioning), you can either specify a tag name, branch name, or commit hash.

To reiterate over what @NielsRogge already said, this is the revision of the *model*, not the repository. You can check the model commits here: https://huggingface.co/roberta-large/commits/main

Revisions also include branches and tags, but this particular model only has a single branch and no tag.

Ah, that makes sense. Thanks for the prompt reply!
In the group by length documentation length is misspelled as legnth
Author: JohnnyC08
Pull request: yes

In the group by length documentation length is misspelled as legnth

## Before submitting
- [X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.



Comments:
Recommended Adafactor settings for T5 cause error
Author: OyvindTafjord
Pull request: no

## Environment info
- `transformers` version: 3.3.1
- Platform: Darwin-19.6.0-x86_64-i386-64bit
- Python version: 3.7.7
- PyTorch version (GPU?): 1.6.0 (False)
- Tensorflow version (GPU?): 2.2.0 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
@sshleifer   (from activity on Adafactor PRs)

## Information

Model I am using (Bert, XLNet ...): T5

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [X] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details below)

## To reproduce

The Adafactor docs recommend the following for T5 : `Adafactor(model.parameters(), lr=1e-3, relative_step=False, warmup_init=True)`

However, the init code then has:
```
        if lr is not None and relative_step:
            raise ValueError("Cannot combine manual lr and relative_step options")
        if warmup_init and not relative_step:
            raise ValueError("warmup_init requires relative_step=True")
```
which makes this setting impossible (as well as just changing to `relative_step=True`). So something seems to be missing either in the recommendations or in the implementation.

Thanks!


Comments:

I think the doc should recommend
```python
Adafactor(model.parameters(), relative_step=True, warmup_init=True, lr=None)
```
want to fix it?

I think what corresponds to the original T5 training code is `Adafactor(model.parameters(), lr=1e-3, relative_step=False, warmup_init=False)`, however that didn't work great for me so far (much slower than Adam, and giving me NaN's even in FP32).

Hello @OyvindTafjord, have you been able to fine-tune T5 with Adafactor? Thanks, Sonali

No, I haven't investigated further regarding the slowness and NaN's I was getting.

This issue persists (i.e. the suggested defaults still produce the error).

I can confirm that `Adafactor(lr=1e-3, relative_step=False, warmup_init=False)` seems to break training (i.e. I observe no learning over 4 epochs, whereas `Adafactor(model.parameters(), relative_step=True, warmup_init=True, lr=None)` works well (much better than adam)
Add Vision Transformer and ViTFeatureExtractor
Author: NielsRogge
Pull request: yes

# What does this PR do?

Opening up a new PR based on #10513 which uses @sgugger's new `image_utils.py` instead of `torchvision` for the image transformations, and is up-to-date with master.

Things to do:

- [x] fix one integration test (currently `ViTFeatureExtractor` converts the numpy arrays into DoubleTensors, but the model expects FloatTensors)
- [x] fix styling (`make style` is not working as expected on my machine, see remaining comments in previous PR)
- [x] perhaps change pooler logic? Design (and updated conversion script) currently at branch "add_pooler_to_vit"

cc @LysandreJik 

Comments:
I've addressed all comments. The pooler now more closely matches the one of `BertModel`. 

Only `make fix-copies` is giving an error on CircleCI for now. Other than that the PR is ready.



Thanks for all your work on this @NielsRogge !
use `BaseModelOutput` as common interface for all different `BaseModelOutputWith*`?
Author: JoanFM
Pull request: no
Labels: name: Feature request

Hello team,

I have been taking a look at the `different` output models from your models, and I wonder if it would make sense to inherit all the `BaseModelOutputWithPool` and all the other flavours of modeling output, instead of using `ModelOutput`.

https://github.com/huggingface/transformers/blob/c301c26370dfa48f6a6d0408b5bb9eb70ca831b3/src/transformers/modeling_outputs.py#L24

We are trying to build a wrapper around many of the public models hosted on hugging face, and it would be useful to know if we can assume that all the potential `outputs` of the models will contain `hidden_states`. Since now they all only inherit from `ModelOutput`  it seems a little confusing.

Am I missing something? Is it not something that can be assumed? 

Comments:
Roberta and XLNet sentence pair training example
Author: azizcu
Pull request: no

I want to use RoBERTa and XLNet for sentence pair input task (like sentence similarity task pair input). Can you explain with code example?

Comments:
Hello, thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discusss.huggingface.co) instead?

Thanks!