noamaanMulla-03/transformers-issues huggingface dataset:
Add attention_scores as output in RoBERTa
Author: sadra-barikbin
Pull request: no

# 🚀 Feature request

Hi, I propose to add `attention_scores`, which is a local variable in `RobertaSelfAttention`'s `forward` method to model output
on user request by adding `output_attention_scores` keyword argument  to RoBERTa model.

## Motivation

Currently, there is `output_attentions` keyword argument which makes model output `attention_probs`, but due to
the softmax being applied to it, it can not get involved in computing losses well. I mean softmax-related numerical instabilities which might arise.

## Your contribution

I would like to do the job, if you will. RoBERTa could be the starting point and then we could do the same to
other models


Comments:
Interesting request -> to enable this, we would ideally enable it across all models which would be quite a bit of work. In which situations do you need the `attention_scores`?

cc @patrickvonplaten @patil-suraj @sgugger 

In my case, I need to change attention distribution of a part of the sentence toward another part of it, using `torch.nn.CrossEntropyLoss` which expects input to be raw unnormalized scores (the `attention_scores` here) to internally compute softmax and log in a stable manner.

Generally, current attention output, namely `attention_probs` seems to serve almost just in visualization purposes. Its precomputed softmax does not let it to be used in certain loss functions.

Can't you just revert the softmax operation? Think PyTorch's softmax implementation is pretty stable

Yes, it can be reverted, but introduces extra computation in certain use cases like Cross Entropy. By the way, I see that the `attention_scores` requirement might not worth needed effort and added complexity to models' inputs.

Hmm, I think given that it's quite an edge-case and that it can be achieved by reverting the softmax operation locally, I don't think it's worth adding more outputs here to be honest. It would create a lot of work and the users would have to get used to another output for every model.
[Flax tests] fix test_model_outputs_equivalence
Author: patil-suraj
Pull request: yes

# What does this PR do?

The `test_model_outputs_equivalence` common test is never run because `recursive_check` call is at the wrong place. This PR calls `recursive_check` in `check_equivalence` and also fixes the `set_nan_tensor_to_zero` by using an equivalent jax function since jax arrays do not support index assignment.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks a lot for fixing it
Make sure custom configs work with Transformers
Author: sgugger
Pull request: yes

# What does this PR do?

This PR makes sure that once can use a custom configuration in Transformers with no trouble in the `save_pretrained`/`from_pretrained` methods. The only requirement is that it must accept kwargs as the `from_pretrained` method calls the init with lots of them (this will be documented soon in a PR that shows how to write a custom model and config).

An alternative would be to make sure that every subclass of `PretrainedConfig` calls the `super().__init__` class with a check like PyTorch, but it would force us to put that calls at the beginning of each init before any attribute is set. This PR opts for a more lenient strategy, and makes sure any config attribute we try to reach is done through a `getattr`.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Add Wav2Vec2 Adapter Weights to Flax
Author: sanchit-gandhi
Pull request: yes

# What does this PR do?

Fixes #15476

- Adds an adapter to the Flax Wav2Vec2 model to reduce the time dimension of the extracted feature vectors beyond that of the standard Wav2Vec2 model. The encoder's output hidden states thus have a time context window that is more similar to that of a subword token instead of just a character. 
- Shape and values of Flax output logits match those of the PyTorch model.
- Flax model uses all PyTorch model weights, including those of the adapter. Running the script in #15476 is resolved to yield identical Flax and PyTorch results (within 4e-2 threshold).

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Great looks good to me! @patil-suraj, could you also take a quick look? 

For reference, PR is the corrected version of #15521

Feel free to merge @sanchit-gandhi, the failing test is unrelated
Fix tests hub failure
Author: sgugger
Pull request: yes

# What does this PR do?

This PR fixes the flaky tests in the test_hub job. 

TL;DR: the two tests changed push to the Hub with async commits which is the source of two flaky errors and one at-exit error:
- 1st flaky error: the pushes may not be finished before we try to access the repo, so we wait for the push in progress to be finished
- 2nd flaky error: the pushes may not all be done, since a commit is skipped when the push in progress is not finished

At-exit error: when exiting the test, `huggingface_hub` accesses the `is_done` property of some of the commands (even if they are finished), which calls another time the `post_method` of those command (`lfs_prune`) on a repo that does not exist anymore. This is fixed on the `huggingface_hub` side by the PR mentioned above.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
update serving_output for some TF models
Author: ydshieh
Pull request: yes

# What does this PR do?

The recent merged PRs #15511 and #15530 forgot to update `serving_output` for the changed outputs.
(sorry about this)

This PR fixes it.

@patrickvonplaten 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

(@Rocketknight1 )
Tokenizers dictionaries not being saved to cache.
Author: m-bain
Pull request: no

Hi,

Sometimes huggingface servers are down. This can break training scripts if you are trying to load a tokenizer because they get the vocabulary through the API everytime.

e.g. 
`tokenizer = transformers.CLIPTokenizer.from_pretrained("openai/CLIP/ViT-B/32")`

If the servers are down the above line of code will fail:

`requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://huggingface.co/api/models/openai/CLIP/ViT-B/32`

Why not save the tokenizer vocab to the local model cache the first time this is called? This would reduce unnecessary calls to the server and make training models less sensitive to huggingface server downtime.



Comments:
Hello 🙂 

Tokenizers get automatically cached when first downloaded, so once you downloaded the tokenizer, you can use `AutoTokenizer.from_pretrained(model_name, local_files_only=True)` and if the tokenizer is present in the cache, you'll get the tokenizer. Otherwise, you'll get `TypeError: __init__() missing 1 required positional argument: 'vocab_file'`.

If you want to work completely offline, you can run your script with the `TRANSFORMERS_OFFLINE=1` environment variable.

ok tbh i feel like the cache should be used automatically without needing the flag, it also seems to not have downloaded for me but ill check further
Ability to save model outcomes in tabular format CSV file?
Author: pratikchhapolika
Pull request: no

I am downloading the model https://huggingface.co/microsoft/Multilingual-MiniLM-L12-H384/tree/main microsoft/Multilingual-MiniLM-L12-H384 and then using it.

Transformer Version: '4.11.3'

I have written the below code:

```
model = tr.BertForSequenceClassification.from_pretrained("/home/pc/minilm_model",num_labels=2)
model.to(device)

print("hello")

training_args = tr.TrainingArguments(
    output_dir='/home/pc/proj/results2',          # output directory
    num_train_epochs=10,              # total number of training epochs
    per_device_train_batch_size=16,  # batch size per device during training
    per_device_eval_batch_size=32,   # batch size for evaluation
    learning_rate=2e-5,
    warmup_steps=1000,                # number of warmup steps for learning rate scheduler
    weight_decay=0.01,               # strength of weight decay
    logging_dir='./logs',            # directory for storing logs
    logging_steps=1000,
    evaluation_strategy="epoch",
    save_strategy="no"
)

print("hello")

trainer = tr.Trainer(
    model=model,                         # the instantiated 🤗 Transformers model to be trained
    args=training_args,                  # training arguments, defined above
    train_dataset=train_data,         # training dataset
    eval_dataset=val_data,             # evaluation dataset
    compute_metrics=compute_metrics
)

```
Is there way to retrieve my ( **I do not want to use tensor-board**) :

1. **Training loss** for every epoch
2. **Validation loss** for every epoch

In a tabular format ( May be in CSV file)

Then I want to plot my train and validation loss graph.

**Please help me what changes do I need to bring in the above code in case its feasible.**




Comments:
You need to add your own logging handler.
Check this: https://github.com/huggingface/transformers/issues/10454

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Add local and TensorFlow ONNX export examples to docs
Author: lewtun
Pull request: yes

# What does this PR do?

This PR builds on #13831 to show examples of:

* Exporting a local checkpoint to ONNX
* Exporting a pure TensorFlow checkpoint to ONNX


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks for the tip @sgugger - I'd totally forgotten about this neat feature of the docs :) I've now implemented the change, so should be good for another pass
bart-large-cnn training related information
Author: MaheshChandrra
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

Hi Team HuggingFace

Can we please know the training data set,,total number of summary pairs used and the configuration used to train BART Summarizer (facebook/bart-large-cnn).

Thanks in advance!!

Comments:
bart-large-cnn is trained on CNN/DM summarization dataset. See this for dataset info https://www.tensorflow.org/datasets/catalog/cnn_dailymail

Also available in the Hugging Face `nlp` [library](https://huggingface.co/datasets/cnn_dailymail) :) 

You can explore the dataset [here](https://huggingface.co/nlp/viewer/?dataset=cnn_dailymail&config=3.0.0)


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


The name is very confusing if it's trained on CNN and Dailymail but named as CNN (only without DM). Please consider changing the name. 

sorry for the confusion. it's too late to change it and the name is copied from fairseq. it's trained on the cnn/dailymail dataset.

I feel that you answered the question about the dataset, but what about the training configuration used?
GPT2 is not jit-traceable
Author: gcompagnoni
Pull request: no

## Information

I would like to use Pytorch tracing on a pretrained GPT2 model, but I run into these warnings for the attention layers:
```
python3.8/site-packages/transformers/modeling_gpt2.py:164: TracerWarning: Converting a tensor to a Python float might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  w = w / (float(v.size(-1)) ** 0.5)
python3.8/site-packages/transformers/modeling_gpt2.py:169: TracerWarning: Converting a tensor to a Python index might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  mask = self.bias[:, :, ns - nd : ns, :ns]
```
The first warning concern the same line as the one reported in #3954 (and fixed by #3955).

## To reproduce

You can run the following:
```
import torch
from transformers import GPT2Tokenizer, GPT2LMHeadModel

tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
model = GPT2LMHeadModel.from_pretrained('gpt2')
model.eval()

tokens=tokenizer('The cat is on the table.', return_tensors='pt')['input_ids']

with torch.jit.optimized_execution(True):
    traced_model = torch.jit.trace(model, tokens)
```

## Environment info
     
- `transformers` version: 3.4.0
- Platform: Linux-5.4.0-52-generic-x86_64-with-glibc2.29
- Python version: 3.8.5
- PyTorch version (GPU?): 1.7.0+cpu (False)
- Using GPU in script?: NO
- Using distributed or parallel set-up in script?: NO



Comments:
Hi! I believe those are warnings and not errors? Does it change the expected results when tracing the model?

You are right, despite the warnings (and my - limited - understanding of what should work inside tracing), the output of the compiled model with different inputs are comparable to the base ones. 

Thanks a lot!

Glad it works!

@gcompagnoni @LysandreJik 
Hi, there
Why do I try the example below occurs error ? issue reported in #15598

```
import torch
from transformers import GPT2Tokenizer, GPT2LMHeadModel

tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
model = GPT2LMHeadModel.from_pretrained('gpt2')
model.eval()

tokens=tokenizer('The cat is on the table.', return_tensors='pt')['input_ids']

with torch.jit.optimized_execution(True):
    traced_model = torch.jit.trace(model, tokens)
```
Compute loss independent from decoder for TF EncDec models (as #14139)
Author: ydshieh
Pull request: yes

# What does this PR do?

Apply the same change in #14139 to TF Encoder-Decoder like models.

(As discussed in [this comment in 14148](https://github.com/huggingface/transformers/pull/14148#discussion_r780361117))

@NielsRogge & @patrickvonplaten  knows better the context
@gante @Rocketknight1 for TF

Comments:
Which model checkpoint should be selected for evaluation？
Author: ARDUJS
Pull request: no

According to this sample：
https://github.com/huggingface/transformers/blob/master/examples/pytorch/summarization/run_summarization.py

I get many checkpoints. 
![image](https://user-images.githubusercontent.com/20811685/148014017-c86d6aff-abb1-49b7-9390-26cf5b627244.png)

Which should be selected for testing

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Training causal language models from scratch without grouping independent data samples into blocks
Author: ritiztambi
Pull request: no

Trying to train a causal language model from scratch in Pytorch using run_clm_no_trainer script.
My dataset has independent sentences separated by newlines.
The current script groups these sentence (which are image captions) while training which creates erroneous results when sampling for generation 
Currently I'm trying to avoid this by adding period mark between these sentences before training and at inference splitting generated result at period. This is more of a hack and has obvious shortcomings.
Is there a way to avoid grouping of text while training. I've gone through the script and I may be missing something really obvious.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
datasets doesn't support NSP? should I implement my custom DataCollator or function for dataset.map?
Author: dancingpipi
Pull request: no

As the title, I haven't find NSP dataset, there is only transformers.TextDatasetForNextSentencePrediction, but it's not work while loading data by `load_dataset(...)`.

So have I to implement my custom DataCollator or function for dataset.map? Which one is better?

Or I miss something?  

Comments:
Pinging @sgugger 

Yes, you have to implement your own function for preprocessing the data.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
TF: Add informative warning for inexistent CPU backprop ops
Author: gante
Pull request: yes

# What does this PR do?

Some TF operations do not support backpropagation on CPU, namely convolutions with the `groups` argument set ([related issue on Keras](https://github.com/keras-team/keras/issues/15713)). Sadly, the exception thrown is very uninformative and makes no mention of hardware problems if there is no GPU on the system (but it does if the machine has a GPU). This PR adds an informative warning in case a user loads the affected models.

Fixes #15114
Fixes #15059


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Example of the warning:
<img width="1512" alt="image" src="https://user-images.githubusercontent.com/12240844/153624301-9b171959-66c1-4a24-bb59-e86673192e6a.png">


(Merging as soon as tests pass, as this just adds a warning)
Enable ONNX export when PyTorch and TensorFlow installed in the same env
Author: lewtun
Pull request: yes

# What does this PR do?

This PR implements one of the proposals discussed in #15620 to enable the ONNX export of pure TensorFlow models (i.e. model repos without PyTorch weights) when TensorFlow and PyTorch are installed in the same environment.

When both frameworks are installed, we default to using an `AutoModel` class (instead of `TFAutoModel`), and then use the `from_tf=True` argument of the `from_pretrained()` method to load the TensorFlow weights.

With this PR, it is now possible to export models like the following without error:

```bash
python -m transformers.onnx --model=keras-io/transformers-qa onnx/
```

The user will see a 404 error in the logs from the `from_pretrained()` method:

```
404 Client Error: Entry Not Found for url: https://huggingface.co/keras-io/transformers-qa/resolve/main/pytorch_model.bin
```

but we can handle this in a separate PR if users report confusion.

Closes #15620 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks for the reviews 😍 !

> 404 Client Error: Entry Not Found for url: https://huggingface.co/keras-io/transformers-qa/resolve/main/pytorch_model.bin
> but we can handle this in a separate PR if users report confusion.

Well, I'm reporting this right away. We need to design a way not to report misleading and alarming info and not deal with it in a reactive way. We already have too much logging noise so this just makes things worse. And most users won't report this and just be worried.

Thanks.

Thanks for the feedback @stas00 - the 404 error is also displayed in the `pipeline()` function with e.g.

```python
from transformers import pipeline

# Load a pure TensorFlow model => see 404 Client Error in logs, but pipeline loads fine
p = pipeline("question-answering", model="keras-io/transformers-qa")
```

I'm happy to take a look at both cases and propose a fix


That would be very awesome of you, Lewis! Thank you!
make style produces a lot of reformatted lines
Author: ydshieh
Pull request: no

## Environment info

- `transformers` version: 4.17.0.dev0
- Platform: Windows-10-10.0.19044-SP0
- Python version: 3.9.5

## Information

I am at commit `3fae83d2` (in `master` branch). When I run `make style`, there are tones of files being reformatted (no local change on my side).

```
reformatted examples\pytorch\speech-pretraining\run_wav2vec2_pretraining_no_trainer.py
reformatted examples\research_projects\jax-projects\wav2vec2\run_wav2vec2_pretrain_flax.py
reformatted examples\research_projects\onnx\summarization\bart_onnx\generation_onnx.py
reformatted examples\research_projects\distillation\run_squad_w_distillation.py
reformatted examples\research_projects\wav2vec2\run_pretrain.py
reformatted examples\research_projects\movement-pruning\masked_run_glue.py
reformatted examples\research_projects\pplm\run_pplm.py
reformatted examples\research_projects\movement-pruning\masked_run_squad.py
reformatted examples\research_projects\lxmert\modeling_frcnn.py
reformatted src\transformers\generation_beam_search.py
reformatted examples\research_projects\visual_bert\modeling_frcnn.py
reformatted src\transformers\generation_flax_utils.py
reformatted src\transformers\generation_tf_utils.py
reformatted src\transformers\models\bart\tokenization_bart.py
reformatted src\transformers\modeling_tf_utils.py
....(many more)
```

Most of them are something like: `1 / (d_head**0.5)` --> `(d_head ** 0.5)`.
(not very sure if this is a problem only on my machine)

### Who can help
@sgugger  @LysandreJik 

Comments:
It looks like you haven't updated your black version. Make sure to run `pip install -e .[quality]` again.

@sgugger Thanks, I am thinking maybe it's the version and trying to verify -- and you already gave the answer!
Fix _configuration_file argument getting passed to model
Author: sgugger
Pull request: yes

# What does this PR do?

Currently, the following code fails:
```
from transformers import LayoutLMv2ForTokenClassification
model = LayoutLMv2ForTokenClassification.from_pretrained('microsoft/layoutxlm-base', num_labels=10)
```
or any code using this checkpoint.

This is because the `_configuration_file` argument passed to the config ends up in the kwargs passed to the model. This PR fixes that and adds a test to make sure we don't regress.

Pb was reported in #15451

Comments:
_The documentation is not available anymore as the PR was closed or merged._
[research_projects] deal with security alerts
Author: stas00
Pull request: yes

for 2 weeks now dependabot keeps on complaining about old PL versions hardcoded in research_projects that have a vulnerability.

This PR may break those abandoned examples but at least give the developers some peace of mind.

Perhaps it's time to move research_projects that have been abandoned to their own repo.

@LysandreJik 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Good idea, @LysandreJik - done

Please review with whitespace-ignore filter on:
https://github.com/huggingface/transformers/pull/15594/files?diff=split&w=1