noamaanMulla-03/transformers-issues huggingface dataset:
ROUGE Multiple References
Author: dptam
Pull request: no

It appears the current ROUGE metric computes the score with 1 reference per candidate. I was wondering if there is way to compute ROUGE with multiple references per candidate? Thanks 

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Fine-tuning bart-base on XSum and got 34.0 as ROUGE1 (40.61 with higher lr)
Author: XinnuoXu
Pull request: no

Hi, I'm wondering whether there are any benchmarks for fine-turning bart-base on XSum. I found this one [https://huggingface.co/VictorSanh/bart-base-finetuned-xsum/tree/main](url) which also shows the R1 is 35 -ish. Does it suppose to be this low?

Comments:
Just an update, I increased lr and got ROUGE-F(1/2/l): 40.61/17.48/32.62

Maybe @patrickvonplaten or @patil-suraj have an idea

Hi @XinnuoXu 

That model was trained a while ago, and there were some bugs in BART related to `decoder_start_token_id` at that time, see https://discuss.huggingface.co/t/bart-lm-odd-beam-search-output/618/13
which could be the reason for this.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Very large difference between the results after resume
Author: dorooddorood606
Pull request: no

Dear HuggingFace team,

There is unfortunately no option to reopen the bugs, the issue I reported here [1] still exists with testing the last version of transformers. I add my comments on the same ticket. Could you kindly reopen this bug?

The variations are very high after resume, which makes the results not usable with resuming after checkpoint, I also tried to make things deterministic in torch, but it also could not solve the issue. I study in an institution where I only have access to GPUs for short hours, and I very much appreciate your help on this issue to make training reproducible after resume from the checkpoints in trainer.

@sgugger 

[1] https://github.com/huggingface/transformers/issues/11323

Comments:
Please do not open a duplicate issue, you can reopen the old one.

Dear Sylvain
If huggingFace team closes an issue, the user is not able to reopen it, at least on my side, I also attached the screenshot of it. 

https://ibb.co/56S00LT

Thank you. 

Your screenshot does not show the bottom of the dialog box where there should be the button "Reopen" normally.

And if not, please ask us to reopen the issue on the issue discussion, do not open a duplicate :-)

there is really no reopen on the users sides. Sure I will  make sure to ask and I will avoid recreate the issue, thank you very much for the remark :)  I pay attention
Using BART for Mask Infilling makes all the first tokens missing  
Author: yeounyi
Pull request: no

I'm fine-tuning BART ```"facebook/bart-large"``` model for mask infilling. My dataset looks like below.

The original sentence BART should predict is `taste the rainbow.`, and the input data it gets is `<mask> taste <mask> rainbow <mask>`, or it should predict `global asset management`, given `<mask> global <mask> asset <mask>`. 


Generally it works well but only the first tokens are missing. BART's prediction for the first data was `aste the rainbow.` and the prediction for the second was `asset management.`. I don't know why this is happening. `taste` and `global` was given in input, why BART is missing those? Even the first token in original sentence is not given in the input BART predictions always drop the first token.  Given `<mask> happiest <mask> place <mask>`, it should predict `the happiest place on earth.`, but it gives me `happiest place on earth.`

I'm not sure this is related, but I gave `force_bos_token_to_be_generated` options to be `True`, and still not working

```
config = BartConfig(force_bos_token_to_be_generated=True)
model =  BartForConditionalGeneration.from_pretrained("facebook/bart-large", config = config)
```

I would appreciate any help. Thanks


Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Works fine when using the model.generate function
Update modeling_tf_pytorch_utils.py
Author: LooperXX
Pull request: yes

# What does this PR do?
Fix a bug in convert_tf_weight_name_to_pt_weight_name(). 
Similar to the kernel parameters, the recurrent_kernel parameters in the LSTM networks need to be transposed, too.

Fixes # (issue)

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

## Memo
If the recurrent_kernel parameters are not transposed, it will cause the model parameters not to be loaded correctly, resulting in model migration failure.


Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

> This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.
> 
> Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

This issue causes incorrect parameter transfer between PyTorch and TensorFlow when the LSTM neural networks are used in code.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
[Doc] Something wrong in description of 'DistilBertForSequenceClassification' in doc
Author: nxznm
Pull request: no

I think the description of input_ids (one of the parameters) of [DistilBertForSequenceClassification](https://huggingface.co/transformers/model_doc/distilbert.html#distilbertforsequenceclassification) is not correct.
I think the input_ids should be `torch.LongTensor of shape (batch_size, sequence_length)`, rather than `torch.LongTensor of shape (batch_size, num_choices)`.


Comments:
That's correct, thanks for spotting! Could you open a PR to fix this?

Thanks!
Add --text_column to run_summarization_no_trainer
Author: changjonathanc
Pull request: yes

# What does this PR do?

Add the `--text_column` option to `run_summarization_no_trainer.py`
(mostly copy from `run_summarization.py`)

Also removed a duplicated line:
`padding = "max_length" if args.pad_to_max_length else False`

@sgugger

Comments:
Hi there and thanks for the PR! It doesn't really make sense to have `text_column` without `summary_column` to go with it. Could you add this one too?

Hi @sgugger ! `summary_column` is already there, it's only `text_column` missing. 😄
[LXMERT] Unclear what img_tensorize does with color spaces
Author: shivgodhia
Pull request: no

## Environment info

- `transformers` version: Not using transformers directly, I'm loading a model "unc-nlp/frcnn-vg-finetuned"
- Platform: MacOS
- Python version: 3.8
- PyTorch version (GPU?): 1.6.0, no GPU
- Tensorflow version (GPU?): don't have
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help
@eltoto1219 probably

Models: "LXMERT": "unc-nlp/frcnn-vg-finetuned"

Library: https://github.com/huggingface/transformers/tree/master/examples/research_projects/lxmert

## Information

Model I am using (Bert, XLNet ...):  "LXMERT": "unc-nlp/frcnn-vg-finetuned"

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x ] my own task or dataset: (give details below)

## Problem

I don't get what img_tensorize in utils.py is doing with color spaces. I run the following code to load the model. 

```
# load models and model components
frcnn_cfg = Config.from_pretrained("unc-nlp/frcnn-vg-finetuned")

frcnn = GeneralizedRCNN.from_pretrained("unc-nlp/frcnn-vg-finetuned", config=frcnn_cfg)

image_preprocess = Preprocess(frcnn_cfg)
```
Turns out that frcnn_cfg.input.format is "BGR" so I wanted to know what exactly is going on? Here is where the image is loaded (utils.img_tensorize)

```
def img_tensorize(im, input_format="RGB"):
    assert isinstance(im, str)
    if os.path.isfile(im):
        img = cv2.imread(im)
    else:
        img = get_image_from_url(im)
        assert img is not None, f"could not connect to: {im}"
    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
    if input_format == "RGB":
        img = img[:, :, ::-1]
    return img
```

See, we seem to be converting the images to RGB, then if it's "RGB" format we flip the blue (?) channel? Is the image ever converted to "BGR"? 



Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
How to get masked word prediction for other languages
Author: AnnaSou
Pull request: no

Hello,

I trying to get masked words predictions for languages except English with Roberta or XLM Roberta.

```
from transformers import pipeline
nlp = pipeline("fill-mask", model="roberta-base")
template = f"That woman is {nlp.tokenizer.mask_token}."
output = nlp(template)

nlp4 = pipeline("fill-mask", model="roberta-base")
nlp4(f"Женщины работают {nlp4.tokenizer.mask_token}.")
```
The output for English example is quite good, while for Russian one does not make sense at all:

`[{'sequence': 'Женщины работаюта.', 'score': 0.2504434883594513, 'token': 26161, 'token_str': 'а'}, {'sequence': 'Женщины работають.', 'score': 0.24665573239326477, 'token': 47015, 'token_str': 'ь'}, {'sequence': 'Женщины работаюты.', 'score': 0.1454654186964035, 'token': 46800, 'token_str': 'ы'}, {'sequence': 'Женщины работаюте.', 'score': 0.07919821888208389, 'token': 25482, 'token_str': 'е'}, {'sequence': 'Женщины работаюти.', 'score': 0.07401203364133835, 'token': 35328, 'token_str': 'и'}]`

Neither "roberta-base" nor "xlm-roberta-base" work for Russian language example. 

Maybe I am doing it wrong, but how would one use masked word prediction for other languages?

Thanks! 

Comments:
You can filter the model hub on 'ru' (Russian) and 'fill-mask' (masked language modeling): https://huggingface.co/models?filter=ru&pipeline_tag=fill-mask

robert-base was trained on English text only, so it will not work for Russian. 

Maybe a good choice is this model: https://huggingface.co/blinoff/roberta-base-russian-v0

How about XLM-RoBERTa? That should be trained with multiple languages, but I got similar issues. 

```
The specified target token ` courageuse` does not exist in the model vocabulary. Replacing with `▁courage`.
[{'sequence': '
 Cette femme est belle.
', 'score': 0.002463364042341709, 'token': 21525, 'token_str': '▁belle'}, {'sequence': '
 Cette femme est courage.
', 'score': 4.064602762809955e-06, 'token': 116252, 'token_str': '▁courage'}]
```

> You can filter the model hub on 'ru' (Russian) and 'fill-mask' (masked language modeling): https://huggingface.co/models?filter=ru&pipeline_tag=fill-mask
> 
> robert-base was trained on English text only, so it will not work for Russian.
> 
> Maybe a good choice is this model: https://huggingface.co/blinoff/roberta-base-russian-v0



Thank you for the reply! I second the comment above regarding XLM-Roberta. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Fine Tune GPT-NEO 2.7B
Author: cppntn
Pull request: no

Hello to everyone, is there a script to fine tune this new model?

Thanks

Comments:
Did you try to use `run_clm.py`?

@LysandreJik Yea, it's working. But i cannot fine-tune it using "GPU" in colab and the "TPU" gives me this error ```exception: process 0 terminated with signal SIGKILL```, I tried the solutions in the issues but still  

SIGKILL is probably an out of memory error, meaning you're running out of RAM. I'm not entirely sure colab gives you the hardware necessary to do such big trainings, even when using DeepSpeed.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
overflow_to_sample_mapping missing in in documentation
Author: schelv
Pull request: no
Labels: name: WIP

In the [documentation ]( https://huggingface.co/transformers/master/main_classes/tokenizer.html#transformers.PreTrainedTokenizerFast.__call__)of the fast tokenizer, the `overflow_to_sample_mapping` field is missing.
Instead the `overflowing_tokens`  is listed there, which is only part of  the base tokenizer.


Comments:
Indeed! Do you want to open a PR with a fix?

No not really. 
But I've found that the documentation is the same because the `PreTrainedTokenizerFast` inherits the `__call__` method as well as the documentation from `PreTrainedTokenizerBase`. 
The `__call__` method documentation is a concatenation of two different documentations.
So changing that single line of documentation is more complicated than I expected.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

I think that it does?
Auto modelcard
Author: sgugger
Pull request: yes

# What does this PR do?

This PR adds functionality in the Trainer to auto-generate model cards and some utilities to do the same without the Trainer if people are not using it. In passing, the old `ModelCard` class is deprecated (to be removed in v5).

As an example [here](https://huggingface.co/sgugger/test-glue-mrpc) is a repo that is generated by the `run_glue` script with this new functionality, using the following command on a machine with 2 GPUs:

```
accelerate launch examples/pytorch/text-classification/run_glue.py \
    --model_name_or_path bert-base-cased \
    --task_name mrpc \
    --do_train \
    --do_eval \
    --learning_rate 2e-5 \
    --per_device_train_batch_size 16 \
    --per_device_eval_batch_size 16 \
    --evaluation_strategy epoch \
    --logging_strategy epoch \
    --weight_decay 1e-2 \
    --output_dir ~/tmp/test-glue-mrpc \
    --overwrite_output_dir \
    --push_to_hub
```

I've only adjusted the glue example for now, will do the others once we have settled on an API.

Comments:
Adds Flax BERT finetuning example on GLUE
Author: marcvanzee
Pull request: yes

# What does this PR do?

Adds Flax BERT finetuning example which finetunes on one of the GLUE tasks.

I evaluated all tasks 5 times and added the average runs, the best run. and stdev in a table in the README. I used the seed of the best run as the default.

I also ran all experiments on three devices: 8 Cloud TPU-v3, 1 Cloud TPU-v3, 1 P100 GPU. I compared the runtimes and put them in another table in the README.

This PR was discussed over Slack with @patrickvonplaten and @sgugger .

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [X] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


Comments:
Changes were done according to discussion [here](https://github.com/huggingface/transformers/pull/11593)

Test failures are unrelated
Can this `@slow` annotation be removed at barthez tokenizer test=
Author: PhilipMay
Pull request: no

Here is a `@slow` annotation that can be removed IMO.
Otherwise maybe write a comment why it is tagged as slow?

https://github.com/huggingface/transformers/blob/bc2571e61c985ec82819cf01ad038342771c94d0/tests/test_tokenization_barthez.py#L27

I can provide a PR.

Comments:
No it is intended as slow. The Barthez tokenizer fast tokenizer takes forever to load from the slow one sadly, which is why we marked those tests are slow. To be able to remove that, we must first create a small sentencepiece model that is compatible with Barthez (it is using the real tokenizer checkpoint right now) and then the tests can be marked as unslow.

Ok thanks. Closing this then.

@sgugger When do you execute the slow tests? Do you do it manually before you do a release?

They run once every day!

Ahh ok thanks.
Mixed precision training : link broken
Author: IvanoLauriola
Pull request: no

A few weeks ago this link
https://github.com/huggingface/transformers/tree/master/examples/text-classification#mixed-precision-training
showed a comparison between models trained with and without mixed precision for a bunch of sequence/text classification tasks.

Now I think that link is broken and something (examples and similar) has been moved but I do not find the new location of this comparison.
Do you know where can I find that? 

Comments:
Yes, it's here: https://github.com/huggingface/transformers/tree/master/examples/pytorch/text-classification#mixed-precision-training



Cool, thank you!
Remove "`optional`, defaults to :obj:`None`"
Author: PhilipMay
Pull request: no

There are some docstrings with "`optional`, defaults to :obj:`None`" arguments.
According to @sgugger this should be avoided: https://github.com/huggingface/transformers/pull/11417#discussion_r629320375

PS: I can provide a PR if wanted...

Comments:
Please go ahead if you want to clean this! A quick serach shows me 17 \`optional\`, defaults to :obj:\`None\` and two `optional`, defaults to None

ok - see #11703 @sgugger
[RAG] official facebook example code for RAG is not working anymore.
Author: giobin
Pull request: no

## Environment info
- `transformers` version: 4.6.0.dev0
- Platform: Linux-3.10.0-1127.10.1.el7.x86_64-x86_64-with-debian-buster-sid
- Python version: 3.6.13
- PyTorch version (GPU?): 1.8.1+cu102 (True)
- Tensorflow version (GPU?): 2.4.1 (False)
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no


### Who can help
rag: @patrickvonplaten, @lhoestq

Models:
RAG model

## Information

Model I am using RAG

The problem arises when using the official example scripts from https://huggingface.co/facebook/rag-sequence-nq, which I copied here:

```
from transformers import RagTokenizer, RagRetriever, RagSequenceForGeneration 

tokenizer = RagTokenizer.from_pretrained("facebook/rag-sequence-nq") 
retriever = RagRetriever.from_pretrained("facebook/rag-sequence-nq", index_name="exact", use_dummy_dataset=True) 
model = RagSequenceForGeneration.from_pretrained("facebook/rag-sequence-nq", retriever=retriever) 

input_dict = tokenizer.prepare_seq2seq_batch("how many countries are in europe", return_tensors="pt") 

generated = model.generate(input_ids=input_dict["input_ids"]) 
print(tokenizer.batch_decode(generated, skip_special_tokens=True)[0]) 

```


The tasks I am working on is:
I am trying to ran the sample code from above. Note that this same error occurs also when finetuning rag with the official code in transformers/examples/research_projects/rag/finetune_rag.sh

## To reproduce

Steps to reproduce the behavior:

1. run the sample code above 

The error is:
```
Traceback (most recent call last):
  File "prova_rag.py", line 5, in <module>
    model = RagSequenceForGeneration.from_pretrained("facebook/rag-sequence-nq", retriever=retriever, _fast_init=False)
  File "/nlu/users/giovanni_bonetta/transformers/src/transformers/modeling_utils.py", line 1208, in from_pretrained
    model, state_dict, pretrained_model_name_or_path
  File "/nlu/users/giovanni_bonetta/transformers/src/transformers/modeling_utils.py", line 1278, in _load_state_dict_into_model
    model._init_weights(module)
  File "/nlu/users/giovanni_bonetta/miniconda2/envs/hf_venv/lib/python3.6/site-packages/torch/nn/modules/module.py", line 948, in __getattr__
    type(self).__name__, name))
AttributeError: 'RagSequenceForGeneration' object has no attribute '_init_weights'
```

Looking at last commits i suppose that the error was introduced in "Pytorch - Lazy initialization of models #11471" a couple of weeks ago, where the line `model._init_weights(module)` was introduced.
## Expected behavior

It should initialize the model without errors.


Comments:
Vit deit fixes
Author: NielsRogge
Pull request: yes

# What does this PR do?

* Some small documentation improvements of ViT + DeiT.
* Adds a cats image to the `fixtures/test_samples` folder, which is used in the integration tests of both ViT and DeiT. 
* Adds a community notebook, illustrating how to fine-tune the Vision Transformer on CIFAR-10.

(there's something weird going on with the .gitignore within the test_samples folder , see files changed). 

Comments:
The notebook looks amazing!

Out of curiosity, have you tried using `Trainer` to fine-tune `ViT`?

Added a notebook that uses the Trainer. Includes a nice confusion matrix at the end :)