noamaanMulla-03/transformers-issues huggingface dataset:
[`Blip`] Fix slow tests and doctests with correct values
Author: younesbelkada
Pull request: yes

# What does this PR do?

Fixes https://github.com/huggingface/transformers/issues/22625

In fact, the `num_attention_heads` of `BlipTextModel` should be 12 and not 8. Hence the models that are on the Hub were producing different logits / generations that the original implementation in some cases.

I made PRs on the Hub:

- https://huggingface.co/Salesforce/blip-itm-large-flickr/discussions/1
- https://huggingface.co/Salesforce/blip-itm-base-coco/discussions/3
- https://huggingface.co/Salesforce/blip-image-captioning-base/discussions/13
- https://huggingface.co/Salesforce/blip-image-captioning-large/discussions/8#642eed4cce2efe48a1aa1497
- https://huggingface.co/Salesforce/blip-vqa-base/discussions/3#642eed68ae8ae35b7a9bbb7f
- https://huggingface.co/Salesforce/blip-vqa-capfilt-large/discussions/5
- https://huggingface.co/Salesforce/blip-itm-large-flickr/discussions/3
- https://huggingface.co/Salesforce/blip-itm-large-coco/discussions/3

And tested them on the slow tests and doctests with `.from_pretrained(xxx, revision="refs/pr/xx")`, this PR fixes the slow tests and doctests with the correct values. Let's merge this PR and I'll merge the PRs that are on the Hub

cc @sgugger 


Comments:
_The documentation is not available anymore as the PR was closed or merged._
Update tiny model summary file for recent models
Author: ydshieh
Pull request: yes

# What does this PR do?

I just created tiny models for recent models. This PR updates tiny model summary file to use them. This includes:
- TFBartForSequenceClassification
- TFBlipForConditionalGeneration
- ClapModel
- MegaModel
- NllbMoeModel
- TFVisionTextDualEncoderModel
- WhisperForAudioClassification

A few tiny changes are included too.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Could you support other distributed training backends from command line, I'm using GLOO.
Author: BruceDai003
Pull request: no

https://github.com/huggingface/transformers/blob/76c4d8bf26de3e4ab23b8afeed68479c2bbd9cbd/src/transformers/training_args.py#L858

Seems like it's hard-coded to NCCL. Could you instead make this an argument? 

@sgugger 

Comments:
You can initialize the backend you want in your script. Accelerate won't re-initialize it in this case.

@sgugger Thanks, I was following the examples provided in examples/pytorch/question-answering/run_qa.py. 
As in 
https://github.com/huggingface/transformers/blob/76c4d8bf26de3e4ab23b8afeed68479c2bbd9cbd/examples/pytorch/question-answering/run_qa.py#L212, it will parse from all the listed allowed arguments including `TrainingArguments`
I think there will be no arguments provided to choose my backend here. And I think it won't use Accelerate anyway. So, in this case, it seems the only way to change it to 'gloo' backend is modifying the source code in training_args.py?

Oh I was confused on where you opened the issue. There is no way to change the backend in the `Trainer` code, as it's not just one argument that could be setup differently. You should use Accelerate for a more customized training loop.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

OK, I just made a PR here to further justify what I meant: https://github.com/huggingface/transformers/pull/14012, please take a look. Make any suggestions or corrections as you need. Thanks.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

AFAIK, this is handled by the xpu_backend `training_args` parameter passed into the `Trainer` class now. Making a note here for future reference.
See https://github.com/huggingface/transformers/blob/ee8e80a060d65ab349743ffcb5842365eb0e5606/src/transformers/training_args.py#L315-L316 for more details
Fix broken link
Author: ngoquanghuy99
Pull request: yes


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_21988). All of your documentation changes will be reflected on that endpoint.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Move labels to the same device as logits for LlamaForSequenceClassification and Blip2
Author: xssChauhan
Pull request: yes

# What does this PR do?
Fixes issue #22561 by moving the labels to the same device as the logits they are compared to for `LlamaForSequenceClassification`, `Blip2`.

@sgugger Could you review this once?

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@sgugger  I have refreshed my permissions, but I still do not see the option of rerunning the pipeline on CircleCI. Is it possible that I cant do that?

You can just push an empty commit: `git commit -m "Trigger CI" --allow-empty`

@sgugger  Is such a long waiting time for CircleCI report expected?

Could you try again? Tests still aren't run.

@sgugger I also added code for Blip2, and the tests now pass.

Perfect, thanks!
Seq2Seq Trainer for QA: No useful metric returned for evaluation
Author: stefan-it
Pull request: no

### System Info

Latest Transformers version from main.

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Hi,

unfortunately, the `run_seq2seq_qa.py` in PyTorch Example folder does not output a useful evaluation metric:

```bash
python run_seq2seq_qa.py \
  --model_name_or_path google/mt5-small \
  --dataset_name mlqa \
  --dataset_config mlqa-translate-train.de \
  --context_column context \
  --question_column question \
  --answer_column answers \
  --do_train \
  --do_eval \
  --per_device_train_batch_size 12 \
  --learning_rate 3e-5 \
  --num_train_epochs 1 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --save_steps 5000 \
  --max_steps 50 \
  --output_dir ./mt5-small
```

I'm running this command and the final output looks like:

```
***** eval metrics *****
  epoch                   =       0.01
  eval_loss               =    15.7694
  eval_runtime            = 0:00:48.60
  eval_samples            =      10584
  eval_samples_per_second =    217.741
  eval_steps_per_second   =     27.218
```

So I'm missing EM and F1-Score: why is it no longer there? This also happens when fine-tuning SquAD 2.0.

### Expected behavior

Useful evaluation metric: EM and F1-Score should be returned.

Comments:
You need to pass along `--predict_with_generate` to use generate in the evaluation, and then get the metrics.

Thanks @sgugger ! I used that option, but after evaluation it outputs:

```bash
Traceback (most recent call last):
  File "run_seq2seq_qa.py", line 724, in <module>
    main()
  File "run_seq2seq_qa.py", line 683, in main
    metrics = trainer.evaluate(max_length=max_length, num_beams=num_beams, metric_key_prefix="eval")
  File "/home/ubuntu/transformers/examples/pytorch/question-answering/trainer_seq2seq_qa.py", line 92, in evaluate
    eval_preds = self.post_process_function(eval_examples, eval_dataset, output)
  File "run_seq2seq_qa.py", line 617, in post_processing_function
    decoded_preds = tokenizer.batch_decode(preds, skip_special_tokens=True)
  File "/home/ubuntu/transformers/src/transformers/tokenization_utils_base.py", line 3445, in batch_decode
    return [
  File "/home/ubuntu/transformers/src/transformers/tokenization_utils_base.py", line 3446, in <listcomp>
    self.decode(
  File "/home/ubuntu/transformers/src/transformers/tokenization_utils_base.py", line 3485, in decode
    return self._decode(
  File "/home/ubuntu/transformers/src/transformers/tokenization_utils_fast.py", line 549, in _decode
    text = self._tokenizer.decode(token_ids, skip_special_tokens=skip_special_tokens)
OverflowError: out of range integral type conversion attempted
```

Ah, I've just found this issue for that problem: https://github.com/huggingface/transformers/issues/22634

So I'll close here!
[WIP] 🌐 [i18n-KO] Translated `sequence_classification.mdx` to Korean
Author: 0525hhgus
Pull request: yes

# What does this PR do?

Translated the `tasks/sequence_classification.mdx` file of the documentation to Korean.
- The file name is `sequence_classification.mdx`, but the document name is `text classification`.

Thank you in advance for your review:)

Part of https://github.com/huggingface/transformers/issues/20179

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

<!-- 제출 전 체크리스트로, 가짜연구소만의 체크리스트도 <details>로 감싸서 만들어두면 더 좋을 것 같아요. -->

## Who can review?
<!-- 가짜연구소 팀원들과 리뷰가 끝난 후에만 허깅페이스 직원들에게 리뷰 요청하는 아래 주석을 노출해주세요! -->
<!-- @sgugger, @ArthurZucker, @eunseojo May you please review this PR? -->

Team PseudoLab, may you please review this PR?

@0525hhgus, @KIHOON71, @sim-so, @gabrielwithappy, @HanNayeoniee, @wonhyeongseo, @jungnerd

Comments:
_The documentation is not available anymore as the PR was closed or merged._

There is a problem with the storage stream, so I will PR it again.. 😢
Extend Callback API for remote execution of ClearML Experiments
Author: thepycoder
Pull request: no

### Feature request

Hi!

Not that long ago, we added support for the ClearML experiment manager as a training callback, and the general feedback has been good so far! However, ClearML is more than an experiment manager alone.

In order to allow users to quickly experiment, ClearML can clone an existing experiment from the UI and then override the originally captured hyperparameters. It does this by injecting the new parameter values into the code at runtime. A user can then schedule and run this edited experiment clone on a remote machine.

But in order for this functionality to work properly, ClearML has to be able to initialize, access and overwrite the training parameters even before they are first used by the Trainer. The current callback implementation does not allow this.

Do you think this is something worth considering to add? I suspect it's rather easy to add a new, very early callback route, but it sounds harder to me to allow a callback to override the training arguments. What do you think?

### Motivation

We received user feedback on our own support slack channel of users trying to run Transformers remotely, but the parameters not being overridden. More advanced ClearML functionality like pipelines and HPO depend on this functionality to properly work.

### Your contribution

We'd be very willing to make a PR, this issue is meant to discuss if you agree that it can be properly added and if so, how you would like to see it practically :) Thank you for the consideration!

Comments:
Hi there. The `Trainer` is not allowed to modify its own `TrainingArguments`, a design choice we made so that reproducibility and resuming from a checkpoint work properly. This is why the callbacks are also not allowed to change the training arguments. It's probably best for this use case if you simply subclass the `Trainer` API and either add a new method or override what you need.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
rag-end2end-retriever Training Time Results
Author: jamesoneill12
Pull request: no

### Feature request

Hi, 

I would suggest including the training runtime for the new version of [RAG](https://github.com/huggingface/transformers/tree/main/examples/research_projects/rag-end2end-retriever) that is end-to-end. The hyperparameters for the results at the end of the README.md (under the "Comparison of end2end RAG (including DPR finetuning) VS original-RAG" section) are that below, but I've no idea the training time for these experiments on SQUAD. Currently my runtime on 2 GPUs is looking like months, so I want to know if there's something I've missed that causing training to be so slow or if it takes this long to run RAG end-to-end.  

--gpus 4 
--train_batch_size 4 
--eval_batch_size 
--max_source_length 128 
--max_target_length 25 
--val_max_target_length 25 
--test_max_target_length 25 
--label_smoothing 0.1 
--dropout 0.1 
--attention_dropout 0.1 
--weight_decay 0.001 
--adam_epsilon 1e-08 
--max_grad_norm 0.1 
--lr_scheduler polynomial 
--learning_rate 3e-05 
--num_train_epochs 10 
--warmup_steps 500 
--gradient_accumulation_steps 4
--distributed_retriever ray 
--num_retrieval_workers 4  

Thanks, James

### Motivation

Saves a lot of time before researchers decide if they've enough resources to work and build on this work or not. Also, suggestions on how to run training with less resources would also be useful. 

### Your contribution

Not sure it requires much help, apart from the original authors (or at least whoever ran the experiments corresponding to the results in the readme) including training time numbers in the readme results section. 

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
save_pretrained crashes when torch_dtype is passed
Author: NikolaBorisov
Pull request: yes

If you have the following code

```
p = pipeline(... torch_dtype=torch.bfloat16)
p.save_pretrained()
```

you get a crash because `torch.bfloat16` is not json serializable in the `tokenizer.save_pretrained()` method.

This PR fixes this.

@Narsil
@ArthurZucker


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_21986). All of your documentation changes will be reflected on that endpoint.

My understanding that the following code should work

```
from transformers import pipeline
p = pipeline("some/model", torch_dtype=torch.bfloat16)
``` 

from what I can see the tokenizer kwargs basically start with copy of the model_kwargs? 

https://github.com/huggingface/transformers/blob/main/src/transformers/pipelines/__init__.py#L869

Indeed, this means that the problem is bigger! Popping the argument is good if we want to keep the pipeline that way, I think we agreed to properly handle the extra tokenizer kwargs @Narsil if you want to take care of it, it will fix this! 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@ArthurZucker @Narsil What do you want to do here. I think this PR does the right thing at the moment because extra arguments to the pipeline are passed around to the tokenizer and then the tokenizer crashes on save_pretrained. 
I think merging this is strictly better than the current state. Sure the code could be improved internally. Conceptually there should be something that decides which kwargs to the pipeline should be passed to each of the components in the pipeline.

Hey! Thanks for reporting the bug, I merged something that seemed more aligned with what we want: torch dtype should just not be passed to the tokenizer, so poping it outside
Atlas: Few-shot Learning with Retrieval Augmented Language Model
Author: ae99
Pull request: no
Labels: name: New model

### Model description

Atlas is a retrieval-augmented seq2seq language model comprised of a Contriever retriever and fusion-in-decoder (FID) architecture (which uses T5), introduced in the paper [Atlas: Few-shot Learning with Retrieval Augmented Language Models](https://arxiv.org/pdf/2208.03299.pdf)

From the papers abstract:

> Large language models have shown impressive few-shot results on a wide range of tasks.
However, when knowledge is key for such results, as is the case for tasks such as question
answering and fact checking, massive parameter counts to store knowledge seem to be needed.
Retrieval augmented models are known to excel at knowledge intensive tasks without the
need for as many parameters, but it is unclear whether they work in few-shot settings. In this
work we present Atlas, a carefully designed and pre-trained retrieval augmented language
model able to learn knowledge intensive tasks with very few training examples. We perform
evaluations on a wide range of tasks, including MMLU, KILT and NaturalQuestions, and
study the impact of the content of the document index, showing that it can easily be updated.
Notably, Atlas reaches over 42% accuracy on Natural Questions using only 64 examples,
outperforming a 540B parameters model by 3% despite having 50x fewer parameters.

### Open source status

- [X] The model implementation is available https://github.com/facebookresearch/atlas
- [X] The model weights are available https://github.com/facebookresearch/atlas

### Provide useful links for the implementation

Open-sourced implementation from Meta https://github.com/facebookresearch/atlas, with weights available.

Authored by @patrick-s-h-lewis and @gizacard

Comments:
Hi all!

Super appreciative of the authors for open-sourcing this model, really exciting stuff.

I'm planning on having a go at implementing this model here. Aware there are others who have been looking at similar models in the past (https://github.com/huggingface/transformers/issues/15387), so thought it good to get this ticket in early in case you are also interested in working on this!

go for it! it shouldnt be too hard to get inference working - training may be more involved - the way we do the distributed index might be a little painful to integrate gracefully.
good luck!

Please make sure that you provide links to the original repo prominently, and try to make sure the models are 1) capable of achieving the same accuracy that they do in our repo 2) mathematically preform the same computations. 



Hello, is ATLAS a part of huggingface now?
Compute Accuracy in clip-roberta
Author: skaulintel
Pull request: no

### Feature request

Is it possible to include an accuracy metric when training clip-roberta?

### Motivation

We would like to have something other than loss to track during training.

### Your contribution

I tried creating a dummy compute_metrics function to pass to GaudiTrainer thusly. 

    metric = evaluate.load("accuracy")
    def compute_metrics(p):
        return 1

get the following error:

Traceback (most recent call last):
  File "run_clip.py", line 553, in <module>
    main()
  File "run_clip.py", line 532, in main
    metrics = trainer.evaluate()
  File "/usr/local/lib/python3.8/dist-packages/transformers/trainer.py", line 2932, in evaluate
    output = eval_loop(
  File "/usr/local/lib/python3.8/dist-packages/optimum/habana/transformers/trainer.py", line 1074, in evaluation_loop
    logits_dtype = get_dtype(logits)
  File "/usr/local/lib/python3.8/dist-packages/optimum/habana/transformers/trainer_utils.py", line 43, in get_dtype
    return [get_dtype(logits_tensor) for logits_tensor in logits]
  File "/usr/local/lib/python3.8/dist-packages/optimum/habana/transformers/trainer_utils.py", line 43, in <listcomp>
    return [get_dtype(logits_tensor) for logits_tensor in logits]
  File "/usr/local/lib/python3.8/dist-packages/optimum/habana/transformers/trainer_utils.py", line 45, in get_dtype
    raise TypeError(f"logits should be of type torch.Tensor or tuple, got {type(logits)} which is not supported")
TypeError: logits should be of type torch.Tensor or tuple, got <class 'transformers.modeling_outputs.BaseModelOutputWithPoolingAndCrossAttentions'> which is not supported



Comments:
Hi @skaulintel! According to what I see in the traceback, this issue is specific to Optimum Habana. Could you move this issue [there](https://github.com/huggingface/optimum-habana/issues) please? And then I'll follow up.
Revert migration of setup to pyproject.toml
Author: sgugger
Pull request: yes

# What does this PR do?

As mentioned in #22599, the migration of the setup to pyproject is causing some issues for editable installs on some setups. This PR reverts that migration and adds the setup.py to the formatted files. (Note that I could not directly revert the original PR due to some of its changes being already reverted in #22587 )

Fixes #22599 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
moved labels to the same device as logits for BLOOM, GPT Neo, GPT NeoX, RoBERTa and VIT models
Author: iamarunbrahma
Pull request: yes

# What does this PR do?

As suggested in the https://github.com/huggingface/transformers/issues/22561, moved labels to the same device as logits for `BLOOM`, `GPT Neo`, `GPT NeoX`, `RoBERTa` and `VIT` models.

@sgugger Could you review this once?

Comments:
_The documentation is not available anymore as the PR was closed or merged._