noamaanMulla-03/transformers-issues huggingface dataset:
Pin ruff
Author: sgugger
Pull request: yes

# What does this PR do?

ruff is fast and is also released on a fast pace 😅 
Those releases sometimes break the CI and all existing PRs (which then require a rebase on main) so let's pin it and we will do an upgrade in 6 months or for 2024.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
KeyError: 'eval_metric-name' in trainer.py, line 2339
Author: pminervini
Pull request: no

### System Info

Latest `transformers` version from the `main` branch, running on Ubuntu

```python
  File "code-cli.py", line 347, in <module>
    main(sys.argv[1:])
  File "code-cli.py", line 281, in main
    train_result = trainer.train(resume_from_checkpoint=checkpoint)
  File "[..]/python3.10/site-packages/transformers/trainer.py", line 1631, in train
    return inner_training_loop(
  File "[..]/python3.10/site-packages/transformers/trainer.py", line 1975, in _inner_training_loop
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch, ignore_keys_for_eval)
  File "[..]/python3.10/site-packages/transformers/trainer.py", line 2236, in _maybe_log_save_evaluate
    self._save_checkpoint(model, trial, metrics=metrics)
  File "[..]/python3.10/site-packages/transformers/trainer.py", line 2339, in _save_checkpoint
    metric_value = metrics[metric_to_check]
KeyError: 'eval_metric-name'
```

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Seq2seq training on NQ

### Expected behavior

```python
  File "code-cli.py", line 347, in <module>
    main(sys.argv[1:])
  File "code-cli.py", line 281, in main
    train_result = trainer.train(resume_from_checkpoint=checkpoint)
  File "[..]/python3.10/site-packages/transformers/trainer.py", line 1631, in train
    return inner_training_loop(
  File "[..]/python3.10/site-packages/transformers/trainer.py", line 1975, in _inner_training_loop
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch, ignore_keys_for_eval)
  File "[..]/python3.10/site-packages/transformers/trainer.py", line 2236, in _maybe_log_save_evaluate
    self._save_checkpoint(model, trial, metrics=metrics)
  File "[..]/python3.10/site-packages/transformers/trainer.py", line 2339, in _save_checkpoint
    metric_value = metrics[metric_to_check]
KeyError: 'eval_metric-name'
```

Comments:
Without seeing the code you run and how it was launched, there is very little we can do to help you.

@sgugger no worries it's just to be able to reference to an issue when/if I submit a pull request
Rescale image back if it was scaled during PIL conversion
Author: amyeroberts
Pull request: yes

# What does this PR do?

Resolves an issue that occurs, when a float image, with values between `[0-1]` is passed into an image processor, its inputs are rescaled back between `[0, 255]` to convert to a `PIL.Image.Image` for resizing and isn't rescaled again. This results in inconsistent outputs depending on whether the `do_resize` flag is `True` or `False`. 

Images of this type are typically fed in when pipelines that convert images to torch tensors using `ToTensor`. 

Fixes #22392

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


Comments:
_The documentation is not available anymore as the PR was closed or merged._
4.27.1 breaks fp16 training of Flaubert
Author: thomas-schillaci
Pull request: no

### System Info

- `transformers` version: 4.27.1
- Platform: Linux-4.18.0-372.26.1.el8_6.x86_64-x86_64-with-glibc2.28
- Python version: 3.9.16
- Huggingface_hub version: 0.12.1
- PyTorch version (GPU?): 1.12.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help?

@sgugger @ArthurZucker

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Using transformers 4.26.1 the following script behaves properly (train and validation loss decreasing), using transformers >=4.27.1, the training loss is always 0, the validation loss always is nan.
Please note that the problem doesn't occur when removing `fp16=True`.

```python
from transformers import AutoModelForSequenceClassification, AutoTokenizer, Trainer, TrainingArguments
from datasets import load_dataset

model_name = 'flaubert/flaubert_base_cased'
model = AutoModelForSequenceClassification.from_pretrained(
    model_name,
    num_labels=2,
    problem_type='single_label_classification'
)
tokenizer = AutoTokenizer.from_pretrained(model_name)


def tokenize_function(example):
    return tokenizer(example['text'], padding=True, truncation=True)


dataset = load_dataset('imdb', split='train[:1%]')
dataset = dataset.train_test_split(test_size=0.2)
dataset = dataset.map(tokenize_function, batched=True, remove_columns=['text'])

train_args = TrainingArguments(
    'out',
    report_to=[],
    logging_strategy='steps',
    logging_steps=1,
    evaluation_strategy='epoch',
    num_train_epochs=1,
    fp16=True
)
trainer = Trainer(model=model, train_dataset=dataset['train'], eval_dataset=dataset['test'], args=train_args)
trainer.train()
```

### Expected behavior

Upgrading to >=4.27.1 should produce a similar training to 4.26.1.
Thank you for your help!

Comments:
Thanks for reporting and providing a clear reproducer! It let me pinpoint the regression to [this PR](https://github.com/huggingface/transformers/pull/21627). I think we shouldn't have touched that modeling code. Let me just consult internally and I will report back here with the next steps soon!

The PR mentions above reverts the commit that introduced the bug. This will be released in a patch (4.27.4) later today.

Awesome, thank you!

Tested on 4.27.4, this issue is fixed, thank you again!

Thanks for letting us know!
🚨🚨🚨   Fix ordering of height, width for BLIP image processor
Author: amyeroberts
Pull request: yes

# What does this PR do?

The BLIP image processor incorrectly passed in the dimensions to resize in the order `(width, height)`. This is reordered to be correct. 

In most cases, this won't have an effect as the default height and width are the same. However, this is not backwards compatible for custom configurations with different height, width settings and direct calls to the `resize` method with different height, width values.

This also updates the docstring as this was incorrect.  

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Skip flaky NLLB Moe test for now
Author: amyeroberts
Pull request: yes

# What does this PR do?

Skips flaky test. See failed CI run here: https://app.circleci.com/pipelines/github/huggingface/transformers/60931/workflows/4642cd50-8d8c-4a6b-bea9-b077ff7400cf/jobs/747651


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

Comments:
cc @sgugger @ArthurZucker 

_The documentation is not available anymore as the PR was closed or merged._

Thanks Amy!
[NLLB-MoE] `model_type` update for auto mapping
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Changes the `model_type` to allow users using the `AutoModel` . 
While we are at it, also moved the testing model to `hf-internal-testing`
Fixes #22461 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Llama: support for `max_position_embeddings`
Author: gante
Pull request: yes

# What does this PR do?

Related to #22433.

`LlamaConfig`, as opposed to other models, did not support `max_position_embeddings`. Instead, the position embedding class was hardcoded to a length of `2048`. This PR rectifies that, keeping the `2048` default.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Initialize OPT 175B model for a long time
Author: lelandfy
Pull request: no

### System Info

Recently I am trying to train OPT 175B (facebook/opt-175b) and found my code needs almost 10 hours to initialize the model weights before starting training. Actually my code is pretty simple
```
...
config = PretrainedConfig.from_json_file('175b.json')
model = OPTForCausalLM(config)
...
```
Is there any issue of my code or I have to configure something to accelerate the initialization? @ArthurZucker , @younesbelkada 

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I already posted the main part to cause the problem in problem description.

### Expected behavior

I think the initialization computation should not be so long.

Comments:
The line takes a very long time because there are 176 billions parameters to initialize. The initialization is also performed several times, which is a bug we recently fixed. If you use the latest main you might see it takes a bit less time.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Docs fix: Multinomial sampling decoding needs "num_beams=1", since by default it is usually not 1.
Author: manueldeprada
Pull request: yes

# Fix error in docs: multinomial sampling decoding strategy

As indicated in the library source code:
https://github.com/huggingface/transformers/blob/228792a9dc0c36f1e82ab441e1b1991d116ee0a0/src/transformers/generation/utils.py#LL1364-L1367

Multinomial sampling needs `num_beams=1`. However, this is not indicated in the docs, potentially leading to execute beam-search multinomial sampling instead of the intended multinomial sampling.

This deviation from the expected behaviour happens quite often, since a lot of models have in their `generation_config.json` the parameter `num_beams` set to something higher than 1. This happens, for example, in the majority of top translation models from the Hub.

Also, I have included "ancestral sampling" as another name for multinomial sampling, since it is the most common name in the decoding algorithms literature.


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).



## Who can review?

Original authors of this piece of documentation: @gante, @sgugger, @stevhliu and @MKhalusova 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Fixed bug to calculate correct xpath_sub_list in MarkupLMTokenizer
Author: silentghoul-spec
Pull request: yes

Fixed the problem where xpath_sub_list was computed incorrectly due to bug in code of MarkupLM Tokenizers.

# What does this PR do?
This PR fixes the bug with function get_xpath_seq method of MarkupLMTokenizer class inside src.transformers.models.markuplm.tokenization_markuplm module. 
Earlier in line number 304 xpath_sub_list was assigned instance of xpath_tag_list which supposedly made embedding of tags with different subscripts similar. Eg li[0] was similar to li[1].
It also fixes the same bug present in src.transformers.models.markuplm.tokenization_markuplm_fast module


Fixes # (issue)
Fixes the issue with incorrect xpath_sub_list computation inside get_xpath_seq method of MarkupLMTokenizer class inside src.transformers.models.markuplm.tokenization_markuplm module and also fixes the same issue in get_xpath_seq method of MarkupLMTokenizerFast class inside src.transformers.models.markuplm.tokenization_markuplm_fast module.

## Before submitting
- [ No] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [Yes ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ No] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ Yes] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ No] Did you write any new necessary tests?


## Who can review?
 @NielsRogge




Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks for fixing!
[Quick poll] Give your opinion on the future of the Hugging Face Open Source ecosystem!
Author: LysandreJik
Pull request: no
Labels: name: WIP

Thanks to all of you, Transformers just passed 75k :star2:  last week!

Since the last survey, a lot has happened: the [diffusers](https://github.com/huggingface/diffusers), [evaluate](https://github.com/huggingface/evaluate) and [skops](https://github.com/skops-dev/skops) libraries were born. `timm` joined the Hugging Face ecosystem. There were 25 new releases of `transformers`, 21 new releases of `datasets`, 13 new releases of `accelerate`.

If you have a couple of minutes and want to participate in shaping the future of the ecosystem, please share your thoughts: 

[**hf.co/oss-survey**](https://docs.google.com/forms/d/e/1FAIpQLSf4xFQKtpjr6I_l7OfNofqiR8s-WG6tcNbkchDJJf5gYD72zQ/viewform?usp=sf_link)

(please reply in the above feedback form rather than to this thread)

Thank you all on behalf of the HuggingFace team! 🤗

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

multi-gpu pre-training in one machine for Large GPT from scratch without horovod (**Model Parallelism**)

Closing as this survey has wrapped up
(Re-)Enable Nightly + Past CI
Author: ydshieh
Pull request: yes

# What does this PR do?

(Re-)Enable Nightly + Past CI

cc @stas00 : I don't think there is something (related to `DeepSpeed`) that really needs your review in this PR. But if you prefer, you can take a look the 2 `Dockerfile` files under `docker` (and more files if you want). Thank you.

p.s. I launched a full run (without TensorFlow past version CIs) [here](https://github.com/huggingface/transformers/actions/runs/4532718828)

Comments:
_The documentation is not available anymore as the PR was closed or merged._

The DS part looks good, @ydshieh

I wonder if you want to continue testing torchdynamo at all. Users wanting to use it should be encouraged to move to torch>=2.0 instead, where it's built in. But a subject for a different PR I guess.

> The DS part looks good, @ydshieh
> 
> I wonder if you want to continue testing torchdynamo at all. Users wanting to use it should be encouraged to move to torch>=2.0 instead, where it's built in. But a subject for a different PR I guess.

From my side, it would be great if I don't have to deal with all the potential (installation/runtime) issues for such 3rd party libraries across with different torch versions (at least, not with previous torch versions). It's best to focus on the torch and torch+DeepSpeed testing results.

oh, I meant not testing torchdynamo in general transformers-wide. For sure you don't need any unrelated packages installed to test deepspeed, other its own deps.


Without TensorFlow Past CI - it takes 2.5 days to run the Nightly CI + PyTorch Past CI.
I put the schedule to trigger the workflow on Sunday and Thursday at 2 AM.

The TensorFlow past CI will only run under push events.
Generate: PT Dynamo without graph breaks in the main greedy/sample loop
Author: gante
Pull request: yes

# What does this PR do?

This PR is part of our PT Dynamo + `.generate()` readiness.

Let's start with the basics: 
1 - Calling generate after `torch.compile()` doesn't explode -- this PR fixes the error seen in https://github.com/pytorch/pytorch/issues/93042
2 - There are no graph breaks in the main generation loop, for greedy search and sample. [Graph breaks are a major source of slowdowns](https://pytorch.org/docs/master/dynamo/faq.html#why-am-i-not-seeing-speedups).

A quick run on GPT2 shows that we gain a ~1.5x speed with `torch.compile()` on `.generate()`, after these changes (~4 mins of compilation time). Please note that this is a quick check, and not a proper benchmark ;)

Comments:
_The documentation is not available anymore as the PR was closed or merged._
`libssl.so.10' not found but ll exsit
Author: co-develop-drv
Pull request: no

### System Info

ImportError: /home/aaa/anaconda3/envs/ner/lib/python3.9/site-packages/tokenizers/../../../libssl.so.10: version `libssl.so.10' not found (required by /home/aaa/anaconda3/envs/ner/lib/python3.9/site-packages/tokenizers/tokenizers.cpython-39-x86_64-linux-gnu.so)

(ner) aaa@:lib$ ll /home/aaa/anaconda3/envs/ner/lib/python3.9/site-packages/tokenizers/../../../libssl.so.10
lrwxrwxrwx 1 aaa aaa 15 Mar 30 21:03 /home/aaa/anaconda3/envs/ner/lib/python3.9/site-packages/tokenizers/../../../libssl.so.10 -> libssl.so.1.0.0

Ubuntu 22.04.2 LTS
conda 23.1.0

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. in another environment of conda 'git clone https://github.com/huggingface/transformers.git' and 'pip install -e .'
2. pip uninstall transformers
3. conda remove -n (that conda env in step 1) --all
4. then `libssl.so.10' not found and I copy libssl.so.1.0.0 from other place in my ubuntu
5. ln -s libssl.so.1.0.0          libssl.so.10

### Expected behavior

just found it

Comments:
it is my mistake：chmod 775
ViTModel extract only last layer's attention weights
Author: Interpause
Pull request: no

### Feature request

See: https://huggingface.co/docs/transformers/model_doc/vit#transformers.ViTModel

I propose a new keyword argument for `forward()`, `output_attention_layers: Tuple[int, ...] = (,)`, that allows one to specify which layers to extract attention weights for when `output_attentions=True`. When specified, other layers' attention weights are **immediately discarded to save VRAM**.

### Motivation

Setting `output_attentions=True` currently returns attention weights for all layers. When using a high resolution, this uses a prohibitive amount of VRAM, resulting in CUDA OOM.

For context, I am developing a model that could benefit from utilizing the last layer attention weights of DINO. However, I currently OOM when I try and train at a higher resolution. After experimenting with the various return options for `forward()`, I realised `output_attentions` and `output_hidden_states` use much more VRAM than just returning the last hidden state. As such, I am unable to feasibly train my model without pre-caching DINO's outputs.

### Your contribution

I might be able to work on a PR during the weekends.

Comments:
Or you can adapt the modeling code of ViT to suit your needs. That is why each model is fully defined in their modeling file.

I see, okay then.
Move common properties to BackboneMixin
Author: amyeroberts
Pull request: yes

# What does this PR do?

First of a series of PRs to enable loading timm checkpoints using the `AutoBackbone` API. This PR moves common logic e.g. `channels` property to the `BackboneMixin` class. This is for two main reasons: 
* Reduce duplicated code
* Enable using similar logic across the transformer and timm backbones


## Series of PRs
- [x] Moving common logic into the `BackboneMixin` class (this PR)
- [ ] Add `out_indices` to backbones  - [PR](https://github.com/amyeroberts/transformers/pull/109/files)
Note: This is an optional design choice and not necessary for loading the timm backbones
- [ ] Add tests for backbone models - [PR](https://github.com/amyeroberts/transformers/pull/110/files) 
- [ ] Add `TimmBackbone` model that can be loaded through `AutoBackbone`  - [PR](https://github.com/amyeroberts/transformers/pull/111/files)
This is where all the important stuff happens 🪄 

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

Comments:
_The documentation is not available anymore as the PR was closed or merged._