noamaanMulla-03/transformers-issues huggingface dataset:
Fix Mamba slow path bug with dtype mismatch.
Author: Adibvafa
Pull request: yes

# What does this PR do?
Fix issue https://github.com/huggingface/transformers/issues/32690

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [X] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [] Did you write any new necessary tests?


## Who can review?
@ArthurZucker 

Comments:
Thanks @Adibvafa for the fix! Do you think we could we add a test for f16/bf16 inference to avoid possible regressions later?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32691). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hi again @Adibvafa , I'm back from holidays :) iiuc there aren't tests for various inference dtypes, do you think they could be added to this PR? then we can merge it!

Hi @molbap, I added a test to Mamba for checking this dtype mismatch.
It seems Mamba2 is suffering from the same bug https://github.com/huggingface/transformers/issues/33409
If the test is good I can add it there too.

@molbap There seems to be an issue with the test. I will fix it.

Thanks for your work on that @Adibvafa !

@molbap I added a test named `test_dtype_mismatch_handled_in_cache` to catch any dtype mismatch between model and cache. It only passes by changing `conv_state[:, :, cache_position] = new_conv_state.to(device=conv_state.device)` to `conv_state[:, :, cache_position] = new_conv_state.to(device=conv_state.device, dtype=conv_state.dtype)` as expected.

Thanks 🤗
`modeling_mllama.py`
Author: HanGyeol-Yoo
Pull request: no
Labels: name: bug

### System Info

v4.45.0

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

In
https://github.com/huggingface/transformers/blob/2e24ee4dfa39cc0bc264b89edbccc373c8337086/src/transformers/models/mllama/modeling_mllama.py#L1438-L1456

The example shows that when printing the shape of last_hidden_state, it outputs "torch.Size([1, 1, 4, 1025, 7680])". 
However, when I run it directly, it outputs "torch.Size([1, 1, 4, 1601, 7680])". Is this an error in the example?


### Expected behavior

It seems the example is incorrect.

Comments:
Hi, it depends on the model configuration, probably you are using instruct or 90B version?

> Hi, it depends on the model configuration, probably you are using instruct or 90B version?

I used the 11B Inst version and got the output "torch.Size([1, 1, 4, 1601, 7680])".
I confirmed that the output is different because the Config of the PT version and the Inst version of the model are different. Thank you!


Yes, it depends on image size parameter, for Instruct and 90B models image size is 560, while for 11B base model it's 448.
When will Transformers version 4.45.0 be released?
Author: hahmad2008
Pull request: no
Labels: name: bug

### System Info

I installed this version 4.45.0.dev0 repo itself, by using:

`pip install -U git+https://github.com/huggingface/transformers.git `

When this tag `4.45.0` will be officially released?

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

-

### Expected behavior

-

Comments:
Probably next week!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

It was released, closing this :)
Generate: move llama `prepare_inputs_for_generation` to `GenerationMixin`
Author: gante
Pull request: yes

# What does this PR do?

Step 3 of #32685 [move as many `prepare_inputs_for_generation` variants as possible into `GenerationMixin`]

This PR does the following changes on `llama`:
- move `prepare_inputs_for_generation` to `GenerationMixin`. The goal is to have a base `prepare_inputs_for_generation` that works on most generate-capable models, as a) this is not modeling code b) most of them are copy-paste;
- move `_prepare_4d_causal_attention_mask_with_cache_position` from a standalone method into `LlamaModel`. `prepare_inputs_for_generation` calls this function and this function may change on a given model, so it must be part of the model object. It is also part of the model forward pass, so I've decided to place it in each model's base model class, as opposed to in `PreTrainedModel`

Because `llama` is central and has many derived models, this change required touching many models. All changes follow the logic above.

👉 review suggestion: `generation/utils.py` -> `llama` -> the rest


_____________________________
✅ slow llama tests are green

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33677). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hey! 🤗 Thanks for your contribution to the `transformers` library!

Before merging this pull request, slow tests CI should be triggered. To enable this:
- Add the `run-slow` label to the PR
- When your PR is ready for merge and all reviewers' comments have been addressed, push an empty commit with the command `[run-slow]` followed by a comma separated list of all the models to be tested, i.e. `[run_slow] model_to_test_1, model_to_test_2`
  - If the pull request affects a lot of models, put at most 10 models in the commit message
- A `transformers` maintainer will then approve the workflow to start the tests

(For maintainers) The documentation for slow tests CI on PRs is [here](https://www.notion.so/huggingface2/CI-for-pull-requests-8335bd9217d24d1e8ed1d3a016f39804).
Refactor image features selection in LlaVa
Author: kenza-bouzid
Pull request: no
Labels: name: Feature request

name: Vision

### Feature request

We would like to wrap image features selection in LLaVa in a separate function to make it easier to override for custom use cases (e.g. applying a layer norm on the image features before projection, etc).

https://github.com/huggingface/transformers/blob/52daf4ec768fb9ffe84a0c373834172a7c54aecc/src/transformers/models/llava/modeling_llava.py#L452-L461

Currently, we need to override the entire forward pass for any custom processing of the image features. Having a separate function would make it a lot easier! 

### Motivation

To make it easier to implement custom image features selection in LlaVa. 

### Your contribution

A custom function for image features selection
```python
    def _get_selected_image_features(
        self, pixel_values: torch.FloatTensor, vision_feature_layer: int, vision_feature_select_strategy: str
    ):
        image_outputs = self.vision_tower(pixel_values, output_hidden_states=True)
        # this is not memory efficient at all (output_hidden_states=True) will save all the hidden stated.
        selected_image_feature = image_outputs.hidden_states[vision_feature_layer]
        if vision_feature_select_strategy == "default":
            selected_image_feature = selected_image_feature[:, 1:]
        elif vision_feature_select_strategy == "full":
            selected_image_feature = selected_image_feature
        else:
            raise ValueError(f"Unexpected select feature strategy: {self.config.vision_feature_select_strategy}")
        return selected_image_feature
```

Comments:
cc @zucchini-nlp for VLMs
prune_heads() method for AutoModelForCausalLM
Author: mnauf
Pull request: no
Labels: name: Feature request

### Feature request

We have `prune_heads()` method for `AutoModel` class, but not for `AutoModelForCausalLM`. Please provide `prune_heads()` method to `AutoModelForCausalLM` class.

### Motivation

Mechanistic interpretability study from the paper [Are Sixteen Heads Really Better than One?](https://arxiv.org/abs/1905.10650) explore how pruning attention heads effect the model's (BERT) performance. I want to do something similar for LLMs. Since LLMs are supported via `AutoModelForCausalLM` class, I am unable to do this experiment.


### Your contribution

Just getting started with HuggingFace. Not confident about implementing this feature on my own as of now. 

Comments:
Hey @mnauf! Over the years we've seen very limited interest in this feature, hence why it wasn't implemented in recent LLMs.

If we see interest for it growing then we might consider it; the best way to approach it might be for you to do a fork with your changes to kickstart your experiments.

edit: found a draft pr here - https://github.com/huggingface/transformers/pull/33377

---

@LysandreJik if not for a prune_heads() method, do you think that, with the recent explorations around llm pruning, we could at least have a structure in place for loading a pruned model? Ideally, depth/width pruned models be loadable. Just how we do it for quantized models, having a support in place for pruned models would be useful. 
I'd love to be pointed towards an already open/closed discussion/thread on the same. 
Thanks.

Yeah having something related to pruning does not sound too bad! You can already load a model with less width, with `ignore_missmatched_sizes = True` in `from_pretrained`. For width, you do need to update the config. 

On the hub we could do stuff tho to link pruned versions !
Workaround for bark issue in pipelines
Author: Rocketknight1
Pull request: yes
Labels: name: run-slow

The way we pass `generation_config` to `Bark` seems to cause some issues with duplicated keys, see #33779. This was caused by changes in #33480

This issue doesn't seem to occur with other `text-to-speech` or `text-to-audio` models, and I got a local workaround working. However, I suspect there's probably a cleaner solution to the issue than what I did here! More importantly, I probably need to investigate why this wasn't flagged in the CI/nightly runs.

cc @gante @sanchit-gandhi @ylacombe 

Comments:
Hey! 🤗 Thanks for your contribution to the `transformers` library!

Before merging this pull request, slow tests CI should be triggered. To enable this:
- Add the `run-slow` label to the PR
- When your PR is ready for merge and all reviewers' comments have been addressed, push an empty commit with the command `[run-slow]` followed by a comma separated list of all the models to be tested, i.e. `[run_slow] model_to_test_1, model_to_test_2`
  - If the pull request affects a lot of models, put at most 10 models in the commit message
- A `transformers` maintainer will then approve the workflow to start the tests

(For maintainers) The documentation for slow tests CI on PRs is [here](https://www.notion.so/huggingface2/CI-for-pull-requests-8335bd9217d24d1e8ed1d3a016f39804).

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33824). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Given the issue,should we do a patch?

@LysandreJik unclear, since it only affects Bark! It's just unfortunate that it's the default pipeline model.

Anyway, merging for now until we come up with a more permanent solution!
Fix device mismatch errors
Author: zucchini-nlp
Pull request: yes
Labels: name: run-slow

# What does this PR do?

Fixes https://github.com/huggingface/transformers/issues/33816, and propagates changes to all other VLMs with masked scatter. Lllava-onevision already has such a device movement, so we should be good to go with these changes


Comments:
Hey! 🤗 Thanks for your contribution to the `transformers` library!

Before merging this pull request, slow tests CI should be triggered. To enable this:
- Add the `run-slow` label to the PR
- When your PR is ready for merge and all reviewers' comments have been addressed, push an empty commit with the command `[run-slow]` followed by a comma separated list of all the models to be tested, i.e. `[run_slow] model_to_test_1, model_to_test_2`
  - If the pull request affects a lot of models, put at most 10 models in the commit message
- A `transformers` maintainer will then approve the workflow to start the tests

(For maintainers) The documentation for slow tests CI on PRs is [here](https://www.notion.so/huggingface2/CI-for-pull-requests-8335bd9217d24d1e8ed1d3a016f39804).

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33851). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Validate the eval dataset in advance.
Author: jackyjinjing
Pull request: yes

# What does this PR do?
Validate whether the eval dataset is empty in advance to avoid wasting the time of training for one epoch only to discover this issue.

Fixes # (issue)
fix #33721

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.
@muellerzr and @SunMarc

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts, @qubvel
- speech models: @ylacombe, @eustlb
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Rocketknight1
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @SunMarc
- chat templates: @Rocketknight1

Integrations:

- deepspeed: HF Trainer/Accelerate: @muellerzr
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33743). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Add include_loss_for_metrics
Author: Manalelaidouni
Pull request: yes

# What does this PR do?
Fixes #32307

This PR includes loss in the `compute_metrics` function via the `include_loss_for_metrics` training argument flag, this is particularly useful for calculating loss dependent metrics, the `EvalPrediction` class now handles optional inputs and losses.

## Before submitting

- [ ]  This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x]  Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),Pull Request section?
- [x]  Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link to it if that's the case.
- [ ]  Did you make sure to update the documentation with your changes? Here are the[documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and[here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ]  Did you write any new necessary tests?

## Who can review?

@muellerzr @SunMarc


Comments:
Hey @SunMarc, thanks for the great feedback! 
I’ve refactored the code as you suggested and updated the EvalPrediction accordingly,

I wanted to ask you about the failing tests (in tests/models/gpt2/test_modeling_flax_gpt2.py) they don't seem related to my changes and the tests passed locally for me, am I missing something here?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33088). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@muellerzr Sounds good to me! 

@muellerzr great idea! I was actually hesitant to suggest removing  `include_input_for_metrics`  but I agree it would make the API much cleaner; 

Just to make sure we’re on the same page before I move forward with your suggestion, so we’ll set `include_for_metrics` as a `(list[str], *optional*, defaults to [])` — 

To set `include_for_metrics` as an empty list and not `None` (to simplify code 
and not have to check if it’s `None` everytime we use it) we’ll mark the field as `default_factory=list`  — 

We'll also add a deprecation warning for `include_inputs_for_metrics` like this:


```python
if args.include_inputs_for_metrics:
    warnings.warn("Using `include_inputs_for_metrics` is deprecated. Please use `include_for_metrics` instead.",FutureWarning)
    args.include_for_metrics.append("inputs")
```


@muellerzr @amyeroberts does this sound good?

@Manalelaidouni Sounds good to me! Only small addition would be that we'd ideally mention when the `include_for_metrics` will be deprecated (v5) and 

Hi @muellerzr, just wanted to follow up on this PR, would appreciate your review when possible, thanks in advance!

Just bumping this - really useful change, would like to see this PR merged if possible

Merging it ! Thanks for your work @Manalelaidouni !
Raise `accelerate` dependency error in case of defaulting `low_cpu_mem_usage=True`
Author: kylesayrs
Pull request: yes

# What does this PR do?

Fixes bugs where the `low_cpu_mem_usage` argument is changed during loading but after checking for accelerate installation.

Set up environment
```bash
python3 -m pip install compressed-tensors
```

In an environment where accelerate is not installed
```python3
from transformers import AutoModel
model = AutoModel.from_pretrained("nm-testing/TinyLlama-1.1B-Chat-v1.0-W8A8_tensor_weight_static_per_tensor_act-e2e")
# `low_cpu_mem_usage` was None, now set to True since model is quantized. 
# NameError: name 'init_empty_weights' is not defined
```

After changes
```python3
from transformers import AutoModel
model = AutoModel.from_pretrained("nm-testing/TinyLlama-1.1B-Chat-v1.0-W8A8_tensor_weight_static_per_tensor_act-e2e")
# `low_cpu_mem_usage` was None, now default to True since model is quantized.
# ImportError: Using `low_cpu_mem_usage=True` or a `device_map` requires Accelerate: `pip install 'accelerate>=0.26.0'`

# Now the user knows to either install accelerate or set low_cpu_mem_usage=False
```

## Who can review?
@mgoin @SunMarc @ArthurZucker
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33830). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
'DepthEstimationPipeline' object has no attribute 'image_size' when num_workers > 0
Author: rsamf
Pull request: no
Labels: name: bug

### System Info

OS: Linux (Ubuntu 22)
transformers: 4.44.2
torch: 2.3.1


### Who can help?

@Narsil 

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

See code snippet below to reproduce:

```python

from transformers import pipeline
import torch

pipe = pipeline(model="LiheYoung/depth-anything-base-hf", device=0, num_workers=1)

# To reproduce, the input needs to be an array of images, and it can just be 1 image in the array.
# A single image not placed in an array will not reproduce the issue.
pipe(["<YOUR IMAGE>"])
```

This results in:
```
File "[...]/site-packages/transformers/pipelines/depth_estimation.py", line 109, in postprocess
    predicted_depth.unsqueeze(1), size=self.image_size[::-1], mode="bicubic", align_corners=False
AttributeError: 'DepthEstimationPipeline' object has no attribute 'image_size'
```

Upon investigation, you will see that `self.image_size` is set in `preprocess()` which is executed by one of the workers while (I believe) `postprocess()` is executed on the main process. Because these 2 processes do not share the same class instance of DepthEstimationPipeline, `self.image_size` ends up never being defined in the main processes's address space. This makes the DepthEstimationPipeline only work when `preprocess()` and `postprocess()` are executed on the same process when num_workers=0.

A quick fix would be for DepthEstimationPipeline to accept an `image_size` argument which can be passed to `preprocess_params --> preprocess()`. Then the user can create a pipeline by doing
```python
pipe = pipeline(model="LiheYoung/depth-anything-base-hf", device=0, num_workers=1, image_size=(256,256))
```
for example. Of course, it's not as clean since it doesn't dynamically infer the image sizes from the user's input, but it should help. Any other ideas?

### Expected behavior

For `DepthEstimationPipeline.__call__()` to execute successfully with multiprocessing.

Comments:
cc @NielsRogge 

Looking into this, I noticed that some pipelines do not have the pretense of matching the input size (e.g. image_to_image), while others that have this requirement (e.g. image_segmentation) achieve so by leveraging the `model_inputs` and `model_outputs` context object.

I believe depth estimation to be better suited for the latter case, so I opened a PR (#33759) that exploits the contexts in a similar way to carry over the `target_size` value all the way to post processing.

cc @amyeroberts and @Rocketknight1 on the above ^