noamaanMulla-03/transformers-issues huggingface dataset:
weird criterion to decide if needed to adjust the padding size
Author: hyusterr
Pull request: no

https://github.com/huggingface/transformers/blob/241c04d36867259cdf11dbb4e9d9a60f9cb65ebc/src/transformers/trainer_pt_utils.py#L93

When working with `Trainer` and model output with `TokenClassiferOutput`, the `Trainer` collect outputs batch by batch in `evaluate_loop` with `nested_concat`.
The criterion to decide whether it is needed to adjust the shape is by tensor.shape[1]. 
However, if user does not exclude `attentions` with `ignore_key` beforehand, the shape[1] of the attentions tensor is always the same. Leading to error occurred with `torch.cat`. 
I think this design is kind of weird, especially because this behavior is not emphasized in the token classification tutorial: https://huggingface.co/docs/transformers/tasks/token_classification


Comments:
cc @SunMarc @muellerzr 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[tests] fix bnb test bug 
Author: faaany
Pull request: yes

## What does this PR do?
Currently, the below test fails:

```bash
pytest -rA tests/models/esm/test_modeling_esm.py::EsmModelIntegrationTest::test_inference_bitsandbytes
```

```bash
>       return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
E       RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index in method wrapper_CUDA__index_select)

../../miniforge3/envs/hf-dev/lib/python3.10/site-packages/torch/nn/functional.py:2551: RuntimeError
----------------------------------------------------------- Captured stderr call -----------------------------------------------------------
The `load_in_4bit` and `load_in_8bit` arguments are deprecated and will be removed in the future versions. Please, pass a `BitsAndBytesConfig` object in `quantization_config` argument instead.
`low_cpu_mem_usage` was None, now default to True since model is quantized.
Loading checkpoint shards: 100%|██████████| 2/2 [00:01<00:00,  1.17it/s]
========================================================= short test summary info ==========================================================
FAILED tests/models/esm/test_modeling_esm.py::EsmModelIntegrationTest::test_inference_bitsandbytes - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument fo...
```
This PR fixes it.

cc @ydshieh 

Comments:
Add Evolla model
Author: zhoubay
Pull request: no
Labels: name: New model

### Model description

# Model Name: Evolla

## Model Specifications
* **Model Type:** Protein-language generative model
* **Parameters:** 80 billion
* **Training Data:** AI-generated dataset with 546 million protein question-answer pairs and 150 billion word tokens

## Architecture
Multimodal model integrating a protein language model (PLM) as the encoder, a large language model (LLM) as the decoder, and a sequence compressor/aligner module.

## Key Features
- Decodes the molecular language of proteins through natural language dialogue
- Generates precise, contextually nuanced insights into protein function
- Trained on extensive data to capture protein complexity and functional diversity

## Applications
* **Protein Function Annotation:** Provides detailed functional insights for proteins
* **Enzyme Commission (EC) Number Prediction:** Assists in classifying enzymatic activities
* **Gene Ontology (GO) Annotation:** Helps in understanding protein roles in biological processes
* **Subcellular Localization Prediction:** Predicts where proteins are located within a cell
* **Disease Association Analysis:** Identifies potential links between proteins and diseases
* **Other Protein Function Characterization Tasks:** Supports various research needs in proteomics and functional genomics

## Performance
Demonstrates expert-level insights, advancing research in proteomics and functional genomics.

## Availability
* **Evolla-10B Weights:** [Hugging Face](https://huggingface.co/westlake-repl/Evolla-10B)
* **Code Repository:** [GitHub](https://github.com/westlake-repl/Evolla)
* **Webserver:** [Chat-Protein](http://www.chat-protein.com/)

## License
MIT License

## Contact
For inquiries, contact the corresponding author(s) via email (e.g., yuanfajie@westlake.edu.cn).

### Open source status

- [x] The model implementation is available
- [x] The model weights are available

### Provide useful links for the implementation

_No response_

Comments:
Fix `LlavaForConditionalGenerationModelTest::test_config` after #36077
Author: ydshieh
Pull request: yes

# What does this PR do?

This might be just a temporary fix though.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36230). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hi ✋ 

In #36077, there is no newly added `check_config_can_be_init_without_params`. There is a newly added test file/class `tests/models/llava/test_configuration_llava.py` and relevant methods, but they are extra tests.

This PR only affects  the original `test_config` in `LlavaForConditionalGenerationModelTest` in `tests/models/llava/test_modeling_llava.py` (which are independent from the new `tests/models/llava/test_configuration_llava.py`).

And it's not dummy: it simply select the `else` branch below (defined in `tests/test_configuration_common.py`)

```
    def check_config_can_be_init_without_params(self):
        if self.config_class.is_composition:
            with self.parent.assertRaises(ValueError):
                config = self.config_class()
        else:
            config = self.config_class()
            self.parent.assertIsNotNone(config)
```
AMD DeepSpeed image additional HIP dependencies
Author: ivarflakstad
Pull request: yes

Add hipblastlt and hipsolver as dependencies to the amd deepspeed image as they are needed to build various extensions.
Upgraded torchlibs with rocm6.2.4 index.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36195). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Use torch 2.5.0 for deepspeed amd image
Author: ivarflakstad
Pull request: yes

torchvision 0.20.0 depends on torch 2.5.0, so 2.5.1 was a mistake.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35931). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks @ivarflakstad for the review.

Since @muellerzr maintains deepeed CI, I will also wait a bit for they to also take a look.
Request to add GTrXL: Stabilizing Transformers for Reinforcement Learning
Author: ashok-arora
Pull request: no
Labels: name: New model

### Model description

Gated Transformer-XL (GTrXL) improves upon Transformer-XL by addressing its instability in reinforcement learning. It introduces gating mechanisms similar to LSTMs, which help stabilize training and improve long-term memory retention. This makes GTrXL more effective in partially observable environments where capturing long-range dependencies is crucial.

### Open source status

- [x] The model implementation is available
- [ ] The model weights are available

### Provide useful links for the implementation

- https://github.com/opendilab/PPOxFamily/blob/main/chapter5_time/gtrxl.py
- https://github.com/alantess/gtrxl-torch 

Comments:
Add missing atol to torch.testing.assert_close where rtol is specified
Author: ivarflakstad
Pull request: yes

The formulae is `|actual - expected| <= atol + rtol * |expected|`, so I don't know if my approach of just copying the value of rtol into atol is good enough.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36234). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Qwen2-5-VL: revert slow image processor
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Fixes https://github.com/huggingface/transformers/issues/36193

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36236). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[tests] remove tf/flax tests in `/generation`
Author: gante
Pull request: yes

# What does this PR do?

`generate` has a few things that are shared across frameworks. Sometimes, when making core changes, we need to update all three frameworks. Otherwise, we get a red CI ([example](https://github.com/huggingface/transformers/pull/36215)).

Since we're deprecating TF/JAX, this PR removes their tests regarding `generate` so that the other frameworks no longer hinder PT-related `generate` development.

cc @LysandreJik @ArthurZucker as we chatted on Slack 🤗 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36235). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Remove loading custom kernel for RT-DETRv2
Author: qubvel
Pull request: yes
Labels: name: cleanup

name: Vision

# What does this PR do?

RT-DETRv2 does not have a custom kernel implemented, but still loads the one for the first version. This PR removes the unnecessary loading.

cc @jadechoghari

Comments:
run-slow: rt_detr_v2

This comment contains run-slow, running the specified jobs: This comment contains run-slow, running the specified jobs:

models: ['models/rt_detr_v2']
quantizations: [] ...

looks good! thanks!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36098). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

run-slow: rt_detr_v2

This comment contains run-slow, running the specified jobs: This comment contains run-slow, running the specified jobs:

models: ['models/rt_detr_v2']
quantizations: [] ...

run-slow: rt_detr_v2

This comment contains run-slow, running the specified jobs: This comment contains run-slow, running the specified jobs:

models: ['models/rt_detr_v2']
quantizations: [] ...
[generate] Fix encoder decoder models attention mask
Author: eustlb
Pull request: yes

# What does this PR do?

https://github.com/huggingface/transformers/pull/35852 introduced a bug where the attention mask is lost for encoder-decoder models.
This PR implements a naïve fix.

cf https://github.com/huggingface/transformers/issues/36020

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36018). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

agree that having a `encoder_mask` would be cleaner, but it diverges a bit with the current convention in the codebase where encoder-decoder model forward's use `attention_mask` for the encoder and ` decoder_attention_mask` otherwise. It is also how it is currently documents:
https://github.com/huggingface/transformers/blob/9d2056f12b66e64978f78a2dcb023f65b2be2108/src/transformers/generation/utils.py#L1957-L1959

@zucchini-nlp As @eustlb wrote, historically the naming convention is for the `attention_mask` to attend to the input (back when there were no multimodal models) 👀 

@eustlb I'm going to rerun the tests until CI gets green, and then merge this PR -- not having this fix is blocking me :D
AutoProcessor loading error
Author: JJJYmmm
Pull request: no
Labels: name: bug

### System Info

Related Issues and PR: #34307 https://github.com/huggingface/transformers/pull/36184
- `transformers` version: 4.49.0.dev0
- Platform: Linux-5.15.0-131-generic-x86_64-with-glibc2.35
- Python version: 3.10.16
- Huggingface_hub version: 0.27.1
- Safetensors version: 0.5.2
- Accelerate version: 1.0.1
- Accelerate config:    not found
- PyTorch version (GPU?): 2.6.0+cu126 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>
- GPU type: NVIDIA H100 80GB HBM3

### Who can help?

@Rocketknight1 

### Information

- [x] The official example scripts
- [x] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Here are the reprodution steps

1. choose a mllm like qwen2.5vl, and download it's config file
2. derive its images processor, processor and model
3. modify the config file and try to use AutoProcessor to load_from_pretrain
4. and the error occurs like #34307

```python
from transformers import Qwen2_5_VLProcessor, Qwen2_5_VLImageProcessor, Qwen2_5_VLForConditionalGeneration, Qwen2_5_VLConfig

class NewProcessor(Qwen2_5_VLProcessor):

    image_processor_class = "NewImageProcessor"

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)

class NewImageProcessor(Qwen2_5_VLImageProcessor):

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)

class NewConfig(Qwen2_5_VLConfig):
    model_type = "new_model"

class NewModel(Qwen2_5_VLForConditionalGeneration):

    config_class = NewConfig

    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)



from transformers import AutoModel, AutoImageProcessor, AutoProcessor

AutoImageProcessor.register(NewModel.config_class, NewImageProcessor)
AutoProcessor.register(NewModel.config_class, NewProcessor)
AutoModel.register(NewModel.config_class, NewModel)


if __name__ == "__main__":
    processor = NewProcessor.from_pretrained("path/to/NewModel_config/")
```

modified config
```
config.json:

"architectures": [
    "NewModel"
],
"model_type": "new_model",

preprocessor_config.json:

"image_processor_type": "NewImageProcessor",
"processor_class": "NewProcessor"

```

I also check the pr https://github.com/huggingface/transformers/pull/36184, it didn't work, because the func _get_class_from_class_name use mapping but the key is string rather than Config class

### Expected behavior

None

Comments:
fix by https://github.com/huggingface/transformers/pull/36184
[tests] remove `test_export_to_onnx`
Author: gante
Pull request: yes

# What does this PR do?

Removes `test_export_to_onnx` -- the export is handled in `optimum` (see [this comment](https://github.com/huggingface/transformers/pull/30375#issuecomment-2178959189))

Most of them were being skipped anyways 👀 

(The `optimum` team was notified on slack)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36241). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[GGUF] Support new architectures/ quantisation schemes in Transformers
Author: Vaibhavs10
Pull request: no
Labels: name: contributions-welcome

### GGUF 🤝 Transformers

Currently, Transformers allows one to load a GGUF directly in Transformers, or alternatively, convert it into Transformers format. [ref](https://huggingface.co/docs/transformers/main/en/gguf)

We currently support LLaMa3, Mistral and Qwen 2 architectures.

Additionally we support the following Quantisation schemes:

- F32
- Q2_K
- Q3_K
- Q4_0
- Q4_K
- Q5_K
- Q6_K
- Q8_0

We appreciate contributions and suggestions on which model architectures & quantisation schemes we should support next.

cc: @younesbelkada @LysandreJik 

Comments:
What about "Importance Matrix"? Is this relevant here?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@Vaibhavs10 Are there any plans to add deepseek v2, v3 as well?
running utills.fx.symbolic_trace on gp2 raised an error: torch.fx.proxy.TraceError: Proxy object cannot be iterated, which does not occur in the previous version 
Author: minkiml
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.48.0
- Platform: Linux-4.18.0-535.el8.x86_64-x86_64-with-glibc2.28
- Python version: 3.9.21
- Huggingface_hub version: 0.27.0
- Safetensors version: 0.4.5
- Accelerate version: 1.2.1
- Accelerate config:    - compute_environment: LOCAL_MACHINE
        - distributed_type: MULTI_GPU
        - mixed_precision: no
        - use_cpu: False
        - debug: False
        - num_processes: 8
        - machine_rank: 0
        - num_machines: 1
        - rdzv_backend: static
        - same_network: False
        - main_training_function: main
        - enable_cpu_affinity: False
        - downcast_bf16: False
        - tpu_use_cluster: False
        - tpu_use_sudo: False
- PyTorch version (GPU?): 2.5.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>
- GPU type: NVIDIA A100-SXM4-80GB

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
from transformers import GPT2Model, GPT2Config
from transformers.utils import fx

config = GPT2Config.from_pretrained("gpt2")
model = GPT2Model(config)

trace = fx.symbolic_trace(model, input_names=["input_ids", "attention_mask"])
```

....
File "/.conda/envs/venv/lib/python3.9/site-packages/torch/fx/proxy.py", line 327, in iter
raise TraceError('Proxy object cannot be iterated. This can be '
torch.fx.proxy.TraceError: Proxy object cannot be iterated. This can be attempted when the Proxy is used in a loop or as a *args or **kwargs function argument. See the torch.fx docs on pytorch.org for a more detailed explanation of what types of control flow can be traced, and check out the Proxy docstring for help troubleshooting Proxy iteration errors

### Expected behavior

No error.

Comments:
cc @muellerzr @SunMarc for torch.compile I think (feel free to ping someone else if you think they'd be better suited)

Thanks for the report @minkiml ! Could you trace back to the commit that caused this issue ? Also we have a `symbolic_trace` function in transformers to trace the model, cc @michaelbenayoun as I saw that you worked on fx

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
from_pretrained fails to save weights.py and layers.py into cache, therefore fails to find them in cache
Author: openyk
Pull request: no
Labels: name: bug

- weights.py and layers.py confirmed exist in local model folder, originally saved via cloning (attempt1) and via save_pretrained (attempt2)
- attempting to set custom cache dir as argument or by TRANSFORMERS_CACHE ENV... both ways no effect
- cache folder and other files are saved successfully, so not a permissions issue

### System Info

- `transformers` version: 4.48.0
- Platform: Linux-6.8.0-51-generic-x86_64-with-glibc2.39 (ubuntu 24.04.1)
- Python version: 3.12.3
- Huggingface_hub version: 0.27.1
- Safetensors version: 0.5.2
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.5.1+cu124 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: no


### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("/path/to/model", trust_remote_code=True, local_files_only=True)
```

### Expected behavior

cache is generated with all relevant model files saved successfully, then loaded from cache for loading the model.

Comments:
Could you update the reproduction instructions with the cache setup too and for which model did you face this ? (if this was model specific) , But, yeah if you update the reproducing steps. That could help the community to come up with a uniformed approach

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Anole add model
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Adds Anole as a new model, a new PR based on https://github.com/huggingface/transformers/pull/32013

Comments:
This PR is ready. One thing to note is that image generation quality is very random, and even with the CFG we have it is not the best. The original repo is neither as good as the latest image generation models, and they have a slighly different CFG for instruct-based models

I can take a look at trying to match at least the original repo quality, but seems like Anole is not top model anymore. Their advantage was in having interleaved generation possible, I believe Janus also support it now given that it is one model doing both modalities. 

So now I am not sure if it is still worth shipping Anole or not? @ArthurZucker  WDYT?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36047). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Up to you, but if janus is easier to add we should prob add it rather than this one! Unless logic is the same! 
Sorry that you had to spend time on it 😾 

Yeah, same feeling here, isn't worth maintaining but this might help @yaswanth19 with Janus shipping

Okay I'll review when I have time then!