noamaanMulla-03/transformers-issues huggingface dataset:
nit: typing use Llama4TextConfig instead of Llama4Config
Author: kmehant
Pull request: yes

# What does this PR do?

nit fix, use text config typing instead.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

- text models: @ArthurZucker


Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.
Add weights_only=True to torch.load
Author: cyyever
Pull request: yes

# What does this PR do?
Use weights_only=True in all code.

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

Most of these conversion scripts are already excluded from release builds, but I think this is good for reducing warnings + adding security. cc @arthurzucker are you okay with it, or is there any risk of breaking things?

PT2.6 defaults weights_only to Ture.

Good point, actually - this PR only affects older versions, so it should be completely safe, and everyone using torch 2.6 is already "testing" it. Going to merge without core maintainers since it should be totally non-breaking!
Add XPU case to is_torch_bf16_gpu_available
Author: cyyever
Pull request: yes

torch.xpu also provides `is_bf16_supported` function.

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

@SunMarc Done

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37132). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

CI is green, so merging. Thank you for the PR!
[agents] remove agents 🧹 
Author: gante
Pull request: yes

# What does this PR do?

`transformers.agents` was deprecated in #36415, scheduled to be removed in `v4.51`. `v4.51` was released last weekend, so this PR removes all traces of `transformers.agents`.

Special note for the docs: I've kept a page for the `agents` and `tools`, which held a deprecation message. They now hold a reference to `smolagents`.

👉 if you were a user of `transformers.agents`: use [`smolagents`](https://github.com/huggingface/smolagents) instead, it is a greatly improved version of it!

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37368). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

With my comments above, I think you'll have everything! You can search for strings `ReactCodeAgent`, `ReactJsonAgent` and `Tool` to double check.

@aymeric-roucher good point -- removed the translations of the (old) agents docs 👍
ImportError: cannot import name 'AdamW' from 'transformers' (unknown location)
Author: rajdeep-biswas
Pull request: no

### Steps to reproduce -  
```python3
!pip install --quiet transformers==4.5.0
!pip install --quiet pytorch-lightning==1.2.7

from transformers import (
     AdamW,
     T5ForConditionalGeneration,
     T5TokenizerFast as T5Tokenizer
 )
```

Throws error -
```bash
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
/tmp/ipykernel_3065/106495151.py in <module>
---> 22 from transformers import (
     23      AdamW,
     24      T5ForConditionalGeneration,

ImportError: cannot import name 'AdamW' from 'transformers' (unknown location)

```

Comments:
Hi, I just tried this and was able to import `AdamW`, make sure you have `torch` installed.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

I am also facing this problem when running the [code](https://colab.research.google.com/drive/1IDwie_Te_2GntHay_zZ-oPvriNsacQ8d#) for pretraining wav2vec2.

edit: found the solution: use transformers==4.20.0

the version of the torch must be >=1.2.0, "AdamW" optim can be used

You should use `torch.optim.AdamW` instead of `transformers.AdamW`.  
Remove `AdamW` from the import, and replace `AdamW` with `torch.optim.AdamW`.

Example:

```python
from transformers import AdamW, get_linear_schedule_with_warmup
optimizer = AdamW(filter(lambda p: p.requires_grad, colbert.parameters()), lr=config.lr, eps=1e-8)
```

to:

```python
from transformers.optimization import get_linear_schedule_with_warmup
optimizer = torch.optim.AdamW(filter(lambda p: p.requires_grad, colbert.parameters()), lr=config.lr, eps=1e-8)
```
Add fuyu Fast Image Processor
Author: Aravind-11
Pull request: yes

# What does this PR do?

This PR introduces `FuyuImageProcessorFast`, providing a faster alternative to the original `FuyuImageProcessor` by leveraging `torchvision` for image transformations.

Key changes include:
- Implementation of `FuyuImageProcessorFast` inheriting from `BaseImageProcessorFast`.
- Adaptation of Fuyu-specific resize and padding logic using `torchvision.transforms.functional`.
- Updates to `tests/models/fuyu/test_image_processing_fuyu.py` to include the fast processor, override save/load tests, and adjust tolerances for equivalence checks. All Fuyu image processing tests now pass.
- Addition of documentation for `FuyuImageProcessorFast` and update of the main usage example to recommend `AutoProcessor` for loading.

Fixes #36978

## Before submitting
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? https://github.com/huggingface/transformers/issues/36978
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

@yonigozlan

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

Re-opened here: https://github.com/huggingface/transformers/pull/37410
Facing issue while getting model from Rag,pretrained
Author: MAHESH18TECH
Pull request: no

**Code**

`# Initialize the tokenizer and model
    model_name = "facebook/rag-token-nq"
    tokenizer = RagTokenizer.from_pretrained(model_name)
    model = RagTokenForGeneration.from_pretrained(model_name)
    
    # Initialize the retriever
    retriever = RagRetriever.from_pretrained(model_name)
    
    # Tokenization function
    def tokenize_function(examples):
        return tokenizer(
            examples['text'], 
            truncation=True, 
            padding='max_length', 
            max_length=512
        )
    
    # Tokenize the dataset
    tokenized_dataset = dataset.map(
        tokenize_function, 
        batched=True, 
        remove_columns=dataset.column_names
    )`

**error**

`HFValidationError: Repo id must be in the form 'repo_name' or 'namespace/repo_name': 'https://storage.googleapis.com/huggingface-nlp/datasets/wiki_dpr/'. Use `repo_type` argument if needed.


The above exception was the direct cause of the following exception:



OSError                                   Traceback (most recent call last)


OSError: Incorrect path_or_model_id: 'https://storage.googleapis.com/huggingface-nlp/datasets/wiki_dpr/'. Please provide either the path to a local folder or the repo_id of a model on the Hub.


During handling of the above exception, another exception occurred:



OSError                                   Traceback (most recent call last)


/usr/local/lib/python3.11/dist-packages/transformers/models/rag/retrieval_rag.py in _resolve_path(self, index_path, filename)
    122                 f"- or '{index_path}' is the correct path to a directory containing a file named {filename}.\n\n"
    123             )
--> 124             raise EnvironmentError(msg)
    125         if is_local:
    126             logger.info(f"loading file {resolved_archive_file}")



OSError: Can't load 'psgs_w100.tsv.pkl'. Make sure that:

- 'https://storage.googleapis.com/huggingface-nlp/datasets/wiki_dpr/' is a correct remote path to a directory containing a file named psgs_w100.tsv.pkl

- or 'https://storage.googleapis.com/huggingface-nlp/datasets/wiki_dpr/' is the correct path to a directory containing a file named psgs_w100.tsv.pkl.`

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Bug in LlaveNextProcessor when using do_pad=False
Author: aarbelle
Pull request: no
Labels: name: bug

### System Info

when using `do_pad=False` the variable `processed_images` is not defined and the code crashes [here](https://github.com/huggingface/transformers/blob/c0f8d055ce7a218e041e20a06946bf0baa8a7d6a/src/transformers/models/llava_next/image_processing_llava_next.py#L723C1-L724C66)

### Who can help?

_No response_

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

Just load the processor with argument `do_pad=False` and try to run.

### Expected behavior

The code should not crash and return a list of images

Comments:
Hey @aarbelle !  If you want unpadded images, you need to set `return_tensors=None`, otherwise it will fail to batch tensors

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
After tokenizers upgrade, the length of the token does not correspond to the length of the model
Author: CurtainRight
Pull request: no
Labels: name: bug

### System Info

#36532 
1. My version is 4.48.1, a relatively new version. After referring to the document and executing it, my reasoning is still abnormal and the result is the same as the original reasoning
`import torch
import transformers
from transformers import PegasusForConditionalGeneration

# 加载 Pegasus 模型
#model = PegasusForConditionalGeneration.from_pretrained('google/pegasus-large')

# 获取模型参数
params = model.state_dict()
print(params)

# 获取词嵌入层权重（Pegasus 的词嵌入层是 `model.shared.weight`）
embeddings = params['model.shared.weight']
print(embeddings)

# 计算扩展前的词嵌入统计量
pre_expansion_embeddings = embeddings[:-3, :]  # 排除最后 3 个 token
mu = torch.mean(pre_expansion_embeddings, dim=0)  # 均值
n = pre_expansion_embeddings.size()[0]  # 词嵌入数量
sigma = ((pre_expansion_embeddings - mu).T @ (pre_expansion_embeddings - mu)) / n  # 协方差矩阵

# 定义多元正态分布
dist = torch.distributions.multivariate_normal.MultivariateNormal(
    mu, covariance_matrix=1e-5 * sigma
)

# 生成新的词嵌入
new_embeddings = torch.stack(tuple((dist.sample() for _ in range(3))), dim=0)
print(new_embeddings)

# 替换词嵌入层的最后 3 个 token
embeddings[-3:, :] = new_embeddings
params['model.shared.weight'][-3:, :] = new_embeddings

# 加载更新后的参数
model.load_state_dict(params)`
3. And I found that on NPU, although the token has changed, there are no errors and the reasoning is normal. Transformers and tokenizers are the same, only the torch version is different. On NPU, it is 2.1.0

### Who can help?

hi,@Rocketknight1 
Please don't close it yet, how can I adjust and adapt to the new version.thanks

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

Please don't close it yet, how can I adjust and adapt to the new version.thanks

### Expected behavior

Please don't close it yet, how can I adjust and adapt to the new version.thanks

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
fix: (llama4) fix no_split_modules to be picked up for fsdpv1 and v2 sharding
Author: kmehant
Pull request: yes

# What does this PR do?

Currently, FSDP sharding both v1 and v2 would fail to automatically infer which modules to wrap since, the current configuring of no_split_modules for llama4 makes it inaccessible. We should essentially move the `_no_split_modules` to common base class i.e. `Llama4PreTrainedModel`.

Though we can get around this by explicitly passing the classes to wrap, for downstream users who rely on automatic wrapping inference (traditionally working good feature of transformers 😉)  would experience bad training memory costs for llama4 + FSDP and wont realize.

While using fsdp activation checkpointing, there are no workarounds.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

- text models: @ArthurZucker
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- trainer: @zach-huggingface and @SunMarc

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR




Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.
[Llama 4] `offloaded_hybrid` fails on main w/ `torch._dynamo.exc.BackendCompilerFailed`
Author: Vaibhavs10
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.52.0.dev0
- Platform: Linux-6.8.0-1024-aws-x86_64-with-glibc2.39
- Python version: 3.11.11
- Huggingface_hub version: 0.30.2
- Safetensors version: 0.5.3
- Accelerate version: 1.6.0
- Accelerate config: 	not found
- DeepSpeed version: not installed
- PyTorch version (GPU?): 2.6.0+cu124 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: yes
- Using GPU in script?: yes
- GPU type: NVIDIA H200

### Who can help?

@Cyrilvallez @ArthurZucker 

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Here's the test script: https://gist.github.com/Vaibhavs10/f7d5bcc45587daad2645870d739a73b1
Here's the test prompt: https://huggingface.co/reach-vb/random-files/blob/main/very_long_context_prompt.txt

### Expected behavior

It should work!

Comments:
As explained offline, this is an issue with torch/flex that appears when using very long contexts. There are several issues that were opened about it, see e.g. https://github.com/pytorch/pytorch/issues/145074. 
In the meantime, the easiest workaround is to use prefill chunking, i.e. set `prefill_chunk_size=chunk_size` to a reasonably small value (i.e. 16k or 32k) in `generate`, to effectively chunk the context!
Fix tests failed with gated repos.
Author: ydshieh
Pull request: yes

# What does this PR do?

Fix tests failed with gated repos by adding `require_read_token`

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37484). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Weights of BlipModel are not initialized from the model checkpoint
Author: sumitmishra209
Pull request: no
Labels: name: bug

### System Info

The code snippet is an example from https://huggingface.co/docs/transformers/model_doc/blip#transformers.BlipModel.get_text_features.example.

The warning that I get is:

Some weights of BlipTextModel were not initialized from the model checkpoint at Salesforce/blip-image-captioning-base and are newly initialized: ['embeddings.LayerNorm.bias', 'embeddings.LayerNorm.weight', 'embeddings.position_embeddings.weight', 'embeddings.word_embeddings.weight', 'encoder.layer.0.attention.output.LayerNorm.bias', 'encoder.layer.0.attention.output.LayerNorm.weight', 'encoder.layer.0.attention.output.dense.bias', 'encoder.layer.0.attention.output.dense.weight',...

The model weights seem to be newly initialised as there's some error with loading the pre-trained weights. Please guide me in solving this issue. I am making a custom model in which I need the last hidden state of output with text as input (outputs = text_encoder(**inputs))

### Who can help?

_No response_

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

from transformers import AutoProcessor, BlipModel

model = BlipModel.from_pretrained("Salesforce/blip-image-captioning-base")
processor = AutoProcessor.from_pretrained("Salesforce/blip-image-captioning-base")

inputs = processor(text=["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt")
text_features = model.get_text_features(**inputs)

### Expected behavior

BlipModel.from_pretrained("Salesforce/blip-image-captioning-base")

this shoul get inialized with weights to run model.get_text_features(**inputs)

Comments:
Hi @sumitmishra209 , it's a known issue with BlipModel, you can check it out here: https://github.com/huggingface/transformers/issues/31199 basically the structure of `BlipModel` does not match the weights here (it's text_decoder instead of text_model, for instance), but `BlipForConditionalGeneration` and the other subclasses should help. Closing this issue for know, let me know if that helps!
Remove fsspec dependency which isn't directly used by transformers
Author: cyyever
Pull request: yes

# What does this PR do?
 
As the title says.

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

cc @arthurzucker @lysandrejik do we know why this was there?

@Rocketknight1 It's used by hf/datasets.
[ci] fix doc builder
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Merging qwen omni broke CI, this PR fixes it

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page). This will assign reviewers and trigger CI.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_37489). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.