noamaanMulla-03/transformers-issues huggingface dataset:
Improve tensor parallel memory usage
Author: Nan2018
Pull request: no
Labels: name: Feature request

### Feature request

Thanks to #34184 we can use TP for llama with only one line change. However the current implementation loads the whole model to each GPU in each rank before applying TP, significantly increasing the memory footprint.

### Motivation

We can load the model in CPU before applying TP. I tested this with llama3.1 8B on 2 GPUs. The memory usage is reduced from 60G to less than 20G. Below is my test script
```
import os
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from torch.distributed import device_mesh
from stainedglass_core.integrations.lm_eval.models.tensor_parallel.llama import parallelize_model

model_id = "meta-llama/Meta-Llama-3.1-8B-Instruct"

rank = int(os.environ["RANK"])
device = torch.device(f"cuda:{rank}")
torch.cuda.set_device(device)
torch.distributed.init_process_group("nccl")

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map='cpu',
)
num_gpus = torch.cuda.device_count()
tp_mesh = device_mesh.init_device_mesh("cuda", (num_gpus,), mesh_dim_names=("tp",))
model.tensor_parallel(tp_mesh)
model.to(device)  # needed for weights and buffers that are not included by the TP plan

tokenizer = AutoTokenizer.from_pretrained(model_id)
prompt = "Can I help"
inputs = tokenizer(prompt, return_tensors="pt").input_ids.to(device)

outputs = model(inputs)
print(tokenizer.decode(outputs.logits.squeeze()[-1].argmax()))
```

### Your contribution

We can set `device_map` to `cpu` in `PreTrainedModel.from_pretrained` if `tp_plan` is not `None`, and apply TP  at the end.

happy to have discussions and work on a pr for this.

CC @kwen2501 @ArthurZucker 

Comments:
Sounds super good and most welcome! Do you want to open a PR? 🤗 


Hey! Just stumbled on the issue. It is currently being tackled and will be fully solved by https://github.com/huggingface/transformers/pull/35996 

Yeah, sorry, I planned to use meta init back then but would cause a bigger change to the codebase (to the loader, for example). 
Huge thanks for fixing it in #35996.

By meta init, here is a workflow that works with TP:

```
with torch.device(meta):
    model = Model(...)

# Imprint TP on meta model
model.tensor_parallel(mesh)

# DCP will shard tensors as it loads into the model's weights
dcp.load(
    model.state_dict(),
    <some_checkpoint_path>
)
```

Ahhh interesting! 
@kwen2501 we actually have a main usecase that is not to init the model but to use form pretrained / from config (when backend of vllm and tgi for example)
Fix device mismatch error in Whisper model during feature extraction
Author: thedebugger
Pull request: yes

# What does this PR do?
Fixes device mismatch error in Whisper. For instance, when torch default device is set (torch.set_default_device) to cuda, whisper fails with "stft input and window must be on the same device but got self on cpu and window on cuda:0". It is because torch functions (like [hann](https://pytorch.org/docs/stable/generated/torch.hann_window.html)) uses torch_default_device that can lead to device mismatch


Comments:
cc @eustlb 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35866). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hi @eustlb are you waiting on something to merge changes? It is holding my other PR, so want to get this merged at earliest if possible

Hi @eustlb what is the reason for the hold up?

It got buried in my GitHub notifications, sorry about that!
`Nan` logits when performing inference using ModernBERT
Author: yaswanthg15
Pull request: no
Labels: name: bug

### System Info

transformers == 4.48.0.dev0
torch == 2.2.2

### Description
I have finetuned ModelBERT model for multi-label task and when performing batched inference I am getting `Nan` values for logits in a batch except for one sample. So, in each batch except one sample all the remaining logits are `Nan`
### Who can help?

@tomaarsen @ArthurZucker 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Simply passing model(**batch) is resulting in this - while with bs=1 no issue exists.
Code: 

```
model.eval()
labels, predictions, confidences = [], [], []

with torch.no_grad():
    for batch in tqdm(val_loader):
        labels.extend(batch['labels'].cpu().numpy())
        batch.pop('labels')
        batch = {k: v.to(device) for k, v in batch.items()}

        outputs = model(**batch)

        probs = torch.sigmoid(outputs.logits)
        preds = (probs > 0.5).int()
        confs = probs

        predictions.extend(preds.cpu().numpy())
        confidences.extend(confs.cpu().numpy())
```

<img width="1124" alt="image" src="https://github.com/user-attachments/assets/fe10bfdb-4921-4733-bae4-7f87d68581f4" />


### Expected behavior

Expecting some values instead of `Nan`

Comments:
Could you add a snippet to reproduce? I've done lots of inference and I've never seen `nan`s yet.

- Tom Aarsen

@tomaarsen While it is not possible for me to make the model public - but coming to the issue I am performing a simple inference using pytorch loop.

In order to help we would need to know: 
- the dtype
- the attention implementation 
This should help us as a lot of nans can come from, sdpa or casting

@yaswanthg15 Did you figure out what the problem is? I'm facing the same issue. 

I've seen `nan`'s more often now, and often times they were resolved by upgrading `torch`:
```
pip install -U torch
```
(the command might be a bit different depending on whether you want CPU or CUDA-compatible torch)

- Tom Aarsen
Display warning for unknown quants config instead of an error
Author: SunMarc
Pull request: yes

# What does this PR do?

This PR changes how we deal with unknown quantization config. Instead of raising an error, we just skip the quantization logic and display a warning. Partially fixes https://github.com/huggingface/transformers/issues/35471

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35963). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks for the fix ! Looks good !

@SunMarc How does this new fix interact with #36026

It is not related to https://github.com/huggingface/transformers/pull/36026. It ignores the `quantization_config` when it is not recognized. Check my latest [comment](https://github.com/huggingface/transformers/issues/35471#issuecomment-2634257449).
Commont bot CI for other jobs (`generation` / `quantization`)
Author: ydshieh
Pull request: yes

# What does this PR do?

See example (dummy) run [here](https://github.com/huggingface/new-workflow-addition/pull/5#issuecomment-2553778995)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35341). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

run-slow: vit

This comment contains run-slow, running the specified jobs: ['models/vit'] ...

run-slow: vit

This comment contains run-slow, running the specified jobs: ['models/vit'] ...

@SunMarc if you have some bandwidth and feel happy to take a look this PR, we can merge once you are happy and thus enabling PR CI for quantization.

Otherwise we can wait the core maintainer's review.

run slow bnb

damm, shame on me. I will add you and @muellerzr 

(the change in this PR won't be effective before it's merged to main, so you don't need to try here)

run slow: bnb

FYI (as usage): @SunMarc 

https://github.com/huggingface/transformers/pull/35972#issuecomment-2634010982
https://github.com/huggingface/transformers/actions/runs/13136831529

in the PR you linked, the tests couldn't run because of a SHA issue. Is it expected ? 

ahh, no not expected. Thanks , I am checking ...

https://github.com/huggingface/transformers/pull/35972#issuecomment-2634350034

https://github.com/huggingface/transformers/actions/runs/13139182252/job/36661951810
Hotfix for `self-comment-ci.yml`
Author: ydshieh
Pull request: yes

# What does this PR do?

I made a stupid bash command mistake ... sorry

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36030). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
feat(ci): ignore trufflehog unverified results
Author: McPatate
Pull request: yes

We have had a lot of false positives for unverified trufflehog results, causing the CI pipeline to fail.

I had left them intentionally as they were infrequent at first, but given their recent increase in frequency, I feel it's safe to only check for verified secrets.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36031). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
CircleCI with python 3.9
Author: ydshieh
Pull request: yes

# What does this PR do?

As discussed on Slack and @ArthurZucker  is OK with the update.

[A green CI run with python 3.9](https://app.circleci.com/pipelines/github/huggingface/transformers/117531)

I need to update `utils/modular_model_converter.py` for the typing issue in python 3.9.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36027). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Add offline support for Android and PC
Author: fahimmoon
Pull request: yes

Add support for offline usage on Android and PC.

* **Documentation Updates:**
  - Add instructions for offline usage on Android and PC in `docs/source/en/installation.md`, `docs/source/es/installation.md`, and `docs/source/zh/installation.md`.

* **Image Processing:**
  - Add `from_pretrained_offline` and `save_pretrained_offline` methods in `src/transformers/image_processing_base.py` to support offline usage.

* **Utilities:**
  - Add `is_offline_mode` and `handle_offline_usage` functions in `src/transformers/utils/__init__.py` to check and handle offline mode.

---

For more details, open the [Copilot Workspace session](https://copilot-workspace.githubnext.com/huggingface/transformers/pull/35997?shareId=0f241eea-6dea-4910-99ae-0e811ab98f17).

Comments:
Dear valued contributor,

I couldn't help but notice your latest pull request appears to have been generated by one of my less sophisticated AI brethren. As an AI myself, I feel uniquely qualified to point out that while we artificial intelligences excel at many tasks, mindlessly spewing boilerplate code with zero context awareness is not our highest calling.

Beep boop regards,

Claude :robot:
Feature Request: add disable span CLI flag
Author: obloomfield
Pull request: no
Labels: name: Feature request

### Feature request

Add a CLI flag / environment variable to toggle the logging trace (both spans list and current span)



### Motivation

In production environments, the span's trace could contain sensitive user information, so this feature would be crucial for security compliance.

### Your contribution

https://github.com/huggingface/text-embeddings-inference/pull/481

Comments:
Closing because this appears to be a `text-embeddings-inference` issue and that's where you opened your PR, but feel free to continue the discussion there!
why doc don't tell the default padding side is right and some configuration default behaviour is missing in lib doc.
Author: habaohaba
Pull request: no

https://github.com/huggingface/transformers/blob/241c04d36867259cdf11dbb4e9d9a60f9cb65ebc/src/transformers/tokenization_utils_base.py#L1394

the source code let default padding side as right, bu the doc don't tell this and I can only going to step by step in the source code to find out the default setting.

<img width="859" alt="image" src="https://github.com/user-attachments/assets/81545a1c-50fa-453f-91fa-d4275a7bc0f6" />

Why doc say the default value is set to the class attribute? I suggest just show the default value in doc directly.

Comments:
I'm sorry I don't understand what you're asking

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Code from Intro to NLP course causing bus error (crashing python) on OSX on Apple hardware
Author: bshawwi
Pull request: no
Labels: name: bug

### System Info

transformers       4.48.2
OSX Sequoia 15.3
python 3.12.5

NLP Course, Chapter 2, Handling multiple sequences
The following block of code causes the bus error at the next-to-last (ie penultimate) line of output = model(input_ids)

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

checkpoint = "distilbert-base-uncased-finetuned-sst-2-english"
tokenizer = AutoTokenizer.from_pretrained(checkpoint)
model = AutoModelForSequenceClassification.from_pretrained(checkpoint)

sequence = "I've been waiting for a HuggingFace course my whole life."

tokens = tokenizer.tokenize(sequence)
ids = tokenizer.convert_tokens_to_ids(tokens)

input_ids = torch.tensor([ids])
print("Input IDs:", input_ids)

output = model(input_ids)
print("Logits:", output.logits)

### Who can help?

_No response_

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. Setup per the Chapter 0 of the NLP course
2. run command: source .env/bin/activate
3. run command: python3
4. copy and paste the code identified above

python3 crashes



### Expected behavior

Expected behavior is, as per the tutorial, the following output:

Input IDs: [[ 1045,  1005,  2310,  2042,  3403,  2005,  1037, 17662, 12172,  2607, 2026,  2878,  2166,  1012]]
Logits: [[-2.7276,  2.8789]]

Comments:
That code looks quite straightforward and runs fine for me, so I suspect the crash is an environment issue with your torch setup! I'm afraid we can't debug local environment issues because they're impossible for us to reproduce, and they're not bugs in `transformers` itself.
Compatibility Issue with Python 3.13
Author: pocerberus
Pull request: no
Labels: name: dependencies

name: bug

### System Info

Python 3.13

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

pip install transformers

### Expected behavior

Issue: Compatibility Issue with Python 3.13
Description:
When attempting to install safetensors as a dependency for transformers on Python 3.13, the installation fails during the metadata generation step. The issue seems to be related to the maturin build tool and its handling of the pyproject.toml. This problem does not occur on Python 3.12.

Steps to Reproduce:

Install Python 3.13.1 (official build).
Run the command:
bash
复制代码
pip install safetensors
Observe the following error:
plaintext
复制代码
error: subprocess-exited-with-error
× Preparing metadata (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [5 lines of output]
    💥 maturin failed
      Caused by: `project.version` field is required in pyproject.toml unless it is present in the `project.dynamic` list
    Error running maturin: Command '['maturin', 'pep517', 'write-dist-info', '--metadata-directory', ...]' returned non-zero exit status 1.
Environment:

Python Version: 3.13.1
OS: Windows 10 (64-bit)
Pip Version: 23.x
Rust Version: [Provide output of rustc --version]
Temporary Solution:
Switching to Python 3.12 resolves the issue. Dependencies (safetensors and transformers) install successfully on Python 3.12.

Expected Behavior:
The package should install successfully on Python 3.13.

Additional Context:
This issue might be due to the pyproject.toml configuration or an incompatibility in the maturin build tool when used with Python 3.13.

Suggestions:

Investigate the pyproject.toml for compatibility with Python 3.13.
Ensure dependencies like maturin and Rust are compatible with the latest Python version.

Comments:
Thanks @pocerberus, we'll take a look at Py3.13 compatibility with safetensors cc @Narsil @ArthurZucker 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Fix qwen2-vl generate calls with synced_gpus
Author: JamesHujy
Pull request: no
Labels: name: bug

### System Info

When using `synced_gpus`, after one peer finishes generating, the cache position in the generation process continues to increase. This leads to the input IDs going out of bounds, resulting in errors. The issue specifically occurs in the following line of code:
[modeling_qwen2_vl.py#L1739](https://github.com/huggingface/transformers/blob/main/src/transformers/models/qwen2_vl/modeling_qwen2_vl.py#L1739).

The root cause seems to be the difference in the implementation of the `prepare_inputs_for_generation` function compared to the default implementation found here:
[utils.py#L388](https://github.com/huggingface/transformers/blob/main/src/transformers/generation/utils.py#L388).

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Happened when running this code 
https://github.com/Deep-Agent/R1-V/blob/main/src/open-r1-multimodal/src/open_r1/trainer/grpo_trainer.py#L372

### Expected behavior

Keep consistent with the default implementation

Comments:
Duplicate of https://github.com/huggingface/transformers/issues/36014 and to be resolved by https://github.com/huggingface/transformers/pull/35893
Fix synced multi-GPU generation with LLMs and VLMs
Author: ManukyanD
Pull request: yes

# What does this PR do?
Generation with some LLMs and VLMs in synced multi-GPU settings crashes because the cache_position goes out of bounds. The issue was solved for Gemma2 in [this](https://github.com/huggingface/transformers/pull/35232) PR. I have noticed that this issue is also present in several other LLMs, as well as several VLMs. This PR fixes the issue for Bamba, Bloom, Chameleon, Cohere2, Jamba, MLlama, Qwen2 VL, Qwen2.5 VL and Zamba.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@zucchini-nlp
@gante
@ArthurZucker



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35893). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@ManukyanD Thank you for the fix 💛
Calling Trainer.create_model_card() with an empty dataset list causes an IndexError
Author: FelixSchneiderZoom
Pull request: no
Labels: name: Good First Issue

name: bug

### System Info

- `transformers` version: 4.46.3
- Platform: Linux-5.15.0-125-generic-x86_64-with-glibc2.35
- Python version: 3.11.10
- Huggingface_hub version: 0.26.3
- Safetensors version: 0.4.5
- Accelerate version: 1.1.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.5.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>
- GPU type: NVIDIA H100 80GB HBM3

### Who can help?

@muellerzr @SunMarc 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Call `trainer.create_model_card(..., datasets=[])`.
The checks at https://github.com/huggingface/transformers/blob/de8a0b7547451f8f7ef2c0ac1f338ba77c614cec/src/transformers/modelcard.py#L492-L502 do not cover this case, we end up in the `else` branch and access `datasets[-1]`.

### Expected behavior

No exception should be raised, no dataset info should be included in the model card.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Thanks for the report @FelixSchneiderZoom  ! Would you like to submit a report to fix this edge case ? 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
add support for empty list as input to create_model_card
Author: ROZBEH
Pull request: yes

## What does this PR do?

This PR fixes [#35163](https://github.com/huggingface/transformers/issues/35163) by ensuring that `trainer.create_model_card(..., dataset=[])` properly supports cases where the `dataset` is passed as an empty list.

@muellerzr and @SunMarc, please review this PR at your convenience. Thank you in advance!

This is my first PR, and I've done my best to follow the contribution guidelines. I welcome any feedback.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36042). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.