noamaanMulla-03/transformers-issues huggingface dataset:
Adding special tokens by default
Author: MostHumble
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.47.1
- Platform: Linux-6.1.85+-x86_64-with-glibc2.35
- Python version: 3.11.11
- Huggingface_hub version: 0.27.1
- Safetensors version: 0.5.2
- Accelerate version: 1.2.1
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.5.1+cu121 (True)
- Tensorflow version (GPU?): 2.17.1 (True)
- Flax version (CPU?/GPU?/TPU?): 0.10.2 (gpu)
- Jax version: 0.4.33
- JaxLib version: 0.4.33
- Using distributed or parallel set-up in script?: False
- Using GPU in script?: False
- GPU type: Tesla T4

### Who can help?

@ArthurZucker 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
checkpoint = "HuggingFaceTB/SmolLM2-135M" # same behavior with gpt2
device = "cuda" # for GPU usage or "cpu" for CPU usage
tokenizer = AutoTokenizer.from_pretrained(checkpoint)
model = AutoModelForCausalLM.from_pretrained(checkpoint).to(device)
inputs = tokenizer("Gravity is", return_tensors="pt", return_special_tokens_mask=True)
```
outputs:
```cmd
{'input_ids': tensor([[22007,  6463,   314]]), 'attention_mask': tensor([[1, 1, 1]]), 'special_tokens_mask': tensor([[0, 0, 0]])}
```

### Expected behavior

I'd expect the tokenized to add the `
` / `bos` by default, setting `add_special_tokens=True` doesn't change the bahavior as it's already the default. Am I missing something?

Comments:
Seems like this has been a design choice [#3311](https://github.com/huggingface/transformers/issues/3311#issuecomment-696129801)
[doctest] Fixes
Author: stevhliu
Pull request: yes

Attempts to fix some of the failing doctests 😅 

Some of the ones I couldn't figure out:

- [CodeLlama](https://huggingface.co/docs/transformers/main/en/model_doc/code_llama#overview) is gated and the doctest job can't access it?
- [Moshi](https://huggingface.co/docs/transformers/main/en/model_doc/moshi#overview), not sure what the `waveform_to_token_ratio` value should be
- [SigLIP](https://huggingface.co/docs/transformers/main/en/model_doc/siglip#overview), FlashAttention2 is not installed so `attn_implementation="flash_attention_2"` is causing it to fail
- [generation_strategies.md](https://huggingface.co/docs/transformers/main/en/generation_strategies#self-speculative-decoding), the `facebook/layerskip-llama3.2-1B` checkpoint is also gated and can't be accessed
- pipeline_tutorial.md and quicktour.md, out-of-memory on the tests?
- task_summary.md, not sure how to resolve `RuntimeError('CuDNN error: CUDNN_STATUS_INTERNAL_ERROR')`


Comments:
[DOC] Fix contamination and missing paragraph in translation
Author: Yosshi999
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixed a translated document. Some paragraphs are missing, and there is a strange sentence (which may be generated by chat based LLM).

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

Documentation: @stevhliu



Comments:
Fix fast image processor warnings in object detection examples
Author: sugendran
Pull request: yes
Labels: name: Vision

name: Processing

# What does this PR do?

The `AutoImageProcessor` class warns that `use_fast` will default to True from v4.48. This PR adds an optional argument to the object detection examples to set the value of use_fast, defaulting to True.


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@qubvel


Comments:
Request to add Co-DETR
Author: daniel-bogdoll
Pull request: no
Labels: name: New model

### Model description

> A collaborative hybrid assignments training scheme, namely **Co-DETR**, learns more efficient and effective DETR-based detectors from versatile label assignment manners. This new training scheme can easily enhance the encoder’s learning ability in end-to-end detectors by training the multiple parallel auxiliary heads supervised by one-to-many label assignments such as ATSS and Faster RCNN. In addition, we conduct extra customized positive queries by extracting the positive coordinates from these auxiliary heads to improve the training efficiency of positive samples in the decoder. In inference, these auxiliary heads are discarded and thus our method introduces no additional parameters and computational cost to the original detector while requiring no hand-crafted non-maximum suppression (NMS).

Quote from their paper: https://arxiv.org/pdf/2211.12860

SotA on **Object Detection on COCO test-dev**: https://paperswithcode.com/sota/object-detection-on-coco

### Open source status

- [x] The model implementation is available
- [x] The model weights are available

### Provide useful links for the implementation

Repository & Weight Links: https://github.com/Sense-X/Co-DETR
MMDet Implementation: https://github.com/open-mmlab/mmdetection/tree/main/projects/CO-DETR

Author: Zhuofan Zong https://github.com/TempleX98

Comments:
cc @qubvel, WDYT?

Sounds like a good idea, but we might lack operations natively implemented in transformers. As far as I see from the config, the RPN head is used with ROI align, which may require a torchvision constraint or a custom kernel. Maybe it was discussed previously, @NielsRogge do you have any information?
🚨🚨🚨 image-classification pipeline single-label and multi-label prob type squashing fns (sigmoid vs softmax) are backwards
Author: rwightman
Pull request: yes
Labels: name: Core: Pipeline

name: bug

name: Vision

# What does this PR do?

Use the proper squashing/activation function to calc probabilities for image-classification pipeline.

Currently 'sigmoid' is used for models with the single-label problem type. This is not correct.

Comments:
LGTM - it does break backward compatibility but it's obviously correct, and the previous behaviour was obviously wrong.

cc @arthurzucker for confirmation before merging!
Split and clean up GGUF quantization tests
Author: Isotr0py
Pull request: yes

# What does this PR do?

- Split up GGUF tests into three subsets: quantization tests, basic integration tests and models tests.
- This PR also consolidates the quantization tests since they are using duplicate code.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@SunMarc @LysandreJik 

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts, @qubvel
- speech models: @ylacombe, @eustlb
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Rocketknight1
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @SunMarc
- chat templates: @Rocketknight1

Integrations:

- deepspeed: HF Trainer/Accelerate: @muellerzr
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc @MekkCyber

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35502). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix typing in audio_utils.chroma_filter_bank
Author: CalOmnie
Pull request: yes

# What does this PR do?

The typing for the `weighing_parameters` for the `audio_utils.chroma_filter_bank` was `Tuple[float]`, however, the actual type expected is `Tuple[float, float]`. This PR fixes that.

## Before submitting
- [X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35888). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix the config class comparison for remote code models
Author: Rocketknight1
Pull request: yes

The config class comparison previously used `str(model_class.config_class)`, which resulted in mismatches because the actual class was loaded from a cache dir and therefore had the cache dir path as part of its full class name string.

This new test just uses `model_class.__name__` which is much less error-prone.

Fixes #35584, related to #29854

Comments:
cc @ArthurZucker for review!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35592). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

gentle ping @arthurzucker, and sorry for the spam!
Close Zamba2Config code block
Author: Rocketknight1
Pull request: yes

A code block in Zamba2Config is not closed, which is causing doc building to fail.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35914). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[docs] Fix Zamba2
Author: stevhliu
Pull request: yes

Fixes an unclosed code block in Zamba2 which is causing the doc-builder to fail

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35916). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
MultiModalityCausalLM does not support Flash Attention 2.0 yet
Author: AlanPonnachan
Pull request: no

### System Info

transformers version 4.47.1
Google colab
Python 3.10.12

-------------------

I attempted to use Flash Attention with the [Janus-1.3B](https://github.com/deepseek-ai/Janus) model, but encountered the following error:

`ValueError: MultiModalityCausalLM does not support Flash Attention 2.0 yet.`

This error was raised by the transformers/modeling_utils.py file:
```
if not cls._supports_flash_attn_2:
    raise ValueError(
        f"{cls.__name__} does not support Flash Attention 2.0 yet. Please request to add support where"
        f" the model is hosted, on its model hub page: https://huggingface.co/{config._name_or_path}/discussions/new"
        " or in the Transformers GitHub repo: https://github.com/huggingface/transformers/issues/new"
    )
```

Installed FlashAttention-2 using the command:
`pip install flash-attn --no-build-isolation`

 Here is the code I used:
```
import torch
from transformers import AutoModelForCausalLM
from janus.models import MultiModalityCausalLM, VLChatProcessor
from janus.utils.io import load_pil_images

# specify the path to the model
model_path = "deepseek-ai/Janus-1.3B"
vl_chat_processor: VLChatProcessor = VLChatProcessor.from_pretrained(model_path)
tokenizer = vl_chat_processor.tokenizer

vl_gpt: MultiModalityCausalLM = AutoModelForCausalLM.from_pretrained(
    model_path, trust_remote_code=True,  attn_implementation="flash_attention_2"
)
vl_gpt = vl_gpt.to(torch.bfloat16).cuda().eval()

conversation = [
    {
        "role": "User",
        "content": "<image_placeholder>\nConvert the formula into latex code.",
        "images": ["images/equation.png"],
    },
    {"role": "Assistant", "content": ""},
]

# load images and prepare for inputs
pil_images = load_pil_images(conversation)
prepare_inputs = vl_chat_processor(
    conversations=conversation, images=pil_images, force_batchify=True
).to(vl_gpt.device)

# # run image encoder to get the image embeddings
inputs_embeds = vl_gpt.prepare_inputs_embeds(**prepare_inputs)

# # run the model to get the response
outputs = vl_gpt.language_model.generate(
    inputs_embeds=inputs_embeds,
    attention_mask=prepare_inputs.attention_mask,
    pad_token_id=tokenizer.eos_token_id,
    bos_token_id=tokenizer.bos_token_id,
    eos_token_id=tokenizer.eos_token_id,
    max_new_tokens=512,
    do_sample=False,
    use_cache=True,
)

answer = tokenizer.decode(outputs[0].cpu().tolist(), skip_special_tokens=True)
print(f"{prepare_inputs['sft_format'][0]}", answer)
```





Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Remove _supports_static_cache = True for some model classes
Author: ydshieh
Pull request: yes

# What does this PR do?

Remove `_supports_static_cache = True` for some model classes. See the comments in changes.

They were True before because it is set simply we can use static cache without torch.compile. But after #34247, static is kind tied to `torch.compile` and we should say it works if it works with torch.compile

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34975). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

kindly ping @ArthurZucker 

run-slow: idefics

This comment contains run-slow, running the specified jobs: ['models/idefics'] ...

Failing tests are irrelevant to this PR and already failing on `main`. Merge PR now.
Add default TP plan for all models with backend support
Author: Cyrilvallez
Pull request: yes

# What does this PR do?

As per the title!


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35870). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Let's test #35844
Use rocm6.2 for AMD images
Author: ivarflakstad
Pull request: yes

Use rocm6.2 base image and pytorch wheels as rocm6.3 only has nightly pytorch wheels atm.

I specify `6.2.4` as the base image because it is the latest of the `6.2.*` alternatives.
Normally I would have just gone for `6.2` and assume that I get the most up-to-date patch version, but I noticed it both has a different hash and size than `6.2.4` and it is about a month older.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35930). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix: loading DBRX back from saved path
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Fixes https://github.com/huggingface/trl/issues/2574 and adds a test for that. Plus fixes one edge case when the composite model is loaded with `torch_dtype=torch.float16` (i.e. the dtype is not string or dict) by adding one small condition


Comments:
Added a comment in existing test that the tiny model dtype is fp32, and we test by loading it with full and half precisions

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35728). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Decompose chat template docs
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

As peer title, this PR decomposes docs on chat templates into separate pages for each topic, because the chat template page was getting too big and I also wanted to add multimodal usage examples making it even worse.

The multimodal template file is a new file, other files are copied from the original `chat_template.md`

Comments:
cc @stevhliu for docs layout!

One possible issue: Maybe the "basics" doc should keep the old `chat_templating.md` name, so that old links / Google searches etc. still find it?

Super nice! ❤️ 

I have a related ongoing PR [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31757/en/conversations) that also decomposes the chat template docs. Let me think a bit on how best to incorporate your additions :)

If it's ok with you, I can add these docs to my PR (should be ready for review soon) or if its a bit more urgent then we can merge this now as well! Let me know what you think :)

@stevhliu if your PR is merged soon, sure we can move multimodal template docs there. I will also be adding the audio modality soon to the docs, but should be oke as is for now
Skip failing QwenVL test for now
Author: Cyrilvallez
Pull request: yes

# What does this PR do?

As per the title. Right now it breaks all the CIs.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35933). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Qwen-2-5-VL: fix CI
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

As per title


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35935). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Llama3.2: Allow batch to have
Author: maximilianmordig
Pull request: no
Labels: name: Feature request

name: VLM

### Feature request

Currently, llama3.2 requires either no images per batch or each example have at least one image. Is there some easy workaround (apart from feeding dummy images which is computationally expensive) to allow some examples to have images and other examples to have no images?

Current error message:
`"If a batch of text is provided, there should be either no images or at least one image per sample"`

from https://github.com/huggingface/transformers/blob/3f860dba553d09a8eb96fded8d940c98a9a86854/src/transformers/models/mllama/processing_mllama.py#L303

### Motivation

Not all examples may have images, but some do.

### Your contribution

Could possibly submit a PR

Comments:
Hey @maximilianmordig !

While it is possible to allow images only in some of the text samples for the processing, that will hurt model performance. The model chooses either to do cross-attn or not depending on image presence, and if we have text-only inputs they are not expected to do any cross attention. AFAIK that was the reason behind it
Fix TP initialization
Author: Cyrilvallez
Pull request: yes

# What does this PR do?

As per the title. When TP was introduced in https://github.com/huggingface/transformers/pull/34184, it used a context manager `with torch.device` to set the model device. However, the context manager actually does NOT put the tensor on the device explicitly (it simply changes the default device if using name without indices, such as "cuda" or .cuda()). This means that the model was still on CPU.

Then `parallelize_module` would automatically switch the model, however it uses the SAME gpu for all child processes. So it's fine for small models, but will OOM for larger ones (e.g. Llama 8B on 4x L4 GPU (24 GB) would OOM, even though it's supposed to take ~16 GB only).

It does not look like we can instantiate only the model shards from the CPU to the different devices to avoid loading the full model on each device at the start.
To avoid that, I believe we could instantiate model on `meta` device, then recursively loading and dispatching each layer, which would avoif loading the full model in RAM then dispatching which goes against parallelization idea. Will try to explore it/see with accelerate team how we can do it.


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35860). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.