noamaanMulla-03/transformers-issues huggingface dataset:
apply class transformers.SequenceBiasLogitsProcessor on Qwen model
Author: buptspig
Pull request: no
Labels: name: Feature request

name: Generation

### Feature request

I notice that your wrote:
In order to get the token ids of the sequences that you want to bias, make sure to set add_prefix_space=True when initializing the tokenizer, and use tokenizer(bad_words, add_special_tokens=False).input_ids. The add_prefix_space argument is only supported for some slow tokenizers, as fast tokenizers’ prefixing behaviours come from pre tokenizers. Read more [here](https://huggingface.co/docs/tokenizers/api/pre-tokenizers).



So if I am using Qwen model, since its tokenizer is based on fast tokenizers, I can't use this bias logits feature?

Is there anyway I can use the Qwen model and still use this feature?

### Motivation

the old code is not working on the new model like qwen, want update.

### Your contribution

none

Comments:
cc @gante WDYT?

any idea on if this feature can get updated?

gentle ping @gante 

I wrote that a long ago :D I'm going to revisit it to see if we still have the same limitation with fast tokenizers.

@buptspig @panguagency 

the docstring was indeed outdated :) check #35699 for the updated docstring, which includes an example with a fast tokenizer (qwen)

@gante   thanks a lot, I will check now, glad that I can use this feature!

@gante   I noticed that in sequence_bias=state.sequence_bias,   if the sequence_bias have large vacabulary, for example 5000, it will getting very slow.  could it be optimized?

chatgpt o1 sugguest a couple of way to improve the speed for a custom_logits_processors. but since I am an entry level programmer so I could not implement it.

Looking forward your reply

@panguagency it can probably be improved, but we're low on bandwidth (contributions are welcome!). Currently, it has a for loop, applying one bias at a time, but the code can probably be vectorized with torch operations.
[generate] update docstring of `SequenceBiasLogitsProcessor`
Author: gante
Pull request: yes

# What does this PR do?

Fixes #35432 

The docstring of `SequenceBiasLogitsProcessor` was outdated -- the example didn't work with fast tokenizers, and fast tokenizers are now the norm. This PR updates the example and tips, especially to be used with fast tokenizers.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35699). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
version 4.47.0 provides different generation results when using quantized awq model
Author: xin3he
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.47.0
- Platform: Linux-5.4.0-169-generic-x86_64-with-glibc2.31
- Python version: 3.9.19
- Huggingface_hub version: 0.26.5
- Safetensors version: 0.4.2
- Accelerate version: 0.27.2
- Accelerate config:    not found
- PyTorch version (GPU?): 2.3.0+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>
- GPU type: NVIDIA A100-SXM4-80GB


### Who can help?

@gante @SunMarc @MekkCyber 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

autoawq_model = "casperhansen/opt-125m-awq"
prompt = "One day, the little girl"
user_model = AutoModelForCausalLM.from_pretrained(autoawq_model).to('cuda:0')
tokenizer = AutoTokenizer.from_pretrained(autoawq_model)
input_ids = tokenizer(prompt, return_tensors="pt")["input_ids"].to('cuda:0')
generate_kwargs = dict(do_sample=False, temperature=0.9, num_beams=4)
gen_ids = user_model.generate(input_ids, **generate_kwargs)
gen_text = tokenizer.batch_decode(gen_ids, skip_special_tokens=True)
target_text = ["One day, the little girl in the back of my mind will ask me if I'm a"]
assert gen_text == target_text, f"Expect: {target_text}\n but get: {gen_text}."
```

### Expected behavior
When version < 4.47.0, it works well. Version 4.47.0 provides different result
```log
Traceback (most recent call last):
  File "/data6/xinhe/fx_test/test.py", line 13, in <module>
    assert gen_text == target_text, f"Expect: {target_text}\n but get: {gen_text}."
AssertionError: Expect: ["One day, the little girl in the back of my mind will ask me if I'm a"]
 but get: ['One day, the little girl in the back of my mind will say, ??I??m so glad you??'].
```

Comments:
Thanks for the report @xin3he, can you check which commit in transformers introduced this issue ? Maybe due to attention refactor 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
transformers.image_transforms.normalize documents and checks for the wrong type for std and mean arguments
Author: CalOmnie
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.49.0.dev0
- Platform: Linux-6.8.0-51-generic-x86_64-with-glibc2.39
- Python version: 3.11.11
- Huggingface_hub version: 0.27.1
- Safetensors version: 0.5.1
- Accelerate version: 1.2.1
- Accelerate config:    not found
- PyTorch version (GPU?): 2.5.1+cu124 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: Neither
- Using GPU in script?: No
- GPU type: NVIDIA GeForce GTX 1060 6GB

### Who can help?

_No response_

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
import numpy as np
from transformers.image_transforms import normalize

mock_img = np.random.rand(3, 10, 10)

# Simulating getting the std and mean from text
std = map(float, "1.5 0. 2.".split())
mean = map(float, "1.5 0. 2.".split())
# According to the doc, this setup should work, but it does not since "map" does not implement `len`

normalize(mock_img, std, mean)
```

### Expected behavior

Use the `Sequence` type instead of Iterable to let users know that `len` needs to be implemented.

Comments:
Question: Am I following the right process to submit very small fix that do not impact the runtime (e.g. typing and documentations)? I noticed a few other mistypes while reading the code and I'd like to do it properly going forward.

Hi @CalOmnie, yes, this was good! For small fixes when the issue is clear, like bugs or obviously incorrect types, you can just submit a PR. We recommend opening an issue first when the change might be controversial or need discussion (for example, adding a whole new model class, or changing the behaviour of existing tools)
Patch moonshine
Author: eustlb
Pull request: yes

# What does this PR do?

This PR:
- fixes typos in documentation
- updates expected logits to the ones expected for T4 runners
- fixes badly handled inputs (when no `attention_mask` is provided) in generate's wrap by simply removing the generate wrap 

Justification concerning the last point: it's been discussed that handling per-batch-idx max_lengths is something that is going to be added directly to `generate` (see https://github.com/huggingface/transformers/pull/35676). In the future, `Moonshine`'s processor will return `max_length=[val_1, ... ,val_n]` that will directly be intended to be passed as `max_length` generate's parameter. For this reason, `max_length` should be computed outside generate and therefore the current wrap is not necessary! The wrap was intended to reduce hallucinations that are an edge case but impact evaluation on a test set compare to original codebase. Model cards and recommanded usage will be updated with the correct usage:

```python
from transformers import AutoProcessor, MoonshineForConditionalGeneration
from datasets import load_dataset

processor = AutoProcessor.from_pretrained("UsefulSensors/moonshine-tiny")
model = MoonshineForConditionalGeneration.from_pretrained("UsefulSensors/moonshine-tiny")

ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
inputs = processor([ds[0]["audio"]["array"], ds[0]["audio"]["array"]], return_tensors="pt")
input_values = inputs.input_values

token_limit_factor = 6.5 / processor.feature_extractor.sampling_rate  # Maximum of 6.5 tokens per second
seq_lens = inputs.attention_mask.sum(dim=-1)
max_length = int((seq_lens * token_limit_factor).max().item())

generated_ids = model.generate(input_values, max_length=max_length)
transcription = processor.batch_decode(generated_ids, skip_special_tokens=False)
print(transcription)
```

Comments:
Fix #35447 Tokenizer does not split text according to newly added input tokens
Author: jiongjiongli
Pull request: yes

# Fix Bug
#35447 : Tokenizer does not split text according to newly added input tokens

# Resolution
Method `Trie.split`: Add steps to ignore partial match that should be removed

@ArthurZucker and @itazap


Comments:
> Hey ! Sorry but I cannot seem to reproduce! ![image](https://private-user-images.githubusercontent.com/48595927/401166272-e8dbbd60-b54f-4844-9e6b-914443ab05eb.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MzY3MjAwOTAsIm5iZiI6MTczNjcxOTc5MCwicGF0aCI6Ii80ODU5NTkyNy80MDExNjYyNzItZThkYmJkNjAtYjU0Zi00ODQ0LTllNmItOTE0NDQzYWIwNWViLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTAxMTIlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUwMTEyVDIyMDk1MFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWU5NDJiNTQzMzFmN2QyNjBiMGUzMmEyNmQ5ZWY5NDdjZWRmZDdmOGU5YjdmYzdmY2ZiMmM4N2NhMGMzMzUwODcmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.N_J9MEZ-ATRtZ3twwXMA8Ls5kn-diTdQI1s4XcnqFbg)

Hello @ArthurZucker , this issue repos using `tokenizer.add_tokens(["red", "e"])` rather than `tokenizer.add_tokens(["e"])` .

Below is repro code and actual result:

```Python
# Code:
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("google-bert/bert-base-cased", use_fast=False)
tokenizer.add_tokens(["red", "e"])
output_tokens = tokenizer.tokenize("read")
print(f"Output tokens: {output_tokens}")

# Actual Output:
Output tokens: ['read']
```
Don't import torch.distributed when it's not available
Author: booxter
Pull request: yes

# What does this PR do?

This is a continuation of 217c47e31bc0cd442443e5b4a62c8bc2785d53ee but for another module. This issue was spotted in nixpkgs (again) when building lm-eval package that used a different path in transformers library to reach the same failure.

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Related: #35133

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@ArthurZucker @kwen2501 

Comments:
Unsure if the failure in CI is relevant. I am able to pass a test run with this test case in nix environment. Is there some instability in CI or I should really dig in here?
Tool calling: support more types
Author: aymeric-roucher
Pull request: yes

# What does this PR do?

Adds more supported types in chat templates.
Why not use an empty type for "any"?
=> Sometimes users want to specify a type hint like this:
```py
def get_weather(location: Any) -> None:
    return "Nice weather today."
```
=> In that case, I think it makes it more explicit to have the type hint being precised in the json schema as "any" rather than left empty as is currently the case.

I don't know if these changes are compatible with the current usage of chat template in transformers (in particular, maybe some models are trained with the convention to let the type for 'Any' blank), but they allow finer definition of inputs which is useful for using transformers chat templates in `smolagents`. Let me know if this is an appropriate change @Rocketknight1!

Comments:
`any` is not part of the [JSON Schema specification](https://json-schema.org/understanding-json-schema/reference/type), so I think this should be left unset.

However setting NoneType to `null` is the right thing to do and seems to have been an omission.

@Rocketknight1 I noticed some of your chat templates are missing the handling of `null` as well, and as I've reused your `json_to_python_type` macro I added this in f.ex. MiniCPM-4B and Dria-Agent models' templates.

Seconding @CISC's comment, I don't think we can use `{"type": "any"}`, but the type for `None` is definitely appreciated!

That's updated @Rocketknight1 ! Thanks to you and @CISC for your feedback! 🤗
add system prompt option in .apply_chat_template()
Author: ONE-THING-9
Pull request: no

### Feature request

Currently, I cannot find the option of adding a system prompt while doing tokenizer.apply_chat_template().

### Motivation

Because of this I have to avoid using apply_chat_template

### Your contribution

we can add this in {'role':'system'..........}

Comments:
Hey, I think it already supports this! see [here](https://huggingface.co/docs/transformers/chat_templating) (you just need to customize the roles) 

Got it, i missed that.

There is a big problem here.  Some of these models reject the system prompt in their chat template.  It's wildly inconsistent and there doesn't seem to be a way to tell if a model accepts a system prompt or not.  You end up getting this error if you pass in a system prompt and format it using the chat template function:

jinja2.exceptions.TemplateError: Conversation roles must alternate user/assistant/user/assistant/

I'm really surprised I haven't seen anybody raise this.  Am I missing something?  How do you tell if a model takes a system prompt or not?  

Thanks for the feedback! cc @Rocketknight1 if that's something you noticed

Hi @unoriginalscreenname - unfortunately, this is an unavoidable consequence of how these models were trained. Some models were trained with system prompts as part of the training data, and other models were not. When a model was not trained with a system prompt, it will not have any tokens that it can use to represent a system prompt, and trying to insert a prompt will confuse the model and probably significantly reduce the output quality.

In the cases when a model was trained without a system prompt, the model's chat template can be configured to raise an error if a `system` message is included in the input, and this is indeed what happens with some models (e.g. LLaMA/Mistral/Mixtral). This is correct and intended behaviour, and there isn't really any way to "fix" it without retraining the models!

The only solution I can suggest is that there is usually a different fine-tune of most models that supports a system prompt. For example, instead of Mistral-instruct you can use [Zephyr-beta](https://huggingface.co/HuggingFaceH4/zephyr-7b-beta), and instead of Mixtral-instruct you can use [Dolphin](https://huggingface.co/cognitivecomputations/dolphin-2.7-mixtral-8x7b). Both of these models were trained with system prompts, and will understand them correctly (and apply them in their chat template).

hi I am also facing a similar problem, where I am using llama 3.1 8B instruct model. I am finetuning using lora models, but there is no way to disable adding the system prompt when finetuning the model, using apply_chat_template to create the text data. I donot want to use system prompt, as I using another library for inference that doesnt need adding system prompts. Is there an easier way than writing custom chat template. Like a flag use_system_prompt=False.

`use_system_prompt=False` is also helpful when using `chat_template` multiple times, e.g. for CoT and then appending another text
Neftune computation is probably wrong with packed training
Author: ccdv-ai
Pull request: no
Labels: name: trainer

name: Feature request

name: bug

### System Info

v4.46.2

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Neftune computation is probably wrong with packed training because the scaling factor is `alpha/sqrt(lenght*d)`. The length is the packed length there:

https://github.com/huggingface/transformers/blob/a06a0d12636756352494b99b5b264ac9955bc735/src/transformers/trainer_utils.py#L126-L149

### Expected behavior

Should take into account the size of each sentence during computation.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

cc @muellerzr @SunMarc 

Thanks for the report ! Would you like to open a PR to add the support for packed training @ccdv-ai ?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

hey @Rocketknight1 ,@SunMarc ,@muellerzr I saw this open & wrote a small fix . hope you can review this and merge if issue is resolved . I'll acknowledge any comments you have on it as well if required .
Fixes, improvements to `timm` import behaviour
Author: rwightman
Pull request: yes

* Add `requires_backends` to TimmWrapperConfig so that we see a useful error message when wrapper is used without timm installed... now it is
```
ImportError: 
TimmWrapperConfig requires the timm library but it was not found in your environment. You can install it with pip:
`pip install timm`. Please note that you may need to restart your runtime after installation.
```

* Fix logic for timm dummy imports, was `not torchvision and not timm`, now `not (timm and torchvision)`

Comments:
Fix all_special_tokens to have special tokens from added_tokens
Author: ita9naiwa
Pull request: yes

# What does this PR do?

for phi-3 models,
![image](https://github.com/user-attachments/assets/0d89f2e5-d8b3-42a9-b33c-5413b72de1d0)

some important special tokens (such as template tokens `
`) are not in Tokenizer.all_special_tokens

but they re in added_vocab
<img width="1222" alt="image" src="https://github.com/user-attachments/assets/9af81b21-8af1-4d23-a8ba-706841b6d959">

 This PR adds those special tokens in added vocab into special tokens.

if I ran correctly, rust tokenizer implementation find special tokens well.

```rust
eprintln!("{:?}", self.special_tokens_set.iter().collect::<Vec<_>>());

> ["
", "
", "
", "
", "
", "
", "
", "
", "
", "
", "
", "
", "
", "
", "
", "
", "
", "<unk>", "
", "
", "
", "
", "
", "
", "
", "
", "
", "
", "
", "
", "
", "
", "
", "
", "
", "
", "
", "
", "
", "
", "
", "
", "
", "
", "
", "
", "
"]
```

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


Comments:
not sure it's intended behavior tho.

CI errors: I'm seeing unexpected errors from where I didn't touch

cc @ArthurZucker @itazap 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32688). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hi, no updates? @itazap 


Awesome, thanks @ita9naiwa ! @ArthurZucker good to merge? 🚀 

Thanks @ArthurZucker ! Something like 
```python
self._additional_special_tokens.extend(new_tokens)
```

in the `SpecialTokensMixin.add_tokens` function?

I think it should only be done for tokenizer_utils (not fast)! 

Hi, sorry I didn't get what should be done more, can you describe more for me to get it and update?

OVerall we should:
> make sure they are added to special_tokens_map_extended when we add tokens no? (can make a difference in terms of performances for models like whisper )

so instead of computing the set of special added tokens every time youcall special tokens extended, we ought add the tokens to the list, when we do  `add_token` . It's alright if we don't, but it's important to measure the performances as this can affect other applications
Fix  "test_chat_template_dict" in video LLMs
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

As per title, the test was failing when we ask to return non-tensor inputs

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35660). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.