noamaanMulla-03/transformers-issues huggingface dataset:
Fix for tokenizer.apply_chat_template with continue_final_message=True
Author: schoennenbeck
Pull request: yes

For a lot of tokenizers in `Tokenizer.apply_chat_template` with `continue_final_message=True` we get a "ValueError: substring not found" if the final message starts or ends in some whitespace. Here is some example code that exhibits the issue:

```python
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4")
messages=[
    {"role": "user", "content": "What is the capital of France?"},
    {"role": "assistant", "content": "Great question! The capital of France is "}
]
tokenizer.apply_chat_template(
    messages, add_generation_prompt=False, 
    continue_final_message=True
)
```

This is due to the fact that the `apply_chat_template`-method looks for the full final message in the rendered chat but many modern chat templates (in particular the Llama3.1-chat-template) actually trim messages before rendering.

This PR strips the final message before looking it up in the rendered string. This fixes the issue. However, this means that the continuation can now happen at a slightly different spot than the user intended. I believe this is the best way to address this issue but I would also be open to simply raise a more descriptive error in case this happens so the user can strip the last message themselves to handle this. 

Either way I think the current failure mode is not ideal.

If required I could also add a test for this behaviour.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

## Potential reviewers (based on affected area)
- tokenizers: @ArthurZucker
- chat templates: @Rocketknight1



Comments:
Alternative or possibly more robust way would be to only do the stripping if the full final message string cannot be found in the rendered chat as is.

@schoennenbeck merged! Thanks again for a clean and helpful PR!
removes decord 
Author: vrnvu
Pull request: yes

The only usage that was left of decord is removed in this PR

- Context:, project abandoned for 2 years: https://github.com/dmlc/decord
- Adds toil to local dev (https://github.com/huggingface/transformers/issues/22617)
  - https://github.com/huggingface/transformers/issues?q=is%3Aissue+decord
- Already removed from a couple of places in favor of av: https://github.com/huggingface/transformers/pulls?q=is%3Apr+decord

The helper functions `read_video_pyav` and `sample_frame_indices` are the used in all the documentation https://github.com/search?q=repo%3Ahuggingface%2Ftransformers%20read_video_pyav&type=code and `video_classification` https://github.com/huggingface/transformers/blob/main/src/transformers/pipelines/video_classification.py#L132



Comments:
Regarding the failing test: 
```python
VideoClassificationPipeline requires the PyAv library but it was not found in your environment. You can install it with:
```
pip install av
```
Please note that you may need to restart your runtime after installation.
FAILED tests/models/timesformer/test_modeling_timesformer.py::TimesformerModelTest::test_pipeline_video_classification_fp16 - ImportError: 
VideoClassificationPipeline requires the PyAv library but it was not found in your environment. You can install it with:
```
pip install av
```
Please note that you may need to restart your runtime after installation.
```
the workflow for pipelines needs a small fix!

Fixed the requirement, I wrongly confused "vision" with "video" and thought I didn't need the `av` requirement. 🤕 

How can I troubleshoot them? Are they flaky?

I dont see how they are related with this PR.

But the tests you edited are skipped by the CIs because they need av (which was not installed)

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33987). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Okay merging then! 🤗
tracker: move `prepare_inputs_for_generation` into the generation mixin 🧹 
Author: gante
Pull request: no
Labels: name: WIP

name: Generation

🧹 This is a tracker regarding the move of `prepare_inputs_for_generation` into the generation mixin 🧹 

# Why?

1. `prepare_inputs_for_generation` is not part of the core modeling, but rather a utility for `generate`
2. it should greatly reduce the need to touch modeling code, on `generate` changes. Fewer modeling changes -> improved model stability
3. greatly reduced number of lines of code 🙏 

# Tracker

Kinda ordered list of tasks:

- [x] 1. Fix related slow tests before we start — all `llama`, `generate`, and `cache_utils` [except sink cache, broken atm] slow tests should be passing to ensure we don’t break anything (#33138)
- [x] 2. `PreTrainedModel` doesn't inherit from `GenerationMixin`, so that `can_generate()` becomes independent of `prepare_inputs_for_generation` being overwritten or not (#33203)
- [x] 3. Move llama’s `prepare_inputs_for_generation` to the generation mixin. This implies moving one function that prepares the 4D mask too (the one that is called there) (#33677)
- [x] 4. Add tests for the generalist `prepare_inputs_for_generation` — currently we don’t test it directly, and we should (decoder-only llms: https://github.com/huggingface/transformers/pull/33870, encoder-decoder llms: https://github.com/huggingface/transformers/pull/34048)
- [x] 5. Address the case of `synced_gpus` in `generate`: when `synced_gpus` and `cache_positions` is out of bounds, take the latest available `input_ids` for dummy computations (https://github.com/huggingface/transformers/pull/34095)
- [x] 6. Delete `prepare_inputs_for_generation` from as many models as possible. There may be merge conflicts here, due to the 4D mask function. Try to iron out as many trivial cases as possible (decoder-only llms: https://github.com/huggingface/transformers/pull/33870, encoder-decoder llms: https://github.com/huggingface/transformers/pull/34048)
- [x] 7. Change `prepare_inputs_for_generation` to forward `**kwargs` from its input to its output. With minimal changes, this should enable most VLMs to use the shared function -- they forward `pixel_values` from the input to the output (support for **kwargs https://github.com/huggingface/transformers/pull/33870)
- [x] 8. By this point most cases of `prepare_inputs_for_generation` should have been removed 🤗  We would need to check the others individually, there may be further simplification patterns available!

Comments:
@ydshieh edit the tracker above as soon as you start working on a task, so we don't risk doing redundant work 🤗 (e.g. with the link to a draft PR)

I'll do the same!

Thanks
Generate: visit non-llm `prepare_inputs_for_generation`
Author: gante
Pull request: yes

# What does this PR do?

Closes #32685 🙌 

This PR does a final pass over the remaining `prepare_inputs_for_generation`:
- makes a few adjustments to the general function to handle trivial corner cases
- removes the cases where the general function is equivalent
- adds a comment on the functions that can't be removed, so we can quickly a) remember that there is a general function b) why the general function doesn't work on the model


👉 After this PR, let's aim at overwriting `prepare_inputs_for_generate` as few times as possible, so we can quickly roll out model-agnostic upgrades 🏎️ and minimize bugs 🐛  

Comments:
@zucchini-nlp / @ylacombe : i've tagged you both so that yoach can double-check audio models, and raushan the others 🤗 

> Are you planning to run slow tests for every models btw ?

Yes :) Now that's approved, I will be running slow tests before merging @ylacombe 

Ran slow tests on all models with significant changes, no regressions -- merging :)
Support Llama 3.2 conversion (text models)
Author: pcuenca
Pull request: yes

# What does this PR do?

Update the existing Llama conversion script to support the new 1B and 3B models.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33778). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

This is now ready for review @ArthurZucker. I exported the Llama 3.2 instruct, base and guard models and verified that the configuration files were identical to what's currently in the Hub.

I converted Llama 2 and only saw a couple of new properties from the default configuration (for example, `mlp_bias`) that look fine to me. The only interesting difference is that the converted weights are now exported in `bfloat16`, while [this repo](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf/tree/main) was exported in `float16`. I believe `bfloat16` is correct, because the [original weights](https://huggingface.co/meta-llama/Llama-2-7b-chat/tree/main) were released with that dtype (and it's consistent with newer models). Let me know if you think it's necessary to override the dtype in this case, but I think it could potentially mess things up when converting fine-tunes exported in the original format.

I found the error message for tokenizer (< 3.x) conversion to be unclear, so I added a explicit reference to requiring `sentencepiece`.

I didn't keep a local copy of the Llama 1 research weights, so I can't test that code path now. I re-requested access to download the weights from Meta, not sure how long it'll take.

No worries, LGTM and I think we can merge! 🤗
Size mismatch for state-spaces/mamba2-130m state_dict load
Author: arghadippurdue
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.45.2
- Platform: Linux-5.14.21-150400.24.116-default-x86_64-with-glibc2.31
- Python version: 3.9.20
- Huggingface_hub version: 0.25.2
- Safetensors version: 0.4.5
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.4.1 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: NO

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("state-spaces/mamba2-130m")

### Expected behavior

Traceback (most recent call last):
  File "python3.9/site-packages/transformers/models/auto/auto_factory.py", line 564, in from_pretrained
    return model_class.from_pretrained(
  File "/python3.9/site-packages/transformers/modeling_utils.py", line 4014, in from_pretrained
    ) = cls._load_pretrained_model(
  File "python3.9/site-packages/transformers/modeling_utils.py", line 4559, in _load_pretrained_model
    raise RuntimeError(f"Error(s) in loading state_dict for {model.__class__.__name__}:\n\t{error_msg}")
RuntimeError: Error(s) in loading state_dict for Mamba2Model:
        size mismatch for backbone.embeddings.weight: copying a param with shape torch.Size([50288, 768]) from checkpoint, the shape in current model is torch.Size([50277, 4096]).

Comments:
The weights are not directly compatible with transformers. You would need to convert them yourself with this script: https://github.com/huggingface/transformers/blob/main/src/transformers/models/mamba2/convert_mamba2_ssm_checkpoint_to_pytorch.py 
( I've converted the 130m model a few weeks back, see https://huggingface.co/AntonV/mamba2-130m-hf )

I have seen this issue multiple times so might be worth to create official associated repos. Originally, the implementation was meant for codestral mamba which is why there are none for the paper ones.

Wdyt? cc @molbap

Converted weights: https://huggingface.co/collections/AntonV/unofficial-mamba2-for-hf-transformers-670fb0892b26b0acd623323b
(most are fp16 since the original ones were also fp16)

Thanks @vasqu for the conversion! I think it's fine to refer people to that issue/these weights when trying to use mamba2 in transformers, yes. One way also could be to open a PR on the hub to add the hf weights directly to https://huggingface.co/state-spaces/mamba-130m, either way is fine! 
@arghadippurdue I'll close the issue then thanks to Anton, let me know if you still encounter an issue using the model.
BLIP: fix input expansion logic
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Fixes https://github.com/huggingface/transformers/issues/34223. Since BLIP expansion happens after the text has been tokenized for specific reasons, we missed this bug. 

I can add one more test for batched expansion but I don't think we need it since this won't probably break again and was a silly bug

Comments:
GGUF support for BERT architecture
Author: Dimmension
Pull request: no
Labels: name: Feature request

### Feature request

I want to add the ability to use GGUF BERT models in transformers.
Currently the library does not support this architecture. When I try to load it, I get an error TypeError: Architecture 'bert' is not supported.
I have done most of the mapping, with some fields I am having difficulty.
Can anybody help me and provide comments on this feature?
### Motivation

I ran into a problem that I can't use gguf models in RASA(rasa uses standard from_pretrained). So I decided to make BERT support

### Your contribution

That's my extended ggml.py file
```python
GGUF_TENSOR_MAPPING = {
    "bert": {
        "context_length": "max_position_embeddings",
        "block_count": "num_hidden_layers",
        "feed_forward_length": "intermediate_size",
        "embedding_length": "hidden_size",
        "attention.head_cgguf>=0.10.0ount": "num_attention_heads",
        "attention.layer_norm_rms_epsilon": "rms_norm_eps",
        # "attention.causal": "",
        # "pooling_type": "",
        "vocab_size": "vocab_size",
    }
}
 
GGUF_CONFIG_MAPPING = {
    "bert": {
        "context_length": "max_position_embeddings",
        "block_count": "num_hidden_layers",
        "feed_forward_length": "intermediate_size",
        "embedding_length": "hidden_size",
        "attention.head_cgguf>=0.10.0ount": "num_attention_heads",
        "attention.layer_norm_rms_epsilon": "rms_norm_eps",
        # "attention.causal": "",
        # "pooling_type": "",
        "vocab_size": "vocab_size",
    }
}
 
GGUF_TOKENIZER_MAPPING = {
    "tokenizer": {
        # "ggml.token_type_count": "",
        # "ggml.pre": "",
        "ggml.model": "tokenizer_type",
        "ggml.tokens": "all_special_tokens",
        "ggml.token_type": "all_special_ids",
        "ggml.unknown_token_id": "unk_token_id",
        "ggml.seperator_token_id": "sep_token_id",
        "ggml.padding_token_id": "pad_token_id",
        "ggml.cls_token_id": "cls_token_id",
        "ggml.mask_token_id": "mask_token_id",
    },
    "tokenizer_config": {       
        "ggml.unknown_token_id": "unk_token_id",
        "ggml.seperator_token_id": "sep_token_id",
        "ggml.padding_token_id": "pad_token_id",
        "ggml.cls_token_id": "cls_token_id",
        "ggml.mask_token_id": "mask_token_id",
    },
}
```

Comments:
Hi @Dimmension, there is dedicated open issue https://github.com/huggingface/transformers/issues/33260. You could take a look how other architectures and tests were added and follow the same logic. Usefull links you may need as well are: https://github.com/ggerganov/llama.cpp/blob/master/convert_hf_to_gguf.py where you can find tensors and config conversion logic, this one is for mapping: https://github.com/ggerganov/llama.cpp/blob/master/gguf-py/gguf/tensor_mapping.py, in order to correctly rename all tensors back.
Support for "add_eos_token" to "apply_chat_template" function in tokenizers
Author: aditya-29
Pull request: yes

# What does this PR do?

The chat templates in some models do not include the <EOS> token at the end. For example, in the CodeGemma instruct model, when we use the apply_chat_template function, the tokenized output does not include the <EOS> token at the end.

This omission creates a problem when we try to use apply_chat_template to format training data for instruction fine-tuning runs, as the model does not learn when to stop generating outputs due to the absence of the <EOS> token in the training data.

By adding an option to include the <EOS> token at the end of the chat template, we can save the manual effort required to add the <EOS> token ID for every example.

Fixes # (issue)
## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
- tokenizers: @ArthurZucker
- chat templates: @Rocketknight1


Comments:
Hi @aditya-29, I don't think we can merge this, I'm sorry! The reason is that **chat templates are supposed to handle these tokens internally in the template itself.**

The expected behaviour of a template is that when `add_generation_prompt=False` and `continue_last_message=False`, the template should format data for training, and add `EOS` at the end of the sequence. In other words:

- `add_generation_prompt=True`: Template should add an assistant message header, with no EOS
- `add_generation_prompt=False`: Template should format the last message correctly, including any EOS tokens, but no extra header at the end.

If the CodeGemma templates aren't doing this, then we should open issues on the Gemma templates to fix them, rather than editing `transformers`. Feel free to refer them to this comment. I may also need to update the template author documentation to make this clearer.
Image-Text-to-Text Support in Transformers Pipeline
Author: chakravarthik27
Pull request: no
Labels: name: Feature request

### Feature request

Implement the new feature to support a pipeline that can take both an image and text as inputs, and produce a text output. This would be particularly useful for multi-modal tasks such as visual question answering (VQA), image captioning, or image-based text generation.

```python
from transformers import pipeline

# Initialize the pipeline with multi-modal models
multi_modal_pipeline = pipeline("image-text-to-text", model="meta-llama/Llama-3.2-11B-Vision-Instruct")

# Example usage
messages = [
    {"role": "user", "content": [
        {"type": "image"},
        {"type": "text", "text": "If I had to write a haiku for this one, it would be: "}
    ]}
]
result = multi_modal_pipeline(messages )
print(result)  # Should return an answer or relevant text based on the image and question
```

### Motivation

- Simplifies workflows involving multi-modal data.
- Enables more complex and realistic tasks to be handled with existing Transformer models.
- Encourages more multi-modal model usage in research and production.

### Your contribution

 **Transformers Integration**
Ensure that the pipeline works well within the Hugging Face Transformers library:

- Implement the custom pipeline class (`ImageTextToTextPipeline`).
- Add support for handling different data types (image, text) and ensure smooth forward pass execution.

```python
class ImageTextToTextPipeline(Pipeline):
  ....
  
```

Comments:
Good timing ;) https://github.com/huggingface/transformers/pull/34170

I'd like to work on this feature request.
Differential Attention implementation for BERT.
Author: Chirayu-Tripathi
Pull request: no
Labels: name: Feature request

### Feature request

The paper "Differential Transformers" implements a differential attention mechanism which calculates the attention scores as the difference between two separate softmax attention maps leading to better long-context modeling and key information retrieval.

LINK : [Paper](https://arxiv.org/pdf/2410.05258)

### Motivation

Although the paper focuses on decoder only models, I think this differential attention mechanism might be helpful with encoder only models as well.

### Your contribution

I can work on this feature, if the HuggingFace team considers it as a valuable addition.

Comments:
was thinking about implementing/using it for Longformers as well