noamaanMulla-03/transformers-issues huggingface dataset:
Vision (Auto)Processor multiple images finetuning example.
Author: lovodkin93
Pull request: no
Labels: name: Examples

name: Feature request

name: Multimodal

### Feature request

Is it possible to upload an example of how to finetune PaLIGemma on multi-image inputs?
Something similar to [multi-image-inference](https://huggingface.co/docs/transformers/main/model_doc/paligemma#multi-image-inference), which shows how to perform multi-image inference over PaLIGemma.

### Motivation

enabling finetuning of multi-image PaLIGemma

### Your contribution

.

Comments:
cc @qubvel 

Hi @lovodkin93, you can probably modify this [example](https://github.com/merveenoyan/smol-vision/blob/main/Fine_tune_PaliGemma.ipynb) for multi images finetuning.
cc @merveenoyan 

@qubvel if you're not working on it I can take a stab

@merveenoyan Sure, go ahead! I'm not currently working on it, so feel free to take a stab.
Feature to configure `stop_strings` in `generation_config.json` or other config files
Author: suhara
Pull request: no
Labels: name: Feature request

name: Generation

### Feature request

The transformer library should offer a way to configure `stop_strings` and the tokenizer for it.

`model.generate()` can take a `stop_strings` argument to use custom stop tokens for generation, but a tokenizer object needs to be passed as well.

```
model.generate(...,
               stop_strings=["<stop token>"],
               tokenizer=tokenizer)
```

If we add `stop_strings` to `generation_config.json`, which can be loaded correctly [code](https://github.com/huggingface/transformers/blob/33868a057c02f0368ba63bd1edb746be38fe3d90/src/transformers/generation/configuration_utils.py#L144-L145), it will return the following error, as it requires a tokenizer object, which cannot be defined in the config file. 

```
>>> from transformers import AutoTokenizer, AutoModelForCausalLM
>>> tokenizer = AutoTokenizer.from_pretrained(model_path)
>>> model = AutoModelForCausalLM.from_pretrained(model_path, device_map="auto")
>>> model.generate(**tokenizer("Hi how are you?", return_tensors="pt", return_token_type_ids=False))

...

ValueError: There are one or more stop strings, either in the arguments to `generate` or in the model's generation config, but we could not locate a tokenizer. When generating with stop strings, you must pass the model's tokenizer to the `tokenizer` argument of `generate`.
```


### Motivation

The user shouldn't be bothered by adding extra arguments to `generate()` or `pipeline`.

For example, [nvidia/Mistral-NeMo-Minitron-8B-Instruct](https://huggingface.co/nvidia/Mistral-NeMo-Minitron-8B-Instruct) needs to use `stop_strings` but so many people simply calls `generate()` without `stop_strings` and share complaints.


### Your contribution

I'd be happy to create a PR but need guidance for the design choice. 

Comments:
cc @gante @zucchini-nlp !

Hmm, maybe we can try to autodetect the tokenizer by model class name and load it within generate, when no tokenizer is provided? Not sure how much of a hassle it can be, as mapping from auto-map is sometimes slow
Add GLM-4 and Later GLM Model (Draft)
Author: zRzRzRzRzRzRzR
Pull request: yes

This is a draft and we will continue work
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [X] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [X] Did you write any new necessary tests?

## Who can review?

@ArthurZucker



Comments:
Hi @zRzRzRzRzRzRzR ! Thanks for drafting the PR. The workflow has been failing due to the usage of TikToken.  Once the converter scripts converts tiktoken configuration to HF tokenizer configuration, you won't need to import tiktoken during inference at tokenization_glm.py 

YEP! Will review today 🤗 

> Hi @zRzRzRzRzRzRzR ! Thanks for drafting the PR. The workflow has been failing due to the usage of TikToken. Once the converter scripts converts tiktoken configuration to HF tokenizer configuration, you won't need to import tiktoken during inference at tokenization_glm.py

Fix this issue now~ Tks

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31977). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Feel free to ping me again for a review! 

BTW @zRzRzRzRzRzRzR, I took over and am currently adding the model. You can find the new PR here https://github.com/huggingface/transformers/pull/33823, should be ready pretty soon
Remove unused test_dataset
Author: thisisiron
Pull request: yes

# What does this PR do?

This PR removes unused code related to the test dataset, improving code clarity.

Fixes # (issue)

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@amyeroberts, @qubvel


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34516). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Revert "Fix Whisper CI"
Author: ydshieh
Pull request: yes

Reverts huggingface/transformers#34541

My bad, there are some torch / flax equivalent issues. Revert for now

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34605). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Remove `@slow` for `test_eager_matches_sdpa_inference`
Author: ydshieh
Pull request: yes

# What does this PR do?

And make it less flaky

- Use smaller (short) inputs
- Use larger epsilon in norm layers


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34558). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Tagging @zucchini-nlp to double-check VLM test changes :)
Changing __repr__ in torchao to show quantized Linear
Author: MekkCyber
Pull request: yes

# What does this PR do?

When a model is quantized using TorchAO and then loaded, the representation of its Linear layers is expected to be different compared to the standard representation. This pull request (PR) modifies the representation of these Linear layers to match the format used in TorchAO's implementation : https://github.com/pytorch/ao/blob/main/torchao/quantization/quant_api.py 

Before : 
`
Linear(in_features=4096, out_features=4096, bias=False)
`
After : 
```
Linear(in_features=4096, out_features=4096, weight=AffineQuantizedTensor(shape=torch.Size([4096, 4096]), block_size=(1, 128), device=cuda:0, layout_type=TensorCoreTiledLayoutType(inner_k_tiles=8), layout_tensor_dtype=torch.int32, quant_min=0, quant_max=15))
```
## Who can review?
cc @SunMarc 

Comments:
cc @SunMarc for review ! Thank you !

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34202). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

We can merge in the mean time 🤗
Fix torchvision interpolation CI
Author: yonigozlan
Pull request: yes

# What does this PR do?
Fix error with make fixup when torchvision is not present in the environment.
<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

## Who can review?

Cc @ArthurZucker
<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts, @qubvel
- speech models: @ylacombe, @eustlb
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Rocketknight1
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @SunMarc
- chat templates: @Rocketknight1

Integrations:

- deepspeed: HF Trainer/Accelerate: @muellerzr
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc @MekkCyber

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34539). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[i18n-ar] Translated file : `docs/source/ar/multilingual.md` into Arabic
Author: AhmedAlmaghz
Pull request: yes


## What does this PR do?
Translated the `docs/source/ar/multilingual.md` file of the documentation to `Arabic العربية`. I use transformers for my side-project and wanted to contribute. As a beginner, this seemed appropriate.

### Some questions:

- I can't access the pr documentation endpoint for `Arabic العربية`. Is there a missing step?
- I left all the other files alone after copying the English docs. Is this acceptable for future updates or should I add contents to the `_toctree.yml` only when translation is complete? Should I delete all other files for a cleaner review?
- In `Arabic`, people use both `Transformer` and `المحولات` to describe the model. However, as this is more of a product of HuggingFace I opted not to translate it. May you please let me know which you would prefer?

Thank you in advance for your review.

Part of #32435

### Before submitting

- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests), Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/) ?Please add a link to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

 ### Who can review?
 @stevhliu , @MKhalusova , @abodacs, may you please review this PR?
 

Comments:
Thank you @abodacs 

Done @stevhliu 
Please, merging

The following pull requests have been added to the current pull request to be merged once
- [x]  Run inference with multilingual models #33048
- [x]  Use model-specific APIs #33030
- [x]  Share a custom model #33031
- [x]  Templates for chat models #33026
- [x]  Interoperability with GGUF files #33037
- [x]  Export to TFLite #33077

@abodacs  @stevhliu 

Done 
@stevhliu 

Thank you @stevhliu 
Done

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33048). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thank you:
@abodacs 
@stevhliu
[i18n-ar] Translated file : `docs/source/ar/serialization.md` into Arabic
Author: AhmedAlmaghz
Pull request: yes


## What does this PR do?
Translated the `docs/source/ar/serialization.md` file of the documentation to `Arabic العربية`. I use transformers for my side-project and wanted to contribute. As a beginner, this seemed appropriate.

### Some questions:

- I can't access the pr documentation endpoint for `Arabic العربية`. Is there a missing step?
- I left all the other files alone after copying the English docs. Is this acceptable for future updates or should I add contents to the `_toctree.yml` only when translation is complete? Should I delete all other files for a cleaner review?
- In `Arabic`, people use both `Transformer` and `المحولات` to describe the model. However, as this is more of a product of HuggingFace I opted not to translate it. May you please let me know which you would prefer?

Thank you in advance for your review.

Part of #32435

### Before submitting

- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests), Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/) ?Please add a link to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

 ### Who can review?
 @stevhliu , @MKhalusova , @abodacs, may you please review this PR?
 

Comments:
Thank you @abodacs
[i18n-ar] Translated file : `docs/source/ar/sagemaker.md` into Arabic
Author: AhmedAlmaghz
Pull request: yes


## What does this PR do?
Translated the `docs/source/ar/sagemaker.md` file of the documentation to `Arabic العربية`. I use transformers for my side-project and wanted to contribute. As a beginner, this seemed appropriate.

### Some questions:

- I can't access the pr documentation endpoint for `Arabic العربية`. Is there a missing step?
- I left all the other files alone after copying the English docs. Is this acceptable for future updates or should I add contents to the `_toctree.yml` only when translation is complete? Should I delete all other files for a cleaner review?
- In `Arabic`, people use both `Transformer` and `المحولات` to describe the model. However, as this is more of a product of HuggingFace I opted not to translate it. May you please let me know which you would prefer?

Thank you in advance for your review.

Part of #32435

### Before submitting

- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests), Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/) ?Please add a link to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

 ### Who can review?
 @stevhliu , @MKhalusova , @abodacs, may you please review this PR?
 

Comments:
[i18n-ar] Translated file : `docs/source/ar/fast_tokenizers.md` into Arabic
Author: AhmedAlmaghz
Pull request: yes


## What does this PR do?
Translated the `docs/source/ar/fast_tokenizers.md` file of the documentation to `Arabic العربية`. I use transformers for my side-project and wanted to contribute. As a beginner, this seemed appropriate.

### Some questions:

- I can't access the pr documentation endpoint for `Arabic العربية`. Is there a missing step?
- I left all the other files alone after copying the English docs. Is this acceptable for future updates or should I add contents to the `_toctree.yml` only when translation is complete? Should I delete all other files for a cleaner review?
- In `Arabic`, people use both `Transformer` and `المحولات` to describe the model. However, as this is more of a product of HuggingFace I opted not to translate it. May you please let me know which you would prefer?

Thank you in advance for your review.

Part of #32435

### Before submitting

- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests), Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/) ?Please add a link to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

 ### Who can review?
 @stevhliu , @MKhalusova , @abodacs, may you please review this PR?
 

Comments:
من هنا نبداء القسم الجديد
 @abodacs 




# DEVELOPER GUIDES

- [x]  Use fast tokenizers from 🤗 Tokenizers #33034
- [x]  Run inference with multilingual models #33048
- [x]  Use model-specific APIs #33030
- [x]  Share a custom model #33031
- [x]  Templates for chat models #33026
- [x]  Trainer #33080
- [x]  Run training on Amazon SageMaker #33071
- [x]  Export to ONNX #33072
- [x]  Export to TFLite #33077
- [ ]  Export to TorchScript #33079
- [ ]  Benchmarks #33023
- [ ]  Notebooks with examples #33049
- [ ]  Community resources #33027
- [ ]  Troubleshoot #33017
- [x]  Interoperability with GGUF files #33037


Thank you @abodacs 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33034). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Stablelm-2-1_6b-chat config extracted from GGUF file differs from source model config
Author: Isotr0py
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.46.0
- Platform: Linux-6.1.85+-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.24.7
- Safetensors version: 0.4.5
- Accelerate version: 0.34.2
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.5.0+cu121 (False)
- Tensorflow version (GPU?): 2.17.0 (False)
- Flax version (CPU?/GPU?/TPU?): 0.8.5 (cpu)
- Jax version: 0.4.33
- JaxLib version: 0.4.33
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@SunMarc 
Also cc @VladOS95-cyber since you added GGUF support for StableLM :)

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
from transformers import AutoConfig

config_hf = AutoConfig.from_pretrained("stabilityai/stablelm-2-1_6b-chat")
config_gguf = AutoConfig.from_pretrained("Crataco/stablelm-2-1_6b-chat-imatrix-GGUF", gguf_file="stablelm-2-1_6b-chat.IQ4_XS.imx.gguf")
print(config_hf)
print(config_gguf)
```

**Outputs**
```text
StableLmConfig {
  ...
  "use_qkv_bias": true,
  "vocab_size": 100352
}

StableLmConfig {
  ...
  "use_qkv_bias": false,
  "vocab_size": 100352
}
```

### Expected behavior

The `stabilityai/stablelm-2-1_6b-chat"` model has `use_qkv_bias=True`. However, the config extracted from `stablelm-2-1_6b-chat` GGUF file has `use_qkv_bias=False`, causing model failed to initialize with qkv_proj bias.

Comments:
Hey @Isotr0py, @SunMarc! By default, use_qkv_bias is always false, because this parameter is not specified in gguf config and there is no logic to convert it somehow in https://github.com/ggerganov/llama.cpp/blob/master/convert_hf_to_gguf.py. Original model can have use_qkv_bias is true, or false as well, depending on attached config to the model. So, in this case, if you want gguf model to be exactly the same as original one, you should explicitly use config with use_qkv_bias = True, at least for now. 

@VladOS95-cyber Thanks for explanation! I think a potential solution is to check if `attn_q.bias` etc present in gguf tensors, and implement it in #34450.

But I'm afraid that this will increase the CPU overhead for GGUF config extraction. WDYT?
🌐 [i18n-KO] Translated `convbert.md` to Korean
Author: ahnjj
Pull request: yes

<!— PR의 제목은 "🌐 [i18n-KO] Translated `<your_file>.md` to Korean" 으로 부탁드립니다 —>
# What does this PR do?

Translated the `convbert.md` file of the documentation to Korean.
Thank you in advance for your review. 

Part of https://github.com/huggingface/transformers/issues/20179

## Before reviewing
- [x] Check for missing / redundant translations (번역 누락/중복 검사)
- [x] Grammar Check (맞춤법 검사)
- [x] Review or Add new terms to glossary (용어 확인 및 추가)
- [ ] Check Inline TOC (e.g. `[[lowercased-header]]`)
- [ ] Check live-preview for gotchas (live-preview로 정상작동 확인)

## Who can review? (Initial)

<!-- 1. 위 체크가 모두 완료된 뒤에만 N조 팀원들에게 리뷰 요청하는 아래 주석을 노출해주세요! 차례대로 1 ~ 5조 -->
<!-- May you please review this PR? -->
<!-- @jun048098, @yijun-lee, @mreraser, @shinhyunji36, @heuristicwave -->
<!-- @win2dvp21, @enchantee00, @fabxoe, @1kmmk1, @junejae -->
<!-- @SeungAhSon, @Jwaminju, @thsamajiki, @4N3MONE, @jungnerd -->
<!-- @mjk0618, @cjfghk5697, @ahnjj, @JinukHong, @nuatmochoi -->
<!-- @010kim, @chhaewxn, @boyunJang, @jeongiin, @harheem -->
@ahnjj, @jun048098, @Jwaminju, @jeongiin, @chhaewxn, @yijun-lee

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review? (Final)

<!-- 2. N조 팀원들과 리뷰가 끝난 후에 아래 주석을 노출해주세요! -->
@stevhliu May you please review this PR? 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34599). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.