noamaanMulla-03/transformers-issues huggingface dataset:
🌐 [i18n-KO] Translated `text-to-speech.md` to Korean
Author: wonhyeongseo
Pull request: yes

<!-- PR의 제목은 "🌐 [i18n-KO] Translated `<your_file>.md` to Korean" 으로 부탁드립니다 -->
# What does this PR do?

Translated the `text-to-speech.md` file of the documentation to Korean 😄 
Thank you in advance for your review!

Part of https://github.com/huggingface/transformers/issues/20179
<!-- 메인 이슈에 기록이 남아요! 가짜연구소 리포를 사용해 연습하실때는 제거해주시면 감사하겠습니다! :smile: -->

## Before reviewing
- [ ] Check for missing / redundant translations (번역 누락/중복 검사)
- [ ] Grammar Check (맞춤법 검사)
- [ ] Review or Add new terms to glossary (용어 확인 및 추가)
- [ ] Check Inline TOC (e.g. `[[lowercased-header]]`)
- [ ] Check live-preview for gotchas (live-preview로 정상작동 확인)

## Who can review? (Initial)

<!-- 1. 위 체크가 모두 완료된 뒤에만 OSSCA 팀원들에게 리뷰 요청하는 아래 주석을 노출해주세요! -->
<!-- Team OSSCA, may you please review this PR?  -->
<!-- @wonhyeongseo, @keonju2, @harheem, @HongB1, @junejae -->

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review? (Final)

<!-- 2. 가짜연구소 팀원들과 리뷰가 끝난 후에만 허깅페이스 직원들에게 리뷰 요청하는 아래 주석을 노출해주세요! -->
<!-- May you please review this PR? -->
<!-- @sgugger, @ArthurZucker, @eunseojo -->

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_24975). All of your documentation changes will be reflected on that endpoint.

Hello, I'm curious about the reason for closing this PR. I'd like to inquire whether I can continue the translation work of this document.
안녕하세요,이 PR을 닫으신 이유가 무엇인지 궁금합니다. 제가 이 문서의 번역 작업을 이어나갈 수 있는지 여쭤보고 싶습니다.
Fix `max_steps` documentation regarding the end-of-training condition
Author: qgallouedec
Pull request: yes

# What does this PR do?

Fixes #26635 


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@muellerzr @pacman100 @stevhliu @MKhalusova


Quick explanation, the document currently states that


> max_steps (`int`, *optional*, defaults to -1):
>         If set to a positive number, the total number of training steps to perform. Overrides `num_train_epochs`.
>         In case of using a finite iterable dataset the training may stop before reaching the set number of steps
>         when all data is exhausted.

But, when you use finite iterable dataset, the training doesn't stop as expected:

```python
import torch
from datasets import Dataset
from torch import nn
from transformers import Trainer, TrainingArguments

class MyModule(nn.Module):
    def __init__(self):
        super().__init__()
        self.linear = nn.Linear(2, 2)

    def forward(self, a, return_loss=True):
        output = self.linear(a)
        return {"loss": output.sum().abs()}

data = torch.tensor([[i, i] for i in range(10)], dtype=torch.float32)  # [[0., 0.], [1., 1.], [2., 2.], ...]
dataset = Dataset.from_dict({"a": data}).to_iterable_dataset()  # finite iterable dataset

args = TrainingArguments(output_dir=".", per_device_train_batch_size=1, max_steps=20)
trainer = Trainer(model=MyModule(), args=args, train_dataset=dataset)
trainer.train()
```

It trains for 20 steps, by looping twice through the dataset twice, instead of "stopping before reaching the set number of steps when all data is exhausted".

It's because the logic in the trainer is the following:

```python
global_step = 0
for epoch in range(num_train_epochs):
    for step, inputs in enumerate(epoch_iterator):
        global_step += 1
        if global_step >= max_steps:
            break
    if global_step >= max_steps:
        break
```

Therefore, the doc should be updated.


Comments:
Enforce pin memory disabling when using cpu only
Author: qgallouedec
Pull request: yes

# What does this PR do?

Fixes #26556 

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@muellerzr

I would skip writing new test, it's really hard to test, plus it's a minor fix



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27745). All of your documentation changes will be reflected on that endpoint.
[Time series] Add patchtst
Author: kashif
Pull request: yes

# What does this PR do?

Adds the patchtst model 

re-opened from closed #25927

Comments:
Hi @amyeroberts , can you review the latest commits? We have made all the changes based on your requests. There is a deadline from our side, so we would appreciate your help to make it faster.

_The documentation is not available anymore as the PR was closed or merged._

Hi @amyeroberts, we have resolved all of your requests. Can you review the latest update? All the tests (checking the slow tests pass), the returns when return_dict=False is passed and seed parameter is removed. Thank you!

@namctin We recently updated our formatting libraries - using ruff instead of black. To make the quality checks pass, you'll need to: 
* Uninstall black: `pip uninstall black`
* Update any necessary formatting settings: `pip install -e .[quality]` 
* Re-run formatting: `make fixup` 
* Push any changes made

For the documentation tests - it seems the tests are failing because the checkpoint being pointed to doesn't exist. Is this deliberate? 

I have created some on my end, let me update the docs to double check

@amyeroberts All the tests have passed. Can you take a look and merge if there is no other requests? Thank you!

@amyeroberts We have either resolved or commented on all of your requests. Can you please review?

Thank you @amyeroberts for your careful review which makes the codes much cleaner.
Model parallelize for LlaMA
Author: pengzju
Pull request: no

The LlaMA model currently does not support model parallel training, when will this feature be added?

'LlamaForCausalLM' object has no attribute 'parallelize'

Comments:
This API is deprecated on existing models and won't be implemented on new ones. Use `from_pretrained(xxx, device_map="auto") to parallelize your model.

Hi sgugger, I wonder if  `from_pretrained(xxx, device_map="auto")` can be used to train models?

Yes it can, as long as it all fits on GPUs (like was the case with `parallelize` before).

Thanks for your reply! I've tried it and it works well. And I noticed that `device_map="auto"` can use CPU or disk offloading. But when I try to train a large model with offloading, I get the GPU OOM error.
I also tried setting a low `max_ memory` to set aside buffer for optimizer states in GPU, but still get the GPU OOM error.
Does `device_map="auto"` enable training large model with offloading?

Yes, if you have the hardware for it. To train a model, you need 4x the size in GPU memory (one for the model, one for the gradients and two for the optimizer state if using Adam). If not, then you will need to look at techniques like Zero-3 in DeepSpeed or FSDP, or use `peft` for fine-tuning.

Thanks. My point is that using `device_map="auto"` can **offload** some modules of large model to CPU or Disk, so do we still need DeepSpeed or FSDP?

As I said before, CPU and disk offload with `device_map="auto"` do not support training, only inference.

Will device_map = "auto" support pipeline parallel training?

> 

Hi，will model paralleize and zero strategy  work together when training? or any plan for that? Thanks!

No, ZeRo-3 from DeepSpeed has its own way of initalizing the model which is supported in Transformers already. You shouldn't mix it up with `device_map="auto"`.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Is there a need to change optimizer if we are using "auto"?


is this model parallel or data parallel?
Introduce SegGPT model
Author: raghavanone
Pull request: yes

Introduce SegGPT model.

#27514 

Comments:
cc @amyeroberts and @niels

Hi @raghavanone - thanks for opening this PR! Let us know when it's ready for review 

@raghavanone Hey, I also was doing the implementation of `SegGPT` to add to the library (you can find my work here https://github.com/huggingface/transformers/pull/27735) I believe my current implementation follows a bit more the library standards (@NielsRogge if you could take a look at my PR to make sure my claim is right). If you want to we can collaborate on my PR

@EduardoPach my implementations is close to completion,the test pass and conversion script is also complete.I don't think it is wise to abandon this.
Wrong image processing behavior of Fuyu
Author: SeungyounShin
Pull request: no

### System Info

At https://github.com/huggingface/transformers/blob/083e36923a19650fa264c4173db2f63ab124bb27/src/transformers/models/fuyu/processing_fuyu.py#L539

image processing should accept size variables if not it will pad image to this extent. 

<img width="1139" alt="image" src="https://github.com/huggingface/transformers/assets/20262536/695f703f-7256-4716-b4a9-1083d166e0ab">

And while fuyu is trained to accept variable resolution it should be not intended behavior I guess. 



### Who can help?

text models: @ArthurZucker and @younesbelkada
vision models: @amyeroberts

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction
```python
from transformers import FuyuProcessor, FuyuForCausalLM
from PIL import Image
import requests
import numpy as np

processor = FuyuProcessor.from_pretrained("adept/fuyu-8b", cache_dir = ".cache")
model = FuyuForCausalLM.from_pretrained("adept/fuyu-8b", cache_dir = ".cache").to("cuda:0")

# prepare inputs for the model
text_prompt = "generate coco-style prompt\n"

url1 = "https://huggingface.co/adept/fuyu-8b/resolve/main/bus.png"
image1 = Image.open(requests.get(url1, stream=True).raw)

inputs = processor(text=text_prompt, images=image, return_tensors="pt").to("cuda:0")
print(inputs["input_ids"].shape)

# autoregressively generate text
generation_output = model.generate(**inputs, max_new_tokens=128)
prompt_len = inputs["input_ids"].shape[-1]
generation_text = processor.decode(generation_output[0][prompt_len:], skip_special_tokens=True)
print(generation_text)
```
### Expected behavior

Expected to be not padded to that extent. 

Comments:
close : I believe the observed behavior is likely expected and necessary for batch processing."
uses dvclive_test mode in examples/pytorch/test_accelerate_examples.py
Author: dberenbaum
Pull request: yes

# What does this PR do?

Fixes tests in examples/pytorch/test_accelerate_examples.py to use test mode for dvclive tracking, so that it does not try to do git operations in tests/CI.

Fixes comments [here](https://github.com/huggingface/transformers/pull/27352#issuecomment-1819131456) and [here](https://github.com/huggingface/accelerate/pull/2139#discussion_r1408383251).


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

Could you take a look @muellerzr? I didn't find anywhere else that needs a change, but you likely know better whether I missed any tests.


Comments:
cc @ArthurZucker this will fix a ton of pesky test failures in the examples :)
Generate: `GenerationConfig` throws an exception when `generate` args are passed
Author: gante
Pull request: yes

# What does this PR do?

Adds an informative exception that would have prevented #27704 🤗 

Fixes #27704 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27757). All of your documentation changes will be reflected on that endpoint.

@ArthurZucker I considered checking it against the signature of `generate`, to account for future changes. However, `.validate()` is called in every `generate` call, and `inspect.signature` is slow -- highly undesirable. 

I'd like to leave it like this for now, and consider adding a more robust (=slow) method in the future if we still have problems :)
Batch QuestionAnsweringPipeline prediction with different postprocess_params (e.g. max_answer_len)
Author: KatHaruto
Pull request: no

Is it possible to change `QuestionAnsweringPipeline.postprocess` parameters such as `max_answer_len` for each question in batch prediction?

In my use case, `max_answer_len` is an important parameter for the output I want. And I want to use batch prediction to improve performance, but currently I can only set a single (pre)`postprocess_params` in one batch.

Comments:
Hey! 🤗 I don't think this is possible out of the box as it's a custom usage! Feel free to share your solution here or ask on the [forum](https://discuss.huggingface.co/)!
Add NMS utilities
Author: NielsRogge
Pull request: yes

# What does this PR do?

As the Mask R-CNN is pretty big (#25348), it's split into smaller PRs.

This PR adds the utilities for NMS (non-maximum suppression), a well-known method in computer vision to remove near-duplicate bounding boxes. 

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Fine tune decoder-only transformers in seq2seq manner
Author: YerongLi
Pull request: no

### Feature request

https://github.com/huggingface/transformers/issues/1464
This posts discuss fine-tuning GPT2, 
With GPT2/LLaMA, by default, we need to input the [prompt label] the whole sentence `model([prompt label])` in fine-tuning and caculate the CrossEntropy on the label part, and the model output the model().logits.

**Are there any ways to input the prompt only and do the fine-tuning in the seq2seq manner?** (`model(prompt)`), this way we minimize the loss of log p(y|x).

Get the feature of `model(prompt)` rather than `model([prompt label])` is the whole point.


### Motivation

seq2seq equivalence fine-tuning workflow for decoder-only transformers.

### Your contribution

I could submit PR with discussion as a guidance.

Comments:
Hey 🤗  thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discuss.huggingface.co/) instead? I'm sure the community will be of help!

Thanks!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
How to convert flax model to pytorch?
Author: GuodongFan
Pull request: no

### Feature request

run_t5_mlm_flax.py
only generate a `flax_model.msgpack`
，but i want to obtain `pytorch_model.bin`.



### Motivation

convert msgpack to bin 

### Your contribution

none

Comments:
Please use the [forums](https://discuss.huggingface.co/) for such questions as we keep issues for bugs and feature requests only.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

how to convert a .msgpack (flax/jax format) model weights to .bin (huggingface format) or .pth (pytorch format)

@jzssz please refrain from asking the same question on so many issues 😅 I answered https://github.com/huggingface/transformers/issues/26813#issuecomment-1835703292
how can I use convert_marian_to_pytorch.py to convert translation model from marian to pytorch model?
Author: xiuzhilu
Pull request: no

I train a translation model by using Marian C++ library, and I get model that model file type is npz, at same time I get model.npz.decoder.yml and model.npz.yml. I want convert the model from Marian C++ library to pytorch model. I find the convert_marian_to_pytorch.py can accomplish it from the document(https://huggingface.co/transformers/model_doc/marian.html). But I can't convert it. How can I use convert_marian_to_pytorch.py to convert it? Thanks

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

I have the same problem, I can't generate the pytorch_model.bin file, it generates model.safetensors. Anyone kind to explain how to do it? Thanks in advance and congratulations for the great work you are doing

Safetensors is the new default for model saving for security and performance reasons! You can de-activate this but we recommend sharing the models via safetensors! 

Thank you for the prompt response. Since the convert_marian_to_pytorch.py file does not make any reference to model.safetensors how can I disable it. I want to generate the pytorch_model.bin for personal use, in case of sharing I will share the safetensors model. A thousand thanks

when you do `model.save_pretrained(...., use_safe_serialization = True)` should suffice!
Tree of Thoughts Tutorial
Author: rajveer43
Pull request: yes

# What does this PR do?

Adding ToT Framework 

<!-- Remove if not applicable -->

Fixes #26726

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada, @gante


Comments:
cc @MKhalusova
Better error message for bitsandbytes import 
Author: SunMarc
Pull request: yes

# What does this PR do ?

This PR adds more context to the `ImportError` that is raised when `is_bitsandbytes_available() `is False. In this [PR](https://github.com/huggingface/transformers/pull/24995), we introduced a GPU check. Hence, `is_bitsandbytes_available() `can be `False` even though `bitsandbytes` is installed. 
Solves issue #26165
cc @NielsRogge as this happened to you too

Comments:
[MusicGen] Fix audio channel attribute
Author: sanchit-gandhi
Pull request: yes

# What does this PR do?

`audio_channels` attribute is stored in the decoder sub-model config, not the overall top-level config. 

The logits test for MusicGen mono previously failed on the daily CI tests (cc @ydshieh): https://github.com/huggingface/transformers/actions/runs/6806630933/job/18508277638
It passes when we fix this bug.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27440). All of your documentation changes will be reflected on that endpoint.