noamaanMulla-03/transformers-issues huggingface dataset:
Make inheritance consistent for classes having a `generate` method
Author: fxmarty
Pull request: no

### Feature request

Hi, I create this issue to see if this may be blocking to other people or not.

`GenerationMixin` does not inherit from `nn.Module`, while `WhisperForConditionalGeneration` does. Both now have a `generate` method.

This issue is to highlight that PRs as https://github.com/huggingface/transformers/pull/21252 may well break the workflow of users that expect `generate` to be defined in `GenerationMixin`, not inheriting from `nn.Module`. Such changes can either result in silent errors, or errors due to the unexpected inheritance.

For example, https://github.com/huggingface/transformers/pull/21252 make it close to impossible to have a `TensorRTModelForSpeechSeq2Seq(GenerationMixin)` class that does not inherit from nn.Module, use transformers generate, and is able to handle several architectures. Which was something possible before.

I don't have a better solution to propose right now, as I understand different models needing different `generate` will be a more common need in the future.

### Motivation

There is no reason to inherit from `nn.Module` to use `generate`.

### Your contribution

/

Comments:
cc @gante @ArthurZucker 

As seen in Slack, let's see if there is interest from others before acting on it, especially as the `generate` method for other modalities than text is prone to evolve to support other use-cases.

If anyone stumbles upon this issue as they're blocked by the above, please comment below to let us know. Thanks!

Thank you for raising the issue @fxmarty! 

I think this is an example of where improving the modularity on `.generate()` could benefit non-standard use cases. In particular for this issue, some models rewrite `.generate()` in the model class itself (`Whisper`, `BLIP`, `RAG`, ...) -- it could be avoided if we had some option to add pre- and post-processing steps to `.generate()`. I have an idea in the back of my mind, but I haven't put it into words.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
fsmt Tokenizer.save_vocabulary Bug
Author: lihaoxin2020
Pull request: no

### System Info

- `transformers` version: 4.26.1
- Platform: Linux-4.18.0-348.23.1.el8_5.x86_64-x86_64-with-glibc2.17
- Python version: 3.8.16
- Huggingface_hub version: 0.12.1
- PyTorch version (GPU?): 1.12.1+cu116 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed

### Who can help?

@ArthurZucker @younesbelkada 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I'm using FSMT and try to reproduce the behavior of basic transformer on WMT16 translation task with [run_translation.py](https://github.com/huggingface/transformers/tree/main/examples/pytorch/translation). It turns out that during the training, both valid loss and training loss go down normally, and valid BLEU scores increase gradually at the beginning, but suddenly dropped a lot and corrupt to ~0. After some investigation, I found the fsmt_tokenizer keeps complaining each time I try to save the training states. 

You should be able to reproduce the complaint as easy as running the short script below: 

```
from transformers import (
    AutoModelForSeq2SeqLM,
    AutoTokenizer
    )

tokenizer = AutoTokenizer.from_pretrained("allenai/wmt16-en-de-12-1")

tokenizer.save_vocabulary("./tmp")
```

### Expected behavior

```
Saving vocabulary to ./tmp/merges.txt: BPE merge indices are not consecutive. Please check that the tokenizer is not corrupted!
```
And the training curves valid BLEU looks like [this](https://api.wandb.ai/links/alanlee/2zddajjr) and valid loss looks like [this](https://api.wandb.ai/links/alanlee/h77vb3bf), which show that there should be no gradient explosion but still broken.

Comments:
Hey, apparently, it is a known issue that the vocabulary has some holes. The problem with your training is probably not related as you mention that it `suddendly` drops and corrupts. Meaning up until some point everything works well no? 😉 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
🌐 [i18n-KO] Translated `tasks/zero_shot_image_classification.mdx` to Korean
Author: HanNayeoniee
Pull request: yes

<!-- PR의 제목은 "🌐 [i18n-KO] Translated `<your_file>.mdx` to Korean" 으로 부탁드립니다 -->
# What does this PR do?

Translated the `tasks/zero_shot_image_classification.mdx` file of the documentation to Korean.
Thank you in advance for your review. 

Part of https://github.com/huggingface/transformers/issues/20179
<!-- 메인 이슈에 기록이 남아요! 가짜연구소 리포를 사용해 연습하실때는 제거해주시면 감사하겠습니다! :smile: -->

## Before reviewing
- [x] Check for missing / redundant translations (번역 누락/중복 검사)
- [x] Grammar Check (맞춤법 검사)
- [x] Review or Add new terms to glossary (용어 확인 및 추가)
- [x] Check Inline TOC (e.g. `[[lowercased-header]]`)
- [x] Check live-preview for gotchas (live-preview로 정상작동 확인)

## Who can review? (Initial)
<!-- 1. 위 체크가 모두 완료된 뒤에만 가짜연구소 팀원들에게 리뷰 요청하는 아래 주석을 노출해주세요! -->
Team PseudoLab, may you please review this PR? @0525hhgus, @KIHOON71, @sim-so, @gabrielwithappy, @HanNayeoniee, @wonhyeongseo, @jungnerd

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review? (Final)
<!-- 2. 가짜연구소 팀원들과 리뷰가 끝난 후에만 허깅페이스 직원들에게 리뷰 요청하는 아래 주석을 노출해주세요! -->
@sgugger, @ArthurZucker, @eunseojo May you please review this PR?

Comments:
_The documentation is not available anymore as the PR was closed or merged._

LGTM! :-)
docs: ko: fix: update `_toctree.yml`
Author: wonhyeongseo
Pull request: yes

# What does this PR do?

Resolve conflicts raised by a recent `_toctree.yml` change (#23049)

* Edited some titles to match the new coherent style.
* Moved sections to match the English table of contents.
* As all of the two removed files were yet to be translated, no work was needed.

Comments:
Closing in favor of #23112 

Checking if same conflicts emerge with these changes as colleague's branch is experiencing.

_The documentation is not available anymore as the PR was closed or merged._

Closing again in favor of #23112 (Checks are passing)

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_23113). All of your documentation changes will be reflected on that endpoint.
gen_kwargs in Seq2SeqTrainer
Author: TJSun009
Pull request: no

https://github.com/huggingface/transformers/blob/b8648290d2d97e7c7dbccd2d4a6a4f44e70d3b63/src/transformers/trainer_seq2seq.py#L257

Hi,

I'm trying to use the Seq2SeqTrainer with generation in the evaluation_loop and it looks like the config isn't being properly passed to the prediction step. I'm having to manually set self._gen_kwargs as this is not initialised anywhere else in the evaluation_loop. It is initialised in the evaluate call but this uses the Trainer evaluate implementation which lacks generation.

Am I missing something?

Comments:
cc @gante 

My bad I realised the preprocess_logits_for_metrics function that I found [here](https://discuss.huggingface.co/t/cuda-out-of-memory-when-using-trainer-with-compute-metrics/2941/13) was truncating the generation output.
[i18n-PL] Translating docs to Polish
Author: nikos-py
Pull request: no
Labels: name: WIP

<!--
Note: Please search to see if an issue already exists for the language you are trying to translate.
-->

Hi!

Let's bring the documentation to all the Polish-speaking community 🌐 (currently 0 out of 267 complete)

Who would want to translate? Please follow the 🤗 [TRANSLATING guide](https://github.com/huggingface/transformers/blob/main/docs/TRANSLATING.md). Here is a list of the files ready for translation. Let us know in this issue if you'd like to translate any, and we'll add your name to the list.

Some notes:

* Please translate using an informal tone (imagine you are talking with a friend about transformers 🤗).
* Please translate in a gender-neutral way.
* Add your translations to the folder called `PL` inside the [source folder](https://github.com/huggingface/transformers/tree/main/docs/source).
* Register your translation in `PL/_toctree.yml`; please follow the order of the [English version](https://github.com/huggingface/transformers/blob/main/docs/source/en/_toctree.yml).
* Once you're finished, open a pull request and tag this issue by including #issue-number in the description, where issue-number is the number of this issue. Please ping @ArthurZucker, @sgugger for review.
* 🙋 If you'd like others to help you with the translation, you can also post in the 🤗 [forums](https://discuss.huggingface.co/).

## Get Started section

- [ ] [index.mdx](https://github.com/huggingface/transformers/blob/main/docs/source/en/index.mdx) https://github.com/huggingface/transformers/pull/20180
- [ ] [quicktour.mdx](https://github.com/huggingface/transformers/blob/main/docs/source/en/quicktour.mdx) (waiting for initial PR to go through)
- [ ] [installation.mdx](https://github.com/huggingface/transformers/blob/main/docs/source/en/installation.mdx).

## Tutorial section
- [ ] [pipeline_tutorial.mdx](https://github.com/huggingface/transformers/blob/main/docs/source/en/pipeline_tutorial.mdx)
- [ ]  [autoclass_tutorial.mdx](https://github.com/huggingface/transformers/blob/master/docs/source/autoclass_tutorial.mdx)
- [ ]  [preprocessing.mdx](https://github.com/huggingface/transformers/blob/main/docs/source/en/preprocessing.mdx)
- [ ]  [training.mdx](https://github.com/huggingface/transformers/blob/main/docs/source/en/training.mdx)
- [ ]  [accelerate.mdx](https://github.com/huggingface/transformers/blob/main/docs/source/en/accelerate.mdx)
- [ ]  [model_sharing.mdx](https://github.com/huggingface/transformers/blob/main/docs/source/en/model_sharing.mdx)
- [ ]  [multilingual.mdx](https://github.com/huggingface/transformers/blob/main/docs/source/en/multilingual.mdx)

<!--
Keep on adding more as you go 🔥
-->


Comments:
Pin numba for now
Author: sgugger
Pull request: yes

# What does this PR do?

Today's release of `numba` broke the audio feature extractors. Not sure if it's because of numba by itself or because it forces an update to Numpy 1.24. Will be investigated later by the audio team but in the meantime pinning `numba` to make `main` green.

cc @ydshieh @sanchit-gandhi for information, will merge this as soon as the CI is green.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
[`Doctest`] Fix pix2struct doctest
Author: younesbelkada
Pull request: yes

# What does this PR do?

Link to failing job: https://github.com/huggingface/transformers/actions/runs/4867713745/jobs/8680544136

This PR fixes the current failing doctest for pix2struct. https://github.com/huggingface/transformers/pull/23051 fixed the issues related with pix2struct and training by changing the way attention masks are computed. Naturally this has changed the value of the expected loss function in the docstring

cc @ydshieh 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Generate: slow assisted generation test
Author: gante
Pull request: yes

# What does this PR do?

`test_assisted_decoding_matches_greedy_search` fails once in a while, which blocks development. This PR removes the blocker by moving it to a slow test. 

Why a slow test (and not redesign the test or add the flaky decorator)?
1. It is impossible to remove at 100% the non-determinism in this test. Some form of masking has to be used by design, which means that there is always a chance for the generations to diverge. When the generated sequences do diverge, the scores should be very similar at the step they diverge, as they are caused by the very small values within the numerical attention masks.
2. I've tried to add the check above (when the sequences diverge, the scores should be similar)... but some models still failed that check quite hard when the sequences didn't match. Some well-established models can run it without observed failures (e.g. 10k runs on GPT2 = 0 sequence mismatches). Others, especially roberta-based models, fail at a high rate. This means I should explore this further.
3. Since there is something to be explored, I believe the slow decorator is more appropriate: we can track failures without risking red CI on PRs.


Comments:
_The documentation is not available anymore as the PR was closed or merged._

> Maybe the test will be less flaky if done on a pretrained checkpoint

Definitely! However, I think there is a deeper problem here, the logits diverge way more than I'd expect on some models, and it's odd that those models rely on the same base code (roberta). After I finish preparing the release for assisted generation, I'll get back to sorting related bugs
A potential bug here found in `BeamSearchScorer.process`
Author: ZachVec
Pull request: no

### System Info

System doesn't matter.

### Who can help?

@gante 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I am new to the transformer and reading the source code of the beam search in `src/transformers/generation/beam_search.py`. And I have a question about the code here: https://github.com/huggingface/transformers/blob/main/src/transformers/generation/beam_search.py#L290

I noticed in PR #21993, the variable `cur` is increased by one before checking if beam hyp is done, but **this variable is increased in a loop**. In other words, this variable would be increased for each sample in this batch. I wonder if there is any particular reason for that.

### Expected behavior

```python
    def process(
        self,
        input_ids: torch.LongTensor,
        next_scores: torch.FloatTensor,
        next_tokens: torch.LongTensor,
        next_indices: torch.LongTensor,
        pad_token_id: Optional[int] = None,
        eos_token_id: Optional[Union[int, List[int]]] = None,
        beam_indices: Optional[torch.LongTensor] = None,
    ) -> Tuple[torch.Tensor]:
        cur_len = input_ids.shape[-1] + 1  # the one should be add up here, instead of in the loop.
        # some code here.
        for batch_idx, beam_hyp in enumerate(self._beam_hyps):
            # some code here.
            self._done[batch_idx] = self._done[batch_idx] or beam_hyp.is_done(
                next_scores[batch_idx].max().item(), cur_len
            )
        # some code here.
```

Comments:
Hey @ZachVec -- I believe you are correct, the implementation is incorrect for batch size > 1. I'll open a PR to fix it :)

Should be fixed now 🤗
Generate: correct beam search length on score calculation for multi batch generation
Author: gante
Pull request: yes

# What does this PR do?

Fixes #23084 

When computing the score with length penalty, the length was (incorrectly) incremented once per batch member. It should only be incremented once -- the length here is `cur_len` (the length of the generated tokens) + `1` (the token being added at the iteration).

Slow tests were ran for BART, T5, GPT2 -- no regressions.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Generate: better warnings with pipelines
Author: gante
Pull request: yes

# What does this PR do?

Addresses comments in https://github.com/huggingface/transformers/issues/23054

This PR adds the following enhancements to generate-related pipeline warnings:
1. Clarifies the `max_length` reduction suggestion in the summarization pipeline
2. Also pipes task-specific configuration to `generation_config` (when applicable), which fixes the warning about relying on `model.config` to parameterize `generate`

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Add resources for LayoutLmV2 and reformat documentation resources
Author: y3sar
Pull request: yes

# What does this PR do?

From #19848 
This PR adds resources to the LayoutLMV2 documentation page.

Also in the LayoutLMV2 documentation page the documentation resources heading was incoherent with the other doc pages so I removed the heading and put the task specific guides under the corresponding task headings. 



Comments:
cc @stevhliu 

_The documentation is not available anymore as the PR was closed or merged._
added farsi lang
Author: mzamini92
Pull request: yes

# What does this PR do?

Added Farsi (fa) to the docs. I have translated `_toctree.yml` and `accelerate.mdx` and `add_new_pipeline.mdx`. the rest also will be translated soon. I will make this pull request so others also can contribute and make this process faster. 

<!-- Remove if not applicable -->

Fixes # (issue)


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.
@sgugger 

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @sgugger

Integrations:

- deepspeed: HF Trainer: @stas00, Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam

Documentation: @sgugger, @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: @sgugger
- TensorFlow: @Rocketknight1

 -->


Comments:
_The documentation is not available anymore as the PR was closed or merged._

> Please only add the translated file in the new folder.

@sgugger only the translated files are now in fa folder. the rest have been removed.

> Thanks a lot! One last comment on the `add_new_model` file.

@sgugger I'm sorry I didn't quite get what you mean? can you please tell me more to do it right away? appreciate it. 

You have translated half the file only. Maybe leave it out of this PR and add it in a new PR when you're fully done?
Add support for beam search's num_return_sequencs flag in flax 
Author: mayankagarwals
Pull request: yes

Fixes part of https://github.com/huggingface/transformers/issues/22696

Comments:
CC @gianlucadetommaso and @gante 

_The documentation is not available anymore as the PR was closed or merged._

@mayankagarwals to make our CI green, you will likely need to rebase with `main`, then run `make fixup`, then commit. 

Also, tag the PR as ready when you're ready for a final check from a core maintainer :)

Hey @gante ,

Yes! Have done those. Wanted to get your views on it before cleaning up the PR. The CI is green now. 

I couldn't find any specific test for this so didn't add but the following script serves as a decent test for functional purposes
```
from transformers import FlaxGPT2LMHeadModel, GPT2Tokenizer

tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
model = FlaxGPT2LMHeadModel.from_pretrained("gpt2", pad_token_id=tokenizer.eos_token_id)

input_ids = tokenizer.encode('I enjoy walking with my cute dog', return_tensors='jax')

beam_output = model.generate(
    input_ids,
    max_length=50,
    num_beams=5,
    no_repeat_ngram_size=2,
    num_return_sequences=2,
    early_stopping=True
)
print("All generated hypotheses:\n")
for sequence in beam_output.sequences.tolist():
    print(tokenizer.decode(sequence, skip_special_tokens=True))
    print("-------")
    
```
    
    
   Output before the change: 
   
```
All generated hypotheses:

I enjoy walking with my cute dog, but I'm not sure if I'll ever be able to walk with him again.

I'm not sure if I'll ever be able to walk with him again. I'm not sure if I'll
-------
```

Output after change: 
```
All generated hypotheses:

I enjoy walking with my cute dog, but I'm not sure if I'll ever be able to walk with him again.

I'm not sure if I'll ever be able to walk with him again. I'm not sure if I'll
-------
I enjoy walking with my cute dog, but I'm not sure if I'll ever be able to walk with him again.

I'm not sure if I'll ever be able to walk with him again.

I'm not sure if
-------
```

> But augment it to return N beams and verify we get the sequences/scores for these beams

Sure, have added the test. Thanks for the reference, made it a 5-minute work :p  @sanchit-gandhi @gante 

ready for a final check, tagging a core maintainer