noamaanMulla-03/transformers-issues huggingface dataset:
Generate: update links on LLM tutorial doc
Author: gante
Pull request: yes

# What does this PR do?

This PR:
1. Adds new links to the LLM tutorial doc (more specifically: to the new text generation speedup doc and to two external libraries that collaborate with us)
2. changes where the link is located in some sentences -- color highlighting is important to quickly find the key piece of information!

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30550). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
DBRX: make fixup
Author: gante
Pull request: yes

# What does this PR do?

#30476 introduced some `make fixup` check failures on `main`, this PR fixes them

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30578). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Amy/Yih-Dar: merging as it is a very small bugfix (that breaks our copies checks in everyone's CI)

@gante Thanks for the quick fix!
Fix generation doctests
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Fixes some doctests that were failing in [this run](https://github.com/huggingface/transformers/actions/runs/8659540805/job/23745774900)


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30263). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

The failing `src/transformers/generation/tf_utils.py::transformers.generation.tf_utils.TFGenerationMixin.sample` is passing for me locally 🤔 

confirmed all passed, thanks a lot!

Let's not bother by tests_pr_documentation_tests  here. It's likely CPU vs GPU issue.

@gante @amyeroberts Now we raise an error is user tries to pass in LogitsProcessor object instead of a kwarg for assistant model. Tests are passing, except for the gpu vs cpu one, as confirmed by @ydshieh above

@ydshieh since the failing doctest is okay on your end, would you be able to merge? 🤗 (I don't have permissions to merge with red CI :D )

Yes, I will merge later today. Thank you  for all the work/review you 3 have done ❤️ 

The run I triggered is 2 weeks ago and since then there is a commit changing something.

Let's merge as I am kind confident it still works! (well I only triggered doctest)

Thanks again!
General PR slow CI
Author: ydshieh
Pull request: yes
Labels: name: run-slow

# What does this PR do?

Extend #30341 to handle general PR slow CI. Now, it could handle

- new model pull request
- or a pull request with the (latest) commit have the form `[run-slow]bert,gpt2`
  - the prefix could be `[run-slow]`, `[run_slow]` or `[run slow]`
  - the remaining part should be comma-separated like `bert,gpt2` or `bert, gpt2` etc.
    - there is no syntax check or error handling of the wrong format (we can add them later if we find necessary)

The script and workflow file names (and their methods or job/step names too) to reflect they handle more general cases instead of just new model cases.

[An example run](https://github.com/huggingface/transformers/actions/runs/8896080103/job/24427938756):
  - (I deliberate make a failing tests)
  - it's trigger with the commit message `[run-slow]bert,gpt2`

IMPORTANT: the label to add in order to trigger this CI is **run-slow**. 

`single-model-run-slow` is no longer used.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30540). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Remove `use_square_size` after loading
Author: ydshieh
Pull request: yes

Remove `use_square_size` after loading.

#27690 removed `use_square_size` but keep a backward compatibility for Kosmos-2's image processor.
However, once we load it with a image processor config file, we could and we should remove that no longer used attribute.

Fix #30522

Comments:
> Wondering if we could remove `use_square_size` at some point?

You mean remove them from Hub repositories first, then at the end from our code?



The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30567). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Yes, although people may have already fine-tuned KOSMOS-2 (e.g. https://huggingface.co/Mit1208/kosmos-finetuned-DocLayNet), both locally and on the hub.

> Yes, although people may have already fine-tuned KOSMOS-2 (e.g. https://huggingface.co/Mit1208/kosmos-finetuned-DocLayNet), both locally and on the hub.

I will check what I can do (regarding the Hub repositories part)

> Thanks for fixing!
> 
> Can you add a test for kosmos to make sure that the correct image sizes are selected?

@amyeroberts  This PR doesn't change (directly) any logic regarding the size, although it does remove a attribute and this might change something.

What test you have in mind? Like 

- load an image processor with `use_square_size`, save it, and reload it again (locally) and see if it works and the results are matched?

@ydshieh OK, if image size isn't affected then we don't need to cover that, but something like what you suggest: saving and reloading the processor to make sure it can be done. Anything which fails on main and passes with this PR :)
[trainer] self.model_wrapped + _model_unwrap
Author: stas00
Pull request: yes

This PR adds:

* [x] adds `self.model_wrapped` - to have access to the outmost module regardless of how many times it was wrapped (e.g. under DeepSpeed there is a double wrapping ` DDP(Deepspeed(Transformers Model))`)
* [x]  makes sure that `self.model` is always set to the normal model
* [x] fixes a bug where under `model_parallel` `self.model` was not set (twice)!
* [x] simplifies the `model_init` checking logic 
* [x] replaces `_actual_model` which couldn't handle multiple wrapping levels with `_model_unwrap` which can and integrate it

Please ignore the small mentions of DeepSpeed, this PR is split of from https://github.com/huggingface/transformers/pull/9211 to get all the non-DeepSpeed related changes into a separate review to make things a bit easier on the reviewers as it was suggested by @sgugger. This PR was made by copying from the other PR and manually removing all the added deepspeed code. 

If possible, let's get it in asap so that I could rebase and we could move on with the DeepSpeed PR. Thank you very much!

@sgugger 

Comments:
This should be good now, @sgugger - thanks a lot for all the suggestions!

@LysandreJik GitHub Reviewers is down, so tagging you instead.

> This PR adds:
> 
> * [x]  adds `self.model_wrapped` - to have access to the outmost module regardless of how many times it was wrapped (e.g. under DeepSpeed there is a double wrapping ` DDP(Deepspeed(Transformers Model))`)
> * [x]   makes sure that `self.model` is always set to the normal model
> * [x]  fixes a bug where under `model_parallel` `self.model` was not set (twice)!
> * [x]  simplifies the `model_init` checking logic
> * [x]  replaces `_actual_model` which couldn't handle multiple wrapping levels with `_model_unwrap` which can and integrate it
> 
> Please ignore the small mentions of DeepSpeed, this PR is split of from #9211 to get all the non-DeepSpeed related changes into a separate review to make things a bit easier on the reviewers as it was suggested by @sgugger. This PR was made by copying from the other PR and manually removing all the added deepspeed code.
> 
> If possible, let's get it in asap so that I could rebase and we could move on with the DeepSpeed PR. Thank you very much!
> 
> @sgugger

yes , i find that  DeepSpeed there is a double wrapping  DDP(Deepspeed(Transformers Model))) in many times , but i see the code , only find ddp wraped code like this self.accelerator.ddp_handler = DistributedDataParallelKwargs(**kwargs), 
but i can not  find the code for deepspeed(Transformers Model),  i am sorry ,where code is to wraped deepspeed ？ thanks
Use text config's vocab size in testing models
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

The PR replaces all the `model.config.vocab_size` with `text_config.vocab_size` if available. This way VLMs do not need to copy vocab size related tests in every modeling file + I see that "vocab_size" is deprecated in Llava and will be removed in "v4.42"

Related to adding new model VideoLlava, shoudl be merged before that

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30568). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Encoder-decoder models: move embedding scale to nn.Module
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

This PR moves embedding scale to nn.Module in encoder decoder models, so that users who want to pass in `inputs_embeds` to the `forward` will get the same results as if they passed `input_ids` via `model.get_input_embeddings()(input_ids)`

The generation from embeds is not supported for these models, that's why we did not see the inconsistency. I think we do not need specific tests and here I can add support for generation from embeds in another PR if needed.

All the tests (+slow) for the changes models are passing on my end

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30410). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@zucchini-nlp also, can you:
1. Check if there is a common test checking that the output of `forward` with `input_ids=input_ids` == output of forward with `inputs_embeds=model.get_input_embeddings()(input_ids)`
2. If there is not, can you add it? 🙏 

We'll tag the core maintainer after we ensure there's a test!

Hmm, okay, I thought the test for generation with inputs-embeds is enough. There is no such test, yes. If that's needed I can add it and that can trigger handling other models/cases if there are peculiarities :( 

Seems like it's ready for core maintainer's review. Reverted the previous commit with skips and tested that checking signature is working for most cases. In cases where it does not work, model-specific test skipping is kept. 

Failing tests are passing for me locally, and are not related to the changes

For the musicgen tests - a fix was pushed to main - rebasing should resolve 

@amyeroberts done, all the NotImplementedError models are skipped in their modeling_test files and the CI is green
Fix Marian model conversion
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Fixes #26216. After a bit of code exploration I found that problem was in the "tie_weights" method. The issue is that "tie_weights" does not clone, but simply change pointer output_embeddings -> input_embeddings. And since we tie weights at least twice during loading weights (once before loading state dict and the other is after), the weights were being loaded incorrectly.


Why the issue is only in MarianMT and not other models that tie weights? -> Because in other models either the weights already hold the same/tied values , or the order of loading parameters is different from MarianMT. When loading state dict into the MarianMT, "output_embeddings" are loaded last and therefore "input_embeddings" weights data is overriden to will hold same data as "output_embeddings". That way we lose access to actual "input_embeddings" weights data
This is somehow related to an [old PR](https://github.com/huggingface/transformers/pull/24200) I found.


I converted the weights with `python src/transformers/models/marian/convert_marian_tatoeba_to_pytorch.py --models fin-eng --save_dir converted` and checked the correctness of results with the below script. Everything works good.

```python
from transformers import AutoTokenizer, MarianMTModel

tokenizer = AutoTokenizer.from_pretrained("/home/raushan/converted/opus-mt-fin-eng/")
model = MarianMTModel.from_pretrained("/home/raushan/converted/opus-mt-fin-eng/")

inputs = ["Hei siellä", "Miten aurinko sanotaan suomeksi?"]
batch_tokenized = tokenizer(inputs, return_tensors="pt", padding=True)
model_output = model.generate(
    **batch_tokenized, max_new_tokens=100
)
batch_detokenized = tokenizer.batch_decode(
    model_output,
    skip_special_tokens=True,
)

print(batch_detokenized)
```

Marian tests including slow are all passsing on my end.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30173). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@ArthurZucker i reverted the change for tying weights, to be consistent with 'main'. It was not the actual solution for Marian models, so I think it does not hurt whatever was the reason for it to be added. Requesting re-review :)
Refactor default chat template warnings
Author: Rocketknight1
Pull request: yes

The Idefics2Processor applies a chat template using its tokenizer, but there's an annoying interaction between the way it calls `tokenizer.apply_chat_template()` and the way we trigger default chat template warnings, which means that even when we correctly add a chat template to `processor_config.json`, the default chat template warning is still thrown. The reason is that the tokenizer method still sniffs `self.default_chat_tokenizer` if `self.chat_template` is not set (which it isn't, because the template is set on the parent processor instead), which results in the warning being thrown.

Rather than doing a refactor of the method to fix this now, I'm going to temporarily silence the warnings for one or two more versions to avoid bad UX, and then once we properly remove default chat templates I can restore `IdeficsProcessor2`, and properly refactor the tokenizer `apply_chat_template` code as well, and all of this should be resolved!

I've also opened PRs to add proper chat templates to the IDEFICS2 repos.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30551). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix QA example
Author: Rocketknight1
Pull request: yes

The QA example assumes the model has a `CLS` token, but many newer models don't. This PR falls back to the `BOS` token if one isn't present, and finally falls back to the start of the sequence if that doesn't work either.

Fixes #30570



Comments:
Will do now before merging!

Tried running it on Falcon myself, but it turns out there are multiple other issues here - in particular, the code also assumes a `PAD` token is present. Fundamentally, this example just isn't suited to fine-tuning a CLM like Falcon!

I think we can maybe still merge the PR (since there are likely MLM models either now or in the future that will be missing CLS), but @ananegru you'll probably need to use a masked language model as a base rather than a causal language model! Maybe one of the DeBERTa models?

Just confirmed that the example still works with this branch and `microsoft/deberta-v3-base`

My bad for going through all this before realizing the root of the problems was a CLM being used in MLM examples, though!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30580). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> My bad for going through all this before realizing the root of the problems was a CLM being used in MLM examples, though!

I didn't notice either! Happy for this to be merged in if it still works with existing models. Thanks again for adapting the scripts so quickly
warm-starting encoder-decoder models using EncoderDecoderModel always giving an empty string after fine-tuning
Author: kawsar-pie
Pull request: no

### System Info

Hi @ArthurZucker  and @patrickvonplaten, I am trying to train a seq2seq model using **EncoderDecoderModel** class and found [this](https://huggingface.co/blog/warm-starting-encoder-decoder) blog very helpful. Thanks to @patrickvonplaten for the nice explanation. Following this blog I fine-tuned a seq2seq model where I used a **BERT** ([BanglaBERT](https://huggingface.co/csebuetnlp/banglabert) an Electra) model as encoder and **[XGLM](https://huggingface.co/facebook/xglm-564M)**  as decoder using [BanglaParaphrase](https://huggingface.co/datasets/csebuetnlp/BanglaParaphrase) data. But after fine-tuning the model always generates an **empty string**. Now I do not understand where the problem is. Please help me to find the bug in the [code](https://www.kaggle.com/code/kawsar17/paraphrase-bert2xglm).

Thanks.

### Who can help?
@ArthurZucker 
@patrickvonplaten 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Here is my [notebook](https://www.kaggle.com/code/kawsar17/paraphrase-bert2xglm)

### Expected behavior

Input-output for my code:
{'target': 'সিপিও আহত থাকায় যুদ্ধ পরিচালনার দায়িত্ব এসে পড়েছিল সেম্প্রোনিয়াসের কাঁধে।',
 'pred_target': ''}
 
 which should be something like this (should give the paraphrased sentence according to the input sentence in Bangla):
 {'target': 'সিপিও আহত থাকায় যুদ্ধ পরিচালনার দায়িত্ব এসে পড়েছিল সেম্প্রোনিয়াসের কাঁধে।',
 'pred_target': 'সিপিও কর্তৃক আহত হয়ে সেমপ্রোনিয়াসের কাঁধে যুদ্ধ পরিচালনার দায়িত্ব আসে।'}

Comments:
Hi @kawsar-pie, thanks for raising an issue! 

This is a question best placed in our [forums](https://discuss.huggingface.co/). We try to reserve the github issues for feature requests and bug reports.

Thank you @amyeroberts for your comment. I created a new topic in the forum for this issue. I hope to find a solution from there.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Generate: consistently handle special tokens as tensors
Author: gante
Pull request: yes

# What does this PR do?

To enable `torch.compile` with `generate`, some special token-related operations have to be rewritten into torch operations. That requires special tokens to be tensors instead of integers or a list of integers. (See https://github.com/huggingface/transformers/pull/29374 for a working prototype)

This PR reworks special token usage in `generate` to consistently treat them as a tensor, as opposed to e.g. keeping track of `eos_token_id` in integer and in tensor form.

👉 Review suggestion: start by reading `_prepare_special_tokens` and how it fits in `generate`.

______________________________________________________________________
Requirements before merging this PR:
- [ ] merge https://github.com/huggingface/transformers/pull/29718 [Fixes doctests]

Tests ran locally:
- [ ] logits processors doctests (`pytest --doctest-modules src/transformers/generation/logits_process.py -vv`), needs requirement to be merged first
- [x] generate doctests (`pytest --doctest-modules src/transformers/generation/utils.py -vv`)
- [x] generate integration tests (`RUN_SLOW=1 py.test tests/generation/test_utils.py -vv`)
- [x] cache integration tests (`RUN_SLOW=1 py.test tests/test_cache_utils.py -vv`) -- same failures as in `main`
- [x] llama slow tests (`RUN_SLOW=1 py.test tests/models/llama/test_modeling_llama.py -vv`)
- [x] whisper slow tests (`RUN_SLOW=1 py.test tests/models/whisper/test_modeling_whisper.py -vv`) -- same failures as in `main`

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29788). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

let's merge https://github.com/huggingface/transformers/pull/29956 first, so the diff here becomes much smaller (the EOS-as-stopping-criteria made the diff more elaborate)

(Arthur -- don't review this one until that is merged, I'll ping you again)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
i cannot find the code that transformers trainer model_wrapped by deepspeed , i can find the theory about model_wrapped was wraped by DDP(Deepspeed(transformer model )) ,but i only find the code transformers model wrapped by ddp, where is the deepspeed wrapped ? thanks ^-^ 
Author: ldh127
Pull request: no

### System Info

i cannot find the code that transformers trainer model_wrapped by deepspeed , i can find the theory about model_wrapped was wraped by DDP(Deepspeed(transformer model )) ,but i only find the code transformers model wrapped by ddp, where is the deepspeed wrapped ? thanks ^-^ 

### Who can help?

i cannot find the code that transformers trainer model_wrapped by deepspeed , i can find the theory about model_wrapped was wraped by DDP(Deepspeed(transformer model )) ,but i only find the code transformers model wrapped by ddp, where is the deepspeed wrapped ? thanks ^-^ 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

i cannot find the code that transformers trainer model_wrapped by deepspeed , i can find the theory about model_wrapped was wraped by DDP(Deepspeed(transformer model )) ,but i only find the code transformers model wrapped by ddp, where is the deepspeed wrapped ? thanks ^-^ 

### Expected behavior

i cannot find the code that transformers trainer model_wrapped by deepspeed , i can find the theory about model_wrapped was wraped by DDP(Deepspeed(transformer model )) ,but i only find the code transformers model wrapped by ddp, where is the deepspeed wrapped ? thanks ^-^ 

Comments:
Closing as duplicate of #30591
remove jax example
Author: ArthurZucker
Pull request: yes

# What does this PR do?
The job only tests a single example, GLUE and that's it. SLOW tests are not even run. Let's remove this ! 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30498). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix canonical model --model_type in examples
Author: amyeroberts
Pull request: yes

# What does this PR do?

#29001 Updated references to canonical models. Some of the gpt2 replacements `gpt2` -> `openai-community/gpt2` should be reverted back, as they refer to the model type, rather than checkpoint

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30480). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix image segmentation example - don't reopen image
Author: amyeroberts
Pull request: yes

# What does this PR do?

Fixes bug in current image segmentation examples, where `Image.open(image)` was called on an image which was already a `PIL.Image.Image` object

Fixes #30467

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30481). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.