noamaanMulla-03/transformers-issues huggingface dataset:
Fix CLAP converting script
Author: ylacombe
Pull request: yes

# What does this PR do?

Following [this hub discussion](https://huggingface.co/lukewys/laion_clap/discussions/3) and the addition of CLAP new checkpoints this summer, I've corrected CLAP converting script and added the new checkpoints to the hub ([here](https://huggingface.co/ylacombe/larger_clap_music), [here](https://huggingface.co/ylacombe/larger_clap_general) and [here](https://huggingface.co/ylacombe/larger_clap_music_and_speech) - yet to be moved to laion organization).

Note that I've also used [this fix](https://github.com/LAION-AI/CLAP/pull/118) of the CLAP library from @Vaibhavs10, that is yet to be merged!  

I've also manually checked that it didn't break the original checkpoint conversion!


## Who can review?

Hey @amyeroberts and @sanchit-gandhi ! cc @Vaibhavs10 as well!

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27153). All of your documentation changes will be reflected on that endpoint.
Added passing parameters to "reduce_lr_on_plateau" scheduler
Author: CharbelAD
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Follow up on #27595. Added support to pass `lr_scheduler_kwargs` to scheduler of type "reduce_lr_on_plateau".

Code example:

```python
train_dataset = RegressionDataset(length=64)
eval_dataset = RegressionDataset(length=64)
args = TrainingArguments(
            "./regression",
            evaluation_strategy = "epoch",
            lr_scheduler_type="reduce_lr_on_plateau",
            lr_scheduler_kwargs={"factor": 0.5, "mode": 'max'},
            learning_rate=0.2,
            warmup_steps=2,
        )
model = RegressionModel()
trainer = Trainer(model, args, train_dataset=train_dataset, eval_dataset=eval_dataset)
trainer.create_optimizer_and_scheduler(num_training_steps=10)
print(trainer.lr_scheduler.factor, trainer.lr_scheduler.mode)
```
Current behavior: prints `0.1 min` which are the default values of the parameters `factor` and `mode` because `lr_scheduler_kwargs` are not being passed.

Expected behavior: prints `0.5 max`.

I did not write any test cases for this since a test case for passing extra kwargs was already added in #27595, however, I can add one if any maintainer finds it necessary. 

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [X] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [X] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@muellerzr, @pacman100, and @ArthurZucker 


Comments:
mark `test_initialization` as flaky in 2 model tests
Author: ydshieh
Pull request: yes

# What does this PR do?

`torch.nn.init.trunc_normal_` is flaky and sometimes produce large value even if `mean=0.0` and `std=1e-10).

 

Comments:
For the record: below code shows the issue

```python
import torch

DIM = 32
STD = 1e-10
B = 2
N_ITER = 100


for _ in range(100):
    for pow in range(4, 15):
        m = 0
        seq_len = 2 ** pow
        for i in range(N_ITER):
            t = torch.zeros(size=(seq_len, DIM))
            o = torch.abs(torch.nn.init.trunc_normal_(t.to(torch.float32), mean=0.0, std=STD, a=-B, b=B))
            n = torch.sum(o >= B)
            m += n

        print(f"seq_len = {seq_len}: {m / seq_len / DIM / N_ITER}")

    print("-" * 80)
```

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27906). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix ExponentialDecayLengthPenalty negative logits issue
Author: pokjay
Pull request: yes

# What does this PR do?

In cases where the model logits are negative, ExponentialDecayLengthPenalty decreases the score of eos_token_id instead of increasing it.
To fix this issue we compute the penalty of the absolute value and add it to the original score, as described in #25416
The test was updated to check for negative logits.

In addition, this PR updates the class documentation and adds examples, as part of #24783

Fixes #25416


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?
@gante 


Comments:
Thanks @gante and @ArthurZucker !

Could you please approve the documentation workflow? I’d like to verify the documentation looks as expected

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_25594). All of your documentation changes will be reflected on that endpoint.

@gante Do I need to do anything more to get this merged?

You could take into account the changes I suggested 😉 code related variable name are always put aroung codeblocks in our doc! 

@ArthurZucker oops, totally missed those! Committed those fixes now, Thanks!

@pokjay since we last spoke, we've added this file to the list of files to be doctested in our PR CI -- it seems like your example's outputs don't match the hardcoded outputs. 

Would you be able to double-check that? :)

(as soon as this gets fixed, we can merge)

@gante Fixed the issues, all checks pass!

@pokjay thank you for iterating 💛
Llama conversion script: adjustments for Llama Guard
Author: pcuenca
Pull request: yes

# What does this PR do?

Small adjustments to the Llama 2 conversion script so it works with the original Llama Guard weights.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

cc @ArthurZucker 

Comments:
fix llava
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Fix the prepare inputs for generationa after the cache PR.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27909). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Allow `resume_from_checkpoint` to handle `auto_find_batch_size`
Author: muellerzr
Pull request: yes

# What does this PR do?

This PR adds the training batch size as part of the `TrainerState`. We do this because the `TrainerState` can be loaded in on `resume_from_checkpoint`, and so if a user has set `auto_find_batch_size` to be `True`, we can keep what that batch size was in there and load it back in if it were saved

Fixes https://github.com/huggingface/transformers/issues/25956


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@amyeroberts 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27568). All of your documentation changes will be reflected on that endpoint.

@ArthurZucker agreed that it's a bit overkill. Would it be better to create a new file instead (something like `training_metadata.json`) instead that only gets made for now when doing something like `auto_find_batch_size` is enabled?

Why don't we just overwrite the arg given from the user? 

@ArthurZucker we still need to store it away somewhere when we do `resume_from_checkpoint`. The assumption is given a fresh run we don't want to have to run through the iteration loop again to find the right batch size if we've found it once during a prior call. It still needs to be saved somewhere outside on the file system

Ah okay, we don't know if the input batch was auto-found or not. Got it. Not sure we want to create a new file for this, fine with loading the state and if we need more meta-data we'll put them there as well I guess!
resume_from_checkpoint may fail with auto_find_batch_size
Author: n-splv
Pull request: no

### System Info

When we resume training from a checkpoint, the process may stale, because the number of already passed steps from the checkpoint may turn out to be greater than the estimated total number of steps. When `auto_find_batch_size` is turned on, the trainer would first try to choose a higher value for the batch size, before running out of memory. 

Consider a simple example:
We want to train a model on 100 samples for 10 epochs. Here is what happens:
1. The trainer tries to work with a larger `batch_size = 8`. The estimated number of steps is 100 * 10 / 8 = 125;
2. We run out of GPU memory, and eventually the batch_size gets reduced to 2. We now have 100 * 10 / 2 = 500 steps to go;
3. At the step 150 we save a checkpoint and stop the training;
4. Later we load the model from a checkpoint and try to continue training with the same params. Now the trainer would once again try to set the batch_size = 8, estimate 125 total steps and... Finish the process immediately, since we have already taken 150 / 125 steps.

### Who can help?

@muellerz, @pacman100

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```
args = Seq2SeqTrainingArguments(
    ...
    auto_find_batch_size=True,
)
train_result = trainer.train(resume_from_checkpoint=CHECKPOINT_DIR)
```

### Expected behavior

The information about the used batch_size should probably be saved somewhere in the checkpoint, and the trainer should be smart enough to account for it when interpreting the number of completed steps. For now, it seems like the only solution is to continue the training by manually providing the same batch size, which is not intuitive and somewhat restricting - suppose, my hardware changed but I want to resume the training from my checkpoint.

Comments:
cc @muellerzr
[Doc] Spanish translation of pad_truncation.md
Author: aaronjimv
Pull request: yes

# What does this PR do?

Add the Spanish version of `pad_truncation.md` to `transformers/docs/source/es`

Fix a typo in the table of `en/pad_truncation.md`

Fixes #15947 


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @pacman100

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc and @younesbelkada

Documentation: @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->

@omarespejel @sgugger @osanseviero @stevhliu

Comments:
Hello. I am open to any feedback.

Hi @stevhliu, thanks!

Hi @osanseviero, thanks for the help. Please let me know anything else.
Speculative decoding functionality
Author: domgri
Pull request: no

### Feature request

Hey 👋 

Idea is to add speculative decoding functionality in assisted generation code path. There are several similar ways how that can be achieved.

Related papers: 
* [Blockwise Parallel Decoding](https://proceedings.neurips.cc/paper/2018/file/c4127b9194fe8562c64dc0f5bf2c93bc-Paper.pdf), by Google Brain
* [Speculative Sampling](https://arxiv.org/abs/2302.01318), by DeepMind
* [Fast Inference from Transformers via Speculative Decoding](https://arxiv.org/pdf/2211.17192.pdf)

### Motivation

- Shows promising inference time improvements.
- Would enable faster expermentation with various models.
- Could become a fundamental place for more elaborate experimentation of speculative decoding.

Related issues/discussions: #26565


Related blog post explaining current assisted generation implementation and its (improvements and) limitations: [here](https://huggingface.co/blog/assisted-generation) by @gante 


### Your contribution

I would like to give an attempt and rase a PR for this (e.g. for Google Brain implementation).

Although, might need some help with:

- Deciding which approach to implement
- Decision for code architecture - either combine with assisted generation, partially combine or implement separately
- Motivation - maybe I am only one interested in this idea 😊 

Any feedback would be greatly appreciated 🙇‍♂️ 

Comments:
Interesting! @gante WDYT? 

Hi @domgri 👋 

I'm very interested in having DeepMind's Speculative decoding when `do_sample=True`, in `assisted_generation`! It should only need a few extra if/elses, to handle the clever token acceptance strategy they came up with :)

As for the others, since they don't seem to bring additional benefits, I would like to avoid adding them -- maintaining different generation methods takes up a significant amount of time, so they need to bring benefits for us to justify adding them to the codebase. 

Finally, FYI, I'm trying to design a strategy for issues like this one: some way to leverage the Hub to add new generation methods than don't need to be approved (or maintained) by the transformers team 🙌 

Thanks for a quick and encouraging feedback!

Then for now I will focus on DeepMind's Speculative decoding and see how it goes 🙌. Though, will be interested to see the outcome of strategy question as well 👍.


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Bug in greedy_search when use_cache=False and use inputs_embeds as model input
Author: TsuTikgiau
Pull request: no

### System Info

transformers 4.30.0, python 3.9, ubuntu

### Who can help?

@gante

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

outputs = llama2model.generate(
    inputs_embeds=inputs_embeds,
    attention_mask=attention_mask,
    do_sample=False,
    use_cache=False
)

### Expected behavior

I'm using LLama2 model to take token embedding directly as input and do greedy search generation (do_sample=False) with no KV cache (use_cache=False). I faced a runtime error "RuntimeError: shape '[-1, 332]' is invalid for input of size 666" at [this line](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/models/llama/modeling_llama.py#L528). 

Note that when use_cache=True, the code works. 

After some debugging I found that this is because inputs_embeds is not updated. After each decode step, the new generated input_id is added at [this line](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/utils.py#L2382) and the new attention mask is added at [this line](https://github.com/huggingface/transformers/blob/v4.30.0/src/transformers/generation/utils.py#L2386). However, the new inputs_embeds is not added. As we set use_cache=False and use the inputs_embeds as input, the original inputs_embeds will be use again in the forward pass with an updated attention mask (length+1), and leads to the size mismatch error. I have checked the latest code in the main branch and notice that the update of inputs_embeds is still missing.



Comments:
BTW, I only checked greedy_search. I'm not sure if this bug exists also in other decode strategies.

Hey @TsuTikgiau 👋  Thank you for opening this issue.

The situation you describe is correct; `input_embeds` is not updated throughout `generate`, `input_ids` is. As such, it requires `use_cache=True` to work -- the relevant corresponding inputs are stored in `past_key_values`. 

Adding support for it would add significant complexity to `generate`. As such, my decision for now is *not* to add it, as I haven't seen any request for it. My advice would be to modify the function locally to suit your needs :)

Other users: if you encounter this issue, let me know (reaction or comment here). If several people request this feature, I might revisit the decision above! 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.