noamaanMulla-03/transformers-issues huggingface dataset:
Batch Generation giving different output when using batch size > 1 or when using padding in MambaForCausalLM
Author: piyushdevlpr
Pull request: no

### System Info

- `transformers` version: 4.41.2
- Platform: Linux-6.5.0-41-generic-x86_64-with-glibc2.35
- Python version: 3.10.8
- Huggingface_hub version: 0.23.3
- Safetensors version: 0.4.3
- Accelerate version: 0.26.0
- Accelerate config: not found
- PyTorch version (GPU?): 2.0.0+cu117 (True)
- Tensorflow version (GPU?): 2.9.0 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help?

@ArthurZucker @gante

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I have trained a MambaForCausalLM model on a custom dataset.
I am using the following code to generate next token in eval mode - 
```
model = MambaForCausalLM.from_pretrained("mamba_custom")
tokenizer = PreTrainedTokenizerFast.from_pretrained("mamba_tokenizer_custom")
tokenizer.padding_side = 'left'

inputs_rl = tokenizer(sentence, padding="max_length", truncation=True, max_length=100, return_tensors="pt").to("cuda:0")
with torch.no_grad():
    outputs = model(inputs_rl["input_ids"], attention_mask=inputs_rl["attention_mask"])
```

### Expected behavior

The tokenizer pads the input towards the left side. When I change the argument padding="max_length" to generate inputs without padding, I get different tokens as prediction.

Using model.generate gives the same issue as well.

Comments:
cc @gante 

@piyushdevlpr 👋 

Mamba, contrarily to transformers models, does not take an attention mask as input (see the signature [here](https://huggingface.co/docs/transformers/main/en/model_doc/mamba#transformers.MambaForCausalLM.forward)). As such, it does not support padding, and will return different values.

(I'm going to open a PR to try to prevent this issue from happening again)
Mamba & RecurrentGemma: enable strict signature
Author: gante
Pull request: yes

# What does this PR do?

Fixes #31540

Mamba accepts `**kwargs`, and thus `attention_mask` can be passed. Many users thus assume it behaves just like other models and can support left-padding.

RecurrentGemma also accept `**kwargs`, but simply not to crash `generate`.

This PR enables a strict signature on Mamba and RecurrentGemma.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31549). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Add MultiStepLR with Warmup Scheduler
Author: penguinwang96825
Pull request: no
Labels: name: Feature request

### Feature request

I would like to propose the addition of a new learning rate scheduler that combines MultiStepLR with a warmup phase. Currently, the Transformers library does not include a scheduler that uses both MultiStepLR and warmup. This feature can be beneficial for training models where the learning rate needs to be adjusted at specific epochs with an initial warmup phase to stabilise training.

### Motivation

In many training scenarios, it is beneficial to start with a warmup phase where the learning rate gradually increases, followed by a phase where the learning rate decreases at specific milestones (steps).

### Contribution

I propose adding a new scheduler, `get_multistep_schedule_with_warmup`, which combines the functionality of MultiStepLR and Warmup. This scheduler will increase the learning rate linearly during the warmup phase and then follow the MultiStepLR schedule. I am more than happy to create a pull request (PR) implementing this feature. Please let me know if this sounds like a valuable addition, and I will proceed with the implementation.

Comments:
cc @muellerzr @SunMarc 

Hi! In general we prefer if you can provide tangible results of improvement via either your own work or a paper referencing it. Can you link any please? thanks!

@muellerzr Thanks for the prompt reply! Sure I present it in detail below.

#### Popularity and Practical Use

The MultiStepLR scheduler is widely used and recognized for its effectiveness in practice, as evidenced by its popularity among PyTorch users. According to Defazio et al. (2023), it is one of the top three most popular schedulers. This piece-wise approach to decreasing the learning rate when progress plateaus has proven to be effective. Many studies incorporate this scheduler as a default choice for learning rate adjustment (Sohn et al., 2016; Wang et al., 2017; Gong et al., 2021).

| PyTorch Scheduler | GitHub Files (K) |
| :---: | :---: |
| ReduceLROnPlateau | 105.0 |
| StepLR | 101.0 |
| MultiStepLR | 37.9 |
| CosineAnnealingLR | 37.1 |
| ExponentialLR | 16.0 |
| OneCycleLR | 14.9 |
| CosineAnnealingWarmRestarts | 10.9 |
| CyclicLR | 9.1| 
| LinearLR | 5.9 |
| ConstantLR | 3.6 |
| MultiplicativeLR | 2.6 |
| PolynomialLR | 1.3 |

#### References

Defazio, Aaron, et al. "When, why and how much? adaptive learning rate scheduling by refinement." arXiv preprint arXiv:2310.07831 (2023).

Gong, Yuan, Yu-An Chung, and James Glass. "Ast: Audio spectrogram transformer." Proc. Interspeech (2021).

Wang, Jian, et al. "Deep metric learning with angular loss." Proceedings of the IEEE international conference on computer vision. (2017).

Sohn, Kihyuk. "Improved deep metric learning with multi-class n-pair loss objective." Advances in neural information processing systems 29 (2016).
Deprecate `vocab_size` in other two VLMs
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Continuing on #31655, let's deprecate the `model.config.vocab_size` for all VLMs for consistency across repo and use `model.config.text_config.vocab_size` in future models.

Currently only Fuyu and Paligemma have `vocab_size` in config file, and I depreacted it to v4.43. Or should it be v4.44? 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31681). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@zucchini-nlp @amyeroberts 
After this update, I am facing an error while making an instance of LlavaProcessor. This is the code I wrote:
'processor = AutoProcessor.from_pretrained("llava-hf/llava-1.5-7b-hf")'

It gives the following error:
'TypeError: LlavaProcessor.__init__() got an unexpected keyword argument 'chat_template''

It was working fine before the update. I have the following versions of transformers and bitsandbytes installed:
'!pip install -q -U transformers==4.37.2'
'!pip install -q bitsandbytes==0.41.3 accelerate==0.25.0'

Is there any issue with the current update or is there any other solution for this problem?

Answered the question in another thread, not related to this PR. Please open an issue if you still encounter the error after updating transformers :)
[Grounding DINO] Add processor to auto mapping
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR adds GroundingDinoProcessor to the AutoProcessor API.

cc @EduardoPach 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31845). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
chore: remove duplicate words
Author: hattizai
Pull request: yes



## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @SunMarc

Integrations:

- deepspeed: HF Trainer/Accelerate: @muellerzr
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc 

Documentation: @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
FX symbolic_trace: do not test decoder_inputs_embeds
Author: fxmarty
Pull request: yes

As per title, fixes https://github.com/huggingface/transformers/pull/31574#issuecomment-2213827446

https://github.com/huggingface/transformers/pull/31574 was meant to support `inputs_embeds`, not `decoder_inputs_embeds`. As some encoder-decoder models, we hit some errors (`ValueError: You have to specify either decoder_input_ids or decoder_inputs_embeds`).

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31840). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Isn't the error happening for `input_embed`?

> /usr/local/lib/python3.10/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py:711: in forward
    raise ValueError("You have to specify either input_ids or inputs_embeds")
E   ValueError: You have to specify either input_ids or inputs_embeds

(here is `BlenderbotSmallEncoder`).

@ydshieh yes, that's because `inputs_embeds` was not in the `input_names` list here https://github.com/huggingface/transformers/blob/ae9dd02ee1a8627d26be32202202b8081e9855a4/tests/test_modeling_common.py#L1141-L1151 for encoder-decoder models, and is then filtered out https://github.com/huggingface/transformers/blob/ae9dd02ee1a8627d26be32202202b8081e9855a4/tests/test_modeling_common.py#L1228.

@fxmarty There's still errors with fx tests happening on main: https://app.circleci.com/pipelines/github/huggingface/transformers/97262/workflows/8822144c-e8ed-4722-a982-72ee876c3eda/jobs/1287254

```
FAILED tests/models/mobilebert/test_modeling_mobilebert.py::MobileBertModelTest::test_torch_fx - RuntimeError: mat1 and mat2 shapes cannot be multiplied (4x192 and 96x64)
FAILED tests/models/mobilebert/test_modeling_mobilebert.py::MobileBertModelTest::test_torch_fx_output_loss - RuntimeError: mat1 and mat2 shapes cannot be multiplied (4x192 and 96x64)
```

Could you take a look?
Support dynamical input size and shape for TrOcr input image
Author: mykolamelnykml
Pull request: no

# 🚀 Feature request

Support dynamical input size and shape for TrOcr input image.

## Motivation

TrOcr has input shape for image 384x384, but can recognise only single line. So It does not effective utilize resources.



Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Hey @kolia1985  actually im implementing TrOCR for  ocr when i am trying to run the error has been showing up as The input dimensions are not matching to model input dimensions do you have idea about thesolution to this error??
generate tests failing with updated safetensors' weights
Author: amyeroberts
Pull request: no
Labels: name: Should Fix

name: Generation

### System Info

transformers main - 4.39.0.dev

### Who can help?

@gante 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Tests started failing because the new safetensors weights added for a test model on https://hf.co/joaogante/tiny-random-gpt2-with-generation-config

Failure message:

FAILED tests/test_modeling_utils.py::ModelUtilsTest::test_generation_config_is_loaded_with_model - ValueError: weight is on the meta device, we need a value to put in on 0.
Example circleci run:
https://app.circleci.com/pipelines/github/huggingface/transformers/86552

A patch was merged in #29521 to keep the CI green. However, the failing safetensors weights need to be addressed 

### Expected behavior

Tests pass loading from safetensors weights

Comments:
Test loading generation config with safetensor weights
Author: gante
Pull request: yes

# What does this PR do?

Fix #29522

This PR changes the model used in a generation config loading test to a model that a) is used on other tests b) has a generation config. The purpose of the test is to check that the generation config is loaded correctly with the model (which broke in the past)

The model used for the test is a dummy model, and it has empty weights that break when we load the model with `device_map="auto"` (because we can call `.to` with empty weights when the source is a `meta` device).



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31550). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
save_pretrained: use tqdm when saving checkpoint shards from offloaded params
Author: kallewoof
Pull request: yes

Even at 9B, doing a save with partially CPU offloaded tensors can take up to 30 seconds. Doing a 70B model can take 5-10 minutes depending on how much VRAM you have around. This adds a progress bar to give the user some visual feedback whenever there are offloaded params.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


Comments:
Update depth estimation task guide
Author: merveenoyan
Pull request: yes

Update depth estimation task guide with DAv2 and ZoeDepth @NielsRogge 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31860). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Add gemma 2
Author: ArthurZucker
Pull request: yes
Labels: name: run-slow

# What does this PR do?
Adds support for gemma2


Comments:
Please excuse this if I'm just not reading the code correctly (?), but I'm struggling to understand the intended function of the hybrid cache. [Here, in modeling_gemma2.py](https://github.com/huggingface/transformers/blob/cf85e86e9ad22ddd02c519d21c4783e3dbf72178/src/transformers/models/gemma2/modeling_gemma2.py#L244) as well as the two other attention functions, the past keys/values are updated like so:

```python
        if past_key_value is not None:
            # sin and cos are specific to RoPE models; cache_position needed for the static cache
            cache_kwargs = {
                "sin": sin,
                "cos": cos,
                "sliding_window": self.sliding_window,
                "cache_position": cache_position,
            }
            key_states, value_states = past_key_value.update(key_states, value_states, self.layer_idx, cache_kwargs)
```

While the function definition for `HybridCache.update` looks like:

```python
    def update(
        self,
        key_states: torch.Tensor,
        value_states: torch.Tensor,
        layer_idx: int,
        cache_kwargs: Optional[Dict[str, Any]] = None,
        sliding_window: Optional[int] = None,
    ) -> Tuple[torch.Tensor]:
        cache_position = cache_kwargs.get("cache_position")
        self.key_cache[layer_idx] = self.key_cache[layer_idx].to(device=key_states.device)
        self.value_cache[layer_idx] = self.value_cache[layer_idx].to(device=value_states.device)
        k_out = self.key_cache[layer_idx]
        v_out = self.value_cache[layer_idx]
        if sliding_window:
            update_fn = self._sliding_update
        else:
            update_fn = self._static_update
    ...
```

It doesn't read the `sliding_window` argument from the kwargs, so the default value is always used and the `_sliding_update` function is never selected, even on layers that use a `sliding_window`. Is this right?

that is a good catch and not intended, we will update this and fix it. The typo is that we should either pass the sliding windo directly, or get the sliding window form the cache kwargs. I think this stems from a will to make it compile compatible, and a type!
Revert "Fix `_init_weights` for `ResNetPreTrainedModel`"
Author: ydshieh
Pull request: yes

Reverts huggingface/transformers#31851

More failures when merged into `main`. I probably made a mistake and didn't trigger a full run on t his PR. Sorry

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31868). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
add warning when using gradient_checkpointing with FSDP full shard
Author: yundai424
Pull request: yes

# What does this PR do?


Add a warning when using FSDP full shard with `gradient_checkpointing` training arg to encourage users to use fsdp config's `activation_checkpointing` instead.

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes #30404


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.
@SunMarc  @muellerzr


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31578). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
fix: Fixed a `raise` statement
Author: Sai-Suraj-27
Pull request: yes

# What does this PR do?
`raise` must be followed by an exception instance or an exception class. Raising a literal (like a `str` in this case) will raise a `TypeError` at runtime and here we are not printing the message also. So, I have updated the code accordingly.

https://github.com/huggingface/transformers/blob/c63f15890377530044bcd2574e3e672907a5152a/src/transformers/models/whisper/convert_openai_to_hf.py#L350

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@amyeroberts


Comments: