TypeError: ModernBertModel.forward() got an unexpected keyword argument 'num_items_in_batch'
Author: Bachstelze
Pull request: no
Labels: name: bug

### System Info

Reopening: https://github.com/huggingface/transformers/issues/35838

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

see https://github.com/huggingface/transformers/issues/35838

### Expected behavior

see https://github.com/huggingface/transformers/issues/35838

Comments:
cc @muellerzr , but also @Bachstelze have you tried installing the latest version from `main` to see if that fixes it?

I installed transformers with those lines after the PR:
```
git clone https://github.com/huggingface/transformers.git
cd transformers
pip install .
```

Hmm, okay, seems like the issue still exists then!

@Bachstelze Can you please provide the full stack trace and code you're running for this instance?

@Pappasad also mentions that TrOCR doesn't forward
the stack trace for modernBERT as encoder in an encoder-decoder model from this [colab notebook](https://colab.research.google.com/drive/1xWER80KU1OGptLkHNH80eAgcM3R0zihb?usp=sharing):
```
TypeError                                 Traceback (most recent call last)

[<ipython-input-9-a71da896f123>](https://localhost:8080/#) in <cell line: 0>()
    131     optimizers=(adam, lr_scheduler)
    132 )
--> 133 trainer.train()
    134 print("training finished", flush=True)
    135 #wandb.finish()

8 frames

[/usr/local/lib/python3.11/dist-packages/transformers/trainer.py](https://localhost:8080/#) in train(self, resume_from_checkpoint, trial, ignore_keys_for_eval, **kwargs)
   2183                 hf_hub_utils.enable_progress_bars()
   2184         else:
-> 2185             return inner_training_loop(
   2186                 args=args,
   2187                 resume_from_checkpoint=resume_from_checkpoint,

[/usr/local/lib/python3.11/dist-packages/transformers/trainer.py](https://localhost:8080/#) in _inner_training_loop(self, batch_size, args, resume_from_checkpoint, trial, ignore_keys_for_eval)
   2489                     )
   2490                     with context():
-> 2491                         tr_loss_step = self.training_step(model, inputs, num_items_in_batch)
   2492 
   2493                     if (

[/usr/local/lib/python3.11/dist-packages/transformers/trainer.py](https://localhost:8080/#) in training_step(self, model, inputs, num_items_in_batch)
   3608 
   3609         with self.compute_loss_context_manager():
-> 3610             loss = self.compute_loss(model, inputs, num_items_in_batch=num_items_in_batch)
   3611 
   3612         del inputs

[/usr/local/lib/python3.11/dist-packages/transformers/trainer.py](https://localhost:8080/#) in compute_loss(self, model, inputs, return_outputs, num_items_in_batch)
   3669                 loss_kwargs["num_items_in_batch"] = num_items_in_batch
   3670             inputs = {**inputs, **loss_kwargs}
-> 3671         outputs = model(**inputs)
   3672         # Save past state if it exists
   3673         # TODO: this needs to be fixed and made cleaner later.

[/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py](https://localhost:8080/#) in _wrapped_call_impl(self, *args, **kwargs)
   1734             return self._compiled_call_impl(*args, **kwargs)  # type: ignore[misc]
   1735         else:
-> 1736             return self._call_impl(*args, **kwargs)
   1737 
   1738     # torchrec tests the code consistency with the following code

[/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py](https://localhost:8080/#) in _call_impl(self, *args, **kwargs)
   1745                 or _global_backward_pre_hooks or _global_backward_hooks
   1746                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1747             return forward_call(*args, **kwargs)
   1748 
   1749         result = None

[/usr/local/lib/python3.11/dist-packages/transformers/models/encoder_decoder/modeling_encoder_decoder.py](https://localhost:8080/#) in forward(self, input_ids, attention_mask, decoder_input_ids, decoder_attention_mask, encoder_outputs, past_key_values, inputs_embeds, decoder_inputs_embeds, labels, use_cache, output_attentions, output_hidden_states, return_dict, **kwargs)
    601 
    602         if encoder_outputs is None:
--> 603             encoder_outputs = self.encoder(
    604                 input_ids=input_ids,
    605                 attention_mask=attention_mask,

[/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py](https://localhost:8080/#) in _wrapped_call_impl(self, *args, **kwargs)
   1734             return self._compiled_call_impl(*args, **kwargs)  # type: ignore[misc]
   1735         else:
-> 1736             return self._call_impl(*args, **kwargs)
   1737 
   1738     # torchrec tests the code consistency with the following code

[/usr/local/lib/python3.11/dist-packages/torch/nn/modules/module.py](https://localhost:8080/#) in _call_impl(self, *args, **kwargs)
   1745                 or _global_backward_pre_hooks or _global_backward_hooks
   1746                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1747             return forward_call(*args, **kwargs)
   1748 
   1749         result = None

TypeError: ModernBertModel.forward() got an unexpected keyword argument 'num_items_in_batch'
```

@Bachstelze can you try via `pip install git+https://github.com/huggingface/transformers@muellerzr-more-models-sadface`

still the same error

@muellerzr don't make a sad face, take your time and enjoy your weekend! ;)

@muellerzr I have a similar issue with Donut

`DonutSwinModel.forward() got an unexpected keyword argument 'num_items_in_batch'`

Looking at the linked PR, does it also fix the Donut model? 

I'm quite new to this, but the changes in the PR are for the `encoder_decoder/modeling_encoder_decoder.py` while Donut seems to be using `vision_encoder_decoder/modeling_vision_encoder_decoder.py`

@muellerzr also hit a similar issue for DistilBert upgrading from 4.46.3 -> 4.49.0
```
def _call_impl(self, *args, **kwargs):
        forward_call = (self._slow_forward if torch._C._get_tracing_state() else self.forward)
        # If we don't have any hooks, we want to skip the rest of the logic in
        # this function, and just call forward.
        if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
                or _global_backward_pre_hooks or _global_backward_hooks
                or _global_forward_hooks or _global_forward_pre_hooks):
>           return forward_call(*args, **kwargs)
E           TypeError: DistilBertForSequenceClassification.forward() got an unexpected keyword argument 'num_items_in_batch'

../pip_deps_torch/site-packages/torch/nn/modules/module.py:1520: TypeError
```

@muellerzr I get the error `ViModel.forward() got an unexpected keyword argument 'num_items_in_batch'` with TrOCR even using your approach:

> [@Bachstelze](https://github.com/Bachstelze) can you try via `pip install git+https://github.com/huggingface/transformers@muellerzr-more-models-sadface`



I got same error after update my transformer's libs

> I got same error after update my transformer's libs

@zheka77111 What was the previous version you used? Did that version work for you?

same error `DeiTModel.forward() got an unexpected keyword argument 'num_items_in_batch'` with transformers == 4.49.0.dev0

![Image](https://github.com/user-attachments/assets/e6cab9d4-fd46-42dc-bb09-a12363d30d03)

> > I got same error after update my transformer's libs
> 
> [@zheka77111](https://github.com/zheka77111) What was the previous version you used? Did that version work for you?

4.36, but now I backup to it and get same error

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

This issue still bugs. Why can't it be reopened?

@Bachstelze can you install the latest version from `main` and confirm it's still occurring? Use `pip install git+https://github.com/huggingface/transformers.git`

cc @SunMarc 

I used` pip install git+https://github.com/huggingface/transformers.git` and I am still getting the bug `"TypeError: DonutSwinModel.forward() got an unexpected keyword argument 'num_items_in_batch'` . I am working with the Donut Model.

Can you share a minimal reproducer @waceke-kc ? 

I have the same error when using qwen2.5-vl-3b-Instruct-unsloth-bnb-4bit and with transformers 4.51.0 to 4.53.0.dev0


```python
import unsloth
from unsloth import FastVisionModel # FastLanguageModel for LLMs
import torch
model, tokenizer = FastVisionModel.from_pretrained(
    "unsloth/Qwen2.5-VL-3B-Instruct-unsloth-bnb-4bit",
    load_in_4bit = True, 
    # Use 4bit to reduce memory use. False for 16bit LoRA.
    trust_remote_code=True
)
model = FastVisionModel.get_peft_model(
    model,
    finetune_vision_layers     = True, # False if not finetuning vision layers
    finetune_language_layers   = True, # False if not finetuning language layers
    finetune_attention_modules = True, # False if not finetuning attention layers
    finetune_mlp_modules       = True, # False if not finetuning MLP layers

    r = 16,           # The larger, the higher the accuracy, but might overfit
    lora_alpha = 16,  # Recommended alpha == r at least
    lora_dropout = 0,
    bias = "none",
    random_state = 3407,
    use_rslora = False,  # We support rank stabilized LoRA
    loftq_config = None, # And LoftQ
    # target_modules = "all-linear", # Optional now! Can specify a list if needed
    device= "cuda"
)

from unsloth import is_bf16_supported
from unsloth.trainer import UnslothVisionDataCollator
from trl import SFTTrainer, SFTConfig

FastVisionModel.for_training(model) # Enable for training!

trainer = SFTTrainer(
    model = model,
    tokenizer = tokenizer,
    data_collator = UnslothVisionDataCollator(model, tokenizer), # Must use!
    train_dataset = final_converted_dataset,
    args = SFTConfig(
        per_device_train_batch_size = 4,
        gradient_accumulation_steps = 4,
        warmup_steps = 5,
        # max_steps = 30,
        # num_train_epochs = 1, # Set this instead of max_steps for full training runs
        learning_rate = 2e-4,
        fp16 = not is_bf16_supported(),
        bf16 = is_bf16_supported(),
        logging_steps = 100,
        optim = "adamw_8bit",
        weight_decay = 0.01,
        lr_scheduler_type = "linear",
        seed = 3407,
        output_dir = output_name,
        report_to = "none",     # For Weights and Biases

        # You MUST put the below items for vision finetuning:
        remove_unused_columns = False,
        dataset_text_field = "",
        dataset_kwargs = {"skip_prepare_dataset": True},
        dataset_num_proc = 4,
        max_seq_length = 2048,
        save_steps = 100,
        save_total_limit = 3,
        # load_best_model_at_end = True,
        # evaluation_strategy = 'steps'
    ),
)
trainer_stats = trainer.train() 

```
the error full track is the following:
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[22], line 1
----> 1 trainer_stats = trainer.train() 
      2 # trainer.train(resume_from_checkpoint=True)

File ~/miniconda3/envs/usul/lib/python3.12/site-packages/transformers/trainer.py:2245, in Trainer.train(self, resume_from_checkpoint, trial, ignore_keys_for_eval, **kwargs)
   2243         hf_hub_utils.enable_progress_bars()
   2244 else:
-> 2245     return inner_training_loop(
   2246         args=args,
   2247         resume_from_checkpoint=resume_from_checkpoint,
   2248         trial=trial,
   2249         ignore_keys_for_eval=ignore_keys_for_eval,
   2250     )

File ~/miniconda3/envs/usul/lib/python3.12/site-packages/transformers/trainer.py:2560, in Trainer._inner_training_loop(self, batch_size, args, resume_from_checkpoint, trial, ignore_keys_for_eval)
   2553 context = (
   2554     functools.partial(self.accelerator.no_sync, model=model)
   2555     if i != len(batch_samples) - 1
   2556     and self.accelerator.distributed_type != DistributedType.DEEPSPEED
   2557     else contextlib.nullcontext
   2558 )
   2559 with context():
-> 2560     tr_loss_step = self.training_step(model, inputs, num_items_in_batch)
   2562 if (
   2563     args.logging_nan_inf_filter
   2564     and not is_torch_xla_available()
   2565     and (torch.isnan(tr_loss_step) or torch.isinf(tr_loss_step))
   2566 ):
   2567     # if loss is nan or inf simply add the average of previous logged losses
   2568     tr_loss = tr_loss + tr_loss / (1 + self.state.global_step - self._globalstep_last_logged)

File <string>:31, in _unsloth_training_step(self, model, inputs, num_items_in_batch)

File ~/kobo/usul/usul_ocr/unsloth_compiled_cache/UnslothSFTTrainer.py:748, in _UnslothSFTTrainer.compute_loss(self, model, inputs, return_outputs, num_items_in_batch)
    747 def compute_loss(self, model, inputs, return_outputs = False, num_items_in_batch = None):
--> 748     outputs = super().compute_loss(
    749         model,
    750         inputs,
    751         return_outputs = return_outputs,
    752         num_items_in_batch = num_items_in_batch,
    753     )
    754     return outputs

File ~/miniconda3/envs/usul/lib/python3.12/site-packages/unsloth/models/_utils.py:1043, in _unsloth_pre_compute_loss(self, model, inputs, *args, **kwargs)
   1037     logger.warning_once(
   1038         f"Unsloth: Not an error, but {name} does not accept `num_items_in_batch`.\n"\
   1039         "Using gradient accumulation will be very slightly less accurate.\n"\
   1040         "Read more on gradient accumulation issues here: https://unsloth.ai/blog/gradient"
   1041     )
   1042 pass
-> 1043 outputs = self._old_compute_loss(model, inputs, *args, **kwargs)
   1044 return outputs

File ~/miniconda3/envs/usul/lib/python3.12/site-packages/transformers/trainer.py:3801, in Trainer.compute_loss(self, model, inputs, return_outputs, num_items_in_batch)
   3799         loss_kwargs["num_items_in_batch"] = num_items_in_batch
   3800     inputs = {**inputs, **loss_kwargs}
-> 3801 outputs = model(**inputs)
   3802 # Save past state if it exists
   3803 # TODO: this needs to be fixed and made cleaner later.
   3804 if self.args.past_index >= 0:

File ~/miniconda3/envs/usul/lib/python3.12/site-packages/torch/nn/modules/module.py:1751, in Module._wrapped_call_impl(self, *args, **kwargs)
   1749     return self._compiled_call_impl(*args, **kwargs)  # type: ignore[misc]
   1750 else:
-> 1751     return self._call_impl(*args, **kwargs)

File ~/miniconda3/envs/usul/lib/python3.12/site-packages/torch/nn/modules/module.py:1762, in Module._call_impl(self, *args, **kwargs)
   1757 # If we don't have any hooks, we want to skip the rest of the logic in
   1758 # this function, and just call forward.
   1759 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1760         or _global_backward_pre_hooks or _global_backward_hooks
   1761         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1762     return forward_call(*args, **kwargs)
   1764 result = None
   1765 called_always_called_hooks = set()

File ~/miniconda3/envs/usul/lib/python3.12/site-packages/accelerate/utils/operations.py:818, in convert_outputs_to_fp32.<locals>.forward(*args, **kwargs)
    817 def forward(*args, **kwargs):
--> 818     return model_forward(*args, **kwargs)

File ~/miniconda3/envs/usul/lib/python3.12/site-packages/accelerate/utils/operations.py:806, in ConvertOutputsToFp32.__call__(self, *args, **kwargs)
    805 def __call__(self, *args, **kwargs):
--> 806     return convert_to_fp32(self.model_forward(*args, **kwargs))

File ~/miniconda3/envs/usul/lib/python3.12/site-packages/torch/amp/autocast_mode.py:44, in autocast_decorator.<locals>.decorate_autocast(*args, **kwargs)
     41 @functools.wraps(func)
     42 def decorate_autocast(*args, **kwargs):
     43     with autocast_instance:
---> 44         return func(*args, **kwargs)

File ~/miniconda3/envs/usul/lib/python3.12/site-packages/peft/peft_model.py:1757, in PeftModelForCausalLM.forward(self, input_ids, attention_mask, inputs_embeds, labels, output_attentions, output_hidden_states, return_dict, task_ids, **kwargs)
   1755     with self._enable_peft_forward_hooks(**kwargs):
   1756         kwargs = {k: v for k, v in kwargs.items() if k not in self.special_peft_forward_args}
-> 1757         return self.base_model(
   1758             input_ids=input_ids,
   1759             attention_mask=attention_mask,
   1760             inputs_embeds=inputs_embeds,
   1761             labels=labels,
   1762             output_attentions=output_attentions,
   1763             output_hidden_states=output_hidden_states,
   1764             return_dict=return_dict,
   1765             **kwargs,
   1766         )
   1768 batch_size = _get_batch_size(input_ids, inputs_embeds)
   1769 if attention_mask is not None:
   1770     # concat prompt attention mask

File ~/miniconda3/envs/usul/lib/python3.12/site-packages/torch/nn/modules/module.py:1751, in Module._wrapped_call_impl(self, *args, **kwargs)
   1749     return self._compiled_call_impl(*args, **kwargs)  # type: ignore[misc]
   1750 else:
-> 1751     return self._call_impl(*args, **kwargs)

File ~/miniconda3/envs/usul/lib/python3.12/site-packages/torch/nn/modules/module.py:1762, in Module._call_impl(self, *args, **kwargs)
   1757 # If we don't have any hooks, we want to skip the rest of the logic in
   1758 # this function, and just call forward.
   1759 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1760         or _global_backward_pre_hooks or _global_backward_hooks
   1761         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1762     return forward_call(*args, **kwargs)
   1764 result = None
   1765 called_always_called_hooks = set()

File ~/miniconda3/envs/usul/lib/python3.12/site-packages/peft/tuners/tuners_utils.py:193, in BaseTuner.forward(self, *args, **kwargs)
    192 def forward(self, *args: Any, **kwargs: Any):
--> 193     return self.model.forward(*args, **kwargs)

TypeError: Qwen2_5_VLForConditionalGeneration.forward() got an unexpected keyword argument 'num_items_in_batch'



@SunMarc , I was using this script from [Phil Schmid](https://www.philschmid.de/fine-tuning-donut) but then I realized that even though I am using pip install git+https://github.com/huggingface/transformers.git locally, the Sagemaker estimator ends up using another version. That is what is causing the issue for me. I ended up using version 4.46 on my Sagemaker estimator code. 

Thanks for the reproducer @abdelkareemkobo ! This is something that unsloth needs to fix in their codebase, so I would recommend opening an issue there. THe issue is that they patched `get_batch_samples` with `_unsloth_get_batch_samples`. However, `_unsloth_get_batch_samples` doesn't take into account when we pass a peft model, unlike in transformers where we check the right forward to decide if we need to calculate `num_items_in_batch`. 

```python
        model_forward = (
            unwrapped_model.forward
            if not _is_peft_model(unwrapped_model)
            else unwrapped_model.get_base_model().forward
        )
        forward_params = inspect.signature(model_forward).parameters
```

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.