tensor mismatch when finetuning a smollm3 model
Author: codefusser
Pull request: no
Labels: name: bug

### System Info

When performing a fine-tuning job with batch size of 4 and max steps of 1000, it errors out with some tokenizer error

```
The tokenizer has new PAD/BOS/EOS tokens that differ from the model config and generation config. The model config and generation config were aligned accordingly, being updated with the tokenizer's values. Updated tokens: {'bos_token_id': None, 'pad_token_id': None}.


  0%|          | 0/500 [00:00<?, ?it/s]Traceback (most recent call last):

  File "/tmp/script.py", line 191, in <module>

    trainer.train()

  File "/root/.cache/uv/environments-v2/script-912247c0edd68a55/lib/python3.12/site-packages/transformers/trainer.py", line 2328, in train

    return inner_training_loop(

           ^^^^^^^^^^^^^^^^^^^^

  File "/root/.cache/uv/environments-v2/script-912247c0edd68a55/lib/python3.12/site-packages/transformers/trainer.py", line 2672, in _inner_training_loop

    tr_loss_step = self.training_step(model, inputs, num_items_in_batch)

                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/root/.cache/uv/environments-v2/script-912247c0edd68a55/lib/python3.12/site-packages/trl/trainer/sft_trainer.py", line 1189, in training_step

    return super().training_step(*args, **kwargs)

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/root/.cache/uv/environments-v2/script-912247c0edd68a55/lib/python3.12/site-packages/transformers/trainer.py", line 4009, in training_step

    loss = self.compute_loss(model, inputs, num_items_in_batch=num_items_in_batch)

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/root/.cache/uv/environments-v2/script-912247c0edd68a55/lib/python3.12/site-packages/trl/trainer/sft_trainer.py", line 1123, in compute_loss

    entropy = torch.sum(per_token_entropy * attention_mask) / attention_mask.sum()

                        ~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~

RuntimeError: The size of tensor a (4) must match the size of tensor b (8) at non-singleton dimension 0


  0%|          | 0/500 [00:01<?, ?it/s]
```
Please what might be wrong?

### Who can help?

_No response_

### Information

- [x] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

sft finetuning with a10-largex2 GPU

training parameters:

# Configure training
config = SFTConfig(
    output_dir="./smollm3-jobs-sft",
    per_device_train_batch_size=4,
    learning_rate=3e-5,
    max_steps=1000,
    logging_steps=50,
    save_steps=200,
    push_to_hub=True,
    hub_model_id="hubsnippetai/smollm3-jobs-sft"
)

# Train
trainer = SFTTrainer(
    model=model,
    train_dataset=processed_train,
    args=config,
)
trainer.train()

### Expected behavior

After loading the datasets, the job should continue running while training the model without erroring out.

Comments:
This looks like a tokenizer pad token issue. The tensor mismatch happens because SmolLM3 doesn't have a proper pad token configured.

**Try this fix:**
```
tokenizer.add_special_tokens({'pad_token': '[PAD]'})

model.resize_token_embeddings(len(tokenizer))

model.config.pad_token_id = tokenizer.pad_token_id

```
Also set `packing=False` in your SFTConfig and consider reducing batch size to 2 if the issue persists.

Happy Hackotoberfest2025!  I’d like to work on this issue. I’d start by ensuring the configs are synchronized and the tensors match in shape. 

Sounds good! I'm actually fine tuning with the transformer library but it seems that modification will require its pytorch implementation, which was built into the library. I should find the matching tokenizer parameter to adjust, though proper "padding" was in place.

I could share the source code, if you are interested.

Thanks

> Sounds good! I'm actually fine tuning with the transformer library but it seems that modification will require its pytorch implementation, which was built into the library. I should find the matching tokenizer parameter to adjust, though proper "padding" was in place.
> 
> I could share the source code, if you are interested.
> 
> Thanks

Moreover, the Hacktoberfest2025 is really an exciting one. Thanks for sharing 👍 

> Also set `packing=False` in your SFTConfig and consider reducing batch size to 2 if the issue persists.

this is a helpful solution to the tensor mismatch issue- Have you tried the suggestions from BKCODE2003 ? 

the ``` packing ``` option is ``` false ``` by default

https://huggingface.co/docs/trl/main/en/sft_trainer#trl.SFTTrainer

Hi, I changed the code block # Compute entropy to ensure that the dimensions of per_token_entropy and attention_mask match before multiplication, added a unit test and opened a PR. Hope this helps- cheers Katharina

Absolutely,  and I think padding could be used to match the dimensions of the tensors before multiplying.

Investigating is still ongoing as fine-tuning progresses with the ``` push_to_hub ``` option set to ``` True ```

please assign this issue

This issue was resolved by handling the chat template of the dataset implicitly using the map function.

However, the ideal pull request is to check for tensor shape before performing an operation in the ```trainer.py``` file.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.