Add LightOnOCR model implementation
Author: baptiste-aubertin
Pull request: yes
Labels: name: New model

name: Vision





# What does this PR do?

Implementation of the LightonOCR model following Modular Transformers architecture.

Our model is a 1B parameter OCR model using Pixtral as the vision encoder and Qwen3 as the LLM decoder.

I still have an issue with auto configuration, which I'm not familiar with:

`🚨 Config not found for lightonocr. You can manually add it to HARDCODED_CONFIG_FOR_MODELS in utils/auto_docstring.py`


<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker @Cyrilvallez
- vision models: @yonigozlan @molbap
- audio models: @eustlb @ebezzam @vasqu
- multimodal models: @zucchini-nlp
- graph models: @clefourrier

Library:

- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- continuous batching: @remi-or @ArthurZucker @McPatate
- pipelines: @Rocketknight1
- tokenizers: @ArthurZucker and @itazap
- trainer: @SunMarc
- attention: @vasqu @ArthurZucker @CyrilVallez
- model loading (from pretrained, etc): @CyrilVallez
- distributed: @3outeille @ArthurZucker
- CIs: @ydshieh

Integrations:

- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization: @SunMarc @MekkCyber
- kernels: @MekkCyber @drbh
- peft: @BenjaminBossan @githubnemo

Devices/Backends:

- AMD ROCm: @ivarflakstad
- Intel XPU: @IlyasMoutawwakil
- Ascend NPU: @ivarflakstad 

Documentation: @stevhliu

Research projects are not maintained and should be taken as is.

 -->


Comments:
Hi @molbap  @zucchini-nlp ! Thanks for the detailed review and the instructions!
Sorry I didn't get back to you last week, I was a little overloaded with the release. 

i went through you comment and I'm still running into test issues. The problem is that modular is trying to reimplement a common attention mechanism for both the Pixtral vision model and the Qwen3 text model, and this is causing some conflicts. Have you dealt with this before? Not sure how to handle that :( 

Hi I think all the tests passed and i rebased the branch :). Please tell me if I need to do something else
Thanks a lot for your help !

run-slow: lightonocr

This comment contains run-slow, running the specified jobs: 

models: ['models/lightonocr']
quantizations: [] ...

There's a couple errors on the `slow tests` I just ran: with 

```python
FAILED tests/models/lightonocr/test_modeling_lightonocr.py::LightOnOCRForConditionalGenerationIntegrationTest::test_model_can_generate_without_images - AttributeError: 'LightOnOCRConfig' object has no attribute 'vocab_size'
FAILED tests/models/lightonocr/test_modeling_lightonocr.py::LightOnOCRForConditionalGenerationIntegrationTest::test_model_forward_with_images - AttributeError: 'LightOnOCRConfig' object has no attribute 'vocab_size'
```

so just a config key missing/a wrong access IMO. The other one which is suspicious is

```python
    def vision_apply_rotary_pos_emb(q, k, cos, sin, position_ids=None, unsqueeze_dim=1):
        """Applies Rotary Position Embedding to the query and key tensors.
        .......
        """
        cos = cos.unsqueeze(unsqueeze_dim)
        sin = sin.unsqueeze(unsqueeze_dim)
>       q_embed = (q * cos) + (vision_rotate_half(q) * sin)
E       RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:1!
```

which is on a multi-GPU setting only but means that a q tensor should be created on a safer device, or at least within this util function to be safe. 

> There's a couple errors on the `slow tests` I just ran: with
> 
> ```python
> FAILED tests/models/lightonocr/test_modeling_lightonocr.py::LightOnOCRForConditionalGenerationIntegrationTest::test_model_can_generate_without_images - AttributeError: 'LightOnOCRConfig' object has no attribute 'vocab_size'
> FAILED tests/models/lightonocr/test_modeling_lightonocr.py::LightOnOCRForConditionalGenerationIntegrationTest::test_model_forward_with_images - AttributeError: 'LightOnOCRConfig' object has no attribute 'vocab_size'
> ```
> 
> so just a config key missing/a wrong access IMO. The other one which is suspicious is
> 
> ```python
>     def vision_apply_rotary_pos_emb(q, k, cos, sin, position_ids=None, unsqueeze_dim=1):
>         """Applies Rotary Position Embedding to the query and key tensors.
>         .......
>         """
>         cos = cos.unsqueeze(unsqueeze_dim)
>         sin = sin.unsqueeze(unsqueeze_dim)
> >       q_embed = (q * cos) + (vision_rotate_half(q) * sin)
> E       RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:1!
> ```
> 
> which is on a multi-GPU setting only but means that a q tensor should be created on a safer device, or at least within this util function to be safe.

i added vocab getter and put the q and k on the same device as cos and sin :) 

Hi @molbap thanks again for your help ! Could you please re trigger the slow tests to be sure that everything pass before the intervention of the last reviewer :) ? Thanks 

run-slow: lightonocr

This comment contains `run-slow`, running the specified jobs: 

models: ["models/lightonocr"]
quantizations: []

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/19101201521)

✅ No failing test specific to this PR 🎉 !


> Hye @baptiste-aubertin , While we're waiting on core maintainer's review, I left a few comments and questions below, mostly about things that looks redundant to me. LMK what you think and if we can delete those?

Hi @zucchini-nlp thanks you ! This time I will try to answer rapidly 😅 

run-slow: lightonocr

This comment contains `run-slow`, running the specified jobs: 

models: ["models/lightonocr"]
quantizations: []

The run-slow command is changing, do not worry about the CI result! Thanks for the fixup :) 
And really thanks for the fast iterations, makes a world of difference on model integrations. 

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/19131524529)

✅ No failing test specific to this PR 🎉 !


hey hey, there's just this 

```bash

Traceback (most recent call last):
  File "/home/runner/work/transformers/transformers/.venv/bin/doc-builder", line 10, in <module>
    sys.exit(main())
  File "/home/runner/work/transformers/transformers/.venv/lib/python3.10/site-packages/doc_builder/commands/doc_builder_cli.py", line 50, in main
    args.func(args)
  File "/home/runner/work/transformers/transformers/.venv/lib/python3.10/site-packages/doc_builder/commands/build.py", line 103, in build_command
    build_doc(
  File "/home/runner/work/transformers/transformers/.venv/lib/python3.10/site-packages/doc_builder/build_doc.py", line 392, in build_doc
    anchors_mapping, source_files_mapping = build_mdx_files(
  File "/home/runner/work/transformers/transformers/.venv/lib/python3.10/site-packages/doc_builder/build_doc.py", line 255, in build_mdx_files
    raise type(e)(f"There was an error when converting {file} to the MDX format.\n" + e.args[0]) from e
ValueError: There was an error when converting ../transformers/docs/source/en/model_doc/lightonocr.md to the MDX format.
Unable to find LightOnOCRTextConfig in transformers. Make sure the path to that object is correct.
```

It is likely caused by the missing config object in `__all__`? Other than that all looks good,cc @Cyrilvallez for core review as this is mergeable soon!

> hey hey, there's just this
> 
> ```shell
> Traceback (most recent call last):
>   File "/home/runner/work/transformers/transformers/.venv/bin/doc-builder", line 10, in <module>
>     sys.exit(main())
>   File "/home/runner/work/transformers/transformers/.venv/lib/python3.10/site-packages/doc_builder/commands/doc_builder_cli.py", line 50, in main
>     args.func(args)
>   File "/home/runner/work/transformers/transformers/.venv/lib/python3.10/site-packages/doc_builder/commands/build.py", line 103, in build_command
>     build_doc(
>   File "/home/runner/work/transformers/transformers/.venv/lib/python3.10/site-packages/doc_builder/build_doc.py", line 392, in build_doc
>     anchors_mapping, source_files_mapping = build_mdx_files(
>   File "/home/runner/work/transformers/transformers/.venv/lib/python3.10/site-packages/doc_builder/build_doc.py", line 255, in build_mdx_files
>     raise type(e)(f"There was an error when converting {file} to the MDX format.\n" + e.args[0]) from e
> ValueError: There was an error when converting ../transformers/docs/source/en/model_doc/lightonocr.md to the MDX format.
> Unable to find LightOnOCRTextConfig in transformers. Make sure the path to that object is correct.
> ```
> 
> It is likely caused by the missing config object in `__all__`? Other than that all looks good,cc @Cyrilvallez for core review as this is mergeable soon!

Hi i added missing configs to __all__ and i rebased :) I hope it will solve the issue ! 

Hi I rebased and adapted the tied weight to the new format :)
I also removed `_init_weights` methods who triggered test fail and seemed to be useless 

A test is not passing but it not seems to be related with our model. Could you please run the slow tests @molbap ?

run-slow: lightonocr

This comment contains `run-slow`, running the specified jobs: 

models: ["models/lightonocr"]
quantizations: []

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/19472508127)

### Model CI Report

#### ❌ Failed tests

- [lightonocr](https://github.com/huggingface/transformers/actions/runs/19472508127/job/55723602369):
    tests/models/lightonocr/test_modeling_lightonocr.py::LightOnOCRForConditionalGenerationModelTest::test_eager_padding_matches_padding_free_with_position_ids
    tests/models/lightonocr/test_modeling_lightonocr.py::LightOnOCRForConditionalGenerationIntegrationTest::test_model_can_generate_without_images
    tests/models/lightonocr/test_modeling_lightonocr.py::LightOnOCRForConditionalGenerationIntegrationTest::test_model_forward_with_images



There's just these:

```bash
 FAILED tests/models/lightonocr/test_modeling_lightonocr.py::LightOnOCRForConditionalGenerationModelTest::test_model_parallelism - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:1!
FAILED tests/models/lightonocr/test_modeling_lightonocr.py::LightOnOCRForConditionalGenerationIntegrationTest::test_model_can_generate_without_images - AttributeError: 'LightOnOCRConfig' object has no attribute 'vocab_size'
FAILED tests/models/lightonocr/test_modeling_lightonocr.py::LightOnOCRForConditionalGenerationIntegrationTest::test_model_forward_with_images - AttributeError: 'LightOnOCRConfig' object has no attribute 'vocab_size'
```

For the last two - thought it was solved? but for the first one it's an "actual" issue, although you should be able to fix it by either creating the tensor in the same device as the first one, or just make sure you don't have any lingering .to(device) as device_map="auto" should be able to handle the device placement!

hi @molbap , i fixed the tests. Can you rerun the slow one, the device test needs to be run on several gpus :) 

I re rabased and I still have tests not related to our model who are failing :/ 

hi I rabased this morning and all the tests are now passing 👍 😄 

We have had very good results fine-tuning on LightOnOCR, and are looking forward to this being merged so we can use it in production.

cc @molbap @zucchini-nlp, feel free to tag me when you believe this is ready for final review!

Hi @Cyrilvallez @molbap , I re rebased today. All the tests seems to pass on my side, can you trigger the slow tests ? :) 

run-slow: lightonocr

This comment contains `run-slow`, running the specified jobs: 

models: ["models/lightonocr"]
quantizations: []

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41621). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/20059722139)

✅ No failing test specific to this PR 🎉 !


Hi @Cyrilvallez I think we are ready for the final review  😄 🔥. Feel free to ping me here or on slack if you need :) 

@Cyrilvallez ready, CI is green 😄 

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, lightonocr

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=41621&sha=83d01f

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, lightonocr

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=41621&sha=8769e8

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, lightonocr

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=41621&sha=7c9f15

Hi @Cyrilvallez, I re rabased and fixed the last elements of @zucchini-nlp 

I can see that we have test failing but it doesn't seems to be related with my implementation :) 

The tokenization tests are failing on main as well, should be fixed soon :)

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=41621&sha=ba4a4a

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, lighton_ocr

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, lighton_ocr

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, lighton_ocr

Hi @Cyrilvallez thanks for the last review ! I correct most of the feedbacks and let comment to two of them that i have trouble with. I still need to update our tokenizer and a module name and it will be okay

I just rebased and all the tests are passing now 💯 

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, lighton_ocr

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=41621&sha=b87fe7

Hi @Cyrilvallez, I updated the tokenizer and the processor. I still have 3 pieces of feedback where I have trouble, I left you comments in your review.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, lighton_ocr

@baptiste-aubertin replied to the two comments and rebased main. Can you make sure that tests are passing and run `make fix-copies` to adjust modular conversion?



**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, lighton_ocr

Hi @molbap and @zucchini-nlp , thanks for your messages ! I resolved the last comments thanks to your messages @zucchini-nlp 😄 . And i also added a usage section in the model doc @molbap . I guess we are ready for the next @Cyrilvallez 

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=41621&sha=940318

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, lighton_ocr

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=41621&sha=bc1339

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, lighton_ocr

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=41621&sha=3a72ae

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, lighton_ocr

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=41621&sha=d8fc0f

run-slow: lighton_ocr

This comment contains `run-slow`, running the specified jobs: 

models: ["models/lighton_ocr"]
quantizations: []

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/20957408720)

### Model CI Report

#### ❌ Failed tests

- [lighton_ocr](https://github.com/huggingface/transformers/actions/runs/20957408720/job/60225786710):
    tests/models/lighton_ocr/test_modeling_lighton_ocr.py::LightOnOcrForConditionalGenerationModelTest::test_sdpa_can_dispatch_on_flash
    tests/models/lighton_ocr/test_modeling_lighton_ocr.py::LightOnOcrForConditionalGenerationIntegrationTest::test_lightonocr_ocr_integration



**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, lighton_ocr

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, lighton_ocr

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, lighton_ocr

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=41621&sha=ad88fc

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, lighton_ocr

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=41621&sha=c8215f

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, lighton_ocr

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, lighton_ocr

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=41621&sha=6c7f30

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, lighton_ocr

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=41621&sha=4739bc

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, lighton_ocr

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=41621&sha=ea55ec

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, lighton_ocr

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, lighton_ocr

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, lighton_ocr

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=41621&sha=c00004

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, lighton_ocr

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, lighton_ocr

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=41621&sha=f3e12d

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, lighton_ocr

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, lighton_ocr

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, lighton_ocr

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, lighton_ocr

run-slow: lighton_ocr

This comment contains `run-slow`, running the specified jobs: 

models: ["models/lighton_ocr"]
quantizations: []

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=41621&sha=a3e0fc

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/20993255398)

### Model CI Report

#### ❌ Failed tests

- [lighton_ocr](https://github.com/huggingface/transformers/actions/runs/20993255398/job/60344101052):
    tests/models/lighton_ocr/test_modeling_lighton_ocr.py::LightOnOcrForConditionalGenerationModelTest::test_sdpa_can_dispatch_on_flash
    tests/models/lighton_ocr/test_modeling_lighton_ocr.py::LightOnOcrForConditionalGenerationModelTest::test_sdpa_padding_matches_padding_free_with_position_ids
    tests/models/lighton_ocr/test_modeling_lighton_ocr.py::LightOnOcrForConditionalGenerationIntegrationTest::test_lightonocr_ocr_integration



**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, lighton_ocr, mistral3

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=41621&sha=033aa1

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, lighton_ocr, mistral3

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=41621&sha=edaf22

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, lighton_ocr, mistral3

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=41621&sha=c6b1dc