Repository: ggml-org/llama.cpp
Pull request: mtmd : fix LightOnOCR image preprocessing

Uses longest edge bicubic resizing instead of the previous 1024 token limit and bilinear resizing. This is in line with the LightOnOCR
[processor_config.json](https://huggingface.co/lightonai/LightOnOCR-2-1B/blob/main/processor_config.json)
```json
    "resample": 3,
    "size": {
      "longest_edge": 1540
    }
```

This fixes the output with the [LightOnOCR-2-1B-bbox](https://huggingface.co/lightonai/LightOnOCR-2-1B-bbox) model (using the GGUF by [noctrex](https://huggingface.co/noctrex/LightOnOCR-2-1B-bbox-GGUF))

<details>
  <summary>Before (boxes)</summary>
  
<img width="1700" height="2200" alt="university-physics-page745-before-hf" src="https://github.com/user-attachments/assets/31c79e04-aae5-4fd4-b66b-92f5d17b1bd0" />

</details>

<details>
  <summary>Before (markdown)</summary>

```md

shock absorbers.

![image](image_1.png)277,70,670,243

**FIGURE 15.24** To counteract damping forces, you need to keep pumping a swing. (credit: Bob Mical)

**Figure 15.25** shows a mass $m$ attached to a spring with a force constant $k$ . The mass is raised to a position $A_0$ , the initial amplitude, and then released. The mass oscillates around the equilibrium position in a fluid with viscosity but the amplitude decreases for each oscillation. For a system that has a small amount of damping, the period and frequency are constant and are nearly the same as for SHM, but the amplitude gradually decreases as shown. This occurs because the non-conservative damping force removes energy from the system, usually in the form of thermal energy.

![image](image_2.png)240,295,740,465

**FIGURE 15.25** For a mass on a spring oscillating in a viscous fluid, the period remains constant, but the amplitudes of the oscillations decrease due to the damping caused by the fluid.

Consider the forces acting on the mass. Note that the only contribution of the weight is to change the equilibrium position, as discussed earlier in the chapter. Therefore, the net force is equal to the force of the spring and the damping force ( $F_D$ ). If the magnitude of the velocity is small, meaning the mass oscillates slowly, the damping force is proportional to the velocity and acts against the direction of motion ( $F_D = bv$ ). The net force on the mass is therefore

$$
ma = -bv - kx.
$$

Writing this as a differential equation in $x$ , we obtain

$$
\frac{d^2x}{dt^2} + b\frac{dx}{dt} + kx = 0.
\quad \text{15.23}
$$

To determine the solution to this equation, consider the plot of position versus time shown in **Figure 15.26**. The curve resembles a cosine curve oscillating in the envelope of an exponential function $A_0 e^{-\alpha t}$ where $\alpha = \frac{b}{2m}$ . The solution is

```  

</details>


<details>
  <summary>After (boxes)</summary>
  
<img width="1700" height="2200" alt="university-physics-page745-after-hf" src="https://github.com/user-attachments/assets/d232a255-25e7-478d-9f21-b33a3a668cc6" />

</details>

<details>
  <summary>After (markdown)</summary>

```md

shock absorbers.

![image](image_1.png)310,103,700,295

**FIGURE 15.24** To counteract dampening forces, you need to keep pumping a swing. (credit: Bob Mical)

**Figure 15.25** shows a mass $m$ attached to a spring with a force constant $k$ . The mass is raised to a position $A_0$ , the initial amplitude, and then released. The mass oscillates around the equilibrium position in a fluid with viscosity but the amplitude decreases for each oscillation. For a system that has a small amount of damping, the period and frequency are constant and are nearly the same as for SHM, but the amplitude gradually decreases as shown. This occurs because the non-conservative damping force removes energy from the system, usually in the form of thermal energy.

![image](image_2.png)270,420,710,645

**FIGURE 15.25** For a mass on a spring oscillating in a viscous fluid, the period remains constant, but the amplitudes of the oscillations decrease due to the damping caused by the fluid.

Consider the forces acting on the mass. Note that the only contribution of the weight is to change the equilibrium position, as discussed earlier in the chapter. Therefore, the net force is equal to the force of the spring and the damping force ( $F_D$ ). If the magnitude of the velocity is small, meaning the mass oscillates slowly, the damping force is proportional to the velocity and acts against the direction of motion ( $F_D = -bv$ ). The net force on the mass is therefore

$$
ma = -bv - kx.
$$

Writing this as a differential equation in $x$ , we obtain

$$
m\frac{d^2x}{dt^2} + b\frac{dx}{dt} + kx = 0.
\quad \text{15.23}
$$

To determine the solution to this equation, consider the plot of position versus time shown in **Figure 15.26**. The curve resembles a cosine curve oscillating in the envelope of an exponential function $A_0e^{-\alpha t}$ where $\alpha = \frac{b}{2m}$ . The solution is


```
</details>

<details>
  <summary>Input file</summary>

<img width="1700" height="2200" alt="page-745" src="https://github.com/user-attachments/assets/2888657d-d8b8-448c-bcea-93a6063c9ffd" />
</details>

The example files are from [University Physics Volume 1](https://openstax.org/details/books/university-physics-volume-1) by OpenStax  licensed under [Creative Commons Attribution License v4.0](https://creativecommons.org/licenses/by/4.0/deed.en)


Demo script: https://gist.github.com/DorianRudolph/4c23020d87e7dac5441576b142b691f9

AI usage: I used codex with gpt-5.4 xhigh to find the cause of the issue and prepare the demo script.

Changed files:
filename: tools/mtmd/clip.cpp
patch: @@ -1161,7 +1161,6 @@ struct clip_model_loader {
                         hparams.set_warmup_n_tokens(16*16);
                     } break;
                 case PROJECTOR_TYPE_PIXTRAL:
-                case PROJECTOR_TYPE_LIGHTONOCR:
                     {
                         // ref: https://huggingface.co/mistral-community/pixtral-12b/blob/main/preprocessor_config.json
                         // TODO: verify the image_min_tokens
@@ -1171,6 +1170,15 @@ struct clip_model_loader {
                         hparams.set_limit_image_tokens(8, 1024);
                         hparams.set_warmup_n_tokens(256); // avoid OOM on warmup
                     } break;
+                case PROJECTOR_TYPE_LIGHTONOCR:
+                    {
+                        hparams.n_merge = 1;
+                        hparams.rope_theta = 10000.0f;
+                        get_u32(KEY_SPATIAL_MERGE_SIZE, hparams.n_merge, false);
+                        hparams.image_longest_edge = hparams.image_size;
+                        get_u32(KEY_PREPROC_IMAGE_SIZE, hparams.image_longest_edge, false);
+                        hparams.set_warmup_n_tokens(256); // avoid OOM on warmup
+                    } break;
                 case PROJECTOR_TYPE_KIMIVL:
                     {
                         hparams.rope_theta = 10000.0f;
@@ -3180,7 +3188,6 @@ bool clip_image_preprocess(struct clip_ctx * ctx, const clip_image_u8 * img, str
 
         case PROJECTOR_TYPE_PHI4:
         case PROJECTOR_TYPE_PIXTRAL:
-        case PROJECTOR_TYPE_LIGHTONOCR:
             {
                 GGML_ASSERT(params.image_min_pixels > 0 && params.image_max_pixels > 0);
                 clip_image_u8 resized_image;
@@ -3196,6 +3203,19 @@ bool clip_image_preprocess(struct clip_ctx * ctx, const clip_image_u8 * img, str
                 normalize_image_u8_to_f32(resized_image, *img_f32, params.image_mean, params.image_std);
                 res_imgs->entries.push_back(std::move(img_f32));
             } break;
+        case PROJECTOR_TYPE_LIGHTONOCR:
+            {
+                GGML_ASSERT(params.image_longest_edge > 0);
+                clip_image_u8 resized_image;
+                const clip_image_size target_size = img_tool::calc_size_preserved_ratio(
+                    original_size,
+                    params.patch_size * params.n_merge,
+                    params.image_longest_edge);
+                img_tool::resize(*img, resized_image, target_size, img_tool::RESIZE_ALGO_BICUBIC);
+                clip_image_f32_ptr img_f32(clip_image_f32_init());
+                normalize_image_u8_to_f32(resized_image, *img_f32, params.image_mean, params.image_std);
+                res_imgs->entries.push_back(std::move(img_f32));
+            } break;
 
         case PROJECTOR_TYPE_LLAMA4:
             {