Repository: ggml-org/llama.cpp
Pull request: clip: (minicpmv) fix resampler kq_scale

# Critical Bug: Incorrect `kq_scale` Calculation in MiniCPM-V Resampler Module

## Summary

The `clip_graph` class calculates `kq_scale` using Vision Encoder (ViT) hyperparameters, but this value is incorrectly reused in the Resampler module which has completely different architectural dimensions. This causes numerical discrepancies and accuracy degradation compared to the official Python implementation.

## Issue Location

- **File**: `tools/mtmd/clip.cpp`
- **Lines**: 540-545 (constructor) and 1195-1224 (resampler attention)

## Root Cause Analysis

### 1. In `clip_graph` Constructor (Lines 540-545)

```cpp
n_embd(hparams.n_embd),                    // e.g., 1152 (ViT embedding dim)
n_head(hparams.n_head),                    // e.g., 16 (ViT attention heads)
d_head(n_embd / n_head),                   // = 72 (ViT head dimension)
kq_scale(1.0f / sqrtf((float)d_head))      // = 1/sqrt(72) ≈ 0.1178 ❌ WRONG!
```

These hyperparameters come from **Vision Encoder config** loaded via GGUF metadata:
- `clip.vision.embedding_length` → `hparams.n_embd`
- `clip.vision.attention.head_count` → `hparams.n_head`

These are **input** dimensions, not projector output dimensions.

### 2. In Resampler Attention Module (Lines 1195-1224)

```cpp
const int n_embd_proj = clip_n_mmproj_embd(ctx);  // e.g., 2048 (projector output dim)
const int d_head = 128;                            // Hardcoded - correct for Resampler
int n_head = n_embd_proj / d_head;                 // e.g., 16 heads for Resampler

// Currently using wrong kq_scale from constructor:
// kq_scale = 0.1178 (based on ViT's d_head=72)
// Should be: 1.0f / sqrt(128) ≈ 0.0884
```

The Resampler uses hardcoded `d_head = 128`, which is the **correct dimension** for its multi-head attention, but it incorrectly uses `kq_scale` computed from ViT's `d_head = 72`.

## Observed Behavior

From debug output:
```
1152___________72
kq_scale: 0.117851___________0.0883883
```

- Left value (0.117851): Incorrect value from ViT parameters (`1/sqrt(72)`)
- Right value (0.0883883): Correct value for Resampler (`1/sqrt(128)`)

## Impact

1. **Numerical Discrepancies**: Incorrect scaling in attention scores affects softmax distribution
2. **Accuracy Degradation**: Vision encoder outputs don't match official Python implementation
3. **Model Performance**: Suboptimal inference quality for MiniCPM-V models

## Affected Models

This architectural pattern may affect multiple models:

- ✅ **MiniCPM-V series** (confirmed affected)
- ⚠️ **MobileVLM** (likely affected if resampler/projector dimensions differ from ViT)
- ⚠️ **Qwen2-VL** (may have similar issues)
- ⚠️ Any model where **projector dimensions** ≠ **vision encoder dimensions**

## Proposed Fix

### Option 1: Local Computation (Recommended)

Compute `kq_scale` locally in the Resampler module instead of reusing the class member:

```cpp
// In build_minicpmv(), around line 1195-1224:
{
    const int d_head = 128;
    int n_head = n_embd_proj / d_head;
    
    // Compute local kq_scale based on Resampler's d_head
    const float resampler_kq_scale = 1.0f / sqrtf((float)d_head);  // = 0.0884
    
    ggml_tensor * Q = ggml_add(ctx0,
        ggml_mul_mat(ctx0, model.mm_model_attn_q_w, q),
        model.mm_model_attn_q_b);
    ggml_tensor * K = ggml_add(ctx0,
        ggml_mul_mat(ctx0, model.mm_model_attn_k_w, k),
        model.mm_model_attn_k_b);
    ggml_tensor * V = ggml_add(ctx0,
        ggml_mul_mat(ctx0, model.mm_model_attn_v_w, v),
        model.mm_model_attn_v_b);

    Q = ggml_reshape_3d(ctx0, Q, d_head, n_head, num_query);
    K = ggml_reshape_3d(ctx0, K, d_head, n_head, n_pos);
    V = ggml_reshape_3d(ctx0, V, d_head, n_head, n_pos);

    cb(Q, "resampler_Q", -1);
    cb(K, "resampler_K", -1);
    cb(V, "resampler_V", -1);

    embeddings = build_attn(
        model.mm_model_attn_o_w,
        model.mm_model_attn_o_b,
        Q, K, V, nullptr, resampler_kq_scale, -1);  // Use local scale ✅
    cb(embeddings, "resampler_attn_out", -1);
}
```

### Option 2: Remove Class Member (More Comprehensive)

Remove `kq_scale` from class members and compute it locally wherever needed. This prevents future confusion.

## Testing Recommendations

1. **Compare outputs** with official Python implementation using identical inputs
2. **Run accuracy benchmarks** on MiniCPM-V evaluation datasets
3. **Audit other models** (MobileVLM, Qwen2-VL, etc.) for similar issues
4. **Add unit tests** to validate attention scaling factors

## References

- MiniCPM-V Resampler: https://huggingface.co/openbmb/MiniCPM-o-2_6/blob/main/resampler.py
- GGUF metadata keys: `clip.vision.embedding_length`, `clip.vision.attention.head_count`

## Reported By

MiniCPM-V Development Team

## Additional Notes

The `clip_graph` class is a generic graph builder used across multiple vision encoder architectures. The current design assumes all attention modules share the same dimensional parameters, which is incorrect for models with separate encoder and resampler/projector components.

## Technical Details

### How Parameters are Loaded

1. **GGUF File Loading** (`clip.cpp` lines 2707-2708):
   ```cpp
   get_u32(string_format(KEY_N_EMBD, prefix), hparams.n_embd);
   get_u32(string_format(KEY_N_HEAD, prefix), hparams.n_head);
   ```
   Where `KEY_N_EMBD = "clip.%s.embedding_length"` and `KEY_N_HEAD = "clip.%s.attention.head_count"`

2. **Projector Output Dimension** (`clip.cpp` line 5098):
   ```cpp
   case PROJECTOR_TYPE_MINICPMV:
       return ctx->model.mm_model_proj->ne[0];  // Returns actual projector output dim
   ```

### Call Stack

```
mtmd_init_from_file()
  └─> clip_init()
      └─> loader.load_hparams(ctx_vision->model, CLIP_MODALITY_VISION)
          └─> Loads n_embd=1152, n_head=16 from GGUF
              └─> clip_graph constructor computes kq_scale from these values
                  └─> build_minicpmv() incorrectly uses this kq_scale
```

## Verification

To verify the fix, compare attention output with Python implementation:

```python
# Python reference
import torch
d_head = 128
kq_scale = 1.0 / torch.sqrt(torch.tensor(d_head, dtype=torch.float32))
print(f"Correct kq_scale: {kq_scale.item()}")  # Should be ~0.0884
```


Changed files:
filename: tools/mtmd/clip.cpp
patch: @@ -1175,10 +1175,11 @@ struct clip_graph {
             cb(K, "resampler_K", -1);
             cb(V, "resampler_V", -1);
 
+            float resampler_kq_scale = 1.0f/ sqrtf(float(d_head));
             embeddings = build_attn(
                 model.mm_model_attn_o_w,
                 model.mm_model_attn_o_b,
-                Q, K, V, nullptr, kq_scale, -1);
+                Q, K, V, nullptr, resampler_kq_scale, -1);
             cb(embeddings, "resampler_attn_out", -1);
         }
         // layernorm