# Code study — multimodal I/O packing

Sources:

- `reference/transformers-inkling/image_processing_inkling.py`  
- `reference/transformers-inkling/feature_extraction_inkling.py`  
- `reference/transformers-inkling/processing_inkling.py`  
- `reference/hub/processor_config.json`  
- `InklingModel.forward` in modular  

---

## Image path

### Processor (`InklingImageProcessor`)

1. Optional long-edge rescale (`rescale_image_frac`, cap often 2048)  
2. Slice into **40×40** patches (`divide_to_patches`)  
3. Pad incomplete patches with **−1.0**  
4. Rescale + **CLIP** mean/std normalize  
5. Repeat temporal axis to **2** (`temporal_patch_size`)  
6. Emit `pixel_values` + `num_patches`  

### Text expansion

For each image: insert `image_token * num_patches` (default `
` × P).

### Tower (`InklingVisionModel`)

`plan_out_scales` → hierarchical fold of (t,h,w) into channel depth → Linear layers → last projects to `text_hidden_size` → RMSNorm → one vector per patch.

### Merge

```
image_features = vision_tower(pixel_values).pooler_output
inputs_embeds.masked_scatter(input_ids == image_token_id, image_features)
```

---

## Audio path

### Feature extractor (`InklingFeatureExtractor`)

From processor_config (authoritative defaults):

| Field | Value |
|---|---|
| sampling_rate | 16000 |
| feature_size (mel) | 80 |
| audio_token_duration_s | **0.05** |
| hop_length | 800 samples |
| window_size / n_fft | 1600 |
| output | log10 mel · shape `[B, T, 80]` + mask |

### dMel quant (`InklingProcessor._extract_dmel_bins`)

```
clamp(mel, -7, 2) → nearest of 16 bin centers → int32 [B, T, 80]
```

### Text expansion

`audio_token * T_valid` (mask-aware).

### Tower

Multi-codebook-style embedding over 80×16 codes + RMSNorm → one vector per **frame** (not per mel bin).

### Merge

```
audio_features = audio_tower(audio_input_ids[mask])
inputs_embeds.masked_scatter(input_ids == audio_token_id, audio_features)
```

---

## Generation behavior

`prepare_inputs_for_generation` attaches `pixel_values` / `audio_input_ids` only when:

- first iteration, or  
- `use_cache=False`  

Later decode steps stream text tokens only; media already absorbed into KV (and conv) state.

---

## Failure modes to know

| Symptom | Likely cause |
|---|---|
| Assertion on feature vs token count | Wrong soft-token expansion vs tower output |
| Silent bad audio | Wrong sample rate (must be 16 kHz) |
| Broken image edges | Padding −1.0 patches mishandled offline |
| Video confusion | Temporal dim present; OOB video lightly evaluated |

---

## Trace exercise

Pick one image URL + one short WAV and, on paper, list:

1. Number of image soft tokens  
2. Number of audio soft tokens (ceil duration / 0.05s)  
3. Which `input_ids` positions get scattered  
4. What tensors disappear after step 0 of `generate`