RiverRider/srt-omni-shared-tower

SRT omni shared tower

One linear tower that places image, audio and video beside text in a single searchable space, read off the frozen hidden states of Qwen3-Omni-30B-A3B at 60% depth. 8 MB.

One shared tower beats one tower per modality on every modality (1,376 holdout items):

	shared	per-modality	n
mixed gallery	0.2885	0.2667	1376
image	0.2902	0.2687	1027
audio	0.4451	0.4207	164
video	0.2486	0.1730	185

Derangement floor 695 +/- 21 against an analytic 688. The widest margin is video (+0.076), the modality with the fewest items, which is what a shared tower borrowing structure from the larger modalities looks like.

Contents

omni_shared_map.pt holds Wi (one 2048 -> 512 item tower, key all), Wt (the matching text tower), mu_img (a separate centering vector per modality) and mu_txt.

import torch
m = torch.load("omni_shared_map.pt", map_location="cpu", weights_only=False)
Wi, Wt = m["Wi"]["all.weight"], m["Wt"]["weight"]
v = (item_state - m["mu_img"]["audio"]) @ Wi.T + m["Wi"]["all.bias"]
q = (text_state - m["mu_txt"])      @ Wt.T + m["Wt"]["bias"]
# cosine between normalised v and q

Two things that will silently break this

Centering is not optional. Raw cosine between unrelated items on these states is +0.869 and raw retrieval sits exactly at chance. Subtract the modality's mu before projecting. The vectors ship with the tower for that reason.

Pool content tokens only. These states pool positions where the input id equals the modality's content token. Average every position instead and the shared chat prompt dominates: unrelated items land at cosine 0.9987 and you are measuring the template.

Scope

Fitted on one backbone, one manifest of 5,000 COCO images, 1,000 AudioCaps clips and 1,000 MSR-VTT videos, one split. Audio carries the highest score on the thinnest evidence: 164 holdout items, from a source where 12.6% of clips downloaded empty because AudioCaps is YouTube-sourced, so the surviving set is "clips still available" rather than a random sample.

States, scripts and results: RiverRider/srt-omni-crossvendor-states. Written up in paper_nla.md section 11.9.