Reversible Primitive–Composition Alignment for Continual Vision–Language Learning
Authors: Canran Xiao
Tianxiang Xu
siyuanma
Yiyang Jiang
Haoyu Gao
Yuhan Wu
Venue: ICLR 2026 Poster (2026)
Keywords: continual learning
vision-language models
catastrophic forgetting
Primary area: foundation or frontier models, including LLMs

Abstract: Vision-language (VL) models are increasingly deployed in non-stationary settings, yet under sequential adaptation they often preserve primitive recognition while losing compositional structure, especially with tight rehearsal budgets and no task IDs. We address this gap by asking how a continual VL system can maintain structurally dependable behaviour while safeguarding zero-shot performance. We introduce Compo-ReAlign, a structure-first recipe built around three components: a reversible composer that maps primitive embeddings to compositions by design, a multi-positive InfoNCE that jointly aligns textual and composed views of the same target, and a spectral trust region that clips updates when alignment sensitivity inflates.  Across compositional DIL and multi-domain MTIL retrieval, Compo-ReAlign sets a new state of the art, improves over the strongest prior by +2.4 R@1, and reduces forgetting by 40%. We provide a compact, reversible alignment head with geometry-aware training for compositionally robust VL continual learning.

Published as a conference paper ICLR 2026
REVERSIBLE PRIMITVEECCOMPSSTION ALIGNMENT FOR CONTINUAL VISION-LANGUAGE LEARNING
Canran Xiao Tianxiang Xu², Siyuan Ma², Yiyang Jiang', Haoyu Gao", Yuhan Wu Shenzhen Campus of Sun Yat-sen University, "Peking University *Nanyang Technological University Hoog Kong Polytechnic University, "Georgia Institute of Technology, *Zhejiang University xi siaocr30me @mail 1. cn
ABSTRACT
Vision Manoonllaguage(VV VL) models are increasingly deployed in non-stationary settings, yet un- der sequential adaptation they often preserve primitive recognition while losing composi- tional structure, especially with tight rehearsal budgets and no task IDs. We address this gap by asking how continual VI. system maintain structurally dependable behaviour while safeguarding zero-shot performance. We introduce COMPO-REALICE first recipe built around three components: a reversible composer that maps primitive em- beddings to compositions by design, a multi-positive InfoNCE that jointly aligns textual and composed views of the same target, and a spectral trust region that clips updates when alignment sensitivity inflates. Across compositional DIL and multi-domain MTIL retrieval, COMPO-RE REMOO-RRALIIN sets a new state of the art, improves over the strongest prior by +2.4 R @ 1, and reduces forgetting by 40%. We provide compact, reversible alignment head with geometry-aware training for compositionally robust VL continual learning.
a structure-
1 INTRODUCTION
Vision-language models (VLMs) (Radford et al., 2021 Guo et al. 2025 are increasingly deployed in non- stationary settings( Zhou al. 2025) new domains, evolving tasks, and shifting data sources in retrieval, assistance, and analytics( Lin et 2he5 2026 Zhan et al. 2025). In these environments, systems must adapt rapidly while preserving generalization and reliability on unseen data. Practical constraints are pronounced: privacy and cost often preclude large- scale rehearsal, memory budgets are tight, and task identitics may be unavailable at test time et al.] 2025b).
Chen et al. 2025 Zhang et al. 2024 Li al. (025)|Peng & Zhang,
Substantial progress has improved continual visual-language learning (VLL through geometry/tpollogy preservation and distillation (Ni et al. [2023] Zheng et al. 2023 Zhu al. 2023 Yao et al. 2023 Gao et al.
2024 Tha et al. 2024).. scalable streaming protocols (Garg et al. (024)), and aware consolidation (Cui
al. 2024). Replay and data-free surrogates (e.g., negative-tent text synthetic pairs) reduce forgetting under
limited memory (Yan et al. 2022 Smith et al., 2023 Lei et al. 2023
Wu et al. 2025); ) parameter-efficiem
prompts/adapters mitigate interference low update cost QQian et al. 2023 Tang et al.) 2024 Yao al.
2024 Xu et al. 20244 Luo et al.) 2025 Huang et al. 2025a). Yet practical pain point persists: under se- quential adaptation, models can maintain overall task/domain competence while degrading in fine- grained, combinatorial generalization, especially when rehearsal is scarce and no task-ID is available. This gap con- cerns how VL representations remain structurally dependable across tasks not merely whether average accuracy or zero-shot scores are preserved.
"Corresponding author

---

Published as a conference paper ICLR 2026
How can a continual VLM maintain structurally dependable behaviour under strict memory and no task IDs, while safeguarding zero-shot performance? We pursue structure-first approach that anchors the meaning of complex inputs across tasks, studies its geometric stability, and leverages small text-centric buffers as symbolic scaffolds.
Our contributions are as follows: (i) Phenomenon and diagnostics. We identify and quantify a recurrent deterioration in structural dependability during continual VI., and introduce light, reproducible diagnos- tics-retention ratios, cycle consistency proxies, and Jacobian-spectrum indicators- reveal tight links between alignment geometry and downstream behaviour. (ii) Simple, budget-friendly recipe. We demon- strate that a minimal training scheme- anchoring multiple textual views of the same target and stabilizing local sensitivity- substantillyy improves retention, lowers forgetting, and preserves zero-shot transfer across DIL/MTIL/VQA tracks, outperforming strong replay/regularization and adapter baselines under the same rehearsal budgets. (iii) Performance (state-of-the-art across settings). Under identical rehearsal budgets, our approach achieves best-in-class results on continual retrieval and VQA across DIL/MTIL tracks-raising compositional retention and sero-shot stability while reducing forgetting.
2 RELATED WORKS
Continual VL under non-stationary streams. Early continual captioning framed forgetting as transient- vs-shared dynamics in sequence models, introducing caskcconditioned gating and gradient masking to pro- tect recurrent states and vocabularies (Del Chiaro al. 2020). For contrastive VL, recent work scales to multi-domain retrieval and pretraining: momentum/ddisillatioon and topology-aware objectives curb drift across datasets and time (e.g.. BMU-MoCo for video-text (Gao al., 2022). Open-VCLIP for zero-shot video (Weng et al., 2023), CTP for VL continual pretraining with compatible momentum and topology preservation (Zhu et al. 2023). ). At web scale, TiC-CLIP shows that warm-starting from the last checkpoint plus replay offers a practical path close to Setraining-from-sceee
(Garg et al.) 2024). For retrieval, DKR emphasizes rectifying mismatched affinitics before distillation to avoid propagating carlier errors (Cui et al., 2024). Much of this line has focused task/domain retention and large-scale training mechanics (Zhang et al. 2025). However, real deployments also require compositional robustness -.c.,, preserving how at- tributes and objects bind bind-when rehearsal is scarce and task identities are unknown.
Zero-shot stability and structure preservation. A second line studies how to keep VL geometry sta- ble so zero-shot transfer remains reliable. Mod-X preserves off-diagonal similarity structure to maintain negative-pair geometry across domains (Ni et al. 2023). ZSCL performs reference-set distillation with weight averaging protect zero-shot predictions (Zheng al. 2023), CTP distils neighbourhoodd/opp-
logical relations (Zhu et al. 2023). and ZAF stabilizes consecutive zero-shot outputs on unlabeled data as a strong anti-forgetting signal (Gao et al. (022)) Probabilistic fine-tuning (CLAP4CLIP) further improves calibration and continual robustness (Jha et al. 2004). These approaches strengthen global stability but still leave open whether the model retains the internal structure that enables binding for instance, whether a composition embedding can reliably support recovering its primitive set and resist counterfactual swaps.
Against this backdrop, this paper targets the above pain point from a structure-firs perspective: we use a minimal head that (i) treats textual and composed representations as joint positives to keep the "meaning of a composition" anchored, (ii) makes the primitive-composition map reversible by design so binding remains recoverable.
3 EXPLORATORY STUDY
Continual VLMs often preserve primitives (attributes/objeets) while forgetting how to compose them
et al. 20255). We ask: Q1: Under a sequence of tasks that preserves the same set of primitives (at-
trboutesohjectsieeaatinn))) but rotates their compositions, do VLMs retain primitive recognition yet forget
2

---

Published as conference paper ICLR 2026
1.0
0:50
6.75
0.70
0.5
200
Taa:: nobo
Telk rOcR
Teat robo
(b) CRR (higher better). 1.95% CT) across tasks for FT, EWC, centric replay slows CRR decay. Replay-Text.
(a) Composition accuracy (mean
(c) Zero-shot composition accuracy on unseen pairs.
Figure I: Exploratory curves with error bands. Primitives are stable; composition degrades with task index, most for FT.
EWC
Replay-lex
0.72
0.8
0.54
2.7
0.58
28
end
g
0.52
00
0.24
012
22
02
62
ea
620
Jacob spoctallattio tuulj
jpostaenpecddd totkrs Bueld
(a) FT: high-crror plateau at large Tmxx and high CCE.
(b) EWC: moderated error surface.
(c) Replay-Text: expanded low-error
basin.
Figure 2: Error contour over Jacobian spectrum vs. CCE. Composition error grows with spectral sensi- tivity and irreversibility; text-centric micro-buffers curb both.
how to bind them? Q2: If forgetting occurs, does it coincide with loss of reversibility between primi- tive and composition embeddings and with an inflation of the alignment Jacobian spectrum? Q3: With strict rehearsal budget, text-centric micro-buffer more effective than an image-centric one, hinting that structure anchoring beats raw memory?
To answer the above questions, we construct continual streams Ti + TT ... where each task reuses the same primitive inventory (e.g., color/shape/materiill for CLEVR-like data; attribute/object for MIT-States) but exposes disjoint or low-overlap compositions. We sequentially tune frozen CLIP-style backbone with lightweight heads/LoRA (no task IDs), under small rehearsal budgets {0, 16, 64, 256) samples per task, comparing SEQFT, EWC, LWF, ADAPTER-ONe ONLY, and REPLAY variants. We evaluate: (i) primi- tive recognition (attributes, objects), (ii) composition accuracy in glasiffeationiettiiva///0.. (iii) binding robustness via hard-negative margins, and (iv) two structural diagnostics: cycle-consistency error (CCB) of primitive++ mappings and Jacobian spectral indicators (e.g., maximal singular value of Òs /0ppp where is the image- text similarity and Ep primitive embedding). Definitions, datasets, baselines, and computation details are given in Appendix A.
Findings. We observe three consistent phenomena across the exploratory setup. (i) Primitives remain stable while composition degrades with task index and the Compositional Retention Ratio drops clearly below one, with zero composition affected the most, and text-centric replay outperforming fine-tuning
3

---

Published as conference paper ICLR 2026
EWC
Replay Text
0.30
Gestoman cimart (ucheamn
The Bictouer boe Dre tate-
0.5
0.2
Not Not.oowe
0.15
6.58
Trak MO0R
1ask mnoos
(a) FT: pronounced subspace drift (b) EWC: drift.
reduced but non-negligible
(c) Replay--Ter Text: smallest drift; struc-
(deep layers, late tasks).
ture best preserved.
Figure 3: Subspace-drift heatmaps (text tower). Colors show Grassmann distances across layers (g) vs. tasks (x))
and EWC under the same budget, which is evident in the error-band curves in Fig. T (ii) Composition emror increases jointly with the Jacobian spectral radius and the cycle-consistency error, and the quiver field reveals descent directions toward a low-error basin, with empirical task means for fine-tuning drifting into higher- risk regions while Replay-Text stays within broadened low-error area, shown by the nonlinear contour (iii) Subspace drift concentrates in deeper layers and late tasks for fine-tuning, is moderated but not climinated by EWC, and is smallest and more localized for Replay-Text, which reflected by the iso-contoured heatmaps and hotspot markers in Fig. 3
maps in Fig. 2
centttrr--bfor--mmmmoe principle: continual VLMs prefer- entially retain first-order primitives while losing higher-order binding structure, and this loss is heralded by reduced reversibility and unstable aligmment geometry. We therefore motivate parameter-efficient head that enforces reversible prinitive+ alignment via cycle consistency while constraining the alignment Jacobian spectrum across tasks.
Takeaway. These observations support
a
4 METHOD
We propose COMPO-REALICH a minimal head for continual VL.Ms built on three ideas: one composer, one objective, and one stabilizer. (1) reversible composer maps a small set of primitive embeddings composition embedding with orthogonal core, hence invertible by construction. (2) A single multi-positive InfoNCE objective treats the text composition and the composed-fron crrtom--iiiiiees embedding positive views for the image, implicitly tying the composition views together without extra cy- cle/set losses. (3) A spectral trust region clips parameter gradients whenever the Jacobian sensitivity to primitive anchors becomes too large, stabilizing alignment geometry without adding losses. A tiny text- centric buffer optionally supplies paraphrastic templates and hard negatives but still reuses the same single objective.
Let fu : : - Rdd and ft y + Rdd be frozen encoders whose outputs we Izznorrmaliz. For an image x, a composition text you and its m primitives {p.).
fx(x)
f((ve)
ft(p)) (p.(()))
2.
llfo(z))l2*
e
lff())))l2
Cp.i
E Rd.
(1)
We denote U, [d(Ae.1);...; (ll:jeAAeppmmm Aepm)) E RRd the adapted primitive stack (row-wise), where A ddxd is light adapter and Φ Rdd Rdd tiny MLP.
4

---

Published as conference paper ICLR 2026
4.1 REVERSIBLE COMPOSER: BIECTION CONSTRUCTION
If a model can compose composition embedding directly from primitives and that embedding behaves like the textual one, binding is preserved. Making the core transform orthogonal turns reversibility into a design property rather than penalty.
We average adapted primitives then mix them through an orthogonal map:
172
R(O) [R(e) u|l2"
((Ae.))
é
(2)
m
where R(O) E Rdxd is orthogonal via the Cayley transform
R( = (I SI+ S) - Rdxd Then ((O) and R(O)-1 R
(3)
T. d is embedding dimension; the number of primitives.
4.2
ONE ORJECTIVE: MUL.TI- INFONCE (Two POSITIVES BY DEFAULT)
The textual composition and the composed embedding é. are two views of the same concept. Using them as joint positives for the image says: "match the image both ways you encode the composition," which implicitly co-locates ee and é. without explicit cycle/set losses.
Let sa, b) T b be cosine similarity since vectors are unit-normalized. For a batch {(Zo,is Ce, 1 and temperature > 0, define the Iwo-positive symmetric InfoNCE:
B
exp ((z (c,))/T +
c
log
(4)
Σ [exp s( ZZ, (r)) + exp( (x
B
exp(s(ee,i, 2n,3 + wx 2w,))7)
Le-v B
log
(5)
Σラ=1 [eeppp eexp(s(ee,ss Zo. )/T) + exp (s(e.,t, (xpps((ee z s//))
1=1
Lm =/2 (Curc Le+e).
(6)
Buffer extension. If a buffered paraphrase v. is available, we simply add e for sample i.e., the numerztoss/denmminttor above receive an extra exp(s(ze,5 f/) and its symmetric counterpart. This generalizes Eq. to a multi multi-positive InfoNCE without adding a new loss.
fx(2)) (b as an extra positive
4.3 GEOMETRY AS A TRUST REGION: SPECTRAL CLIPPING
The exploratory study shows composition failure correlates with large Jacobian spectra. We therefore clip the step whenever local sensitivity becomes too large, instead of adding another loss.
Let vec(Up) mRd be the stacked adapted primitives and
i s(zz, Ca)
Jp
E ER. 1p1xdd
a vvc(UU))
We estimate @max ~ || with one or power iterations a random unit vector v. Given target
y > 0, we rescale the gradient of parameters 0 E (0,A, Φ) as
go
go
a
min
(8)
maax
This spectral trust region caps harmful sensitivity while keeping the objective Las unchanged.
5

---

Published as a conference paper ICLR 2026
4.4 TRAINING IN CONTINUAL STREAMS
At task we update only θ, A, á with encoders frozen and task IDs. For each minibatch: (i) encode (I, ye: {); (ii) compose e. via Eqs. 2 (iii) compute Lri in 2 current samples (optionally adding buffered paraphrases extra positives); (iv) estimate max and apply spectral clipping Eq. 8 (v) take an optimizer step. Refer to Appx. B.1.3 for the detailed calculation process.
5 EXPERIMENTS
5.1 EXPERIMENTAL SETUP
Benchmarks and Protocols. We evaluate COMPO-REAL on three complementary continual tracks:
(i) Compositional DIL without task-ID: CLEVR/CoGenT (controlled primitives), MIT-States and VAW/VG- Attr Xttrributexobeeee and ConStruct-VL SVLC sequences (structured concepts; ITM) with ARO/Sug- arCrepe as probe-only compositional tests; (ii) Multi-domain retrieval MTIL: COCO - Flickr30K
+ ECommerce- T2I + RSICD (union-of-domaing testing); (ii) Continual VQA: CLOVE (scene- and function-incremeente incremental) and VQACL (skill x All streams expose the same primitive inventory but rotate compositions or domains. Details of splits, task orders and memory budgets appear in Appx. A.2
Metries. We report @@ 1/5/10, MRR/mR for retrieval/ITM, VQA accuracy (Avg/Last/AF), continual summaries (Avg/Last/ Largl//aggettigg/WWWW WTT///TTT Zero-Shot Transfer Degradation (ZSTD) and Compositional Retention Ratio (CRR). Formal definitions are in Appx. A.2
Baselines. We compare against strong replay (IncCLIP, SGP, ConStruct- VL, QUAD, GIFT, TiC-CLIP strategies), regulariaation//isilllaio (Mod-X, 7SCL,, CTP, DKR, ProXy-FDA, MG-CLIP, CLAP4CLIP, S&D, 7.AF, C-CLIP), and adapters/Mof: (RATTT, TRIPLET, DDAS, DIKI, RAIL,, LADA, CL-MoE) meth- ods, plus generic SogFT/EWC/LW//Repp We mark methods that require a task-ID at inference with "P". Full citations and per-method settings are in Appx.
5.2
MAIN RESULTS
Across all tracks, COMPO-REAL delivers the best average performance and the strongest structure re- tention. The retrieval/matching table (Tab. T) shows that COMPO-REALIGH sets new state of the art across compositional DIL and multi-domain MTIL, improving Avg R@ 1 (Image->Text) by +2.4 absolute over the strongest prior (C-CLIPIDIKI) and reducing forgetting by roughly 40% relative (AF 3.2 vs. 5.0-5.1). No- tably, CRR rises to 0.91, indicating substantially better preservation of attribute-ojeec binding, and ZSTD is the smallest in magnitude, evidencing minimal harm to zero-shot transfer. On continual VQA (Tab. 2) COMPO- REALIGN surpasses recent prompt/MoF approaches, yielding consistent gains on CLOVE-scene, CLOVE-function, and VQACL with the lowest AF.
5.3 SINGLE-FACTOR ABLATION
We conduct single-factor factor ablations to verify the contribution of each design choice. As shown in Tab. we can observe that: (i) Two-positive alignment is the main driver. Removing the composed positive incurs the largest drops on retrieval (R@ I-T 11.9, T-+I 11.9; CRR -0.04; AF +0.8) and VQA (CLOVE-scene -1.9, VQACL- -18),, confirming that treating textual and composed views joint positives is critical for binding retention. (ii) Spectral trust region guards stability. Disabling clipping barely changes top- re- trieval but increases forgetting notably and worsens ZSTD, showing it acts geometry safety valve rather than a pure accuracy booster. (iii) Orthogonal core matters for structure. Replacing the Cayley core with a linear mix consistently reduces CRR (-0.03) and harms both retrieval and VQA (= 1-1.6 point drops),
6

---

Published as conference paper ICLR 2026
Table Retrieval ITM results on compositional DIL (Track A) and multi-domain MTIL (Track B). We report averages across their respective task streams. 1 higher is better; AF and ZSTD & lower (closer to O for ZSTD) is better. CRR measures compositional binding retention.
Avg R@I
Method
CRR
AF L
ZSTD +
Image-
Text
Text Text->Image
Image
SeqFT
41.2
29.4
0.72
16.3
8.7
EWC
45.0
33.1
0.77
12.4
6.9
LwF
34.2
0.78
11.7
6.0
Replay-Text Text
51.8
38.7
0.84
7.5
4.1
ConStruct-Ve VL Smith et al.
2023
50.9
37.4
0.83
7.9
4.6
IncCLIP Yan
2022
53.1
41.2
0.86
6.7
-3.3
Mod-X
Ni
al. 2025
52.7
39.5
0.85
6.9
-3.8
ZSCL
Zheng ct al. 2023
54.2
40.8
0.86
6.1
-2.9
DKR Cui ct al.
2024
55.0
42.1
0.87
5.6
-2.
GIFT
Wu el al.
2025
55.6
42.5
0.88
5.3
2.3
ZAF al. 2022
54.7
42.0
0.87
5.4
2.0
C-CLIP el al.
(2025a)
56.4
43.0
0.88
2.1
DIKI Tang et al.
2024)
56.0
43.2
0.89
5.0
-19
COMPO-REALIGN (ours)
58.8
45.1
-1.
Table 2: Continual VQA (Track C). Average accuracy (%) on CLOVE-scene (DIL), CLOVE-function (TIL), and VQACL concep),, plus average forgetting AFU.
Method
C1.OVF-xcene Avg
CI.OVF-function Avg 1
VQACI. Avg
AF
SeqFT
54.2
49.5
46.3
9.8
EWC
56.7
51.0
48.2
8.0
LwF
57.4
52.1
49.0
7.6
SGP (2023
60.2
54.8
$1.3
6.3
TRIPLET Jan el al.
2023
61.0
56.5
53.1
5.8
QUAD Maroul et al
2025
62.3
57.1
54.0
5.2
CL-MoE Hual et al 2025
63.5
59.2
55.4
4.7
COMPO-REALLIGNOunnS
65.1
60.4
56.8
3.6
supporting "reversibility by design" as a robust inductive bias. (iv) Tiny text buffer is high leverage. Elimi- nating the buffer hurts across the board, indicating that symbolic anchors are far more memory-efficiee than image storage. (v) Mean vs. attention pooling. Attention yields near-identical accuracy with higher latency, validating
peean-pooling default for simplicity and speed. (vi) Primitive shaper and temperature modest but helpful. Removing //A or drifting т trades away about point on average; both mainly affect CRR and AF, consistent with their roles in smoothing primitive geometry and hardness.
Overall, these ablations corroborate the minimal recipe: one composer, one objective, one stabilizer-achh contributes complementary gains.
5.4 MECHANISM VALIDATION
Geometry-Structure Coupling We quantify how geometric sensitivity and reversibilit--estimaat Ja- cobian spectral radius maax (Jp) and cycle consistency emror (CCE)-relate to compositional performance (R @ CRR, ZSTD). We report task-wise statistics and correlations, and visualize (i) scatter plots with
7

---

Published as a conference paper at ICLR 2026
Table 3: Single-factor ablations across Tracks A+B (Retrieval/TTM) and Track C (Continual VQA).
Metrics (left): Avg R@1 (two directions), CRR 1, AF 4. ZSTD L; Metrics (right): CLOVE-
scene/func/VQAAI. accuracy 1. AF Each row toggles exactly one component away from the full model.
Track A+R: Retricaal ITM (averaged)
Track Continual VQA (overagri)
Vanaan
ReI [-+T1
RØI T--f
CRR 1
AF I
7STD
CLOVE-oore
CLOVVE-fue:
VQACT- 1
AF L
Dizure cocn FetKOi,
Tul ksrn
Damuu CLEWE coe, CUMT Sxedon VoAcE
Full (ours)
58.8
45.1
65.1
568
composed poeiive (esttoelyy InfoNCR)
c-1.9
0.87 (-001)
40 (10.)
(-60
(-1.0
(-1.0)
(-1.8
4.3 (+0.))
who specinal ast segien (no clipping)
57.9 c-0.)
44.3 (-0.)
039 (-002)
(+1.))
-16 (-43)
64.2 (-0.)
59.3 (-6.)
56.0 (-0.6)
4.2 (+0.0)
othoggnnal coee liner
Cayley)
-1.011111
0888 (o00)
3.8 (30
-1. (-4.5
63.6
59.1
55.6 (-1.0
4.1 (eb.0
hufler
6-.5
(-00)
477 (+1.)
-19
4.6
menn sticstion poillng
c-..)
449 (-0.))
(-000)
333 (+0.)
-1.8 (-4.6
(-0.)
(-6.0)
56.7 (-6.1
37 (+0.3)
wil primitive shaper (ol and A removedd)
57.6 c-12)
44.1 (-10)
088 c-000
399 (10.7)
-1. (-6.3)
64.0
59.3 (-1.3)
55.7 (-1.3)
4.1 (-2.5)
tompenttrre 0.10 idefoult 0.007)
(7.4.-1.0)
43.9
0.89 (-
3.7 (+0.))
-1.6 (-6.3)
64.1
59.4 (-1.0)
55.8
39 (+0.3)
0.21
mama 2-22.1
Sprrsnn in 0.61
Scnaman $:0.00
0.5
0.04
0.00
5.4
50.
0.40
1.2
Compus
0.5
17
0.01
1.1
0.2
56
0.01
0.09
0.3
0.3
Sea00
cct
(a) Fmax vS R@
(b) mmax vs CRR.
(c) CCE vs |ZSTD|
(d) Contour trajectory.
Figure 4: Geonetry-structure coupling. Three scatter panels annotate Pearson/Spearman coefficients; the contour panel overlays the task trajectory (Ti + To), which remains in a low-error basin under COMPO- REALIGN.
gression lines and Pearson/Spearman coefficients, and (ii) an error contour over (amax CCE) with the task trajectory overlaid. Across tasks, xmax is strongly anti-correlated with R@1 @1 and CRR Fig. 4bb)), and CCE is positively correlated with [ZSTD] (Fig. 4b) The deeper-layer correlations are stronger (Li0-L12), indi- cating that late-layer alignment geometry pivotal for preserving composition. The trajectory in Fig. 4 stays within a low-error basin, consistent with our structuree-efore--meee account.
Invertible Readout and Binding Robustness We test whether the composed embedding é. admits an invertible readout of the underlying primitive set and whether such invertibility translates into binding ro- bustness under counterfactual perturbations. We measure: (i) Readout accuracy: from é we predict the multi-hot primitive set via the inverse map Ipt-c and report Top-k accuracy, PR-AUC and ROC-AUC. (ii) Counterfactual margins: we measure the contrast margin 7 = s(z,, text - max
cfEN max s(z,, text,r) under attribute-swap and object-swap candidates N. We compare the full model to ablations: (linear mix), text-only positive (remove composed positive), and no spectral clipping. We adopted the pass- ing criterion: T substantially higher than ablations and significantly larger counterfactual margins (Wilcoxon, p < 0.01).
orthogonal core
The inverse readout from é. achieves strong Top-3/Top-5 and area metrics, with PR/ROC curves in Fig. 5b and cleellyy dominating ablations. Removing the composed positive yields the largest drop, indicating that two-view alignment (textual e, and composed é.) is key to identifiability. Under counterfactual swaps,
8

---

Published as conference paper ICLR 2026
Cco
120
622
C4
(a) PR of primitive (b) ROC curves of primitive (c) Attribute-swap margins (d) Object-swap margins 7. readout from cc.
readout from éc.
Figure 5: Readout quality & counterfactual robustness in one figure. (a-b) Invertible readout quality: PR/ROC curves for full, w/o orthogonal, text-only, and w/o spectral variants from e. (c-d) Binding ro- bustness under counterfactuals: Boxplot--strip overlays of attribute/object swap margins the full model shifts the distribution right with fewer hard-negative hits.
0.07
(SGAR)
JAG2 ADSIT
0.08
+ L50
3,40
0U41
6.445
3.642
3,50
0.04
0,00
304
0.89
645
664
644
0.00
400
15
13
72
0.000
(a) Diversity vs ACRR (scatter den- (b) Template task ACRR heatmap. sity contours).
(c) Multilingual improvements.
Figure 6: Text as structural anchors. (a) higher semantic diversity correlates with larger ACRR; (b) gains persist across template morphology and tasks; (c) advantages hold across EN/ZH/ES.
the full model produces significantly larger margins and fewer negatie reversals. Fig. 5c and 5d show that reversibility improves binding discriminability, rather than superficial alignment.
Evidence from Text-Centric Micro-Buffer as "Structural Anchors" With a fixed rehearsal budget M-64 text snippets per task, we manipulate three factors the text-centric micro-buffer: semantic diversity (coverage of primitive pairs and lexical entropy), (ii) template morphology ("attr-obj" vs. "obj with attr"), and (iii) language (EN/ZH/ES). We then measure changes relative to an image-only buffer with the same budget. If text acts as siructural anchor, expect diversity positively correlate with compo- sitional retention ACRR, and advantages to persist across templates and languages. The results show that: (i) Diversity helps structure. The -eatter--desity in Fig. [6 shows clear positive trend. (ii) Robust to surface form. Fig. 66 shows both "attr-obj" and "obj with attr" templates improve CRR across tasks with minimal gap. (iii) Cross-language holds. Fig. 6 indicates consistent gains for EN/ZH/ES. with modest variation due to tokenizer overlap. These support the view that text anchors structure more efficiently than pixels under the same budget.
9

---

Published as a conference paper ICLR 2026
6 CONCLUSION
We tackled the core challenge of preserving compositional structure in continual vision-tanguage learning under strict memory and no task IDs, proposing COMPO-REALIGN, Our approach consistently improves compositional retention, reduces forgetting, and attains state-of-the-art retrieval and VQA under identical rehearsal budgets, while maintaining zero-shot stability. Empirically, the tight coupling we observe between Jacobian- Spectrum/CCE indicators and downstream performance highlights geometry reliable handle for safeguarding structure.
Future work will explore lightly unfreezzig encoders under geometric constraints, and extensions to stream- ing video and multilingual settings for real-world deployment.
Ethics Statement This work adheres the ICLR Code of Ethics. Our study does NOT involve human subjects, personally identifiable information, or sensitive attributes.
REFERENCES
Zhangquan Chen, Xufang Luo, and Dongsheng Li. VisrI: Intention-driven visual perception via reinforced reasoning. arXiv preprint arXiv:2503.07523, 2025.
Zhenyu Cui, Yuxin Peng, Xun Wang, Manyu Zhu, and Jiahuan Zhou. Continual vision-language retrieval via dynamic knowledge rectification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 11704-11714 2024.
Riccardo Del Chiaro, Bartlomiej Twardowski, Andrew Bagdanov, and Joost Van de Weijer. Ratt: Recurrent attention to transient tasks for continual image captioning. Advances in Neural Information Processing Systems, 167361666774, 2020.
Yizhao Gao, Nanyi Fei, Haoyu Lu, Zhiwu Lu, Hao Jiang, Yijie Li, and Zhao Cao. Bmu-moco: Bidirectional momentum update for continual video-language modeling. Advances in Neural Information Processing Systems, 35:22699-2211 227122, 2022.
Zijian Gao, Xingxing Zhang, Kele Xu, Xinjun Mao, and Huaimin Wang. Stabilizing zero-shot prediction: tasks. Advances in Neural Information Pro-
A novel antidote to forgetting in continual cessing Systems, 17:1121-21188888 2024.
Saurabh Garg, Mehrdad Farajtabar, Hadi Pouransari, Raviteja Vemulapalli, Sachin Mehta, Oncel Tuzel, Vaishaal Shankar, and Fartash Faghri. Tic-clip: Continual training of clip models. In The Twelfth Inter- national Conference on Learning Representations, 2024.
Haiyang Guo, Fanhu Zeng, Ziwei Xiang, Fei Zhu, Da-Han Wang, Xu- Yao Zhang, and Cheng-Lin Liu. Hide- Ilava: Hierarchical decoupling for continual instruction tuning of multimodal large language model. arXiv preprint arXiv:2503. 12941, 2025.
Tianyu Huai, Jie Zhou, Xingjiao Wu, Qin Chen, Qingchun Bai, Ze Zhou, and Liang He. Cl-moe: Enhancing multimodal large language model with dual momentum mixture-of-expente for continual visual question answering. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 19608- 19617, 2025.
Chen Huang, Skyler Seto, Hadi Pouransari, Mehrdad Farajtabar, Raviteja Vemulapalli, Fartash Faghri, On- cel Tuzel, Barry-John Theobald, and Joshua M Susskind. Proxy-fda: Proxy-based feature distribution alignment for fine-tuning vision foundation models without forgetting. In Forty-second International Conference Machine Learning, 2025a.
10

---

Published as a conference paper ICLR 2026
Linlan Huang, Xusheng Cao, Haori Lu, Yifan Meng, Fei Yang, and Xialei Liu. Mind the gap: Preserving and compensating for the modality gap in clip-based continual learning, arXiv preprint arXiv:2507.091118 2025b.
Saurav Jha, Dong Gong, and Lina Yao. Clap4clip: Continual learning with probabilistic finetuning for vision-language models. Advances in neural information processing systems, 17:119944-1291888 2024.
Stan Weixian Lei, Difei Gao, Jay Zhangjie Wu, Yuxuan Wang, Wei Liu, Mengmi Zhang, and Mike Zheng Shou. Symbolic replay: Scene graph as prompt for continual learning on vqa task. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 1250-1259, 2023.
Hao Li, Yuhao Wang, Xiantao Hu, Wenning Hao, Pingping Zhang, Dong Wang, and Huchuan Lu. Cad- track: Learning contextual aggregation with deformable alignment for robust rght tracking. arXiv preprint arXiv:2511.17967 2025.
Yuxin Lin, Mengshi Qi, Liang Liu, and Huadong Ma. Vlm-assisted continual learning for visual question answering in self-driving. arXiv preprint urXiv:2502.00843 2025.
Wenzhuo Liu, Fei Zhu, Longhui Wei, and Qi Tian. C-clip: Multimodal continual learning for vision-
language model. In The Thirteenth International Conference on Learning Representations, 2025a.
Yuyang Liu, Qiuhe Hong, Linlan Huang, Alexandra Komez-Villa,, Dipam Goswami, Xialei Liu, Joost van de Weijer, and Yonghong Tian, Continual learning for vims: A survey and taxonomy beyond forgetting. arXiv preprint WxXv:2508.04427,, 2025..
Mao-Lin Luo, Zi-Hao Zhou, Tong Wei, and Min-Ling Zhang. Lada: Scalable label-specific clip adapter for continual learning. In Forty-second International Conference on Machine Learning. 2025.
Imad Eddine Marouf, Enzo Tartaglione, Stéphane Lathuilière, and Joost van de Weijer. Ask and re- member: A questions-only replay strategy for continual visual question answering. arXiv preprint WwXiv:2502.04469, 2025.
http//axxiv..o//ppt....o /ttp:////wwpp//////222/22/ /20.2/25222.02 2402.04469
Zixuan Ni, Longhui Wei, Siliang Tang, Yueting Zhuang, and Qi Tian. Continual vision-language repre- sentation learning with off-diagonal information. In International Conference on Machine Learning, pp- 26129-26149. PMLR, 2023.
Wenshuo Peng and Kaipeng Zhang. Harmonidpo: Video-guided audio generation via preference-optimizzed diffiusion: W. peng, k. zhang. International Journal of Computer Vision, 134(2):77, 2026.
Zi Qian, Xin Wang, Xuguang Duan, Pengda Qin, Yuhong Li, and Wenwu Zhu. Decouple before interact: Multi-modal prompt learning for continual visual question answering. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2953-2962, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763, PmLR, 2021.
James Seale Smith, Paola Cascante- Bonillaa, Assaf Arbelle, Donghyun Kim, Rameswar Panda, David Cox, Diyi Yang, Zsolt Kira, Rogerio Feris, and Leonid Karlinsky. Construct-vi: Data-free continual structured vi concepts learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14994-15000 2023.
11

---

Published as a conference paper ICLR 2026
Longxiang Tang. Zhuotao Tian, Kai Li, Chunming He, Hantao Zhou, Hengshuang Zhao, Xiu Li, and Jiaya Jia. Mind the interference: Retaining pre-trained knowledge in parameter efficient continual learning of vision-language models. European conference on computer vision, pp. 346-365. Springer, 2024.
Zejia Weng. Xitong Yang, Ang Li. Zuxuan Wu, and Yu-Gang Jiang. Open-vclip: Transforming clip an open-vocabulary video model via interpolated weight optimization. In International conference on machine learning, pp. 18978-36988 PMLR, 2023.
Bin Wu, Wuxuan Shi, Jinqiao Wang, and Mang Ye. Synthetic data is an elegant gift for continual vision- language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 2813- 2823, 2025.
Yicheng Xu, Yuxin Chen, Jiahao Nie, Yusong Wang, Huiping Zhuang, and Manabu Okumura. Advanc- ing cross-domain discriminability in continual learning of vision- vision-language models. Advances in Neural Information Processing Systems, 37:51552-51555 2024.
Shipeng Yan, Lanqing Hong. Hang Xu, Jianhua Han, Tinne Tuytelaars, Zhenguo Li, and Xuming He. Gener- ative negative text replay for continual vision-language pretraining. In European Conference on Computer Vision, pp. 22-38. Springer, 2022.
Jiawei Yao, Chuming Li, Keqiang Sun, Yingjie Cai, Hao Li, Wanli Ouyang, and Hongsheng Li. Ndc- scene: Boost monocular 3d semantic scene completion in normalized device coordinates space. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9421-9431. IEEE Computer Soci- ety, 2023.
Jiawei Yao, Chuming Li, and Canran Xiao. Swift sampler: Efficient learning of sampler by 10 parameters. Advances in Neural Information Processing Systems, 37:59030-59000 59990,,3-555,,,. 2024.
Jiazuo Yu, Yunzhi Zhuge, Lu Zhang, Ping Hu, Dong Wang, Huchuan Lu, and You He. Boosting continual learning of vision-language models via mixture-of-experts adapters. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23219-23230. 2024a.
Yu-Chu Yu, Chi-Pin Huang, Jr-Jen Chen, Kai-Po Chang, Yung- Hsuan Lai, Fu-En Yang. and Yu- Chiang Frank Wang. Select and distill: Selective dual-teacher knowledge transfer for continual learning on vision-language models. In European Conference on Computer Vision, pp. 219-236. Springer, 2024b.
Donglin Zhan, Haoting Zhang, Rhonda Righter, Zeyu Zheng, and James Anderson. Collaborative bayesian optimization via wasserstein barycenters. In 2025 IEEE 64th Conference on Decision and Control (CDC), pp. 6284-6291, IEEE, 2025.
Haoting Zhang, Donglin Zhan, Yunduan Lin, Jinghai He, Qing Zhu, Zuo-Jun Shen, and Zeyu Zheng. Daily physical activity monitoring: Adaptive learning from multi-source motion sensor data. In Conference on Health, Inference, and Learning, pp. 39-54. PMLR, 2024.
Xi Zhang, Feifei Zhang, and Changsheng Xu. Vqael: A novel visual question answering continual learning setting. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 19102-19112 12, 2023.
Xiaofeng Zhang, Fanshuo Zeng, Yihao Quan, Zheng Hui, and Jiawei Yao. Enhancing multimodal large language models complex reason via similarity computation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 10203-10211, 2025.
Zangwei Zheng, Mingyuan Ma, Kai Wang, Ziheng Qin, Xiangyu Yue, and Yang You. Preventing zero-shot transfer degradation in continual learning of vision-language models. In Proceedings of the IEEE/CVF international conference computer vision, pp. 19125-19136 2023.
12

---

Published as a conference paper ICLR 2026
Da-Wei Zhou, Yuanhan Zhang, Yan Wang, Jingyi Ning, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu. Learn- ing without forgetting for vision-language models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025.
Hongguang Zhu, Yunchao Wei, Xiaodan Liang, Chunjic Zhang, and Yao Zhao. Ctp: Towards vision- language continual pretraining via compatible momentum contrast and topology preservation. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pp.
2023.
13

---

Published as conference paper ICLR 2026
A DETAILS OF THE EXPERIMENTAL SETUP
A.1 EXPLORATORY STUDY SETUP
A.1.1 DATASETS, TASK STREAMS, AND MODELS
Synthetic. CLEVR-like with Color x Shape x Material x Size; tasks reuse the same primitive marginals but rotate disjoint composition subsets. Real. MIT-States compositional probes plus GQA subset for attribute/relational VQA templates. Streams. K E [4, 6] tasks; each task samples a fresh composition set with limited or no overlap: primitive coverage remains stable. Backhone and heads. Frozen CLIP ViT-B/16 (unless specified), with lightweight projection head + LoRA on text tower and final projection. No task IDs; identical step budgets per task. Baselines. SEQFT, EWC, LwF, ADAPTER-ONLY and REPLAY (Text-centric vs. Image-centric) under matching rehearsal budgets {0, 16, 64, 256 /task.
with ARO/SugarCrepe-sy
A.1.2 METRICS
Primitive/Comppsitionn Accuracy. Report per-task accuracy for attributes/objects and for compositions (pair or multi-attribute bindings), and compute forgetting (max previous minus current). Compositional Retention Ratio (CRR), Let A A and A (f) be accuracies task
puir
A
(t) puir
CRR(1)
(2) (0
obj
Binding Contrast Margin (BCM). For true image-text pair, = s(x, Vuue) maX s(x, y) where N' is a set of hard negatives from counterfactual compositions (swap attribute/ohject). Cycle Consistency
Error (CCE). Fit two light maps between text embeddings: Reep reconstructs composition embedding from its primitives; Ryec recovers primitives from a composition. Define
CCE = F.p Rptc (e
(10)
with symmetric variants on the image side if desired. Jacobian spectral indicators. For similarity (z), fi)) and primitive embedding Cp, compute Jp Ös/Oe, and track Vmax(1p) and condition number. Subspace drift. Use principal angles/CCA to measure Grassmannian distance between the current and historical composition subspace (per tower/layer).
A.1.3 TRAINING PROTOCOL AND HYPERPARAMETERS
Frozen backbone; AdamW; LoRA ranks E {8, 16); head LR x 10 LoRA LR 10-1 weight decay 10 10-2; ;; batch size 256; per-task steps fixed across methods. Text-centric buffers store composition templates and hard-negativy variants; image-centric buffers store images/patches under the same item budget.
A.2 MAIN EXPERIMENTAL SETUP
A.2.1 BENCHMARKS AND TASK CONSTRUCTION
Track A - Compositional DIL (no task-ID). CLEVR/COGenT. We follow CoGenT A-B remaps Soolors+>hapes/mattee
to stress compositionality under matched primitive marginals. Tasks expose dis- joint low-overlap attribute-ojject compositions. MIT-States (AttrxObj) and VAt/VG-Attr. We fix the attribute and object vocabularies; each task rotates the visible pairs. ConStruct-VL SVLC SyLt,Smith et al. (2023). Using Visual Genome/ Binome//AWWddee derived sequences, adopt the official order over colormmateria/sste,,
14

---

Published as conference paper ICLR 2026
spatial relations, action relations, and state. Each task is image-text matching (ITM) with balanced posi- tives/negatives. Probe-only suites. ARO and SugarCrepe are used for compositional probing each round; they do not participate training. Rehearsal budgets. {0. 16, 64, 256} text snippets per task (default text- centric); when a baseline requires images we match its memory cap.
Track B Multi-domain retrieval MTTI. COCO - -ECommerce-12 RSICD, follow-
ing Cui et al. 2024)) [Ni et al. (2023); Zheng et al. (003). Each round introduces new domain; test queries are drawn from the union all seen domains. We do not supply domain-ID at test unless a method mandates it (marked "{").
Track C- Continual VQA. CLOVE et 2023): scene-incremental (DIL) with evolving environ- ments and function-incremeene (TIL) with evolving skills; model across tasks. VQACL Zhang al. (2023): outer tasks are reasoning skills CCoun/Collo//Locationn
and within each skill the object classes partitioned into groups that arrive over time; evaluation requires transferring the learned skill unseen concept groups. We follow authors' official splits and answer vocabularies.
A.2.2 EVALUATION METRICS AND CONTINUAL SUMMARIES
Retrieval ITM. Recall@ K (R@K, K E (1 10)), mean reciprocal rank (MRR), and mean rank (mR).
We report per-task and averaged scores.
VQA. Exact-match accuracy (%). Following Zhang et al. (2023) Lei et al. (023): Qian al. (2023):
Huai et al. (2025), we summarize with Avg (mean over tasks), Last (atter the final task), and AF (average forgetting).
CL summaries. Let At.. denote performance on task after finishing task u. For T tasks,
Avg T ACT, Last AT,,,
T-1
Forgetting
max Ar ALT). BWT T Σ aE(!.....T)
(Ar, - At.))
T- t=1
[=1
FWT the pre-training performance on unseen tasks relative to a zero-shot reference A7:: FWT
- CA - )
Zero-shot transfer. ZSTD Zheng al. 22023) is the drop in zero-shot accuracy on held-out classification sets (e.g., ImageNet variants) measured before after each task.
Compositional diagnosties. CRR
Apair
(Sec. 3 higher indicates preserved binding beyond inde-
Aus
pendent primitive accuracy. We also track ARO/SugarCrepe scores and non-optimized structural correlates (inverse readout accuracy; estimated Jacobian spectral radius distribution).
A.2.3 COMPARED METHODS
We group methods by learning principle and use official code or faithful re-implemennations with authors' validated hyperparameters; task-ID-at-test baselines are marked "f".
Replay, IncCLIP Yan et al. (2022) ; SGP Lei et al.
2023): J ConStruct-VL. Smith et al. (023)
M Marouf al. (025 GIFT Wu et al. 2025
TCC-LIP strategies Garg et al. (2024).
15

---

Published as a conference paper ICLR 2026
Regularzatiio/iistllttiio Mod-X Ni al. (002)): ZSCL Zheng al. 2023); CTP Zhu al.
(2023): DKR Cui et al. (2024):
Proxy-FFD FDA Huang et al. (2025a): MG-CLI Huang et al. (2025b)
CLAP4CLIP Jha et (2024):
S&D Yu al. (2024b) ZAF Gao et al.
(0024) C-CLIP
al.
(2025a)
RATT
Del Chiaro al.
2020
TRIPLET Qian al.
2023
DDAS Yu et al. (2024a); DIKI Tang et al. )2024; RAIL Xu et al.
2024); LADA I Luo et al.
2025
CL-MoEE Huai et al. (2025).
Generic CL baselines. Seq EWC, LwF, Repgay-Image/Tex, and Joint upper bound.
A.2.4 HYPERPARAMETERS
Text-centric micro-buffer. We maintain tiny buffer B of size M E {0, 16, 64, 256} per task, containing short composition templates and a 1:1 mix of hard negatives. Hard negatives are mined online by nearest- neighbor swap on the text side (replace the attribute or relation while keeping the object). Each step we sample by snippets (default by = 32) and reuse the same objective (Eq. 4h: buffered paraphrases are simply added as extra positives in the aumerators/denominator.
Objective (two positives by default). The single training loss is the symmetric multi-positiv InfoNCE of Eq. 4 with temperature 7 = 0.07. Unless noted, we use only in-batch negatives (no queue) to keep the method minimal. For VQA, the image acts as the key and each candidate answer text acts query; é. is computed from the primitive set implied by the question type.
Spectral trust region. We stabilize geometry by clipping the step rather than adding a loss (Eq. 8 Imple- mentation uses directional derivative of s(zv: ē.) = z é. wrt. vec(Up) IR=ad draw a random unit vector compute max R || with one power-iteration using autograd. grad (create-grat graph-True), and scale parameter gradients by min(1, 1 m/x.x) We ? 6.0 for ViT-B/16 and y 7.5 for ViT-L/14. Overhead is < 2% wall time.
Optimization & schedules. We use AdamW (31 0.9, 32 0.98, weight decay 10 10-2) with cosine decay and 5% warmup on the first task only; subsequent tasks warm-start without warmup (per time-continual evidence). Global batch size is B 256 for retrieval/ITM and B 128 for VQA (achieved via DDP gradient accumulation). We train 20k steps per task on Tracks A/B and 10k on Track C. with early stopping the current task's validation. Mixed precision scaling. We apply gradient-norm clipping at 1.0.
BF16 when available, otherwise FP16 with loss
Data processing. Images are resized to 224 with RandomRest ietiiommeeizedCrooo zedCrop and horizontal flip p = 0.5. We avoid color jitter in attribute-heavy tasks (MIT-States. SVLC) to prevent color-label leakage; for generic retrieval we use a mild ColorJitter (brighnees/conrrasssttua and cetetttttiionooomaiien we do not paraphrase on-the-fly beyond the buffer.
0.2). Text is lowercased
identity-initialized, @ uses Kaiming uniform, and small random skew with scale 10-3 so that R(O) = I at start. Attention-pooling parameters (W.. w) are zero-initialized to start from mean pooling.
Initialization.
Hardware & reproducibility. We train on 8xA100 80GB (retricval/ITM) and 4xA100 80GB (VQA). Distributed data parallel with find unused parp Fess=False. We fix seeds 1,,1, enable cuDNN deterministic, and control dataloader workers for repeatability. All reported numbers are std over seeds.
16

---

Published as conference paper ICLR 2026
Ablation toggles. We vary: pooling (mean vs. attention), temperature {0.03, 0.05, 0.07, 0.10), spec- tral threshold E {5. 6, 7,8). buffer size M E {0, 16, 64, 256), and optional LoRA on CLIP projections with rank E {4, 8, 16). Unless stated, defaults are mean pooling, 0.07, ? 6.0/7.5 (B/L), and
M = 64.
B SUPPLEMENTARY TECHNICAL DETAILS
B.1 IMPLEMENTATION DETAILS
B.1.1 BACKBONES HEADS.
We freeze both image and text encoders of CLIP-style model and learn only a tiny head. Results are reported with ViT-B/16 and ViT-L/14; the representation size the native CLIP projection (no extra projection layers). Let fu X' + R2 and ft y Id be the frozen encoders with L2-normalized outputs. Our head COMPO-REALIGN contains three lightweight parts:
Primitive shaper A. We use single-hidden-Laee MLP
((u) W2 GELU(W, LN(u)), Wi, W2 gdxd
with dropout 0.1 and residual connection A E IRd (identity init.). For m primitives {p:)-=
+ Φ(u). It is preceded by linear adapter
fe(p))
Cp.i
Up = ((Ap))).
(po)l)
Permutation-invariint composer. By default we average then mix (Eq. 2
R(O) ū
Up
é
[[() u
R
We also implement an iitentiin--polinng variant for completeness:
exp( w tanh (Wa Up.:))
Thst Σ octlppi
= no=oorn(RC) R(O) uau):
04 En exp(1x (w tanh Wa Up.s))
with W. E Raddd w E Rd. We found attention matches mean pooling but adds latency; mean thus the default.
Orthogonal core via Cayley. We parameterize R(O)= E Rdddd as (Eq. 3)
R(O) = (I +)
S= ] )
R I by construction and R R We compute (I + S) with single LU factor- ization per forward pass and add +&I with = = 10 for numerical safety; orthogonality holds to machine precision.
B.1.2 TOKENIZATION & PROMPTS.
We use CLIP's tokenizer. For text compositions ye adopt class-agnostic templates that expose primitives explicitly, e.g.,
MIT-StatesVAAW/VV-AAtt "a photo of {aur) {obj}"
17

---

Published as conference paper ICLR 2026
Algorithm I COMPO-REALIGN, Training Tass i (no task IDs)
1: Inputs: dataset Dr with triples (x,, Ye: {pr )==):: frozen encoders fe: ft:
trainable head params (θ, A, Φ); temperature spectral cap 7 (optional) lext micro-buffer Boufi power-iteration steps pow E { 21
2: for epoch 1, ...... do for mini-batch (I Ve,is {pr.))=))) C D,do do
3:
4:
Encode & normalize
(zi)
felye,.)
Ep,i. fe(pee)
5:
Zudi (x=)2=))
C 11fe(20,))122
l1f((1)))l2
6:
Adapt primitives & average
1 L Uiie
7:
u t ((Aepiie) for 8 1,..,,, m;
U, f-
8:
Reversible composition (Cayley core)
(Fqs. 3
R
9:
S ( R(O) SI+ Ces
|R(O)u
10:
Add paraphrase positives
SelVlasss)
11:
P. - (E',,,h from Bouf with e'c,,
[
12:
Multi-positivy symmetric InfoNCE
(Eq
13:
P; (ee, 6e,) U P;; compute Lin over (zu.; P.) )
14:
Estimate local sensitivity (power iteration on JVP)
15:
sample unit v E gud, mmax.i +-
16:
for / 1, ..... T pow do
17:
+- V vooc(Upp.)) (s.c(,,))((Z. ee,))] v (JVP via autodiff)
18:
f- w/ w//lw w 12: m +- 2012
19:
Spectral trust region (per batch)
(Eq. 8
max 1
20:
- 2 max.:: a t- min,,,
21:
scale head gradients: ge,A, +- a ge.4. Update (head only; encoders frozen)
22:
23:
A. Φ) + OPTIMIZER.STEP (
(
24:
Output: updated head (θ, A, at task / (with fo, frozen)
ConStruct- VL (SVLC): "the image describes {concepr)" where fconcept) is color/material/size or a relation clause ("[objl} left of (obj2}")
Retrieval on COCo/Flick:30K/ett. standard CLIP prompts plus two paraphrases per concept to reduce prompt bias.
For VQA, the question is encoded as text; answers come from the task's closed set and are scored by image-ter text similarity. When primitives are needed (e.g.. "Color of {obj]?"), we use dataset metadata when available; otherwise light rule-based extractor maps adjectives/nouns in the question to fattr, obj}.
Parameter footprint. On ViT-B/16, Ø and together add 2dh parameters and the skew form adds d(d-1) This 1% of the frozen backbone. We do not use LoRA by default to keep the method minimal; (o) projection layers is included only in ablations.
18

---

Published as a conference paper at ICLR 2026
B.1.3 PSEUDOCODE
Computation flow. For each mini-batch, we first encode and nor-nora the image xy. the target compo- sition text Vc, and each primitive Pt, using frozen fvr füi gradients do not flow into encoders. The primitive embeddings are then adapted and pooled: each Ep,i, is passed through a tiny adapter-MLP stack (A, o) to yield use = o ((Aep.))); the primitive set is summarized by the mean prototype ü; = 2 Ee= uie to pre- serve permutation invariance and stabilize gradients. We then perform reversible composition via an orthog-
onal core R(@) parameterized with the Cayley map in Eq. 3 in practice, computing (I + - (I S)
is implemented as single linear solve avoid explicit matrix inversion and to keep the cost at O(d2) per sample (often batched and fused). The resulting composed embedding ēe, is T-g-normalized (Eq. 2
guaranteeing R(0)-1 = R(@) so that information about primitives is not collapsed by the composer.
Next, formulate single, symmetric multi postii-ossitive InfoNCE (Eq. 4 where the positive set for each image is P. = {en,i, 3 U P.. Here P: contains optional text paraphrases from the micro-buffer Buut; these are included only as additional positives and require new losses. Negatives are the remaining texts in the mini- batch for both the textual and composed views, yielding a denominator that aggregates |P,] terms per sample j; the loss computed in both directions (v-+c and c->v) with a shared temperature stabilization. To stabilize alignment geometry, we estimate the local sensitivity of s(zu,; Ce,i) to the adapted primitives through one-two Jacobian- -vector power iterations (JVPs) per batch, which have the cost of a few reverse-mode passes but do not materialize the full Jacobian. The resulting estimate max sets a spectral trust region that rescales the head gradients by a = min [1, 7 7/ōmax] J (Eq. (). capping harmful sensitivity while leaving the objective unchanged. Finally, we perform an optimizer step on head parameters only (e, A, σ); encoders remain frozen, no task IDs are used, and the rehearsal budget is enforeed by restricting P.| and the buffer sampling policy. This pipeline yields parameter-efficien updates
r and log-sum-exs
that anchor the meaning of compositions to their primitives, preserve reversibility, and maintain zero-shot stability under strict memory.
C ADDITIONAL EXPERIMENTS AND RESULTS
C.I PARAMETER SENSITIVITY
sensitivity to key hyperparameters by varying one factor at a time while keeping others at their defaults (Sec. A.2.4). For each configuration train across all streams and report mtd over 3 seeds. Retrieval is averaged R@ (Image-TTex) on Tracks A+B; VQA Avg is the mean across CLOVE-scene, CLOVE-function, and VQACL; also report CRR 1, AF 0 is better). Pooling lemperature Tpool controls the sharpness of permutation-invariant aggregation (Tpool=0 equals uniform mean; Tpool + xo approaches max).
We evaluate
and ZSTD + (closer to
As shown in Fig. 7 We can observe the following conclusions: Temperature. A clear optimum at 7 0.07: smaller 7 over-emphasizes hard negatives and destabilizes multi-positive logits, larger softens contrast and weakens gradients, lowering CRR and accuracy. Spectral threshold. 7y balances plasticity and stability. Tight clipping (7 S 5) slightly reduces AF and improves ZSTD magnitude but underfits retrieval, loose clipping (7 2 7) increases AF and degrades CRR. Buffer size. Text-centric anchors are high-leverage: even M=16 recovers most gains, M is saturaaaur,,ion, M-128 256 brings small, consistent improvements. LoRA rank. Optional LoRA on projections yields marginal gains up to r=8 then saturates, the minimalist head already preserves structure. Batch size. Larger B slightly improves in- negatives and stabilizes training but platcaus beyond B=256. Learning rate. The sweet spot is 2 10 4, larger rates inflate spectral sensitivity and forgetting despite clipping, smaller rates underfit. Positives per sample. Moving from two three positives (adding one paraphrase) consistently boosts CRR and both tasks with negligible ZSTD cost, more than three yields diminishing returns. Pooling temperature. Uniform mean (Fp is optimal, sharper aggregation drifts toward "max" and hurts stability/CRR. Power iterations. One step suffices to
19

---

Published as conference paper ICLR 2026
Tengeature
Spertaltrooooddy
14
(D)
AT -2-T22TDD (1)
B851 VODA CRR
03/23STD (2)
14
23-
0
mik (gng)
Leanuin 100
VON/COORR (1)
14
AD 23TD 03
/25223TDDD 03
(1) positvo Ape
30.0
(3) Pooling kemperstare Tpis
Faration
03-
017
ART 28TT (1)
BuS VON CRRR
AF 23TT
13
Fevirval Avg Rigi
68
2670
Figure 7: Parameter sensitivity (mean over seeds). One factor varies at a time. Retrieval: Avg
R@I l (I+T) 1. VQA Avg: mean of CLOVE-scene, CLOVE-function, VQACL
estimate the spectral scale, additional iterations do not change outcomes, confirming the sensitivity map low-rank in practice.
C.2 ORDER SENSITIVITY (TASK& & DOMAIN PERMUTATIONS)
Protocol. To rule out "lucky ordering," " we evaluate five permutations for each continual stream. Track
A (Compositional DIL) permutations: A1 CLEVR/CoGenT + MIT-States + VAW/VG-Attr - SVLC;
A2 MIT-States > CLEVR/CoGenT -> SVLC VAW/VG-Attr; A3 VAW/VG-Aur - MIT-States -> CLEVR/CoGenT + SVLC: A4 SVLC -> VAW/VG-Attr MIT-States CLEVR/COGenT: A5 MIT- States > SVLC VAW/YG-Attr - CLEVR/CoGenT. Track
retrieval) permutations: B1 COCO + Flickr30K EComm-T21 RSICD; B2 Flickr30K - COCO - RSICD - E-omm-TT221. B3 EComm-
T21 + COCO Flickr30K - RSICD; B4 RSICD - EComm-T21 - COCO - Flickr30K; B5 COCO + RSICD EComm-T21 -> Flickr30K. For each method and permutation report Avg/Last R@1 (I~T, T-I), CRR, AF and ZSTD. We summarize order sensitivity by the sample standard deviation Std, across permutations
20

---

Published as conference paper ICLR 2026
(a) std across
Parallel bands
(Track A).
(b) Parallel bands (Track B). Baselines for all exhibit wider spreads than ours.
(c) Std heatmaps across orders (shared color scale). Lower is better.
orders
metries.
Figure 8: Order sensitivity overview. MEMPO-REALIGN produces tighter bands across metrics and lower variability than strong baselines on both tracks.
Frose buud uge the eetnet
De) Cies Orouge
-M
(a) Task-wise error bands (Track B) for R@ in both direc- tions (std over five orders).
(h) AF (left axis, lower better) and CRR (right axis, higher better) distributions across orders.
Detailed order effects. Our method maintains higher means and narrower uncertainty bands as lasks accrue and achieves the smallest AF spread with the highest CRR.
Figure 9:
Figures summarize robustness to task/domain permutations on Tracks A/B: (i) the parallel-band views (Figs. 8a-88 show that COMPO-REALICE forms tightly bundled trajectories across all metrics, whereas baselines spread substantially, especially on AF and ZSTD. This is corroborated by the standard-deviation heatmaps (Fig. Sch: across-order std for Avg R@ I (I+T) drops to 0.26 on Track A and 0.24 Track B (ours) versus 0.55-0.52 for strong baselines, while AF variability shrinks from 0.27-0.28 (DIKI) to 0.15 (ours). (ii) task-wise error bands (Fig. Pap indicate stability under accumulation: R@1 stays consistently above baselines and the shaded uncertainty narrows, suggesting reduced order- induced drift rather than reliance on lucky sequence. (iii) distributional views (Fig, 9b) reveal that AF (forgetting) not only centers lower but also exhibits the tightest interquartile range, while CRR concen- trates higher with smaller dispersion consistent with our geometry-stabeee stabilizing design.
tasks accrue, mean
C.3 CROSS-DOMALE DOMAIN ZERO-SHO SHOT STEADY STATE
Protocol. To test whether geometric stability extrapolates to unseen domains, we evaluate zero-shot perfor- mance on held-out distributions [ImageNet (IN), IN-Sketch, IN-Renditions (IN-R), IN-Adversarial (IN-A), ObjectNet, DomainNer-Skert tion ZSTD (lower magnitude better; closer to 0 is best) and the alignment spectral radius T estimated on late layers (1.10-112) of the text tower. We plot !ZSTD| versus the layer-mean o = E1=00 Fmax- Our criterion is to occupy the lower-left quadrant (smaller a, smaller (ZSTD) across domains.
(DNet-Sketch)), ). For each method, we report the zero-shot transfer degrada-
the per-domain scatter matrix (Fig. 10a), the points for COMPO-REALICH consistently lie in the lower- left quadrant- sattrrantt-smuuuannussss/ smaller o and smaller [ZSTD] while baselines drift toward higher and/or
21

---

Published as conference paper ICLR 2026
Soplo Sm
18T
On
Gldbe ovepling gooneetry and sere den.t
CCLP
Dhe-Sath
110000
30-
2.
(a) Scatter matrix: [ZSTD] per domain (median lines form quad- rants).
Global scatter domains with per- method markers.
Figure 10: Cross-domain zero-shot steadiness. Lower-left is better. Ours concentrates in the low-o, low- [ZSTD| region across domains.
Wittindommin empling Wvs. (2STD]
Quadiant occupancy lmedias
1.54
Domatre Innerdeet (ont.rm
1.82
1.00
066
Reyliy
cGUPP
(a) Per-domain Pearson correlation between 77 and [ZSTD] (b) Lower-left quadrant occupancy across (higher indicates stronger coupling).
domains
(median-based) Ours: 6/6.
Geometry-zero-shot coupling diagnostics. Strong within-domain coupling and consistent lower-left occupancy for Ours.
Figure 11:
larger IZSTD). The global scatter (Fig. 10p) shows a clear positive trend between geometry and zero- shot drift; all methods align with this slope, but Ours forms compact cluster strictly below and to the left of the baseline clouds. Finally, the correlation bars and quadrant-occupann plot (Fig. [T] indicate consistently positive within-domain coupling and 6/6 lower-left occupancy for Ours, evidencing stable geometry-zero-shee relationship across held-out domains.
C.4 TRAINING DYNAMICS MONITORING
Protocol. We track the alignment sensitivity Ōmax at every training step for the late text layers (L10-L12).
A clipping trigger occurs step & and layer l whenever maxx (t) y (trust-region threshold y=1.35). We visualize (i) a stepxlayer time-heatmap @axx, and (ii) the per-step trigger rate (fraction layers Late (401-600).
=(t)
exceeding y). We segment training into phases: Warmup (steps 1-200), Mid
22

---

Published as conference paper ICLR 2026
Plaace arommaged
(a) Time-heatmaps of max (1.10-L112 600 steps). Left: Compo-Rest ReAlign; Middle: No-Clip ablation; shared col- orbar right.
(b) Per-step trigger rate (smoothed) with phase bands (left) and embedded phase summary table (right).
Training dynamics: sensitivity bursts & clipping responses. Trust-region clipping suppresses and shortens late-layer spikes, yielding fewer and shorter episodes above Ty.
Figure 12:
In the time- heaemahs (Fig. 12a COMPO-REALIN REALIGN exhibits sparse, short spikes confined to early steps and L12, while No-Clip shows broad, persistent bands above 7, especially late. The trigger-rate plot with phase bands (Fig. 12b) indicates a rapid decay and low variance for Ours, versus sustained high triggering for No- Clip. Together, these visuals show the trust region intercepts sensitivity bursts critical stages, preventing late-layer geometry blow-ups and stabilizing training.
D THEORETICAL ANALYSIS
D.1 IDENTIFIABILITY AND A CRR LOWER BOUND
We formalize when the proposed reversible composer preserves sufficient information to recover the primi- tive set of composition and how this yields lower bound for structural Compositional Retention Ratio (CRR).
Let = {u....., uM} C gd-1 denote the adapted primitive dictionary with unit vectors ú, =
d(Aepa)
[o((Ae
For composition C [M] of size m. define the (unrotated) mean
= ((S)
u ((S)
= m
c(S)
[[
E Sd-1
(11)
ES
The learned composer applies an orthogonal (d) (Cayley core) produce &g = Rc(S); since known and R = R-1, the canonicalized composed embedding is R & = c(S). We decode primitives by top-m correlation:
((c) == Top-m indices of {(c, )-
(12)
Define coherence μ == max,4 (u,, uz)l E [0, 1) and the structural CRR for single composition as
CRR(S) = |s S
E [0,
(13)
For brevity write = u(S) and e:= c(S). We write a S b to hide absolute constants.
We first show that coherence alone guarantees separation between members and non-members.
Lemma (Norm of the mean and in/out correlations). Let S c [M] with [S| = m and coherence μ. Then
1 (m 1)
T
N
(14)
m
23

---

Published as conference paper ICLR 2026
for any S, b S.
1
(m 1)p
(ū, ua) 2
(ū, ub) S μ.
(15)
m
Consequently,
(m 1)
Cc Uay
(c ub)
1 1 (m - 1)11
(16)
m
member score
non-menber scon!
Proof. Since luull2 and
((u,, uz}l | S for i +
|
1
(Ui, ug} 2 m²
m m(m 1) =
1 (m 1)1)
(17)
m
1.ES
which is Eq. 14 For any S.
(ū, ua)
1
(uus Ua) +
Σ iES|aa}
(Ug Unl 2 m
1 (1 1 (m 1)(0
(18)
and for any E S.
(ū, ub)
(uis u6)
S m
mu n.
(19)
m
iES
Divide both by |ū|l and use Eq. 14 to obtain Eq. Theorem 1 (Exact identifiability). Under coherence μ Z for any S with [S] m the decoding rule satisfies S(c) S. Moreover, the margin separating members from non-members obeys
16
1 1 (2m 1)p
Do
min(c, un)
max ub)
(20)
aES
bEs
mm / (m 1)3
The condition μ < 2mm1 is necessary (up equality) for uniform separation across all S.
Proof. By Lemma 1 for any a E S. b S.
(m 1)p
1 1 (2m 1)fs
(c, un) (c, up)
(21)
m
(m 1)p
V (m 1)
which is Eq. 20 The right-hand side positive iff > (2m 1)j, i.e., /
2m-I I which guarantees all members outrank all non-members and hence S(c) S. For necessity, if p 2m
can construct u;
with pairwise inner products saturating μ on two (oo+1))-tuple such that the bound in Eq. 20 is non-positive, preventing uniform separation for the worst-case S.
Remark. Inequality Eq. attains equality on equiangular tight frames where off-diagonal inner products constant =y. so the bound is tight in the worst case.
We next allow perturbations in the composed vector before normalization (e.g., training noise or small modeling mismatch). Let the canonical (unrotated) pre-normalized vector be and suppose the model produces
=
+ n
E idd
(22)
n/|
the decoder uses č in place of
24

---

Published as conference paper ICLR 2026
Lemma (Lipschitzness of normalization). ||//112 S =||lū|l2 with (0, 1), then
| - dl2 S 1
(23)
Consequently, for any unit gd-1,
2s
(G.0) (c, n>} 1 I-8
(24)
Proof. Write a == u. x == a a, ||a||a 2, t == |2|||2 Then
a S |a{= - 111 |
| - sl
|
|||||
2||
(25)
6
Since t 2 ||a|12 2 (1 =)ss obtain ||ē Cauchy-Schwand
c||2 -
2||n||a
2a
The inner- anner-product bound follows by
(1-e]s
I-
Theorem 2 (Robust identifiability & deterministic CRR). Let μ
and define the clean margin Do in
2m
Eq. 20 If mlla S & e|oū||2 with
D
(26)
A
then 3(2) = S and hence CRR(S)= = 1. More generally, the perturbed margin satisfies
4
min uo) max(c, us) aES
2 Au
(27)
BES
1-E
Proof. By Lemma for any j.
25
(z) - (c. uz}س
5
(28)
1-
Therefore, for any S, E S.
48
48
(G ua) (ē, u.) 2 ((c, ua) - (c, un))
A
(29)
which is Eq. 27 If the right-hand side is positive then every member still outranks every non-member, so I- for yields Eq. 26
( S. Solving Ag
To translate perturbations into CRR bound, suppose ni is an isotropic sub-Gat Sob-Gaussian vector with parameter o² (i.c., (n, v) is o-sub-Gaussian for all |ella = 1). Using standard norm tails, for some absolute e > O,
probabilistic CRR lower bound.
Pr (llll12 1) < 2exp( ( cl2 (exp0=)(/02
11> 0.
(30)
Define the separation radius r" for all (a, b) E S x ([M) \ S) whenever ||a||2 r* Hence, by a union bound over m(M m) pairs,
م+ي0 | By Theorem 2 the pairwise ranking (2, (a) > (2, us) holds
Ag
cr+ Pr (S(2) * S) < m(M m) Pr (11ll1 2 r*) 2m(M m) exp -crtt 202)
(31)
Using Eq. 14 and Eq. 20
1 (m 1)1z
(2m - 1)1c
Ao 2
(32)
m
m (m 1)4
25

---

Published as conference paper ICLR 2026
Therefore,
1+A0 )=²=²²² (1
Ag
(m 1)p)
EE [[RR(())] 2 2m(M - mm) exp
(33)
g2
m
In particular, if the dimensionless constant 4+ and CRR is near 1.
Im-I and 0² x (m 1)R). then the failure probability decays exponentially in
Dimension- corollary. ui...., uM are i.d. uniform on gd-1 (or sub- Gaussian normalized), then with probability at least M-2
logM M
A S Cy d
(34)
for an absolute constant C>> 0. Thus, whenever
d Z (2m 1)a log
(35)
we have μ 2m-1 with high probability, and Theorems ? apply. Substituting this μ into Eq. 20 and
Eq. 33 yields explicit d-M-m trade-offs: the margin scales Ao Vm C²
1)
log ad md
and CRR
concentrates near provided o² <
The orthogonal composer R renders reversibility algorithmic (RT), while mean aggregation plus low coher- ence produce a tight member/non-ment member margin Eq. [20] The normalization is stable (Lemma 2). so small perturbations preserve identifiability (Theorem 2}. This yields the exponential CRR lower bound Eq. 33 plaining why text-centric buffers that reduce effective coherence (semantic diversity) or shrink perturbations (spectral clipping) improve compositional retention.
LLM USAGE
We used a large language model for minor English editing (gramnar/wordin//llatt))) and small, localized code fixes (e.g., resolving syntax errors, adding missing imports). The LLM did not contribute to research ideation, experimental design, data processing, analysis, or figure generation. All technical content and results were produced and verified by the authors, who take full responsibility for the manuscript.
26


Reviews:
[{"review_id": "JM27Bw01M0", "reviewer": "Reviewer_fozS", "summary": "Brief Summary: The paper tackles the task of vision-language continual learning. The authors show through the exploratory study (section 3) that in continual streams, the primitive understanding (such as attributes/objects) remain stable but compositional aspect particularly zero-shot drops significantly. This the core motivation behind their method Compo-Realign which is essentially an alignment head with reversible composer, trained with info-nce loss, and clipping function. Training is done in a task-free manner. \n\nExperiments on multiple tracks such as composition, multi-domain retrieval, and continual VQA show proposed Compo-Realign significantly outperforms existing baselines by 2-5 points.", "questions": "Q1. For ablation in Table 3, instead of spectral trust region clipping, what happens if we do naive fixed clipping?\n\nQ2. In table 3, it seems text-buffer plays a significant role (2.5 points), and noted in L321. Can the authors expand why this is the case? Then it would mean, text replay is the most useful not the reversible mapping as the original claim? Maybe I am misunderstanding something here.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Pros:\n\n1. The investigation and the key problem of losing compositionality but not single-attributes/primitives makes sense to me. The Compo-Realign method is well motivated, albeit in a constrained setting of Lora adaptation. \n\n2. The proposed Compo-Realign method outperforms existing strong and recent baselines (Table 2), and authors provide single-factor ablation studies (Table 3), showing the benefits of composed positive in both retrieval and vqa settings. A range of datasets are considered including both synthetic and real images. Different downstream-tasks are considered such as composition, vqa, retrieval.\n\n3. The authors have detailed ablation and analysis with visualization graphs in both main paper and supplementary. In particular, detailed ablations on parameter sensitivity (fig7), and order sensitivity (fig8) are appreciated.", "weaknesses": "Cons: \n\n1. My main concern is that the paper mostly looks at frozen encoders (CLIP) with lightweight heads and not other options such as:\n\n(i) full-fine-tuning (also noted in Conclusion section under future work).\n\n(ii) other backbone other than CLIP, such as SigLIP or SigLIP2\n\n(iii) non-ViT CLIP (such as ResNet based) even though ViT-CLIP is the more commonly used model\n\n2. Similar to above point, while the authors are looking into Continual Vision-Language learning, generative models such as VLMs are not really considered. \n\n3. Currently, the authors assume the primitive set is already known but that might not be the case for free-form text. To my understanding, the closest would be vg-attr but that is only object + attribute pair and not free form text. Perhaps the COCO/Flickr settings are relevant? But it isn't clear how the primitives are being extracted from entire coco-captions/flickr-captions?\n\n4. The overall improvement over existing baselines is somewhat marginal (+2 points on average recall @1). \n\n---\n\nOverall Rating: 6/10\nThe paper provides an interesting setup and investigates the issue of retaining primitives but forgetting composition. The authors provide a very reasonable solution and show empirically it improves over existing baselines. The paper can be significantly strengthened by additional experiments with more backbone models, using VLM generative setup, experiments with image-text continual learning setup.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Zwq96EB6Hb", "reviewer": "Reviewer_M3AB", "summary": "This paper addresses the challenge of continual vision-language learning. The authors propose a novel method, COMPO-ReALign, designed to improve compositional generalization and structural robustness under stringent memory constraints and in the absence of task identifiers. Both empirical results and theoretical analysis are provided, making the study compelling and convincing.", "questions": "1. Can the authors provide evidence or justification to support the theoretical assumption mentioned in Weakness 1?\n2. Can the authors report detailed training time for spectral clipping, as highlighted in Weakness 2?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Clear motivation. The paper demonstrates that current vision-language (VL) models tend to forget compositional structures and proposes three diagnostic approaches to address this issue.\n2. Strong performance. The method achieves state-of-the-art results across several benchmarks.\n3. Theoretical analysis. Under certain assumptions, the paper provides a theoretical analysis to support its findings.", "weaknesses": "1. Weak theoretical assumption. In Line 1056, the paper directly assumes the range of coherence $\\mu \\lt \\frac{1}{2m-1}$ without providing a detailed analysis or justification in the empirical results.\n2. Incomplete evaluation of spectral clipping. Since spectral clipping requires the calculation of Jacobian spectra, it is necessary to report detailed training time. The current version only briefly mentions this in Line 684, which is insufficient.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "arJVTa1EpD", "reviewer": "Reviewer_6oVT", "summary": "This paper addresses the critical problem of compositional forgetting in continual vision-language learning, where models retain knowledge of primitive concepts but lose the ability to bind them together correctly. The authors introduce COMPO-REALIGN, a novel and parameter-efficient head for frozen vision-language models. The method is built on three core ideas: 1) a reversible composer that maps primitive embeddings to a compositional embedding via an orthogonal transformation, ensuring invertibility by design; 2) a multi-positive InfoNCE objective that aligns the image with both the textual composition and the synthesized compositional embedding, implicitly enforcing structural consistency; and 3) a spectral trust region that stabilizes alignment geometry by clipping parameter gradients when local sensitivity becomes too high. Through extensive experiments across three distinct continual learning tracks (compositional DIL, multi-domain retrieval, and continual VQA), the authors demonstrate that COMPO-REALIGN achieves new state-of-the-art results, significantly improving performance while substantially reducing forgetting and preserving compositional structure.", "questions": "1. Could you comment on the applicability of COMPO-REALIGN in a scenario where parts of the vision-language backbone are also finetuned? Do you foresee the core principles, particularly the spectral trust region, being adaptable to stabilize the backbone representations themselves, or is the method fundamentally designed as a post-hoc head for a fixed feature extractor?\n\n2. The spectral threshold is a crucial hyperparameter for balancing stability and plasticity. Based on your sensitivity analysis (Figure 7), could you provide more intuition or a general guideline for setting this value? Does it depend more on the model architecture (e.g., ViT-B vs. ViT-L), the task nature (e.g., retrieval vs. VQA), or the data distribution?\n\n3. Have you considered alternatives to mean-pooling, such as an attention mechanism, to create the initial composed representation? While this might add complexity, it could potentially handle more complex sentences and weigh primitives more intelligently. Would it be possible to integrate such a mechanism while preserving the desirable properties of the reversible orthogonal core?\n\n4. Your theoretical analysis in Appendix D provides a compelling lower bound on CRR under certain coherence assumptions. How does the proposed training scheme, specifically the use of a text-centric buffer with diverse examples and the spectral stabilizer, actively help the model satisfy these low-coherence and small-perturbation assumptions in practice?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper's primary strength lies in its novel formulation of a significant, real-world problem. It shifts the focus from standard catastrophic forgetting to the more nuanced and practical challenge of preserving fine-grained compositional structure. This is a crucial step towards building more reliable and robust VLMs for dynamic environments. The proposed solution is highly original, particularly the concept of enforcing \"reversibility by design\" through an orthogonal composer and stabilizing alignment geometry via a spectral trust region. This geometry-first approach is a novel and powerful paradigm for continual learning.\n\n2. The technical quality of the work is outstanding. The proposed method, COMPO-REALIGN, is theoretically well-motivated and elegant in its simplicity. The synergy between the three components—the composer, the objective, and the stabilizer—is clearly articulated and justified. The experimental validation is exceptionally thorough and rigorous. It includes multiple, diverse benchmarks, a comprehensive set of strong baselines, detailed ablation studies (Table 3), and insightful mechanism validation (Section 5.4, Figures 4 & 5). The introduction and use of diagnostic metrics like Compositional Retention Ratio (CRR) provide a much-needed tool for quantifying the specific problem being addressed, adding a layer of depth to the analysis that is often missing in CL literature.\n\n3. The paper is exceptionally well-written and organized. The motivation is clearly established through an exploratory study (Section 3), which effectively diagnoses the problem and guides the development of the proposed solution. The methodology (Section 4) is described with precision, and the mathematical formulations are clear and easy to follow. The figures and tables are informative and well-designed, effectively conveying the key results and insights. The overall narrative is compelling and logically structured.", "weaknesses": "1: The method is exclusively evaluated on a frozen CLIP backbone. While this is a common and practical setup for parameter-efficient continual learning, it limits the scope of the findings. In more challenging scenarios where the representation backbone itself must adapt (e.g., via full-model or partial finetuning), it is unclear how COMPO-REALIGN would perform or interact with backbone-level forgetting. The head might preserve compositional logic, but if the primitive representations from the backbone degrade, the overall system would still fail.\n\n2. The current composer relies on mean-pooling to aggregate primitive embeddings before the orthogonal transformation. While effective for the attribute-object pairs and simple relations tested, this approach might not scale gracefully to more complex, hierarchical, or nested linguistic structures (e.g., \"a man in a blue shirt standing next to the car that is parked behind the house\"). A discussion on the limitations of mean-pooling and potential extensions would be beneficial.\n\n3. The spectral trust region is a key component, but estimating the Jacobian spectral norm via power iteration, even if efficient, introduces implementation complexity and computational overhead compared to simpler regularization techniques. Although the authors state the overhead is <2%, a more detailed discussion on the trade-offs, especially in the context of much larger models or distributed training setups, would strengthen the paper's practical claims.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gTkS9xXge0", "reviewer": "Reviewer_KY7V", "summary": "The paper tackles continual VLM’s tendency to remember primitives (attributes/objects) but forget their compositions. It proposes COMPO-REALIGN: a tiny, plug-and-play head with a reversible composer (orthogonal via Cayley), a multi-positive InfoNCE tying textual and composed views, and a spectral trust-region to stabilize alignment—backbones stay frozen. Across compositional retrieval and continual VQA, it achieves SOTA averages with higher compositional retention and lower forgetting at <1% extra params.", "questions": "Please refer to Weakness", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1）  Clear structure-first formulation with explicit reversibility. Turning the composer’s core orthogonal via the Cayley map makes invertibility a design property rather than a penalty—clean and principled. \n\n2）  Minimal yet effective. The single multi-positive InfoNCE neatly aligns textual and composed views; spectral clipping acts as a geometry safety valve rather than an accuracy crutch. \n\n3）Practical deployment appeal. Frozen encoders, <1% extra params, and text-centric micro-buffers as structural anchors that are more memory-efficient than images; multilingual and template-form robustness are demonstrated.", "weaknesses": "1） Scope beyond retrieval/VQA? The current streams are comprehensive, yet mainly retrieval/ITM and VQA. How would the reversible composer behave for generation-style continual tasks or dense VL (e.g., referring expressions) where composition interacts with decoding? \n2） The method assumes a stable primitive inventory. If primitives are noisy, overlapping, or evolve over time, do the reversibility guarantees or CRR improvements hold? Could the composer adapt as the primitive set grows? \n\n3）Spectral thresholding choices. Spectral clipping stabilizes late-layer spikes, but how robust are results to γ and the accuracy of σ̂_max estimation with few power iterations, especially under longer streams? Any auto-tuning strategies? \n\n4）Decoding primitives from compositions at scale. The identifiability/CRR analysis and top-m readout are compelling; Are there any failure cases where coherence assumptions break down?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)