CoMem: Compositional Concept-Graph Memory for Vision–Language Adaptation
Authors: Heng Zhou
Jing Tang
Jusheng zhang
Yanshu Li
Canran Xiao
Liwei Hou
Zong Ke
Jiawei Yao
Venue: ICLR 2026 Poster (2026)
Keywords: VLM
Vision Language Learning
Continual Learning
Primary area: foundation or frontier models, including LLMs

Abstract: Continual vision–language learning is crucial for multimodal tasks such as image–text retrieval, visual question answering, and grounded reasoning in dynamic environments, yet deployed systems must learn from non-stationary streams under strict privacy and memory budgets, where naïve finetuning forgets and harms transfer. We aim to sustain stable yet plastic capability in this setting without storing raw data, enabling reuse and recombination across domains and tasks. We present CoMem, a framework that treats compositional structure as the unit of memory and rehearsal: it incrementally organizes knowledge into a compact graph of concepts and relations and rehearses directly in feature space by conditioning practice signals on sampled subgraphs. A lightweight compositional consistency objective keeps part–whole predictions coherent, while teacher-informed, uncertainty-aware filtering limits off-manifold drift. Across cross-domain retrieval, structured concept learning, and continual multimodal VQA, CoMem achieves state-of-the-art retention and transfer alongside consistent gains on SVLC and VQACL/CLOVE under matched memory and parameter budgets. By casting structure as memory and rehearsing where learning happens (feature space), CoMem provides a privacy-friendly and testable paradigm for reliable continual adaptation without raw exemplars.

Published as a conference paper at ICLR 2026
COMEM: COMPOSITIONAL CONCEPT-GRAPH MEM- ORY FOR VISION-LANGUAGE ADAPTATION
Heng Zhou'l Jing Tang Jusheng Zhang
Yanshu Li Canran Xiao Liwei Hou Zong Jiawei Yao?
1 National University of Singapore, HHuazhong University of Science and Technology
Sun Yat-sen University, "Brown University, 'Shenzhen Campus of Sun Yat-sen University "Hunan Airon Tech 'University of Washington heng. zhou@u. nus edu; xiaocr 3@ma 1 sysu. edu. en
ABSTRACT
Continual vision-language learning is crucial for multimodal tasks such as im- age-text retrieval, visual question answering. and grounded reasoning in dynamic environments, yet deployed systems must learn from non-stationary streams un- der strict privacy and memory budgets, where naïve finetuning forgets and harms transfer. We aim sustain stable yet plastic capability in this setting without storing raw data, enabling reuse and recombination across domains and tasks, We present COMEM, framework that treats compositional structure the unit memory and rehearsal: it incrementally organizes knowledge into a com- pact graph of concepts and relations and rehearses directly in feature space by conditioning practice signals on sampled subgraphs. A lightweight composi- tional consistency objective keeps part-whole predictions coherent, while teacher- informed, uncertainty-ave domain retrieval, structured concept learning, and continual multimodal VQA, COMEM achieves state-of of-the-attttee art retention and transfer alongside consistent gains on SVLC and VQACL/CLOVE under matched memory and parameter budgets. By casting structure memory and rehearsing where learning happens (feature space), COMEM provides a privacy-friendly and testable paradigm for reliable continual adaptation without raw exemplars.
filtering limits off-manifold drift. Across cross-
INTRODUCTION
Foundation dision-languree models (VLMs) such as CLIP (Rdforrd et al. 2021) have become widely used as standard backbones for variety of multimodal tasks, including image-text retrieval (Yang et al., 2024), visual question answering (Hu al. 2024 Xu et al. 2025 Zhang et al. 20256), grounded reasoning (Zhu et al. 2024; Shi et al.] 2025 Zhang et al. 2025d Ma et al. 2024a), and others(Liu et al. 2025b; Ma et al. 2024b 2025). These models enable robust cross modsl under- standing by jointly embedding visual and textual information. However, in practical deployment, these systems often face challenges arising from non-stationary and domain-shifting data streams, strict privacy and memory budgets that limit the ability to retain historical samples, and heteroge- neous objectives that frequently lack reliable task identifiers (Mao et al. 2022).
These challenges can lead significant issue: catastrophic forgetting, where fine-tuning on new tasks causes degradation in performance previously learned tasks. This problem worsens un- der conditions such as domain shifts. where models struggle maintain zero-shot performance, and with the distortion of cross-modal geometry that is crucial for transfer learning (Zheng et al.) 2023 Ni et al., 2023). Studies have shown that when pretraining continues without effective re- tention mechanisms, forgetting tends to accumulate over time, compounding the issue and further undermining performance in real-world tasks (Garg et al. 2024).
Existing solutions typically focus on three approaches: (i) preserving cross-modal geometry and limiting parameter drift (Zheng et al. 2023 Ni et al. 2023 Zhu et al. 2023), (ii) replacing raw-
"Corresponding author
"These authors contributed equally this work

---

Published as a conference paper at ICLR 2026
data replay with symbolic synthetic surrogates (Smith et al. 2023 Wu et al. 2025 Zhang al. 2025c), and (iii) reducing the number of trainable parameters via adapters or prompts (Liu et al, 2025a). While these methods offer improvements in retention and performance, they often fail to directly address the central issue of maintaining stable yel plastic compositional competence when dealing with non-stationary. multi-domain data streams, especially under strict privacy and memory constraints (Zhang et al., [202]] Xiao et al.) 2025). In structured-concepl and skill-object tasks, models still encounter difficulties in reusing knowledge learned from earlier stages, particu- larly when there are shifts in the data distribution or limited supervision (Smith et al. 2023 Zhang jt al., 2023 2021). Moreover, surrogate replay methods can inherit biases from the teacher model and provide little control over what is rehearsed, while geometry-only objectives tend to preserve alignment without fostering generalization. Parameter-efficient tuning often results in task-specific adjustments, which limits the ability to reuse learned structures. As such, there is a clear need for a unified approach that combines semantically grounded rehearsal signals with mechanisms to pre- serve transferability across domains. Existing solutions typically focus three approaches: (i) pre- serving cross modal geometry and limiting parameter drift (Zheng et al.) 2023 Ni et al. 2023 Zhu et al. 2023). (ii) replacing raw-data replay with symbolic synthetic surrogates (Smith al. 2023
Wu et al. 2025), and (iii) reducing the number of trainable parameters via adapters or prompts (Liu al. 2025a). While these methods offer improvements in retention and performance, they often fail to directly address the central issue of maintaining stable yer plastic compositional competence when dealing with non-stationary, multi-domain data streams, especially under strict privacy and memory constraints (Xiao et al. (025).. In structured-concee and skill-object tasks, models still encounter difficulties in reusing knowledge learned from earlier stages, particularly when there are shifts in the data distribution or limited supervision (Smith et al.) [023] Zhang et al., (202)). More- over, surrogate replay methods can inherit biases from the teacher model and provide little control over what is rehearsed, while geometry-only objectives tend to preserve alignment without fostering generalization. Parameter-efficient tuning often results in task-specific adjustments, which limits the ability to reuse learned structures. As such, there is a clear need for unified approach that com- bines semantically grounded rehearsal signals with mechanisms to preserve transferability across domains.
To address these gaps, we introduce COMEM, a novel continual learning framework for vi- sion-language tasks. In COMEM, we treat compositional structure the unit of memory and re- hearsal. Rather than storing raw examples, COMEM incrementally organizes tasks into a compact graph of concepts and relations. Rehearsal is then conducted in feature space, conditioning practice signals on graph substructures. This method enables the model to revisit informative combinations of concepts and relations, even under strict privacy and memory constraints. Additionally, we in- troduce a compositional consistency objective that ensures predictions remain compatible across concepts and relations, enabling the model to reuse learned structures effectively across shifting tasks. Teacher-informed filtering and uncertainty-awar distillation mechanisms are incorporated to balance the trade-off between plasticity and stability in continual learning.
The main contributions are follows:
We recast continual VLM learning as organizing a compact graph of concepts and relations, then rehearse in feature space by conditioning on its substructures. This yields targeted, privacy-friendly practice signals without storing raw images and provides a scalable unit of reuse across tasks.
1) Structure-as-mentt
2) Compositional stability. We propose a training principle that maintains consistency between parts and wholes while using teacher- and uncertainty innfommm filtering to balance plasticity and stability. The approach is complementary to geometry- based objectives and compatible with parameter-efficient tuning.
3) Reliable gains under fair budgets. On cross-domain retrieval, structured concept learning, and continual VQA, COMEM delivers higher recall and accuracy with lower forgetting under matched memory and trainable-paramete budgets, and exhibits stable behavior across seeds and reasonable hyperparameter ranges.
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORKS
Geometry and regularization. A line of methods stabilizes CLIP-like models by constraining representation geometry or parameter drift. ZSCL preserves zero-shot transfer via unlabeled- reference distillation with weight averaging and introduces an MTIL benchmark (Zheng et al.) 2023}; Mod-X aligns off-diagonal similarity structure (Ni et al., 2023); CTP adds a compatible momentum branch with
distillation (Zhu et al.) 2003)); DKR rectifies teacher affinities for retrieval (Cui et al. 2024). Probabilistic finetuning with sanguage-guieed consolidation (CLAP4CLIP) (Jha et al. 2024) and replay-frce zero-shot stability (ZAF) (Gao et al. 2024), well as LoRA-based consolidation (C-CLIP) and modality-gan modeling (MG-CLIP) (Liu et al. 2025a Huang et al. (2055)), further improve stability under domain/class shift. Yet these Peature/parameter space approaches seldom model reusable concepts and typed relations, limiting compositional trans- fer. COMEM complements them by inducing typed concept graph and enforcing composition via relation-aware replay and consistency.
Replay without raw data. Replay mitigates forgetting without storing raw data, meeting privacy and memory constraints (Zhang al. 2025a). IncCLIP synthesizes hard negative texts with cross- modal distillation (Yan et al. 2002); ConStruct-VL offers a data-free SVLC benchmark with adver- sarial aseudo-replay and Layered-LoRA (Smith et al. (023); for VQA, SGP replays scene-gre -rape prompts with pseudo QA pairs (Lei et al. 2023); diffusion based synthesis (GIFT) distills on gen- erated image-text pairs with adaptive consolidation (Wu et al. (025). Yet symbolic or pixel-level surrogates weakly encode relations and provide limited control in the feature space where learning occurs. COMEM instead replays feature-level samples conditioned on sampled subgraphs, enabling structured, on on-manifold rehearsal under tight memory budgets.
Parameter-efffcieet adaptation. Parameter-efficient adaptation (adapters, prompts, MoF) limits trainables while mitigating forgetting. TRIPLET decouples multimodal prompts for continual VQA Qian et al. 2023). DDAS routes inputs to MoE-adapters with frozen-CLLD fallback for OOD Yu et al. 2024). C-CLIP couples LoRA with contrastive consolidation {Liu et al.] (025a), and CL-MoE introduces dual-router momentum experts for MLLM VQA (Huai et al.] 2025). Recent advances include Proxy-FDA, which aligns neighborhood structure via proxy features (Huang et al., 2025a), and LADA, which appends label-specific memory units to frozen encoder (Luo et al. 2025). COMEM is orthogonal to PEFT and can pair with adapters/LoRA under matched parameter or memory budgets.
METHOD
We address continual learning over a stream of multimodal tasks (De)T= where each task sup- plies supervision image-text pairs D, {(x ((")) together with task-specific labels (c.g., classes, masks). Rather than storing instances, we maintain a concept graph memory that encodes reusable atomic concepts and typed relations, and perform rehearsal by generating feature-level samples conditioned on compositional subgraphs, As overviewed in Fig. [ the training loop at task t consists of three stages: (i) concept induction from (x, with noise-aware, teacher-frozen ver- ifier to update the graph memory; (ii) graph-conditioned replay that samples a subgraph and syn- thesizes features from the memory: and (ii)) joint optimization on real and synthetic batches with multi-objective regularization to preserve multimodal alignment and compositional consistency.
3.1 PRELIMINARIES AND NOTATION
We denote the vocabulary of attributes and entities C AUE and set of relation types by R. The
image encoder fing(+ o) maps an image to a tokenized feature map Z = fing (æ; Φ) E R Pxd with P patch tokens, and the text encoder (ix(::)) maps text to token embeddings = fixe (1) E RLxd For tasks that operate on global descriptor, we use a pooling operator x /xd - Rd and write 2 = (Z). task head h(::w) consumes either & (e.g., classification, retrieval) or Z (e.g.. dense prediction via lightweight decoder). We collect trainable student parameters as 0 (ò, yp, w) and maintain a frozen teacher (the checkpoint after task t-1). Generator and aggregator parameters kept separate i and
3

---

Published as a conference paper at ICLR 2026
Timeline of Continual Vision-Language Learning
No access to raw past data after finishing task; Fixed memory budget
non shatlonery pery: new conceptsinelations may emerge
Dret
man riding motooroycle
speeding rexing
ettul
Irage-text poirs
O
troining
jample-
M
Teacher
Enfinies
Nade counts Anchere A,
Shats PMi2/oo- accurrence
Budget
Updated Node counts f.-
Archars A.
Shats:
PMI//0-
Student
Protatyse
escurrence Pratetype
Merrory
Prototype
archar budget
Edge counts
Updated Edge counts n,
budget
Workflow of task (stegle-step training)
Ceup
Coep
trai
End of step memory
Cmn
Cor
Cromp
Cepiay
D
Image
verifier
and
nac por
Text Encoder
Feature
shart fectual pic)
prompt
Text Porser
Condidanes (nxr)
Concept- Gnoph
Groph Aggragatar
M,
Merrery
Shage Conzapt Induction
Stage Sraph Canditionee Reply
Strge Optimization
Figure 1: Training and Replay Pipeline in COMEM. Top (timeline): After completing task the model snapshot 0:-1 becomes the teacher for task i, with no access to past raw data. The model trained task using fixed memory budget B and updated concept-graph memory ,M,- At the end, O, is saved for the next task. Bottom (task-t workflow): The process involves three stages: (1) Concept Induction: Extract concept triplets (a, e, from the image-text pair and update memory. Replay: Sample a subgraph S and generate replay features ž. (3) Joint Optimization: Optimize the model with real and synthetic batches, applying losses for supervision, multimodal alignment, replay, and compositional consistency. The updated model is then written to memory.
(2) Craph-Cooditionn
3.2 NOISE-AWARE CONCEPT INDUCTION AND VERIFICATION
For each pair yz,y) extract scored concept triplets
{ (a, ae EEE, TER, We [0.
T(x, v)
(I)
Candidates (a, e, r) come from lightweight text parser 1Leex (3) (prompted Information Extraction, prompted IE), followed by visual verifier evaluated on the teacher 0 to avoid confirmation bias.
Let p(c) be short prompt for concept or relation C U R. and f. fixe (xx)((()); () E Rd the teacher embedding. We use a shared, low-rank projection for verification:
W ABT, A.BER dxd
d,
Saligu (cl Z) of LSE,E(P) (W Zp, te)
(2)
where saiggn (c Z) is the alignment score of concept with respect to the feature set Z, calculated using cosine similarity between Z, and teacher embedding L. controlled by the temperature param- eter LSE is log-sum-exp: is the sigmoid, and 7 temperature. Triplet confidence aggregates the three verifications with calibrated temperatures Ta Te: Ty. (estimated on a held-out split):
w((a e, r) [Salign (a) 7) Salign (el 2) Salign (r 7 a))r) 1/(au tac+o,)
where w(a, r, is the weighted triplet confidence score based concept alignment, Oar dex O,,- the weights for attribute, entity, and relation alignments. These weights control the contribution of each part of the triplet in the alignment calculation. We keep only triplets passing dual thresholds: w 2 y and teacher-come consistecy H[ (T(. +(Z))) < E. where predictive entropy; otherwise they queued for recheck.

---

Published as a conference paper at ICLR 2026
Concept Graph and Evidence Reservoirs. Verified triplets update typed graph g (V, E) with nodes V and directed edges SVV V x V. Each node stores a prototype μe R count n. and anchor reservoir A. R2 of most B, token features (not images). Each edge (u 4 v) keeps an interaction embedding the [R and count Re.
We update prototypes with EMA using token-level supports:
Me - (1 (1-aa) a1e + a Se, že
Σ
softmax, ((W te)) tp:)))) Z,
(4)
[Sul
(2.).....w)))...
where Se collects supports for concept Anchors are maintained online by a budgeted k-center objective with time-decay AE (0, 1).
A. f aig
min min SC SS sEsupp(c) aES
max
all2 with sampling weight Wi = v w(a e, r).
(5)
To control drift and synonymy, we periodically merge nodes with high textual cosine and prototype similarity (union-find with thresholding), and we apply age-based decay ne. neg.
Concept Graph Memory. M (9, (Ace is the concept memory, consisting of the concept
graph 9 and a set of anchor reservoirs Acsev with global budget E. Be, where each concept has its associated anchor reservoir.
Budgets are reallocated proportional to uncertainty (higher variance = larger Be).
3.3 SUBGRAPH SAMPLING
Replay subgraphs should be likely under observed co-occurrence while remaining diverse. We define a positive, normalized plausibility score using normalized Pointwise Mutual Information (NPMI) and edge counts:
Φ(Vs, Es) exp (11 Σ NPMM(( + X2
Σ log(1
Log(1 + +e))).
(6)
cEV3
eEFs
plausibility
where NPMI E [-1, is estimated from decayed co-occurrences in M and clipped to [0, 1]. For diversity, we adopt a DPP-style term on node prototypes:
A(Vs) det
del (vv),
Kg exp( ||Me Me, 12/11//p) 9z>
(7)
diversity
with qualities zi x Tee Our target (unnormalized) sampler is 4(S) x &(Vs, Es) A(Vs), with [vs Kmax-
Two-Stage Approximate Sampler. (1) k-DPP node selection: sample k ~ Unif (2. K mmax) and select Vs by greedy k-DPP MAP on K (log-det gains). (2) Connectivity projection: connect Vs by adding a minimum-cost Steiner tree over edges with cost 1/(1 + n.); if necessary, expand via BFS to reach a connected induced subgraph. We accept/reject the resulting S with a single Metropolis Hastings step using (S)) debias the greedy approximations.
3.4 PRAPH-CONDDTIONED FEATURE GENERATOR
We synthesize features in the representation space where learning occurs. Given a connected sub- graph S (Vs, Es), we form textual conditioning tokens [tu and (4r) (u-T)jeE using the student text encoder for compatibility during training. A graph aggregator computes
hg = (S) = Σ
Yu U tu +
Σ
Yuer V Prailta, ty, tw),
(8)
weV.
(u (uJEEE
where U, V E Raxr with r r drel = MLP((I-IHH) and y. are attention weights (sum within node/edge groups).
5

---

Published as a conference paper at ICLR 2026
Teacher-Guided Conditional Generator. We parameterize conditional Gaussian with separate parameters 1::
(a V(;; po(hs), ( & - pa((s) o ( Oe.
To encode relations beyond a union of node anchors, we train pa with a relation-aware Maximum Mean Discrepancy (MMD):
Egen MMD?
Kndd (( Zs), Krel(u, u; S) exp( --
Arail Pret(u, S) rel (v, 5)
(10)
where 2s Uc)
E.) pools node anchors and edge anchors E (MLP(a.lk..)
au Au, a, E EAoh and (r ; S) projects features into relation-awan space via learned bilinear maps. We add a support regularizer to keep samples close to the anchor hull:
Sup hull = max 0, dist (2, conv conv(2s)) }).
(11)
where conv(.) is approximated by nonnegative least-squares projection. We do not backpropagate Greplay (defined below) into avoid coa-r--oooooo--ff-f--- mismatch.
3.5 TRAINING OBJECTIVES
Each mini-batch interleaves real features and graph-conditioned replay: B {(Z,, Tú, label;)) U ((Em, }m)} with Sm (S)) and fm ~pol- Sm) The total loss
= (mup(h( (M(Z:w) + Amm mmm + AroCrephyy Acomphaomp Ggen Abull Cuup (12)
Task Supervision. Ssup is task-dependent; for dense tasks apply it Z via lightweight decoder, for global tasks on = T (2).
Multimodal Alignment Real and Replay. We symmetrie InfoNCE(Oord al. Lum CintoNCE(Z,T)) + LintoNCE(2, ts),
2018):
(13)
where L5 = Agg text (S) is the text-side aggregation of (lw) [1,} using the same attention as in Eq.
Replay Distillation (Teacher-Filteredd We preserve the teacher's behavior on replay while
weightegg uncertain samples:
Greplay Es,s ws,s [L (T(( m 1 + p]] ())o(()))0033111) s0(3)]2]
(14)
W8, = ([((T((()))))) E)) S E).
Compositional Consistency. We instantiate two complementary constraints.
(i) Log-Probability PoE Consistency. Let (c SS) denote the marginal concept distribution under subgraph S (using a concept head on -~<pa( 5)). For two subgraphs Si, S2 and union Su.
fo = E(S..s))
(To(
(S)) || norm(po(- (1)) @ po(. S2))) + -K.(.) rov
(15)
(ii) Relation Satisfaction via Typed Contrast. For each (a 4 e) E Es, define a tri-linear score se s, 1) (R, (0(2)), la) + (R, go (ās), te) with R, diagonal or low-rank. We use typed InfoNCE with hard negatives:
exp(sela, r. 5))
Caubgraph =
Σ log exp(sa(a,r,e 8)) + (a' Rnex exp(se(a' r,e') (S)
(16)
(o.r.)EEE
where Queg samples typed negatives sharing (a, r) or (r, e) but unseen in Es, penalized by NPMI to avoid implausible pairs and filtered by teacher consistency. The final Leomp Lpue Laubgraph-
We optimize 6, z by SGD on Eq. 12 while updating v only by V(AgenLgea + Anan Supll (no gradients from ). A two-phase schedule improves stability: warm-up for E epochs with Aromp=0 =0 and X small, then enable Leomp and ramp Aye. The pseudocode for training of COMEM shown in Algorithm|]

---

Published as a conference paper at ICLR 2026
Table 1: Cross-domain retrieval (mR higher is better). We report per-domain mR and contin- ual metrics (Avg mR, AF). Best/sccond/third are shaded from dark to light gray.
Method
COCO f Flickr30K t TAPR
RSICD
EComm
Avg mR t AF +
Mod-X Ni et al. 2023
71.5
74.2
63.8
60.4
58.9
65.8
5.3
ZSCL Zheng et al. (2023
74.1
78.0
66.9
63.2
61.0
68.6
3,9
CTP (Zuu ct al. 2023)
73.0
76.5
65.1
62.1
60.2
67.4
4.2
DKR (Cul ct al. 2024
75.0
78.5
67.4
64.6
61.9
69.5
3.5
CLAP4CLIP Jha
2024
74.2
77.1
66.1
63.0
60.6
68.2
3.7
C-CLIP [Liu al.
2025a
79.6
82.3
70.8
68.1
65.2
73.2
2.7
MG-CLIP (Huang el
2025b)
81.5
70.1
67.4
64.3
72.3
2.9
GIFT (Wu ét al.
2025
79.1
82.0
71.2
68.5
65.8
73.3
2.5
COMEM (ours)
83.2
86.5
73.1
71.4
68.9
76.6
1.9
EXPERIMENTS
4.1
EXPERIMENTAL SETUP
Datasets and Streams We evaluate COMEM three complementary streams stressing compo- sitionality, domain shift, and reasoning: (i) SVLC CConStruct-LL)- data-free sequence from VG/VAW where each task is binary image-text matching focused one concept family (Color, Material, Size, Spatial, Action, State), probing retention and recomposition without storing im- ages (Smith et al. 2023). (ii) Cross-domain retrieval followinn ZSCL/CTPZheng et al. 2023 Zhu et al. 2023). a multi-domain sequence across COCO, Flickr30K, IAPR TC-12, RSICD, and ECommerce-T2 with the same retrieval objective, testing zero-shot retention and robustness; also include a time-continus subset TTCC-DatCComp/ReCaap
for pretraining ablations (Garg al. 2024). (iii) VQA skills & trAnSfer-VQACL uses a 10-skill outer sequence with per-skill object-group sub-tasks (skillsx concepts grid), while CLOVE contrasts acene-incremente (DIL.) and
function-incremmental (TIL.) on VQA v2/TDIUC using authors' splits (Zhang et al. 2023
Lei al,
2023
Evaluation Protocols and Metrics We evaluate using the following evaluation metrics: (i) Re- trieval Recall@ 1/5/10 (R@K), mean Recall (mR), and mAP. Continual metries as Zheng et al. 2023): Last (final performance), Averuge (across tasks), and Transfer (zero-shot retention on unseen/new domains). We also report Average Forgetting (AF) and Backward/Forward Transfer (BWT/FWT) where applicable. (ii) SVLC Binary matching accuracy. AUROC, AUPRC per task and macro-averaged, continual AF/BWT/FWT. (iii) VQA Overall VQA accuracy and per-type ac- euracy (skills); continual AF, Last, Average. For VQACL we also report cross-composition accuracy where the (skill, object-group) pair was unseen during training (Zhang et al.) 2023
Baselines and fairness.
We compare COMEM against recent SOTAs for continual VLL: InCCLIP Mod-X (Ni al. 22033, ZSCL (Zheng al., (003).. CTP (Zhu al 2023). DKR
Yan al. 2022).
Cui et al. 2024). CLAP4CLIP (ha et al. 2024). ZAF (Gao et al. 20024), C-CLIP (Liu et al. 2025a).
GIFT (Wu et al. 2025). Proxy-FDA (Huang al. 2025a), LADA (Luo al. 2025 ENGINE (Zou
2025). and MG-CLIP (Huang et al.) 2025b) for retrieval / SVLC streams; and VQACL (Zhngg ct al.
2025). and CL-MoE 1.2 to see our settings for comparison with baselines, and the implementation of COMEM can be found in A.1.2
al.
2023), Symbolic Replay (SGP) (Smith al. 2023), QUAD (Marouf
(Hual et al. 2025) for VQA streams. Refer A.
4.2 MAIN RESULTS
Cross-domain retrieval. Table [ reports mean Recall (mR, %) per domain and the continual metrics. COMEM achieves the best average mR across five domains with the lowest forget- ting. improving the strongest baseline (GIFT/C-CLIP track) by +3.3 mR average and reducing AF by an absolute 0.6. Gains are consistent on both near (COCO/Flickr30K) and far NAPRRRSSCDDECComeee)) domains.

---

Published as a conference paper at ICLR 2026
Table 2: Structured concepts (SVLC) and continual VQA. Best/second/third are shaded from dark to light gray. SVLC reports macro Acc/AUROC and AF; VQA reports overall Acc on VQACI/CLOVE and AF. "_" indicates a method not applicable to that stream.
SVLC (ConStruct-VLL VL)
Continual VQA
Method
Ace t AUROC AF
VQACL Ace f CLOVE Ace AF 4
SGP Smith el al.
2023
77.3
84,9
49.5
60.1
ZAF
Gao el al. 2024g
80.3
87.1
2.6
51.2
61.0
2.4
C-CLIP (Liu el al. 2025a
79.8
86.5
2.9
50.6
60.7
2.7
GIFT
(Wu ct al. 2025)
79.9
86.9
2.7
52.0
61.4
2.9
L-MöB (Hual
20255
54.1
62.3
COMEM (ours)
82.5
88.8
2.1
55.8
63.7
1.7
SVLC and VQA. Table 2 summarizes results structured VL concepts (SVLC, ConStruct- VL) and continual VQA (VQACL/CLOVE) On SVLC, COMEM outperforms recent data-free and PEFT methods by +2.2 Acc and +1.7 AUROC while yielding the lowest AF, indicating both better concept retention and calibration. On VQA, COMEM achieves the best overall accuracy on VQACL/CLOVE with the lowest AF; compared to the strong MLLM-based CL-MoE, COMEM +1.7 (VQACL) and +1.4 (CLOVE) higher while being parameter- and memory-efficient due to feature-level replay.
Table 3: Single-factor ablations.
Retrieval
SVLC VQACL
Ablation (remove or modify one component)
Avg mR 1 AF Acc
Acc 1
COMEM (full)
76.6
1.9
82.5
55.8
Generator / Replay
w/o relation-awer aware MMD (vanilla RBF)
75.7
2.3
81.6
54.9
w/o support-hul. regularizer
76.1
2.2
82.0
55.3
stop-grad disabled (allow Creplay grads d)
75.8
81.7
55.0
node anchors only (no edge anchors Ee)
75.9
2.4
81.8
55.0
Distillation Consistency
w/o entropy gate in Creplay
75.3
2.8
81.2
54.6
w/o compositional consistency (Ccomp)
74.9
2.9
80.3
54.0
PoE only (no relation contrast)
75.4
2.5
81.0
54.5
relation contrast only (no PoE)
75.7
2.4
81.3
54.7
Verifier / Sampler
student verifier (no teacher freeze)
75.6
2.5
81.1
54.6
per-concept dense W. (no shared low-rank)
76.1
82.0
55.2
uniform node sampling (no k-DPP / Steiner MH)
75.2
2.7
81.0
54.4
no MH accept (keep k-DPP Steiner)
76.2
2.1
82.2
55.5
4.3 ABLATION ANALYSIS
We perform single-factor ablations and report Retrieval (Avg mR, AF), SVLC (Acc), and VQACL
(Acc), averaged over seeds under $4.1 (higher mR/Ace is better, lower AF is better). From Table we observe: (1) Structured replay is essential. Removing relation-aware MMD ( (-0.9 mR, +0.4 AF) or edge anchors (-0.7 mR. +0.5 AF) hurts both accuracy and retention. (2) Stability mecha- nisms matter. Disabling the entropy gate raises AF from 1.9 to 2.8, and allowing freplay gradients to the generator yields AF 2.6. (3) Compositional consistency complementary. Removing all consistency terms drops Avg mR to 74.9 (-1.7) and SVLC Acc to 80.3 (-2.2); PoE-only or relation-only recover part of the gap. (4) Plausible, diverse subgraphs help. Uniform sampling costs -1.4 mR and +0.8 AF; M-DPP+Steiner+MH is best, and removing only MH gives small decline (-0.4 mR). (5) Teacher-frozen. shared low-rank verification reduces forgetting. Using the student verifier dense per-concept projections increases AF by +0.6~0.7.
8

---

Published as a conference paper at ICLR 2026
Senntisity Bodge -26
Sconiisity Suhgraph
Scontivity
24
Anp
-2
Andher Stalet okkes))
Suhgrggh Steo Keas
Vetdae Rack
Sensitivity analysis. Shaded bands show variability across seeds; COMEM exhibits smooth trends and broad plateaus near the chosen defaults.
Figure 2:
4.4 SENSITIVITY ANALYSIS
Sensitivity to hyperparameters. Figure shows how Arg mR (1) and AF (J) vary with anchor budget B, subgraph size K maxx and verifier rank r. Curves are smooth with narrow variability bands and broad flat region near our defaults, indicating low hyperparameter sensitivity, (i) Increasing B from 8K~644 raises mR 75.8 77.77 and lowers AF 2.4-+188, 1.8, then both plateau; adopt B-64K for the best accuracy-memory trade-off. (ii) Kmax= =6 is a broad optimum (76.7 mR, 1.8 AF); <3 under-covers compositions, while 28 slightly raises AF (~2.0-2.2), supporting k-DPP+Steiner with a moderate K Kmax- Figure 5 in 1.2 further confirms robustness to loss-weight choices.
Long-Horizon Forgetting We build 18-task MTIL--tyle sequence (retrieval domains inter- leaved with SVLC families) and report, per task E, Last (avg mR over seen tasks), AF@I, BWT@1, and FWT@ I.As shown in Figure over 18-task stream, COMEM remains stable: Last @I quickly plateaus around 76.6% while AF grows slowly to only 2.2 at 1188. Its transfer dynamics are favor- able- B---BWTii is the least negative (-0.11) and FWT the highest (0.60) -iddiiiin stronger reuse on unseen tasks.
Coleres Parforaance Dear Traks
Parformance Comparkon
Forpettnn and Bacisord Townder
Forward Transte
poccposoonoocoooo
argis
170
30-
Longred PHR SJ
T45
35
Figure 3: Long-Term Stability and Performance Comparison
4.5 MEMORY- AND PARAMETER-FAN FAIR COMPARISONS
We conduct two crittca studies: (A) equal memory (MB) under fixed budget, and (B) equal PEFT parameters (trainable M). Both use the retrieval stream with the same task order.
Equal Memory (MB) match the total memory budget eeeeooo--pptcyyess--e--meeededdd for COMEM) (24, 49, 98, 196) MB and compare exemplar replay (ER/DER++), synthetic replay (GIFT-style), and small-cache variants of CLIP finetuning (CLAP4CLIP, C-CLIP), COMEM stores token anchors only, and raw images are kept.
As shown in Figure under strictly matched MB, COMEM outperforms exemplar replay by +1.3 mR and 0..7 AF at 98MB and remains competitive in wall-clock time. The advantage persists across budgets and saturates near 100MB, indicating that token-level anchors plus relation-ane aware synthesis provide more informative replay per MB than pixels generic synthetic images.

---

Published as a conference paper at ICLR 2026
20
24
26
66
Memory Budges (WB)
Figure Performance and Efficiency Comparison Continual Learning Methods
Equal PEFT Parameters We fix trainable parameter budgets at (2M, 4M, 8M, 16M} and com- pare CLiP-based PEFT methods against COMEM (whose trainables are primarily the aggregator el and generator 8). All methods use ViT-B/16, identical tokenization and schedules. Table 4 shows that COMEM consistently dominates under equal trainables c.g., at 8M parameters This supports that graph-conditioned replay and compositional constraints improve retention beyond parameter- count scaling. SVLC Acc gains at the same budget indicate stronger compositional transfer.
Table 4: Equal PEFT budget. With the same trainable parameters, COMEM yields higher mR and lower AF. At moderate budgets (8M), COMEM also leads on SVLC Acc.
2M trainables
4M trainables
SM trainables
16M trainables
Method
Avg mk 1
AP
Avg mk
AF
Avg mk
AF
Avg mR
AF
C-CLIP
0.29
7.22
0.17
0.16
2.3
LADA
0.13
0.17
76.2
2.09
76.8:0.122
2.100.06
76.1 0.09
2.1:0.04
ENGINE
0..3
0.08
75.0
2.12
7.00
2.1 0.03
2.000.06 7..0+0.11
COMEM (ours)
0.18
2.0
7.09
0.05
7.66
1.4
0.07=0008
1.800.4
CONCLUSION AND FUTURE WORK
We introduced COMEM to address continual vasion-language learning, which treats compositional structure as the unit of memory by organizing a compact concept-relation graph and rehearsing directly feature space with a lightweight consistency objective. Across cross-domain retrieval, structured concept learning, and continual multimodal VQA, COMEM consistently reduces for- getting and improves transfer under matched memory and parameter budgets, indicating that se- mantically grounded, feature-space rehearsal is more effeetive primitive than exemplar generic synthetic replay.
Our study still relies lightweight text parsing and teacher filtering and assumes fixed relation schema, which may constrain coverage in open-world settings. Future work will explore end-to-end concept discovery, integration with instruction-tunel MILMs and federated/streaming pretraining, and deployments to privacy-critical applications such as search, assistive agents, and robotics.
Ethics Statement This work adheres to the ICLR Code of Ethics. Our study does NOT involve human-subjects research, the collection of personally identifiable information, or the annotation of sensitive attributes, and we do not create any new human data. All experiments are conducted on publicly available, widely used vision- language benchmarks (COCO, Flickr30K, IAPR TC-12, RSICD, ECommerce-T2N ConStruct- SerSStructVVLSVLEE VL/SVLC, VQACL, CLOVE, and TiC-DataComp/RedCap)) strictly under their respective licenses and terms of use.
Reproducibility Statement We organize the paper and appendix enable step-by-step repro- duction. The complete experimental ders appears ders-appears in $4.1; memory- and parameter-fair comparisons detailed in $4.5 with matching rules in Appendix IA.1.2] Implementation details (backbones/okkenizztio.. verifier, memory and anchor accounting, subgraph sampler, aggregator/generatoo architectures, loss weights/scheduules,
darisclldddatsstssssteeaan metrics, baselines, and task or-
10

---

Published as a conference paper at ICLR 2026
batch composition, and hardware) are provided in Appendix $A.1.2; the full training loop is sum- marized in Algorithm [ For exact replication we fix and report seeds (42/43/44). software versions (PyTorch 2.3, CUDA 12.1), and determinism flags, and we enumerate all key hyperparameters used in main runs (e.g., anchor hudgets B-64K for retrieval and 48K for VQA; verifier rank r= 7=0.07, 7=0.6, 1=1.5; m K 166 warm-up = and loss weights in $A.1.2). Due to ongoing commercial use, we do not release source code or binaries during the review period. Upon acceptance, we will open-source a de-identified COMEM reference implementation (training, infer- ence, and logging), together with pinned environment files (Docker/Conda), task-order files, seed lists, and one-click scripts/configs that reproduce every table/figure.
REFERENCES
Zhenyu Cui, Yuxin Peng. Xun Wang, Manyu Zhu, and Jiahuan Zhou. Continual vision-language retrieval via dynamic knowledge rectification. In Proceedings the AAAI Conference on Artificial Intelligence, volume 38, pp. 11704-11712, 2024.
Zijian Gao, Xingxing Zhang, Kele Xu, Xinjun Mao, and Huaimin Wang. Stabilizing zero-shot prediction: A novel antidote to forgetting continual vision-language tasks. Advances Neural Information Processing Systems, 37: 1-8422282288888 2024.
Saurabh Garg, Mehrdad Farajtabar, Hadi Pouransari, Raviteja Vemulapalli, Sachin Mehta, Oncel Tuzel, Vaishaal Shankar, and Fartash Faghri. Tic-clip: Continual training of clip models. In The Twelfth International Conference on Learning Representations, 2024.
Wenbo Hu, Yifan Xu, Yi Li, Weiyue Li, Zeyuan Chen, and Zhuowen Tu. Bliva: simple multimodal Ilm for better handling of text-rich visual questions. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 2256222662222 2264. 2024.
Tianyu Huai, Jie Zhou, Xingjiao Wu, Qin Chen, Qingchun Bai, Ze Zhou, and Liang He. Cl-moe: Enhancing multimodal large language model with dual momentum mixture-of-experts for contin- ual visual question answering. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 19608-19617, 2025.
Chen Huang, Skyler Seto, Hadi Pouransari, Mehrdad Farajtabar, Raviteja Vemulapalli, Fartash Faghri, Oncel Tuzel, Barry-John Theobald, and Joshua M Susskind. Proxy-fda: Proxy-based feature distribution alignment for fine-tuning vision foundation models without forgetting. In Forty-second International Conference on Machine Learning, 2025a.
Linlan Huang, Xusheng Cao, Haori Lu, Yifan Meng, Fei Yang, and Xialei Liu. Mind the gap: Preserving and compensating for the modality gap clip-based continual learning. arXiv preprint arXiv:2507.09118 2025b.
Saurav Jha, Dong Gong, and Lina Yao. Clap4clip: Continual learning with probabilistic finetuning for
models. Advances in neural information processing systems, 37:129 146- 129186, 2024.
Stan Weixian Lei, Difei Gao, Jay Zhangjie Wu, Yuxuan Wang. Wei Liu, Mengmi Zhang, and Mike Zheng Shou. Symbolic replay: Scene graph as prompt for continual learning on vqa task. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 1250-1259, 2023.
Wenzhuo Liu, Fei Zhu, Longhui Wei, and Qi Tian. C-clip: Multimodal continual learning for vision- language model. In The Thirteenth International Conference on Learning Representations, 2025a.
Yunqing Liu, Wenqi Fan, Xiaoyong Wei, and Qing Li. Glprotein: Global-and and-ooal structure aware protein representation learning. In Findings of the Association for Computational Linguistics: EMNLP 2025, pp. 4355 4372, 2025b.
Mao-Lin Luo, Zi-Hao Zhou, Tong Wei, and Min-Ling Zhang. Lada: adapter for continual learning. In Forty-second International Conference on Machine Learning, 2025.
Scalable label-specific clip
11

---

Published as a conference paper at ICLR 2026
Yue Ma, Yingqing He, Xiaodong Cun, Xintao Wang, Siran Chen, Xiu Li, and Qifeng Chen. Follow your pose: fose-guided text-to-video generation using pose-free videos. Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pp. 4117-4125, 2024a.
Yue Ma, Hongyu Liu, Hongfa Wang, Heng Pan, Yingqing He, Junkun Yuan, Ailing Zeng, Chengfei Cai, Heung- Yeung Shum, Wei Liu, et al. Follow-your-emoji: Fine-controllable and expressive freestyle portrait animation. In SIGGRAPH Asia 2024 Conference Papers, pp. 1-12, 2024b.
Yue Ma, Zexuan Yan, Hongyu Liu, Hongfa Wang. Heng Pan, Yingqing He, Junkun Yuan, Ailing Zeng, Chengfei Cai, Heung-Yeun Shum, et al. Follow-your-emoj-fasseeee Towards efficient, fine-controllable, and expressive freestyle portrait animation. arXiv preprint arXiv:2509. 16630, 2025.
Yuren Mao, Yaobo Liang, Nan Duan, Haobo Wang, Kai Wang, Lu Chen, and Yunjun Gao. Less- forgetting multi-lingual fine-tuning. Advances Neural Information Processing Systems, 35: 14917-14928, 2022.
Imad Eddine Marouf, Enzo Tartaglione, Stéphane Lathuilière, and Joost van de Weijer. Ask and re- member: A questions-only replay strategy for continual visual question answering. arXiv preprint arXiv:2502.04466 04069,, 2025. UtLlttsss: http:////aaiivooo///..pff. org/ 2502. 04469
Zixuan Ni, Longhui Wei, Siliang Tang, Yueting Zhuang, and Qi Tian. Continual vision-language representation learning with off-diagonal information. In International Conference on Machine Learning, pp. 36129-26149, PMLR, 2023.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv: 0001///77077335 2018
Zi Qian, Xin Wang, Xuguang Duan, Pengda Qin, Yuhong Li, and Wenwu Zhu. Decouple before interact: Multi-modal prompt learning for continual visual question answering. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2953-2962, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- PmLR, 2021.
Chuancheng Shi, Shangze Li, Shiming Guo, Simiao Xie, Wenhua Wu, Jingtong Dou, Chao Wu, Canran Xiao, Cong Wang, Zifeng Cheng, et al. Where culture fades: Revealing the cultural gap in text-to-image generation. arXiv preprint arXiv:2511.17282 2025.
James Seale Smith, Paola Cascante-Bonille Assaf Arbelle, Donghyun Kim, Rameswar Panda, David Cox, Diyi Yang, Zsolt Kira, Rogerio Feris, and Leonid Karlinsky. Construct-vi: Data- free continual structured vl concepts learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14994-15004, 2023.
Bin Wu, Wuxuan Shi, Jinqiao Wang, and Mang Ye. Synthetic data is elegant gift for continual vision-language models. In Proceedings of the Computer Vision and Pattern Recognition Confer- ence, pp. 2813-2823, 2025.
Canran Xiao, Jiabao Dou, Zhiming Lin, Zong Ke, and Liwei Hou. From points to coalitions: Hierar- chical contrastive shapley values for prioritizing data samples, arXiv preprint arXiv:2512.19363. 2025.
Mengzhu Xu, Hanzhi Liu, Ningkang Peng, Qianyu Chen, and Canran Xiao. Affordance- first decomposition for continual learning in video-language understanding. arXiv preprint arXiv:2512.00694, 2025.
Shipeng Yan, Lanqing Hong. Hang Xu, Jianhua Han, Tinne Tuytelaars, Zhenguo Li, and Xum- ing He. Generative negative text replay for continual vision-language pretraining. European Conference Computer Vision, pp. 22-38. Springer, 2022.
12

---

Published as a conference paper at ICLR 2026
Zhenyu Yang, Dizhan Xue, Shengsheng Qian, Weiming Dong, and Changsheng Xu. Ldre: Llm- based divergent reasoning and ensemble for zero-shot composed image retrieval. In Proceedings of the 47th International ACM SIGIR conference research and development in information retrieval, pp. 80-90, 2024.
Jiazuo Yu, Yunzhi Zhuge, Lu Zhang, Ping Hu, Dong Wang, Huchuan Lu, and You He. Boosting continual learning of vision- laiinnl models via mixture-of experts adapters. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23219-23230, 2024.
Haoting Zhang, Jinghai He, Donglin Zhan, and Zeyu Zheng. Neural network-assisted simulation optimization with covariates. 2021 Winter Simulation Conference (WSC), pp. 1-12. IEEE, 2021.
Haoting Zhang, Haoxian Chen, Donglin Zhan, Hanyang Zhao, Henry Lam, Wenpin Tang, David Yao, and Zeyu Zheng. Socrates: Simulation optimization with correlated replicas and adaptive trajectory evaluations. arXiv preprint 0at::25/1.00065 2025a.
Jusheng Zhang, Yijia Fan, Wenjun Lin, Ruiqi Chen, Haoyi Jiang, Wenhao Chai, Jian Wang, and Keze Wang. GAM-agent: Game-theoretic and uncertainty-awart collaboration for complex visual reasoning. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, net forum?
2025b. |
ioso
Jusheng Zhang, Xiaoyang Guo, Kaitong Cai, Qinhan Lv, Yijia Fan, Wenhao Chai, Jian Wang, and Keze Wang. Hybridtoken-vimm Hybrid token compression for vision-language models, 2025c.
URL wttphhtts//axiiii org/ abs 2512. 08240
Jusheng Zhang, Zimeng Huang. Yijia Fan, Ningyuan Liu, Mingyan Li, Zhuojie Yang, Jiawei Yao, Jian Wang, and Keze Wang. KABB: Knowledge-awer bayesian bandits for dynamic expert coor- dination in multi-agent systems. In Forty-second International Conference on Machine Learning, 2025d. URL /tttpss:/oppereevieeee http:/eewweiiie...eeeeeeeereeeee wwt wtw.rruummiiAkvyyyaa jho
Xi Zhang, Feifei Zhang, and Changsheng Xu. Vqacl: A novel visual question answering continual learning setting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19102 19112, 2023.
Zangwei Zheng, Mingyuan Ma, Kai Wang, Ziheng Qin, Xiangyu Yue, and Yang You. Preventing zero-shot transfer degradation in continual learning of vision-language models. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 19125-19136, 2023.
Da- Wei Zhou, Kai-Wen Li, Jingyi Ning, Han-Jia Ye, Lijun Zhang, and De-Chuan Zhan. External knowledge injection for clip- clip-based class-incremental learning, arXiv preprint arXiv:2503.08510 2025.
Chenming Zhu, Tai Wang, Wenwei Zhang, Kai Chen, and Xihui Liu. Scanreason: Empowering 3d visual grounding with reasoning capabilities. In European Conference Computer Vision, pp- 151-168. Springer, 2024.
Hongguang Zhu, Yunchao Wei, Xiaodan Liang. Chunjie Zhang, and Yao Zhao. Cip: Towards vision- language continual pretraining via compatible momentum contrast and topology preservation. In P Proceedings of the IEFF/CVF International Conference on Computer Vision, pp. 22257-22267, 2023.
13

---

Published as a conference paper at ICLR 2026
A
APPENDIX
A.1 SUPPLEMENTARY TECHNICAL DETAILS
A.1.1 PSEUDOCODE FOR COMEM TRAINING
Algorithm [ trains COMEM on task by first initializing the student 0 from the frozen teacher 0 and setting aggregator & and generator 8. For each mini-batch, it encodes images/text, proposes triplets via prompted IE, and verifies them with teacher- frozen shared low-rank projector, accepting only weth--oofiddemc/looweeeee
items to update the concept- graph memory (EMA prototypes, budgeted token/edge anchors, counts with merge/decay). It then samples connected subgraphs using two-stage sampler and synthesizes replay features with text-conditioned generator, fitting the generator via relation-aware MMD and a support-hull regularizer. The student is optimized on mixed real+replay batches using the task loss, multimodal InfoNCE (real and replay), entropy-gated distillation, and compositional consistency (PoE + typed relation contrast). We update 0, vb with the total loss, and update 1 only from generator losses (no gradient from dissillatio/ccosisteeeyy. The procedure outputs the updated 0r and memory M,, Tab, summarizes the main symbols used in this paper.
Let |C] be the number of concepts, |R] relation types, and B total anchor budget. The memory stores O (lC)d + |R|d + Bd) floats for storing prototypes, edge embeddings, and anchor tokens for each concept and relation. Here, [C] is the number of concepts, [R] is the number of relations, and B is the anchor budget. Verification uses a shared low-rank W AB with cost ((Prr) per sample (vs. (PP for dense W). The k-DPP node selection is O O(K mayd) with greedy log-det gains and cached kernels; the Steiner projection is near-lineal in the local neighborhood size. Replay sampling and Gaussian synthesis are (dd) per synthetic instance. Overall, the method scales sublinearly with data volume via budgeted reservoirs and low-rank/shared projections.
Alg Algorithm 1 COMEM Training at Task (Noise-Aware, Relation-Condition Replay)
1: Inputs: D,, teacher 0, memory M
2: Initialize student 0 + f: initialize s), i
3: for epoch l....... E do
4:
for mini-batch C DD do
5:
Encode Z fing (z; の). T fex(
6:
Extract candidate triplets by 1ltext (y): verify with Eq. 2 keep if Eq. 3 and entropy pass
7:
Update M: prototypes (Eq. 4 anchors (Eq, 5) counts/merge/ddcca
8:
Sample subgraphs (Smm } M
m= via two-stage sampler ($3.3
9:
Generate replay 2m ~p Sm): compute Leen (Eq. 10P and Loup hull (Eq. 11
10:
Compute Cyup- C.m (Eq. [3}. Creplay (Eq. 14). and Coomp (Eq. 15 16
11:
Update 0, v by SGD Eq. 12 update 9 only by V (Agen Lgenn + Amull Smpp,hluu
12:
Output: Updated 6,, memory M₁
A.1.2
IMPLEMENTATION DETAILS
Backbone and tokenization We use CLIP ViT-B/16 as image encoder and paired text encoder (frozen teacher snapshots and trainable student). Patch tokens yield Z E Pxd (P=196, d=768). Results with ViT-L/14 are included in ablations.
Fair comparison settings We match backbone, input resolution, and optimizer sched- ule; for CLIP-based methods we use VIT-B/16. For memory-based baselines (e.g.. Inc- CLIP/CLAP4CLIPPGII we cap exemplar or synthetic-replay memory in MB to equal our anchor memory (anchors + prototypes relation embeddings), and for LoRA-style methods (C-CLIPLLADVVNGIIN) we equalize PEFT budgets (same total trainable parameters; default rank r=16, a232). Domain- ID usage follows each method: ZSCLLCTP/CLAPACLIIPG uated in their native (DIL/MTIL) protocols; when method assumes domain/task ID at test time, also report the domain- free variant when defined. All methods use identical task orders/splits and trained under the same hardware budget.
are eval-
14

---

Published as a conference paper at ICLR 2026
Category
Symbol
Description
C
Set of concepts (attributes and emities)
Concepts and Relations
A
Set of attributes (subtype of concepts)
E
Set of entities (subtype of concepts)
R
relations between concepés
M
Concept memory (graph and anchor reservoirs)
Memory Graph
Concept graph (nodes: concepts, edges: relations)
V
Set of nodes in the concept graph & (concepts)
Feature representation an image (oulput image
Embeddings and Projections
encoder)
T
Feature representation
(output encoder)
Shared low rank projection matrix for concept verificattio
Training Variables
Model parameters (student parameters)
Teacher model parameters (frozen)
Og Op
Weight parameters for aligament of attributes, entities, and relations
Weight Parameters
Threshold for triplet weight conlidence (see Eq. 3)
Temperature parameter controlling the sottuess alignment
fa Z)) srfo, 6,
Alignment score concept with respect to Z Triplet confidence score attribute o, entity e, relation
Triplet and Alignment
Mangr(a 2), sugele 2), Singe [2] T(.
Aligmment scores altribute, entity, relation
Set of triplcts (sttribute, catity, relation) generated from input pair (#.3)
A Bg
Anchor reservoir for concept
Replay and Memory Update
Anchor hudget concept
Tey mhe
Counts concept e and relation the memory
Lup Leamp
Supervised loss (task specitlc loss)
Functions
Comppssitional consistency loss (PoE and relation contrast)
Kuax
Maximum number nodes sampled subgraph
Sampling and Generator
Plausibility score a sampled subgraph S Diversity score for a sampled subgraph S (DPP score)
(Vs)
R@K
Recall rank K (retrioval metric)
Performance Metries
mR
Mcan recall (retrieval metric) Average forgetting (continual learning metric)
Table 5: List of important symbols.
Concept induction and verification Prompted IE runs with a constrained vocabulary for at- urbutesventiieesreltiiosss Visual verification uses the teacher-frozen shared low-rank projector W=AB" with rank r=64; temperature ==0.07; dual gate thresholds y=0.6, entropy cutoff 1=1.5 nats (validated on the first task's val split). Node/edge counts employ exponential decay (half-life tasks).
Memory and anchors Per-node token anchor cap B. S 8; total anchor budget B < 64K tokens for retrieval streams and B < 48K for VQA (fewer concepts per task). Prototypes use EMA with a=0.1. Online k-center uses farthest-firs with time-decay weight >-0.95. Edge anchors E. are formed by 2-layer MLP (hidden 512, GELU).
Subgraph sampling K nodes. Two-stage sampler: greedy k-DPP (quality q & Tha RBF kernel bandwidth from median heuristic on prototypes) - Steiner connectivity (edge cost 1/(1+n,)) + single-step MH accept using g(S) x Φ A. NPMI clipped to [o, 1] with Laplace smoothing 1=
Graph aggregator and generator Aggregator uses single-head attention with U.V E [Rdxr (r=64); Φiet 2-layer MLP (hidden 1024). Conditional Gaussian generator pa E S) outputs mean/diag-var via 2-layer MLPs. Relation-aware MMD uses an RBF kernel with bandwidth from the median heuristic on Zs, plus a relation projection term weight -.55 Per subgraph we draw K=16 synthetic features. The support-hull margin is 5=0.1 (features &g fy-normalized).
Loss weights and schedules We optimize the total loss in Eq. 12 with Amm = 1.0,, A -0.5, =0.5, Apau=0.1, 3=0.5. Two-phase schedule: warm-up E,, epoch per task with Acomp=pp =0 and small =0.2, then full weights. We stop gradients from Lreplay Ceomp to the generator 9. We sweep each loss coefficient. Results are reported as mean=std over 3 seeds on the retrieval (Avg mR, AF), SVLC (Ace), and VQACL (Ace) tracks. Figure 5 shows broad plateaus
15

---

Published as a conference paper at ICLR 2026
around the defaults, indicating low sensitivity. (i) Amm: under- weightigg cross-modal alignment (0.0) reduces retrieval mR by -0.9 and raises AF to 2.30, as image-text geometry drifts; over- weighting (1.5) brings gains. (ii) Are distillation is the main driver of eetention-removeee raises AF to 2.40; too large (1.5) slightly reduces plasticity (mR () while improving AF, matching the plasticity-stabiliyy trade-off. (iii) Acomp:' turning off compositional constraints hurts compositional generalization (SVLC 80.8%; VQACL 54.1%), confirming their role in subgraph-wise structure transfer. (iv) Agen and Abull: both shape replay quality. Without generator loss, replay distribution narrows (mR 76.0; AF 2.20); without the hull regularizer, off- manifold samples increase forgetting (AF 2.10). Larger Agen/Abull brings minor changes, suggesting stable synthesis. (v) 3: combin- ing logit- and feature-level distillation (3x0.5) is most robust; pure logit distillation raises AF, while very large 3 slightly reduces mR, consistent with over-constraining representations. Over- all, COMEM exhibits a broadly flat response around the defaults; coarse tuning suffices to obtain near-optimal performance across tasks.
40
10-
50
50-
40-
40-
20-
20
1.0
0.0
6.5
1.5
0.00
6.25
0.30
1.00
Value
h_bull
80
60
50-
40-
40-
20
20-
0.25
1,50
0.10
Value
Retieeal
SVLC
VQJCCL
Figure 5: Loss-weight sensitivity (mean We vary one coefficient at a time and measure retrieval (Avg mR/AF), SVLC (Acc), and VQACL (Acc).
Optimization and training length AdamW with decoupled weight decay 1e-4, cosine LR, For retrieval and SVLC: LR 5e-5 (student encoder/head), 1e-4 (aggregator), 2c-4 (generator). For VQA: LR scaled by 0.6. Batch size: 256 real pairs + 128 replay features per step (gradient ac- cumulation when needed). Epochs per task: COCO/Flickr30K IAPRORSICD/ECommere 6, ConStruct-VL families 4 each, VQACL/CLOVE 4. Mixed precision (bf16), gradient clipping at 1.0.
Hardware and software Experiments 8XA100-80GB, PyTorch 2.3, CUDA 12.1. We report means over seeds (42/43/44). Wall-clock time and memory are profiled with Py PyTorch profiler.
Baselines and fairness. We re-implement or use official code where available for IncCLIP (ann et al. 2022), Mod-X (Ni et al. 2023). ZSCL (Zheng et al.) 2023). CTP (Zhu et al. (023). DKR (Cui et al. 2024). CLAP4CLIP Uha et al. 2024), and ConStruct-Ve VL (Smith al. (223). matching backbone, input resolution, and memory budgets. When a method requires exemplars, we cap its exemplar memory to match anchor memory (in MB) for apples-to-apples comparison.
16

---

Published as a conference paper at ICLR 2026
A.2 ADDITIONAL EXPERIMENTS AND RESULTS
A.2.1 FINER COMPOSITIONAL TRANSFER
accuracy unseen (skill, object-group) pairs; and (ii) SVLC unseen pairs: accuracy/AUROC on concept pairs that never co-oceur training (e.g., Color Material.... We report: Acct, AUROC1, and relative gains vs. PoE-only and Relation-only ablations. Figure shows that: (i) On VQACL unseen (skill, group) pairs, COMEM improves macro
We evaluate (i) VQACL cros--cmpoostionn
VODCCL Drease Conquaaiion Parfarnnee
AVX Useanee Aocaseo
BAN ALMOO
Ooldan
Coller
Figure 6: Cross-Composition Performance Unseen Pairs: VQACL and SVLC Benchmarks
Ace by +2.0pp over PoE-only and+1.0pm +1ppp over Relation-only. with the biggest gains on Color/Count where attribute selection and object-shift composition are critical. (ii) On SVLC unseen pairs, COMEM yields consistent margins, especially on attributexattribute and spatial state/action where union reasoning and edge satisfaction must co-exist. These results validate our claim that PoE (marginal compatibility) and relation satisfaction (edge-level constraints) are complementary, their joint enforcement via Leomp and relation-aware replay key to robust compositional transfer.
A.2.2 SUBGRAPII SAMPLING MECHANISM
We conducted experiments verify whether DPP/Steiner/MH are necessary. With fixed proposal budget (IK proposals/epoch, m6x we compare six samplers: {Uniform, NPMI-only, DPP- only, NPMI+DPP, NPMI+DPP4Steiner, NPMI-DDP--Sténer+MME]) A proposal is accepted if satisfies the plausibility/diversity thresholds used across methods (for MH, acceptance follows the MH rule). We log: retrieval Avg mR/AF, acceptance rate (%), mean prototype distance within subgraph (avg pairwise fa across node prototypes), rare relation coverage (%, edges from the bottom 20% of relation frequencies), and rare concept coverage (%, nodes from the bottom 20% of concept frequencies).
Sampler
Avg mR (Si
AF
Acceptance (%)
Proto Dist 1
Rare Rel. Cov. (%) Rare Concept Cov, (%)
Uniform
62.1
0.71
NPML-only
DPP-only
75.9
2.3
70.2
0.78
16.1
22.4
NPMI + DPP
76.3
2.1
72,8
0.79
22.7
23.1
N -PP-
DPP Steiner MH (ours)
68.4
0.80
Table 6: Sampler comparison (fixed proposals/epoch). NPMI boosts plausibility and rare-edge coverage; DPP increases prototype spread (diversity): Steiner reduces poor-connectivity samples and improves acceptance; final MH step slightly lowers acceptance but improves sample quality, yielding the best mR and AF. "Proto Dist" is the mean pairwise distance among node prototypes within a sampled subgraph (higher implies more diverse concepts).
Table 6 shows a clear compositional effect: (i) Plausibility (NPMI) raises acceptance and rare re- lation coverage, yielding lower AF; (ii) Diversity (DPP) increases iubra-subgraph prototype spread, improving mR but without NPMI it undersamples rare relations; (iii) Steiner improves connectiv- ity/feasibility, lifting acceptance back up and reducing AF; (iv) MH trades modest acceptance
17

---

Published as a conference paper at ICLR 2026
drop (-7.5 pp) for the best quality per accepted subgraph, delivering the highest mR and lowest AF. Overall, NPMI (plausibility) + DPP (diversity) + Steiner (connectivity) + MH (quality control) is necessary to achieve both high accuracy and low forgetting under fixed sampling budget.
A.2.3 TEACHER STRATEGY
The motivation of this experiment is to show that "teacher-fitered" replay is principled rather than ad hoc. We compare three teachers for gating replay on synthetic features 2: (i) Prev the previous-task snapshot 0=0,-1, (ii) Init the original pretrained model, and (iii) EMA an exponential moving average of within task t (decay 0.999). We sweep the entropy thresh- old { {1.0, 1.5, 2.0,2.5) (nats) in the indicator ws -
8)) S E.. We adopted the following metrics: Average Forgetting (AF4), filtered ratio (% of replay removed by the gate; higher means stricter), and the support-hull statistics of w.r.t. 3ş: on-manifold rate (% with (2, cZnv(Zs)) 8), mean hull distance, and 90th-eecenttle distance.
Eakaay EHAM
Baway TMAM
Barep Dlesseol (ara
Browgy Tonchaad (a00)
Figure 7: Teacher choice and entropy threshold ç. EMA yields the lowest AF and hull distances across broad & range; the snapshot teacher is close second with sweet spot at f=1.5; the pretrained teacher either over-filters low (too few replays) or admits off off-manifold samples at high& & (higher AF). On-manifold judged by dist(2, conv( S 5 with 8-0.1.
Figure 7 shows that EMA teachers provide the most reliable filter: 1=1.5 they minimize AF (1.70) while maximizing on-manifold rate (86%) and lowering hull distances, indicating cleaner replay. The snapshot teacher performs nearly well with similar sweet spot: making & too strict (1.0) under-rehearses. while too loose admits off-manifold samples and increases forgetting. The pretrained teacher suffers from domain/task mismatch: it either over-filters (high filtered%) over- admits off-manifold replay (large hull distances), yielding consistently higher AF, Overall, "teacher- filtered" replay is effective when the teacher tracks recent tasks (EMA/snapshot), and moderate entropy gate (5E(1.., 2.0) offers a broad, stable optimum.
A.2.4 ANCHOR CONTRIBUTION
To pinpoint which replay sources drive compositional generalization, we compare three configu- rations in the generator target set Zs: (i) A node token anchors only; (ii) A node anchors plus edge anchors -MLP(a.,
(iii) ts (text-only) - no visual anchors, re- play conditioned only on aggregated text ts from the subgraph. We adopted the following metries: (a) SVLC relations: AUROC (1) on relation-centric families; (b) VQA relations: accuracy (1) on relation- foccued skills.
Figure 8 localizes the source of replay gains: (1) Edge anchors matter for relations. Eedge adds explicit interaction evidence, yielding +1.7 pp AUROC (SVLC) and +1.6 pp Acc (VQA) over node- only. (2) Text-only is insufficient. Conditioning on is without visual anchors underperforms node- only, indicating that relation transfer needs visual grounding in addition to textual compatibility. (3) Interpretation. Node anchors capture object/atrribute priors; edge anchors inject pairwise structure that the relation-and aware MMD can align to, improving composition where "who-does what- what-wheree" decisive.
18

---

Published as a conference paper at ICLR 2026
(AUROC,
Relation-Ceatric Skills (Accuracy. 1)
85
65.5
Accuraicy
12
Relation Familliss
Relation-Cottrt Skills
Figure 8: Performance comparison of different anchor configurations on SVLC and VQA relation tasks.
A.2.5 ADVERSARIAL TASK ORDERS
We evaluate three 18-task streams with identical content but different orders: (1) Default (balanced mix); (2) Long- Long-Til-Firrss (rare concepts first, head later); (3) Low Low->High NPMI (from least plau- sible to most plausible compositions). We report the difference of AF curves relative to Default, i.e., AAF@/ (ordr dd@autlault We also summarize AF@ 18, area under AAF, peak AF, and final Last @ mR.
Sa
wig NPSE
Af-rslated Metrics
AUCIAFTI
Notrics
Pertaak flelative Delaut Over Time
3.0
256
Lang-au--Fed Low--tghtMt
1:9
3.5
AUCOAA 1044
4.2
2.70
3.6
Matica
Nac
Vania
Teah
Figure 9: Performance Comparison of Different Task Orders
Figure 9 shows that adversarial orders amplify forgetting compared Default: Long- Long-Tail-Fin Tail-First raises AF moderately (AF@ 18 +0.30), while Low High NPMI harsher (AF@ 18 +0.40; larger low- www-plusibilltyy
AUC()AF).. This aligns with COMEM'S mechanism: early exposure compositions yields fewer reliable anchors and more off-manifold replay, inflating AF until mem- ory densifies. Despite this, COMEM remains stable: Last mR drops only 0.3-0.5 pp 18-8, and AAF plateaus rather than Snverging-sugggeee our plausibility-awaz sampling, entropy- gated distillation, and relation-aware replay effectively contain order-induced drift.
A.2.6 BACKBONE SCALE
Industrial deployment requires clear accuracy-cost trade-offs and scalability. We compare COMFM on CLIP ViT-B/16 vs. ViT-L/14 under (A) equal memory (anchor budget in MB) and (B) equal PEFT parameters (trainable M). both settings we fix data, schedule, and optimizer.
From Figure 10 (i) Scalable gains with moderate cost. VIT-L/14 improves mR by +0.6++1.1 pp
and reduces AF by 1-0.1 across regimes. The best AmR/resource appears at mid budgets (49- (49-98MB or 4 8M trainables), where AmR/GB x 0.1 and AmR/GPU-h z= 0.33-0.45. (ii) Diminish- ing returns. At high memory (196MB) or high PEFT (16M), accuracy saturates while cost contin- ues to rise, lowering the ratios. (iii) Because COMEM'S replay and consistency operate in feature
19

---

Published as a conference paper at ICLR 2026
Peokeranen Expparive Nemar
How Creramplin Menerr
(A) Kilviiees Equel Moreers
-13
-13
Seigh DR)
www.rronneCepprree.keekoo PEPT)
<31 Homerry Cveramarion PRpT)
201 Arfaeeno Comprrivent-emnt
133114 Stawe
11:
134
Trausdier
Figure 10: Comparison of ViT-L/14 and ViT-B/16 under equal memory and equal PEFT budget conditions
space, stronger image features (ViT-L/144 enlarge the anchor manifold and improve relation- alignment-yieldiee stable, compute-aware improvements without changing the algorithm.
A.2.7 ROBUSTNESS TO FINE-TUNING STRATEGY
We compare two regimes: (i) full fine-tuning of the student encoders/heads (our default), and (ii) parameter-effiient fine-tuning (PEFT), where cap trainable parameters and primarily train the aggregator v and generator & (no LoRA/adapters are required by COMEM, but this setting matches PEFT budgets used by baselines). Across settings, COMEM remains strong: full fine-tuning yields the best absolute performance, and even under tight PEFT budgets our method surpasses recent SOTAs.
Table 7: CoMem under different fine-tuning strategies. Retrieval reported as Avg mR (1) and AF (L), averaged over seeds. PEFT budgets denote total trainables. Full fine-tuning gives the highest absolute accuracy; PEFT keeps mR nearly unchanged while slightly lowering AF.
Strategy
Retricval Avg mR (1)
(L)
Full fine-tuning (encoders head)
76.6
PEPT-2M trainables (mainly v/, 1
75.8 + 0.10
2.0
0.05
PEPT-4M trainables (mainly vh (
76.1 + 0.09
1.9 + 0.05
PEFT-8M trainables (mainly 1/
76.6 + 0.08
1.8
+ 0.04
PEFT-16W. 16M trainables (mainly
76.7 0.08
1.8 + 0.04
We can find: (1) Full fine-tuning is best in absolute terms. On the cross-domain retrieval stream, COMEM attains the highest Avg mR with low forgetting. It also leads on SVLC and VQACL in the full-FT setting (cf. Tables and 2. (2) With only 8M trainables, COMEM matches full-FT retrieval (76.6 mR) while further reducing AF to 1.8. and exceeds PEFT SOTAs by +0.8-1.0 mR and 10.3-00.6 AF. On compositional benchmarks, COMEM +2.1 Acc on SVLC and +3.9 Acc VQACL over the best competing PEFT baseline. (3) Why it holds up with fewer trainables. Treating structure as memory and rehearsing in featurr space makes learning less sensitive to the size of the updateable parameter graph-conditioned replay supplies targeted, on-manifold practice
20

---

Published as a conference paper at ICLR 2026
Table 8: PEFT (8M trainables): comparison with recent SOTAs. COMEM retains near-full- FT retrieval while improving forgetting and compositional transfer. Retrieval on the cross-domain sequence (Avg mRt/AFL); SVLC reports macro Acet; VQACL reports overall Acct.
Method (PEFT, 8M)
Retrieval Avg mR /F AF (1)
SVLC Acc (1)
VQACL Ace (t)
C-CLIP Liu et al.
2025a
75.6 / 2.4
79.3
50.9
LADA Luo el al 2025
75.8 2.1
80.0
51.5
ENGIN 2025
75.7 2.1
79.6
51.2
COMEM (ours)
76.6 1.8
82.1
55.4
Ablations on structural design choices. All results are reported on the cross cross-domain retrieval and structured concept learning setting, with average retrieval mR (Avg mR, t), average forgetting (AF, J), and accuracy downstream concept tasks (Acc., 1).
Table 9:
Ablation
Method
Avg mR (1)
AF (L)
Acc. (t)
CoMem (Fixed Relation Schema) CoMem (Dynamic Relation Schema)
76.6
1.9
82.5
Relation schema
73.4
3.2
79.7
GIFT (Baseline)
74.1
3.0
79.5
CoMem Parsing (Automated)
76.6
1.9
82.5
Parsing strategy
CoMem w/ Manually Curated Data
76.2
2.1
82.2
GIFT (Baseline)
74.1
3.0
79.5
signals, while entropy-gadd gattpy distillation curbs off-manifold drift-so both accuracy and retention remain robust even when trainables are constrained.
A.2.8 STRUCTURAL DESIGN: RELATION SCHEMA AND PARSING STRATEGY
As shown in Tab. switching from a fixed dynamic relation schema substantially increases forgetting (AF from 1.9 to 3.2) and lowers both Avg mR and accuracy, confirming that a fixed relation vocabulary acts stabilizing regularizer under tight memory and parameter budgets. In contrast, replacing automated parsing with manually curated concepts yields only marginal changes in Avg mR and AF, indicating that CoMem is robust the specific concept-induction mechanism and that lightweight parsing is a practical but not brittle design choice.
A.2.9 SENSITIVITY TO LOSS BALANCING AND PARAMETER BUDGETS
The loss ablation Tab. TO shows that CoMem's 's performance is stable across a broad range of (Ncomp: )re) choices: disabling either compositional consistency replay mildly degrades Avg mR and increases AF, while completely removing both leads to the largest drop, confirming that both components contribute but that the overall objective is not overly sensitive The parameter-budget study in Tab. further indicates that CoMem consistently outperforms strong baselines under both 1M and 2M trainables, with lower AF and higher Avg mR, and that gains persist when scaling up the trainable- parameeer budget, suggesting that the design yields robust improvements even in low-maraeeter regimes.
weight values.
-emoory
A.3 THEORETICAL ANALYSIS
We formalize COMEM's 's training at round t E (1. as one step of projected gradient descent over a convex parameter set C RR with diameter : supp,e'ex 100 0' 0'|12:
0.41 Ix m.9)) ge f..(0.) ArO En-Q. vr(0.=)
(17)
(R.().)
Here fe is the (convex) instantaneous task loss (on real data step t) and 2( is convex dis- tillation/replay potential evaluated at feature-level replay z E Rd The distribution Q is the graph- conditioned generator used by COMEM step
21

---

Published as a conference paper at ICLR 2026
Table 10: Sensitivity of CoMem to loss-weight configurations. We vary the compositional consis- tency weight doooy and replay/distilation weight Aye and report average retrieval mR (Avg mR, 1), average forgetting (AF, 1), and accuracy on SVLC and VQACL (t).
Loss Weights
Avg mR (1)
AF (L)
SVLC Acc (1)
VQACL Acc (?)
005, = (default)
76.6
1.9
82,5
55.8
Acomp 0.0, = (no comp)
75.5
2.3
81.0
54.5
Acomp
=1.0, =055
76.2
2.0
82.0
55.1
Aomp
0.5, =2-
76.3
2.1
82.2
55.3
Aamp 1.5, =0.5
75.9
2.2
81.8
54.8
Aamp 0.0, =0. (no comptreplay)
74.8
2.5
80.3
53.2
Table 11: Effect vrinable-paramett budgets on retrieval performance. We compare several con-
tinual VI. baselines and CoMem under 1M and 2M trainable parameters, reporting average retrieval mR (Avg mR, 1) and average forgetting (AF, J) with mean + standard deviation over multiple runs.
1M trainables
2M trainables
Method
Avg mR (1)
AF (L)
Avg mR (1)
AF (C)
C-CLIP LADA
72.4 0.12
2.7 0.11
73.9 0.20
2.5 1 0.14
7
2.6
0.16
2.3 0.12
ENGINE
78.0
2.12
2.4 0.16
CoMem (ours)
+ 0.10
2.06
75.8
2.0
For the ideal retention term we define
(0) == E.-P, r(0;2), PT :- 1.0)
(18)
where Pa- is the (infeasible) mixture of all past feature distributions up to t-1 (obeying data- governance). Let 0; E arg minee: F2 (0) be dynamic comparator and Vr Σ 10 87111/2 la its path-variation. We measure the dynamic regret on the ideal objective, Regr E (O.) (0:))-
We work under standard online convex optimization regularity with saplay-specific discrepancy con- trol inherited from COMEM'S structured memory.
Assumption 1 (Smoothness, Lipschitzness). Each St is convex, Ly-smooth and Gy-Lipschitz on X. The potential r(:22) is convex and Lp-stooth in uniformly in z, and its gradient in L.-Liscchitz: ||Vr(0; Vr(0; 2' z2)l2 I Lzllz 2/112. Moreover, for the ideal retention, supeek [V (8) S B.
Assumption (Replay discrepancy via anchors and MMD). Let Pane denote the anchor-inducel empirical distribution maintained by the concept-graple memory and TB the anchor coverage ra- dius in feature space: every past feature z lies within distance TB to couv(supp(( Panc)). Let Krel be the relation-aware kernel used by COMEM. Assume Vr(0;-) E Huza with RKHS norm [Vrr(0:))ll $ Ar for all f E K. If Qu is the generator distribution at step t, define &, : MMD (Qt, Panc) and Smax = max, Et-
Assumption (Teacher-gane gated stability). The distillation uses an entropy gate (as 335) There exists Kigg (0, 1] such that the effective gradient magnitude satisfies supa [[ Vr(0, 2)]22 & Kig B and the smoothness constant of r on accepted replays is at most Ly-
A replay-bias decomposition.
Define the gradient bias (ideal minus used):
(0) = Are (0) VR;
(19)
The next lemma quantifies b, terms of (i) anchor coverage I'B and (ii) generator-vs-anche MMD, both under the same relation kernel used in COMEM'S RAMMD loss. Lemma (Bias via anchor coverage and MMD). Under Assumptions
2 for any 0 E K.
[bx(0)]122 < Aro (L. Larn + Sr < Xmn,
A = L Tn + Ar Smax-
(20)
22

---

Published as a conference paper at ICLR 2026
Proof. Add and subtract Panea Vr(0; 2) and apply the triangle inequality:
| (0) - (R (O)1 - |Bq. Vr(0;2) Epa.. 2r(0;2)1| + Paanc Vr(0;2) - Ep.., Vr(O;2)
(1)
(1
For (i), by Vr(O;:) E Han and the reproducing property, (u, Q. Vr - Pp... Vr)
[Vr((:))llx 5t I A for any unit vector u, hence (t) I A, For (t), anchor coverage implies
every past feature can be written as 2 = E, a, te with dg ~ Pancs a; 2 0, Σ, a,= = and ||e| <
a;
rg. By convexity and Jensen, p,_, Vr(B: 2) = E Vr(0: E, 0;0; te). Using the L=-Lipschitzness of Vr its second argument yields || Vr([0; Σ; a;æ; e) - Vr(0; E, a.a;)) S L Lzelel| s LarB.
and averaging gives (t) < L.TB. Multiplying by Are completes the proof.
A.3.1 DYNAMIC REGRET BOUND
Theorem 1 (Dynamic regret under approximate replay). Let Assumptions 13 hold and suppose
? 1/ + )veler.. Define G. == supt,e fi() ).o (0)ll) S G5 + b and A in
Lemmal] 0
Then the dynamic regret on the ideal objective satisfies
T
Reg
dyn
=
Σ t=1
(0))
|0 - -8111
+
7
Σ 1=1
T
|1 +
T Vr D D bb)(l))l
(22)
2m
2
replay bias term
Consequently, using |190|| S G. +
|bpl| S G. + Aro and Lemma 7
112
1fT (G, + )ro
+ D __v + D AT.
Regann 5
+
(23)
2n
Choosing 17" min +G. Lythmote 1 / yields
Reg
dyn & D DGx + Am + (G, + Ano VTr + Aro AT.
(24)
Proof. By convexity of Ft and the identity gr (Ft( bo(),,,
(0.) (0) < (O.), 0: 0,) = (gt, 0: - 0T) (be)))), 0, - 0,)
(gt, 0, - 07) D [0.)
(25)
where we used Cauchy-Sehwarz Schwarz and ||0g - 0, S D for the second term. For the first term, apply the standard projected gradient inequality exanssxpass of Ixx):
||0 0;11 110++1 1:11
+ "
(g.. 6g 07>
<
(26)
2m
2
Because the comparator drifts, expand ||0++1 - 0:112
10+++1
0i+1 + +01+1 - 0l2 and bound
the cross term by 2ab < a² + 1 and the norm |0e,1
fit1l S D:
0 - 07 1 < -110.+1 0 112 2D |10 0
+ 110 - 1711..
(27)
Plugging Eq. 27 into Eq. 26 and summing Eq. 25 over 1. 1,..., and yields
T telescopes the squared distances
(0.) (T0) <
||0| 12
|1r+11
07-111 112
17
11ge 12
(28)
21)
21j
D T
+
100,
07-11
+
1
Σ
10
0,
12
+
1, (0.)).
"
2n
(29)
Dropping the neg-tegetive -110r41 07 12/) and the additional
Σ
|00
0:-1 ? gives
2n
Eq. [2 Bounding 9/ and ||b,]| by G. + A. and A,.AA (Lemma gives Eq. 23 Optimizing the quadratic in 7) under the smoothness constraint gives Eq. 24
23

---

Published as a conference paper at ICLR 2026
Interprefation. The regret has three components: (i) the usual VT term scaled by the gradient budget G. + A; (ii) a path-variation penalty (G, + VT capturing non-stationarity; (ii)) an additive linear term D Are A T stemming from replay bias. By Lemma [ A is jointly reduced by smaller coverage radius rB (larger/better anchors) and smaller generator MMD m (better relation-aware RAMMD fitting).
Corollary 1 (Strongly convex retention). If FÝ is j--ttrongly convex (e.g., via an le penalty strongly-convex proxy of the distillation term) and 7) S ( + XreLr), then
dyn RegT
S
(G, + A)
I1+1 + (TT) + DV + D AT.
(30)
2
7)
(Sketch.) Apply the standard strongly-conve OGD analysis with the biased gradient V FF; + (0e) and proceed exactly in Theorem using E, ( s (G, + (1 In(1 /T
A.3.2 FORGETTING BOUND FOR A PAST TASK
Let < t and consider the loss of the s-th task evaluated at time t, f(t),, compared to its own optimum 6%
Theorem (Forgetting control via stepwise drift and replay). Under Assumption 7 for any < t and any stepsize nj S 1 + AreL)).
L1 " 2
fe(0,) - fo() - (f.(0.) - f.
+ Σ
Gynllaull +
w=B
opt. evror at stop
S (S.(0.) - Sx(0)) + (1 8) (G + And + 2 (G. + Xre
L
(31)
If. in addition, the (accepted) replay potential is Pte-styyngly convex in on average, then the quadratic term improves to L1-2unnee, 1/ [|gu 112, reducing the drift when Ayo is moderately large.
Proof. By L smmoohhhsssss of Ss.
(.(0uti) (ow) (V),(Ow),0utz -ba) + T.f |10ut1 -ull?
(32)
Projection is non-expansive, so 110011 Oul S yllsull and (V 0u+1
0,,
[f.)ll [0u+1 - < G [lg Summing s,...,t t-1 yields the first line of Eq. 31 The second line uses |ل S G. + A.e If R. is -tronnlyy convex in (after gating and expecta- tion), the standard co-coercivity inequality gives (VR,(P.,) 0u+1 6.) < Pre 10 improving the quadratic coefficient by- 21prrerre-
0m /,.
Eq. 31 shows forgetting grows at most linearly in the horizon (L-8), with slope controlled by the effective step budgel G, + Are By Lemmal [ reducing the anchor radius and the generator MMD max exaclly what COMEM's k-center anchors and RAMMD regularizer do-tightens both dynamic regret and forgetting. The optional Pre term formalizes the stabilizing role of teacher- filtered replay: a moderately large Aye contracts inter-step drift.
A.4 LLM USAGE
We employed large language model for minor English editing- as improving grammar, word- ing, and clarity-as well as for small, localized code fixes, including resolving syntax errors and adding missing imports. The LLM played no role in research ideation, experimental design, data processing, analysis, or figure generation. All technical content and results were created and verified by the authors, who assume full responsibility for the manuscript.
24


Reviews:
[{"review_id": "7u7gqQuAf1", "reviewer": "Reviewer_mzsw", "summary": "The paper proposes a continual learning framework that builds a concept–relation graph from image–text pairs by extracting (attribute, entity, relation) triplets. It then performs subgraph-conditioned replay in the representation space, combined with teacher-guided filtering and compositional consistency losses, to mitigate forgetting. Experiments on cross-domain retrieval, structured concept matching, and continual VQA report improvements over several baselines.", "questions": "Please see the weakness.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "Clear motivation for feature-level replay with a graph memory under data-retention constraints.\n\nUse of teacher confidence/entropy gating and compositional consistency is conceptually reasonable.\n\nBroad empirical coverage with multiple tasks and ablations indicates non-trivial engineering effort.", "weaknesses": "Method design is somehow too complex but the presentation is poor.\nFigure 1 (method overview) is hard to read.\nThe diagram is overcrowded: font is too small, visual hierarchy is unclear, and symbols in the figure do not align cleanly with those in the text. It is difficult to grasp the training and replay flow from the overview alone.\n\nHeavy notation but unclear definition.\nThe paper uses many symbols (for concepts/relations/subgraphs/generator variables, temperatures, loss weights, etc.) without clear definition (e.g., s_align in Eq.(2), three α in Eq. (3) are undefined). Some symbols appear to change meaning across sections.\n\nA large number of hyperparameters with no systematic tuning protocol.\nThe method includes multiple thresholds (confidence/entropy), loss weights (distillation/consistency/contrastive), structural choices (subgraph size, anchor budget, low-rank dimension), and kernel/sampling parameters. The paper should clarify how hyperparameters are chosen, validation splits, and search budgets.\n\nNot all captions can be parsed into complete (a, e, r) triplets (or maybe in some cases, more than one entity); handling of such cases is unspecified.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eGd4TWoNaT", "reviewer": "Reviewer_cTbP", "summary": "COMEM proposes an innovative framework for continual vision-language learning, whose core idea is to treat ​​compositional structure​​ as the fundamental unit of ​​memory​​ and ​​rehearsal​​, rather than storing raw data. The core methodology involves organizing knowledge by structuring the data stream into a compact graph of concepts and relations, and conducting rehearsal by directly generating replay samples in the feature space based on sampled subgraphs. This approach integrates a lightweight compositional consistency constraint and a teacher model filtering mechanism, effectively balancing stability and plasticity. Under strict privacy and memory constraints, the method achieves superior retention and transfer performance across multiple challenging tasks, including cross-domain retrieval, structured concept learning, and continual visual question answering, significantly mitigating the problem of catastrophic forgetting in continual vl learning.", "questions": "refer to Weakness", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "-1. Innovative memory mechanism design and privacy friendly learning paradigm: COMEM's biggest innovation lies in using composite structures as memory units, which not only saves storage space but also captures the intrinsic connections between concepts. Due to not storing any raw image or text data, COMEM naturally meets strict privacy protection requirements. All replay operations are performed in the latent feature space, avoiding the risk of sensitive data leakage and making it particularly suitable for deployment in privacy sensitive scenarios such as healthcare and finance.\n\n-2. COMEM's component design has great flexibility: orthogonal to parameter- efficient methods, it can be used in conjunction with adapters such as LoRA. Supporting different subgraph sampling strategies, combining compositional consistency constraints,  teacher- and uncertainty-informed filtering mechanisms to improve its anti forgetting ability.\n\n-3. Resource friendly: Only a total anchor budget of 64K is needed to achieve good experimental results.\n\n-4. Excellent experimental performance: Significant improvement in cross domain retrieval tasks, structured concept learning, and continuous VQA tasks, robust to different hyperparameters and ViT scales, and stable performance curves in the long-horizon learning process of 18 tasks.", "weaknesses": "-1. The core of COMEM relies on a Fixed Relation Schema, which means that the identification and organization of its concepts and relationships are carried out within a predefined framework. This method is highly effective in handling known and well structured data streams, but may limit its adaptability in fully open environments.\n\n-2. Bias of Teacher Model: Although the Teacher Informed Filtering mechanism can train stably, it may also transfer the cognitive biases or knowledge blind spots of the teacher model itself to the student models.\n\n-3. COMEM is a multi-component complex system, and its training involves multiple stages such as concept induction, subgraph sampling, feature generation, and multi-objective optimization. This complexity brings a high threshold for engineering implementation\n\n-4. The appendix explores the impact of Task Orders, where COMEM's forgetting degree (AF) increases significantly when tasks appear in an adversarial order. This indicates that the stability of the model depends to some extent on the \"friendliness\" of the data flow. This affects its generalization in real-world scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ttjHrZPwg6", "reviewer": "Reviewer_hXmH", "summary": "This paper introduces a novel model framework, CoMem, designed to address the privacy and catastrophic forgetting challenges inherent in real-world Continual Vision-Language Learning (CVLL) deployment. CoMem achieves its goals through a structured, three-stage process. First, during Concept Induction, the model extracts attributes, entities, and relations from new data to update the Concept-Graph Memory. This memory is key, as it stores abstract prototypes and anchors instead of sensitive raw data, making the framework privacy-friendly while mitigating forgetting. Second, in Graph-Conditioned Replay, a small subgraph is selected from memory to generate synthetic features. The model then uses these features to \"rehearse\" past knowledge, effectively preventing catastrophic forgetting of previously learned data. Finally, during Joint Optimization, CoMem trains on a mixed batch of real data from the new task and the synthetic features from memory. A comprehensive total loss function, formulated as a weighted sum of distinct loss components, is implemented to balance the acquisition of new knowledge with the retention of previously learned information. CoMem was evaluated against state-of-the-art methods across three continual learning benchmarks: cross-domain retrieval, structured concept learning (SVLC), and continual Visual Question Answering (VQA). Across all evaluations, CoMem demonstrated superior results, achieving both the highest average performance and the lowest average forgetting rate.", "questions": "see weaknesses", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. State of the Art performance: The proposed model achieves state-of-the-art results, demonstrating the best retention and lowest average forgetting (AF) across multiple benchmarks.\n\n2. Novel Approach: The proposed model achieves its results with a novel “structure-as-memory” approach to effectively solve both privacy and memory issues.", "weaknesses": "1. Reliance on Upfront Parsing: The current method relies on a \"lightweight text parsing\" step to extract concept and relation candidates from the text.\n\n2. Fixed Relation Schema: The framework assumes a \"fixed relation schema\". This weakness, as the authors note it \"may constrain coverage in open-world settings\" where new, unseen types of relations might emerge.\n\n3. How do you ensure the generated features are semantically rich and diverse enough to capture the complex, nuanced interactions within a subgraph, and not just an average or blurry representation?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EgcpN2Seh3", "reviewer": "Reviewer_YWUY", "summary": "This paper introduces CoMem, a continual vision–language learning framework designed for dynamic, privacy-constrained environments where models must learn from non-stationary multimodal data streams without storing raw samples. Unlike traditional fine-tuning methods that cause catastrophic forgetting, CoMem conceptualizes compositional structure as the core unit of memory and rehearsal. The model incrementally builds a compact concept–relation graph, using feature-space rehearsal conditioned on sampled subgraphs to sustain knowledge retention. A compositional consistency objective ensures coherence between parts and wholes, while teacher-informed and uncertainty-aware filtering helps balance plasticity and stability. Experiments across multiple multimodal tasks, including cross-domain retrieval, structured concept learning, and continual VQA, demonstrate that CoMem achieves state-of-the-art retention and transfer performance under matched memory and parameter budgets.", "questions": "See weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. Novel Methodology: The idea of treating compositional structure as memory and conducting rehearsal directly in feature space is innovative and well-motivated for privacy-limited continual learning.\n\n2. Comprehensive Analysis: The paper clearly articulates how the proposed consistency and filtering mechanisms work together to preserve stability and plasticity, offering solid theoretical and empirical insight.\n\n3. Strong Experimental Validation: Results across multiple multimodal tasks demonstrate consistent improvements in both retention and transfer, with fair ablation comparisons.", "weaknesses": "1. The introduction could be better structured and written. It does not clearly establish the relevant background or smoothly motivate the authors’ claims. As a result, the logical flow is somewhat fragmented, making it difficult for readers to follow the argument and understand the setting.\n\n2. The CoMem framework involves multiple components in its training objective. It is unclear how sensitive the overall performance is to the balance among these losses. This complexity might hinder the practical application of CoMem in new scenarios. The authors are encouraged to provide a more analysis or discussion on the stability of these hyperparameters and their effect on robustness.\n\n3. In Table 4, increasing the number of trainable parameters does not appear to yield clear performance improvements. The authors could strengthen their claims by including experiments with smaller parameter budgets to further analyze how performance gains scale with parameter count across different methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)