Pi-CCA: Prompt-Invariant CCA Certificates for Replay-Free Continual Multimodal Learning
Authors: Jiayu Zhang
Chuangxin Zhao
Canran Xiao
Ruibo Duan
Wenyi Mo
Haoyu Gao
Wenshuo Wang
Venue: ICLR 2026 Poster (2026)
Keywords: vision–language learning
VQA
replay-free
Primary area: foundation or frontier models, including LLMs

Abstract: When deployed on non-stationary data streams, foundation vision-language models require continual updates without access to past data. However, naive fine-tuning undermines their zero-shot recognition capabilities and prompt robustness. We seek a replay-free principle that preserves pre-trained cross-modal generalization under domain/prompt shifts. We introduce Prompt-Invariant CCA Certificates (Pi-CCA), a geometry-first approach that summarizes image--text alignment with a compact certificate capturing the top-k canonical spectrum and subspace. During adaptation, we match this summary using only mini-batch statistics and induce prompt robustness via averaging over perturbations. Across MTIL, X-TAIL, VLCL, and ConStruct-VL, Pi-CCA achieves state-of-the-art performance among replay-free methods.
By optimizing alignment invariants rather than proxy signals, Pi-CCA provides a simple, generator-free, constant-memory path to continual adaptation with strong zero-shot retention and resilience to prompt/style shifts.

Published as a conference paper ICLR 2026
PI-CCA: PROMPT-INVARIANT CCA CERTIFICATES FOR REPLAY-FREE CONTINUAL MULTIMODAL LEARNING
Jiayu Zhang" Chuangxin Zhao??, Canran Xiao Ruibo Duan
Wenyi Mo², Haoyu Gao%, Wenshuo Wang**
Peking University, Institute of Automation, Chinese Academy of Sciences
3 Shenzhen Campus of Sun Yat-sen University, "Seoul National University
South China University of Technology, GGeorgia Institute of Technology 2023648 702510211111 email. scut. edu.cn
ABSTRACT
When deployed on non-stationary data streams, foundation vision-language models re- quire continual updates without access to past data. However, naive fine-tuning under- mines their zero-shot recognition capabilities and prompt robustness. We seek replay- free principle that preserves pre-trained cross-modal generalization under domain/promptt shifts. We introduce Prompt- Invarian CCA Certificatex(PI-CCAA. a geometry first ap- proach that summarizes image-text text alignment with a compact certificate capturing the top-k canonical spectrum and subspace. During adaptation, we match this summary us- ing only mini-batch statistics and induce prompt robustness via averaging over perturba- tions. Across MTHI,,, X-TAIL,, VLCLL, and ConStruct-VL, Pr-CCA achieves state-of-the- art performance among replay-free methods. By optimizing alignment invariants rather than proxy signals, PI-CCA provides a simple, generator-free, constant-memon path to continual adaptation with strong zero-shot retention and resilience to prompt/style shifts.
1 INTRODUCTION
Foundation lision-language models (VLMs) (Radford al. 2021 Awais et al.
2025
enable zero-shot
recognition and retrieval across changing domains (atel et al. 2023; Feng et al. 2024 Yada et al., 2025
Chan et al. 2025 Zhang et al. 2025 Shi et al., 2025 Ni et al. 2025 LIU al. 2026).
In practice, they must be continually ly adapted to non-stationary streams without storing past data (privacyliceesinnccost) cotiieess--oniiioooss that standard fine-tuning often vio- lates (Yao et al. 2023 Li & Ke 2025) Peng et al. 2026). This vision-language continual learning (VL-CL) setting (Zheng et al., 2023) presents two core challenges: avoiding catastrophic forgetting of cross-modal alignment (and thus zero-shot ability) and maintaining robustmess to promp//distribution shifts, typically without task IDs and under tight memory budgets (Liu et al.
while preserving zero-shot transfer and prompt
these challenges could
allow VI.Ms to be applied in wider range of fields (Xu et al. 2025 Zhhng et al. 2024 Zhan et al. 2025).
Prior VI. CL research has made notable progress via proxy constraints or architectural mechanisms: dis- tributional/logit distillation and off diagonal similarity alignment to stabilize representations (Zheng et al.)
2023 Ni et al.) 2023) Zhu et al. 2023; Cui et al.] 2024; [iu et al.] 2025) Gao et al. (0024). parameter-
efficient or router-based adapters to separate old and new knowledge Yu al. [024] Tang et al.
2024 Xu
et al. 2024). and replay or stream benchmarks to mitigate data unavailability Yan et al. 2022
Lei et al.
*Corresponding author
These authors contributed equally this work

---

Published as a conference paper ICLR 2026
2023 Yao al., 2024; Smith al., 2023 Zhang et al. 2023 Garg et al. 2024). Yet these proxies leave a persistent structural weakness: they regularize outcomes (similarities, logits, weights, routes) rather than di- rectly controlling the alignment object that underlies pross-modal generalization. As a consequence, current methods can (i) permit slow drift of the alignment geometry that drives zero-shot performance, (ii) depend on reference corpora, generators, or task metadata that are not always available, and (iii) remain brittle to prompt/style variation even when average metrics improve. This gap suggests the need for replay-free principle that preserves alignment as an invariant, not merely as a byproduct of surrogate objectives.
We ask: Can continual adaptation preserve cross-modal generalization by explicitly maintaining the ge- ometry of image-text alignment, without storing past data? Our answer is a replay-free, geometry- -iisst framework that treats alignment first-class invariant and constrains its spectral and subspace structure with a compact, task-agnostic summary. In parallel, we target robustness to prompt variation through an invariance mechanism that averages over prompt perturbations at training time.
Our contributions are as follows: (i) Insight. We recast forgetting in VI.CI. as alignment-geonen drift instead of matching proxy quantities. This idea offers principled route retain zero-shot transfer der distributional and prompt shifts. (ii) Capability. We provide a replay-free and constanl-memory con- solidation mechanism that is agnostic downstream objectives and compatible with parameter-efficient adaptation (e.g., LoRA), while introducing an explicit prompt-robustness component that reduces sensi- tivity to phrasing. (iii) Performance and Evidence. Across MTIL, X-TAIL, VLCL, and ConStruct- VL, our approach attains state-of- oftte-of-the-ar results among replay-free methods and we furnish analyses linking alignment-geomety stability to retention/transfer trends, clarifying why the method is effective.
2 RELATED WORKS
VL-CL. Early multimodal CL studied forgetting and order effects in VQA with linguistically motivated 1119)/nm Jin et al. 2020), and used task-aware gated recurrent models to approach near-zero forgetting without replay (Del Chiaro et al., 2020). With CLIP-era VLMs, the focus shifted to retaining zero-shot ability while learning new domains. Regularization aligns similarity distributions or pa- rameters (Mod-X (Ni al. 2023), ZSCL (Zheng et al. 2023), CTP (Zhu et al.) (023)) DKR (Cui al. 2024 ). Architectural and efficient variants adopt
task sequences (Greco et al. 2019;
tuning (Yu et al. 2024 Tang et al. 2024 or analytic adapters with training-free fusion for X-TAIL (Xu et al. 2024). Recent work further con- solidates via contrastive knowledge (C-CLIP (Liu et al., 2025) or stabilizes zero-shot on unlabeled data (ZAF (Gao al., 2.244). Despite progress, these methods act on proxy signals (similarities, logits, param- eters, routing) and often depend on reference data or teacher ensembles, rather than preserving the canon- ical cross-modal alignment geometry of the whitened image- text cross-covariance that underpins CLIP's retrieval and recognition. PI-CCA instead directly tracks and constrains alignment invariants (canonical correlations and subspaces) under replay-free streams.
Data-free or replay-light consolidation. When past data cannot be kept, prior work uses symbolic or synthetic stand-ins: scene-graph prompts for VQA (Lei et al. 2023), data-free benchmark with adversarial pseudo-replay and layered LoRA (Smith et al. (023). negative-text replay and bidirectional momentum for image/video pretraining (Yan et al. [2022] Gao et al. (022), diffusion--yntthsizze pairs for distillation (Wul et al. 2025), questions-only replay for VQACL (Zhang al.) 2023), and time-continual pretraining showing cumulative replay is competitive when feasible (Garg et al. (202).. Despite gains, these routes add generators and pipelines, raise privacy concerns, certificate summarizes past alignment and regularizes updates using only mini-batch statistics.
task specific. Pi PI-CCA is replay- and generator generator-free: compact
Geometry-aware preservation and prompt robustness. (SV)CCA/PCCA and CKA (Raghu et al.) 2017] Morcos et al. 2018 Korblith et al. [019 Andrew et al.) 2013) quantify subspace or spectral shifts but largely diagnostic in CL. In VL CL., Mod-X is geometry-
Representatioonssiillarty measures such
2

---

Published as conference paper ICLR 2026
inspired yet matches contrastive off-diagonals rather than canonical spectra/subspaces Ni
2023 ;
Proxy-FDA preserves local neighborhoods with proxies (Huang et al. 2025a). Prompt methods (CoOp, MaPLe) learn (multi-modal) prompts curb sensitivity (Zhou et al. 2022; Khattak al. 2023), and prompt- based CL for VQA adds modality-aware routing (Qian et al.. Overall, consolidation still targets proxy signals, not invariants of the whitened cross-met mossll covariance, leaving brittleness to prompt/style changes. PI-CCA instead uses sketched, replay-free CCA certificate: it maintains the canonical spectrum and subspaces across tasks (via EMA) and attains prompt invariance by averaging text projectors, preserving the alignment skeleton with constant memory and no past data.
Location
Count
Color
Where is
How many
A stream of task
color
refresh S
Prompt perturbations
Text Adapters
Imoge encoder
"Which color the robjects in the image?"
constroins
PI-CAA certificate
6-P
Adapters
Losses & update
7,,
Crask
Spee
Sketch projectors
Centering covariances
Cpi
R.IRR
Q2 Qt
EMA
Φ. d2
x
Figure I: PI-CCA framework. stream of tasks processed without replay. The text/image encoders fir fo are adapted via LoRA (backbones frozen), producing embeddings Z₁, Zy that yield mini-batch covari- ances with an EMA update. The whitened cross-covariance is summarized in the sketch & projectors block via fixed R, /R, to obtain Q., Q: R (from h x k bases). The Prompt perturbations module samples SNP, forms (Q("))) and their mean Q: to drive the prompt-invariance loss. compact PI-CCA certificate (Pice) Si Si constrains training while its text basis S; is refreshed from prompt perturbations. Losses Lusk Lopeci Luth Cgi} are combined to update only the LoRA parameters Φu. Φ2.
3 METHOD
We propose Prompt-Invariant Canonical Correlation Analysis Certificates (PI-CCA), replay-free con- tinual learning framework that preserves the cross-modal alignment subspace of a lasion-language model. As illustrated in Fig. [ the core idea is to summarize the geometry of image-text alignment by a compact CCA certificate that stores (i) the top-k canonical correlations and (ii) a sketch of the corresponding canon- ical subspaces. During training
tasks, we enforce spectral and subspace-angle consistency with the certificate using only mini-batch statistics, without accessing past data. Prompt invariance is achieved by averaging the certificate randomized prompt perturbations.
3.1 PRELIMINARIES AND NOTATION
Let Jo X' R. and Jt : W Rd denote the image and text encoders, parameterized with LoRA adapters Hu et al. 2022): we freeze the backbone weights 0,, O, and update only the low-rank adapter
parameters Φu. Φ: (i.c., 0. (0.. Φ.) and = (0,, Φt)). Given a mini-batch w.),,, define
centered embeddings Z, 1..,1...... Zv BBxd Z, [2.,1.... Et,B. E Bxde where 2,,
3

---

Published as conference paper ICLR 2026
fo(x)) By and Zt, fi(w;; zt with Sy, Re being batch means. Let
Σ. B-1 z Zu - I En, = -- Zi +wd,
- ΣNT = B-1 Z Zr
(1)
where Teo > 0 are ridge shrinkage coefficients ensuring positive definiteness.
The whitened cross-
covariance
M = - Eee En B
(2)
whose top-k singular value decomposition (SVD) M & Ü diag(Pis)) & defines the canonical correla- tions Pi:k (p1 2 2 Px)) and the (whitened) canonical directions UR E Bd V% E IRd with orthonormal columns.
Pi. are orthogonal projectors in the original feature spaces; S are sketched bases; Q() S are sketched Gram projectors; unless stated otherwise, distances are computed in the /--dimensional sketch space. Economy-size QR decomposition (QR) is used for orth(.).
3.2 THE PI-CCA CERTIFICATE
VLM zero-shot retrieval and open- vocabulaay recognition rely on the geometry of cross-modes alignment. Rather than storing data or distilling past logits, we capture the alignment skeleton by (i) the top-k canonical correlations (spectral invariants) and (ii) the canonical subspaces (directional invariants).
Let the reference (pre-continual) CCA quantities be Pi:k E [0, Ut E Rd, and Vi E Rd from Eq. 2 Define the original- space projectors
P = (U)T Rd.xd, P² Rdixd.
(3)
To make storage constant in de, de, we use random orthonormal sketches R. E RR. and Rg E Rdex with h < dy, (e.g., Gaussian orthogonal or subsampled Hadamard transforms). The certificate is
Pi-CCA-Cert (pi S ST),
= UT h
where 5; is prompt-invariant text sketch defined below. Equivalently, one may store sketched projectors Q = (S;) R. P. R. and Qi S; (S;)
Prompt-invariant certificate via projector averaging. Let ō P denote a prompt perturbation (syn- onym/template variation). For M perturbations {6, IM -== form original-space projectors (8m) ( Vz (6m)) and their sketches Q7(6m) = R. (dm R. Define the average sketched projector
M
Q = M Σ (6m))
(5)
m=1
and take its top-k eigenvectors:
S: eigvecs Qi) E Bock
Qi = S;;;)
(6)
Averaging projectors eliminates sign/rotation ambiguity within the canonical subspace (no Procrustes align- ment needed). By default we maintain global certificate (one per model) constructed from diverse anchor prompt set.
4

---

Published as conference paper ICLR 2026
3.3 REPLAY-FREE ALIGNMENT PRESERVATION LOSSES
Given mini-batch, compute M and its top-k SVD (Ük, Pi:k. V&). Define sketches S. R. Uk. S.
R2 Vs, Q = S.ST Q S.S. The total loss is
L Lask + X Lupse + X2 Lou + X3 Cpi ,2, 0.
(i) Permutation-stable spectral preservation Eypec. Directly pairing indices can be unstable under near- degenerate singular values. We adopt a permulation-imvarion metric and an efficient pairing surrogate. Let s-rl,()) denote sorting in descending order. Define
2
Sspee
|| (ssort((p.x)) Pisk
+
Σ
Ps
P
(8)
1=1
sorted paizing (optimal for cenvex costs)
Ky-Fan-k & sum aliguament
where & [0, 1] balances pairwise and aggregate spectral matching. For exact permutation invariance one may replace the first term by min E, (Px(()) - )² (solvable by the Hungarian algorithm, (1:9)); we default to the sorted surrogate for speed. Optionally, polynomial spectral moments be added:
Lmom Ews (r((M My) tr (MMM) My)))²
(9)
which depend only on {pt}: we use J in practice.
(ii) Subspace-angle preservation Lsub- For original-spann -sginal-space orthogonal projectors P. Q onto k-dimensional subspaces, 1|IP- Q Σ/= sin" 0; (principal angles 0,). In the h--imensional sketch space, Q. are not exact projectors of the original subspaces; we therefore use their Frobenius distance as a surrogate that preserves order/angles under near-isometric sketches (e.g., Gaussian/SRHTT
Sub = Q - 0:122 + 1 Q. - -000-111.
(10)
We further stabilize by spectral clipping: after forming each Q we project its eigenvalues to [0, I] and re- symmetrize.
(iii) Prompt-invariance Cpi-
Sample i.i.d perturbations 6m P. compute (m) and Q
R V (m) (m))
(m)T R. We align the mean projector and contract its dispersion:
M
M
M
1 2
Q
mm)
Qi
Q m)
1
270. t=1
Q
(0)
(11)
M
2M
M
(iv) Task loss Linsk- InfoNCE (Oord al. form; gradients from Eq. backpropagate jointly into Sei
We use the task's standard objective (e.g.. Information Coise-Contrastive Estimation, 2018), classification cross-entropy, or detection losses). PI-CCA is agnostic to its
3.4 STREAMING ESTIMATION WITHOUU REPLAY
To stabilize estimates across batches without storing past samples, we maintain exponential moving averages (EMA) of covariance factors:
Σ (1 - (1-3))((-") (1-1) + E END: I (1 (1-3)=(( +,3 E.e, Σ(O) (1 (1-B)E((-1 + Emer (12)
(1-1)
5

---

Published as a conference paper ICLR 2026
with 3 (0, 1]. We then form M )-12 (1())) -1/2 and compute its top-k SVD. Ridge To: Y are either fixed or adapted (e.g., Ledoit-Woll).
Stable whitening and differentiation. We implement E E 1/2 by (i) eigendecomposition with eigenvalue floor and symmetric reassembly, or (ii) r-step Newton-Schul iteration on the normalized covariance, both are followed by stop-gradient on the inverse square root if needed. Differentiable SVD is realized via Too steps of block power iteration with re-orthogonalization (QR) each step, gradients are propagated M (and hence to E..), not through the certificate.
We maintain streaming EMAs and refresh the certificate every step using a slow EMA to preserve the alignment skeleton while allowing controlled plasticity:
S orth (art(((-a) (I-a)
St orth (1-a) S ta 1 Em S(m)) M
Pi: (1-a) Pi: to Picks
(13)
where (0, 1); orth(.) returns economy-size QR basis and does not backpropagate gradients.
Full optimization PI-CCA and certuficate-refresh details are deferred to Appendix A.1 including the complete training procedure in Algorithm !
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Datasets. We evaluate PI-CCA across four widely used VL-CL tracks: (i) MTIL (multi-domail task- incremental classification) the 11-domain suite introduced by ZSCI et (2023); we follow their standard task orders. (ii) X-TAIL(cross-dome domain task-agnostic classification) the task-agnostic protocol [Xu et al. (024). where test images come from the union seen and unseen domains without any domain hint, (iii) VLCL(continul image-text retrieval)- eight sequential image-caption tasks used by C-C et al. (025) (we report both I2T/T2I). (iv) ConStruct- (strructrreed VL concepts, replay)(Smith et al. [2023] the 7-task sequence over VG/VAW for attributehrelationshipp matching. We additionally report a time-continual study on medium-scall split of TiC- CFCC//////ddss to assess temporal robustness of alignment. Exact domain list and sample counts provided in Appendix $A.2
Evaluation Protocols and Metrics. For MTIL/X-TAIL report: Average (mean accuracy over steps), Last (mean accuracy the final step), and Transfer (mean accu- racy on not-yet-seen domains at each step). For VLCL we report I2T/T2I Recall @K (pri- mary: R@ I; R@5/16 in the appendix) per task and the final-step average across tasks. For ConStruct-VL we report Final Accuracy (FA) and Average Forgetting (AF). To quan- tify zero-shot retention, follow prior work and report the performance drop (PD) on held-out zero-shot suite after the final step.
Table I: Classification tracks. PI-CCA sets a new replay- free state the art on MTIL and X-TAIL.
MTIL(())
X-TAIL
Method
Avg
Last Transfer
Avg
Last Transfer
PI-CCA(ours)
76.8
75.5
73.2
68.1
66.9
64.7
C-CLIP
2025
75.2
73.8
70.9
66.3
62.7
MG IP Hiiang al.
202.5b
73.6
72.0
70.0
66.3
65.1
63.0
Proxy-FDA Huang
2025a
72.9
71.5
69.3
64.2
61.8
LADA [Tmo
74.2
73.0
70.7
66.8
66.0
63.3
DIKI
Tang el al 20/4
74.9
73.6
71.4
67.1
65.8
63.8
74.3
70.5
ZAF
Gio al
21144
73.7
72.5
71.9
66.1 64.9
63.5
DDAS
eY al. 2024
74.1
74.1
70.6
66.5
66.1
63.1
7SCL.
Zheng
2023
72.5
71.2
69.0
65.6
64.3
63.9
Baselines. We compare against strong,
Mod-X
[NI al.
2023
73.3
72.1
69.6
65.8
64.6
62.6
replay-free SOTAs across categories: (i) Reg- ularizationn/issiilattoo ZSCL(Zheng et al.)
2023
Mod-X(Ni et al. 2023), CTP et al.
2023
ZAF(Gao al. 2024), DKR al.
2024
6

---

Published as conference paper ICLR 2026
Method
VI.
VL.CL R01
ConStruct-
ConStruct- (U)
PI-CCA(ours)
48.6 = 1,0
37
75.2 + 1.3
2,7 = 0.2
GIFT Wu et. 2025
47.3 1.2
36.5 0.7
73.9 : 1.5
3,3 4 0.3
C-CLIP Liu 2025
46.1 1,4
35,7 + 1.2
72.4
3,9
ENGINE Zhou al.
2025
44.7 111
34,5 1.6
71.3
4.4
MG CLPP (Huang 2025b
45.0 + 1.6
34.8 +
71.6
4.2
Proxy FDA (Huang 20250
43.6 + 1.7
33.8
70.5 1.9
4.6
DKR Cui 2024
45.2 1.5
35.2
71.8
4.1
ZAF
34.0
72.0
3.6
Mod-2
X
2012
4.5
Table Retrieval and structured-concept tracks. Final-step retrieval (VLCL) and ConStruct- VL. results. Pr-CCA delivers the highest I2T/T21 R@I and the best FA/AF pair while remaining replay-free. Best, second-best, and third-best cells are shaded dark gray, medium gray, and light gray, respectively. f denotes synthetic replay.
Variant
MTIL Avg (1) MTIL Last (1) VLCL I2T R@1 (t) ConStruct- VL AF (L)
Pi-CCA (full)
76,8
75.5
48.6
2.7
w/o spectral term (A1=0)
74.3 (2.5)
73.1 (2.4)
46.3 (2.3)
3,8 (L.))
w/o subspace term (Λz=0)
74.6 (2.2)
73.4 (2.1)
45,9 (2.7)
3,9 (1.2)
w/o prompt invariance (Aş=0, M-0)
75.3 (1.5)
74.0 (1.5)
47.1 (1.5)
3.3 (0.6)
w/o certificate EMA (a=0)
75.6 (1.2)
74.1 (1.4)
47.7 (0.9)
3.1 (0.4)
w/o covariance EMA (8=0)
74.1 (2.7)
72.7 (2.8
46.1 (2.5)
3.7 (1.0)
spectral moments (J=0)
76.1 (0:7)
74.9 (0.6)
48.0 (0.6)
2.9 (0.2)
Hungarian pairing (exact)
76.7 (0.1)
75.4 (0.1)
48.5 (0.1)
(0.1)
SRHT sketches (vs. Gaussian)
76.6 (0.2)
75.2 (0.3)
48.4 (0.2)
2.9 (0.2)
Table 3: Single-factor ablations. Performance drops relative the full Pi-CCA model shown in blue for each variant. Removing spectral or subspace terms causes the largest performance degradation.
Proxy-FDA(P) Hunng et al. 2025a). (ii) Purameter:fficien//Arriicee
MoF--Aappeess Yu et al.
2024 DIKI et al. 2024), C-C et al. 2025), LADA(Luo et al., 2025), ENGINE Zhou et al. al. 2025b), and the analytic adapter of RAIL(Xu et al. 2024 (with X-TAIL). For completeness we also report replaylyyuthetc-rrele
2025
(G-CLLP(HHang
references: CLAP4CLIP(ha et al. 2024) (small
memory) and GIFT Wu et al. 2025 (diffusion
(diffusion-generated) generated replay).
4.2
MAIN RESULTS
Tables T [and report our comparisons on classification style continual learning (MTIL, X-TAIL), continual imate-text retrieval (VLCL), and structured concept matching (ConStruct-VL.). Across all tracks, PI-CCA achieves the top performance among replay-free methods. MTHL PI-CCA yields the highest step- averaged and final-step accuracies while maintaining strong Transfer. Under the task-agnostic X-TAIL pro- tocol, it consistently narrows the cross-domain gap and improves zero-shot retention. For VLCL retrieval, Pt-CCA outperforms recent replay-free approaches and surpasses synthetic-replay method (GIFT) without storing or generating data. On ConStruct- VL, PI-CCA attains both the highest Final Accuracy and the lowest Average Forgetting.
7

---

Published as conference paper ICLR 2026
76.
15.6
11c
20
15.00
15.73
16:01
14:21
19.90
19.75
10.00
1723
(a) 3D Pareto: peak memory (GB, D) step time (ms, 1). MTH. Avg (t); color encodes AF (L). Filled markers are non-dominated points under (mem, time, Avg)-
(b) 2D Pareto envelope: MTIL Avg (1) versus peak mem- ory (GB, J); the curve traces the efficient frontier.
Figure 2: Certificate capacity Pareto views. (a) A robust ridge emerges for & e [48, 96], E [192, 320]; (b) the 2D envelope shows the same efficient frontier. The configuration (A:, h)) (64, 256) lies near the knee.
4.3 ABLATION STUDY AND ANALYSIS
Component-wise ablation. Table 3 removes or alters one component at a time. Removing either the spectral preservation term (Aạ =0) or the subspace-angle term (A2 = 0) causes the largest drops on MTIL and retrieval, highlighting that both spectrum and directions are necessary to preserve alignment. Disabling prompt invariance (A3 = 0, M 0) mainly hurts retention while slightly reducing retrieval, consistent with its role in mitigating prompt sensitivity. Turning off certificate EMA (a = 0) or the streaming covariance EMA (3= = 0) degrades stability, and the latter is more severe. Low-order spectral moments (J >>> 0) provide small but consistent gains over J = O. Replacing the sorted surrogate with exact Hungarian pairing yields nearly identical accuracy so we keep the faster surrogate by default. Gaussian and SRHT sketches behave similarly, with a slight edge Gaussian at our budget. In addition, Appendix $A.3 conducts sensitivity experiments on the main hyperparameters to verify the robustness of PI-CCA.
Scale and Efficiency. We sweep the certificate capacity over h E {16, 32, 48, 64, 80, 96, 128) and h {128, 192, 256, 320, 384) while keeping all other settings fixed. We report MTIL Avg. MTIL Last, VLCL 12T R@ I (all t), and ConStruct- VL AF (L). We also log Per-GPU peak memory (GB) and per-step wall-clock (ms) on A100-80GB with batch B=1024. The 3D Pareto plot in Fig. [2 highlight non-dominated settings under the joint objectives of low memory, low time, high Aug, and low AF (AF visualized color). Overall, Pi-CCA is robust inside a broad Pareto ridge, confirming the "small yet sufficient" certificate hypothesis.
Geometry > Performance: correlation evidence.
geometry drifts per suboog--suuaace-annleee drift Dang Ei= sin² 0; and spectral drift D, |p Pi relate them to performance drops AAvg (MTIL step-averaged accuracy drop, in percentage points) and AR @ (VLCL 12T R@I drop, p.p.) relative to the default knee configuration (k, hi) = (64, 256) of Pi-CCA. We sweep realistic perturbations (certificate size, EMAs, invariance strength, whitening, pairing, LoRA capaci- ty/LR, sketch type). As shown Fig. 3 larger angle/spectral drifts generally imply larger drops in Avg and R@I 1, with Dang typically the stronger predictor. In addition, $A.4 provides theoretical explanation.
We measure
Prompt invariance stress test. We stress Cpi by increasing prompt perturbation strength E [0, 1] (token- level synonym http://acc-trasslatonneeeeeee an ablated model without invariance (A==0, M-0). report VLCL 12T R@ I (t), zero-shot PD (J). and AF ConStruct- VL ({) under (i) ID templates (CLIP-style variants) and (ii) OOD templates (Appendix $..2
jitter ratio), and compare Pi-CCA (with X3=0.2, M-41
8

---

Published as conference paper ICLR 2026
3.2
2.
255
3.3
14-
0.5
6.5-
355
02
0.1
32
6.1
0.
04
Zarte
0.-18-0'k
0y-16-21k
Figure 3: Geometry + performance correlation. Each panel shows scatter, least-squares fit, and 95% CL. Pearson/Spearman are annotated. Clear positive trends with realistic scatter support that preserving CCA geometry (angles & spectrum) predicts retention rather than being a coincidental regularizer.
templates:
AF)
templates: {R@1 AF)
with Lp!
pt
Rgg RgLinnh Lpi
tre
5.0
4.25
4.00
455
45
RB (198)
47-
3.50
4.0
PO
3.25
335
3,00
44
3.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.5
0.8
1.0
Perturbation strongth
Perfurbation strength
(a) ID templates:
(R@1, PD, AF).
(b) OOD templates: s f 1, PD, AF).
Figure Prompt invariance stress curves. Lpi flattens degradation slopes for both and OOD prompts. At s= 1.0, O, Pi ICCA improves R@1 by +2.44 p.p. (ID) +2.51 p.p. (OOD) and reduces AF by =1.10 (ID) 0.96 (OOD) vs. no Cgi.
As shown in Fig we find that: (i) Invariance cuts the slope of accuracy decay: s=1.0, R@ retains 46.9 (ID) vs. 44.5 without invariance; OOD shows similar gaps. (ii) Forgetting and zero-shot drift (AF/PD) grow with 8, but Gni consistently dampens both, especially under OOD styles. (iii) The curves suggest a practical operating range 0.6 where performance remains close to nominal with invariance.
Task-order sensitivity. To examine whether PI-CCA "gets lucky" with task order, evaluate 20 independently shuffled MTIL sequences (11 domains; orders listed in Appendix $A.2). We use the configu- ration (k, h) (64, 256). Fig. summariies the ccross-order distributions, find: the interquartile ranges are small, the between-orde order span (max min) is modest, supporting that PI-CCA's 's retention is robust to task-order.
5 CONCLUSION
We addressed replay-free continual adaptation of lasion-l models by reframing forgetting as frigmment-geometty drift and introduced PI-CCA, which preserves cross-modal generalization via com-
9

---

Published as conference paper ICLR 2026
Avg across orders (1)
Last across orders (1)
AF across orders (4)
3.0
76.00
2.9
75.50
76.8
75.25
76.6
76.4
76.2
74.75
76.0
Figure 5: Task-order sensitivity. Boxplots over 20 random orders for Avg/Last (1) and AF (L). Dots show per-order means (3 seeds). Narrow IQRs indicate low order sensitivity.
pact, prompt-invariant certificate of canonical spectra and subspaces. Across standard VL-CL protocols, directly constraining these invariants maintains zero-shot behavior and reduces forgetting while remaining compatible with parameter-efficien tuning. Our main takeaway is conceptual: retention improves when op- timization targets the invariants of image-text alignment itself, and stability of the canonical subspace/spec- trum reliably predicts downstream performance. Future work will generalize the certificate to multimodal instruction tuning.
Ethics Statement This work adheres the ICLR Code of Ethics. Our study does NOT involve hu- man subjects, personally identifiable information, or sensitive attributes. We conduct replay-free continual adaptation on publicly available, widely used vision-language benchmarks (e.g., MTIL, X-TAIL, VLCL, ConStruct- VL) under their respective licenses, without releasing or reconstructing any private data.
Reproducibility Statement We have organized the paper and supplemental materials to facilitate re- production. The full experimental protocol, datasets, metrics, baselines, and task orders specified in $4.1 with additional implementation and optimization details in Appendix A (Algorithm [ and Ap- pendix $A.2 (backbones/adapters. hyperparameters, prompt perturbations, hardware, and random seeds). Our theoretical results are stated with explicit assumptions and complete proofs in the Theory section, en- abling independent verification. Dataset preprocessing and evaluation scripts are documented in the supple- mentary; we rely only on publicly available benchmarks listed in $4.1 To ensure exact-step reproducibility. we report all key hyperparameters, EMA rates, sketch dimensions, and power-iteration settings, and we provide the task-order permutations used in our sensitivity analyses (Appendix $A.2). Due to ongoing com- mercial use, we cannot release the code during review. The code package (training, evaluation, and logging) will be released in the camera-ready version subject de-identification and removal of any proprictary de- pendencies. Upon acceptance we will (i) open-source the PI-CCA reference implementation, (ii) upload configuration files and seed lists reproducing every table/figure, and (iii) include scripts regenerate all results from raw datasets.
REFERENCES
Galen Andrew, Raman Arora, Jeff Bilmes, and Karen Livescu. Deep canonical correlation analysis. In International conference on machine learning, pp. 1247-1255. PMLR, 2013.
Muhammad Awais, Muzammal Naseer, Salman Khan, Rao Muhammad Anwer, Hisham Cholakkal, Ming-Hsuan Hsuan Yang, and Fahad Shahbaz Khan. Foundation models defining a new era
Mubarak Shah, Ming- in vision: survey and outlook. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025.
10

---

Published as a conference paper ICLR 2026
Chak-Fu Chan, Peter Kok-Yiu Wong, Xiaowen Guo, Jack CP Cheng, Jolly Pui-Ching Chan, Pak-Him Leung, and Xingyu Tao. Context-aware vision-languee model agent enriched with nomain-speciffe ontology for construction site safety monitoring. Automation in Construction, 177:106305, 2025.
Zhenyu Cui, Yuxin Peng, Xun Wang, Manyu Zhu, and Jiahuan Zhou. Continual vision- languae retrieval via dynamic knowledge rectification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 11704-11712, 2024.
Riccardo Del Chiaro, Bartlomiej Twardowski, Andrew Bagdanov, and Joost Van de Weijer. Ratt: Recurrent attention to transient tasks for continual image captioning. Advances in Neural Information Processing Systems, 33:16736-16748, 2020.
Chen Feng, Georgios Tzimiropoulos, and Ioannis Patras. CLIPCleaner: Cleaning Noisy Labels with CLIP. In The 32nd ACM International Conference on Multimedia (ACM MM), 10 2024. doi: 10. 10.1145/3664647. 3680664.
Yizhao Gao, Nanyi Fei, Haoyu Lu, Zhiwu Lu, Hao Jiang, Yijie Li, and Zhao Cao. Bmu-moco: Bidirectional momentum update for continual video-language modeling. Advances in Neural Information Processing Systems, 35:22699-223 12, 2022.
Zijian Gao, Xingxing Zhang, Kele Xu, Xinjun Mao, and Huaimin Wang. Stabilizing zero-shot prediction: A novel antidote to forgetting in continual vision- langun- tasks. Advances in Neural Information Pro- cessing Systems, 37:128462-128488. 2024.
Saurabh Garg, Mehrdad Farajtabar, Hadi Pouransari, Raviteja Vemulapalli, Sachin Mehta, Oncel Tuzel, Vaishaal Shankar, and Fartash Faghri. Tic-clip: Continual training of clip models. In The Twelfth Inter- national Conference on Learning Representations, 2024.
Claudio Greco, Barbara Plank, Raquel Fernández, and Raffaella Bernardi. Psycholinguistics meets continual learning: Measuring catastrophic forgetting in visual question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 3601-3605, 2019.
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Repre- sentations, 2022.
Chen Huang, Skyler Seto, Hadi Pouransari, Mehrdad Farajtabar, Raviteja Vemulapalli, Fartash Faghri, On- cel Tuzel, Barry-John Theobald, and Joshua M Susskind. Proxy-fda: Proxy-based feature distribution alignment for fine-tuning vision foundation models without forgetting. In Forty-second International Conference on Machine Learning, 2025a.
Linlan Huang, Xusheng Cao, Haori Lu, Yifan Meng, Fei Yang, and Xialei Liu. Mind the gap: Preserving and compensating for the modality gap in clip-based continual learning, arXiv preprint arXiv:2507.091188 2025b.
Saurav Jha, Dong Gong, and Lina Yao. Clap4clip: Continual learning with probabilistic finetuning for vision-language models. Advances in neural information processing systems, 37:129146-1299 2024.
Xisen Jin, Junyi Du, Arka Sadhu, Ram Nevatia, and Xiang Ren. Visually grounded continual learning of compositional phrases. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 2018 2018-2029, 2020.
Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Maple: Multi-modal prompt learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 99113-19122 2023.
11

---

Published as a conference paper ICLR 2026
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International conference on machine learning, pp. 3519-3529. PMIR, 2019.
Stan Weixian Lei, Difei Gao, Jay Zhangjie Wu, Yuxuan Wang. Wei Liu, Mengmi Zhang, and Mike Zheng Shou. Symbolic replay: Scene graph as prompt for continual learning on vqa task. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 1250-1259, 2023.
Zichao Li and Zong Ke. Domain meets typology: Predicting verb-final order from universal dependencies for financial and blockchain nlp. Proceedings of the 7th Workshop on Research in Computational Linguistic Typology and Multilingual NLP,, pp. 156-164, 2025.
Wenzhuo Liu, Fei Zhu, Longhui Wei, and Qi Tian. C-clip: Multimodal continual learning for vision-
language model. In The Thirteenth International Conference on Learning Representations, 2025.
Yunqing LIU, Yi Zhou, and Wenqi Fan. Enhancing molecular property predictions by learning from bond modelling and interactions. In The Fourteenth Internationaa Conference Learning Representations, 2026. URL https::tpss...m...tt. /
Mao-Lin Luo, Zi-Hao Zhou, Tong Wei, and Min-Ling Zhang. Lada: Scalable label-specific clip adapter for continual learning. In Forty-second International Conference on Machine Learning, 2025.
Ari Morcos, Maithra Raghu, and Samy Bengio. Insights on representational similarity in neural networks with canonical correlation. Advances in neural information processing systems, 31, 2018.
Chaojun Ni, Cheng Chen, Xiaofeng Wang, Zheng Zhu, Wenzhao Zheng, Boyuan Wang, Tianrun Chen, Guosheng Zhao, Haoyun Li, Zhehao Dong, et al. Swiftvla: Unlocking spatiotemporal dynamics for lightweight vla models at minimal overhead. arXiv preprint arXiv:2512.0090 2025.
Zixuan Ni, Longhui Wei, Siliang Tang, Yueting Zhuang, and Qi Tian. Continual lision-language repre- sentation learning with off-diagonal information. In International Conference on Machine Learning, pp. 26129-26149, PMLR, 2023.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint 1rXiv 037448880378... 2018
Dhruvesh Patel, Hamid Eghbalzadeh, Nitin Kamra, Michael Louis Iuzzolino, Unnat Jain, and Ruta Desai. Pretrained language models as visual planners for human assistance. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15302-15314, 2023.
Wenshuo Peng, Gongxuan Wang, Tianmeng Yang, Chuanhao Li, Xiaojie Xu, Hui He, and Kaipeng Zhang. Svbench: Evaluation of video generation models on social reasoning, 2026. URL
orgg hogg/ab//2512, 21507
Zi Qian, Xin Wang, Xuguang Duan, Pengda Qin, Yuhong Li, and Wenwu Zhu. Decouple before interact: Multi-modal prompt learning for continual visual question answering. In Proceedings of the IEEE/CVF International Conference on Computer Vision pp. 2953-2962, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkín, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748 -7633 PmLR, 2021.
Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. Sveca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. Advances in neural information processing systems, 30, 2017.
12

---

Published as a conference paper at ICLR 2026
Chuancheng Shi, Shangze Li, Shiming Guo, Simiao Xie. Wenhua Wu, Jingtong Dou, Chao Wu, Canran Xiao, Cong Wang, Zifeng Cheng, et al. Where culture fades: Revealing the cultural gap in text-to-image generation. arXiv preprint arXiv:2511.1728., 2025.
James Seale Smith, Paola Cascante- Bonilla, Assaf Arbelle, Donghyun Kim, Rameswar Panda, David Cox, Diyi Yang, Zsolt Kira, Rogerio Feris, and Leonid Karlinsky. Construct-vl: Data-free continual structured vi concepts learning. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 14994-15004, 2023.
Longxiang Tang, Zhuotao Tian, Kai Li, Chumming He, Hantao Zhou, Hengshuang Zhao, Xiu Li, and Jiaya Jia. Mind the interference: Retaining pre-trained knowledge in parameter efficient continual learning of vision-language models. In European conference on computer vision, pp. 346-365. Springer, 2024.
Bin Wu, Wuxuan Shi, Jinqiao Wang, and Mang Ye. Synthetic data is an elegant gift for continual vision- language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 2813- 2823, 2025.
Mengzhu Xu, Hanzhi Liu, Ningkang Peng, Qianyu Chen, and Canran Xiao. Affordance-first decomposition for continual learning in video-language understanding. arXiv preprint arXiv:2512.00694, 2025.
Yicheng Xu, Yuxin Chen, Jiahao Nie, Yusong Wang, Huiping Zhuang, and Manabu Okumura. Advanc- ing cross-domain discriminability in continual learning of vision-language models. Advances in Neural Information Processing Systems, 37:51552-51576, 2024.
Yuki Yada, Sho Akiyama, Ryo Watanabe, Yuta Ueno, Yusuke Shido, and Andre Rusli. Improving visual rec- ommendation on e-commerce platforms using vision- lsiounlanguage models. In Proceedings of the Nineteenth ACM Conference on Recommender Systems, pp. 975-978, 2025.
Shipeng Yan, Lanqing Hong, Hang Xu, Jianhua Han, Tinne Tuytelaars, Zhenguo Li, and Xuming He. Gener- ative negative text replay for continual vision-language pretraining. In European Conference on Computer Vision, pp. 22-38. Springer. 2022.
Jiawei Yao, Chuming Li, Keqiang Sun, Yingjie Cai, Hao Li, Wanli Ouyang, and Hongsheng Li. Ndc- scene: Boost monocular 3d semantic scene completion in normalized device coordinates space. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9421-9431. IEEE Computer Soci- ety, 2023.
Jiawei Yao, Chuming Li, and Canran Xiao. Swift sampler: Efficient learning of sampler by 10 parameters. Advances in Neural Information Processing Systems, 37:59030-59053, 2024.
Jiazuo Yu, Yunzhi Zhuge, Lu Zhang, Ping Hu, Dong Wang, Huchuan Lu, and You He. Boosting continual learning of vision-language models via mixture-of-experts adapters. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 23219-23230, 2024.
Donglin Zhan, Haoting Zhang, Rhonda Righter, Zeyu Zheng, and James Anderson. Collaborative bayesian optimization via wasserstein barycenters. In 2025 IEEE 64th Conference on Decision and Control (CDC), pp. 6294-6291. IEEE, 2025.
Haoting Zhang, Donglin Zhan, Yunduan Lin, Jinghai He, Qing Zhu, Zuo-Jun Shen, and Zeyu Zheng. Daily physical activity monitoring: Adaptive learning from multi-source motion sensor data. In Conference on Health, Inference, and Learning, pp. 39-54. PMLR, 2024.
Xi Zhang, Feifei Zhang, and Changsheng Xu. Vqael: A novel visual question answering continual learning setting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19102-19112. 12, 2023.
13

---

Published as conference paper ICLR 2026
Xiaofeng Zhang, Fanshuo Zeng, Yihao Quan, Zheng Hui, and Jiawei Yao. Enhancing multimodal large language models complex reason via similarity computation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 10203-10211, 2025.
Zangwei Zheng, Mingyuan Ma, Kai Wang, Ziheng Qin, Xiangyu Yue, and Yang You. Preventing zero-shot transfer degradation in continual learning of vision-language models. In Proceedings of the IEEE/CVF international conference on computer vision, pp- 19125-19136, 2023.
Da-Wei Zhou, Kai-Wen Li, Jingyi Ning, Han-Jia Ye, Lijun Zhang, and De-Chuan Zhan. External knowledge injection for clip-based class-incremental learning. arXiv preprint AXXiv:55330081000 2025.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 130(9):2337-2348, 2022.
Hongguang Zhu, Yunchao Wei, Xiaodan Liang, Chunjie Zhang, and Yao Zhao. Ctp: Towards vision- language continual pretraining via compatible momentum contrast and topology preservation. Pro- ceedings of the IEEE/CVF International Conference Computer Vision, pp. 2225222226777 2023.
14

---

Published as conference paper ICLR 2026
A APPENDIX
A.1 SUPPLEMENTARY TECHNICAL DETAILS
Optimization and Certificate Update Algorithm I outlines training at task t. Each iteration (Lines 7 builds M from centered embeddings, extracts PI.k, Vx) via a differentiable block power iteration, forms sketches/projectors (Line 8) evaluates Sspees Lub.. Lgi with the task loss, and updates parameters (Line (2). We maintain streaming EMAs and refresh the certificate every step using slow EMA to preserve the alignment skeleton while allowing controlled plasticity (Lines [5).
Optimization. All experiments use AdamW with weight decay 0.05 and cosine schedule. We use an initial learning rate of 1.5 10-1 for the image-side LoRA parameters and 1.0 x 10-1 for the text-side LoRA parameters, with mixed precision in bolo 1 6 and gradient clipping at 1.0. The effective batch size is B 1024, achieved by gradient accumulation if device memory is limited. For time-continual training on TiC splits, warm up only on the first temporal chunk and keep the same maximum learning rate for all subsequent chunks to follow established practice. Unless otherwise stated, small datasets receive one three epochs per task, and large datasets receive about one epoch per task, with early stopping the current-task validation set.
Algorithm 1 Pi-CCA Training at Task t
1: Inputs: dataset Dú; encoders for fo with params O.., Oú; certificate (pisk- St, Sţ); sketches Ry Rii hyperparams (A1, 22, λ3. W.W1J, 77, a, 0,,377 Yt k, h, M, Tpow)
2: for epoch 1.... do
3: for mini-batch B {1x; w;) w BB C Dj do
4: Encode & center: Z (2;))
2n
Z, - (fx 2,
5:
Covariances: E..= B-1 Z Zr + IoI,
Σx: :==B-11 H-1 Z, Ze II Σn1 B-1 z Z.
6:
Whitened cross-cov.: M -/11/2 Evt Lit
5-1/2
(Eq 2
7:
Top-k SVD (Ü,, Pi:k, Vk)
SVD+(M) via Tpow block power steps with QR re-
orthogonalization
Sketches/projector:: =, ÜR, S. 3.=RA R. Vn. Q = S.S Q
8:
Prompt perturbations: sample {dm Jm= compute {Q) (m)) Jm=1 and their mean Q
9:
Σ m ((
M
10:
Losses: Lapee (Eq. 8 Lmom (optional, Eq. 9); Sub (Eq.
10
Cpi (Eq.
11:
Total loss: L= Cusk + d Cs + A2 + Aph
(Eq.
12:
Update params: i,, O, t- optimizer step( Vo,,o, C)
Streaming EMAs: update x(() (t)) it
x(() using Eq.
13:
12
14:
Certificate refresh: Piu f (1 a) P + appii
S ortth(((1- a
aa)
15:
S (1 S
MM
S (m)
(Eq. 13
16:
Output: updated encoders fv, fo and certificate (pi S St)) at task t
A.2 EXPERIMENTAL SETUP (SUPPLEMENTA))
Backbone & adapters. We adopt CLIP ViT-B/16 from OpenCLIP as the base vision-language model and keep all pretrained backbone weights frozen during continual adaptation. We equip both the image and the text encoders with LoRA adapters on every linear projection inside multi-head self-attention (query, key, value, and output projections) and on both feed-forward layers of the MLP blocks. LoRA weights are
15

---

Published as a conference paper ICLR 2026
Table 4: Datasets and task orders used in our experiments. MTIL and X-TAIL are evaluated with zero task domain hints at inference. VLCL follows eight-dataset order for continual retrieval and additionally reports zero-shot retention on a held-out suite. ConStruct-VV VI. comprises seven structured concept subsets built from VG and VAW, and TiC applies chronological splits probe temporal robustness.
Track
Order
Dataset / Subset
Key stats and notes
(A) MIIL: multi domain inceeennceeetttll classification (default alphabetical order)
MTIL.
RGVC-Aircraft
100 classes, 10k images, fine-grained aircraft vari- ants.
MTIL
2
Caltech101
102 categories. 9.146 images, object recognition.
MTIL
CIFAR-100
classes, 50k train 10k test images 32x32.
MTIL
4
DTD
47 texture categories, 5,640 images.
MTIL
5
EuroSAT
10 land-use classes, 27k images (RGB option). 102 classes, 8.189 images, fine-grained flowers.
MTIL
Flowers-100
MTIL
7
Food-101
101 classes, 101k images.
MTIL
8
MNIST
10 classes, 60k train and 10k test images.
MTIL
9
Oxford-IIT Pets
37 classes, 7.349 images.
MTIL
10
Stanford Cars
classes, 16,185 images.
MTIL
11
SUN397
397 scene categories, 108,754 images.
(B) X-TAIL: cross-domain lask-agnostic classification
Aircraft, Caltech101, DTD, EuroSAT, Flowers, F0o1101, MNIST, Pets, Cars, SUN397
X-TAIL
1-10
Same as MTIL except CIFAR- excluded. Test- time label space union of seen/unseen domains.
(C) VLCL: continual image-text retrieval
VLCL
1
Flickr30K
31,783 images with five captions each, Karpathy splits.
VLCL
COCO Captions
123,287 images with five captions each, 5k val/test.
VL.CI.
3
Pets
Oxford-IIIT Pets in caption form, domain shift. Al-generated images and prompts. synthetic im- agery.
VLCL
4
Lexica
VLCL
5
Simpsons
Cartoon frames and captions, style shift.
VLCL
WikiArt
Artwork images with descriptions, art domain. E-commerce clothing with captions, fashion do-
VLCL
7
Kream
main.
VLCL
8
Sketch
Sketches paired with text.
(D) ConStruct- VL: structured VL concepts
ConStruct-V VL
Relation: spatial
Triplets from VG/VAW; size 1k-31k per subset. Attribute- focused triplets: VG, VAW. VG+ Attribute triplcts; VG and combined sets.
ConStruct-V. VL
2
Attribute: size
ConStruct-VL
Autribute: material
ConStruct-VLL VL.
4
Relation: action
Inter-object action relations.
ConStruct-V. VL
5
Attribute: color
Color understanding triplets.
ConStruct-VE
Object state
State-focused triplets. Single-object action attributes.
ConStruct-Vi VL
7
Attribute: action
(E) TIC: time-contiuaal pretraining
TIC
2016-2017
First temporal chunk TIC-YFCC/RedCapp Second temporal chunk. Third temporal chunk. Final temporal chunk.
TIC
2
2018
TIC
3
2019-2020
TIC
2021-2022
initialized with the standard zero-init scheme so that the initial network is exactly the frozen backbone, and the adapters gradually inject task-specific updates training proceeds. The adapter rank is set 16
16

---

Published as a conference paper ICLR 2026
with scaling a 16 and modest dropout rate of 0.05 applied on adapter outputs. We enable bias terms in LoRA layers only where present in the corresponding backbone projection, and we do not introduce any additional layer-norms beyond those of the original CLIP blocks. This keeps the parameter footprint small and the optimization stable while allowing Pt-CCA to steer the representation through low-dimensional control surface.
Pr-CCA hyperparameters. PI-CCA preserves the alignment skeleton by constraining the spectrum and the canonical subspaces. We use the top = 64 canonical components, which balances fidelity and cost on ViT-B features, and we form h 256-dimensional orthonormal sketches for both modalities so that subspace distances are computed a near-isometric space. Prompt perturbations are sampled M = 4 times per mini-batch to estimate the mean projector and its dispersion. We maintain two levels of exponential moving averages: a certificate EMA with rate a = 0.01 that slowly refreshes the stored spectrum and sketched bases, and covariance EMA with rate 3 = 0.01 that stabilizes the streaming covariance factors. To guarantee well-posed whitening, we add ridge shrinkage an eigenvalue floor of 10-3 during the inverse square-root computation. We obtain the top-k singular vectors of the whitened cross-covariance via a differentiable block power iteration with pow = 3 steps and QR re- orthogonalization at each step. The loss composition uses Z 1.0 for spectral preservation, X2 1.0 for jubspace-angle preservation, and λ3 0.2 for prompt invariance. We include Ky-Fan alignment term with weight ę = 0.1 and low-order spectral moments with 2 and weights (w1,w2) = (0.2, 0.1) to stabilize near-degenerate spectra. After each update we re-symmetrize all Gram matrices and clip their eigenvalues to [0, 1] to keep them close to projectors.
"/ 10-3 to the batch covariances and apply
Datasets and orders. 4ists the task sequences used in this paper. For MTHL we adopt the 1 1-domain suite and follow the alphabetical order by default. X-TAIL uses the same domains except that CIFAR- 1000 is removed, and the label space at test time is the union of seen and unseen domains. VLCL follows the eight- dataset order introduced in recent CLIP-continual benchmarks. ConStruct-VL. uses a seven-task sequence over structured VL concepts that covers attributes, relations, and states. TiC adopts four temporal splits to probe time-continual robustness. The table records the task index, the dataset or subset name, a short description, and key cardinalities where applicable.
Hardware and protocol. We run all experiments on eight NVIDIA A100 80 GB GPUs with PyTorch 2.3 and CUDA 12 under NCCL data parallelism. Each configuration is repeated with three different random seeds, and we report the mcan and the standard deviation. PI-CCA never stores replays past-task samples. When baseline explicitly requires reference or wild unlabeled data, we follow its original procedure and keep these resources strictly outside of PI-CCA.
Prompts and perturbations. For classification -tylee evaluations we use the standard CLIP class templates and we ensemble across small pool of and template jitters that preserve class semantics while varying phrasing, and these perturbations are used only inside the projector averaging and the prompt-invariance loss. For retrieval-style evaluations we leave captions unchanged, and we apply perturbations the text encoder solely for forming the prompt-invariant certificate, which prevents any leakage of label or caption content into the training targets.
variants. Prompt perturbations are realized by synonym
Out-of-distributinn (OOD) prompt templates We evaluate OOD prompts that deviate from CLIP-style class templates. Below is non-exhaustive set used in $4.3 (placeholders in {):
17

---

Published as conference paper ICLR 2026
Prompt Templates
Prompt Templates
Instructional:
"Identify the main object: (class).
Provide a brief capticn."
"Task: detect (class) in the picture and sumnarize it."
Narrative:
"I'm looking at scene where {class) appears."
"This moment captures a (class) in context."
Keywords:
"[class), high detail, natural light, candid, outdoors."
Caption:
"A candid shot featuring a (class).*
Hashtag:
"#(class) #dailyshot #photography"
Meta:
"User: describe image that includes (class). Assistant: ....
s
Translation:
English \rightarrew Chinese \rightarrow English variants
remplate:
(class)) Context-uuknowny Describe briefly."
Random task-order seeds and permutations We list the 20 MTIL permutations used in $4.3) Domains: Aircraft (Air), Caltech101 (Cal), CIFAR100 (CIF), DTD (DTD), EuroSAT (Eur),
Flowers (F1o), Food101 (Foo), MNIST (MNI), OxfordPets (Pet), StanfordCars
(Car), SUN397 (SUN).
Table 5: Order seeds (I) domain sequence). Abbreviations above.
Order ID
Permutation of 11 domains
S-1027
Air, Cal, CIF, DTD, Eur, Flo, Foo, MNI, Pet, Car, SUN Car, Pet, Foo, Eur, DTD, Air, Cal, CIF, SUN, Flo, MNI
S-1132
S-1219
SUN, Cal, Car, Foo, Pet, Eur, DTD, CIF, Air, Flo, MNI DTD, Eur, Cal, Air, CIF, Flo, SUN, Pet, Foo, Car, MNI Cal, Air, DTD, Pet, Car, SUN, Foo, Eur, CIF, Flo, MNI Foo, Flo, CIF, Eur, Air, SUN, Cal, Car, Pet, DTD, MNI Pet, Car, Air, Cal, CIF, Foo, DTD, Eur, SUN, Flo, MNI Eur, DTD, Foo, Cal, Air, Pet, Car, SUN, CIF, Flo, MNI CIF, DTD, Eur, Car, Pet, Foo, Cal, Air, SUN, Flo, MNI Car, Foo, DTD, Cal, Eur, Air. CIF, Pet. SUN, Flo, MNI Air, EUR, Pet. Foo, Cal, DTD, Car, CIF, SUN, Flo, MNI Flo, Foo, Cal, Air, Car, Pet, Eur, DTD, CIF, SUN, MNI
S-1305
S-1402
S-1508
S-1603
S-1701
S-1806
S-1904
S-2001
S-2107
S-2209
Pet. SUN, Foo, Flo, Cal, Air, Car, DTD, Eur, CIF, MNI
S-2311
CIF, Cal, Air, Foo, DTD, Eur, Pet, Car, Flo, SUN, MNI SUN, Air, Foo, Cal, DTD, Eur, Car. Pet, CIF, Flo, MNI Air. DTD, Flo, Foo, Pet, Cal, Car, Eur, CIF, SUN, MNI
S-2415
S-2512
continued on next page
18

---

Published as conference paper ICLR 2026
Order ID
Permutation of 11 domains
S-2608
Cal, Foo, Air, Car, DTD, Eur, Pet, CIF, SUN, Flo, MNI
S-2704
Eur, Cal, CIE, Air, Flo, Pet, Car, Foo, DTD, SUN, MNI
S-2809
Foo, Car, Cal, Eur, SUN, DTD, Air, Pet. CIF, Flo, MNI
S-2913
DTD, Air, Cal, Foo, Pet, Car, Eur, CIF, SUN, Flo, MNI
A.3 ADDITIONAL EXPERIMENTS AND RESULTS
A.3.1 HYPERPARAMETER SENSITIVITY.
We summarize the core factors of C-CCCC--lignnan geometry (kt, h)), prompt invariance (M, >3). stream- ing stability (a, 3), and spectrum/subspace balancing A2)aand report mean over three seeds on representative metrics. Trends in Fig. [6 show: (i) moderate canonical rank and sketch size (k-64, h-256) best capture the alignment skeleton; (11) prompt averaging
(M)) and small invariance weight (Ag) substan- tially reduce forgetting without hurting retrieval; (iii) small but nonzero EMAs (a, B) are crucial for stable whitening and certificate refresh; and (iv) balanced spectral/subspace weights (A== maximize reten- tion- coasppatyy trade-offs. Variations around the defaults lead to modest performance changes, indicating robustness.
K: Icanoniccal compa)
h (sketch diml
M (promp p perturbations)
33 (Invariance)
15
12
72-
70
40-
36
50
36
123
64
228
726
354
522
3.5
03
65
0.3
0.4
(cort. EMA)
[cow. EMA)
(spactral)
(cubspace)
2
136
AFF 1ibla
12
114
0-
66-
12
60-
58
0.
0.79
1.3
0.55
3.0
2.5
Paramnttr Value
PTL RUC. 2T (11
Figure 6: Core hyperparameters. Finer-grained sweeps confirm robustness around the defaults. Geometry (k, h) and invariance (M, control fidelity and prompt sensitivity, EMAs (a, 3) stabilize streaming esti- mates, balanced losses (A1, Az) maximize retention retention-plastiityy Changes are modest across a broad range of values.
19

---

Published as a conference paper ICLR 2026
As summarized in Fig 7 whitening most stable at 7=10-3 e=10-5 with =3 exact Hungar-
ian pairing matches the sorted surrogate within noise. Mild global spectrum regularization (gE 0.2]. ~=1~2) slightly lowers AF, and Gaussian sketches edge SRHT by = 0.2 R@ Around r=16, OLORA =1,, 16, Pimp 0.05, capacityloptimization changes yield < 0.4-pt shifts. Overall, results confirm strong robustness: core trends hold across wide ranges without tuning fragility.
Uiiveling poppriig
Cracs
5. 5
0.4
03
Edpe
Powe ckp
Parrs rama
-28
450
CordnuoiiLal
Figure 7: Other hyperparameters. Incremental gains and robustness from whitening, global-spectrum regularizers, sketch choice, and adapter/optimizztion knobs; core conclusions remain unchanged.
A.3.2 CERTIFICATE REFRESH STRATEGIES
We compare five strategies over the 11 11sttep MTIL stream: NR (no refresh, a=0), SR-T (slow text-only refresh, a=0.01), FR-T (fast text-only, a=005),, SR-TV (slow text+vision, a=0.01 both), and FR-TV (fast text+vision, a=0.05 both). We log subspace-angle drift Dang = ,isin" 2 0; and spectral drift D, | 111 per step, alongside Arg (1) and AF (L). For global vs. local certificates we contrast a single Global Pi-CCA certificate versus Class-local and Concept-local variants (per-classppr--coneep sketches), comparing accuracy and resource cost. As shown in Fig g and 9 SR-T minimizes geometry drift and delivers the best Avg/AF over steps. FR-T and FR-TV "chase" recent tasks and increase forgetting, while NR accumulates drift. Global certificates balance performance and cost, class-/concept-locs variants add memory/time and slightly reduce Avg, suggesting unnecessary specialization.
A.3.3 PAIRING STRATEGY BOUNDARY
We compare the sorted surrogate (descending sort of P) against the Hungarian optimal assignment un- der controllable spectral crowding. We bin runs by the minimum singular-gap gappar &min = min;
1s+1)= E {0.0005, 0.0010. 0.0015.0 0.0025, 0. 0.0040, 0.0060. 0.0080, 0.0100, 0.0120} and sweep spectral jit- ter 77 E {0.00, 0.15, 0.30, 0.45, 0.60, 0. 75, 0.90) with 6 replicates per (ōmin- 17.. then aggregate per 6min- For each run we record metric iffferrecee(HHungrian Figure [T shows that: (i) Under very small gaps (8min < 0.004), Hungarian yields tiny but sometimes sig- nificant improvements. (ii) For gaps of practical size (6min 0.006), the Sorted and Hungarian algorithms are statistically indistinguishable with AAF remaining approximately zero across the board. (iii) the sorted surrogate is the recommended method, it is both safe and faster. The Hungarian algorithm only shows advantage in contrived scenarios with tightly crowded spectra, offering no meaningful benefit in practical applications.
Sorted): AAv (p-p.), AR@ (R@1 I (p.p.), and AAF (p.p.).
20

---

Published as conference paper ICLR 2026
60
AuS: PAT
76.5
Avg
0.25
0.20
75.0
2.8
10
Taak stop
(a) Drift vs. step: Dang and D... Shaded bands: randomized (b) Performance vs. step: Avg (left axis) and AF (right axis).
Figure 8: Refresh strategy analysis. Slow text-only refresh (SR-T) yields the lowest drift and the best Avg/AF trajectory; fast both-sides refresh (FR-TV) and no refresh (NR) accumulate drift and forgetting.
Aug !11
Verrery (GB)
17.5
15.0
50
12.5
(K)
40
10.0
7.
5,0
2.5
0.0
Global
Class-occa
Conceptiocal
Figure 9: Accuracy vs. cost by certificate granularity. Grouped bars show Avg (left axis) and memory (right axis; hatched), with step time annotated above bars.
A.3.4 CERTIRICATE GEOMETRY: SKETCHING, INITIALIZATION, AND UPDATE CHOICES
We first study how the CCA certificate behaves under different sketch constructions, initializations, subspace losses, and update rules.
Sketch randomness and sketch type. Teblet MTIL and VLCL performance over 5-10 runs with different sketch RNG seeds and two sketch families (Gaussian vs. SRHT). The standard deviations are very small, indicating that Fi-CAA is robust to sketch randomness and sketch type.
Certificate initialization. Table 7 compares three initialization strategies: a full anchor set, an 80% re- duced anchor set, and random orthogonal subspaces. Thanks EMA updates, Pi-CCA converges to useful invariant even from weak or random initializations, with only modest gaps in final performance and geometry drift.
21

---

Published as conference paper ICLR 2026
Pairing difference heatmap (Hungarian Sorted)
AAvg (pp)
0.05
0.04
Soried
AR@1 (pp)
0.02
0.01
AAF (pp)
-0.01
0,0005
0.0010
0.0025 0.0040
0.0100 0.0120
singular gap
Figure 10: Sorted vs. Hungarian under spectral crowding. Heatmap of mean (Hungarian Sorted, in p.p.) per ômin bin for AAvg @R@l I, and AAF. Stars mark Holm--ooffrrooi-cooreeett significance: p<.05, ** p<.01. Tiny gains appear only when 8win S 0.004. For òmin 2 0.006, differences vanish.
Table 6: Effect of sketch type and sketch RNG seeds MTIL and VLCL. Mean and std are 5-10
Sketch type
MTIL Avg (1)
VL.CL 12T R@I (1)
Std (MTIL)
Std (VLCL.)
Gaussian SRHT
76.8
48.6
#0.2
#0.1
76.9
48.4
+0.3
+0.2
Subspace loss variant. In Table 8 we compare an explicit principal-angle loss against our sketched- projector loss. Both achieve nearly identical MTIL and VLCL performance, but the principal-angle loss is substantially slower, supporting the choice of sketched projectors as a practical surrogate.
Gradient flow through certificate update. Table compares differentiable variant that backpropagates through EMA+QR with our default stop-gradient update. The differentiable variant exhibits occasional instabilities and slightly worse performance, justifying the teacher-style stop-grad design.
A.3.5 SCALING WITH BACKBONE AND ADAPTER CAPACITY
We next study how Pi-CCA scales when the backbone and adapter capacity are increased.
Backbone size. Table 10 evaluates Pi-CCA on ViT-B/16, ViT-I/14, and VIT-L/14@3366 Performance improves with larger backhones, while the additional time and memory remain moderate and the certificate size stays fixed.
Adapter configuration. Table 11 shows that Pi-CCA remains effective under higher LoRA ranks and when partially fully finetuning the backbone: the geometry-based losses consistently improve perfor- mance with modest extra cost.
A.3.6 PROMPT INVARIANCE: PERTURBATION COUNT AND ROBUSTNESS
We now examine the role of the prompt-invariance term, both under random perturbations and adversarial shifts.
22

---

Published as conference paper ICLR 2026
Table 7: Effect of certificate initialization on MTIL, X-TAIL, and geometry drift.
Initialization
MTIL Avg (1)
MTIL Last (1)
X-TAIL R@l (1)
Geometry drift (1)
Full anchor set Reduced anchor (80% removed)
76.8
75.5
68.1
2.1
75.4
74.1
67.6
3.0
Random orthogonal subspaces
75.2
73.9
67.3
3.5
Table 8: Comparison of subspace loss variants. Relative step time is normalized default.
Subspace loss lype
MTIL Avg (1)
VLCL 12T R@I (1)
Relative step time (x) (L)
Explicit principal-angle loss
76.9
48.7
1.36
Sketched projector loss (ours)
76.8
48.6
1.00
Number of prompt perturbations M. Table 12 aggregates the effect of M across X-TAIL, VLCL, and ConStruct-V VL., Increasing M from improves robustness, while further increases yield marginal gains but noticeable extra cost.
Adversarial prompt shifts. Table 13 under gradient-bass based ad adversarial prompt perturbations on X-TAIL. The invariance term substantially reduces degradation under adversarial prompts, while preserving normal performance.
evaluates Pi-CCA with and without the prompt-invariance loss Lii
A.3.7 OVERHEAD, REGULARIZATION BASELINES, AND ANCHOR CONFIGURATION
Overhead and memory footprint. Table 14 quantifies Pi-CCA's overhead relative to LoRA-only base- line. Time and memory increases modest, while the certificate storage is tiny compared to typical replay buffers.
Stronger regularization baselines. To test whether Pi-CCA's gains come from "more regularization", Table 15 compares Pi-CCA to LoRA with strong generic feature regularizers and to proxy similarity alignment baseline. Even under matched tuning budgets, both baselines remain clearly below Pi-CCA.
Anchor set size and diversity. Table 16 ablating the anchor prompt set (single default template, 50% templates dropped, full set) shows that Pi-CCA is not overly sensitive to anchor diversity: even minimal label-derived set recovers most the gains.
A.3.8 STATISTICAL SIGNIFICANCE AND TERKTASK RESULTS
Paired t-tests. Table 17 reports two-sided paired t-tests (3 seeds) between Pi-CCA and the strongest replay-free baselines on key metrics, All p-values below 0.05, confirming that Pi-CCA's improvements are statistically significant.
Per-task results across benchmarks. To show that improvements not driven by a single "lucky" task, Table 18 aggregates per-task results across MTIL (11 domains), VLCL (8 datasets), and ConStruct-VI. (7 subsets). Pi-CCA consistently matches outperforms C CLIP across all three benchmarks.
23

---

Published as conference paper ICLR 2026
Table Effect of backpropagating through the certificate update.
Certificate update variant
MTIL Avg (1)
VLCL 12T R@i (1)
Stability
Grad-through EMA orth
76.0
48.0
occasional spikes
Stop-grad EMA + orth (ours)
76.8
48.6
stable across seeds
Table 10: Scaling Pi-CCA to larger CLIP backbones. Time is wall-clock seconds per step; memory is peak GPU usage.
Backbone
MTIL Avg (T)
VLCL 12T R@I
Time (s/step) (J)
Memory (GB) (L)
VIT-B/16
76.8
48.6
3.2
16.4
ViT-L/14
78.2
49.1
4.0
24.1
VIT-L/14@336
78.4
49.3
4.2
28.5
A.4 THEORETICAL ANALYSIS
Let so. ft be the (frozen-backbonee, LoRA-adapted) image/text encoders, and let u(x) E Rdd and v(w) E dR denote their whitened, centered embeddings within mini-batch: ENe B-I Zo + %0l Exe 1 B-1 2 Ze rel, Eet Z. Zr. The whitened cross-covariance is
M E//// Evt Eu E-1/2 Rd.xd.
(14)
Let the rank-k SVDs be Mr = Us (PLa((p))) and M Už V with orthoprojectors
P. = U Pt = V.VT Pt = U Pł = V. We denote by O, (resp. θ,) the diagonal matrix of principal angles between span(Uk) and span(U?) (resp. Va and VY), and recall the identity || P*1 /2 sin O||F-
Given pair (z, w). define the zero-shot score s (z, == (u(x), M v(w)) and the task loss E: R - R+. The (population) zero-shot risk under distribution D
R(M) B( (2,
(15)
Assumptions.
(A1) (Bounded whitened embeddings) [|u(z)]l2 and [[ - almost surely.
(A2) (Lipschitz loss in the score) f is Le-Lipschitz: (e(a) - E(b)] S Lela - 6).
(A3) (Rank-k structure) We compare M and reference M through their top-k SVD factors above; denote Pmax (P. p < 1.
A.4.1 SINGLE-STEP EXCESS RISK BOUND FROM SPECTRAL AND SUBSPACE DRIFT
We first quantify how changes in canonical spectrum and canonical subspaces control the zero-shot risk.
Lemma (Risk is Lipschitz in M under (A1)-(A2)). For any M, M',
(R(M) R(M') < Le I|M - M'
(16)
Proof. By (A2) and Jensen,
RRM) R(M') ([[(((, - E(uu, M'),) L. B[Ku, (M (M-M"))ll) M')o
(17)
By Cauchy-Schwarz
Schwarz and (A1), (u, (M My)o) & IIM M2 hence the claim.
24

---

Published as conference paper ICLR 2026
Table 11: Effect of adapter configuration on Pi-CCA.
Configuration
MTIL Avg (1)
VLCL 12T R@ (1)
Time (s/step) (L)
Memory (GB) (L)
LoRA rank 16 (default)
76.8
48.6
3.2
16.4
LoRA rank
77.2
48.9
3.5
17.1
LoRA rank 64
77.5
49.1
3.8
18.0
Full finetune (last layer)
77.0
48.8
3.9
17.5
Full finetune (all layers)
77.4
49.0
4.2
19.0
Table 12: Effect of the number of prompt perturbations M across benchmarks. For M 0 we do not apply
prompt perturbations on X-TAIL (entries marked "-").
M
X-TAIL R@I (1) X-TAIL AF AF
VLCL 12T R@I (1)
ConStruct- VL (J)
Rel. step time (x) (1)
0 (no Cp)
47.1
3.3
1.00
67.8
3.6
47.9
3.1
1.02
2
68.4
3,4
48.4
2.9
1.06
4
(default)
69.2
3.2
48.6
2.7
1.12
8
69.1
3.3
48.7
2.7
1.21
Lemma (Geometric decomposition of the rank-k part). Let Ap == sort (p1:k) Pi.k. Then
M Mi 2l S ||Aplla
2pmax (1) sin θ 1/2 + sin O.llz).
(18)
Proof. Write = diag(p1:)), D* = diag(pis)) By the triangle inequality.
DU - DVD 1/2 $ DV - -U DV lle UU - DV Tle + |UE(D - D
(4)
(19)
For (D
|D D2 ||lll permuutation-invariin pairing by sorting).
For (A), insert
Pt PP-))::
(A) = 1(1 ([)- DV. P;Ux DV, - UE DVV ll2 PU)Uk
(20)
$ | P3)ull |||2 + (UE Up T)
(21)
s|n e.ll
Since U,. U& has eigenvalues cos o(v) we use |1 1 cos 0| < sin to get | Up - Il]2 S || 0, Thus
(A) < 2|ID[|2 || sin O,l12 5 2pmax sin O
The term (B) is symmetric on the text side, giving (B) S 2pmax || sin 02|12- Combining the three bounds yields the result.
Lemma 3 (Tail energy identity). For any matrix M, ||M - Mall2 0k+1 Hence
| - M l| I || Mt 2 + ( + I
(22)
Theorem (Alignment-goomeetr drift single-step excess-risk bound). Under (A1)-(A3),
R(M) R(M*) Lı [|Ap|l2 2/maxx (Il sin O + sin Oel12) (M4+1(M] (x
(23)
25

---

Published as conference paper ICLR 2026
Table 13: Effect of prompt invariance under adversarial prompt shifts on X-TAIL.
Method
Adv. R@ (1)
Adv. AF (L)
Normal R@ (1)
Normal AF (1)
Pi-CCA with Cx.
56.2
3.1
69.2
3.2
Pi- Pi-CCA Cpi
49.1
4.2
69.1
3.3
Pi Pi-CCA with Lx (no perturbation)
69.2
3.2
69.2
3.2
Table 14: Overhead and memory footprint of Pi-CCA vs. LoRA baseline. Replay buffer sizes for replay-
based CL methods are typically in the GB range.
Method
Time increase (1)
Peak memory increase (1)
Certificate storage
Replay buffer
Pi-CCA (ours)
R 8%
6%
50 KB
N/A
LoRA baseline
N/A
N/A
N/A
GB (replay methods)
Equivalently, using orthoprojectors.
R(M) - -R(M* s Le |All2 + Pmax
v2 (IPo
P + || PFlls) xx+1(M) + (
(24)
Proof. By Lemma 3 |M M I | MM + 0 (M) + (M+)) Apply Lemma to bound M Mg 2, then Lemma 1 to convert spectral deviation into risk deviation. For the projector form, |P |F / sin F.
Interpretation. If Ap = 0 and Us = Ut V& = V the excess risk is controlled purely by tail energy; when the CCA spectrum decays fast beyond k, zero-shot ability is rigidly preserved.
A.4.2 DYNAMIC REGRET OVER A NON-STATIONARY TASK SEQUENCE
We now consider a stream (DrYT= with models { {M, produced by any adaptation rule (e.g., PI-CCA). Let the per-step comparator be M, (e.g., the best rank-k model for D, within the same hypothesis class). Define the dynamic regret
T Regr = Σ (R((M.))((MM))) Re(M))) R((M) = (( [e((u,
(25)
For each I, denote Ap: == (ort Pl On. == @(Un,,, Uk.). O.. = O(VE,, Vt.)) Pmax,: = max(pr.1, PI,1 }, and & := (M) + Jkt1 (MT)).
Theorem (Dynamic regret bound from geometric drift). Under (AI)-(A3), for any sequence {Mi} and comparators {M.}.
T
Regr < L
Apella
+ Pinanit (l
Pt,llE + [Pr,, - Phelle) + Se.au
(26)
Proof. Apply Theorem 1 to (M, M.) under D, for each f, then over t I,....
26

---

Published as conference paper ICLR 2026
Table 15: Comparison of Pi CCA with strong regularization and proxy-alignment baselines.
Method
MTIL Avg (1)
MTIL Last (1)
VLCL 12T R@ (1)
LoRA (plain finetuning)
71.2
69.9
42.0
LoRA strong regularizers (L2, cosine)
72.4
71.1
43.5
LoRA+ proxy alignment (Mod-X style)
73.6
72.2
45.0
LoRA Pi CCA (ours)
76.8
75.5
48.6
Table 16: Effect of anchor prompt configuration Pi-CCA.
Anchor configuration
MTIL Avg (1)
MTIL Last (1)
VLCL 12T R@ (1)
ConStruct-VL. AF (L)
Default-only
76.4
75.0
48.2
2.9
50% dropped
76.6
75.2
48.4
2.8
Full (main setting)
76.8
75.5
48.6
2.7
Plug-in control via certificate-based regularization. Let the training losses
Espec (t) (ort4 | 1
Laub (t) Pr,, poert 1/ 311 Pooert
(27)
be computed against a slowly refreshed certificate (plort : poert Peeet). By triangle inequality,
|| S V (t) + pliist
Pl.1& 12,,
||a,t P! S 2 Csub(t) + ppert
Pl,lee. (28)
If the certificate tracks the instantaneous comparators (e.g., by slow EMA) so that the residual terms leffet Pl,2:1l2 and Peert P.,.ll remain small, then Theorem implies
Regr < Le Σ 1=1
((V (t) + V (t) + Le Σ 8emmi + (small tracking error).
(29)
This formalizes the empirical observation that stabilizing the CCA spectrum and subspaces controls forget- ting and reduces dynamic regret replay-free continual adaptation.
A.5 PYTHON SCRIPT FOR PI-CCA
The following Python script demonstrates the core functionality of Pi-CCA. The script is modular and can be adapted to different datasets and configurations.
Listing 1: Compact Python Script for Pi-CCA
import torch
import torch. nn. functional as F import numpy as np
from sklearn. deteapoccccooooiiiionn import PCA
from sklearn. ptteecm.preprocessinn import StandardScaler
Load pre-trained model (e.g., CLIP) for image and text embeddings Here, assume the of toy dataset like MNIST CIFAR-10
def load ()ta
27

---

Published as conference paper ICLR 2026
Table 17: Paired t-tests between Pi ICCCA and strongest replay-free baselines on key metrics. Pi-CCA means
reported in the main text.
Metric
Baseline
Baseline mean = std
p-value vs. Pi-CCA (J)
MTIL Avg (T)
C-CLIP
75.2 = 0.7
0.019
MTIL Last (1)
DDAS
74.1 0.8
0.023
MTIL Transfer (T)
ZAF
71.9 0.6
0,017
X-TAIL Avg (1)
RAIL
=
0,021
X-TAIL Last (1)
C-CLIP
66.3 = 0.7
0.028
X-TAIL Transfer (f)
RAIL
64.2 0.6
0.024
VLCL 12T R (1)
C-CLIP
46.1 = 1.4
0.017
VLCL T21 R@ (1)
C-CLIP
35.7 1.2
0.021
ConStruct-Ve VL (1)
C-CLIP
1.9
0.013
AF AF
ZAF
3.8 1 0.6
0.008
Table 18: Per-task results for MTIL, VLCL, and ConStruct-VL: Pi-CCA vs. C-CLIP.
Benchmark
Task / Dataset Subset
Metrie
Pi-CCA (1)
C-CLIP (T)
MTIL
FGVC-Aircraft
Acc
75.7
73,8
MTIL
Caltech101
Acc
79.2
77.8
MTIL
CIFAR-100
Acc
75.0
73.6
MTIL
DTD
Acc
73.3
71.3
MTIL
EuroSAT
Acc
76.9
74.8
MTIL
Flowers-100
Acc
78.5
76.3
MTIL
Food-101
Acc
75.8
74.3
MTIL
MNIST
Acc
80.0
78.8
MTIL
Oxford-IIIT Pets
Acc
74.7
73.1
MTIL
Stanford Cars
Acc
80.1
78.9
MTIL
SUN397
Acc
75.9
74.8
VLCL
Flickr30K
12T R@!
49.7
48.7
VLCL
COCO Captions
R@1
51.6
50.6
VLCL
Pets
12T R@1
47.8
46.4
VLCL
Lexica
12T R@
50.1
48.4
VLCL
Simpsons
12T R@1
42.8
41.4
VLCL
WikiArt
12T R@
49.4
47.9
VLCL
Kream
R@1
51.7
50.7
VLCL
Sketch
12T R@1
45.9
44.4
ConStruct-V
Relation: spatial
FA
75.9
75.8
ConStruct-VI
Attribute: size
FA
74.4
72.3
ConStruct- VL
Attribute: material
FA
73.7
72.5
ConStruct- -V
Relation: action
FA
75.1
73.3
ConStruct-VL VL
Attribute: color
76.9
75.7
ConStruct-Ve VL
Object state
FA
74.1
73.1
ConStruct-Ve VL
Attribute: action
FA
76.2
74.3
Example: 1oad MNIST CIFAR-10 and precompute image and text embeddings using CLIP
# For simplicity, using random data for demonstration purposes num_samples = 100
13
num_features 512 # Feature dimension
28

---

Published as conference paper ICLR 2026
15
Random data: [num samples x num features]
image_ amage_data = np. random. rand (num_samples, num featuee) text_ data np. random. rand (num_samples) num num_features) return image_data, text_data
17
19
20
covariance computation
21
def immpute__ovariances saaage_embeddinge, text_ _ext_embeddings, batch_ size =32)
32
# Compute covariance matrices for image and text embeddings in mini -batches
23
B = len simage_embeddann
24
lenge_enmed consor (imaage_mbeddiings)
25
text_embeddings = torch. tensor (text_embedding)
26
27
Cov torch.zerow [ (imagge_ebbeddingss... [1], image_embeddinga
+ shape [1]))
cov_tt torch. zeros (text_embedddings.stll
Lext_embeddingee
4 shape [1]))
cov_vt torch. zeros (imagge_ebbedddins....aap [1], text_embeddings.
4 shape [1]))
31
for in range (0, B, batch_size)
32
batch_image image_emeeddingg (i:i+bacch_sizze
33
batch_text = text_embeddinge [i: : +batch_size)
34
35
# Compute covariance for mini-bate cov_vv += torch. cov (batch_image.1) cov_tt += corch.cov (batch_text.27
36
37
38
cov_vt += torch, mm (batcchhmaaaceeT batch_text)
40
# Normalize covariance
COV_V /= B
42
cov_tt /= B
cov_vt /=
44
45
return COV_VV, cov_tt, cov_vt
46
47
Whitening and CCA certificate computation
48
def whiten_ar and ced___ooput_eeeee cca (cov_ve cov_tt, cov_vt, k=64)
49
# Perform whitening covariance matrices
inv_cov_vv = inv_cov_tt corch,inverse (cov_tt)
(cov_vv)
51
53
# Compute whitened cross-covariance matrix
wttpchhhhmmiiicooo__oeo1 (inv_cov_w, cov_vt), inv_cov_tt) M torch.mm (torch. mm
55
# Perform SVD the whitened cross-covariance matrix
57
U, S, V torch.sve (M)
59
# Extract top-k singular values and vectors (Pi-CC. certificate)
29

---

Published as conference paper ICLR 2026
60
hop_k_singular_vale values S[:k]
top_k_u U[:, :k] top_k_ v[:, :k]
62
64
# Return the compact certificate (canonical correlations and subspaces) return top_k_ weingnlaaraaaleeeee t top_k_
65
67
Update the certificate using mini-batch statistics def update_certificcate (image_embeddingg text_embeddings, k=64, 4 bbttc_size=32):
# Step 1: Compute covariance matrices cov_vV, cov_tt, cov_vt conpute_covariances (image_embeddingg text_embeddings, batch sircchhsize-aachh_aS _izze
70
72
# Step 2:
Whitening and SVD to get Pi-CCA certificate
bog___singular_value top_k_U, top_k_V= = whiten_and_compute__eeoooo cov_vv, cov_tt, cov_vt, k-k)
75
# Return the updated PI-CCA certificate return top_k_ Boo_k_singular_aauees top U_, V_
77
Main function run the Pi-CCA process
79
def main():
90
# Load data (e.g., MNIST or CIFAR-10, here we use random embeddings
+
81
image_data, text_data load_data ()
83
# Update the certificate (this would typically be done iteratively + over tasks)
84
top_k_singular_vallnn top_k_U, top_k_ update_certiificaee Saage_data, text_dataa k=64)
85
# Output the resulting certificate
57
print "TTop=K_sinnulle_aallsss print ("Top- K___ (Image__ubspacce: :", top_k_0)
top_k_siinpllr__values
89
print ( ("Top-Ke K, K.V. (uttpasubsaace) :", top_k_V)
90
if name = main ":
92
main
A.6 LLM USAGE
We used large language model for minor English editing (gramnar/wordng(llaitt)) and small, localized code fixes (e.g., resolving syntax errors, adding missing imports). The LLM did not contribute to research ideation, experimental design, data processing, analysis, figure generation. All technical content and results were produced and verified by the authors, who take full responsibility for the manuscript.
30


Reviews:
[{"review_id": "KbrGtaOBo3", "reviewer": "Reviewer_YmbX", "summary": "- Proposes PI–CCA, a replay-free continual adaptation method for vision-language models that preserves cross-modal geometry via compact CCA certificates (spectral and subspace info).\n\n- Uses LoRA adapters, EMA covariance updates, and sketched projections for constant memory.\n\n- Adds a prompt-invariance term by averaging projectors over prompt perturbations.\n\n- Shows strong results across VL-CL, retrieval, and structured concept tasks, with analyses linking geometry drift to performance.", "questions": "- the paper uses synonym/template jitters and M=4. what happens for (i) larger M, (ii) structured paraphrase generators (back-translation ensembles), or (iii) adversarial prompt shifts?\n\n- how sensitive are results to the initial certificate computed from the frozen model? if the initial certificate is imperfect (noisy prompts or limited anchor set), does PI–CCA converge to a useful invariant or can it get locked to a poor alignment?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Directly preserves the alignment object (canonical spectrum + subspace) instead of indirect proxies (logits/similarity), giving a principled invariant to target during replay-free continual updates\n\n- Compact, practical design: random orthonormal sketches, LoRA-only updates, and streaming EMAs give constant memory and low overhead, making the method compatible with large frozen backbones used in practice. Empirical Pareto plots show a usable knee (e.g., k=64,h=256) balancing mem/time and performance.\n\n- multi-track evaluation (classification, retrieval, structured concepts, time-continual splits) and consistent ablations demonstrating which components matter (spectral & subspace terms are critical; prompt invariance helps retention). Results place PI–CCA at SOTA among replay-free baselines on their benchmarks.", "weaknesses": "- Certificate dependence on sketching and hyperparameters is described but the randomness/variance from sketch seeds and sketch types needs deeper quantification: how often does a small sketch RNG seed change final retention?\n\n- Experiments focus on ViT-B/16 + LoRA; unclear how PI–CCA scales to much larger backbones or different adapter types (other LoRA ranks, full-finetune, or other PETs).\n\n- Reproducibility caveat: paper promises to release code only at camera-ready due to commercial constraints. for reproducibility during review, providing a compact reference script or a verified pseudocode runner (or smaller toy logs) would greatly help reviewers reproduce core claims.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GFiEiAKcRt", "reviewer": "Reviewer_TQCn", "summary": "This paper proposes a continual learning method for vision language model.\nThe proposed method (PI-CCA) attempts to maintain the geometry of image-text alignment without using the past data during continual learning.\nTo this goal, PI-CCA penalizes the change of the covariance of embeddings of the mini-batch data, which is obtained through SVD.\nExperiments demonstrate that PI-CCA outperforms baselines on multi-domain task incremental classification, cross-domain task-agnostic classification, and continual image–text retrieval.\nAdditionally, the paper evaluates PI-CCA through Ablation study including correlation between geometry and performance.", "questions": "What impact will the proposed method have in broader fields? \nAre there any ideas that could be applied to other areas of machine learning?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper is well written. The related work section is well-organized, and the paper maintains a consistent focus on the problem from the introduction to the experiments.\n- The proposed method appears moderately novel. The core idea might not be very surprising, but the details of the proposed method seems well designed.\n- The experiments demonstrate that PI-CCA is superior, and the ablation study supporting the claim is also well-conducted.", "weaknesses": "- I think the explanation of the proposal method could be improved to be a bit easier to understand. \nThe concept of a “certificate” is used, but since I am not familiar with continual learning, I do not understand the definition. An explanation of the term is necessary.\n\n- There are no statistical tests for the main results. There is the following sentence:\n\"Each configuration is repeated with three different random seeds, and we report the mean and the standard deviation.\"\nHowever, I cannot find the standard deviation.\n\n- The proposed method is a collection of multiple ideas, and it might remain a bit unclear whether the core idea possesses the generality or universality to influence future research in a broad research area.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CEqlwPAWtu", "reviewer": "Reviewer_pV7E", "summary": "This paper introduces PI-CCA, a replay-free continual learning framework for vision-language models (VLMs) that aims to preserve cross-modal alignment geometry during adaptation. The method proposes a compact CCA-based certificate that captures the top-k canonical correlations and subspaces of image-text embeddings, and enforces their consistency across tasks using spectral and subspace-angle losses. Additionally, prompt robustness is encouraged through averaging over prompt perturbations. The approach is evaluated on multiple VL-CL benchmarks and shows strong performance compared to replay-free baselines.", "questions": "*The use of Frobenius norm between sketched projectors as a surrogate for subspace-angle preservation is intuitive, but can you justify its theoretical soundness? How does it compare to more principled metrics like principal angle distributions?\n\n*The certificate update mechanism involves QR-based orthogonalization and EMA smoothing. Could you clarify whether gradients are propagated through these steps, and if not, how this affects learning stability?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The proposed CCA certificate is compact, aligning well with the constraints of continual learning.\n2. Extensive experiments across multiple benchmarks (MTIL, X-TAIL, VLCL, ConStruct-VL) demonstrate competitive performance, and the ablation studies are thorough.", "weaknesses": "1. The core components prompt perturbation averaging and subspace-angle preservation are not novel and have been extensively studied in prior work on representation similarity and prompt tuning. Their application to continual learning is incremental and lacks a strong theoretical or empirical justification for novelty. e.g., Orthogonal Over-Parameterized Training (CVPR20),  Controlling text-to-image diffusion by orthogonal finetuning (NeurIPS23) DeLoRA: Decoupling Angles and Strength in Low-rank Adaptation (arxiv 24) PAID: Pairwise Angular-Invariant Decomposition for Continual Test-Time Adaptation\n\n2. The paper does not convincingly demonstrate that preserving CCA geometry is fundamentally more effective than existing regularization or distillation-based methods. The observed gains could be attributed to careful tuning rather than a conceptual breakthrough.\n\n3. The prompt invariance mechanism is relatively simplistic (averaging projectors) and does not explore more sophisticated or adaptive strategies.\n\n4. The method assumes access to a diverse anchor prompt set for certificate construction, which may not be feasible in real-world streaming scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nJP2PRBhCw", "reviewer": "Reviewer_Z4s6", "summary": "The paper proposes a new method of replay-free continual learning for CLIP-like vision-language models. The core idea is to retain the statistics of feature correlations called CCA certificates, consisting of top-k singular values and the corresponding subspace in the feature space during continual learning. Additionally, they propose to apply low-rank random projections to the CCA certificates, resulting in the quantities called PI-CCA. Also they propose to marginalize these quantities by perturbations in prompts. By leveraging the PI-CCA, it is expected that the proposed continual learning may be more efficient than explicitly leveraging actual data. Experiments show that the proposed method is superior to the existing replay-free methods in average.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper is clearly written and easy to follow.\n- The core idea of retaining correlation statistics seems simple and easy to implement.\n- Experimental results show the superiority of the proposed method in average, compared to previous replay-free methods.", "weaknesses": "- The paper appears to violate the official format for submissions, which makes unclear whether the submitted content exceed the paper limit or not if the manuscript were appropriately reformatted.\n- The proposed method seems to introduce some computational/storage burden by SVD and the PI-CCA quantities, compared to previous continual learning methods. Also, since retaining of statistic information can also be seen as some kind of replay, it should be compared with replay-based continual learning for fair evaluation.\n- One of the motivation of the replay-free method has been privacy protection as discussed in Introduction, but it is unclear whether the proposed method can provably prevent the privacy leakage from the correlation statistics.\n- In experiments, the superiority of the proposed method is only shown by averaged results, which may be dominated by a single task with high accuracy by luck.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)