RLAP-CLIP: Continual Multimodal Learning with Prototype Adaptation and Difficulty-Aware Routing
Authors: Ruikun Luo
Jiarui Wang
Yuan Gao
Jing Yang
Jieming Yang
Song Wu
Hai Jin
Xiaoyu Xia
Venue: ICLR 2026 Poster (2026)
Keywords: Continual Multimodal Learning; Prototype Optimization; Mixture-of-Experts
Primary area: transfer learning, meta learning, and lifelong learning

Abstract: Vision-language models, such as CLIP, achieve strong zero-shot performance through contrastive pre-training but face significant challenges in class-incremental image classification scenarios. When learning new tasks sequentially, current methods suffer from degradation in prototype quality due to passive averaging and underutilize their visual adaptation capabilities. We propose RLAP-CLIP, which addresses these limitations through three components. First, Reinforcement Learning-based Prototype Optimization (RLPO) formulates prototype construction as a reinforcement learning problem to actively optimize class separability rather than relying on simple averaging. Second, difficulty-aware cross-modal fusion uses a mixture-of-experts to route samples through specialized processing pathways based on complexity. Third, dual-modal prompting balances visual and textual adaptation. Experiments on eight image classification benchmarks demonstrate consistent improvements, with RLAP-CLIP achieving average accuracy gains of 3.72-4.46 points and final accuracy improvements of 0.49-4.48 points over other methods, validating that RLAP-CLIP achieves state-of-the-art performance.

Published as a conference paper at ICLR 2025
RLAP-CLIP: CONTINUAL MULTIMODAL LEARNING WITH PROTOTYPE ADAPTATION AND DIFFICULTY- AWARE ROUTING
Ruikun 1234, Jiarui Wang 1234 Yuan G Jing Yang", Jieming Yang", Song Wu1231, Hai Jin 1231, Xiaoyu Xia
National Engineering Research Center for Big Data Technology and System
"Services Computing Technology and System Lab *Cluster and Grid Computing Lab
"School of Computer Science and Technology, Huazhong University of Science and Technology
5 Zhengzhou University
6 RRoya Melbourne Institute of Technology
frkluo, gumorming, wusong, hjin] edu. en, (yuangaohnu, yangjinghust yjmlaile Malaaile@ggmee gm il com,
xi siaoyu. xia rmi edu.au
ABSTRACT
Vision-language models such as CLIP achieve strong zero-shot performance through contrastive pre-training but face significant challenges in class- incremental image classification scenarios. When learning new classes sequen- tially, current methods suffer from degradation in prototype quality due to passive averaging and underutilize their visual adaptation capabilities. We propose RLAP- CLIP, which addresses these limitations through three components. First, Rein- forcement Learning-basee Prototype Optimization (RLPO) formulates prototype construction reinforcement learning problem to actively optimize class sep- arability rather than relying on simple averaging. Second, difficult--waan cross- modal fusion uses mixture-of-expprts architecture route samples through spe- cialized processing pathways hased on complexity. Third, dual-modal prompt- ing balances visual and textual adaptation. Experiments on eight image classifi- cation benchmarks spanning general classification, fine-grained recognition, and domain-shift scenarios demonstrate consistent improvements, with RLAP-CLIP achieving average accuracy gains of up to 4.52 percentage points and final accu- racy improvements of up 6.26 percentage points over state-of of-the-art methods. Our source code is available at RLAP-CLIP.
INTRODUCTION
Vision-language models like CLIP Radford et al. (2021) have achieved remarkable zero-shot per- formance through large-scale contrastive pre-training, learning align visual and textual represen- tations in a shared semantic space. However, when deployed in continual learning scenarios where new tasks arrive sequentially, these models face the classic stability-plasticin plasticity dilemma Shi et al. (2024); Kim & Han (2023); Buzzega et al. (2020): they must acquire new knowledge while preserv- ing previously learned concepts without catastrophic forgetting.
Current continual learning approaches for llniuan models predominantly rely on parameter- efficient adaptation He al. (2023); Zhou al. (2022b;a); He al. (2025), freezing pre-trained weights while introducing minimal learnable parameters for new tasks. Methods like CoOp Zhou al. (2022b) learn continuous prompts in the text encoder, while approaches such as MaPLe Khattak et al. (2023) extend prompting to both modalities. While this parameter- freezing strategy theoreti- cally prevents catastrophic forgetting Zhou al. (2023), our analysis reveals two critical limitations that prevent these methods from reaching their potential.
"Corresponding author.

---

Published as a conference paper at ICLR 2025
First, existing methods suffer from prototype quality degradation due passive averaging. Since storing complete historical data violates memory constraints, continual learning methods rely on prototype-bascee knowledge retention Douillard et al. (2020), where each class is represented by the mean embedding of limited exemplar samples Zhou et al. (2025b); Zhu et al. (2021). However, as new tasks arrive and feature spaces evolve, prototypes computed through simple averaging may drift into suboptimal regions Masana et al. (2022), leading to progressive performance degradation and increased forgetting of previously learned classes Zhou al. (2023).
Second, current approaches exhibit asymmetric multimodal exploitation, predominantly adapting textual representations while treating visual encoders as static feature extractors Zhou al. (2022b); Wang et al. (2022); Jiang et al. (2025). This text-centric bias may break down in fine- tne-grained recog- nition where discriminative visual patterns lack precise linguistic descriptors Song et al. (2020).
To address these challenges, propose RLAP-CLIP, which targets the stability-plastiiie dilemma through complementary mechanisms. For stability, we introduce RLPO, which transforms proto- type construction from passive averaging into active optimization based reinforcement learning. RLPO maintains robust class representations that resist degradation as new tasks arrive by learn- ing to weight samples based on their contribution to class separability. For plasticity, we employ difficulty-awan cross- moosll fusion with mixture-of-expert. routing, enabling adaptive processing based sample complexity, providing enhanced capacity for challenging boundary cases while efficiently handling easy samples. Bridging both objectives, our enhanced dual-modal prompting balances visual and textual adaptation, ensuring the model captures discriminative patterns across modalities without over-relying on either, thus maintaining flexibility for new tasks while preserving learned representations. Our main contributions are as follows:
We identify and mitigate prototype quality degradation by formulating reinforcement learn-
ing- based optimization framework;
We propose lifficulty-awarl sample routing strategy leveraging mixture-of-expprts architecture to enhance feature processing.
We reveal the importance of balanced multimodal adaptation for continual learning in vi- sion-language models.
Experiments across eight benchmark datasets demonstrate consistent improvements, with RLAP- CLIP achieving average accuracy gains of up to 4.52 points and final accuracy improvements of up to 6.26 points over state-of-the the-art methods, validating that RLAP- CLIP achieves state-of of state-of-the-art performance.
MOTIVATION
When deployed in continual learning scenarios, vision-language models show significant perfor- mance gap compared to their theoretical potential Yi et al. (2024). Our empirical analysis using CLIP model Radford et al. (2021) reveals two critical issues that limit their effectiveness.
Prototype Quality Deteriorates Over Sequential Tasks. To overcome memory constraints, contin- ual learning methods maintain class representations through prototypes Zhou et al. (2025b); Gomez- Villa et al. (2024), mean embeddings computed from a small set of stored exemplar samples per class. As Figure la demonstrates, this passive averaging strategy leads to substantial degradation across sequential tasks. While theoretical upper bounds computed using complete training datasets show only modest decline due to inherent feature space evolution, simple averaging exhibits se- vere performance drops and accelerating forgetting rates. The performance gap between uniform weighting and the theoretical optimum progressively widens as more tasks accumulate, revealing that passive averaging overlooks the need for active optimization based on class separability. This degradation becomes particularly problematic in continual learning scenarios where maintaining discriminative class boundaries crucial for preventing catastrophic forgetting.
Visual Adaptation Capabilities Remain Underutilized. Current continual learning approaches ex- hibit asymmetric adaptation patterns: they primarily focus textual prompts (learnable tokens that guide text encoder adaptation) Liu et al. (2025b); Kim et al. (2024); Smith et al. (2023); Zhou et al. (2022b) while treating visual encoders fixed feature extractors. Visual prompts, which adapt the visual encoder through learnable tokens appended to image patches, remain largely unexplored de- spite their potential for capturing discriminative patterns that cannot be expressed linguistically. As
2

---

Published as a conference paper at ICLR 2025
dmode didd
Repp000 Kate: 200
80-
-0-
00
(96 gtoed
100
4 Number of Taaks
Namber Takks
Imags 9655.
Cara
UCP-422
CU2-300 C1A8-220
(a) Prototype quality analysis using CLIP VIT-B/16 on CIFAR-I00 across eight sequential tasks, showing performance degradation (left) and accelerating for- getting rates (right) under different strategies.
(b) Performance comparison across different prompt- ing strategies on six datasets, demonstrating that vi- sual prompts provide substantial benefits to textual prompting
Comprehensive analysis demonstrating the importance of both prototype quality (a) and visual prompting strategies (b) in continual learning scenarios.
Figure 1:
shown in Figure Ib, When combining textual prompting with visual prompting(dual-monnn prompt- ing), performance increases markedly, 84.2% CUB-200 and 66.9% on Aireraft. The t-SNE vi- sualization in Figure 2 further confirms that dual-modal prompting produces more compact and well-separated class clusters, reducing overlap compared to single-modality approaches.
No Prompting
Text-Only Prompting
Visual- Only Prompting
Dual-Modi Prompting
seree Mtringin
(a) No prompting
(b) Text-only prompting (c) Visual-only prompting (d) Dual-modal prompting
Figure 2: t-SNE visualization of learned feature representations on CUB-200 dataset under different prompting strategies. Colors represent 12 kinds of main bird species. Dual-modal prompting (d) achieves better class separation compared no prompting (a), text-only (b), or visual-only (c).
To quantify the clustering improvements visualized in Figure 2, we measure intra-class compact- ness and inter-class separation across prompting strategies on CUB-200. Table shows that dual- modal prompting achieves 32.2% tighter intra-class distance compared to no prompting while si- multaneously increasing inter-class separation by 10.9%. The separation ratio, defined as the inter- class to intra-class distance ratio, improves from 1.90 3.11, representing 63.7% enhancement. This quantitative analysis confirms that combining visual and textual adaptation produces signif- icantly more discriminative feature representations than single-modality approaches, with dual- modal prompting outperforming both text-only (ratio: 2.74) and visual-only (ratio: 2.40) strategies.
These findings highlight that existing methods fail ex- ploit the potential of vision- language models fully. The reliance on passive proto- type construction, coupled with asymmetric modality adaptation, results a per- sistent performance bottle- neck. Overcoming these challenges requires rethink- ing of both knowledge repre-
Table Quantitative clustering quality analysis on CUB-200 dataset. Lower intra-class distance indicates tighter clusters, higher inter-class distance indicates hetter separation, and higher separation ratio indicates superior overall clustering.
Prompting Strategy
Intra-class Distance
Inter-class Distance
Separation Ratio
No Prompting
2.08
3.95
1.90
Text-Only Prompting
1.56
4.28
2.74
Visual-Only Prompting
1.72
4.12
2.40
Dual-Modal Prompting
1.41
4.38
3.11
sentation and the effective utilization of multimodal capabilities in continuing learning.
3

---

Published as a conference paper at ICLR 2025
Bual-Modal Prompting with Expandable Projections
Difficulty Aware Cross Loass Fusion
Vision Prompts
Expondoble Projections
Easy Samples
Weasy
Expert
Imagee Encader
P
Difficuity
Scoren
Rouber
Hard Samples
Gross-Modee Fusion
Export
Whard
Center-based
P
Excmmiar Selection
P2
photo of (class)
Text +
Prototype
Pollcy Netwonk
Encoden
Update
Text Prompts
Protokype = Wi fi
Instance
Projected Prototype Feature
Calculare Roward
Policy Network
Update
Projected
Projected
p
Image
p.
Text Feature
Reward
RLPO Mechanism
Figure 3: Overview of the RLAP-CLIP architecture for continual multimodal learning. comprises enhanced dual-modal prompting with expandable projections, RLPO mechanism, and difficulty- aware cross-modal fusion.
RLAP-CLIP
To address these challenges, we propose RLAP-CLIP, unified framework that integrates three key components: Reinforcement Learning-based Prototype Optimization (RLPO) to refine class representations and mitigate prototype degradation actively, difficulty-awarr cross-modal fusion to dynamically route samples based on complexity, and enhanced dual-modal prompting to balance visual and textual adaptation. RLPO provides theoretical convergence guarantees in Appendix C.
3.1 REINFORCEMENT LEARNING-BASER PROTOTYPE OPTIMIZATION
Traditional prototype-basee based methods compute class representatives through uniform averaging: -
Pe B. Ex,eE, fx;)) where & denotes the exemplar set for class c and f(-) represents the fea- ture extraction function. This approach treats all samples equally, leading to suboptimal prototypes when exemplars contain outliers or represent multi-modal distributions. We instead cast prototype construction sample-weighted optimization problem, where a policy network with parame- ters learns importance weights w; for each sample based on its contribution to class separability:
Σ wifes w, =o(fi, and Σ wi #:
Pe
#:0:=0
Here, fo denotes the feature embedding of sample i, y4 is its class label, and P {pi...., Pc} represents the set all class prototypes. The policy network considers both individual sample features and global prototype relationships.
We define reward function that encourages high intra-class similarity while maintaining inter-class separation. For sample i with label yt. the reward is:
sim (on(p...))) Pj)
R; = sim(fs (f.
max mim j
Σ
--1
where sim(..)) denotes cosine similarity, C' is the total number of classes, and À controls the reg- ularization strength. The first term maximizes similarity between sample and its class prototype Py,, the second term maximizes the margin to the nearest incorrect prototype, and the third term penalizes excessive similarity between different class prototypes prevent collapse.
To stabilize training across tasks with varying reward scales, we normalize the rewards within each batch. Specifically, for batch of samples with rewards { (Ri, ..., Ra), compute normalized
4

---

Published as a conference paper at ICLR 2025
advantages:
R: PR
A;
On +
where MR B E R3 is the batch mean, OR deviation, and e prevents division by zero.
B V Lizi (R, -MR)² is the batch standard
The policy network xa is trained to maximize the expected reward while preventing drastic changes from reference policy. The optimization objective is:
1 Σ log (w (ww f.. P) A, AKL (Te (Teel
CRIPO
where the first term encourages the policy to assign higher weights to samples with positive advan- tages (samples that improve class separation), and the second term the Kullback-Leible diver- gence:
B Σ (o(ws][i,
To (w Sú, P) Tel W1 [f; P)
DEL
( (Ter)
(5)
using Layer- Norm and GELU activation between layers. Output weights are normalized via softmax. The ref- erence policy Tref is maintained through exponential moving average and updated after each task to prevent drastic policy shifts. The reference policy Ter is initialized as a uniform distribution (equal weights) and updated after each task to the learned policy, ensuring smooth adaptation across sequential tasks while preventing catastrophic changes.
We implement To four-layer MLP with dimensions
3.2 DUAL-MODAL PROMPTING WITH EXPANDABLE PROJECTIONS
Existing methods predominantly adapt textual representations while treating visual encoders as fixed feature extractors Liu et al. (2025b); Kim et al. (2024), creating an asymmetry that hinders perfor- mance on fine-grained recognition tasks. We address this by introducing dual-modal prompting with task-specific projections to jointly adapt visual and textual features.
For each task t, we maintain learnable visual prompts V {of .q..... x 12 KxD shared across all classes, where K denotes the prompt length and D is the embedding dimension. These prompts are appended the patch embeddings of input image z,, yielding the augmented visual input:
0 = [m;; y*]
(6)
where | denotes concatenation. Similarly, we employ learnable textual prompts pretended fixed template. For class c, the textual input follows the template structure:
Te T @ "A photo of a [CLASS]"
where T = [ (5, .... t'r } E RMXD are learnable textual prompts with length M, and @ denotes the concatenation operation the token sequence. This design allows the learned prompts to provide context before the class-specing template, enabling better adaptation task- task-specific characteristics.
To prevent interference between tasks while enabling adaptation, we employ task-specific projection layers that transform the frozen CLIP features:
f2.(0 (i.))) (t.(t) = = (E.(T))
where E, and Eg denote the frozen visual and textual CLIP encoders respectively, P:: R + R D and P RR IR' are task-specific linear projection layers that map features task-adapted space of dimension D'. By maintaining separate projection parameters for each task while keeping the backbone frozen, preserve previously learned knowledge and avoid interference between tasks.
To maintain representative samples the feature space evolves, dynamically update exemplar sets for each class. Let &, denote the current exemplar set and S. the new samples for class c. We compute the centroid of the combined
Σ (x))+ + Σ E (rlt)())))
(9)
E. + |S.
ZyES.
5

---

Published as a conference paper at ICLR 2025
where (-) represents the visual feature extraction through the encoder and projection. We then select the m samples nearest to Pe in the feature space to form the updated exemplar set e.m
Cnew
arg min
Σ (l (z)
Pell2
(10)
IS TE
This exemplar selection strategy, which we term center-based selection, ensures that stored samples remain representative of their class as the feature space evolves across tasks. By prioritizing samples closest to the class centroid, we maintain exemplars that capture core discriminative characteristics while filtering outliers that could degrade prototype quality. This approach becomes particularly important when combined with RLPO, as the policy network relies on exemplar quality to learn effective sample weighting strategies.
3.3 DUFFCULTY-AWARE PROSS-MODAL FUSION WITH MIXTURE OF EXPERTS
Beyond prototype optimization, effective continual learning also requires adaptive processing that accounts for varying sample complexity within each task. While our dual-modal prompting captures discriminative patterns across modalities, not all samples benefit equally, straightforward examples may be over-prove weer-proesssed while challenging boundary cases receive insufficient attention. To address this imbalance, we design mixture- mixture-offexxprtt (MoE) mechanism that dynamically routes samples through specialized processing pathways based on their difficulty, providing the most benefit for maintaining class boundaries during continuing learning.
Firstly, we quantify sample difficulty using the distance from a sample to its corresponding class prototype. For sample with class label ye, we define the difficulty score as:
dj 11 sim(J; PH.) ) + sim(J?,
(11)
where -im(-.- denotes cosine similarity, f." and f are the visual and textual features of sample i respectively, and Py py, are the visual and textual prototypes for class yı obtained from RLPO. Samples with high difficulty scores (far from prototypes) are considered hard, whereas low-difficulty samples are deemed easy. This metric leverages the optimized prototypes from RLPO to provide reliable difficulty estimates.
Based on this difficulty metric, we employ two specialized experts: lightweight expert Eeacy imple- mented as a single linear layer for simple samples, and a deep expert Frand consisting of a three-layer feed-forward network for complex samples. The routing probability for sample a. to the easy expert is computed as:
(12) where (-) is the sigmoid function, [0, 1] is learned threshold parameter that determines the difficulty boundary, and a > O controls the sharpness of the routing decision. The probability of routing to the hard expert is P( Fhand (r:) processing combine outputs from both pathways:
P P(Eaasy (z:) = a-a(dd -))
(Feesyy The final visual features after expert
prreper = P(Eeasy)z)) Eeasy + P(Enunllz;) (fa)
(13)
For cross-modal fusion, employ an attention mechanism adaptively weight the contributions from different modalities. We first concatenate the expert-processed visual features f." preeppent the prototype of visual features fvisual proto- and the textual features frextual i
h; = [f." expert, fvisual proto fixuual proto]
(14)
where | denotes concatenation. The attention mechanism computes importance weights for each modality:
[W,, Wo, W.] SoftMax (atta(xAAAtttton(t))
(15)
where the attention function is implemented feed-forward network with ReLU activation that maps the concatenated features to three scalar weights. The weighted features are then computed as:
W. pleerper Frisual proto = Wo fvisuul proto Freeual proto = W. fiextual proto
(16)
These weighted features f. Frisual proto, and fisecual proto used for computing similarities in the final classification, allowing the model to dynamically emphasize different information sources based sample characteristics.

---

Published as a conference paper at ICLR 2025
3.4 TRAINING AND INFERENCE
Our model freezes CLIP encoders throughout training, adapting only task-speeific prompts (V', Th,, projection layers (Pr. Pf), the RLPO policy network (Fa), MoE experts (Brasy, Ehard), and cross-modal attention weights. During training, prompts augment encoder inputs, projections map features to task-adapted space, and difficulty scores route samples through MoE pathways. RLPO jointly optimizes prototype weights via our multi-objective loss (Appendix D). At inference, we classify samples by nearest prototype matching after identical feature processing.
4 EXPERIMENTS
We evaluate RLAP-CLIP on diverse continual multimodal learning tasks. The model builds on CLIP with ViT-B/16 as the visual encoder, using pre-trained weights from OpenAl and LAION- 400M Radford et al. (2021). Each task is trained for 20 epochs per task using A opti- mizer Khattak et al. (2023). Following standard continual learning principles, we maintain 20 ex- emplars per class for fair comparison with other methods Rebuffi et al. (2017).
Datasets and Baselines. Experiments are conducted on eight datasets spanning different visual domains. Their details can be found in Appendix A and Appendix B. General classification tasks include CIFAR- 100 Krizhevsky et al. (2009) and ImageNet-R Hendrycks et al. (2021). Fine-grained recognition is evaluated on CUB-200 Wah et al. 12) Aircraft Maji et al. (2013), and Cars Krause al. (2013). Specialized domains are covered by Food101 Bossard et al. (2014), UCF101 Soomro et al. (2012), and ObjectNet Barbu et al. (2019). Tasks arrive sequentially, requiring the model to learn new classes while retaining knowledge of previously encountered ones. We compare against three kinds of methods: Classical continual learning approaches, including various finetune vari- ants Zhai et al. (2022); Zhou et al. (2022b); Prompt-based methods such as CODA-Prompl Smith et al. (2023) and DAP Jung et al. (2023); and Vision-language continual learning methods, including SimpleCIL Zhou et al. (2025a), PROOF Zhou et al. (2025b), DKR Cui et al. (2024), and C-CLIP Liu et al. (2025a).
Table 2: Performance comparison across eight datasets. Results show average accuracy (Avg) and final accuracy (Final) as percentages.
Methods
Exemplar
CIPAR-100
CUB-200
Airoralt
Food-101
UCF-101
ImaggeNet-R
ObjectNet
Avg
Avg
Final
Ang
Final
Arg.
Final
Avg
Finul
Arg
Final
Aog
Final
Avg
Final
Finetune
1.72
0.54
Tinetuee CoOp (20226)
24.03
24.74
16.21
Fnctune LTT (2022)
27.69
7.67
51.95
35.96
83.08
78.25
25.10
13.77
29.74
81.79
57.75
29.77
32.85
SimpleCIL es (2025a)
50.20
76.63
79.75
77.52
88.96
86.85
84.73
88.12
55.68
76.54
45.11
CoOp
(20226)
73.04
74.09
87.98
M6.001
N1.74
M8.36
77.13
40.40
PLOT Chen (2012)
78.35
72.03
$2.43
75.39
72.49
52.91
08.24
41.85
CODA-Prompi
(2023)
81.33
79:1
74.7%
87
$4.70
73.46
92.73
90.28
79.5
Jung al. (2023)
76.57
75.39
74.09
83.15
41,45
81.68
78.
87.64
77.23
42.47
32.05
DKR et (202g)
80.17
77.35
78.94
76.23
88.75
86.92
51.28
56.47
85.91
82.73
89:32
87.15
80.47
78 18
47.23
41.30
FROOF Zhou at (2025k)
82.92
78.87
81
79.18
90.53
89
63.59
58.81
87.52
84.74
93.56
91.32
82.50
80.30
49.64
43.65
C.CLIP Liu al. (20025a)
81.75
79,92
82.14
79,83
92.18
90.45
55.73
02.15
87.08
$4.21
91
89.67
83.15
81.06
51.37
47.52
RLAP-CLIP
86.64
79.41
85.78
83.67
94.82
93.15
70.25
68.41
88.34
$6.88
97.68
95.80
85.63
22.22
53.89
48.91
4.1
MAIN RESULTS
Table 2 summarizes the experimental results across all eight datasets, reporting both average accu- racy (performance across all tasks) and final accuracy (performance on all classes after learning the final task). Traditional fine-tuning approaches suffer from severe catastrophic forgetting, with final accuracies ranging from 0.47% on CUB-200 to 2.46% on CIFAR- indicating that parameter- based adaptation alone is insufficient for continual learning in vision-language models. Among prompt-based methods, CODA-Prompt shows better performance across most datasets, reaching 90.28% final accuracy on UCF-101. However, these approaches struggle with fine-grained dis- crimination, as seen in CODA-Prompt's 54.70% final accuracy on Aireraft compared to 90.28% on UCF-101. Besides, vision-language continual learning methods achieve better overall performance. PROOF establishes strong baselines with 78.87% final accuracy on CIFAR-100 and 91.32% on UCF-101, while C-CLIP demonstrates competitive results, particularly on visual recognition tasks like Cars (90.45%) and ImageNet-R (81.06%). Our RLAP-CLIP consistently outperforms all base- lines across datasets and metrics. On general classification tasks such as CIFAR-100, it achieves
7

---

Published as a conference paper at ICLR 2025
86.64% average accuracy, improving over PROOF by 3.72 points. The improvements are more pro- nounced on fine- pree-rained recognition: on CUB-200, RLAP-CLIP reaches 85.78% average accuracy compared C-CLIP's 82. 12%, representing a 3.64 point gain. Similar improvements appear Cars (+2.64 points over C-CLIP) and Aircraft (+4.52 points over C-CLIP). On challenging datasets like ObjectNet, RLAP-CLIP achieves 53.89% average accuracy, outperforming C-CLIP by 2.52 points. The improvements across both average and final accuracy metrics indicate that RLAP-CLIP not only learns new tasks effectively but also maintains better long-term knowledge retention, effec- tively addressing the stability-plasticity trade-off in continual learning.
4.2 ABLATION STUDIES
Component Analysis. We evaluate each component's contribution through incre- mental ablation. The baseline (Base) uses PROOF Zhou al. (2025b) as the start- ing configuration, consisting of frozen encoders with text-only prompting and sim- ple prototype averaging. Ta- ble 3 reports the performance as components are progres-
Table 3:
Component wise ablation study. Numbers in parentheses indicate gains over the previous configuration. CMF denotes cross- modal attention fusion.
Dutaset
Base
+VP
+VP+MoEE
+VP-MoE+RLPO
RLAP-CLIP
CIFAR-100
82.92
83.21 (40.20)
84.05 (40.84)
85.47 (0) .42)
86.64 (+1.17)
CUB 200
R1.67
82.34(+0.67)
83.52 (+1.18)
(+1.37)
85.75 (+0.89)
Class
90.53
91.28 (40.75)
92.67 (+1.399
93.94 (+1.27)
94.82 (40.88)
Aireratt
63.59
66.45
70.25
Food-101
(40
(40.16)
87.95
(40.29)
UCF-101 101
(+0.56)
ImageNNe-R
82.52
82.73
(40.1)
(+1.211
(+0.9113
95.60 (+0.75)
ObjectNet
49.54
50.89
(+1.25) 52.47 (+1.58)
53.21 (+0.74)
53.89 (+0.66)
Average
78:90
79.64
(40.67)
80.79 (+1.13)
81. (.1
82.87 (+0.89)
sively incorporated. Visual Prompts (VP) provide an average gain of 0.67%, with larger improve- ments on fine-grained datasets such as Aircraft (+1%2% and smaller gains on coarse-grained tasks like Food-101 (+0. 166),, indicating that VP captures discriminative patterns crucial for fine ine-grained recognition. The Mixture-of-
(MoE) routing mechanism contributes an additional 1.15% av- erage improvement, with the largest gain on ObjectNet (+1.58%), by routing hard samples through deeper experts while efficiently processing easy ones to enhance feature quality. RLPO yields further contribution (+1.19%), particularly on fine-grained tasks such as Aircraft (+2.28%) and action recognition like UCF-101 (+1,46%) by actively optimizing prototypes to preserve subtle class boundaries. Finally, cross-modal attention fusion (CMF), which adaptively weights visual, prototype, and textual features through the learned attention mechanism. provides additional +0.89% on average, with the largest gains on Aircraft (+ (1552% and CIFAR-100 (+1.17%). The full RLAP-CLIP model, incorporating all four components, achieves consistent improvements across all datasets, confirming that each component contributes complementary benefits.
CIPAR 110
CUB-30
CIFAR
CH18-369
CIEAR-IE
CUB (8)
CHEAR
CUR-360
Alrendi
Airesst
Airralt
basachessi
Alreni
Objeethee
Objecthitt
Objenther
(a) Simple averaging
(b) Weighted averaging
(c) RLPO (Ours)
(d) Overall comparison
Figure& 4: Comparison of prototype construction strategies. Performance ranges from 40% (center) to 100% (outer edge). Larger areas indicate better performance across datasets.
Prototype Construction Strategies. Figure 4 compares three prototype construction approaches across eight datasets. Simple averaging assigns equal weights to all exemplars, providing a basic baseline but yielding limited performance on ObjectNet (41.27%) and Aircraft (56.32%). Quality- weighted averaging refines prototypes by assigning higher weights to exemplars closer to the class centroid, assuming they better represent the class, leading to improved results (CUB-200: 79.86%, Aircraft: 61.48%). Our RLPO method goes further by learning weights through reinforcement learning. The policy network adaptively adjusts weights to maximize both intra-class cohesion and inter-class separation, achieving 83.42% on CUB-200, 92.18% on Cars, and 67. 67.85% on Aircraft. As shown in Figure 4d, RLPO consistently outperforms passive averaging across all task types,
8

---

Published as a conference paper at ICLR 2025
demonstrating its abilitt to adapt weighting strategies to task-specific characteristics and effectively mitigate prototype degradation.
RLAP-CLIP maintains reasonable computational efficiency with 8.5M trainable parameters and 41.7-minute training time per task, achieving a favorable balance between performance and cost (Appendix 1). Our dual-modal prompting strategy outperforms parameter-efficient alternatives in- cluding LoRA by 10+ points (Appendix F), though visual prompting exhibits task-dependent trade- offs on fine-grained variants (Appendix G). Additional analyses in the appendix address practical deployment considerations. Appendix H examines performance under varying memory constraints (30/20/10/5/0 exemplars per class), demonstrating robust performance with graceful degradation. Appendix presents t-SNE visualizations on Stanford Cars dataset, illustrating how RLPO achieves superior prototype alignment compared to simple mean averaging.
4.3 HYPERPARAMETER SENSITIVITY ANALYSIS
We analyze four key hyperparameters understand their impact on model performance.
RLPO Weight (ARLPO). This parameter controls the influence of reinforcement learning- -based prototype optimization in the total loss (Eq.4) 4). Figure 5a shows that all datasets achieve optimal per- formance at ÁRLPO 0.2, with sharp degradation at higher values. UCF-101 drops from 97.68% to 92.15% ÀRLFO increases to 1.0, while fine-grat fine-grained datasets like CUB-200 decline by 5.44 points. This uniform sensitivity suggests that excessive prototype optimization disrupts the learned feature space regardless task type, with moderate reinforcement providing the best balance between class separation and feature stability.
80-
02
60-
150
Reinorrament Learnng Wegen
Officity Weight Fuder
Test homntt jangt
ViuN Frmpt
(a) RL weight ARIPPO
(b) Difficulty weight
(c) Text prompt length
(d) Visual prompt length
Figure 5: Hyperparameter sensitivity amalysis across eight datasets. Optimal values vary by task
characteristics but show consistent patterns within task categories.
Difficulty Weight (3). This factor amplifies the loss contribution of hard samples. Figure 5b reveals consistent optimum at = 0.2 across all datasets. Fine-grained recognition tasks exhibit higher sensitivity, CUB-200 drops 4.44 points from peak to 8 = 1.0, while Aircraft declines by 4.58 points. Hign difficulty weighting causes overfitting to boundary cases and outliers, particularly detrimental for tasks requiring subtle discrimination between visually similar classes.
Text Prompt Length and Visual Prompt Length. Text prompts (Figure 5c) show task-dependent optima, action recognition (UCF-101) peaks at 5 tokens, as action concepts require minimal lin- guistic elaboration. Fine-grained tasks optimize
10 tokens, while distribution-shiftt datasets (ImageNet-R, ObjectNet) achieve best performance at 50 tokens. This pattern reflects varying lin- guistic complexity, where actions require simple descriptors, fine-grained categories need moderate detail, and distribution shifts benefit from extensive contextual guidance. Visual prompts (Figure 5d) uniformly optimize 10 tokens across all tasks, with ame-grained datasets showing strongest gains (Aircraft: + +1.91 points from I to 10 tokens). This consistency indicates that 10 visual tokens provide sufficient adaptation capacity without overfitting, particularly crucial given limited exemplar storage in continual learning.
RELATED WORK
Continual Learning in Vision-Language Models. Recent approaches continual learning in vision-language models primarily adopt eframeter-efficient strategies to mitigate catastrophic for- getting Wang et al. (2023); Ostapenko al. (2022). CoOp Zhou al. (2022b) introduces learnable

---

Published as a conference paper at ICLR 2025
continuous prompts for the text encoder while keeping CLIP backbone frozen. MaPLe Khattak et al. (2023) extends this to both visual and textual modalities with multi-modal prompting. CODA- attention-based prompt pool that dynamically composes
Prompt Smith et al. (2023) develops task-specific prompts, while DAP Jung et al. (2023) generates instance-level prompts for improved adaptability. SimpleCHI Zhou et al. (2025a) demonstrates that frozen CLIP features reasonable performance without adaptation, leading to methods like C-CLIP Liu et al. (2025a) and PROOF Zhou et al. (2025b) that combine prompt learning with prototype- classification.
achieve
Prototype Construction and Management. Prototype-basee approaches have been critical and common to continual learning, with iCaRL Rebuffi al. (2017) pioneering the use of class mean embeddings computed from stored exemplars. Subsequent works improve exemplar selection Re- buffi et al. (2017) or correct representation bias Zhu et al. (2021), but continue treating prototype construction as a passive averaging process. PODNet Douillard et al. (2020) uses pooled outputs for distillation while maintaining prototypes for classification. Recent lision-languuge methods like PROOF Zhou et al. (2025b) and DKR Cui et al. (2024) adopt prototype-based classification but still compute prototypes through uniform averaging. This passive construction becomes problematic as feature spaces evolve across tasks Masana et al. (2022), leading to prototype drift and degraded performance. We reformulate prototype construction as an active optimization problem using re- inforcement learning, where sample weights are learned maximize class separability rather than assigned uniformly.
Mixture of Experts and Adaptive Processing. Mixture of Experts (MoE) architectures demon- strate that routing inputs through specialized pathways improves both efficiency and perfor- mance Fedus et (2022); Zhou et al. (2022c). Recent work shows MoE's effectiveness in large language models Dai et al. (2024); Shao et al. (2024) and multimodal settings Li et al. (2025); Pióro al. (2024). In computer vision, adaptive processing has been explored through dynamic networks that adjust computation based on input complexity. However, these ideas have not been applied to continual learning vision-language models, where samples vary significantly difficulty both within and across tasks. We introduce difficulty-aware routing that allocates deeper processing to challenging samples near class boundaries while efficiently handling straightforward examples, ad- dressing the varying complexity inherent in sequential task learning.
6 DISCUSSION AND CONCLUSION
We present RLAP-CLIP, which reformulates prototype construction as an active optimization prob- lem using reinforcement learning rather than passive averaging. Experiments on eight datasets demonstrate consistent improvements, with the apparent improvements observed on fine-grained recognition tasks where visual adaptation is most impactful. Our results highlight three key find- ings: (I) actively optimized prototypes preserve class separability more effectively than uniform averaging, (2) dual-modal adaptation outperforms text-centric approaches, with visual prompting contributing substantial gains, and (3) difficulty-awarc mixture-of-expe of-eeperts routing efficiently allo- cates computational resources while enhancing feature quality. While our method is evaluated under class-incremental settings, the varying improvements across task types suggest that the benefits of vi- sual prompting task-dependent. Extending RLAP- APIIIIP other incremental learning paradigms and further exploring when visual adaptation offers the greatest benefit remain promising directions for future work.
ACKNOWLEDGMENTS
We sincerely thank the AC and reviewers for their constructive and valuable feedback. This re- search was supported in part by the National Science Foundation of China (NSFC) under grant No. 62302130 and 62402188, the China Postdoctoral Science Foundation under Grant No. 1025T180424, the Postdoctoral Fellowship Program of CPSF under Grant No. GZC20240542 and DEZC20251000 and the Postdoctoral Project of Hubei Province under Grant No. 2024HBB- HCXA026.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. Advances in neural information processing systems, 32, 2019.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101-mining discriminative compo- nents with random forests. In European conference on computer vision, pp. 446-461. Springer, 2014.
Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark ex- perience for general continual learning: strong, simple baseline. Advances in neural information processing systems, 33:159200 13:15920-15930, 2020.
Guangyi Chen, Weiran Yao, Xiangehen Song, Xinyue Li, Yongming Rao, and Kun Zhang- Plot: Prompt learning with optimal transport for vision-language models. arXiv preprint arXiv:2210.01255. 2022.
Zhenyu Cui, Yuxin Peng, Xun Wang, Manyu Zhu, and Jiahuan Zhou. Continual vision-language retrieval via dynamic knowledge rectification. In Proceedings the AAAI Conference on Artificial Intelligence, volume 38, pp. 11704-11712, 2024.
Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Yu Wu, et al. Deepseekmoe: Towards ultimate expert specialization in mixture- of-experts language models. arXiv preprint arXiv:2401. 106066, 2024.
Arthur Douillard, Matthieu Cord, Charles Ollion, Thomas Robert, and Eduardo Valle. Podnet: Pooled outputs distillation for small-tasks incremental learning. In European Conference on Com- puter Vision, pp. 102102..5 Springer, 2020.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120);1-39, 2022.
Alex Gomez-Ville Dipam Goswami, Kai Wang, Andrew D Bagdanov, Bartlomiej Twardowski, and Joost van de Weijer. Exemplar-free continual representation learning via learnable drift compen- sation. In European Conference on Computer Vision, pp. 473-490. Springer, 2024.
Jiangpeng He, Zhihao Duan, and Fengqing Zhu. CI-lora: Continual low-rank adaptation for rehearsal-free class-incremental learning. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 30534 30544, 2025.
Xuehai He, Chunyuan Li, Pengchuan Zhang, Jianwei Yang, and Xin Eric Wang. Parameter-efficient model adaptation for vision transformers. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 817-825, 2023.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, al. The many faces of robustness: A criti- cal analysis of out-of-distribttion generalization. Proceedings of the IEEE/CVF international conference on computer vision, pp.
2021.
Shengqin Jiang, Daolong Zhang, Fengna Cheng, Xiaobo Lu, and Qingshan Liu. Dupt: Rehearsal- based continual learning with dual prompts. Neural Networks, 187:107306, 2025.
Dahuin Jung, Dongyoon Han, Jihwan Bang, and Hwanjun Song. Generating instance-level prompts for rehearsal- free continual learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1 11847-11857, 2023.
Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shah- baz Khan. Maple: Multi-modal prompt learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 191 19112219 2023.
11

---

Published as a conference paper at ICLR 2025
Beomyoung Kim, Joonsang Yu, and Sung Ju Hwang. Eclipse: Efficient continual learning in panop- tic segmentation with visual prompt tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3346-3356, 2024.
Dongwan Kim and Bohyung Han. On the Saaiilitypplasticie dilemma of faassinncrementl learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp- 20196-20204, 2023.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- sie--graied categorization. In Proceedings of the IEEE international conference on computer vision work- shops, pp. 554-561, 2013.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Yunxin Li, Shenyuan Jiang, Baotian Hu, Longyue Wang. Wanqi Zhong, Wenhan Luo, Lin Ma, and Min Zhang. Uni-moe: Scaling unified multimodal Ilms with mixture of experts. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025.
Wenzhuo Liu, Fei Zhu, Longhui Wei, and Qi Tian. C-clip: Multimodal continual learning for vision- language model. In The Thirteenth International Conference on Learning Representations, 2025a.
Yunfei Liu, Yunziwei Deng, Angi Liu, Yanan Liu, and Shengyang Li. Fine-grained multi-modal prompt learning for vision-language models. Neurocomputing, 636:130028, 2025b.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5151, 2013.
Mare Masana, Xialei Liu, Bartlomiej Twardowski, Mikel Menta, Andrew D Bagdanov, and Joost Van De Weijer. Class-incremental learning: survey and performance evaluation on image clas- sification. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(5):5513-5533, 2022.
Oleksiy Ostapenko, Timothee Lesort, Pau Rodriguez, Md Rifat Arefin, Arthur Douillard, Irina Rish, and Laurent Charlin. Continual learning with foundation models: An empirical study of latent replay. In Conference on lifelong learning agents, pp. 60-91. PMLR, 2022.
Maciej Pióro, Kamil Ciebiera, Krystian Król, Jan Ludziejewski, Michal Krutul, Jakub Krajewski, Szymon Antoniak, Piotr Milos, Marek Cygan, and Sebastian Jaszezur. Moe-mamba: Efficient selective state space models with mixture of experts. arXiv preprint arXiv:2401.04088, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp- 8748-8760 8763. PmLR, 2021.
Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 2001-2010, 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
Haizhou Shi, Zihao Xu, Hengyi Wang, Weiyi Qin, Wenyuan Wang, Yibin Wang, Zifeng Wang, Sayna Ebrahimi, and Hao Wang. Continual learning of large language models: A comprehensive survey. ACM Computing Surveys, 2024.
James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Cascante- -onilla Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, and Zsolt Kira. Coda-prompt: Continual de- composed attention based prompting for rehearsal frree continual learning. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pp. 1190--11999 2023.
12

---

Published as a conference paper at ICLR 2025
Kaitao Song, Xiu-Shen Wei, Xiangbo Shu, Ren-Jie Song, and Jianfeng Lu. Bi-modal progressive mask attention for fine one-grained recognition. IEEE Transactions on Image Processing, 29:7006- 7018, 2020.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint 1212.002, 2012.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 2011.
Liyua Wang, Jingyi Xie, Xingxing Zhang, Mingyi Huang, Hang Su, and Jun Zhu. Hierarchical de- composition of prompt-based continual learning: Rethinking obscured sub-optimality. Advances 2023.
in Neural Information Processing Systems,
Yabin Wang, Zhiwu Huang, and Xiaopeng Hong. S-prompts learning with pre-trained transformers: An occam's 's razor for domain incremental learning. Advances in Neural Information Processing Systems, 35:5682-5695, 2022.
Chao Yi, Yu-Hang He, De-Chuan Zhan, and Han-Jia Ye. Bridge the modality and capability gaps in vision-language model selection. Advances in Neural Information Processing Systems, 37: 2024.
Xiaohua Zhai, Xiao Wang. Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 18123-18132 2022.
Da- Wei Zhou, Qi-Wei Wang, Zhi-Hong Qi, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu. Deep class-inc learning: A survey. arXiv preprint arXiv:2302..06648 1(2):6, 2023.
Da-Wei Zhou, Zi-Wen Cai, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu. Revisiting class- incremental learning with pre-trained models: Generalizability and adaptivity are all you need. International Journal of Computer Vision, 1313)):0122-0322 2025a.
Da-Wei Zhou, Yuanhan Zhang, Yan Wang. Jingyi Ning, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu. Learning without forgetting for vision-language models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025b.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for Lasiun-anguuag models. In Proceedings of the IEEE/CVF conference on computer vision and pattem recognition, pp. 168 16825, 2022a.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning prompt for vision- language models. International Journal of Computer Vision, 130(9))2337-2348 2022b
Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M Dai, Quoc V Le, James Laudon, et al. Mixture-of-experts with expert choice routing. Advances in Neural Information Processing Systems, 35:7103-7114 2022c.
Fei Zhu, Zhen Cheng, Xu-Yar Zhang, and Cheng-lin Liu. Class-incremental learning via dual augmentation. Advances neural information processing systems, 34: 13306-13318, 2021.
APPENDIX
For a better understanding of the main paper, we provide additional details in this supplementary material, which is organized as follows:
Detailed specifications of the eight evaluation datasets, categorized into general classi- fication tasks (CIFAR-100. ImageNet-R), fine-grained recognition tasks and specialized domain tasks (Food-101, UCF-101, ObjectNet). Each dataset description includes structural characteristics, inherent challenges, and task partitioning strategies.
SA Dataset:
Aircraft, Cars),
$B Comparison Methods: Comprehensive overview of baseline methods spanning classical con- tinual learning approaches (Finetune variants), prompt-hased continual learning methods (CoOp,
13

---

Published as a conference paper at ICLR 2025
CODA-Prompt, DAP, PLOT), and satte-of-the-aal vision-languaee continual learning techniques (SimpleCIL, DKR, C-CLIP, PROOF).
Rigorous mathematical foundation for the Reinforcement Learning- based Prototype Optimization (RLPO) method, including problem formulation, convergence analy- sis, and theoretical guarantees with formal proofs establishing convergence stationary points and practical convergence properties.
SC Theoretical Analysis:
Detailed description of the multi-objective training strategy that simultaneously optimizes classification accuracy, maintains vision-language alignment, learns op- timal prototype weights, and balances expert utilization. This section includes the mathematical formulation of all loss components and their integration.
SD Multi-Objective Training:
SE Mixture-of-Expeee -fxxertsss Routing Analysis: Detailed examination of difficulty assessment mecha- nisms for sample routing in the mnixxrreof-exppets architecture, comparing random routing, entropy- based routing, and iistance-based routing strategies with comprehensive performance analysis
across datasets.
SF Parameter- Efficient Adaptation Comparison: Comparison with parameter-efficien fine- uniig methods including LoRA and single-modalit prompting variants, validating that dual-modal prompting design achieves substantial improvements through synergistic cross- cross-modal adaptation.
SG Visual Prompting Trade-off Analysis: Fine- tieeggrineed analysis of cases where visual prompting helps versus hurts classification performance, revealing that dual-modal prompting provides a 2.27 to I benefit-to-harm ratio with concentrated challenges in same-family numerical variants where symbolic text distinctions dominate.
SH Exemplar Budget Analysis: Performance evaluation across different exemplar budgets (30, 20, 10, 5, and exemplars per class), demonstrating that 20 exemplars provide near-optimal perfor- mance while maintaining reasonable memory overhead, and showing graceful degradation to CLIP zero-shot baseline when exemplars are unavailable.
$I Scalability and Computational Complexity: Analysis of AAAI''s computational effi- ciency under standardized conditions, demonstrating 2x faster training than DKR with only 9% in- ference overhead and 7% additional FLOPs, achieving the best accuracy accuracy-eficienccy trade-off among all compared methods.
SJJ Visualizing Prototype Quality in Feature Space: t-SNE visualizations comparing RLPO
prototypes against simple mean prototypes on Stanford Cars, demonstrating improved alignment with both class centers and text embeddings, directly explaining performance gains through active optimization that prevents prototype degradation.
A DATASET
We evaluate on eight diverse datasets spanning three categories: general classification (CIFAR-100) ImageNet-R), fine-grained recognition (CUB-200, Aireraft, Cars), specialized domains (Food- 101, 1CF, ObjectNet).
A.1 GENERAL CLASSIFICATION TASKS
CIFAR-100 contains 60,000 32x32 images across 100 classes organized into 20 superclasses. De- spite low resolution, the dataset requires robust representations for discriminating closely related classes.
ImageNet-R comprises 30,000 images across 200 classes featuring artistic renditions including paintings, sketches, and sculptures. The dataset tests generalization beyond photographic domains through substantial visual style variations.
14

---

Published as a conference paper at ICLR 2025
A.2 FINE-GRAINED RECOGNITION TASKS
CUB-200 contains 11,788 images of 200 bird species with high visual similarity requiring discrim- ination of subtle plumage patterns, beak shapes, and color distributions. Intra-class variation from gender dimorphism and seasonal changes demands invariant species-speciffe representations.
Aireraft encompasses 10,200 images of 100 aircraft variants differing only in subtle details like engine configurations. Aircraft from the same manufacturer share nearly identical designs, with distinguishing features visible only from specific angles.
Cars contains 16,185 185 images covering 196 make/model/year combinations. Discrimination requires sensitivity minor design elements like grille patterns and year-specific modifications while main- taining view invariance across angles and lighting.
A.3 SPECIALIZED DOMAIN TASKS
Food-101 contains 101, ,000 images of 101 food categories with high variability from preparation methods and presentation styles. Many dishes share ingredients or similar appearances, requiring discrimination of specific combinations and presentation cues.
UCF-101 comprises 13,320 video clips of 101 human actions, processed frame sequences. The dataset covers diverse action types requiring understanding of poses, object relationships, and scene context. Many actions share similar static appearances, demanding comprehensive visual cue anal- ysis.
ObjectNet contains 50,000 images across 313 object classes captured in natural environments with objects rotated, occluded, and placed unusual contexts. The dataset challenges models trained canonical object views by presenting objects from non-standard viewpoints and backgrounds, testing robustness to distribution shift.
B COMPARISON METHODS
We compare RLAP-CLIP against comprehensive baselines spanning different continual learning paradigms, from classical approaches to state-of-the-ar vision-language techniques.
B. CLASSICAL CONTINUAL LEARNING BASELINES
Finetune represents the naive baseline with sequential fine-tuning on each task without forgetting prevention. This approach lacks memory retention mechanisms, leading to severe catastrophic for- getting as optimization overwrites parameters critical for previous tasks. The standard version up- dates all parameters including the CLIP backbone, typically resulting in complete forgetting with final accuracy below 3% on most datasets.
Finetune LiT freezes the visual encoder while allowing text encoder fine-tuning, motivated by CLIP's visual encoder learning more general features. However, updating the text encoder still causes significant forgetting textual representations drift from their original configuration, break- ing vision-langual alignment for previous tasks. While showing improvement over full fine-tuning (30-50% final accuracy). it demonstrates that freezing one modality is insufficient for effective con- tinual learning.
Finetune CoOp adapts Context Optimization by learning task-specific prompts prepended text encoder inputs while keeping CLIP frozen. For each task. new prompt vectors are learned from scratch with previous prompts discarded. This preserves base model knowledge better than full fine-tuning but lacks prompt retention, resulting in moderate performance (20-40% final accuracy).
B.2 PROMPT-BASED CONTINUAL LEARNING METHODS
CoOp with exemplar memory extends the basic approach by maintaining memory buffer of ex- emplars, enabling knowledge retention through replay. The method learns task-specific prompts preserved across tasks, with a unified prompt used during inference. The optimization combines current task loss with distillation loss on replay samples: L Lrsst ACreplay- However, exclusive
15

---

Published as a conference paper at ICLR 2025
focus on textual prompts limits visual adaptation capabilities, with performance plateauing around 70-80%.
CODA-Prompl introduces attention- based prompt pool mechanism for dynamic task-specific prompt composition. The method maintains a shared prompt pool P = {Pi,..., Pvv with attention- based selection: wi = softmax(y" k// /d), where selected prompts are weighted and combined. This enables knowledge sharing across tasks while allowing task-specific combinations, achieving 75-90% final accuracy on several benchmarks.
DAP combines architectural expansion with prompt learning by adding lask-specific adapters: h' = h+ (h),, where adapters use bottleneck architectures. Prompts work synergistically with adapters, providing task-specific context influencing both frozen backbone and adapter computa- tions. While competitive, the growing architectural complexity becomes computationally expensive as tasks increase.
PLOT formulates prompt selection an optimal transport problem for better task-awareness. The method computes optimal transport plans minimizing transportation cost C₁₁ = Ilʃ; - Py 113 between input features and prompt representations. This provides principled prompt combination from dif- ferent tasks, enabling knowledge transfer when tasks share concepts, though performance depends on transport cost computation quality.
B.3 VISION-LANGUAGE CONTINUAL LEARNING METHODS
SimpleCIL leverages CLIP's zero-shot capabilities without parameter updates, storing class names and using zero-shot classification by comparing image embeddings with text embeddings of class descriptions. While avoiding catastrophic forgetting, cannot adapt to task-specific characteristics or domain shifts. Performance depends heavily on pre-trained representation quality and semantic clarity of class names, achieving 75-85% accuracy on well-aligned datasets but struggling with specialized domains.
DKR introduces dynamic memory management with separate buffers for visual and textual infor- mation. The method stores image exemplars and augmented text descriptions, employing a dual- stream architecture: friset = a Jvisual + (1 [ Knowledge retention combines exemplar re- play with cross-modal distillation maintaining consistent predictions: Laisill
KI-(p' (Iphew) +
(p'ullllp)
C-CLIP adapts CLIP through parameter-efficient fine-tuning and prototype-based classification. The method introduces lightweight task-specific adapters: f = Ex(x) + A", (E,()) and fi = Ex(x) (E, Class prototypes are maintained as exemplar means with nearest-prototype old
classification. Regularization maintains similarity structure between prototypes: Legg = En, 15 achieving over 90% accuracy on several datasets.
new k
PROOF represents current state-of- of-the-art, combining prompt learning, prototype management, and open-world recognition. The method maintains class-specific and task-specific prototypes, optimiz- ing prompts maximize margins: Lmargin max(0, m-+ 8neg - Spoc). A two-stage inference process first identifies tasks then performs class-specific classification. While achieving 78-91 % final accu- racy, PROOF still relies on passive prototype construction through averaging and focuses primarily on textual adaptation, limitations our RLAP-CLIP addresses through active prototype optimization and balanced multimodal adaptation.
C THEORETICAL ANALYSIS
In this section, provide theoretical foundation for the Reinforcement Learning baaning-bbsed Prototype Optimization (RLPO) method, establishing its convergence propertics and analyzing its advantages over traditional uniform averaging approaches.
16

---

Published as a conference paper at ICLR 2025
C.I PROBLEM FORMULATION
Traditional continual learning methods construct class prototypes through uniform averaging:
Pc
uniform
f
(17)
Eel
IEE
where & denotes the exemplar set for class and fi E Rd represents the feature vector of sample This passive approach treats all samples equally, which is suboptimal when exemplars contain outliers, exhibit multi-modal distributions, or have varying discriminative importance.
RLPO reformulates prototype construction as a weighted optimization problem guided by reinforce- ment learning. We define policy network xp I& x P + R2 parametrized by θ C IR where P {pI....,Per C R2 represents the current prototype set. The weighted prototype for class c becomes:
(0( Wifi,
w, = To (fr
Σ w =
(18)
REE
iEE
The reward function for sample with feature f, and true label y, is designed to maximize intra-class similarity while minimizing inter-class confusion:
sim Lim(Py.:P))
(P( sim(fs. Pv.)
-mxx sim (f.( 37
(19)
C --1
where sim(.,.) : Rdd x R2d - [- denotes cosine similarity, and À > regularization coefficient that encourages prototype separation.
The key insight is that RLPO optimizes the global objective:
J(0) B [R[, ys P(O))]
(20)
subject the constraint that prototypes depend on 0 through Eq. 18, creating non-stationary opti- mization landscape.
Assumption (Regularity Conditions) We assume the following conditions hold:
1. Bounded features: There exists By > 0 such that [[f]l S By for all f E F.
2. Lipschitz policy: The policy network ny is Læ-Lipschitz continuous in 8: | (f. P) Te, (f, P)) & L [|] - 0.2l.
3. Lipschitz similarity: The similarity function is Li-Liscchitz in both arguments.
4. Bounded policy gradient variance: The policy gradient estimator satisfies Ellgg (0J( S o² for some <, Co.
5. Non-degeneracy: For all i Ec. we have To (ft,P) Emin 0.
Lemma (Prototype Smoothness) Under Assumption the prototype mapping 0 pc(() is Lp- Lipschitz continuous with:
(0:) (e())) S T.p |0 02
(21)
where Lp 2B
To establish this result, we need to show that small changes in the policy parameters 0 lead to bounded changes in the resulting prototypes. This crucial property for analyzing convergence, as it ensures that the optimization landscape is well-behaved despite the non-stationary nature of the problem.
Let w. (k) To (Je, P) denote the weight assigned to feature fa under parameter setting 0g for
I 2), Since the weights are normalized, we have E.es. w (k) for both k 1,2. From
Eq. 18, we can write:
Po(01) Po(02)
Σ
(w (1)))
w.
(22)
ieE.
17

---

Published as a conference paper at ICLR 2025
Applying the triangle inequality:
1p-(0:) -Po(2)] -
10
(2)
wi |1
(23)
KEE
By the Lipschitz property of the policy network (Assumption I(i))::
l w (2) = no, (fi, P) -T0, To,(fi P)I S L 100 - 02ll
(24)
Using the bounded features assumption (Assumption 1(i):
[pc(01) -pe(62)) = Σ
T. O2l By | T. B -22l
(25)
3EE.
Since the weights satisfy Liee, w 1 and each w (8( z Emin by Assumption I(v), we have [Eel S ___ Moreover, the weight redistribution is bounded by the normalization constraint, yield-
Emin
ing:
2B 110 02ll
(pe(0))- - pp.
(26)
6min
This completes the proof with Lp
2B,
Lemma (Reward Function Properties) The reward function (P) satisfies:
1. Bounded gradient: [[VRR,(P)I S LR 2L XL,
2. Alignment property: For the optimal prototype set Pt, we have [[R((P")] 2 [R[R, (Pimitii))
We prove each property separately. The key insight is that the reward function in Eq. 19 is composed of cosine similarity terms, each of which has well-understood Lipschitz properties.
Proof of (i): The reward function can be written as:
sim(pn, Ps)
(P(P) = sim simmff.. (f. Pv.)
max sim (f.,P) - Σ
(27)
C C-1
J
Ti
For the first term Ti Sim(fe.Py..) the gradient with respect to Py: satisfies:
|V Till L,
(28)
by the Lipschitz property of the similarity function (Assumption 1(ii)).
For the second term T- = maxj4y, sim.fsPjj) at differentiable points:
1 Tal S L.
(29)
where j = arg maxjs/y, sim(fi.pj)
For the third term Ta. the gradient with respect Py; is:
Pv, sim(Pue:P) Pi)
Tg = Σ
(30)
Pv.
C-1
Since there are at most (C non-zero terms, each bounded by Jo
ba
IV P., Tall X Σ C-1
AL.
(31)
14
18

---

Published as a conference paper at ICLR 2025
Similarly, for each Py where # ys:
XL. IV Tsll - C-1
(32)
The total gradient involves most three non-zero components: Gradient with respect to Pa: bounded by L, + A Gradient with respect to py-: bounded by L, Gradients with respect to other prototypes in T3: each bounded by ALe
C-1
Therefore:
[pR((P)] L. + (2
(33)
We set LR = 2L, + AL for consistency.
Proof of (ii): The alignment property follows from the fact that RLPO optimizes over all possible weighting schemes, including uniform weighting. Since the optimal solution O is obtained by maximizing ((0) over all f θ, and uniform weighting corresponds
specific 0g e:
J(0)) = BiR(f.y; p.) 2 ((0)) = E[R(f, y; Puniform)]
(34)
This completes the proof.
C.2 CONVERGENCE ANALYSIS
The main challenge in analyzing RLPO lies in the non-stationary nature of the optimization land- scape. Unlike standard stochastic optimization problems where the objective function fixed, in RLPO the objective depends the current prototypes, which themselves depend the policy parameters being optimized.
Theorem (Convergence to Stationary Points) Consider the stochastic gradient ascent update:
Oc+1 = 02 + 7.9t
(35)
where 9s unbiased estimate of (0) and {nd satisfies E. T) and E, n < 00. Under Assumption 1, the sequence {0.} converges to a stationary point of (0) almost surely. Moreover, for the step size Trt Vi
no we have:
Ci VT
CologT
T
Σ E(I
+
+ dplas
(36)
T
where C1, C2 are constants depending on problem parameters, and the bias term satisfies:
L T-p
ōpiux
O
(37)
Emin
We analyze the convergence using Lyapunov function approach adapted for non-stationary objec- tives.
Step 1: Gradient Decomposition. The true gradient can be decomposed as:
(o) = (RU. Vo log To P(O)) + -V VoP(O)
(38)
The first term the standard REINFORCE gradient, while the second captures the indirect effect through prototype changes.
Step 2: Lyapunov Analysis. Define V(0) = J(0*) - -((0) where 0* a global optimum. For the
update 0.+1 0, 17:1:
L (0² (oJ(()1) + 17 Soias (39)
=
where Toy is the Lipschitz constant of J (bounded by T. Lp from Lemmas 1 and 2).
19

---

Published as a conference paper at ICLR 2025
Step 3: Bias Analysis. The bias arises because practical gradient estimates ignore the prototype dependency:
0 LRL2
bins (IIVp (.P((.))))V)PP())1) & LR Lpp =0
(40)
Cmin
using the bounds from Lemmas and 2.
Step 4: Convergence Rate. For 7)t S :t:
(( < ElV (e)) TK (VeJ + LITÑo²
+ medbias
(41)
Summing from 1 to T and using 1)t Vt
1 T
2V1
L
28
E VoJ
+
(42)
noTT
VT
170
This yields the stated convergence rate with appropriate constants C1 and C22-
Corollary (Practical Convergence) In practical scenarios with large number of classes C and sufficient policy network capacity, the bias term becomes negligible relative to the main convergence terms:
Spias C
As C oo,
O
(43)
This indicates that RLPO achieves near-optimal convergence for large-scale continual learning problems.
From Theorem 1, Spiu O
(LRLy)
fiin The constants LR. Lp, and Cuin depend the feature
dimension d and network architecture but not directly on C. When normalized by the number of classes (which often appears in practical loss functions), the relative impact the bias diminishes as C' increases, ensuring that RLPO's convergence approaches that of standard policy gradient methods in large-scale settings.
D MULTI-OBJECTIVE TRAINING
RLAP-CLID employs multi-objective training strategy that simultaneously optimizes classifica- tion accuracy, maintains vision-language alignment, learns optimal prototype weights, and balances expert utilization. The total training objective combines four complementary loss terms:
Cerall Lets + Adip + ARLPOCRLPO + AMoe
(44)
where Actip- AREPO, and AMOE are hyperparameters controlling the relative importance of each loss component.
D.1 CLASSIFICATION LOSS
The classification loss weights samples according to their difficulty scores to emphasize challenging cases that are more likely define class boundaries:
B
1
Cats
B Σ
(1 + d:) p(g/l())
(45)
where B denotes the batch size, 7 > 0 is a hyperparameter controlling the emphasis on difficult samples, d, J0, 1] is the difficulty score for sample i, and p(u(2;) represents the predicted prob- ability for the true class Yi given input I;, The probability is computed using softmax over cosine similarities between the sample's features and all class prototypes:
20

---

Published as a conference paper at ICLR 2025
p(us(z4) exp(sim(ffr Pv,
)/ Tcss
(46)
Ee= exp(sim (fr. Pe) pc)/Tab)
where f; represents the weighted multimodal features after cross-modal fusion, Pe denotes the pro- totype for class c, and Tels 0 is temperature parameter for the classification softmax.
D.2 CONTRASTIVE LOSS
To maintain the vision-language alignment learned during CLIP pre-training, we employ a bidirec- tional contrastive loss that ensures matched image-text pairs have higher similarity than mismatched pairs:
B
log exp(sim(ff (oossim(f".f ff)/T) + log exp(sim(ff) ff)/ Σ exp(sim( (59/TT
Delip
1=1 Σ/= exp(sim([f Σ
(47)
2B
where f. and f denote the visual and textual features for sample respectively, sim(..) computes cosine similarity, and r > O is learned temperature parameter that controls the sharpness of the similarity distribution. The first term represents the image-to-text contrastive loss, while the second term represents the text-to-image contrastive loss.
D.3 PROTOTYPE OPTIMIZATION LOss
The reinforcement learning-basee based prototype optimization loss LRLPO trains the policy network to learn optimal sample weights for prototype construction. This loss takes the form:
1 Σ log (wi A, DKL (To (Tel)
CRPO
(48)
where Ty the policy network with parameters w; is the weight assigned to sample i, A; normalized advantage (reward minus baseline) for sample i, and ÅXL > 0 controls the strength of the KL regularization that prevents the policy from deviating too far from the reference policy Teet-
D.4 EXPERT BALANCING Loss
To ensure efficient utilization of both experts in our mixture-of-experts architecture and prevent mode collapse where one expert processes all samples, introduce expert balancing loss:
B
P(e 1) 2
L
Σ
(49)
B
eE (wwsyy.auaa bard)
where P(e[a.) denotes the routing probability for expert given sample x;- This loss penalizes deviations from equal expert utilization, encouraging each expert process approximately half of the samples. The quadratic penalty ensures smooth gradients and stable optimization.
E MIXURE-OO-EXPERITS ROUTING ANALYSIS
The effectiveness of mixture-of-expertt architectures largely depends on the quality of the routing mechanism that determines which samples should be processed by which expert. our framework, employ two specialized experts: lightweight expert for simple samples and a deep expert for complex samples. The key challenge lies in accurately assessing sample difficulty to enable optimal resource allocation.
21

---

Published as a conference paper at ICLR 2025
We evaluate three distinct approaches for difficulty assessment and sample routing:
Random Routing serves as a baseline where samples are assigned to experts uniformly at random with equal probability. This approach helps isolate the architectural benefits of having multiple processing pathways from the benefits of intelligent routing. While random routing ensures balanced expert utilization, it cannot adapt processing depth to sample characteristics, potentially wasting computational resources on simple samples while under- peoder-poosssing challenging ones.
Entropy-based Routing prediction uncertainty as a proxy for sample difficulty. The entropy of the model's output distribution is computed as H -E Pe log predicted probability for class c. Samples with entropy above a learned threshold are considered difficult and routed the deep expert. The intuition is that uncertain predictions indicate challenging samples requiring enhanced processing. However, this approach has limitations: early in training, many samples produce uncertain predictions regardless of their intrinsic difficulty, and prediction uncertainty may not accurately reflect the discriminative challenges specific continual learning scenarios.
Pc. where Pe represents the
Distance-based Routing (Ours) directly measures sample difficulty by computing the distance from each sample to its class prototype in both visual and textual feature spaces. The difficulty score
is defined as d, = sim(f"" py, +imml P..)) This approach captures the intuition that samples far from their class centers are more challenging to classify correctly and would benefit from deeper processing. The method leverages multimodal information and directly relates to the class bound- ary structure, making it particularly suitable for fine-graie grained recognition tasks where subtle visual differences matter.
2
Table 4: Performance comparison of difficulty assessment mechanisms for mixture-of-expprtt rout- ing across eight datasets.
Dataset
Random Routing
Entropy-based
Distance-based (Ours)
UCF-101
93.87
94.23
97.68
CUB-200
83.52
84.87
86.32
CIFAR-100
84.16
85.12
86.64
Cars
92.52
94.18
96.62
Aircraft
66.03
67.89
70.25
Food-101 101
87.75
88.01
88.24
ImageNet-R
83.15
84.47
85.63
ObjectNet
53.33
55.29
57.79
Table reveals distinct patterns across different task types. Random routing establishes baseline performance, with results ranging from 53.33% ObjectNet 93.87% UCF-101, reflecting the inherent difficulty variations across datasets.
Entropy-based routing shows modest improvements over random routing, with gains ranging from 0.26% on Food-101 to 1.96% on ObjectNet. The limited improvements suggest that prediction uncertainty alone is insufficient for effective sample routing. This occurs because entropy reflects the model's 's current confidence rather than the intrinsic structural difficulty of the sample. High entropy can result from insufficient training rather than genuine sample complexity, leading to suboptimal routing decisions.
Distance-basee routing achieves substantial and consistent improvements across all datasets. The most significant gains appear on fine-grained recognition tasks: Aircraft (+4.22% over random), Cars (+4.10%) and CUB-200 (+2.80%). These improvements highlight the method's effectiveness for tasks requiring subtle discrimination. where samples class boundaries particularly benefit from enhanced processing through the deep expert.
The superior performance stems from distance-based routing's direct connection to the geometric structure of the feature space. By measuring proximity to class prototypes, this approach identifies samples that are genuinely challenging for the current representation rather than those producing uncertain predictions due to model limitations. This alignment ensures computational resources allocated where they provide maximum benefit for maintaining discriminative class boundaries,
22

---

Published as a conference paper at ICLR 2025
which is crucial for continual learning scenarios where new tasks continuously reshape the feature space.
F
ADAPTATION COMPARISON
We compare RLAP-CLIP against other parameter-efficient fine-tuning (PEFT) approaches val- idate our dual-modal prompting design. Table 5 presents results on CIFAR-100 and CUB-200 datasets.
Low-rank adaptation (LoRA) with 1.77M parameters achieves only 63,47% and 64.92% final ac- curacy on CIFAR-10 100 and CUB-200 respectively. This demonstrates that low-rank weight updates alone are insufficient for continual learning in vision-language models. LoRA lacks task-specific modality alignment mechanisms, leading to severe performance degradation across sequential tasks as the learned low-rank matrices fail to preserve discriminative boundaries for previously learned classes. RLAP-CLIP's dual-modal prompting design, with equivalent parameters (8.5M), achieves 79.41%/83.6679 final accuracy, representing 10-12 percentage point improvements modality variants. This substantial gain validates that synergistic cross-modal adaptation, where visual and textual prompts jointly guide both encoders, is the critical factor for effective continual learning in lision-language models. The combination enables the model to leverage complementary information from both modalities while maintaining alignment in the shared embedding space.
single-
Table 5: Comparison with parameter-efficiet fine-tuning methods.
Method
Trainable Params (Million)
CIFAR-100
CUB-200
Avg Acc
Final Acc
Avg Acc
Final Acc
LoRA
1.77
76.23
63.47
75.68
64.92
Text Prompts Only
8.42
78.15
66.82
77.94
68.35
Visual Prompts Only
8.45
79.87
69.24
79.61
71.58
RLAP-CLIP (Dual-Modal)
8.5
86.64
79.41
85.78
83.67
G VISUAL PROMPTING TRADE-OFF ANALYSIS
While our main results demonstrate consistent improvements from dual-modal prompting, under- standing when and why visual adaptation may introduce challenges provides important insights into the method's behavior. We conduct a one-grained analysis on the Aircraft dataset to characterize cases where visual prompting helps versus hurts classification performance.
Table 6: Overall performance comparison on Aircraft dataset across 3,333 test samples.
Mode
Accuracy (%)
Correct Predictions
Text-only Prompting
65.02 25.02
2,167
Dual-modal Prompting
68.41
2,280
Improvement
+3.39
+113
We compare text-only versus duallmodal predictions across 3,333 test samples. Table 6 shows that dual-modal prompting achieves 68.41% accuracy compared to text-only's 65.02%, representing net gain of 3.39 percentage points. Table 7 breaks down the outcomes into four categories. Visual prompting helps 200 samples (6.00% of total) where text-only fails but dual-modal succeeds, which term HELP cases. Conversely, 88 samples (2.64% of total) represent HURT cases where text- only succeeds but dual-modal fails. This yields a 2.27 to benefit-to-harm ratio. Importantly, the 88 hurt cases represent only 4.06% of text-only's correct predictions, indicating that dual-modal prompting preserves the vast majority of text-only's 's capabilities while adding substantial new correct predictions.
To understand where visual prompting introduces errors, we analyze the distribution of HURT cases across aircraft classes. Table 8 shows the ten classes with the most HURT cases. These top ten
23

---

Published as a conference paper at ICLR 2025
Table 7: Classification outcome breakdown across four categories based on prediction correctness.
Category
Count
Percentage (
Both modes correct
2,079
62.37
HELP cases (text fails, dual succeeds)
200)
6.00
HURT cases (text succeeds, dual fails)
88
2.64
Both modes incorrect
966
28.99
classes account for 72.7% of all hurt cases, revealing a clear concentration in specific aircraft cate- gories. The classes share common characteristics that explain why visual prompting struggles while text-only succeeds. Table categorizes these patterns into three groups. fame-family numerical variants account for 46.6% of HURT cases, such as Boeing 737-300 versus 737-400 which differ by approximately 3 meters in fuselage length. Regional jet series represent 21.6% cases, including models like CRJ-700 and ERJ 145. Manufacturer systematic naming conventions contribute 31.8%, encompassing variants like the Airbus A318, A319, A320, and A321 sequence.
Table Top ten aircraft classes ranked by number HURT cases.
Rank
Aireraft Variant
Hurt
Help
Net
1
Boeing 737-300
2
2
Boeing 737-400
10
2
88
3
Boeing 757-200
8
2
-6
4
Boeing 767-300
7
3
-4
5
Airbus A330-200
6
1
-5
6
MD-87
5
2
7
CRJ-700
5
1
-4
8
Boeing 737-500
2
-3
9
ERJ 145
4
1
-3
10
A321
2
Total (top ten)
64
18
-46
The concentration of HURT cases in same-family numerical variants reveals fundamental trade- off dual-modal prompting. Text encoders excel at processing symbolic distinctions. The class names "Boeing 737-400" and "Boeing 737-300" create well-separated token embeddings despite referring to visually similar aircraft, as the numerical suffixes provide clear discriminative signals. In contrast, visual prompts trained on only 20 exemplars per class learn family-level features such as shared cockpit design, engine configuration, and tail shape that are common across variants. This strengthens antra-family similarity, making it difficult distinguish between Boeing 737-300 and 737-400 when they differ primarily by fuselage length, a subtle difference barely perceptible at 224x224 resolution.
Table 9: HURT cases categorized by pattern type with representative examples.
Pattern Type
Cases
Percentage (%)
Same-family numerical variants
41
46.6
Regional jet series
19
21.6
Manufacturer systematic naming
28
31.8
When cross-modal fusion combines modalities, attention weights redistribute from text-dominant weights of approximately 0,95 in text-only mode to balanced weights of approximately 0.35 for visual, 0.45 for text, and 0.20 for prototype channels in dual-modal mode. This dilution of the text signal occasionally pulls predictions toward family centroids where visual differences are imper- ceptible, causing confusions between numerical variants. However, the analysis reveals that visual prompting's 's benefits far outweigh its limitations. The 2.27 to benefit-to- harm ratio, combined with the fact that HURT cases represent only 4.06% of text-only's correct predictions, demonstrates that
24

---

Published as a conference paper at ICLR 2025
dual-modal prompting provides substantial overall improvement with net accuracy gain of 3.39 percentage points while preserving 62.37% of samples that remain correct under both modes. The hurt cases occur in highly specific scenarios involving came-family numerical variants where sym- bolic text distinctions dominate over visual features. For the vast majority of fine-grained, recognition tasks where discriminative patterns span both modalities, dual-modal prompting provides substantial and consistent improvements.
H EXEMPLAR BUDGET ANALYSIS
We conducted experiments across different exemplar budgets: 30, 20, 10, 5, and 0 exemplars per class. The zero-exemplar setting effectively reduces our method CLIP's zero-shot classification baseline. Table 10 presents the complete results across all eight benchmark datasets.
Table 10: Performance comparison across different exemplar budgets. Average accuracy (%) af- ter learning all tasks. A(30~220) and A(20+11) show accuracy changes when reducing memory budget.
Dataset
30 Ex.
20 Ex.
10 Ex.
5 Ex.
Ex.
A(30-220)
A(20-+10)
CIFAR-100
87.15
86.64
81.23
72.46
66.00
-0.51
-5.41
CUB-200
86.24
85.78
79.31
67.85
51.50
-0.46
-6.47
Cars
95.38
94.82
90.15
83.28
70.01
-0.56
-4.67
Aireraft
71.03
70.25
59.87
48.32
34.13
-0.78
-10.38
Food-101
82.15
81.67
76.42
68.19
58.24
-0,48
-525
UCF-101
84.32
83.89
78,95
71.03
62.45
-0.43
-4.94
ImageNet-R
79.67
79.21
74.38
66.84
56.73
-0.46
-4.83
ObjectNet
73.84
73.26
68.15
60.47
49.97
-0.58
-5.11
Average 84.95
84.37
77.64
67.98
55.41
-0.58
-6.73
The results demonstrate that "20 exemplars" sits important point: reducing 10 exemplars (50% memory reduction) causes substantial 6.73% average accuracy loss, whereas increasing to 30 exemplars (50% memory addition) yields only 0.58% marginal gain. This reveals that 20 emplars provide near-optimal performance while maintaining reasonable memory overhead, con- sistent with the standard setting adopted by other continual learning methods (iCaRL, PODNet, PROOF). At O exemplars, RLAP-CLIP achieves 55.4% average accuracy and these results closely match CLIP zero-shot baselines (e.g., CIFAR-100: 65%, CUB-200: 50%), indicating that when RLPO's ceempaar-depndeer components disabled, the method gracefully degrades to baseline performance rather than catastrophically failing.
I SCALABILITY AND COMPUTATIONAL COMPLEXITY
We analyze RLAP-CLIP's computational efficiency and scalability characteristics. Table 11 com- pares RLAP-CLIP against baseline methods under standardized conditions single NVIDIA GPU.
Table 11: Computational efficiency comparison. All metrics are reported relative to SimpleCII. as the baseline (Ix).
Method
Trainable Params
Train Time/Task
Inference Latency
FLOPs/Imaae
SimpleCIL
0
1x
1x
1x
CoOp
0.03x (~-8)
2.5x
1.01x
1.01x
PROOF
1x (~003MM)
4x
1.02x
1.02x
DKR
22x (~-6.6M)
12x
1.12x
1.15x
RLAP-CLIP
20x (~-6M)
6x
1.09x
1.07x
RLAP-CLIP introduces 6M trainable parameters and achieves 3.72-4.46 average accuracy gains over PROOF, translating approximately 0.65 accuracy points per million parameters. In contrast, DKR
25

---

Published as a conference paper at ICLR 2025
requires comparable parameters (6.6M) but yields lower accuracy. For training efficiency. RLAP- CLIP is 2x faster than DKR while achieving superior accuracy across all datasets, and the 1.5x overhead compared to PROOF is justified by consistent 3-4 point improvements grained datasets such as Aircraft (+6.66 points) and CUB-200 (+4.11 points). For inference, RIAP- CLIP adds only 9% latency, which is 3% lower than DKR, since diffieulty scores computed directly from existing features without additional forward passes. RLAP-CLIP also requires only 7% additional FLOPs compared to 15% for DKR, achieved through adaptive computation where the lightweight easy expert handles the majority of samples efficiently. In summary. RLAP-CLIP achieves the best eecuracy-efficiene
challenging fine-
trade-off with 2x faster training and 8% lower FLOPs than DKR, while providing 3.72-4 46 point improvements over PROOF with only 1.5x additional training time and 5% extra inference cost.
Bet in ardeniinge
1AAE D1
Figure 6: t-SNE visualization comparing prototype construction methods Stanford Cars. RLPO prototypes (b) achieve tighter alignment with both class centers (distance: 1.13->0.4) and text embeddings (distance: 1.24 1.24-+0.93) compared simple mean prototypes (a).
J VISUALIZING PROTOTYPE QUALITY IN FEATURE SPACE
Figure 6 visualizes prototype quality on Stanford Cars using t-SNE projections of 12 fine-grained categories. Circles denote image embeddings, triangles denote text embeddings, and stars denote learned prototypes. Panel (a) shows simple mean prototypes; panel (b) shows RLPO prototypes.
RLPO prototypes position closer to class centers and text embeddings. The distance to center de- creases from 1.13 0.94, and distance text reduces from 1.24 0.93. This occurs because RLPO learns to weight representative samples higher while down-weighting outliers, whereas uni- form averaging allows noisy exemplars degrade prototype positions. For visually similar classes (BMW/Audi in upper-left, Toyota Camry/Honda Accord in upper-right), RLPO maintains clear inter-class separation while simple averaging produces overlapping prototypes that increase con- fusion. This improved prototype positioning directly explains RLAP-CLIP's 2.64-point gain on Cars and 4.56-point gain on Aircraft over C-CLIP, demonstrating that active optimization prevents the prototype degradation that causes catastrophic forgetting in continual learning.
26


Reviews:
[{"review_id": "7uKkYm2Ti4", "reviewer": "Reviewer_hrcE", "summary": "The paper introduces RLAP-CLIP, a continual multimodal learning framework that (i) replaces passive prototype averaging with reinforcement-learning-based prototype optimization (RLPO), (ii) incorporates dual-modal prompting for visual and textual inputs, and (iii) employs difficulty-aware mixture-of-experts (MoE) routing. Experiments on eight datasets show consistent improvements in both average and final accuracy over strong CLIP-based baselines, with ablation studies confirming the contribution of each component.", "questions": "- How do results change with 10/5/0 exemplars per class? Could RLPO operate with pseudo-exemplars (e.g., feature replay) instead of images?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The prototype drift argument is convincing, and the analysis figures clearly show how simple averaging causes performance degradation.\n\n- Framing prototype construction as a reinforcement learning weighting problem, where rewards promote intra-class cohesion and inter-class separation, offers an interesting new perspective. The use of KL regularization toward a reference policy provides reasonable stability.\n\n- The study demonstrates that visual prompts are important in continual learning and that dual-modal prompting outperforms text-only and visual-only approaches.", "weaknesses": "- RLPO introduces a policy network and reward normalization, while MoE adds routing and a deeper hard path. However, the paper does not provide a clear comparison of training time, inference latency, or FLOPs and parameter counts against the baselines under the same hardware and memory conditions, which is crucial for evaluating methods in continual learning settings.\n- The paper focuses on class-incremental classification, However, it remains unclear how RLPO and MoE would perform in task-agnostic or open-world scenarios involving unknown classes, or multimodal image–text continual learning.\n- The results depend on exemplar buffers of 20 samples per class. Please evaluate performance under smaller memory budgets or exemplar-free settings (e.g., with synthetic replay) to demonstrate the robustness of RLPO.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hZQOzUIgMv", "reviewer": "Reviewer_vX6H", "summary": "The paper proposes a novel reinforcement learning based framework, RLAP-CLIP, for improving vision-language models on continual image classification tasks. They introduce three different components 1) RLPO to mitigate prototype degradation in continual classification tasks, difficulty-aware cross-modal fusion for better cross-modality integration for training, and enhanced dual-modal prompting to resolve modality imbalance. The authors present results on a variety of image classification tasks to support their claims and also provide theoretical guarantees to strengthen them.", "questions": "Please refer to the questions raised above in the weaknesses.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper is well-written and easy to follow, with each design choice for RLAP-CLIP clearly explained.\n2. The authors show a comparison with a variety of past approaches that strengthen their work.\n3. I also like that they clearly introduced the problem first by showing how forgetting increases in vision-language models as tasks increase and conventional averaging-based approaches are not ideal to resolve this.", "weaknesses": "1. The effect of dual modal prompting in forming better compact clusters for each class is difficult to see. Can you provide some quantitative measure of how effective the correct cluster formation is?\n2. The idea of normalized advantages and comparing between intra-class and inter-class seems quite similar to GRPO [1] reward optimization. What is the novelty in RLPO, and how does it compare with this RL fine-tuning approach?\n3. How do other dual prompting compare with other parameter-efficient finetuning approaches like LoRA, Prefix tuning?\n4. Also, the proposed framework is limited to classification tasks, whereas other methods like C-CLIP work on a variety of different tasks, like retrieval and captioning. This limits the generalizability of RLAP-CLIP.\n5. Please add some qualitative examples from the datasets used for benchmarking that help understand the advantage of RLAP-CLIP better.\n\n[1] DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bg5qaqOAu0", "reviewer": "Reviewer_LqUL", "summary": "This paper proposes RLAP-CLIP, a framework for continual multimodal learning with vision-language models (e.g., CLIP). aiming to alleviate prototype quality degradation and asymmetric adaptation in sequential task learning. The method introduces (1) RLPO, a reinforcement learning-based approach to actively optimize prototype construction for better class separability; (2) a difficulty-aware, mixture-of-experts mechanism for dynamic sample routing; and (3) dual-modal prompting to balance textual and visual adaptation. Experiments on eight datasets across diverse domains and tasks demonstrate RLAP-CLIP outperforms strong baselines, especially in fine-grained and out-of-distribution settings.", "questions": "1. Can the authors clarify policy network architecture details for RLPO (exact layer sizes, normalization)? How does the policy adapt when exemplar set sizes are very small, or as class counts grow?\n\n2. Are there task types (e.g., language-driven tasks) where visual or dual-modal prompting hurts? Figure 2 and Table 2 suggest continuous improvement, but are there more fine-grained trade-offs?\n\n3. Could you contextualize the paper against more recent, related works that were not discussed, such as [1] which introduces a mixture-of-experts network for improved sample efficiency in visual RL, and [2], which proposes strategies for continual learning also using RL?\n\nReferences:\n\n[1] Huang, S., Zhang, Z., Liang, T., \"MENTOR: Mixture-of-Experts Network with Task-Oriented Perturbation for Visual Reinforcement Learning\" (2025)\n\n[2] Liu, Z., Fu, G., Du, C., \"Continual Reinforcement Learning by Planning with Online World Models\" (2025).", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Strong Motivation and Illustration: Section 2 gives a nice layout of the prototype quality degradation issue and comparison of different prompting strategies in class separation, effectively highlighting the failure cases of existing continual learning methods for VLMs. \n\n2. Clear Presentation and Novel Designs: Section 3 gives a thorough introduction of different components of the RLAP-CLIP framework. Particularly, the RLPO module transforms prototype averaging into a reinforcement learning problem, with well-articulated mathematical objectives and proof. \n\n3. Comprehensive Experiments and Solid Results: RLAP-CLIP is benchmarked on a wide range of datasets, showing consistent improvement over state-of-the-art baselines. Additionally, a stepwise ablation is provided, attributing improvements to each module Hyperparameter sensitivity analysis is also conducted to verify the robustness of the framework.", "weaknesses": "1. Lack of Details on Prototype Policy: The paper provides some equations but is somewhat vague regarding the policy network architecture itself, such as policy hyperparameters and normalization details.\n\n2. Limited Analysis of Scalability: While RLPO's theoretical soundness is established, discussion of potential computational bottlenecks (especially for large-scale, real-world continual learning) is lacking. For example, how does RLPO's policy network scale for hundreds/thousands of classes and when data distributions heavily shift?\n\n3. Limited Task Scenarios: The experimental setup follows class-incremental protocols with a fixed number of exemplars per class, but there is minimal exploration of memory constraints or more severe task shift scenarios (e.g., open-world settings). Additionally, the paper could benefit from presenting qualitative failure cases or edge conditions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1dKV3TlTRW", "reviewer": "Reviewer_5YDU", "summary": "The paper proposes a reinforcement learning framework for continual image classification. The method, RLAP-CLIP, uses Reinforcement Learning–based Prototype Optimization (RLPO) to actively refine class representations and mitigate prototype (representation of each class or a embedding of class) degradation. It incorporates difficulty-aware cross-modal fusion to route samples by complexity and enhanced dual-modal prompting to balance visual and textual adaptation. Experiments across multiple image classification benchmarks show RLAP-CLIP outperforms prior methods.", "questions": "- What is center-based example selection in the Figure 1. This part is not explained in the paper.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The reinforcement learning–based prototype update method empirically outperforms prior work. Enhanced dual-modal prompting and difficulty-aware sample handling further contribute to the gains, as demonstrated in experiments.\n\n- The paper provides empirical and theoretical evidence of training stability—covering hyperparameter sensitivity and convergence—which is critical for reinforcement learning frameworks.", "weaknesses": "- The method improves continual image classification by combining established components—prototype updates, reinforcement learning, learnable prompts, and routing. While the gains on benchmarks are clear, the pipeline is heuristic and tailored only for classification, making its generalization to other continual recognition tasks (e.g., retrieval, detection...) uncertain.\n\n- Key terminology should be clearly defined (e.g., “prototype,” “center-based exemplar selection”) to improve readability and understanding. Only objectives and architectures are only explained in the paper. Training and inference process are not explained. Please more clarify what are frozen or learnable weights in the framework.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)