CoCoA-Mix: Confusion-and-Confidence-Aware Mixture Model for Context Optimization
Authors: Dasol Hong
Wooju Lee
Hyun Myung
Venue: ICML 2025 poster (2025)
Keywords: Context optimization
Prompt tuning
Vision-language models
Primary area: applications->computer_vision
TL;DR: CoCoA-Mix enhances specialization and generalization in prompt tuning using CoA-loss for refined decision boundaries and CoA-temp for confidence-based scaling.

Abstract: Prompt tuning, which adapts vision-language models by freezing model parameters and opti- mizing only the prompt, has proven effective for task-specific adaptations. The core challenge in prompt tuning is improving specialization for a specific task and generalization for unseen domains. However, frozen encoders often produce misaligned features, leading to confusion between classes and limiting specialization. To overcome this issue, we propose a confusion-aware loss (CoA-loss) that improves specialization by refining the decision boundaries between confusing classes. Additionally, we mathematically demonstrate that a mixture model can enhance generalization without compromising specialization. This is achieved using confidence-aware weights (CoA- weights), which adjust the weights of each prediction in the mixture model based on its confidence within the class domains. Extensive experiments show that CoCoA-Mix, a mixture model with CoA-loss and CoA-weights, outperforms state-of-the-art methods by enhancing specialization and generalization. Our code is publicly available at https://github.com/url-kaist/CoCoA-Mix

CoCoA-Mix: Confusiorand-CCnffdeeee--www
Mixture Model for Context Optimization
Dasol Hong'
Wooju Lee
Hyun Myung'
Abstract
??
4
prob.
A
prob.
Prompt tuning, which adapts laioon-language models by freezing model parameters and opti- mizing only the prompt, has proven effective for task-specific adaptations. The core challenge in prompt tuning is improving specialization for a specific task and generalization for unseen do- mains. However, frozen encoders often produce misaligned features, leading to confusion between classes and limiting specialization. To overcome this issue, propose confusion-aware loss (CoA-loss) that improves specialization by refin- ing the decision boundaries between confusing classes. Additionally, we mathematically demon- strate that a mixture model can enhance general- ization without compromising specialization. This is achieved using confidence-awald weights (CoA- weights), which adjust the weights of each predic- tion in the mixture model based its confidence within the class domains. Extensive experiments show that CoCoA- -Mxx, mixture model with CoA-loss and CoA-weights, outperforms state- of-the-art art methods by enhancing specialization and generalization. Our code publicly available at http:://gituub..o///lkkkiiiic/ce
C
OA
confusing
clear
aa)
seen unseen
spocialized prediction generalized prediction final prodiction
seen unseen
aaC
domain
domain
trade-off
win-win
(b)
Figure 1: (a) Effect of CoA-loss. The left and right sides show probabilities for the hatched sample on the decision boundary. In the middle figure, dashed and solid lines indi- cate the current and updated decision boundaries. CoA-loss assigns higher weights to this confusing sample, enhancing specialization in prompt tuning. (b) Effect of CoA-weights. The left and right sides show performance across domains. CoA-weights increase confidence in the specialized predic- tion for seen domains while reducing for unseen domains, preserving both specialization and generalization.
on large-scale datasets, enabling remarkable zero-shot ca- pabilities. However, their reliance on generic embeddings often limits their effectiveness in task-specific applications.
1. Introduction
Pre-trained vision-language models (VLMs) have achieved remarkable results in diverse downstream tasks, such as im- age classification (Fu et al., 2022), object detection (Zhong et al., 2022; Gu et al., 2021), and visual question answer- ing (Cho et al., 2021; Lin et al., 2022). These models align vi- sual and textual embeddings through extensive pre-training
To mitigate this limitation, prompt engineering has emerged practical way to enhance performance on specific task (Wang et al., 2021; 2022a). The method involves manually designing a task-specific text template for the model input. For example, in image classification, hand- crafted prompts such as "a photo of a [CLASS), type flower" help associate visual embeddings with the correct class labels, such as "hibiscus" or "sword lily." (Zhou et al., 2022c) Although effective, the manual process is labor-intensive and requires domain expertise, limiting scalability across diverse tasks.
Urban Robotics Lab, School of Electrical Engineer- ing. Korea Advanced Institute of Science and Technol- ogy, Republic of Korea. Correspondence lo: Dasol Hong wd..hon@@kaist.ac..oo
Wooju Lee <dnwn24@ kaist Hyun Myung <hmyung kr.>..
Proceedings of the nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
Prompt tuning has become a scalable alternative to manual prompt engineering by replacing hand-crafted prompts with

---

CoCoA-Mix: Coafussin-andd-Coffddnn---- Mixture Model for Context Optimization
learnable prompts (Zhou et al., 2022c; Zhu al., 2023; Zhou et al., 2024; Zhang et al., 2024a). This method allows VLMs to effectively align textual embeddings with the cor- responding visual embeddings by optimizing the prompts for a specific task while freezing the model parameters.
2. Related Work
2.1. Textual Prompt Tuning
Prompt tuning has emerged as a method to reduce the re- liance on human expertise while improving the performance of VLMs on specific tasks. The method can be categorized into visual prompt tuning (Jia al., 2022; Bahng al., 2022), textual prompt tuning (Zhou et al., 2022c;b; Zhu et al., 2023; Yao et al., 2023; Zhang et al., 2024b), and visual-textual prompt tuning (Khattak et al., 2023a;b). In this paper, we focus on textual prompt tuning, which re- places hand-crafted prompts with learnable prompts and optimizes them using few-shot training data, while keeping the model parameters frozen.
However, the existing methods have two limitations. First, existing methods do not explicitly address confusing cases arising from the frozen visual encoder. The frozen visual en- coder often fails to capture task-specific features, struggling to distinguish between different classes. However, most ex- isting methods rely on standard cross-entropy loss, which is ineffective in handling confusing cases and consequently limits their specialization (Zhou et al., 2022c;b). Second, achieving generalization to unseen domains is crucial in prompt tuning, but current methods often sacrifice special- izatiin to improve generalization. Most studies inherently assume specialization and generalization are competing ob- jectives. As a result, the trade-off problem remains an open challenge.
2.2. Loss for Prompt Tuning
In textual prompt tuning, the frozen encoders map inputs to generic emheddings, which can lead to misaligned vision- text embeddings for specific tasks. A common strategy to mitigate this issue is to employ a standard cross-entropy loss that aligns textual prompts with their corresponding visual embeddings. CoOp (Zhou et al., 2022c) pioneered textual prompt tuning by optimizing prompts for specific tasks us- ing cross-entropy loss. Subsequent works employed regu- larization via hand-crafted prompts, which enhanced spe- cialization by preventing prompts from learning unintended patterns (Zhu et al., 2023; Yao et al., 2023; Zhang et al., 2024b). However, such regularization methods may prevent prompts from fully capturing task task-specific patterns, limiting specialization in complex tasks. Other approaches (Khat- tak et al., 2023a; Zhang et al., 2024b) introduce additional network parameters to enhance the representational capac- ity of the model. MaPLe (Khattak et al., 2023a) leverages interactions via coupling function, while DePT (Zhang et al., 2024b) employs dual- architec- ture to decouple task-specific and task-shared knowledge into separate feature spaces. Although these methods im- prove generalization, they increase the number of learn- able parameters, leading to overfitting when training data is scarce and limiting scalability. In contrast, we aim to enhance both specialization and generalization without in- troducing additional network components.
To overcome these limitations, we propose a confusion- and-confidence-awaal aware mixture model (CoCoA-Mix)) which combines confusion-aware loss (CoA-loss) and confidence- aware weights (CoA-weights), We first introduce mixture model that combines predictions from individual prompts, providing a theoretical framework to analyze prompt tun- ing in terms of specialization and generalization. Building on this theoretical insight, CoA-loss improves specializa- tion by applying larger gradient to confusing cases, refining decision boundaries as shown in Figure 1(a). Then, CoA- weights achieve generalization without compromising spe- cialization by scaling mixture model weights based on the confidence of individual prompts across class domains, as shown in Figure ((b). As a result, our method enhances both specialization and generalization by ensuring that the error of the mixture model remains lower than the minimum error of individual prompts. Main contributions are as follows:
We provide a mathematical framework demonstrating that specialization and generalization can be improved simultaneously.
We propose a CoCoA-Mix framework, consisting of CoA-loss and CoA-wwights. CoA-loss boosts special- ization by improving classification for confusing cases, while weights improve generalization by adjust- ing the weights of individual prompts in the mixture model based on their confidence over class domains.
2.3. Mixture of Prompts
Prompt ensembling has been studied to improve general- ization in VLMs. Allingham et al. (2023) propose zero- shot prompt ensembling (ZPE), which automatically as- signs weights to hand-crafted prompts from a large pool, leading to improved zero-shot accuracy. Lu et al. (2024) extend this to model-level ensembles suited to varying re- source settings, but this requires multiple forward passes inference time. Despite their generalization benefits, neither
The proposed method achieves average harmonic mean improvements of 15.28% and 3.28% over zero-shot CLIP in base-to-new generalization and cross-datase transfer, respectively; it also improves the average accu- racy in shot incremenaal learning by 5.6%p.
2

---

CoCoA-Mix: Confasionaaud.Cootdeeee---wa Mixture Model for Context Optimization
approach explicitly addresses the challenge of achieving task-specific specialization. In contrast, our method jointly improves specialization and generalization, while remain- ing computationally efficient by avoiding multiple forward passes.
Lemma 3.3. Let the class set of the target domain Dr be partitioned into K disjoint subsets, with correspond- ing sub-domains DTu: DT...... .Twx such that Dr _ Dr. Then, the expected error of p7 given by:
ET ET(17)
Σ A (fT),
3. Proposed Method
3.1. Preliminary
where X -i(x..)--yy [(x,]) Dr. denotes the prob- ability that a sample from the target domain DDr belongs to the sub-domain Dr, satisfying Σi= X = 1. Based on Theorem 3.2, the error of the mixture model in the arbitrary target domain can be upper-bounder as follows:
In downstream tasks, CLIP employs visual and textual coders to map images and textual prompts into shared embedding space. The textual prompts can be generated as hand-crafted prompts by embedding class labels into prede- fined templates, such as "a photo of a [CLASS]. With t, the probability of the image belonging to the class Pe (I), defined as follows:
(1) Evey exp ((T(T))T)
exp (8+(T)/T)
(1)
cT(p7)
S
Σ
X
TT cr.
cr..
T cn (Pey)
specialization
1=
where is the temperature scale, y represents the of classes, and (1) denotes the cosine similarity between the visual and textual embeddings of the class generated by t.
generaiisation
(6)
where x.t denotes the mixing weight of the prompt t, for its own domain DT., and n denotes the mixing weight of the prompi tا + i) when applied to the domain D;. Here, TT Σ 1= OTTT 1.
3.2. Decomposing Specialization and Generalization
The expected enror (p) of predictive distribution ý in arbitrary target domain Dr is defined using the Kullback- Leibler (KL) divergence as follows:
Let y denote the set of all classes in the arbitrary tar- get domain Dr, which is partitioned as y = Vi. Specifically, we assume that labeled data is provided for 31 .......V.. whereas no supervision is available for Vo. Due to the absence of labeled data in DDn- the associ- ated prompt to cannot be specialized through training. In- stead, we utilize a generalized hand-crafted prompt such as a photo of [CLASS]
(p) (xx log p(y)),
where y is the ground-truth label for the image
We introduce a mixture model to derive an upper bound for the error in the target domain, providing framework for analyzing specialization and generalization.
Definition 3.1 (Mixture model). Let K + different prompts be given by T = {to, t..... ,tx), and let π = {Tu, ".,... k } denote set of non-negative weights sat- isfying Σ = 1. The mixture model p7 is defined as a weighted combination of the individual prompts:
Building this, we detail the proposed method for special- ization and generalization in prompt tuning. In Section 3.3, we propose confusion -ware loss (CoA-loss), which focuses on confusing cases in prompt tuning and effectively reduces the specialization error eT; (fx,) for each sub-domain Dr:- Subsequently, Section 3.4 enhances generalization by opti- mizing confidence-awee the confidence of each prompt the given class domain, tightening the upper bound of er ). As result, the pro- posed method enhances specialization and generalization simultaneously. Figure 2 shows the overall framework of the confusion-and-- wwnd-ooniddnce--nnee aware mixture model (CoCoA- Mix), which combines CoA-loss and CoA-weights.
cxp EER S1 (()((
weights (CoA-weights)) based on
(1) Evey exp
k m,Bt, (1)/T)
Theorem 3.2. The expected error of the mixture model p7 can be bounded as follows:
Σ mer((r.)
(4)
3.3. Confusion-Aware Loss for Specialization
1=0
Let P be a predictive distribution and Ds be the source domain. According Nguyen et al. (2022), the expected
The proof provided in Section
3

---

CoCoA-Mix: Confusionaaud.CCtiddnnrr-www Mixture Model for Context Optimization
Visual Encodder
Loss for learnable prompt 4;
image X
For in classs
prompl (x, Vr Pt.)
Cai
Bind
2
for out-class
Loss for in class weights "
in-class
cui-cliss
CCE(X, y/ y:
Ma
[class]
Pt.
Text Encoder
Loss for out-class weights m)"
learnahle prompe f; class tokens
AA photo of")
[class].
to Pto
Lun (x: Pt. Pto
prompt by class
ittcclass
Figure 2: CoCoA-Mix framework integrates confusion-aware loss (CoA-loss) for specialization and confidence-awarc weights (CoA-weights for generalization, ensuring performance improvements without trade-offs. The learnable prompt 4; is optimized with CoA-loss to specialize in distinguishing confusing classes within the training domain. CoA-weights adjusts prediction confidence by increasing Th for in-class and decreasing क:"h! for out-class. At inference, the specialized predictions .r,, adjusted via CoA-weights, are combined with the generalized predictions Pto to ensure generalization while preserving specialization.
error (p) on the target domain Dy is bounded as follows:
C.0
1/-
C (L(pr (z) +8, V2
cr( S cs())++++++++++>)))}>
where is defined as a visual embedding; C con- stant that bounds log l(), ensuring each class probabil- ity is least exp(-C); pr and ps are the marginal distribution of z for the target and source domains, spectively; and 8 denotes the conditional misalignment (x)(x) [KL (tsli/)ssssssss))) (z/x)))) which is typically small.
1//
0.5
0.0
0.0
0.5
1.0
0.3
0.5
P+(y)
Pe #
(a)
(b)
Figure 3: Gradient component of Cprompt with respect to (a)
We focus on textual prompt tuning, which cannot optimize the visual embedding due to the frozen encoders. Thus, Equation (7) suggests that minimizing (p) can reduce (p),, enabling error minimization in the mixture model via source domain specialization.
st (c + y) and (b) 8x (v)), where w = represents standard cross-entropy.
where w a hyperparameter that balances the contribution of CoA-loss.
Most existing methods use standard cross-entropy, defined as follows, for the specialization in prompt tuning:
To illustrate how CoA-loss refines the decision boundary between confusing classes, we analyze the gradients for the correct class and an incorrect class c. The gradients of fpompp with respect to the similarities se(y) and se(c + y) are as follows:
CCE(x, yi Dt) = log (w))
Cross-entropy assigns gradients based on class probabilities, assigning larger gradients to misclassified samples. While effective for misclassified cases, it relies only on individual probabilities and does not consider inter-class relationships, limiting its ability to handle class confusion. Addressing confusing classes is crucial in prompt tuning with limited training data because they significantly impact the deci- sion boundary. To overcome this limitation, we propose -wware
= - (x(x)) (1 - wV)())),
1
Os+(y)
(11)
1 (1) wpx Ds (c Fy)
OC 3hpmmpt
Figure shows the gradients for classes y and c. When w O, the loss corresponds standard cross-entropy, As implied by Equation (11), standard cross-entropp assigns gradients based on individual probabilities, ignoring inter- class relationships. In contrast, CoA-loss increases the gra- dient for when (x(y) approaches 0.5 and for class e when pe(c) nears pe Pe()), especially as w increases. These scenarios (10) represent confusing cases where the correct class probabil-
confusion- aware loss (CoA-loss), defined as follows:
Leor (x, yr Px) = 11- (v)-
(9)
The overall loss Lppumps for optimizing the prompt t is given by:
4ppampi (x, y: Po) = Ler + wLCors

---

CoCoA-Mix: Confusion-and.Cootdeeee----- Mixture Model for Context Optimization
ity is uncertain where the incorrect class probability is similar the correct class, making them difficult distin- guish. CoA-loss thus induces larger gradient updates for these confusing cases.
generated by combining random strings or retrieving ran- dom words. According Assumption 3.4, the temperature TH for out-class domains is optimized using entropy loss LEat which compares the entropy of Pr. and Pt. as follows:
3.4. Confidence-Aware Weights for Generalization without Trade-Off's
T nut arg min (xx,))-Ds, [ (x: Pr,, Pto)l:
(14)
Equation (6) suggests that optimizing can minimize the upper bound of the expected error (r(}T) of the mixture model. Achieving this requires assigning a higher weight to the prediction with lower error in the target domain. In this section, we propose confidence-aware weights (CoA- weights), which adjust the weight of each prediction based on the class domain.
CEm = max (0. H (Pto) - (ft.) + d).
(15)
where d is a margin and H (p) is the normalized entropy of i over the out-class set, i.e., ( = Σ ye -((c) log p(c)/ log (
The entropy loss LEnt ensures that the entropy of Pe.so ex- ceeds that of Pt. by a margin of d. Entropy measures un- certainty in the prediction, with higher values indicating lower confidence. As a result, x. is optimized to make specialized predictions less confident than generalized ones.
Assumption 3.4. The specialized prediction Pe. for Dr, satisfies the following relationships:
Cr, (Pe.) S er, (Peegar and (ijee(er S Cryai (Pr.).
(12) The first inequality reflects that a prediction Pe. optimized for a specific domain D7, always performs better than pre- dictions pe made by prompts optimized for other domains. Conversely, the second inequality assumes that the gener- alized prediction Pt. is more effective for unseen classes. This assumption aligns with the principle that a single pre- diction specialized for a task cannot excel at all tasks, a claim further supported by statistical evidence presented in Appendix B.
4. Experiments
4.1. Datasets and Implementation Details
We validate the effectiveness of our method in three tasks: (1) base-to-new generalization, (2) few-shot class- incremental learning (FSCIL), and (3) cross-dataset transfer.
Datasets We evaluate base-to-new generalization and cross-dataset transfer performance using 11 datasets: Ima- geNet (Deng et al., 2009), Caltech101 (Fei-Fei et al., 2004), OxfordPets (Parkhi et al., 2012), StanfordCars (Krause et al., 2013), Flowers 102 (Nilsback & Zisserman, 2008), Food101 (Bossard et al., 2014), FGVCAircraft (Maji et al., 2013), EuroSAT (Helber et al., 2019), UCF101 (Soomro,
Equation (6) and Assumption 3.4 suggest that minimizing the upper bound of cr requires increasing nn for in- classes and decreasing π out for out-classes. To achieve this, we propose CoA- CoA-weights which adjusts the weights nrth and n; of Pt. in the mixture model.
2012), DTD (Cimpoi al., 2014), and SUN397 (Xiao et al.,
2010). For FSCIL,, we use CIFARI00 (Krizhevsky et al.,
Optimizing n.th for In-Class Domains The weight z/n" for in-class domains is optimized by minimizing the cross- entropy loss of the mixture model over the training domain Ds.:
2009). Following Tao et al. (2020), we split the classes into
60 Base and 40 New classes and adopted 5-shot 5 5-way setting, resulting in a total of 9 training sessions.
Training Details The prompt length M is initialized ran- domly and set to 16 unless specified. The out-class set y for optimizing न.. is generated by sampling the same num- ber of random words as the in-class set 3's using the API (Re- bguns, 202 Prompt tuming is performed using the Adam optimizer (Kingma, 2014) with a learning rate of 0.002. Op- timization for the CoA-weights is conducted with SGD. Fur- ther details of implementation are provided in Appendix D.
TT = arg min E(xa))Dss (x (13)
With this optimization, weight .!!' is increased when the spe- cialized prediction fr, outperforms the generalized predic-
tion Pegy: for the in-class domain and decreased otherwise. Consequently, the weight Λ."" for in-classes is optimized. Further details on the cross-entropy effect in the mixture model are provided in Appendix C.
4.2. Performance Comparison
Base-to-New Generalization We evaluate prompt tuning performance over classes in 4-shot setting. Each dataset is evenly split into two disjoint subsets: base classes (Base) for tuning and unseen new classes (New). Accuracy measured
Optimizing कद for Out-Class Domains Because only in-classes are available during training, out-class set y must be generated optimize лशप.. The out-class set can be
5

---

CoCoA-Mix: Contusionant--Cottdeen----- Mixture Model for Context Optimization
Table I: Performance comparison on datasets in the base-to-new benchmark. H represents the harmonic mean.
AVERAGE
IMAGENET
CALTECH1O1
METHOD
BASE
NEW
BASE
NEW
BASE
NEW
H
CLIP
65.14
68.78
66.82
64.43
60.04
62.16
90.64
91.16
90.90
COOP
77.23
68.56
71.33
73.72 + 0.29
64.94
0.87
69.05
97.16 0.16
93.92 - 0.80
95.51
PROGRAD
78.74
72.19
75.06
74.81 + 0.29
66.68
0.26
70.51
97.50 + 0.08
95.49
96.48
КGCOOP
78.67
74.62
76.38
75.44 + 0.08
09.43 0.29
72.31
97.61 0.33
91.00
96.18
MAPLE
77.14
74.69
75.40 1: 0.29
7.12
72.83
97.47 0.31
93.77 1.11
95.57
DEPT
79.20
66.36
71.78
73.50 1: 0.22
70.00 = 0.16
71.71
97.83 1: 0.05
95.83 0.25
96.82
COA-LOSS
79.12
73.66
76.15
75.68
= 0.00
67. 0.31
71.62
97 97.94
1 0.14
94.54 = 0.24
96.21
CoCoA-MIX
79.31
75.10
77.03
75.47 1 0.09
68.92 0.10
72. 04
98.02 + 0.03
94.39 + 0.10
96.17
OXPORDPETS
STANFORDCARS
FLOWERS 102
METHOD
BASE
NEW
BASE
BASE
NEW
H
CLIP
90.01
94.24
92.07
55.37
66.65
60.49
69.23
73.90
71.49
CoOp
94.10 0.73
94.42
4.17
94.16
69.54
0.75
71.39
70.44
90.60
67.00
1.04
77.01
PROGRAD
95.00 + 0.31
97.36
0.42
0.42
96.16
71.45 + 0.39
73.16 + 0.58
72.29
91.36 + 0.63
74.92 + 0.90
82.32
KGCOOP
94.65 + 0.15
97.59 +
96.10
68.64 + 0.35
74.96 + 0.53
71.66
90.09 + 0.63
76.31 + 0
82.63
MAPLE
94.80
97.67
96.21
67.97 0.29
74.40
71.04
88.03 1.62
73.43 0.499
80.06
DEPT
94.00 + 0.29
88.63 0.78
91.23
71.83 + 0.52
59.27 = 0.76
64.94
94.53 0.53
66.30 + 1.42
77.92
COA-LOSS
94.90 = 0.49
97.93 + 0.08
96.39
72.70 + 0.11
73.07 1.27
72.87
88 15.89 + 1.75
75.58 = 1.31
81.67
CoCoA-MIX
95.16 0.38
97.60 + 0.09
96.36
73.09 + 0.25
74.97 + 0.08
74.01
91.04 + 1.79
77.37 + 0.38
83.64
FOOD101
FoVCAIRCRAFT
SUN397
METHOD
BASE
NEW
BASE
NEW
H
BASE
NEW
H
CLIP
83.58
$4.95
84.26
19.51
24.60
21.76
66.76
70.52
68.59
CoOp
89.19 0.19
88.45 + 0.89
88.81
26.17 + 7.89
19.50 11.94
11.46
77.37
0.66
72.06 + 1.56
74.60
PROGRAD
89.33 + 0.08
89.93 + 0.58
89.63
34.21 + 1.99
28.53
= 2.08
30.97
79.16
0.36
74.34
76.20
KGCOOP
90.26 0.11
91.25 0.15
90.75
33.43 + 0.56
32.27=1.11
32.81
79.07 0.24
76.78
77.91
MAPLE
89.37 0.54
0.54
31.67
0.66
33.13
32.29
78.33 0.21
0.45
78.00
DEPT
89.80 + 0.08
88.10 1
88.94
35.93 1 0.93
24.33 0.09
29.01
79.10 0.22
67
1 0.46
72.70
COA-LOSS
90.11 + 0.18
90.87
+ 0.42
90.49
33.91 1 0.68
32.47
0.37
33.17
78. 70
1 0.25
75.43 1 0.72
77.03
CoCoA-MIX
90.09 0.16
90.93
0.09
90.50
33.51 1 0.28
34.15 0.14
33.83
78.51 1 0.17
76.60 + 0.24
77.54
DTD
EUROSAT
UCF101
METHOD
BASE
NEW
BASE
BASE
NEW
H
CLIP
53.24
54.71
53.97
54.79
66.21
59.96
69.03
69.61
69.32
CoOp
71.22 1.13
53.62
3.45
61.03
79.93 + 1.07
64.79
71.19
80.58
0.66
: 64.11
71.32
PROGRAD
72.07
0.29
50.56 + 2.43
59.35
81.29 + 3.36
69.81 + 5.56
74.80
80.97 + 0.29
73.32 + 1.85
76.93
KGCoOp
72.92 1.05
59.14 1.14++10 1.53
65.28
83.20 + 0.72
70.51 9.30
75.61
80.09 0.24
77.75 0.40
78.90
MAPLE
70.40 2.57
58.40 1..0++3.00 3.00
63.71
76.50 3.85
55.70 3.19
64.27
78.57 +..77+2111
1.56 76.60
77.53
DEPT
0.88
53.13
1.07
61.98
78.70 + 1.56
50.53 + 5.71
61.08
81.57 0.84
66.53 0.87
73.28
COA-LOSS
73.23 + 2.02
58.09 + 0.81
64.76
83.38 + 0.49
70.07 = 2.49
76.09
80.83 4 0.80
74.22 - 0.91
77.38
CoCoA-Mix
72.80 1.89
64.29 1.25
68.25
83.49 + 0.66
3.10 69.11
75.54
81 28 + 0.95
77.75 + 0.24
79.47
independently on Base and New, and their harmonic mean (Xian et al., 2017) is calculated to evaluate the trade- off between them. The final performance is reported as the average of three random seeds for fair evaluation.
2023a) and DePT (Zhang et al., 2024b) focus on improving performance on both Base and New by increasing model capacity. However, with only 4-shot training samples, they often overfit, limiting generalization. Our CoA-loss, when combined with naive ensemble, improves Base perfor- mance but offers limited generalization as it only focuses specialization. By incorporating CoA-weights, our CoCoA- Mix achieves the highest average performance on both Base and New without trade-offs. Notably, CoCoA-Mix surpasses the specialization performance of DePT while using 2.8% of its parameters.
Table shows that CoOp (Zhou et al., 2022c) achieves higher average performance than CLIP Base but lower average performance New, highlighting the need for generalization in prompt tuning. ProGrad (Zhu et al., 2023) and KgCoOp (Yao et al., 2023), which incorporate hand-crafted prompts during training, improve average per- formance both Base and New. MaPLe (Khattak al.,
6

---

CoCoA-Mix:
Contusionaan--Coondeee----wa Mixture Model for Context Optimization
Table 2:
Performance comparison on CIFARI00 in the FSCIL benchmark. Mean represents the average accuracy across all sessions, and PD indicates the performance difference between the first and last sessions.
ACC(%)
METHOD
MEANT
PDJ
3
4
5
L2P
89.9
86.0
81.8
80.3
80.0
74.6
73.2
72.6
65.0
78.2
24.9
CLIP-ZSL
77.9
COOP-FSCIL
88.6
78.9
77.5
76.0
76.8
78.3
79.2
79.8
79.3
79.4
9.3
FACT W/ CLIP
87.8
84.0
81.4
78.0
77.8
76.3
75.0
72.5
71.9
78.3
15.9
FSPT-FSCIL
86.9
83.1
81.9
80.7
80.4
79.9
80.1
79.9
79.4
81.4
7.5
CoCoA MIX (OURS)
88.2
85.6
84.6
82.7
82.8
82.5
82.3
81.8
80.8
83.5
7.4
Table 3: Performance comparison in cross cross-datased transfer
Base sec.
78.65
78.75
78.78
79.
77.52
METHOD
SOURCE
TARGET
FL
GČE
CE
CE+MAE CE+CoA
CLIP
66.73
64.89
63.97
COOP
69.06 + 0.43
59.88
61.52
Figure 4: Comparison of Base performance by loss function
PROGRAD
70.21 + 0.16
62.36
63.58
КGCOOP
70.52 + 0.05
64.45
65.17
MAPLE
69.53 + 0.39
65.24
65.26
DEPT
68.03 + 0.09
65.06
64.42
reduces target accuracy, highlighting the need for gener- alization. ProGrad and KgCoOp similarly enhance source accuracy but fail to outperform zero-shot CLIP the target dataset, indicating limited transferability of prompt knowl- edge. MaPLe and DePT improve transfer performance by introducing additional parameters but show limited source accuracy. CoCoA-Mix achieves the highest performance on both source and target datasets, demonstrating effec- tive knowledge transfer across different datasets. Notably, CoCoA-Mix outperforms existing methods in generaliza- tion while utilizing only 0.26% of the parameters used by MaPLe. Appendix E. provides detailed results on 10 datasets.
COCOA-MIX
70.85 0.09
65.27
66.07
Few-Shot Class-Incremental Learning (FSCIL) We evaluated our method on FSCIL tasks to verify its effec- tiveness in incremental learning with limited data. In FSCIL, the number of prompts K was increased incrementally, with each prompt specializing in its session. For a fair compar- ison, CoCoA-Mix uses prompts with M = per session. has fewer parameters than methods with M 16 un- til session 6 but requires
parameters from session 7. Table shows the performance of FSCIL the CI- FAR100 dataset. L2P (Wang al., 2022b), which dynami- cally selects prompts from pool, shows high initial perfor- mance but suffers significant knowledge forgetting as new classes are added, leading to the highest performance dif- ference (PD). CoOp-FSCII. (Zhou al., 2022c) and FACT w/ CLIP (Zhou al., 2022a) outperform zero-shot CLIP in early sessions but affected by knowledge forgetting in later sessions. FSPT-FSCIL (Ran al., 2024) outperforms zero-shot CLIP in all sessions by leveraging a brain-inspired strategy in prompt tuning. CoCoA-Mix performs lower in the first two sessions due to fewer parameters but achieves the highest performance in later sessions, outperforming the state-of-the-arr methods. Details of the FSCIL implementa- tion provided in Appendix D.4.
4.3. Effectiveness of Our Method
Performance Comparison with Existing Loss Functions We compared Base performance across various loss func- tions to demonstrate the effectiveness of CoA-loss for specialization in prompt tuning. Figure shows the aver- age Base performance for each loss function. Focal loss (FL) (Ross & Dollár, 2017) emphasizes misclassified sam- ples but struggles to learn from well-classified ones, leading to poor performance compared with CE in prompt tuning. Generalized cross entropy (GCE) (Zhang Sabuncu, 2018), which generalizes and mean absolute error (MAE), lim- its optimization on misclassified samples, restricting spe- cialization. CE achieves better performance prompt tun- ing by balancing well-classified and misclassified samples. MAE combined with CE treats all samples more equally and ensures consistent learning even in the few-shot setting, slightly improving performance. However, these loss func- tions fail to address confusing cases that significantly affect decision boundaries, limiting their performance. Adding
Cross-Datan Transfer We evaluate cross-dataset trans- fer by training on ImageNet with 1, 000 classes in 4-shot setting and testing on 10 different datasets. Table shows the accuracy and harmonic mean for source and target datasets. CoOp improves source accuracy over zero-shot CL.P but

---

CoCoA-Mix: Contision-aud.CCotdeeee----- Mixture Model for Context Optimization
100.0
57.75
hard (25.2%)
rasy ap0.
99.8-
confi teing
test abc.
57.50
ousy (37 (15)
confusiin (17.8%)
57.25
99.6
CE
CK CE
CR ChA
CE
57.00
100
0
100
training epoch
training epoch
training epoch
(a)
(b)
(c)
(d)
Figure 5: (a) Proportion of predictions by zero-shot CLIP on EuroSAT. (b) Accuracy on easy test samples correctly predicted
by zero-shot CLIP. (c) Accuracy on confusing test samples misclassified by zero-shot CLIP with probability gap below 0.2. (d) Accuracy on all test samples.
Table Comparison of performance based on the loss function used train prompts and the strategy mixing prompts. TEa refers the tuning ensemble methods (Lu et al., 2024). ViT-B/16 shows the average performance over datasets, while RN5O+RNIO1+ B reports the average over 10 datasets.
Backbone
Method
Loss
Ensemble
Base
New
H
CLIIP
65.1
68.8
66.8
CLIP (w/ Ensemble)
Uniform Ensemble
70.6
74.3
72.3
VIT-B/16
CoA-loss (w/o Ensemble)
CoA-loss
78.6
68.5
72.9
CoA-loss (w/ Ensemble)
CoA-loss
Uniform Ensemble
79.1
73.7
76.2
CoCoA-Mix
CoA-loss
CoA-weights
79.3
75.1
77.0
Tea CoCoOp
CoCoOp
Sample-Aware Weight Generator
84.1
75.5
79.2
RN50 RN101 ViT-B/16 + ViT-B/32
TEa CoA-Loss
CoA-loss
Sample-Awar Weight Generator
85.3
75.2
79.5
CoCoA-Mix
CoA-loss
MeA-weights
85.4
76.3
80.3
CoA-loss to CE achieves the best performance by effec- tively focusing on confusing cases. Detailed results and analysis are provided in Appendix E.3.
ious ensembling strategies and loss functions across both single- and multi-backbone configurations. In the ViT-B/16 setting, uniform ensembling improves generalization perfor- mance over the CLIP baseline by aggregating predictions from multiple prompts. CoCoA-Mix, which integrates CoA- loss and CoA-weights, outperforms all other variants. This highlights the importance of confidence-awand weighting. In the model-level ensemble setting, CoCoA- achieves the best performance, outperforming the tuning ensemble (Tew) with sample-aware weight generator (Lu et al., 2024). Notably, CoA-weights achieve this with only two learn- able parameters, compared with over 205,204 parameters required by the sample-aware weight generator.
Effectiveness of Confusion-Aware Loss We analyze the effect of CoA-loss on model predictions using the FuroSAT dataset. Figure 5 compares training progress with and with- out CoA-loss. Figure 5()) shows the proportion of predic- tions by zero-shot CLIP. Easy samples are correctly clas- sified, while confusing samples are misclassified with probability gap of less than 0.2 between the correct and in- correct classes. Hard samples misclassified with larger probability gap. Figure 5(b) and Figure 5(c) present per- formance on easy samples and confusing samples. While CoA-loss performs similarly to CE on easy samples, it sig- nificantly improves accuracy on confusing ones. Figure 5(d) shows that CoA-loss outperforms CE on test data by address- ing confusing cases, enhancing class decision boundaries. The performance improvement on confusing samples with CoA-loss for each dataset is provided in Appendix E.2
Conclusion
We proposed CoCoA-Mix, a framework combining CoA- loss and CoA-weights enhance specialization and general- ization prompt tuning. CoA-loss improves specialization by addressing confusing cases, while CoA-weight adjust the confidence of predictions to enhance generalization with- out sacrificing specialization. We believe our method sets new direction for effective prompt tuning.
Effectiveness of Confidence-Aware Weights We eval- uate the effectiveness of CoA-weights in the context of prompt ensembling. As shown in Table 4, we compare var-
8

---

CoCoA-Mix: Confusiin-and--Conddden---- Mixture Model for Context Optimization
Acknowledgements
Dosovitskiy, A. An image is worth 16x16 words: Trans- formers for image recognition scale. arXiv preprint arXiv:2010. 19929, 2020.
This work was supported by Korea Evaluation Institute Of Industrial Technology (KEIT) grant funded by the Korea govemmenn(MOTIE) 200.250,,455, Development of Coop- erate Mapping, Environment Recognition and Autonomous Driving Technology for Multi Mobile Robots Operating in Lcrge-scale Indoor Workspace). This research was sup- ported by the KAIST Convergence Research Institute Oper- ation Program.
Fei-Fei, L., Fergus, R., and Perona, P. Learning generative visual models from few training examples: An incremen- tal Bayesian approach tested on 101 object categories. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshop, pp. 178-178, 2004.
Fu, J., Xu, S., Liu, H., Liu, Y., Xie, N., Wang, C.-C., Liu, J., Sun, Y., and Wang, B. CMA-CLIP. Cross cross-modality atten- tion CLIP for text-image classification. In Proceedings of the IEEE International Conference on Image Processing, pp. 2846-2850, 2022.
Impact Statement
Prompt tuning has emerged as promising method for VLMs, enabling efficient adaptation to diverse tasks by op- timizing a small set of parameters while keeping the core model frozen. However, real-world scenarios with unseen classes require more generalized solutions. We proposed a mixture model-based approach enhance generality and improve performance in practical applications. We believe the outcomes of this research are largely positive, making it unnecessary to highlight specific negative impacts in this paper.
Gildenblat, J. and contributors. PyTorch library for CAM methods. sttps://githan com/ jacobgil/ pytorch-grad-on 2021.
Gu, X., Lin, T.Y.., Kuo, W., and Cui, Y. Open-vocabulary object detection via vision and language knowledge dis- tillation. arXiv preprint arXiv:2104. 13921, 2021.
Helber, P., Bischke, B., Dengel, A., and Borth, D. EurOSAT: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Se- lected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2226, 2019.
References
Allingham, J. U., Ren, J., Dusenberry, M. W., Gu, X., Cui, Y., Tran, D., Liu, J. Z., and Lakshminarayanan, B. A simple zero-shot prompt weighting technique to improve prompt ensembling in text-image models. In Interna- tional Conference on Machine Learning, pp. 547-568. PMLR, 2023.
Jensen, L. W. V. Sur les fonctions convexes et les inégalités entre les valeurs moyennes. Acta mathematica, 30(1):175-199, 1906.
Jia, M., Tang, L., Chen, B.-C.. Cardie, C., Belongie, S., Hariharan, B., and Lim, S.-N. Visual prompt tuning. In Proceedings of European Conference on Computer Vision, pp. 709-727, 2022.
Bahng, H., Jahanian, A., Sankaranarayanan, S., and Isola, P. Exploring visual prompts for adapting large-scale models. arXiv preprint arXiv:2203.172777 2022.
Bossard, L., Guillaumin, M., and Van Gool, L. Food- 101- mining discriminative components with random forests. In Proceedings of European Conference on Computer Vision, pp. 4614 2014.
Khattak, M. U., Rasheed, H., Maaz, M., Khan, S., and Khan, F. S. MaPLe: Multi-modal prompt learning. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19113-19122, 2023a.
Cho, J., Lei, J., Tan, H., and Bansal, M. Unifying vision- lnd-lung tasks via text generation. In International Conference on Machine Learning, pp. 1931-1942, 2021.
Khattak, M. U., Wasim, S. T., Naseer, M., Khan, S., Yang, M.-H., and Khan, S. Self-regulating prompts: Foun- dational model adaptation without forgetting. In Pro- ceedings of the IEEE/CVF International Conference Computer Vision, pp. 15200- 2023b.
Cimpoi, M., Maji, S., Kokkinos, I, Mohamed, S., and Vedaldi, A. Descríbing textures in the wild. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3606-3613, 2014.
Kingma, D. P. Adam: A method for stochastic optimization. arXiv preprint 1rxiv::412 69800 2014.
Krause, J., Stark, M., Deng. J., and Fei-Fei, L. 3D ob- ject representations for fine-grained categorization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshop, pp. 554 551-561, 2013.
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. ImageNet: A large-scale hierarchical image database. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 248-255, 2009.

---

CoCoA-Mix: Canfusson-and-CCotteeeee--aa Mixture Model for Context Optimization
Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Wang. H., Wang, Z., Du, M., Yang. F., Zhang, Z., Ding, S., Mardziel, P., and Hu, X. Score-CAM: Score-weighted visual explanations for convolutional neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshop, pp. 24-25, 2020.
Lin, Y., Xie, Y., Chen, D., Xu, Y., Zhu, C., and Yuan, L. Revive: Regional visual representation matters in knowledge-based visual question answering. In Proceed- ings of the Annual Conference on Neural Information Processing Systems, volume 35. pp. 11560-10571, 2022.
Wang, P., Yang, A., Men, R., Lin, J., Bai, S., Li, Z., Ma, J., Zhou, C.,, Zhou, J., and Yang. OFA: Unifying architec- tures, tasks, and modalities through simple sequence- to-sequence learning framework. In International Con- ference Machine Learning, volume 162. pp. 23318- 23340, 2022a.
Lu, Z., Bai, J., Li, X., Xiao, Z., and Wang. X. Beyond sole strength: Customized ensembles for generalized vision- language models. In International Conference on Ma- chine Learning, pp. 32924-32933 PMLR, 2024.
Wang, Z., Yu, J., Yu, A. W., Dai, Z., Tsvetkov, Y., and Cao, Y. Sim Simple visual language model pretraining with weak supervision. arXiv preprini urXiv:2108. 10904, 2021.
Maji, S., Rahtu, E., Kannala, J., Blaschko, M., and Vedaldi, A. Fine-gre Fine-grained visual classification of aireraft. arXiv preprint 130..5553 2013.
Nguyen, A. T., Tran, T., Gal, Y., Torr, P., and Baydin, A. G. KL guided domain adaptation. In Proceedings of the International Conference on Learning Representations, 2022.
Wang. Z., Zhang, Z., Lee, C.-Y., Zhang, H., Sun, R., Ren, X., Su, G., Perot, V., Dy, J., and Pfister, T. Learning to prompt for continual learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 139-149, 2022b.
Nilsback, M.-E. and Zisserman, A. Automated flower clas- sification over large number of classes. In Indian Con- ference on Computer Vision, Graphics, and Image Pro- cessing, pp. 722-729, 2008.
Xian, Y., Schiele, B., and Akata, Z. Zero-shot learning- the good, the bad and the ugly. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2017.
Parkhi, O. M., Vedaldi, A., Zisserman, A., and Jawahar, C. Cats and dogs. In Proceedings of the IEFE/CVF Conference on Computer Vision and Puttern Recognition, pp. 3498-3505, 2012.
Xiao, J., Hays, J., Ehinger, K. A., Oliva, A., and Torralba, A. SUN database: Large-scale scene recognition from abbey to zoo. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 3485- 3492, 2010.
Ran, H., Gao, X., Li, L., Li, W., Tian, S., Wang, G., Shi, H., and Ning. X. Brain-inspired fast-and slow-update prompt tuning for few-shot caas-innrremetal learning. IEEE Transactions on Neural Networks and Learning Systems, 2024.
Yao, H., Zhang, R., and Xu, C. Visual-language prompt tuning with mowledge-guided context optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6757-6767, 2023.
Rebguns, M. Wonderwords. https://pypi.. project/ http://wwowwwdd/e///ooo February 18, 2021.
Zhang. J., Ma, X., Guo, S., Li, P., Xu, W., Tang, X., and Hong, Z. Amend to alignment: Decoupled prompt tun- ing for mitigating spurious correlation in vision- -annuala models. In International Conference on Machine Learn- ing, 2024a.
Ross, T.-Y. and Dollár, G. Focal loss for dense object de- tection. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2980- 2988, 2017.
Zhang, J., Wu, S., Gao, I.., Shen, H. T., and Song, J. DePT: Decoupled prompt tuning, In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 12924-12933, 2024b.
Soomro, K. UCF101: A dataset of 101 human ac- tions classes from videos in the wild. arXiv preprint 1212.0402 2012.
Tao, X., Hong, x., Chang, X., Dong, S., Wei, X., and Gong, Y. Few-shot class-incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12183-12192, 2020.
Zhang, Z. and Sabuncu, M. Generalized cross entropy loss for training deep neural networks with noisy labels. In Proceedings of the Annual Conference on Neural Infor- mation Processing Systems, volume 31, 2018
10

---

CoCoA-Mix: Confasionaat-CCntdeeee----a Mixture Model for Context Optimization
Zhong, Y., Yang, J., Zhang, P., Li, C., Codella, N., Li, L. H., Zhou, I.., Dai, X., Yuan, L., Li, Y., et al. RegionCLIP: Region-based language-image pretraining. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 16793-16803, 2022.
Zhou, D.-W., Wang, F.-Y,, Ye, H.JJ,, Ma, L.., Pu, S., and Zhan, D.-C. Forward compatible few-shot class- incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 9046-9056, 2022a.
Zhou, K., Yang, J., Loy, C. C.,, and Liu, Z. Conditional prompt learning for vision-languane models. In Proceed- ings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 16816-16825, 2022b.
Zhou, K., Yang. J., Loy, C. C., and Liu, Z. Learning to prompt for vision-language models. International Jour- nal of Computer Vision, 130(9):2337-2348, 2022c.
Zhou, Z., Yang, M., Shi, J.-X., Guo, L.-Z., and Li, Y.-F. DeCoOP: Robust prompt tuning with out-of-distribution detection. In International Conference on Machine Learn- ing, volume 235, pp. 62161-62177, 2024.
Zhu, B., NIu, Y., Han, Y., Wu, Y., and Zhang, H. Prompt- aligned gradient for prompt tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 15659-15669, 2023.

---

CoCoA-Mix: Confasionaat--Cafideeeee--aa Mixture Model for Context Optimization
A. Proof of Theorem 3.2
Consider K + individual prompts T {to, b,...., bx} and a mixture model #7 with non-negative weights æ where Σi=g = 1. Let Dr be an arbitrary target domain. The expected error cr(77) of the mixture
(To. -1,-.., Tx) model on the target domain is defined as follows in terms of the Kullback-Leibler (KL) divergence:
er (pT) (xxx)))DD |- log p7 (y)]
where y is the ground-truth label for the image x.
exp (18/7 = Evey exp (D nid0, (1))
Using the definition of the mixture model, (T) two terms follows:
the expected error can be decomposed into
et(p7) = B(x,y)-Dr |- log PT (3)l
exp EE
ii= Ti8t, (v)/T)
B(x.y)>Dr
log
EV'y exp
EE 11 (1)/T
K
K
=B(x))~Dr
(u)/T + +log
Σ
exp
Σ
T (1')
1=0
P'EY
1==0
K
-xxx))DDrr
8t (V)/T T+ Σ n
log
Σ
exp (sz, (1)//T
N'E
K
K
+ D(
1g,og
exp (St, (l')/) + log
Σ exp
Σ
Ti (1)/7)
P'y
PEY
The first term is rewritten using the definition of the individual predictive distribution Pe. for the prompt t,, given as (er,,,.))// as follows:
Pr. (v) Evey (1)7)
B(x,)>Dr
Σ 81
(1)/T
/ + Σ 1g
exp (8x,
NEY
K
=E(x.)/Dr
8v,(U)/T log
Σ
exp (82, (1)/T)
i=0
FEY
K
=E(xx))DDT
Σ
Ti
logexp (st, ()))T) - log Σ
exp (SL, (sL, (1)/7))
im0
KEY
exp (St. TT)
Σ T: TE (x.y)>DD
log
Evey exp (8t, (1')/T)
K
mer (pr.).
As a result, the first term is equivalent to a convex combination of the expected errors of the individual predictive distributions with weights π.
For the second term, Jensen's inequality (Jensen, 1906) can be applied to bound it, as log Σ exp a convex function:
12

---

CoCoA-Mix: Contusson-amt..Coonddoorrvvvv Mixture Model for Context Optimization
(xx,))>Dr
n,log
exp (st, (1'/T) log
Σ
Σ
T (1)/TT)
1=0
PEy
P'EY
i=0
K
K
2 D(
Σ
1ylog
Σ
exp
(1)/T) +
T
log
Σ
exp (0t, (197))
1=0
KEy
L'EY
0.
By combining the results from the first and second terms, we conclude that the expected error of the mixture model on the target domain is bounded as follows:
cT Σ (p..)(e))
B. Statistical Validation of Assumption 3.4
To empirically verify Assumption 3.4, we conducted a statistical experiment using the CIFAR-100 (Krizhevsky et al., 2009) dataset and the CLIP model. Specifically, we randomly partitioned the 100 classes into 50 in-classes and 50 out-classes. We then trained specialized prompts on the in-class subset using prompt tuning and compared their predictions against the zero-shot CLIP baseline on both domains. This process was repeated over 10 random splits.
Figure 6 presents box plot summarizing the performance differences between the specialized prompt on the in-class domain Dr, and the generalized zero-shot prompt tg. The results show that t; consistently outperforms to on in-class samples from Dr. whereas to achieves higher accuracy on out-class samples from Dzy
To assess statistical significance, we performed one-sided paired t-tests on the per-split accuracy gaps. The resulting p-values were 9.25 10-12 12 for the in-class domain comparison (acc,, > acct,) and 2.06 10-10 10 for the out-class domain comparison (aco,, > acc;,). both significantly below the standard threshold of 0.05. These results allow to reject the null hypothesis and confirm that both inequalities in Assumption 3.4 hold with strong statistical confidence.
These findings support the assumption that specialized predictions are more effective within their domain, whereas generalized predictions are preferable for unseen class domains.
Accuracy Difference
0.28
0.26
0.24
acCt; aCCto
acCt
acCt;
Figure 6: Box plots of accuracy differences across 10 random splits. The left shows performance gains of specialized over generalized predictions on in-class domain Dr, i.e. acct, acCt, The right shows gains of the generalized prediction on out-class domain DT, i.e. acC₁, acCt,
13

---

CoCoA-Mix: Contisionanud.Cottdeen----- Mixture Model for Context Optimization
C. Effect of Cross-Entropy in the Mixture Model
The derivative of the cross-entropy LCE for the mixture model 77 with respect to "q is as follows:
( log p7 (v))
-1
Op (y)
17 rz.
K (EiooTi Ti88, (1)/T
1
exp
17 (v) θ5.
Σrey exp
Ei= Ti (1111))
8e, (v)(F))( - (T)
Σ
p7 (F(()ss10)) /
(v(
LEY
St, (v) -DF())O(()) /T
IEV
(sL, (v) BT.)TT,
where d5, the importance-weighhed similarity defined as a weighted sum of the predicted probability of the mixture model and the similarity derived from the prompt L, i.e. â1, = Eigy 177 (1) (1). For example, if the mixture model predicts class with the highest probability, št, approximates the similarity st, (1*) for class /" derived from prompt &,. Here, we explain how the CoA-weights T, for in-classes is optimized through the cross-entropy of the mixture model. For simplicity, assume St. Rs St, (I"), where l" = arg max, (1).
In the case St, (y) > St,, the prompt t; predicts the correct class y with high similarity. Therefore, when the mixture model misclassifies, i.e., 1 * y. the other prompts tyæt provide low similarities for the correct class y. This case results in an increase in I. through gradient updates, encouraging the mixture model to rely more on t4.
Conversely, if St. (v) < St., the prompt predicts the correct class y with low similarity. When the mixture model correctly classifies, i.e. 1" y,, it suggests that the other prompts liz' provide high similarities for the correct class y, while the prompt t, underperforms. This case decreases ny, allowing the mixture model trust the other prompts tzzs more.
14

---

CoCoA-Mix: Contusson-amt-CCConddoorrvvvar Mixture Model for Context Optimization
D. Implementation Details
D.I. Details CoA-Weights Optimization
There exist multiple strategies for optimizing the CoA-weights л. To evaluate the effectiveness of different optimization strategies for CoA-weights, we compared two implementation approaches: one-stage optimization and two-stage opti- mization. Specifically, for one-stage optimization, we design the optimization process such that the temperature scale is weighted by Wi, i.e., T; 7. r. We fix To to the temperature of the pre-trained CLIP model, i.e. 7D 0.01, and optimize тı jointly with prompt parameters. This enables the computation of π satisfying Σ मे 1, ensuring compatibility with the standard temperature scaling framework of CLIP. The optimization of a single scalar 71 suffices to determine both л and the global temperature T, via the relation T; n. TT However, the temperature scale not fixed during optimization, which may introduce instability when K 2 2. In contrast, the two-stage strategy decouples prompt tuning and foeghtss optimization. Here, π is parameterized using a softmax of parameters o0. OtK- Concretely, we fix the pre-softmax logit ag = and optimize a; under fixed temperature т = 0.01. The comparison of these strategies for the K = 1 case is reported in Table 5. The two strategies yield comparable performance, suggesting that the proposed loss function is robust to the specific choice of optimization parameterization for -weighsss In this paper, we used the one-stage strategy for most benchmarks and adopted the two-stage approach specifically for FSCIL tasks due its stability advantages.
Table 5: Comparison of various optimization strategies with respect to лı and for Base and New domains. In one-stage optimization, To is set to the temperature scale from pre-trained CLIP, i.e. 0.01. = {mo, æ1 and 7 are determined by optimizing Ti. In two-stage optimization, Ou is set to 0, and π and are determined by optimizing 01
T
Base
New
H
One-Stage Optimization
TT.TT
79.3
75.1
77.0
T1
Two-Stage Optimization
0.01
79.4
75.0
77.0
(xp e (ex)((a))
Table 6: 11 datasets used for base-to-ne generalization and cross-dataset transfer
Datasct
CLASSES
TASK
DESCRIPTION
EXAMPLE CLASSES
Large-scale dataset for object classifica- tion with diverse categories
tench, goldfish, great white shark, tiger shark, Accordion, Airplane, Brain, But- terfly. Crab, Motorbike, etc. Bengal, Persian, Beagle, Ameri- can Bulldog, etc.
IMAGENET
1, 0,000
Object recognition
Variety of object categories with random background images.
1ALTECH101
100
Object recognition
Fine-grained object recognition Fine-grained object recognition
Classification of pet breeds including cats and dogs. Images of various vehicle types, anno- tated by model.
OXFORDPETS
37
2000 AM General Hummer SUV, 2007 BMW X5 SUV, etc. Daffodil, Pink Primrose, Tiger Lily, Yellow Iris, Apple Pie, Waffles, Sushi, Chocolate Cake, Bibimbap, etc. Boeing 717, DH-82, Falcon 2000, etc.
STANFORDCARS
196
Fine-grained object recognition
FLOWERS 102
102
Classification of various flower species.
User-uploaded real-world food photos with varied backgrounds and noise. Aircraft images annotated hicrarchically by variant, family, and manufacturer. Covers diverse scenes including indoor, urban, natural environments. Real-world texture images annotated with descriptive attributes.
FOOD101
101
Object recognition
Fine-grained object recognition
FGVCAIRCRAFT
100
Abbey, Airport Terminal, Bed- room, Harbor, Bar, etc. Striped, Dotted, Cracked, Fi- brous, Scaly, Zigzagged, Annual Crop Land, Forest, Highway Road, River, etc.
SUN397
397
Scene recognition
Texture attribute recognition
DTD
47
Land use and land cover classification
Satellite images from Sentinel-2 focus- ing land use and cover types.
EUROSAT
10
Video clips human actions collected from TouTube in dynamic, real-world environments.
Apply Eye Makeup, Basketball Dunk, Playing Piano, etc.
UCF101
101
Action recognition
15

---

CoCoA-Mix: Confussin-and..Cotiidooovvwaava Mixture Model for Context Optimization
D.2. Details of Dataset
Table lists datasets for base-to-new generalization and cross-dataset transfer. Evaluation across 11 datasets highlights generalizability and efficiency beyond specific tasks domains.
D.3. Base-to-New Generalization
Experiments were performed utilizing CLIP with ViT-B/16 (Dosovitskiy, 2020) backbone. Training was conducted over 50 epochs with batch size of 32. The prompt t was optimized using the Adam optimizer with a learning rate of 0.002 and optimized using SGD with the same learning rate, momentum of 0.9, and weight decay of 5 x 10-4 The weight for LC,A was set to t 5.0, the weight for Lim was set to 10.0, and the margin was set to d = 0.2. The prompt length M was set to 16.
weight decay of x 10-1 CoA-weight
D.4. Few-Shot Class-Incremental Learning
Following Ran et al. (2024), experiments were conducted using CLIP with ViT-L/14 (Dosovitskiy, 2020) backbone. CoCoA-Mix used prompts of length M = 2 per session and accumulated them across sessions, requiring fewer parameters than the baseline except in the final two sessions. Each prompt was trained for specialization within its session, and the final prediction used all prompts from previous sessions. Tuned prompt L for each session, along with CoA- foA-weights n.." and T; were stored and reused. To ensure scaling stability across sessions, we used the two-stage optimization strategy for CoA-weights. Considering the number of iterations per session, CoA-weights were optimized for 2 epochs in the initial session, and for 100 epochs in all subsequent sessions. The margin d of the loss Cem set to 0.1. In the initial session, the out-class set was generated using random words, and in subsequent sessions, it consisted of classes from previous sessions. All other settings followed those of base-to-new generalization.
D.5. Cross-Dataset Transfer
Experiments used CLIP with ViT-B/16 backbone. The weight for Leas was set to = 7.0, while the loss weight for z was set to 2.0. Other settings followed those of base-to-new generalization.
16

---

CoCoA-Mix: Confusion-and-CCaiideee------ Mixture Model for Context Optimization
Table 7: Performance comparison on 11 datasets in cross-dataset transfer.
SOURCE IMAGENET
TARGET AVERAGE
TARGET
Method
CALTECH101
OXFORDPETS
STANFORDCARS
FLOWERS102
CLIP
66.73
64.89
93.27
89.18
65.56
68.05
CООP
69.06
59.88 (-5.01)
91.06 (-2.21)
86.74 (-2.44)
59.84 (-5.72)
62.38 (-5.67)
PROGRAD
70.21
62.36 (-2.53)
92.41 (-0.86)
87.90 (-1.28)
62.94 (-2.62)
66.98 (-1.07)
КGCOOP
70.52
64.45 (-0.43)
93.55 (+0.28)
89.86 (+0.68)
65.61 (+0.05)
68.33 (+0.28)
MAPLE
69.53
65.24 (+0.35)
93.43 (+0.16)
89.77 (+0.59)
65.70 (+0.144)
71.17
(+3.12)
DEPT
68.03
65.06
(+0.17)
94.07 (+0.80)
89.43 (+0.25)
65.87 (+0.31)
69.93 (+1.8
CoCoA-Mo MIX
70.85
65.27 (+0.38)
93.46 (+0.19)
89.07 0.11)
65.59
(+0.03)
68.72
(+0.67)
Target
Method
FOOD101
FGVCAIRCRAFT
SUN397
DTD
EUROSAT
UCF101
CLIP
85.43
24.81
62.61
44.09
48.36
67.51
COOP
83.29 (-2.14)
16.71 (-8.10)
59.40 (-3.21)
38.44 (-5.65)
39.24
(-9.12)
61.66 (-5.85)
PROGRAD
84.37 (-1.06)
17.10
(-7.71)
62.67 (+0.06)
39.87 (-4.22)
45.39
22.97)
63.98 (-3.53)
KGCOOP
85.83 (+0.40)
21.18
(-3.63)
64.84 (+2.23)
44.30 (+0.21)
44.64
(3.72)
66.39 (-1.12)
MAPLE
86.13 (+00.70)
23.27
(-1.54)
66.43 (+3.82)
44.83 (+0.744)
43.73
-4633)
67.93 (+0.42)
DEPT
86.27 (+0.84)
22.10
(-2.1)
65.77 (43.16)
45.53 (+1.44)
44.00
(4.36)
67.60 (+0.09)
COCOA-MIX
85.78 (+00335)
24.10
(-0.7)
63.61 (+1.000
46.41
(+2.32)
48.18
-0.18)
67.78 (+0.27)
E. Additional Experimental Results
E.1. Cross-Dataset Transfer
Table presents the accuracy of the method trained on single source dataset and evaluated both source and target datasets. Parenthesized values represent the performance difference relative to zero-shot CLIP. CoCoA-Mix achieves the highest accuracy on the source dataset, highlighting the effectiveness of CoA-loss in specialization. It also achieved the highest average accuracy across 10 target datasets, validating the effect of CoA-weights. For FGVCAircraft, DTD, and EuroSAT, where zero-shot CLIP shows significant performance gaps between source and target datasets, CoCoA-Mix exhibits the most stable and effective transferability, maintaining high performance with minimal accuracy degradation compared with previous methods. This suggests that CoCoA-Mix particularly effective under significant domain shifts. The relatively lower generalization improvement over base-to-new generalization is discussed in Appendix H.
17

---

CoCoA-Mix: Confusion-and.CCttidaaavvwww Mixture Model for Context Optimization
E.2. Performance Improvement Confusing Samples of CoA-loss
We define confusing samples as those misclassified by zero-shot CLIP with probability gap of 0.5 less between the correct and incorrect classes. Figure 7 presents the performance of confusing samples from the test data across 11 datasets for the Base classes. While slight performance drops are observed datasets such as F0od101, SUN397, and DTD, CoA-loss consistently enhances specialization across most datasets, with notable improvements on OxfordPets, Flowers and UCF101. This demonstrates the effectiveness of CoA-loss in handling confusing cases while enhancing the performance across diverse datasets.
OxfordPets
7.45%
Flowers102
5.98%
UCF111
4.65%
Ca1tech101
3.06%
ΕuroATT
2.02%
ImageNet
0.36%
FGVCAirCraft
0.33%
StanfordCars
0.25%
-0.27%
Food101
-0.91%
SUN397
-1.56%
DTD
Figure 7: Performance improvement on confusing samples, defined as instances with a probability gap S 0.5 between correct and incorrect labels or misclassified by zero-shot CLIP.
18

---

CoCoA-Mix: Confusion-and-CCufiddec------ Mixture Model for Context Optimization
E.3. Base Performance Comparison According to Loss Function
We compared CoA-loss with four loss functions: focal loss (FL) (Ross & Dollár, 2017), generalized cross-entropy loss (GCE) (Zhang & Sabuncu, 2018), cross-entropy (CE), and mean absolute error (MAE). The results are presented in Figure 8. FL modifies CE by weighting with (1 - 7)7 follows:
(x, y: p) - (1-p(g))) log ply),
(16)
where 7 2 is a focusing parameter. FL. prioritizes misclassified samples but limits learning from well-classified ones, leading to lower performance in prompt tuning.
GCE generalizes CE and MAE follows:
(xCE(x, y: 1 "11
(17)
where q E (0, 1] transitions from CE (q - 0) to MAE (q- = 1). While designed to mitigate the overfitting of CE and slow
convergence of MAE, GCE underperforms CE in prompt tuning.
CE, widely used in prompt tuning, is defined follows:
LCR (x, v1 = - log p(y).
(18)
Its gradients with respect to the similarity s3)) and s(c + y) are as follows, respectively:
OLer - (1- p(v)) and Ōs(c # v)
Occr
p(c),
(19)
0s(y)
where 7 is the temperature. CE updates gradients based only on individual class probabilities, ignoring inter-class relation- ships.
Adding MAE to CE, the loss defined as follows:
CCE+MAE (x, y: log p(y)
1 y
Σ 1i=y p(1),
(20)
where w controls the contribution of MAE. While CE emphasizes difficult samples, MAE ensures uniform learning over all samples, improving robustness to noise. With optimized w, CE+MAE slightly improves performance, but excessive MAE weighting reduces focus on difficult samples, degrading performance.
CoA-loss, similar MAE, applies only the correct class y and combined with CE follows:
CCE+COA y: = log ply) w -yy)),
(21)
where controls the contribution of CoA-loss. Unlike CE, CoA-loss considers both p(c * y) and p(v) when updating p(c * y), focusing on confusing cases where p(y) and p(c * y) approach 0.5. By focusing learning near decision boundaries, CE+CoA achieves the highest specialization performance.
79.8
0.1
0.5
10
20
10 3
79.0
11I
1.0
1.0
77.8
2.0
77.2
FL
GCE
CE
CE+MAE
CE+CoA
Figure 8: Performance comparison across various loss functions with varying hyperparameters. The hyperparameter setting for each loss shown above each data.
19

---

CoCoA-Mix:
Mixture Model for Context Optimization
Table 8: Effect of CoA-weight on Base and New classes
Bose ace (%)
75.00
79.00
74.80
New acc.
74.60
BASE
NEW
H
X
79.12
73.66
76.15
78.75
73.85
0
5
0.0
0.1
0.2
0.3
0.4
x
79.30
73.81
76.32
d
79.31
75.10
77.03
(a)
(b)
Figure 9: Sensitivity analysis of hyperparameters. The dotted line indicates baseline performance. (a) Effect of CoA-loss weight Base accuracy. (b) Effect of margin d on New accuracy.
F. Ablation Studies
F1. Effectiveness of Confidence-Aware Weights
To evaluate the effectiveness CoA-weight, we conducted an ablation study on optimizing weights for Base classes (T:"") and New classes (xz"")) separately and jointly. Table 8 shows that optimizing T,"" for Base classes improves performance over the baseline. This indicates that confidence adjustment in the in-class domain provides additional benefits for specialization. When both x, and π; are optimized, the performance improves further, achieving the best harmonic mean between the Base and New classes. These results demonstrate that adapting weights for both in-classes and out-classes is crucial for achieving generalization without compromising specialization.
F22 Sensitivity of Hyperparameters
We analyzed the sensitivity of method hyperparameters by evaluating performance under different configurations. Specifically, investigated the effect of varying weights w for CoA-loss Base classes and the margin d for CoA-weights on New classes.
Figure ((a) shows the Base performance for different w values, with the red dashed line representing the standard cross- entropy baseline. Introducing CoA-loss improves specialization, and tuning w effectively balances the focus on confusing cases, achieving superior performance. Figure 9(b) shows the New performance with varying margins The red dashed line represents the baseline without ="" Across all hyperparameter settings, our method outperforms the baseline, indicating that performance is relatively insensitive the choice of margin d.
F.3. Generation of Unseen Classes
We conducted an ablation study to evaluate the effectiveness of different strategies for generating unseen classes. The study compares three methods: (1) no generation (None), (2) generating random strings (Random String), (3) generating total of N classes, with half as random strings and half sampled from the random word API (Rebguns, 2021) (Random String and Word), and (4) sampling N words from the random word API (Rebguns, 2021) (Random Word). As represented in Table 9, sampling random words yields the best performance, outperforming other methods. This suggests that generating out-classes using semantically meaningful words, rather than arbitrary strings, further enhances generalization. This occurs because semantically meaningful words better align with learned embeddings, providing clearer semantic cues for distinguishing unseen classes, whereas arbitrary strings introduce noise into the embedding space.
Table 9: Ablation study comparing different strategies for generating unseen classes. The table reports accuracy on New classes
NONE
RANDOM STRING
RANDOM STRING AND WORD
RANDOM WORD
ACCURACY
74.12
75.00
75.04
75.10
20

---

CoCoA-Mix: Confusson-aud-CConddeee----aaaa Mixture Model for Context Optimization
G. Qualitative Visualizations
To better understand model behavior, we provide ScoreCAM (Wang et al., 2020) using the PyTorch library by Gildenblat et al. (2021), comparing CoCoA-Mix with zero-shot CLIP and CoOp. The visualization results are shown in Figure 10. the Flowers102 dataset (New), CoCoA-Mix more accurately attends to semantically meaningful regions in out-class samples, suggesting that CoA-weights effectively enhance generalization. In the FGVC Aircraft dataset (Base), CoA-loss focuses more precisely fine-grained details such as text on airplane wings, outperforming zero-shot CLIP in specialization. These qualitative results demonstrate that CoA-loss and CoA-weights contribute to generalization and specialization, respectively.
CoA-Looss
Image
Zero-Shot CLIP
CoOp
CoCoA-Mix
(w/ Ensemble)
Flowers 102 (Base)
english marigold
mexican aster
barbeton daisy
english marigold
english marigold
Flowers102 (New)
water lily
lotus
lotus
lotus
water lily
Flowers102 (New)
columbine
gaura
hee halm
columbine
columbine
FGVCAireraft (Base)
727-200
Boeing 717
707-320
727-200
727-200
FGVCAírcraft (New)
30G
Cessna 560
Embracr Legacy 600
Falcon 2000
Cessna 560
Cessna 560
Figure 10: ScoreCAM results on Flowers and FGVCAircraft. Red indicates higher activation. The leftmost column shows the image and ground-truth label. Other columns show the ScoreCAM results and predicted labels for each method.
21

---

CoCoA-Mix: Contssoo-snu.CCatddeecccvaaaar Mixture Model for Context Optimization
H. Limitation and Future Work
This study focused on textual prompt tuning. To demonstrate that improved performances in the source domain lead to better performance in the target domain, we adopted the following relation from Nguyen al. (2022):
C (LL(pp(z))ps()) 8,
(p( S cs(p)
(22)
V2
where z is defined as visual embedding; C' is a constant that bounds log p({), ensuring each class probability is at least exp(-C); and ps are the marginal distribution of z for the target and source domains, respectively;
This assumes that visual embeddings z remain consistent across domains, implying similar embeddings for images of the same semantic but differing styles. However, pre-trained CLIP can generate distinct visual embeddings for images with varying styles. As result, while the proposed method ensures that prompts specialized for the source domain perform well in the target domain from class perspective, it does not guarantee performance across target domains with different styles. This limitation is reflected in the results, where cross-dataset transfer shows weaker generalization compared with base-to-new generalization. Future work could address this by optimizing visual prompts to align embeddings of images with different styles but the same semantics. Such methods could enhance specialization and generalization across both classes and styles, enabling more robust performance across diverse domains.
22


Reviews:
[{"review_id": "ACZQUH0QrS", "reviewer": "Reviewer_AKUY", "summary": "This paper addresses the challenge of improving both specialization and generalization in prompt tuning for vision-language models. It proposes a confusion-aware loss (CoA-loss) that refines decision boundaries between confusing classes, enhancing specialization. Additionally, they introduce a confidence-aware temperature (CoA-temp) mechanism within a mixture model to improve generalization by adjusting prediction weights based on confidence levels. The proposed method, CoCoA-Mix, integrates these components and demonstrates superior performance over state-of-the-art approaches in balancing specialization and generalization.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## `W-1` \n\nWe agree that Remark 3.3 should state $\\\\epsilon\\_T(\\\\hat{p}\\_t^\\\\pi)=\\\\min\\_i\\\\epsilon\\_T(\\\\hat{p}\\_{t\\_i})$, not $\\leq$. However, our main claim regarding the simultaneous improvement of specialization and generalization **relies on Theorem 3.2 and our method, not Remark 3.3**. \n\nThe figure [here](https://anonymous.4open.science/r/CoCoA-Mix-B466/Equation-target_error_bounds.png) illustrates how $\\alpha_\\text{in}$, $\\alpha_\\text{out}$, and $\\\\epsilon\\_{S\\_v}(\\\\hat{p}\\_v)$ determine the upper bound of the target error in the mixture model. CoA-Loss reduces $\\\\epsilon\\_{S\\_v}(\\\\hat{p}\\_v)$ and enhances specialization, while CoA-Temp optimizes $\\alpha_\\text{in}$ and $\\alpha_\\text{out}$ via ***Eq. 15*** and ***Eq. 16*** to improve generalization. Consequently, CoA-Loss and CoA-Temp jointly reduce the upper bound of $\\epsilon_T(\\hat{p}_t^\\pi)$, thereby enhancing specialization and generalization simultaneously.\n\n**Even without CoA-Temp, $\\\\epsilon\\_{T\\_v^\\\\text{out}}(\\\\hat{p}\\_t^\\\\pi)=\\\\min\\_i\\\\epsilon\\_{T\\_v^\\\\text{out}}(\\\\hat{p}\\_{t\\_i})$ and $\\\\epsilon\\_{T\\_v^\\\\text{in}}(\\\\hat{p}\\_t^\\\\pi)=\\\\min\\_i\\\\epsilon\\_{T\\_v^\\\\text{in}}(\\\\hat{p}\\_{t\\_i})$ guarantee performance at least as good as zero-shot CLIP for out-class domains and learned prompts for in-class domains simply by setting $\\alpha_\\text{out}=0$ and $\\alpha_\\text{in}=1$**. This demonstrates that the mixture model can *preserve* both specialization and generalization. With CoA-Loss and CoA-Temp, these can be further *improved* simultaneously. **This is also supported by the empirical results in *Tables 1 and 3***, where CoCoA-Mix consistently outperforms CLIP across both *Base* and *New* domains.\n\n---\n## `W-2`\n\nAs described in L222–224 (right column), $\\tau_{\\boldsymbol{h}}$ is fixed to the temperature of the pre-trained CLIP model. Therefore, optimizing $\\tau_{\\boldsymbol{v}}$ is sufficient to determine both $\\alpha$ and $\\tau$ via ***Eq. 14***: $\\tau=\\frac{\\tau_h\\tau_v}{\\tau_h+\\tau_v}$ and $\\alpha=\\frac{\\tau_h}{\\tau_h+\\tau_v}$. The design was chosen to ensure compatibility with CLIP's standard temperature scaling framework.\n\n**We also evaluated the direct optimization of $\\alpha$ using a softmax.** The results can be found [here](https://anonymous.4open.science/r/CoCoA-Mix-B466/Table_I-optimization_strategies.png). Specifically, we fixed the pre-softmax logit of $\\boldsymbol{h}$ to zero (i.e., $\\alpha_h'=0$) and set $\\tau$ to the temperature of the pre-trained CLIP model. Table I shows that both approaches yield similar performance, indicating that **our objective is not sensitive to the choice of optimization strategy for $\\alpha$.**\n\n---\n## `W-3`\n\nThe temperatures $\\tau_{\\boldsymbol{v}}^\\text{in}$ and $\\tau_{\\boldsymbol{v}}^\\text{out}$ are optimized simultaneously in the same loop, but each temperature is optimized with its own loss function and optimizer, as described in \"[Overall process]\" of our response to Reviewer *P3tC*’s comment `W-1`.\n\n---\n## `W-4`\n\nAt inference time, given the optimized prompt $\\boldsymbol{v}$ and the temperatures $\\\\tau\\_{v}^\\\\text{in}$ and $\\\\tau\\_{v}^\\\\text{out}$, we compute the mixture weight as follows:\n\n$$\n\\\\alpha(l)=\\\\begin{cases}\n\\alpha\\_\\text{in} = \\\\tau/\\\\tau\\_{v}^\\\\text{in}&\\\\text{if $l\\\\in\\\\mathcal{Y}\\_{S\\_v}$} \\\\\\\\\n\\alpha_\\text{out} =\\tau/\\tau_{v}^\\text{out}&\\text{otherwise}\n\\end{cases}.\n$$\nThe predictive probability is then:\n\n$$\n\\hat{p}_{t}^{\\pi}(l) = \\\\frac{\\\\exp\\\\left((1-\\\\alpha(l))\\\\boldsymbol{s}\\_{h}(l)/\\\\tau+\\\\alpha(l)\\\\boldsymbol{s}\\_{v}(l)/\\\\tau\\\\right)}{ \\\\sum\\_{l'\\\\in\\\\mathcal{Y}}\\\\exp\\\\left((1-\\\\alpha(l))\\\\boldsymbol{s}\\_{h}(l)/\\\\tau+\\\\alpha(l)\\\\boldsymbol{s}\\_{v}(l)/\\\\tau\\\\right)}.\n$$\n\n---\n## `W-5`\n\nWe provide **ScoreCAM visualizations** comparing CoCoA-Mix with CLIP and CoOp to improve interpretability. The visualization results are available [here](https://anonymous.4open.science/r/CoCoA-Mix-B466/Figure_I-scoreCAM.png). \n\nIn *Flowers102 (New)*, **CoCoA-Mix attends more accurately to semantic cues in out-class domains**, highlighting that CoA-Temp improves generalization. In *FGVC Aircraft (Base)*, **CoA-Loss attends to fine-grained details** such as wing text more effectively than zero-shot CLIP. These results show that CoA-Temp and CoA-Loss enhance generalization and specialization, respectively.\n\n---\n## `Relation To Broader Scientific Literature` \n\nWhile prompt mixtures have been studied in prior work, our method differs notably in design and objective. Please see our response to Reviewer *P3tC* (`W-2`) for a detailed comparison.\n\nFurthermore, most existing prompt tuning methods are designed for specialization-generalization trade-offs or incremental learning, making it difficult to achieve both within a single framework. In contrast, as shown in ***Table 2***, our method naturally extends to incremental learning without catastrophic forgetting, bridging the two objectives. Therefore, we believe this work makes a meaningful contribution to prompt tuning.", "claims_and_evidence": "The claim that the provided mathematical framework demonstrates that specialization and generalization can be improved\nsimultaneously is not supported by convincing theoretical evidence. \n\nThe inequality derived in Theorem 3.2 ensures that \\( \\epsilon_{T}(\\hat{p}_t^{\\pi}) = \\min_i \\epsilon_T(\\hat{p}_{t_i}) \\) when \\( \\pi \\) is optimized. However, it does not guarantee that \\( \\epsilon_{T}(\\hat{p}_t^{\\pi}) < \\min_i \\epsilon_T(\\hat{p}_{t_i}) \\). Consequently, Theorem 3.2 does not establish that specialization and generalization can be improved simultaneously.", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes, the proofs for the theoretical results are correct.", "experimental_designs_or_analyses": "Yes, there are no outstanding issues in the designs and analyses of this paper.", "relation_to_broader_scientific_literature": "The mixture model in prompt tuning is straightforward and has been explored in previous literature; therefore, the contributions of this work to the relevant community are relatively modest.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "**Strengths:**\n\n1. The proposed mixture model is straightforward and effective in achieving a better trade-off between specialization and generalization in prompt tuning for vision-language models.\n\n2. The empirical results demonstrate the superiority of the proposed methods over state-of-the-art prompt tuning approaches.\n\n**Weaknesses:**\n\n1. The inequality derived in Theorem 3.2 ensures that \\( \\epsilon_{T}(\\hat{p}_t^{\\pi}) = \\min_i \\epsilon_T(\\hat{p}_{t_i}) \\) when \\( \\pi \\) is optimized. However, it does not guarantee that \\( \\epsilon_{T}(\\hat{p}_t^{\\pi}) < \\min_i \\epsilon_T(\\hat{p}_{t_i}) \\). Consequently, Theorem 3.2 does not establish that specialization and generalization can be improved simultaneously.\n\n2. The design of confidence-aware temperature requires further clarification: why is the weight $\\alpha$ in the mixture model adjusted by optimizing the temperature rather than optimizing $\\alpha$ directly?\n\n3. The classification of in-class and out-class domains requires further discussion. In the implementation stage, are the optimization problems in Equations (15) and (16) solved simultaneously or separately in different cases?\n\n4. The details of how to calculate the predictive probability of input images using the tuned mixture model at the inference stage should be discussed in Section 3.\n\n5. Some visualization results in the evaluation section should be included to interpret the superiority of the proposed mixture model over existing methods, such as by visualizing the features on which the proposed algorithm relies.", "other_comments_or_suggestions": "Please refer to the weaknesses listed in the previous section.", "questions_for_authors": "Please refer to the weaknesses listed in the previous section.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Nrr2uyDKJ4", "reviewer": "Reviewer_d4Lr", "summary": "This paper proposed a series of techniques to tackle the problem of improving specialization in prompt tuning, including confusion-aware loss, mixture-model using confidence-aware temperature. Extensive experiments are conducted to show the performance of the proposed method.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## `Other Strengths And Weaknesses`\n\nFirst, the \"CoA-Loss\" in ***Table 1*** corresponds to a naive ensemble model with a fixed $\\\\pi=\\\\{0.5,0.5\\\\}$. This naive mixture prediction can perform well when the equal weighting is near-optimal for a given dataset. In contrast, CoCoA-Mix optimizes CoA-Temp based on the class domains, thereby enhancing specialization and generalization. In some datasets, the advantage over fixed-weight ensembles may appear limited, particularly under few-shot settings with high variance. To verify this, we evaluated both methods across six random seeds. The results (see below) show that **CoCoA-Mix consistently matches or outperforms the naive ensemble across all datasets when averaged over more trials.** This suggests that the lower performance on the three datasets is mainly due to variance resulting from limited trials.\n\n|  | Caltech101 |  |  | OxfordPets |  |  | EuroSAT |  |  |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n|  | Base | New | H | Base | New | H | Base | New | H |\n| CoA-Loss (w/o mixture model) | 97.2±0.1 | 94.1±0.5 | 95.7 | 94.6±0.7 | 97.3±0.3 | 96.0 | **84.5±0.8** | 62.7±1.5 | 72.0 |\n| CoA-Loss (seed1,2,3) | 97.9±0.1 | **94.5±0.2** | **96.2** | 94.9±0.5 | **97.9±0.1** | **96.4** | 83.4±0.5 | **70.1±2.5** | **76.1** |\n| CoCoA-Mix (seed1,2,3) | **98.0±0.0** | 94.4±0.1 | **96.2** | **95.2±0.4** | 97.6±0.1 | **96.4** | 83.5±0.7 | 69.1±3.1 | 75.5 |\n\n\n|  | Caltech101 |  |  | OxfordPets |  |  | EuroSAT |  |  |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n|  | Base | New | H | Base | New | H | Base | New | H |\n| CoA-Loss (seed1,2,3,4,5,6) | 98.1±0.2 | 93.6±1.4 | 95.8 | 94.6±0.5 | **97.6±0.7** | 96.2 | 84.1±1.4 | **66.9±4.1** | 74.4 |\n| CoCoA-Mix (seed1,2,3,4,5,6) | **98.2±0.2** | **94.3±0.4** | **96.2** | **95.0±0.4** | **97.6±0.3** | **96.3** | **84.4±1.4** | **66.9±3.6** | **74.5** |\n\n---\n## `Other Comments Or Suggestions`\n\nWe appreciate the suggestion. To empirically verify ***Assumption 3.4***, $\\\\epsilon\\_{T\\_v^\\\\text{in}}(\\\\hat{p}\\_v)\\\\leq \\\\epsilon\\_{T\\_v^\\\\text{in}}(\\\\hat{p}\\_h)$ and $\\\\epsilon\\_{T\\_v^\\\\text{out}}(\\\\hat{p}\\_h)\\\\leq\\\\epsilon\\_{T\\_v^\\\\text{out}}(\\\\hat{p}\\_v)$, we conducted a statistical experiment using the CIFAR-100 and the CLIP model. We randomly split the 100 classes into 50 in-class and 50 out-class domains, trained $\\hat{p}_v$ using prompt tuning on the in-class subset, and compared it with the zero-shot $\\hat{p}_h$ on both domains. This process was repeated over 10 random splits. A box plot summarizing the results is available [here](https://anonymous.4open.science/r/CoCoA-Mix-B466/Assumption3.4-statistical%20experiment-boxplot.png). The results show that $\\hat{p}_v$ consistently outperforms $\\hat{p}_h$ on in-class samples, while $\\hat{p}_h$ achieves better performance on out-class samples. \n\nTo assess statistical significance, we conducted one-sided paired $t$-tests on the per-split accuracy gaps. The resulting $p$-values were $p_{\\text{in}} = 9.25 \\times 10^{-12}$ and $p_{\\text{out}} = 2.06 \\times 10^{-10}$, both well below the standard threshold of 0.05. These results allow us to reject the null hypothesis and confirm that the inequalities in ***Assumption 3.4*** hold with strong statistical significance. We will include a summary of this verification and the corresponding visualizations in the appendix of the final version.\n\n---\n## `Methods And Evaluation Criteria` \n\nThank you for your feedback. If you could point us to the specific uncertainty-aware loss functions you are referring to, we would be happy to provide a detailed comparison and clarify the differences or contributions of our approach. Additionally, we provide a comparison with focal loss, generalized cross-entropy, and MAE in ***Appendix D.3***, including both formulation and impact on specialization.", "claims_and_evidence": "Yes. The paper claimed that it focuses on the problem of improving specialization for specific domains. Based on this motivation, the authors proposed several techniques to help prompts adapt to the target domains better.", "methods_and_evaluation_criteria": "The method part is basically reasonable, because it focuses on the task-specific ability in prompt tuning and the generalization of universal embedding, and the loss ideas it uses are similar to some uncertainty-aware loss function ideas in the field of domain adaptation. The disadvantage is that the loss function is not novel enough, because similar ideas have already appeared in the traditional field of domain adaptation. The more interesting point is the proof related to the mixture-model and the setting of confidence-aware temperature.", "theoretical_claims": "I checked the corresponding proof and found no errors. The corresponding proof makes use of Jensen inequality and simple splitting, and its assumption is reasonable.", "experimental_designs_or_analyses": "The experiments are extensive, with a total of 11 datasets tested. I have no doubts about the experimental performance of the method.", "relation_to_broader_scientific_literature": "This article solves the adaptation problem of prompt-tuning for different tasks to a certain extent, and provides some new analysis for the combination of domain adaptation and prompt-based methods.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "The experimental performance of the method is very good, which is worthy of recognition. What needs to be discussed is that the performance of $CoCoA-Mix$ is lower than that of $CoA-Loss$ in the three data sets in the experiment, and the author needs to make some clarification and discussion in this regard.", "other_comments_or_suggestions": "I don't know if it's possible to do some statistical experiments on the Assumption $3.4$, such as using a certain data set to see if the Assumption $3.4$ is met. Although I intuitively think that the assumption is reasonable, it may need further verification.", "questions_for_authors": "None.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GSMRGxUmDl", "reviewer": "Reviewer_P3tC", "summary": "The authors explore the prompt tuning problem of vision-language models and propose a training method, CoCoA-Mix, to improve generalization and specialization simultaneously.\nCoCoA-Mix consists of a confusion-aware loss to enhance specialization for confusing classes and a confidence-aware temperature strategy that aims to improve generalization.\nThe authors provide a theoretical derivation for the proposed method.\nMoreover, the authors provide experimental results based on CLIP to demonstrate the effectiveness of CoCoA-Mix.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## `Theoretical Claims`\nIn our paper, we refer to Eq. 12 in [1], which relates distributional shift to generalization performance:\n$$\nl\\_\\\\text{test} \\\\leq l\\_\\\\text{train} + \\\\frac{M}{\\\\sqrt{2}} \\\\sqrt{\\\\text{KL} [p\\_T(z)|p\\_S(z)] +\\\\mathbb{E}\\_{p\\_T(x)}\\\\left[ \\\\text{KL}[p\\_T(y|x)|p\\_S(y|x)] \\\\right] },\n$$\nwhere $l\\_\\\\text{train}=\\\\mathbb{E}\\_{(x,y)\\\\sim p\\_S(x,y),z\\\\sim p(z|x)}[-\\\\log\\\\hat{p}(y|z)]$; $l\\_\\\\text{test}=\\\\mathbb{E}\\_{p\\_T(x,y)}[-\\\\log\\\\hat{p}(y|x)]$; $p\\_S$ and $p\\_T$ are joint data distributions of source and target domains, respectively; and $z$ is a representations derived from $x$. \nIn our setting, $l_\\text{train}$ and $l_\\text{test}$ correspond to $\\epsilon_S(\\hat{p})$ and $\\epsilon_T(\\hat{p})$ by ***Eq. 2***. \nWe assume the CLIP encoder yields sufficiently informative representations via large-scale pretraining, satisfying Assumptions 1 and 2 in [1]. This allows us to apply Eq. 12 in [1] to bound the target error $\\epsilon_T(\\hat{p})$ as follows:\n$$\n\\epsilon_T(\\hat{p}) \\leq \\epsilon_S(\\hat{p}) + \\frac{M}{\\sqrt{2}} \\sqrt{\n\\text{KL}(p_T(\\mathbf{z})|p_S(\\mathbf{z}))+\\lambda\n}.\n$$\n\nWe hope this clarifies our method. If we misunderstood your point, we would appreciate further clarification.\n\n[1] Kl guided domain adaptation, Nguyen *et al*., ICLR 2022.\n\n---\n\n## `W-1`\n\n**[Confidence-Aware Temperature for Generalization without Trade-Offs]**\n\nWe would appreciate clarification on which part of the 'detailed description' was confusing so that we can address it more precisely in the final version. \n\nTo clarify, CoA-Temp adjusts the weight between specialized and generalized predictions based on whether the class is in the in-class or out-class domain. The in-class is defined as the set of classes in the training dataset, while the out-class consists of random words not included in the training dataset. \n\nThe in-class weight $\\alpha_\\text{in}$ is optimized using cross-entropy loss on in-class training samples (***Eq. 15***), while the out-class weight $\\alpha_\\text{out}$ is optimized using an entropy-based loss (***Eq. 16***) under Assumption 3.4, which assumes generalized predictions are more reliable in out-class domains. For details on $\\alpha$ optimization and its use during inference, see our responses to Reviewer *AKUY* (`W-2`) and (`W-4`), respectively.\n\n**[Overall process]**\n\nOur method optimizes three components and **each component is optimized simultaneously with a separate loss function and optimizer** :\n\n1. Prompt $\\boldsymbol{v}$ via cross-entropy and CoA loss ***(Eq. 9)***,\n2. In-class temperature $\\tau_{\\boldsymbol{v}}^\\text{in}$ via a cross-entropy loss ***(Eq. 15)***,\n3. Out-class temperature $\\tau_{\\boldsymbol{v}}^\\text{out}$ via an entropy-based loss ***(Eq. 16)***.\n    \n**[Clarification on $q$ in Eq. (12)]**\n\n$q$ is the probability that a target sample belongs to the in-class domain $\\mathcal{D}_T^\\text{in}$ (L250–251). It is used only in the theoretical analysis *(Eq. 12)* to decompose the target error into in-class and out-class components.\n\n---\n## `W-2`\n\n**[Prompt mixture models in VLMs]** \n\nRecent works such as Allingham *et al*. (ICML'23) [2] and Lu *et al*. (ICML'24) [3] have explored prompt ensembling using hand-crafted prompts or multiple backbones to improve generalization. These methods generally do not target specialization and often require additional inference-time costs. In contrast, our method (1) explicitly improves specialization and generalization and (2) is more efficient than backbone-level ensembling [3], requiring no extra forward passes during inference, and (3) provides a mathematical framework for specialization and generalization in prompt tuning. \n\nWe also compared CoA-Temp with the mixing strategy proposed in [3] and the results are provided [here](https://anonymous.4open.science/r/CoCoA-Mix-B466/Table_II-mixing_strategies.png). While the sample-aware weight generator in [3] requires $205,204$ learnable parameters, CoA-Temp achieves superior performance using only two parameters.\n\n[2] J. U. Allingham, *et al*. “A simple zero-shot prompt weighting technique to improve prompt ensembling in text-image models,” in *ICML*, 2023.  \n[3] Z. Lu, *et al*. “Beyond sole strength: Customized ensembles for generalized vision-language models,” in *ICML*, 2024.\n\n---\n## `W-3`\n\nWe have included CoCoOp in our evaluation under the same setting as ***Table 1***. The average performance is as follows:\n\n|  | Base | New | H |\n| --- | --- | --- | --- |\n| CoCoOp | 77.23 | 68.56 | 71.33 |\n\nOur goal is to demonstrate that **CoCoA-Mix performs well even under challenging low-shot conditions**. We found that the benefits of our method are more pronounced when the training data is scarce. Nevertheless, we evaluated the 16-shot setting (results available [here](https://anonymous.4open.science/r/CoCoA-Mix-B466/Table_III-16shots_results.png)). Despite fewer learnable parameters, CoCoA-Mix achieved competitive performance on Base and clearly outperformed others on New.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes, I think the proposed method makes sense for the fine-tuning problem of VLMs.", "theoretical_claims": "I check the proof of Theorem 3.2 and I think it is correct.\nHowever, I have doubts about the introduction of the target domain. The paper [1] shows a distribution shift between the target domain and the source domain. This shift generally does not refer to the gap between the empirical error and the expected error.\n\n[1] Kl guided domain adaptation, Nguyen et al., ICLR 2022.", "experimental_designs_or_analyses": "Yes, I checked the results and settings, and the authors provided the source code.", "relation_to_broader_scientific_literature": "This submission discusses the weakness of previous prompt tuning methods and proposes CoCoA-Mix to improve generalization and specialization simultaneously.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:\n1. The authors study prompt tuning problems and propose CoCoA-Mix to enhance generalization and specialization simultaneously.\n2. The authors provide theoretical support to make the method more convincing.\n\nWeakness:\n1. The writing is not clear. The detailed description of Confidence-Aware Temperature for Generalization without Trade-Offs in the submission is too confusing, which makes it difficult for readers to intuitively understand its content. In addition, the authors did not provide a description of the overall process of the method, and even a description of the overall loss function. (e.g., q only appears once in Eq. 12 but has no specific description.)\n2. There are too few works in the related work section, which basically only include a few comparative methods, and cannot give readers a full picture of the field.\n3. Lack of baseline CoCoOp. Why is the 4-shot setting used in the experiment in Table 1 instead of the commonly used 16-shot setting?", "other_comments_or_suggestions": "please refer to Other Strengths And Weaknesses.", "questions_for_authors": "please refer to Other Strengths And Weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes CoCoA-Mix, which aims to enhance specialization and generalization in prompt tuning for vision-language models. It introduces two key components: a confusion-aware loss (CoA-Loss) that sharpens decision boundaries for confusing classes, and a confidence-aware temperature mechanism (CoA-Temp) that adaptively weights predictions in a mixture model based on in-class and out-class confidence. The method is supported by theoretical insights and evaluated across eleven datasets spanning diverse domains.

The paper tackles an important and practically relevant challenge in prompt tuning—balancing task specialization with generalization across domains. Reviewers acknowledged the breadth of the empirical results and the general applicability of the framework. In particular, CoA-Temp was seen as a novel and effective design for improving generalization without compromising specialization, and the empirical performance across base and novel domains was consistently strong. The theoretical formulation was considered mathematically correct, though reviewer AKUY questioned whether the central claim of simultaneous improvement was fully supported by the presented result in Theorem 3.2. Nonetheless, the authors provided clear clarifications and an illustrative breakdown of their bound-based interpretation.

The main concerns come from two reviewers who gave weak rejection. Reviewer AKUY challenged the interpretation of the theoretical result, but the authors clarified that their claim rests on Theorem 3.2 (not the disputed Remark 3.3), and supported it with visual and statistical evidence. Reviewer P3tC pointed out writing clarity issues, missing baselines, and limited related work coverage. The authors responded comprehensively by adding CoCoOp as a baseline, reporting new 16-shot results, and clarifying the optimization process and architectural pipeline. Reviewer d4Lr, who gave a weak accept, found the experiments strong and the theoretical design valid, raising only minor concerns about novelty. Although reviewers did not update their scores after rebuttal, the authors’ detailed and well-substantiated response addresses the core concerns, and the lack of further engagement should not be treated as a sign of unresolved issues.

Considering the above aspects, the AC recommends acceptance to ICML.
Decision: Accept (poster)