NeuCLIP: Efficient Large-Scale CLIP Training with Neural Normalizer Optimization
Authors: Xiyuan Wei
Chih-Jen Lin
Tianbao Yang
Venue: ICLR 2026 Poster (2026)
Keywords: Representation Learning
Contrastive Learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Accurately estimating the normalization term (also known as the partition function) in the contrastive loss is a central challenge for training Contrastive Language-Image Pre-training (CLIP) models. Conventional methods rely on large batches for approximation, demanding substantial computational resources. To mitigate this issue, prior works introduced per-sample normalizer estimators, which are updated at each epoch in a blockwise coordinate manner to keep track of updated encoders. However, this scheme incurs optimization error that scales with the ratio of dataset size to batch size, limiting effectiveness for large datasets or small batches. To overcome this limitation, we propose NeuCLIP, a novel and elegant optimization framework based on two key ideas: (i) **reformulating** the contrastive loss for each sample **via convex analysis** into a minimization problem with an auxiliary variable representing its log-normalizer; and (ii) **transforming** the resulting minimization over $n$ auxiliary variables (where $n$ is the dataset size) via **variational analysis** into the minimization over a compact neural network that predicts the log-normalizers. We design an alternating optimization algorithm that jointly trains the CLIP model and the auxiliary network. By employing a tailored architecture and acceleration techniques for the auxiliary network, NeuCLIP achieves more accurate normalizer estimation, leading to improved performance compared with previous methods. Extensive experiments on large-scale CLIP training, spanning datasets from millions to billions of samples, demonstrate that NeuCLIP outperforms previous methods. Code is available at https://github.com/Optimization-AI/NeuCLIP.

Published as a conference paper at ICLR 2026
NEUCLIP: EFFICIENT LARGE-SCALE CLIP TRAINING WITH NEURAL NORMALIZER OPTIMIZATION
Xiyuan Wei
Chih-Jen Lin
Tianbao Yang
Texas A&M University xwe @tamu. edu
National Taiwan University
Texas A&M University t cianbao-yang@ttam edd
ej 1 in@cst e ntu. edu. Mohamed bin Zayed University of Artificial Intelligence chihjen. lii@mbbuui........ ac.ae
ABSTRACT
Accurately estimating the normalization term (also known as the partition fune- tion) in the contrastive loss is central challenge for training Contrastive Language-Image Pre-training (CLIP) models. Conventional methods rely on large batches for approximation, demanding substantial computational resources. To mitigate this issue, prior works introduced per-sample normalizer estimators, which are updated at each epoch in blockwise coordinate manner to keep track of updated encoders. However, this scheme incurs optimization error that scales with the ratio of dataset size batch size, limiting effectiveness for large datasets or small batches. To overcome this limitation, we propose NeuCLIP, novel and elegant optimization framework based on two key ideas: (i) reformulating the contrastive loss for each sample via convex analysis into minimization prob- lem with an auxiliary variable representing its log-normalizer, and (ii) trans- forming the resulting minimization over auxiliary variables (where n is the dataset size) via variational analysis into the minimization over a compact neural network that predicts the log-normalizers. We design alternating optimiza- tion algorithm that jointly trains the CLIP model and the auxiliary network. By employing tailored architecture and acceleration techniques for the auxiliary network, NeuCLIP achieves more accurate normalizer estimation, leading to im- proved performance compared with previous methods. Extensive experiments large-scale CL.IP training, spanning datasets from millions to billions samples, demonstrate that NeuCLIP outperforms previous methods. Code is available
INTRODUCTION
Since its introduction, Contrastive Language-Image Pretraining (CLIP) (Radford et al. 2021 has emerged the de facto standard for vision-language representation learning. The strong capability to align images with corresponding texts has made CLIP valuable for a wide range of real-world applications, including zero-shot classification (Qian Hu 2024). cross-modal retrieval (Zeng &
Mao 2022). ), text-to-image generation (Ramesh et al. (022). and high-quality dataset selection (Fang al. 2023ah. With the rise of large language models (LLMs), CLIP has also been widely adopted to equip LLMs with the ability to interpret visual inputs Bai et al. 2025).
fundamental impediment training CLIP models is their extensive dependence on huge datasets: attaining competitive performance typically mandates access to millions or even billions of image- text pairs (Fang et al. 2023a Wang al., (0256). mainstream approach for training CLIP mod- els is to optimize bimodal contrastive loss, which contrasts each positive image-text pair against numerous negative pairs. To enable training on billions of samples, two primary strategies have emerged to approximate the prohibitive normalization term required for contrastive loss gradient calculation. The first strategy, which relies on massive GPU resources, uses an extremely large batch size to construct contrastive loss within each batch for backpropagation. This strategy is used by many works, including OpenAl CLIP (Radford et al. (02I)) and OpenCLIP (Cherti et al.] 2023 The second strategy addresses the high resource demand of the first by directly optimizing

---

Published as a conference paper at ICLR 2026
global contrastive loss, which contrasts each positive pair with all negative pairs. To tackle the com- putational challenge, an estimator of the normalizer for each sample's s contrastive loss is maintained and updated using moving average formula following rigorous framework of finite-sum coupled compositional optimization. It was first proposed by Yuan et al. (2022) for unimodal contrastive self- self-supervised learning, and later adopted by Wei al. (2024) with significant improvements for CLIP training, yielding the FastCLIP method. While being less resource demanding, this strategy suffers an inherent limitation that the optimization error scales with the ratio of dataset size batch size, constraining its effectiveness for large datasets or small batch sizes.
Recently, there emerge some new ideas for CLIP training. For example, Zhai et al. 2023) proposed SigLIP with sigmoid- based contrastive loss by formulating the problem as a binary classifica- tion problem, which avoids the computation of the normalization term involving numerous nega- tive pairs. However, SigLIP still requires a large batch size to achieve competitive performance. Sun et al. (2025) proposed AmorL.I that leverages a lightweight network to predict the normal- izer of each contrastive loss. While conceptually appealing, AmorLIP's objective for training the lightweight network still faces the challenge of estimating the log- peg-partition function for the gradient calculation of the lightweight network, leading to chicken-and-ees problem.
This paper aims to address the limitations of FastCLIP and AmorLIP for CLIP training through a principled approach towards optimizing the global contrastive loss with neural normalizer. Our method is built on two key ideas: (1) using convex analysis, we reformulate the contrastive loss of each anchor data minimization problem with an auxiliary variable, whose optimal solution corresponds to the log-normalizer; and (2) using variational analysis, we transform the minimization over n auxiliary variables (where dataset size) into minimization over compact network that directly predicts the log-normalizers, referred to as the normalizer- prriiierrppiiiiiinn network (NPN).
Compared with FastCLIP and AmorLIP, our method offers several notable advantages. First, the objective for learning the encoders and the NPN is unified, and its gradient avoids any nonlinear dependence on the partition function. This allows traditional stochastic gradient methods to be employed for updating both the encoders and the NPN without incurring gradient estimation bias. Second, the unrestricted optimal solution of the auxiliary variable motivates us to inject inductive bias into the design of the NPN, resulting in simple yet effective architecture: a feedforward layer on top of the encoders followed by a timization framework enables key acceleration techniques, including alternating optimization with multiple NPN updates and periodic re- re-initialization of the NPN's parameters using sampled updated embeddings, which yields significantly better normalizer approximation. The contributions of this paper are threefold:
pooling layer. Moreover, this seamless op-
We reformulate the contrastive loss into an equivalent form in which the normalization terms are explicitly exposed as optimization variables. This reformulation provides a principled foundation for efficient neural normalizer approximation.
We introduce a joint optimization problem that learns the encoders and a compact normalizer- prediction network (NPN) with unified objective, which is derived from variational analysis. We also develop an efficient algorithm for alternatively optimizing the NPN and the CLIP encoders.
We validate the effectiveness of our approach through extensive experiments on large-scale datasets, showing consistent improvement over existing methods. Comprehensive ablation studies are also conducted to highlight the contribution of different components in our framework.
2 RELATED WORKS
Efficient Training of CLIP Models. Numerous approaches have been proposed to enhance the efficiency of CLIP model training. Prior works include curating high-quality datasets (Schuhmann) et al. 2022 Fang et al., 2023a| Xu et al. 2024 Wang et al., 2024), designing more efficient vision encoder architectures (Fang et al.) 2023b Alabdulmohsin et al. 2023 Chen et al. 2024), applying image-token masking to reduce computational cost (Li et al. 2023b a), and modifying the geome- try of the embedding space (Chou & Alam 2025 Pal et al. 2025). Additional strategies leverage knowledge distillation train compact student models (Vasu et al. 2024) or employ pretrained reference model to steer and accelerate the training of a target model, thereby improving scaling laws (Wei et al. 2025). In contrast, work is orthogonal to these directions: focus improv-
2

---

Published as a conference paper at ICLR 2026
ing the optimization process itself by providing more efficient and stable method for minimizing the contrastive loss.
Optimizing the Global Contrastive Loss. The global contrastive loss first introduced by Yuan ct al. (2022) to address the large batch-size requirement of SimCLR (Chen et al. 2020). They proposed an efficient optimization algorithm, SogCLR, with provable convergence guarantees for unimodal self-supervised contrastive learning. Notably, SogCLR with a batch size of 256 matches the performance of SimCLR with a batch size of 8,192 on ImageNet. Subsequent work by QQu al. (023) provided a distributionally robust optimization (DRO) interpretation of the global contrastive loss, leading to a constrained DRO formulation with individualized temperature optimization. Build- ing on this perspective, Weii et al. (2024) proposed a simplified variant that unifies the temperature parameters into a single scalar. Another line of work explains the global contrastive loss from probabilistic perspective, showing it as the maximum likelihood estimation of discriminative model (Wang et al. 2025ap. Despite these different viewpoints, all methods rely on optimization techniques similar to SogCLR, which maintain and update per-sample estimators for the normaliza- tion term of the contrastive loss. Consequently, they share a key limitation: the optimization error scales with the ratio between the dataset size and the batch size.
Learning with Auxiliary Networks. Leveraging auxiliary networks to facilitate training has been widely explored (Shen et al. 2024 He et al., 2020; Su et al. 2025 Kim et al. 2024 Evans et al. 2025| Qiu et al. 2024]] Sun et al. 2025). We highlight two closely related works (Qiu et al.) 2024
Sun et al., (025). Qiu et al. (2024) introduced TempNet, a network designed to predict personal-
ized temperatures for each sample when training CLIP models with a robust global contrastive loss. Their approach was also motivated by variational analysis that led the joint optimization of en- coders and TempNet. Nevertheless, their optimization algorithm, built on SogCLR, still requires maintaining and updating per-sample estimators of the contrastive loss normalization term, and thus inherits the same limitations of SogCLR. Sunn et al.] (2025) proposed AmorLIP, which optimizes a robust global contrastive loss similar to that of Wei et al. (0024) by jointly learning a lightweight network to approximate the partition function. However, Im differs from our method in sev- cral key aspects: (i) the objective for training the lightweight network is heuristically defined as divergence minimization between its predictions and the true partition function values, which still involves a non-linear function of the normalizer, leading to the chick-and-egg problem; (ii) Amor- LIP simply employs a Multi-Layer Perceptron (MLP) with few layers for the NPN, while our design leverages inductive bias to improve performance, as validated in ablation studies. Furthermore, AmorLIP requires maintaining exponential moving average (EMA) network of NPN to mitigate the chicken-and-egg issue, since its auxiliary objective still involves estimating a nonlinear function of the normalizer.
3 PRELIMINARY
Notations. We denote by to the parameters of the CLIP model. Let = {(x,, ((r;, 2;)) be a training dataset of samples, where æ; is an image and z, is its corresponding text description. The features of image x; and text zj output by the CLIP model are denoted by e1,i = e1 (w: x.) E Rd and e2.j = e2 2,) E IRd respectively, where ej (w; and (w; denote the image encoder and the text encoder, respectively. The cosine similarity between features e1,i and e2.j is denoted as $t,j = cos(ee,.. e2.)).
The convex conjugate of function f R + R is given by (z) == max y z- fz)). From the Fenchel-Moreau theorem (Rockafellar 1997 Theorem 12.2) we know that if f is proper, lower semi-continuous and convex function, the convex conjugate of f is equivalent to f, i.e., f(x) = f*'((' (I:) == max. x (v).
Global Contrastive Loss. Following Wei et al. 2024). we consider optimizing a robust global contrastive loss defined below:
min T S
Σ ESES
log E + gi (w, Tii, +
Σ
log E + + T:i, s.(w).:i.S)) + 2rp, (1)
S
Z:ES
where
01
(1)))
Σ
exp
8i.j - 8ig
(2(1)
exp
88, - 84.6
[8|
-1
T
S
T
EiE
IE, ESJ
3

---

Published as a conference paper at ICLR 2026
is the temperature parameter, To: small constants, and is hyperparameter. In order to solve this problem, we need to compute an estimator of the gradient. In particular, the gradient w.rt. w is given by
+ |S| E ((w, E
1
(w, 1;
Σ E fel,,Tii, (gz(w, T; S
1
We can see that the terms e + (w, T; T; S) and e (; serve as normalizers of the gra- dient calculation for image æ; and text 2,. A key challenge is that 91 (w, r: i) and 92(W, T; i, cannot be computed exactly, as they depend on all other data samples. This necessitates approxi- mating these normalizers using only batch of samples.
Mini-batch Approximation. A naive approach is to simply use a mini-batch approximation (Rad-) ford et al. 2021, Cherti al. 2023). i.e., sampling subset C and computing the following gradient estimator
Σ B
e gn (1, (w,, +
1 [B]
Σ B
(w;,
92(w, T;
This is equivalent to performing backpropagation miiii-btchh contrastive loss
Σ
log(s + (w; r)) +
log(s + 92(w, 1;,,
B
B
B:EB
#:EB
However, this gradient estimator is biased as its expectation does not give the true gradient Equa- tion (2) due the non-linearity of the reciprocal function. As a consequence, it requires large batch size (Yuan et al. 2022).
Moving-average Approximation. To address the large batch issue, Yuan al. (022 proposed the SogCLR algorithm, which maintains two sequences estimators {u], 22,) for each 2; 2; approximate + 91 (wo(), (0)0))))(0 ,, () and e + (2((w((), i the t-th iteration, respectively. At t-th iteration with solutions (oo(t),((0) for (a,, 2;) in the sampled batch g(t), their normalizer
estimators are updated as follows
(++1)
(1 (1-7)ui, 7(5 + 91 (w) ,1);; (())),
1,
(3)
(++1) (1 7 7(5 + (2(w(0, ,()), B()),,
2,4
where E E [0, 1] is hyperparameter. Then, the gradient estimator for ((() () is computed by
7()
(())
Σ
(++1) (w)i(w(() (0;,,,()) +
Σ
Vwgz(wol)
[801
[B(
(++1)
2:EB()
"i,
EIEBO
u2
It has been shown that the optimization error of SogCLR converges zero Yuan et al. 2022). Built on this idea, Wei et al. (2024) proposed FastCLIP, an efficient distributed CLIP training framework with several improvements including the temperature optimization and the learning rate schedule for y. However, the convergence error of FastCLIP suffers from a scaling factor of ((n/ B) on the standard rate Yuaan et al. 2022), where = [B()) is the mini-batch size per-iteration. This property not desirable since the error will increase when increases and B decreases.
4 NEUCLIP: CLIP TRAINING WITH NEURAL NORMALIZER OPTIMIZATION
In this section, we first present reformulation of the contractive loss as a minimization problem. Then we derive a joint optimization problem from variational analysis to learn the encoders and the normalizer- predition network (NPN). Finally, we present optimization algorithm.
4.1 REFORMULATING THE CONTRASTIVE LOSS
Without lose of generality, let us consider the individual contrastive loss for an image anchor data 2; given by F(w,, r; m²) = log(s + 9i (w, i)). Since f(-) log(-) is convex function,
4

---

Published as a conference paper at ICLR 2026
can leverage the conjugate transformation f(x) max, v z- fy) with (") = log( [c] -
to reformulate the above individual contrastive loss as follows (by setting I = E (: i))
F(w,T; m;) = log(e + 9i (w,T; S)), - max (e + 91 (;, - (())
--maxfy (5 + 91 (w, T;4, + log(-y) + = minf (s + 91 (w, (w, - loog(- -y) - 1}
y
min min(cxp(-a) (E + 91 (w, T; i.) + 1},
(5)
where the last equality uses a change of variable a = - log(-y). y). It not difficult to derive that the optimal solution a" the last optimization problem is given by a² log(e + gl (w, i; (cf. Appendix A.II, which is exactly the log-normalizer. We note that the above reformulation can be viewed as special case of the optimized certainty equivalent (OCE) (Ben- Tal & Teboulle 2007).
Substituting the above reformulation of each contrastive loss in Equation we get the following equivalent form of the global contrastive loss:
min exp((a111) 9i + +01,1 +01 0n
min 00
S
RIES
(6)
+
1
Σ 2,ES
min minexp(-a2.)) (5 (i,, + 02.1 - 1 +2Tp.
S
Indeed, the update of u1, U2 sequences of SogCLR in Equation be recovered by solving the
above problem using stochastic block mirror descent method (Lan 2020 Section 4.6.2). We provide detailed derivation in Appendix A.2
4.2 NEURAL NORMALIZER OPTIMIZATION
Maintaining and updating {a, } in a coordinate-wise manner is the root that leads to a scaling factor of (n / B) the convergence error FasICLIP. To mitigate this issue, our idea is grounded in the following theorem from variational analysis.
Theorem (Rockafellar& & Wets 2009 Theorem 14.60). Let F be a space of measurable functions from St to I that is decomposable relative to finite measure Let f :9 x R + R be normal integrand. Then, as long as IZEa (x, ((x)))(dx)) + oo for all a(-) E, we have
inf [ [(x (Lz,,())))((e)) =f JEO o())F NEn
-fr inf ((x, m(dr).
.ER
Moreover, if the above infimum not -o,, then (-) arg aig Srea f(a ((x)) if and only if of (x) E arg min [xx a)) for p-almost every x E
The above equality indicates that the minimization over individual variables a for each x on the right hand side within an integral of can be translated into searching for function a(-) e that minimizes the whole integral over all x E Ω.
Our reformulated contrastive loss (Equation 6) shares similar structure to the right hand side of Equation (7) when the measure is probability measure, with minimization over (1,ú 02 and then the average over all samples. Hence, Theorem iimplies that
min exmeeeeppaaai (s gi (w, (), + t11 -1 0n
T.
S
SIES
min a1(F)F
exp(-an (a:)) (e (Wi,; + on (nss)) -
S
#:ES
As result, Equation
6 can be transformed into:
exp(-or(x;)) (1 (m; -1}
min wst
min (1))
(8)
S
ES
cxp(-az(z)) (e gaw + +z((2)) -1 + 2pr.
+ r
S
E,ES
5

---

Published as a conference paper at ICLR 2026
Neural Normalizer Optimization. Directly solving is not easier than solving b due to the constraint 01 (2))) E F. Our strategy is to approximate these functions using parameterized neural networks. Specifically, solve the problem by restricting a (-)- E Fw, and ax2()) E Fw.., where Fw, and Fw, denote the function classes represented by neural networks parameterized by Wi and W2, respectively. This raises the question of how to design the architecture of the neural network. A naive idea is to use simple feedforward neural networks. It is guaranteed by universal approximation theory that a neural network can approximate any continuous function arbitrarily well as long as the network is wide enough. However, this could increase the burden of training. Instead, design a network with inductive bias, i.e., by leveraging the
we draw insights from Theorem structure of the problem. Specifically, we want
01 (2x) E arg min exp(-a1.)) (E + 91 (w,,;;i T;i, + ta11 -1 a1,
cI -
log
E
Σ
exp
S
2, E8,
where the last equality is from the optimality condition (cf. Appendix A.I and the definition of 91- Since 01,4. E2, are readily available from the CLIP encoders, we only need a model that compresses the information of all 62,j- Inspired by this, define the following network architecture with parameter W1 E Rdxm, where m is the number of neurons of the hidden layer:
cos(e1... W1.y ) - ef,, )) (10)
log E e+
Σ exp
where W1,5' denotes the j'-th column of W. This can be seen compact network built top of the encoders, processing their output embeddings {er,s. e2 with a feedforward layer parameter- ized by Wi and followed by log-sum--xponential pooling layer. Compared with the unrestricted optimal solution of (x;;) in Equation (9) we can view Wi,,,.., Wi.. as prototypical embeddings that summarize the texts {z This supported by existing studies self-supervised representa- tion learning, which show that the learned embeddings of training samples from the same class tend
concentrate around their class means (Ben-Shaul et al. 2023). Similarly, we use the following network with an additional parameter W2 Raxm to approximate a2(2;) by
cos(e2, W25') - ef,1 e((exx) )) (1I)
1
a2(2;) 00 (22: ei.i, e2,)) = log
exp
j'=1
Finally, our unified objective for learning the encoders and the NPN becomes
min T.
(exp(-a1 (W1,,,,, e2.)) (e gi (w, an (Wi,, +
W. W...W
S
ESES
(exp( a2 e1,, e2.)) (e 9e + a2 (2
2T 1).
S
E.ES
(12)
4.3 ALTERNATING OPTIMIZATION AND ACCELERATION
straightforward approach is update w, T, W,, W2 simultaneously by using stochastic gradient based methods. However, we find that this approach does not work well in practice (see Appendix B.3 for empirical results). The reasons are multi-fold: (i) the overall objective landscape of w. r. W.. 2r is much more complicated than the original objective in terms of w. r: (ii) the NPNs' predictions also rely on the output embeddings of the encoders, which makes the predicted normalizers from one step update of W,, 1, W2 not good enough for updating the pa- rameters w. F. A natural idea to address this issue is to split the parameters w, т, W1, Wz into two blocks (w, r) and (W1, Wz), and use alternating optimization scheme update two blocks one by one. The method is proved to enjoy a convergence guarantee (Grippof & Sciandrone 1999 The- orem 6.3) and similar strategies have been used for other problems that exhibit two natural blocks,
To solve the problem 12

---

Published as a conference paper at ICLR 2026
Algorithm 1: The NeuCLIP Algorithm
Input: CLIP model w("), temperature ,("), NPNs ( / iterations T, restart frequency T, and number of updates T. for NPNs
dataset S, number of
for = - -- do
Randomly sample mini-batch B(() C.S;
2.
if't mod T, 0 then
// Restart Reset ((()) ( with (e2, (e and (ei,) respectively: :
Set w ((() and w/(.0) w! for t' = 0, To 1
// Multiple updates
12 w.r.t w(lit), w:
7
Compute mini-batch gradient of the objective
Update (1,',++ w(tt'+1)) with an optimizer:
8
Set w((+)) (((+.Te) and w(t-1) w((.Ta)
Compute al_ (z,) and a2
a² 1 (z.) for (æ,, z, E B(O);
10
Compute mini-batch gradient of the objective
12
w.r.t (0(),,,)
Update ((++)) and -(++1) with an optimizer;
12
e.g., non-negative matrix factorization Lin 2007 However, the method is not implementable as exactly solving the optimization problem over one block given another block is unrealistic. To ad- dress this, we apply only one or several stochastic gradient updates to approximately minimize the problem over each block. We present practical method in Algorithm [ which is referred to as NeuCLIP. For comparison, we present FastCLIP in Algorithm
2 in Appendix A.3
Acceleration. We develop two techniques to accelerate training. First, we perform multiple NPN updates before updating the CLIP model. To avoid additional cost, we use the same batch of data from the given iteration for updating the NPNs. Since each update of the encoders changes the loss landscape the NPNs, multiple updates enables the NPNs to maintain the same pace as the encoders and produce more accurate normalizers. In practice, updates (e.g.. Т, 10) is sufficient. Since the NPNs are lightweight networks, the additional cost minimal (ef. Appendix [B. for empirical results). Second, we apply periodic re-initialization of the NPNs by using randomly sampled text embeddings {e2.;} to reset W, and their corresponding image embeddings }ei,i} to reset W2. This also helps mitigate the convergence gap between the CLIP model and the NPNs This procedure is motivated by the observation that W1 and W2 act as compact summaries of all text and image embeddings. Together, these two techniques ensure that the NPNs remain well-aligned with the evolving encoders, leading to more effective training.
find that small number of
Convergence of NeuCLIP. We analyzed the convergence property of an abstract algorithmic frame- work of Algorithm in Algorithm (cf. Appendix for details). Let f(w, T, Wi, W2) denote the function considered in Equation
Theorem we show that after
O(e (4) iterations,
we can find an e-stationary point such that Eho E V (w)(), ,(t), w w( ||" E
under proper assumptions.
5 EXPERIMENTS
Experiment Settings. In all the experiments, we train a CLIP model on an image-text dataset with a given compute budget (i.e., number of samples to be processed) using NVIDIA H100 GPUs. The text encoder of the CLIP model is Transformer (Vaswani et al., (017). and the image encoder either a VIT (Dosovisskiy et al., (02T) or ResNet (He et al. (016). We consider five datsets, including CC3M (Changpinyo et al. 202 lb, CC12M (Sharma al. 2018) and three subsets of the DEN dataset at different scales (Fang et al. 2023a), ranging from 14M to 192M and IB. The details of the experiment settings, including batch size and training budget for each dataset, can be found in T Ablation studies are conducted on the CC3M dataset and the DFN-14M dataset.
Evaluation Metries. Throughout the section,
evaluate the performance of trained models on zero-shot classification and retrieval tasks. Specifically, we leverage the Datacomp bench- mark (Gadre et al. 2023) and report the average performance on its 38 tasks (denoted as Datacomp
7

---

Published as a conference paper at ICLR 2026
Table 1: Details of experiment settings. "Size" denotes the number of image-text pairs we success- fully downloaded. "Samples" denotes the number of samples that are processed for training, which equals the total number of iterations times global batch size. "Batch Size" denotes the global batch size.
Dataset
Dataset Size
Samples Seen
Vision Encoder
Batch Size
CC3M
2.7M
100M
ResNet-50
1,024
CC12M
9.2M
300M
ViT-B/32
2,048
DFN-14M
13.7M
320M
ViT-B/32
4,096
DFN-192M
192M
1.3B
VIT-B/16
5,120
DFN-IB
1.0B
3.0B
VIT-B/16
5,120
OpenCLIP
FastCLIP
SigLIP
NeuCLIP
40
Datacomp Average
ImageNet & Variaatts
25
30
35
30
15
25
10
20
100
200
300
100
200
300
100
200
300
Samples
Samples (M)
Samples (M)
(a)
(b)
(c)
Figure 1: Performance curves of different methods on DFN-14M (a): Datacomp Average perfor-
mance. (b): ImageNet& & Variants performance. (c): Retrieval performance.
Average). Moreover, report the average performance two subsets of the 38 tasks: (1) Ima- geNet & Variants, which consists of classification tasks on ImageNet-related datasets; (2) Retrieval, which consists of retrieval tasks. More information can be found in Appendix B.2
Hyperparameters. For the NPNs, we set the number of hidden neurons as m 4, 096, restart frequency T, 500, and number of updates per iteration T, 10. We the Adam op- timizer (Loshchilov Hutter) 2019) to train the CLIP model and the AdaGrad optimizer (Duchi al. 2011) train the NPNs. We provide more details on other hyperparameters Appendix B.
5.1 COMPARISON WITH BASELINES
In this subsection, we provide comparison between our proposed NeuCLIP and several strong base- lines, including OpenCLIP (Cherti et al. 2023), FasICLIP (Wei al. 2024). SigLIP (Zhai et al., 2023) and AmorLIP (Sun et al.) 2025). For OpenCLIP and SigLIP, we use the implementation from open clip (Ilharco et al. 202T For FastCLIP and AmorLIP, we use their released code, respec- tively. For experiments on CC3M, CC12M and DFN-14M, we repeat each method three times with different random seeds and report the mean. The Datacomp Average performance of different meth- ods the different datasets presented in Table 2 and the full evaluation results deferred to Appendix B.5 Additionally, we plot the performance curves during training in Figure
The first observation we have is that NeuCLIP outperforms all other methods all datasets, indi- cating the effectiveness of our approach. Secondly, from Figure D we find that NeuCLIP achieves larger improvement at the later stage of training. Note that in Algorithm ! we optimize the NPNs ( (((() given fixed CLIP model w!!), (t), At later stage training, the change in (*) (()) becomes smaller, enabling the learning of NPNs be more efficient for the updated encoders. Third, for AmorLIP. we observe differences between our results on CC3M and CC12M and those reported in Sun et al. (2025). This is because
reran the AmorLIP training the same datasets used for the other methods, whereas our CC3M and CC12M datasets differ from those in Sun et al. (2025). they come from different downloaded snapshots. (ef. Appendix B.5
8

---

Published as a conference paper at ICLR 2026
Table 2: Datacomp Average performance of different methods trained on different datasets.
Method
CC3M
CC12M
DFN-14M
DFN-192M
DFN-IB
OpenCLIP 21.84
27.91
37.78
54.58
56.25
FastCLIP
24.74
31.50
38.45
54.72
56.68
SigLIP
22.19
28.60
37.23
54.26
56.32
AmorLIP
22.89
29.86
37.53
53.83
56.24
NeuCLIP
25.08
31.89
39.16
54.90
57.34
OpenCLIP
RaSTCLIP
NeuCLIP
Dataromp Average
39.5
Our NPN
MLP
20
12.8
39.0
9.4 f
Enror
Datacomp
38.48
6.2
38
10
38.5
0.7
1.9
38.0
Unified Separate
0.02
0.1
0.5
25
5
10
20
50
500
1000
10
Objective
(x1000)
Tu
Batch Size
Dataset Size (M)
(a)
(c)
Figure 2: (a): Ablation study of training objective and architecture of NPNs. (b): Ablation study of restart frequency of NPNs (left) and number of updates (right). (c): Estimation error of NPNs.
5.2 ABLATION STUDY
In this subsection, we conduct ablation study of different components in NeuCLIP. We run all the experiments DFN-14M CC3M, where the setting is the same in Table
Comparison with AmorLIP's Design. The main differences between the design of AmorLIP and NeuCLIP lie in the training objective and model architecture of the NPN: (1) AmorLIP employs two separate objectives to train the CLIP model and the NPNs respectively, while we leverage a unified objective. Specifically, AmorLIP the following objective train their NPNs:
2|5 Σ m..,EE (1001 - log(e + 91 7; 8))1 + ||02, log(s + 92(w, (1;,)))1)
where 01 is the predicted log-normalizer for and 022 is the predicted log-normalizer for (2) Amori.IP chooses Multi-Layer Perceptrons (MLPs) as their NPNs, while NeuCLIP we use single-layered NPNs that take advantage of the inductive bias in the optimal solutions of a. In or- der to provide a comparison between these design choices, we conduct the following experiments: (1) In Line 7 of Algorithm D we compute the gradient of the NPNs using the above equation, in which case the objectives for training the CLIP model and the NPNs are not unified anymore. (2) We instantiate the NPNs with MLPs, and initialize them with random weights, which follows the practice of Sun et al. (2025). We present the Datacomp Average performance of different objectives and architectures in Figure 2a From the results tive yields better performance than learning with two separate objectives, and our inductive- NPN design outperforms MLPs. We also conduct the same experiments on CC3M and CC12M. The results, along with full results on DFN-14M, are presented in Tables to 16 where we reach similar conclusion.
observe that learning with the unified objec-
Restart Frequency. To investigate the impact of the restart frequency T,, we conduct experiments with different values of T,, where T, oo means the NPNs are never i--itiialiied. We plot the Datacomp Average performance of different T, in the left part of Figure 2b and present the full evaluation results Table 17 Appendix B.6 From the results we can observe that when T, > 500, the performance decreases. Also, T, 500 gives better performance than no re-initialization (i.e.. T, oo). This is probably because the NPNs lack behind the updated encoders, making their estimations less accurate for updated encoders. Moreover, when T, is small, the NPNs frequently set to the mini-batch features. this case the output of the NPNs is close to the mini- batch estimators, which also leads degraded performance.
Multiple NPN Updates, Another strategy to mitigate the gap of convergence speed between the NPNs and the encoders to update the NPNs multiple (T.)) times before updating the encoders.

---

Published as a conference paper at ICLR 2026
Specifically, we use the same batch of data to compute the stochastic gradient w.r.t. the NPNs' pa- rameters across multiple updates to avoid expensive forward passes of the CLIP model. We conduct experiments with different Tu and plot the results in the right part of Figure 2b From the results we can see that as the number of updates increases, the performance first increases, and starts to de- crease when Tu 10. The decrease is expected since we using the same batch of data update the NPNs, which will overfit to the batch and provide inaccurate estimation for other samples.
Estimation Error of Normalizers. From Section increases when the dataset size increases or when the batch size decreases. To demonstrate the effectiveness of our approach, we compare the estimation error of normalizers OpenCLIP. Fast- CLIP, and NeuCLIP under the following two settings. Firstly, we run each method on CC3M using two batch sizes (512 and 1024). For each run, we select five checkpoints such that the corresponding checkpoints across batch size settings have seen the same number of samples. At each checkpoint, we compute the estimation error as the mean squared error between the logarithm of the predicted normalizers and the true normalizers, and then report the mean across checkpoints. More details on the computation the estimation error are presented in Appendix [B6]] As shown on the left part of Figure 2c, the error of NeuCLIP increases only marginally when the batch size decreases, whereas OpenCLIP and FastCLIP exhibit a much larger increase. Secondly, we run each method on two datasets of different sizes: subset of DFN-14M (n 1.37M) and full DFN-14M
know the estimation error of FastCLIP
13. 13.7M). On the right part of Figure 2c we plot the average error of five checkpoints selected using the same pro- cedure as above. The results show that NeuCLIP only slightly affected by the increase in dataset size, in contrast to OpenCLIP and FastCLIP that suffer significant degradation.
CONCLUSION
In this paper, have studied the problem of efficiently approximating the normalizers in the trastive loss for training CLIP models. We proposed a novel objective that allows us to jointly learn CLIP encoders and compact networks that predict the nog-oormalizers of image and text data. We proposed an alternating optimization algorithm to learn the encoders and the compact networks effi- ciently. We conducted extensive experiments to demonstrate the effectiveness of our algorithm, and reveal insights on training of the network through ablation study.
ACKNOWLEDGMENTS
We thank anonymous reviewers for constructive comments. This work used GPU resources at TAMU ACES and NCSA Delta through allocation CIS230245 from the Advanced Cyberinfrastrue- ture Coordination Ecosystem: Services & Support (ACCESS) program, which is supported by U.S. National Science Foundation grants #2138259, #2138286, #2138307, #2137603, and #2138296. Xiyuan Wei and Tianbao Yang were partially supported by National Science Foundation award #2306572. Tianbao Yang was partially supported by Stephen Horn '79 Engineering Excellence award at Texas A&M University. Chih-Jen Lin was supported in part by National Science and Technology Council of Taiwan grants DSTC-I13-2222--0002000-
and NSTC-114-2634-FF
002-007.
REFERENCES
Ibrahim M Alabdulmohsin, Xiaohua Zhai, Alexander Kolesnikov, and Lucas Beyer. Getting vit in shape: Scaling laws for compute-optimal model design. Advances in Neural Information Pro- cessing Systems, 36:16406-
2023.
Shuai Bai, Keqin Chen. Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502. 13923, 2025.
Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the lim- its of object recognition models. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Bur E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.. 2019. URL / wwwp:/ceeee.innc................ cctpapeccppppppppppppee pddf
fi hills//paper/2019///i Trcaa//eSSSSSOaaaaaaeaoaaaaaaaa0
10

---

Published as a conference paper at ICLR 2026
Ido Ben-Shaul, Ravid Shwartz-Ziv, Tomer Galanti, Shai Dekel, and Yann LeCun. Reverse engineer- ing self-superrised learning, 2023. URL https://axxiv... org/ wttg/bb///////.oo abs/ 2305 15614
Aharon Ben-Tal and Marc Teboulle. An old-new concept of convex risk measures: The optimized certainty equivalent. Mathematical Finance, 17:449-476, 02 2007. 200770031
doi: 10.1111jj1166-99960
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12M: Pushing web- scale image-text pre-training to recognize long-tail visual concepts. In CVPR, 2021.
Jieneng Chen, Qihang Yu, Xiaohui Shen, Alan Yuille, and Liang-Chieh Chen. Vitamin: Designing scalable vision models in the vision-language era. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2295-112966 June 2024.
Ting Chen, Simon Korbllith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. CoRR, abs/2002.05709, 2020. URL https: 2202/ 05709
Xinlci Chen, Hao Fang, Tsung- Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv: 504.00325, 2015.
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gor- don, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pultern Recognition (CVPR), pp. 2818-2829 2829, June 2023.
Jason Chuan-Chil Chou and Nahid Alam. Embedding geometries of contrastive language-iman -mnaa pre-training. In Alessio Del Bue, Cristian Canton, Jordi Pont-Tuset, and Tatiana Tommasi (eds.), Computer Vision ECCV 2024 Workshops, pp. 39994166 Cham, 2025. Springer Nature Switzer- land. ISBN 978-3-031-915855
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. leee, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recogni- tion at scale. In International Conference on Learning Representations, 2021. URL https: //openzeview. net www.://ee//ormmmiidmeemmmmm
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, (1(6))2121-21583 2011. URL.
httt:://mm>>
www./prrgpppp..eee. e 2 2/dcchiuulh .tm...mllllmmm.
Talfan Evans, Shreya Pathak, Hamza Merzic, Jonathan Schwarz, Ryutaro Tanno, and Olivier J. Hénaff. Bad students make great teachers: Active learning accelerates large-scale visual under- standing. In Aleš Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and Güll Varol (eds.), Computer Vision ECCV 2024, pp. 280, Cham, 2025. Springer Nature Switzerland. ISBN 978-3-031--7644-99
Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander Toshev, and Vaishaal Shankar. Data filtering networks. arXiv preprint arXiv:2309.17425 2023a.
Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xin- long Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 19358 19369, June 2023b.

---

Published as a conference paper at ICLR 2026
Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smymis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang. Eyal Orgad, Rahim Entezari, Giannis Daras, Sarah Pratt, Vivek Ramanujan, Yonatan Bitton, Kalyani Marathe, Stephen Mussmann, Richard Vencu, Mehdi Cherti, Ranjay Krishna, Pang Wei W Koh, Olga Saukh, Alexander Ratner, Shuran Song, Hannaneh Hajishirzi, Ali Farhadi, Romain Beau- mont, Sewoong Oh, Alex Dimakis, Jenia Jitsev, Yair Carmon, Vaishaal Shankar, and Ludwig Schmidt. Datacomp: In search of the next generation of multimodal datasets. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 17092-277111 Curran Associates, Inc., 2023. URL https:://prott httpps:/pprocceedingg.. neur ips cc cc/paper.pppp.....ceece / 313324440d5aaaaaaa42244422232213 Dateee-aatta--aatsssssssm htw.eatttttttteeeeeeeeaaaaaaaea. Bww.BBenhhhass....e
pdf
Luigi Grippof and Marco Sciandrone. Globally convergent block-coordinngn techniques for uncon-
strained optimization. Optimization Methods and Software, 10(4):587-637, 1999, doi: 1080/ 10556789908805730. URL
http:///oo..ogg./////// 10. 10.
Zhishuai Guo, Yi Xu, Wotao Yin, Rong Jin, and Tianbao Yang. Unified convergence analysis for adaptive optimization with moving average estimator. Machine Learning, 114(4):1-51, 2025.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang. Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 8340-8349, October 202
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adver- sarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15262-15271, June 2021b.
Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, July 2021. URL pattps://dae /tt///0000rr//1..22212111 per.doo5543773 If you use this software, please cite below.
Minsu Kim, Joohwan Ko, Taeyoung Yun, Dinghuai Zhang, Ling Pan, Woo Chang Kim, Jinkyoo Park, Emmanuel Bengio, and Yoshua Bengio. Learning to scale logits for temperature-condiionn GFlowNets. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Rexearch, pp. 24248-24270 24270. PMLR, 21-27 Jul 2024. URL https: hattps://ppoceedinngg .ml.. mlr press/ #235/kim244 html
Guanghui. Lan. First-order and Stochastic Optimization Methods for Machine Learning. Springer Series the Data Sciences. Springer International Publishing, Cham, 1st ed. 2020. edition, 2020. ISBN 3-030-39568-5.
Xianhang Li, Zeyu Wang, and Cihang Xie. An inverse scaling law for clip training. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.). Advances in Neu- ral Information Processing Systems, volume 36, pp. 49068 49087. Curran Associates, Inc., http:///ppppppeeeeeeeee////////
2023a. URL
12

---

Published as a conference paper at ICLR 2026
Yanghao Li, Haoqi Fan, Ronghang Hu, Christoph Feichtenhofer, and Kaiming He. Scaling language-image pre-training via masking. In Proceedings of the IEEE/CVE Conference on Com- puter Vision and Pattern Recognition (CVPR), pp. 10900-2300,, June 2023b.
Chih-Jen Lin. Projected gradient methods for nonnegative matrix factorization. Neural Comput., 19(10):2756-2779 October 2007. ISSN 7667. doi: (0.1162/ecc02207.19..00
URL,
https: /oi. www.ooggg/....o. 1ww.111222/20 www.e////ncc....e 2007 .7.66.77
Tianyi Lin, Chi Jin, and Michael I. Jordan. Two-timescale gradient descent ascent algorithms for nonconvex minimax optimization. Journal of Machine Learning Research, (6(11):1-44 2025.
URL http: fmtrr//mmmr.. www.roorg/pppae /ptppraapr/////////eee/eeaa/eeee /
Chaoyue Liu, Dmitriy Drusvyatskiy, Misha Belkin, Damek Davis, and Yian Ma. Aiming to- wards the minimizers: fast convergence of sgd for overparametrized problems. In A. Oh, T. Naumann, A. Globerson, K. Saenko. M. Hardt, and S. Levine (eds.), Advances in Neu- ral Information Processing Systems, volume 36, pp. neurips cc/ ccpppper.ppppppccppppp filess/paper/2033 2023. URL. http:t/////rrreeeeossss.eeeereep
Curran Associates, Inc.,
7 99ec 3ttp::/22......1......a..a333333
Iya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer-
ence on Learning Representations, 2019. URL httpsss//ppeeeeeeeee..oo http:///nwevee...eetttrururaaaaa
Bkg6RiCqY?
Maher Nouiehed, Maziar Sanjabi, Tianjian Huang, Jason D Lee, and Meisam Razaviyayn. Solving a class non-convex min-max games using iterative first order methods. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL httpss://prooeeedings.sct neurip....uuuusrcoucos..ss ccppaper files//aperr20019/ pdf
Avik Pal, Max van Spengler, Guido Maria D' A'mell di Melendugno, Alessandro Flaborea, Fabio Galasso, and Pascal Mettes. Compositional entailment learning for hyperbolic vision-language models. In The Thirteenth International Conference on Learning Representations, 2025. URL
littps://one http:s///penreview... net forum?id=3111 31e32v
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary De Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Bun E. Fox, and R. Garnett (eds.). Advances in Neural Information Processing Systems, volume 32, Cur- ran Associates, Inc., 2019. URL httpttss:://///m http://ps./ppr.ceaigg......o wwwp:::ss...nuurr........ees file http:///w.....ooooo..//////./a2.
files/
Qi Qian and Juhua Hu. Online zero-shot classification with clip. In Aleš Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and Gül Varol (eds.), Computer Vision ECCV 2024, pp. 462-477, Cham, 2024. Springer Nature Switzerland. ISBN 978-3-031-72280-5.
Zi-Hao Qiu, Quanqi Hu, Zhuoning Yuan, Denny Zhou, Lijun Zhang, and Tianbao Yang. Not all se- mantics created equal: Contrastive self-supervised learning with automatic temperature indi- vidualization. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference Ma- chine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 28421. mlr. wtt...ppres///222/2222222222
PMLR, 23-29 Jul 2023. URL Intps:: pttps::/proceediin
html
Zi-Hao Qiu, Siqi Guo, Mao Xu, Tuo Zhao, Lijun Zhang, and Tianbao Yang. To cool or not to cool? temperature network meets large foundation models via DRO. In Forty-first International Conference on Machine Learning, 2024. URL YT.:
net hwt.:orumteeeeeeoeoo ffo.mmioooii
13

---

Published as a conference paper at ICLR 2026
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 7748-8763. PMLR, 18-24 Jul 2021. html
URL sttpss:/proceeddigg www.a/iinnnmmm...e........... wtw..p.ee./vvii..... ra11i3//ra
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do ImageNet classifiers generalize to ImageNet? In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.). Proceedings the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 5389-5-80 9400. PMLRR, 09-15 Jun 2019. URL https: htttps://proceetee ngs.
m www.ppewsssv/iemmm recht 1 9a. html
R Tyrrell Rockafellar and Roger J-B Wets. Variational analysis, volume 317. Springer Science & Business Media, 2009.
R.T. Rockafellar. Conver Analysis. Princeton Landmarks in Mathematics and Physics. Prince- ton University Press, 1997. ISBN 9780691015866. URL https::/books...ootttt hwtp:/bboooo..ool....... com/
books2i
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278 222294 2022.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypemymed, image alt-text dataset for automatic image captioning. In Proceedings ACL, 2018.
Maohao Shen, Subhro Das, Kristjan Greenewald, Prasanna Sattigeri, Gregory W. Wornell, and Soumya Ghosh. Thermometer: Towards universal calibration for large language models. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scar- lett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 44687-44711 4.,.,.,MRR., PMLR, 21-27 Jul 2024. BtLhttts:: http:///cccccceee.s.eeom www.mlr...p.. www.pprees...... v235///v235///// htt.:/5555hee22cc.m html
Junhao Su, Changpeng Cai, Feiyu Zhu, Chenghao He, Xiaojie Xu, Dongzii Guan, and Chenyang Si. Momentum auxiliary network for supervised local learning. In Ales Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and Güil Varol (eds.), Computer Vision ECCV 2024, pp. 276-292, Cham, 2025. Springer Nature Switzerland. ISBN 978-3-031-77261-3.
Haotian Sun, Yitong Li, Yuchen Zhuang, Niao He, Hanjun Dai, and Bo Dai. Amorlip: Efficient language-image pretraining via amortization. arXiv preprint arXiv:2505. 10255555.188... 18983, 2025.
Pavan Kumar Anasosalu Vasu, Hadi Pouransari, Fartash Faghri, Raviteja Vemulapalli, and Oncel Tuzel. Mobileclip: Fast image-text models through multi-modee reinforced training. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15963-15974, June 2024.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention all you need. I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus. S. Vishwanathan, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https::/prroeediigg..coo o/t.://ceee.ea.......iie....pppe
files/paper/2017.
File/ FLL12E1222251111111003sssaaaaa22 Paper Pappr..
Bokun Wang, Yunwen Lei, Yiming Ying, and Tianbao Yang. On discriminative probabilistic modeling for sulf-supe representation learning. In The Thirteenth International Confer- ence on Learning Representations, 2025a. URL
net/ forum?id
s1 515grgcrr
14

---

Published as a conference paper at ICLR 2026
Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representations by penalizing local predictive power. In H. Wallach, H. Larochelle, A. Beygelzimer. F. d'Alché-Bue, E. Fox, and R. Gamett (eds.), Advances in Neu- ral Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL httpss://proceedinng hwurtiiinnnnnnnu.oo.o... neurips cc/ / Beetccb067ee6155222355222229---3
pdf
Xiao Wang, Ibrahim Alabdulmohsin, Daniel Salz, Zhe Li, Keran Rong, and Xiaohua Zhai. Scaling pre-training to one hundred billion data for vision language models. arXiv preprint arXiv:2502.07617, 2025b.
Yiping Wang, Yifang Chen, Wendan Yan, Alex Fang. Wenjing Zhou, Kevin Jamieson, and Simon Shaolei Du. Cliploss and norm-based data selection methods for mul- timodal contrastive learning. In A. Globerson, L.. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang (eds.), Advances in Neural Information Pro- cessing Systems, volume 37, pp. 15028-15069. Curran Associates, Inc., 2024. URL
neur Soreaaaeeeeeccaaaaaaaoeooooooeer
Xiyuan Wei, Fanjiang Ye, Ori Yonay, Xingyu Chen, Baixi Sun, Dingwen Tao, and Tianbao Yang- Fastelip: A suite of optimization techniques to accelerate clip training with limited resources. arXiv preprint arXiv:2407.0144: 2024.
Xiyuan Wei, Ming Lin, Fanjiang Ye, Fengguang Song, Liangliang Cao, My T. Thai, and Tian- bao Yang. Model steering: Learning with reference model improves generalization bounds and scaling laws. In Forty-second International Conference on Machine Learning, 2025. URL
net forummiduiioooooooo -tt.://ewwe..eettooooe-e------- Fo
Hu Xu, Saining Xie, Xiaoqing Tan, Po- Yao Huang, Russell Howes, Vasu Sharma, Shang-Wor Wen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Demystifying CLIP data. In The Twelfth International Conference on Learning Representations, 2024. URL https:
7 /ooenzeview.
infElg
Peter Young. Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78, 2014.
Zhuoning Yuan, Yan Yan, Rong Jin, and Tianbao Yang. Stagewise training accelerates convergence of testing error over sgd. Advances in Neural Information Processing Systems, 32, 2019.
Zhuoning Yuan, Yuexin Wu, Zi-Hao Qiu, Xianzhi Du, Lijun Zhang, Denny Zhou, and Tianbao Yang. Provable stochastic optimization for global contrastive learning: Small batch does not harm performance. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 25760 25782. PMLR, 17-23 Jul 2022. URL htt.://ee/pppeeeeere...eeeeeeees
mlr www.ppressss/ie
yuan html
Zhixiong Zeng and Wenji Mao. A comprehensive empirical study of vision- language pre-trained model for supervised cross-modal retrieval. arXiv preprint wwxXv:2201.002777 2022.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEFF/CVF international conference on computer vision, pp. 11975-1 1986, 2023.
15

---

Published as a conference paper at ICLR 2026
A TECHNICAL DETAILS
A.1 DERIVATION OF OUR NEW OBJECTIVE AND THE OPTIMAL SOLUTION
When f(-) log(-), we have
("( = max x f(x) = maxy x log(z).
2
x
From the first-order optimality condition we know y + 1/2" which is -1/y. Substituting the optimal solution of z into the definition of (1)) we get (1) log( (og(-y) - 1.. Since log(-) a convex function, we have
f(x) = f (z) = max y- f- max r y log( (2)( + 1.
Plugging in x s 91 (w, T, S) into the above equation, get F(w, 7; o;); log(e + gi T))
(- log(s= 91 (ww, (w,,;;,,))
max (s gi (w, -;;; + log( (y) + 13
min miin--y-- (5 ( S)) - log( -y) 1}
min min (exp(-epppppp (e (w :; S)) + +a- 1},
where the last equality uses change of variable - log(-y). This completes the derivation of the new objective. Moreover, to derive the optimal solution of a, we can leverage the first-order optimality:
exp(-a") (E + (Iww,,;,,)) + 0,
which gives a" log(s + (ww,
A.2 INDUCING THE u UPDATE OF SOGCLR FROM EQUATION
Indeed, the sequence update in Equation 3 and the gradient estimator of w in Equation 4 can be derived for optimizing Equation (6. illustrate this, we consider a stochastic block mirror descent
update of J1,+ exp(-a1,1) and U2 exp(-axp a22x) for (Ix, 24) 5 at the t-th iteration:
(++1) arg min U1 (e + g1 (un ,,, ,0,,0 (()) log(y1,)) + D(n v..).
V1.i
91.4
"7
(I+1) arg min U2 (e + (t) ,10;1 B()) log(y2,))
D(y2,s, (1
22.
92.1
17
where D(u, v) = - (og(u)++og(l)) + (u v) is the Bregman divergence induced by - log(-) (also known as the Itakura Saito distance), and r > hyperparameter. We can derive closed-form ((-1) [(++]] as follows. By the first-order optimality condition, we have
updates प|. ,V2,
+ (1) (0) (,)
1
= 0.
(++1)
(t+1)
(8)
Vi.5
17
V1,i
01.5
Rearranging the terms, we get
"
(e (i (,, B()))
(++))
I+1
1
which can be mapped to Equation with 1 = 1/ and 7 = 1/(11 + +1). With (1+1). ((+1).
vi.5 992,
the stochastic gradient of Equation 6 w.r.t (4) is given by
T (1).
B(O
(1+1) 31
V
9 (w 7 (t), (
B
F2,1
((+1)
v (t) (,
(EBO)
iEB
which is exactly in Equation
with (1)
u
1/
(1)
16

---

Published as a conference paper at ICLR 2026
Algorithm 2: The FastCLIP Algorithm
Input: Model w (0). temperature 7!0, estimators u1, Ug dataset S, number of iterations T
for t = O,..., T
do
Randomly sample mini-batch BtO) C:;; (t++1)
Update ui,i (++11 and 22,1 using Equation for i B());
Set ui [++1] "i and u2, u2 for & B(0;
(t+1) (e)
Compute gradient estimators for w(t), 7(1) using Equations and 13). respectively
log(u1, ((+11) +
log(us ((+1)) + 2p
B
iEB())
AEB())
+r¹
V V. (w (1) ,10);, )())
|B
[++1)
(13)
iEB()
1,a
1
+ (t)
V (2 (())).
[B0
(++1
iEB()
22
Update w (t+1) and (t-1) using optimizer;
Table 3: Hyperparameters for training the CLIP model.
Dataset
Ir
Ir of
wd
warmup
p
CC3M
le-3
1.25e-4
0.1
10,000
6.5
CC12M
4e-4
5e-5
0.1
10,000
8.5
DFN-14M
5e-4
6.25e-5
0.2
500
11.0
DFN-192M
3.125e-4
3.9e-5
0.2
500
11.0
DFN-1B
3.125e-4
3.9e-5
0.2
500
11.0
A.3 DETAILS OF THE FASTCLIP ALGORITHM
The FastCLIP algorithm (Wei et al. 2024 is presented in Algorithm
B ADDITIONAL EXPERIMENT RESULTS
B. DETAILS OF EXPERIMENT SETTINGS
Hy pyperparameters. In Table 3 present hyperparameters for training the CLIP model. The hyperparameters are selected according to Datacomp Average performance. For each dataset, we tune the learning rate (Ir) between le-4 and 1e-3, and tune the temperature parameter's Ir between 1/8 and of the model weights' Ir. We set the weight decay (wd) following previous works (Gadre al., 2023 Wei et al. 2024). We apply warmup to the Ir at the beginning of training, by linearly increasing the Ir from 0 to peak value across given number of iterations (as specified by the warmup hyperparamter). For CC3M and CC12M,
set the value of p following Wei al. 2024). For the DFN datasets, we tune between 8.0 and 13.0. In Table 4 we present hyperparameters for training the NPNs. We tune the learning rate between le-4 and 100.0 and the weight decay between 0.0 and 1.0. We tune the number of prototypes m between 1 1,024 and 8,192, the restart frequency T, between 0 (i.e., no restart) and 12,500, and the number of updates T, between and 50.
Table 4: Hyperparameters for training the NPNs.
Optimizer
Ir
wd
T,
T
AdaGrad 1.0
0.0
4,096
500
10
17

---

Published as a conference paper at ICLR 2026
Algorithm 3: Simple (Stochastic) Gradient-based Algorithm for Solving Equation (12
Input: Model w², Temperature 79, Prooootypss W W Dataset S, Number of iterations T
for O,..., T
1 do
Randomly sample mini-batch B(() C S;
Compute mini-batch gradient of 12) w.r.t w(t) ,((), w w
Update w(t+1), (((++),,((++1) W1 (1+1)
W (++1
with AdamW;
Simultaneous
Alternating
25.0
Datacomp Average
ImageNet & VVariants
25
22.5
15
Retrieval
20
20.0
10:
17.5
10
15.0
5
100
100
100
Samples Seen (M)
Samples Seen (M)
Samples Seen (M)
Figure 3: Comparison between simultaneous and alternating optimization.
B.2 DETAILS OF THE DATACOMP BENCHMARK
The Datacomp Benchmark (Gadre et al., 2023 consists of 38 tasks in total, including 35 zero-shot image classification tasks and zero-shot retrieval tasks. The performance metric for classification tasks is top-1 accuracy, and the performance metric for retrieval tasks is the average of image recall at and text recall at 1. The "ImageNet & Variants" subset consists of ImageNet-1K (Deng et al. 2009) and 6 distribution shift datasets (Wang al.] 2019 Recht al., 2019, Hendryycks et al. 2021a b: Barbu et al. 2211)) and the "Retrieval" subset consists of FlKckr30K (Young et al. 2014 and MSCOCO (Chen et 2015).
B.3 SIMULTANEOUS VS. ALTERNATING OPTIMIZATION
In this subsection, we provide comparison for two approaches solving Equation 12) The first approach is simple gradient-based algorithm that treats all parameters as a whole and updates them simultaneously. which is presented Algorithm The second one is our NeuCLIP algorithm (Algorithm[I) D). which optimizes the CLIP model and the NPNs in an alternating manner. We conduct experiments on CC3M and plot the performance of the two approaches in Figure 3 From results we can see that the joint optimization algorithm performs much worse than the alternating optimization algorithm.
B.4 TRAINING COST OF THE WWWAALIZER-PREICCIEOO NETWORK
In this subsection, we investigate the additional training cost incurred by the NPNs. We profile the running time of NeuCLIP with ResNet50, VIT-B/32 and V as the vision encoder respectively using the PyTorch (Paszke et al. 2019) Profiler. We present the results in Table from which we can observe that the additional cost of the NPNs remains low. In addition, profile the peak memory of NeuCLIP and OpenCLIP during training. The results, reported in Table 6 suggest that memory overhead of NeuCLIP is negligible.
B.5 COMPARISON WITH BASELINES
In this subsection, provide experiment results of different methods various datasets. In Tables 7 we present the evaluation results of different methods trained on CC3M to DFN-1B.
Variation in Performance of AmorLIP. On CC3M and CC12M, find that the results of our reproduction of AmorLIP is different from those reported in the AmorLIP paper. For example, their reported results of FastCLIP are lower than ours while that of AmorL are higher (c.f. Table 12.
18

---

Published as a conference paper at ICLR 2026
Table 5: Running time NeuCLIP under different settings. NPN Time denotes the training time of the NPNs in iteration. Total Time denotes the running time of one iteration. Overhead denotes the portion of NPN Time in terms of Total Time.
Vision Encoder
Embedding Dimension
NPN Time (ms)
Total Time (ms)
Overhead
ResNer50
1,024
49.23 1.10
529.09 + 21.79
9.30%
ViT-B/32
512
54.17 + 3.98
897.79 + 31.89
6.03%
ViT-B/16
512
56.43 2.83
944.41 + 29.40
5.98%
Table 6: Memory consumption of NeuCLIP under different settings. Memory denotes the peak memory of different methods during training. Overhead denotes the memory overhead of NeuCL.IP over OpenCLIP.
Vision Encoder
Embedding Dimension
Memory (MB)
Overhead
NeuCLIP
OpenCLIP
ResNet50
1,024
10,573.2 + 11.10
10,340.1 + 12.47
2.25%
ViT-B/32
512
20,721.0 + 23.98
20,550.0 + 23.47
0.83%
ViT-B/16
512
55,171.3 + 33.28
55,006.5 + 36.13
0.30%
Table 7: Evaluation results of different methods trained CC3M. We each method for 3 times with different random seeds, and report the average performance over training runs along with standard deviation.
Method
Datacomp Average
ImageNet & Variants
Retrieval
OpenCLIP
21.84 + 0.23
14.73 + 0.22
22.25 + 0.38
FastCLIP
24.74 + 0.35
19.09 + 0.20
29.56 + 0.25
SigLIP
22.19 + 0.11
16.08 = 0.16
22.13 = 0.52
AmorLIP
22.89 + 0.20
17.78 = 0.43
24.32 = 0.53
NeuCLIP
25.08 + 0.39
19.85 + 0.37
30.53 + 0.37
Table 8: Evaluation results of different methods trained on CC12M. We run each method for 3 times with different random seeds, and report the average performance over training runs along with standard deviation.
Method
Datacomp Average
ImageNet & Variants
Retrieval
OpenCLIP
27.91 +:2 0.77
21.21 + 0.04
26.94 + 0.59
FastCLIP
31.50 + 0.43
24.61 = 0.05
32.33 = 0.48
SigLIP
28.60 + 0.35
22.67 = 0.05
27.99 = 0.25
AmorLIP
29.86 + 0.83
23.48 + 0.62
28.97 + 0.77
NeuCLIP
31.89 + 0.15
25.09 + 0.12
32.93 + 0.16
Table 9: Evaluation results of different methods trained on DFN-14M. We run each method for times with different random seeds, and report the average performance over 3 training runs along with standard deviation.
Method
Datacomp Average
ImageNet & Variants
Retrieval
OpenCLIP
37.78 + 0.05
32.65 + 0.20
24.33 + 0.06
FastCLIP
38.45 + 0.06
33.03 = 0.06
25.15 = 0.11
SigLIP
37.23 + 0.10
32.89 + 0.17
23.97 + 0.20
AmorLIP
37.53 0.33
33.35 + 0.04
24.58 + 0.19
NeuCLIP
39.16 + 0.20
33.79 0.07
26.60 = 0.28
19

---

Published as a conference paper at ICLR 2026
Table 10: Evaluation results of different methods trained on DFN-192M.
Method
Datacomp Average
ImageNet & Variants
Retrieval
OpenCI.IP
54.58
55.
50.42
FastCLIP
54.72
53.44
50.53
SigLIP
54.26
55.09
50.49
AmorLIP
53.83
55.09 55.88
51.43
NeuCLIP
54.90
51.39
Table 11: Evaluation results of different methods trained on DFN-IB.
Method
Datacomp Average
ImageNet & Variants
Retrieval
OpenCLIP
56.25
57.49
55.27
FastCLIP
56.68
57.87
56.05
SigLIP
56.32
57.20
55.33
AmorL.IP
56.24
57.12
55.23
NeuCLIP
57.34
58.69
56.71
We would like note that the training datasets used by [Sun et al. (2025) and us are in fact differ- ent. This is because the two datasets are distributed with their metadata only, i.e. texts and links to their corresponding images. During the downloading process, not all images would be success- fully downloaded. Specifically, (Sun al. (2025) reported size of 2,274,566 samples for CC3M and 8,059,642 samples for CC12M, while our CC3M and CC12M datasets contain 2,723,840 and 9,187,328 samples, respectively. We tune the parameters of AmorLIP and still could not achieve the reported performance, so we believe the difference in performance is due to variation in datasets. On the other hand, if compare their reported results with the results of NeuCLIP, still observe an improvement of NeuCLIP over AmorLIP.
Comparison under Same Amount of Compute. In Table 13 we present the comparison between NeuCLIP and other baselines trained for the same amount o GPU hours. The main focus to offset the computation overhead of NPN update (c.f. Table
We assume OpenCLIP. FastCLIP and SigLIP consume the same amount of compute at each iteration since they only differ in loss computation from image and text features (though Wei et al. (202)) showed that FastCLIP is slightly faster than OpenCIIP). And we reduce the number of iterations of NeuCLIP by the overhead of NPN update shown in Table 5 so that the total amount of GPU hours of NcuCLIP matches that of OpenCLIP, FasICLIP and Sig From Table 13
observe that the performance of NeuCLIP slightly decreases compared with Table 2 but still outperforms other baselines. We did not provide results of AmorLIP, but due to the additional cost of updating the small network in AmorLIP, it is mostly likely that under the same amount of GPU hours, the performance of AmorLIP is lower than that in 2
B.6 ABLATION STUDY
In this subsection, present detailed results of the ablation study. In Tables 14 16 present evaluation results of different objectives and architectures on CC3M, CC12M and DFN-1 14M, re- spectively. In Tables 17 to 19 we present the ablation results DFN-14M of the restart frequency T, the number of updates Tu and the number of prototypes m, respectively. In Tables 20 and 21 present the ablation results on CC12M of the restart frequency T, and the number of prototypes
Table 12: Datacomp Average performance of FastCLIP and AmorLIP from Sun et al. 2025 and
our reproduction.
Method
Source
CC3M
CC12M
FastCLIP
Sun et al. (2025)
23.46
29.00
FastCLIP
Ours
24.74
31.50
AmorLIP
Sun et al. 2025
24.11
30.66
AmorLIP
Ours
22.89
29.86
20

---

Published as a conference paper at ICLR 2026
Comparison of Datacomp Average performance of NeuCLIP trained under the same amount of compute as baselines.
Table 13:
Method
CC3M
CC12M
DFN-14M
DFN-IB DFN-192M
OpenCLIP
21.84
27.91
37.78
54.58
56.25
FastCLIP
24.74
31.50
38.45
54.72
56.68
SigLIP
22.19
28.60
37.23
54.26
56.32
NeuCLIP 25.06
31.75
39.16
54.85
57.28
mi, respectively. In Figure we plot the estimation error of normalizers of different methods dif- ferent number of samples seen. In Figure 3. we plot the Datacomp Average performance of different restart frequency T, on CC12M.
Computation of the Estimation Error of Normalizers, In order to obtain the estimation error of given model on given dataset, we first obtain the embeddings e1, e2,; for 2;, 2 in the whole dataset using the model, which is done by performing forward pass on all the images and texts. Then for given image a; and given model, its true normalizer computed using Equation (9) Similar procedure is applied for obtaining true normalizer for given text. Thus the true normalizer does not incur bias or variance. To obtain the estimation error a given model, we randomly sample 10K data points, and for each data point, compute its true normalizer and estimators from corresponding algorithms (OpenCLIP, FastCLIP and NeuCLIP).
Table 14: Ablation of training objective and model architecture CC3M.
Objective
Architecture
Datacomp Average
ImageNet & Variants
Retrieval
Unified
Our NPN
25.08
19.85
30.53
Unified
MLP
24.84
19.28
29.50
Separate
Our NPN
24.19
19.20
29.38
Separate
MILP
24.02
19.09
29.08
Table 15: Ablation of training objective and model architecture on CC12M.
Objective
Architecture
Datacomp Average
ImageNet & Variants
Retrieval
Unified
Our NPN
31.89
25,09
32.93
Unified
MLP
31.43
25.04
32.30
Separate
Our NPN
31.05
25.04
31.34
Separate
MLP
30.94
24.77
31.12
Table 16: Ablation of training objective and model architecture on DFN-14M.
Objective
Architecture
Datacomp Average
ImageNet & Variants
Retrieval
Unified
Our NPN
39.16
33.79
26.60
Unified
MLP
38.58
33.19
26.35
Separate
Our NPN
38.63
33.70
25.98
Separate
MLP
38.26
33.12
25.86
21

---

Published as a conference paper at ICLR 2026
Table 17: Ablation of restart frequency I, on DFN-I4M.
T,
Datacomp Average
ImageNet & Variants
Retrieval
20
38.41
33.14 14
26.30
100
38.54
33.37
26.05
500
39.16
33.79
26.60
2,500
39.06
33.29
26.25
00
38.48
33.18
25.79
Table 18: Ablation of number of updates T, on DFN-14M.
Datacomp Average
ImageNet Variants
Retrieval
39.02
33.50
26.65
5
39.10
33.60
26.48
10
39.16
33.79
26.60
20
38.68
33.57
26.05
50
38.03
33.35
26.46
Table 19: Ablation of number of prototypes m on DFN-I4M.
m
Datacomp Average
ImageNett Variants
Retrieval
1,024
38.57
33.38
25.92
2,048
38.56
33.73
26.28
4,096
39.16
33.79
26.60
8,192
39.25
33.90
26.28
Table 20: Ablation of restart frequency T, on CC12M.
T,
Datacomp Average
ImageNet & Variants
Retrieval
20
31.59
24.76
32.74
100
31.64
24.96
32.80
500
31.89
25.09
32.93
2500
31.47
24.88
32.65
00
31.22
24.55
32.32
Table 21: Ablation of number of prototypes m on CC12M.
m
Datacomp Average
ImageNet & Variants
Retrieval
1024
31.52
24.73
32.12
2048
31.57
25,00
32.83
4096
31.89
25.09
32,93
8192
32.12
25.24
32.85
22

---

Published as a conference paper at ICLR 2026
Algorithm 4: Simplified Version of the NeuCLIP Algorithm for Analysis
Input: Initial point (x("), ()) step sizes 7) 72, momentum parameter 3, initial momentum (00 number of iterations T, number of updates K. for t ................................
do
Set y (t.0) (*);
for == 0, ..... K do
Randomly sample g(t,k);
Update y ((,,+1) (t. 1/2 (2)(()()) ( c(i,)));
Set ( ((,.K);
Randomly sample g(t);
Update (1++-)) = (1 - () + (()))
Update (++1) (t) 1/1 ((+))
OpenCLPP
FastCLIP
NeuCLIP
1024
512
512
Subset
DFN-14M
80
Error
20
0
25
50 75
100
100
200
300
0
25 50
75
100
Samples Seen
Samples Seen (M)
Samples Seen (%)
(b)
(c)
Figure 4: (a): Estimation error of normalizers of different methods with batch size 1,024 (solid lines) or 512 (dashed lines) on CC3M. (b): Estimation error of different methods with batch size 4,096 (solid lines) or 512 (dashed lines) on DFN-14M. (c): Estimation error of different methods on subset of DFN-14M (solid lines) DFN- 14M (dashed lines).
3.20
31.89
patacomp Avera
31.8
31.4
31.22
31.2
0.02
0.1
T, (x1000)
Figure 5: Ablation study restart frequency NPNs on CC12M.
C CONVERGENCE ANALYSIS OF NEUCLIP
In this section, we provide the convergence analysis of Algorithm the proof in Guo et al. (025)).
The analysis mainly follows
Notations:
We let denote the Euclidean norm. We use V and V2 denote the gradient w.r.t. the first and second argument of a function, respectively. We use Ex[-] to denote the expectation w.r.t all the randomness up to iteration and
E[-] to denote the total expectation.
We cast the objective in Equation [12 as a function of two blocks of variables
f(x 3) == [f y.
where x = (w, T). y = (W, W2), and denotes random sample. Then Equation 12
becomes
minn.y y fxx, y). We would like show that converges e-stationary point of the function
23

---

Published as a conference paper at ICLR 2026
F(x) := miny (z, y), i... + E B||VF(2"0111 < E² after T iterations of optimization.
Instead of directly analysing Algorithm| Algorithm foor ease of analysis:
we analyze Algorithm 4 which a slight modification of !
In Algorithm 4 we only consider using one sample to compute the stochastic gradients for both a and y. instead of using mini-batches in Algorithm] T
Algorithm we use Momentum SGD optimizer for a..
We would like to emphasize that the analysis is not the main contribution of this work, and the analysis mainly serves as theoretical justification of the proposed NeuCLIP method. To derive the convergence of Algorithm 4 we need the following assumptions.
Assumption 1. The following conditions hold:
(a) For any given æ, y. let y (ar, y) E arg miny fæx y') be the optimal solution closest to y.
It satisfies
y (x, y), V2f v)) l D" (æ, y)ll.
(b) Vif(a, y) is Ln Lipschitz continuous in æ and L continuous in y, i..,
|V y) v Vif(=', vl s Lullæ æ'1l + L y y'll.
Vx, x', y. y
(c) V2f(2, y) is L2 Lipschitz continuous in æ and L22 Lipschitz continuous in i.c., |22((2 y) 22f(2', b')ll = Le || x'1l + Lzz y y'll.
Yx, x', y y'
(d) There exist 02 such that the stochastic gradients have bounded variance, i.e.,
V v.E) Vif(=z y) | of. Eg V2/ V2/(0, r) S o². Væ, y.
(e) 1" ming F(x)
Remark 1. Assumptions [ [e) are standard conditions for analyzing algorithms that optimize
non-convex problems with two blocks of variables (Guo et al. 2025 Lin et al. (0025.. When the models are updated in a bounded domain and have smooth activation functions, Assumptions [ and [ will hold. Assumption
a is a mild condition that has been shown to hold for wide neural
networks (Liu al. 2023
With the above assumptions, get the following convergence results.
Theorem 2. Under Assumption let LF == L11 +
hallaatte and setting
B 50
min
T.2
801,2
21 11
B
13/2 1//21122
1/2
(1 1 )
1/1 min
2LF
4LF
8V21, (T12 + T221)
8v
T max
10 (0(F((x1))) (x:
F
5
E
FF
VF
40L 12
EE
|10000
3
(x (0)
Be2
s2
then after T iterations of Algorithm
we have
T-1
Σ
+ + 10001p/]
1 S
Corollary 1. Under the same assumptions as Theorem 2 achieve an e-stationary point, i.e., + -I-1 E [[ < s2, the total number of stochastic gradient evaluations of f(x y E)
(s-11).
24

---

Published as a conference paper at ICLR 2026
Proof. Ignoring the constants, we know that I is in the order 0 0 Bea 0
( ((
whichever
m ). 1/2 = O and 7 = OEe")
is the largest. The conditions of Theorem 2 imply that 3 = O(=*),
Then we know T Ole Setting k O(1), we know that the total number of stochastic gradient evaluations is O(E
Lemma 1. Under Assumptions a and c the function F(x) is differentiable with VF(x) VIf(æ, y (æ)), where (x) is any solution in arg miny f(x, y¹). And the gradient V FF()) is
LF = L11 + Lalln+La Lipschitz continuous.
Proof. From Yuan al. (2019 Lemma 9), we know that under Assumptions a and [c] the function f(x,,) . satisfies / PL. condition for any given æ. According to Nouiehed et al. (2019)
Lag
Lemma A.5), we know that F(x) is differentiable with (F
Vif y (z)), and its gradient
Lipschitz continuous with constant LE L11 +
La + Lm) This completes the proof.
/
Lemma 2. Under Assumptions a to (d) for any given a. xt and y² (x:) arg min,, ( v).
there exists y² (a') E arg min, f( y) such that
Lea L21) | w'1. ||-
(1'(x) y' (x')l
pb
Proof. The proof directly follows from (Nuuiehed et al.
2019
Lemma A.3) and the fact that f(x, :)
satisfies | -PL condition for any given x Yuan et al.
Les
2019
Lemma 9).
Lemma 3. Under Assumptions 1 b
and
(c)
for mn
S ZLF we have
((z()+1)) < (0)
+
7h 2
2 (((1)) (F()
1|²
11
722 / ?
(14)
2
4
Proof. From the -mosohhhssss of F' we have
((x((++1) {F(x + (V)((((")) (0(11))) (0) + LE (b(t)))) - - 1|
2
=f((()) - (v)( v((t1), (e+1) - (0))) + (6((t1))20((+1))) - (1)))
LF (1+1) - (0)) 2
+ 2
=F( 711 (VF(x - (1(+1)) (0(1)),(1)))) x((+1))) - 1/1 -. ) /olct.u//
=F((o(") + vF(z)(1) - b -m (v)()(1)) - ((t+1), vF(z(())
-(m
mm LFUR |o(tta)
Note that we have
(V)( ((+1) VF(x (1 - + { (1 {
Thus we get
((11) (1) + + - - 17
o((t1)/2
2
(((()()) - w/ctu112 +
[)) /|
{) + マ/ 2
()(x - , ((+
7 2
2
where the last inequality is due Lrni
2 < zit by choosing 'n
1
This completes the proof. 1
ZLF
25

---

Published as a conference paper at ICLR 2026
Lemma 4. Under Assumptions arg miny f (xl(++), y²), then we have
[
1 [dd
let y² (x(1-11) be any given optimal solution in
Ex+1 || | - -VF(a(tt)) 5(1 (1--3)) B) -
2L3 1= +
1
8
(15)
+ +431% (y(tt)) (104))-8((0 )((
+ 0.
Proof. We have
1 ((+2) -V
(F(x
E441
())((tt)) 1 + a
(())))) (itt ((n-)''raff Waa (itii
(1- 3io
E141
 (etti) Suijii Ah A + ( - Segltae
اللويو -
Ex+1
االl اlltia (1+,)))+++))))))
(+)))
(ltttt (1+))²) Aa + ((+)))
= =1 3) ( A
|| | (e ((t)))
+ ( (.ll (ill ((t))a(t++))))) Keiigen
M -
((th)) (
+ +88
1.+1 [11- 3) (
/
Vit Sestttee ((+))) (4-1).(0 ((+)))
-F(e)(t)  + AA + ( "((+ 
(11 -1-3))((++) -
+ E || [[ ((((+)))))(1 y(++l) ((+t)) Vi   [
$ (1 -3) (F(x((+1) + (V yلtt)) + 3 io
where the last equality comes from the fact that
E+11 [(0 - -V(o -- (ttt( ((tt))))()))) ((t1))),
Vif(2((+11, ((tt)) ((+t),)(++)))) - y ((D)))
=(1- - (++ - (F(z((+1)) + pV vV
((+)) y(tl))
E+t1 (1 y(++)) elt+l)) ((f(2((+11) ((+1)))) = 0.
26

---

Published as a conference paper at ICLR 2026
and the last inequality is due to Assumption [
Furthermore, have
(1 ()0((11) V VF((x(())) / (x( ((t1)) (1-5)(((tt VF (1- +
() -VP(((
+8(Vi/(2 , ((+tt)) VF( 2
<(1-8)p(1 - 3) - 2
+ 1+ 1
8 (1- (V=((())(> vF(x + 3(V ((V1 ,
y(tt) - (F(ee(())) ?
<(1 - 8) o((t1) )F( /
2(1 + 31 -3)²
( -VF(((t) ?
2(1 + 3),3²
, - [F(2((+1))) 2
+ 3
<(1 - 8) - (
222
0 (1(+1) 2 +3 V ((f((())11 gltt))) v (((
B
where the first inequality is due the Young's inequality ||a b112 < (1 c)||a|1 (1 +
1/c)]lbl?, vall, b and c > 0, and the last inequality is due to (1 - 3) + 8) I 1 and the Lip-
schitz continuity of (=)) (Lemmal) I and the update rule of æ. Thus we get
B++1
(((+2) ( |0]
<(1 <(1- 3)
[0(t2)) -r(a))) +
2
1 o((-1)
B
+ 48 V (10 (((t1)) - (V(( /|²
+ 82 of
[((4)) (112)) VF(= + 263.
<(1 8)
B
+ +431%
(((+ 3" (2(())))
|
2 + 82 oz
where the last inequality comes from Assumption lbb) and the fact that (F(x(t+)) (=((t)))1), y² (x(++))) for any y (x(t+1)) arg miny (x((11)) y²). This completes the proof.
Lemma 5. Under Assumption i for 1/2 S min Lgg 1. ler yery" pory"xx' (xx()) be any given optimal solution in arg min,, f(x(1), yy), then there exists y (x(++)) E arg min, f(x(t++1) y²) such that
E,
(et))
3 (c(((+1))
(1)12 K (1) --- (1))
+ AL3g(L1 + 221) E,
<
(16)
2
113122
2(1 - (1 ) K 21l o²
27

---

Published as a conference paper at ICLR 2026
Proof. Let yy (x. 3) E mrg f(x, y²) be the optimal solution closest to y. We have
(((ttt))) - (xx!) y ( |1] - Ee,t
|| | (y(Att)
3" (((""") y (())))
Ec,
(tab)) 1/2 (z) , (/(()) ,(c,))) )0(("") (x)()) (
E
E
3 (1,)) 72 22 (9, ,y((k)) - g"(o(") y)((.)) + m2 2 fcx(2= (,000) ((A))
1/ (x (1) y) k1h gl(kk) [
= =||o((A)
72 (2((xx()) (1) (((( - vz(( (y((k)) /
+ m/ E,, || [2f(E()) , ((())) v2f(x!!) , ,((,b)) |ll
((y((it)) 1/2 V2) , y((a))) - 7" v(("! y y(an)) Vel(a! ((th))) - v2((2. (m (((A))))
+ 2012 Be,
- -1y((()) 7/2 (2) (((h)) - )'(((() y ((,i))]
+ n2 L.,, [[
()) y (( (2)( (((.k) [9]
< 12
(() g"(x0) 2y (() + n o
(17)
where the first inequality holds since g' (x)("), ("x(") (((++)))) is the optimal solution closest to y (f.++))) the last equality is due to the fact that
12 (z)0 y - g (z) y V2f(x10) (0(("",y((t))) ((i)) - (2f(2!)) y(ch), c(())))) (1) 20 ((hh)) 8 (x) (,0l)) ((.))) (t)) 12 (2)
Ey (3 (f.2)
E., V2 ( y (()))) (a)(x
()) 3 (1,)) ((())))) = O,
and the last inequality is due to Assumption [ Furthermore, we have
(t,b)) 72 (x) , )((,k)
(((((") , y((A)))
() 72 22 (x)
y ((())))
(x 2 ((()) + 12 22f "'""""""""""" (='"),(1)A)))) ()(,,)))]] |
(x) y (100) 2
||
+ - (2)()) ((())) v f g (2) (((
20z (t)) - ÿJ (x) (((())y((()))) (((,))), / (z. (t) ((()))))
$ (1)))))) (1((1
g + m
V2l(2 (t), ((t))) (2)(1 y (2x(1) y((t)))
20012 / (((0))))))))))) |
<(1 + 1222122 - 24umn) (p(e))) E (2)()) ( (()ll 2 (1 - mm7) ((( -x(((())) ( (()))))
||?
(18)
where the first equality is due to the fact that (x)(x((), D (x(()), (((+))))= O. the first inequality is due to Assumption [a] the second inequality is due to the Lig-Lipschitz continuity of (2f(x..) and the last inequality comes from the choices of n22 S Laz and Equation (18). we get E y ((++))) 5" (x) , ((RDD)/ [19] <1 - -212) Ee [ I- (2) y ((.0))1
and n2 V 1 Combining Equation (17
+ n2 o3.
28

---

Published as a conference paper at ICLR 2026
Telescoping over &o = O..... K 1, and noting that y -y((10) ,y(+t)) =y((,k), we have
E. (1-1) D (x y (+11)) // ||] - (1-wmn))*
|
yy()) 8" (x((1) 8 (0)) 1|= (1- (1 pm2) (m 02
(19)
From Lemma
2 we know there exists y (xx(x)1)) min, f fax (1t),,,,, y) such that
I1 (xx y" (cx( Lee(b12 + L21) ||x!(1) 11
M
By considering this y (zx(+++))) we have
y (c0( [
E.
y (++1)
1 (++1) (x((" y ((t + 8" (x)")) ,y(tt)) y (x (x((t))) |
E+
y (++1) (1)11 y ((++)) )(ctee
147722
+
2
E1 |105 (x y (++1)) y ( (1
1 + H1712
(3/0) (0 /|1 + (1 (1 p1mp)) 1 ym ož
(1 (1- (272)
2
2
L32 (L12 + -221)2
|0010 x |3
B2
122
1u7/2)
(1) y "0(0
4132(Tn1
122 n2n 1a1112
E, |
2(1 (1 1m21 M2 of
where the first inequality is due to Young's ss inequality ||a+b|22 < (1+c) (+tc)lo (1+1//++1//]] Va, and > O, the second inequality comes from Equation (1)) and Lemma 2 and the last inequality is due to the fact that ÿ" (x((1) (()) is cloeset to y(!), the update rule of the choice that 1/2 S 1 and the fact that
(1 + 1912
(1 -mm))
( ( 1112
(1 1mm) -1m12)
K-1
1 p12
(1 -022)=11 S 5(1
10722 K
2
This completes the proof.
Next we present the proof of Theorem 2
Proof of Theorem 2
Multiplying Equation 15 by =
J 2/3
Equation (66 by b =
2m L.?
and adding them to Equation
we have
((x((+11) + a E,41
[1(+2)) VF
[[]]
bb
(1)(tt)) y² (2(
(F)
+ ?11
2 [((tt)) -VF(2 ||
71
VF (20)
11?
111
|
4
+ +a(1 - 3) p((+1)) -FF ||²
2L7
(0(0))
1
(2(((+))) ?
+
8
1 K (g)00) 34 (0(200))
1|?
4L32
L2i Pnp
|0/0|2
+b
+
2
1232
2(1 (1 1112) K )012 03
+ +a,9 of b
29

---

Published as a conference paper at ICLR 2026
Taking total expectation on both sides and rearranging the terms, we have
|| | -VFaa(t>)) + (b 40BL72) E | 2 (z(((())) ||]
E| E[F(x((tD)) + a
[)] + a(1 - -8)+ + ") TE | V
1 ( 19
b (1
1072)
K E
(1()) - y (
2
!1 E
||
+ (a
1
4L32 L12 + L221) )2p
E [|ocoo]]]
b
2
3
p2:122
3 of +6 b 2(1 (1- )m1)) K
11 02
With the choices of a, b, we have
a(l- -1 3) + マタ
1
1 +
1
7n- =
2/ m = a,
2B
2
2
b- 40 = b 2 = b (1 - (1 - (1
11712 xk)) = b (1- 117p2 K
2
2
Setting
r =- E(Fx(xx (1) ) + +aB |o(tt1) - -VF 1²] +(b- - 40.l E || 11,100- y" (z<1))) [1]]
we have
VF ||] 111
12 111
8 112 + 1122 )212
"/ r(++)) (T(()
4 +
2
32
p (1 (1 - u [k]
m 3
27/1 Li
2(1 - (1 - pm2)) K )11 o²
2
(1 (1-
2 )K
1
(20)
If
1. 1/2 (1 - (1 mn ) K)2/2
B
112
711
S
m S
4LF
8 8/21 (L12 + L21)
we have
12
711
8L12
L12 + L21 121) 1
7/1
<
B2
4
1 (1 (1 1072 K)
16
4
16
8
Plugging the above inequality into Equation 200 and rearranging the terms, we have
m
VF
11?
< (+)) ((+ 3B
4mp 21 - (1 p172))
) F 172 02
E
E
2
2
p(1 (1 2
2922 K
Telescoping over t
O...., T 1. and dividing both sides by m,, we have
T-1
)))
||=
+ 1 # 110000
< r()) r(T)
Bot
4T (1 (1 - p1971 K 1122 02
E
+
T
2
Tm
2
p(1 - (1 - uure )K)
t=0
Note that with our choices of a, b, we have a 2 0,b - 40,313 > 0, which implies T(T) > F. Then we get
(FF(2((0))
+ 4 |000|]/]
1
E
T
2(r(0) P
8T (1 - 11711) K ym oZ
+ +Bo
Tm
111 - (1 - 2 )k)
())
F*)
8T3 (1 (1 - (u7/2) (K )012 oz
+ +00
Tm
M(1 (1 1m2 )k)
2
1 1
v (0) VF (0)) ||]
1 (1 - arz )k 4L E
2² (z (0)) 100
+ T B
E
+ T (1 (1 10g )k)
2
(0)
30

---

Published as a conference paper at ICLR 2026
Combining the conditions in Lemmas
and for
8²
1e²2 1 1
B
122 min
501
L22 80L ps
B
1
1/2
1/2 (1 (1 1922 )112
111 min
2
4LF
8v 8/2L LL12 + L21)
T = max
10(F(x(0) F*))
5
E
(0)))
VF(x
[1] ||]
40 E
||2000
(z)(0)
|
T71E2
A
pupez
we have
(()
- F*)
2²
E²
T 8 E -00) /F(0
Bo
Tmı
5
5
5
2 K 4
| (0) - y (a: 1//0
T (1 (1 - un )R)
1
4
(00 y (a 000 1|/
c²
(1 2il E
T
(1
5
Additionally, we can show that
1 - (1 11772) jk
22 21,
(21)
(1 ) K
and thus we have
8 (1 - (1 (ultl) K )1p2 o3
8²
11 - (1 - ) k
5"
To show Equation
21 note that the equality holds for K = 1. Assume the inequality holds for
some K, then for K we have
(1 pm1) K+1
(1 (1 - (m22) Σ kk=o(1 (1 - p17/2) )*
1 (1
1022
2
K+I
(1 (1 433 ))
M122 >k (1
=2 Σ K k (1- 11722)* )
2,
Σ 2k=0 1822 )
where the inequality comes from the fact that above inequalities, we get
2 0 and 1
pup 1 1
1922
Combining the 2
+ b S2
1
E
T
t=
This completes the proof.
D THE USE OF LARGE LANGUAGE MODELS (LLMS)
We use LLMs to help find recent applications of CLIP models, and to help search for works that leverage auxiliary networks in other fields than CLIP training, such Computer Vision and Natural Language Processing. We also use LLMS to help polish up writing.
31


Reviews:
[{"review_id": "dzhqz77f3B", "reviewer": "Reviewer_LKoG", "summary": "- Proposes NeuCLIP, a framework for efficient CLIP training by learning a neural normalizer that predicts log-normalization terms in contrastive loss.\n\n- Reformulates the contrastive loss via convex and variational analysis, turning per-sample normalizer estimation into learning a compact neural network.\n\n- Introduces an alternating optimization algorithm that jointly updates CLIP encoders and the normalizer-prediction network (NPN).\n\n- Demonstrates consistent improvements over OpenCLIP, FastCLIP, SigLIP, and AmorLIP on datasets from 3M to 1B pairs.", "questions": "- What is the per-step wall-clock and memory overhead compared to FastCLIP and OpenCLIP?\n\n- How sensitive are results to NPN hyperparameters?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is based on an elegant theoretical foundation combining convex conjugate and variational principles to remove per-sample normalizer tracking.\n\n- We see strong empirical validation - NeuCLIP consistently outperforms baselines on multiple datasets and scales favorably to billion-sample training.", "weaknesses": "- Some hyperparameters (restart frequency, number of prototypes m) seem tuned per dataset; robustness to such choices is not discussed.\n\n- It might be worth discussing the computational overhead of the extra NPN updates, restarts, and parameter sync cost to give practitioners more insights to use in reality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fikIaJvg2v", "reviewer": "Reviewer_erJT", "summary": "The paper tackles the difficulty of estimating the partition function (normalizer) in CLIP’s global contrastive loss without relying on very large batch sizes. The authors (1) use convex conjugacy to rewrite each per-sample log-normalizer as the solution of a simple one-dimensional minimization, and (2) replace the n per-sample variables by a compact “normalizer-prediction network” (NPN) learned jointly with the CLIP encoders through a single objective. They propose an alternating optimization routine with two accelerators: several NPN steps per encoder update and periodic re-initialization of NPN “prototypes.” On several CLIP training scales (millions to a billion samples), NeuCLIP reports better Datacomp averages than OpenCLIP, FastCLIP, SigLIP, and AmorLIP under fixed compute budgets.", "questions": "1. **Compute and speed**. At matched Datacomp accuracy, what is the wall-clock time and GPU-hours for NeuCLIP vs. FastCLIP and SigLIP?\n\n2. **Scaling to 1B+.** DFN-1B shows smaller gains than DFN-192M under the stated budget. Is this due to fewer total seen samples (1.0B vs. 1.3B) or to NPN capacity limits?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The convex-variational reformulation removes the reciprocal-of-estimator bias from mini-batch CLIP and avoids per-sample moving averages in FastCLIP. The unified loss couples encoder and NPN training without needing a separate consistency target for the normalizer.\n\n2. Alternating updates with multiple quick NPN steps and periodic NPN restarts is easy to implement and, per their appendix, gives better stability than simultaneous updates.", "weaknesses": "1 **Limited accounting of compute and wall-clock.** Results are reported “under the same budget” and with “8 × H100,” but the paper does not provide thorough wall-clock and energy numbers for NeuCLIP vs. strong baselines at equal accuracy.\n\n2. **Breadth of baselines and settings.**  SigLIP is included, but the study would benefit from (i) larger-batch SigLIP/OpenCLIP points at matched compute, and (ii) comparisons under stronger data filtering or with modern data recipes, since normalizer accuracy can interact with data quality. The paper notes some dataset mismatch with AmorLIP but does not fully normalize the comparison.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iby8hnms3z", "reviewer": "Reviewer_fUVb", "summary": "The paper proposes NeuCLIP, a training framework for CLIP that replaces per-sample moving-average normalizer estimates with a neural normalizer–prediction network (NPN) trained jointly with the encoders. The authors (1) use convex conjugate analysis to rewrite each per-sample contrastive loss as a minimization over an auxiliary log-normalizer variable, and (2) use a variational argument to replace the set of per-sample variables by a function class, then parameterize that function with compact networks operating on encoder embeddings. Training alternates between several fast NPN updates (with periodic proto-reinitialization) and encoder/temperature updates. Experiments on CC3M, CC12M, and DFN at 14M/192M/1B scales report higher Datacomp averages than OpenCLIP, FastCLIP, SigLIP, and AmorLIP under the authors’ setups; ablations examine unified vs. separate objectives, NPN architecture, restart frequency, and NPN update count, and profiling suggests single-digit percent overhead for the NPN step.", "questions": "1. How is the “true” (log) normalizer computed for the error analyses at scale, and what bias/variance does that estimator have relative to the exact full-dataset partition values?\n2. What are the peak memory and wall-clock contributions of the NPN as a function of prototype count m and embedding dimension, and how do these scale for larger backbones (e.g., ViT-L/H)? Please include per-step FLOPs and activation memory. \n3. Does the alternating schedule with Tu>1 and periodic re-initialization admit any convergence guarantee (e.g., to a stationary point) under realistic stochasticity, or can it cycle?** Empirical stability diagnostics would help. \n4. How sensitive are results to m and to the prototype initialization strategy? Would K-means/K-medoids over a streaming buffer of embeddings outperform random re-seeding? \n5. Please report Datacomp and normalizer-error curves to validate the claimed robustness to small batches. \n6. Does the NPN inadvertently learn to absorb temperature effects, and if so, how do you regularize to prevent degeneracy between τ and α(·)? \n7. Can you provide ablations isolating the two accelerations (multiple NPN updates vs. periodic restart) across datasets, and quantify which contributes most under fixed compute?\n8. Since dataset discrepancies materially affect AmorLIP/FastCLIP comparisons, will you release the exact URL lists and filtering scripts for each run so the community can reproduce your numbers byte-for-byte? \n9. Does the NPN introduce a biased gradient for the encoder parameters relative to the true global objective? If unbiasedness does not hold, can you bound or characterize the induced bias and its dependence on m, Tr, and Tu? \n10. How portable is the approach to multilingual CLIP, video-text, or retrieval-only training? Are any parts of the derivation tied to cosine similarity or image–text symmetry that would need adjustment? \n\n**If you address my concerns, I will consider raising my score.**", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The loss is reformulated via a Fenchel conjugate into a per-sample minimization whose optimizer equals the log-normalizer; a variational theorem is then invoked to justify searching over functions and learning an NPN. The paper provides the derivations and the induced FastCLIP update as a special case. \n\n\n2. Results span multiple datasets (millions→billions of pairs), report the Datacomp average plus subset scores, include ablations of hyperparameters (e.g., restart frequency and update count), and list batch sizes, samples processed, and optimizer choices. \n\n\n3. A profiler table reports the NPN adds about 6–9% per-iteration time across representative encoders, do", "weaknesses": "1. The paper notes dataset download discrepancies vs. AmorLIP (different numbers of successfully fetched samples) yet still compares scores; large-scale web datasets are sensitive to crawl state, which can materially shift results. The largest-scale runs report single numbers without variability, and some baselines rely on third-party code with possible configuration drift—together reducing the strength of “method X > method Y” claims. \n\n\n2. The estimation-error plots rely on a “true normalizer”; for global losses that sum over all negatives, computing (or approximating) this quantity exactly is costly and may itself introduce bias. The procedure for obtaining this target at scale is not fully detailed, which affects how to interpret the error curves. \n\n\n3. While block-coordinate convergence is cited for idealized alternating optimization, the implemented procedure adds multiple NPN steps per batch and periodic re-initializations. There is no formal convergence rate or stability guarantee for this specific schedule, making the method’s robustness to optimizer/hyperparameter choices theoretically unclear. \n\n\n4. Performance depends on the number of prototypes m, restart frequency, and update count; the paper shows drops when these deviate (e.g., too frequent restarts or too many NPN updates leading to batch-overfit), but provides limited guidance for transfer to different hardware/batch regimes. This can be a practical failure mode in low-resource or non-H100 settings. \n\n\n5. All main experiments use 8×H100 GPUs and large global batches (e.g., 4096–5120). Claims about efficiency versus large-batch training would be stronger with results under constrained hardware and strict parity of training budgets and data filters across methods; otherwise, improved scores might partly reflect schedule choices rather than the optimization change alone.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "13x51shboZ", "reviewer": "Reviewer_TYsU", "summary": "This paper looks into a fundamental issue in CLIP model training and its dependence on large batch size for good performance. It looks into existing techniques that are less dependent on large batch size but their normalization approximation for log-partition estimation has room for improvement. The authors proposes a normalizer prediction network (NPN). The training objective jointly learns the encoder and a small network that predicts the log-normalizer for image and text", "questions": "See weakness", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper has novel contribution and addresses an important issue in CLIP training. This is helpful for the research community specially when they are looking to train on a smaller budget\n2. The method beats others in Datacomp. Although in some cases (table 9, 10) the improvement in marginal", "weaknesses": "1. Probably needs some more in related work. For example https://arxiv.org/abs/2409.13079 and https://openreview.net/pdf?id=3i13Gev2hV and how this work is tackling a different aspect of image-text pretraining\n2. The paper has novel contribution but it is not clear how those improved results number actually help in retrieval. Probably some quantitative examples will be helpful\n3. Minor: typo in line 58, 60. SigCLIP should be SigLIP\n4. Having insight on the limitation of the proposed approach and how it can be improved further will be helpful", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)