AmorLIP: Efficient Language-Image Pretraining via Amortization
Authors: Haotian Sun
Yitong Li
Yuchen Zhuang
Niao He
Hanjun Dai
Bo Dai
Venue: NeurIPS 2025 poster ()
Keywords: Contrastive Learning
Language-Image Pretraining
Primary area: deep_learning

Abstract: Contrastive Language-Image Pretraining (CLIP) has demonstrated strong zero-shot performance across diverse downstream text-image tasks. Existing CLIP methods typically optimize a contrastive objective using negative samples drawn from each minibatch. To achieve robust representation learning, these methods require extremely large batch sizes and escalate computational demands to hundreds or even thousands of GPUs. Prior approaches to mitigate this issue often compromise downstream performance, prolong training duration, or face scalability challenges with very large datasets. To overcome these limitations, we propose AmorLIP, an efficient CLIP pretraining framework that amortizes expensive computations involved in contrastive learning through lightweight neural networks, which substantially improves training efficiency and performance. Leveraging insights from a spectral factorization of energy-based models, we introduce novel amortization objectives along with practical techniques to improve training stability. Extensive experiments across 38 downstream tasks demonstrate the superior zero-shot classification and retrieval capabilities of AmorLIP, consistently outperforming standard CLIP baselines with substantial relative improvements of up to 12.24%.

AMORLIP: Efficient Language-Image Pretraining via Amortization
Haotian Sun', Yitong Li', Yuchen Zhuang', Niao He Hanjun Dai Bo Dai
GGoorgi Institute of Technology *Swiss Federal Institute of Technology haotian.. sunggatech edu, bodai@cc. gatech. edu
PPrecurai
Abstract
Contrastive Language-Imad Pretraining (CLIP) has demonstrated strong zero-shot performance across diverse downstream text-image tasks. Existing CLIP meth- ods typically optimize a contrastive objective using negative samples drawn from each minibatch. To achieve robust representation learning, these methods require extremely large batch sizes and escalate computational demands to hundreds even thousands of GPUs. Prior approaches to mitigate this issue often compromise downstream performance, prolong training duration, or face scalability challenges with very large datasets. To overcome these limitations, we propose AMORLIP, efficient CLIP pretraining framework that amortizes expensive computations involved in contrastive learning through lightweight neural networks, which sub- stantially improves training efficiency and performance. Leveraging insights from a spectral factorization of energy-based models, we introduce novel amortization objectives along with practical techniques to improve training stability. Exten- sive experiments across 38 downstream tasks demonstrate the superior zero-shot classification and retrieval capabilities of AMORLIP, consistently outperforming standard CLIP baselines with substantial relative improvements of up to 12.24%.
Introduction
6
Contrastive language-image pretraining methods, such as CLIP [51 36) and ALIGN (38), have emerged as powerful paradigms for learning general- purpose lasion-language representations from large- scale image-text pairs sourced from the web. By optimizing contrastive objective, these approaches effectively align representations from image and text modalities within shared embedding space, thereby facilitating robust zero-shot transfer to di- verse downstream tasks, such image classification and modal retrieval [55
ImageNet Shifts
Retrieval
Overall
ingrovemen (%)
IN-R
IN-O
IN-1K
IN-K
IN/2
CnyN
INA
Fiek
coco
38 Tacks
Figure AMORLIP consistently delivers performance gain over CLIP across various tasks. The bar plot represents the absolute per- formance improvements (%) over CLIP [51] in ImageNet classification, retrieval, and overall across the 38 DataComp tasks 23
69
In practice, training CLIP models typically involves optimizing a ranking-based Noise Contrastive Esti- mation (NCE) objective [47 28, 22), where negative pairs are sampled from within each minibatch. This minibatch- based negative sampling inherently requires very large batch sizes (e.g., 32K samples or larger [36] to ensure sufficient diversity among negatives for effective representation learning. A limited number of negative samples can introduce noisy gradient estimates and result in slower convergence and suboptimal downstream task performance. Consequently, CL.IP-based models often require significant computational resources, typically involving hundreds or even thousands of GPUs
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

consisting of paired or TPUs [5T] 36,77 71,, thus severely limiting accessibility for practitioners with constrained resources. Moreover, the CLIP objective requires computing similarity scores between all combinations of samples within minibatches before evaluating the loss for each sample pair. This inherent dependency prevents parallel per-sample computations and further hinders training efficiency.
To mitigate these computational barriers, existing works have explored memory-ellicient techniques
such as unimodal pretraining [72
36], image masking and rescaling 43
22 63 42), and gradient
accumulation methods [T 36
12 16]. Though reducing memory consumption, these methods
typically compromise downstream performance or prolong training. Alternatively, recent approaches approximate a larger negative sample set via non con-parametric estimation with offline buffers [70] [50] 66 [4]. However, these approaches face scalability challenges, as maintaining buffers comparable to the entire training dataset becomes increasingly impractical when training with billions of samples.
We propose AMORLIP, an efficient CLIP pretraining framework that introduces amortization alle- viate the need for large sets of negative samples and significantly enhance training efficiency [2] 58) We reformulate the CLIP training from an energy-based model perspective and derive an efficient representation for the partition function using spectral factorization. Motivated by this formulation, AMORLIP employs lightweight neural networks amortize partition functions effectively. We optimize AMORLIP via two-stage process, alternating updates between lightweight amortiza- tion networks and backbone encoders. Through continuous amortization over rolling minibatches, AMORLIP progressively incorporates richer sample information across batches and enables efficient training with minimal overhead. Additionally, we introduce and thoroughly analyze two amortization objectives, accompanied by practical techniques to further enhance training stability and efficiency.
Extensive experiments conducted across 38 downstream tasks demonstrate the robust zero-shot classification and retrieval performance of AMORLIP, achieving substantial relative improvements of up to 12.24% over CLIP. As illustrated in Figure [ these performance gains are consistent across diverse evaluation settings. Furthermore, comprehensive ablation studies confirm the effectiveness of the proposed representation parameterization for partition functions and validate the impact of our training techniques.
We summarize main contributions follows: (1) We propose AMORLIP, an efficient CLIP pretraining framework that amortizes costly computations from CLIP learning via lightweight neural networks, which substantially enhances training efficiency and improves model performance. (2) Leveraging an efficient representation derived from a spectral factorization perspective, AMORLIP effectively approximates partition functions, thereby alleviating the large-batch requirement inherent in CLIP training. (3) Extensive empirical evaluations demonstrate that AMORLIP consistently and significantly outperforms existing CLIP-based methods across diverse downstream tasks.
2 Preliminaries
CLIP as energy-based learning In this section, we formulate the CLIP objective with energy- based model (EBM) learning. Given a dataset D = (
(1) uu
(u)t)) un }}
images (9) and corresponding textual descriptions uy we pretrain two modality-specifie encoders ())()) and (-)(-) ) to generate representations within a shared embedding space. Let (1r) E T and (ur) E [R denote the fz-normalized embeddings for images and texts. For notation simplicity, we let / {1, represent an arbitrary modality of either image (!) or text (1)). Given modality 1, we denote the complementary modality as l' A We also use u, up {ur, WT] to denote the corresponding input. Specifically, both encoders jointly optimized using the CLIP objective (5)
to align representations of matching pairs
while pushing apart representations of non-
matching pairs "1 .uf"). where j a i. Generally, the CLIP model fits the conditional distributions P(uluu). We adopt an energy-based parameterization of these conditional distributions as:
P (ullur) = TP (ui) exp
(reha (MI) T trr (ur) - log Zr (ur))
(1)
Zy (up) ET(w)) exp (7 (w)T dy (wx)) AI E (I,T),
(2)
where P..) denote certain negative sampling distributions from the dataset DD and T is a learnable temperature parameter commonly adopted in CLIP-like models [51 71 66); Zy (ur) is the partition

---

function to ensure P (24 (yy) is a valid distribution. The ranking- basedd NCE objective 47
28
29
employed by CLIP can be formulated as follows:
(NCE
2r W1
vy (up
Σ
log Σ exp
(uyy (u²)
4/y
(3)
IE(LTT
P(Om)(
Existing CLIP implementations [36] 51 typically adopt in-batch negative sampling by contrasting
Ui (6)
within each minibatch B C D. Notably, the NCE objective inherently requires large batch sizes ensure sufficiently diverse set of negative samples, thereby facilitating effective representation learning.
each positive pair against all sample combinations
Let p and q denote two probability distributions. Given convex function f R+ R satisfying f(1) = 0 and strict convexity around 1, the f-divergence between q and is defined as:
f-divergence
Dy (q.p) == Ep(x) [f [(m)
(4)
p(a)
which measures the discrepancy between the distributions and L. Many widely used divergences fall under this framework through specific choices of f(-). For instance, the Kullback-Leibler (KI.) divergence corresponds to f() t llog t, and the f(t) = 2 (tlog' t - (t + log 1
(JS) divergence corresponds to
3 AMORLIP: Efficient Amortizations for Partition Functions
In this section, we introduce AMORLIP, an efficient contrastive language-inen image learning framework that employs lightweight amortization of the partition functions. We briefly outline the proposed framework and defer some detailed derivations and proofs of the preceding statements to Appendix B
3.1 AMORLIP Framework
Despite its widespread adoption, the CLIP objective in Eq. 33) presents two significant challenges: i) Estimation bias: ÉNCE in Eq. (3) is estimated using only the limited number of negative samples from each minibatch, which potentially results in biased gradients, particularly in small-batch scenarios [1] 770. Consequently, CLIP models employing Eq. (3) require large batch sizes to achieve good contrastive learning performance. ii) Inter-sample dependency: The nonlinear logsumexp operation in Eq. (3) needs computation over all negative samples prior to evaluating the loss for each individual pair. This inherent anter-sample dependency prevents parallel computations of ÊNCE and restricts computational efficiency.
To address these challenges, reformulate the representation learning objective from EBM perspective. One straightforward approach is Maximum Likelihood Estimation (MLE) on (urlw))
MLLE == - 2T(Ep(un.rr)) (ur (wx) vv (uv)
+
Σ
Ex(u)) [log Zi (w))
LETT,T)
(5)
<> 2rBBg(ur (ab (wr) th (wx)
Σ
(E()))())))
exp(TVi(u)) (y(())) stop_grad (z(e))
(E(I,T)
where P (w, uy) denotes the joint sampling distribution; stop_grad (-) stands for the stop-gradient operation; the "<+" indicates gradient equivalence between the two formulations for encoder updates.
While the MLE objective in Eq. (5) effectively models the target conditional distribution P(ur (()) the computation of the partition function Z, (w) involves summation over all possible samples un making MLE optimization computationally intractable. To make Eq. (5) practical, we aim to construct learnable representation Xo (w:) to approximate Z, (u)) and offload its computation from the MLE optimization. We refer to this strategy as amortization, as we amortize the estimation cost of Z (w)) by separately optimizing λo (w)) over training steps, rather than recomputing it during each forward pass. Concretely, instead of directly optimizing Eq. (5) we decompose the optimization into a modular two-stage training pipeline:
Stage (Amortization) We first optimize designed amortization objective lamor to train Xo, (wr) in approximating Zi (w)) iee. ming, lamor (Λo, (w)). In the following section, we explore several design choices for the amortization loss with biri-variance trade-offs.
3

---

Stage II (Representation Learning)
We substitute the optimized Ao, (w) for Zí (2t) in (5)
plamor) MLE
=
-2T Br(u, f/ (w) dy (ur)
+ Σ (p(u))((up))
(x) ((e())))
(w)
(6)
(E(I,T)
During training, we alternate optimizations of Stage I and Stage II within each minibatch, with P(u up) and (P()) set to the joint and marginal sampling from each minibatch. Since amortization progresses concurrently with representation learning, λe, (w) continuously aggregates information across rolling minibatches. This design alleviates the computational burden of repeated partition function calculations executed at each optimization step, thereby mitigating gradient bias during representation learning.
3.2
Amortization with Efficient Representation
To establish effective learning objectives for the amortization stage, first develop efficient representation of the amortization target Z (ua) from a kernel-based perspective. Recognizing the substantial variance introduced by the learnable temperature scalar r, we further propose two amortization objectives with the bias-variance trade-off.
Spectral representation for Zi (w) The EBM parameterization in Eq. (1) naturally leads spectral representation of the partition function Z, (u)). Specifically, interpreting Eq. (I as Gaussian kernel and employing random Fourier features [52 18 73,, we obtain:
P(ullur x P (w)) (ow (17 (ui) 9w (vrv (ur))"
where N (0, Ia) are the d-dimensional random features and the corresponding transform &w (1i (u4) = exp (ivFw" " (ui) exp (T/2) E IR (Detailed derivation in Appendix B.I
Proposition The partition function is linearly representable by dw (ths (u)), i.e. Zi (ux) (ow (y/2 (u)), v1) p(w)
Proof. From Eq. 7 there exists vector U/ E such that
Zi (w:) Ez(ay) Kow (vr (ui)) dws (whr (ur))" p(o)) (ow (1h (ux)) Er(ny) e (vv (uy ()=))))) p(w)
Motivated by Proposition I as MLPo, (+)) E R, on top of each feature representation (2x) to approximate ij (v)())) Additionally, the amortization target Zı (w)) can exhibit substantial variance across minibatches due to the learnable temperature scalar т, which varies considerably during training and may increase up to 100 (5I) To mitigate this numerical instability, we further adopt log-space parameteriza- tion: log λe; (u)) = MLPa, (wr(uu) (w:).. Specifically, log λe, (w) learns scalar within a numerically range, and the actual estimate for Zi (w) is subsequently recovered via λe, (w) exp (MLPs, (vr (). As demonstrated later in Section 4. the proposed small-scale MLPs effectively approximate Zi (w)) with negligible computational overhead during training. In the following, we discuss two design choices for the amortization objective.
we introduce lightweight multi-layer perceptrons (MLPs), denoted
stable (f1oaat32-representabll
Divergence objective for amortization The learnable function λo (wi) amortizes the effect of the partition function Z, (w) and defines an amortized conditional distribution analogous to Eq. ( i.e., Qa, (ur (w) = P (u)) exp (T (w4) thy (ur) - log λa, (u:) We formulate an f-divergence objective based on Eq. to minimize the discrepancy between Qa, (ur/ur) and IP (ur (w):
ming, L div Ep(w)) [Dy (Q (urlu), (ur [ut)]
Pp( exp (reby (u)) 1/v (ur) - log Zi (ui)
(8)
Xx
The divergence objective introduces an unbiased estimator for f-divergence. With a proper choice of f. this objective potentially improves numerical stability. For example, we can write the KI. divergence with ftt t flog
Lamor, kf-div ((we) (xxp (Tub (w:) T th (wr) - log do, (wi) (10g Zi(u)) doy (mt)
)
(9)
Alternatively, JS divergence is another suitable choice with inherent boundedness that may help further mitigate numerical issues.

---

Algorithm 1: AMORLIP Framework
Input:
Dataset D; Initial encoders v/ and amortization networks Xo; for I {1,T);
Number of epochs: T; Number of steps per epoch: K; Number of Ao update per batch: Tx; Update interval for λØ, Tonlino: Update interval for λ6, Turget-
2 Maintain target networks 100) 0
(1)
for t 1,... T do Update (b-11) for k = ,. do
11 6.
and re-initialize X 1
K
Sample B& from D and get ( (ws) via Eq. 12 for each ut E BR and / E {1,T}
/* Stage I: Amortization */
k = 0 (mod Tomlnn) then Optimize 2 via Eq. 10 or Eq. for Tx iterations
k = O (mod Targes) then Update 2. via Eq. (T1)
/* Stage II: Representation Learning */
Optimize encoders 15t via Eq. 6 for each u E Be and I E (1,
12-log objective for amortization Observing that each Ag, (w)) is a unary function w.r.t. iy we can also directly fit Ao; (u)) by matching its log-value at each input point uy:
mine; Lamor, 12-log == (p(() [Illog de, (u)) log Zi (u4)ll
(10)
where aamor, 12-log corresponds the family of Lamor, f-div described in Eq. (8) with the specific choice of f(t) 1/4 (logt): t)² (see Appendix B.3 for details).
Remark (Connection between Cammrr, kt-div and fammrr, 12-log): Empirically, Eq. 10 introduces biased estimator for the KL divergence with potentially reduced variance compared to the Monte- Carlo approximation amor, Ad-iv in Eq. (9). This variance reduction arises because the optimization in Eq. (10) occurs entirely in log-space log Ao, (w), mitigating potential numerical issues caused by the exponential operation exp(-) presented in Eq. (9). Additionally, famor, 12-log exhibits relatively low bias, as it closely approximates the KL divergence up to second order under mild conditions (see Appendix B.4 for details). Overall, both amor, kf-div and Lamor, 12-log effectively enhance numerical stability. Thus, we adopt both formulations as design choices for amortization objectives.
3.3 Training Techniques for AMORLIP
Training stability The training procedure AMORLIP involves stochastic approximations at two distinct time scales, alternating optimization between the encoders 1 (-) and the partition function estimator Ao (-). Consequently, the optimization frequencies of these components are crucial for stable training. To improve training stability, we introduce target network XØ (-) that slowly updates 9. Within each training epoch, update Aş (.)
its parameters towards the online network 48 26
every T Tunger steps using exponential moving average (EMA) of the online model parameters 44 8 + a 1) + (1 a)){k))
(11)
where a denotes the EMA decay factor, and k represents the current update step. We then substitute
Ag (.) into Eq. 6 place of the online network λo, (ur). Additionally, the rapidly increasing
temperature т elevates the variance of Zi (u) in the amortization objectives. Denote b. (-) and
T (-) as the target amortization network and the partition function at the I-th epoch, respectively.
We assume the outputs from the target network at the previous epoch, X
(t-1) (-). have a magnitude
similar to those of the current online network X0; (+). Thus, we naturally introduce the following
weighted combination to replace 714)) (w:) in Eq. 8 and Eq. 10
(12) where 3 is initialized with small value when is low and gradually increased up Br
7(0) (w:) 7, 3,1
(1z)+(1 B.)Z(') (u)),
grows larger. Empirically, we employ a cosine scheduling strategy similar to that [66]: B, =
37 - Br (1 cos )). The resulting weighted estimate 2 (w) is thus effectively "flattened"
by the target amortization predictions, thus facilitating the optimization of amortization objectives.
5

---

Table 1: Performance comparison (%) across (1) top-1 aceuracy of zero-shot classification tasks on ImageNet and six distribution shifts, (2) recall@) I of retrieval tasks on Flickr30k and MS-COCO, and (3) overall performance on all DataComp tasks. The results are reported for two training scales. Highest scores are highlighted in bold, and second-best scores are underlined. The proposed AMORLIP consistently outperforms baseline methods across all evaluated tasks.
Tasks (-)
Image Net & Dist. Shifts
Retrieval
Avg. Tasks
Method (1)
IN-Ik IN-Sk IN-V2 IN-A IN-O IN-R ObjN Avg. Flickr COCO
Avg.
ResNet-50 Pretrained on CC3M
CLIP [51]
16.84
10.30
13.96
3.69
21.70
20.71
11.00
14.03
25.79
13.93
19.86
21.48
SigLIP 71
17.74
10.34
15.43
3.88
23.10
22.96
12.01
15.07
26.73
14.86
20.80
21.32
SogCLR 770
19.91
11.91
17.90
4.27
26.05
25.69
13.51
17.03
27.51
16.57
22.04
21.47
FastCLIP 666
20.58
13.03
18.09
4.15
27.10
27.22
14.04
17.74
34.31
19.80
27.06
23.46
AMP [-diiv] (F-divv
21.16
13.57
18.30
4.99
27.65
28.45
14.34
18.35
35.30
19.91
27.61
24.08
AMORI.IP (12-log)
21.50
14.30
19.45
5.20
28.10
29.22
14.64
18.92
35.01
19.67
27.34
24.11
ViT-B/32 Pretrained on CC12M
CLIP [11
25.26
15.70
21.30
4.36
29.95
33.88
12.67
20.45
34.32
17.89
26.10
27.65
SigLIP 71
25.42
16.60
22.08
4.79
30.90
33.85
13.00
20.95
32.91
17.86
25.39
26.91
SogCLR 70
27.59
18.28
23.01
4.83
30.45
35.43
14.14
21.96
33.01
17.33
25.17
26.97
FastCLIP
66
27.74
18.33
22.51
4.72
32.45
35.72
13.77
22.18
36.64
20.78
28.71
29.00
AMORLIP (S-div)
29.21
19.80
24.55
5.29
34.25
39.24
15.59
23.99
37.93
22.11
30.02
29.91
AMORI. (12-bog)
29.93
20.11
24.70
5.51
34.90
39.38
15.10
24.23
38.70
21.47
30.09
30.66
Training efficiency Unlike the encoders di requiring optimization at each minibatch, optimize Ao every Tonline encoder optimization steps. Furthermore, the proposed AMORLIP inherently supports efficient multi-GPU training via distributed data parallelism (DDP). Conventional CLIP models 36 (6 (2) require invoking all (.l_her(((())) operations at every step when optimizing the NCE loss in Eq. In contrast, AMORLIP triggers the gathering operation only during the amortization stage, executed merely 1/Tonline times as frequently as CL.P. During the contrastive learning stage, A MORLIP computes the amortized partition function using lightweight MIPs independently each device without calling all_ (..aher(.)). Consequently, with a suitably large Tanline. AMORL.IP effectively reduces computational overhead and enhances overall training efficiency. We summarize the proposed AMORLIP in Algorithm
4 Evaluation
Training setups We compare AMORLIP against widely adopted language-image baselines, includ- ing CLIP [51), SigLIP (71), SogCLR (70), and FastCLIP [66]. We use the OpenCLIP (36 codebase and original implementations for these models. Following the experimental setups from [66]. we pre- train models at two scales: medium-scale experiment using ResNet-50 [31 trained on Conceptual Captions 3M (CC-3M; (57]) with batch size of 1024 for 30 epochs, and large-scale experiment using ViT-B/32 trained Conceptual Captions 12M (2C- [00) with batch size of 2048 for 33 epochs. Due to expired source links, our downloaded datasets contain 2,274,566 samples for CC-3M and 8,059,64 9,642 samples for CC-12M. All experiments are conducted using NVIDIA H100 GPUs with 80GB VRAM. Additional training details can be found in Appendix C available at https://gihubb.cm/aaatiii
Our implementation is
4/AmorLIP.
AMORLIP implementation In AMORLIP, we implement λø using a three-layer MLP for each modality / E {1,T), operating in parallel to the respective text and image encoders. Each MLP takes the corresponding encoder's d-dimensional feature as input and outputs scalar representing the amortized partition function. We control the network's 's width through a dimension factor fa. setting the intermediate layer dimension as fa d. Specifically, we choose fa = 0.5 for the medium- scale setting and fa = 1.0 for the large-scale setting. For amortization hyperparameters detailed in Algorithm [] we set Tx = and Turget = 2 for both training scales, while Tonline is set to 8 for medium-scale and for large-scale experiments. Regarding techniques described in Section 3.3 the EMA factor a set to 0.999 for medium-scale and 0.92 for large-scale training. The parameter BT universally fixed at 0.8.

---

20
ImageNet et & Shifts
Retrieval
DataComp 38 Tasks
Improvement (%)
15
10
-10
Inansttt- Inarahi
Imagania Imagete x2
Okyniter
Imagonies.
MSC300
Presca VOC 2007
kiT venta Delance
Flowa Florar (02
GFAR- BI Pet
Cambel--to
101
oramB
SUNSee
CPAR-100)
GoogE
CLEVA Delance
Desctbabbl Tartums
TOTAL
STL-10 10
Starbord Caus
MIRST
Cruntryt REGISCES
Waltcam Winetr
FGVC Alond
FMo
SVNN
Caraeyon 17
Deter Street
CLEVA EV County
Putth Eurodhr
Figure 2: Breakdown of absolute improvement (%) made by AMORLIP over CLIP model on all 38
DataComp Tasks 23 under large scale setting.
Evaluation metrics We evaluate AMORLIP and baseline methods using the DataComp bench- mark (23), comprising 38 widely used text-image tasks. Specifically, we report top-1 zero-shot classi- fication accuracy on ImageNet (IN-IK; (55) and six of its distribution-shife
variants: ImageNet- Sketch (IN-Sk; (65)), ImageNet- V2 (2 (53)) ImageNet-A (IN-A; [33). ImageNet-O (IN-O; B3), ImageNet-R (IN-R: [32), and ObjectNet (ObjN; 4 Additionally, we evaluate retrieval performance via recall@) I on Flickr30k (Flickr; (691) and MSCOCO (COCO; [14]). Finally, we report average performance across all 38 DataComp tasks (Avg.38). To assess the training efficiency of AMORLIP, we also measure per-step training time and GPU memory (VRAM) usage.
4.1 Main Results
Table presents the performance of text-image models across the 38 downstream tasks of Dat- aComp [23]. Consistently, across different encoder architectures and dataset scales, AMORLIP outperforms all baselines in most zero-shot classification and retrieval tasks. Specifically, AMORLIP achieves improvements in top-1 accuracy of up to 4.67% on ImageNet zero-shot classification tasks and up to 4.32% on its distribution-shifee variants. In retrieval tasks, AMORLIP surpasses other experiments and 3.92% for the large-scale
methods by an average of 7.75% for the medium-sett experiments. Overall, AMORLIP exhibits substantial relative improvements over CLIP, with 12.24% in the medium-scale setting and 10.89% in the large-scale setting. Additionally, AMORLIP using the 12-log objective demonstrates slightly more consistent performance gains compared to the f-div objective, while the f-div objective achieves comparable even superior performance in retrieval tasks. Figure 2 further illustrates that both AMORLIP objectives can achieve up to a 20% abso- lute improvement on 30 out of the 38 evaluated tasks. These results collectively highlight that the amortization of AMORLIP can effectively enhance multimodal representation pertaining.
4.2 Learning Efficiency
20
Acnurasy (%)
Faster training convergence Figure 3 illustrates the evolu- tion of model performance (reflected by classification accu- racy) over training epochs. In both evaluated settings, AMOR- LIP consistently achieves higher final performance than the baseline models. In the medium-scak, setting shown in Fig- ure 3a| AMORLIP surpasses the best baseline performance (20.58% by FastCLIP) using only 26 epochs, and achieves convergence around 13.3% faster than all baselines. Further training up to 30 epochs improves the accuracy to 21 21.50%
15
FastCLIP AmorLiPa og
10
SigLIP
SegCLR
10
Training Epoch
(a) ResNet-50 CC3M
In the large-scale setting depicted by Figure 3bb AMORLIP extends this training speed advantage significantly, reaching comparable performance about 10 epochs earlier and equiv- alently at least 30.3% faster than all baselines. The AMORL.P ultimately achieves 7.89% relative performance gain over the best- best-performing baseline. Notably, the efficiency benefit becomes more pronounced the large-scale scenario, as the increased number of iterations and samples better facilitates amortization optimization. Additionally, AMORLIP initially trails some baselines, such as SogCLR, but begins to outper-
IN Accurany (%))
25
20
CLIP
FasiCUP
S SgL
AmorL PP
15
SogCLR
Amer LP-a
10
15
20
25
30
Training Epoch
(b) VIT-B/32 on CC12M
Figure 3: ImageNet classification ac-
curacy (%) of models at two scales.
7

---

Table Performance (%) and relative training overhead (%) of AMORLIP at medium-scat under different amortization settings, evaluated on one H100 GPU. The relative overhead is depicted by per-step training time and total VRAM usage (including encoders and amortization), relative to CLIP.
(a) Online update frequency oaline
(b) Dimension factors fa
T onlline
IN&Shifts Retrieval Avg. 38
A ATim
fa
IN&Shifts Retrieval Avg. 38
AVRAM
1
18.28
27.97
23.33
4.47%
2
18.78
28.29
23.97
0.60%
1/2
18.59
28.03
23,40
4.06%
I
18.61
28.38
23.25
0.42%
1/8
18.92
28.10 10
24.11
2.16%
0.5
18.92
28.10
24.11
0.33%
1/32
18.83
28.39
24.25
0.23%
0.25
18.56
27.65
24.09
0.26%
CLIP
14.03
19.86
21.48
844.48 ms
CLIIP
14.03
19.86
21.48
75.91 GiB
form them around 60K encoder training iterations for both scales. We hypothesize that after this point, the encoder's 's output features stabilize sufficiently, thereby enhancing amortization optimization and enabling AMORLIP exhibit superior performance gains. Overall, AMORLIP demonstrates faster convergence and increased relative efficiency larger scales.
Lightweight amortization Table 2 further quantifies the additional time and memory overhead of AMoRLIP relative to CLIP by examining critical werhead-related factors. Specifically, when the amortization network is updated less frequently (lower Tomlee and utilizes fewer parameters (smaller dimension factor fa), the extra GPU time and memory overhead become effectively minimized and eventually negligible (with only 0.26% higher memory usage and 0.23% additional training time), while consistently outperforming the baseline in downstream tasks. Interestingly, reduced amortization network complexity or update frequency generally corresponds to improved model performance compared larger amortization networks. A potential explanation that smaller networks may mitigate overfitting during the amortization stage. Furthermore, this highlights that AMORLIP provides an effective representation for partition functions, thus achieving robust performance even with lightweight amortization implementations.
4.3
Ablation Studies
In the ablation studies, we evaluate the impact of several key factors associated with amortization. Unless otherwise specified, adopt the medium-scale setting with the 12-log objective in Table
Impact of stability techniques Figure 4 evaluates the impact of the two stability techniques introduced in Sec-
IN Shifts
Retrieval
Avg. 38 Tasks
Performance (%)
tion 3.3 As shown in Figure 4a| using a larger EMA factor for the target model generally enhances both classification and overall performance. A larger EMA factor smooths updates of the target model parameters, which effectively preserves valuable information from previous batches. Fig- ure 45] further examines the effect of different combination weights (3r) for amortization based on previous-epoch infor- mation. moderately high value of ßr, such as 87 0.8, facilitates amortization learning and particularly improves retrieval performance. However, excessively large values (e.g., Br = 1.0) hinder effective updates from new batches and negatively affect overall performance. The ablations presented in Figure 4b suggest that 37 = 0.8 provides the best balance with optimal overall performance.
22
22
0.92
0.999
(a) Target EMA factor in Eq.
Shifts
Retrieval
Avg. 35 Tunko
Performance (%)
22
Amortization objective We compare two proposed amor-
Br
tization objectives in Section 3.2 As shown in Table [ and Figure 3 the 12-log objective exhibits slightly better performance and smoother training dynamics than the f- divergence objective at both scales. As discussed in Sec-
(b) Combination weight Br in Eq.
12
Figure 4: Performance (%) of AMOR-
LIP with different values of parameter a and Br in stability techniques.

---

although the 12-log objective introduces bias, it effectively reduces variance. This indicates that the proposed parameterization of AMORLIP can successfully amortize the target partition func- tion even under a biased objective. Additionally, the reduction in variance contributes to improved training performance.
tion 3.2
Amortization farget By default, the amortization ob- jective targets the entire partition function, incorporating both positive and negative samples from each minibatch, Table the effect of amortizing only the partition functions computed from negative samples while using positive samples directly from the current batch. Results demonstrate that amortizing the full partition function con- sistently leads to improved performance across all three task groups. This outcome likely arises because positive samples typically yield higher similarity scores and may significantly influence the magnitude of the partition function.
Table 3: Performance (%) with amortiz- ing only negative samples (Neg.) vs, the entire batch (Pos.+Neg.).
Amor. Target IN&Shifts
Ret.
Avg. 38
Neg. Pos. Neg.
18.02
27.92
23.17
18.92
28.10
24.11
4.4 Empirical Guidelines for Hyperparameter Setup
Based our empirical evaluation above, summarize concise and practical guidelines for instanti- ating hyperparameters:
Amortization loss:
We proposed two amortization loss variants: f-divergence (unbiased) and 12-log (biased but with reduced variance). We recommend the 12-log objective when the learnable temperature varies significantly during training (as in standard CLIP pretraining, temperature from 14.27 to 100 [5l). In scenarios with stable temperature (e.g.. fine-tuning), the unbiased f-divergence objective may perform better.
The EMA decay factor a determines how much past model information retained. Theoretically, a larger stabilizes model training by smoothing gradient estimation noise and effectively preserving information from previous batches. Based on the empirical results in Figure 4a| we recommend using relatively large a, typically 0.999, which achieves the best results in small or medium scales.
EMA decay factor ot
Empirically, moderately high value of 3r, such as 8r 0.8, facilitates amortization learning and particularly improves retrieval performance. However, excessively large values (e.g., 3T 1.0) may hinder effective updates from new batches and negatively affect overall performance. Figure 4b suggests that 3T 0.8 provides the best balance with optimal overall performance.
Combination weight 37:
Update frequency T T The amortization network should be updated less frequently than the encoder models (Table 2ay. Empirically, frequency of Tom < offers an optimal balance between performance and computational overhead for small and medium scales.
Capacity (fa) of the amortization network: Reduced network complexity (smaller fa) mitigates potential overfitting and minimizes memory overhead. Based on empirical results, we recommend setting fa to 0.5 or even 0.25 to achieve the best balance between efficiency and performance.
Finally, we emphasize that AMORLIP consistently outperforms baseline methods even with subopti- mal hyperparameter settings. The effectiveness of IMORLIP helps reduce the necessity for extensive hyperparameter tuning.
5 Related Work
Efficient CLIP training response to the rapid growth of data and model scales in CLIP train- ing 33 6 62 several studies aim at improving training efficiency. Methods such as LiT (72). FLIPP 43), and CLIPA [4] aim to lower computational complexity through strategies like model freezing, token masking, resolution rescaling. Other approaches have modified the contrastive loss,

---

including the decoupled softmax loss in DCL [68] pairwise sigmoid loss in SigLIP 771, or aggregat- ing local losses computed on subsets of each minibatch [T] 36] 16]. However, these techniques often compromise downstream task performance
extend training durations. Another group of work leverages non-parametric estimation of the partition function, such as DeCL. [T] SogCI.R 77,, iSogCLR [50], and NuCLR (64) Furthermore, system-level approaches, such as modifications to gradient implementations [59
49 25 and distributed parallel training frameworks [63] 36 [5 56 [6),
have attempted to scale CLIP models with larger batch size across large clusters of GPUs or TPUs. Nevertheless, these two types of approaches generally suffer from substantial overhead either in space (e.g., maintaining large offline buffers) time (e.g., inter-device communication), potentially limiting their practical scalability and efficiency.
Amortization in self-supervised learning In general self-supervised learning (SSL), many promi- nent methods reuse or approximate computationally expensive operations, such as computing large- scale similarity matrices or offload these tasks to auxiliary models. We refer to this overarching strategy as amortization. One common approach involves maintaining a memory buffer to store representations of previously encountered samples, thereby reducing per-batch computation and alleviating large batch size requirements [1]. Such memory banks are widely employed in contrastive and clustering- -baied SSL frameworks [67
For instance, the MoCo family [30] 13, along with other related methods 66 [), utilizes a queue-based buffer to amortize negative representation computations effectively. More recently, several methods [70] 50 664 have adopted larger memory buffers that span the entire training set, amortizing the expensive partition function computation at a per-sample granularity. Another amortization strategy involves momentum encoder updated slowly via an EMA model of the online encoder parameters 30 13 27). The EMA model is also central to serrative-sample free methods such as BOL (77. Additionally, recent work (37 has proposed amortizing reconstruction tasks via meta-learning to further enhance SSL performance across multiple modalities. Unlike most existing methods with nonparametric amortization, the proposed A AMORLIP directly learns the amortization targets by optimizing lightweight neural networks. This parametric amortization ensures greater flexibility while free of maintaining a large memory buffer.
EBM Learning. Energy-based models [EBM))[[1) flexibly represent probability distributions using an energy function defined over data points. Specifically, a conditional EBM takes the form:
P((l(=)
((x)
where the energy function f(x, typically parameterized by deep neural networks, assigns lower energy values to more probable data points; the partition function (xx) ensures P(y|z) to be valid
probability distributions. Training EBMs generally involves several techniques [60], such as MLE
via MCMC sampling [34 21), score matching [55 46). and NCE [47 28 29]. To further improve efficiency, amortization techniques have been introduced into EBM training: SteinGAN[45) amortizes negative sample generation for MLE with a jointly trained sampler; ADE 17 leverages a primal-dual perspective MLE to learn efficient sampling strategy for exponential family distributions; and ALOE [9] introduces an amortized sampler inspired by local search to estimate gradients for EBMs on discrete structured data efficiently. Recently, CLIP-JEM (24) introduces an image-text joint-energy function in the CLIP representation space to enable text-to-image generation capabilities.
6 Conclusion
In this paper, we proposed AMORLIP, a novel amortization framework that effectively decouples the estimation of partition functions from minibatch- optimization through lightweight neural networks. Extensive experimental results demonstrated that AMORLIP consistently outperforms existing CLIP-like baselines across 38 diverse downstream tasks, achieving substantial relative improvements of up to 12.24%. AMORLIP significantly enhances training efficiency and leads to more resource-efficient contrastive language-image pretraining.
Acknowledgments and Disclosure of Funding
This work was supported in part by the ONR grant N000142512173, NSF grants ECCS: 2401391 and IIS: 2403240. Dolby support, and computing resources received from the National Supercomputing Center (CSCS) and the Swiss Al initiative.
10

---

References
[1] S. M. Ali and S. D. Silvey. A general class of coefficients of divergence of one distribution from another. Journal of the Royal Statistical Society: Series B (Methadological), 28(1):131-142. 1966.
[2] B. Amos. Tutorial amortized optimization, 2025.
[3] Y. M. Asano, C. Rupprecht, and A. Vedaldi. Self-labelling via simultaneous clustering and representation learning, 2020.
[4] A. Barbu, D. Mayo, J. Alverio, W. Luo, C. Wang, D. Gutfreund, J. Tenenbaum, and B. Katz. Objectnet: large-scale bias-controlled dataset for pushing the limits of object recognition models. Advances in neural information processing systems, 32, 2019.
[5] M. BARTLETT. Approximate confidence intervals. Biometrika, 40(1-2):12-19, 06 1953.
[6] R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108. 07255, 2021.
[7] M. Caron, P. Bojanowski, A. Joulin, and M. Douze, Deep clustering for unsupervised learning of visual features, 2019.
[8] M. Caron, I. Misra, J. Mairal, P. Goyal, P. Bojanowski, and A. Joulin. Unsupervised learning of visual features by contrasting cluster assignments, 2021.
[9] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging properties in self superviid vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650-9660, 2021.
[10] S. Changpinyo, P. Sharma, N. Ding, and R. Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pages 3558-3568, 2021.
[11] C. Chen, Zhang, Xu, L. Chen, J. Duan, Y. Chen, S. Tran, Zeng, and T. Chilimbi. Why do we need large batchsizes in contrastive learning? a gradient-bias perspective. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 338755 Curran Associates, Inc., 2022.
[12] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations, 2020.
[13] X. Chen, H. Fan, R. Girshick, and K. He. Improved baselines with momentum contrastive learning, 2020.
[14] X. Chen, H. Fang, T.-Y. Lin, R. Vedantam, S. Gupta, P. Dollar, and C. Zitnick. Microsoft coco captions: Data collection and evaluation server, 2015.
[15] Y. Chen, X. Qi, Wang. and L. Zhang. Disco-clip: A distributed contrastive loss for memory efficient clip training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22648 -2667,, 2023.
[16] Z. Cheng, H. Zhang, K. Li, Leng. Z. Hu, Wu, D. Zhao, X. Li, and L. Bing. Breaking the memory barrier: Near infinite batch size scaling for contrastive loss, 2024.
[17] B. Dai, Z. Liu, H. Dai, N. He, A. Gretton, I.. Song, and D. Schuurmans. Exponential family estimation via adversarial dynamics embedding. Advances in Neural Information Processing Systems, 32, 2019.
[18] B. Dai, B. Xie, N. He, Liang, Raj, M.-F Balcan, and Song. Scalable kernel methods via doubly stochastic gradients. Adv. in Neural Info. Processing Systems, 27:3041-3049, 2014.
11

---

[19] H. Dai, R. Singh, B. Dai, C. Sutton, and D. Schuurmans. Learning discrete energy based models via auxiliary-variable local exploration. Advances in Neural Information Processing Systems, 33: 15:10443-10555 2020.
[20] A. Dosovitskiy, L.. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale, 2021.
[21] Y. Du, S. Li, J. Tenenbaum, and I. Mordatch. Improved contrastive divergence training of energy based models. arXiv preprint arXiv:2012.01316, 2020.
[22] Y. Fang, W. Wang, B. Xie, Q. Sun, L. Wu, X. Wang, T. Huang, X. Wang, and Y. Cao. Eva: Exploring the limits of masked visual representation learning at scale, 2022.
[23] S. Y. Gadre, G. Ilharco, A. Fang, J. Hayase, G. Smyrnis, T. Nguyen, R. Marten, M. Wortsman,
D. Ghosh, Zhang, E. Orgad, R. Entezari, G. Daras, S. Pratt, V. Ramanujan, Y. Bitton, K. Marathe, S. Mussmann, R. Vencu, M. Cherti, R. Krishna, P. W. Koh, O. Saukh, A. Ratner, Song, H. Hajishirzi. A. Farhadi, R. Beaumont, S. Oh, A. Dimakis, J. Jitsev, Y. Carmon, V. Shankar, and L. Schmidt. DataComp: In search of the next generation of multimodal datasets. arXiv, 2023. DataComp.
[24] R. Ganz and M. Elad. Text-to-iman -mage generation via anergy-based clip, 2024.
[25] L. Gao, Y. Zhang, J. Han, and J. Callan. Scaling deep contrastive learning batch size under memory limited setup, 2021.
[26] J.-B. Grill, F. Strub, Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271-21288 2020.
[27] J.-B. Grill, F. Strub, F. Altchrr, C. Tallec, P. H. Richemond, E. Buchatskaya, C. Doersch, B. A. Pires, Z. D. Guo, M. G. Azar, B. Piot, K. Kavukcuoglu, R. Munos, and M. Valko. Bootstrap your own latent: A new approach to self-supervised learning, 2020.
[28] M. Gutmann and A. Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the thirleenth international conference on artificial intelligence and statistics, pages 297-304. JMLR Workshop and Conference Proceedings, 2010.
[29] M. U. Gutmann and A. Hyvärinen. Noise-con estimation of unnormalized statistical models, with applications to natural image statistics. The journal of machine learning research, 13(1):377-311, 2012.
[30] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation learning, 2020.
[31] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition, 2015.
[32] D. Hendrycks, Basart, N. Mu, S. Kadavath, Wang. E. Dorundo, Desai, T. Zhu, S. Parajuli, M. Guo, D. Song, J. Steinhardt, and J. Gilmer. The many faces of robustness: A critical analysis of out-o--distributtonn generalization, 2021.
[33] D. Hendrycks, K. Zhao, S. Basart, J. Steinhardt, and D. Song. Natural adversarial exam- (2019, arXiv preprint es. ct..LG/1907.07774 5(6), 2019.
[34] G. E. Hinton. Training products of experts by minimizing contrastive divergence. Neural computation, 1478)1777))880, 2002.
[35] A. Hyvärinen, J. Hurri, P. O. Hoyer, A. Hyvärinen, J. Hurri, and P. O. Hoyer. Estimation of nor-noraa statistical models. Natural Image Statistics: A Probabilistic Approach to Farly Computational Vision pages 419 2009.
12

---

[36] G. Ilharco. M. Wortsman, R. Wightman. C. Gordon, N. Carlini, R. Taori, A. Dave, V. Shankar, H. Namkoong, J. Miller, H. Hajishirzi, A. Farhadi, and L. Schmidt. Openclip, July 2021. If you use this software, please cite it as below.
[37] H. Jang, J. Tack, D. Choi, J. Jeong, and J. Shin. Modality-agnosiie self supervuue learning with meta-learned masked auto-encoder, 2023.
[38] C. Jia, Y. Yang, Y. Xia, Y.TT. Chen, Z. Parekh, H. Pham, Q. V. Le, Y. Sung, Z. Li, and T. Duerig. Scaling up visual and vision-language representation learning with noisy text supervision, 2021.
[39] J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford,
J. Wu, and D. Amodei. Scaling laws for neural language models, 2020.
[40] D. Kingma and J. Ba. Adam: method for stochastic optimization, 2017.
[41] Y. LeCun, S. Chopra, R. Hadsell, M. Ranzato, Huang, et al. A tutorial on energy-based learning. Predicting structured data, 1(0), 2006.
[42] X. Li, Z. Wang, and C. Xie. An inverse scaling law for clip training, 2023.
[43] Y. Li, H. Fan, R. Hu, C. Feichtenhofer, and K. He. Scaling language-image pre-training Via masking. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 23390-23400, 2023.
[44] T. P. Lillicrap. J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning, 2019.
[45] Q Liu and D. Wang. Learning deep energy models: Contrastive divergence vs. amortized mle, 2017.
[46] S. Lyu. Interpretation and generalization of score matching. arXiv preprint arXiv: arXiv: 1205.262,, 2012.
[47] Z. Ma and M. Collins. Noise contrastive estimation and negative sampling for conditional models:
Consistency and statistical efficiency. arXiv preprint
2018
[48] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis. Human-level control through deep reinforeement learning. Nature, 518(7540):5295333 2015.
[49] H. Pham, Z. Dai, G. Ghiasi, K. Kawaguchi, H. Liu, A. W. Yu, J. Yu, Y.-T. Chen, M.-T. Luong, Y. Wu, M. Tan, and Q. V. Le. Combined scaling for zero-shot transfer learning, 2023.
[50] Z.-H. Qiu, Q. Hu, Z. Yuan, D. Zhou, L. Zhang, and T. Yang. Not all semantics are created equal: Contrastive self-supervised learning with automatic temperature individualization. arXiv preprint arXiv:2305.1 1965, 2023.
[51] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
[52] A. Rahimi and B. Recht. Random features for large-scale kernel machines. Adv. in Neural Info. Processing Systems, 20, 2007.
[53] B. Recht, R. Roelofs, L. Schmidt, and V. Shankar. Do imagenet classifiers generalize to imagenet? In International conference machine learning, pages 5389-5400. PMLR, 2019.
[54] T. Ren, T. Zhang, C. Szepesvári, and B. Dai. A free lunch from the noise: Provable and practical exploration for representation learning. arXiv preprint arXiv:2 11022/11.. /485, 2021.
[55] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 1 115:211-252 2015.
13

---

[56] C. Schuhmann, R. Beaumont, R. Vencu, C. W. Gordon, R. Wightman. M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, Schramowski, S. R. Kundurthy, K. Crowson, L. Schmidt, R. Kaczmarczyk, and J. Jitsev. LAION-5b: An open large-scale dataset for training next generation image-text models. Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
[57] P. Sharma, N. Ding. S. Goodman, and R. Soricut. Conceptual captions: A cleaned. hypemymed, image alt-text dataset for automatic image captioning. In I. Gurevych and Y. Miyao, editors, Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 2556-2565, Melbourne, Australia, July 2018. Association for Computational Linguisties.
[58] R. Shu, H. H. Bui, S. Zhao, M. J. Kochenderfer, and S. Ermon. Amortized inference regulariza- tion, 2019.
[59] N. S. Sohoni, C. R. Aberger, M. Leszczynski, J. Zhang, and C. RÁI. Low-memory neural network training: A technical report, 2022.
[60] Y. Song and D. P. Kingma. How to train your energy-based models. arXiv preprint arXiv:2101.03288. 2021.
[61] A. Srinivas, M. Laskin, and P. Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning, 2020.
[62] A. Srivastava, A. Rastogi, A. Rao, A. A. M. Shoeb, A. Abid, A. Fisch, A. R. Brown, A. Santoro, al. Beyond the imitation game: Quantifying and extrapolating
A. Gupta, A. Garriga-Alons the capabilities of language models. arXiv preprint arXiv:2206.044 04611,, 2022.
[63] Q Sun, Y. Fang, L. Wu, X. Wang. and Y. Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303. /5389, 2023.
[64] B. Wang, Y. Lei, Y. Ying, and T. Yang. On discriminative probabilistic modeling for self- supervised representation learning, 2024.
[65] H. Wang, S. Ge, E. P. Xing, and Z. C. Lipton. Learning robust global representations by penalizing local predictive power, 2019.
[66] X. Wei, F. Ye, O. Yonay, X. Chen, B. Sun, D. Tao, and T. Yang. Fastclip: A suite of optimization techniques to accelerate clip training with limited resources, 2024.
[67] Z. Wu, Y. Xiong. S. Yu, and D. Lin. Unsupervised feature learning via non- non-parametric instance- level discrimination, 2018.
[68] C.-H. Yeh, C.-Y. Hong, Y--C. Hsu, T.-L. Liu, Y. Chen, and Y. LeCun. Decoupled contrastive learning, 2022.
[69] P. Young, A. Lai, M. Hodosh, and J. Hockenmaier. From image descriptions to visual denota- tions: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78, 2014.
[70] Z. Yuan, Y. Wu, Z.-H. Qiu, X. Du L. Zhang, D. Zhou, and T. Yang. Provable stochastic optimization for global contrastive learning: Small batch does not harm performance. In International Conference on Machine Learning, pages
PMLR, 2022.
[71] X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer. Sigmoid loss for language image pre- training. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11975-11986, 2023.
[72] X. Zhai, X. Wang, B. Mustafa, A. Steiner, D. Keysers, A. Kolesnikov, and L. Beyer. Lit: Zero-shot transfer with locked-image text tuning, 2022.
[73] T. Zhang, T. Ren, C. Xiao, W. Xiao, J. E. Gonzalez, D. Schuurmans, and B. Dai. Energy-based predictive representations for partially observed reinforcement learning. In Uncertainty in Artificial Intelligence, pages 2477-2487. PMLR, 2023.
14

---

A Limitations and Broader Impacts
A.1 Limitations and Future Work
In this work, we proposed AMORLIP, a novel amortization paradigm to enhance CLIP training efficiency. Despite demonstrating effectiveness and efficiency, our proposed method exhibits several limitations:
Resource constraints Due computational resource limitations, our evaluations of AMORLIP and baseline methods were constrained to datasets with to ten million samples and models to the scale of ViT-B/32. Although experimental results these scales consistently demonstrate the advantages of our method, further evaluation at billion-scale datasets and larger backbone models could better highlight AMORLIP's scalability and efficiency. We plan to address this in our future work.
Data privacy and licensing We pretrained and evaluated AMORLIP using publicly available datasets in compliance with their respective licenses and intended use policies. Nevertheless, given the extensive scale of these datasets comprising millions of text-image pairs collected from the web, there remains potential risk of encountering unintended unfiltered content. This could inadvertently lead to privacy concerns or inadvertent exposure of sensitive information.
A.2 Broader Impacts
Potential positive societal impacts The proposed AMORLIP addresses important challenge in contrastive anguuae-iimae pretraining, i.e., the extensive computational resource requirements that have limited accessibility and scalability. As demonstrated both theoretically and empirically, AMORLIP significantly reduces reliance on large batch sizes through efficient amortization techniques. Consequently, AMORLIP facilitates effective and efficient pretraining of rision-language models and enables a wider range of individuals and organizations to develop and deploy high-quality multimodal models across diverse downstream tasks, including cross-modal retrieval, zero-shot classification, and text-to-image synthesis.
Potential negative societal impacts However, the proposed method may also lead to misleading results in downstream applications. Despite achieving superior representation performance compared to baselines, AMORLIP does not guarantee perfect accuracy recall downstream tasks. For example, when deployed in text-to-image retrieval scenarios, AMORLIP could potentially return mismatched or inappropriate images, leading to unintended consequences such as privacy leakage or exposure to harmful content. We therefore recommend deploying AMORLIP alongside robust data privacy protection and content moderation tools to effectively mitigate these risks.
A.3 Ethical statements
When conducting research presented in the paper, we have fully conformed with the NeurIPS Code of Ethics. Additionally, our use models and datasets strictly adheres their corresponding licenses and usage guidelines.
15

---

B Theoretical Derivations
B.1 Derivation of Random Features in Eq.
We begin by rewriting T with the following equivalent energy-based parameterization 54 Since each da (ur) is f2-normalized, we have 1/v// (un)ll²= Then, one can write:
(((ulue) x I (ux) exp
(TV: (ur) t/v (ux))
11 (w))l
T 11ub2 (ux))l²
exp IIV (w) VTbv (ur)
= =P (ur) exp
2
2
|IViih (wi) VTev (wr)
=P (ux) exp
exp (T)
where exp (- Fiy - Tith 12 /2) is the Gaussian kernel and induces the spectral decomposition
by applying random Fourier features. Denote ō1,y (VFih vFbv) E R for notation simplicity. One can write:
VVTuh Viebelll
exp
2
(di,r))
exp
|
||
15 1|2
=(2T
exp
JR exp
2
2
||w112
-(mm)
exp
+ iw
our
dw
Rd
2
=RunN(0,Le) (exp (iw 01,r0]]
= E -Bo..( (exp (ivTw" 4/1) exp T
which leads to Eq.
B.2 Derivation of Divergence Objective in Eq. 8
With the definition of Dy (..:) Section one can write:
L Lamor, f-div
-Ep(ua)
Dy (Q) (1yr (mx) (uv (urlu)))
f
exp (T7 (w))
wh (wrr))
exp
(Ture (u)) wh (ur)) /ea, (ws)
-Er(w))
P (Pwv))
du(uv)
Zi (wz)
(Tu (uu) vo (uv) /Z (w)
exp
exp (Thh (w) tby (wr)
Zi (w) de, (w)
-Ep(n))P(ure
Zi (w))
where a is base measure for uy.
16

---

B.3
Connection between aamor, f-div and Lamor, 12-1og
We show that Eq. (10 can be included into the family of f-divergence in Eq. (8) with f(t)
(log t)²:
(P(u)) exp (re' (w)) th (uv) - log Zi (ui) /
Zi (w)) Ag (w))
Ep((a)) eexp (ruh (wi) thu (ur))
log
Zi (2x) de (ux)
2 (r()
Zi (wi)
(u))
|log Ao, (u)) log Zi (un)1
which recovers Eq. 1T).
B.4 Connection between Lamor, ki-div and Lamor, 12-log
We demonstrate that ammor, 12-log closely approximates the KL divergence up to second order within the general f-divergence framework. For simplicity of notation, we denote Po (z) as the fixed probability distribution and (ar) as the distribution parameterized by f. We assume Qo, closely approximates Fo at 0 8o. i.e., Pu Qe. Additionally, we define the score function sp(x) and the Fisher information matrix Gg for Qa, as:
so (I = V log (z)). Go [se(r)se(z)]]
Considering DyQQo Po) as scalar function of 0 that is at least twice continuously differentiable in a neighborhood around the point tg. we apply the second-order Taylor expansion to obtain:
(0) (0)) (0 0o) ( VaD,(00) - 0o) (0,) (0 00) O(100) (13)
Here, the first term vanishes since (0,)) = f( To f(1) Then, we show that the first-order
gradient in Eq. (13) also zero at Og Specificallyy, one can write:
VoD, Po) Ve / po(x))
(x(x)
dp(x))
Po(x)
Qa (a(x))
Po(x)Vo
du(x)
(D(x)
/ po(o)))"
(x)
2o
Vego (z)
du(())
(14)
Po
Po(
/ (x)))
go
(o(x)du(z)
Po (x)
Egg l'
(0()))
po(
Evaluating Eq. (4) at Og yields:
(00) = 8 (1) Ego [so(x)]
= f (1) / (0(xr) Vogo(=) du(xz)
(x(x)
= (1) Vo (0(z)d)(x)) 0.
Similarly, derive the Hessian matrix in Eq.
/
qo (x) po(x)) po (x
p"
(qo(x) so(x)) (go(x) sx()))
l' (0(z))
qo (x) (so(z)s0(z)T Vose(z))
dp(x))
(15)
po(x))
= =f (0((2)
fr1
90
go s83g + p 90 (sesg + Voso)
du(()).
Po
17

---

Evaluating Eq. 15 at 8u yields:
(00) 5 (1) [
+ (1) Engen [N00 T1 + Bugo 800] (1)Gr, (1) (Gao Gb (1)
(Voool en.))
where Eggo Vosol 0o
= - Goo due to Bartlett's second identity 5 Therefore, for any variant within the f f-divergence family. Eq. (13) simplifies to:
Dy(0) 2 f (1) 0b) To, 6o) 0(101)).
(16)
For Sal.div (1) I log and S(2 (1) 1 (log 1)², it is straightforward to verify that (1)
tog (1) 1. Thus, the 12-log estimator closely approximates the KL divergence up to second order.
C Detailed Experimental Setup
For training the encoders, we employ the AdamW optimizer [40] with learning rate of 1 x 10-3 for the medium-scale setting and 4 x 10-4 for the large-scale setting. For updating the amortization network,
the Adam optimizer [40] universally set at learning rate of 1 10 Following the temperature scaling technique proposed in [66]. we rescale the contrastive loss and adopt an additional regularizer p. i.e., (NCE, rescaled INCE/
+ P/T. Consistent with [66], set p = 6.5 for the medium-scale setting and p = 8.5 for the large-scale setting. Additionally, we introduce temperature annealing to further improve learning efficiency. Specifically, to -80 during the last quarter of epochs in the medium-scale setting and to 8.55 during the last third of epochs in the large-scale setting. For AMORLIP f-div, we add 12-log loss regularizer with coefficient of 0.1 and sweep between two divergence formulations js} and report the best results in Table |I
reset p
ImageNet Shifts
Retrieval
DataComp 38 Tasks
AmorLPrratsi
Improvement (%)
a e 6
AmorLiPpp4w
0
Inageltet.t Imeyotie-
emagttte up insgettei 1t
Oblecthel Imsgreeerdd
Flete Mecoco
Gatesh- Of RESRCC Oxfont-l77 Pel SUNKeT
Osted Flowrrs (22
HITTI Vancie Draenua
MitSTT
Flast
Foret-o
GTL.10 Eurohir
GIFRR sig
TOTAL
OL BLEVR Counto
Dasociabll tasture carriod
POiG Alonn
Rariitred So
Patoccame Boscaamppon CLE UP Clatercc Daller stivel
swww Heniitii GTORB Camerjon17
Passal Voc
Figure 5: Breakdown of absolute improvement (%) made by AMORLIP over CLIP model on all 38
DataComp Tasks 23 under medium scale setting.
D More DataComp Results Breakdown
Figuree]jsooeasee the absolute improvement delivered by AMORLIP over CLIP across all Datacomp tasks in the medium setting.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: comprehensively demonstrate the effectiveness and efficiency the pro-
posed AMORLIP, we provide detailed theoretical derivations (Sections 3.1 3.2 3.3 Ap-
pendix B) alongside extensive empirical validations (Section Appendices and D
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In Appendix A we discuss the limitations of this work, including the method-
ological scope, potential socictal impacts, and computational efficiency considerations.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
19

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: We provide detailed proofs and clearly stated assumptions in Appendix to support the theoretical claims presented in the main text rigorously.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We detail the proposed algorithm in Section 3 and provide comprehensive disclosures of hyperparameter settings, training configurations, evaluation metrics, and an in-depth analysis of experimental results in Section and Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibilityy can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibiliity In the case of closed-source models, it may be that access to the model is limited in
20

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We open-sourced our codebase with the paper. In the meantime, have provided detailed description of the proposed algorithm in Section 3 We have also described hyperparameter settings, training configurations, evaluation metrics, and an in- depth analysis experimental results Section 4 and Appendix C
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines SaceeggiideeOodeSSSubiiiiiieeeee
(https://nips.co hnttos:/nips.ccppubiicqueeeeee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurIPS code and data submission guidelines (https://nips.. pouubiccuuidds/ooodeuuuiiiiiiiii
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
We have provided detailed description of the experimental settings, such hyperparameter settings, training configurations, evaluation metries, and in-depth analysis of experimental results in Section aadd Appendix
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[No]
Justification: Though error bars are not explicitly reported in our evaluations, we have assessed our method across multiple training scales, diverse evaluation metrics, and broad range of downstream tasks Section 4 to reduce the variability in supporting the effectiveness and efficiency of the proposed method.
21

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We detail the compute resources in Section 4 and Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the htt.hh/ecwrssccccuuuii/cttaccaaa
NeurlPS Code of Ethics
Answer: [Yes]
Justification:
The research conducted in the paper fully conforms with the NeurIPS Code of
Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
22

---

Answer: [Yes]
Justification: We have discussed both potential positive societal impacts and negative societal impacts in Appendix A
Guidelines:
The answer NA means that there no societal impact of the work performed.
the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative socictal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
The algorithm described in this paper primarily targets text-image represen- tation learning and thus does not inherently bear significant risks of misuse. Nonetheless, we acknowledge potential indirect societal impacts related to large-scale vision-language models Appendix A
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All models and datasets employed in this paper, including ResNer 3i ViT 20, and the Conceptual Captions datasets [57 10), are publicly accessible and utilized
in full compliance with their respective licenses.
23

---

Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
asseets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paperswithcode.com ppperswithcodd.ccm//ttaatett com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper primarily contributes methodology enhancement in text-image representation learning and does not rely on the release of any specific new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

---

Answer:
[NA]
Justification:
for This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy httts://beerriasscccoofeeeeeeees what should or should not be described.
25


Reviews:
[{"review_id": "ds1NSqlVVU", "reviewer": "Reviewer_h4E3", "summary": "This paper introduces AmorLIP, an efficient framework for contrastive language-image pretraining that reduces reliance on large batch sizes in standard CLIP training. By amortizing partition function computations through lightweight neural networks inspired by spectral factorization, the authors propose a modular two-stage training procedure alternating between learning amortization networks and updating backbone encoders. The paper analyzes two amortization objectives, presents stability enhancements, and demonstrates improved zero-shot classification and retrieval across 38 multimodal tasks.", "questions": "- Could the authors discuss or report any experiments using dramatically larger pretraining datasets (e.g., LAION-400M or larger), or clarify whether the observed improvements are expected to hold in settings with hundreds of millions or billions of samples?\n- Additionally, there are several tasks in Figure 2 where the improvements are negative or negligible. Could the authors analyze these cases and provide possible explanations or hypotheses about the conditions under which the proposed method may not offer significant benefits?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n- The approach is technically sound and well-grounded in energy-based modeling and amortization, with clear derivations and practical reformulation.\n- The method is original, leveraging spectral factorization and neural amortization, and stands out from standard buffer-based schemes.\n\n**Weaknesses**\n- All experiments are conducted with pretraining only on CC3M or CC12M, and do not empirically verify scalability to larger datasets like LAION-400M, which would strengthen the claims.\n- Although the method aims to reduce dependency on large batch sizes, experiments only use batch sizes of 1k and 2k. It would be better to show how AMORLIP and CLIP perform across a wider range of batch sizes.\n- SigLIP also partially addresses the large batch size issue, but Table 1 shows SigLIP underperforms CLIP at small batch sizes, which is inconsistent with previous reports. The reason for this discrepancy should be clarified.", "was_revised": true, "final_justification": "I appreciate the authors' thorough responses, which have addressed the majority of my concerns. Regarding the scaling experiments with larger datasets, I commend the authors' efforts and strongly recommend including these results in the final version. Based on the comprehensive responses and other reviewers' feedback, I will raise the rating.", "rebuttal": "We thank the Reviewer for their valuable suggestion.\n\n  \n## [Experiments using larger pretraining datasets]\n\n> All experiments are conducted with pretraining only on CC3M or CC12M, and do not empirically verify scalability to larger datasets like LAION-400M, which would strengthen the claims.\n\nIn our submission, we comprehensively evaluated AmorLIP and baselines using medium- (CC3M) and large-scale (CC12M) datasets across 38 tasks. As shown in Table 1 and Figure 3, AmorLIP consistently outperforms all baselines and demonstrates robust and transferable gains. Specifically, AmorLIP surpasses CLIP by 2.63% (CC3M) and 3.01% (CC12M) on average across DataComp tasks.\n\nTo further investigate scalability to larger datasets, as the reviewer suggested, we have initiated additional experiments on LAION-400M with the x-large setup from [1]:\n\n| Visual Encoder | Training Epochs | Batch Size | GPUs     |\n|----------------|-----------------|------------|----------|\n| ViT-B/16       | 42 Epochs       | 640×8      | 8 × H100 |\n\nUnfortunately, despite our best efforts to secure additional compute resources, our current computational constraints mean these experiments require more than four weeks (with an average step time of ~0.9s) and cannot be completed during the rebuttal period. We appreciate the Reviewer for this suggestion and commit to finalizing these experiments and including detailed results and analyses in an updated version of our paper if permitted.\n\n## [Ablation study on training batch size]\n\n> Although the method aims to reduce dependency on large batch sizes, experiments only use batch sizes of 1k and 2k. It would be better to show how AMORLIP and CLIP perform across  a wider range of batch sizes.\n    \nWe thank the Reviewer for the valuable suggestion. As our motivation was indeed to reduce dependency on large batch sizes, we initially focused on moderate batch sizes (1k and 2k). Following the Reviewer’s recommendation, we conducted additional ablation studies across a wider range of batch sizes (512 to 2048) under the medium-scale setting. We report the zero-shot IN-1K accuracy as follows:\n\n| Model | Batch size=512 | Batch size=1024 | Batch size=2048 |\n|------------------|--------|---------|---------|\n| CLIP | 15.55% | 16.84% | 18.40% |\n| FastCLIP | 19.62% | 20.58% | 21.42% |\n| **AmorLIP (l2-log)** | **20.94%** | **21.50%** | **22.07%** |\n\nThese results demonstrate that AmorLIP consistently outperforms CLIP and FastCLIP[1] across varying batch sizes. Notably, AmorLIP surpasses CLIP by at least 3.68% for all batch sizes, with even greater improvements at smaller batch sizes. This further confirms the effectiveness of AmorLIP in achieving strong performance with significantly reduced batch sizes.\n\n## [Clarification on SigLIP results]\n\n> SigLIP also partially addresses the large batch size issue, but Table 1 shows SigLIP underperforms CLIP at small batch sizes, which is inconsistent with previous reports. The reason for this discrepancy should be clarified.\n    \nThank you for raising this point. We evaluated all baseline methods, including CLIP and SigLIP, using the original OpenCLIP implementation [2], following hyperparameters from [1]. SigLIP indeed outperforms CLIP on zero-shot ImageNet&Shifts classification, aligning with observations in both the original SigLIP paper [5] and related works on similar-scale datasets (e.g., CC3M and CC12M in [3]).\n\nDespite SigLIP’s better performance on certain classification tasks, our experiments show it may underperform CLIP across the broader set of 38 DataComp tasks. This observation aligns with the theoretical analysis presented in [4]. Specifically, [4] indicates that SigLIP’s binary-NCE (classification-based) loss requires an additional input-dependent bias term for self-normalization and model consistency, which can significantly impact performance. This theoretical insight matches empirical observations from the original SigLIP paper ([5], Sec. 4.9), where it was reported that performance is sensitive to the initialization and learning dynamics of this bias term.\n\nIn our experiments, we initialized SigLIP’s bias term at -10 and made it learnable throughout training, consistent with the original implementation. However, we observed that after approximately 46k training steps, SigLIP’s contrastive validation loss began to increase, indicating potential overfitting or instability under small to medium-scale settings. We hypothesize this is due to the learning complication of the bias term under small/medium scale, aligning with empirical findings from [4].\n\nIn contrast, we emphasize that the proposed AmorLIP is built based on ranking-based NCE, which is explicitly self-normalized by design. This design choice eliminates the need for carefully tuning an additional bias term and thus avoids these stability issues, with relative small batch size, sharing the benefits of both CLIP and SigLip\n\n## [Explanation on negative improvements]\n\n> Could the authors analyze these cases and provide possible explanations or hypotheses about the conditions under which the proposed method may not offer significant benefits?\n    \nWe observe that the proposed AmorLIP underperforms on tasks requiring recognition of highly specialized local image patterns with minimal global semantic meaning, such as PatchCamelyon and Camelyon17 (metastatic tissue classification). We hypothesize this is because these tasks emphasize repetitive local structures rather than broader semantic context.\n\nHowever, we emphasize that AmorLIP achieves positive gains on nearly 80% of tested tasks across both scale settings, with more than 60% of tasks showing absolute performance gains exceeding 2%. Additionally, AmorLIP demonstrates consistent improvements on widely adopted tasks, including ImageNet & Shifts and retrieval tasks on Flickr and MSCOCO, even with small batchsize, and thus, less memory cost.\n\nThis highlights the effectiveness of our method. In future work, we will explore extensions of AmorLIP to better address image pattern-focused tasks, such as those encountered in PatchCamelyon.  \n\n## Reference\n\n[1] Wei, Xiyuan, et al. \"Fastclip: A suite of optimization techniques to accelerate clip training with limited resources.\" arXiv preprint arXiv:2407.01445 (2024).\n\n[2] G. Ilharco, M. Wortsman, R. Wightman, C. Gordon, N. Carlini, R. Taori, A. Dave, V. Shankar, Namkoong, J. Miller, H. Hajishirzi, A. Farhadi, and L. Schmidt. Openclip, July 2021.\n\n[3] Wang, Bokun, et al. \"On discriminative probabilistic modeling for self-supervised representation learning.\" arXiv preprint arXiv:2410.09156 (2024).\n\n[4] Ma, Zhuang, and Michael Collins. \"Noise contrastive estimation and negative sampling for conditional models: Consistency and statistical efficiency.\" arXiv preprint arXiv:1809.01812 (2018).\n\n[5] Zhai, Xiaohua, et al. \"Sigmoid loss for language image pre-training.\" Proceedings of the IEEE/CVF international conference on computer vision. 2023."}, {"review_id": "GT9ZPESR7M", "reviewer": "Reviewer_31TJ", "summary": "This paper addresses a major bottleneck in Contrastive Language-Image Pretraining (CLIP): the immense computational cost and the requirement for extremely large batch sizes. To overcome this, the authors propose AMORLIP, an efficient pretraining framework that significantly reduces the dependency on large batches. The core idea is to \"amortize\" the most computationally expensive part of the contrastive loss—the partition function (the denominator in the softmax)—using lightweight neural networks. Extensive experiments on the 38 tasks of the DataComp benchmark show that AMORLIP not only matches but consistently and significantly outperforms standard CLIP and other efficiency-focused baselines.", "questions": "1. The paper compares two amortization objectives: an f-divergence-based objective and an L2-log objective. The L2-log objective appears to perform slightly better and more consistently. Is there a theoretical or intuitive reason to prefer one over the other? For instance, are there specific data regimes or training dynamics where the unbiased but potentially higher-variance f-divergence estimator might be advantageous?\n\n2. The stability of the training relies on a target network for the amortization function, updated via EMA. How sensitive is the training process to the EMA decay factor α? Figure 4a shows that α=0.999 is best, but is there a \"cliff\" where a slightly different value leads to instability, or is the performance relatively robust within a certain range?\n\n3. The core idea is to learn a parametric model (λ_θ) of the partition function. How does the capacity of this model (i.e., the MLP size) affect performance? The ablation in Table 2b suggests that smaller networks can be better, which is somewhat counter-intuitive. Could you elaborate on the hypothesis that this mitigates overfitting during the amortization stage?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths\n\nSolves a Critical and Practical Problem: The computational barrier to entry for training large-scale vision-language models like CLIP is a significant issue for the research community. This paper directly tackles this problem, proposing a solution that makes high-performance CLIP training more efficient and accessible.\n\nPrincipled and Novel Methodology: The approach is theoretically well-grounded. Reformulating the contrastive objective as an EBM and using amortization for the partition function is a clever and powerful idea. The derivation of an efficient representation for the partition function via spectral factorization provides a solid theoretical foundation for the method, moving beyond heuristic solutions.\n\nStrong and Comprehensive Empirical Results: The paper's claims are backed by extensive experiments. AMORLIP is shown to outperform strong baselines (CLIP, SigLIP, SogCLR, FastCLIP) across 38 diverse downstream tasks at two different scales (Table 1). The performance gains are not marginal, with relative improvements of up to 12.24% over CLIP. This demonstrates the robustness and generalizability of the learned representations.\n\n\nWeaknesses\n\nScalability to Billion-Scale Datasets: While the experiments on CC3M and CC12M are substantial, the ultimate test for a CLIP training method is its performance at the scale of modern foundation models (e.g., LAION-5B). The authors acknowledge this limitation. While the trends strongly suggest the benefits would hold or even increase at a larger scale, demonstrating this empirically remains an important next step.\n\nLack of Statistical Significance Reporting: The paper does not report error bars or conduct statistical significance tests for its results. Although the consistency of improvements across 38 tasks provides strong evidence, reporting the variance across multiple training runs with different random seeds would have further strengthened the claims of superiority over baselines.\n\nComplexity of the Framework: The proposed AMORLIP framework, with its EBM formulation, target networks, EMA updates, and specialized amortization objectives, is considerably more complex to implement than the standard CLIP InfoNCE loss. While the paper provides clear algorithmic details, this added complexity could be a minor barrier to adoption compared to the simplicity of the original CLIP.", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the Reviewer for providing insightful comments. Please find the responses below:\n\n## [Statistical significance reporting]\n\n>  Although the consistency of improvements across 38 tasks provides strong evidence, reporting the variance across multiple training runs with different random seeds would have further strengthened the claims of superiority over baselines.\n  \nWe thank the Reviewer for raising this important point. To address the concern, we conducted additional experiments with two more seeds on CC3M and report the average performance (AVG) across 38 DataComp tasks along with standard deviation (STD):\n\n| Model | AVG | STD | Improvement over CLIP |\n|------------------|---------|--------|----------------------|\n| CLIP | 21.76% | 0.30% | - |\n| FastCLIP | 23.38% | 0.25% | +1.62% |\n| AmorLIP (l2-log) | 24.07% | 0.24% | **+2.31%** |\n\nThese results consistently demonstrate that AmorLIP achieves a substantial performance improvement of over 2.3% compared to CLIP, which significantly exceeds the observed variance (STD < 0.3%).\n\n## [Complexity of the framework]\n\n>  While the paper provides clear algorithmic details, this added complexity could be a minor barrier to adoption compared to the simplicity of the original CLIP.\n    \nThank you for the feedback. We acknowledge that the proposed method introduces additional components to the conventional NCE implementation. The main complexity arises from the amortization stage, where the function $\\lambda_{\\theta}(\\cdot)$ is optimized using the selected amortization objective. Since our proposed objectives (KL-divergence and l2-log) are inherently supported by popular ML frameworks such as PyTorch and JAX, integrating these should not pose significant difficulties. In contrast, the modifications to the contrastive learning stage are minor, as we simply replace the partition function with predictions from $\\lambda_{\\theta}(\\cdot)$.\n\nTo ensure reproducibility and ease of implementation, we have provided detailed implementation guidelines in Sec. 4 and Appendix D. Additionally, we included extensive ablation studies covering target networks, EMA updates, and amortization objectives. We hope these studies can serve as practical guidance for optimal parameter selection. Finally, we will release our implementation code, accompanied by comprehensive comments, along with the updated version of the paper to further assist our readers.\n\n## [Comparison between two amortization objectives]\n\n>   The L2-log objective appears to perform slightly better and more consistently. Is there a theoretical or intuitive reason to prefer one over the other? For instance, are there specific data regimes or training dynamics where the unbiased but potentially higher-variance f-divergence estimator might be advantageous?\n   \nEmpirically, the L2-log objective demonstrates better training dynamics and stability. We provide a theoretical explanation for this observation as follows.\n\nIn Appendix B3, we first show that the L2-log objective is a biased estimator of a specific instance of the f-divergence objective (i.e., KL divergence). However, as discussed in Appendix B4, the bias of the L2-log objective remains relatively small, as it closely approximates the KL divergence up to second order under mild conditions. Additionally, the L2-log formulation (Eq.10) ensures that optimization occurs entirely in log-space, effectively mitigating numerical issues associated with the exponential operation $\\exp(\\cdot)$ present in the KL divergence (Eq.9). Consequently, the L2-log objective reduces training variance induced by the rapidly changing temperature factor. The empirical results shown in Table 1 align well with this theoretical reasoning.\n\nIn summary, we recommend using the L2-log objective when the learnable temperature undergoes significant changes during training, as commonly observed in raw CLIP pretraining (where temperature ranges broadly from approximately 14.27 to 100 according to prior efforts [2]). Conversely, in scenarios where the temperature remains stable or changes minimally (e.g., CLIP fine-tuning), the unbiased nature of the f-divergence objective could potentially offer advantages. We plan to investigate this scenario further in future studies.\n\n## [Discussion on the EMA decay factor]\n\n>   How sensitive is the training process to the EMA decay factor α? Figure 4a shows that α=0.999 is best, but is there a \"cliff\" where a slightly different value leads to instability, or is the performance relatively robust within a certain range?\n   \nThe EMA decay factor $\\alpha$ determines how much past model information is retained. Theoretically, a larger $\\alpha$ stabilizes model training by smoothing gradient estimation noise and effectively preserving information from previous batches. In Figure 4(a), we clarify that the rightmost column (average accuracy across all 38 tasks) serves as the most reliable performance indicator since it aggregates results over more tasks compared to ImageNet & Shifts (5 tasks) and Retrieval (2 tasks). This overall metric clearly demonstrates a consistent trend of improved performance with increasing $\\alpha$, which supports our theoretical analysis. Additionally, we observe robust performance across a wide range of $\\alpha$ values, consistently surpassing baseline methods. For instance, within ImageNet & Shifts, varying $\\alpha$ introduces at most a 1.1% performance difference, and even the lowest achieved accuracy (17.81% at $\\alpha=0.92$) significantly exceeds the CLIP baseline (14.03%). This confirms the effectiveness and broad applicability of our stability technique across diverse hyperparameter settings.\n\n## [Capacity of the parametric model]\n\n>   How does the capacity of this model (i.e., the MLP size) affect performance? The ablation in Table 2b suggests that smaller networks can be better, which is somewhat counter-intuitive. Could you elaborate on the hypothesis that this mitigates overfitting during the amortization stage?\n    \nThank you for raising this important point. The parametric model $\\lambda_{\\theta}(\\cdot)$ maps a high-dimensional feature representation (e.g., 512-dimensional vector for ViT-B/32) into a positive scalar. Given this setup, the model could potentially be prone to overfitting. Reducing the model size acts as a form of capacity control and architectural regularization.\n\nA smaller network provides architectural regularization in two perspectives. First, it constrains the model to simpler functional mappings and enhances generalization. Second, it introduces minor prediction noise into the estimated partition function, effectively acting as implicit regularization for the representation learning objective (Eq. 8-10). This subtle perturbation may encourage the encoders to learn more robustly.\n\nAdditionally, we emphasize that performance differences resulting from varying the model size are relatively minor (within 0.86%). This indicates the robustness of our proposed parameterization, which consistently achieves strong performance across different hyperparameter choices.\n\n  \n## Reference\n\n[1] Wei, Xiyuan, et al. \"Fastclip: A suite of optimization techniques to accelerate clip training with limited resources.\" arXiv preprint arXiv:2407.01445 (2024).\n\n[2] Radford, Alec, et al. \"Learning transferable visual models from natural language supervision.\" International conference on machine learning. PmLR, 2021."}, {"review_id": "pz6n0EuqNQ", "reviewer": "Reviewer_g7oi", "summary": "This papers focus on conventional image language pretraining stage. Different from contrastive learning based CLIP model with larger GPU memory and batch size, this work focus on improving the training efficiency and performance. The main idea is sourced from energy-based models.", "questions": "1. Can you provide more discussion about the difference between Sigmoid Loss for Language Image Pre-Training and this work in technique?\n2. Also, provide more discussion with Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Pros:\n\n1. Strong Performance Gains:\nThe proposed method demonstrates significant improvements, particularly in image classification and retrieval tasks. It is extensively evaluated across all DataComp benchmarks, providing a thorough and convincing performance comparison.\n2. Comprehensive Baseline Comparisons:\nBeyond standard CLIP, the work includes fair and detailed comparisons against recent strong baselines such as FastCLIP, SigLIP, and SogCLR, highlighting the robustness and effectiveness of the approach.\n3. Innovative Use of Energy-Based Models:\nThe integration of an energy-based function into the CLIP framework is a novel and promising direction, potentially offering new insights into contrastive learning and model efficiency.\nCons:\n\nScalability on Larger, High-Quality Datasets Not Demonstrated:\n1. A key concern is the limited scale and quality of the pretraining datasets. While CC3M and CC12M are used with rule-based filtering, they still contain significant noise and lack the quality of more modern datasets like DataComp-1B, LAION-COCO, or ShareGPT4V. To convincingly demonstrate scalability and robustness, it is recommended to pretrain on at least 100M+ high-quality samples or preferably on the full DataComp-1B set. Addressing this limitation could significantly enhance the impact and credibility of the work.\n2. Limited Downstream Task Diversity:\nThe evaluation is primarily focused on classification and retrieval tasks within the DataComp benchmark. To better position this method as a general-purpose vision encoder, it would be valuable to include results on broader multimodal tasks such as visual question answering (VQA), image captioning, or referring expression grounding.\n3. Different model parameters scale is also not explored.", "was_revised": true, "final_justification": "The paper sets out to challenge prominent models like CLIP and SigLIP and the work is interesting. But the CC datasets are somewhat dated and the absence of code limits the ability to fully assess the generalizability and broader impact. I tend to accept this work.", "rebuttal": "Thank you for the insightful comments. Please find our responses below:\n\n## [Experiments using larger pretraining datasets]\n\n>  To convincingly demonstrate scalability and robustness, it is recommended to pretrain on at least 100M+ high-quality samples or preferably on the full DataComp-1B set. Addressing this limitation could significantly enhance the impact and credibility of the work. \n\nIn our submission, we comprehensively evaluated AmorLIP and baselines using medium- (CC3M) and large-scale (CC12M) datasets across 38 tasks. As shown in Table 1 and Figure 3, AmorLIP consistently outperforms all baselines and demonstrates robust and transferable gains, despite the noise present in the datasets. Specifically, AmorLIP surpasses CLIP by 2.63% (CC3M) and 3.01% (CC12M) on average across DataComp tasks.\n\nTo further investigate scalability to larger datasets, as the reviewer suggested, we are working on validating the effectiveness of our proposed method with much larger-scale datasets. We have initiated additional experiments on LAION-400M with the x-large setup from [1] (listed in the following Table) and plan to continue on the full DataComp-1B dataset afterwards.\n\n| Visual Encoder | Training Epochs | Batch Size | GPUs |\n|----------------|------------------|------------|----------|\n| ViT-B/16 | 42 Epochs | 640×8 | 8 × H100 |\n\nUnfortunately, despite our best efforts to secure additional compute resources, our current computational constraints mean these experiments require more than four weeks and cannot be completed during the rebuttal period. We appreciate the Reviewer for this suggestion and commit to finalizing these experiments and including detailed results and analyses in an updated version of our paper if permitted.\n\n## [Downstream task diversity]\n\n>   To better position this method as a general-purpose vision encoder, it would be valuable to include results on broader multimodal tasks such as visual question answering (VQA), image captioning, or referring expression grounding.\n\nThank you for this valuable feedback. Our evaluations primarily focused on zero-shot classification and retrieval tasks using the standard DataComp benchmark, consistent with prior works [1-3], to clearly demonstrate AmorLIP’s effectiveness in foundational representation learning.\n\nWe agree with the Reviewer that evaluating broader multimodal tasks, such as visual question answering (VQA), image captioning, or referring expression grounding, can further establish our method as a general-purpose vision-language encoder. However, since these tasks typically require additional generative modeling or specialized fine-tuning steps, they introduce complexities that might obscure direct comparisons in foundational representation pretraining. Thus, we chose to maintain a clear and direct evaluation scope within classification and retrieval tasks.\n\nNevertheless, we acknowledge the importance of extending our evaluation to general multimodal tasks and will actively explore these in our future work.\n\n## [Different model scales]\n\n>   Different model  parameters scale is also not explored.\n    \nThank you for highlighting this important aspect. Following common practices in prior work [1], our experiments evaluated AmorLIP across two dataset scales (CC3M and CC12M) and two encoder architectures (ViT and ResNet). As shown in Table 1 and Figure 3, our results indicate that the advantages of AmorLIP consistently hold or even strengthen as model and dataset scales grow. For fairness, we kept all comparisons to baselines with the same model size. Additionally, we studied the impact of varying the parametric amortization model’s scale (Table 2b) and confirmed its robustness.\n\nWe appreciate the Reviewer’s suggestion and acknowledge that exploring different encoder model parameter scales would further demonstrate the scalability and effectiveness of AmorLIP. Currently, we are actively scaling our experiments to larger datasets (e.g., LAION-400M), as mentioned previously, and we plan to systematically investigate varying encoder scales in future experiments.\n\n## [Differences between AmorLIP and SigLIP[5]]\n\n>   Can you provide more discussion about the difference between Sigmoid Loss for Language Image Pre-Training and this work in technique?\n    \nWe thank the Reviewer for highlighting this important distinction. Below, we clarify and summarize the core differences more concisely:\n\n**Core difference (contrastive learning objective):** The fundamental difference between AmorLIP and SigLIP lies in their respective contrastive objectives for representation learning. Specifically, AmorLIP learns representations by parameterizing the target conditional distribution $\\mathbb{P}(u_l\\mid u_{l’})$ using a spectral factorization view from energy-based models (EBMs). This representation can be efficiently optimized through ranking-based noise contrastive estimation (ranking-based NCE), as adopted in CLIP. In contrast, SigLIP employs a binary classification-based NCE, implicitly assuming the partition function is independent of conditional inputs.\n\n**Learning complications:** As analyzed in [4], SigLIP’s binary NCE (classification-based) formulation requires an additional bias term for self-normalization and model consistency. This bias term can significantly affect performance, as shown in the original SigLIP paper ([5], Sec. 4.9). The observations also align with our experiments that AmorLIP shows consistent performance gain over SigLIP. In our experiments, we observed that after approximately 46k training steps, SigLIP’s contrastive validation loss began to increase, indicating potential overfitting or instability under small to medium-scale settings. We hypothesize this is due to the learning complication of the bias term under small/medium scale, aligning with empirical findings from [4]. In contrast, AmorLIP’s ranking-based NCE is inherently self-normalized and does not rely on an additional bias term, thus avoiding the stability and performance sensitivity observed in SigLIP.\n\n**Computational efficiency differences:** Conventional CLIP implementations rely heavily on the costly all_gather(·) operation at each step to estimate the partition function (Eq. (3)). SigLIP reduces this computational burden by dividing sample pairs into independent blocks, though at the expense of assuming an input-independent partition function. AmorLIP similarly enables efficient blockwise computation of the loss by parameterizing the partition function as a lightweight network dependent solely on the conditional inputs. This approach retains computational efficiency while preserving the theoretical advantages of ranking-based NCE.\n\nWe will further highlight these key differences clearly in future paper updates.\n\n## [Discussion with Spectral Contrastive Loss[6]]\n\n>   Also, provide more discussion with Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss.\n    \nThank you for raising this insightful comparison. Below, we concisely summarize the theoretical and empirical distinctions between AmorLIP and the spectral contrastive loss (SCL) from [6]:\n\n**Core difference (contrastive objective):** SCL derives its contrastive objective via an SVD-based factorization of the target conditional distribution $\\mathbb{P}(u_l\\mid u_{l’})$. The key difference between SCL and AmorLIP lies in the second term of their respective objectives. Specifically, AmorLIP employs an exponential operation (Eq. 5 in our paper), which effectively maps features into an infinite-dimensional space, enhancing model expressiveness. In contrast, SCL uses a quadratic term (Eq. 6 in [6]), expanding features into only a finite polynomial-dimensional space, which may limit expressiveness. Thus, SCL often requires significantly larger feature dimensions to achieve comparable performance (e.g., the feature dimension of 8192 is used in [6] for ImageNet classification.)\n\n**Gradient variance analysis:** The gradient of SCL’s quadratic term introduces an unnormalized weighting term $2\\psi_l(u_l)^\\top \\psi_{l’}(u_{l’})$. This potentially causes high gradient variance and instability during optimization. AmorLIP, on the other hand, employs a softmax weighting via its NCE loss, resulting in bounded and more smoothed gradient estimates.\n\n**Empirical validation:** To further validate our analysis, we conducted ablation studies by replacing CLIP’s loss with SCL and pretrained on CC3M. The ImageNet-1K classification results below highlight SCL’s limited expressiveness and AmorLIP’s representation capability.\n\n| Model | SCL | CLIP | AmorLIP (l2-log) |\n|------------------|---------|---------|------------------|\n| Acc. | 11.64% | 16.84% | **21.50%** |\n\n  \n  \n\n\n## Reference\n\n[1] Wei, Xiyuan, et al. \"Fastclip: A suite of optimization techniques to accelerate clip training with limited resources.\" arXiv preprint arXiv:2407.01445 (2024).\n\n[2] Radford, Alec, et al. \"Learning transferable visual models from natural language supervision.\" International conference on machine learning. PmLR, 2021.\n\n[3] Zhai, Xiaohua, et al. \"Sigmoid loss for language image pre-training.\" Proceedings of the IEEE/CVF international conference on computer vision. 2023.\n\n[4] Ma, Zhuang, and Michael Collins. \"Noise contrastive estimation and negative sampling for conditional models: Consistency and statistical efficiency.\" arXiv preprint arXiv:1809.01812 (2018).\n\n[5] Zhai, Xiaohua, et al. \"Sigmoid loss for language image pre-training.\" Proceedings of the IEEE/CVF international conference on computer vision. 2023.\n\n[6] HaoChen, Jeff Z., et al. \"Provable guarantees for self-supervised deep learning with spectral contrastive loss.\" Advances in neural information processing systems 34 (2021): 5000-5011."}, {"review_id": "GRBx9F58SE", "reviewer": "Reviewer_ePbH", "summary": "AMORLIP proposes a contrastive language-image pretraining framework that amortizes the partition function using a lightweight network, in order to eliminate the rigid dependence on large batches of negatives as in CLIP training. From the perspective of energy-based models, the partition function Z is expressed as a linear form of random features, and a small MLP is used to amortize its computation in a two-stage alternating optimization scheme. The framework also introduces two objective variants—f-div and l2-log—to balance the bias-variance trade-off. Experiments are conducted on both CC-3M + ResNet-50 and CC-12M + ViT-B/32 settings, covering 38 downstream DataComp tasks. Compared to CLIP, AMORLIP achieves up to a 12.24% improvement in average zero-shot performance.", "questions": "In visual self-supervised contrastive pretraining, it has traditionally been assumed that a large number of negative samples is necessary. However, some studies [1,2] have shown that if certain hyperparameters—such as the temperature coefficient, loss scaling, or gradient weighting—are properly tuned, contrastive learning can still perform as well as or even better than the original large-batch setting, even with only a few negatives per pair.\nDo you think similar effects could be achieved in vision-language contrastive pretraining? If not, what differences do you think might account for this?\n\n[1] EqCo: Equivalent Rules for Self-supervised Contrastive Learning\n\n[2] Are All Negatives Created Equal in Contrastive Instance Discrimination?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n- The proposed method is both novel and well-motivated. By parameterizing the partition function in contrastive learning using spectral decomposition and a lightweight MLP, it avoids reliance on large batches and all-gather operations. Although the NCE community has long treated Z as a parameter and approximated it with conditional networks, AMORLIP is the first to effectively transfer this idea to large-scale cross-modal contrastive learning, resolving distributed bottlenecks via \"two-stage amortization + sparse all_gather.\"\n- The experimental evaluation is thorough, spanning 38 downstream tasks across two scales, and comparing systematically with four efficient CLIP baselines (e.g., SigLIP, SogCLR, FastCLIP).\n\n### Weaknesses\n- The use of two amortization losses and multiple hyperparameters (α, β_T, T_online, f_d) increases the tuning burden, which may hinder scalability in large-scale pretraining. Some ablation studies show the performance is sensitive to β_T. It would be helpful to provide empirical guidelines for setting these hyperparameters based on data scale and model size.\n- Only zero-shot classification and retrieval results are reported, with no fine-tuning experiments or analysis.", "was_revised": true, "final_justification": "I have carefully read the authors’ response and the other reviewers’ comments, and my questions have been largely resolved. Overall, I find this paper solid compared with others in its field, so I will keep my score unchanged.", "rebuttal": "We sincerely appreciate the Reviewer for providing valuable feedback. Please find the responses below: \n\n## [Empirical guidelines for hyperparameter setup]\n\n>  It would be helpful to provide empirical guidelines for setting these hyperparameters based on data scale and model size.\n    \nThank you for this valuable feedback. Based on our empirical evaluation (Sec. 4), we provide concise and practical guidelines for choosing these hyperparameters:\n\n-   **Amortization loss:** We proposed two amortization loss variants: f-divergence (unbiased) and l2-log (biased but with reduced variance). We recommend the l2-log objective when the learnable temperature varies significantly during training (as in standard CLIP pretraining, temperature from ~14.27 to 100 [1]). In scenarios with stable temperature (e.g., fine-tuning), the unbiased f-divergence objective may perform better.  \n      \n    \n-   **EMA decay factor $\\alpha$:** The EMA decay factor $\\alpha$ determines how much past model information is retained. Theoretically, a larger $\\alpha$ stabilizes model training by smoothing gradient estimation noise and effectively preserving information from previous batches. Based on the empirical results in Figure 4(a), we recommend using a relatively large $\\alpha$ value, typically 0.999 achieves the best results in small or medium scales.  \n      \n    \n-   **Combination weight $\\beta_T$:**  Empirically, a moderately high value of $\\beta_T$, such as $\\beta_T=0.8$, facilitates amortization learning and particularly improves retrieval performance. However, excessively large values (e.g., $\\beta_T=1.0$) may hinder effective updates from new batches and negatively affect overall performance. Figure 4b suggests that $\\beta_T=0.8$ provides the best balance with optimal overall performance.  \n      \n    \n-   **Update frequency $T_{\\text{online}}^{-1}$ for amortization network:** The amortization network should be updated less frequently than the encoder models (Table 2a). Empirically, a frequency of $T_{\\text{online}}^{-1}\\leq\\frac{1}{8}$ offers an optimal balance between performance and computational overhead for small and medium scales.  \n      \n    \n-   **Capacity ($f_d$) of the amortization network:** Reduced network complexity (smaller $f_d$) mitigates potential overfitting and minimizes memory overhead. Based on empirical results, we recommend setting $f_d$ to 0.5 or even 0.25 to achieve the best balance between efficiency and performance.  \n      \nFinally, we emphasize that AmorLIP consistently outperforms baseline methods even with suboptimal hyperparameter settings. The effectiveness of AmorLIP helps reduce the necessity for extensive hyperparameter tuning.\n\n  \n\n## [Additional fine-tuning experiments]\n\n>  Only zero-shot classification and retrieval results are reported, with no fine-tuning experiments or analysis.\n    \n\nThank you for this suggestion. Our evaluations primarily focused on zero-shot classification and retrieval tasks using the standard DataComp benchmark, consistent with prior works [1-3], to clearly demonstrate AmorLIP’s effectiveness in foundational representation learning.\n\nAs suggested by the Reviewer, incorporating fine-tuning experiments would indeed make our evaluation more comprehensive and provide valuable insights into how the learned representations transfer when fine-tuned on downstream tasks. However, our work, consistent with prior studies [1-3], primarily focuses on developing a high-performing pre-training approach that is dataset-agnostic. Fine-tuning stage adapts representations to each specific dataset, which could potentially compensate for or mask weaknesses in learning general and robust representations during pre-training. Additionally, the complexities introduced by fine-tuning tasks might obscure direct and clear comparisons of foundational representation quality. Thus, we intentionally chose to maintain a clear and direct evaluation scope using only zero-shot classification and retrieval tasks.\n\nNevertheless, we acknowledge the importance of fine-tuning evaluations and will actively pursue them in future extensions of our work.  \n\n## [Transferability of Hyperparameter Tuning in AmorLIP]\n\n>   Do you think similar effects could be achieved in vision-language contrastive pretraining? If not, what differences do you think might account for this?\n    \nWe thank the Reviewer for this insightful question. Indeed, both vision-language (VL) and visual self-supervised learning (SSL) rely fundamentally on contrastive (NCE) loss; thus, insights such as loss scaling and temperature tuning are potentially valuable across both domains. However, we would like to highlight a crucial difference specific to VL pretraining, which makes directly transferring these techniques non-trivial.\n\nIn VL datasets, one image can correspond to multiple valid captions, and multiple images can share similar descriptions, leading to a significant issue of false negatives [4]. These false negatives (semantically related pairs mistakenly treated as unrelated) introduce complexities beyond the visual SSL setting. Consequently, directly applying aggressive loss scaling or gradient weighting as in visual SSL [5,6] might unintentionally amplify the impact of false negatives, potentially degrading VL model performance.\n\nThat said, certain principles from SSL remain beneficial for VL contrastive learning. Specifically, in AmorLIP, our amortized partition function can be viewed as an adaptive, gradient-free weighting mechanism. The amortization network dynamically learns these weights based on input samples and carefully designed amortization objectives, to inherently capture cross-modal relations. This suggests that adaptive loss scaling and gradient weighting can still be beneficial for VL, provided they account explicitly for cross-modal complexities.\n\nFurthermore, we emphasize that techniques such as loss scaling and temperature tuning are orthogonal to our method, and thus could potentially be combined with AmorLIP to further boost performance. For instance, as detailed in Appendix D, we have already incorporated a rescaled contrastive loss to enhance training stability: $\\ell_{\\text{NCE, rescaled}}=\\ell_{\\text{NCE}}/\\text{stopgrad}(\\tau)+\\rho/\\tau$, where $\\tau$ is the learnable temperature and $\\rho$ is the scaling factor, following best practices from [2].\n\nIn summary, while our framework provides a more generalized methodology for VL pretraining, integrating insights from well-established visual SSL techniques remains promising. We plan to explore these connections further in our future work.\n\n  \n\n## [Paper formatting]\n\nThank you for pointing out the typos. We will correct them in our updated version.\n\n  \n\n##  References\n\n[1] Radford, Alec, et al. \"Learning transferable visual models from natural language supervision.\" International conference on machine learning. PmLR, 2021.\n\n[2] Wei, Xiyuan, et al. \"Fastclip: A suite of optimization techniques to accelerate clip training with limited resources.\" arXiv preprint arXiv:2407.01445 (2024).\n\n[3] Zhai, Xiaohua, et al. \"Sigmoid loss for language image pre-training.\" Proceedings of the IEEE/CVF international conference on computer vision. 2023.\n\n[4] Bulat, Adrian, Yassine Ouali, and Georgios Tzimiropoulos. \"FFF: Fixing Flawed Foundations in contrastive pre-training results in very strong Vision-Language models.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[5] EqCo: Equivalent Rules for Self-supervised Contrastive Learning\n\n[6] Are All Negatives Created Equal in Contrastive Instance Discrimination?"}]

Author rebuttal:

Meta-review: This work addresses a critical and practical problem in contrastive learning, where immense computational cost and the requirement of large batch sizes for effective training.  The authors introduces AmorLIP, an efficient training framework for contrastive language-image (CLIP). AmorLIP significantly reduces the dependency on large sizes in standard CLIP training by amortizing partition function computations (the denominator in the softmax, which is the most computationally expensive part of the contrastive loss) through lightweight neural networks inspired by spectral factorization. The authors propose a modular two-stage training procedure alternating between learning amortization networks and updating backbone encoders  The authors evaluate AmorLIP on 38 zero-shot classification and retrieval tasks from the DataComp benchmark and show improved performance with AmorLIP.

Strength of the paper: 
- An original method leverages spectral factorization and neural amortization. Approach is also technical sound. The use of Energy-Based Models is also an innovative idea.
- Good selection of downstream evaluation tasks provides useful understanding and support of efficacy of proposed approach.
- The work solves a critical and practical problem to reduce computational barrier to entry the area of training contrastive models like CLIP. This work proposes a solution that makes high-performance CLIP training more efficient and accessible.

Reviewers align on most of the strengths of the paper from the high level and see its novelty, contribution, and relevance to NeurIPS venue. Thus we recommend this paper to be accepted to the conference.

After most questions being addressed and clarified during the rebuttal and discussion stage, reviewers feel a few areas that can improve the paper.  

To improve:
- AmorLIP was evaluated on CC3M and CC12M, while the frontier datasets used in the area has been scaled to billions (e.g., DataComp-1B, MetaCLIP). It's understandable that training with such large scale datasets could be too costly for an exploratory paper. However, we would recommend the authors 1) discussing such limitation and the gap between datasets/scale used in this work and datasets used at frontier practices, and what could be the path to close such gap in future, and 2) consider expanding experiment scale in future work. This is also applicable to the model sizes.
- Many discussion between reviewers and authors are valuable to include in the final manuscript
Decision: Accept (poster)