Maximizing Incremental Information Entropy for Contrastive Learning
Authors: Jiansong Zhang
Zhuoqin Yang
Xu Wu
Xiaoling Luo
Peizhong Liu
Linlin Shen
Venue: ICLR 2026 Poster (2026)
Keywords: Self-supervised Learning; Contrastive Learning; Incremental Entropy Maximization; Information Bottleneck
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Contrastive learning has achieved remarkable success in self-supervised representation learning, often guided by information-theoretic objectives such as mutual information maximization. Motivated by the limitations of static augmentations and rigid invariance constraints, we propose IE-CL (Incremental-Entropy Contrastive Learning), a framework that explicitly optimizes the entropy gain between augmented views while preserving semantic consistency. Our theoretical framework reframes the challenge by identifying the encoder as an information bottleneck and proposes a joint optimization of two components: a learnable transformation for entropy generation and an encoder regularizer for its preservation. Experiments on CIFAR-10/100, STL-10, and ImageNet demonstrate that IE-CL consistently improves performance under small-batch settings. Moreover, our core modules can be seamlessly integrated into existing frameworks. This work bridges theoretical principles and practice, offering a new perspective in contrastive learning.

Published as a conference paper at ICLR 2026
MAXIMIZING INCREMENTAL INFORMATION ENTROPY FOR CONTRASTIVE LEARNING
Jiansong Zhang! Zhuoqin Yang!.), Xu Wu', Xiaoling Peizhong Liu", Linlin S
1.34
1 School of Computer Science and Software Engineering, Shenzhen University, Shenzhen, China
2 School of Engineering, Huaqiao University, Quanzhou, China
3 Computer Vision Institute, School of Artificial Intelligence, Shenzhen University, Shenzhen, China
4 Guangdong Provincial Key Laboratory of Intelligent Information Processing, Shenzhen University, Shenzhen, China
ABSTRACT
Contrastive learning has achieved remarkable success in self-supervised represen- tation learning, often guided by information -theoretic objectives such mutual information maximization. Motivated by the limitations of static augmentations and rigid invariance constraints, we propose IE-CL (Incremental-Entrooy Contrastive Learning). framework that explicitly optimizes the entropy gain between aug- mented views while preserving semantic consistency. Our theoretical framework reframes the challenge by identifying the encoder as an information bottleneck and proposes a joint optimization of two components: learnable transformation for entropy generation and an encoder regularizer for its preservation. Experiments on 100ARO STL-10, and ImageNet demonstrate that IE-CL consistently improves performance under small-batch settings. Moreover, our core modules can be seamlessly integrated into existing frameworks. This work bridges theoretical principles and practice, offering new perspective in contrastive learning.
INTRODUCTION
Self-supervised contrastive learning has emerged as a cornerstone paradigm for representation learn- ing, enabling models to extract rich semantic features without explicit labels. At its core, contrastive learning constructs latent space where semantically similar samples converge while dissimilar ones diverge Wang & Isola (2020); Le-Khac et al. (2020). Despite rapid advances in this field, fundamental tension persists between theoretical understanding and practical implementation. While recent works have decomposed contrastive objectives into alignment and uniformity principles WWng & Isola 2020); Zhang al. (2024), they offer limited insight into the dynamic information landscape that unfolds during the learning process.
Current contrastive frameworks such as SimCLR Chen et al. (020aa), MoCo et al. (020), and their variants rely heavily static, human-engineered augmentations and large batch sampling to enforce invariance and representational diversity. These approaches impose rigid constraints on the learning dynamics: augmentations must balance semantic preservation with transformational complexity, while batch scaling faces inevitable hardware limitations. Despite substantial engineering efforts to refine augmentation strategies Chen & He (202T): Chen al. (202T): Tian et al. (2020a), these methods fundamentally lack a principled mechanism for adaptively expanding the representational capacity of each instance while maintaining semantic coherence.
We address this limitation by reconceptualizing contrastive learning through the lens of incremental information entropy, novel framework that quantifies the expansion of representation space during learning. Inspired by information -tteorrttttt objectives Hjelm et al. (2019): |arddes et al. (2022). we focus on how additional controllable uncertainty is gained between augmented views to strengthen learning. Our key insight is that optimal contrastive learning could maximize the conditional entropy gain between positive views while preserving their mutual information. The effectiveness of maximizing this incremental entropy is contingent on its preservation through the deep encoder,
Corresponding author: 11 shee @szu.edu.cm

---

Published as a conference paper at ICLR 2026
Anchor
Encoder
projector
IE-Contrastive Learning
fa
Z
InfoNCE IncEnrr+IncEEnroppy
Momentim Update
z z'Zz
Augmentation
Encoder
projector
Chelesingg Space
Incrementai
4
Query
Incremental
Loarnable parameter
Entroyy
Jacobion i constratot
Aeyooonaaiin
Quary Reprevsntatinn
Figure 1: Overview of the proposed IE-CL. We define incremental entropy as the absolute change in entropy induced by classical contrastive augmentations (see Definition B.2). To optimize the contrastive learning process, we propose the Sample A Augmentation Incremental Block (SAIB), learnable module that ensures the local Jacobian determinant 1. By incorporating sample-level incremental entropy into contrastive optimization, we establish principled framework that improves the effectiveness of supervieee representation learning.
which often acts as an information bottleneck. We therefore propose framework that jointly optimizes two synergistic components: a learnable transformation for entropy generation and an explicit regularization of the encoder for entropy preservation. This principled approach highlights the overlooked trade-off between semantic invariance and representational expressivity.
Based on this theoretical insight, we introduce IE-CL (Incremental-Enne Entrrop Contrastive Learning), framework that explicitly optimizes for controlled entropy gain. To achieve this, we design learnable nonlinear transformation module (SAIB) that adaptively expands each sample's local representation manifold by guaranteeing strictly positive Jacobian determinant. Crucially, to ensure this generated entropy not lost during encoding, this module is paired with explicit encoder regularization mechanism that encourages information preservation. These components work in concert with a Kullback-Leibler divergence constraint to balance entropic expansion against semantic consistency. IE-CL operates efficiently under small batch sizes (e.g., 256), enabling broader applicability without the hardware burden of large-batch training.
The contributions this work can be summarised as: (1) We propose a new theoretical framework for contrastive learning that identifies the deep encoder as information bottleneck. We posit that effective representation learning requires jointly optimizing for both entropy generation at the input and entropy preservation during encoding. (2) Based on this framework, we design novel model, IE-CL. featuring two key components: a learnable transformation (SAIB) to generate rich input-level entropy, and an encoder regularizer (e.g., Spectral Normalisation) to ensure its faithful propagation. (3) We provide a detailed empirical analysis demonstrating IE-CL's effectiveness, particularly in small-batch settings. We also show that our core module can enhance other self-supervised models in plug-and-pla. manner.
Our work bridges the gap between informationntheoreti principles and practical contrastive learning, offering more complete theoretical understanding and algorithmic innovations that significantly advance the field self-supervised representation learning.
RELATED WORK
Self-supervised Paradigm Self-supervised learning has emerged as a prominent paradigm for feature extraction without reliance on labeled date[Liu et al. (2022); Wang et al. (2023) Yang et al. (2024). A central research focus has been the development of effective encoding frameworks that facilitate rich representation learning in the absence of supervision. Notable approaches include contrastive learningChen et al. (2020b c): Chen & He (2021); | Chen et al. (2021): Caron et al. (202T)
Oquab et al. (2024); | Wu et al. (2023), masked autoencoder Zhou et al. (2022b); Xie et al. (2022)) Wei
et al. 2022 g Chen et al. (2024). and advances loss function desigr Ermolov et al. (2021) Zbontar
al. 2021); Tian al. 2020b) Ozsoy et al. (2022): [ardes et al. 2022) Among these, contrastive
2

---

Published as a conference paper at ICLR 2026
learning has become dominant paradigm due to its ability extract rich features through well- designed pretext tasks within a dual-encoder frameworkGarrido et al. (2023).It It has frequently served as benchmark for evaluating self- supervupee learning methods. Recently, the emergence of masked pretext tasks has opened new avenues for learning representations in a label-free setting. Works such as He et al.] (2022) and [ao et al. (2022) creatively adapted masking strategies from NLP to vision, enabling image reconstruction from masked tokens using spatial priors and positional embeddings. Following this, Jinghao Zhou Zhou et al. (2022a) further abstracted feature representations in image self self-supervised learning using a know
distillation- masking learning strategy, also demonstrating the effectiveness of masking strategies in dual-track sepersseee frameworks like contrastive learning. Concurrently, the realm of non-masking pretext Mo et al. (2023); Huang et al. (2022); Oinar et al. (2023) in self-supervised learning has witnessed numerous novel contributions. Notably, Tong et.a Tong et al. (2023) employed an extremely high number of patches as a self-supervised signal, proposing a self- self-supervised learning framework requiring only one epoch. The remarkable success of these works is largely attributable to researchers' deepening understanding of data processing methods in self-supervised learning.
Contrastive Learning Theory The empirical success of contrastive learning has spurred extensive theoretical investigations. Early work focused on analyzing the mathematical foundations of con- trastive loss. Saunshi et al. da,[anhhhi al. (2019) were among the first to show that contrastive learning can produce linearly separable representations under certain conditions. Wang and Isola Wang & Isola (2020) decomposed the InfoNCE loss into two interpretable terms allgnmeett and unifor- mity where alignment promotes similarity between positive pairs and uniformity mitigates feature collapse. This formulation offered a unified lens for understanding contrastive learning and inspired connections to broader informationntheorrtte frameworks, such as mutual information maximiza- tion Tian ) and noise contrastive estimation Hu et al. (2023). From a spectral graph theory viewpoint, Chen et al. HaoChhn al. (2021) revealed that contrastive learning implicitly learns the Laplacian of the data graph, showing equivalence to spectral clustering objectives. This was later extended to dynamic graphs (Shen et al. (2022) and connected to kernel methods |Wang al.) (022).. Tan et al. TTn et al. (2024) introduced a-order mutual information to unify contrastive and non-contrastive losses, bridging matrix-based contrastive methods (e.g.. Barlow Twins Zbontar et al. (2021) VICReg Bardes et al. )2022) with classical dimensionality reduction techniques such as ISOMAP. Beyond spectral perspectives, Zimmermann et al. Zimmermann et al. (2021) proposed a probabilistic interpretation, viewing contrastive learning as reverse engineering the data generation process under the assumption of a uniform latent prior. This aligns with the framework of noise contrastive estimation Gutmann & Hyvärinen (20T0) and sheds light on its generalization behavior.
Lee et al. Lee et al. (2021) further established a link between contrastive loss and the variational lower bound of the data likelihood using latent variable models, As non-contrastive approaches such as BYO Grill et al. (220) and Barlow Twins Zbontar et al.] (2021) gained popularity, recent efforts have focused on theoretically characterizing the distinction between contrastive and non-contrastive paradigms Zhang et al. (2024).
3 METHOD
3.1 INFORMATION ENTROPY IN CONTRASTIVE LEARNING
Contrastive Learning Objectives The primary goal of contrastive learning is optimize the similarity between positive pairs (anchor and query) while repelling negative samples, thereby enabling effective cuf-ssuper representation learning under the assumption of independently and identically distributed (i.i.d)) samples within a mini-batch. For a given batch of original images B = (a;
1,2,... ..,,... N). the representation 2; E Z, denotes the embedding of image v,, computed via the encoder fe,- This embedding typically originates from the anchor branch in dual-stream contrastive architecture. The representation 2,* denotes the positive sample, whereas 2; E Zziith j # i) corresponds to negative samples derived from different instances in the batch. These negative and positive representations are encoded by the second branch, fo,, and are collectively referred to as the query set. The standard form of the objective can be expressed as:
exp(sim (2,, 2 2/7/ Σ log 1=1
N
L(Z,, Zz) = N
(1)
Σnn exp(siii 2/)T)
3

---

Published as a conference paper at ICLR 2026
where sim( 2y) is the similarity function, and is the temperature parameter that controls the sharpness of the probability distribution. Cosine similarity is often employed, defined as:
sim (1, zz)
The optimization objective seeks minimize the distance between each anchor and its positive counterpart (2,, 2,t), while maximizing the separation from all negative samples 2j + 2," thereby facilitating effective self-supervised learning.
Mutual Information Theory Mutual information provides a principled framework for analyzing self--upprvised learning objectives, as discrete probability distributions can be interpreted as samples drawn from underlying continuous distribution.
Lemma 3.1 (Equivalence between InfoNCE minimization and mutual information maximization). Let Z = fe(X) be the embedding of input and Z the corresponding positive sample. Then, representation, the mutual information satisfies
based the
min LIfoNCE max IZ I(Z:Z+) 2 log LigoNCC
Proof. The InfoNCE loss for positive pair (z,2*) can be written as
(o0ssi(2222)//55
L -Bp(p.z11 log exp(sim(2, 2 //r) + Ey=1 exp expp(sim(2) 2/T)
(3)
Using the Varahann representation,
I(Z sup (o(2.a))(T(,,zT)) - log E Ep()))(++) (e2()))))
(4)
T
Choosing T(z, sim(2,**)/ //T yields the lower bound
[(2; 2+) 2 log LInfONCE-
(5)
Thus, minimizing TIaONNE equivalent to maximizing T(Z; 2+).
3.2 INCREMENTAL ENTROPY IN CONTRASTIVE LEARNING
evident that optimizing the distributions of and Zg fundamentally depends on obtaining effective and discriminative feature representations. From an cooommaiiontthooeeee standpoint- abstracting away encoder-specific inductive biases, the learning objective can be intuitively framed as minimizing the conditional entropy H + [Z) while maximizing the marginal entropy H(Z ). The incremental entropy is thus defined first from the input side.
Definition 3.2 (Based on the concept of Shannon Entropy, the change in information entropy of given sample X after a transformation is applied, resulting in X', is referred to as the Incremental Information Entropy).
A H(X) H(X), H(X) = Σ
p(xs)) -(x)=--pte
log p(x;)
The relationship between a transformation and the change in entropy can be precisely quantified. For linear transformation g represented by matrix A, the incremental information entropy is given by:
( = H(((x))) - H(X) = log det A
Proof. When the transformation g is linear function, the probability density function of can be written as:
p'x('') (x' -b) -b))
(6)
dee A
Replacing 1/x (x') with HX'):
4

---

Published as a conference paper at ICLR 2026
H(X)= =-
10 1-'ff'xx'' (=''('") de'
(7)
--/ --/Px(A (x -b))
log (x -b))
die
|det
|det Al
Logarithmic term expansion:
-1x')=-fa Dx -b)
(8)
[ogex x (A (A' b)) - loo det Al de'
det Al
Split into two parts:
H(x) = 1000=-fpp (A
(9)
log (A 1 (a' b)) dx' log det Al Perform permutation on the variable u A Ar' b) with de' |eet Aldu:
1
det A
H(X'= - --
px (u) log px (u) du + log det A
(10)
To wit:
H(X')= (X) + log det A|
(11)
Incremental information entropy is:
AH(X) = H(X') - H()) = log det A]
(12)
This relationship makes clear why standard augmentations have limitations. When the transforma- tion g linear isometry (such rotation, cropping, mirroring, etc.), its matrix representation A has a determinant det A| = 1, which leads to AH O. such cases, these augmentations can enrich sample diversity the batch-level without altering the instance-level entropy.
However, a critical challenge arises from the nature of deep encoders themselves. In information theory, the Data-Processing Inequality states that post-processing cannot increase information. For differential entropy, this implies that the entropy of a variable's representation Z = f(X) is bounded by the entropy of the original variable X. Specifically, for deterministic function f, the change in entropy is governed by:
H(F(X)) S (E llog det [x))]]]
(13)
where (y( is the Jacobian of the transformation at This inequality highlights crucial issue in representation learning: a deep encoder, acting as the function f. can potentially become an information bottleneck, diminishing the entropy of its input. Any diversity generated at the input level is not guaranteed to be preserved in the final representation space. To address this, we introduce the IE-CL framework, a holistic approach that pairs an entropy generation module with an entropy-preservve preserving encoder. We formalize this approach in the following proposition.
Proposition 3.3 (Principle of Constrained Incremental Entropy Maximization). Let X be a negative sample, 96 be non-linear transformation, and z'- folg.(X-)) be the final representation encoded by an encoder fo. To robustly increase the representation entropy H HZZ"),, maximizing the input-level incremental entropy (X(X)) alone is insufficient. A joint condition is required: (1) Input Entropy Generation: The transformation 9o must be optimized to maximize the incremental entropy (X (2) Encoder Entropy Preservation: The encoder fo must be simultaneously constrained to preserve the entropy of its input. Satisfying both conditions provides principled path toward maximizing the diversity of negative representations for effective contrastive learning.
Theoretical Argument. Our argument is based on the Data- Pata-Psoeessing Inequality for differential entropy. Let X' = g
the transformed input to the encoder. The entropy of the final representation, - = (XX). is bounded as follows:
(X- (x
H(Z')) H(fo(X') & H(X') + [lg det (x')
(14)
This inequality reveals the core challenge. The first condition, maximizing AH(X ") is equivalent to maximizing (X') since H(X-) is constant with respect the parameters & of go. However,
5

---

Published as a conference paper at ICLR 2026
even if (X') is large, the second term, which depends on the Jacobian of the encoder fe, can be large negative value, effectively nullifying the gains from the first term. This occurs the encoder acts strong information bottleneck, aggressively compressing its input space.
Therefore, to guarantee that a large H (X') induces a correspondingly large H(Z")), we introduce the second requirement: constraining the encoder. Specifically, by regularizing fo be entropy- preserving (e.g.. via Lipschitz continuity constraints), we effectively bound the term E [loog] det Jf l). thus preventing it from becoming excessively negative. This condition ensures that the entropy injected by Is is faithfully propagated to the final representation space.
Consequently, the joint optimization of an sntropy-generaatng transformation and an entropy- preserving encoder is necessary and sufficient strategy to robustly increase the final representation entropy H (Z"
3.3
MAXIMIZING INCREMENTAL INFORMATION ENTROPY
Based the framework established in Proposition 3.3 our goal is to co-optimize both the generation of incremental entropy and its preservation through the encoder. While encoder regularization is implemented via standard techniques such spectral normalization, the core of our contribution lies in the design of a learnable, discussed, cannot linearly provide incremental information entropy. To address this, we propose nonlinear transformation implemented via batch-level pixel-wise operations, explicitly designed to induce positive entropy increments in the query branch.
transformation I6- Isometric transformations, as
Sample Augmentation Incremental Block (SAIB) Our objective is to maximize mutual in- formation by minimizing the conditional entropy I!t Z) on the query side. To inject a peeeattiss-rreervingg but intropy-expansse transform into the query branch we introduce the SAIB module, which couples ViT-style positional encoding Dosovitskiy et al. (202I) with a non-linead residual stack. The input X E R3xHxW is first patchified into a matrix P E RC H/p (/P)x(e") (as in VIT, C 3), where the mini-batch occupies the channel dimension. 1 x 1 - Conv + 3 x 3- Conv + 1 x 1 Conv wwthh channel expansion ratio 2-is wrapped by two skip connections (see Appendix Figure (). Owing to the channel-expanding residual design, the local Jacobian A of SAIB satisfies det Al > 1 almost everywhere (Appendix Ib guaranteeing positive incremental entropy (H(P) > O. After the non-linear block we reshape P back to the spatial layout and add troisième skip connection x'=X + reshape(P').
sequence of
KL regularisation to avoid degenerate ga- entropy expansion may lead to distributional drift. We therefore penalise the Kullback-Leibler divergence
Because Io acts only on the query branch, aggressive
DRIlP' ll ()) = f log (z(2) ((z)
dz,
(15)
where =z) is the SAIB-transformed query distribution and q(z) - =p( =z) is the anchor distribution. Assuming q is Gaussian with mean μ and variance TfT,
DKL. (p& 9) H(Z-") |lue all
d
log 2TTO)
(16)
20
where ns E[ and is the feature dimension.
Overall objective. Our final objective function holistically integrates all components of the frame- work established in Proposition 3.3 We minimise the combined loss:
Linal = LIMGONEE + 3 Dxr. (ps ll 4) - > H( + Creg encoder + 7 R(go)
(17)
with X, 1,,,,7777 > O. Here. the InfoNCE loss drives the primary representation learning task, while the KL-divergence term ensures that the transformations induced by SAIB maintain semantic consistency, The negative entropy term, -X H(Z directly optimizes for greater diversity in the representation
'The query branch corresponds the lower path in Fig. 1, whose encoder parameters are fea-

---

Published as a conference paper at ICLR 2026
space, serving the practical objective for maximizing incremental entropy. Crucially, the novel encoder regularizer, 7) Creg encoder. operationalizes the entropy preservation principle central to our framework, ensuring that the diversity generated by SAIB is not lost during encoding. The final term, y R(ga), ). is an optional weight-decay penalty on the SAIB module's parameters. This unified objective enables an end-to-end optimization of both entropy generation and preservation, yielding more robust representations.
4 EXPERIMENT & RESULT
4.1 EXPERIMENTAL. SETUP
Implementation Details We conducted upstream self-supervised learning experiments CIFAR- 10 Krizhessky & Hinton (2009), JUTAR-100Kriizevee & Hinton (2009), STL-10 Coates et al. (2011), and ImageNet Deng et al. (2009), followed by downstream evaluation the PASCAL VOC. To ensure fair comparison, we used BeesNet-based encoders across all experiments and fixed the random seed to 42 for reproducibility. For large-scale pretraining, employed ResNet-50 as the backbone and followed the standard MoCo configuration on ImageNet for consistent evaluation protocol. For ablation and scalability analysis, we used ResNet-15 18 with batch size of 256 across CIFAR-10, 10, CIFAR- 100, STL-10, and ImageNet, enabling controlled comparisons under limited capacity settings. This phase primarily showcases the effectiveness of the proposed method under smaller batch settings across varying dataset distributions.
Table 1:
The comparison of the proposed method with ResNet-50 as the backbone under different numbers of pre- teeitrriiin iterations. Using BOLD and Underline formatting to highlight the best and second results.
Method
100
200
400
800
Batch Size
SimCLR (CML)20)
Chen
al.
2020u
66.5
68.3
69.8
71.1
4096
SwAV (NamlPS'a0)
Lron al
20A0
66.5
69.1
70.7
71.0
4096
McCo-v2 (CVPR-20
Chen
20/0
67.4
70.9
71.3
256
SimSiam (ICCV221
21211111 Tec & HD
2021
68.1
70.0
70.8
71.7
256
NNCLR (ICCV'21)
Dinbbed
All4Onz (ICCV223)
Estopa a
65.4
66.0
66.6
68.9
1024
Matrix SSL (ICML 24
Zhangg
2024
69.2
69.9
71.1
71.9
512
Ours
68.3
70.9
71.7
73.2
256
Model Architectures IE-CL was implemented top of the MoCo framework, incorporating momentum encoder and symmetric contrastive loss as in SimCLR Chen & (202I). A ResNet backbone with the classification head removed was used symmetrically branches. The output features are -66-dmmensiona,, obtained by global average pooling. Each branch uses symmetric lhyee-layer projector with an MLP-BN architecture. The hidden dimension is set to 4096, and the final projection is 5 512-dimensional. The anchor encoder and SAIB module are updated via backpropagation, while the query encoder updated using momentum-basee moving averages. The pseudo-code of IE-CL is shown in
both anchor and query
Optimization and Hyperparameters We trained IE-CL using AdamW with batch size of 256, a base learning rate of 0.3, weight decay of le-5, and momentum of 0.9. Learning rates were scheduled via cosine annealing. The momentum coefficient m for the momentum encoder was set to 0.9. The regularization weights for our final objective (Eq. 17) were configured as À = 0.2 for entropy maximization, 3 0.09 for the KL.-divergence, and y Te for the SAIB weight decay. Crucially, the entropy-preserving Spectral Normalization to every convolutional layer of the encoder fe. and its corresponding weight was set used SGD with batch sizes of 512, learning rate of 0.03, momentum of 0.9, and weight decay of le-5. Cosine annealing was also used for scheduling. The linear classifier trained for 200 epochs, and we report the final epoch accuracy. All experiments were conducted on 8 x NVIDIA Tesla V100 GPUs (32GB), using Pororc 1.13 and Python 3.8.
encoder regularizer, Frey encoders was implemented by applying
1.0 as it is an architectural constraint rather than a loss term. For linear evaluation, we
7

---

Published as a conference paper at ICLR 2026
Table 2: Comparison of self-supervised learning methods on various datasets (left) and segmenta- tion/detection performance on PASCAL VOC2012 (right).
(a) Comparison based on ResNet--18 with batch size is 256.
(b) Results PASCAL VOC2012 with ResNet-50 SSL pretrained.
Method
CIFAR-10
CIFAR-100
STL-10
Imagehet
DespCluster (BCCV"/8 18 Caron ot 2018
84.1
50.1
79.
41.1
SimCLR (ICML/20 oh AUAQ
91.1
65.3
90.1
52.4
Pretrained
mioU
mAP
McCo-22 (CVPR
BYOL. (NeurIPS 200mmi
PAN
01.0
91.3
Supercisal
73.76
SimSiam (ICCV"2)
Ches &
912
61.1
90.5
33.2
Random
38.35
40.90
W-MSE (ICML221
21
002
90.6
64.5
87.7
47.2
SimCLR
76.74
73,17
(ICCV22
2021
McCo-v
$30C (TNNLS
91.0
63.2
91.4
SimSium
77.09
73.45
MinEaa (PR 23 di
90.
66.1
91.5
Light McCo (CME
2023
57.9
1.21)
0.68)
Oars
92.1
69.5
91.9
59.4
4.2 MAIN RESULTS
Linear Eyaluation
We adopt the standard linear evaluation protocol Chen et al. (2000a; Grill al. (2020) He al. (202)) in which the pretrained anchor encoder is frozen, and linear classifier is trained on top. The anchor encoder, with the backbone network parameters frozen, is used for the linear evaluation process. A linear layer appended and trained using supervised signals while keeping the backbone fixed. Training data is augmented via random horizontal flipping, random cropping 224 x 224, and layer normalization. For evaluation, input images are resized from 256 x 256 to 224 x 224. Table T reports Top-1 accuracy after IE-CL pretraining on ImageNet using ResNet-50 over 100, 200, 400, and 800 epochs. Table 2 shows linear probe results on other datasets using ResNet-18 (trained for 300 epochs ImageNet and 1,000 on smaller datasets). IE-CL consistently outperforms previous baselines across all settings.
Blr Rotets Noree Color Crop SAIB
51.1
49.8
42.9
29.7
33.7
8.1
35.9
80
45.5
46.4
38.9
-40
Query
31.2
21.0
26.5
15.2
13.4
9.4
19.5
800
25.8
-20
18.4
9.5
6.8
9.3
6.5
5.5
9.2
10
21.9
4.0
Reax
(II Ciaaviaa Notec
GC Chansian Sia:
SU SAIDH490
Crop
Cotor
Noise
Rotate
Blr
SAIB
Average
Anchor
Figure 2: Illustration the data augmentation opera- tors studied. The non-isometric transformation operator SAIB has learnable parameters, enabling non-prior aug- mentation for contrastive learning. Visualizing changes from epochs (d) to 400 epochs (h) shows KL divergence effectively constrains incremental entropy, preventing collapse.
Figure Ablation tests the relationship between SAIB and the previous pretext task. The image was resized to 224x224, and augmentation strength settings from Chen et al. (2020a) were applied, followed by two-by-two tests with SAIB placed on both sides the contrastive learning.
Transfer Learning To assess the transferability of the learned representations, evaluate IE-CL on two downstream tasks from PASCAL VOC 2012 Everingham et al. (2010); object detection and semantic segmentation. We use Faster R-CNN Ren et al. (2016) for object detection and DeepLab- v3 Chen et al. (2018) for segmentation, both with ResNet-50 backbones pretrained via IE-CL. For segmentation, training samples are augmented with random cropping and contrast-based enhancement. Adam is used with learning rate of 3 x 10-4 For detection, the model is trained using SGD with a learning rate of 44
Augmentation Dependency As SAIB is implemented at the data loading stage (see Appendix Algorithm[]] it can be interpreted as a learnable augmentation layer, contrasting with traditional pretext-hased augmentation schemes used in prior contrastive methods. Figure Preuue3presentss
presents comparative
8

---

Published as a conference paper at ICLR 2026
results for various augmentation strategies ImageNet-100, under 100 epochs of pretraining and linear evaluation. Our method exhibits robust performance gains under limited augmentation.
Table 3: Ablation study of the IE-CL, components on ImageNet-1k k using MoCo-v2 with a ResNet-18 backbone. We incrementally add our proposed components: the SAIB module for entropy generation. KL regularization for semantic consistency, and an Encoder Regularizer (implemented via Spectral Normalization) for entropy preservation.
Configuration
SAIB
KL Reg.
Encoder Reg.
Top-1
MoCo-v2 (Baseline)
X
X
x
52.50
Entropy Generation
X
X
58.80
X
Semantic Consistency
59.15
IE-CL (Full Framework)
59.41
Ablation Study To assess the contribution of each component in IE-CL, we performed an ablation study based on a MoCo-v2 baseline with a ResNet-18 backbone on ImageNet-1k, k.. As shown in T progressively adding the core modules Proposition 3.3 yields consistent gains. Introducing the SAIB module for entropy generation produces the largest improvement, confirming the benefit of maximizing input-level incremental entropy, while the KL-divergence term for semantic consistency further enhances performance by mitigating distributional drift. Crucially, the final row demonstrates that incorporating our proposed entropy preservation mechanism via an encoder regularizer ('Encoder Reg. ') provides an additional performance boost top of the already strong SAIB+KL configuration. This result provides strong empirical evidence for the central tenet of our framework: that optimal performance is achieved by jointly optimizing for both entropy generation at the input and entropy preservation through the encoder. We also found that cascading multiple SAIB modules offered diminishing returns, shown Table 4 thus we use single module in our main configuration.
Plug and Play Table demonstrates the plug-and ability of SAIB when integrated into other self-supervised learning frameworks on ImageNet- 100, including non-contrastive methods such as BYOL and SimSiam. At this point, SAIB is placed the Target side, similar to its placement on the Anchor side in contrastive learning. We further visualize entropy gains during training in Figure and 5 showing accelerated convergence and performance improvement attributed SAIB.
Table 5: Based the theory of maximizing inere- mental information entropy with non-isometric trans- formations, SAIB can seamlessly integrated to en- hance other self-supervised paradigms.
Table 4: Ablation on the number of cascaded SAIB modules within the full IE-CL framework. Perfor- mance slightly degrades with more than one module, indicating diminishing returns.
Method
Topl
Butch Size
Epoch
MoCo-v2
66.29
256
200
Configuration
SAIB Cascade
Top-1
BYOL
67.95
255
200
SimCLR
256
TE-CL. (Full Framework)
Ix 59.41
SimSiam
66.25
256
200
IE CL with more modules
2x
58.62
McCosv2 -
67.54 1.25)
256
58.71
BYOL + SAIB
68.76 (7 0.81)
255
200
SimCLR + SAIB
64,02 (1 0/68)
256
200
SimSiam
5 DISCUSSION, LIMITATION, CONCLUSION AND FUTURE WORK
This work introduces Sample Incremental Information Entropy and presents new framework, IE-CL, to advance mutual information maximization in contrastive learning. addresses the critical challenge of the encoder information bottleneck by jointly optimizing for entropy generation, via novel learnable transformation module (SAIB), and entropy preservation, via an explicit encoder regularizer. Our approach yields consistent improvements across various datasets, though several aspects merit further study. SAIB operates at the patch level and induces local pixel-space variations, which preserve semantic consistency but may limit expressiveness in modeling complex structures or higher- resolution tasks. Its reliance on convolutional priors also raises challenges for extension to vision transformers Vision Kolmogorov-Arnot Networks Yang et al. (2026). Accordingly, exploring

---

Published as a conference paper at ICLR 2026
Salf-Suparvised Training Loss on ImagaNet-1K
Incremental Entrapy vs. InfeMCE on ImagaNet-1k training
Via 5AB (leamina Ar1
Without SA
1.4
Cont Losa
Lis
ko
INOONEE
Incwass
Wen SA-8 56-41%
123
0.0g
Iteratoon (=10%)
Figure Comparison of SSL. training loss drop curves based on the proposed maximized incremental information entropy (SAIB) on ImageNet-Ie IK, using MoCo-v2 the baseline.
Figure 5: The variation of the incremental entropy A (X) the Query side and InfoNCE throughout the ilerations is shown.
broader cross-domain integration strategies and adapting them to diverse visual backbones Yang et al. 2025) may offer a promising direction for future research. Nonetheless, the core principle of IE CL, explicitly modeling and maximizing sample entropy, provides principled perspective for augmentation design and entropy-aware optimization, enriching representation diversity and deepening the information-theorrtie understanding of self-supervised learning.
ACKNOWLEDGMENTS
This work was supported by the National Natural Science Foundation of China under Grant 62576216 1,6, Guangdong Provincial Key Laboratory under Grant 2023B 212060076, and also supported by the Intelligent Computing Center of Shenzhen University.
ETHICS STATEMENT
This work does not involve human subjects, personally identifiable information, or sensitive medical data. All experiments conducted on publicly available benchmark datasets (CIFAR-10/100, STL- 10, ImageNet, and PASCAL VOC), which are widely adopted in the research community. We adhere strictly the ICLR Code of Ethics and the licensing terms of the datasets used. Our proposed method, IE-CL, is intended for advancing self suf-rvueee learning research in computer vision and does not present foresceable risks harmful misuse. We disclose all relevant implementation details, maintain academic integrity, and ensure that our research complies with ethical standards of reproducibility, transparency, and fairness.
REPRODUCIBILITY STATEMENT
We have made extensive efforts ensure the reproducibility of our results. A detailed description of the proposed method, IE-CL, including the theoretical derivations (Section 3), algorithm design (SAIB module), and the overall objective function, is provided in the main text. The experimental setup, datasets, and evaluation protocols are described in Section 4, with optimization hyperparameters and implementation details explicitly listed. Additional pseudo-code and derivations are included in the Appendix. We will release the source code, training scripts, and configuration files after the paper accepted, as supplementary materials to enable full reproducibility. Random seeds and hardware specifications are also reported to facilitate consistent replication of our experiments.
REFERENCES
Hangbo Bao, Li Dong. Songhao Piao, and Furu Wei. Beit: Bert pre- tre-training of image transformers. In International Conference on Learning Representations (ICLR), 2022.
10

---

Published as a conference paper at ICLR 2026
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: for supervieee learning. In International Conference on Learning Representations (ICLR), 2022.
regularization
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsuper- vised learning of visual features. In Proceedings of the European conference on computer vision (ECCV), pp. 132-149, 2018.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In Advances in Neural Information Processing Systems, volume 33, pp. 9912-9924. Curran Associates, Inc., 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9650-9660, 2021.
Liang- Ching- Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder- decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 801-818, 2018.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations, In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning (ICML), volume 119 of Proceedings of Machine Learning Research, pp. 1597-1607. PMLR, 2020a.
Ting Chen, Simon Kornblith, Kevin Swersky, Mobammad Norouzi, and Geoffrey Hinton. Big self-supervised models are strong semi- supersuper learners. In Advances in Neural Information Processing Systems, volume 33, pp.
Curran Associates, Inc., 2020b.
Xiaokang Chen, Mingyu Ding, Xiaodi Wang, Ying Xin, Shentong Mo, Yunhao Wang, Shumin Han, Ping Luo, Gang Zeng, and Jingdong Wang. Context autoencoder for self-supervised representation learning. International Journal of Computer Vision, 132(():208-22) 223, 2024.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15750-15758, 2021.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04299 2020c.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), pp. 9640-9649, 2021.
Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsuper- vised feature learning. Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, Proceedings of Machine Learning Research, pp. 215-223, PMLR, 2011.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 245,255,200 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representutions (ICLR), 2021.
Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. With little help from my friends: neighot- contrastive learning of visual representations. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9588- 2021.
11

---

Published as a conference paper at ICLR 2026
Aleksandr Ermolov, Aliaksandr Siarohin, Enver Sangineto, and Nieu Sebe. Whitening for self- supervised representation learning. In Proceedings of the 38th International Conference on Machine Learning (ICML), volume 139 of Proceedings of Machine Learning Research, pp. 3015- 3024. PMLR, 2021.
Imanol G. Estepa, Ignacio Sarasua. Bhalaji Nagarajan, and Petia Radeva. All4one: Symbiotic neighbour contrastive learning via self-attention and redundancy reduction. Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), pp. 16243-16253, 2023.
Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International Journal of Computer Vision, 88: 303-338, 2010.
Quentin Garrido, Yubei Chen, Adrien Bardes, Laurent Najman, and Yann Lecun. On the duality between contrastive and non-contrastive self-supervised learning. In International Conference on Learning Reprexentations (ICLR), 2023.
Cean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent a new approach self-supervised learning. In Advances Neural Informatison Processing Systems, volume 33, pp. 21271 11171-21184, Curran Associates, Inc., 2020.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the thirteenth international conference artificial intelligence and statistics, Proceedings of Machine Learning Research, pp. 297-304. PMLR, 2010.
Jeff Z HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for self supervvee deep learning with spectral contrastive loss. In Advances in Neural Information Processing Systems, volume 34, pp. 5000-5011. Curran Associates, Inc., 2021.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9729-9738, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEF/CVF Conference Computer Vision and Puttern Recognition (CVPR), pp. 16000-16009, 2022.
R Devon Hjelm, Alex Fedorov, Samuel
Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In International Conference on Learning Representations (ICLR), 2019.
Tianyang Hu, Zhili Liu, Fengwei Zhou, Wenjia Wang, and Weiran Huang. Your contrastive learn- ing is secretly doing stochastic neighbor embedding. In International Conference on Learning Representations (ICLR), 2023.
Junqiang Huang, Xiangwen Kong, and Xiangyu Zhang. Revisiting the critical factors of augmentation- invariant representation learning. In European Conference on Computer Vision (ECCV), pp. 42-58. Springer, 2022.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. 2009.
Phuc H Le-Khac, Graham Healy, and Alan F Smeaton. Contrastive representation learning: A framework and review. IEEE Access, 8: 8.193907-193934 2020.
Jason D Lee, Qi Lei, Nikunj Saunshi, and JIACHENG ZHUO. Predicting what you already know helps: Provable self-supervised learning. In Advances in Neural Information Processing Systems, volume 34, pp. 309-323. Curran Associates, Inc., 2021.
Shuo Li, Fang Liu, Zehua Hao, Licheng Jiao, Xu Liu, and Yuwei Guo. Minent: Minimum entropy for self-supervised representation learning. Pattern Recognition, 138: 10836, 2023.
12

---

Published as a conference paper at ICLR 2026
Shuo Li, Fang Liu, Licheng Jiao, Puhua Chen, and Lingling Li. Self-supervised self-organizing clustering network: a novel unsupervised representation learning method. IEEE Transactions on Neural Networks and Learning Systems, 35(2):187711871 2024.
Wenye Lin, Yifeng Ding, Zhixiong Cao, and Hai-Tao Zheng. Establishing a stronger baseline for lightweight contrastive models. In IEEE International Conference on Multimedia and Expo (ICME), pp. 1062-1067, 2023.
Yixin Liu, Zhao Li, Shirui Pan, Chen Gong, Chuan Zhou, and George Karypis. Anomaly detection attributed networks via contrastive self- self-suue learning. IEEE Transactions Neural Networks and Learning Systems, 33(6):2378-2392) 2022.
Shentong Mo, Zhun Sun, and Chao Li. Multi-level contrastive learning for self- self-super vision transformers. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 2778-2787 2023.
Chingis Oinar, Binh M. Le, and Simon S. Woo. Expectation-maxiiizatie via pretext-invariant representations. IEEE Access, 11:65266- 266566666526,, 2023.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo. Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang- Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision. Transactions on Machine Learning Research, 2024.
Serdar Ozsoy, Shadi Hamdan, Sercan Arik, Deniz Yuret, and Alper Erdogan. Self-supervised learning with an information maximization criterion. In Advances Neural Information Processing Systems, volume 35, pp. 35240-35255 Curran Associates, Inc., 2022.
Prajit Ramachandran, Barret Zoph, and Quoc V Le. Searching for activation functions. arXiv preprint arXiv:1710.05941, 2017.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. IEEE transactions on pattern analysis and machine intelligence, (996))1137-11-10 2016.
Nikunj Saunshi, Orestis Plevrakis, Sanjeev Arora, Mikhail Khodak, and Hrishikesh Khandeparkar. A theoretical analysis of contrastive unsupervised representation learning. In Proceedings of the 36th International Conference Machine Learning (ICML), volume 97 of Proceedings of Machine Learning Research, pp. 6628-56377 PMLR, 2019.
Kendrick Shen, Robbie M Jones, Ananya Kumar, Sang Michael Xie, Jeff Z HaoChen, Tengyu Ma, and Percy Liang. Connect, not collapse: Explaining contrastive learning for unsupervised domain adaptation. In Proceedings of the 39th International Conference on Machine Learning (ICML), volume 162 of Proceedings of Machine Learning Research, pp. 19847-19878. PMLR, 2022.
Zhiquan Tan, Yifan Zhang, Jingqin Yang, and Yang Yuan. Contrastive learning is spectral clustering similarity graph. In International Conference Learning Representations (ICLR), 2024.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In European Conference on Computer Vision (ECCV), pp. 776-794. Springer, 2020a.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? In Advances in Neural Information Processing Systems, volume 33, pp. (827-6839, Curran Associates, Inc., 2020b.
Yuandong Tian. Understanding deep contrastive learning via coordinate wise optimization. In Ad- vances in Neural Information Processing Systems, volume 35, pp. 19511-19522 Curran Associates, Inc., 2022.
Shengbang Tong. Yubei Chen, Yi Ma, and Yann Lecun. Emp-ssl: Towards self-supervised learning in one training epoch. arXiv preprint 200.::2304.0397,, 2023.
13

---

Published as a conference paper at ICLR 2026
Shirui Wang, Wenyi Hu, Pengyu Yuan, Xuqing Wu, Qunshan Zhang, Prashanth Nadukandi, Ger- man Ocampo Botero, and Jiefu Chen. A self-supervised deep learning method for seismic data deblending using blind-trace network. IEEE Transactions Neural Networks and Learning Systems, 34(7):3405-33 -4,14, 2023.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. Proceedings of the 37th International Conference on Machine Learning (ICML), volume 119 of Proceedings of Machine Learning Research, pp. 9929-9939 PMLR, PMLR, 2020.
Yifei Wang, Qi Zhang, Yisen Wang, Jiansheng Yang, and Zhouchen Lin. Chaos is a ladder: A new theoretical understanding of contrastive learning via augmentation overlap. In International Conference on Learning Representations (ICLR), 2022.
Chen Wei, Haoqi Fan, Saining Xie, Chao- Wu, Alan Yuille, and Christoph Feichtenhofer. Masked feature prediction for self-supervised visual pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14668-14678 2022.
Zhirong Wu, Zihang Lai, Xiao Sun, and Stephen Lin. Extreme masking for learning instance and distributed visual representations. Transactions on Machine Learning Research, 2023.
Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu.
Simmim: A simple framework for masked image modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9653-9663 2022.
Zhengeng Yang. Hongshan Yu, Yong He, Wei Sun, Zhi-Hong Mao, and Ajmal Mian. Fully convolu- tional network-based self-supervised learning for semantic segmentation. IEEE Transactions on Neural Networks and Learning Systems, 35(1): (:11))1322 132-142, 142, 2024.
Zhuoqin Yang, Jiansong Zhang, Xiaoling Luo, Xu Wu, Zheng Lu, and Linlin Shen. Medkan: An advanced volmogorov-arnokk network for medical image classification. In IEEE International Conference Bioinformatics and Biomedicine (BIBM), pp. 3090-3097. IEEE, 2025.
Zhuoqin Yang, Jiansong Zhang, Xiaoling Luo, Xu Wu, Zheng Lu, and Linlin Shen. Vision kan: Towards an attention-free backbone for vision with kolmogorov-arnold networks. arXiv preprint arXiv:2601.2155 2026.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stephane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In Proceedings of the 38th International Conference on Machine Learning (ICMI., volume 139 of Proceedings of Machine Learning Research, pp. 12310-12322 PMLR, 2021.
Yifan Zhang, Zhiquan Tan, Jingqin Yang, Weiran Huang, and Yang Yuan. Matrix information theory for self-supervised learning. In Proceedings of the 41st International Conference on Machine Learning (ICML), volume 235 of Proceedings of Machine Learning Research, pp. 59897-5991 8. PMLR, 2024.
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong, Image bert pre-training with online tokenizer. In International Conference on Learning Representations (ICLR), 2022a.
Qiang Zhou, Chaohui Yu, Hao Luo, Zhibin Wang, and Hao Li. Mimco: Masked image modeling pre-training with contrastive teacher. In Proceedings of the 30th ACM International Conference on Multimedia, MM '2, pp. 4487-4495. Association for Computing Machinery, 2022b.
Roland S Zimmermann, Yash Sharma, Steffen Schneider, Matthias Bethge, and Wieland Brendel. Contrastive learning inverts the data generating process. In Proceedings of the 38th International Conference on Machine Learning (ICML), volume 139 of Proceedings of Machine Learning Research, pp. 12979-12990 PMLR, 2021.
14

---

Published as a conference paper at ICLR 2026
SUPPLEMENTARY MATERIALS
A THE USE OF LARGE LANGUAGE MODELS (LLMS)
In preparing this manuscript, we employed large language models (LLMs) solely for language polishing and grammar refinement. The LLMS were not involved in idea generation, theoretical development, algorithm design, experimental implementation, result analysis. All technical content, experiments, and conclusions presented this work are entirely the contribution the authors.
B THEORETICAL JUSTIFICATION FOR THE IE-CL FRAMEWORK
This appendix provides a detailed theoretical argument for our proposed Incremental Entropy Con- trastive Learning (IE-CL. framework. We first establish why maximizing the entropy of negative sample representations, H (Z")). ). is desirable objective within the InfoNCE framework. We then use the Data-Processing Inequality to formally demonstrate why naively maximizing input-level entropy is insufficient due to the information bottleneck deep encoders. Finally, we show how our full IE-CL objective function provides principled and complete solution to this challenge.
B. THE GOAL: MAXIMIZING NEGATIVE ENTROPY FOR BETTER CONTRASTIVE LEARNING
The standard InfoNCE loss for a positive pair (z, and a set of negative samples {2g drawn from a distribution q(= is:
N- Tk=1
exp(s(z,2*)/3
LInfONCE = E log
(18)
exp(s(2, 2+),, (N 1)E (xp(=(2,=-)/7))
Our core premise is that increasing the entropy of the negative distribution, HZ where g is the distribution of Z'- makes the contrastive task more challenging and thus compels the model to learn better representations. Let's formalize this.
The denominator of the InfoNCE loss can be implies that the negative samples = are more diverse and spread out in the representation space. This increased diversity makes it statistically more likely for negative samples to be close the anchor 2, thus increasing the expected value of the negative scores, E, [xxp(s(222 2-) )/r]].
partition function. A higher entropy H(Z"-)
This directly increases the value of the denominator, which turn increases the InfoNCE loss. To compensate for this more difficult learning signal (i.e., to minimize the loss), the optimizer is forced adapt the encoder parameters (01, 02) create sharper separation. This is primarily achieved by increasing the similarity of the positive pair, s(2, ++) f.
increased positive pair similarity implies that given an anchor z, its positive counterpart =² becomes more predictable. In conditional entropy, (Z)+ZZ) 4. According to the definition of mutual information, T(Z: Z = I(Z+) H IZ). decrease in conditional entropy (while the marginal entropy H(Z+) is kept non-trivial to prevent collapse) leads to an increase in the mutual information, (Z; t. This the ultimate goal of InfoNCE-based contrastive learning.
terms, this corresponds to a reduction in the
Thus, we have established the following desirable causal relationship:
max HZZ")) min H [Z+ Z2) = max I(2; z*) + min LIMONCE
(19)
This confirms that maximizing the entropy of negative representations is valid and principled objective for improving contrastive representation learning.
B.2 THE CHALLENGE: THE INFORMATION BOTTLENECK IN DEEP ENCODERS
Having established goal, the naive strategy would be to simply maximize the entropy the input of the encoder, H("")), using our SAIB module, 9a- However, this approach is fundamentally flawed because it ignores the transformative effect of the deep encoder, fo.
15

---

Published as a conference paper at ICLR 2026
The Data- Processing Inequality for differential entropy provides formal tool analyze this. Let X = 9 be the transformed imput. The entropy of the final representation, Z' (X
is bounded by the entropy of its input (X""))
H(Z")) = H(fo(X")) )) & H(X")) + (epe) det J,, (x')
(20)
where (x²) is the Jacobian of the encoder function fa evaluated at a'.
This incquality reveals the core challenge. While our SAIB module is designed to maximize H (X"-). the second term, Eloog det fo |l. which depends entirely on the encoder, can be a large negative value. This occurs if the encoder acts as severe information bottleneck, aggressively compressing or collapsing its input space. In such a scenario, the entropy gained at the input level via SAIB would be nullified by the entropy lost during the encoding process.
Therefore, we conclude that maximizing the input-level incremental entropy AH (and thus ((x")) )) necessary but not sufficient condition. To robustly increase the final representation entropy H(2' ), mechanism to control the encoder's information-compresiit behavior is essential.
B.3 THE IE-CL SOLUTION: A SYNERGISTIC OPTIMIZATION FRAMEWORK
Our IE-CL framework provides a complete solution by reformulating the objective to jointly optimize both entropy generation and preservation. We re-state final loss function from the main text:
Limal = LINOONE + (Pou(Pellg) - (Z" + 7) encoder
(21)
Let's analyze how this objective creates optimization landscape that solves the challenge described in Sec. [.2 The goal of the optimizer is to minimize Limal. which is dominated by the term -XH (2" effectively becoming an objective maximize H (Z'"). To achieve this, the optimizer can adjust the parameters of SAIB (Ø) and the encoder (0).
1. Optimizing SAIB (o): To maximize the final entropy H (Z'-), the optimizer is incentivized to maximize the input entropy H H(X as established by the bound in Eq. 20 The SAIB module, Pø. is specifically designed for this task. As shown in the appendix, its design as volume-expanding map det Jn > 1) directly translates maximizing the incremental entropy ( This the entropy generation part of our framework.
[yCreeeecnnder 2. Optimizing the Encoder (0): The term rf encoder directly constrains the encoder. By implementing this regularizer via Spectral Normalization, we constrain the Lipschitz constant of the encoder's layers. A smaller Lipschitz constant leads to "smoother" transformation, which in turn prevents the Jacobian determinant term E[log det f from becoming excessively negative. This term directly counteracts the information bottleneck, serving as the entropy preservation part of our framework.
3. Semantic Constraint (Dx The KL--divergence term acts as a crucial regularizer on SAIB, ensuring that the entropy maximization process does not push the transformed samples X" into semantically meaningless or out-of-distribution space.
In conclusion, the IE-CL objective function does not assume a naive carry-over of entropy. Instead, it creates synergistic system where the only effective way for the optimizer to maximize the final representation entropy H ((Z")) is to simultaneously use SAIB to generate rich input entropy and constrain the encoder to faithfully preserve it. This provides a principled and robust solution to learning diverse representations for contrastive learning.
C JACOBIAN DETERMINANT OF THE SAIB
C.I DETAILS OF THE SAIB
Block definition. Let x E RDD be the flattened patchified tensor. Within a fixed ReL.U activation pattern the block acts linearly:
f(x) = Ax,
A == W.M.WWM.WWMW.
(22)
where
16

---

Published as a conference paper at ICLR 2026
W1 Wx W3 W E.ERDX2D are x 1-convs;
W R2DxD is a 3 3-conv that doubles the channel dimension;
Mj diagonal 0/1 masks coming from ReLU derivatives.
Step A lower bound on ||A||2- Because W2 maps RD -R with i.dd Gaussian initialisation
of variance 2/fansu, random matrix theory gives
Prfo (W2) 2 v2
(23)
All other W; are square and full rank by construction, so |||| 2 /2 WWM 2|2 almost surely.
>
Step 2: Singular values of the Jacobian. The Jacobian of f
==
(24)
Let be the right singular vector of associated with mmax (A) =: > 1. Then
| I|u + Aull2 2 ||Au||2 - ||ul|2 8
(25)
and by triangle inequality also ||Jull2 2 Hence the largest singular value of J satisfies Tmax (J) 1+ 2.
Step 3: Determinant strictly greater than Since J is the sum of identity and matrix of full column rank, every singular value of J is 2 (see Weyl's monotonicity theorem). With at least one singular value we get
det J
II (x) > 2x1 > 1.
(26)
Therefore the block is locally volume-expanding almost everywhere, and its differential entropy change AH = E[log det J|] > 0.
Even if some ReLU masks set entire channels zero, the 2x expansion ensures that at least one singular value of A remains > with high probability, keeping the argument intact.
Remark.
ReLU
[x],Com: C...Cen.Chhnel- Morchetze
Ivl,GannChhmme
Ixl,Com, Chanaal
I.C.11
Ix3,Camm, Natchette
Is1.Ctan.Clonef
Figure 6: Structure of Sample Augmentation Incremental Block (SAIB). Note that due to the patching of the 3-channel image, the batch occupies the position of the original channel. Therefore, it is possible to drive the enter-batch information to communicate by changing the original convolutional channels. All the convolutions in this block are cascaded with BN layers and SwashaRamaaaandran ett al. (2017) to achieve nonlinear augmentation capability. And then convolved and non-linearly processed, and finally reconstructed back the original position through positional coding.
17

---

Published as a conference paper at ICLR 2026
D TRAINING COST
We show a comparison of the training time consumed for the proposed strategies Figure and Figure 8 respectively. Figure 7 shows the different methods 256 batch setting with resnet50 as backbone on ImageNet- k The time required to train one epoch. All parameters were kept at the optimal settings declared at the time of their release, and time spent was evaluated using mixed precision on V100 (32G).
Figure illustrates the additional computational time consumption associated with the SAIB plug- and-play existing approach. Due to the differences in the self-supervised paradigms, we observe that for the encoder half-update paradigm (MoCo-v2, SimSiam, BYOL), adding SAIB to maximise the incremental information entropy results in only slight additional computation time (within 10%), whereas for the full-parameter update approach that relies heavily on the batch scaling function (SimCLR), adding SAIB increases the training time by 12.3%. Overall, SAIB is able to balance the performance improvement of the model with the increased training time.
Training Time Compartson for Seff Supervisdd Learinng Mathods (Bx V100 32GB GPUs)
Time epoch (mirute))
05.5 ml
60.5 min
50.0
47.9mm
NNCLI
alowar Baan Mctaowi
Figure 7: Comparison of the time taken by different methods to train an epoch on ImageNet- 1k with batch 256. The proposed IE-CL, although it includes an additional non-isometric transform module SAIB, still spends less training cost compared to the previous methods because it uses momentum to update the Query encoder.
SAIB Overhend:
Training Comparison: Original vs.
Original
7.
SAU
Time per spoch (minutu))
x
(4.7%)
MoCo-2
EYOL
SinCLR
SinSlam
Self-Supervised Learning Methods
Figure 8: As plug-and-play module, SAIB enhances the performance of existing contrastive learning methods with limited additional computational overhead. Overall, it achieves effective performance gains within acceptable increase in training time on average, approximately 8.8% more compare to the original models (see Table 5 in the main text).
18

---

Published as a conference paper at ICLR 2026
E
PSEUDO CODE
Algorithm 1: PyTorch pseudo-code of IE-CL (Corrected)
Q: anchor encoder (updated by backprop)
K: query encoder (updated by mementum) momentum hyparparameter for str: contrastive loss function (e.g., InfoNCE)
SAIB: sample augmentation incremental block optimizer: updates and SAIB paraneters
H: entropy estimator
Initialize s's parameters from s's k. 1_lod_state_diiee (Q. state
for x in leader:
# Create two augmented views
x_anchor, x_query aug (x), aug (x)
Apply SAIB to the query view to increase entropy queqquurryt_ transforeed = SAIB (x query)
- Forward Pass -
Q computes features for anchor and transforned query a_anh Q(x_anchor)
a_query_transformed Q qix_query_transsoomedl
к computes features for transformed query (no gradients)
with forch.no_grad/l
q_quer <is_.uuey__transtomme
- Loss Calculation (matches Equation 25) 11. InfoNCE Loss
Littonee ctr (q_ (q_anchor, k_query)
divergence for regularization
Lx klososs sggperry_transfommeeddee detrach O), g_anchor)
Incremental Entropy Maximization
Lontropy_nax trrg_
Total lcss
loss LinfeNee B* Ca. Cent nox
- Backward Pass Optimizer Step
loss bassbbackaad())
ootimizer.steple optimmizer.zeroogradll
Momentun Update
with torch.no_grad())
for paran_q, param_k in zip (Q. parameetesss (), paaaanetees ()):
param_k.data param_ k. data + param_q.data (1.0 - m)
19


Reviews:
[{"review_id": "X48q0BPzJ8", "reviewer": "Reviewer_LX2z", "summary": "The paper proposes IE-CL (Incremental-Entropy Contrastive Learning) for self-supervised contrastive learning that explicitly models entropy generation and preservation in the learning process. The central idea is to inject entropy at the input level via a learnable augmentation block (SAIB), then constrain the encoder to preserve this expanded entropy through spectral normalization. This is motivated by an information-theoretic decomposition of contrastive learning objectives under the Data Processing Inequality (DPI), suggesting that maximizing representational informativeness requires both entropy creation (from augmentations) and controlled propagation (through the encoder). The model comprises three main components: 1) SAIB (Sample Augmentation Incremental Block): A learnable transformation enforcing \"volume-expanding\" Jacobian to expand local volume in the feature manifold, thereby increasing input-space entropy. 2) KL Regularizer: Maintains semantic consistency between entropy-expanded samples. 3) Encoder Preservation (Spectral Norm): Constrains the encoder’s Lipschitz constant to prevent information collapse.\n\nThe provided experimental results on CIFAR-10/100, STL-10, and ImageNet-100 demonstrate consistent performance improvements over SimCLR, BYOL, MoCo-v2, and SimSiam. The ablation studies show the SAIB block contributes the most, while the encoder regularizer has only a marginal effect.", "questions": "The authors are suggested to respond to those raised in **Weaknesses.*\n\n**Additional Questions**\n\nThe proposed SAIB module is implemented as a small convolutional network. How can the IE-CL framework be extended to deal with Vision Transformer architectures?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed formulation leads to a new perspective on entropy control in SSL. Unlike prior works (e.g., InfoMax-SSL, VICReg, Matrix-IB) that maximize representation-level entropy, IE-CL proposes to inject entropy at the input level through a learnable augmentation mechanism. This shift from output-space to input-space entropy control is novel and conceptually meaningful.\n2. The proposed method is established based on sound theoretical motivation. The information-theoretic derivation connecting augmentation entropy, encoder Jacobian, and DPI is mathematically correct and clearly stated. The idea of treating entropy propagation as an “incremental” process is intuitive and reasonable.\n3. The empirical validation is comprehensive. The method is tested on multiple datasets and integrated into different contrastive baselines (e.g., SimCLR, BYOL, MoCo-v2, SimSiam). The experimental results show consistent gains, and training overhead is minimal.\n4. The presentation includes sufficient implementation and training details. The ablation studies reveal transparent module effects and interactions.", "weaknesses": "1. The claim that encoder preservation is necessary appears overstated. The paper argues that spectral normalization of the encoder is required to prevent the loss of generated entropy (Section 3.3). However, Table 3 shows that removing this component leads to only a marginal change in performance (0.26 percent difference). This result indicates that the encoder likely already preserves entropy through existing normalization layers and the contrastive objective itself. Therefore, the Encoder Preservation module should be characterized as helpful rather than necessary, since the theoretical argument appears to overextend a sufficient condition into a claim of necessity.\n2. The plug-and-play experiment in Table 5 evaluates only the addition of the SAIB module to other baseline methods, without reporting results for the combined configuration of “SAIB plus Encoder Regularizer.” This omission further reinforces the impression that the SAIB module is the sole component contributing meaningfully to performance improvements, while the Encoder Regularizer has little demonstrable effect.\n3. The empirical gains in the experiments are limited. For example, ImageNet improvements are visible but not substantial (+1.3 \\% over Matrix-SSL at 800 epochs). Also, multi-seed or statistical analysis is not provided to confirm the significance of the performance gains.\n4. Although the principle of maximizing information entropy is well established, the novelty of IE-CL lies primarily in where the entropy is introduced, namely at the input level rather than the representation level. The paper would benefit from emphasizing this distinction more clearly and from explicitly differentiating its approach from prior InfoMax and variance-regularized methods such as VICReg, InfoMin, and EMP SSL.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pnbFsxrYr8", "reviewer": "Reviewer_bSVD", "summary": "This paper introduces IE-CL, which is a framework designed to overcome the limitations of static augmentations and large batch sizes in CL. So the authors show that maximizing the \"incremental entropy\" (the entropy gain between augmented views) while preserving semantics is equivalent to minimizing the InfoNCE loss.\n\nTo achieve this, they propose the SAIB which is a lightweight, trainable module that plugs into the query branch. To my understanding, I think SAIB learns to expand the representation space (increasing entropy), while a KL divergence regularizer ensures the new view remains semantically consistent with the original. The authors have conducted several experiments show SAIB improves linear evaluation performance, works well in small-batch settings, can enhance other non-contrastive methods, and helps with downstream tasks like segmentation.", "questions": "See the weakness section.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I think the paper has some strengths:\n\nFirst, I think the theoretical foundation is a major plus. The authors provide a proof for their core claim, linking the minimization of contrastive loss to the maximization of incremental entropy. In my opinion, this reframing of contrastive learning as a trade-off between entropy expansion and semantic alignment is a novel information-theoretic perspective.\n\nAlso, I think the SAIB module itself is a contribution. It's not just an arbitrary layer but a lightweight block designed to execute the paper's goal (inducing positive entropy increments) while being regularized to preserve semantics.\n\nFinally, I think the experiments seem comprehensive. The authors validate their method across a wide range of datasets and tasks. I think it's particularly strong that they test not only linear probing but also show effectiveness in small-batch settings (a key weakness of many CL methods) and on downstream transfer tasks like segmentation and detection.", "weaknesses": "I think the paper's primary weakness is its narrow experiment, which doesn't fully support the broad claims of a improved \"framework.\" My main issue is that all experiments are confined to ResNet architectures. The self-supervised learning field has largely migrated to Vision Transformers (ViTs), and their complete absence here is a glaring omission. In my opinion, I feel this makes the work somewhat dated and raises a critical question: is this incremental entropy principle a general SSL concept, or is it just a clever trick that happens to work well with the inductive biases of CNNs? The claim of generalizability is a little bit undermined when the method isn't tested on the field's dominant architecture.\n\nMy other concern is that the analysis of the method's new hyperparameters is too thin. The SAIB module adds a new layer of complexity, particularly the KL regularization weight, but the provided ablation study is basic. I think It's hard to tell how robust the method is or what the practical tuning cost would be for a new user. I think the paper misses the chance to investigate the interplay between the new module and existing crucial hyperparameters. For example, the temperature (τ) is important to tell how the model works. How does adding SAIB affect the model's sensitivity to temperature? Does the optimal τ change? I think demonstrating the analysis is needed for anyone trying to implement this method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oo2yyVE1kZ", "reviewer": "Reviewer_gtzf", "summary": "Existing self-supervised contrastive learning methods mainly rely on augmentation-based invariance constraints, which limit representation expressiveness. This paper introduces entropy as a measure to preserve semantic consistency and improve expressiveness. The proposed Sample Augmentation Incremental Block (SAIB) and Incremental Information Entropy (IncEntropy) objective capture entropy gain, reflecting the diversity of information lost after encoding. The main contribution is using the encoder’s information bottleneck (via SAIB) to extract semantic information while reducing noise.", "questions": "1. Could the authors provide insights or results on the performance of the proposed method in larger-scale settings?\n2. Have the authors considered whether there is any way to extend this approach to ViT-based backbones, despite the limitation that Spectral Normalization cannot be applied?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The approach is interesting, particularly in introducing the concept of entropy into self-supervised learning through SAIB and IncEntropy. In particular, SAIB presents an appealing way to apply entropy, showing the largest performance gain in the ablation study (Table 3).\n\n- The method is also simple and can be easily integrated into existing SSL frameworks, as demonstrated in Table 5.", "weaknesses": "1. Although the effectiveness of the proposed method is demonstrated throughout the paper, most experiments (except Table 1) are conducted on relatively small-scale settings such as ResNet-18 or ImageNet-100. Since Table 3 highlights the strong effect of entropy generation through SAIB, it would be valuable to evaluate the method on larger or standard-scale benchmarks. The same applies to Table 5.\n\n2. It would also be helpful to include an ablation study with semantic consistency only, to better isolate and verify the effectiveness of SAIB.\n\n3. It is well known that dense prediction tasks, such as semantic segmentation, differ significantly from image classification, and standard SSL methods often struggle to generalize well to these tasks. Therefore, to more convincingly demonstrate the effectiveness and general applicability of the proposed method, evaluation on additional (dense) prediction benchmarks beyond the Pascal dataset would be beneficial.\n\n4. As the authors also mentioned, SAIB relies on Spectral Normalization, which is designed for convolutional priors. Therefore, this approach cannot be directly applied to ViT-based backbones.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LtVeCQDKU4", "reviewer": "Reviewer_DnaM", "summary": "The paper introduces IE-CL, a novel information-theoretic framework for self-supervised learning. The motivation stems from two perceived limitations in existing Contrastive Learning methods: the inflexibility of static data augmentation and the representation compression caused by the deep encoder acting as an \"information bottleneck.\"", "questions": "See weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The SAIB module is a genuinely clever algorithmic contribution. It replaces the inherent rigidity of manually designed data augmentations with a learnable, instance-specific mechanism.\n\n2. The paper is well-grounded in information theory. By explicitly framing the deep encoder as a bottleneck, the authors move beyond the standard alignment/uniformity analysis to propose a more granular, principled objective focused on incremental information gain.\n\n3. The empirical results show improvements upon previous SOTAs.", "weaknesses": "1. The IE-CL loss function is highly complex, requiring the delicate balancing of at least four major hyperparameter terms. There needs to be more ablations on the sensitivity.\n\n2. While the method aims to make CL more accessible by excelling in small-batch scenarios, the introduction of the SAIB module, the complex loss terms, and the explicit regularization undoubtedly incur additional computational overhead. The paper critically omits a quantitative analysis of the increase in FLOPs, training time, or memory consumption relative to baselines. This lack of efficiency analysis weakens the overall practical value, as the potential computational cost might negate the performance gain, particularly in the resource-limited settings it targets.\n\n3. The transferability of the learned augmentation should be discussed. As the standard augmentations used are not combined with specific datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Wh6NBWWA4L", "reviewer": "Reviewer_Trhc", "summary": "This work modifies the view generation process for contrastive learning by modfiying the query view generation process with a view-generating neural network which functionally acts as an augmentation generator. This view generator is trained with entropy maximization, making the negatives harder and therefore increasing the loss.  Overall, this results in stronger performance when combined with self-supervised methods.", "questions": "* How could this be extended to Vision Transformers, to modernize the method?\n\n* I'm curious if the gains could be attributed largely to spectral regularization? In the ablation table, what would be the result of ONLY encoder regularization?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* There is a mathematical intuition which, at a high level, is sensible is a novel motivatation for the method.\n* Empirical evaluations show small but consistent improvements.\n* It is not too costly, computationally.", "weaknesses": "* There are limited transfer learning results, which is the main application of self-supervised pre-training. In particular, classification results are missing. \n\n* ViTs are not evaluated. Would the method work with a Vision Transformer backbone? Vision Transformers are ubiquitous. \n\n* There are missing strong self-supervised baselines, such as DINO[1] style training. \n\n[1] Oquab, Maxime, et al. \"Dinov2: Learning robust visual features without supervision.\" arXiv preprint arXiv:2304.07193 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)