Projection Head is Secretly an Information Bottleneck
Authors: Zhuo Ouyang
Kaiwen Hu
Qi Zhang
Yifei Wang
Yisen Wang
Venue: ICLR 2025 Poster (2025)
Keywords: Projection Head
Contrastive Learning
Self-supervised Learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Recently, contrastive learning has risen to be a promising paradigm for extracting meaningful data representations. Among various special designs, adding a projection head on top of the encoder during training and removing it for downstream tasks has proven to significantly enhance the performance of contrastive learning. However, despite its empirical success, the underlying mechanism of the projection head remains under-explored. In this paper, we develop an in-depth theoretical understanding of the projection head from the information-theoretic perspective. By establishing the theoretical guarantees on the downstream performance of the features before the projector, we reveal that an effective projector should act as an information bottleneck, filtering out the information irrelevant to the contrastive objective. Based on theoretical insights, we introduce modifications to projectors with training and structural regularizations. Empirically, our methods exhibit consistent improvement in the downstream performance across various real-world datasets, including CIFAR-10, CIFAR-100, and ImageNet-100. We believe our theoretical understanding on the role of the projection head will inspire more principled and advanced designs in this field.  Code is available at \url{https://github.com/PKU-ML/Projector_Theory}.

Published as a conference paper at ICLR 2025
PROJECTION HEAD IS SECRETLY AN INFORMATION BOTTLENECK
Zhuo Ouyang 1x
Kaiwen Hu?*
Qi Zhang'
Yifei Wang'
Yisen Wang
College of Engineering, Peking University
2 School of EECS, Peking University
3 State Key Lab of General Artificial Intelligence,
School of Intelligence Science and Technology, Peking University
4 MIT CSAIL
5 Institute for Artificial Intelligence, Peking University
ABSTRACT
Recently, contrastive learning has risen to be a promising paradigm for extract- ing meaningful data representations. Among various special designs, adding a projection head top of the encoder during training and removing it for down- stream tasks has proven to significantly enhance the performance of contrastive learning. However, despite its empirical success, the underlying mechanism the projection head remains under-explored. In this paper, we develop in-depth theoretical understanding of the projection head from the information-theone perspective. By establishing the theoretical guarantees on the downstream perfor- mance of the features before the projector, we reveal that effective projector should act as an information bottleneck, filtering out the information irrelevant to the contrastive objective. Based on theoretical insights, we introduce modifica- tions projectors with training and structural regularizations. Empirically, our methods exhibit consistent improvement in the downstream performance across various real-world datasets, including CIFAR-10. 10, CIFAR-100 and ImageNet-1000 100. We believe our theoretical understanding on the role of the projection head will inspire more principled and advanced designs in this field. Code is available at https: //tp:/ggtttta/ cwm/ ML/Poo Theooy
INTRODUCTION
In recent years, contrastive learning has emerged promising representation learning paradigm and exhibited impressive performance without supervised labels (Chen et al., 2020; He et al. 2020) Zhontar et al., (021). The core idea of contrastive learning is quite simple, that is to pull the aug- mented views of the same samples (i.c.. positive samples) together while pushing the independent samples (i.e., negative samples) away. To improve the downstream performance of contrastive learn- ing, researchers have proposed various special training objectives and architecture designs (Grill al. 2020; Wang et al. 2021 Guo al. 2023 Wang et al. 2023 2024 Du al. 2024). Among them, one of the most widely-used techniques is the projection head (i.e., projector) (Chen et al. 2020). which is shallow layer following the backbone during pretraining and is discarded in down- stream tasks like image classification and object detection. It has been shown that the features before the projector (denoted as encoder features) exhibit much better downstream performance than the features after the projector (denoted as projector features) across various applications (Ting al.) 2021 Gupta et al. 2022). Inspired by the success of the projection head contrastive learning, researchers also extend this architecture to other representation learning paradigms and achieve sig- nificant improvements (Sariyildiz al. 2022; Zhou al. 2021). However, although the projection head has been widely adopted, the understanding of the underlying mechanism behind it is still quite limited. In this paper, we aim to establish a theoretical analysis of the relationship between the projection head and the downstream performance of contrastive learning.
Among various theoretical understandings of contrastive learning (Arora et al. 2019 Oord et al.
2018 HaoChen et al. 2021 Wang et al. 2022 Cui et al. 2023 information theory provides
"Equal Contribution.
Corresponding Author: Yisen Wang (yisen.wang@@uallll..
wang wang@pku.edu.co

---

Published as a conference paper at ICLR 2025
Supervised Label
Input Sample
Encoder
Projection Head
Contrastive
Y
x
Z
Zz
R
Loss
Self-supervised Target
Downstream
Task
Figure I:
The general construction of contrastive learning can be displayed as the information flow model above, where Y denote the ground-truth labels in downstream tasks, X are the input samples, Z1, Z2 denote the encoder and projector features, and represent the self-supervised targets.
useful framework for analyzing its downstream performance (Oord et al. 2018 Tian et al. 2020 Specifically, as shown in Figure [ the overall information flow in contrastive learning transmits from Y R. where represents the natural semantics that generates input samples X (e.g., the supervised labels). 21 (encoder features) and Zz (projector features) respectively denote the features before and after the projector, and R represents the self-supervised targets in contrastive learning. has been proven that neural networks can capture most of the essential information needed for downstream tasks by minimizing the contrastive objective, i.e., maximizing the mutual information I [Z2; R) leads to maximizing [[2; Y') (Tian et al. 2020). However, crucial question remains: what the influence of discarding the projection head downstream tasks?
By establishing theoretical guarantees, we identify some critical factors affecting the downstream performance of the encoder features. Specifically, the downstream utility of encoder features, quan- tified by I(Z;; Y), improves as the mutual information between encoder features and projector features 1(2;; Z2) decreases while the mutual information between encoder features and the con- trastive objective T [ZZ;; R) increases. To balance these two factors, theoretical analysis suggests an essential design principle for the projector: should function as an information bottleneck, se- lectively filtering out information that is irrelevant to the contrastive objective.
Based the theoretical principle, we propose two approaches optimize the designs of projectors, ie., modifying the training objective and adjusting the projector architectures. To be specific, for controlling the mutual information between encoder and projector features (I(Z) : Z2)) with new training objectives, we estimate the mutual information with the surrogate metric proposed by Tan et al. (2023) and incorporate it as a regularization term. For structural adjustments, we decrease the mutual information between encoder and projector features (Z;; Z2) by promoting the sparsity of the projector outputs with discretization (Liu al., (2022) and sparse autoencoder (Ng et al. ( architectures. With the proposed improvements on the projectors, our methods exhibit significantly improved downstream performance across various real-world datasets and contrastive frameworks.
Our contributions are summarized as follows:
We develop a new theoretical understanding for the role of the projection head in contrastive learning from the information-thhorrtie perspective. We establish both lower and upper bounds for the downstream performance of features before projector.
Our findings indicate theoretical principles for designing an effective projection head: it should act as an information bottleneck, filtering out the irrelevant information and pre- serving the essential information for the contrastive objective.
Based on theoretical principles, we propose two categories of methods improve projector design, namely training regularization and structural regularization. Various experiments different datasets and contrastive methods demonstrate that modified projectors sig- nificantly improve the downstream performance of contrastive learning.
RELATED WORK
Contrastive Learning, Recent advancements in contrastive learning algorithms have revolutionized the field of representation learning by leveraging the principle of instance discrimination. Represen-
2

---

Published as a conference paper at ICLR 2025
tative algorithms such SimCLR (Chen et al. 2020) and MoCo (He et al.
2020) have demonstrated
the power of contrastive approaches by maximizing alignment between difterently augmented views of the same data and minimizing similarity between different instances. The following frameworks including BYOL (Grill et al., 2020), Barlow Twins (Zbontar et al., (02I), SimSiam (Chen & He, 202T] extend contrastive learning with different objectives, structures, and empirical tricks. Among various designs in contrastive learning, an essential component is the projection head, which em- ploys a multi-layer perceptron (MLP) on top of the backbone network during the pretraining process and discards it in the downstream task. The projection head has been found to be crucial in improv- ing the quality of learned representations. The features before the projection head exhibit benefits in generalization (Chen et al., 2020), robustness (Gupta al. (022)) and avoiding dimensional col- lapse (Jing et al., 2021). Due to the impressive performance of the projection head, researchers extend this structure to different sulf-supe paradigms (Zhou et al. 2021] Zhang et al. 2022
Yue et al. 2023, Zhang et al.) 2023), and even to supervised learning (Sariyildiz et al. 2022) lead- ing to significant improvements across various scenarios. However, although the projection head has achieved impressive empirical success, the theoretical understanding remains quite limited.
Theory of Contrastive Learning. As contrastive learning algorithms have achieved remarkable empirical success across various downstream applications. researchers try to understand the mech- anisms behind them from different theoretical perspectives. [Wang & Isola (2020) understand con- trastive objectives by reformulating the InfoNCE loss as two terms: the alignment of positive pairs and the uniformity of negative samples. Arora et al. (2019) establish the first theoretical guarantee between the contrastive loss and the downstream classification loss. [HoCChen al. (2021) evalu- ate the downstream performance of the optimal solution in contrastive learning by connecting the contrastive objective to matrix factorization. (Saunshi et al. (2022) point out that understanding con- trastive learning methods requires incorporating inductive biases. Besides, several works discuss the selection of positive samples (Tian et al.] 2020), introduce new training objectives (Bardes et al.) 2021). and analyze the learning dynamics TTn et al. [023] Wang et al.] 2023). Additionally, for an- alyzing the special architectures in contrastive learning. previous works establish theoretical analysis of the prediction head in contrastive learning without negative samples (Wen & Li 2022 Zhuo et al.) 2023). investigate the effect of the projection head in both supervised and self-supervised learning (Xue et al. 20244) and apply the concepts of expansion and shrinkage to explain the effectiveness of projection head (Gui et al. 2023).
3 AN INFORMATION-THEORER
ANALYSIS ON THE ROLE OF PROJECTORS
Among various theoretical frameworks for contrastive learning (Arora al. 2019 HaoChen al., 2021 Oord et al. 2018: Wang et al. 2022), the information-theome
perspective is popular and of- fers many insightful understandings (Tian et al. 2020 Tan et al. 2023). However, previous analyses usually overlook the existence of the projection head and discuss the downstream performance of projector features, which leads to an obvious gap between theory and practice. In this paper, we still use the information-therree perspective, but focus on providing a theoretical understanding of encoder features.
In this section, we start with an introduction to the information flow in contrastive learning to lay a foundation for our analysis. After that, we establish the theoretical guarantees on the downstream performance of the features before the projector, which characterize the role of the projection head in contrastive learning and indicate the principle for designing an effective projector.
3.1 PROBLEM SETUPS
Informittion-heerrfiic Formulation. As shown in Figure [] the overall information flow in con- trastive learning transmits from Y to R. To elaborate, Y represents the natural semantics that generates input samples X (c.g., the supervised labels). Z₁ and Ze are the encoder features and projector features, and R are the self-supervised targets in contrastive learning (positive samples in contrastive learning are pulled together while negative samples are pushed away). Following HaoChen et al. (202T), D. we assume that the set of X finite but exponentially large set for the simplicity of analysis. During the contrastive pretraining process, the backbone network encodes the input samples and obtains the encoder features Z1. After that, the projector is applied following the encoder and we obtain the projector features Zg. Eventually, we calculate the contrastive objec- tive on the projector features. Consequently, our information flow model can be summed up as
y X 21 Zz R.
3

---

Published as a conference paper at ICLR 2025
Mutual Information Definitions. We then introduce the basic concept of mutual information. We
adopt Shannon information, where the entropy of a random variable X is defined as (
-Ep(x) log P(X), and the mutual information between two variables X1 and X2 is (2) = ( - ( = ( (1) Intuitively, T(X;; X2) conveys the reduction in the entropy of X1 when the information about X2 is given. When I Xx) = O, X1 and X2 independent variables, and when they are closely related, their mutual information tends to be large.
Mutual Information Dynamics in Contrastive Learning. We now utilize the concept of mutual information to revisit the contrastive learning process. During pretraining, we train the projector fea- ture with infoNCE loss, aligning similar objects while distancing dissimilar ones, which increases the mutual information (22; R) (Oord et al., 2018). It has been proved that minimizing the con- trastive objectives enables the neural network obtain essential information for downstream tasks (e.g., the supervised labels in classification tasks), i.e., ((2;; Y) increases with the larger [(2;; R) (Tian et al. 2020). However, after pretraining is completed, the common contrastive pipeline dis- cards the projection head and uses the encoder feature 21 for downstream tasks, which suggests that we should focus on establishing the theoretical guarantees of (21;
3.2 THEORETICAL GUARANTEES OF DOWNSTREAM TASK PERFORMANCE
In this section, we will first theoretically derive a lower bound of [(Y; 21), which provides new understanding of how pretraining the projection head can benefit downstream tasks.
Theorem 3.1. The downstream task performance of encoder features can be lower-bounded by T(Y; 7 2 T(71; R) T(21; ; 72) + T Y);
Proof Sketch. We provide a proof sketch of this lower bound as it is the core of our theoretical analysis. We first utilize the unidirectional information flow model to illustrate that Y and R are conditionally independent when given the encoder feature, i.e., I(Y; R(Z,) O. Besides, observe that Z₁ learns less from than from Z2, i.e. 1(21; R) S 1(21; With these two observations, we derive the lower bound.
I(Y: 21) T(Y: 2 + T(Y: R(Z,)
(I(Y; R(Z) 0)
T(Y:7,, R)
= H (Z1; R) - R(Y)
2 HZ1,, 22) -
(1(Z); R) s I(21;
(I)) H(2)) = =H(Z] (2) - ([([)))
-H(,, H(Zi)R,
2 *H(,, (2) H((] (M(I)))((((+++)))) + (Z)) (H(Z)R)) N H(ZRRR
=T(Z;; R) T(Z;; 72) + ((R;Y).
According to Theorem 3.1 the lower bound comprises three components. As the mutual infor- mation between the ground-truth labels and supervidd targets (I(R; Y)) can be regarded constant in contrastive learning (Arora et al. 2019). we focus on the first two terms, i.e., I(Z1; R) and [(2;; Zz).
The Projector Needs to Preserve Sufficient Information Related to Contrastive Objectives. The first term, [(Zi; Rt,, measures the amount information that the encoder feature acquires from the contrastive learning objective. To improve the lower bound of downstream task performance, the encoder feature need to share sufficient knowledge of the self-supervised target R. As the contrastive objective is calculated on the projector features, this term implies that the projector should avoid forgetting the essential information, which is consistent with the practical design where the projector usually a shallow architecture.
The Projector Needs to Filter out Unrelated Information. As for the second term, the theorem states that the downstream performance improves with less mutual information between encoder and projector features. The result indicates the necessity of the projection head. Without the pro- jector, this term achieves the maximum, which indicates the downstream performance significantly declines. In practice, several studies improve the downstream performance of contrastive learning by implicitly decreasing the mutual information between encoder and projector features. For exam- Jing et al. (2021) discard part of the projector features when calculating the contrastive loss and Lavoie et al. 2022) project the encoder features into simplicial representation space.
4

---

Published as a conference paper at ICLR 2025
Combining two terms, the theorem delivers principle for designing projector: it should act as an information bottleneck (Tishby al. 2000) that filters out the irrelevant information of the con- trastive objectíve.
Furthermore, we also provide upper bound of I(Y; Z1) to analyze the influence of the projection head. Theorem 3.2. The downstream task performance encoder of fectodeeaaur can be upper bounded by (1) & - H(1))
Proof Sketch. We provide proof sketch of this upper bound. Analogous to the proof of the lower bound, we observe that Y and Zz are conditionally independent wen given the encoder feature, i.e., I(Y; ZY; 21) = O. We then obtain the upper bound with some simple deduction.
Z1) I(Y; 21, (2)
(21; -0)
= I(Y; + 22(;
S H(Z) Z2) + +[YY; 72)
I(Y; 72) + (()) - (71;
Theorem 3.2 liniiates that the downstream performance gap between the encoder features and pro- jector features is decided by two terms, i.e., 7(Zı; Z2) and H(z)). Similar the lower bound of the downstream performance, the first term / (Z1 Z2) implies that an effective projector should decrease the mutual information between encoder and projector features. As for the second term (H (Z1)), the bound implies that the encoder features with better downstream performance should be more uniform in the representation space, which consistent with the empirical findings (Ma) et al. (202)). The upper bound further verifies the principle for designing effective projector, I.e., the projector should be able filter information unrelated contrastive objectives.
In the following, we first evaluate the effectiveness of the theoretical analysis with existing projectors and then propose several improvements based on the theoretical principle.
3.3 EMPIRICAL VERIFICATIONS ON THEORETICAL GUARANTEES
In this section, we will justify our theoretical guarantees by validation experiments. Due the diffi- culty in computing mutual information with high precision, we adopt the matrix mutual information surrogate metric (Tan et al.) 2023). We introduce their definitions of matrix entropy and matrix mutual information follows.
Definition 3.3 (Matrix-based a-order (Rényi) entropy). Suppose matrix A E R which A(i, i) for every i = ...n a is a positive real number. The a-order (Rényi) entropy for matrix A defined as follows:
1
log tr ((1))
A
(A)
1
where we use a = 2 as the default value.
Definition 3.4 Matrix-based mutual information). Suppose matrix A, R"x= which A(i, B(i,)) = 1 for every = 1, a is a positive real number. The a-order (Rényi) mutual information between A and B is defined as follows:
(A; = (A( + (B (A
where @ represents the Hadamard product of two matrices.
Following Tan et al. (2023), 3, we respectively estimate the mutual information ((21; Z2) and entropy
H(2)) ) with the matrix surrogate metric I. (2, 2T: 2225) and H, (2, (2,2,) where 2,, 2, are the normalized encoder and projector feature matrices of the samples. As for estimating I I(Z; R), calculate the contrastive loss of the encoder features as surrogate metric (Oord al., 2018))
Tendency during Training. We start with observing the changing tendency of estimated theoretical bounds during the pretraining process of contrastive learning. In practice, we take the widely-used contrastive method SimCLR (Chen et al., (020) as an example and train the ResNet-18 on CIFAR- 100. As shown in Figure 2 both lower and upper bounds continuously increase during training. Meanwhile, the rate of increase is rapid initially and gradually slows down, which is consistent with
5

---

Published as a conference paper at ICLR 2025
Validation Accuracy
144
Validation Accuracy
-52
8
lower Bound
Upper Bound
54
40
1.0
-54
76
-5.
0.6
-5.0
lower Bound
Upper Bound
Validation Accuracy
Validation Accuracy
Epochs
Epochs
(a) Estimated Lower Bounds
(b) Estimated Upper Bounds
Figure 2: The change processes of estimated guarantees and practical downstream performance of encoder features trained with SimCLR on CIFAR-100 for 200 epochs. The trend indicates that the bounds provide fairly accurate estimation of the variations in downstream performance.
Lower Bound
Bound
1.5
Lower Bound
1.4
0.87
1.0
0.79
p=0.81
1.0
pm0.68
89
Downstream Tisk Accuraty
Downstream Task Accuracy
Downstream Task
Downstream Task Accuracy
(a) Lower Bound of Sim-
(b) Upper Bound of Sim-
(c) Lower Bound of Sim-
(d) Upper Bound of Sim-
CLR CIFAR-100
CLR CIFAR- 100
CLR CIFAR-10
CLR CIFAR-10
Figure 3: Correlation between downstream task accuracy and the estimated theoretical bounds on CIFAR-10 and CIFAR-100. Different points represent the encoder features learned by SimCLR with different projectors.
the tendency of the downstream task performance. The empirical results indicate that the estimated theoretical bounds accurately characterize the change process of the downstream performance with encoder features.
Correlation between Bounds and Downstream Accuracy. We then investigate the correlation between the theoretical guarantees and downstream accuracy with different projectors. Specifically, we compare projectors with different depths, different widths, different training parameters, and different implementations. As shown in Figure both the estimated lower and upper bounds exhibit strong correlations to the practical downstream accuracy, which further verifies the effectiveness of our theoretical estimation on the downstream performance of encoder features. More empirical details can be found in Appendix B.1
NEW PROJECTORS WITH PRINCIPLED MODIFICATIONS
The thcoretical analysis in Section 3 indicates that effective projection head should be in- formation bottleneck and filter out information irrelevant to contrastive objectives. Built on this principle, we optimize the designs of projection heads through two approaches: training regu- larization and structural regularization. With modified projectors, evaluate the performance of contrastive learning in various real- world datasets.
4.1 TRAINING REGULARIZATION
To control the mutual information between encoder and projector features, straightforward method estimate the mutual information and apply as regularization term contrastive loss. As the accurate estimation of mutual information is not accessible, we still adopt the matrix mutual information introduced in Section 3.3 the surrogate metric and denote it as the bottleneck regu-

---

Published as a conference paper at ICLR 2025
Table I: Linear evaluation accuracy (%) of contrastive learning representations learned by the objec- tives with/without regularizations on the mutual information between encoder and projector features.
Dataset
Framework
Baseline
Bottleneck Regularizer
Gains
SimCLR 87.47
87.73
+0.26
CIFAR-10
Barlow Twins
89.00
89.32
+0.32
SimCLR 58.12
59.38
+1.26
CIFAR-100
Barlow Twins
64.16
66.31
+2.15
SimCLR
67.36
68.42
+1.06
ImageNet-
Barlow Twins
69.96
71.3
+1.34
larizer:
- 1a ;225))
(1)
Taking the widely-used InfoNCE loss example, the modified objective is:
f((t)(
Liotal = -Bg, A: 1i log exp(S(x) ((T))) E exp([(=)[((=T) X breg:
where (x, )), x") are the positive and negative pairs in contrastive learning, À is a hyper- parameter control the scale of regularization.
Setup. We conduct our experiments on CIFAR-10, CIFAR-IO0, and ImageNet-100, with Resnet- 18 as our backbone. We evaluate the classification based on two representative contrastive frameworks, i.e. SimCLR (Chen al. 2020) and Barlow Twins (Zbontar al.) 2021). On CIFAR-10 and CIFAR-100, we train the model for 200 epochs with batch size 256 and weight decay 10 10-4 The coefficient of the regularization term (A) is set to 0.0001. On ImageNet- -000e,0,, we train the model for 100 epochs with batch size 128 and weight decay 10 10-4 The coefficient of the regularization
set to 0.01. More details can be found in Appendix B.2
Empirical Results. In Table [ we compare the performance of the original InfoNCE loss with ours across different benchmarks. Under the SimCLR framework, we find that the classification accuracy increases by 0.26% on CIFAR-10, 1.26% on CIFAR-100, and 1.06% on ImageNet- 100 with the regularization term. Mcanwhile, under the Barlow Twins framework, the classification accuracy increases by 0.32% on CIFAR-10, 2.15% on CIFAR-100, and 1.34% on ImageNet- 100. The experimental results consistently suggest that adding the bottleneck regularizer to control the mutual information between encoder and projector features effectively improve the downstream performance across different datasets and contrastive frameworks.
4.2 STRUCTURAL REGULARIZATION
In addition to applying a regularizer to training objectives, we also explore controlling the mutual information between encoder and projector features by modifying the projector structures. Specifi- cally, we use two common structures that can control the amount of preserved information: feature discretization (Liu et al. 2002) and sparse autoencoder (Ng et al. (011).
4.2.1 DISCRETIZED PROJECTOR
As a well-known technique remove irrelevant information (Kotsiantis & Kanellopoulos 2006), discretization converts continuous features to discrete points in the representation space, where the number of discrete points determines how much information is retained. For simplicity, we fol- low the implementation of discretization proposed by Mentzer al.) (2023) and adopt the Finite Scalar Quantization structure (FSQ). Specifically, we discrete each dimension of projector features follows:
(9((())) -.2
(3)
where f(x);; is the i-th dimension of the original projector feature, g(=) = [./2] tanh(z). We note that the tanh function enables the projector features to satisfy that tanh(2;) (-1,1). Con- sequently, decides the number points in the discretization space, i.e., the mutual information between the encoder and projector features is controlled by L. When L decreases, more information from the encoder features is discarded.

---

Published as a conference paper at ICLR 2025
Table 2: Linear evaluation aceuracy (%) of contrastive learning methods with the original and dis- crete projectors on CIFAR-10, CIFAR- 100 and ImageNet-100.
Dataset
Framework
Baseline
Discretized Projector
Gains
SimCLR
87.47
88.5 89.38
+1.03
CIFAR-10
Barlow Twins
89.00
+0.38
SimCLR
58. 12
60. 16
+2.04
CIFAR-100
Barlow Twins
64.16 16
65.18
+1.02
SimCLR
67.36
68.38
+1.02
ImageNet- 100
Barlow Twins
69.96
71.68
+1.72
Setup. Similar to the training regularization setting, we adopt SimCLR (Chen et al. 2020 and Bar- low Twins (Zbontar et al. 202I) as our baseline frameworks, and measure the results on CIFAR-10, CIFAR- 100 and ImageNet-1000 The parameters are consistent with those of the training regulariza- tion setting. For the discretized projector, discrete each dimension of the projector feature to 30 points in SimCLR and 3 points Barlow Twins. More details can be found in Appendix B3.
Empirical Results. We compare the performance of the original projectors with our discretized ones across different benchmarks. As displayed in Table [ the classification accuracy increases by 1.03% on CIFAR-10, 2.04% on CIFAR- 100, and 1.02% on ImageNet- with the discretized projector under the SimCLR framework. Mcanwhile, under the Barlow Twins framework, the classification accuracy increases by 0.38% on CIFAR-10, 1.02% on CIFAR-100, and 1.71% % ImageNet- 100. The experimental results on these benchmarks indicate that employing the discretized projector can promote downstream performance across different datasets and contrastive frameworks.
Theoretical Understanding. Beyond empirical improvements, we also provide theoretical un- derstanding on the gains discretized projectors. The following theorem demonstrates that the discretized projector indicates superior downstream performance of encoder features. The proofs can be found in Appendix
Theorem 4.1. The downstream performance of aead can be lower-bounded by
I(Y;21) - (H(2) I(B)))
(4)
The above theorem gives a theoretical explanation for the discretized projector. Generally, the con- strastive loss (i.e., 1 (Z2; R)) is fairly low and can be regarded as a constant. Meanwhile, I (R; Y) is also constant. Thus, in order increase the lower bound, the projector feature needs to be more concentrated, so that H (22) becomes smaller. Intuitively, discretizing the projector feature to sev- eral points can achieve this and greatly benefit downstream performance. Meanwhile, the existence of sweet point is well explained by the theorem. On the one hand, if the point number (e.g,, one single point), then the contrastive learning process will collapse (i.e., T(Zz; R) becomes very small since Z2 can not acquire adequate information from R). On the other hand, if we do not discretize the projector feature, then Zz will become uniform (i.e., (Z2) is very large value), reducing the lower bound dramatically. The analysis above provides theoretical perspective to understand why there is peak when the point number grows from one to infinity.
few
4.2.2 SPARSE PROJECTOR
Besides the discretization, we explore applying another powerful method for filtering our irrelevant information: the sparse autoencoder (Ng al. (20II), which reconstructs the original features from a sparsely activated bottleneck layer. To be specific, we use the top-k autoencoder (Gao et al. 2024) as:
h Top
bpre)),
(5)
22 Waech + bpre:
where Wene- bpre. and Wdec are trainable parameters, TopK is an activation function that only keeps the k largest hidden representations. Serving as an information bottleneck, this approach decreases the mutual information between encoder and projector features when à becomes smaller. Meanwhile, we figure out that the hidden layer only preserves the information of the acp-activated features, which helps to retain the crucial aspects of encoder features, i.c., the information related to the contrastive objectives. Based on the theoretical analysis in Section 3 we believe that the sparse
8

---

Published as a conference paper at ICLR 2025
Table 3: Linear evaluation accuracy (%) of contrastive learning methods with the original and sparse projectors on CIFAR- 10, CIFAR 100, and ImageNet-
Dataset
Framework
Baseline
Sparse Projector
Gains
SimCLR 87.47
88.16
+0.69
CIFAR-10
Barlow Twins
89.00
89.48
+0.48
SimCLR
58.12
61.99
+3.87
CIFAR-100
Barlow Twins
64.
68.15
+3.99
SimCLR
67.36
68.48
+1.12
ImageNet-
Barlow Twins
69.96
71.86
+1.90
62
Sparse Projector
Sparse Projector
61
Accuracy
60
Discretized Projector
Accuracy
Discretized Projector
Bottleneck Regularizer
Bottleneck Regularizer
59
Baseline
Baseline
58
57
56
No Projector
X
No Projector
1.0
1.5
2.0
2.5
3.5
6.5
6.0
5.5
-5.0
-4.5
Mutual Information
Lower Bound
(a) Mutual Information between Encoder and Projec- Estimated Lower Bounds of Downstream Perfor- tor Features
mance
Bottleneck Regularizer
Discretized Projector
Sparse Projector
62
59.0
Accuracy (%)
80.0
60
Accuracy
50.5
59.5
59
56
58.0
59.0
54
57.5
10-4
10-1
0.05
30 Point Num
52
10'
10'
10100
10"
10F
Lambda
Top-k
(c) Scales of Training Regularizers
(d) Numbers of Discrete Points
(e) Numbers Activated Features
Figure 4: Empirical understandings of proposed methods. (a), (b) show the correlation between
estimated theoretical guarantees and downstream performance. The proposed methods improve the downstream performance by filtering out the irrelevant information, leading to a better downstream performance guarantee. (c), (d), (e) demonstrate the influence of different regularization parameters. With stronger regularizations, the downstream performance increases first and then decreases. The experiments are conducted on the models trained by SimCLR on CIFAR-100, 100.
autoencoder is an effective projector as we can filter out irrelevant semantics and only preserve the information related to the contrastive objectives.
Setup. The datasets and most parameters in this section are the same those of the discretized projector. For the sparse projectors. we set aggressive ratios of unactivated neurons. Denote the dimension of the hidden layer h in the sparse autoencoder as d, we set k: = 0.001d for SimCLR on CIFAR-10. 10, CIFAR- 100, and ImageNet- For Barlow Twins, and l: 0.2d on CIFAR-100 and ImageNet- 100. More details can be found in Appendix B.4
set = 0.001d CIFAR10
Empirical Results. We compare the performance of the original projectors with our sparse ones across different benchmarks. As shown in Table under the SimCLR framework, the sparse pro- jector achieves increase classification accuracy by 0.69% on CIFAR-10. 3.87% on CIFAR-100. and 1.12% on ImageNet-100 100. Under the Barlow Twins framework. it also manages to achieve an increase in classification accuracy by 0.48% CIFAR-10, 3.99% CIFAR-100, and 1.90%

---

Published as a conference paper at ICLR 2025
ImageNet-1000 100. The outcomes consistently suggest that the sparse projector can successfully im- prove the downstream performance across different datasets and contrastive learning frameworks.
4.3 EMPIRICAL UNDERSTANDINGS OF PROPOSED METHODS
In previous sections, we modify the projectors by limiting the mutual information between encoder and projector features with training and structural regularizations. The empirical results show that our methods significantly improve the performance across different real-world datasets and con- trastive frameworks.
In this section, we aim to establish further understanding on the relationship between the empirical benefits and theoretical principles. The experiment details can be found in Appendix B.5
Validation of Proposed Methods. We begin by analyzing the relationship between the impact of our proposed methods on theoretical guarantees and practical performance. With the features trained by the SimCLR with no projector, the default projectors and our proposed projectors, we collect the downstream accuracy, the estimated lower bound and the estimated mutual information. In Figure Saaure4aaaan and and 4b] we observe that our methods reduce the mutual information I [21; Z2) and obtain a higher theoretical estimation of downstream performance. Furthermore, the degree of practical improvements strongly correlates with the improvements in theoretical guarantees. These results further validate the effectiveness of both our theoretical principles and proposed modifications.
Ablation Study of Regularization Strengths. We then conduct ablation study on the effect of different parameters in our methods. To be specific, we respectively vary the coefficient of the bottleneck regularization term, the point number of discretization, and the activation number of the sparse projector for the three methods. The results are displayed Figure 4c 4d and 4e For all of three methods, with the increase of discarded information by the regularized projectors, the downstream performance increases first and then decreases. The empirical results indicate that the original projector preserves too much information irrelevant to the contrastive objective and we can find the sweet point by gradually enhance the strength of regularizations.
CONCLUSION
Among various special designs in contrastive learning, the projection head is crucial component contributing to its impressive performance across various downstream tasks. However, the underly- ing mechanism behind the projection head is still under-explored. In this paper, we present new theoretical understanding of the projection head from an iotommaiion-thhoreee perspective. Par- ticularly, by establishing the lower and upper bounds of the downstream performance, we reveal that an effective projector should act as an information bottleneck and filter out information irrele- vant to contrastive objectives. Based on the theoretical analysis, we propose training and structural regularization methods to control the mutual information between encoder and projector features. Empirically, the proposed methods achieve consistent improvement different datasets and contrastive frameworks. With the fruitful insights presented in this paper, we believe it has great potential and can inspire more principled designs in contrastive learning.
the downstream task across
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of this paper, we detail the theoretical and empirical parts of our results in the main paper and the appendix. In Section of the main paper, we introduce the basic notaions and models used in the theoretical analysis. Furthermore, in Appendix [A] we provide the detailed proofs of the theoretical guarantees. For empirical details, we briefly introduce the modified projec- tors Section 4 of the main paper. In Appendix B we elaborate the parameters of our modified projectors, including the bottleneck regularizer, the discrtized projector and the sparse projector, and the settings where we pretrain and evaluate the models.
ACKNOWLEDGEMENT
Yisen Wang supported by National Key R&D Program of China (2022ZD0160300), Na- tional Natural Science Foundation of China (92370129, 62376010), and Beijing Nova Program (20230484344, 20240484642). Yifei Wang was supported in part by the NSF AI Institute TILOS, and Alexander von Humboldt Professorship.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saun- shi. A theoretical analysis of contrastive unsupervised representation learning. arXiv preprint arXiv:1902.09229, 2019.
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Varraae--mvrriincc--vvvinnnnneee heg.aaaaaaccc..toaaaaacaaaaaaaaa for self-supervised learning. In ICLR, 2021.
Ting Chen, Simon Komblith, Mohammad Norouzi, and Geoffre Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning, In CVPR, 2021.
Jingyi Cui, Weiran Huang, Yifei Wang, and Yisen Wang. Rethinking weak supervision in helping contrastive learning. In ICML, 2023.
Tianqi Du, Yifei Wang, and Yisen Wang. On the role of discrete tokenization in visual representation learning. ICLR, 2024.
Leo Gao, Tom Dupré la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. arXiv preprint (rXiv:2406.04093, 2024.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a approach to self-supervised learning. In NeurlPS, 2020.
Yu Gui, Cong Ma, and Yiqiao Zhong. Unraveling projection heads in contrastive learning: Insights from expansion and shrinkage. arXiv preprint arXivv2336.03333 2023.
Xiaojun Guo, Yifei Wang, Tianqi Du, and Yisen Wang. Contranorm: contrastive learning per-
spective on oversmoothing and beyond. In ICLR, 2023.
Kartik Gupta, Thalaiyasingam Ajanthan, Anton van den Hengel, and Stephen Gould. Under- standing and improving the role of projection head in self-supervised learning. arXiv preprint arXiv:2212.11493 2022.
Jeff Z HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for self self-supervised deep learning with spectral contrastive loss. In NeurIPS, 2021.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. In ICLR, 2021.
Sotiris Kotsiantis and Dimitris Kanellopoulos. Discretization techniques: A recent survey. GESTS International Transactions Computer Science and Engineering, 32(1):47-58, 2006.
Samuel Lavoie, Christos Tsirigotis, Max Schwarzer, Ankit Vani, Michael Noukhovitch, Kenji
Kawaguchi, and Aaron Courville. Simplicial embeddings in self-supervised learning and down- stream classification. arXiv preprint arXiv:2204.0066 2022.
Huan Liu, Farhad Hussain, Chew Lim Tan, and Manoranjan Dash. Discretization: An enabling technique. Data Mining and Knowledge Discovery, 6:393 423, 2002.
Jiajun Ma, Tianyang Hu, and Wenjia Wang. Deciphering the projection head: Representation eval- uation self-supervised learning. arXiv preprint arXiv:2301. /2/89, 2023.
Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. Finite scalar quantiza- tion: Vq-vae made simple. arXiv preprint
2023.
Andrew Ng et al. Sparse autoencoder. CS294A Lecture notes,
2011.
Aaron van den Oord, Yazhe Li. and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint 100.vv:8070077440 2018.

---

Published as a conference paper at ICLR 2025
Mert Bulent Sariyildiz, Yannis Kalantidis, Karteek Alahari, and Diane Larlus. No reason for supervision: Improved generalization in supervised models. arXiv preprint arXiv:2206. 75369, 2022.
Nikunj Saunshi, Jordan Ash, Surbhi Goel, Dipendra Misra, Cyril Zhang, Sanjeev Arora, Sham Kakade, and Akshay Krishnamurthy. Understanding contrastive learning requires incorporating inductive biases. ICML, 2022.
Zhiquan Tan, Jingqin Yang, Weiran Huang, Yang Yuan, and Yifan Zhang. Information flow in self-supervised learning. In ICML, 2023.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? In NeurIPS, 2020.
Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv preprint physics/00040057, 2000.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In ICML, 2020.
Yifei Wang, Zhengyang Geng, Feng Jiang, Chuming Li, Yisen Wang. Jiansheng Yang, and Zhouchen Lin. Residual relaxation for multi-view representation learning. In NeurlPS, 2021.
Yifei Wang, Qi Zhang, Yisen Wang, Jiansheng Yang, and Zhouchen Lin. Chaos is ladder: A new theoretical understanding of contrastive learning via augmentation overlap. In ICLR, 2022.
Yifei Wang, Qi Zhang, Tianqi Du, Jiansheng Yang. Zhouchen Lin, and Yisen Wang. A message passing perspective on learning dynamics of contrastive learning. In ICLR, 2023.
Yifei Wang, Qi Zhang, Yaoyu Guo, and Yisen Wang. Non-negative contrastive learning. In ICLR, 2024.
Zixin Wen and Yuanzhi Li. The mechanism of prediction head in non-contrastive self-supervised learning. In NeurlPS, 2022.
Yihao Xue, Eric Gan, Jiayi Ni, Siddharth Joshi, and Baharan Mirzasoleiman. Investigating the benefits of projection head for representation learning. In ICLR, 2024.
Xiaoyu Yue, Lei Bai, Meng Wei, Jiangmiao Pang, Xihui Liu, Luping Zhou, and Wanli Ouyang- Understanding masked autoencoders from a local contrastive perspective. arXiv preprint arXiv:2310.0199 2023.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In JCMI., 2021.
Qi Zhang, Yifei Wang, and Yisen Wang. How mask matters: Towards theoretical understandings of masked autoencoders. In NeurIPS, 2022.
Qi Zhang, Yifei Wang, and Yisen Wang. On the generalization of multi-modal contrastive learning. In ICML, 2023.
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. arXiv preprint arXiv:2111.07832. 2021.
Zhijian Zhuo, Yifei Wang, Jinwen Ma, and Yisen Wang. Towards a unified theoretical understanding of non-contrastive learning via rank differential mechanism. ICLR, 2023.
12

---

Published as a conference paper at ICLR 2025
A OMITTED PROOFS
A.1 AUXILIARY LEMMAS
To prove the lower and upper bounds, we first introduce some basic lemmas.
Z1 Z2 R. Lemma A.1 (Information cutoff theory). With information flow Y X
have I(;; 21) = and I(Y; Z1)
Proof. From model, obtain simplified information chain: Y Z1 + R. Given Z₁, the information channel between Y and R is blocked out, i.e.,
P(RY, P(R
where P(.) denotes the probabilities. Consequently, we have
P(R, YIZ) P(Y (()))-(((Y,,, (( P(Y]Z)) P(R[Z))
where the first equation the Chain Rule of Conditional Probability. According to the definition of conditional mutual information, we have
T(Y; R|Z)
Σ Σ Σ PER #1E21
ply,r, 21)00 )p)g)
p(a.r(as) 0.
The second equation in this lemma can be proved by the same technique.
Lemma A.2 (Information processing theory). With information flow Zi + Z2 + R and the infor- mation processing inequality, we have
1(2n (2) 2 R(Z;;R
and I(Z,, R) ( (21;
Proof. With information flow Z1 Z2 R and lemma A.1 have ([21;R]Zz) 0 and T(Zz; R(Z = 0. Consequently, the following two inequalities holds:
T(7;: 72) T(Z1; 22) [(21; 2(22) I(21; R, 22) 2 [(7;; R),
T(R: 72) [R:: 72) + [(R; Z1 72) T(R; Z1, 22) 2 T(R: 71)
A.2
PROOF OF THEOREM 3.1
Proof.
With lemma A.1
we know that 7(Y; R1) 0, which yields
(Y;7)) 7(Y; + T(Y; R|Zz) T(Y; Z1, R).
Additionally, lemma A.2 claims that 7(Z1; R) < T(21: 72). which further gives
HZ1, R) ((z)) + HHR) R[4;;
2 ((1)) + 1 (211;42)
(6)
H(1;; -
1---Ho -H((2) + H(R).
Thus, we can deduce the lower bound as follows:
I(Y; (1) = I(Y; Zi, R)
H(2,, R) H(Z,, R|Y)
2 H(( H(Zz) H(R) - H(2,. R(Y)
-H(Zi;72) H(Z2) - -H(R) - H(R(V) H,Z
2 =H(Z)(Z)) [(Z[R)++ I(R:Y)
-(Zi; R) 1(Z1: 22) +I(RYY).
13

---

Published as a conference paper at ICLR 2025
Validao Accuracy
Val Validation Accuracy
lower Bound
Upper Bound
lower Bound
Upper Bound
Validda
OUP 3t.ed
Apper
Upper Baued
Epochs
Epochs
Epochs
Epochs
(a) Lower Bound Ten- (b) Upper Bound Ten- (c) Lower Bound Ten- (d) Upper Bound Ten-
dency During Training on dency During Training on dency During Training on dency During Training on
CIFAR-10
CIFAR-10
ImageNet- 100
ImageNet-100
Figure 5: Tendency during training of SimCLR CIFAR-I0 and ImageNet-100.
A.3 PROOF OF THEOREM 3.2
Proof.
With lemma A.1
we know that [(;; Zz = 0, which yields
IY:2)) 21) + (2) 21;
Then we can deduce the upper bound as follows:
T(Y: 21) = T(Y: 72)
= =[(Y; + 22(;
$ Z2))) +[(Y;Z))
I(Y; (1(2) 1(21;22).
B EXPERIMENT DETAILS
In this section, we detail the setting of each individual experiment in this work. All experiments conducted with at most two NVIDIA RTX 3090 GPUs.
B. EXPERIMENT DETAILS OF EMPIRICAL VERIFICATION ON THEORETICAL GUARANTEES
In this part, we justify our theoretical guarantees by two kinds validation experiments. On the one hand, we manage verify that the tendency of bounds fits quite well with that of validation accuracy. On the other hand, demonstrate that both bounds are highly correlated with downstream task performance.
Tendency during training, In this experiment, we aim to show the tendency of the lower and upper bounds during the pretraining process and then compare them with the trend of validation accuracy. We conduct the experiment under the SimCLR framework on CIFAR- 100 with a total run of 200 epochs, using ResNet-18 8 the backbone and an MLP as projector. As for the hapereparameters detail, we use learning rate 0.4, weight decay 10-4, InfoNCE temperature 0.2. Then we record both bounds and validation accuracy every 10 epochs and plot the first 100 epochs.
As supplemental aspect for the experiments of tendency during training, we plot the tendency of baselines from SimCLR and Barlow Twins on CIFAR-10, CIFAR-100, and ImageNet-1000 To be
more specific, Figure 2a 5 and 6 respectively represent the baseline of SimCLR on CIFAR- 100, the baselines of SimCLR CIFAR-10 and ImageNet- 100, and the baselines of Barlow Twins CIFAR-100. CIFAR-10, 10, ImageNet-100.
Correlation between bounds and downstream accuracy. We investigate the correlation between the upper and lower bounds and downstream task accuracy with different projectors. In these CIFAR-100 with ResNet- as the backbone and train for 200 epochs using batch size 256. As for the hyper-parameters, we use learning rate 0.4, weight decay 10-4 InfoNCE temperature 0.2, augmentation cropsize 32.
periments, we utilize the SimCI.R framework on both CIFAR-10
14

---

Published as a conference paper at ICLR 2025
Validation Accuracy cy
Valida tion Accuracy
10-
Validation Accuracy
-42
lower Bound
Upper Bound
Bound
-20
-30-
Bourd
Upser Bound
Bourd
Edatiinn
Epochs
Epochs
Epochs
(a) Lower Bound Tendency During Training CIFAR- 100
(b) Upper Bound Tendency During (c) Lower Bound Tendency During
Training CIFAR- 100
Training CIFAR-I0 10
Validation Accuracy
Validation Ac accuracy
Validation Accuracy
Upper Bound
lower Bound
14
Upper
18
Upper Va Isationonn Accurac
bier Bound
Idattion Accurscy
12-
Va Soation Accuracy
Upper Bound
Epochs
Epochs
Epochs
(d) Upper Bound Tendency During Training on CIFAR-10
(e) Lower Bound Tendency During (f) Upper Bound Tendency During Training on ImageNet-100
Training on ImageNet-1000
Figure 6: Tendency during training of Barlow Twins on CIFAR-10, CIFAR-100 and ImageNet- 100. 100.
To be more specific, we adopt linear projectors with different output dimensions (128, 256, 512, 1024), MLP projectors (i.e., 1024), no projector (i.e., use the encoder feature calculate the contrastive loss), a deepened-MLPP projector Rettear-RLLU-Lineaar--a ReLU-Linear) and two improved projectors addentit-mmapinn and DirectCLR) both CIFAR- 10 and CIFAR-100 As replenishing experiments, change learning rate to 0.2, weight decay to 10 -3 InfoNCE temperature to 0.1, augmentation crop size to 64 re- spectively on both CIFAR-10 10 and CIFAR-100. In total, these choices of parameters correspond to 16 experiments and 16 dots Figure 3 We calculate the covariance between the lower and upper bounds and downstream task accuracy, using it as a criterion to validate the effectiveness of our theoretical estimation the downstream performance of encoder features.
with different output dimensions (128, 256, 512,
As a supplemental aspect for the experiments of correlation, we also investigate the bounds and downstream performance of different projectors of SimCLR on ImageNet-100 and Barlow Twins on CIFAR-100. We plot the results in Figure 7 in which different points represent the experi- ments conducted with different projectors. To be more specific, for SimCLR on ImageNet- adopt linear projectors with different output dimensions (128, 256, 512. 1024), MLP projectors (i.e.. Reeea--RLUULLnearr with different output dimensions (128, 256, 512. 1024), projector (i.e., the encoder feature to calculate the contrastive loss). For Barlow Twins on CIFAR-100, we adopt full Barlow Twin's projectors with different hidden dimensions (1024, 4096) and different ouput dimensions (1024, 4096), half of Barlow's projectors (i.e., -inear-BN--eLL--Linee
we
with different hidden dimensions (1024, 4096), output dimensions (1024, 4096), baseline (with hidden dimension 2048 and output dimension 2048) and no projector.
B.2 EXPERIMENT DETAILS OF TRAINING REGULARIZATION
In this experiment, add regularization term the loss function to optimize the training of the projector. We conduct this experiment on CIFAR-10, 10, CIFAR-100 and ImageNet- using SimCLR and Barlow Twins as our frameworks. For all experiments, we adopt ResNet-18 as the backbone and use MLP projector whose structure dependent the framework and will be introduced the following. During the pretraining process, we simultaneously train classifier, which is a single-layer linear head and is trained without influencing on the rest of the network.
Recall Definition 3.4 and that the regularization term is Creg = La (2, 2,2.), where we take = the default value. As for the original loss functions of SimCLR and Barlow Twins,
15

---

Published as a conference paper at ICLR 2025
1.4
1.0
p=0.71
Lower Bound
Upper Bouna
1.2
Lower Bound
-5.5
Upper Boung
0.8
1.0
0.6
0.8
-6.0
p=0.75
pm0.65
p=0.72
0.6
-12
60
65
60
65
55
60
65
65 Acuuscy
Downstream Task Accuracy
Downstream Task Accuracy
Downstream Task Accuracy
(a) Lower Bound Sim- CLR on ImageNet-100
(b) Upper bound of Sim- (c) Lower Bound of Bar-
(d) Upper Bound of Bar- low Twins on CIFAR-100
CLR on ImageNet-100
low Twins on CIFAR-100
Figure 7: Correlation between downstream task accuracy and the estimated theoretical bounds of SimCLR on ImageNet-100 and Barlow-Twins on CIFAR-I00.
have
Ssimelr - EE ao -2- Min log
exp(f(x)T + E exp(((=)TT(x))))
and
LBT C(- + ΣΣ C
J3L
and _ denote the projector features of two views, 6 is the batch size and 7 is scaling factor. We then produce our loss functions by adding the regularization term, i.e.
where Cip (Es(=))) /E.(=
Lsimelr_tote
Lsimetr + Mrreg and LBT totat LBT + XCreg
For experiments conducted on CIFAR-10 and CIFAR-100, we use batch size 256 and train for 200 epochs, while for those conducted on ImageNet- 100, we use batch size 128 and train for 100 epochs.
SimCLR. We detail other hyper-parameters in the SimCLR framework. On CIFAR- 10 and CIFAR- 100, we use learning rate 0.4, weight decay 10-4, InfoNCE temperature 0.2, and set to 10-4 Our projector adopts a Linear-ReLU--LE
structure, where we use 2048 as the hidden dimension and 256 as the output dimension. On ImageNet-100, we use learning rate 0.3, weight decay 10-4, InfoNCE temperature 0.2, and set À to 0.01. We use the same projector structure but change the hidden dimension to 4096 and the output dimension to 512.
Barlow Twins. We detail other hyper-parameters in the Barlow Twins framework. On CIFAR-10 and CIFAR-100, we use learning rate 0.3, weight decay 10 scaling factor x 10 33 and set À to 10-3, Our projector consists of three linear layers separated by two pairs of batch normalization and ReLU layers, where we use 2048 as the hidden dimension and the output dimension. On ImageNet- 100, we use learning rate 0.3, weight decay 10-1, scaling factor 5 x 10-3 and set À to 0.01. We use the same projector structure the
CIFAR datasets.
B.3 EXPERIMENT DETAILS OF DISCRETIZED PROJECTOR
In this experiment, we discretize each dimension of the projector to a certain number of points, which we denote as L. We conduct this experiment on CIFAR-10, CIFAR- 100 and ImageNet- 100, using SimCLR and Barlow Twins as those in Appendix B.2 For experiments conducted on CIFAR- 10 and CIFAR-100, we use batch size 256 and train for 200 epochs, while for those conducted on ImageNet- 100, we use batch size 128 and train for 100 epochs. We respectively take 30 under the SimCLR framework and = under the Barlow Twins framework.
frameworks. All the network settings
exactly the same
B.4 EXPERIMENT DETAILS OF SPARSE PROJECTOR
In this experiment, we employ the sparse autoencoder in our projection head, adding sparsity to the projector feature so reduce the similarity between encoder and projector features (i.c.. decrease T(7;; 72)). To be specific, let us recall the formular of top-k autoencoder,
h Topk(W Wene (f) Uprc));
16

---

Published as a conference paper at ICLR 2025
Shorederd Neadtr'
Soase
Sarta momele
07.
27.
07.0
Lowe
(a) Mutual Informa- tion between Encoder and Projector Fea- tures of SimCLR on CIFAR-10
(b) Estimated Lower Bounds of Down- stream Performance SimCLR on CIFAR-10
(c) Mutual Informa- tion between Encoder and Projector Fea- tures SimCLR on ImageNet-100
(d) Estimated Lower Bounds of Down- stream Performance SimCLR on ImageNet-100 Im
Figure 8: Supplemental experiments on the correlation between estimated theoretical guarantees and downstream performance of SimCLR on CIFAR-10 and ImageNet-100.
in which TopK is an activation function that only maintains the k largest hidden representations and keeps the rest inactivated (i.c., set them to 0). We conduct this experiment on CIFAR-10, CIFAR-100 and 100, utilizing SimCL. and Barlow Twins as our frameworks. For all experiments, we adopt ResNet-18 as the backbone, while the structure of the projector is dependent on the framework we choose and will be introduced in the following in detail. For experiments conducted on CIFAR- 10 and CIFAR-100 100,
train for 200 epochs using batch size 256, while for those carried out ImageNet- 100, we train for 100 epochs with batch size 128.
SimCLR. Under the SimCLR framework, we directly employ the sparse autoencoder as our pro- jector, totally deleting the original projector of SimCLR. Our projector adopts structure similar to but we activate only a tiny number of hidden representations. On CIFAR-10 and CIFAR-I00 100, we use learning rate 0.4, weight decay 10-4, and InfoNCE temper- ature 0.2. To be specifie, on CIFAR-10, a variety of different k. On CIFAR- 100, we set the hidden dimension to 10 and = 102 after testing different & from 10 to 10 On ImageNet- 100, we use learning rate 0.3, weight decay 10 InfoNCE temperature 0.2, fix the hidden dimension to 10", and then set k 10?.
SimCLR (i.e., Mnnear-ReLU--nneen
hidden dimension 10a and set k = 100 after trying
Barlow Twins. Under the Barlow Twins framework, we preserve half of its original projector and structure. On CIFAR-10 and CIFAR-100, use learning rate 0.3, weight decay 10-4 4 and scaling factor 10 Similar to the SimCLR section, we use hidden dimension 10y on CIFAR- 10 and 10 on CIFAR- 100, and then set k = 2 x 10s on CIFAR-10 and k = 10x on CIFAR-100. On ImageNet- 100, we use learning rate 10
employ the sparse autoencoder following it, i.e., a Linear-BNt
0.3, weight decay 10 scaling factor 10 hidden dimension 10 and set i
B.5 EXPERIMENT DETAILS OF EMPIRICAL UNDERSTANDINGS OF PROPOSED METHODS
In this section, we use ResNet-18 as the backbone and SimCLR as our framework. During the pretraining process, we simultaneously train classifier, which is single-layer linear head and is trained without influencing on the rest of the network. As default parameters, we take learning rate 0.4, weight decay 10 and InfoNCE temperature 0.2, batch size 256, training epoch 200 on CIFAR-10 and CIF,R-100, and learning rate 0.3, weight decay 10-4 InfoNCE temperature 0.2, batch size 128, training epoch 100 on ImageNet-100. The projectors of our proposed methods follow the corresponding designs in previous sections.
Validation of Proposed Methods. In this part, we focus on the correlation between the estimated theoretical guarantees and downstream performance. The experiments are conducted on CIFAR- 10, CIFAR-100 and ImageNet-1006 We now detail the core parameters. On CIFAR-100, we set = 10-3 for the bottleneck regularizer, L 20 for the discretized projector and sparse projector. On CIFAR-10, 10, we set À = 10-4 for the bottleneck regularizer, L = 20 for the discretized projector and & 5 x 10 for the sparse projector. On ImageNet- 100, we take À 10 for the bottleneck regularizer, L = 30 for the discretized projector and k = 10 for the sparse projector. The result on CIFAR- 100 is displayed in Figure 4a and 4b while the rest can be found in Figure 8
10a for the
Ablation Study of Regularization Strengths. In this part, we study the effect of differ- ent parameters in our methods on CIFAR-100. 100. For the bottleneck regularizer, we take X = 0, 10-4, 10-3, 0.01, 0.05, 0.1 to form the line chart. As for the discretized projector,
17

---

Published as a conference paper at ICLR 2025
-4.245 s1(2;;) -4.255
-3.995 ((21:)) 4.005
-3.745 s (111;
3.755
1,05
1.35
1.53
1.30
0.78
1.05
1.25
0.780
1.87
0.75
0.09
1.20
0.770
p=-0.97
0.80
p=-0.99
1.15
pm-0.85
0.755
0.83
1.10
0.760
0.83
44.4 44.6 44.8 45.0 452 45.4
45.5
54.0
54.5
55.0
55.5
56.0
Val-acc1
Val-acc1
Val-acc1
(a) Fix (21;; H) to around -4.250
(b) Fix (21; to around -4.000 (c) Fix ((21; H) to around -3.750
Figure 9: Relationship between downstream performance and [(4 Z2) with fixed (Z1; R).
take L 5, 7, 10, 30, 50, 100. 10, 100, 10 5 x 10S, 10 10
Regarding the sparse projector, explore ho with values in
B.6 CALCULATION OF MUTUAL INFORMATION
In our experiments, we take a when calculating mutual information. In this part, we clarify on the reason of our choice. Following (Tan et al. 2023). the 1-order (Rényi) entropy for matrix A where a 1 is defined
A log 12
(A) - tr
Table 4: Linear probing accuracy of the models trained with bottleneck regularization (implemented by selecting different a:) on CIFAR-100.
ACC 59.06
59.38
58.59
This combined with Definition 3.3 yields the complete definition of matrix entropy and thus matrix mutual information. We then use the training regularization setting test the downstream accuracy of SimCLR on CIFAR-100 with different choices a, where the coefficient of the regularization term 10-1 In Table 4 we find that the downstream accuracy of the case with a = 2 is slightly tends to be better choice in the sense of
higher than that of the one with downstream performance. Moreover, we find that training with a is way more time-consuming because of Singular Value Decomposition, which further motivates us to choose a = 2 eventually.
1,
B.7 ABLATION STUDY OF INFORMATION BOTTLENECK [Z1 Z2)
In order to rigorously verify that the downstream performance indeed mainly influenced by [[Z;; Z2), collect data points with several fixed values of I[Z1; R) and study the relation- ship between downstream performance and 7(7;; Z2). Typically, we choose three cases where 1(21: x -.25, -3.70 (with eπrоrs no more than 0.01) and collect 6 points for each case. The results in Figure indicate that the downstream accuracy indeed rises 7(7; 72) falls, and the absolute value correlation between [(2; Zz) and the downstream accuracy when (21; R) is fixed is close to 1, which further justifies the role of [[2;; Zz) as an information bottleneck.
B.8 CORRELATION BETWEEN THE ONLINE ACCURACY OF ENCODER AND PROJECTOR FEATURES
In this part, aim investigate the correlation between I(Y; Z1) and Z2) For this, attach new classifier after the projector to obtain the classification accuracy the projector feature (i.e.. [(Y: Zz)). As shown in Figure [0 the correlation between the online accuracy of encoder and projector features is fairly small, indicating that T(Y; 7z) may not have fixed relationship with
18

---

Published as a conference paper at ICLR 2025
Accuracy of Projector Features
p=0.35
57
56
55
54
58
60
Accuracy Encoder Features
Figure 10: The correlation between the online accuracy of encoder and projector features.
I(Y; Z1): This further implies that although the inequality I1) I(Y; Z2) holds, maximizing T(;Z2) does not necessarily mean maximizing T(Y; Z1).
B.9 COMPARISON BETWEEN TRAINING AND STRUCTURAL REGULARIZZTION
Although both training and structural regularization can effectively improve downstream perfor- mance, we observe better results with structural regularization, which motivates us explore the comparison between these two types of regularization. We conjecture that the underlying reason why training regularization has inferior downstream performance is that though it lowers the value poor lower bound and thus poor downstream
of I ; Z2), fails protect 7 [Z;; R), resulting accuracy. To prove such an insight, we use larger coefficients for the regularization term to align the mutual information I 1(; Z2) with those of the structural regularization methods. Typically, set À 0.1 for comparison with discretized and sparse projectors. As shown in Table 5 we find that even if / (21; Z2) of the bottleneck regularizer is lower than those of structural regularization methods, it has inferior lower bound and poorer downstream performance due to low 1(Z;; R), which accords with our conjecture.
Table 5: Comparison between Training and Structural Regularization.
Method
I(Z1; Zz)
1(Z Rt)
lower bound
ACC
Bottleneck Regularizer (A 10-4)
1.28
-3.70
-4.98
59.38
Bottleneck Regularizer (A = 0.1)
0.36
-5.39
-5.75
50.72
Discretized Projector
1.16
-3.52
-4.68
60.16
Sparse Projector
1.02
-3.49
-4.51
61.99
B.10 GENERALIZATION OF OUR METHODS TO SUPERVISED LEARNING
In this section, we generalize our training and structural regularization methods to supervised learn- ing. To elaborate, we add new single-layer classifier to the end of the projection head, where it has access the ground truth labels of images. During training, we simply shift the initial InfoNCE loss to the cross entropy loss of the new classifier. Note that the other classifier attached to the end of the encoder remains and our accuracy is measured from this classifier instead of the new one. We collect the classification accuracy SimCLR on CIFAR-100. In Table we find that both adding regularization term (with the coefficient À = 10-4) and using a discretized projector (with discrete point number L 10) outperforms the bascline, indicating that our proposed methods can be well generalized to the supervised learning setting.
Table 6: Classification accuracy of the features before the projector in supervised learning.
Method
Baseline
Bottleneck Regularizer
Discretized Projector
ACC
71.85
72.65
72.85
19

---

Published as a conference paper at ICLR 2025
B.11 SCPPLEMENTARY RESULTS WITH NONLINEAR CLASSIFIERS
In order to further verify that the downstream performance bottlenecked by ([1;22) instead of the capability of the classifier, we conduct additional experiments and test the downstream accuracy of our methods with nonlinear classifiers. To be more specific, we finetune offline classifier for 200 epochs after pretraining under the SimCLR framework on CIFAR-100 100. We set the coefficient A = 10 for the bottleneck regularizer, discrete point number L 30 for the discretized projector, and activated features k 5 x 10 for the sparse projector. As displayed in Table the accuracies with both linear and nonlinear classifiers improve with the use of our methods, which indicates that it not the classifier but the mutual information (21; Zz) that restrains the downstream performance.
Table 7:
Fine-tuning accuracy of ResNet-18 pretrained by SimCLR with the original and our pro- posed projectors on CIFAR- 1000
Method
Baseline
Bottleneck Regularizer Discretized Projector
Sparse Projector
Finetuning ACC
72.31
72.88
73.27
73.48
B.12
COMBINATIONS OF DIFFERENT REGULARIZATION METHODS
Given that we have achieved a certain amount of improvement by means training and struc- tural regularization, it is intriguing to investigate whether the combination of several of the pro- posed methods can yield even better performance. With the default parameters. we conduct experi- ments using different combinations and collect the downstream accuracy SimCLR on CIFAR- 100, which are listed in Table 8 We observe that combining any two of our proposed methods can bring further improvement to downstream performance.
Table 8: Linear probing performance of the combination of our proposed projectors on CIFAR- 100. BR: bottleneck regularizer; DP: discretized projector; SP: sparse projector.
Method
BR
DP
SP
BR+DP
BR+SP
DP+SP
BR+DP+SP
ACC
59.38
60.16
61.99
60.68
63.23
62.42
62.49
C
THEORETICAL VALIDATION ON THE DISCRETIZATION METHOD
In this section, we provide a proof for theorem discretized projector.
which gives theoretical illustration for the
Proof. We consider the trade-off between I [[2;; R) and [2)
I(11; ((21; 22) HRR (R(Z)) (((2) (Z2)
H[) H(22) (HH())) (Z22/Z1)
= H(R) - -H(22) - -(H(R)Z)) - H(Z2 R. Z1)
= -H((2)) + 1(22;; R) + HHH Zi)
2 -H(Z2) R
It remains be shown that H HRR Z1) to lemma A.I we know that Z, and R are independent random variables when given Zz, i.e.
H(Z
H(R H(Z, R, Z1) holds. According
= HR Z2, Z1). Therefore, we have
H(R (Z2 Z.) = H() Z1) (1)
(H, Z2|2,) ((22)21) ((2
=(H(R, 72 H(Z, R, -
= H(R[Z)) H(Z2/Z))
Thus the proof of theorem is finished and provides theoretical perspective to understand how the discretized projector works and why there is a peak in downstream performance when the discrete point number grows from one to infinity.
20


Reviews:
[{"review_id": "cBivenZKK2", "reviewer": "Reviewer_DcxU", "summary": "This paper studies the design choices of the projection head in contrastive learning, and shows that the projection head should serve as a proper information bottleneck that keeps only \"useful\" information related to the class label or the contrastive objective.\n\nFor the problem setup, consider the information flow of $Y \\rightarrow X \\rightarrow Z_1 \\rightarrow Z_2 \\rightarrow R$, where $Y$ is the semantic info (e.g. class label), $X$ is the observable/input, $Z_1$ is the pre-projection head features, $Z_2$ is the projection head output, and $R$ is the contrastive label.\nThe paper argues that we should focus on $I(Y; Z_1)$ (rather than $I(Y; Z_2)$), since the projection head is discarded in downstream training.\n\nThe main results are as follows:\n- It provides **upper and lower bound on the downstream performance** for the pre-projection head features.\n  - Lower bound: $I(Y;Z_1) \\geq I(Z_1; R) - I(Z_1; Z_2) + \\text{const}$, where $\\text{const} = I(R;Y)$ is not affected by the learned features.\n  - Upper bound: $I(Y; Z_1) \\leq I(Y; Z_2) - I(Z_1; Z_2) + H(Z_1)$.\n- We want to adjust the terms on the right hand side so that the bounds are tighter, which leads to **two modifications**: 1) adding a mutual-information-based regularization term in the training objective, and 2) modifying the projector architecture.\n  - The **mutual information regularizer** $I(Z_1; Z_2)$: estimated using the matrix mutual information (Renyi entropy) as the surrogate. The matrices are $\\hat{Z}_1\\hat{Z}_1^\\top, \\hat{Z}_2\\hat{Z}_2^\\top$, where $\\hat{Z}_1, \\hat{Z}_2$ denote the normalized feature matrix before and after the projection head.\n    - Result: The accuracy improvements are slightly higher for Barlow Twins than for SimCLR.\n  - 2 types of **structural regularization**, both meant to bottleneck the information allowed by the projection head.\n    - Discretized projector: the output is discretized into $L$ integer values.\n    - Sparse projector: using the top-k activation function, which keeps the $k$ largest hidden representation. \n        - The experiments use $k=0.001d$ or $k=0.2d$, where $d$ is the projector's latent dimension.", "questions": "- Fig 2: the curves are only shown up to 100 epochs, at which point the validation accuracy hasn't saturated. What happens to epoch 100-200 (Sec 4 mentions that the models are trained for 200 epochs on CIFAR-100)?\n- Do you observe the same trends on other datasets (i.e. CIFAR-10 and ImageNet-100, which are partially shown in Fig 3) and from Barlow Twins?\n- Fig 3: are these results varying across training hyperparameters (e.g. learning rate, weight decay, augmentation strengths, temperature of the contrastive objective)?\n- For the accuracy: since the paper is based on mutual information, the downstream experiments should also consider nonlinear evaluation, i.e. making the downstream classifier sufficiently powerful to make sure the performance is actually bottlenecked by the \"information\" in the representation rather than by the classifier.\n- Does combining several of the 3 proposed changes lead to further improvements?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed changes are motivated by the terms in the mutual information-based lower and upper bounds.\n- The proposed changes lead to an 0.26 to 3.99 increase in the accuracy of SimCLR and Barlow Twins on CIFAR-10, CIFAR-10, and ImageNet-100.", "weaknesses": "Since the theoretical results of the paper are not strong enough to serve as a major contribution, the empirical results should be strong.\n\nHowever, the relation between the derived bounds and the empirical results are not strong: the correlation is computed based on a small number of points (please see the question below), and there is no correlational result on Barlow Twins.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7yIiEnZnBY", "reviewer": "Reviewer_7rGv", "summary": "- The paper proposes that contrastive methods using separate representation layers (for downstream tasks) and projection heads (for computing the contrastive loss) are actually taking advantage of an Information Bottleneck between the representation layer and the projection head.\n- The authors derive lower and upper bounds on I(Y;Z_1) (the Mutual Information between the downstream target, Y, and the representation Z_1) that show an I(Z_1;Z_2) term being minimized (where Z_2 is the projection layer’s representation).\n- The authors show a correlation between their bounds and the validation set accuracy to attempt to justify their theoretical claims.\n- The authors then use an estimator of I(Z_1;Z_2) as a regularizer in the contrastive loss and show empirically that using that regularizer improves performance on the downstream classification task for three image classification problems.\n- The authors then compare with two previously-studied “structural” regularizers that (they argue) also minimize I(Z_1;Z_2) and show that those regularizers also improve performance on these classification tasks.", "questions": "- Line 250: why do you set \\alpha=2? Wouldn’t \\alpha=1 correspond to the Shannon Mutual Information? Is there some benefit to distorting the Shannon Information in this setting?\n- Given that the “structural” regularizers are more effective than the direct minimization of (an estimate of) I(Z_1;Z_2), my sense is that this paper needs to do more work to demonstrate that the “structural” regularizers improve performance for the stated reason (that they minimize I(Z_1;Z_2)). I don’t contest that they *are* reducing I(Z_1;Z_2), but I would like to know that there isn’t some alternative hypothesis that better explains the improvement in performance. Alternatively, perhaps it’s worth considering a variational approach to minimizing I(Z_1;Z_2), such that you can properly bound I(Z_1;Z_2) – perhaps the relatively poor performance of your objective-based approach is due to using an estimate of I(Z_1;Z_2) rather than a proper bound? All that being said, I think the underlying hypothesis is plausible, so I welcome your pushback on my concerns about the paper. I’m open to good arguments to increase my rating.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The writing is generally clear and easy to follow.\n- The paper gives an Information Theoretic explanation for why the projection head is important in many contrastive loss settings.\n- The paper proposes a method to directly minimize the claimed quantity of interest, I(Z_1;Z_2) and shows that using that minimizer can improve accuracy on the downstream classification task.\n- The paper provides an explanation for the effectiveness of aggressive bottlenecking between the representation layer and the projection head.", "weaknesses": "- Line 154: The simplest relationship of interest is that I(Y;Z_1) >= I(Y;Z_2), by the data processing inequality, so maximizing I(Y;Z_2) also maximizes I(Y;Z_1), which you acknowledge as the quantity of interest. Since by assumption we don’t have access to Y, we are limited to maximizing I(X;Z_2), which is what many SSL objectives do (including InfoNCE); again by the data processing inequality, I(X;Z_1) >= I(X;Z_2). But in the limit when I(X;Z_2) approaches H(X), I(Y;Z_2) must be maximized. While my view is that correct compression is important for learning good representations, this kind of information maximization argument is common in SSL (since you don't know what the downstream task is, why would you want to throw out any information in the representation?), so I think it is important to be careful how you justify your approach.\n- Thm 3.2, line 209: You appear to be using I(Y;Z_1|Z_2) = H(Z_1|Z_2) - H(Z_1|Y,Z_2) and then assuming that H(Z_1|Y,Z_2) >= 0, but that is not guaranteed unless Z_1 is discrete, which it probably isn’t. So it seems to me that the bound is broken, or at least that your proof of the bound may be incorrect. Have I misunderstood your justification for the inequality in line 209?\n- Figure 2a: The lower bound estimate is non-monotonic with a noticeable (but small) peak at 80 epochs, even though validation accuracy increases monotonically through epoch 100. This indicates that you should probably run more experiments to get more robust statistical results, or that your analysis is not perfectly accurate. Or perhaps some other explanation, but it doesn’t appear to be addressed in the paper, as you claim (line 264) that the lower bound “continuously increase[s]”.\n- More generally, the “Tendency during training” paragraph is weak, as there are many quantities that can generally increase during training while accuracy is increasing, without real explanatory power, such as weight magnitudes.\n- The two “structural” regularizers were already known to improve downstream task performance, so it is not clear to me that the empirical results are adding much to the presentation, although they do show that (in the settings you consider) the explicit minimization of I(Z_1;Z_2) is less effective than just using a strong architectural bottleneck.\n- It might be helpful in Figure 1 to show how Z_1 is used in downstream tasks – i.e., add an edge Z_1 \\to \\hat{Y} or something like that.\n- Line 152: “pipline” => “pipeline”.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RRpzb7MuxV", "reviewer": "Reviewer_7XZ7", "summary": "The paper analyzes the performance of encoder features (before projection head) in contrastive learning. Using an information-theoretic perspective, it derives bounds on the mutual information of the encoder features with the labels. Based on these bounds, it concludes that a good projection head removes unnecessary information from encoder features. It then proposes training and structural modifications to this end, and validate the effectiveness empirically on a few datasets.", "questions": "I wonder if/how the analysis can be applied to supervised learning, e.g. could features from before the last layer in supervised learning also be improved in a similar fashion?\n\nWhat is the impact/motivation for the choice of $\\alpha$ in the matrix based mutual information?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper provides a novel analysis of pre-projection features in contrastive learning based on mutual information. The proposed modifications to contrastive learning are well motivated by the theory, and the experiments help validate the theory. \n\nThe paper is overall pretty easy to follow.", "weaknesses": "See questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "R4rccOQ35r", "reviewer": "Reviewer_7oxV", "summary": "This paper aims to understand the role of the projection head in contrastive learning from an information-theoretic perspective. Treating the mutual information between the supervised label Y (potentially in downstream tasks) and the learned representations Z1 (before the projection head) as the metric for downstream task accuracy, this paper presents both the lower and upper bounds based on mutual information between features at different layers and proposes that the term I(Z1;Z2) as the information gap plays an important role. To boost the performance of downstream tasks, this paper proposes two types of modified loss functions to regularize the term I(Z1;Z2).", "questions": "1. [Theorem 3.1 and 3.2] Regarding the lower and upper bounds for the mutual information I(Y,Z1) as the proxy of the downstream accuracy, there are three terms in each bound, in which, the effect of I(Z1;R) (or I(Y;Z2)) and I(Z1;Z2) are mixed. Also, as  is shown in Figure 2, both the upper and lower bounds are not tight, it is not sufficiently convincing to claim that the role of the projection head is only revealed via I(Z1;Z2). I was wondering if we could design experiments (e.g. fixing the encoder) such that other terms in the bounds are fixed to study the role of I(Z1;Z2).\n2. [Calculations of mutual information] Suppose there exist perfect features Z1 such that Z1 = Y almost surely. This corner case with infinity mutual information should be taken care of.\n3. [Figure 3] It seems that points in the right-top corner are more close to a horizontal line. Is it because the upper and lower bounds are not tight enough?\n4. In addition, there are several related papers investigating the role of the projection head:\n\t- Saunshi, Nikunj, Jordan Ash, Surbhi Goel, Dipendra Misra, Cyril Zhang, Sanjeev Arora, Sham Kakade, and Akshay Krishnamurthy. \"Understanding contrastive learning requires incorporating inductive biases.\" In International Conference on Machine Learning, pp. 19250-19286. PMLR, 2022.\n\t- Gui, Yu, Cong Ma, and Yiqiao Zhong. \"Unraveling Projection Heads in Contrastive Learning: Insights from Expansion and Shrinkage.\" arXiv preprint arXiv:2306.03335 (2023).\n\t- Wen, Zixin, and Yuanzhi Li. \"The mechanism of prediction head in non-contrastive self-supervised learning.\" Advances in Neural Information Processing Systems 35 (2022): 24794-24809.\n\t- Xue, Yihao, Eric Gan, Jiayi Ni, Siddharth Joshi, and Baharan Mirzasoleiman. \"Investigating the Benefits of Projection Head for Representation Learning.\" arXiv preprint arXiv:2403.11391 (2024).\n\nI am open to raising my score if the more explicit connection between the mutual information I(Y;Z1) and the downstream accuracy can be presented in some concrete settings, and if fine-grained analysis can be provided to study the tightness of the lower and upper bounds.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. It is novel to view the projection head as information bottle neck, which provides one way to regularize the infoNCE loss to directly boost the downstream performance\n2. Different types of regularization are proposed in this paper, which enables the cross-comparison and validates the consistent improvement over the canonical infoNCE loss.", "weaknesses": "1. The performance of downstream task (e.g. classification accuracy) is related to mutual information implicitly. Thus, using mutual information between Y and Z1 as the target to justify the claim seems insufficient to me. It would be very helpful to provide concrete toy examples to relate the downstream accuracy and the mutual information in an explicit way. In this sense, the information gap I(Z1;Z2) seems to be a reasonable motivation for the regularized loss, but is not strong enough as a theoretical explanation to fully understand the role of the projection head.\n2. The message conveyed in experiments is not clear enough. \n\t- For example, in Figure 2 and 3, instead of the comparison between lower bound VS downstream accuracy, as the claim of this paper is the role of I(Z1;Z2), it would be more insightful to plot downstream accuracy VS I(Z1;Z2).\n\t- In addition, in Figure 4, it would be more helpful to compare the proposed methods in terms of the downstream accuracy with more datasets to see if the ranks of all methods remain consistent.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)