MetaMask: Revisiting Dimensional Confounder for Self-Supervised Learning
Authors: Jiangmeng Li
Wenwen Qiang
Yanan Zhang
Wenyi Mo
Changwen Zheng
Bing Su
Hui Xiong
Venue: NeurIPS 2022 Accept (2022)
Keywords: self-supervised learning
representation learning
multi-view
contrastive learning
dimensional analysis

Abstract: As a successful approach to self-supervised learning, contrastive learning aims to learn invariant information shared among distortions of the input sample. While contrastive learning has yielded continuous advancements in sampling strategy and architecture design, it still remains two persistent defects: the interference of task-irrelevant information and sample inefficiency, which are related to the recurring existence of trivial constant solutions. From the perspective of dimensional analysis, we find out that the dimensional redundancy and dimensional confounder are the intrinsic issues behind the phenomena, and provide experimental evidence to support our viewpoint. We further propose a simple yet effective approach MetaMask, short for the dimensional Mask learned by Meta-learning, to learn representations against dimensional redundancy and confounder. MetaMask adopts the redundancy-reduction technique to tackle the dimensional redundancy issue and innovatively introduces a dimensional mask to reduce the gradient effects of specific dimensions containing the confounder, which is trained by employing a meta-learning paradigm with the objective of improving the performance of masked representations on a typical self-supervised task. We provide solid theoretical analyses to prove MetaMask can obtain tighter risk bounds for downstream classification compared to typical contrastive methods. Empirically, our method achieves state-of-the-art performance on various benchmarks.

MetaMask: Revisiting Dimensional Confounder for Self-Supervised Learning
Jiangmeng Li", Wenwen Qiang", Yanan Zhang
University of Chinese Academy of Sciences Institute of Software Chinese Academy of Sciences
Southern Marine Science and Engineering Guangdong Laboratory (Guangzhou) {jiangmeng2019, wenwen2018, yanan2018)@@ssee ac cn
Wenyi Mo
Gaoling School of Artificial Intelligence
Renmin University of China 2022101010eruc. edu.
Changwen Zheng
Institute of Software Chinese Academy of Sciences Southern Marine Science and Engineering Guangdong Laboratory (Guangzhou) changwendiscas. ac
Bing Sul
Gaoling School of Artificial Intelligence
Renmin University of China
Beijing Key Laboratory of Big Data Management and Analysis Methods
subingats@gmail. com
Hui Xiong
Thrust of Artificial Intelligence The Hong Kong University of Science and Technology (Guangzhou) Guangzhou HKUST Fok Ying Tung Research Institute xionghui@ust. Brigh .k
Abstract
As successful approach self-supervised learning, contrastive learning aims learn invariant information shared among distortions of the input sample. While contrastive learning has yielded continuous advancements in sampling strategy and architecture design, still remains two persistent defects: the interference task-irrelevant information and sample inefficiency, which related to the recur- ring existence trivial constant solutions. From the perspective of dimensional analysis, we find out that the dimensional redundancy and dimensional confounder are the intrinsic issues behind the phenomena, and provide experimental evidence support our viewpoint. We further propose simple yet effective approach MetaMask, short for the dimensional Mask learned by Meta Meta-learning, to learn representations against dimensional redundancy and confounder. MetaMask adopts the redundancy-reducteo technique to tackle the dimensional redundancy issue
"Equal contributions.
t "oorrsppnniigg author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

CIFAR-10
ImggeNe
ENL
EMH
E.50
Feature Dimensional Dicforezee
1.3
912 K0H 360 3022
9100 122/0
512 304 3M0 400 0000 103
Dinataccaalt pwdsedoin huad
Diocestanatty ofpegistion
(b)
(a) The visualization of the representations learned by SimCLR [ and our method using the cetundancy-reductionn technique 2 on CIFAR-10, respectively. The learned features projected into a color image in RGB format, where different colors represent different types of features. The abscissa axis represents the feature dimensions, and the ordinate axis represents samples of different classes. The greater the color contrast, the lower the dimensional feature similarity. The left plots present the contribution of different dimensions specific category classification, and the right plots present the similarity between dimension features within a batch. Compared with SimCLR, our method using eddundancy-rduuctio can indeed learn representations with decoupled dimensions. (b) The experimental results obtained by Barlow Twins on CIFAR- 10 and ImageNet datasets. We intuitively increase the dimensionality of the projection head, which leads the dimensions of representations to model more different information. Note that we adopt the official code of Barlow Twins and train on 8 GPUs of NVIDIA Tesla V100.
Figure 1:
and innovatively introduces dimensional mask reduce the gradient effects of specific dimensions containing the confounder, which is trained by employing meta-learning paradigm with the objective of improving the performance of masked representations on typical self- ical analyses to prove MetaMask can obtain tighter risk bounds for downstream classification compared to typical contrastive methods. Empirically, our method achieves wate-of-the-ee art performance on various benchmarks/
task. We provide solid theoret-
Introduction
fundamental idea behind self-supervised learning learn discriminative representations from the input data without relying on human annotations. Recent advances in visual self-supervise demonstrate that unsupervised approaches can achieve competitive performance over supervised approaches by introducing sophisticated learning paradigm contrastive learning g [
learning 3 5 6
superviidd tasks. A representative
I2 [3], which aims to learn invariant information from different views (generated by data augmentations) by performing instance-level contrast, i.e., pulling views of the same sample together while pushing views of different samples away. Various 15 16. 17] techniques have been explored to address the problem of trivial solutions to self-supervised contrastive learning, e.g., constant representations. However, state-of- the-art methods still suffer from two crucial defects, including the interference of task-irrelevant information and sample inefficiency. The trivial solution problem is related to such defects, e.g., small number of negative samples large proportion of task-irrelevan information may lead the model to learn constant representations. We revisit the current self-supervised learning paradigm from the perspective of dimensional analysis and argue that the intrinsic issues behind the defects of self-supervised learning are the dimensional redundancy and dimensional confounder
9
I0 IT
'The implementation available at/attps///iithub www.//oowmwmmeeeeeeeeooooooo Ma0000088///taMaaa 'Throughout this paper, the term confounder is used in its idiomatic sense rather than the specific statistical in Structural Causal Models (18).

---

CIEAR-10
InnageNet
13.4
73.30
152
73.0
452-
442-
12.4
Bakee
20 25
45
0.5
3
8
20
30
40
Random dimensional mask rate (16)
Random dimensional mask rate (%)
Figure 2: Experimental scatter diagrams obtained by Barlow Twins [2] with randomly masked dimensions on CIFAR-10 10 and ImageNet datasets, where Baseline and the red dashed lines denote the performance achieved by the unmasked representation of Barlow Twins. Every single point denotes an independent experimental result achieved by randomly masking the original representation at the dimensional level with a specific mask rate. Note that the original representation is consistently fixed. Following the experimental protocol of [2 [6], we evaluate the performance CIFAR-10 and ImageNet by adopting KNN prediction and linear probing, respectively, which further proves the general existence of dimensional confounders.
From the perspective of information theory, each dimension holds a subset of the representation' 's information entropy. The dimensional redundancy denotes that multiple dimensions hold overlapping information entropy. To tackle this issue, Barlow Twins [2] takes advantage of a neuroscience approach, i.e., redundancy-rrcuuttoon [19], in self-supervise learning. It makes the cross-correlation matrix of twin embeddings close to the identity matrix, which is conceptually simple yet effective to address the dimensional redundancy issue. Figure (a) supports the superiority of redundancy- reduction in addressing the dimensional redundancy problem. Since Barlow Twins encourages the dimensions of learned representations to model decoupled information, it naturally avoids the collapsed trivial solution that outputs the same vector for all images.
However, only reducing dimensional redundancy still suffers from the problem of dimensional confounder. The dimensional confounder indicates a set of dimensions that contains "harmful" information and further degenerates the performance of the model, c.g., the dimensions simply capturing chaotic and worthless background information from the input. To prove the existence of such dimensional confounders, we conduct motivating experiments from two perspectives. It is stated in ( that Barlow Twins keeps improving as the dimensionality of the projection head increases, since the backbone network acts dimensionality bottleneck to constrain the dimensionality of the representation so that the learning paradigm of Barlow Twins can promote the representation capture more fask-irrelevant information with the same dimensionality. Counterintuitively, our observation from the motivating experiments is in stark contrast with the conclusion in [2]. As shown in Figure | (b), the performance of Barlow twins as a function of the dimensionality presents a convex pattern on either 1I0 or ImageNet. After reaching the peak, the performance gradually decreases as the dimensionality increases until it collapses. We reckon that since the task-relevant semantic information is limited, as the information captured in different dimensions is gradually decoupled, more task-irrelevant noisy information encoded into the representation, i.e., the dimensional confounder increases, so the performance of the learned representation is getting worse until collapse.
To further validate our inference, we randomly mask several dimensions by frankly setting the masked values to 0, and then we evaluate the performance of the masked representation. The experimental results are demonstrated in Figure 2 where every single point denotes the result of sampled mask. The representations with some specific masked dimensions can indeed achieve better performance than the unmasked representation, and such an observation consistent on a small-scaled dataset, e.g., CIFAR- 10, and large-scale dataset, e.g., ImageNet. The dimensional mask rate that can improve the performance of the model on ImageNet is on average lower than that on CIFAR- 10. We understand that this phenomenon depends on an ingredient: the representation learned on CIFAR-10 10 models discriminative information for 10 categories in 512 dimensions, while the representation uses 2048 dimensions to model information for 1000 categories on ImageNet, so the dimension-to-doe -llcass ratio on CIFAR-10 (51.2) is much higher than that on ImageNet (2.048). Therefore, the presence of dimensional confounders naturally remains at a low level on ImageNet since supporting 1000- category classification requires a large amount of heterogeneous discriminative information. This empirical evidence further supports our assumption of the dimensional confounder.
3

---

To this end, we intuitively propose simple yet effective approach MetaMask, short for the Meta- learned dimensional Mask. to adjust the impacts of different dimensions in self-paced manner. In order to learn representations against dimensional redundancy and confounder, we regard the cross-correlation matrix constraint of Barlow Twins as dimensional regularization term and further introduce a dimensional mask, which simultaneously enhances the gradient effect of the dimensions containing discriminative information and reduces that of the dimensions containing the confounder during training. We employ the meta-learning paradigm to train the learnable dimensional mask with the optimization objective of improving the performance of masked representations on typical self- supervised tasks, e.g., contrastive learning. Consequently, the dimensional mask incessantly adjusts the weights of different dimensions with respect to their gradient contribution to the optimization, so as to make the model focus the acquisition of discriminative information, which can address the trivial solution caused by treating all dimensions equally in optimization, especially for high-dimensional representations. Empirically, we conduct head-to-head comparisons on various benchmark datasets, which prove the effectiveness of MetaMask. Concretely, the contributions of this paper are four-fold:
We present a timely study on the adaptation of the state-of-the-arr self-supervised models downstream applications and identify two critical issues associated with the learned representation, i.e., dimensional redundancy and dimensional confounder.
We propose novel approach to tune the gradient effects of dimensions in a self- safed manner during optimization according to their improvements on a self-supervised task.
Theoretically, we provide theorems and proofs to support that MetaMask achieves relatively tighter risk bounds for downstream classification compared typical contrastive methods.
We conduct extensive experiments to demonstrate the superiority of MetaMask over bench- mark methods on downstream tasks.
2
Related Work
Sele-super learning achieved impressive success in the field of representation learning with- out human annotation by constructing auxiliary self-supervised tasks. Typically, sule-supervised approaches be categorized as either generative or discriminative [ Conventional genera- tive approaches bridge the input data and latent embedding by building a distribution over them, and the learned embeddings are treated image representations [21 23 24 25 26). However, such a learning paradigm learns the distribution at the pixel-level, which is computationally expensive. The detailed information required for image generation may be redundant for representation learning.
For discriminative methods, contrastive learning achieves state-of-the-ar performance on various 29 30 31 32,
33 [5]. Deep InfoMax [3] explores maximize the mutual information between the input and output of an encoder. CPC (5) adopts noise-contrastive estimation [34] as the contrastive loss to train the model by maximizing the mutual information of multiple views, which is deduced by using the Kullback-Leiler divergence (35). CMC [ and
downstream tasks 6 27 28
AMDIM 4 employs contrastive learning on the multi-view data. SimCLR (I and MoCo p 36
use large batch or memory bank to enlarge the amount of available negative features. To sample informative negatives, DebiasedCL [10] and HardCL [II) design sampling strategies by using positive- unlabeled learning methods [37 38 Motivated by (39]. 40 provides an information theoretical framework for self-supervised learning and proposes information bottleneck- based approach. However, these methods rely on a large number of negative samples to learn valuable representations while avoiding trivial solutions, i.c., constant representation. SimSiam [66] and BYOL [14) jointly waive negatives and avoid training degradation by using the techniques of asymmetric siamese network and stop-gradient. SSL-HSIC 41 provides a unified framework for negative- contrastive learning based on Hilbert Schmidt Independence Criterion [42) The representation learning paradigm employed by current contrastive methods is contrary to the coding criteria proposed by [19 and
43 respectively. Therefore, Barlow Twins [2] proposes to constrain the correlation matrix between the features to be close to the identity matrix, which can effectively reduce the dimensional redundancy. Yet, such an approach still suffers from the dimensional confounder issue, and we thus propose a new meta-learning- based method to attenuate the over-learning of dimensional confounders by learning mask to filter out irrelevant information.

---

and o8 updiate Job 9"T (.
g (-)
Make commtation mukils close the aduniity
fo
A
14
M
93 (
~ V.
f-
controst
h
updatte M via a meta learning technigue
Figure The architecture of MetaMask. The solid line pointing backwards represents the regular training step, where the encoder and projection heads are trained by jointly back- pack-propagating Larr and Ceontrast- The solid blue line pointing backwards represents the meta-learning step, where dimensional mask M is trained by performing the second-derrvative technique on Ccontrast- All learnable networks are trained until convergence.
Preliminaries
| [1,N [1. where
Formally, suppose the input multi-view dataset as X'
N denotes the number of samples, and .M denotes the number of views. Therefore, x represents the j-th view of the i-th sample following x! X. The multiple views are built by performing
identically distributed data augmentations 8 2 i.c., [t, T.
3.1 Contrastive Learning
We recap the preliminaries of contrastive learning [1. which aims to learn embedding that jointly maximizes agreement between views the same sample and separates the views of different samples in the latent space. Given minibatch of N examples sampled from the multi-view dataset X, positives represent the pairs (a/ x where j.j' E [1,M], and negatives represent {æ/ (2'" where i + V. Conventional contrastive methods feed the input æ into a view-shared encoder f+(+) to learn the representation h;, which is mapped into feature z! by projection head g*(+), where and 0 are the network parameters. fo(-) and g+)-) are trained by using the contrastive loss
exp
[2],
/T]
Lcontrast
Σ
Σ
log
EVELL,N Eyep, 1GARv1441 exp ( (
/T
JE[LM-] 1] XE[L,N] M+
where d(.) a discriminating function to measure the similarity of the feature pairs, (i44/13441) denotes an indicator function equalling if i' or j j and is a temperature parameter valued by following [ In inference, the projection head go (-) is discarded, and the representation h is directly used for downstream tasks.
5

---

3.2 Dimensional Redundant Reduction
Following the principle proposed by 19
introduces redundanny-reduucion objective function:
Carr
Σ MEI jt +
Σ kE[1,D]
C1-Ck lkk
11
+X
Σ
(Cl 2 kil
(2)
where À is positive hyper-paree parameter valued by following (2), and C' is the cross-correlation matrix computed between the outputs of the encoder along the batch dimension:
EiE[L,N Z/.0 2
C
EEELLAN (Z(Ae)
. EiELLNN
)
x ((,K))
where k and /' index the dimension of features. C' square matrix with the size of D. which denotes the dimensionality of the projected feature z. The value range of the elements in C is [-1,1]. i.e., complete anti-correlation complete correlation.
Methodology
Our goal is to Iearn discriminative representations against dimensional redundancy and dimensional confounder from multiple views by performing self-supervised learning. To this end, we propose MetaMask to automatically determine the optimal dimensionality of the representation by decoupling information into different dimensions and filtering out those dimensions with irrelevant information. MetaMask addresses the dimensional redundancy issue by adopting a couuddancy-redcctin regu- larization. To tackle the dimensional confounder problem, MetaMask learns a dimensional mask to adjust the gradient weights of dimensions according to their contributions to improving specific self-supervised task during optimization, which achieved in meta-learning manner.
MetaMask's architecture shown Figure Specifically, the sample x! is first fed into the encoder to obtain its representation h/. Then, we build a learnable dimensional mask M = {w>x | E [1, DI}.
which is introduced to assign weight each dimension of the representation h by
h
h
X M
(4)
where h denotes the masked representation, and E is element-wise Hadamard product function. h is then fed into separate projection heads gdrr() (.) and (.). respectively. g8 (-) plays the same role as in conventional contrastive learning methods, i.c., the contrastive loss Ccontrast in Eq. (T) is
applied the output features of g (-). For the projection head (d))(-), the redundancy-reduucion loss Ldrr in Eq. is performed to its output features to tackle the dimensional redundancy.
In training, the encoder fe(.), the projection heads gd (-) and (-), M are trained a meta-learning manner consisting of two steps. In the first regular training step, we train To(-).9) o Arr (-) and gō (+) by jointly minimizing the redundancy-reducction and contrastive losses, which is formalized by
(-), well as the dimensional mask
Pregutar Larr + a Leontrast
where a is a coefficient that controls the balance between Larr and Lcontrast. and we evaluate the influence of this hyper-parameter in Section
In the second meta-learning-baase based step, we update M by using the second-derivative technique to solve a bi-level optimization problem. We encourage M to mask (reduce the gradient weights of) the specific dimensions containing task-irrelevant information, which are regarded as dimensional confounders, that the encoders can better explore the discriminative information in training. To this end, M can be updated by computing its gradients with respect to the performance of fo(-) and gg' (.). The corresponding performance is measured by using the gradients fo(.) and g (.) during the paokapropn of contrastive loss. Formally, we update the dimensional mask M by
arg min Leontrast (98m.ct (focrial (X') @ M))
(6)
M

---

Algorithm 1 MetaMask
Input: Multi-view dataset X. minibatch size N, and a hyper-parameter a.
Initialize The neural network parameters: 0 for fol-), it for (.), ydr for (drr()), and M
{wx//k k E DI). The learning rates: lo and Lg, etc.
repeat
for t-th training iteration do
Iteratively sample minibatch X' with N examples from X. # regular training step, fix M
arg ming. 0.0 Ldvr gg (fo (X") M)) +a Leontrast (g) (fo (2") @ M))
# compute trial weights and retain computational graph, fix A and
Orrial 0 lo VoVa Wooeontrast (g (X') 1 M))
Otrial = VoV &oontross (ge (So (x") s M))
# meta training step using second derivative
arg min Leontrast (grress (Jorsn (X') & M))
end for
until 0, 0, gdrt, and M converge.
where X' represents a minibatch sampled from the training dataset X, Otriat and Otrial represent the trial weights of the encoders and projection heads after gradient update using the contrastive loss defined in Equation|] 1 respectively. We formulate the updating of such trial weights as follows:
Osrial 0-6 V Coontrast (g (So (x") M)) Strial 9-la V Loontrast (g3 (Jo (X') 8 M))
(7)
where tg and to are learning rates. The intuition behind such a behavior is that we perform derivative over the derivative (Hessian matrix) of the combination {0, Jt to update M, i.c., the second-derivative trick. Specifically, we compute the derivative with respect to A by using a retained computational graph of {6, 0) and then update M by back- pockppropagating the derivative as Equation 6
We train M only based on the performance of contrastive learning for two reasons: 1) the dimensional
redundancy reduction is a regularization term with strong constraints and generally stable convergence, and it is independent of downstream tasks; 2) such meta-learning-basee training approach that promotes contrastive learning performance empowers our method to constrain the upper and lower bounds of the cross-entropy loss on downstream tasks, which ensures that M can partially mask the gradient contributions of dimensions containing iask-itreeevant information and further promote the encoder focus on learning task-relevant information, which demonstrated in Section
The two steps for updating fo(.), (-), and g (-) and updating .M are iteratively repeated until convergence. The training pipeline is detailed by Algorithm
5 Theoretical Analyses
Current contrastive methods follow the fundamental assumption of multi-view learning (Assumption 1 in (39], Assumption in [40], and Assumption 4.1 in (45)), i.e., views (or distortions) of same sample holds the invariant discriminative label-relevant information. Theorem 4.2 in (55 further proposes the guarantees for general encoders in the learning paradigm of regular contrastive learning, which proves that the contrastive loss in the self-supervised learning stage can constrain the upper and lower bounds of the cross-entropy loss in the supervised learning stage for downstream tasks. To elaborate the behavior of our MetaMask, we extend this Theorem to build connection between the proposed masked representation and downstream performance by
Theorem 5.1. (Connecting Masked Representations Downstream Cross- Pooss-Entropy Loss). Under the general assumption of self-supe multi-view learning, when M is optimal, for any coupled h, h E Lerossentropy (h) for downstream classification can be bounded by Lcontrast 5 (h)
7

---

Table 1: Comparison of different methods classification accuracy (top 1). We follow the experi- mental setting of g and adopt conv and as the backbones in the experiments.
NN-200
46
STL- 10
47
CIFAR- 10 46
CIFAR-100
46
Model
conv
fc
conv
conv
fc
conv
Barlow Twins 2
39.81
40.34
80.97
81.43
76.63
78.49
52.80
52.95
MetaMask, 1.93
42.91
+3.10
42.06
84,96 3. 99 86.13
85.24 4-4.3+4.70 7.
4..61
86.90 48. 41 58.02
56.78
SWAV 6
39.67
39.02
74.65
75.30
72.19
72.36
46.58
46.75
+ MetaMask 5.000
40.28
0..6 61
40.05
1.03
81.34
82.19+6
80.37 +-..77+5... 18 77.22
4..6
52.07
52,98
SimCLR 0
36,24
39.83
75.57
77.15
80.58
80.07
50.03
49.82
+ MctaMask, +3.00
40.56
40.82
81.37
82,90 45.77
82.85 4-2.7
,1.64,
53.70
43..7
54.24
CMC 8
41.58
40.11
83.03
85.06
81.31
83.28
58.13
56.72
MeMaMMsk
1.73
42.91
42.06
84.96
93
86.13 1.07
85.24 4.3.9
86.90
58.02
0.09
56.78
BYOI. 14
41.59
41.90
81.73
81.57
77.18
80.01
53.64
53.78
MetaMask
42.72.+
43.29
83.01
83,44
79.2642..0
0.
82.73
55.68
54.72+0.94
SimSiam 16
41.03
41.27
80.91
81.88
78.14
81.13
52.55
53.52
MetaMask
41,82
42.67
82.35
83,96
80.03
s.
81.39
54.70
22. 16 55.02
DCL [T]
38.79
40.26
77.09
78.39
80.89
80.93
51.38
51.09
MetaMask
40.28
41.69
80.94
81.06
81.33
4-.44
81.34
53.05
53.74
HCL [T]]
40.05
41.23
79.86
80.20
82.13
82.76
52.69
53.13
MetaMask +1.14 40.43
41.01
0.22
81.52
82.09
+1.80
83.65
83.95
54.03
34 54.26+1.13
in self-supervised learning:
Loontrast 9
(1))
V
(1)
h
y
2
Σ RE[1,D]
/
(Log'
(1 8) -0
(M(M-t)
S Lerossentropy
(h) + log
< Leontrast
( (ho) +
go
h O M
v) +
D
where M denotes the quantity of negative samples, D denotes the dimensionality of the representation,
h denotes the unmasked representation. h denotes the masked representation obtained by Equation
y denotes the target label, (-1y) Ep(y) [E (- E.(())))] represents the conditional
variance function where 8 (:,-)
-) is discriminating function to measure the difference between the
terms. e.g., 8 (..-) -III- in tow-dimensional embedding space or 6 (+,-) = -loo (cos (---))
- -log 1)((-) in high-dimensional embedding space, 1-1 is function acquiring k-th dimension |-
vector, log (M) is constant, and O (M-+) the approximation error's 's order.
Theorem 5.1 states that reducing the risk of contrastive learning can improve the performance on downstream tasks, which further supports our intuition to make the model focus on the acquisition of discriminative information by learning a dimensional mask with the objective of improving the performance of masked representations on contrastive learning is theoretically sound. Refer to Appendix A. I for the elaboration of MetaMask's 's behavior to mask the gradient contribution of specific dimensions containing confounder information. Hence, we derive
Theorem 5.2. (Guarantees for Reduced Conditional Variance of Masked Representations), When M is optimal, for any coupled h, h E R, given label y, the conditional variance of h is reduced:
T
90
h
00 (h) v) yet T (L08
11) = (Lo8 (1))*
hn
Theorem 5.2 states that given the label y, the masked representation h has smaller conditional variance than h in contrastive learning. We bring Theorem 5.2 into Theorem 1. derive a conclusion: our approach can better bound the downstream classification risk, i.e., the upper and lower bounds of supervised cross-entropy loss obtained by MetaMask are tighter than typical contrastive learning methods. Please refer to Appendix A.2 for the corresponding proofs.

---

6 Experiments
6.1 Benchmarking MetaMask with Various Backbones
Implementations. For the comparisons demonstrated in Table we uniformly set the batch size as 64, and we adopt a network with the 5 convolutional layers in AlexNet [48] as conv and network with 2 additional fully connected layers as fc. We straightforwardly adopt the same data augmentations in [8] and [49], and the memory bank [50] is adopted to facilitate calculations by retrieving 4,096 past negative features during training. Following the experimental principle of linear probing, remove the projection head and re- train an MLP as the classification head for the learned representation.
For the experiments in Table 2 the batch size is valued by 512, and ResNet-18 [1 is used as the backbone encoder. We adopt the data augmentation and other experimental settings following (2].
Note that the KNN classifier is adopted for evaluation.
Discussion of the comparisons using conv and fe encoders. Table shows the comparisons on four benchmark datasets, and the corresponding positive or negative margins are reported. We observe that MetaMask beats the best prior methods by a significant margin on most downstream tasks, which demonstrates that MetaMask can efficiently improve the performance of the learned representation when the suffsvieerisinn is insufficient, i.e., the limited batch size and weak encoder.
Discussion of the comparisons using the ResNet encoder. We further perform elassification comparisons using strong encoder, i.e., ResNet. Table shows that MetaMask can consistently improve the state-of-the-art methods, which indicates that the proposed method has strong adapt- ability to different encoders. The experimental evidence proves our assumption of the dimensional confounder. Thus, given fixed dimensionality of the projection head, MetaMask can indeed improve the discriminativeness of the learned representation by masking the dimensional confounder.
6.2 Validation of the Robustness of MetaMask against Dimensional Confounder
We conduct experiments to prove that MetaMask is able to mitigate the negative impact of the dimensional confounder on the discriminativeness of the learned representations, and the results are shown in Figure 4 The intriguing outcome demonstrates two merits of the proposed method: I) even the dimensionality of the projection head changes, MetaMask consistently robust to the dimensional confounder, i.e., the model maintains relatively consistent performance; 2) MetaMask can alleviate the negative impact of curse of dimensionality [52 53 c.g., given a fixed number of examples, as the dimensionality of the representation increases during training. the performance of the model on downstream tasks instead degenerates. Concretely, Figure 2 demonstrates that regardless of whether representation learning suffers from the curse of dimensionality, dimensionality confounding factors are widely present in learned representations, and hence, the defined dimensional confounder is more generalized issue of the curse of dimensionality. Refer Appendix A.3] for the further discussion. Moreover, we observe a surprising result that MetaMask achieves better performance. We provide an understandable explanation: the essence of the curse of dimensionality is that network with a large number of parameters is prone to over-fitting to the training data. Therefore, compared with the annealing strategy, a fixed and relatively large learning rate can prevent the model from falling into local optimum (over-fitting) during optimization. Our further experiments show that MetaMask collapses abruptly at larger dimensional settings, e.g., 16384, which demonstrates that the trick of using a fixed learning rate can only unsteadily improve the performance of MetaMask, but this behavior is not essential. Figure Asoows shows that under the different experimental settings, vanilla MetaMask consistently counteracts the negative effects of the dimensional confounder.
7 Conclusion and Future Discussion
From the dimensional perspective, we first clarify that the dimensional redundancy and confounder issues are the essence of the current sulf--uuer learning paradigm's 's defects. Multiple motivating experiments are conducted to support our viewpoint, and the intriguing results are opposite to the traditional conclusions. To jointly tackle the dimensional redundancy and confounder issues, we propose a heuristic learning approach, called MetaMask, which leverages a second-derivative technique to learn a dimensional mask in order to adjust the gradient weights of different dimensions during training. The proposed theoretical analyses demonstrate that MetaMask obtains relatively

---

Table 2: Comparison of different methods on classification accuracy
(top 1). We adopt ResNer- /8 as the backbones.
Model
CIFAR-10
CIFAR-100
STL-10
IN-200
Barlow Twins [
83.72
60.31
73.62
45.10
+ MMttaMas 1.25
4
61.42
73.97
46.71
VAN
SimCLR [
81.73
58.27
70.09
44.46
+ MetaMask 4.332
86.01
61.03
74.90
45.87
BYOL 14
88.05
60.94
72.04
46.72
Figure 4: Comparisons of varying the dimensionality of the projection head CIFAR-10 by using ResNet- 18. denotes MetaMask us- ing a trick of fixed learning rate instead of the cosine an- nealing strategy.
2048 4006 8142
+ -eetaMak44..25
87.53-0.52
61.422+0.8
73.97+1.93
46.81 46.81+0.111
DCL [1]]
85.61
59.29
71.18
45.77
+ MetaMask
86.47 4.0.86
60.82
74.79
46,35
HCL M
85.27
61.21
71.92
46.90
+ MMetaMask
85.97
61.63+4..4
74.03
46.06+0005
NNCLR [54]
81.44
61.64
71.89
47.10
+ MettaMask41.63
85.78
61.89
74.02
46.90-o.20
tighter risk bounds for downstream classification compared conventional contrastive methods. From the experiments, we observe a consistent improvement in the performance of our method over sfate-of-the-arr methods on benchmark datasets.
One of our critical contributions is proposing heuristic learning paradigm for self-supervised learn- ing: regarding the dimensional redundancy reduction a regularization term and further exploring to mask dimensions containing the confounder. In the current methodology, we train the learnable dimensional mask with the objective of improving the performance of masked representations on a specific self-supervised task, i.c., contrastive learning, and provide the theoretical analysis to prove that such learning approach can indeed prompt the model focus on acquiring discriminative information. However, contrastive learning is immutable for the sepervueee task in MetaMask.
commonly acknowledged that masked image modeling (MIM) methods [55] [56] [5]] achieve
impressive performance in self-supervise learning. Although the theoretical contribution bridging the gap between MIM loss and cross-entropy loss on downstream tasks has not yet emerged, we still deem combining MIM approaches with our proposed paradigm is attractive to the community.
Acknowledgements
The authors would like to thank the anonymous reviewers for their valuable comments. This work is supported in part by National Natural Science Foundation of China No. 61976206 and No. 61832017, Key Special Project for Introduced Talents Team of Southern Marine Science and Engineering Guang- dong Laboratory (Guangzhou) No. GML2019ZD0603, National Key Research and Development Pro- gram of China No. 2019YFB 405100, Foshan HKUST Projects (FSUST21-FYTRIO1 A, FSUST21- FYTRI02A), Beijing Outstanding Young Scientist Program NO. BJJWZYJHO1200 201 111199000009, Beijing Academy of Artificial Intelligence (BAAI), the Fundamental Research Funds for the Central Universities, the Research Funds of Renmin University of China 21XNLG05, and Public Comput- ing Cloud, Renmin University of China. This work is also supported in part by Intelligent Social Governance Platform, Major Innovation & Planning Interdisciplinary Platform for the "Double-First Class" " Initiative, Renmin University of China, and Public Policy and Decision-makinn Research Lab of Renmin University of China.
References
[1] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning. ICML 2020, 13-18 July 2020, Virtual Event, Proceedings of Machine Learning Research. PMLR, 2020.
[2] Jure Zbontar, Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self- supervised learning via redundancy reduction. In Proceedings of the 38th International Con- ference on Machine Learning, ICMI. 2021, 18-24 July 2021, Virtual Event, Proceedings of Machine Learning Research. PMLR, 2021.
10

---

[3] R. Devon Hjelm, Alex Fedorov, Samuel Lavoie Mavoie-Marchildon, Karan Grewal, Philip Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
[4] Philip Bachman, R. Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. In NeurlPS 2019, 2019.
[5] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 1007.08307.3374 2018.
[6] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Minan-Tienna Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
[7] Jiangmeng Li, Wenwen Qiang. Changwen Zheng, Bing Su, Farid Razzak, Ji- Ji-Rong Wen, and Hui Xiong. Modeling multiple views via implicitly preserving global consistency and local complementarity. IEEE Transactions on Knowledge and Data Engineering, pages 1-17, 2022.
[8] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. arXiv preprint 1rXiv:1906.05849 2019.
[9] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. In 2020 JEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, Computer Vision Foundation/ / IEEE, 2020.
[10] Ching- Cho Chuang, Joshua Robinson, Yen-Chen Lin, Antonio Torralba, and Stefanie Jegelka. Debiased contrastive learning. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tiee Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
[11] Joshua David Robinson, Ching-Yar Yao Chuang. Suvrit Sra, and Stefanie Jegelka. Contrastive learn- ing with hard negative samples. 9th International Conference on Learning Representations, ICIR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
[12] Jiangmeng Li, Wenwen Qiang, Changwen Zheng, Bing Su, and Hui Xiong. Metaug: Contrastive learning via meta feature augmentation. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings PMLR, 2022.
of Machine Learning Research, pages
[13] Wenwen Qiang, Jiangmeng Li, Changwen Zheng, Bing Su, and Hui Xiong. Interventional contrastive learning with meta semantic regularizer. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 18018-18030, PMLR, 2022.
[14] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Ávila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent- A new approach to self supervsed learning. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
[15) Aleksandr Ermolov, Aliaksandr Siarohin, Enver Sangineto, and Nicu Sebe. Whitening for self self-supervised representation learning. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning. ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 3015-3024 PMLR, 2021.
[16] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021. Computer Vision Foundation/ / IEEE, 2021.

---

[17] Hang Gao, Jiangmeng Li, Wenwen Qiang, Lingyu Si, Fuchun Sun, and Changwen Zheng. Bootstrapping informative graph augmentation via meta learning approach. In Luc De Raedt, editor, Proceedings of the ThirttyFirrs First International Joint Conference on Artificial Intelligence, IJCAI 2022, Vienna, Austria, 23-29 July 2022, pages 3001-3007. ijcai.org, 2022.
[18] Madelyn Glymour, Judea Pearl, and Nicholas P Jewell. Causal inference in stafistics: A primer. John Wiley & Sons, 2016.
[19] H. B Barlow. Possible principles underlying the transformations of sensory messages. In Sensory Communication. The MIT Press, 1961.
[20] Carl Doersch, Abhinav Gupta, and Alexei A. Efros. Unsupervised visual representation learning by context prediction. 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015. IEFE Computer Society, 2015.
[21] Pascal Vincent, Hugo Larochelle. Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In Machine Learning, Proceedings of the Twenty-Fifth International Conference (ICML 2008), Helsinki, Finland, June 5-9, 2008. ACM, 2008.
[22] Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
[23] Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, 2014.
[24] Jeff Donahue and Karen Simonyan. Large scale adversarial representation learning. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, 2019,
[25] Vincent Dumoulin, Ishmael Belghazi, Ben Poole, Alex Lamb, Martín Arjovsky, Olivier Mas- tropietro. and Aaron C. Courville. Adversarially learned inference. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26. 2017, Conference Track Proceedings. OpenReview.net, 2017.
[26] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
[27] Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
[28] Olivier J. Hénaff. Data-efficient image recognition with contrastive predictive coding, In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 4182-4192. PMLR, 2020.
[29] Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant repre- sentations. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020. Computer Vision Foundation / IEEE, 2020.
[30] Junnan Li, Pan Zhou, Caiming Xiong, and Steven C. H. Hoi. Prototypical contrastive learning of unsupervised representations. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
[31] Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E. Hinton. Big self-supervised models are strong semi- semi-supervised learners. In Advances in Neural Information Processing Systems 33: Annual Conference Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
[32] Vikas Verma, Thang Luong, Kenji Kawaguchi, Hieu Pham, and Quoc V. Le. Towards domain- agnostic contrastive learning. In Proceedings of the 38th International Conference on Machine Learning, ICML. 2021, 18-24 July 2021, Virtual Event, Proceedings of Machine Learning Research. PMLR, 2021.
12

---

[33] Tete Xiao, Xiaolong Wang, Alexei A. Efros, and Trevor Darrell. What should not be contrastive in contrastive learning. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
[34] Michael Gutmann and Aapo Hyvärinen. principle for unnormalized statistical models. In Yee Whye Teh and D. Mike Titterington, editors, Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, AISTATS 2010, Chia Laguna Resort, Sardinia, Italy, May 13-15, 2010, volume 9 of JMLR Proceedings, pages 297-304. JMLR.org, 2010.
estimation: A new estimation
[35] Goldberger, Gordon, and H. Greenspan. An efficient image similarity measure based on approximations of kl-divergence between two gaussian mixtures. In IEEE International Conference on Computer Vision, 2003.
[36] Xinlei Chen, Haoqi Fan, Ross B. Girshick, and Kaiming He. Improved baselines with momen- tum contrastive learning. CoRR, 2020.
[37] Charles Elkan and Keith Noto. Learning classifiers from only positive and unlabeled data. In Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Las Vegas, Nevada, USA, August 24-27, 2008. ACM, 2008
[38] Marthinus Christoffel du Plessis, Gang Niu, and Masashi Sugiyama. Analysis of learning from positive and unlabeled data. In Advances in Neural Information Processing Systems 27: Annual Conference Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, 2014.
[39] K. Sridharan and S. M. Kakade. An information theoretic framework for multi-view learning. Conference on Learning Theory, 2008
[40] Yao-Hung Hubert Tsai, Yue Wu, Ruslan Salakhutdinov, and Louis-Philippe Morency. Self- supervised learning from multi-view perspective. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
[41] Yazhe Li, Roman Pogodin, Danica J. Sutherland, and Arthur Gretton. Self-supervised learning with kernel dependence maximization. CoRR, 2021.
[42] Gretton A., Bousquet O., Smola A., and B. Schölkopf. Measuring statistical dependence with hilbert-schmidt norms. ALT, Springer, 2005.
[43] Yaodong Yu, Kwan Ho Ryan Chan, Chong You, Chaobing Song, and Yi Ma. Learning diverse and discriminative representations via the principle of maximal coding rate reduction. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
[44] Shikun Liu, Andrew J. Davison, and Edward Johns. Self-supervised generalisation with meta auxiliary learning. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' d'Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 1677-1687, 2019.
[45] Yifei Wang, Qi Zhang, Yisen Wang, Jiansheng Yang, and Zhouchen Lin. Chaos is a ladder: new theoretical understanding of contrastive learning via augmentation overlap. arXiv preprint arXiv:2203.33457, 2022.
[46] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[47] Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsuper- vised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, 2011.
[48] A. Krizhevsky, I. Sutskever, and G. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012.
[49] Jiangmeng Li, Wenwen Qiang, Changwen Zheng, Bing Su, and Hui Xiong. Metaug: Contrastive learning via meta feature augmentation. arXiv preprint arXiv:2203.055 159 2022.
[50] Zhirong Wu, Yuanjun Xiong, Stella X. Yu, and Dahua Lin. Unsupervised feature learning via non parametric instance discrimination. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 3733-3742. Computer Vision Foundation/ / IEEE Computer Society, 2018
13

---

[51] Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. In The IEEE International Conference on Computer Vision (ICCV), Oct 2017.
[52] R. Bellman and R. Karush. Dynamic programming: A bibliography of theory and application, Dynamic Programming A Bibliography of Theory and Application, 1964.
[53] Vincent Spruyt. The curse of dimensionality in classification. 2014.
[54] Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. With little help from my friends: Nearest-neighhor contrastive learning of visual representa- tions. In 2021 IEEE/CVF International Conference on Computer Vision, 2021, Montreal, QC, Canada, October 10-17, 2021, pages 9568-9577. IEEE, 2021.
[55] Hangbo Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint M.Xiv:2106.08255: 2021.
[56] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. arXiv preprint arXiv:2 ///////..677, 2021.
[57] Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: simple framework for masked image modeling. arXiv preprint arXiv:2111.09888 2021.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 7
(c) Did you discuss any potential negative societal impacts of your work? [No] Since this work presents a general technique tackle the dimensional confounder for self- supervised learning, we did not see any particular foreseeable ethical consequence.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section 5]
(b) Did you include complete proofs of all theoretical results? [Yes] See Appendix A. and Appendix A.2
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed reproduce the main perimental results (either in the supplemental material or as a URL)? [Yes] See the supplementary files.
(b) Did you specify all the training details (e.g., data splits, hyper- syper-paramettrs, how they were chosen)? [Yes] See Section 6 Appendix A.4 and the supplementary files.
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [No] The experimental results provided by benchmark methods do contain such error bars. We also follow this principle, and we reckon the reason behind such behavior is that the comparisons are conducted in balanced datasets, so the difference between results is not excessively large.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Section and Appendix A.44
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Section Section
Section 5 and Section 6
(b) Did you mention the license of the assets? [No] The assets we adopted are totally
established and public to the community.
(c) Did you include any new assets either the supplemental material or as URL? [Yes]
See the supplementary files.
14

---

(d) Did you discuss whether and how consent was obtained from people whose data you't 'e using/curating? [No] The datasets are all public to the community.
(c) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [No] The datasets have been discussed, validated, and used by many researchers.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
15


Reviews:
[{"review_id": "mhcrWMxos_y", "reviewer": "Reviewer_FKJ2", "summary": "This paper proposed a hybrid joint-embedding framework, which adds learnable masks on representations and applies both contrastive loss and redundancy reduction loss. This mask is updated when fixing the rest of the network. \nThe experiments show that the proposed method improves most of the recent frameworks (BarlowTwins, SimCLR, SwAV, BYOL, etc.) on various datasets (CIFAR-10, CIFAR-100,  STL-10, IN-200)", "questions": "1. What's the reason behind MetaMask's collapse for large embedding dimensions? This is not observed in standard BarlowTwins.\n\n2. Why is this considered \"meta\"? It's basically second-order optimization.\n", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. The authors point out that joint-embedding methods will learn harmful features. This is a widely ignored problem in the self-supervised learning method.\n2. Experiment results show consistent improvement of the proposed method overall frameworks on various datasets. \n3. The second-order optimization on the mask is supported by theoretical proof. \n\n\nWeakness\n1. The masks are trainable means that in the end, they are fixed and will ignore several features known as confounders. However, this needs to be verified via ablation experiments. For example, a linear probe on the whole unmasked features should give lower accuracy.  \n2. The experiments are not convincing due to a lack of fair comparison. The authors use a very customized setting, e.g., AlexNet on various joint-embedding frameworks. No hyperparameter tuning is conducted on these models. \n3. Second-order optimization creates significant computational overhead. The authors need to show how much more training time is needed for each model.\n4. No experiment details. \n\n\n====== post rebuttal comments =====\n\nI've carefully read the authors' responses. All my major concerns are addressed. Though the AlexNet setting on 1 GPU is not convincing enough, the fair amount of controlled experiments in the paper convinces me of the effectiveness of the proposed idea.\n\nI've increased my score.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "jsfIm5ch8xa", "reviewer": "Reviewer_m2DS", "summary": "This paper first investigates the problem of dimensional confounder in contrastive learning, which refers to a subset of dimensions learning only task-irrelevant background information. The author shows the existence of the dimension confounder, where the performance of Barlow Twin starts to drop and eventually collapses after the dimension of the projection head reaches a certain point. The author then proposes to learn a dimensional mask approach, MetaMask, consisting of learnable weights to reweight each dimension of the encoder output in contrastive learning, so that the dimensional confounder can be assigned lower weights. The authors show theoretically that the proposed MetaMask achieves tighter risk bounds (lower and upper) for downstream classification tasks, and empirically improves the performances on downstream tasks and robustness towards dimension sizes by applying MetaMask to the prior models. ", "questions": "Q1: Figure 1(a) is interesting -- the proposed MetaMask seems to have much higher contributions on most dimensions than SimCLR -- why is this and is this desirable?\n\nQ2: How the experiments for Figure 1(b) are a little unclear. How many layers were you using? Were you using the same dimension for all the layers of the projection head, or did you shrink the dimension at the last layer to serve as a dimension bottleneck? These are important because your results sharply contrast with Barlow Twins'. \n\nQ3: Could you provide the dimensional mask rate that improves the performance on both ImageNet and CIFAR-10? Although you claim the ImageNet rate is higher, the difference seems indistinguishable from Figure 2.\n \nQ4: The claim you made, that (Line 50) \"the dimensionality of the projection head ... acts as a dimensionality bottleneck\" is not the same as the claim from Barlow Twin, \"the output of the ResNet is kept fixed to 2048, which acts as a dimensionality bottleneck\". The bottleneck, according to Barlow Twin, seems to be the ResNet output dimension, rather than the projection head dimension. Could you clarify this?", "limitations": "The authors discussed the limitations of the work. The limitation states that the proposed MetaMask is specific towards improving contrastive learning but could not yet show the theoretical guarantees of MetaMask on other self-supervised learning methods, such as masked image modeling. This is a correct assertion and the discussions of limitations are helpful for the community.\n\nThe authors have not shed light on the negative societal impact.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality: \n\nStrengths: the idea is novel, addressing the issue of dimensional confounders by reweighting encoder outputs using meta-learning techniques.  The idea is related to MAXL[1], but the two approaches are different. MAXL applies a mask with weights on the softmax logits in a hierarchical binary prediction setup, where the logits are single scalars. On the other hand, this work applies a mask with weights on the encoder output for contrastive learning, where the output has a large dimension.\n\nThe reviewer is not familiar with meta-learning literature and only evaluates the originality from a contrastive learning perspective.\n\nQuality: \n\nStrengths: the idea overall has technical soundness, with theoretical justification on the proposed MetaMask creating tighter lower and upper bounds of risk, and shows reducing the risk can improve downstream performances. \n\nWeaknesses: The reviewers have some additional clarification questions listed in the next section.\n\nThe reviewer cannot verify the correctness of Equation 8. \n\nClarity: \n\nStrengths: the paper is very well written, with clear motivation, thorough related work, a succinct method section, and well-organized theoretical analysis and experimental results.\n\nSignificance: \n\nStrengths: the community can adopt the method to reduce the impact of the dimensional confounder, especially in large models. The performance improvements are significant in some cases (Table 2, SimCLR on STL-10 and NNCLR on CIFAR-10).\n\n[1] Liu, Shikun, Andrew Davison, and Edward Johns. \"Self-supervised generalisation with meta auxiliary learning.\" Advances in Neural Information Processing Systems 32 (2019).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Zf9nhuzfVu", "reviewer": "Reviewer_ZgDJ", "summary": "The authors propose a self-supervised learning scheme with a mask on top of the contrastive learning and the Barlow-twin method to perform dimensional reduction and confounder elimination. The concept is to use a meta-learning approach to seek a mask deleting redundant features while optimizing the feature representation and model parameters on the fly. The training loss is based on a combination of contrastive and Barlow-twin.", "questions": "I think the improvement is good, but I want to know that this meta-learning type is more essential than just optimizing a loss of Barlow-twin and contrastive loss. Also, the theoretical point that mask includes identity seemingly makes the theorem vacuous, and I wonder if the authors can respond to my concern.", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe proposed method seems easy to implement based on existing works and achieve good performance improvement. The method has been validated with theory.\n\nWeakness:\n\nThe ablation study is insufficient, as the method uses both Barlow-twin and contrastive loss. Still, it's unclear that this meta-learning alike optimization scheme is essential compared with training on a loss combination of Barlow-twin and contrastive loss. The comparisons have only been made against either one of them. Also, in Figure 1, the performance improvement by the random mask is not significant enough to persuade me that the feature redundancy is sufficiently essential (only within 0.1% improvement on ImageNet). Finally, why is the performance not tested on full ImageNet while the motivation figure is?\n\nTheoretically, the authors show that the conditional variance can be reduced with the optimal mask. Still, since the identity matrix (no mask) is also a special case of masks, this is a seemingly trivial statement.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "H41PzShwI9", "reviewer": "Reviewer_rGkj", "summary": "This paper handles the problems which existing self-supervised learning algorithms suffer. Authors demonstrate that existing self-supervised learning algorithms suffer dimensional redundancy and dimensional confounder. They propose MetaMask which learns a dimensional mask through meta learning to decrease the learning signals of features having confounders. For the rationales of MetaMask, they provide theoretical support where MetaMask has tighter risk bounds in downstream tasks compared to baselines. They demonstrate that MetaMask combined with competitive self-supervised algorithms has superior performance to baselines on multiple benchmark datasets under two architectures. ", "questions": "1. (ablation study) Barlow Twins handles dimensional redundancy but suffers dimensional confounder. MetaMask mitigates dimensional confounder by learning and applying a dimensional mask. There would be performance gain only from alleviating dimensional confounder. Can authors show the performance of MetaMask without redundancy-reduction objective function? (I understand that Barlow Twins could exacerbate dimensional confounder.)", "limitations": "It would be great if the authors suggest possible issues from the adoption of bi-level optimization such as computational costs.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper has several strong and weak points.\n\nStrengths:\n1. This paper shows that the existing self-supervised learning algorithms suffer dimensional redundancy and dimensional confounder. The problem is also quite intuitive.\n2. Theoretical support for the reasons why MetaMask works is provided. (I have not carefully checked the proofs)\n3. Authors conduct extensive experiments and demonstrate that MetaMask combined with existing methods improves performance in most cases.\n\nWeaknesses:\n1. MetaMask adopts bi-level optimization, so it would necessarily require significant amounts of additional computational cost during computing second-derivatives. However, authors do not provide any analysis of computational costs.\n2. Authors maintain that MetaMask reduces the gradient effect of the dimension containing the confounder (in lines 76-80), but they do not explicitly demonstrate that masked dimensions (having low weights in MetaMask) have confounder (non-discriminative features such as backgrounds). It would be great if the authors suggest experiments about it.\n3. The improvement of MetaMask seems marginal in important cases although MetaMask improves baseline performance by combining with them in most cases. Specifically, in Table 2, MetaMask combined with existing algorithms in the modern architecture (ResNet18) shows comparable performance to the most competitive baselines (such as BYOL in CIFAR-10, NNCLR in CIFAR-100, and NNCLR in IN-200).\n4. (simple baseline) Applying dropout on $h_i$ in Barlow Twins might be the simple baseline for MetaMask and it would be effective in that BarlowTwins with randomly masked dimensions outperform naive BarlowTwins in Figure 2. Would authors provide the performance of Barlow Twins + dropout?\n\nMiscellaneous minor issues:\n1. In Figure 1, (a) and (b) are missing in the figure (only mentioned in the caption). \n\n---\n\n[After reading authors' answers] I appreciate the detailed responses and authors address several of my concerns. Accordingly, I change my rating to weak accept.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept