Self-supervised Learning from a Multi-view Perspective
Authors: Yao-Hung Hubert Tsai
Yue Wu
Ruslan Salakhutdinov
Louis-Philippe Morency
Venue: ICLR 2021 Poster (2021)
Keywords: Self-supervised Learning
Unsupervised Learning
Multi-view Representation Learning

Abstract: As a subset of unsupervised representation learning, self-supervised representation learning adopts self-defined signals as supervision and uses the learned representation for downstream tasks, such as object detection and image captioning. Many proposed approaches for self-supervised learning follow naturally a multi-view perspective, where the input (e.g., original images) and the self-supervised signals (e.g., augmented images) can be seen as two redundant views of the data. Building from this multi-view perspective, this paper provides an information-theoretical framework to better understand the properties that encourage successful self-supervised learning. Specifically, we demonstrate that self-supervised learned representations can extract task-relevant information and discard task-irrelevant information. Our theoretical framework paves the way to a larger space of self-supervised learning objective design. In particular, we propose a composite objective that bridges the gap between prior contrastive and predictive learning objectives, and introduce an additional objective term to discard task-irrelevant information. To verify our analysis, we conduct controlled experiments to evaluate the impact of the composite objectives. We also explore our framework's empirical generalization beyond the multi-view perspective, where the cross-view redundancy may not be clearly observed.

Published as a conference paper at ICLR 2021
SELF-SUPERVISED LEARNING FROM A MULTI-VIEW PERSPECTIVE
Yao-Hung Hubert Tsai, Yue Wu, Ruslan Salakhutdinov, Louis-Philippe Morency Machine Learning Department, Carnegie Mellon University
ABSTRACT
As a subset of unsupervised representation learning, self-supervised representation learning adopts self-defined signals as supervision and uses the learned representa- tion for downstream tasks, such as object detection and image captioning. Many proposed approaches for self-supervised learning follow naturally multi-view per- spective, where the input (e.g., original images) and the self-supervised signals (e.g.. augmented images) can be seen as two redundant views of the data. Building from this multi-view perspective, this paper provides work to better understand the properties that encourage successful self-supervised learning. Specifically, we demonstrate that self supervised learned representations can extract task-relevant information and discard task-irrelevan information. Our theoretical framework paves the way larger space of self-supervised learning objective design. In particular, we propose a composite objective that bridges the gap between prior contrastive and predictive learning objectives, and introduce an additional objective term to discard lask-irrelevant information. To verify our analysis, we conduct controlled experiments to evaluate the impact of the compos- ite objectives. We also explore our framework's ' empirical generalization beyond the multi-view perspective, where the vrews-view redundancy may not be clearly observed.
anfformationntheorrttin frame-
INTRODUCTION
supervier learning (SSL) (Zhang al., 2016; Devlin al., 2018; Oord al., 2018; Tian et al., 2019) learns representations using proxy objective (i.e., SSL objective) between inputs and self-defined signals. Empirical evidence suggests that the learned representations can generalize well to wide range of downstream tasks, supervision during training. For example, SimCLR (Chen et al., 2020) defines a contrastive loss (i.e., SSL objective) between images with different augmentations (i.e., one the input and the other as the self-supervised signal). Then, one can take SimCLR as features extractor and adopt the features various computer vision applications, spanning image classification, object detection, instance segmentation, and pose estimation (He et al., 2019). Despite success in practice, only a few work (Arora al., 2019; Lee et al., 2020; Tosh et al., 2020) provide theoretical insights into the learning efficacy of SSL. Our work shares similar goal to explain the success of SSL, from the perspectives Information Theory (Cover& & Thomas, 2012) and multi-view representation'
when the SSL objective has not utilize any downstream
To understand (a subset of) SSL, we start by the following multi-view assumption. First, we regard the input and the self-supervised signals as two corresponding views of the data. Using our running example, in SimCLR (Chen et al., 2020), the augmented images (i.e., the input and the supe-supe signal) are an image with different views. Second, we adopt a common assumption in multi-view learning: either view alone is (approximately) sufficient for the downstream tasks (see Assumption 1 in prior work (Sridharan & Kakade, 2008)). The assumption suggests that the image augmentations (e.g., changing the style of an image) should not affect the labels of images, analogously, the self- supervised signal contains most (if not all) of the information that the input has about the downstream tasks. With this assumption, our first contribution is to formally show that the self-supervised learned
'The work (Lee et al., 2020; Tosh et al., 2020) are done concurrent and parallel, and part of their assumptions/ conclusions are similar to ours. We will claborate the differences more in the related work section. We discuss the limitations of the multi-view assumption Section 2.1.

---

Published as a conference paper at ICLR 2021
Gool: Discard task-imolevant Info
Dicard tnok-imelevant info gap XXX: 7)
min H(Zy1S)
X
5
optimal Zi
reletions with downstream task.? 7
mii KZx Fy(00
N14x 5) (S)
Redundancy JX:: (XX Gyub
Extract task-relovant info from X with potential loss Eyto
Goal: Extract task-relevant Info
Minimal and Sufficient Sett- Seff-supervizion
Figure High-level takeaways for our main results using information diagrams. (a) We present to learn minimal and sufficient self-supervision: minimize H(Zx|5 [S) for discarding task-irrelevant information and maximize x2x;; S) for extracting task-relevant information. (b) The resulting learned representation Zx* contains all task relevant information from the input with a potential loss cinfo and discards task-irreeevant information with fixed gap (X;SIT) (c) Our core assumption: the self-supervised signal is approximately redundant to the input for the task-relevant information.
representations can 1) extract all the task information (from the input) with potential loss; and 2) discard all the task-irrelevant information (from the input) with a fixed gap. Then, using classification task as example, we are able the quantify the smallest generalization error (Bayes error rate) given the discussed task-relevant and irreleantt information.
As the second contribution, 2018; Bachman et al., 2019; Chen et al., 2020; Tian et al., 2019) and predictive learning (Zhang et al., 2016; Vondrick et al., 2016; Tulyakov et al., 2018; Devlin al., 2018) approaches; and 2) paves the way to a larger space of composing SSL objectives to extract task-relevant and discard iaskeinrelevant information simultaneously. For instance, the combination between the contrastive and predictive learning approaches achieves better performance than contrastive- or predictive-alone objective and enjoys less over-fitting problem. We also present a new objective discard task-irrelevant information. The objective can be easily incorporated with prior self-supervised learning objectives.
analysis 1) connects prior arts for SSL contrastive (Oord et al.,
We conduct controlled experiments on visual (the first set) and visual-textual (the second set) self- supervised representation learning. The first set of experiments are performed when the multi-view assumption likely to hold. The goal is to compare different compositions of SSL objectives on extracting tesk-relevant and discarding task-irrelevant information. The second set of experiments are performed when the input and the self-supervised signal lie in very different modalities. Under this modsl-t setting, the task-relevant information may not mostly lie in the shared information be- tween the input and the self-supervised signal. The goal is to examine SSL objectives' generalization, where the multi-view assumption is likely to fail.
A MULTI-VIEY INFORMATION-THEORETUUE THEORETICAL FRAMEWORK
Notations. For the input, we denote its random variable as X, sample space as X, and outcome as x. We learn representation (Zx/ 2/ 2₂) from the input through deterministic mapping Fx:
Zx Fx (X). For the supe-supe signal, we denote its random variable/ sample space/ outcome as S/ S/ s. Two sample spaces can be different between the input and the self-supervised signal: X o S. The information required for downstream tasks referred to as "task-relevant information": T/ T/ t. Note that SSL has access to the task-relevant information. Lastly, we use I(A: B) to represent mutual information, [(A; (CC) to represent conditional mutual information, (AA) to represent the entropy, and (AAB) to represent conditional entropy for random variables A/BIC. We provide high-level takeaways for our main results in Figure 1. We defer all proofs to Supplementary.
2.1 MULTI-VIEW ASSUMPTION
In our paper, we regard the input (X) and the self-supervised signals (S) as two views of the data. Here, we provide a table showing different X/S in various SSL frameworks:
Promwwrk
BIRT (Destin al.,
Look (Anangeloic Zinzeman, 2017)
SimUL hChen ol.,
Colonzzatio (Zhing 2010)
Impuic (N)
Non-mtoked Words Manked Weeds
Image
Image
Innag: Lightnsse Image Cohar
ScHf-ggerviaal Sigrals (5)
Audio Sezam
Same Image with Augreenttiznn
We note that not all SSL frameworks realize the inputs and the self supervieee signals as corresponding views. For instance. Jigsaw puzzle (Noroozi & Favaro, 2016) considers (shuffled) image patches as the input and the positions of the patches as the self-supervised signals. Another example is Learning by Predicting Rotations (Gidaris et al., 2018), which considers image (rotating with specific
2

---

Published as a conference paper at ICLR 2021
angle) as the input and the rotation angle of the image as the self-supervised signal. We point out that the frameworks that regard X/S as two corresponding views (Chen et al.; 2020; He et al., 2019) have much better empirical downstream performance than the frameworks that do not (Noroozi & Favaro, 2016; Gidaris et al., 2018). Our paper hence focuses on the multi-view setting between X/S.
Next, we adopt the common assumption (i.e., multi-view assumption (Sridharan & Kakade, 2008; Xu et al., 2013)) in the multi-view, learning between the input and the self-supervised signal:
Assumption 1 (Multi-view, restating Assumption I in prior work (Sridharan & Kakade, 2008)). The self-supervised signal is approximately redundant to the input for the task-relevant information. In other words, there exist an Gifo > 0 such that T(): S Ginfo
Assumption states that, when Cinto is small, the task-relevant information lies mostly in the shared information between the input and the self-supervised signals. We argue this assumption is mild with the following example. For self-supervised visual contrastive learning (Hjelm et al., 2018; Chen al., 2020), the input and the self-supervised signal are the same image with different augmentations. Using image augmentations can be seen as changing the style of an image while not affecting the content. And we argue that the information required for downstream tasks should only be retained in the content but not the style. Next, we point out the failure cases of the assumption (or have large Cinto): the input and the self-supervised signal contain very different task-relevant information. For instance, drastic image augmentation (e.g., adding large noise) may change the content of the image (e.g., the noise completely occludes the objects). Another example is BERT (Devlin et al., 2018), with too much masking, downstream information may exist differently in the masked (i.e., the self-supervised signals) and the non-masked (i.c., the input) words. Analogously, too much masking makes the non- maskead words have insufficient context to predict the masked words.
2.2 LEARNING MINIMAL AND SUFFICIENT REPRESENTATIONS FOR SELF-SUPERVISION
We start by discussing the supervised representation learning. The Information Bottleneck (IB) method (Tishby et al., 2000; Achille & Soatto, 2018) generalizes minimal sufficient statistics the representations that are minimal (i.e., less complexity) and sufficient (i.e., better fidelity). To learn such representations for downstream supervision, we consider the following objectives:
Definition 1 (Minimal and Sufficient Representations for Downstream Supervision). Let 2 be the sufficient supervised representation and Z ZauPaii
be the minimal and sufficient representation:
Z X
arg max T) and ZauPmin arg min H (Zx (T) s.t. (Zx: is maximized. Zx
Zx
To reduce the complexity of the representation Zx, the prior methods (Tishby et al., 2000; Achille & Soatto, 2018) presented to minimize / [2x; X) while ours presents to minimize H ([x|TT) We provide a justification: minimizing H(Zx|T) reduces the randomness from T to Zx, and the randomness is regarded a form of incoppressibilityy (Calude, 2013). Hence, minimizing H (ZZ (T) leads to a more compressed representation (discarding redundant information)". Note that we do not constrain the downstream task T' as classification, regression, clustering.
Then, we present SSL objectives to learn sufficient (and minimal) representations for self-supervision: Definition 2 (Minimal and Sufficient Representations for Self-superrisiion). Let Z be the sufficient self-supervised representation and salliin be the minimal and sufficient representation:
7 arg max S) and Z arg min H (S) s.t. 7 [(xx; S) is maximized. Zx
2x
Definition 2 defines our self-supervised representation learning strategy. Now, we are ready to associate the supervised and self-supervised learned representations:
Theorem (Task-relevant information with a potential loss Emto)). The supervised learned represen- tations (i.e., z and Zaup="")) ) contain all the tesk-relevant information in the input (i.e., IT); The self-supervised learned representations (i.e., Z and Zaslmin) ) contain all the task-relevant information in the input with a potential loss Cinto- Formally.
x
I(X;T) = 1(2XP ,T) I(2 (T) )) 2 [(2x) T) 2 I(2 :T) 2 I(X:T) Cinfo-
We do not claim H (2x ( minimization is better than [(xx: X) minimization for reducing the complexity in the representations 7x- In Supplementary, we will show that ( (T) minimization and 7(xx; X)
minimization interchangeable under our framework's ' setting.
3

---

Published as a conference paper at ICLR 2021
(Fowwed Predictive Leaming AgiFal)) Er., Bog (1512,))
X
S
Extracting
x
Fy
Task-Relevant Info
S
Contmetive 115;35 22-5000
Aecomitrction
insere prodicton
contrpatie
fouters mapgonu
propection
loeed
Discarding Task-Irrelevant Info
S
[twwesed Predictie Leaming Ex., (lg 153)
RIZy
prediction
Freuand
S
Figure 2: Remarks contrastive and predictive learning objectives for self-supervised learning. Between the representation Zx and the self-supervised signal S. contrastive objective performs mutual information maximization and predictive objectives perform log conditional likelihood maximization. We show that the SSL objectives aim at extracting task-relevant and discarding iise-irrelevant information. Last, we summarize the computational blocks for practical deployments for these objectives.
When Cinto is small, Theorem 1 indicates that the self-supervise learned representations can extract almost as much task-relevant information as the supervised one. While when info non-trivial, the learned representations may not always lead to good downstream performance. This result has also been observed in prior work (Tschannen et al., 2019) and InfoMin (Tian et al., 2020), which claim the representations with maximal mutual information may not have the best performance.
Theorem (Task-irrelevant information with fixed compression gap ;ST)) The sufficient self-supervised representation (i.e. T( T)) contains more task-irrelevan information in the input than the sufficient and minimal self- sepeevsueee representation (i.e., I(Z (T). The latter contains an amount of the information, I(X; S) that cannot be discarded from the inpul. Formally, I(z*) x = S(T) + I(Zx X|S,T) 2 I(Z ((T) = ((X: 2 I( : = 0.
Theorem 2 indicates that a compression gap (i.e., ((T)) exists when we discard the task- irrelevant information from the input. To specific, / is the amount of the shared informa- tion between the input and the self-supervised signal excluding the task-relevant information. Hence, (X; S(T) would be large if the downstream tasks requires only a portion of the shared information.
2.3 CONNECTIONS WITH CONTRASTIVE AND PREDICTIVE LEARNING OBJECTIVES
Theorem and 2 state that our self-supervised learning strategies (i.e., min H(Zx(S) and max 7 T(Zx;5) S) defined in Definition 2) can extract task-relevant and discard task-irrelevant informa- tion. question emerges:
"What are the practical aspects of the presented self- supervisd learning strategies?"
To answer this question, we present 1) the connections with prior SSL. objectives, especially for contrastive (Oord et al., 2018; Bachman et al., 2019; Chen et al., 2020; Tian et al., 2019; Hjelm et al., 2018; He et al., 2019) and predictive (Zhang et al., 2016; Pathak et al., 2016; Vondrick al., 2016; Tulyakov et al., 2018; Peters et al., 2018; Devlin et al., 2018) learning objectives, showing that these objectives extracting task-relevant information; and 2) a new inverse predictive learning objective to discard task-irrelevant information. We illustrate important remarks in Figure 2.
Contrastive Learning (is extracting task-relevant information). Contrastive learning objec- tive (Oord et al., 2018) maximizes the dependenccy/contrsttvvee tion Zx and the self-supervised signal S, which suggests maximizing the the mutual information [(Zx;S) S). Theorem 1 suggests that maximizing (Zx; S) results in Zx containing (approximately) all the information required for the downstream tasks from the input X. To deploy the contrastive learning objective, suggest contrastive predictive coding (CPC) (Oord et al., 2018)., which is mutual information lower bound with low variance (Poole et al., 2019; Song & Ermon, 2019):
between the learned representa-
T.CL =
Bl (ten (Zg,77xx) Zx (X),
max ns = (8).
log IE (laa))
(1)
where : + Z is a deterministic mapping and G is project head that projects representation
in into a lower-dimensional vector. If the input and self-supervised signals share the same
4 *Other contrastive learning objectives can other mutual information lower bounds such as DV-bound or NWJ-bound (Belghazi et al., 2018) or its JS -ivivergeecce (Poole et al., 2019: Hjelm et al., 2018) variants. Among different objectives. Tschannen et al. (2019) have suggested that the objectives with large variance (e.g., DV-/NWI-hound (Belghazi et al., 2018)) may lead to worsen performance compared to the low variance counterparts (e.g., CPC (Oord et al., 2018) and JS-dix. (Poole al., 2019)).
4

---

Published as a conference paper at ICLR 2021
sample space, i.e., X' = S, learning (Chen et al., 2020)). The projection head, G. can be an identity, a linear, or a non linear mapping. Last, note that modeling equation 1 often requires large batch size (e.g., large in equation 1) to ensure a good downstream performance (He et al., 2019; Chen et al., 2020).
impose Fx = Fs (e.g., self-supervised visual representation
Forward Predictive Learning (is extracting task-relevant information). Forward predictive learning encourages the learned representation Zx to reconstruct the self-supervised signal S, which suggests maximizing the log conditional likelihood Ers xx [log P(S]Zx)) By the chain rule, S) = (S) - Zx) where (()) is irrelevant to Zx. Hence, maximizing / (2x;5) is equivalent to maximizing Zxx
EEps. 2x [log X()] which is the predictive learning objective. Together with Theorem if 2n can perfectly reconstruct for any (s, ²x) 2s, then Zx contains (approximately) all the information required for the downstream tasks from the input X. A common approach to avoid intractability in computing PSsE [log ((S) (xxxx is assuming variational distribution ( with e representing the parameters in (-1-))) Specifically, we present to maximize EPs., x [log (x) which a lower bound of Ep [log P(S)Zx) )}s (+(+I-) can be any distribution such as Gaussian or Laplacian and e can be a linear model, a kernel method, or neural network. Note that the choice of the reconstruction type of loss depends on the distribution type of (-(-),), and is not fixed. For instance, if we let Q (S) be Gaussian
N [())) o1) with aI as a diagonal matrix", the objective becomes:
LFp
maX (X)
E. Be,So-PP,zy
||+ R(=
(2
where R: 3 - is deterministic mapping reconstruct S from Z and we ignore the constants derived from the Gaussian distribution. Last, in most real-world applications, the self- supervuue signal S has much higher dimension (e.g.. a 224 224 x image) than the representation Zx (e.g.. a 64-dim. vector). Hence, modeling conditional generative model (+()) will be challenging.
Inverse Predictive Learning (is discarding task-irrelevant information). Inverse predictive learning encourages the self-supervised signal S to reconstruct the learned representation Zx, which suggests maximizing the log conditional likelihood Eps, xx [log P P(Z Given Theorem logether with - -H = EPs.zy [log P(Zx|5)) we know if s can perfectly reconstruct 2 for any (s, 2g) ~ Ps.zx under the constraint that I(2x; S) is maximized, then Zx discards the task-irrelevant information, excluding T(X; S) Similar the forward predictive learning, we use Ep3,2 (log Q (x [S)] as a lower bound of Ers [log P(Zx [S)]. In our deployment, we take the
advantage of the design in equation and let Q (Zx (S) be Gaussian N [xx (ss)),o)
E Ba
1| 20112
(3)
(X)
Note that optimizing equation alone results in a degenerated solution, e.g., learning Zx and Zg to be the same constant.
Composing SSL Objectives (to extract task-relevant and discard fask-irrelevant information simultancously). So far, we discussed how prior self-supervised learning approaches extract task- relevant information via the contrastive or the forward predictive learning objectives. Our analysis also inspires a new loss, the inverse predictive learning objective, to discard insk-inrelevant information. Now, We present a composite loss combine them together:
LSSL ACtLOL + APPLEP + AIPLIP,
(4)
where ACL. App, and dIp are hyper-parameters. This composite loss enables us to extract task- relevant and discard task-irrelevas information simultaneously.
EEry xx Dog P(8) )] max KPs xx [log Qa (812x)) + PKu. P (P(8)2x) 1 Qg((1(x)) 2 max Frs.yy Dog (8)
Qe
'The assumption identity covariance in the Gaussian only a particular parameterization the distri- bution Q(-l-). Other examples are MocoGAN (Tulyakov et al., 2018), which assumes Q is Laplacian (i.c., & reconstruction loss) and Φ is deconvolutional network (Long et al., 2015). Transformer-A XL (Dai et al., 2019) assumes Q is a categorical distribution (i.c., cross entropy loss) and ø is Transformer network (Vaswani al., 2017). Although Gaussian with diagonal covariance not the best assumption, it is perhaps the simplest one.
5

---

Published as a conference paper at ICLR 2021
2.4 THEORETICAL ANALYSIS BAYES ERROR RATE FOR DOWNSTREAM CLASSIFICATION
In last subsection, we see the practical aspects of our designed SSL strategies. Now, we provide an theoretical analysis on the representations generalization enror when T is categorical variable. We use Bayes error rate as an example, which stands for the irreducible error (smallest generalization error (Feder & Merhav, 1994)) when learning an arbitrary classifier from the representation infer the labels. specific, let P. be the Bayes error rate of arbitrary learned representations Zx and T as the estimation for T from our classifier, Pa == E 2 x2
max P( ([2,)]-
EET
To begin with, we present a general form of sample complexity with mutual information (I(Zx;5) estimation using empirical samples from distribution Pzx.8- Let 2x.3 denote the (uniformly
[Fo(z,, 8)] with so being the sampled) empirical distribution of Pzx.s and !! (Zx: Epgw 2x:8 estimated log density ratio (i.c., log p(s) pp)))
Proposition 1 (Mutual Information Neural Estimation, restating Theorem by Tsai al. (2020)). Let 0 < 8 < 1. There exists E N and a family of neural networks F == {fo B E e Rd) where compact, so that 30" O, with probability least 1 6 over the draw of [Zai: 1a p
7 8- (2x:5) 1 (Zx: = (V d1log(1/6)
This proposition shows that there exists a neural network 8", with high probability. I3 (Zx: S) can approximate I (5) with n samples at rate O(1/ /n). Under this network 0" and the same parameters d and 6, we ready to present our main results the Bayes enror rate. Formally, let [T] be T's cardinalitiy and Th (xx) = min (max fa, 0}, 1/ } as a thresholding function:
Theorem (Bayes Error Rates for Arbitrary Learned Representations). For an arbitrary learned representations Zx, Pe = (h(P with
(- (IIT) ((X:ST) + ((;:X) -1.") (Zx: + O(V d 1oB(1/0))) ))
Pa S 1-11eep
Given arbitrary learned representations (Zx), Theorem suggests the corresponding Bayes error rate (Pe) is small when: I) the estimated mutual information (I (4x; is large; 2) a larger number of samples are used for estimating the mutual information; and 3) the task-irrelevant information (the compression gap T(X; S)T) and the superftuous information I(Z; X|S, T), defined in Theorem 2) is small. The first and the second results supports the claim that maximizing 1(2x; S) may learn the representations that are beneficial to downstream tasks. The third result implics the learned representations may perform better on the downstream task when the compression gap is small. Additionally, Z is preferable than 7 since T [(Zedlunn X|S, T) = 0 and T T(Z X|S, T) 2 O.
Theorem (Bayes Error Rates for Self-supervised Learned Representations). Let Paup /pealmm be the Bayes error rate of the supervised or the self-supervised learned representations Z /7
/ 17 Then, pasl = TT((Peel) and paslmin Th( (h(Pellmn)) with
log (1 - P.uP) + log2 log (ITI)
< (Peal poelain) < -1-exp 1-loog2+ poup log [T| + Emaoo)
Given our self self-supervised learned representations (Z and 2 sool Theorem 4 suggests smaller upper bound of pesl (or pasloin) when the redundancy between the input and the self-supervised signal (Cinto, defined in Assumption 1) small. This result implies the self-supervised learned representations may perform better on the downstream task when the multi-view redundancy is small.
3 CONTROLLED EXPERIMENTS
This section aims at providing empirical supports for Theorems and 2 and comparing different SSL objectives. In particular, we present information inequalities in Theorems 1 and 2 regarding the amount of the task-relevant and the task-irrelevant information that will be extracted and discarded when learning self-supervised representations. Nonetheless, quantifying the information is notoriously hard and often leads to inaccurate quantifications in practice (McAllester & Stratos, 2020; Song &

---

Published as a conference paper at ICLR 2021
( Omnnggior (Contnastvve with Inverse Predictiv))
06) CIFAR10 (Contrastive with Inverse Predictve) (0g Ormiggit (Contrattvv & lnward Predictiel
$L.3
Teet Aocuray
Accuray
Test Accurany
x
4 6.05li
Sec only
(Cemmaationl
Iforwrrd Pretctive
09
1.
4000
0030
30000
soice
Age Lay
Applap
Ape in Lay. + Applag
Traiiing Eposh
Figure 3: Comparisons for different compositions of SSL objectives on Omniglot and CIFARIO.
Ermon, 2019). Not to mention the information we aim to quantify is the conditional information, which is believed to be even more challenging than quantifying the unconditional one (Póczos & Schneider, 2012). To address this concern, we instead study the generalization error of the self- supervised learned representations, theoretically (Bayes error rate discussed in Section 2.4) and empirically (test performance discussed in this section).
Another important aspect of the experimental design is examining equation 4, which can be viewed as a Lagrangian relaxation to learn representations that contain minimal and sufficient self-supervision (see Definition 2): weighted combination between I [Zx; S) and - (Zx (S). In particular, the contrastive loss Lez and the forward- prediictive loss LFP represent different realizations of modeling T(Zx;S) and the inverse-predine preeiccere loss Lpp represents a realization of modeling - -2xx (S).
We design two sets of experiments: The first one is when the input and self-supervised signals lie in the same modality (visual) and are likely to satisfy the multi-view redundancy assumption (Assumption 1). The second one is when the input and self-supervised signals lie in very different modalities (visual and textual), thus challenging the SSL objective's generalization ability.
Experiment 1- - Visual Representation Learning. We use Omniglot dataset (Lake et al., 2015)² 7 in this experiment. The training set contains images from 964 characters, and the test set contains 659 characters. There are characters overlap between the training and test set. Each character contains twenty examples drawn from twenty different people. We regard image as input (X) and generate self-supervised signal (5) by first sampling image from the same character as the input image and then applying translation/ rotation to it. Furthermore, we represent task-relevant information (7) by the labels of the image. Under this self-supervised signal construction, the exclusive information in X or S are drawing styles (i.e.. by different people) and image augmentations, and only their shared information contribute to T. To formally show the later, if representing the label for X/S, then X) and P(T]S) are Dirac. Hence, T IL S|X and T TT X|S, suggesting Assumption holds.
mapping let R(-) be symmetrical to (-), and G(-) be an identity mapping. On the test set, we fix the mapping and randomly select examples per character as the labeled examples. Then, we classify the rest of the examples using the 1-nearest neighbor classifier based on feature (i.e., Zx Fx (X)) cosine similarity. The random performance on this task stands at 659 & 0.15% One may refer to Supplementary for more details.
We train the feature mapping (-) with SSL objectives (see eq. equation 4), set Fs(-) = Fx(-).
b Results& & Discussions. In Figure 3. we evaluate the generalization ability on the test set for different SSL objectives. First, examine how the introduced inverse predictive learning objective LIp can help improve the performance along with the contrastive learning objective Lct. We present the results in Figure (a) and also provide experiments with SimCLR (Chen et al., 2020) on CIFARIO (Krizhevsky et al., 2009) in Figure (b), where App = 0 refers to the exact same setup as in SimCLR (which considers only LCL). We find that adding LiP in the objective can boost model performance, although being sensitive to the hyper-parameter parameter AIp. According to Theorem 2, the improved performance suggests a more compressed representation results in better performance for the downstream tasks. Second, we add the discussions with the forward predictive learning objective T. We present the results in Figure 3 (c). Comparing to L.P,, LCL, 1) reaches better test accuracy; 2) requires shorter training epochs to reach the best performance; and 3) suffers from overfitting with long-epoch training. Combining both of them (Ler + 0.005L brings their advantages together.
Experiment II- Visual-Textual Representation Learning. We provide experiments using MS COCO dataset (Lin et al., 2014) that contains 328k multill images with 2.5 million labeled
7 'More complex datasets such as CIFARIO (Krizhevsky et al., 2009) or ImageNet (Deng et al., 2009), to achieve similar performance, require a much larger training scale from contrastive to forward predictive objective. For example, on ImageNet. MoCo (He et al., 2019) uses 8 GPUs for its contrastive objective and ImageGPT (Chen et al.) uses 2048 TPUs for its forward predictive objective. We choose the Omniglot to ensure fair comparisons among different self-super learning objectives under reasonable computation constraint.
7

---

Published as a conference paper at ICLR 2021
(a) MS COCD (Using Luza, SSL abjective)
(b) Rew BERT Pre-trained ResNet (Contrastive with inverse Predictive)
Setting
Micro KOC AUC
Subset Acc.
Closs modaliy Self supervised Leasingg
Exm
ResNet
000007
Med PessOOcAu
Pre-caited BERT RoNee
0.5015
00003
Riee HETT Per the-tnnned RonNer
0.7MI 0.000
020081
00001
Prodisired REXTT + Pos-miieed RodNt
0.7005
0.0090
0,2155 - 0.0010
Nin Self upeerised Leuening
Only Proerinee ReeNas
0.6761 0.0015
0.1719=00000
0.0015
02
33
04
43
Figure 4: Comparisons for different settings on self-supervised visual-textual representation training. We report metrics on MS COCO validation set with mean and standard deviation from 5 random trials.
instances from 91 objects. Each image has 5 annotated captions describing the relationships between objects in the scenes. We regard image as input (X) and its textual descriptions as self-supervised signal (S). Since vision and text are two very different modalities, the multi-view redundancy may not be satisfied, which means Ginfo may be large in Assumption 1.
We adopt LCL Lip) as SSL objective. We use ResNet18 (He et al., 2016) image encoder for (-) (trained from scratch or fine-tuned on ImageNet (Deng et al., 2009) pre-trained weights), BERT- uncased (Devlin et al., 2018) text encoder for (-s()) (trained from scratch or BookCorpus (Zhu et al., 2015//iiipeddia pre-trained weights), and a linear layer for G(-). After performing self-supervised visual-textual representation learning, we consider the downstream multi-label classification over 91 categories. We evaluate learned visual representation (Zx) using downstream linear evaluation protocol (Oord al., 2018; Hénaff et al., 2019; Tian al., 2019; Hjelm et al., 2018; Bachman et al., 2019; Tschannen et al., 2019). Specifically, a linear classifier is trained from the self-supervised learned (fixed) representation to the labels on the training set. Commonly used metrics for multi-label classification are reported on MS COCO validation set: Micro ROC-AUC and Subset Accuracy. One may refer to Supplementary for more details on these metrics.
Results & Discussions. First, Figure 4 (a) suggests that the SSL strategy can still work when the input and self-supervised signals lie in different modalities. For example, pre-trained ResNet with BERT (either raw the pre-trained one) outperforms pre-trained ResNet alone. We also see that the self-supervised learned representations benefit more if the ResNet is pre-trained but not the BERT. This result is in accord with the fact that object recognition requires more understanding in vision, and hence the pre- pre-trained ResNet is preferrable than the pre-trained BERT. Next, Figure 4 (b) suggests that the supervisdd learned representations can be further improved by combining LCL and I,r, suggesting L,P may be a useful objective to discard task-irrelevant information.
Remarks on AIp and Lyp- As observed in the experimental results, AIp is a sensitive hyper- parameter to the performance. We provide an optimization perspective to address this concern. Note that one of the our goals is to examine the setting when learning the minimal and sufficient representations for self-supervision (see Definition 2): minimize H [Zxl S) under the constraint that T[Zx; S) is maximized. However, this constrained optimization is not feasible when considering gradients methods in neural networks. Hence, our approach can be seen as its Lagrangian Relaxation by weighted combination between Ler (or LpP, representing 5)) and Lyp (representing H[xx(S) with the A1p being the Lagrangian coefficient.
The optimal Arp can be obtained by solving the Lagrangian dual, which depends on the parametriza- tion of Lel (or LFP) and Lpp. Different parameterizations lead to different loss and gradient landscapes, and hence the optimal 'yp differs across experiments. This conclusion is verified by the results presented in Figure 3 (a) and (b) and Figure 4 (b). Lastly, we point out that even not solving the Lagrangian dual, an empirical observation across experiments is that App which leads to the best performance is when the scale of Lyp is one-tenth to the scale of Ler (or LFP).
4 RELATED WORK
Prior work by Arora et al. (2019) and the recent concurrent work (Lee et al., 2020; Tosh et al., 2020) are landmarks for theoretically understanding the success of SSL. In particular, Arora et al. (2019); Lee et al. (2020) showed decreased sample complexity for downstream supervised tasks when adopting contrastive learning objectives (Arora et al., 2019) or predicting the known information in the data (Lee et al., 2020). Tosh et al. (2020) showed that the linear functions of the learned representations are nearly optimal on downstream prediction tasks. By viewing the input and the self-supervised signal two corresponding views of the data, we discuss the differences among these works and ours. On the one hand, the work by Arora et al. (2019); Lee et al. (2020) assume strong independence between the views conditioning the downstream tasks, i.e. 5(T: 0.
8

---

Published as a conference paper at ICLR 2021
On the other hand, the work by Tosh et al. (2020) and ours assume strong independence between the downstream task and one view conditioning on the other view, i.e., [ z Prior work (Balcan et al., 2005; Du et al., 2010) have compared these two assumptions and pointed out the former one (T(X; S) s 0) is too strong and not likely to hold in practice. We note that all these related work and ours have shown that the self-supervised learning methods are learning extract lask-relevant information. Our work additionally presents to discard task-irrelevant information and quantifies the amount of information that cannot be discarded.
Our method also resembles the InfoMax principle (Linsker, 1988; Hjelm et al., 2018) and the Multi- view Information Bottleneck method (Federici al., 2020). The InfoMax principle aims at preserving the information of itself, while ours aims at extracting the information in the self supervised signal. the other hand, to reduce the redundant information across views, the Multi-view Information Bottleneck method proposed to minimize the conditional mutual information T(2x; X S)) while ours propose to minimize the conditional entropy H (Zx15)) The conditional entropy minimization problem can be easily optimized via our proposed inversed predictive learning objective.
Another related work is InfoMin (Tian et al., 2020), where both InfoMin and our method suggest to learn the representations that contain "not" too much information. In particular, InfoMin presents to augment the data (i.c., by constructing learnable data augmentations) such that the shared information between augmented variants is minimal possible, followed by the mutual information maxi- mization between the learned features from the augmented variants. Our method instead considers standard augmentations (e.g., rotations and translations), followed by learning representations that contain no more than the shared information between the augmented variants of the data.
On the empirical side, we explain why contrastive (Oord et al., 2018: Bachman al., 2019; Chen et al., 2020) and predictive learning (Zhang et al., 2016; Pathak et al., 2016; Vondrick et al., 2016; Chen al.) approaches can unsupervised extract task-relevant information. Different from these work, we present an objective to discard task-ing inseeirreeevan information and show its combination with existing contrastive predictive objectives benefits the performance.
5 CONCLUSION
This work studies both theoretical and empirical perspectives self-supervised learning. We show that the self-supervised learned representations could extract task resk-relevant information (with a potential loss) and discard iiseeirreeevant information (with fixed gap), along with their practical deployments such as contrastive and predictive learning objectives. We believe this work sheds light on the advantages of self-supervised learning and may help better understand when and why self-supervised learning is likely to work. In the future, we plan to connect our framework and recent SSL methods that cannot be easily fit into our analysis: c.g., BYL (Grill et al., 2020), SWAV (Caron et al., 2020), and Unifromality- Alignment (Wang & Isola, 2020).
ACKNOWLEDGEMENT
This work was supported in part by the NSF IIS176356,, NSF Awards #1750439 #1722822, Na- tional Institutes of Health, IARPA D17PC00340, ONR Grant N000141812866 and Facebook PhD Fellowship. We would also like to acknowledge NVIDIA's GPU support.
REFERENCES
Alessandro Achille and Stefano Soatto. Emergence of invariance and disentanglement in deep representations. The Journal of Machine Learning Research, 19(1):147-1980. 2018.
Relja Arandjelovic and Andrew Zisserman. Look, listen and leam. In Proceedings of the IEEE International Conference on Computer Vision, pp. 609 617, 2017.
Sanjeey Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saunshi. theoretical analysis of contrastive unsupervised representation learning, arXiv preprint arXiv:1902.09220 2019.
Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. In Advances Neural Information Processing Systems, pp. 1.500-15519, 2019.

---

Published as a conference paper at ICLR 2021
Maria-Florina Balcan, Avrim Blum, and Ke Yang. Co-training and expansion: Towards bridging theory and practice. In Advances neural information processing systems, pp. 89-96, 2005.
Peter L Bartltt. The sample complexity of pattern classification with neural networks: the size of the weights is more important than the size of the network. IEEE transactions on Information Theory, 44(2):225-536, 1998.
Mohamed Ishmacl Belghazi, Aristide Baratin, Sai Rajeswar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mine: mutual information neural estimation. arXiv preprint 101v::8001.00622 2018.
Cristian S Calude. Information and randomness: algorithmic perspective. Springer Science & Business Media, 2013.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. arXiv preprint arXiv:2006.09%8 2020.
Mark Chen, Alec Radford, Rewon Child, Jeff Wu, and Heewoo Jun. Generative pretraining from pixels.
Ting Chen, Simon Komblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint 2xXiv:2002.05709 2020.
Thomas M Cover and Joy A Thomas. Elements of information theory. John Wiley & Sons, 2012.
Zihang Dai, Zhilin Yang. Yiming Yang. Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. Transformer-xl:
Attentive language models beyond a fixed-length context. arXiv preprint arXiv: 1901.02800, 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Li, Li Fei-Fei. Imagenet: A large-scalt hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Icce, 2009.
Jacob Devlin, Ming-
Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training deep bidirectional transformers for language understanding. arXiv preprint arXiv:18/0.04805, 2018.
Jun Du, Charles X Ling, and Zhi-Hua Zhou. When does cotraining work in real data? IEEE Transactions on Knowledge and Data Engineering, 23(5):788-799 2010.
Tom Fawcett, An introduction roc analysis. Pattern recognition letters, 27(8):861-874, 2006.
Meir Feder and Neri Merhav. Relations between entropy and error probability. IEEE Transactions Information Theory, 40(1):259-266, 1994.
M Federici, Dutta. P Forré, Kushmann, and Z Akata. Learning robust representations via multi-view information bottleneck. International Conference on Learning Representation, 2020.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations, arXiv preprint arXiv:1803.07723 2018.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallee, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: new approach to self-supervised learning. arXiv prepriat arXiv:2006.07733 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recogmition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. arXiv preprint 1rXiv:1911. 1.0572,, 2019.
Olivier Hénaff, Ali Razavi, Carl Doersch, SM Eslami, and Aaron den Oord. Data-efficient image recognition with contrastive predictive coding. arXiv preprint arXiv:1905.0923 2019.
R Devon Hjelm, Alex Fedorov, Samuel Lavoie- Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint 1808 2018.
Kurt Homik, Maxwell Stincheombe, Halbert White, al. Multilayer feedforward networks are universal approximators.
Alex Krizhevsky et al. Learning multiple layers of fcatures from tiny images. 2009.
Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenhaum. Human-level concept learning through prohabilistic program induction. Science, 33006266) 2015.
10

---

Published as a conference paper at ICLR 2021
Jason D Lee. Qi Lei. Nikunj Saunshi, and Jiacheng Zhuo. Predicting what you already know helps: Provable self-supervised learning. arXiv preprint (XXi:2008.01066 2020.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona. Deva Ramanan, Piotr Dollár, and Lawrence Zitmick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Ralph Linsker. Self-organization in a perceptual network. Computer, 21(3):005-117 1988.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3431-3440 2015.
David McAllester and Karl Stratos. Formal limitations the mcasurement of mutual information. In Interna- tional Conference Artificial Intelligence and Statistics, pp. 875-884, 2020.
Sudipto Mukherjee, Himanshu Asnani, and Sreeram Kannan. Cemi: Classifier based conditional mutual information estimation. In Uncertainty in Artificial Intelligence, pp. 1083-1093. PMLR, 2020.
Mehdi Noroozi Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles, In European Conference on Computer Vision, pp. 69-84. Springer. 2016.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. 2018.
arXiv preprint
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning inpainting. In Proceedings of the IEEE conference on computer vision and puttern recognition, pp. 2536-2544 2016.
Matthew Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. arXiv preprint XXx::1802.05365, 2018.
Barnabás Póczos and Jeff Schneider. Nonparametric estimation of conditional information and divergences. In Artificial Intelligence and Statistics, 914-923. PMLR, 2012.
Ben Poole, Sherjil Ozair, Aaron van den Oord, Alexander A Alemi, and George Tucker. On variational bounds of mutual information. arXiv preprint arXiv: arXi::1055.0600 1905.0692,, 2019.
Shai Shalev-Shwartz and Shai Ben-David. Understanding machine learning: From theory to algorithms. Cambridge university press, 2014.
Jiaming Song and Stefano Ermon. Understanding the limitations of variational mutual information estimators. arXiv preprint orXiv:/910.066 2019.
Mohammad S Sorower. A literature survey on algorithms multi-label learning.
Karthik Sridharan and Sham Kakade. An information theoretic framework for multi-view learning. 2008
Yonglong Tian, Dilip Krishnan, and Phillip Isola. arXiv:1906.05849 2019.
Contrastive multiview coding.
arXiv preprint
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning. arXiv preprint arXiv:2005. 10243, 2020.
Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv preprint physies/0000057. 2000.
Christopher Tosh, Akshay Krishnamurthy, and Daniel Hsu. Contrastive learning, multi-view redundancy, and linear models. arXiv preprint arXiv:2008. 10150, 2020.
Yao-Hung Hubert Tsai, Han Zhao, Makoto Yamada, Phill-p Morency, Ruslan Salakhutdinov. Neural methods for point-wiss dependency estimation. arXiv preprint arXiv:2006.05553, 2020.
Michael Tschannen, Josip Djolonga, Paul K Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual information maximization for representation learning. arXiv preprint 1907v 2019.
Sergey Tulyakov, Ming Yu Liu, Xiaodong Yang. and Jan Kautz. Mocogan: Decomposing motion and content for video generation. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 1526-1535, 2018.
11

---

Published as a conference paper at ICLR 2021
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances neural information processing systems, pp. 5998-6008, 2017.
Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Generating videos with scene dynamics. In Advances in neural information processing systems, pp. 613 621, 2016.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. arXiv preprint arXiv:2005. 10242, 2020.
Chang Xu, Dacheng Tao, and Chao Xu. survey on multi-view learning. arXiv preprint arXiv:1304.5634, 2013.
Richard Zhang, Phillip Isola, and Alexei Efros. Colorful image colorization. In European conference on computer vision, pp. 649-666. Springer, 2016.
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference on computer vision, pp. 19-27, 2015.
A REMARKS ON LEARNING MINIMAL AND SUFFICIENT REPRESENTATIONS
In the main text, we discussed the objectives to learn minimal and sufficient representations (Definition 1). Here, we discuss the similarities and differences between the prior methods (Tishby et al., 2000; Achille & Soatto, 2018) and ours. First, to obtain sufficient representations (for the downstream task T), all the methods presented to maximize [Zx; T). Then, to maintain minimal amount of information in the representations, the prior methods (Tishby et al., 2000; Achille & Soatto, 2018) presented to minimize I [2x; X) and the ours presents to minimize (Zx (T). Our goal is to relate [[2x; X) minimization and H (Zx|[T) (T) minimization in framework.
To begin with, under the constraint (2x; is maximized, we see that minimizing I [(Zx; X) is equivalent minimizing T(Zx: X The reason is that T(Zx; X) = I(Zx; X(T) + T(Zx; X) where (Zx;;;;]) = I(Zx:T) T) due to the determinism from to Zx (our framework learns deterministic function from X to Zx) and [(Zx; T') is maximized in our constraint. Then, [[Zx;X[T) [T)x ([xx|X, T), where H (Z) contains no randomness (no infor- mation) as Zx being deterministic from X. Hence, I(Zx: minimization and H (Zxll (T) minimization are interchangeable.
The same claim can be made from the downstream task T the self-supervised signal S. In specific, when X to Zx is deterministic, / [(2x; XS)) minimization and H[ minimization are interchangeable. As discussed in the related work section, for reducing the amount of the redundant information, Federici et al. (2020) presented to use I(5) minimization and ours presented to use H HZx minimization. We also note that directly minimizing the conditional mutual information (i.e., [(Zx; X\S) requires a min-max optimization (Mukherjee et al., 2020), which may cause instability in practice. To overcome the issue, Federici et al. (2020) assumes a Gaussian encoder for X - Zx and presents upper bound of the original objective.
B PROOFS FOR THEOREM AND 2
We start by presenting a useful lemma from the fact that Fx(-) (:) is a deterministic function:
Lemma (Determinism). If P(Zx is Dirac, then the following conditional independence holds: T IL Zx|X |X and S 11 Zx [X, inducing a Markov chain S ++ T X Zx.
Proof. When Zx is a deterministic function of X. for any A in the sigma-algebra induced by Zx we have E[ [ K (T, S)] E I [| = [[ eAI which implies T 1L Zx X and S 11 Zx X.
Theorem and in the main text restated:
12

---

Published as a conference paper at ICLR 2021
Theorem 5 (Task-relevant information with potential loss Cufos restating Theorem 1 in the main text). The supervised learned representations (i.e., I xx : T) and I 1 ( contain all the rele-relevant information in the input (i.e., T();)). The self- supervised learned representations (i.e., I 12xx; ') and I 1xx ;T)) contain all the task-relevant information in the input with a potential loss Einfo- Formally,
I(X;T) 1x ;T) Pm ; I( T) I(Zgluin ,)) 2 I(X:T)
Einfo-
Proof. The proofs contain two parts. The first one is showing the results for the supervised learned representations and the second one is for the self-supervised learned representations.
Supervised Learned Representations: Adopting Data Processing Inequality (DPI by Cover & Thomas (2012)) in the Markov chain S ++ T + X + Zx (Lemma 1), T(Zx: T') is maximized at T(X; T)). Since both supervised learned representations (Z and x maximize (Zx;TT), we conclude I P ;T) = 1X T (X;
Sepf-supervised Learned Representations: First, we have
I(Zx; = I - [Zx; (I I - -(2x)
and
I - I(X;)) I(X;T:S) ((T: By DPI in the Markov chain S 7' + + Zx (Lemma 1), know
5) is maximized at ((X;5)
[[xx;S; S: is maximized at 5;T)
5(7) is maximized at 5(X:
Since both self-supervised learned representations (Z and Z maximize [(2x; we have
[(Zx): S) 1 (2 S) [(X (); Hence, ; S;T) 1 8:11) = 1(X; S; T') and
I(Z SIT) (Z STT I(;; Using the result / (Z S: [ (Z* S;T)
((X;5)7))
we get
[ TT = 10 - + + T|S)
and
I I(Zx min; ; T) (X; (X::7])) + 1(2x) min; T1S). Now, we are ready to present the inequalities:
1. 7(X;7) T(Z 7) due to T(X; T[$) 2 T( TIS) by DPI.
2. ;( 2 T(Z T) due to I T( T|$) 2 T( TIS) = 0. Since H(Zx|S) is
minimized at Z I(Z TIS) = 0.
3. (mn 2 I(X;T) info due to I(X:T) I(X:T|S) + I(Z TIS) -Xx:))I(X::]]5)) I(X:T)
Cinfos
where I(X:TI5) < Cinto by the redundancy assumption.
STT), restating Theorem Theorem 6 (Task-irrelevant information with fixed compression gap :
2 in the main text). The sufficient self-superviser representation (i.e., [(2x)) (T) contains more task- irrelevant information in the input than then the sufficient and minimal self-supervised representation (i.e., 1Z ; T)). The later contains amount of the information, 1(X; SYT). that cannot be discarded from the input. Formally,
1(2x1) (T) ((); : (Z min X I STT) > I(Zaupmin : ( = 0.
13

---

Published as a conference paper at ICLR 2021
Proof.
First, see that
= T(xx:
+ [[2x:
where ( = 8(Tx by DPI in the Markov chain S ++ T + X + Zx.
We conclude the proof by combining the following:
From the proof Theorem 5, we showed 1 5(7) = 1 1( (Z S(7) (())))
Since H [Zx (S) is minimized
Z Z 1(Zx (X),TT) : 0.
Since [2x (T) is minimized at X ZauPmin (SUPiii X(T) = 0.
1
C PROOF FOR PROPOSITION
Proposition 2 (Mutual Information Neural Estimation, restating Proposition in the main text). Ler 6 < There exists d E N and a family of neural networks == {So 0 E E S Rd) where ② compact, so that 30 E O with probability least 8 over the draw of F:lis
poi
M -
S O (V (ogg)(1//)
Sketch of Proof. The proof is a standard instance of uniform convergence bound. First, we assume the boundness and the Lipschitzness of fo. Then, we use the universal approximation lemma of neural networks (Hornik et al.). Last, combing all these two along with the uniform convergence in terms of the covering number (Bartlett, 1998), we complete the proof.
We note that the complete proof can be found in the prior work (Tsai et al., 2020). An alterna- tive but similar proof can be found in another prior work (Belghazi et al., 2018), which gives
us 7 (Zx:8) -(xx:::s)))))) S O dlog 1+1og(11/) The subtle difference between them is
that, given a neural network function space S Rd and its covering number N((,, 17), Tsai al. (2020) has N(O,1)) ((n)- by Bartlett (1998) and Belghazi al. (2018) has O(O,n) n) ((1//a))- by Shalev-Shwarzz & Ben-David (2014). Both are valid and the one used by Tsai et al. (2020) is tighter.
D PROOFS FOR THEOREM 3 AND 4
To begin with, that
(Zx;7) T(2x; - -
1(2x;
1[Zx;5) (a;;5))-(xx:::: [|)) X(Tx:
-T(Zx;5) (18)
2 5(2x;5)- - I(xx;
where T[x = I(Zx: = due to the wwteeeeeeeennniiii from to Zx. Then, in the proof Hence,
of Theorem have shown hhhomm//|2:::kp111133333333377:-
1(x::T) 2 ([ -1
2 5) -(x;S))) -(2x:x(1,,,
where ((xx;517) < ((X; by DPI.
Theorem and 4 in the main text restated:
14

---

Published as a conference paper at ICLR 2021
Theorem 7 (Bayes Error Rates for Arbitrary Learned Representations, restating Theorem 3 in the main text), For an arbitrary learned representations Zx. P. = Th(P.) with
Pe < 1 exp ([U0++/ASST--11EEE1:111111-11110
(+log(1//5)
Proof. We use the inequality between
Pa and Zx) indicated by Feder & Merhav (1994):
-log(1 [ - Pe) - H ZT)
Combining with [(x:T) ((T) H Zx) and 1(xx;T) Z 1(6x:5) (()
we have
log(1
Pe) 2 HTT) + I(Zx; - ((X; I(2x:X||ST))
Hence,
(1ITHH1X1111))--222133337 1(2x58))
Pe exp
Next, by definition of the Bayes error rate, we know 0 S Pe 5
We conclude the proof by combining Proposition 2, 7(m)
(Zx: (xx:))-((Zx;5))
o (1/6
Theorem 8 (Bayes Error Rates for Self-super Learned Representations, restating Theorem 4 in the main text). peup/Pet /peal/p be the Bayes error rate of the supervised or the self-supervised learned representations 2x 12xx1 /Zashmin Then, pesl (h((Ps)) and pashmnn Th(peslmin) with
log (1 PS-p) + log 2
< {Pael pealmin } S - exp 2logt [T]+t)))))
log (T))
Proof. We use the two inequalities between Pa and Zx) by Feder & Merhav (1994) and Cover & Thomas (2012):
-log(1 - Pa) < H(T 2x)
and
H I log 2 + P.log[T).
Combining the results from Theorem 5:
1 T)) 2 ((2)) (( (Z ;T) 2 1 (T) - Einfor
we have
the upper bound of the self-supervised learned representations' Bayes error rate:
{-log(1 ppsl). -log(1 psslanin }} S (H(T)ZX) H (T) 2 (elmm
}}
H (T + Ginfo
< log 2 + P-uPlog|T] + Cifoor
which suggests pesl, pesluni)) } < exp -log 2 log [T] (T]=)n))))
the lower bound of the self-supervised learned representations' Bayes error rate:
-log(1 (1 - paup) < ((T)
< (H H H (xT))))))))
{log 2 peal log T). s flog + peeluin log
which suggests
log (0-(1-pee prop-
peel peshmin }.
log (171)
We conclude the proof by having Pe lie in the feasible range: 0 < P. <
15

---

Published as a conference paper at ICLR 2021
E TIGHTER BOUNDS FOR THE BAYES ERROR RATES
We note that the bound used in Theorems and 8: -log(1 - Pe) E
(x) P.log T
not tight. A tighter bound is H (Pa) HT Zx) S H (P.) with
k 6-1
k 1
H (P.) = 11 k:1 PP) + k(1 Pe Pe)logk k when
< Pe 5
1
[ -1
k
and where (x) xlog(()) -1- 2)og x).
1 (P.) : H(P.) + Polog (IT) - 1),
It is clear that -log(1 (1 - Pe) - (P.)) and H (Pe) - log 2 + Polog((T1).
Hence, Theorem and can be improved follows:
Theorem (Tighter Bayes Error Rates for Arbitrary Learned Representations). For an arbitrary learned representations Zx, P. Th(P.) with P. S Peupper Peupper is derived from the program
log(L/0)
arg max H1 S H(T) - ( (Zx) - (X: + (2x:
P.
T2
Theorem 10 (Tighter Bayes Error Rates for Self-supervised Learned Representations). Let Peup/ peeel/pppppplaaaee be the Bayes error rate of the supervised or the self-supervised learned rep-
resentations Z (2x11x// Then, peal = -Th(Paa) and pesluin = Th (an)min) with
P sal
I (Peal prstmiu)
Pa Peupper
Flower
P Plower is derived from the following program
arg min H (P S H (peal)
Peal
and Pe Peupper is derived from the following program
arg max H (P:sl) S H (P:uP) + Cinfo-
posl
F MORE ON VISUAL REPRESENTATION LEARNING EXPERIMENTS
In the main text, design controlled experiments self-supervised visual representation learning to empirically support our theorem and examine different compositions of SSL. objectives. In this section, we will discuss 1) the architecture design; different deployments of contrastive/ forward predictive learning; and 3) different self-supervised signal construction strategy. We argue that these three additional set of experiments may be interesting future work.
F.1 ARCHITECTURE DESIGN
The input image has size 105 x 105. For image augmentations, we adopt 1) rotation with
degrees from -1 to +10%; 2) translation from -15 pixels to +15 pixels; 3) scaling both width and height from 0.85 to 1.0; 4) scaling width from 0.85 to 1.25 while fixing the height; and 5) resizing the image to 28 x 28. Then, a deep network takes a 28 x 28 image and outputs 1024-dim. feature vector. The deep network has the structure:
Conv - BN ReLU - Conv - BN ReLU MaxPool Conv BN ReLU- MaxPool - Conv
-BN ReLU MaxPool Flatten Linear - L2Norm.
Conv has 3x3 ker- nel size with 128 output channels, MaxPool has 2x2 kernel size, and Linear
is 1152 to 1024 weight matrix.
R(.) is symmetric to Fx which has
Linear - BN - ReLU - UnFlatten - DeConv - BN - ReLU - DeConv BN - ReLU - DeConv
-BN ReLU DeConv. R(-) has the exact same number of parameters as Fx Note that
use the same network designs in .(.,)) and (-I--) estimations. To reproduce the results in our experimental section, please refer to our released code".
'https:/ thub. www.tttt......ch.ooeeeeoo www..//owwaacttttteeeeee Supervi sed learn wew.ennnggoolttiiinineo Multviee
16

---

Published as a conference paper at ICLR 2021
one ger teee masie Copecrve)
10 JoCoing of pinweet Faraar Pedicos Copcc.u
3o Crrngee palsen Dorplatio
(Cunposing Cbiectives with
Ofgeris
Trave
Kew
530 gnobs
29:0 panks
Fet. Sip
231 spods
TEP NO
2000 spddl
5436-2 IVAR
Tee:
Les
Ler I.Gr à Ees
spodc
Figure Comparisons for different objectives/compositions SSL objectives on self-supervised visual representation training. We report mean and its standard error from 5 random trials.
F.2 DIFFERENT DEPLOYMENTS FOR CONTRASTIVE AND PREDICTIVE LEARNING OBJECTIVES
In the main text, for practical deployments, we suggest Contrastive Predictive Coding (CPC) Oord al. (2018) for Lezz and assume Gaussian distribution for the variational distributions in Lpp/ L.p- The practical deployments can be abundant by using different mutual information approximations for Lez and having different distribution assumptions for Lpp/ Lyp. In the following, we discuss a few examples.
Contrastive Learning. Other than CPC Oord et al. (2018), another popular contrastive learning objective is JS Bachman al. (2019), which is the lower bound of Jensen-Shannon divergence between P(Zs, Zx) and P(Zx) (a variational bound of mutual information). Its objective can be written
max E Bp(Zs,Zx) (5)
prrz (softplus (or G(2)),
where we use softplus to denote softplus (1) = log (1 + exp (x)).
Predictive Learning. Gaussian distribution may be the simplest distribution form that we can imagine, which leads to Mean Square Error (MSE) reconstruction loss. Here, we use forward predictive learning example, and we discuss the case when S lies in discrete {0, 1) sample space. Specifically, we let (o(5) be factorized multivariate Bernoulli:
max E Es 2x=Fx(X),F
Σ
log [R(zz)]s + (1 st) log [1 A(Zx)k
(5)
This objective leads to Binary Cross Entropy (BCE) reconstruction loss.
If assume each reconstruction loss corresponds to particular distribution form, then by ignoring which variatioinal distribution we choose, we are free to choose arbitrary reconstruction loss. For instance, by switching and eq. equation 5, the objective can be regarded as Reverse Binary Cross Entropy Loss (RevBCE) reconstruction loss. In our experiments, we find RevBCE works the best among (MSE, BCE, and RevBCE). Therefore, in the main text, we choose RevBCE as the example reconstruction loss as Lpp.
More Experiments. We provide an additional set of experiments by having (CPC, JS) for Le and [MSE, BCE, RevBCE] reconstruction loss for Lpp in Figure 5. From the results, we find different formulation of objectives bring very different test generalization performance. We argue that, given particular task, it is challenging but important to find the best deployments for contrastive and predictive learning objectives.
F.3 DIFFERENT SELF-SUPERVISED SIGNAL CONSTRUCTION STRATEGY
In the main text, we design a self self-supervised signal construction strategy that the input (X) and the self-supervised signal (S) differ in [drawing styles, image augmentations). This self-supervised signal construction strategy different from one that is commonly adopted most self-supe visual representation learning work Tian et al. (2019); Bachman et al. (2019); Chen et al. (2020). Specifically, prior work consider the difference between input and the self-supervised signal only in image augmentations. We provide additional experiments in Fig. 6 to compare these two different self-supervised signal construction strategies.
We that, comparing to the common self-supervised signal construction strategy Tian al. (2019); Bachman et al. (2019); Chen et al. (2020), the strategy introduced our controlled experiments has much better generalization ability to test set. It is worth noting that, although construction
17

---

Published as a conference paper at ICLR 2021
Omniglot (Different Self-supervised Signal Construction)
80
Test Accuracy
70
60
50
40
Strategy ours (controlled exps)
Strategy II similar to SimCLR
100 200 300 400 500 600 700 800 900 1000 Training Epoch
Figure 6: Comparisons for different self-supervised signal construction strategies. The differences between the input and the self-supervised signals |drawing styles, image augmentations] for construction strategy and only (image augmentations) for SimCLR Chen (2020)'s strategy. We choose Lel as our objective, reporting mean and its standard errоr from 5 random trials.
strategy has access to the label information (i.e., we sample the self-supervised signal image from the same character with the input image), our SSL objectives do train with the labels. Nonetheless, since we implicitly utilize the label information in our self-supe construction strategy, it will be unfair to directly compare our strategy and prior one. An interesting future research direction is examining different self-supervised signal construction strategy and even combine full/part of label information into self-supervised learning.
G METRICS IN VISUAL-TEXTUAL REPRESENTATION LEARNING
Subset Aceuracy (A) Sorower, also know the Exact Match Ratio (MR), ignores all partially correct (consider them incorrect) outputs and extend accuracy from the single label case the multi-label setting.
MR
1m=m m]]
Mícro AUC ROC score Fawcett (2006) computes the AUC (Area under the curve) of receiver operating characteristic (ROC) curve.
18


Reviews:
[{"review_id": "z3cRCEyQc8e", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper explores both theoretical and empirical perspectives on self-supervised learning. The authors attempt to prove that self-supervised learning could extract task-relevant information while discard task-irrelevant information. They also provide a composite objective that includes contrastive and predictive loss functions along with an additional regularization. Controlled experiments are conducted to support the presented theorems. \n\nThere are three major concerns for me.\n\n1)\tThe experiments are conducted in a controlled way, including visual representation learning and visual-textual representation learning. Traditional uncontrolled experiments, such as unsupervised learning on Cifar10 or ImageNet are suggested. \n\n2)\tThe results in Fig. 3 and Fig. 5 demonstrate that the performance is sensitive to the hyper-parameter of $L_{IP}$. How to set the hyper-parameter $\\lambda_{IP}$  in practice? Besides, the best performance achieved by $L_{IP}$ is only marginally better to those without it.\n\n3)\tSome previous related works, such as InfoMin [Tian et al., 2020], are suggested to discussed and compared with more details. As far as I know, InfoMin analyzed influence of different view choices using mutual information theory. Therefore, it would be helpful to compare the similarities and differences between this work with it. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "KB0SWOiO6A", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n\tAnalyze the self-supervised learning methods from extracting task-relevant information and discarding task-irrelevant information from the input theoretically and clearly.\n\tPropose a new objective to discard task-irrelevant information explicitly. And it seems easy to combine with other SOTA self-supervised methods.\n\tConduct experiments on visual setting and multi-modal setting.\n\nWeakness:\n\n\tI like the idea of discarding the redundant task-irrelevant information to improve the self-supervised learning. However, the composite objective proposed in this paper seems just like a simple combination of three tasks, which is not strikingly novel. \n\tanother concern of this paper is the lack of persuasive experiment results to prove the effectiveness of the proposed method. In fig.3, the improvements on two dataset are marginally, which can not convince me. \n\tThe \\lambda (λ_IP) in proposed objective function seems not robust to different datasets, which makes me doubt about the generalization of this method. I hope the authors provide more explanation about this.", "extra_scores": {}, "extra_text": {}}, {"review_id": "EIKDfLpMnsh", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this work the authors focus on self-supervised representation learning. This work proposes a composite loss function which includes contrastive loss, predictive loss, and a novel inverse predictive loss. The authors have provided theoretical analysis as well as experimental results on multiple datasets under control environment.\n\nPros:\n\nThis work presents a very detailed theoretical analysis for self-supervised learning objectives. \n\nThe idea of inverse predictive learning for filtering task irrelevant information is interesting.\n\nCons:\n\nThe inverse predictive learning appears to be similar to contrastive learning if we ignore the negative samples in the contrastive learning formulation and use the projection head from contrastive learning for inverse predictive learning. How will the authors differentiate between these two formulations?\n\nThe variation in the performance shown in Figure 3 is very marginal. Also, only selective results are shown in Figure 3. No results are shown on CIFAR-10 which uses all the three losses at the same time.  Figure 5 a shows some results on Omniglot, but the improvement shown there is very marginal. Also, how was the loss weights determined? Experiment on a large scale dataset, maybe ImageNet will be very useful to demonstrate the effectiveness of the proposed loss.\n\nThe weights required for inverse predictive learning in the loss formulation is not trivial. As shown in Figure 3, it varies with different datasets and varies a lot (1 for Omniglot and 0.1 for CIFAR10). Is there a simple way to determine this weights without exhaustive search on target dataset?\n\nThis work mainly focuses on the theoretical aspect of semi-supervised learning and have also presented an interesting inverse predictive learning task for self-supervision which can dis-regard task irrelevant information. However, it is not clear from the experimental results if this is really effective. The results shown in Figure 3 and Figure 4 are not sufficient to demonstrate its effectiveness. Also, quantitative comparison with existing approaches will also strengthen this work, which is only partially addressed. A more in-depth experimental analysis (apart from the theoretical analysis) is required to show its usefulness.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "6basoTDBQss", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: \nAuthors give an information-theoretical abstraction to represent various self-supervised learning methods and understand them better. Specifically, authors claim that self-supervised learning methods can extract task-relevant information and discard task-irrelevant information. Authors also do some controlled experiments and try to support their theoretical analysis. \n\nRecommendation: I recommend to accept the paper (rating 7). I liked the abstraction proposed by authors and particularly liked the way authors set up the Definition 1 and analysis afterwards. Ratings can be improved further if authors can relate experimental setup more to the theory which I find slightly disconnected.  \n\n1) In assumption 1: I(X;T|S) <= \\epslion. If task-relevant information lies mostly in the shared information between the input and the self-supervised signals then I(S;T|X) <= \\epslion, should also be true, right?\n2) Can authors give an intuitive proof sketch for Theorem 1?\n3) In Proposition 1 and Theorem 3, it looks like d can be arbitrarily large and even scale with n. Is there any upper bound in d? Would this invalidate all claims?\n4) In Figure 3 a and 3b, waht happens when you don’t use L_{CL} but use L_{FP} instead?\n5) Figure 3 c is not clear. Why L_{CL} overfits and why does adding L_{FP} avoid that over-fitting? Why did you use \\lambda_{FP} = 0.005?\n6) From figure 3, It looks like L_{CL} is good enough if early stopping is used properly. How does one justify the use of FP and IP? One can also use some regularization along with CL (would the regularization act as “discarding  task-irrelevant  information”)?\n7) How were hyper-parameters (learning rate, batch size, etc) chosen for all the experiments? Was proper hyper-parameter tuning done for all the methods? Can one achieve the same best results as in Figure 3a and 3b with proper tuning of method when \\lambda_{IP} is zero?\n8) It is not very convincing that controlled experiments in section 3 actually support Theorem 1 and 2. “Theorem 1 indicates that the self-supervised learned representations can extract almost as much task-relevant information as the supervised on” and “Theorem 2 indicates that a compression gap (i.e.,I(X;S|T)) exists when we discard the task-irrelevant information from the input”. Can authors clearly point out how conclusions from Figure 3 and Figure 4 are exactly supporting the respective statements of theorem 1 and 2 respectively?\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper received borderline reviews, but all lean toward acceptance.

The reviews highlighted strengths in the paper, citing that they liked the main idea and its mathematical treatment:
* R3: "I liked the abstraction proposed by authors and particularly liked the way authors set up the Definition 1 and analysis afterwards"
* R3 post-discussion: "I recommend accept because authors have a solid theory which would be useful for the self-supervised learning community."
* R4: "This work presents a very detailed theoretical analysis for self-supervised learning objectives. The idea of inverse predictive learning for filtering task irrelevant information is interesting."
* R2: "I like the idea of discarding the redundant task-irrelevant information to improve the self-supervised learning"

However, there was a consensus among reviewers that the experimental validation was weak, both in terms of not showing enough improvement on enough examples and in terms of studying the effect of certain hyperparameters:
* R2: "lack of persuasive experiment results to prove the effectiveness of the proposed method. In fig.3, the improvements on two dataset are marginally, which can not convince me. The \lambda (λ_IP) in proposed objective function seems not robust to different datasets, which makes me doubt about the generalization of this method."
* R3: "Ratings can be improved further if authors can relate experimental setup more to the theory which I find slightly disconnected"
* R3 post-discussion: "All reviewers have concerns about lack of solid experimental evidence [...]  I can not improve my score further because of weak experimental evidence."
* R1: "The experiments are conducted in a controlled way [...] Traditional uncontrolled experiments [...] are suggested."
* R4: "The variation in the performance shown in Figure 3 is very marginal. [...] Figure 5 a shows some results on Omniglot, but the improvement shown there is very marginal. [...]"
* R4: "weights required for inverse predictive learning in the loss formulation is not trivial. [...] Is there a simple way to determine this weights without exhaustive search on target dataset?"
* R4: "However, it is not clear from the experimental results if this is really effective."

The authors' revisions aim to improve the discussion of the $\lambda_\text{IP}$ parameter.

Given these experimental limitations, my recommendation is for acceptance but with a low confidence score.
Decision: Accept (Poster)