Self-Supervised Learning via Maximum Entropy Coding
Authors: Xin Liu
Zhongdao Wang
Ya-Li Li
Shengjin Wang
Venue: NeurIPS 2022 Accept (2022)

Abstract: A mainstream type of current self-supervised learning methods pursues a general-purpose representation that can be well transferred to downstream tasks, typically by optimizing on a given pretext task such as instance discrimination. In this work, we argue that existing pretext tasks inevitably introduce biases into the learned representation, which in turn leads to biased transfer performance on various downstream tasks. To cope with this issue, we propose Maximum Entropy Coding (MEC), a more principled objective that explicitly optimizes on the structure of the representation, so that the learned representation is less biased and thus generalizes better to unseen downstream tasks. Inspired by the principle of maximum entropy in information theory, we hypothesize that a generalizable representation should be the one that admits the maximum entropy among all plausible representations. To make the objective end-to-end trainable, we propose to leverage the minimal coding length in lossy data coding as a computationally tractable surrogate for the entropy, and further derive a scalable reformulation of the objective that allows fast computation. Extensive experiments demonstrate that MEC learns a more generalizable representation than previous methods based on specific pretext tasks. It achieves state-of-the-art performance consistently on various downstream tasks, including not only ImageNet linear probe, but also semi-supervised classification, object detection, instance segmentation, and object tracking. Interestingly, we show that existing batch-wise and feature-wise self-supervised objectives could be seen equivalent to low-order approximations of MEC. Code and pre-trained models are available at https://github.com/xinliu20/MEC.

Self-Supervised Learning via Maximum Entropy Coding
Xin Liu
Zhongdao Wang Yali Li
Shengjin Wang"
Beijing National Research Center for Information Science and Technology (BNRist) Department of Electronic Engineering, Tsinghua University {xinliu20, mcd17)0mails tsinghua. edu. cn fliyali13, ggsgg}&tsinghua edu. cn
Abstract
A mainstream type of current self-supervvsed learning methods pursues general- purpose representation that can well transferred to downstream tasks, typically by optimizing on given pretext task such instance discrimination. In this work, we argue that existing pretext tasks inevitably introduce biases into the learned representation, which in leads biased transfer performance on various downstream tasks. To cope with this issue, we propose Maximum Entropy Cod- ing (MEC), a more principled objective that explicitly optimizes on the structure of the representation, so that the learned representation is less biased and thus gen- eralizes better to unseen downstream tasks. Inspired by the principle of maximum entropy in information theory, we hypothesize that a generalizable representation should be the one that admits the maximum entropy among all plausible represen- tations. To make the objective end-to-end trainable, we propose to leverage the minimal coding length lossy data coding as a computationally tractable surrogate for the entropy, and further derive scalable reformulation of the objective that allows fast computation. Extensive experiments demonstrate that MEC learns a more generalizable representation than previous methods based on specific pretext tasks. It achieves tasks, including not only ImageNet linear probe, but also semi-supervised classifi- cation, object detection, instance segmentation, and object tracking. Interestingly, show that existing batch-wise and feature-wis self-supervised objectives could be seen equivalent to low-order approximations MEC, Code and pre-trained models are available https: : //t///gittt thub. com/xinl u220/MEC.
performance consistently on various downstream
Introduction
Self-supervised learning (SSL) aims to learn rich and meaningful representations without relying on human annotations. Pursuing purrpoe representations, SSL. models are typically used as pre-trained weights for providing good initialization downstream tasks. In this sense, SSL [12, 31, 10, 29, 15, 14, 86, 30] has seen great progress in computer vision, and can achieve competitive or even better performance on various downstream tasks compared to its supervised counterparts.
At the core of current SSL methods is the design of pretext tasks. A pretext task is a (usually candtcr learning objective in which the supervision signals could be mined from the data itself, with the aim of applying the learned representation to other downstream tasks. Early attempts of pretext tasks typically discard certain property of the input data and then force the model to predict the discarded property. For instance, one can convert an RGB image gray-scale one, and train the model to predict the original color [87, 88]; or apply random rotation to image patches and ask the
"ooresponning author
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

SmcCR
MoCo-v2
BYYOL
SwiVy
Sim Saam
Barlow Twirss
MBC
InDD
SimCLR
MoCo-2
InfoMin
BYOI
SWAV
MEC
Figure I: Comparison of transfer learning performance five image-based tasks (top row) and five video-based tasks (bottom row). Along each axis we plot the performance ranking of the represen- tation specific downstream task, a polygon with larger area means better generalization capacity on various downstream tasks, across the board.
model to predict the rotation angles [27]. In both cases, to fulfill the pretext task, the model must learn meaningful representation that can describe object texture, shape even category, therefore the learned representation transfers well to downstream tasks related to these features.
Despite successes of existing pretext tasks, find that they inevitably introduce biases into the learned representation, which conflicts with the original aim of "general-purpose" For example, representations trained with the most prevalent image-level task, instance discrimination [20, 80], are found biased to image-level tasks such as image classification, while by contrast degenerate in patch- or pixel-level tasks like object detection and semantic segmentation [81]. Even being transferred to level tasks, such representations still suffer from domain gaps in cases of different data distributions [26], e.g., classification unseen categories other than ImageNet objects.
In this work, we are curious of what makes for generalizable representations, and pursue an explicit optimization with criterion that directly measures the structure of representations, with the aim of minimizing the biases brought by the pretext task. To this end, we propose Maximum Entropy Coding (MEC). Inspired by the principle of maximum entropy in information theory, the basic hypothesis in MEC is that a generalizable representation should be the one that admit the maximum entropy among all plausible representations. Accordingly, optimizing towards maximum entropy leads to representations with good generalization capacity. The main challenge confronting us is that it is difficult, and computationally expensive, if possible, to estimate the distribution of given representation (usually a finite set of high-dimensional vectors), so in turn is difficult to estimate the entropy. To cope with this issue, replace the optimization objective from the originally defined entropy to a computationally tractable surrogate, i.e., the necessary number of bits needed to encode the representations via lossy data coding [16]. The log-determinant term costs the most computation in the coding length function. By leveraging Taylor expansion of matrix, we further approximate this term with polynomial functions, leading significant speed up and thus makes large-scale pre-training possible.
In contrast to previous SSL methods that mainly evaluate ImageNet classification, experiment on a wide variety of vision tasks to show the good generalization capacity of MEC. The considered tasks span across not only image-level recognition various data distributions, but also patch- or pixel-level tasks like object detection, instance segmentation, and object tracking. We show MEC generalizes well consistently across all tasks considered, while representations learned with previous pretext tasks usually perform well on closely related downstream tasks but degenerate on less related ones (see Figure l). Besides empirical results, we find interesting equivalence between low-order approximations of MEC and existing batch-wise (e.g., SimSiam [14]) or feature-wise objectives (e. g., Barlow Twins [86]), which provides new perspective for unified understanding of prevalent SSL methods. In summary, the main contributions of our work are as follows:
To learn representations generalizable to various downstream tasks, we introduce the principle of maximum entropy into self-supervised learning.

---

Maximum Entropy Coding
Batch-wise Objectives
21
Pal Tr(pAZ
21
21
Cuse
22
logdde Zs)
T
encoder f
plogdeth, x2) 2.)
logdee//L AZ Zi!
21
J/2 22
22
Feature-wisc Objectives
Figure 2: Illustration of our MEC and its relation to batch-wit and feature-wiss objectives.
We propose Maximum Entropy Coding (MEC), which explicitly optimizes on representations based on the principle of maximum entropy, and leverages the minimal coding length in lossy data coding as a computationally tractable surrogate for the entropy.
We reformulate the log-deterninant term in coding length function into a scalable form, which makes large-scale pre-training possible, and unifies existing batch-wise and feature-wise objec- tives as low-order approximations of our method.
We show MEC representation generalizes well on wide variety of image- and video-based downstream tasks, achieving state-of-the-an on most tasks considered.
Method
In this section, we start with illustrating the maximum entropy principle, and then introduce computationally tractable surrogate of the information entropy for high-dimensional vectors. We then present a scalable reformulation of the proposed surrogate, which makes large-scale training possible. And we further incorporate the view consistency prior for maximum entropy coding. Finally, we demonstrate how the proposed method can unify existing batch-wise and feature-wise SSL objectives. Please refer to Appendix E for more details about the proofs in this section.
2.1 Maximum Entropy Coding
The maximum entropy principle. The main purpose of this work is to improve the generalization capacity of self-supervised learning representations across unseen downstream tasks and data dis- tributions, and reduce the biases brought by specifically designed pretext tasks as much as possible. This naturally raises a question, i..,, what makes for a generalizable representation? To answer this question, are particularly inspired by the maximum entropy principle in information theory, which states that the probability distribution that best represents the current state of knowledge about a system is the one with largest entropy, given testable information (such accuracy) and in this way no additional bias or assumptions is introduced [38, 39, 59]. We therefore hypothesis that a general- izable representation is the that has the maximum entropy among all plausible representations. Intuitively, we are able to express the entropy in a closed form, the maximum entropy principle then can serve optimization objective and supervise the representation learning.
Minimal coding length as a surrogate for entropy. Entropy originally defined on probability distributions [63], i.e., H=)) & p(=) (ogg((z)dd) for continuous random variables. However, it is very difficult to estimate the true distributions p(z) of representation [6, 54], from a finite
set of high dimensional vectors Z | 2² 2 E Rdxmm A handy fact is that entropy is conceptually equivalent to the minimal number of bits required to encode the data losslessly, so the minimal lossless coding length could be used to represent the entropy. However, lossless coding of continuous random variables infeasible in our case since it often requires an infinite number of bits, breaking the numerical stability. Instead, we exploit the coding length in lossy data coding [ [16] as computationally tractable surrogate for the entropy of continuous random variables. Given set of samples Z, the minimal number of bits needed to encode Z subject to distortion is given by the

---

b
Figure 4: Effects of the distortion measure on maximum entropy coding (MEC), (a): Encoding the representations is akin to packing e-balls into the representation space. (b): T-SNE [71] visu- alization of the representations learned with large (left plot, c = 0.12) and small (right plot, &a = 0.01). (c): Linear and kNN accuracy and the spectral norm w.t the degree of distortion
following coding length function [47, 72]:
L + d log det I
Z' me²
Z
(I)
where I denotes the identity matrix with dimension m. and is the upper bound of the expected
decoding error between
and the decoded 2, i.e., E [||= Ø||2] S
We note that the computation of log-determinant of high dimensional matrix in Equation (1) is highly expensive and may cause numerically unstable results for ill-conditioned matrix, which inhibits its ap- plication to large-scale pre-trane pre-training (e.g., over million images). Therefore, scalable and stable refor-
[Im + AZ Z where μ m++d and À mez. Utilizing the identical equation (ct A)) exp(Tr(A)) [35].
mulation of Equation (1) required. We first rewrite Equation (1) as L. = i log det
d
we obtain L = Tr ( log [m + AZ z) 2)). where Tr stands for the trace of the matrix. Finally, apply Taylor series expansion to expand the logarithm of the matrix and obtain
8
(-1)ktt
TT
11
(AZ 2
h
k=1
with convergence condition:
AZ T z
< 1, and this can
spprox.
origh
[8.7%]
be achieved by adjusting the hyperparameter e (detailed in Section 2.2). Compared Equation (1), there only matrix multiplication and addition in Equation (2), which significantly speeds up the computation process and avoids the numerical unstability. To verify this, in Figure 3, we make a comparison of running time and relative approx- imation error between Equation (1) and (2) (using the first four terms). The results show that our reformulation can approximate the original coding length function with negli- gible error (all errors are well below 0.5%), and accelerate the computation considerably (over 50x acceleration for all cases), thus making large-scale pre-training possible.
Lime (mal topecaaee
(5.19))
80
2.0
0.9
dm
Figure 3: Comparison of running time and relative approximation error be- tween Equation (1) (origin) and Equa- tion (2) (approx.) for different number of samples in Z (dim).
2.2 Combining with the View Consistency Prior
It should be noted that the necessary premise of the max-
imum entropy principle is that testable information given as prior. For example, the testable information could be the accuracy of predictive model: the most generalizable model should be one with maximum entropy, but it is only when a group of models reaches a given accuracy. Otherwise, simply optimizing towards maximum entropy will lead to trivial solutions such as uniform distributions. In Equation (1), the prior not considered. introduce prior, we employ a common practice in SSL by augmenting Z into two different views Z₁ and Zz. More specifically, given

---

set of images D, an image x' is sampled uniformly from D, and two augmented views x1 and zg are obtained from x via a distribution of data augmentations T. Then they are fed to an encoder f. consisting of backhone and projector network, which produces lz-normalized embeddings of žι
and 22. For a batch of m images, we have Z1 2f....... 21"] and similarly for Zz, which are two observations of the same Z. MEC aims to minimize the following loss:
1) (AZ,2z) 212
MMEC = --- det (Im+ + X Z22
2- - 1 k
where the same notations in Section 2.1 apply, and n. is the order of Taylor expansion. Compared with Equation (1), the formulation in (3) considers not only maximizing entropy, but also the view consistency prior mined from the data itself, therefore learning meaningful representations.
As noted in Section 2.1, the convergence condition of Taylor expansion requires ||C||22 < 1, where
C' = AZ Z2 and À = me mey. We show such condition can be strictly satisfied by setting 3) >>> 1 because of the inequality ||C'|l2 S V/ < 1. In practice, we empirically find that the Taylor expansion converges over a wide range of ca (Figure 4(c)) with a linear warm-up. From the preliminary experiments on CIFAR- 10 [41] (detailed in Appendix B), also find that the distributions of representations show progressive finer granularity as ca decreases (Figure 4(b)). This can be interpreted by the practical meaning of the distortion ea (Figure 4(a)), i.e., smaller ed encourages the representation space to be encoded in finer granularity (and hence more uniform). By contrast, a small 6d might break the semantic structures of similar images (i.e., tolerance). Therefore, a good choice of ea is needed to compromise the uniformity and tolerance properties [75, 74J of representations, which shares the same role as the temperature [80] term in contrastive learning.
An overview of MEC is illustrated in Figure and PyTorch-like pseudocode is provided in Appendix A. The algorithm describes the minimalist variant of MEC, which can be further improved by integrating momentum encoder and asymmetric networks (detailed in experiments).
2.3 A Unified View of Batch-wise and Feature-wise SSL Objectives
Current SSL methods based on Siamese networks can be roughly divided into two categories: batch- wise methods [12, 31, 13, 15, 14, 10] and feature-wise methods [86, 24, 36]. The former aims to minimize the distance between augmented views of the same sample while maximizing the distance between different samples, which can be viewed decorrelating the different features batch, The latter, in contrast, tries to decorrelate the different vector components in the representation. The relationship between them has not been fully understood. Our work builds bridges between these two types of methods through the following derivation:
CMEC = - log det Im AZ Z2) = - log det Ia+
hatch-wise
fenture-wise
which can be proved since z Zz E Rm and Z Z
E Rxd have the same nonzero eigenvalues. In Figure 2, under the framework of MEC, we show the equivalence between batch wise and feature- wise methods using two examples, SimSiam [14] and Barlow Twins [86]. By taking Taylor expansion (Equation (2)) of the left side of Equation (4) and before the trace operation, the diagonal elements of
the leading term (i.e., pAZ Zz) measure the similarity between the views of the same images in batch, and the objective of SimSiam [14] is equivalent to maximizing the trace of this term. Similarly, the leading term of the right side expansion models the correlation between dimensions of the feature, and the objective of Barlow Twins [86] is equivalent to the second-order expansion of L. With the above derivation, our method naturally subsumes the two different kinds of objectives its low-order expansions, and we show in experiments that better downstream task performance can be achieved with higher-order approximations. We further show in Appendix that our MEC can also bridge other self-supervised objectives. And we hope the direct tying of family of objectives to very grounded mathematical concept can inspire more new methods.
3 Experiments
We perform self-supervised pre-training using the proposed MEC on the training set of the Ima- geNet ILSVRC-2012 dataset [17]. After pre-training, conduct extensive experiments examine
5

---

Table 1: Linear evaluation. All methods are based on standard ResNet-50 [33] pre- pre-trained with two x24 224 views on ImageNet train- ing dataset. We perform pre-training with four different length of epochs following [14].
Table Semi-supervised classification. We finetune the pre-trained model using 1% and 10% training samples of ImageNet follow- ing [12, 29], and the top-1 and top-5 accuracy on ImageNet val dataset are reported.
Method
Pre-training epochs
Mothod
1%
10%
100
200
400
800
Top 1
Top 5
Top
Top
SimCLR [12]
66.5
68.3
69.8
70.4
Supervised
25.4
48.4
56.4
80.4
MoCo v2 [13]
67.4
69.9
71.0
72.2
SimCLR [12]
48.3
75.5
65.6
87.8
BYOL [29]
66.5
70.6
73.2
74.3
BYOL [29]
53.2
78.4
68.8
89.0
SwAV [10]
66.5
69.1
70.7
71.8
Barlow [861
55.0
79.2
69.7
89.3
SimSiam [14]
68.1
70.0
70.8
71.3
DINO [[i]
52.2
78.2
68.2
89.1
Barlow [86]
67.3
70.2
71.8
73.0
VICReg
54.8
79.4
69.5
89.5
.EC
70.6
71.9
73.5
74.5
MEC
55.9
79.6
70.3
89.7
Table 3: Transfer learning on object detection and instance segmentation tasks. We fine-tune
the pre-trained model end-to-end on target datasets and tasks, following the standard protocol described in [13, 14, 29, 86). We use Faster R-CNN [61] for VOC detection tasks and Mask R- R-CNN [32] (1 x schedule) for COCO detection and instance segmentation tasks. All Faster/Mask R-CNN
models are with the C4-backbone [79]. Bold entries are within 0.2 below the best.
VOC 07 detection
VOC 07+12 detection
COCO detection
COCO instance
Pro-train
AP
AP
AP
AP50
AP
AP%s
APse
AP
AP1s
Aprik
Ap
AP
Scratch
35,9
16.8
13.0
60.2
33.8
33.1
44.0
26.4
27.8
46.9
29.3
30.8
Supervised
74.4
42.4
42.7
81.3
53.5
58.8
58.2
38.2
41.2
54.7
33.3
35.2
SimCLR [12]
61.4
35.3
McCo v2 [13]
77.1
48.5
52.5
82.5
57.4
64.0
58.9
39.3
42.5
55.8
34.4
36.5
BYOL [29]
77.1
47.0
49.9
81.4
55.3
61.1
57.8
37.9
40.9
54.3
33.2
35.0
SwAV [10]
75.5
46.5
49.6
82.6
56.1
62.7
58.6
38.4
41.3
55.2
33.8
35.9
Barlow [86]
75.7
47.2
50.3
82.6
56.8
63.4
59.0
39.2
42.5
56.0
34,3
36.5
SimSiam [14]
77.3
48.5
52.5
82.4
57.0
63.7
59.3
39.2
42.1
56.0
34.4
36.7
MEC
77.4
48.3
52.3
82.8
57.5
64.5
59.8
39.8
43.2
56.3
34.7
36.8
the learned representations on various tasks. The tasks considered include four image-level tasks: ImageNet linear probing, ImageNet semi-supervised classification, object detection, and instance seg- mentation; and five video-based tasks: single object tracking (SOT) [78], video object segmentation (VOS) [56], multi-object detection (MOT) [48], multi-object detection and segmentation (MOTS) [73] and pose tracking (PoseTrack) [1]. An overview comparison with existing SSL representations across different tasks is presented in Section 3.1. Moreover, in Section 3.4, we present extensive ablations studies to demonstrate the behaviors and properties of the proposed method. All the implementation details of the aforementioned experiments are deferred to Appendix C.
3.1 Overview Comparison
We evaluate MEC together with multiple previous SSL representations the four image-based tasks and the five video-based tasks, plot their performance ranking along a axis for a specific task and obtain two sets of radar-like charts in Figure 1. Intuitively, it can be seen that the larger polygon in a radar chart is, the better the corresponding representation generalize across downstream tasks. MEC consistently outperforms previous representations both image- and video-bare based tasks considered, suggesting its good generalization capacity to various downstream tasks and data distributions. Below detail experiments each task. And additional experiment results can be found in Appendix D. including pre-tran pre-training with different strategies and datasets, transfer learning on fine-grained classification benchmarks.
3.2 Evaluation on ImageNet
Linear probing. We train a linear classifier on top of frozen representations of the pre-trained model on the ImageNet training set, and report the top-1 accuracy on the ImageNet validation set, which is

---

method
MBC
KNN
lincar
101 SCCE
83.5
86.1
SimSiam 14]
84,9
87.3
simsiam w/o mec simsiam mec
simcir w/o mec
84.6
85.9
simnir w/
SimCLR [12]
87.0
88.2
epochs
epochs
Figure Effects of MEC regulation on other SSL objectives on CIFAR-10 [41]. Left plot: accuracy of kNN classifier as monitor of the pre-training process. Right table: linear probing and kNN classification accuracy of the pre-trained models with v.s. without our MEC regulation.
standard and important protocol in SSL [88, 12, 31, 14, 86]. We evaluate the learned representations of models with four different pre-training epochs following [14], and make comparisons in Table with state-of-the-art methods, whose results are reported [14] and better than their original papers. Even so, our method MEC achieves highest accuracy under all pre-training epochs among those methods. particular, with only 200-epoch pre-training, MEC outperforms several 800-epoch pre-trained methods, including SimCLR [12]. SWAV [10] and SimSiam [14], which indicates the pre- training efficiency of our method. Additionally, by incorporating other pre-training strategies, such multi-crop [10, I1], method can attain even better results, which are detailed in Appendix D. Furthermore, for fixed given downstream task (i.e., image classification), MEC also improves generalization across different data distributions, which can be concluded from the transfer learning experiments on 11 fine-grained classification datasets in Appendix D.
Semi-super seee-supereised classification. We fine-tune the pre- pre-trained model on a small subset of ImageNet classification task. Specifically, we adopt the same fixed splits of 1% and 10% of ImageNet training set as in [12] and report both top-1 and top-5 accuracies in Table 2. The experiment results show that our method consistently outperforms previous approaches in both 1% and 10% settings. And also note that all SSL methods surpass the supervised counterparts by large margin (e.g.. over 20% accuracy using 1% labeled data), which emphasizes the importance of leveraging large amount of unlabeled data.
3.3 Transfer learning
Object detection and instance segmentation. We transfer the pre-trained model by end-to-end fine-tuning on object detection and instance segmentation tasks, following the common practice of previous methods [31, 13, 14, 29, 86]. We report the results in Table 3. Similar to the observations semi-supervised classification, the SSL methods outperform or are on par with the supervised counterparts in all tasks. Our method achieves better performance than previous approaches in most of the tasks and metrics considered. In particular, MEC significantly better than the supervised baseline (+1.6 AP), SimCLR (+1.9 AP), and previous best method MoCo v2 (+0.5 AP) on COCO detection task. These results validate that the model traitranne by MEC can generalize very well across different image-based tasks and datasets.
Video-based tasks. To further evaluate whether MEC's representation generalizes beyond image- based tasks, we transfer the pre-trained model to a series of video tasks based on a recently proposed evaluation platform named UniTrack [76]. In contrast above evaluation processes, UniTrack does not require any additional training and hence provides more direct comparison between different representations. We report the results in Table 4 and compare different methods using radar plots in Figure 1. As noted in UniTrack, there is no significant correlation between linear probe accuracy and tracking performance. However, our method achieves strong performance across linear probing and all five video tracking tasks. Specifically, MEC achieves the highest rank on four our of five tasks, and ranks among the top methods on seven metrics out of all nine metrics. These results substantiate that the learned representations of MEC are more peneral-purpose and versatile. We attribute this important property MBC to the principle of maximum entropy, the basis of our method, which enables the model to make best use of data for unseen downstream tasks during pre-training.
7

---

Table 4: Transfer learning on five video tracking tasks. All methods use the same ResNet-50 backbone [33] and are evaluated based on UniTrack [76]. For each cell, we report the results obtained from features at [1ayer3/ 1,yer4], and the best performance between the two is bolded. We then use the best of the two to rank the top 2 models (underlined) in each column.
SOT
VOS [6]
MOT (18)
MOTS [73]
BrackTac [i]
Pro-trrin
AUCyo
T ALC
J-meant
IDFI
ROTAT
IDF1
HOEAT
IDFT
1Ds1
Rand. Init.
10,3/9.0
28.0/20.0
29.3/3339
8./899
8.4/8.5
20.8/23.1
25.9/ 28.7
40.2/38.5
88792 9096.
Supervised
$8.6/49.5
62.0/53.9
02.3/57: 57.9
756673 73.2
633/6118
68.4/69.4
70.2/710
73.3/733
6969 /
InsDis [80)
47.6/44.3
61.//1.1
62.6/80.1
66.7/7399
57.9:619
68.4/68.0
69.6/70.3
72.4/73.9
7106 7015
McCo [31]
47.9
/53.7
61./
69.1 74.1
16.3
71.6/
SimCLR [12]
57.3/5199
61.3/50.7
60.5 55.55
66.91 75.6
63.2
65.816770
67.7/695 69.5
72.3/73.5
7084/ 7367
MoCo [13]
44.7
65.3/
61.2/ 54.0
72.0/74.9
61.2 62.8
67.5/66.3
69.6 69.6
73.0
6932. 7702
Infomin [68]
48.5 46.8
612//1.9
56.4/51.1
66.2/73.4
57.5/619
68.5 68.5
72,5/74,0
7066:
BarLow [86]
60.51
61.7
72.3/74.3
BYOL [29]
48.3
58.91 55.8
/4.3
65.3/74.9
$6.8 62.9
666.8
70.8/ 59.3
72:4/ 73.8
7213
SwAV [10]
19.2/ 52.4
65,5
59.4155 /9.0
65.6/74.4
$6.9/623
68.8/67.3 667.0
69.9 /69.5
72.7/7344 73.6
7025 7377
PixPro [82]
40.5
57.41 19.3
56,4/ 52.2
61.7/6777
51.3/
61.2 / 66.2
66.1
72.4/78. 73.1
7163/ 6953
DeCoo [81]
47.1
59.0
02.3 56.1
753/7229
02.8/ 61.6
70.0/
73.9
MRC
56.8
60.6
62.0/57.5
72.7/75.1
61.2/633
70.7666.5
/7.11
6665
Table 5: MEC ablation experiments with ResNet-50 on ImageNet. We report the results of linear probing. The default settings of these experiments are marked in gray
(a) Smaller epochs.
(b) Smaller batch sizes.
(c) The distortion Edt
epochs
70
100
batch
128
256
512
1024
0.02
0.06
0.25
1.0
acc.
62.1
67,5
69.2
70.6
acc.
69.5
70.1
70.4
70.6
70.5
70.6
70.2
69.5
(d) The momentum coefficient.
(e) The order of expansion.
The projector network.
0.9
0.99
0.996
0.999
order
2
proj
asym.
sym.
4096
8192
acc.
69.0
70.1
70.4
70.6
70.2
acc.
70.0
70.3
70.5
70.6
70.6
ncc.
70.6
70.1
70.7
71.2
3.4 Ablation studies
Batch size and training epoch. The computation and memory resources required for training self-supervised models are demanding. Previous methods often need a large batch size (e.g., 4096 in SimCLR [12]) and long training time (e.g., 1000 epochs in Barlow Twins [86]) to work well. We hence test the efficiency of our method by pre-training the model with smaller training epochs (Table 5a) and batch sizes (Table 5b). The results show that our method can effectively avoid those prohibitive requirements. In particular, Our 50-epoch pre-trained model has already outperformed those 100-epoch pre-trained models of other methods (e.g., SimCLR [12], BYOL [29], Barlow Twins [86]). Moreover, MEC is more robust to diffcrent batch sizes, and even a batch size of 128 performs well (with 1.1% accuracy drop). These results noticeably different from those methods based on the pretext task of instance discrimination, where large number of negative samples is required (e.g., over 4% accuracy drop of SimCLR with batch size of 128). This behavior also liberates MEC from those complex designs, such as memory queue [13] and online clustering [10].
Siamese networks play a critical role in discriminative SSL, and different methods often rely on different architectures to prevent mode collapse. We investigate the effects of different designs of Siamese networks by varying the momentum coefficient (Table 5d) and the projector network (Table 5f). Compared to other methods [13, 14, 29, 86], MEC still works well when the Siamese networks are direct weight-sharing (m=0) and symmetric (sym.). In contrast, BYOL [29] requires the momentum encoder to prevent collapse (0.3% accuracy without it), and asymmetric architectures are essential for SimSiam [14] to work. These results demonstrate that the objective of MEC naturally avoids representational collapse without relying on special design of Siamese networks. We also observe that the performance can be further improved with larger projector network, consistent with the conclusions in previous methods [86, 5].
Taylor series approximation. The computation of log-determinant of high-dimensional matrix can be significantly accelerated by exploiting Taylor series approximation. And we study two important hyper- parameeersss of the process, i.e., the order of Taylor expansion (Table 5e) and the distortion
8

---

Table Comparison of SSL methods with different architectures. (ConvNets v.s. Transformer)
All methods are vre-trained with two 224 crops on ImageNet training set, and we report the results of linear probing on ImageNet validation set.
model
MЕC
IBOT (90)
MAE (30)
DINO [11]
MoCo v3 [15]
SimCLR [12)
BYOL [29]
SwAV [10]
R-50, 800-ep
74.5
73.8
70.4
74.3
71,8
VIT-S, 300-cp
73,4
73.2
68.2
70.9
72.5
69.0
71.0
67.1
BTFB 300-ep
76.5
75.6
69.3
72.6
76.5
73.9
73.9
71.6
measure Ed (Table 5c). We find that the accuracy increases with higher order of expansion, since it produces more accurate approximation. Besides the linear probing, we find the COCO detection performance can be improved by a large margin (+1.5 AP) when the order is increased from to 4. We also note that the extra two orders (from 2 to 4) benefit less than lifting from the first order the fourth order approximation. It is reasonable since the relative approximation error of second-order expansion is already quite decent, lower than 0.5% (Figure 6), almost the same as fourth-ordes expansion (Figure 3). As for the distortion hyper-parameter, on one hand, it should be large enough so that the Taylor expansion is convergent, and we empirically observe this condition satisfied over a wide range of degree of distortion. On the other hand, a good choice of e is also needed to improve the representation quality, because it compromises the uniformity and tolerance properties of representations as depicted in Figure 4 and discussed in Section 2.2.
Generalization across different backbones and objectives. Given the recent progress in Vision Transformers (ViT') [21], it is worthwhile to examine the generalization ability of our method on both VITs and ConvNets, though previous methods usually focus on only one architecture. To this end, make a straightforward extension by replacing the ConvNet encoder with a ViT encoder, following the practice of [15]. The comparisons are made in Table 5 with previous methods based on Siamese networks [12. 29, 10, 11, 31J, and also recently proposed masked auto-encoding methods [30, 90] tailored for Transformers. And the results show that MEC performs equally well with both architectures. Furthermore, since MEC is designed for optimizing the representation quality, it should be complementary to other methods that focus on pretext tasks. To validate this hypothesis, we apply MEC as regulation term on the representations of a contrastive method, SimCLR [12], and a non-contrastive method, SimSiam [14]. The results in Figure 5 demonstrate that MEC improves both linear and kNN accuracy by large margin (e.g., 2.3% linear probing accuracy for SimCLR) with negligible computation overhead (~1.00 xrunnnng time).
4 Related Work
Maximum entropy principle states that the probability distribution which best represents the current
state of knowledge the one with largest entropy, in the context of precisely stated prior data. This
principle originates in Laplace's "Principle of Insufficient Reason" and is popularized by Jaynes [38, 391. has a wide variety of applications in different fields [40], ranging from statistical physics [58,
51]. economics [2, 62], to reinforcement learning [77, 49] and supervised learning [57, 64, 22]. To the best of our knowledge, the usage of this principle has not been explored the context of SSL.
We notice closely related work MCR² [85] exploits the same coding length function (Equation (1)), but the motivation behind quite different: they do not originate from maximum entropy, but instead base their method on coding rate reduction. Furthermore, MCR" experiments clustering tasks, and is difficult to generalize to modern SSL settings due to the expensive and unstable computation. In contrast, our method is tailored for SSL and we reformulate the coding length function into a scalable and stable form for large-scale pre-training. Another closely related concept in information theory is mutual information, which has been widely used in SSL [34, 53, 68, 67, 69] based on the infomax principle [45]. Our method can also be viewed as an application of the infomax principle that maximizes the representation information, but without using mutual information estimators, which might result in worse transfer performance [70].
classification and
Pretext tasks are one of the core components in SSL. They refer to a type of tasks that mine certain structure of the data, and then force learners to predict such structure from degenerated input data. Labels are mined from the data itself, therefore dispensing with the need of human annotations. In natural language processing, the most commonly used pretext task masked language modeling (MLM) [18, 60], where random portion of the input sequence is masked and the model learns to

---

predict these masked content. In computer vision, however, there emerge a much wider variety of pretext tasks [87, 88, 52, 83, 20, 80, 55, 19, 30, 4, 42], since images contain much more information than languages so that there are much more intrinsic structures to be mined as free learning signals Solving pretext tasks requires the model to understand the visual concepts present in images and thus useful representations can be learned. In this work, however, we argue that those handcrafted pretext tasks, like handcrafted feature engineering methods, might introduce undesired bias into the representations. So we pursue a straightforward approach to optimize considering that the ultimate goal of self-supervised learning is a general-purposs representation, explicitly encourage the generalization ability on downstream tasks and minimize the bias in the formulation of the pretext task, by introducing the maximum entropy principle.
representations. And
Siamese representation learning uses two identical neural networks to learn representations by comparing entities, in supervised manner [8, 65, 7]. or recently in self-supervised manner [12, 13, 14, 29, 10, 11, 86, 5, 23, 433. In Siamese SSL., the two branches are weight-sharing [12, 14], or one is a moving average of the other [31, 29]. One crucial problem that needs to carefully addressed in Siamese SSL is mode collapse, where all data is mapped to the same representation. Contrastive methods, such as SimCLR [12] and MoCo [31], prevent the trivial solution by minimizing the distance between augmented views of the same images while maximizing the distance between different images. Clustering-bast -based methods [9, 3, 10] enforces consistency between cluster assignments of the views from same images, where the cluster centers work as negative prototypes. Non-contrastive methods [14, 29] remove the use of negative pairs and SimSiam [14] concludes that the stop- gop-gradient operation is critical to preventing mode collapse. Our work also leverages the basic view consistency prior inherent in Siamese representation learning, which gives testable information for the application of maximum entropy principle. Compared to previous work, method of maximum entropy coding naturally avoids the low-entropy mode collapse problem and can be easily implemented without requiring instance discrimination [20, 12], memory queue [80, 31, 13] or online clustering [9, 3, 10]. Furthermore, we find interesting equivalence between low-order approximations of MEC and existing batch-wise feature-wise objectives, which provides a new perspective for a unified understanding of SSL methods.
5 Conclusion
Whut makes for generalizable representations? In this work, we argue that the best generalizable representation is supposed to be the one that admits the maximum entropy. Based on this assumption, we propose MEC, a novel pretext task that aims explicitly to generalize well. MEC pursues a representation with maximum entropy in all eligible ones that satisfies the view consistency prior of Siamese representation learning. The minimal coding length in lossy data coding is leveraged as computationally tractable surrogate for the entropy; and in order to facilitate large-scale pre- training, we reformulate the coding length function into a scalable form. With the reformulation, we further demonstrate that MEC bridges the batch-wiss and ceature-wise SSL objectives. Extensive experiments shows MEC generalizes well across various image/video tasks, various data distributions, and different network architectures. Furthermore, MEC can be readily plugged into existing SSL methods such as SimCLR and SimSiam, and brings consistent improvements upon these strong representations. We hope our exploration will inspire the community to rethink the possibility of optimizing generalization explicitly in self-supervised representation learning.
6 Funding Transparency Statement
Acknowledgement: This work is supported by the state key development program in 14th Five-Year under Grant No. 1021YF00602003 2021 YFF0602102, 2021QY We also thank for the research fund under Grant No. 201 2019GQG0001 from the Institute for Guo Qiang, Tsinghua University.
References
[1] M. Andriluka et al., "Posetrack: A benchmark for human pose estimation and tracking," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018,
pp. 5167-5176.
10

---

[2] C. Arndt, S. Robinson, and F. Tarp. "Parameter estimation for a computable general equilibrium model: maximum entropy approach," Economic Modelling. vol. 19, no. 3, pp. 375-398, 2002.
[3] Y. M. Asano, C. Rupprecht, and A. Vedaldi, "Self-labellingg via simultaneous clustering and representation learning," arXiv preprint arXiv:1911.0537/ 2019.
[4] H. Bao, L. Dong, and F. Wei, "Beit: Bert pre-training of image transformers," arXiv preprint 0rX45:2106.08254, 2021.
[5] A. Bardes, J. Ponce, and Y. LeCun, "Vicreg: Variaace-iivariane-ccvvaaa regularization for self-supervised learning," arXiv preprint arXiv:2105.044 2021.
[6] J. Beirlant, E. J. Dudewicz, L. Györfi, E. C. Van der Meulen, et al., "Nonparametric entropy estimation: An overview," International Journal of Mathematical and Statistical Sciences,
vol. 6, no. I, pp. 17-39, 1997.
[7] L. Bertinetto, Valmadre, Henriques, A. Vedaldi, and P. Torr, "Fully-convolutiona siamese networks for object tracking," in European conference on computer vision, Springer, 2016, pp. 850-865.
[8] J. Bromley, I. Guyon, Y. LeCun, E. Säckinger, and R. Shah, "Signature verification using a" siamese" time delay neural network," Advances in neural information processing systems, vol. 6, 1993.
[9] M. Caron, P. Bojanowski, A. Joulin, and M. Douze, "Deep clustering for unsupervised learning of visual features," in Proceedings of the European Conference on Computer Vision (ECCV),
2018, pp. 149.
[10] M. Caron, I. Misra, Mairal, P. Goyal, P. Bojanowski, and A. Joulin, "Unsupervised learning of visual features by contrasting cluster assignments," Advances Neural Information Processing Systems, vol. 33, pp. 9912-9924, 2020.
[11] M. Caron al., "Emerging properties in self-supervised vision transformers," " in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 9650-9660.
[12] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, "A simple framework for contrastive learning of visual representations," in International conference on machine learning, PMLR, 2020, pp. 1597-1607.
[13] X. Chen, H. Fan, R. Girshick, and K. He, "Improved baselines with momentum contrastive learning," arXiv preprint arXiv:2003.0422 0449977 2020.
[14] X. Chen and K. He, "Exploring simple siamese representation learning," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 15 750-15 758. [15] X. Chen, S. Xie, and K. He, "An empirical study of training self-supervised vision transformers," in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 9640- 9649.
[16] T. M. Cover, Elements of information theory. John Wiley & Sons, 1999.
[17] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, "Imagenet: A large-scale hierarchical image database," in 2009 IEEE conference on computer vision and puttern recognition, leee, 2009, pp. 248-255.
[18] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, "Bert: Pre-training of deep bidirectional transformers for language understanding." arXiv preprint arXiv: 2018.
[19] C. Doersch, A. Gupta, and A. A. Efros, "Unsupervised visual representation learning by context prediction," in Proceedings of the IEEE international conference on computer vision. 2015, pp. 1422-1430.
[20] A. Dosovitskiy, J. T. Springenberg, M. Riedmiller, and T. Brox, "Discriminative unsuper- vised feature learning with convolutional neural networks," " Advances in neural information processing systems, vol. 27, 2014.
[21] A. Dosovitskiy et al., "An image is worth 16x16 words: Transformers for image recognition at scale," arXiv preprint arXiv:2010. 1929,, 2020.
[22] A. Dubey, O. Gupta, R. Raskar, and N. Naik, "Maximum-entropyy fine grained classification," Advances in neural information processing systems, vol. 31, 2018
[23] D. Dwibedi, Y. Aytar, J. Tompson, P. Sermanet, and A. Zisserman, "With a little help from my friends: Nearest-neighhor contrastive learning of visual representations." in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 9588-9597.
11

---

[24] A. Ermolov, A. Siarohin, E. Sangineto, and N. Sebe, "Whitening for self-supervised represen- tation learning," " in International Conference on Machine Learning. PMLR, 2021, pp. 3015- 3024.
[25] M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman, "The pascal visual object classes (voc) challenge," International journal of computer vision, vol. 88, no. 2,
pp. 303-338, 2010.
[26] D. Fu et al., "Unsupervised pre-training for person re-identification." CVPR, 2021.
[27] S. Gidaris, P. Singh, and N. Komodakis, "Unsupervised representation learning by predicting image rotations," arXiv preprint
2018.
[28) P. Goyal et al., "Accurate, large minibatch sgd: Training imagenet in hour," arXiv preprint
arXiv:1706. (ECi::7706006677.. 2017.
[29] J.-B. Grill al., "Bootstrap your own latent-a new approach to self-supervised learning." Advances in Neural Information Processing Systems, vol. 33, pp. 21 271-21 284, 2020.
[30] K. He, X. Chen, S. Xie, Li,, P. Dollár, and R. Girshick, "Masked autoencoders are scalable vision learners," Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
[31] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick, "Momentum contrast for unsupervised visual representation learning," " Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 9729-9738.
[32] K. He, G. Gkioxari, P. Dollár, and R. Girshick, "Mask r-cnn," in Proceedings of the IEEE international conference on computer vision, 2017, pp. 2961-2969.
[33] K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition." in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770-
778.
[34] R. D. Hjelm et al., "Learning deep representations by mutual information estimation and maximization," " arXiv preprint arXiv:1808. 06670, 2018.
[35] A. Horn and C. R. Johnson, Matrix analysis. Cambridge university press, 2012.
[36] T. Hua, W. Wang, Xue, Ren, Y. Wang, and Zhao, "On feature decorrelation in self- supervised learning," in Proceedings of the IEEE/CVF International Conference Computer Vision, 2021, pp. 9598-9608.
[37] S. loffe and C. Szegedy, "Batch normalization: Accelcrating deep network training by reducing internal covariate shift," in International conference on machine learning, PMLR, 2015,
pp. 448-456.
[38] E. T. Jaynes, "Information theory and statistical mechanics," Physical review, vol. 106, no. 4,
p. 620, 1957.
[39]
E. T. Jaynes, "Information theory and statistical mechanics. ii," Physical review, vol. 108, no.
p. 171, 1957.
[40]
J. N. Kapur, Maximumn-nntrop models in science and engineering. John Wiley & Sons, 1989.
[41] A. Krizhevsky, "Learning multiple layers of features from tiny images," Master's thesis, University Tront, 2009.
[42] Lee, K. Lee, K. Lee, Lee, and J. Shin, "Improving transferability of representations via augmentation-awwrre self-supervision," in Advances in Neural Information Processing Systems, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, Eds., 2021. [Online]. Available: https: //tp:///ppeeeiiieeee www/ pttt/forumidduuAr(jjpi Jnm..
[43] K.-H. Lee, A. Arnb,, S. Guadarrama, J. Canny, and I. Fischer, "Compressive visual repre- sentations," Advances in Neural Information Processing Systems, vol. 34, pp. 19 138-8-19 552, 2021.
[44] T.-Y. Lin et al., "Microsoft coco: Common objects in context," in European conference on computer vision, Springer, 2014, pp. 740-755.
[45] R. Linsker, "Self-organization in a perceptual network," Computer, vol. 21, no. 3, pp. 105-117, 1988.
[46] I. Loshchilov and Hutter, "Sgdr: Stochastic gradient descent with warm restarts," arXiv preprint 108.:36608.03398, 2016.
[47] Y. Ma, H. Derksen, W. Hong, and J. Wright, "Segmentation of multivariate mixed data via lossy data coding and compression." IEEE transactions on pattern analysis and machine intelligence, vol. 29, no. 9, pp. 1546-1562, 2007.
12

---

[48] A. Milan, L. Leal-Taixe I. Reid, S. Roth, and K. Schindler, "Motl6: A benchmark for multi- object tracking," arXiv preprint 2xXiv:603300831 2016. [49] V. Mnih et al., " "Asynchronous methods for deep reinforcement learning," in International conference on machine learning, PMLR, 2016, pp. 1928-1932
[50] V. Nair and G. E. Hinton, "Rectified linear units improve restricted boltzmann machines," in loml, 2010.
[51] T. K. Nakamura, "Statistical mechanics of a collisionless system based on the maximum entropy principle," The Astrophysical Journal, vol. 531, no. 2. p. 739, 2000.
[52] M. Noroozi and P. Favaro, "Unsupervised learning of visual representations by solving jigsaw puzzles," in European conference on computer vision, Springer, 2016, pp. 69-84.
[53] A. v. d. Oord, Y. Li, and O. Vinyals, "Representation learning with contrastive predictive coding," " arXiv preprint arXiv: 0774488 2018.
[54] L. Paninski, "Estimation of entropy and mutual information," Neural computation, vol. 15, no. 6. Pp. 1191-1253, 2003.
[55] D. Pathak, P. Krahenbuhl, J. Donahue, T. Darrell, and A. A. Efros, "Context encoders: Feature learning by inpainting." " in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 2536-2544
[56) F. Perazzi, J. Pont- Tuset, B. McWilliams, L. Van Gool, M. Gross, and A. Sorkine-Hornung, "A benchmark dataset and evaluation methodology for video object segmentation." " in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 724-732.
[57] G. Pereyra, G. Tucker, J. Chorowski, L. Kaiser, and G. Hinton, "Regularizing neural networks by penalizing confident output distributions," arXiv preprini arXiv: 170,00470,,06548, 2017. [58] S. Pressé, K. Ghosh, J. Lee, and K. A. Dill, "Principles of maximum entropy and maximum caliber in statistical physics," Reviews of Modern Physics, vol. 85, no. 3, p. 1115,
[59] Principle of Maximum Entropy, https : // enn wikipedia. org/ wwkii/iiiiieeee httr///rrncpppe.ottt maximum entropy.
[60] A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever, "Improving language understanding by generative pre-training," 2018.
[61] S. Ren, K. He, R. Girshick, and J. Sun, "Faster r-cnn: Towards real-time object detection with region proposal networks," Advances in neural information processing systems, vol. 28, pp. 91-99, 2015.
[62] E. Scharfenaker and J. Yang, "Maximum entropy economics," " The European Physical Journal Special Topics, vol. 229, no. 9, pp. 1577-1590, 2020.
[63] C. E. Shannon, "A mathematical theory of communication," The Bell system technical journal, 1948.
vol. 27, no. 3. pp.
[64] J. Shawe-Taylor and D. Hardoon, "Pac- "Pac-bayes analysis of maximum entropy classification," in Artificial Intelligence and Statistics, PMLR, 2009, pp. 480-487.
[65] Y. Taigman, M. Yang, M. Ranzato, and I.. Wolf, "Deepface: Closing the gap to human-level performance in face verification," " in Proceedings of the IEEE conference on computer vision and pattern recognition, 2014, pp. 1701-1708.
[66] C. Tao et al., "Exploring the equivalence of siamese self-supervised learning via unified gradient framework," arXiv preprint arXiv:21 2021.
[67] Y. Tian, D. Krishnan, and P. Isola, "Contrastive multiview coding," in European conference on computer vision, Springer, 2020, pp. 776-794.
[68] Y. Tian, C. Sun, B. Poole, D. Krishnan, C. Schmid, and P. Isola, "What makes for good views for contrastive learning?" arXiv preprini arXiv:2005. 10243, 2020.
[69] Y.H.. H. Tsai, M. Q. Ma, M. Yang. H. Zhao, L.-P. Morency, and R. Salakhutdinov, "Self-supervised representation learning with relative predictive coding," arXiv preprint arXiv:2103. 1275, 2021.
[70] M. Tschannen, J. Djolonga, P. K. Rubenstein, S. Gelly, and M. Lucic, "On mutual information maximization for representation learning," arXiv preprint arXiv:1907.1366 2019.
[71] L. Van der Maaten and G. Hinton, "Visualizing data using t-sne.." " Journal of machine learning research, vol. 9, no. 11, 2008
[72] R. Vidal, Y. Ma, and S. Sastry, "Generalized principal component analysis (gpca)," IEEE transactions on pattern analysis and machine intelligence, vol. 27, no. 12, pp. 1955-1959, 2005.
13

---

[73] P. Voigtlaender et al., "Mots: Multi-object tracking and segmentation," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 7942-7951.
[74] F. Wang and H. Liu, "Understanding the behaviour contrastive loss," Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp.
[75] T. Wang and P. Isola, "Understanding contrastive representation learning through alignment and uniformity on the hypersphere," in International Conference on Machine Learning, PMLR,
2020, pp. 9929-9939.
[76] Z. Wang. H. Zhao, Y.-L. Li, S. Wang, Torr, and Bertinetto, "Do different tracking tasks require different appearance models?" Advances in Neural Information Processing Systems, vol. 34, 2021.
[77] J. Williams and J. Peng, "Function optimization using connectionist reinforcement learning algorithms," Connection Science, vol. no. pp. 241-268, 1991.
[78] Y. Wu, Lim, and M.-H. Yang. "Online object tracking: benchmark." in Proceedings of the IEEE conference on computer vision and pattern recognition, 2013, pp. 2411-2418.
[79] Y. Wu, A. Kirillov, F. Massa, W.-Y. Lo, and R. Girshick, Detectron2, https: /tgpt//gg//t//////gg/gg thub. com/ 2019.
f www.kkreeerrccdeteeeeooo
[80] Z. Wu, Y. Xiong, S. X. Yu, and D. Lin, "Unsupervised feature learning via non-parametric instance discrimination." in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 3733-3742.
[81] E. Xie el al., "Detco: Unsupervised contrastive learning for object detection," in arXiv:2102. 04803, 2021.
[82] Z. Xie, Y. Lin, Z. Zhang, Y. Cao, S. Lin, and H. Hu, "Propagate yourself: Exploring pixel-level consistency for unsupervised visual representation learning," in CVPR, 2021.
[83] D. Xu. J. Xiao, Zhao, Shao. D. Xie, and Y. Zhuang, "Self-supervised spatiotemporal learning via video clip order prediction," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 10 10334-10343,
[84] Y. You, I. Gitman, and B. Ginsburg, "Large batch training of convolutional networks," arXiv preprint arXiv:1708. 10X.v:1708.03888, 2017.
[85] Y. Yu, K. H. R. Chan, C. You, C. Song, and Ma, "Learning diverse and discriminative representations via the principle of maximal coding rate reduction," Advances in Neural Information Processing Systems, vol. 33, pp. 9422 9434, 2020.
[86] J. Zbontar. L. Jing. I. Misra, Y. LeCun, and S. Deny, "Barlow twins: Self-supervised learning via redundancy reduction," in International Conference on Machine Learning, PMLR, 2021,
pp. 32310-12 320.
[87] R. Zhang, P. Isola, and A. A. Efros, "Colorful image colorization," in European conference on computer vision, Springer, 2016, pp. 649-666.
[88] Zhang, Isola, and A. Efros, "Split-brain autoencoders: Unsupervised learning by cross-channed prediction," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 1058-1067.
[89] B. Zhou, A. Khosla, A. Lapedriza, A. Torralba, and A. Oliva, "Places: An image database for deep scene understanding," arXiv preprint arXiv: 161.v2/600.02055 2016.
[90] J. Zhou et al., "Ibot: Image bert pre-training with online tokenizer," arXiv preprint arXiv:2111.07822, 2021.
14

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Please see Appendix F.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] Please see Appendix G.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] A pseudocode is provided in Appendix and code will be released. All instructions are Appendix C. (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] All implementation details described Appendix C.
(c) Did you report error bars (e-g., with respect the random seed after running experi- ments multiple times)? [Yes] We report the average performance over three seeds.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Please see Appendix C.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you eite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] Please see Appendix H.
(c) Did you include any new assets either in the supplemental or as URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating?11
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "b6QZR_vTPX", "reviewer": "Reviewer_Z5uT", "summary": "This paper proposed a joint-embedding objective called Maximum Entropy Coding, which is close to MCR^2 [74]. This method directly optimizes the information content by minimizing the coding length function. This proposed method unifies batch-wise objectives (SimSiam) and feature-wise objectives (BarlowTwins). Practically, this is implemented as a Taylor series expansion. \nThe proposed method shows strong performances (when combined with all existing techniques, including exponential moving average and asymmetric network) on a wide range of experiments, including ImageNet linear probe, semi-supervised classification, transfer learning on video tasks, and object detection. \nHowever, this paper actually uses all the tricks, including exponential moving average from BYOL, but only mentions it in the ablation study/supplementary material. It is obvious that the strong result of this method mostly comes from this mixture of existing design components.", "questions": "Do you consider n=2 (Taylor expansion order) as a novel model? If not, then how is the performance of this baseline across all experiments. If yes, how is it different from BarlowTwins?", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The proposed method is strongly supported by theoretical motivation: the maximum entropy principle. \n2. The proposed method shows strong performances on a wide range of experiments (when combined with all existing techniques, including exponential moving average, asymmetric network), including ImageNet linear probe, semi-supervised classification, transfer learning on video tasks, and object detection.\n3. The authors provided experimental details, including pseudo-code and all hyperparameters. \n\nWeaknesses:\n1. The strong result mostly comes from a mixture of designing components, e.g., exponential moving average and asymmetric architecture. It is questionable how much advantage comes from the maximum entropy coding loss. In fact, the only contribution of the proposed method over existing methods is that it has a higher-order correction. Table 5e clearly shows that the extra two orders (2->4) only increase accuracy by 0.3%.\n2. It's unclear what the main contribution is to this paper. \n(a) The \"principle of maximum entropy\" is no different from requiring independent features, which is already proposed in MCR^2, BarlowTwins, VICReg, and whitening SSL. \n(b) The higher-order approximation, as mentioned above, only has a 0.3% linear probe improvement. All other experiments fail to verify the advantage. \n(c) The practical advantage of this mixture model. The authors only compare their work to fundamental frameworks (BYOL, BarlowTwins, SwAV), but there are many similar mixture models, e.g., arXiv:2204.07141, arXiv:2104.14548, arXiv:2109.12909, arXiv:2012.13493, arXiv:2201.05119\n\n====== after rebuttal comments =====\n\nI am more convinced by the authors that the empirical advantage is misleading. The empirical advantage is exactly just 0.3%. Actually, in response point 1, Barlow Twins are supposed to reach 73.5%. So here, the extra two orders approximately indeed only provide a 0.1% improvement.\n\nI think this is totally ok. A good theory unifies several ideas and provides incremental improvements. But it is unacceptable that the paper tries to hide this point.\n\nThe main figure and most of the results tables will mislead readers that the advantage of the proposed method over Barlow Twins / SimCLR is due to the proposed loss. In fact, leveraging other tricks like momentum encoders is the main reason but is only mentioned in L159 once. Not in the figure, not in the main text, not in the experimental detail.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "WYBS_UVgqaJ", "reviewer": "Reviewer_CgUD", "summary": "This paper proposes a self-supervised learning method dubbed Maximum Entropy Encoding (MEC) which leverages the principle of maximum entropy to learn unbiased representations of an image dataset (experiments done on ImageNet). The authors combine a maximum entropy with the augmentation-invariance objective of contrastive learning, which is justified as a view consistency prior which gives testable information. \nThe exact resulting loss has a log determinant, which they approximate using a Taylor series. They show that various orders of this approximation are in fact equivalent to existing self-supervised methods such as SimSiam and Barlow Twins.\nExperiments are conducted with pretraining on ImageNet and suitably diverse downstream tasks: linear evaluation on ImageNet, semisupervised classification on ImageNEt, transfer learning on object detection (VOC + COCO) and segmentation (COCO), as well as video tracking (Table 5).", "questions": "From Strengths-Quality: Can you clarify where the increase in metrics from cited sources comes from? Was I looking in the wrong place in the cited works or are there improvements stemming from more modern implementations?\n\nFrom Weaknesses-Quality:\nAre you able to provide any results with pretraining on moderate-to-large scale datasets besides ImageNet?\nI'd also be curious if this objective can provide improvements for self-supervised learning on smaller-scale datasets (this latter point is an extra ask, if pressed for space in the rebuttal feel free to ignore)", "limitations": "Appendices E and F provide a succinct but accurate and precise description of limitations and the possible relation to societal impact", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nOriginality:\nThe community has been iteratively constructing contrastive or similar self-supervised learning methods for several years now. This paper subsumes several previous works by providing a novel unified view that allows for further adaption and exploration. While the resulting loss code is not markedly different from existing work, the theoretical approach it is derived from and the resulting flexibility/opportunity it yields constitute originality in my eyes.\n\nQuality:\nThe experiments in this paper are very clean, with straightforward well-accepted experimental settings. Appendix C describes the methodology well without introducing any complicating bells and whistles. The results of the new method are consistently equal or superior to existing objectives all while being very motivated.\nI cross-referenced numbers with original works and the authors match (or in some cases exceed) the original citations in table referenced numbers (example: the arxiv version of SimCLR lists 69.3 as its top1 imagenet acc while this work gives up to 70.4). I assume these discrepancies are from improved/additional facets in the training, but would appreciate clarification on this point.\nI feel extremely comfortable that I could replicate the experiments of this paper with minimal effort and would see comparable results.\n\nClarity:\nThis paper is extremely well-written. I was able to understand all of the concepts on the first read-through. The order of presentation was logical and descriptions were enlightening without being overly wordy. As previously mentioned, I feel comfortable that I could replicate the results quickly, especially given the useful pseudocode in Appendix A.\n\nSignificance:\nThe direct tying of a family of objectives to a very grounded mathematical concept is highly significant in my eyes as it brings the iterative line of research to a convergence point from which more novel lines of research can be taken. There could have been several more iterations of self-supervised learning papers converging to this objective by empirical motivation but this work shortcuts that and opens up exciting possibilities for future work (especially given the prominence of contrastive learning objectives in current vision-language works).\n\nMisc:\n- Figure 4 was an extremely useful illustration\n- Appendix was extremely informative and helpful\n- The ablations in Table 5 (particularly (e)) largely match my intuition which is nice\n\nWeaknesses:\nOriginality:\nThis paper consolidates existing lines of exploration and opens the door for further \"contrastive\" methods and as such the ultimate objective is not technically very different from existing work. Given the importance and significance of the consolidation though, this is not truly a weakness in my opinion.\n\nQuality:\n*I do wish that there were at least some pretraining experiments on datasets besides ImageNet (either uncurated web imagery or something like Places365). In my opinion this is the biggest weakness of this paper.*\nAs stated above, overall I was very impressed with the quality of this work. The only typo that jumped out to me while reading was \"variaty\" instead of \"variety\" in L60.\n\nClarity:\nI am very familiar with the contrastive learning so am probably a biased estimator here, but I thought the explanations and experiments were extremely straightforward and non-confusing. I have no complaints about the clarity. My only related suggestion is that a pointer to Appendix H is placed in the main text (see Misc below)\n\nSignificance:\nAs stated in the above originality section, it's difficult for a consolidation paper to have earth-shattering impact as the value largely lies in the tying together of work that necessarily exists. I don't think that this by any means is a net negative; I would rank this paper as one of the more significant ones that I've come across in the last 2-4 weeks of arxiv, but it does cap it below what a Conference Best Paper might look like.\n\n\nMisc:\n- Appendix H (starting from L724) is extremely interesting and I was wishing for a similar section within the main text. While incorporating that many lines is impractical at this stage of the paper; I would suggest that a reference to Appendix H with a preview of what it contains is added to the main text ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "W95okT9-Gr", "reviewer": "Reviewer_QNyH", "summary": "This paper aims to learn generalizable representations without labels. To this end, this paper proposes Maximum Entropy Coding (MEC), inspired by the principle of maximum entropy in information theory. MEC uses the minimal lossy coding and the Taylor series approximation to make the maximum entropy estimation feasible. Extensive experimental results show that MEC consistently outperforms existing SSL methods under various downstream tasks. Furthermore, this paper demonstrates that MEC is robust to various hyperparameters (e.g., smaller batch sizes) and architectures (e.g., ViTs).\n", "questions": "- Too small top margins in p2-3. **This seems to violate formatting rules.** This should be modified in the rebuttal period. Also, other spaces (e.g., caption spaces) seem too narrow.\n- Could you test more transfer learning experiments with fine-grained classification benchmarks? These benchmarks are also important to evaluate the transferability of the learned representations. For the transfer setup, I recommend seeing SimCLR and BYOL papers.\n", "limitations": "This paper well addressed the limitations and the potential negative societal impact.\n", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The proposed method has several strengths: (1) simplicity and scalability, (2) high performance across various downstream tasks, and (3) robustness to various hyperparameters and architectures. I think the extensive experiments well demonstrate these strengths. Furthermore, MEC can be interpreted as batch-wise and feature-wise SSL methods, so I feel this part is also interesting.\n\nOne major concern with this paper is that what information the proposed method can learn while existing SSL methods cannot is unclear. This paper mentioned that existing SSL methods introduce some \"biases\" while MEC does not. What are the biases exactly? Could you provide some empirical evidence supporting that MEC can learn less-biased representations? I think this paper does not explain why and how MEC can outperform other SSL methods.\n\nSome minor concerns are provided in the **Questions** section.\n\nTo sum up, although some explanation seems insufficient, I feel the empirical results are strong. Hence, I vote for Weak Accept.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept