Multi-Segmental Informational Coding for Self-Supervised Representation Learning
Authors: Chuang Niu
Ge Wang
Venue: Submitted to ICLR 2023 (2023)
Keywords: self-supervised learning
representation learning
unsupervised learning
deep learning

Abstract: Self-supervised representation learning aims to map high-dimensional data into a compact embedding space, where samples with similar semantics are close to each other. Currently, most representation learning methods maximize the cosine similarity or minimize the distance between different views from the same sample in an $\ell^2$ normalized embedding space, and reduce the feature redundancy via a linear correlation constraint. In this study, we propose MUlti-Segmental Informational Coding (MUSIC) as a new embedding scheme for self-supervised representation learning. MUSIC divides an embedding vector into multiple segments to represent different types of attributes, and each segment automatically learns a set of discrete and complementary attributes. MUSIC enables the estimation of the probability distribution over discrete attributes and thus the learning process can be directly guided by information measurements, reducing the feature redundancy beyond the linear correlation. Our theoretical analysis guarantees that MUSIC learns transform-invariant, non-trivial, diverse, and discriminative features. MUSIC does not require a special asymmetry design, a very high dimension of embedding features, or a deep projection head, making the training framework flexible and efficient. Extensive experiments demonstrate the superiority of MUSIC. 

Under review as a conference paper at ICLR 2023
MULTI-SEGMENTAL INFORMATIONAL CODING FOR SELF-SUPERVISED REPRESENTATION LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Self-supervised representation learning aims to map high-dimensional data into a compact embedding space, where samples with similar semantics are close to each other. Currently, most representation learning methods maximize the cosine sim- ilarity or minimize the distance between different views from the same sample in an l² normalized embedding space, and reduce the feature redundancy via a linear correlation constraint. In this study, we propose MUlti-Segmentall Informational Coding (MUSIC) as a new embedding scheme for self-supervised representation learning. MUSIC divides an embedding vector into multiple segments to rep- resent different types of attributes, and each segment automatically learns set of discrete and complementary attributes. MUSIC enables the estimation of the probability distribution over discrete attributes and thus the learning process can be directly guided by information measurements, reducing the feature redundancy beyond the linear correlation. Our theoretical analysis guarantees that MUSIC learns transform-invarian non-trivial, diverse, and discriminative features. MU- SIC does not require special asymmetry design, a very high dimension of embed- ding features, or deep projection head, making the training framework flexible and efficient. Extensive experiments demonstrate the superiority of MUSIC.
INTRODUCTION
Self-supervised representation learning (SSRL) is now recognized as a core task in machine learn- ing with rapid progress over the past years (Bengio et al., 2013; LeCun et al., 2015). Deep ral networks pre-trained on large-scale unlabeled datasets via SSRL have demonstrated impressive characteristics, such as strong robustness (Hendrycks al., 2019; Liu al., 2021) and generaliz- ability (Mohseni et al., 2020), and improving various downstream tasks. Among various pretext tasks, an effective approach is drive semantically similar samples (i.e., different transformations of the same instance) close to each other in the embedding space (Dosovitskiy et al., 2014; Wu et al., 2018; Tian et al., 2020b; et al., 2019; Dwibedi et al., 2021). Simply maximizing the similarity or minimizing the Fuclidean distance between embedding features of semantically similar samples tends to produce trivial solutions; e.g., all samples have the same embedding features. Recently, various excellent methods have been proposed to learn meaningful representations feature and avoid trivial solutions. Contrastive learning (Hadsell et al., 2006; Oord et al., 2018) based methods, such as SimCLR (Chen et al., 2020a;b) and MoCo (He et al., 2020), have achieved great success by additionally minimizing the similarity between embeddings of the reference and negative samples, which requires either relatively large batches Maaten, 2020) for negative samples. To avoid using negative samples, BYOL (Grill et al., 2020) and SimSiam (Chen & He, 2021) developed clever techniques, such as the asymmetry network ar- chitecture, stop gradients, and momentum weight updating. Subsequent theoretical analysis (Wang & Isola, 2020; Zhang et al., 2021a; Richemond et al., 2020; Tian al., 2021) have demonstrated why these techniques avoid trivial solutions and learn meaningful representations from different as- pects. Clustering-
large memory bank (Wu et al., 2018; Misra &
methods DeepCluster (Caron et al., 2018), SELA (Asano et al., 2019), SWAV (Caron al., 2020) alternatively compute the cluster assignment of one view and optimize the network to predict the same assignment for other views of the same sample, where trivial solu- tions can be avoided via the
assignment of samples over different clusters. In another direction, W-MSE (Ermolov et al., 2021) and Barlow Twins (Zbontar et al., 2021) propose to drive self- or cross-correlation matrices towards the identity matrix, reducing the feature redundancy and learning

---

Under review as a conference paper at ICLR 2023
0.33 DUO
Sa--:Okjer Object Part
Seg-:: Testure
Say-S: Shape
Figure I: Illustration of the MUSIC vector with piano keys. The MUSIC embedding vector consists of multiple segments (Seg-1, ... Seg-S) representing different types of attributes shown in different colors. Each segment is associated with a set of discrete attributes; e.g., Seg-2 represents the texture attribute, and different units in 2 specify different textural patterns, like dots, stripes, etc. Each segment is discretized with a one-hot vector (8,:).
meaningful features without requiring the asymmetry design. Most recently, VICReg (Bardes et al., 2022) constructs novel loss function with three terms, i.e., invariance, variance, and covariance constraints that explicitly suppress trivial solutions. A theoretical study (Shwartz-Ziv et al., 2022) has given a deep analysis on why VICReg works well.
Fundamentally different from the current SSRL methods that normalize embedding features onto the unit hypersphere via 6² norm and use cosine similarity as the metric, we propose MUlti-Segmental Informational Coding (MUSIC) for representation learning in novel way. The motivation is based on the observation that an object can be represented by a set of attributes (Russakovsky & Fei-Fei, 2010). As illustrated in Fig. 1, we construct an embedding vector with multiple segments to rep- resent different types of attributes; e.g., Seg-1, Seg-2, and Seg-S represent object configuration, texture, and shape, respectively, and each segment instantiates set of specific attributes; e.g. Seg-2 represents samples with different textural patterns (dots, stripes, etc.). In other words, we discretize the feature variable by segment of the one-hot vector implemented with softmax function, and the whole embedding vector consists of multiple such one-hot vectors. By doing so, MUSIC makes discrete units of each segment that the infor-
possible to estimate the probability distribution mation measures defined on probability distributions can be directly computed for both optimization and theoretical analysis. Two general properties desired behind the illustration in Fig. 1) sam- ples can be classified into a set of different and discrete attributes in each segment; and 2) different segments discriminate samples using different classification criteria, which means that the mutual information between different segments is minimized, or equivalently the information/eetrooy of embedding features is maximized. To automatically learn such MUSIC embeddings for SSRL, we propose an entropy-based loss function based on the empirical joint probability distribution. Our cermaation-thoorede analysis reveals why such mcaningful features can be promoted while trivial solutions are avoided, which are consistent to the qualitative results in Appendix C.
The contributions of MUSIC are as follows. (1) MUSIC presents a new embedding scheme and enables new information -teooretie optimization framework for SSRL. (2) Theoretical analysis sures that the MUSIC embeddings are optimized to be transform- invarriat,, non-trivial, diverse, and discriminative. Importantly, MUSIC minimize any form of dependency between feature vari- ables beyond the linear correlation in current methods (7bontar et al., 2021; Bardes et al., 2022). (3) Similar to Barlow Twins and VICReg, MUSIC does not require an asymmetry network architecture, negative samples in a large batch or a memory bank, gradient stopping, or momentum updating. (4) MUSIC does not need very high dimension of embedding features or a deep projection head, significantly reducing the computational cost. (5) Extensive experimental results demonstrate the superiority of MUSIC on representative datasets in various evaluation settings.
METHODOLOGY
2.1 SELF-SUPERVISED LEARNING FRAMEWORK
Similar to W-MSE and Barlow Twins, in this study we adopt twin architecture to learn the em- bedding features. where the same network is shared between two branches, as shown in Fig. 2. During training, input images X [ are mapped to two distorted sets X' (a() and
2

---

Under review as a conference paper at ICLR 2023
57
52
X'
Encoder
Projector
51
S2
S3
S4
53
51
S4
Multi-Segmental
Transform
Informational Coding
S2
S7
.2
Encader
Projector
S4
53
Maximize Joint Entropy Orange Elements
54
SSRL framework through multi-segmenntl informational coding optimized with maximum entropy. Here there are four segments and each segment consists of four units for illustration.
Figure 2:
x" (a]) where N is the batch size. A common transformation distribution, which covers random crops combined with color distortions, the same as that in (Bardes et al., 2022), is used to generate training samples. Then, the two of distorted images X' and X" are respectively fed to two branches, each of which consists of an encoder f(; 0y) and a projector g( 0g), where 0 and 0g respectively denote the parameters of the encoder and projector to be optimized. The outputs of the encoder are commonly used as the representation features. The projection head maps the representation features into the embedding space. Note that the presented method is not limited to this twin architecture, which can be extended to the two branches with different parameters, hetero- geneous networks, or even different input modalities (e.g., text, audio, etc.) as studied in VICReg.
2.2 MULTI-SEGMENTAL INFORMATIONAL CODING
Segmi--egmental Informational Coding (MUSIC) is novel embedding scheme for SSRL. The embedding features of two transformed images are x = g ((f(2);; 0g) E RD, and x g(f(=":;p) ()))),,)) E R respectively, where D is the feature dimension. Most of the existing SSRL methods normalize embedding features in the l norm and then maximize their cosine similarity between the two transformed versions. Motivated by the observation described in Fig. 1, we divide the embedding feature z; into multiple segments to represent different types of attributes, denoted by (s, d), 1, S. d 1.... D.. where S is the number of segments, D is the dimension of the 8t segment. In this study, we evenly split the embedding vector, i.e., Ys, D, = Ds. and the dimension of the whole embedding space is D Dş S. In principle, uneven partitions can be applied as well given prior knowledge. To make attributes discrete and complementary, one-hot encoding applied each segment. Specifically, each segment normalized score vector using the softmax function:
dis'' d') = exp(2 ('',
(1)
Eee exp(2((s' (s),
where dd(s'd")) denotes the score of the image x! belongs to the d'-th instantiated attribute in the -thh segment. The score vector q (s("",) for the other branch is computed in the same way. Thus, the MUSIC scheme can be interpreted as a combination of multiple classifiers or cluster operators that implement different classification criteria learned in data-driven fashion.
2.3 ENTROPY Loss
Since each type of attribute has a finite number of discrete instantiations, it is possible to estimate the probability distribution of set of samples over each and every segment. Specifically, we can compute the empirical joint distribution P(s',66 s" d'' d')) between every two instantiated attributes within and across segments based on set of samples as follows:
N
1
P(s' "'''."".'' d'dd) = N
(s's d''',d'n d')q""""" dd")
(2)
1=1

---

Under review as a conference paper at ICLR 2023
where P(s', s,,"', dd''''' dd") is computed the statistical frequency of the sample having both the attribute-d" in the segment-s' and the attribute-d" in the segment-s" over N samples. With the em- pirical joint probability distribution, coormation-theorette metrics can be directly computed. Here two versions of the loss function are defined. The first version Lent is a pure joint entropy loss:
Dg Ds 3 20
1
Lent
P(8 8" d' d²)) log( P(s', , d))) (3)
52
where 1s"=s",d'' is an indicator function that equals to if = s" and d' + d"; otherwise, it equal to O. The empirical joint distribution can be denoted by block matrix as shown in Fig. 2, where (1 ,===""d'' means keeping the diagonal elements of the diagonal blocks and all elements of the off off-diagonal blocks, as indicated by the orange area. Therefore, minimizing this loss function is maximizing the joint entropy over the selected elements. The following subsection will demonstrate the properties of the embedding features learned by optimizing this loss function.
To enhance the transformation invariance of features, we introduce an additional term to maximize the inner product between the embedding features from two transformations. Then, the second version of the loss function is defined as
N ΣΣ 1=1
De
I. = Lent NS
log(g dsd"(s d)),
(4)
where balancing factor. By default, we set X which is in principle neither too small nor too large for a good balance. Minimizing the transformation invariance loss enforces the em- bedding features between two transformations of the same image to be consistent and encourages the embedding feature within each segment to be one-hot. Clearly, this additional term promotes transformation invariance and confident assignments over different attributes. Different from the statistical entropy measurement, this transformation invariance term imposes sample-specific con- straint. The transformation invariance can be also achieved by minimizing the cross-entropy between the two embedding vectors, i.e., NS Σ E Eds (s, d) log( ds))). However, by using the cross-entropy the performance would be degraded, reported in Subsection 4.2.
Our proposed method can be easily implemented, with PyTorch-style pseudo-code in Appendix A. Next, let us theoretically analyze why the entropy loss optimizes informational embedding features as illustrated in Fig. 1.
2.4 ANALYSIS
The entropy loss function consists of two parts, including the entropy over diagonal elements of diagonal blocks and the entropy over all elements of off-diagonal blocks as illustrated by the orange area in Fig. 2, and can formally expressed as
Lent Σ
Σ
P(s.s"" d' d') d"' d"d."
80 s''="" " d!!
(5)
E EPdd, d' d') log P( d'
+ S(S S(S-1) 1)
4'' d'dd"
For the first part, can be demonstrated that its optimal solution is that Vi, s, d, (( = d. (s, d). :s and q²' (s) are one-hot vectors, the statistical probability of the th attribute type taking the dth instantiation is p(s, d) = N ΣN (8, d) = De and P(s, 8, d, d) = Dg The proof can be found in Appendix B. For the second part, it is intuitive that the optimal solution to maximize the joint entropy over the off-diagonal block items is Vs', s" d', d" 8' & s", P(s', 8" dd' ) (DS)
i... batch of samples are evenly assigned over each off-diagonal block.
Transformation invariance: The solution that Vi, :s) q (s) are one-hot vectors means that the learned MUSIC embeddings are invariant transformations, and a sample tends to be confidently represented by a single instantiated attribute within each and every segment.
Non-trivial solution: The solution that + EA g, d) D5 means that a batch of samples are evenly assigned over different attributes in each segment as (s,:) and q.' (s.:) are one-hot vectors.
4

---

Under review as a conference paper at ICLR 2023
Thus, the trivial solution that all samples have the same embedding features be avoided. The discriminative encoding analyzed below also ensures MUSIC emebddings are non-trivial.
Minimum redundancy: As described in Fig. 1, different segments of the MUSIC embedding vector are expected to focus on diverse and complementary attributes. In other words, the redundancy or mutual information between any segments should be minimized, which is popular measure for feature selection (Peng et al., 2005). Specifically, it can be demonstrated that the redundancy or mutual information between any two segments is minimized when the optimal solution obtained. Specifically, the mutual information (s'',") 8") between any two segments s and s" is
1($',5" = =s')) + 1I((s())))))))) s(',
Dg Σ p'(s', log(p' (s'p ,())
Dg Σ d" d"" log(p" dos(""""
d"=1
D3 Ds
(6)
P(s', ss, d', d: log ,('',"" dd,'""))))
d=1dd==
1 0.
-- log Dg
log
+ log (Ds)²
Ds
Thus, the mutual information is minimized and diverse attributes are learned. From another per- spective, Vs', s", d', d" s * s", we have P(s', s" d', dd)) TD3 p(s', d'' d'(" dd",, and thus the variables qs',, , d'') and q(s", dd") are independent. The redundancy constraint was studied for W-MSE, Barlow Twins, and VICReg by minimizing the covariance linear correlation. In con- trast, our antropy-based loss function reduces the redundancy or mutual information in a non-linear way. Moreover, it can be derived that the optimal MUSIC embedding features have zero covariance between any two features in different segments and negative covariance between the features within the same segment; for details, please see Appendix B.
Discriminative encoding: Contrastive learning or instance discrimination has proven very effective for representation learning by maximizing the similarity between different transformations of the same instance while discriminating the reference from other instances. It is underlined that MUSIC consistent to contrastive learning and discriminating instances in a novel way. Specifically, the optimal MUSIC embedding can totally encode (Ds)² different samples. In our default settings Ds = 80, S 102 (See Section for details), MUSIC can represent 80102 different samples. The maximized the joint entropy means that any two units from every two segments have the equal pos- sibility to co-occur, that is, a batch of samples are evenly assigned to all possible embeddings. Since the number of all possible embeddings is much larger (2,048 80202) than the batch size, it will be enforced to encode different instances with different embeddings. Like contrastive learning, it en- sures non-tric non-trivial solutions. The difference lies in that contrastive learning differentiates instances by pushing the reference away from its negative instances, while MUSIC intrinsically assigns instances with different attribute codes in an information-pane prrinippled manner.
In Appendix C, the individual MUSIC embeddings and the empirical joint probability matrix learned on the ImageNet dataset are visualized, showing that the empirical results are consistent with the above theorectical analysis. In summary, the MUSIC embedding features optimized with the entropy-based loss are transform-invariann, non-iiivial, diverse, and discriminative.
3 IMPLEMENTATION DETAILS
For fair comparison, we closely followed the implementation settings in VICReg to train MUSIC models. Specifically, the standard ResNet-50 backbone (He et al., 2016) was used as the encoder that outputs representation vector of 2,048 units in the same training settings, including the data augmentation (random cropping. horizontal flip, color jittering, grayscale, Gaussian blur, solariza- tion, with the same parameters in VICReg), the optimizer of LARS (You al., 2017; Goyal et al., 2017) with a weight decay of 10 and the learning rate of Ir = batch_size, size/256 2566 x base_rr, and the cosine decay schedule (Loshchilov & Hutter, 2016) from 0 with 10 warmup epochs towards the final value of 0.002. The base learning rate base ir was set to 0.6 in our study. By default, we used two-layer MI. projector (8, 1,289,18,111 160)), the number of segments S 102, the segment dimension
5

---

Under review as a conference paper at ICLR 2023
Ds 80, and D = Ds x 8, 160 (similar the feature dimension of 8, 192 used by VICReg and Barlow Twins). The results were respectively analyzed for different feature dimensions, depths of projectors. batch sizes, segment dimensions, and numbers of training epochs. The effect of the single extra hyperparameter Dy of MUSIC was evaluated as well. The SSRL models were trained on the 1,000-classes ImageNet dataset without labels and evaluated in various downstream tasks. All implementation details for downstream tasks are in Appendix G. Code will be made available.
4 RESULTS
4.1 EVALUATION RESULTS IN DIFFERENT TASKS
Table 1: Comparison of SSRL methods on ImageNet for linear and semi-supervised classifica-
tion. Top-1 and Top-5 accuracies (in %) are reported. The best three results are underlined.
Linear Classification
Semi-supervised Learning
Methods
Top-1
Top-5
Top-1
Top-5
1%
10%
1%
10%
Supervised
76.5
25.4
56.4
48.4
80.4
MoCo (He et al., 2020)
60.6
57.2
83.8
PIRL (Misra & Maaten, 2020)
63.6
CPC v2 (Henaff, 2020)
63.8
CMC (Tian et al., 2020a)
66.2
SimCLR (Chen et al., 2020a)
69.3
89.0
48.3
65.6
75.5
87.8
MoCo v2 (Chen et al., 2020c)
71.1
90.1
SimSiam (Chen & He, 2021)
71.3
SWAV (Caron al., 2020)
71.8
InfoMin Aug (Tian et al., 2020b)
73.0
91.1
BYOL (Grill et al., 2020)
74.3
91.6
53.2
68.8
78.4
89.0
Barlow Twins (Zbontar et al., 2021)
73.2
91.0
55.0
69.7
79.2
89.3
VICReg (Bardes et al., 2022)
73.2
91.1
54.8
69.5
79.4
89.5
MUSIC (Ours)
73.6
91.4
54.0
69.1
78.9
89.1
Linear Classification on ImageNet. Linear probing is the commonly used evaluation protocol that trains linear classifier top of the frozen representations to evaluate the performance of SSRL. methods. Being consistent with Barlow Twins and VICReg. ResNet-50 backbone was trained with the batch size of 2,048 for 1,000 epochs the training set of ImageNet, and the linear classification results including Top-1 and Top-5 accuracies of different methods on the evaluation set are reported in Table 1. The difference from Barlow Twins and VICReg is that MUSIC used two-layer MLP projector (8, 192-8,1 160) instead of three layers (8, 192-8, 192-8,1911191 on par with the state of the method BYO that uses asymmetric techniques, such additional predictor and a momentum encoder. Note that the results of some excellent methods (Caron et al., 2020; Gidaris al., 2021) based on crattiicroommmliiipiie positivve techniques are included in Table 1. These techniques can usually boost performance further. The comparative results show that MUSIC achieves better results than Barlow Twins and VICReg, where all these three methods trained a twin architecture without using negative pairs or any asymmetric techniques.
The performance of MUSIC is
Semi-Supervised Classification on ImageNet. We also evaluated MUSIC in the semi-supervised learning setting, where the pre-trained ResNet-50 with MUSIC was fine-tuned on subsets of Ima- geNet, including 1% and 10% the full ImageNet dataset respectively, and all reported methods used the same subset images. Currently, MUSIC is not good Barlow Twins and VICReg in the semi-supervised learning settings, while better than BYL and other compared methods, and significantly better than purely supervised learning without SSRL pretraining.
Transfer Learning. Transfer learning is another popular way for the evaluation of SSRL methods, including object detection, instance segmentation, and linear classification. Our results are reported in Table 2. It is noted that various studies have different setups for the object detection and instance segmentation tasks. Here closely followed (Zbontar al., 2021) selecting the same compari-

---

Under review as a conference paper at ICLR 2023
Table 2: Transfer Learning. For object detection and instance segmentation tasks, SSRL models pre-trained on ImageNet were used to initialize the backbone of the object detection and instance segmentation models on COCO. Mask R-CNN (He al., 2017) with the C4 backbone variant (Wu et al., 2019) was fine-tuned using the schedule. AP metrics defined by COCO are reported here. For the linear classification task, Top-1 accuracy (in %) for Places205 (Zhou et al., 2014) and mAP for VOC07 (Everingham et al., 2010) are based on the frozen representations pre-trained on ImageNet. The best results are in bold.
Object Detection
Instance Segmentation
Linear Classification
Methods
Aph
AP
AP
Apm
AP
AP
VOC2007
Places205
Sup.
38.2
58.2
41.2
33.3
54.7
35.2
87.5
53.2
MoCo-v2
39.3
58.9
42.5
34.4
55.8
36.5
86.4
51.8
SWAV
38.4
58.6
41.3
33.8
55.2
35.9
86.4
52.8
SimSiam
39.2
59.3
42.1
34.4
56.0
36.7
BT
39.2
59.0
42.5
34.3
56.0
36.5
86.2
54.1
MUSIC (Ours)
39.3
59.1
42.6
34.4
55.8
36.6
86.5
54.8
son methods in the same settings. MUSIC performs par with the current methods and slightly better than Barlow Twins on the object detection and segmentation tasks. On the other hand, the linear classification results on VOC2007 and Places205 datasets show that MUSIC achieved better results than the selected methods. Also, similar the other SSRL methods, MUSIC can effectively improve the downstream tasks in the transfer learning settings. All implementation details for the reproduction of transfer learning results are in Appendix G.2.
Table 3: KNN classification. Top-1 accuracy with 20 and 200 nearest neighbors are reported. The best results highlighted in bold.
Table 4: Batch Size. Top-1 accu- racy (in %) results for linear classi- fication on ImageNet were obtained based on ResNet50 with 100 pre- training epochs. The best results are highlighted in bold.
Method
20-NN
200-NN
NPID (Wu et al., 2018)
46.5
LA (Zhuang al., 2019)
49.4
PCL (Li et al., 2021)
54.5
Batch Size 512
1024
2048
4096
BYOL (Grill et al., 2020)
66.7
64.9
SWAV (Caron al., 2020)
65.7
62.7
SimSiam
68.1
68.0
67.9 9
64.0
BT (Zbontar et al., 2021)
64.8
62.9
VICReg
68.2
68.3
68.6
67.8
VICReg (Bardes et al., 2022)
64.5
62.9
MUSIC (Ours)
67.0
64.9
MUSIC
68.3
69.3
69.4
68.7
KNN Classification ImageNet. Another common protocol for evaluating representation learn- ing methods is by K-Neerest-NNiihbbrrs (KNN) classification on ImageNet. We followed the recent studies (Wu al., 2018: Zhuang al., 2019; Caron al., 2020; Bardes al., 2022) that built KNN classifiers with the learned representations on the training set of ImageNet and evaluated the KNN classification results on the validation set ImageNet. The results with 20 and 200 nearest neighbors are reported in Table 3, showing that MUSIC achieved the best performance among the comparison methods. Since the KNN classifier determines the class of a sample by directly search- ing its nearest samples in the feature space, the representation features learned by MUSIC is more semantically similar to each other among the nearest neighbors than those learned by other meth- ods. Thus, MUSIC has the potential superiority when applied to the downstream tasks based on the nearest neighbors search.
All the above results demonstrate the effectiveness and superiority of MUSIC strategy with the information theoretic optimization framework. In the following subsections, the characteristics and superiority of MUSIC will be further discussed.
embedding

---

Under review as a conference paper at ICLR 2023
4.2 EMPIRICAL ANALYSIS
In this subsection, we comprehensively evaluate the proposed MUSIC method in various settings and compare it with other SSRL methods if the corresponding results in the same or comparable settings were already reported. All the models were evaluated with linear classification on ImageNet.
Effect of Batch Size. SSRL methods usually require large batch size or a memory bank especially for contrastive learning. Here we evaluated MUSIC with different batch sizes and the results are reported in Table 4. shows that MUSIC achieved consistently better results than the latest method VICReg over different batch sizes. As discussed in Subsection 2.4, an intrinsic property of MUSIC to discriminatively encode different instances, making work well without large number of contrastive samples.
Table 5: Training epochs. Top-1 accuracy (in %) of linear classification ImageNet using ResNet- 50. The best results are highlighted in bold while the second best results are underlined.
Methods
SimCLR
MoCo v2
BYOL
SwAV
SimSiam
MUSIC
100 epochs
66.5
67.4
66.5
66.5
68.1
69.4
200 epochs
68.3
69.9
70.6
69.1
70.0
71.8
400 epochs
69.8
71.0
73.2
70.7
70.8
73.1
800 epochs
70.4
72.2
74.3
71.8
71.3
73.4
Effect of Epoch Number. The SSRL methods in different studies do not always use the same training epochs due to different computational costs and environments. MUSIC was evaluated with
different numbers of training epochs as reported in Table 5. MUSIC is consistently better than most of the existing methods on all different training epochs. When the numbers of training epochs were small (100 and 200), MUSIC can converge to the best results.
Table 6: Projector Depth. The best results are highlighted in bold.
Table 7: Feature Dimension. The best Top-1 accuracies are highlighted in bold.
Depth
2
3
4
DVICKey
1024
2048
4096
8192
16384
DMUSIC
960
2000
4080
8160
16320
Top-1
69.4
68.5
67.9
VICReg
62.4
65.1
67.3
68.6
68.8
Top-5
89.3
88.3
87.9
MUSIC
64.1
66.6
69.2
69.4
69.1
Time/100ep
8.5h
9.6h
10.8h
Time/100en
7.6h
7.7h
8.0h
8.5h
10.9h
Memory/GPP
10.4G
11.5G
12.5G
Memory/GPU
7.6G
8.0G
8.5G
10.4G
15.9G
Zbontar et al., 2021; Bardes et al., 2022) show that using three-layer MLP as the projector achieved the best results. However, MUSIC has different behavior that two-layer MLP achieved the best results shown in Table 6. may be due to the discriminability and diversity of MUSIC embeddings, allowing it to learn infor- mational representations more effectively. At the same time, the computational cost be reduced, especially for the fully-connected MI.P with nigh-dimensional inputs and outputs. The running time per 100 epochs and the peak memory per GPU for different projector depth are reported in Table 6, where the computational environment is described in Appendix E. Moreover, the comparison re- sults of different methods in Appendix F show that MUSIC cannot only reduce the running time and memory cost but also achieves better performance than Barlow Twins and VICReg.
Effeet of Projector Depth. The existing studies (Chen & He, 2021;
Effect of Feature Dimension. In the previous Barlow Twins and VICReg studies, it was found that a very high-dimensional embedding vector is necessary for improving the representation learn- ing performance. For MUSIC, the feature dimension plays an important role as well. The results of different feature dimensions for VICReg and MUSIC are reported in Table 7, where the the di- mensions of MUSIC embeddings are similar to those of VICReg embeddings while keeping the dimension of each segment the same, Ds = 80. It can be seen that MUSIC achieved consistently better results than VICReg on different embedding feature dimensions. Importantly, when the em- bedding feature dimension was reasonably large (4,09 and 8, 191),,, MUSIC achieves the best results that are even better than the best results of VICReg using the larger dimension of 16,384. This is because that minimizing linear correlation by the existing methods cannot ensure the minimized
8

---

Under review as a conference paper at ICLR 2023
Table Loss terms. The best results are highlighted in bold.
Table 9: Segment Dimension. The best results are highlighted in bold.
Loss
DE+OE
OE+TI DE+OE+TIC DE+OE+TI
Ds 32
64
80
96
128
Top-1
65.4
64.1
68.3
69.4
Top-1 67.8
69.1
69.4
69.2
68.4
Top-5
86.9
86.4
88.6
89.3
Top-5 88.5
89.1
89.3
89.1
88.5
non-linear dependency while MUSIC can minimize any form of dependeney between any two fea- ture variables. Therefore, the redundancy between MUSIC feature variables tends to be lower than the existing methods that feature dimension can be reduced for even better results. In principle, the large embedding feature dimension (i.e.. 16,384) significantly increases the computational and memory cost for Barlow Twins. VICReg, and MUSIC that compute the covariance or joint proba- bility matrix, which was also discussed in the Barlow Twins study (Zbontar et al., 2021). )) This point demonstrated in Table 7 by evaluating running time and memory cost, where the computational environment is described in Appendix E. Thus, MUSIC is both efficient and effective.
Effect of Loss Function. The effect of different loss terms was evaluated in Table 8, where DE, OE, TIC, and TI denote the diagonal entropy loss, off-diagonal entropy loss, transformation invariance loss implemented with cross-entropy, and transformation invariance loss implemented with inner- product, respectively. As described in Subsection 2.4, only optimizing the entropy loss (DE+OE) allows MUSIC to avoid trivial solutions and learn informational representations. This theoretical analysis is consistent with the empirical results in Table 8 that 65.4% Top-1 was achieved using the entropy loss only, comparable to some methods reported in Table Adding the enhanced transformation invariance constraint in instance-
significantly improved the performance, as also discussed the Subsection 2.3. Without adding the DE loss, the results were significantly degraded, the DE loss not only enhances the transformation invariance but also helps learn non- trivial and complementary attributes in each and every segment. our experiments, minimizing the cross-entropy degraded the performance compared with the mnner-product implementation.
Effect of Segment Dimension. Finally, the effect of our unique hyperparameter, i.e., segment di- mension, evaluated. Our empirical results with different segment dimensions in Table 9 indicate that Ds = 80 achieved the best results, where the dimension of the whole embedding vector was kept the same. It shows that the representation performance is not sensitive to this hyperparameter.
DISCUSSIONS AND CONCLUSION
From pairwise independence to mutual independence. Although MUSIC is minimizing the pair- wise independence, the minimum mutual independence among multiple feature variables cannot be ensured (Gallager, 2013). In other words, redundancy may still exist among multiple feature vari- ables. Similar to the study (Niu & Wang, 2022) that imposes high-order moment constraints on the embedding features, maximizing joint entropy among multiple variables could be implemented to reduce the redundancy further for even better self self-learning performance.
From representation learning to hierarchical clustering. Each segment in MUSIC can be re- garded as a clustering head, while different segments promote different and independent clustering criteria. In this study, we mainly focus on the general representation learning task, where each seg- ment has the same number of clusters and every two segments are independent. This idea could be extended to hierarchical clustering; e.g., different segments may have different numbers hierar- chical clusters and the independence constraint between segments can be adapted with task prior.
More discussions and comparisons between MUSIC and related methods are Appendix
In conclusion, we have presented the multi-segment informational coding (MUSIC) scheme that discretizes feature variables and a new cettmattion-hheorede SSRL framework. Theoretical analysis ensures that the optimized embedding features are transform-invariant. non-trivial, diverse, and dis- criminative. Importantly, MUSIC can minimize any form of dependency between feature variables beyond the linear correlation in current methods. Various evaluation results have clearly shown the effectiveness and superiority of MUSIC in terms both accuracy and efficiency. MUSIC could be adapted more downstream tasks, such clustering and dense prediction tasks.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Yuki Markus Asano, Christian Rupprecht, and Andrea Vedaldi. Self-labelling via simultaneous clustering and representation learning. orXiv:/91.00537/ 2019.
Adrien Bardes, Jean Ponce, and Yann LeCun. Unsupervised representation learning by predicting image rotations. In ICLR, 2022.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions pattern analysis and machine intelligence, 35(8):1798-15) 1828, 2013.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsu- pervised learning of visual features. In ECCV, pp. 132-149, 2018.
Mathilde Caron, Piotr Bojanowski, Julien Mairal, and Armand Joulin. Unsupervised pre-training of image features on non-curated data. In ICCV, pp. 2959-2968, 2019.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. NeurIPS, 33:9912- 9924, 2020.
Pengguang Chen, Shu Liu, and Jiaya Jia. Jigsaw clustering for unsupervised visual representation learning. In CVPR, pp. 11526-1155 11535, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. ICML, volume 119, pp. 1597-1607, 2020a.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. Big self-supervised models are strong semi-supervised learners. NeurIPS, 33:22243-22255, 2020b.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR, pp-
15750-15758 June 2021.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv:2003.04297, 2020c.
Carl Doersch, Abhinav Gupta, and Alexei Efros. Unsupervised visual representation learning by context prediction. In ICCV, pp. 1422 1430, 2015.
Alexey Dosovitskiy, Jost Tobias Springenterg, Martin Riedmiller, and Thomas Brox. Discriminative unsupervised feature learning with convolutional neural networks. NeurIPS, 27, 2014.
Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. With little help from friends: Nearest-neighbor contrastive learning of visual representations. In ICCV, pp. 9588-9597 2021.
Aleksandr Ermolov, Aliaksandr Siarohin, Enver Sangineto, and Nicu Sebe. Whitening for self- supervised representation learning. In ICML, pp. 3015-3024 2021.
Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88(2): 303-338, 2010.
Robert G Gallager. Stochastic processes: theory for applications. Cambridge University Press, 2013.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In ICLR, 2018.
Spyros Gidaris, Andrei Bursuc, Nikos Komodakis, Patrick Pérez, and Matthieu Cord. Learning representations predicting bags of visual words. In CVPR, pp. 6928-6938, 2020.
Spyros Gidaris, Andrei Bursuc, Gilles Puy, Nikos Komodakis, Matthieu Cord, and Patrick Perez. Obow: Online wiisal-visualwwwodd generation for seff-super learning. CVPR, pp. 6830- 6840, 2021.
10

---

Under review as a conference paper at ICLR 2023
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, An- drew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv: 706. neeee:7776..27.,.0 2017.
Bean-Bastiens Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. NeurIPS, 33:21271- 21284, 2020.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. In CVPR, volume 2, pp. 1735-1742, 2006.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, June 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV, pp. 2961- 2017.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, June 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, pp. 16000-16009, 2022.
Olivier Henaff. Dfta-efficient image recognition with contrastive predictive coding. ICML, pp- 4182-4192, 2020.
Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song. Using self-supervised learning can improve model robustness and uncertainty. NeurIPS, 32, 2019.
Jiabo Huang, Qi Dong, Shaogang Gong, and Xiatian Zhu. Unsupervised deep learning by neigh- bourhood discovery. In ICML, pp. 2849-2858, 2019.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521(7553))36-4444 2015.
Junnan Li, Pan Zhou, Caiming Xiong, and Steven Hoi. Prototypical contrastive learning of unsuper- vised representations. In ICLR, 2021.
Hong Liu, Jeff Z HaoChen, Adrien Gaidon, and Tengyu Ma. Self-supervised learning is more robust to dataset imbalance. arXiv:21 2021.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv: 03383, 2016.
Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant representa- tions. In CVPR, pp. 6707-6717, 2020.
Sina Mohseni, Mandar Pitale, JBS Yadawa, and Zhangyang Wang. Self-supervised learning for generalizable out-of-distribution detection. In AAAI, volume 34, pp. 5216-5223, 2020.
Chuang Niu and Ge Wang. Home: High-order mixed-moon -ot---mmmmeettteeee based embedding for representation learning. 0rXiv:2207.07743, 2022.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning visual representations by solving jigsaw puzzles. In ECCV, pp. 69-84, 2016.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv:1807.03744 2018.
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In CVPR, pp. 2536-2544, 2016.
11

---

Under review as a conference paper at ICLR 2023
Hanchuan Peng, Fuhui Long, and Chris Ding. Feature selection based mutual information criteria of max dexeendennency max-releeance, and redundancy. IEEE Transactions on pattern analysis and machine intelligence, 27(8):1226-13 1238, 2005.
Pierre Richemond, Jean-Bastien Grill, Florent Altché, Corentin Tallec, Florian Strub, Andrew Brock, Samuel Smith, Soham De, Razvan Pascanu, Bilal Piot, et al. Byol works even without batch statistics. arXiv:2010.10211,222 2020.
Olga Russakovsky and Li Fei-Fei. Attribute learning in large-scale datasets. In ECCV, pp. 1-14. Springer, 2010.
Ravid Shwartz- Ziv, Randall Balestriero, and Yann LeCun. What do we maximize in self-supervised learning? arXiv:2207. 10081, 2022.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding, In ECCV, pp- 776-794. Springer, 2020a.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? In NeurlPS, volume 33, pp. 6827-6839, 2020b.
Yuandong Tian, Xinlei Chen, and Surya Ganguli. Understanding self-supervised learning dynamics without contrastive pairs. In ICML, pp. 10268-10278, 2021.
Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. Feos: Fully convolutional one-stage object detection. In ICCV, pp. 9627-9636, 2019.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In ICML, pp. 1096-1 1096-1103, 2008.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In ICML, volume 119, pp. 1929-9939, 2020.
Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li. Dense contrastive learning for self-supervised visual pre-training. In CVPR, pp. 3024-3033 2021.
Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. Detectron2. 2019.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. CVPR, pp. 3733-3742, 2018.
Xueting Yan. Ishan Misra, Abhinav Gupta, Deepti Ghadiyaram, and Dhruv Mahajan. Clusterfit:
Improving generalization of visual representations. In CVPR, pp. 6509-6518, 2020.
Mang Ye, Xu Zhang, Pong C Yuen, and Shih-Fu Chang. Unsupervised embedding learning via invariant and spreading instance feature. In CVPR, pp. 6210-6219, 2019.
Yang You, Igor Gitman, and Boris Ginsburg. arXiv: /708. 03888, 2017.
Large batch training of convolutional networks.
Yuhang Zang, Chen Huang, and Chen Change Loy. Fasa: Feature augmentation and sampling adaptation for long-tailes instance segmentation. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 3457-3466, 2021.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In ICML, pp. 12310-12320, 2021.
Chaoning Zhang, Kang Zhang, Chenshuang Zhang, Trung X Pham, Chang D Yoo, and In So Kweon. How does simsiam avoid collapse without negative samples? a unified understanding with self- supervised contrastive learning. In ICLR, 2021a.
Richard Zhang, Phillip Isola, and Alexei A. Efros. Colorful image colorization. In ECCV, volume 9907, pp. 649-666. Springer, 2016.
12

---

Under review as a conference paper at ICLR 2023
Richard Zhang, Phillip Isola, and Alexei A Efros. Split-brain autoencoders: Unsupervised learning by cross-channel prediction. CVPR, pp. 1058-1067, 2017.
Wenwei Zhang, Jiangmiao Pang, Kai Chen, and Chen Change Loy. K-net: Towards unified image segmentation. NeurIPS, 34:10326-10338, 2021b.
Wenwei Zhang, Jiangmiao Pang. Kai Chen, and Chen Change Loy. Dense siamese network for dense unsupervised learning. 2022.
Bolei Zhou, Agata Lapedriza, Jianxiong Xiao, Antonio Torralba, and Aude Oliva. Learning deep features for scene recognition using places database. NeurlPS, 27, 2014.
Chengxu Zhuang, Alex Lin Zhai, and Daniel Yamins. Local aggregation for unsupervised learning of visual embeddings. In ICCV, pp. 6002-6012, 2019.
13

---

Under review as a conference paper at ICLR 2023
APPENDIX A PYTORCH PSEUDOCODE
An exemplary implementation for MUSIC in the PyTorch-style is described in Algorithm 1.
Algorithm 1: PyTorch-style pseudocode for MUSIC
f:
network function
lambda: weight on the transformation invariance loss term
N: batch size
D: dimensionality of the embedding vector
DLS: dimensionality of each segment
S-D/D.S: number of segments
select: select the diagonal elements of diagonal blocks and all elements of off-diagonal blocks
for x in loader: # load batch with N samples
two randomly auqmented versions of x x', x = augment (x)
compute embeddings z' (x')
g'r - f(x'))
mu 1i multiissemental informational coding
x torch, reshape (x', [N, -1, D.S]) # NXSxD.S
x f torch reshape (x'', [N, -1, D.S]) # NXSXD.S
q' = torch. softmaxx (x', dim=2) # softmax normalization
q'
(x'', dim=2) softmax normal liatiii
compute transformation invariance loss
TI = -torch.leg (q' +q' sum (dim=2)) mean ()
compute entropy loss
q' torch. reshapees [N, DI) # N
q' rorch.reshape (q'1, [N, D]) $ N X
P torch. exrchmeinsu// 'np, nq->pq', [q', () N compute
empirical joint probability distribution P.s = (e)ect(()
loss_ent (P_s torch. (P_s)) sum
(S S)
final loss
loss = loss.ent lambda * loss.TI
lambda=1 by default
optimization step
loss backward optimizer ()ep
APPENDIX B THEORETICAL ANALYSIS
The optimal solution to the first part in Tents As described in Subsection 2.4, the entropy loss function consists of two parts: (1) the entropy diagonal elements of diagonal blocks and (2) the entropy over all elements off-diagonal blocks, as illustrated in Fig. 2. Now, let us minimize the first part, which formulated
Tent S
ΣN E
P(s 8", d' log P(s s". d')))
(B-7)
3'= 8e di!!!
14

---

Under review as a conference paper at ICLR 2023
Since every diagonal block has the same optimal solution, we only need to consider the st diagonal block, and the objective function can be simplified as
Dg
Lent (8, Σ
P(s, s, s, d, log( P(s, (og(
(B-8)
d=1
where 0 S minimizes this objective function; i.e., V8, d, P(s,& 8, d, d)
& 1,, < EdE P(s, s, d. d) 1 Then, it easy to find the solution that
- Thus, V8, d, we have
D3
D5
E P(s = Σ d
=1
(B-9)
Ds
d=1
As defined in Eqs. (1) and (2), we have Ys, 5 ((ss S 1,0 S (s, < 1, E2 ((ss
1, > ( d) = 1, and P(s, 8, d, d) = + > d(s, (s,
Given the above conditions, let us next prove that for Vs. 3d, g d) (x d) by contradic- tion.
< 1, then have ss, d, either d((s,d) d) < Ed= d'ss, = 1, or q (s,d) < Ed= (s, d=)) For (s. <
If its negation is true, i.e. vs, d. either 0 S d(s df < or 0 S (s.
Ed' (s, ((s, = 1, we have
Ds
Ds
N
P(., 8, d.
=
Σ
dis d) (s.d)
N
d=1
N
Ds
1
d(ss, d) (s.d)
N
(B-10)
1=1
N
D8
Ds diss d) > (s,dd")
That is. Σ, P(s,s,d,d d) < I, which leads to a contradiction with Eq. (B-9). Similarly, we have the same contradiction for (((s', d) < Ed= ( d') 1. Therefore, the statement that Vs, 3d, qi(s.dl d) = q d) = is true. It means that for Vs. (s. and (s,:) are one-hot vectors and equal each other.
Because Vs, d, P(s, s, d, d) = Ty. ds = (s, d). and q' (s. and (s :) are one-hot vectors, then P(s, d, d) = 1 ΣE' diss, d) (s, d) = # EN (s d) = p(s, d) = D.
Covariance the optimal solution. The optimal solution to maximize the joint entropy over the off-diagonal blocks for the second part is Vs' + s", P(s', 8", d', d"'d") E(g(s',
Elq(s','")q( dd")q("" "d"" =
(De) According to the above proof, we have Vs, dl,,, ( d) = FF d)] De We can the- oretically demonstrate that the covariance is zero between any two units from different segments. Specifically,, Vs', s,, ",,d d', d", o s", we have covlg(s', d'),q(s' dd) = Elg(s', , d' d')q(""
"q("",d")) dd")) - Flg(s' E's""""
1 (Ds) Ds Ds
1 1 = 0.
(B-11)
Since Vs, d', d" Ex s( d') and d',d' d','', d' d" p(s. Dz then Vs d", d' + d", "(sss,,s dd)) O. We can demonstrate that any units within the same segment are negatively correlated. Formally, Vs, d',d","""" + d", we have
cov(g(s,d") q(s, d'') Elg(s, d')q(s, d²)) Elg(s, d' E(q(ss P(s, d²) - d'))(sddd)
(B-12)
1 x Ds 2
D5 That is, every unit within each segment encodes discriminative and complementary features, while the units from different segments encode unrelated and diverse features.
15

---

Under review as a conference paper at ICLR 2023
APPENDIX C VISUALIZATION OF MUSIC EMBEDDING
02
145
125
(b)
(c)
Figure C.3: Visualization of MUSIC elements. (a) A partial joint probability matrix, where blue and yellow respectively represent small and large values, (b) two transformations the same image, and (c) partial embedding vectors corresponding to the images in (b).
In Fig. C.3, we visualize the MUSIC elements including an empirical joint probability matrix and individual embeddings, where the empirical joint probabilities ageNet train dataset and only the left-upper partial matrix of 400 x 400 and the first 800 units of embedding features selected for visualization. The theoretical analysis in Subsection 2.4 demon- 1 strates that the embedding statistics are enforced to be uniform; i.e., Vs, d, P(s, d, d) Dy meaning that the probabilities of the diagonal elements in all diagonal blocks are equal and those of off-diagonal elements are zeros. Also, the probabilities of all elements of the off-diagonal blocks are equal, i.e., Vs', s", d', .d", * 5", P(s', s", """"") = (Dg) The empirical joint probability matrix visualized in Fig. C.3-(a) is consistent with theoretical analysis although not perfect match. Furthermore, Figs. C.3-(b) and (c) show that the embedding features in each segment tend to be one-hot and invariant to the transformations, which are also consistent with the theoretical analysis.
computed over the whole Im-
24
25
134
13M
Segment-1 (0-79)
Segment-2 (80-159)
Figure C.4: Visualization of learned MUSIC features on ImageNet validation set. The left side shows the samples assigned to the features indexed by 6, 8, 24, and 25 in the first segment. The right side shows the samples assigned to the features indexed by 88, 124, 134, and 138 the second segment.
To qualitatively evaluate if meaningful embedding features are learned, in Fig. C.4 we show some examples assigned to specific units in the first two segments, where the whole embedding vector has 8,160 units including 102 segments, and each segment has 80 units. Specifically, some features in
16

---

Under review as a conference paper at ICLR 2023
the first segment represent different types of textures; e.g., the feature unit indexed by 24 represents the dot style textures, and the units 6, 8, and 25 correspond to other specific textures/patterss. In the second segment, some features represent different shapes; e.g., the unit 124 abstract a "n" shape, and the units 88, 134, 138 represent other shapes/patterns. Obviously, the first and second segments use different principles to group samples; e.g., the image containing red mushrooms in the first segment is grouped (indexed by 24) with the objects having similar textures, while in the second segment is grouped (indexed by 134) with the images having twin/repeated objects. These visual results indicate that the learned MUSIC embedding features are indeed meaningful and consistent to the general properties of Fig. I, which are ensured by the theoretical analysis.
APPENDIX D RELATED WORK
For self-supervised representation learning (SSRL), various pretext tasks were designed such as denoising auto-encoders (Vincent et al., 2008), context auto-encoders (Pathak et al., 2016), col- orization and cross-channel auto-encoders (Zhang al., 2016; 2017), masked auto-encoders (He et al., 2022), rotation (Gidaris et al., 2018; 2020), patch ordering (Noroozi & Favaro, 2016; Do- ersch et al., 2015; Chen et al., 2021), clustering (Caron et al., 2018; 2019: Asano et al., 2019; Yan et al., 2020: Huang et al., 2019; Zhuang et al., 2019; Gidaris et al., 2021), and instance dis- crimination (Dosovitskiy et al., 2014; Wu et al., 2018; Tian et al., 2020b; Ye et al., 2019; Dwibedi et al., 2021). Here we compare MUSIC with the most related SSRL methods, including contrastive learning, asymmetric non-contrastive Learning, clustering- based SSRL, dense SSRL, and the more recent non-asymmetric and non-contrastive learning methods.
Contrastive Learning. The contrastive learning based SSRL methods (Chen et al., 2020a; He et al., 2020) need to directly compare the features between negative pairs. Thus, large batch sizes are required, such as for SimCLR (Chen et al., 2020a). MoCo (He et al., 2020) uses a memory bank to store large number of features negative samples so that small batch sizes can be used, while it requires a momentum updating technique. The theoretical analysis for contrastive learning is based on estimating the lower bound mutual information between different views (Oord et al., 2018). In contrast, MUSIC doesn't it need negative samples, memory bank, or momentum updating, while can still discriminate instances. Without directly comparing large number of negative pairs for instance discrimination,
naturally encodes different instances with different embeddings via maximizing the joint entropy, as demonstrated in our theoretical analysis. MUSIC can directly use information measurements for both optimization and analysis instead of estimating the lower bound.
Asymmetric non-contrastive Learning, BOL (Richemond et al., 2020) and SimSiam (Chen & He, 2021) demonstrate that meaningful representations can be learned without using negative pairs. However, these methods depend on asymmetric architectures and stop gradient techniques to avoid trivial solutions. The follow-up theoretical analysis (Wang & Isola, 2020; Zhang et al., 202 a; Richemond et al., 2020; Tian al., 2021) leverage various concepts under different assumptions to demonstrate why these methods can avoid trivial solutions. MUSIC requires neither negative samples nor asymmetric designs. Moreover, MUSIC enables new information--heerrtiic SSRL framework where the information measures are directly used for both numerical optimization and theoretical analysis, which intrinsically avoids trivial solutions and learns meaningful features.
Clustering- based SSRL. DeepCluster (Caron et al., 2018) iteratively performs clustering with an extra kmeans algorithm on the features extracted in the previous step, and updates the weights of the network using the cluster assignments as supervision. To avoid trivial solutions, random sam- ples are selected for the empty cluster to compute the centroid. is time consuming for kmeans to cluster the whole large datasets, and the random selected samples is hard to form a meaningful cluster. Similarly, SELA (Asano et al., 2019) leverages the Sinkhorn-Know algorithm to iteratively perform clustering and optimize the clustering networks with the assigned cluster labels in an on- line manner. SWAV (Caron et al., 2020) alternatively computes the cluster assignment of one view and optimize the network to predict the same assignment for other views of the same sample. As a contrastive learning method, SwAV still requires a lot of prototype vectors for negative compar- isons between embeddings and codes. In can be seen that all these clustering methods require to compute a large number of extra cluster centers and leverage extra algorithms to compute assign- ments. From the clustering perspective, each segment in the MUSIC embedding can be regarded as
17

---

Under review as a conference paper at ICLR 2023
cluster assignment, and multiple segments can be regarded as performing multiple clustering si- multaneously, where different segments have different clustering principles. Different from current clustering based methods, MUSIC does not require computing a large number of cluster centers or an extra algorithm to estimate the cluster assignments iteratively.
Dense SSRL. Some excellent studies, such as DenseCL (Wang al., 2021) and DenseSiam (Zhang et al., 2022), design seffssuuprriisdduuuuervsee learning methods for improving dense prediction tasks, including object detection (Tian et al., 2019), instance segmentation (Zang al., 2021), and semantic segmentation (Zhang et al., 202.b). The basic idea is to enhance the pixel-level and region- level consistency in the self- wwwssssuse/uuuusueeeeeeeee
learning setting. DenseCL (Wang et al., 2021) proposes to optimize a pairwise contrastive (diilattyy loss at the pixel level between two views of input images. Most recently, DenseSiam (Zhang et al., 2022) proposes to optimizes the con- sistency of different levels based on the simple Siamese network without needing negative pixel pairs, momentum encoders or heuristic masks. Synergistically, MUSIC can be also used for dense prediction tasks by discretizing the feature vector of each sub-region/pixel and learning dense rep- resentations in the cnnormation-thooreen framework.
Non-asymmetric and non-contrastive learning. Recently, WMSE (Ermolov et al., 2021), Bar- low Twins Zbontar et al. (2021), and VICReg (Bardes et al., 2022) propose to train a simple twin network architecture using covariance matrix based loss functions without needing any asymmetric or constrastive learning techniques. Specifically, WMSE (Ermolov et al., 2021) proposes to min- imize the MSE distance between different views and enforce the self-covariance matrix to be an identity-matric matrix. Barlow Twins (Zbontar et al., 2021) optimizes the cross-covariance matrix to be an identity-matrix. VICReg (Bardes et al., 2022) proposes three loss terms including invariance, variance, and covariance. The theoretical analysis for Barlow Twins (Zbontar et al., 2021) assumes the Gaussian distribution assumption of embedding features, then the loss function can be inter- preted with the information measures under some approximations. One of the key ideas of these methods is to reduce the redundancy between feature variables by minimizing the linear correlation. In contrast, MUSIC discretizes the feature variables making the probability distribution estimable can be directly used
so that the information- measuress (entropy/mutuul information/(inddeeedeeet for both numerical optimization and theoretical analysis without assuming the Gaussian distribution. Importantly, our theoretical analysis shows that MUSIC is exactly minimizing the mutual informa- tion any dependence between feature variables, which is beyond the linear correlation constraints used in current methods. That is why MUSIC can use a shorter embedding vector and achieve even better results at lower computational cost.
APPENDIX E
COMPUTATIONAL ENVIRONMENT
MUSIC models were distributively trained on four nodes, each of which has the system information:
2x 20 core 2.5 GHz Intel Xeon Gold 6248;
* 8x NVIDIA Tesla V100 GPU each with 32 GiB HBM;
768 GiB RAM per node;
Dual 100 Gb EDR Infiniband
APPENDIX F RUNNING TIME
In Table 10, the computational cost of MUSIC was evaluated and compared with other methods. All methods were rum on 32 Tesla V V100 GPUs These methods offer different trade-offs among running time, memory and performance. SWAV with multi-crop and BYOL achieve better performance at the additional computational cost and memory usage. Barlow Twins and VICReg have balanced results with less memory than BYO and SwAV (multi-crop), faster speed than Sw AV (multi-crop), but a slightly worse performance. Compared with the most related Barlow Twins and VICReg methods, MUSIC cannot only reduce the running time and memory usage significantly. but also improve the performance. It is due to that MUSIC can use a shallower fully-comnected MLP head for a better performance as discussed in Subsection 4.2. The computational cost of MUSIC will be significantly reduced further when using a (x2) lower dimension for embeddings, and the performance would be degraded very slightly, as discussed in Subsection 4.2.
18

---

Under review as a conference paper at ICLR 2023
Table 10: Running time and peak memory. Comparison of different methods in terms of the running time over 100 epochs, the peak memory a single GPU, and the top-1 accuracy (%) on linear classification on top of the frozen representations. All models were distributively trained on 32 Tesla V100 GPUs.
Method
Time / 100epochs
Peak memory GPU
Top-I I accuracy
SwAV
9h
9.5G
71.8
SWAV (w/multi-roop)
13h
12.9G
75.3
BYOL
10h
14.6G
74.3
Barlow Twins
12h
11.3G
73.2
VICReg
1 11h
11.3G
73.2
MUSIC
8.5h
10.4G
73.6
APPENDIX G IMPLEMENTATION DETAILS
G.1 EVALUATION ON IMAGENET
Linear classification: For all evaluation experiments ImageNet linear classification, followed the standard procedure that a linear classifier was trained on top of the frozen backbone of a ResNet- 50 pre-trained with MUSIC. The SGD optimizer decay, weight decay of 10 6, batch size of 256, and 100 training epochs. In the training stage, the images were augmented by the composition of random cropping and resizing of ratio 0.2 1.0 for size x24 x 224, and random horizontal flips. In the testing stage, the images were simply cropped from the image center and resized to 224 x 224.
used with learning
of 0.02, cosine
Semi-supervised learning: In the semi-supervised learning setting, a linear classifier was appended to the pre-trained backbone with MUSIC, and the network was fine-tuned using 1% and 10% of the labels. The SGD optimizer was used with weight decay and batch size of 256, and the model was trained for 20 epochs. In the 1% of labels case, we used a learning rate of 0.08 for the encoder and 0.1 for the linear head. In the 10% of labels case, we used 0.02 for the encoder and 0.1 for the linear head. Both these learning rates followed a cosine decay schedule. The data augmentation steps for training and testing followed the same settings of the linear evaluation.
G.2 TRANSFER LEARNING
Object detection and instance segmentation: Mask R-CNN (He et al., 2017) with the C-4 back- bone was trained the COCO 2017 train split and tested the validation We used learning rate of 0.1 and kept the other parameters the same as in the schedule in detectron2.
Linear classification: We followed the exact settings from PIRL (Misra & Maaten, 2020) in evalu- ating linear classifiers on the Places-205 and VOCO7 datasets. For Places-205, a linear classifier was trained using the SGD optimizer for 14 epochs with a learning rate of 0.01 reduced by a factor of 10 at epochs 5 and 10, weight decay of 5 x 10 and a momentum of 0.9. For VOC2007 dataset, we trained SVM classifiers, where the C values were computed using pross-validation
Table 11:
Top-1 linear classification accuracies on CIFAR1O. Here the embedding dimension for all models was set to 1024. The MUSIC results with different segment dimensions are reported.
MUSIC
Models
Barlow Twins
VICReg
8
16
24
32
48
64
TOP-1
88.2
88.5
88.5
88.6
88.6
89.2
89.2
88.6
APPENDIX H RESULTS ON CIFARIO
Here further evaluated the characteristies of MUSIC in terms of the segment dimension the CIFAR10 dataset. Without using any asymmetric or contrastive techniques, Barlow Twins and VI- CReg are regarded the baseline methods for MUSIC. Specifically, the batch size was 512 and
19

---

Under review as a conference paper at ICLR 2023
the total dimension of the embedding vector was 1,024, the base learning rate was 0.1, the number of training epochs was 800, and all other paraa--arameters were kept the default settings for MU- SIC, Barlow Twins, and VICReg. Top-1 accuracy results reported in Table 11, showing that the segment dimension should be adjusted according to the target dataset, which is similar to that the network architecture and the total feature dimension
usually associated with the scale and com- plexity of target datasets. Nevertheless, MUSIC achieved superior results than the baseline models over large range of segment dimensions under the same evaluation setting.
20


Reviews:
[{"review_id": "qydV-Fwxpr", "reviewer": "Reviewer_f3VU", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes multi-segmental informational coding as a new embedding scheme for self-supervised representation learning which divides an embedding vector into multiple segments to represent different types of attribute, and as a result each segment automatically learns a set of discrete and complementary attributes. The proposed model enables the estimation of the probability distribution over discrete attributes and thus the learning process can be directly guided by information measurements, reducing the feature redundancy beyond the linear correlation.", "summary_of_the_review": "I have found the paper very interesting. The paper is theoretically justified. Some experimental or evidential issues evidences are there (mentioned in the weaknesses section) should be improved.", "strength_and_weaknesses": "**Strengths**\n\n(1) The proposed method is based on an interesting idea of segmenting representation and estimating the probability distribution over discrete attributes, which I think is relatively new.\n\n(2) The paper has considered standard benchmarks for experimentations to demonstrate the effectiveness of the proposed method, where the MUSIC model has performed consistently.\n\n(3) The paper is theoretically well grounded and the authors have shown that the optimal MUSIC embedding features have zero covariance between any two features in different segments and negative covariance between the features within the same segment.\n\n**Weaknesses**\n\n(1) The proposed method have achieved minor improvement on very few datasets, whereas on the other datasets, the method couldn't surpass the state-of-the-art methods, which fails to justify the superiority of the proposal.\n\n(2) The method is nicely motivated by the idea of multi-segmental discretisation, but the evidence that the learnt representation follows intuition is largely missing. At the moment, there is only one example in appendix C, but more such visualisation would have been useful and interesting.\n\n(3) It is not clear or evident what information those individual segments are learning. Also the number of segments should be correlated with datasets and also on the size of crops, which have not been ablated in the paper at this stage and decided in an adhoc manner.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "U5uCAjMd4K", "reviewer": "Reviewer_ZcT3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presented a new method for self-supervised image representation learning. Specifically, a multi-segmental informational coding (MUSIC) scheme was proposed to achieve a new feature embedding, which divided the projected feature vector into multiple segments to represent different attributes. The loss function was then designed to maximize the entropy between segments and units within them. Experimental analysis on a few public datasets over benchmark vision tasks shows the effectiveness of the proposed method. The main contribution is the proposed MUSIC coding scheme.", "summary_of_the_review": "The main idea of this paper is interesting and seems to make sense in self-supervised visual representation learning. But it was not well presented with sufficient evidence to justify the claims (the segments and attributes part). The experimental performance was also insufficient to support the claimed effectiveness of the proposed method. The overall technical novelty and contributions are also a bit limited.\nAs a result, this paper is considered to be not ready for publication. But the authors are encouraged to provide more details and justifications of the proposed method and the claimed contributions and re-submit to a future venue.", "strength_and_weaknesses": "## Strengths\n\n\\+ The proposed idea of minimising the mutual information between different attributes is interesting.\n\n\\+ An analysis section (Sec. 2.4) was presented to help better understand the proposed method.\n\n\\+ An empirical analysis (Sec. 4.2) was provided to evaluate the components within the proposed method.\n\n\\+ The paper is in general well written and easy to follow.\n\n\n## Weaknesses\n\n\\- The proposed method seems to be a disentangle approach for the projected feature vector, which was defined to be within several semantic categories (i.e. the segments as illustrated in Fig. 1). But it is unclear how it was guaranteed to have these semantic meanings for a projected vector. The evenly splitting of the vector was also not well justified.\nOn the other hand, although the projected vector was divided into several segments/attributes, the whole vector is in principle representing the same feature as without such partition. It is unclear, as a result, why this would introduce the claimed attributes information into the representation learning. The experimental results also did not suggest any noticeable improvement over previous works in representation learning.\nIt would be better if some examples/qualitative results could have been shown to validate that the claimed segments were actually related to the particular semantics.\n\n\\- The novelty and contributions of the proposed method are a bit limited. Considering that the general idea and framework are quite similar to prior works on self-supervised representation learning (e.g. SimCLR, BYOL, Barlow twins to name a few), the only difference is the final projected feature processing and the following loss to supervise the network training. Whereas apart from the segments part, even the entropy maximization is quite similar to the approach proposed in Barlow twins and VICReg in a high-level idea.\n\n\\- It is a bit unclear how the \"theoretical analysis guarantees\" the proposed model learns non-trivial, diverse and discriminative features, especially the discriminative features.\n\n\\- The performance of the proposed method is only comparable to several prior works and even worse than some of them (e.g. BYOL, Barlow Twins, VICReg). It is hard to tell and be convinced of the effectiveness of the proposed method and its significance.\n\n\\- The authors discussed the idea of hierarchical clustering in the last section, which is nice to see. This also relates the proposed method to the clustering-based approaches. It would be better to discuss and differentiate the proposed segment-based idea and those clustering-based ideas (e.g. DeepCluster, SwAV etc.) for visual representation learning, especially what is the benefit of the proposed method over those approaches.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "TszVK76rzV", "reviewer": "Reviewer_yN39", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a self-supervise learning method using an entropy loss across multi-segments embedding vectors (MUSIC). This method automatically learn different types of attributes in each of multi segments.  Unlike the contrastive learning and its variants, it does not require negative samples in a large batch memory and asymmetric network, gradient stopping, or momentum update. Theoretical analysis support that the MUSIC embedding is transform-invariant, non-trivial, diverse, and discriminative. A set of downstream tasks show that the propose representation learning method is comparable to or outperforms previous states-of-the-arts. An empirical analysis investigates the efficiency in training , the effect of projector depth and feature dimension, and the necessity of the proposed loss function. \n\n\n    ", "summary_of_the_review": "This paper seems to be a good contribution as a self-supervised learning method. However, the motivational idea, which the model learns disentangled attributes over segmented embedding space, was not validated. Also, some relevant references are missing. I hope the authors revise the current version to make up the weaknesses.    \n", "strength_and_weaknesses": "Strengths\n- The information-theoretic entropy loss is well supported by the theoretical analysis  \n- Model training is efficient as it does not require a large match memory to include negative samples\n-The overall performance is comparable to or outperforms previous states-of-the-arts in several downstream tasks \n- The empirical study is comprehensive to understand the capability of the proposed method. \n\nWeakness\n- Some related work is missing. Recently, there have been a decent number of dense contrastive learning which divides the embedding space into small multi-vectors and exploits their correspondence. For, example,\"Dense Contrastive Learning for Self-Supervised Visual Pre-Training\", Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, Lei Li, CVPR, 2021,  \"Dense Siamese Network for Dense Unsupervised Learning\", Wenwei Zhang, Jiangmiao Pang, Kai Chen, Chen Change Loy, ECCV, 2022\n- The paper argues that MUSIC divides an embedding vector into multiple segments that represent different types of attributes such as object part, texture, and shape. However, this hypothesis was never proved. There is no insightful analysis that the multiple segments capture the different visual attributes exclusively.  Although the hypothesis and analogy in Figure 1 may help delivering the \"intended idea\", it would be risky to insist that the model actually learn the features in that manner. \n- Some part of writing is not clear (detailed below)\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject