Unsupervised visualization of image datasets using contrastive learning
Authors: Niklas Böhm
Philipp Berens
Dmitry Kobak
Venue: ICLR 2023 poster (2023)
Keywords: data visualization
contrastive learning

Abstract: Visualization methods based on the nearest neighbor graph, such as t-SNE or UMAP, are widely used for visualizing high-dimensional data. Yet, these approaches only produce meaningful results if the nearest neighbors themselves are meaningful. For images represented in pixel space this is not the case, as distances in pixel space are often not capturing our sense of similarity and therefore neighbors are not semantically close. This problem can be circumvented by self-supervised approaches based on contrastive learning, such as SimCLR, relying on data augmentation to generate implicit neighbors, but these methods do not produce two-dimensional embeddings suitable for visualization. Here, we present a new method, called t-SimCNE, for unsupervised visualization of image data. T-SimCNE combines ideas from contrastive learning and neighbor embeddings, and trains a parametric mapping from the high-dimensional pixel space into two dimensions. We show that the resulting 2D embeddings achieve classification accuracy comparable to the state-of-the-art high-dimensional SimCLR representations, thus faithfully capturing semantic relationships. Using t-SimCNE, we obtain informative visualizations of the CIFAR-10 and CIFAR-100 datasets, showing rich cluster structure and highlighting artifacts and outliers.

Published as a conference paper at ICLR 2023
UNSUPERVISED VISUALIZATION OF IMAGE DATASETS USING CONTRASTIVE LEARNING
Jan Niklas Böhm, Philipp Berens & Dmitry Kobak
University of Tübingen, Germany
(jan-niklas boehm, philipp berens, dmitzy. kobak htt.:unnnnttuiiinnddd...
ABSTRACT
Visualization methods based on the nearest neighbor graph, such as t-SNE or UMAP, are widely used for visualizing high-dimensional data. Yet, these ap- proaches only produce meaningful results if the nearest neighbors themselves are meaningful. For images represented in pixel space this is not the case, dis- tances in pixel space are often not capturing our sense of similarity and therefore neighbors not semantically close. This problem can be circumvented by self- supervised approaches based on contrastive learning, such as SimCLR, relying data augmentation to generate implicit neighbors, but these methods do not produce two-dimensional embeddings suitable for visualization. Here, we present new method, called I-SimCNE, for unsupervised visualization of image data. t-SimCNE combines ideas from contrastive learning and neighbor embeddings, and trains parametric mapping from the high- sigh-dimensional pixel space into two dimensions. We show that the resulting 2D embeddings achieve classification accuracy comparable the of-e-of-thhearr tations, thus faithfully capturing semantic relationships. Using t-SimCNE, we ob- tain informative visualizations of the CIFAR- 10 and CIFAR- 100 datasets, showing rich cluster structure and highlighting artifacts and outliers.
nigh-dimensional SimCLR represen-
INTRODUCTION
As many research fields producing ever larger and more complex datasets, data visualization methods have become important in many scientific and practical applications (Becht et al.) 2019
Diaz-Papkovich et al. 2019 Kobak & Berens 2019 Schmidt 2018). Such methods allow concise summary of the entire dataset, displaying a dime-dimansoonal dataset as a 2D embedding. This oow-dimesiional representation often convenient for data exploration, highlighting clusters and relationships between them. In practice most useful are neighbor embedding methods, such as I-SNE (van der Maaten Hinton 2008) and UMAP (Mcinnes al. 2018)), that aim preserve nearest neighbors from the high-dimensional space when optimizing the layout in 2D.
data martaalien
CIFAR-10
h;
1024
7.5
Taking
1250
Findurrng
Restitt
5.0
Linerr beyer
1 +
22
2,
pojection
metiaation
dess
1p ares
500
opach
10000
1500
Figure I: Left: t-SimCNE. Two augmentations of the same image are fed through the same ResNet and fully-connected projection head get representations z; and z,. The loss function pushes z; and 2j together to maximize their Cauchy similarity. Middle: Embedding of CIFAR-I0 10. The dashed arrows point to the locations of z; and z, from the left. Right: Training loss. The optimization consists of three stages: (1) pre-training with a 128D output for 1000 epochs; (2) fine-tuning only the 2D readout layer for 50 epochs; and (3) fine-tuning the entire network for 450 epochs.

---

Published as a conference paper at ICLR 2023
Unfortunately, for image datasets, nearest neighbors computed using the Euclidean metric in pixel space are typically not worth preserving. Although t-SNE works well on very simple image datasets such MNIST (van der Maaten & Hinton 2008 Figure 2a), the approach fails when considering more natural image datasets such as CIFAR- images, new visualization approaches are required, which use different notions of similarity.
(Sup A.I To create 21 embeddings for
Here, we provide such a method based on the contrastive learning framework. Contrastive learning is currently the state-of-the-art approach to unsupervised learning in computer vision (Hadsell al.) 2006). The contrastive learning method SimCLR (Chen et al. 2020) uses image transformations to create two views of each image and then optimizes a convolutional neural network so that the two views always stay close together in the resulting representation. While this method performs very well in benchmarks- - is typically high-dimensional (e.g. 128 128-dimensional), hence not suitable for visualization.
such as linear or INN classification accuracy, - the computed representation
We extend the SimCLR framework to directly optimize 2D embedding. Taking inspiration from I-SNE, we use the Euclidean distance and the Cauchy (t-distribution) kernel to measure similarity in 2D. While using 2D instead of 128D output may not seem like a big step, we show that optimizing the resulting architecture is challenging. We develop an efficient training strategy to overcome these challenges, and only then are able to achieve satisfactory visualizations. We call the resulting method t- t- (Fig. I) and show that it yields meaningful and useful embeddings of CIFAR-10 and CIFAR- 100 datasets (Krizhevsky) 2009).
Our code is available g wtt.uub..cco/berrssaaaaa-ta----a
(see iclr202222 branch).
2 RELATED WORK
Neighbor embeddings (NE) have a rich history dating back to locally linear embedding (Roweis)
& Saul 2000) and stochastic neighbor embedding (SNE; Hinton & Roweis 2003). They became widely used after the introduction of the Cauchy kernel into the SNE framework (van der Maaten & Linder- Hinton 2008) and after efficient approximations became available (van der Maaten 2014]Linder man et al. 2019). A number of algorithms based on that framework, such as Large (Tang et al., 2016), UMAP (Meinnes et al. 2018). and TriMap (Amid & Warmuth 2019) have been developed and got widespread adoption in recent years in a variety of application fields. All of them are closely related to SNE (Böhm et al. 2022 Damrich et al.) 2023) and rely on the ANN graph of the data.
NE algorithms have been used to visualize latent representations of neural networks trained in supervised setting (e.g. Karpathy 2014, Mnih al. 2015). This approach is, however, unsuitable for data exploration as it is supervised. NE algorithms can also be applied to an unsupervised (also known as self-supervised) representation of a dataset obtained with SimCLR, or to a representation obtained with neural network pre-trained on generic image classification task such as ImageNet (e.g. Narayan et al. 2015). The downside is that these approaches would not yield a parametric mapping to 2D. In this work, we are interested in an unsupervised but parametric mapping that allows embedding out-of-sample points. See Discussion for further considerations.
Conceptual similarity between SimCLR and t-SNE has recently been pointed out by Damrich al. (2023). The authors suggest interpreting INN graph edges as data augmentations, and show that t-SNE can also be optimized using the InfoNCE loss (van den Oord al. 2018) used by SimCLR, and/or using a parametric mapping. Equivalently, one can think of SimCLR as a parametric SNE that samples edges from an unobservable neighbor graph. We were motivated by this connection when developing t- t-SimCNE. Further motivation comes from a recently described phenomenon called di- mensional collapse (Jing et al.) al.] [022] Tian 2022). which suggests that there is redundant information in the output of SimCI.R Hence we reasoned that it should be possible to achieve good represen- tation even with drastically reduced output dimensionality.
Two closely related works appeared during preparation of this manuscript: Zang et al. (2022) suggest an architecture similar to SimCLR for 2D embeddings, but use a more complicated setup to impose 'local flatness' and, judging from their figures, obtain qualitatively worse embeddings of CIFAR datasets than we do (we were unable to quantitatively benchmark their method). Hu et al. (2023)
suggest to use the Cauchy kernel in the SimCLR framework (calling it t-SimCLR), but in terms of 2D visualization, obtain worse results than we do (Hu et al. 2023 Fig. B. shows CIFAR-10 100 reported kNN accuracy 57% vs. our 89%).
2

---

Published as a conference paper at ICLR 2023
3
CONTRASTIVE LEARNING FOR VISUALIZATION
3.1 SIMCLR OVERVIEW
SimCLR relies on creating two views of each data sample using random data augmentations Fig.. For image data, this means that the image is randomly cropped, flipped, and on. The method feeds a mini-batch containing pairs of augmented images through a ResNet to obtain a representation of the images. The parameters of the network are optimized such that the paired images are placed close to each other in the output space, while keeping all other images further apart.
In mathematical terms, the loss function used by SimCLR, known as InfoNCE loss (van den Oord al. 2018 is defined as
exp (sim(( Z,, 21)/T)
(smcLL(((.)) llgg
Lkys exp (sim(zi, 2x.)/T)
2b
simimmz; 2,)/T + log
exp (imm (s)m((;,24)/7
(2)
Here, indices i and j correspond to two data augmentations of the same original image, and denotes network output. For batch size b, there are 26 samples in the batch because each sample augmented twice. The similarity function function used by SimCLR is the cosine similarity: sim(x,y y) x //([|x| [[yl)). We follow Chen et al. al. (2020) in using 7 0.5.
The SimCLR cost function is intimately related to the SNE loss. Note that the cosine distance cosdist(x, y) = 1 sim(x, y) is equal to the half of the squared Euclidean distance between unit- normalized x and y. Let à = a//l Then
exp cosdist(z;, 2,)
|2 2
exp
const exp
(3)
2r
which is precisely the expression used by SNE (a Gaussian kernel of the Euclidean distance) to measure similarity between embedding vectors (Hinton & Roweis, 2003). As shown by DDamiihh et al. (2023), SNE can also be optimized using the InfoNCE loss and mini-batch training; in the SNE setup, and j are two points connected by a ANN graph edge.
The loss functions of both SimCLR and t-SNE give rise to attractive force between similar points i and j. enforced by the InfoNCE numerator in Eq. (1) and a repulsion between all points, enforced by the denominator. WWan & Isola (2020) called this the alignment and the uniformity aspects of the
loss. In the neighbor embedding literature, this is referred to as attractive and repulsive forces acting
on the embedding points (Böhm et al. 2022 Damrich & Hamprecht 2021
Wang al. 2021
3.2 I-SIMCNE LOSS FUNCTION FOR 2D VISUALIZATION
To achieve our goal of adapting SimCLR to create 2D embeddings of images, reduce the dimen- sionality of the linear output layer from 128 to 2 (Fig. 1j. This change, however, requires changing the similarity function as well, as using cosine similarity would effectively constrain the embedding to the unit circle S1, which is not suitable for data visualization.
Instead, we remove the normalization from the last expression in Eq. (3)
allowing the output
vectors to lie anywhere in ? In other words, we replace the cosine distance with the Euclidean distance, which is natural choice for measuring distance between points in 2D embedding. While one could still use Gaussian kernel to transform Euclidean distance into similarity as in Eq. (3) we
follow t-SNE's 'ss example (yan der Maaten & Hinton 2008) and use the Cauchy (t-distribution with one degree of freedom) kernel instead, the heavy-tailed Cauchy kernel reduces the 'crowding problem' that SNE had with the Gaussian kernel (van der Maaten & Hinton 2008). We call the resulting method I-SimCNE.
3

---

Published as a conference paper at ICLR 2023
Table 1: Model performance the CIFAR-10 dataset. The columns are: model type (see text), dimensionality of Z, the total number of training epochs, linear classification accuracy in H, ANN classification accuracy in Z (k- 15; see Fig. A.3 for k E (1], the final loss value, training time. Standard deviations correspond to three runs with different random seeds (for the loss, the standard deviation was always smaller than 0.05). Each experiment was run on single GeForce RTX 2080 Ti GPU (the 5000 epoch experiment was run V100 GPU).
# Model
dim
Epochs
Linear in
ANN in
Loss
Time (hr.)
Cosine (SimCLR)
128
1000
93.1 0.1%
91.1 + 0.2%
5.8
12.0 + 0.1
2
Cosine
3
1000
87.6 0.3%
74.0 = 1.4%
6.3
12.3 + 0.4
3
Euclidean
128
1000
90.7 0.3%
90.1 + 0.2%
2.3
12.9 + 1.6
4
Euclidean
2
1000
84.6 0.3%
80.0 0.2%
3.7
11.9
5
Euclidean
2
5000
8
-
2.9
65.4 0.2
6
Cos. Doss-EEuclidee
Euclidean
2
1500
93.0 0.3%
90.1 0.4%
3.2
17.5 7
7
Eucl. Euclidean
2
1500
90.6 00.6:00 0.2% 0.2%
89.4 0.4%
2.7
18.9 + 2.3
8
-uull--Euclidann
2
1000
90.3 0.3%
88.3 = 0.4%
2.9
11.8 =
9 Fuel. Fuclidean
2
500
88.7 + 0.1%
85.8 + 0.1%
3.3
6.0 0
We denote the Euclidean distance between z, and z, dij ||, -zyll- The corresponding Cauchy similarity is 1/(1 d.y). With that, we define the (-SimCNE loss as
1/(1 + diy)) l SimcNEE(.. - -log 26 Aigh
(4)
1/(1 + d
= log
log
(5)
de
d? 8k
Aigh
We will refer to the SimCLR loss as 'cosine' loss and to the t-SimCNE loss as 'Euclidean' loss. Note that the numerical loss values between them are not directly comparable, because the minimum of the Euclidean loss is zero, whereas the cosine loss, Eq. (2) is bounded from below by
26
SiMCLR (i,j) = -1/7 + log (xx((1/T)+ +
Σ k
exp(- exp(-1/7))
-/7/ + log (xp((1/) (2b 2) exp(-1/7))
(6)
which is ~3.65 for b 1024 and 0.5.
3.3 IMPLEMENTATION DATASETS, AND PERFORMANCE METRICS
We used our own Py Torch Paszke et al. 2019 version 1.12.1) implementation of SimCLR. As the backbone, we used a ResNet18 8 (He kernel size in the first convolutional layer of the ResNet18 8 from x to 3 as in Chen et al. (2000). For the fully-connecter projection head we used one hidden ReLU layer with 1024 units, and linear output layer with 128 units. We optimized the network for 1000 epochs using SGD with momentum 0.9. The initial learning rate was 0.03 b/256 0.12, with linear warm-up over ten epochs (from to 0.1 and cosine annealing (Loshchilov & Hutter (017) down to 0 for the remaining epochs. We used batch size = 1024 and the same set of data augmentations as in Chen et al. (2020).
2016), which has 122-dimensional output. We reduced the
We used CIFAR- 10 and CIFAR-100 datasets (Krizhevsky) 2009) for all our experiments. Each dataset consists of n 60 000 colored and labeled 32 x 32 images. CIFAR-10 has 10 classes Fig. A.2), while CIFAR-100 has 100 classes grouped into 20 superclasses. We used the dataset classes CIFAR10 and CIFAR100 00 provided by the tor corchvist ion package.
To quantitatively assess the embedding quality, not only report loss values but also the test-set KNN accuracy in the projection head output space as our main performance metric. Note that in the contrastive learning literature (e.g. Chen al. 2020). representation quality is usually assessed
4

---

Published as a conference paper at ICLR 2023
a
Euclidean
Euclidean (5000 epochs)
Cosine + Euclidean
d
Euclidean Euclidean
arplane dog trop
aztomobile
100
100
100
100
Figure 2: Different training strategies for t-SimCNE on CIFAR-10. 10. (a) Optimizing the 2D Euclidean loss for 1000 epochs. (b) Optimizing the 2D Euclidean loss for 5000 epochs. (c) Pretraining with cosine loss in 128D and fine-tuning with Euclidean loss in 2D. (d) Pretraining with Euclidean loss in 128D and fine-tuning with Euclidean loss in 2D.
Euclidean
b
Cosine Euclidean
Euclidean Euclidean
aquatic mammalss
lange natural
fush
flowers
and worts
food conrainers
modium sized
vojetalles
neninaest
housmnots eisc-
inverich otos
househcli
furiitre
Insecs
large sariiv
vetiriee
outdorr th ngs
veticcie
100
100
100
Figure 3: Different training strategies for L-SmCCNE on CIFAR-100. The colors correspond to 20 su- perclasses and not to the fine-grained labels. (a) Optimizing the 2D Euclidean loss for 1000 epochs. (b) Pretraining with cosine loss in 128D and fine-tuning with Euclidean loss in 2D. (c) Pretraining with Euclidean loss in 128D and fine-tuning with Euclidean loss in 2D.
via linear classification accuracy in the ResNet output space H. In the Z space, we prefer to use the ANN classifier, as the 2D embedding can be considered good even classes are separable but not linearly separable. For kNN accuracy, we used the scikit-learn (Pedregosa al, 2011) imple- mentation with A: = 15 (any k E [1,30] gave qualitatively the same results, Fig. A.3) For the linear accuracy, we used the Logi st i cRegressi on class from scikit-learn, with the SAGA solver (Defazio et al.) 2014) and no penalty. For CIFAR- 10 (CIFAR- 100). our SimCLR implementation achieved 93% (67%) test-set linear accuracy H Tables and A.1 line 1) which is state of the art Appendix D and da Costa et al. 2022 Table 1).
for SimCLR with ResNet18 8 (e.g. Chen & He 2021
We heavily used Matplotlib 3.6.0 (Hunter 2007). NumPy 1.23.1 (Harris et al.
2020 and open'SNE
0.6.2 (Poličar et al. 2019), which, in turn, uses Annoy (Bernhardsson) 2013).
3.4 1-SIMCNE REQUIRES DIMENSIONALLIY ANNEALING FOR OPTIMAL RESULTT
We used the setup described in the previous section to train t-SimCNE on the entire CIFAR- 10 dataset. However, we found that naïve training of 2D t-SimCNE resulted in a suboptimal embedding layout (Fi.. 2p) and achieved only 80% &NN accuracy (Table I line 4).
When we increased the number of training epochs from 1000 to 5000, the embedding improved Fig. 2p), with İ:NN accuracy reaching 87% (Table 1] line 5). However, 5000 epochs took a long time (almost three GPU days) this approach is not very practical. Moreover, we felt the embedding was still suboptimal, with many clusters seemingly unable to break apart from each other.
We noticed that in both cases above, the linear accuracy in Hf was markedly lower than with standard SimCLR (85% and 89% vs. 93%, cf. [Talle I] lines 4, 5, and 1). We reasoned that it may be beneficial to pretrain the model with 128D output, achieving high-quality representation in H, and then fine- tune the model with the 2D output. This can be seen as 'dimensionality annealing'. Moving from 128D output to 2D requires changing the linear output layer in the projection head. For this,
5

---

Published as a conference paper at ICLR 2023
Euclidean
b
Cosine
400
Figure 4: L2 norms of the representation in 128-dimensional space after training on CIFAR-10 with the Euclidean loss (a) and with the cosine loss (b). Standard SimCLR uses the cosine loss. Colors in Fig. 2 There similar difference in the H space, but less pro- nounced.
200
3
dass Index
class Index
initialized the new 2D output layer randomly, froze the entire network apart from this layer, and trained the last layer for 50 epochs (learning rate 0.12, no warm-up, no annealing). This ensured that the new output layer was reasonably aligned with the rest of the projection head. Afterwards, the entire model was unfrozen and trained for another 450 epochs (initial learning rate 0. 0.12/1000, warm-up for 10 epochs, cosine annealing down to 0).
We experimented with two different options for 128D pre-training, either using standard SimCLR with the cosine loss, using the Euclidean loss. We found that using Euclidean similarity for pretraining resulted in the final t-SimCNE loss 2.7 (Table I line 7), vs. 3.2 when using the cosine similarity Table I] line 6). Euclidean loss also resulted in visually more pleasing embedding with crisp and well-separated clusters Fig. 2 vs. Fig. 2b). As the difference in the final loss was large, (even though the final KNN accuracy was slightly lower compared to the cosine pretraining: 89% vs. 90%).
believe that this training strategy is the optimal
We confirmed this conclusion by applying t-SimCNE to CIFAR-IOM Fig. 3p. We found that without any pretraining, kNN accuracy on the superclass level was 50% Fig. 3d) with SimCLR pretraining
using cosine loss, was 65% (Fig. 3D); and with Euclidean pretraining, was 68% (Fig. 3b). On the level of classes, the ANN accuracy was 33%, 47%, and 51 ,,, respectively (Table A. lines 4-6).
Even though the cosine similarity led a higher quality the representation for both CIFAR-10 and CIFAR-100 (93% vs. 91 % on CIFAR- 10; TTalle 1, lines 1 vs. 3) compared to the Euclidean similarity; and even though the spherical constraint has theoretically appealing properties (Wang & Isola| 2020), the Euclidean similarity worked better for t-SimCNE pretraining. This may have to do with the norm distribution of the embedding vectors. Standard SimCLR with cosine loss has no discernible structure in the embedding norms (Fig. 4p), which is not surprising as the z E 128 vectors are normalized when computing the cosine similarity. But with Euclidean loss, the norms of z vectors strongly differ between classes (Fig. 4b). This leads to reasonable 2D embedding even before the linear fine-tuning (Fig. A.4p and Fig. A.5)), and fine-tuning is able to work more effectively (Fig. A.4p.c and Fig. A.5p.c). Regarding dimensional collapse, it was less pronounced with Euclidean loss compared to the cosine loss (Fig, A.6).
3.5 ADDITIONAL EXPERIMENTS
We explored the effect of training budget on the resulting representations. The training strategy described above totaled 1500 epochs and took almost 20 hours on our hardware. Even when the training length was drastically reduced, pretrained models achieved a better performance than end- to-end training 2D. With the total budget of 500 epochs (400 + 25 + 75), the final kNN accuracy was 86% Table 1] line 9); with 1000 epochs in total (775 +5 200), it was 89% (Table I] line 8) better than end-to-end training for 5000 epochs. Visually, all three budgets resulted in qualitatively similar embeddings (Fig. A.7).
To test the stability of t-SimCNE, we repeated the training procedure three times with different random seeds. We found that while the exact layout and cluster arrangement differed between runs, qualitatively the embeddings were very similar (Fig. A.8).
SimCLR itself could in principle be used for data visualization if the output dimensionality is set to 3, so that the embedding lies on the 2-sphere S However, we found that this setup only resulted in &NN accuracy of 74% (Table line 2) and the embedding itself looked poor (Fig. A.9)

---

Published as a conference paper at ICLR 2023
Bright horses
Horse heads
White degs
airplane
digg horse
automabile
cat
bird
ship
Dark
dogs
pets
Mounted horses
Red cars
Black
Firetrucks
Sailbosts
Bird heads
Creaking frogs
Duplicate cars
Frags, white backgroun
Ratites
Calorful
Boats (aerial
Boats Iside view)
Grounded planes
Figure 5: Annotated t-SimCNE embedding of the CIFAR-10 dataset. We manually annotated some of the prominent clusters by inspecting the images. Shown images are random selection from the 15 nearest neighbors of the line tip.
EXPLORATORY ANALYSIS SHOWS THE POWER OF t-SIMCNE
The ultimate goal of t-SimCNE is be a tool for exploratory data analysis. In this section demonstrate its power using the same CIFAR datasets as above.
4.1 f-SIMCNE REVEALS SUBCLASS STRUCTURE IN CIFAR-10
We manually annotated the 1-SimCNE embedding of CIFAR-10 for additional structure beyond the 10 originally defined classes (Fig. 5} We found that the embedding exhibited rich cluster structure, with many of the original classes (colors) splitting into several distinct clusters, or 'islands'.
We inspected the images within these clusters and found that they corresponded to meaningful se- mantic concepts. For example, within the 'ship' class, sailboats and other boat types differed clearly, such that sallboats formed isolated cluster. Within the main 'ship' cluster, acrial view and level photographs varied systematically in their position in the 2D embedding. Similarly, the 'birds' class split into three well-separated islands: bird heads, ratites" and small birds. In the "horse' class, horse heads and mounted horses formed separate clusters, and the remaining horse images were separated by color. Importantly, this within-class structure was possible to infer from the original class labels alone, and we did not expect to find it when we started the project. Therefore, t-SimCNE helped us discover additional latent structure in the dataset.
Although neighbor embedding methods are notoriously umreliable in preserving global structure of the data (Wattenberg et al., (2016). we found that t-SimCNE adequately represented some of the between-class structure. For example, the 'automobile' class was split into three main clusters: metallic cars, more colorful cars, and mostly bright red cars. The latter formed separate cluster in the upper part of the orange island, next to one of the 'truck' clusters, consisting of bright red firetrucks.
One feature of the embedding that catches the eye, is a distinct, stripe-like orange cluster on the left (Fig. 5). Its oddly elongated and suspiciously dense shape suggests that it may be an artifact. Indeed, we found that this cluster consisted of near-duplicate pictures of the same three cars, such that three pictures appeared multiple times with only minor variations. A previous study focused specifically near-duplicates in CIFAR datasets (Barz & Denzler 2020 and identified 55 near-
Large running birds such ostriches and emus.

---

Published as a conference paper at ICLR 2023
sel wied
spka
pine tree
maple treu
willow tree
Koa
unu
ar
cak
aeccon
truck.
tank
train
bur
woll
mouse
chimpan
house.
camel
hamsttr
bottle
elsphant
television
castle
wardrobe
lon
rabbit
clock
possum
lamp
Cattle
table
nger
plate
bear
cup
seal
bed
otter
couch
eeopard
caver
squimel
lawn mower
kangaroo
skyscrape
ro
porcupine
snall
tel telephone
butterrly
spider
anake
keyboard
robster
shrew
codrracch
plain
hee
beetle
sera
arad
mountard
bor
appl
whale
tulip
Slonk
dappy
sundoo ainflower
archid
eaterpilll atterpillar
crocadia
dinnsqur
Deap
delphin
shark
surtie
sweer pepper
trouss
aquanum fish
muuhroomm
rase
Figure 6: Annotated I-SimCNE embedding of the CIFAR-100 dataset. Class labels were positioned on the periphery the order atan2(y, x) where (x, y) is the mode of the kernel density estimate of embedding coordinates within each class.
duplicates the "automobile' class (Barz & Denzler 2020 Table 1). With our exploratory analysis found 163 almost identical images in the small orange island (Fig. A.I0). Another study on labeling errors in standard computer vision datasets including CIFAR-I0 missed the near-duplicates (Northeutt et al., (021). This suggests that t-SimCNE is more intuitive and more sensitive than other, more conventional, ways to explore image datasets, and can be useful for quality control in actual practice.
The L-SimCNE visualization also highlighted what parts of the dataset SimCLR-like models struggle with. For example, the model seemed to be confused between dogs and pets, black pets). It is less
parts of these
two classes were merged into one (upper-right corner: black-and noticeable, but within the cluster airborne planes there were some images of birds. Those were pictures of birds flying in the sky, which exhibit lot of visual similarity to airborne planes, hence the model's confusion.
While overall the embedding exhibited a pronounced cluster structure, there were a few outlier points located amidst the white space, which did not seem to belong to any one cluster. We investigated these images and found some obvious examples of outlier images. For example, one image, labeled as 'ship', depicts a person riding a jet ski (Fig. A.I.p). Another image labeled as 'truck', but we unable to make out what actually depicted (Fig. A.II These examples would be hard to find within a dataset of 000 images, but they clearly stick out the 2D t-SimCNE visualization.
4.2 1-SIMCNE ELUCIDATES INTER-CIAS RELATIONS IN CIFAR-100
Similar observations apply to the f-SimCNE embedding of the CIFAR-100 dataset with its much more heterogeneous structure compared to CIFAR-10 (Fig. 6). In terms of the large-scale organi- sation, different animal species were placed mostly on the right side of the embedding, while man- made objects could be found on the left side. In terms of the fine-scale organization, the superclasses were separated very well in the embedding (Fig.3b. as were many individual classes within a single superclass. For example, in the superclass 'flowers' the embedding clearly distinguished between orchids, sunflowers, roses, tulips, and poppy. Similarly, the superclass 'large natural outdoor scenes". images of plain, sea, mountain, cloud, and forest were mostly non-overlapping.
8

---

Published as a conference paper at ICLR 2023
On the other hand, in some other superclasses, individual classes were mixed together. For example, in the superclass 'people' (the topmost cluster), all five classes (boy, man, girl, woman, and baby) were intermingled, suggesting that the model struggled in this region of the feature space. Curiously, the same cluster also contained the 'flatfish' class. We found that the 'flatfish' images often show fishermen holding their catch, explaining their appearance in the 'people" island (Fig. A.12
There were further examples of classes that t-SimCNE positioned next to a seemingly wrong su- perclass. The 'forest' class was separated from other members of the 'large natural outdoor scenes' superclass and placed next to the pine, oak, maple, and willow tree classes. This makes sense, as forest close to trees both semantically and in terms of the image statisties.
These examples suggest that the class and the superclass definitions in the CIFAR-100 dataset do not always form a reliable ontology capturing all aspects of an image.
DISCUSSION
We developed a new self-supervised method, t-SimCNE, visualize image datasets in 2D, and showed that it yields meaningful and interpretable visualizations of CIFAR-10 (Fig. 5) and CIFAR- 100 (Fg.. 6) datasets (kNN classification accuracy 89% and 51% respectively). We are not aware of any other unsupervised parametric methods that could yield comparably good 2D embeddings:
I-SNE embeddings in pixel space are typically very poor as the Euclidean metric is not meaningful for image data Fig.. A.I). Any other neighbor embedding method, be UMAP (Melnnes al. 2018), Large Vis (Tang et al. 2016). TriMap (Amid & Warmuth 2019),
etc., or any of their parametric versions (van der Maaten [Ch et al. 2018 Ding
et al. 2018: [Szubert et al., 2019 Kalantidis et al. 2022 Sainburg al. 2021 Damrich et al.
2023). would have the same issue.
t-SNE embeddings of the trained SimCLR representation are reasonably good (Fig. A.13 however (i) we found them qualitatively to be less interpretable than the t-SimCNE embed- dings, as outliers, artifacts, and subclass structure were less noticeable; and (ii) they are not parametric, i.e. do not allow positioning out-of-sampp images into an existing embedding, which is often relevant in applications.
SimCLR with 3D output yields an embedding on 2-sphere S but it was outperformed by t-SimCNE in terms of ANN accuracy and visual class separation, and also is cumbersome to visualize on a 2D plane as it requires a map projection (Fig. A.9).
I-SNE embeddings of a trained ResNet-based classifier representation Fig. A.14 are not unsupervised and not useful for data exploration, as they do not show much structure within each of the pre-defined classes.
Finally, using readily available off thel ResNet-based classifier pretrained on Ima- geNet and visualizing its representation with i-SNE (or UMAP, TriMap, etc.) is very fast alternative approach to t-SimCNE. It unsupervised a sense that it does not use dataset labels (but is based on supervised ImageNet pretraining). While it is, again, not a para- metric method, we found the resulting embeddings to have good ANN accuracy, especially for larger ResNets, such as ResNet-152 Figs. A. 15 and 16). As it is much faster than t-SimCNE, this approach can make sense for initial exploration of the data, however we found that it is strongly outperformed by t- t-SimCNE in terms of visual class separation, as measured by the clustering- adjusted Rand index (Fig. A.17).
Optimizing our architecture with 2D output was challenging and required a carefully tuned training setup. Our training scheme yielded good 2D embeddings, but led to the worse representation in the Hl space compared to standard SimCLR. This suggests that it may be possible to further improve the training protocol and the fine-tuning schedule, which we leave for future work.
In the future it will be interesting to apply t-SimCNE to larger and more complex datasets, such as ImageNet (Russakovsky et al.) 2015) or subsets, e.g. Tiny ImageNet. It will also be interesting to apply it to real-life scientific data, e.g. in a biomedical domain. Our hope is that methods like t-SimCNE will be able to aid scientific exploration and discovery.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
We thank Wieland Brendel and Evgenia Rusak for discussions on the topic.
This research was funded by the Deutsche Forschungsgemeinnccatt (DFG, Germany's Research Foundation) via the Excellence Cluster 2064 "Machine Learning: New Perspectives for Sci- ence" (390727645), by the German Ministry of Education and Research (Tübingen Al Center, 011S18039A), and by the Cyber Valley Research Fund (D.30.28739).
The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS- IS) for supporting Jan Niklas Böhm.
REPRODUCIBILITY
The code for generating our figures is available at bttps://githab. combmbbee/bbbeeelaa///// t d-simmne// wwwccnn//tre/iioe /irre213 Note that the runs on GPU are deterministic, due nonde- terminism introduced by optimized GPU kernels. Hence a perfect reproduction is impossible, but we confirmed that multiple runs yield very similar results. We always report the mean and standard deviation across several runs, and also show multiple visualizations in Fig. A.8
We implemented user-friendly class TSimCNE following the sklearn-style API; a minimal working example is shown in the Github repository. TSimCNE. fit () follows the setup described in this paper, save for one minor detail. We ran all experiments using dataset-level normalization of pixel values. After the experiments were completed, we realized that this normalization does not have any noticeable influence on the results, hence we omit it in the TSimCNE code. Future development of t-SimCNE will happen in the main branch of the Github repository: https: / com/ berenslab/St stt/:/tssimmmrr simcne.
REFERENCES
Ehsan Amid and Manfred K. Warmuth. TriMap: Large-scale Dimensionality Reduction Using Triplets. arXiv preprint 1rXiv:1910.00200 2019.
Björn Barz and Joachim Denzler. Do we train on data? Purging CIFAR of near-duplicates.
Journal of Imaging, 6(6), 2020.
Etienne Becht, Leland McInnes, John Healy, Charles-Antoine Dutertre, Immanuel WH Kwok, Lai Guan Ng, Florent Ginhoux, and Evan W Newell. Dimensionality reduction for visualizing single-cell data using UMAP. Nature Biotechnology, 37(1):38, 2019.
Erik Bernhardsson.
2013.
Jan Niklas Böhm, Philipp Berens, and Dmitry Kobak. Attraction-repulsion spectrum in neighbor embeddings. Journal of Machine Learning Research, 23(95):1-32, 2022.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference Machine Learning, pp. 1597-1607. PMLR, 2020.
Xinlei Chen and Kaiming He. Exploring simple Siamese representation learning. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15745-15753, 2021.
Hyunghoon Cho, Bonnie Berger, and Jian Peng. Generalizable and scalable visualization of single- cell data using neural networks. Cell Systems, 7(2):185-199) 2,,,
Victor Guilherme Turrisi da Costa, Enrico Fini, Moin Nabi, Nicu Sebe, and Elisa Ricci. solo- learn: A library of self-supervised methods for visual representation learning. Journal of Machine Learning Research, 23(56):1-6, 2022.
Sebastian Damrich, Jan Niklas Böhm, Fred A Hamprecht, and Dmitry Kobak. From t-SNE to UMAP with contrastive learning. In International Conference on Learning Representations, 2023.
10

---

Published as a conference paper at ICLR 2023
Sesbastian Damrich and Fred Hamprecht. On UMAP's true loss function. In Advances in Neural Information Processing Systems, 2021.
Aaron Defazio, Francis Bach, and Simon Lacoste-Julien. SAGA: A fast incremental gradient method with support for non-strongly convex composite objectives. In Advances in Neural Infor- mation Processing Systems, volume 27, 2014.
Alex Blaz-Papkovvchh Luke Anderson-Troeed Chief Ben-Eghan, and Simon Gravel. UMAP re- veals cryptic population structure and phenotype heterogeneity in large genomic cohorts. PLoS Genetics, 15(1 te 1008432, 2019.
Jiarui Ding, Anne Condon, and Sohrab P Shah. Interpretable dimensionality reduction of single cell transcriptome data with deep generative models. Nature Communications, 9(1):1-13, 2018.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. 2006 IEEE Computer Society Conference Computer Vision and Pattern Recognition (CVPR'06), 2:7732-1742, 2006.
Charles R. Harris, K. Jarrod Millman, Stéfan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Couraapeau, Eric Wieser, Julian Taylor, Sebastian Berg. Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fernández del Río, Mark Wiebe, Pearu Peterson, Pierre Gérard-Marchant, Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant. Array program- ming with NumPy. Nature, 585(7825):357-362 September 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp- 770-778, 2016.
Geoffrey E Hinton and Sam T Roweis. Stochastic neighbor embedding. In Advances in Neural Information Processing Systems, pp. 857-864, 2003.
Jerry Hintze and Ray Nelson. Violin plots: A box plot-density trace synergism. The American Statistician, 52(2):181-18 1998.
Tianyang Hu, Zhili LIU, Fengwei Zhou, Wenjia Wang, and Weiran Huang. Your contrastive learn- ing is secretly doing stochastic neighbor embedding. In International Conference on Learning Representations, 2023.
Lawrence Hubert and Phipps Arabie. Comparing partitions. Journal of Classification, 2:193-218. 1985.
John D. Hunter. Matplotlib: A 2D graphics environment. Computing in Science & Engineering, 9 (3):90-95, 2007.
Li Jing, Paseal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. In International Conference Learning Representations, 2022.
Yannis Kalantidis, Carlos Eduardo Rosar Kos Lassance, Jon Almazán, and Diane Larlus. TLDR: Twin learning for dimensionality reduction. Transactions on Machine Learning Research, 2022.
Andrej Karpathy, t-SNE visualization of CNN codes, 2014. URL https::/cc.staafoot
edu edu/ httpp/ppppeaappatt////eeeeeee cnnppaaethy//ccmmd//////
Dmitry Kobak and Philipp Berens. The art of using t-SNE for single-cell transcriptomics. Nature Communications, 10:5416, 2019.
Alex Krizhevsky, Learning multiple layers of features from tiny images. Technical report, University of Toronto, April 2009.
George C "Linderman, Manas Rachh, Jeremy G Hoskins, Stefan Steinerberger, and Yuval Kluger. Fast interpolation-base t-SNE for improved visualization of single-cell RNA-seq data. Nature Methods, 16(3):243, 2019. 2019.
11

---

Published as a conference paper at ICLR 2023
Iyy Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts, In Inter- national Conference on Learning Representations, 2017.
Leland McInnes, John Healy, and Steve Astels. hdbscan: Hicrarchical density based clustering. The Journal of Open Source Software, 2(11): 2017.
Leland McInnes, John Healy, and James Melville. UMAP: Uniform manifold approximation and projection for dimension reduction. arXiv: /802 082200342,, 2018.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wier- stra, Shane I Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 2015.
Karthik S Narayan, Ali Punjani, and Pieter Abbeel. Alpha-beta divergences discover micro and macro structures in data. In International Conference on Machine Learning, pp. 796-804, 804, 2015.
Curtis G Northcutt, Anish Athalye, and Jonas Mueller. Pervasive label errors in test sets destabi- lize machine learning benchmarks. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Vetito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, volume 32, 2019.
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit-lear: Machine learning in Python. The Journal of Machine Learning Research, 12:2825-2830, 2011.
Pavlin Gregor Poličar, Martin Strazar, and Blaz Zupan. TSen a modular Python library for t-SNE dimensionality reduction and embedding. bioRsiiv:311877, 2019.
Samuel Roweis and Lawrence Saul. Nonlinear dimensionality reduction by locally linear embed- ding. Science, 290(5500): 2326, 2000.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 1 115(3)211 1112222, 2015.
Tim Sainburg, Leland McInnes, and Timothy Q Gentner. Parametric umap embeddings for repre- sentation and semisupervised learning. Neural Computation, (3(1)):288129011 2021.
Benjamin Schmidt. Stable random projection: Lightweight, general-purpost dimensionality reduc- tion for digitized libraries. Journal of Cultural Analytics, 2018
Benjamin Szubert, Jennifer E Cole, Claudia Monaco, and Ignat Drozdov. Structure-pesserving visualisation of high dimensional single-cell datasets. Scientific Reports, 911):1-10, 2019.
Jian Tang, Jingzhou Liu, Ming Zhang, and Qiaozhu Mei. Visualizing large-scale and high- dimensional data. In International Conference on World Wide Web, pp. 287-297, 2016.
Yuandong Tian. Understanding deep contrastive learning via coordinate-wise optimization. In Al- ice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv 1807 0877.03748, 2018
Laurens van der Maaten. Learning parametric embedding by preserving local structure. In Pro- ceedings of the International Conference on Artificial Intelligence and Statistics, pp. 384-391. PMLR, 2009.
12

---

Published as a conference paper at ICLR 2023
Laurens van der Maaten. Accelerating t-SNE using tree-based algorithms. The Journal of Machine Learning Research, ((1):3221-32455 2014.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of Machine Learning Research, (NNvv):559-22605 2008.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 9929-9939, 13-18 Jul 2020.
Yingfan Wang, Haiyang Huang, Cynthia Rudin, and Yaron Shaposhnik. Understanding how dimen- sion reduction tools work: An empirical approach to deciphering t-SNE, UMAP, TriMap, and PaCMAP for data visualization. Journal of Machine Learning Research, 22(201):1-73, 2021.
Martin Wattenberg, Fernanda Viégas, and lan Johnson. How t-SNE effectively. Distill, 2016.
Zelin Zang, Siyuan Li, Di Wu, Ge Wang, Kai Wang, Lei Shang, Baigui Sun, Hao Li, and Stan
Dime: Deep local-flatness manifold embedding. In Computer Vision ECCV 2022, 576-592, 2022.
13

---

Published as a conference paper at ICLR 2023
APPENDIX A
SUPPLEMENTARY TABLES AND FIGURES
Table A.I: Model performance on the CIFAR-100 dataset. See Table
for description.
#
Model
dim
Epochs
Linear in H
kNN in Z
Loss
Time (hr.)
Cosine (SimCLR)
128
1000
66.7 0.5%
58.0 = 0.2%
5.8
12.0 0.1
2
Cosine
3
1000
52.4 + 0.1%
15.7 + 0.2%
6.3
11.9 + 0.0
3
Euclidean
128
1000
59.5 0.2%
54.5
2.5
11.9 0.1
Euclidean
2
1000
0..4%
-.2%
4.3
11.7
5
Cos. Euclidean
2
1500
65.1 0.1%
3.6
17.5
Eucl. Bell->Euue Euclldean
2
1500
59.3 = 0.4%
+.11%
2.9
17.4 0.1
CIFAR-10
CIFAR-10D
Figure A.I: t-SNE embedding the CIFAR-IO (a) and CIFAR- datasets in pixel space. Each image is 32 x 32 with three color channels, corresponding to vectors of dimensionality 32-32-3 3072. They were embedded using openTSNE (Policar al. 2019) with default parameters. Colors correspond to classes as in Figs. and 6. ANN accuracies were 33% and 13% (CIFAR- 100 classes) in the embedding, and 33% and 15% when measured directly in the 3072-dimensional pixel space. For reference, our method I-SimCNE obtained 89% and 51 % respectively.
airplane
automoble
bird
catt
deer
frog
* horse
truck
Figure A.2: Ten random images from each class of the CIFAR-10 dataset.
14

---

Published as a conference paper at ICLR 2023
CIFAR 10
b
CIFAR 100
0.90
0.85
0.80
Euclidean 1280
Euc Euc
0.70
Euclidean 2D Cosine
15
Figure A.3: kNN accuracy in the Z space of various models listed in Tables and A.1 as function of k. Shading shows standard deviations across three runs. (a) CIFAR-IO. (b) CIFAR-100.
fine-tuning
b
fine-tuning
fine-tuning
10
100
100
Figure A.4: Different stages of fine-tuning when training (-SimCNE on CIFAR-10. (a) The embed- ding after random initialization of the 2D output layer. (b) The embedding after training the output layer for 50 epochs, while the rest of the network stays frozen. (c) The final embedding after fine- tuning the entire network for 450 epochs. Colors as in Fig.
Before fine-tuning
Linear fine-tuning
fine-tuning
100
100
100
Figure A.5: Different stages of fine-tuning when training t-SimCNE CIFAR-100. (a) The em- bedding after random initialization of the 2D output layer. (b) The embedding after training the output layer for 50 epochs, while the rest of the network stays frozen. (c) The final embedding after fine-tuning the entire network for 450 epochs. Colors as Fig.
15

---

Published as a conference paper at ICLR 2023
CIFAR-10
CIFAR-100
10a
Casine
10:1-
100
eigenvalue index
eigenvalue index
Figure A.6: Eigenvalue spectrum of the covariance matrix in the Z space for 128-dimensional mod- els trained with the cosine and with the Euclidean losses. (a) CIFAR-10. (b) CIFAR-100.
500 epochs (400, 25. 75)
b
1000 epochs (775, 25. 200)
1500 epochs (default) (1000, 50, 45D)
100
100
100
Figure A.7: 1-SimCNE embeddings of CIFAR-10 dataset with different runtime budgets. (a) 500 epochs in total (400 for pretraining, 25 for the readout training. 75 for fine-tuning). (b) 1000 epochs
in total (775 for pretraining, 25 for the readout training, 200 for fine-tuning). (a) 1500 epochs
in total (1000 for pretraining. 50 for the readout training, 450 for fine-tuning). Embeddings were flipped along the z- and/or y-axis to maximize the alignment. Colors as in Fig. 2]
100
100
100
Figure A.8: t-SimCNE embeddings of CIFAR- 10 dataset, trained from scratch with three different random seeds. Embeddings were flipped along the r- and/or y-axis to maximize the alignment. Colors as in Fig.
16

---

Published as a conference paper at ICLR 2023
b
Figure A.9: 3D visualization of CIFAR-IO obtained with SimCLR with 3D output. (a) Embedding vectors on the unit sphere. (b) Hammer projection (equal-area) dimensions. Colors in Fig. 2
Figure A.10: Near-duplicates in CIFAR-IO. A zoom-in into the 4-SimCNE embedding. There are three distinct images, duplicated many times with small variations:
(1) front view of car, (2) rear view of a car, and (3) side view of car. Colors as in Fig. 2. For this image the aspect ratio of the embedding has not been preserved.
Figure A.11: Outliers the L-SimCNE embed- ding of CIFAR-10. (a) A cat, almost blurred into the background. (b) An animal, labeled as 'deer', emerging from the water. (c) A per- son on a jet ski, labeled as 'ship' (d) Mud in the foreground, and a truck unloading it in the background. (e) A cat silhouette. (f) Unknown, labeled 'truck'. (g) Unknown. labeled as 'truck'. (h) Unknown, labeled as 'plane'. Col-
ors as in Fig. 2
17

---

Published as a conference paper at ICLR 2023
Figure A.12: Images of the class 'flatfish' in 1IFAR-100 located next to the 'people" super- class in t-SimCNE embedding. Colors as in Fig. 6 For this image the aspect ratio of the embedding has not been preserved.
CIFAR-10
b
CIFAR-100
100
100
Figure A.13: t-SNE of the 512-dimensional standard (with cosine loss) SimCLR representation of CIFAR- 10 (a) and CIFAR-100 (b). We used openTSNE (Poličar et al. 2019) with default parame- ters. Colors correspond to classes as in Figs. 2 and 6 The kNN accuracies were 91 % for CIFAR-10
(a) and 57% for CIFAR-100 classes (b).
CIFAR-10
b
CIFAR-100
Figure A.14: t-SNE embedding of the CIFAR-10 (a) and CIFAR-100 (b) representations obtained with a ResNet-18 18 trained with the supervised cross-entropy loss on the same CIFAR data. The network architecture was the same as for the SimCLR experiments, but the projection head was mapping to ten (for CIFAR-10) or 100 (for CIFAR-100) softmax dimensions. We trained each network for 100 epochs, with initial learning rate 0.1 and linear annealing down to We used the same set of data augmentations as used for contrastive learning (Chen et al.) 2020). Only training images were used for training; the test set accuracy after training was 92.8% for CIFAR-10 and 72.4% for CIFAR-100 classes. The ResNet output layer H (see Fig. 1) was used the input to t-SNE (both training and test images together). We used open'l openTSNE (Poličar et al. 2019) with default parameters. Colors correspond to classes as in Figs. and
18

---

Published as a conference paper at ICLR 2023
CIFAR-10 UMAP
L.SNE
Arn-80%
b
Kin 74%
TriM ap
Arow 70%
100
10
CIFAR-100
d
1:SNE
Ann 51%
UMAP
Rnn 38%
f
TriMap ap
km 29%
100
10
100
Figure A.15: Embeddings of the CIFAR-I0 (a-c) and CIFAR-100 (d-f) representations obtained with ResNet-18, 18, pretrained on the ImageNet classification task. We used pretrained weights avail- able in PyTorch (Paszke et al. 2019). The input images were first resized to 256 x 256 pixels and then center-cropped to 224 X 224 pixels, following He et al. (2016). The ResNet output layer H (see Fig. 1 was used as the input to the visualization algorithms. We used OSEn'TNNE (Poličar al.) 2019). UMAP (McInnes et al. 2018). and TriMap (Amid & Warmuth] 2019) with default parame- ters. Colors correspond to classes in Figs. 2 and 6 KNN classification accuracies indicated in the corner of each panel (for CIFAR- (00, is the accuracy on the class level).
19

---

Published as a conference paper at ICLR 2023
CIFAR-10 (SNE(ResNe118)
L-SimCNE
b
89%
82%
100
100
d
(-5NE/ResNet50)
(R5NE((esN)
f
(-SNE(ResNe:152)
82%
83%
knaw 88%
100
100
CIFAR-100
L-SimCNE
h
(SNE(ResNe118)
i
(-5NE(ResNeL34)
52%
52%
57%
100
100
100
(:SNE\)esNet50)
k
(-SNE(ResNet101)
(-SNE(ResNet152)
505
Figure A.16: Visualizations of t-SimCNE and of tNE applied to the representation obtained using Imageeeet-retrraiedd ResNet networks of different size. (a-f) CIFAR-10. (g-1) CIFAR- 100.
20

---

Published as a conference paper at ICLR 2023
CIFAR 10
b
SimONE
SNE(PesNet18)
(NE(ResNe1101)
SNE(ResNee152)
0.8
71%
adjusted foird tredex
0.6
41% 40%
0.0
aibe
clatte 285 manssamllss 5
duster mi somales 65
net "aaaro es 125
rencsamales 95
CIFAR-100
t-SiMCNE
t-SNE(ResNet18)
t-SNE(ResNec101)
t-SNE(ResNec152)
adjasted Rand index
0.1
100
100
CONERWIED
clarter 215 IEE ciuceers
clatter 245
duater HEn samples
mh samples
SE
Figure A.17: Class separability comparison between t-SimCNE and i-SNE applied to the rep- resentation obtained using ImageNet-pretrained ResNet networks. We used clustering sess how strongly classes were separated from each other in 2D. (a) The adjusted Rand in- dex (ARI; Hubert & Arabie] 1985) between the clusters and the class labels on CIFAR-10. The clusters were obtained with HDBSCAN (McInnes et al.) (017) using the parameter grid (5, 15, 295} mi mainmammpll 1 es E (5, 15,1..... 145} with the con- straint min.samples <
The violin plots (Hintze & Nelson 1998) show the distribution of the ARI values across different HDBSCAN parameter combinations, with the best value indicated above. The black dot denotes the ARI of the default HDBSCAN parameters (min.cluster.s -siee = min
min-saaples amples = 5). We used ResNet weights that are available in Py- Torch. (b-e) The 2D visualizations of CIFAR-10 10 using t-SimCNE and t-SNE of pretrained ResNet representations. The cluster centroids corresponding to the best clustering (in terms of ARI) are shown as black cross marks, and the corresponding HDBSCAN parameters given below. (f-j) The same for CIFAR-100.
21


Reviews:
[{"review_id": "dG93g3fhxi", "reviewer": "Reviewer_LCjY", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Interesting paper concerning unsupervised visualization of image data using neighbour embeddings that borrows ideas of ideas from contrastive learning.\nIt is motivated by the need of mapping from the high-dimensional pixel image  space into lower dimensions (two, in this study), where, as the authors pose it, \"distances in pixel space are often not capturing our sense of similarity\", so that \"neighbors are not semantically close\".", "summary_of_the_review": "Interesting paper concerning unsupervised visualization of image data using neighbour embeddings that borrows ideas of ideas from contrastive learning.\nIt is motivated by the need of mapping from the high-dimensional pixel image  space into lower dimensions (two, in this study), where, as the authors pose it, \"distances in pixel space are often not capturing our sense of similarity\", so that \"neighbors are not semantically close\".\nThe paper is quite well written and its structure is quite fresh and didactic.\nIt uses contrastive learning SimCLR method that generates high-dim image representations and it is used to directly optimize a 2D embedding.\nThe main novel contribution is the pretraining strategy required for this proposed model to work, specially given that two other recent papers seem to be chasing the same idea (Zang et al., Hu et al., both from this year) Such pretraining strategy is a bit too trial and error for my taste (in the sense that it makes you wonder whether it should be tailored to the analyzed dataset at hand), althoug, arguably, understandably so.\nThe literature background section, by the way, is excellent.\nIn section 3.3, for embedding quality assessment, authors use kNN and indicate that \"For kNN accuracy, we used the scikit-learn [...] implementation with k = 15\". Why k=15? What's the rationale for that choice? Did authors investigate other alternative values of k?\nThe experimental body, even if not being fully conclusive, provides very good evidence of the data exploratory potential of the model.\n\nOther:\nIt would be useful to clarify in the introduction that Figures referenced as A.x refer to an appendix.\nFigure 1 should be better located at the end of the introduction.\np.2, section 2: \"In this work, we are interested in an unsupervised but parametric mapping that ALLOWS embedding ...\"", "strength_and_weaknesses": "Strengths:\nQuite well written.\nGood discursive didactic style.\nMarginal but useful technical novelty.\nExcellent review of existing literature.\nCode available.\n\nWeaknesses:\nMarginal/incremental novelty\nA bit too much setting-cooking, which arises some doubts as to the generalizability of the proposal.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "EuzINSqk5C", "reviewer": "Reviewer_TNpm", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper reasons about the relationship between SimCLR, a recent self-supervised learning method, and tSNE, a popular dimensionality technique for data visualization. At the intersection of these two methods, the paper proposed a new method, based on self-supervised learning, for high-dimensional data visualization (to be used instead of tSNE). The new method, known as t-SimCNE, removes the 1-norm constraint on the standard SNE loss, replacing cosine with Euclidean loss, and trains the SimCLR network in two stages to achieve better performance. Compelling results with respect to tSNE are demonstrated in the CIFAR-10 and CIFAR-100 datasets, both qualitative and accuracy.\n", "summary_of_the_review": "The paper offers a novel and thorough empirical and mathematical analysis of tSNE and SimCLR and derives a learning-based dimensionality reduction method. \n\nEDIT: After reading the response and comments by the other reviewers, I believe that the contribution is marginal. The paper would benefit from a more thorough discussion and comparisons to prior work, as well as ablation studies with different backbone settings. I am also concerned about the large runtime of the method, which might prevent it from being used more widely in practice.", "strength_and_weaknesses": "Strengths\n- A novel application of self-supervised learning, dimensionality reduction, is discussed and explored. \n- Paper is clearly written\n- A mathematical relationship between the SNE and simCLR loss is derived\n- Ablation study of the different training strategies (number of training epochs, type of loss, type of training/fine-tuning) elucidates when does the proposed method works and does not work\n- Experiments are conducted on two standard public image benchmarks (CIFAR-10 and CIFAR-100 datasets)\n- Results are compelling and convincing\n\nWeaknesses\n- There is no computational analysis provided. tSNE is known as a relatively lightweight technique that can be used to quickly browse a large dataset. It is not clear whether the proposed method, t-SimCNE, can be run in a similar amount of time and amount of computational resources (batch size of 1024 is large and requires high-memory GPU). A discussion of when the proposed method would be more suitable, in comparison to tSNE, is missing. \n- The key weakness of tSNE pointed out in the intro is that it does not work well on large datasets (reader is pointed to figure A.1). However, in Figure A1, it is not clear what the problem is? Perhaps a zoom in to the graph to illustrate the mix of the labels would help\n- In Equation (3), a relationship between the stochastic neighbor embedding (SNE) and the SimCLR loss is derived. What is the relationship between SNE and t-SNE?\n- What happens if the pre-training on SimCLR is done on a different dataset (e.g., ImageNet), as a strategy to save training time?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "5hQOXnWnhYn", "reviewer": "Reviewer_nGad", "summary": "", "questions": "", "limitations": "", "rating": 10, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In recent years, neighbor embedding methods, such as t-SNE and UMAP, are popular for visualizing high-dimensional data. Motivated by the success of contrastive learning, the authors propose a new method t-SimCNE for unsupervised visualization of image data. The proposed method builds on the popular SimCLR, but replaces the loss with a heavier-tail loss as is used in t-SNE, because it is better for visualization 2D representation. \n\nThe authors detailed training strategies for training/pre-training the new model, showing that the proposed strategies are crucial for resolving nontrivial implementation issues. The authors also present thorough experimental results, showing that t-SimCNE is good at showing revealing fine-grained representation information and subclass structure.", "summary_of_the_review": "The authors provide a very useful method for visualizing image data by combining SimCLR and t-SNE in a nontrivial way (implementation-wise). I think that this paper has potential impact, and would highly recommend acceptance of this paper.", "strength_and_weaknesses": "For unsupervised visualization of image data, one challenge is how to provide as much information as possible in a 2D space. By utilizing the t-SNE loss function in the neighbor embedding literature, the authors propose a natural way to embed and visualize images, which they call t-SimCNE.\n\n**Strengths:** It is clearly shown that it is nontrivial to train the model with a loss similar to t-SNE. Moreover, the authors provide several pre-training strategies and demonstrate that their pretraining/fine-tuning with Euclidean loss strategy works the best. See Figure 2 and 3.\n\nFurthermore, it is clearly shown that data visualization t-SimCNE displays fine-grained structures that is very useful for data diagnostics and interpretation. \n\n**Weaknesses:** I do not find major weaknesses. Perhaps the authors make comments on (1) if there is dimensional collapse issue during training, especially if we do not follow the proposed training strategies, and (2) what information is lost when embedded data is restricted to 2D instead of the initial 128D.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "1hW-702qshE", "reviewer": "Reviewer_25XC", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a method for unsupervised visualization of image datasets. The proposed method relies on first training a NN using contrastive learning and using its representation to minimize another contrastive loss to project the embedding to 2D. The results are validated on standard image datasets.", "summary_of_the_review": "The proposed approach has a significant amount of overlap with TriMap, a contrastive-loss-based dimensionality reduction method. The authors need to clarify how their method differs from TriMap, what advantages (if any) it provides, and perform a comparison.\n\n\nUPDATE: I'm increasing my score to acknowledge the authors' efforts to include additional results. However, I still believe the contributions of the paper are very limited. Please see the response to the authors' feedback.", "strength_and_weaknesses": "I'm surprised that the authors have missed an important contrastive-loss-based dimensionality reduction method, called TriMap [1], that relies on a very similar principle for generating 2D embeddings. The proposed method can be seen as a variant of TriMap where the input representation is generated by a ResNet.\n\n[1] Ehsan Amid and Manfred K. Warmuth. \"TriMap: Large-scale dimensionality reduction using triplets.\" arXiv preprint arXiv:1910.00204 (2019).\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster