TLDR: Twin Learning for Dimensionality Reduction
Authors: Yannis Kalantidis
Carlos Eduardo Rosar Kos Lassance
Jon Almazán
Diane Larlus
Venue: Accepted by TMLR ()

Abstract: Dimensionality reduction methods are unsupervised approaches which learn low-dimensional spaces where some properties of the initial space, typically the notion of “neighborhood”, are preserved. Such methods usually require propagation on large k-NN graphs or complicated optimization solvers. On the other hand, self-supervised learning approaches, typically used to learn representations from scratch, rely on simple and more scalable frameworks for learning. In this paper, we propose TLDR, a dimensionality reduction method for generic input spaces that is porting the recent self-supervised learning framework of Zbontar et al. (2021) to the specific task of dimensionality reduction, over arbitrary representations. We propose to use nearest neighbors to build pairs from a training set and a redundancy reduction loss to learn an encoder that produces representations invariant across such pairs. TLDR is a method that is simple, easy to train, and of broad applicability; it consists of an offline nearest neighbor computation step that can be highly approximated, and a straightforward learning process. Aiming for scalability, we focus on improving linear dimensionality reduction, and show consistent gains on image and document retrieval tasks, e.g. gaining +4% mAP over PCA on ROxford for GeM- AP, improving the performance of DINO on ImageNet or retaining it with a 10× compression.

Published in Transactions on Machine Learning Research (06/2022)
TLDR: Twin Learning for Dimensionality Reduction
Yannis Kalantidis
NAVER LABS Europe
Carlos Lassance
carlos.lasencceehneeerebb..co
NAVER LABS Europe
Jon Almazán NAVER LABS Europe
jon.clmaccannnnnneriiais..cm
Diane Larlus NAVER LABS Europe
Hiane.lartur @wwweelabs.cco......mmc.o..
Reviewed on OpenReview: https: openreview. net/ het/foruntt forun? id= 86fhqdB0b
B
B
B
d
ENCODER
PROJECTOR
LT
d'
8
B
B
ENCODER
PROJECTOR
CROSS CORR. MATRIX
IDENTITY MATRIX
d'
HIGHER DIMENSIONAL REPRESENTATION
LOVER-DIMENSIONA REPRESENTATION
Figure 1: Overview of the proposed TLDR dimensionality reduction method. Given feature vectors from a generic imput space, we use nearest neighbors to define a set feature pairs whose proximity we want to preserve. We then learn dimensionality--eeuttiii function (the encoder) by encouraging neighbors the input space have similar low dimensional representations using the Barlow Twins loss (Zbontar et al., 2021).
Abstract
Dimensionality reduction methods are unsupervised approaches which learn low-dimensional spaces where some properties of the initial space, typically the notion of "neighborhood", are preserved. Such methods usually require propagation on large A-NN graphs or complicated optimization solvers. On the other hand, self-supervised learning approaches, typically used to learn representations from scratch, rely on simple and more scalable frameworks for learning. In this paper, we propose TLDR, a dimensionality reduction method for generic input spaces that is porting the recent self-supervised learning framework of Zbontar et al. (2021) to the spe- cific task of dimensionality reduction, over arbitrary representations. We propose to use near- neighbors build pairs from training and a redundancy reduction loss learn an en- coder that produces representations invariant across such pairs. TLDR is method that is sim- ple, easy to train, and of broad applicability; it consists of an offline nearest neighbor computa- tion step that can be highly approximated, and a straightforward learning process. Aiming for scalability, we focus on improving linear dimensionality reduction, and show consistent gains on image and document retrieval tasks, e.g. gaining +4% mAP over PCA on ROxford for GeM- AP, improving the performance of DINO on ImageNet or retaining with a 10x comprossion.
Code available at: attps:// //:::://gttte gightu......... cwm/ com/naver/tlde

---

Published in Transactions on Machine Learning Research (06/2022)
1 Introduction
Self-supervised representation learning (SST.) has been shown to produce representations that are highly transferable to wide number of downstream tasks via encoding invariance to image distortions (Chen et al., 2020a; He et al., 2020; Caron et al., 2020). Methods like BYOL (Grill et al., 2020), DINO (Caron et al., 2021) or Barlow Twins (Zbontar et al., 2021) start from structured inputs (i.e. a 2D grid of pixel intensities) and learn a large encoder with hundreds of millions of parameters that transforms this input into a reprosentation vector useful for downstream lasks. Central to their success is the scalable and easy-lo-pplinize learning framework that such methods adopt, often based on pairwise loss functions with or without constrasting pairs.
Dimensionality reduction refers to set of (generally unsupervised) approaches which aim learning low-dimessional spaces where properties of an initial dimessoll imput space, e.g. proximity "neighborhood", are preserved. Unlike SSL, dimensionalityy reduction starts from any trustworthy, potentially blackbox, representation and learns a simple (often linear) encoder without any further assumptions on the nature of the input space beyond the fact that structure should be preserved. Dimensionality reduction is a crucial component in many areas as diverse as biology or AT, and used for tasks ranging from visualization and compression, to indexing and retrieval. This is bocause, more often than not, learning a small encoder on top of an existing representation is easier than training a lower dimensionality representation end-to-end; in many cases, training from scratch or fine-tuming a representation is too costly or simply not possible features might be produced by blackbox module or sensor. Considering how manifold learning methods lack scalability and usually require propagation on large k-NN graphs or complicated optimization solvers, it is only natural to wonder: Can we borrow from the highly successful learning frumeworks of self-supervised representation learning to design dimensionality reduction approaches?
In this paper, we propose Twin Learning for Dimensionality Reduction or TLDR, generic dimensionality- reduction technique where the only prior is that data lies on reliable manifold we want to preserve. Tt is based on the intuition that comparing data point and its nearest neighbors is a good "distortion" to learn from, and hence good way of approximating the local manifold geometry. Similar to other manifold Icarning methods (Roweis & Saul, 2000; Van der Maaten & Hinton, 2008; Belkin & Niyogi, 2003; Donoho & Grimes, 2003; Hadsell et al., 2006) we use Euclidean nearest neighbors as a way of defining distortions of the imput that the dimensionality reduction function should be invariant to. However, unlike other manifold learning methods, TLDR does not require eigendecompositioon, negatives to contrast, or cumbersome optimization solvers; it simply consists of an offline nearest neighbor computation step that can be highly approximated without loss in performance and straightforward stochastic gradient descent learning process. This leads to highly scalable method that can learn linear and non-linear encoders for dimensionality reduction while trivially handling aut-of-sample generalization. We show an overview of the proposed method in Figure 1.
TLDR is meant as a post-hoc dimensionality reduction method, i.e. on top of learned reprosentations. We show that it provides an easy way of not only compressing representations, but also improving the performance of large seate-of-the-arr models relying on those reduced representations, without the need fine-tune large encoders. Training end-to-end or fine-tuning requires large resources, specially when starting from large pre-trained models composed of millions or billions of parameters, and application tasks requires non-trivial know-how. Those issues fade when large models are treated as feature extractors, and we only learn a linear or MLP encoder to compress those features vectors in a representation that is more suited to the desired task or dataset. This highlights the natural link between representation learning and dimensionality reduction: we could use the output of models like Barlow Twins, BYOL, DINO, BERT or any reprosentation lcarning method as the input for our TLDR encoder. Yet. those two steps are not interchangeable and we focus on the latter.
certain domains
Aiming at large-scale search applications, we focus on improving linear dimensionality reduction with a compact encoder, an integral part of the first-stage of most retricval systems, and an area where PCA (Pearson, 1901) is still the default method used in practice (Tolias et al., 2020; Weinzaeplel et al., 2022). We present a large set of ablations and experimental results on common benchmarks for image retrieval, as well as on the natural language processing task of argument retrieval. We show that one can achieve significant. gains without altering the encoding and search complexity: for example we can improve landmark image retrieval with GeM-AP (Revaud et al., 2019) on ROxford5K (Radenovié et al., 2018a) by almost 4 mAA P
2

---

Published in Transactions on Machine Learning Research (06/2022)
points for 128 dimensions, commonly used dimensionality,, by simply replacing PCA with linear TLDR encoder. Similarly, we are able to improve the state-of-the-art, retrieval performance of DINO (Caron et al., 2021) representations on ImageNet (Russakovsky et al., 2015), even when compressing the vectors tenfold.
We also perform extensive evaluations using DINO models with a ViT backbone and report state-of-the-art retrieval results for fully self-supervised learning methods on multiple datasets like ROxford, RParis, and ImageNet. We further show that, given access to unlabeled data from the downstream domain, simply learning linear TLDR encoder on top of generic DINO representations leads to large performance gains on the downstream task, without the need to fine-tune the backbone. Finally, we show that TLDR. is robust to both approximate neighbor pair generation, as well as subsequent vector quantization.
Contributions. We int TLDR, a dimensionnlitt reduction method that achieves neighborhood embedding learning with the simplicity and effectiveness of recent self-supervised visual representation learning losses. Aiming for scalability, we focus on large-scale image and document retricval where dimensionality reduction is still an integral component. We show that replacing PCA (Pearson, 1901) with a lineor TLDR methods without any additional computational
encoder can greatly improve the performance of state-of-the complexity. We thoroughly ablate parameters and show that our design choices allow TLDR to be robust to large range of hyper- paree-ppaaeteerss and is applicable to a diverse set of tasks and input spaces.
2 Twin Learning for Dimensionality Reduction
Starting from a set of unlabeled and high-dimennional features, our goal is to learn a lower-dimensional space which preserves the local geometry of the larger input space. Assuming that we have no prior knowledge other than the reliability of the local goometry of the input space, we use ncarest neighbors to define a set of feature pairs whose proximity we want to preserve. We then learn the parameters of a dimenniionalii---edulliio function (the encoder) using loss that encourages neighbors in the input space to have similar representations, while also minimizing the redundancy between the components of these vectors. Similar to other works (Chen et al., 2020b;a; Zbontar et al., 2021) we append a projector to the encoder that produces a representation in a very high dimensional space, where the Barlow Twins (Zbontar et al., 2021) loss is computed. At the end of the learning process, the projector is discarded. All aforementioned components are detailed next. We call our method Twin Learning for Dimensionality Reduction or TLDR, in homage to the Barlow Twins loss. An overview is provided in Figure 1 and in Algorithm
Preserving local neighborhoods. Recent self-supervised learning methods define positive pairs via hand-cralted distortions that exploit prior information from the input space. In absence of any such prior knowledge, defining local distortions can only be achieved via assumptions on the input manifold. Assuming a locally linear manifold, for example, would allow using the Euclidean distance as a local measure of on-manifold distortion and using nearest neighbors over the training set would be a good approximation for local neighborhoods. Therefore, we construct pairs of neighboring training vectors, and learn invariance to the distortion from one such vector to another. Practically, we define the local neighborhood of each training sample as its & nearest neighbors. Although defining local neighborhood in such uniform way over the whole manifold might seem naive, we experimentally show that not only it is sufficient, but also that our algorithm is robust across a wide range of values for k (see Section 3.2).
Using nearest neighbors is of course not the only way of defining neighborhoods. In fact, we show alternative results with a simplified variant of TLDDR (denoted as TLDRgg where we construct pairs by simply adding Gaussian noise to an input vector. This a bascline resembling denoising autocncoders, although in our case we are using a) an asymmetric encoder-decoder architecture and b) the Barlow twins loss instead of reconstruction loss.
Notation.
Our goal is to learn an encoder fo : RR. - Rd that takes as input a vector x E RD and outputs a corresponding reduced vector z = [x( E Rd, with d << D. Without loss of generality, we define the encoder to be a neural network with learnable parameters 0. Let X be a (training) set of datapoints in RRD, the D-dimensional input space. Let x E R be a vector from X. Ny(xx) is composed of the k ncarest neighbors of z. For vector y E X from the training set: y Nx(x) * y E arg, mingex d(æ, g), where d(.--) denotes the Fuclidean distance. Although the definition above can be trivially extended to non non-Fuclidean
3

---

Published in Transactions on Machine Learning Research (06/2022)
Algorithm 1: Twin Learning for Dimensionality Reduction (TLDR)
Input: Training set of high-dimensional vectors X, hyper-parameter l
Output: Corresponding set of lover-dimensional vectors 2,
dinensionalliy--eduttio
function Sa RDD + Rd
1: Calculate the k (approximate) nearest neighbors, i.e. (x)x),, for every a E R.
2: Create positive pairs (z, y) by sampling y from the set (x)..
3: Learn the parameters 0 of fo and & of gs by optimizing the Barlow Twins' loss (Eq. (1)).
distances and adaptive neighborhoods (e.g. defined by a radius), without loss of generality we present our method and results with pairs from k Euclidean neighbors. We define neighbor pairs as pairs (2,,) X x X where y E ENx
Learning à la Barlow Twins. Although contrastive losses were proven highly successful for visual representation learning, explicitly minimizing the redundancy of the output dimension is highly desirable for dimensionality reduction: having a highly informative output space is more important than a highly discriminative one. We therefore choose to learn the parameters of our encoder by minimizing the Barlow Twins loss function (Zbontar et al., 2021), C: which suits C: the problem perfectly, Similar to (Zbontar et al., 2021), we append projector Is to the encoder fo, allowing to calculate the loss in (third) representation space which is not the one that will be used for subsequent tasks. That exlended space can possibly be much larger. We detail the encoder and the projector later. Let 8 = (fo(()) be the output vector of the projector, & E Rd., Given a pair of neighbors (x")x") and the corresponding vectors 2 2 after the projector, the loss function LBT is given by:
24 25 Es b,t b.j
LBT -(1-ca) (
+ ch where Cip (tj
(1)
V V
where b indexes the positive pairs in batch, and j are two dimensions from Rd 1 i, d') and x is a prper-parameter. C is the d' x d' cross-correlation matrix computed and averaged over all positive pairs from the current batch. The loss is composed of two terms. The first term encourages the diagonal elements to be equal to 1. This makes the learned representations invariant to applied distortions, i.e the datapoints moving along the input manifold in the neighborhood of a training vector are encouraged to share similar representations in the output space. The second term is pushing off-diagonal elements towards 0, reducing the redundancy between output dimensions, a highly desirable property for dimensionality reduction.
The redundancy reduction term can be viewed as a soft-whitening constraint on the representations and, as shown in Zbontar et al. (2021), it works better than performing "hard" whitening on the representa- tions (Ermolov et al., 2021). Finally, it is worth noting that understanding the dynamics of learning without contrasting pairs is far from trivial and beyond the scope of this paper; we refer the reader to the recent work by Tian et al. (2021) that studies this learning paradigm in depth and discusses why trivial solutions are avoided when learning without negatives as in Eq. (1).
The encoder fa. We consider number of different architectures for the encoder:
The most straight-forward choice for encoder fe is a linear function parametrized by D x d weight matrix W and bias term b, i.e. fo(x) = Wx + b. Beyond computational benefits, and given that we are mostly interested in medium-sized output spaces where d E (8...., 512), we argue that, given a meaningful enough input space, a linear encoder could suffice in preserving neighborhoods of the input.
linear:
factorized linear: Exploiting the fact that batch normalization (BN) (loffe & Szegedy, 2015) is linear during inference,' we formulate fe as a multi-layer lincar model, where fo is a sequence of layers, each composed of a linear layer followed by a BN layer. This model introduces non-linear dynamics which can
1 AAtthough batch normalization is non-lincar during training because of the reliance the current batch statistics, during inference and using the means and variances accumulated over training, it reduces to a linear scaling applied to the features, that can be embedded the weights of adjacent linear layer.
4

---

Published in Transactions on Machine Learning Research (06/2022)
potentially help during training but the sequence of layers can still be replaced with single linear layer after training for efficiently encoding new features.
MLP: So can be a multi-layer perceptron with batch normalization (BN) (loffe & Szegedy, 2015) and rectified linear units (reLUs) as non-linearities, i.e. fo would be sequence of I Sinear-B--ree triplets, each with H' hidden units (i t 1. followed by linear projection.
Our main goal is to develop a scalable alternative to PCA for dimensionality reduction, we are mostly interested in linear and factorized linear encoders. It is worth already mentioning that, as we will show in our experimental validation, gains from introducing an MLP in the encoder are minimal and would not justify the added computational cost in practice.
The projector Ia- As also recently noted in Tian al. (2021), a crucial part of learning with non-contrastive pairs is the projector. This module is present in a number of contrastive self- self-supervise learning methods (Chen et al., 2020a; Grill et al., 2020; Zbontar et al., 2021; Tian et al., 2021). It is usually implemented as an MLP inserted between the transferable representations and the loss function. Unlike other methods, however, where the projector takes the representations to an even lower dimensional space for the contrastive loss to operate on (i.e. for SimCLR (Chen et al., 2020a) and BYOL (Grill et al., 2020), d' < d), for the Barlow Twins objective, operating in large output dimensions is crucial.
In Section 3, we study the impact of the dimension d' and experiment with a wide range of values. We empirically verify the findings of Zbontar et al. (2021) that calculating the de-correlation loss in higher dimensions (d' >> d) is highly beneficial. In this case, and shown in Figure 1, the transferable representation is now the bottleneck layer of this non-symmetrical hour-glass model. Although Eq. (1) is applied after the projector and only indirectly decorrelates the output representation components, having more dimensions to decorrelate leads to a representation that is more informative: the bottleneck effect created by the projector's output being in a much larger dimensionality implicitly enables the network to learn an encoder that also has more dccorrelatod outputs.
3 Experimental validation
In this section, we present a set of experiments validating the proposed TLDR on both visual and textual representations. In Section 3.1 we first present a summary of the tasks we evaluate on. Then in Sections 3.2 and 3.3 we present results for visual and textual applications, respectively. Finally, in Section 3.4 we study the different typer-parameter of our method, compare to several manifold learning methods, and present results when approximating the nearest neighbor pairs, as well as results after subsequent vector quantization.
3.1 Summary of tasks and evaluation protocol
A summary of all the tasks, datasets and representations that we consider in this section is presented in Table 1. We explore tasks like landmark image retrieval on ROxford and RParis, object class retrieval on ImageNet, and argument retrieval on ArguAna. For all experiments, we start from reliable feature vectors; it is a pre-requirement for the dimensionality reduction task. We assume that any structured data (images or documents) is first encoded with a suitable representation, and, without loss of generality, we assume the Euclidean distance to be meaningful, at least locally, in this input representation space.
In Section 3.2 we evaluate TLDR on popular image retrieval tasks like landmark image retrieval on datasets like ROxford, RParis (Radenović et al., 2018a), or k-NN retrieval on ImageNet (Russakovsky et al., 2015). We start from both specialized, relrievvl-oriennted representations like GeM-AP (Revaud et al., 2019), as well as more generic representations learned via self-supervised learning like DINO (Caron et al., 2021). For all experiments, we use the DINO and AP-GeM models as feature extractors to encode images for visual tasks; we use global image features from publicly avalable models for GeM-AP2 or DINO3, For the textual domain, we focus on the task of argument retrieval (Section 3.3). We use 768-dimenlional features from an
5

---

Published in Transactions on Machine Learning Research (06/2022)
Table 1: Tasks, models and summary for linear dimensionality reduction. 1 denotes the use of the dataset without labels. The GeM-AP and DINO models are from Revaud et al. (2019) and Caron al. (2021), respectively. Different values of d for TLDR and PCA are denoted in parenthesis next to the TLDR performance when + 128. input space D 384 for ViT-S/16; input space D = 768 for BERT.
Retrieval dataset (Metric)
Representation
Dim. red.
Result Summary
Task
TLDR (d = 128)
vs PCA,
vs Orig.
Model
Dataset
dataset
(d = 128)
(D = 2048)
GeM-AP (ResNet-50)
Landmarks
GLD-v2
0.49
4.1%
0.7%
ImageNet
0.22
2.9%
0.8%
DINO (ResNet-50)
ImageNet
ROxford (mAP)
GLD-v2
0.29
3.7%
6.8%
Laudhark Retrieval
ImageNet
0.25
2.6%
I 0.2%*
ImageNet
DINO (ViT-S/16)
GLD-v2
0.31
2.7%
t 5.4%
GL.D-v2
GL.
0.43 (256)
t 1.9%
t 4.6%*
RParis (mAP)
GeM-AP (ResNet-50)
Landmarks
GLD-v2
0.67
1.2%
0.6%
ImageNet Retrieval
ImageNet
DINO (ReaNet-50)
ImageNet
ImageNet
68.4 (512)
t 1.3%
0.9%
(K-NN Top-1 Acc.)
DINO (VIT-S/16)
ImageNet
ImageNet
74.8 (256)
t 0.6%
t 0.3%
Argument Retrieval
ArguAna 10e)0100)
ANCE (BERT)
MSMarco
WT200
94.5 (64)
t 0.8%
f 0.5%
off-the-shelf Bert-Siamese model called ANCE" (Xiong et al., 2021) trained for document retrieval, following the dataset definitions from Thakur et al. (2021).
We do not use any supervision for learning TLDR nor any of the methods we compare to. Given a trained dimensionality reduction encoder, we then encode all downstream task data for each test datasct, and evaluate them in "zero-shot" manner, using non-parametric classifiers (k-NN) for all retrieval tasks. Specifically, for landmark image retrieval on ROxford/RPaait (2018a), where specific qucries are defined. For every query, we mcasure the mcan average procision metric across all other images depicting the same landmark. For ImageNet Russakovsky et al. (2015), we follow the exact process used by Caron et al. (2021) and others: The gallery is composed of the full validation set, spanning 1000 classes, and each image from this validation (val) set is used turn as a query. Once again, we use k-NN to compute a list of images ranked by their relevance to the query, and we aggregate the labels from the top 20 images, assigning the predicted query label to the most prominent class. In Table 1 report. summary of the most notable results in each case. We chose d = 128 (resp. 64) for the visual (resp. textual) domains as the most commonly used setting in practice.
we use the common protocols presented by Radenović et al.
Implementation details. We do not explicitly normalize representations during learning TLDR; yet, we follow the common protocol and L22nnrmalize the features before retrieval for both tasks. Results reported for PCA use whitened PCA; we tested multiple whitening power values and kept the ones that performed best. Further implementation details are reported in the Appendix. It noteworthy that we used the exact same huper-parameters for the learning rate, weight decay, scaling, and À suggested in Zbontar et al. (2021), despite having very different tasks and encoder architectures. We were able to run PCA up to millions of data points and hundreds of dimensions using out-of-the-box tools. More precisely, we use the PCA implementation from scikit-learn. 5 For large matrices (> 500x500, which is our case) it uses the randomized SVD approach by Halko et al. (2011), which is an approximation of the full SVD. This has been the standard way of scaling PCA to large matrices.
Measuring variance. The proposed method has some inherent stochasticity, e.g. from SGD or from the neighbor pair sampling step (Step 2 in Algorithm 1). To make sure we properly measure variance, we each variant of TLDR shown in Figures 2 and 6 for five times and average the output results; the error bars report the standard deviation across those 5 runs. The reason we only report error bars for TLDR is
"nttpe://wwx sbert. http://wwww.beett..e//////////// http.ttte//bceepeeerineedddde models. models.htal hnttpe://sciktt learn. wtwprttteaaaaaaretetrataieeo hwaam..rr/ttablllennnaaeee/////e wwt.s/eeeddeeeeeeeeee.emm www.:eaarr..commmttitioooooo hCA.html

---

Published in Transactions on Machine Learning Research (06/2022)
Table 2: Compared Methods. For unsupervised methods, the objective is based on reconstruction. friighoor-uperrvise methods use nearest neighbors as pseudo-labels guide the learning. Denoising learns to ignore added Gaussian noise. Note that in Figure 6a, we compare to a few additional manifold learning methods (f.e. ICA, LLE, UMAP, LPP and LTSA) that could not scale to large output dimensions.
Method
(Self-) superviston
Encoder
Projector
Loss
Notes
PCA
Recoustruction MSE orthogouality
Used for dimensionnlity reduction in SoTA methods like DELF, GeM, GeM-AP and HOW
(Pesrson, 1901)
unsupervised
linear
linear
DeLim
neighhor-eupeervieed
MLP
None
Contrastive
(Hadbell et al,, 2006) (very low performanae)
Contrustive
weighbor-supees supervised
lincer
MLP
Contrastive
Hadsell et al. (2006) with projector
MSE
unsupervised
linear
MLP
Reconstruction MSE
TLDR with MSE loss
TLDR,
denolsing
linear
MLP
Barlow Twins
TLDR with nolse distortion
TLDR TLDR,,2
weighboorsuperrieeed neighbor-supperviee
lincur
MLP
Barlow Twins
fact. linear
MLP
Barlow Twins
TLDR, 2.2
neighboor-supervieed
MLP
MLP
Barlow Twins
because it has some noticeable variance; all other methods either correspond to deterministic algorithms or had negligible variance and therefore the error bars are not visible.
Further ablations, results on FashionMNIST and 2D visualizations. Additional results and inter- esting ablations can be found in the Appendix. In particular, explore the effect of the training set size, the batch size and report results on another NLP task: duplicate query retrieval. Morcover, and although beyond the scope of what TLDR is designed for, in Appendix D we present results on FashionMNIST when using TLDR on raw pixel data and for 2D visualization. We show that for cases where the input pixels are forming an informative space, TLDR can achieve top performance for d 2 8.
3.2 Results image retrieval
We first focus on landmark image retrieval. For large-scale experiments on this task, it is common practice to apply dimensionality reduction to global normalized image representations using PCA with whitening (Jégou & Chum, 2012; Tolias et al., 2016; Revaud et al., 2019; Tolias el al., 2020). For the experiments in this section, we simply replace the standard PCA step with our proposed TLDR. We conduct most comparisons and ablations using the optimizod, retrieval-oriented GeM-AP represcntations proposod by Revaud al. (2019) that are tailored to the landmark image retrieval task and present results in Section 3.2.1. We then report results using the generic DINO representations that are learned in self-supervised manner in Section 3.2.2.
3.2.1 Landmark image retrieval using GeM-AP representations
Experimental protocol. We start from 2048-dimen features obtained from the pre-trained ResNet-50 of (Revand et al., 2019), which uses Generalized- Meenniiiidd--nnnnn-aaaa pooling (Radenović et al., 2018b) and has been specifically trained for landmark retrieval using the AP loss (GeM-AP). To learn the dimensionality reduction function, we use GLD-v2, a dataset composed of 1.5 million landmark images (Weyand et al., 2020). We learn different output spaces whose dimensions range from 32 to 512. We evaluate these spaces on two standard image retrieval benchmarks (Radenovié et al., 2018a), the revisited Oxford and Paris datasets (ROxford and RParis). Each datasct comes with two test sets of inercasing difficulty, the "Medium" " and "Hard". Following these datasets' protocol, we apply the learned dimensionality reduction function encode both the gallery images and the set of query images whose 2048-dim features have been extracted beforehand with the model of Revand et al. (2019). We then evaluate landmark image retrieval on ROxford5K and RParis6K and report mean average precision (mAP), the standard metric reported for these datasets. For brevity, we report the "Mean" mAP metric, iee. the average of the mAP of the "Medium" and "Hard" test sets; we include the individual plots for "Medium" and "Hard" in the Appendix for completeness.
Compared approaches. We report results for several flavors of our approach. TLDR uses a linear projector, TLDR1 uses factorized linear one, and TLDR; an MT.P encoder with hidden layer. As an alternative, we
7

---

Published in Transactions on Machine Learning Research (06/2022)
ROxford5K
RParis6K
0.55
0.7
0.5
0.65
mAP
mAp
0.45
0.6
TLDR
TIDRR,
TLDR
TLDR;
0.4
TLDR; TLDRę
0.55
TLDRi
TLDRg
MSE
Contrastive
MSE
Contrastive
PCA.
X GeM-AP
PCA, X GeM-AP
0.35
0.5
32
64
128
256
512
2048
32
64
128
256
512
2048
Figure Landmark image retrieval experiments using GeM-AP representations. Mean average procision (mAP) on ROxford (left) and RParis (right) as function of the output dimensions d. We report TLDR with different encoders: linear (TLDR), factorized linear with hidden layer (TLDR1), and a MLP with 1 hidden layer (TLDR)), the projector remains the same (MLP with 2 hidden layers). We compare with PCA with whitening, two basclines based on TLDR, but which respectively train with reconstruction (MSE) and a contrastive (Contrastive) loss, and also with TLDRg, a variant of TLDR which uses Gaussian noise to synthesize pairs. The original GeM-AP performance is also reported.
also report TLDRg, which uses Gaussian noise to create synthetic neighbour pairs. All variants use an MLP with 2 hidden layers and 8192 dimensions as projector.
We compare with a number of un- and self-supervised methods (see also Table for a summary). First, and foremost, we compare to reducing the dimension with PCA with whitening, which is still standard practice for these datasets (Revaud al., 2019; Radenović et al., 2018b; Tolias et al., 2020). We also report results for our approach but trained with the Mean Square Error reconstruction loss instead of the Barlow Twins' (as we discuss in Scction 4, PCA be rewritten learning linear encoder and projector with reconstruction loss), and refer to this method as MSE. In this case, the projector's output is reduced to 2048 dimensions in order to match the input's dimensionality. Following a mumber of approaches that use nearest neighbors jsuperviioiii for contrastive learning (Hadsell et al., 2006), the Contrastive approach uses a contrastive loss on top of the projector's 's output. This draws inspiration from Hadsell ct al. (2006), and is a variant where we replace the Barlow Twins loss, with the loss from Hadsell al. (2006). is worth noting that we omit results from a more faithful reimplementation of Hadsell al. (2006), t.e. using a max-margin loss directly applied on the lower dimensional space and without projector, they were very low. Note that we considered other manifold learning methods, but none of the ones tested was able to neither scale, nor outperform PCA in output dimensions d 2 8; we present comparisons for smaller d in Section 3.4. Finally, we report retrieval results obtained on the initial features from Revaud et al. (2019) (GeM-AP), f.e. without dimensionality reduction. For all flavours of TLDR, we fix the number of nearest neighbors to A = 3, although, and as we show Figure C, TLDR performs well for a wide range of number of neighbors.
Results. Figure reports mean average precision (mAP) results for ROxford and RParis; as the output dimensions d varies. We report the average of the Medium and Hard protocols for brevity, while results per protocol are presented in Appendix B.1. We make number of observations. First and most importantly, we observe that both lincar flavors of our approach outperform PCA by significant margin. For instance, TLDR improves ROxford5K retrieval by almost 4 mAP points for 128 dimensions over the PCA baseline. The MLP flavor is very competitive for very small dimensions (up to 128) but degrades for larger ones. Even for the former, it not worth the extra-computational cost. An important observation that we are able retain the performance of the input representation (GeM-AP) while using only 1/16th of its dimensionality. Using a different. loss (MSE and Contrastive) instead of the Barlow Twins' in TLDD degrades the results.
8

---

Published in Transactions on Machine Learning Research (06/2022)
0.3
0.4
0.25
mAP
DINO
mAP
0.3
0.2
TLDR
0.15
TLDR (GLD-v2)
PCA (GLD-v2)
PCA.
TLDR (ImNet)
PCA. (ImNet)
DINO
0.1
32
64
128
256 384
32
256
Figure Self-supervised landmark retrieval performance on ROxford using DINO VIT-S/16 backbones. Left: mAP on ROxford as function of the output dimensions d using representations from DINO pretrained ImageNet; dimensionality reduction is learned
either ImageNet (dashed lines) GLD-v2 (solid lines). Right: Performance for TLDR and PCA when lcarning dimensionality reduction GLD-v2 over representations from DINO model pretrained on GLD-v2. No labels are used al any stage.
These approaches are comparable to or worse than PCA. Finally, replacing true neighbors by synthetic ones, as in TLDRg, performs worse.
3.2.2 Landmark image retrieval using DINO representations
In the previous section, we started from representations pretrained using supervision and tailored to the landmark retrieval task; we then Icarned dimensionality reduction on top of those an unsupervised way. It is however interesting to also study the fully unsupervised oase and measure how well the method performs if we use representations learned a self-supervised way. In this section, we assume that images are represented using DINO (Caron et al., 2021), state-of-the-art self-supervised approach that whose representations have lead to impressive results for wide variety of transfer learning tasks, including image retrieval.
Results. In Figure 3 we report results on ROxford when learning dimensionality reduction top of DINO features from a ViT-S/16 backbone. In all cases, we follow the evaluation protocol presented in the previous section. In Figure: (left) we present results starting from the publicly available ViT DINO model trained on ImageNet;; similar to the GeM-AP case, we treat the ViT as feature extractor and learn linear encoder on top using either the GLD-v2 or ImageNet datascts. are learned without any supervision. We that TLDR shows strong gains over PCA with whitening, the best performing competing method, and that gains are consistent across multiple values of output dimension d and across all setups. In fact, we see that if one assumes access to unlabeled data from the downstream landamark domain, one can achieve a large +5.4 mAP gain over DINO, by simply learning a linear encoder on top, and without the need to fine-tune the VIT model. Tt also noteworthy that TLDR is able to match the DINO VIT performance on ROxford using only 16-dimensions. Results for RParis are presented in the appendix.
all cases, both fcatures and dimensionality reduction
In Figure 3 (right) we start. from publicly available DINO model trained in unsupervised way on GLD-v2. Caron et al. (2021) evaluate their representations ROxford and RParis using global image features from this model; they report 37.9% mAP on ROxford for the average setting (0.52/0.24 for medium/ codium/hard hard splits). We see that TLDR can improve on that result and is able to achieve state-of ofttte-rtttettt performance on ROxford when using self-supervised learning, í.e. 0.43 mAP (0.57/0.28 for medium /hard splits) for d = 256, +4.6% mAP higher than using the original DINO features learned on GLD-v2.
3.2.3 Results on ImageNet
We further evaluate the performance of the proposed TLDR on ImageNet retrieval using k-NN. We follow the protocol from Caron al. (2021); Wu al. (2018) and run the corresponding evaluation scripts provided in the DINO codebase. We query with all images in the ImageNet val set, using the training set the database;
9

---

Published in Transactions on Machine Learning Research (06/2022)
0.5
VIT-S/16
Top-1 Accuracy
ResNet-50
70
0.4
mAP
65
TLDR
0.3
TLDR,
TLDR PQ
PCAp
PCA PQ
60
X DINO
PQ
0.2
128
384
512
2048
32
d
Bytes
(a) k-NN Accuracy on Image Net using unsupervised learning.
(b) ROxford mAP after quantization.
Figure 4: Retrieval results on ImageNet, and after vector quantization. Left: Top-1 Accuracy as function of the output dimensions for k-NN retrieval on ImageNet following the protocol from Caron et al. (2021) and using DINO ResNet-50 and ViT representations trained on ImageNet. Right: Performance ROxford after PQ quantization of the reduced features (d = 128), a function of the output vector size.
similar to DINO, we then report results using 20-NN. We report Top-1 Accuracy in Figure 4a for the DINO ResNet-50 and ViT-S/16 models. We see that TLDR is consistently better than PCA also this case, and for ResNet-50 gives +9.4/5.1/2.7% gains over PCA for d=32/64/128. For ViT, performance is higher and gains smaller, but still consistent. Generally, linear TLDR encoders (lincar and factorized lincar) improve DINO's retrieval performance on ImageNet k-NN for both backbones and for all output dimension over d 128; this is not the case for PCA, which is not able to improve DINO's performance. We further see that TLDR is able to outperform the original 2048-dimensional features with only 256 dimensions, i.e. achieving a 10x compression without loss in performance for ResNet-50, or reach approximately 75% Top- 1 Accuracy on ImageNet with 256-dimensional features for VIT-S/16 DINO.
3.3 Results on first stage document retrieval
For document retrieval, the process generally divided into two stages: the first one selects a small set of candidates while the second one re-ranks them. Because it works on a smaller set, this second stage can afford costly strategies, but the first stage has to scale. The typical way to do this is to reduce the dimension of the representations used in the first retvleval stage, often in supervised fashion (Khattab & Zaharia, 2020; Gao et al., 2021). Following our initial motivation, we investigate the use of unsupervised dimensionality reduction for document retrieval scenarios where a supervised approach is not possible.
Experimental protocol. We start from 768-dimensional features extracted from a model trained for Question Answering (QA), í.e. ANCE (Xiong et al., 2021). We use 2020; Wachsmuth et al., 2017) a conversational argument dataset composed of 380k documents to learn the dimensionality reduction function. Compared approaches. We report results for three flavors of our approach. TLDR uses a linear encoder while TLDR1 and TLDR, use factorized linear one with respectively one hidden layer and two hidden layers. We compare with PCA, which was the best performing competitor from Section 3.2. We also report retrieval results obtained with the 768-dimensional initial features.
(Bondarenko et al.,
Results. Figure 5 reports retrioval results on ArguAna, for different output dimensions d. We observe that the linear version of TLDR outperforms PCA for almost all values of d. The lintar-fctoorzeed ones outperforms PCA in all scenarios. We that the gain brought by TLDR over PCA increases as d decreases.
10

---

Published in Transactions on Machine Learning Research (06/2022)
100
100
90
90
Recall@100 100
Recall@100
80
80
TLDR
TLDR2
70
TLDR:
70
TLDDR2
TLDR2
TLDR2,6=10
60
PCA.
60
PCAw
X ANCE
ANCE
8 16
32
64
128
768
8 16
32
64
128
768
Figure 5: Argument retrieval results on ArguAna for different values of output dimensions On the left we vary the amount factorized layers, with fixed k =, on the right we fix the amount of factorized layers and test k = =3, 10, 100). Factorized linear fixed to 512 hidden dimensions.
Note that we achieve results equivalent to the initial ANCE representation using only 4% of the original dimensions; PCA, neods twice as many dimensions to achieve similar performance.
3.4 Analysis and Impact of hyper-parameters
Impact of hyper-parameters. We observed surprising robust.ness for TLDR across multiple hyper- parameters. For brevity, merely summarize the main findings here, while complete figures presented in the appendix. When vary the architecture of the projector Jø, important module of TLDR, we see that having hidden layers generally helps. As also noted in Zbontar et al. (2021), having a high auxiliary dimension for computing the loss is very important and highly impacts performance. When it comes to the numbers of neighbors k, we see that TLDR is surprisingly consistent across a wide range of We observe the same stability across several batch sizes. More details are presented in the appendix.
Comparisons to manifold learning methods smaller output dimensions. In Figure 6a present results for TLDR when the output dimensionality is d d 32; in this regime, few more manifold lcarning methods can be run, e.g. ICA, Locally Linear Embedding (LLE) (Roweis & Saul, 2000), Local Tangent Space Alignment (LTSA) (Zhang & Zha, 2004), Locality Preserving Projections (LPP) (He & Niyogi, 2003) and UMAP (MeInnes et al., 2018). Unfortunately, even at smaller output dimensions we had to subsample the dataset to run some of the methods, due to their scalability issues. Specifically, we are forced to use ouly 5% of the training set (~75K images) for learning L.E. and LTSA, 10% for I.P (~150K this setting gives the best results) and 50% (~750K images) for UMAP. We see that TLDR generally dominates for d 2 8, and that linear methods tend to be the highest performing ones as the output dimensionality grows.
How sensitive are TEDR-reduced vectors to subsequent quantization? In Figure 4b we verify that the gains from TLDR persist after compressing the dimensionality reduced vectors to only few bytes, e.g. via Product Quantization (PQ) (Jegou et al., 2010). We conduct the study on ROxford, starting from the GeM-AP ResNet-50 representations. First of all, we see that is highly desirable first use dimensionality reduction and then PQ-compress the vectors; starting from the 2048-dimensional vectors results to much lower performance. Secondly, see that TLDR once again highly outperforms PCA, with +8.16% and +4.28% in mAP for compression to 64 or 128 bits, respectively.
How sensitive is TLDR to approximate nearest ncighbors? To verify that our system is robust to an approximate computation of nearest neighbors, we test. its performance using product quantization (Jegou et al., 2010; Ge et al., 2013) while varying the quantization budget (i.e. the amount bytes used for each image during the nearest neighbor search). Compression done using optimized product quantization (OPQ) (Ge et al., 2013) via the FAISS library (Johnson al., 2017) and results are reported in Figure 6b. We present results ROxford, starting from the GeM-AP ResNet-50 representations. We that TLDR
11

---

Published in Transactions on Machine Learning Research (06/2022)
TLDR
0.4
PCA=
0.5
ICAs
mAP
LLE
mAP
0.2
LTSA
0.48
UMAP
0.1
0.46
TEDR
TLDRtp
16
96.9
98.4
99.2
99.6
Compression rate (%)
(a) Comparisons to manifold learning methods.
(b) Effect of approximate nearest neighbors.
Figure 6: Left: Comparisons to manifold learning methods for small output dimensions d 128; report Mean average precision (mAP) on ROxford using GeM-AP representations
function of the output dimensions d. Right: The effect of nearest neighbor approximation for d = 128. We plot mAP as function of the compression rate used during nearest neighbor computation. Note that the bascline (compression rate = 0) is using the 2048-dimensional (8192 bytes) GeM-AP representations during nearest neighbor compulation.
is quite robust to quantization during the nearest neighbor search and that even when the quantization is pretly strong (1/64 the default size or merely 16 bytes per vector) TLDR still retains its high performance.
4 Related work
The basic idea behind TLDR is embarrassingly simple and links to a large number of related methods, from PCA to manifold learning and neighborhood embedding. In this section we discuss a few such relations; more are discussed in Appendix E.
Linear dimensionality reduction. We refer the reader to Cunningham & Ghahramani (2015) for an extensive review of linear dimensionality reduction. It is beyond the scope of this paper to exhaustively discuss many such related works, we will therefore focus on PCA (Pearson, 1901) which is the de facto standard linear dimensionality method, in particular for large-scale retrieval.
One can derive the learning objective of PCA (Pearson, 1901) by setting fo(x) = WT and g(x) Wa in the model of Figure i.e. linear encoder and projector with W E RDxd and optimize W via minimizing the Frobernius norm of the matrix of reconstruction errors over the whole training set, subject to orthogonality constraints:
W" = arg min || s(fo(z))lle, s.t. WTW = Ia-
W
This equation has a closed form solution that can be obtained via the eigendecomposition of the data covariance matrix and then keeping the largest d eigenvectors".
Unlike PCA, TLDR does not constrain the projector to be a linear model, nor the loss to be reconstruction loss. In fact, the redundancy reduction term in the Barlow Twins loss encourages the whitening of the batch representations as soft constraint (Zbontar et al., 2021), in way analogous to the orthogonality constraint of (2). We see from Figure C that part of the performance gains of TLDR over PCA is precisely due to this asymmetry in the architecture, i.e. when the projector is an MLP with hidden layers. Looking at MSE results Figures 2, f.e. a version of TLDR with a reconstruction loss, we also see that the Barlow Twins loss and the flexibility of computing it an arbitrarily high d' d'-dimensional space further contributes to this gain. One can therefore interpret TLDR as more generic way of optimizing a linear encoder, i.e. using an arbitrary decoder and approrimating the constraint of Eq-(2) in soft way, further incorporating weak notion of whitening.
We refer the reader Chapter of the Deep Learning book (Goodfellow al., 2016) for the derivation of the PCA solution.
12

---

Published in Transactions on Machine Learning Research (06/2022)
Another way of learning a linear encoder is presented in He & Niyogi (2003) where Locality Preserving Projections are given by the optimal linear approximations to the eigenfunctions of the Taplace-Beltrami Beltramii operator on the data manifold. We find the TLDR optimization to not only be more scalable, but also more versatile as it is stochastically and independently optimizing for different neighborhoods of the manifold.
Manifold learning and neighborhood embedding methods. Manifold learning methods define ob- jective functions that try to preserve the local structure of the input manifold, usually expressed via a k--N graph. Non-linear unsupervised dimensionalliy--ecuttoo methods usually require the K-NN graph of the imput data, while most further require eigenvalue decompositions (Roweis & Saul, 2000; Donoho & Grimes, 2003; Zhang & Zha, 2004) and shortest-path (Tenenbaum et al., 2000) or computation of the graph Laplacian (Belkin & Niyogi, 2003). Others involve more complex optimization (McInnes et al., 2018; Agrawal et al., 2021). Moreover, many manifold learning methods were created to solely operate on the data they were learned on. Although "out-of-sample" extensions for many of such methods have been proposed (Bengio et al., 2004), methods like Spectral Embeddings, pyMDE (Agrawal al., 2021) or the very popular t-SNE (Van der Maaten & Hinton, 2008) can only be used for the data they were trained on. Finally, UMAP (McInnes et al., 2018) was recently proposed as not only competitor of t-SNE on 2-dimensional outputs, but as a general purpose dimension reduction technique. Yet, all our experiments with UMAP, even after exhaustive hyperparameter tuning, resulted in very low performance for d 2 8 for all the tasks we evaluated.
Nearest neighbors as "supervision" for contrastive learning. The seminal method DrLIM (Hadsell et al., 2006) uses a contrastive loss over neighborhood pairs for representation learning. Experimenting only on simple datasets like MNIST, it learns a CNN backbone and the dimensionality reduction function in single stage, using a max-margin loss. TLDR resembles DrLIM (Hadsell et al., 2006) with respect to the encoder input and the way pairs are constructed; a crucial difference, however, is the loss function and the space in which it is computed: DrLIM uses a contrastive loss which is computed directly on the lower dimensional space. Despite our best effort to make this approach work as described, performance was very low without a projector. Using the contrastive loss from Hadsell et al. (2006), together with the projector we use for TLDR, we were able lo get more meaningful results (reported as Contrastive in our experiments), although still underperforming the Barlow Twins loss. This difference may be due two reasons: first, and as discussed above, Barlow Twins encourages the whitening of the representations which makes it more suitable for this task. Second, and like many other pair-wise losses, the contrastive loss further requires sampling hard/ htt/meaannggfll negatives (Wu et al., 2017: Radenović et al., 2018b), Very recently, Tsoen et al. (2022) use features space neighbor consistency as a regularization term for learning under label noise by encouraging the prediction of each example to be similar to its nearest neighbours'.
Relation to Deep Canonical Correlation Analysis. The MLP variant of TLDR closely related to the Deep Canonical Correlation Analysis (DCCA) introduced by Andrew et al. (2013). The Barlow Twins loss can be seen as more modern, simpler way optimize DCCA. First, the addition of the projector is shown to signif- icantly and positively impact the results. Second, the gradient of DCCA involves computing the gradient of the trace of the squarc-root of a matrix, step which is not required for computing the Barlow Twins loss. Morcover, Deep CCA obtains optimal results with full-batch optimization and L-BFGS, í.e. a far less scalable setting.
Temporal neighborhoods. Component analysis methods that take into account the temporal structure of the data are also related, something critical for time scries. Dynamical Components Analysis (DCA) (Bai et al., 2020), for example, uses lemporal neighborhoods, to discover a subspace with maximal predictive information, defined the mutual information between the past and future. Similarly, Deep Autoencoding Predictive Components (DAPC) (Clark et al., 2019) builds on the same notion of predictive mutual information from temporal neighbors, but further adopts a masked reconstruction task to enforce the latent representations to be more informative.
Graph diffusion for harder pairs. Iscen et al. (2018b) improve the method from Hadsell et al. (2006) by mining harder positives and negative pairs for the contrastive loss via diffusion over the k-NN graph. Similar to Hadsell et al. (2006), they learn (fine-tune) the whole network and not just the dimensiionality--ecccioon layer. Although it would be interesting to incorporate such ideas in TLDR, we consider it complementary and beyond the scope of this paper. Methods for learning descriptor matching are also related; e.g. (Simonyan et al., 2014) formulates dimensionality reduction as a convex optimisation problem. Although the redundancy
13

---

Published in Transactions on Machine Learning Research (06/2022)
reduction objective can be formulated in many ways, e.g. via stochastic proximal gradient methods like Regularised Dual Averaging in Simonyan et al. (2014), we believe that the simplicily, immediacy and clarity in which the Barlow Twins objective optimizes the output space is a strong advantage of TLDR.
5 Discussion
The Barlow Twins and related SSL losses. We chose the Barlow Twins loss from Zbontar et al. (2021) as we believe it fits dimensionality reduction exceptionally. The second term of the loss does not only provide Barlow Twins with an elegant way of avoiding collapse, something not trivial for other related losses like SimSiam (Chen & He, 2020; Zhang ct al., 2022), but decorrelating the output space is a property that highly suits the dimensionality reduction task. Moreover, from our experiments and ablations, we see that the Barlow Twins loss is highly robust to most pyper-parameters. We could however devise a variant of TLDR that uses any other "symmetrical" loss, e.g. SimCLR (Chen et al., 2020a), SimSiam (Chen & He, 2020) or the recently proposed VICReg (Bardes et al., 2022) that builds on Barlow Twins. Yet, leveraging SSL losses that use self-distillation and the Exponential Moving Average trick such as BYOT. (Grill et al., 2020) or MoCo (He et al., 2020) with neighbor pairs might be far from trivial.
TLDR beyond dimensionality reduction. As we discuss in the previous paragraph, we do not consider TLDR to be method for learning large visual representation models from scratch, where neighbors would sim- ply replace image transformations. We however believe that this signal can be useful as another way of perform- ing model distillation, by e.g. sampling neighbors from a larger model or model trained on more data. It is also clear from the results of TLDR. learning on top of the generic IaageNNet-trainee DINO and the GLD-v2 dataset in Figure 3 (left), that our method can be used for unsupervisod domain adaptation. TLDR is also agnostic to the way neighbors are extracted, and other similarity measures could be used. The choice of this measure would then be a crrper-prrameer potentially taking prior information into account. TLDR could then learn an (out- put) embedding space for which the Euclidean distance is suitable, highly desirable property for many tasks,
A scalable manifold learning method. By "scalability" we refer to the ability of the proposed TLDR to be efficiently applied to arbitrary large datasets and output spaces. We believe this to be an important property, given that most manifold learning methods are unable to "scale" to large datasets or output dimensions. This is why, in Figure 6a for example, we had to subsample the training dataset so some of the competing methods could fit in memory and/or or converge within a couple of days (see also Appendix B.6 for training time comparisons). TLDR uses mini-batch Stochastic Gradient Descent and has lincar time and memory complexity with respect to both the training database size and the output dimension. In that regard, this is a highly scalable manifold learning method.
Computational cost and training time. i.e. learning the dimensionality reduction encoder, is generally orders of magnitude less expensive than than e.g. learning visual representation. We always start from pre-extracted feature vectors and learn a small (linear or MLP) encoder and an MLP projector. There is no large backbone learn and even datasets like GLD-v2 (Weyand et al., 2020) (compased of 1.5M vectors in 2048 dimensions for the GeM-AP model) can easily fit. in memory. When it comes to training time, TLDR is noticeably slower than PCA. For example, for d= 128 and for the GLD-v2 dataset, learning PCA takes approximately 18 minutes (multi-core), while learning a linear TLDR over 100 epochs takes approximately 63 minutes (on a single GPU). The latter includes 13 minutes for computing (exact) k--Ns for the dataset.
In terms of computation, it is worth noting that training TLDR,
Although slower at training time, linear TLDR highly outperforms PCA, while all other manifold lcarning methods that we compare to can neither reach the performance of PCA nor be applied to large training datasets for d d 32. More importantly, as we show in Appendix B.4, the performance of PCA saturates early and, unlike TLDR, does not benefit from adding more data; Our study on a dataset of 1.5 million images suggests that the gap with PCA increases as the training set gets larger. Finally, although the training time of TLDR is higher than PCA, both methods share the same linear encoding complexity during testing, something highly important as this is a process repeated at every single retrieval request.
TLDR out of its comfort zone. Although TLDR can be seen as a way of generalizing recent self-supervisod visual representation learning methods to cases where handcrafted transformations of the data are challenging
14

---

Published in Transactions on Machine Learning Research (06/2022)
impossible to define, we want to emphasize that it is not suited for self-supervised representation learning from pixels; augmentation invariance is a much more suited prior in that regard, while it is also practically impossible to define meaningful neighboring pairs from the input pixel space. Additionally, although visualization is a common manifold learning application, TLDR is neither designed not recommended for 2D outputs; there are other methods like I-SNE, UMAP or MDE that specialize for such lasks (see also Appendix).
What is TLDR suitable for? TLDR excels for dimensionality reduction to mid-size outputs, e.g. when d is between 32 and 256 dimensions, range to which the vast majority of manifold learning methods cannot scale. This is very useful in practice for retrieval. TLDR further provides a computationally efficient way of adapting pre-trained representations, coming e.g. from large pre-trained models, to a new domain or a new task, without the need for any labeled data from the downstream task, and without fine-tuning large encoders.
6 Conclusions
In this paper we introduce TLDR, a dimensiionaliyy-rceetiii method that combines neighborhood embodding learning with the simplicity and effectiveness of recent self-supervised learning losses.
By simply replacing PCA with TLDR, one significantly increase the performance of generic (DINO) specialized (GeM-AP) models on retricval tasks like ImageNet or ROxford, especially if one has access to unlabeled data from the reamstream retrieval domain. Those gains are achieved by learning a linear encoder on top of pre-extracted features and without the need to fine-tune a large backbone. To summarize, TLDR offers a number of desirable properties:
i) Scalabilily: learned via stochastic gradient descent, TLDR can easily be parallelized across GPUs and machines, while for even the largest datasets, approximate nearest neighbor methods be used to create input pairs in sub-linear complexity (Babenko & Lempitsky, 2014; Kalantidis & Avrithis, 2014), ii) Simplicity: The Barlow Twins (Zbontar et al., 2021) objective is robust and easy to optimize, and does not have trivial solutions, iii) Out-of-sample generalization, and iv) Linear encoding complerity: TLDR is highly effective with a linear encoder, offering a direct replacement of PCA without extra encoding cost.
Acknowledgements
The authors want to sincerely thank Christopher Dance for his carly comments that helped shape this work. We also want thank our colleagues at NAVER LABS Europe" for providing feedback that made this work significantly better. Specifically Rafael, Gregory, Riccardo and Florent for reviewing our work and providing us with constructive comments. Last but not least, we want to thank Audi for hand-drawing our gorgeous teaser Figure. Finally, we want to thank all anonymous reviewers, whose comments also made this work better. This work was supported in part by MIAlGrreooble Alpes (ANR-19-P3IAA-0003),
would like to thank Gabriela, Jérome, Boris, Martin, Stéphane, Bulent,
References
Akshay Agrawal, Alnur Ali, and Stephen Boyd. Minimum-distortie embedding. Foundations and Trends in Machine Learning, 11(3):21,-378 2021.
Ehsan Amid and Manfred K Warmuth. Trimap: Largo-scale dimensionality reduction using triplets. arXio preprint 1r110 00200,, 2019.
Galen Andrew, Raman Arora, Jeff Bilmes, and Karen Livescu. Deep canonical correlation analysis. In International Conference on Machine Learning, 2013.
Artem Babenko and Victor Lempitsky. The inverted multi-index. IEEE Transactions on Pattern Analysis and Machine Intelligence, 124671227- 2014.
15

---

Published in Transactions on Machine Learning Research (06/2022)
Junwen Bai, Weiran Wang, Yingbo Zhou, and Caiming Xiong. Representation learning for sequence data with deep autoencoding predictive components. In International Conference on Learning Represenlations, 2020.
Adrien Bardes, Jean Ponce, and Yann LeCun. VICreg: Variaane-ivariannc--vvaaaaca regularization for sclf-supervised learning. In International Conference on Learning Representations, 2022.
Mikhail Belkin and Partha Niyogi. Laplacian eigenmaps for dimensionality reduction and data representation. Neural computation, 15(6))1373-136,, 2003.
Yoshua Bengio, Jean-Frangois Paiemmnt, Pascal Vincent, Olivier Delallcau, Nicolas Le Roux, and Maric Ouimet. Out-of-sample extensions for lle, isomap, mds, eigenmaps, and spectral clustering. Conference on Neural Information Processing Systems, 16:177-184, 184, 2004.
Alexander Bondarenko, Maik Fröbe, Meriem Beloucif, Lukas Gienapp, Yamen Ajjour, Alexander Panchenko, Chris Biemann, Benno Stein, Henning Wachsmuth, Martin Potthast, et al. Overview of touché 2020: Argument retrieval. In Proc. CLEF, pp. 384 395. Springer, 2020.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In Conference on Neural Information Processing Systems, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In International Conference on Computer Vision, 2021.
Ting Chen, Simon Korublith, Mohammad Norouzi, and Geoffrey Hinton. simple framework for contrastive learning of visual representations. In International Conference on Machine Learning, pp. 1597-1607, 2020a.
Xinlci Chen and Kaiming Hc. Exploring simple siamese representation learning. urXiv:2011.105600 2020.
arXiv preprint
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved basclines with momentum contrastive learning. arXiv preprint arXi::2003.4420 2020b.
David Clark, Jesse Livezey, and Kristofer Bouchard. Unsupervised discovery of temporal structure noisy data with dynamical components analysis. In Conference on Neural Information Processing Systems, 2019.
John P Cumningham and Zoubin Ghahramani. Linear dimensionality reduction: Survey, insights, and generalizations. Journal of Machine Learning Research, 16(1):2859-22 2200009000 2015.
David L Donoho and Carrie Grimes. Hessian eigenmaps: Locally linear embedding techniques for high- dimensional data. Proceedings of the National Academy of Sciences of the United States of America, 100 (10):5591-5596, 2003.
Aleksandr Ermolov, Aliaksandr Siarohin, Enver Sangineto, and Nicu Sebe. Whitening for self-supervised representation learning. In International Conference Machine Learning. PMLR, 2021.
Zhiyuan Fang, Jianfeng Wang, Lijuan Wang, Lei Zhang, Yezhou Yang, and Zicheng Liu. SEED: Self-supervised distillation for visual representation. In International Conference on Learning Representations, 2021.
Luyu Gao, Zhuyun Dai, and Jamie Callan. Coil: Revisit exact lexical match in information retrieval with contextualized inverted list. In North American Chapter of the Association for Computational Linguistics, 2021.
Tiezheng Ge, Kaiming He, Qifa Ke, and Jian Sun. Optimized product quantization. IEEE Transactions Pottern Analysis and Machine Intelligence, 36(4):744-755, 2013.
Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1. MIT press Cambridge, 2016.
16

---

Published in Transactions on Machine Learning Research (06/2022)
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Cheshlaghi Azar, et al. Bootstrap your own latent: A new approach to self-supervised learning. In Conference on Neural Information Processing Systems, 2020.
Aditya Grover and Jure Leskovec. node2vec: Scalable feature learning for networks In ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 855-864, 2016.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning invariant mapping. In IEEE Conference Compuler Vision and Pallern Recognilion, volume 2, 2006.
Nathan Halko, Per-Gunnar Martinsson, and Joel A Tropp. Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions. SLAM reciew, (3)))2117-288, 2011.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 9729 9739, 2020.
Xiaofei He and Partha Niyogi. Locality preserving projections. In Conference on Neural Information Processing Systems, 2003.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXie preprint arXiv:1503.0253 2015.
Doris Hoogeveen, Karin Verspoor, and Timothy Baldwin. Cqadupstack: Gold or silver. In International ACM SIGIR Conference on Research and Development in Information Retrieval, volume 16, 2016.
Sergey Toffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, 2015.
Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, Teddy Furon, and Ondrej Chum. Efficient diffusion region manifolds: Recovering small objects with compact emn representations. In IEEE Conference Compuler Vision and Pottern Recognition, pp. 2077-2086 2017.
Ahmet Iscen, Yannis Avrithis, Giorgos Tolias, Teddy Furon, and Ondrej Chum. Fast spectral ranking for similarity search. In IEEE Conference Computer Vision and Pattern Recognition, pp. 7632-7641. 2018a.
Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, and Ondřej Chum. Mining on manifolds: Metric learning without labels. In IEFE Conference on Computer Vision and Pattern Recognition, pp. 7642-7651, 2018b.
Ahmet Iscen, Jack Valmadre, Anurag Arnab, and Cordelia Schmid. Learning with neighbor consistency for noisy labels. In IEEE Conference on Computer Vision and Pattern Recognition, 2022.
Hervé Jégou and Ondrej Chum. Negative evidences and co-occurrences in image retrieval: the benefit of PCA and whitening. In European Conference on Computer Vision, pp. 774 787. Springer, 2012.
Herve Jegou, Matthijs Douze, and Cordelia Schmid. Product quantization for nearest neighbor search. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(1):117-128, 2010.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with gpus. arXiv preprint 1rXiw:1702.087 2017.
Yannis Kalantidis and Yannis Avrithis. Locally optimized product quantization for approximate nearest. neighbor search. In IEEE Conference Computer Vision and Pattern Recognition, pp. 2321 2328, 2014.
Omar Khattab and Matei Zaharia. Colbert: Efficient and effective passage search via contextualized late interaction over bert. In International ACM SIGIR Conference on Research and Development in Information Retrieval, 2020.
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations, 2017.
17

---

Published in Transactions on Machine Learning Research (06/2022)
Chinh Lin, Jheng-Hong Yang, and Jimmy Lin. Distilling dense representations for ranking using tightly-coupled teachers. arXie preprint or 0iv:20000. 11886, 2020.
Chundi Liu, Guangwei Yu, Maksims Volkovs, Cheng Chang, Himanshu Rai, Junwei Ma, and Satya Krishna Gorti. Guided similarity separation for image retrieval. In Conference on Neural Information Processing Systems, 2019.
Leland Mclnnes, John Healy, and James Melville. UMAP: Uniform manifold approximation and projection for dimension reduction. arXiv preprint 2XX::1802.03480 2018.
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. Ms marco: A human generated machine reading comprehension dataset. In CoCo@NeurIPS, 2016.
Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In IEEE Conference on Computer Vision and Pallern Recognition, pp. 3967 3976, 2019.
Karl Pearson. LIII. On lines and planes of closest fit to systems of points in space. The London, Edinburgh, and Dublin Philosophicol Magazine and Journal of Science, 2(11):559-557 1901.
Bryan Perozzi, Rami Al-Rfou, and Steven Skicna. Deepwalk: Online learning of social representations. In ACM SIGKDD International Conference on Knowledge Discovery and Dala Mining, pp. 701-710, 2014.
Filip Radenović, Ahmet Iscen, Giorgos Tolias, Yannis Avrithis and Ondřej Chum. Revisiting Oxford and Paris: Large-scale image retrieval benchmarking. In IEEE Conference on Compuler Vision and Pullern Recognition, pp. 5706 07166-57,,777 2018a.
Filip Radenović, Giorgos Tolias, and Ondřej Chum. Fine-tuning CNN image retrieval with no human annotation. JEEE Transactions on Pattern Analysis and Machine Intelligence, 41(7):1655-1688 2018b.
J. Revaud, J. Almazan, R.S. Rezende, and C.R. de Souza. Learning with average precision: Training image retricval with listwise loss. In International Conference Computer Vision, 2019.
Sam T Roweis and Lawrence K Saul. Nonlinear dimensionality reduction by locally linear embedding. Science, 290(5500): :2323 2326, 2000.
Olga Russakovaky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3)) 2015.
Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Learning local feature descriptors using convex optimisation. IEEE Transactions on Pallern Analysis and Machine Intelligence, 36(3):1573-155 1585, 2014.
Jian Tang, Meng Qu, Mingzhe Wang, Ming Zhang, Jun Yan, and Qiaozhu Mei. Line: Large-scale information network embedding. Tn Proceedings of the 24th International Conference on World Wide Web, pp. 1067-1077, 2015.
Joshua B Tenenbaum, Vin De Silva, and John C Langford. A global geometric framework for nonlinear dimensionality reduction. Science, 290((500)) 2319-2333, 2000.
Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Tryna Gurevych. BEIR: A heterogencons benchmark for zero-shot evaluation of information retricval models. In Thirty-ffthh Conference on Neural Information Processing Systems Dalasels and Benchmarks Track, 2021.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In International Conference on Learning Representations, 2020.
Yuandong Tian, Xinlei Chen, and Surya Ganguli. Understanding self-supervised learning dynamics without contrastive pairs. In International Conference on Machine Learning, 2021.
Giorgos Tolias, Ronan Sicre, and Hervé Jégou. Particular object retrieval with integral max-pooling of CNN activations. In International Conference on Learning Representations, 2016.
18

---

Published in Transactions on Machine Learning Research (06/2022)
Giorgos Tolias, Tomas Jenicek, and Ondřej Chum. Learning and aggregating deep local descriptors for instance-level recognition. In European Conference Computer Vision, pp. 460-477. Springer, 2020.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of Machine Learning Research, 9(11), 2008
Henning Wachsmuth, Martin Potthast, Khalid Al Khatib, Yamen Ajjour, Jana Puschmann, Jiani Qu, Jonas Dorsch, Viorel Morari, Janek Bevendorff, and Benno Stein. Building an argument search engine for the web. In Proceedings of the 4th Workshop on Argument Mining, pp. 49-59, 2017.
Henning Wachsmuth, Shahbaz Syed, and Benno Stein. Retrieval of the best counterargument without prior topic knowledge. In Proceedings of the Araual Meeting of the Association for Compulational Linguislies, pp. 941-251, 2018.
Philippe Weinzaepfel, Thomas Lucas, Diane Larlus, and Yannis Kalantidis. Learning super-features for image retrieval. In International Conference on Learning Representations, 2022.
T. Weyand, A. Araujo, B. Cao, and J. Sim. Google Landmarks Dataset v2 A Large-Scale Benchmark for Instanco-Level Recognition and Retrieval. In IEEE Conference Computer Vision and Pattern Recognition, 2020.
C-Y Wu, Manmatha R., Smola A.J., and Krahenbuhl P. Sampling matters in deep embedding learning. International Conference on Computer Vision, 2017.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In IEEE Conference on Computer Vision and Pottern Recognition, 2018.
Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwijk. Approximate nearest neighbor negative contrastive learning for dense text retrieval. In International Conference on Learning Representations, 2021.
Yuning You, Tianlong Chen, Zhangyang Wang, and Yang Shen. When does self-supervision help graph convolutional networks? In International Conference Machine Learning, 2020.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In International Conference on Machine Learning, 2021.
Chaoning Zhang, Kang Zhang, Chenshuang Zhang, Trung X Pham, Chang D Yoo, and In So Kweon. How does simsiam avoid collapse without negative samples? a unified understanding with self-supervised contrastive learning. In International Conference on Learning Representations, 2022.
Zhenyue Zhang and Hongyuan Zha. Principal manifolds and nonlinear dimensionality reduction via tangent. space alignment. SIAM journal on scientific computing, 26(1):313-33,, 2004.
19

---

Published in Transactions on Machine Learning Research (06/2022)
Appendix
A
Appendix Summary
In this appendix we present a number of additional details, results and Figures that we could not fit in the main text:
We report additional experiments for the landmark retrieval task in Appendix B. Specifically, we present results for the med/hart splits separately for GeM-AP (Appendix B.1), experiment with oracle neighbors (Appendix B.2), experiment with features from larger ResNet-101 backbone (Appendix B.7). We also present some additional results with DINO representations (Appendix B.8).
We present ablations when vary the projector architecture and the number of neighbors k (Appendix B.3), as well as when varying the training set size (Appendix B.4) and the batch size (Appendix B.5). We also compare the training times of TLDR and other manifold learning methods as we vary the size the training set (Appendix B.6)
We report additional experiments for the document retrieval in Appendix C. Specifically, we extend our evaluation protocol and report result on a new task: duplicate query retrieval. We further investigate not only dimensionality reduction for the same task, but also the case of dimensionality reduction transfer.
Although TLDR is not suited for such applications, as a proof of concept we present results on the FaSSionMNIST dataset in Appendix D, L.e when learning from raw pixel data. We also present results when using TLDR for visualization, f.e. when the output dimension is d = in Figure N.
We extend Section with further discussion related topics and more related works in Appendix E. We conclude with brief discussion on limitations of TLDR in Appendix F.1
Finally, in Appendix F we give pytorch-style pseudocode for initializing and training TLDR.
Additional experiments: Landmark image retrieval
B.1 Results on Medium and Hard protocols separately
In Figure A we report the mAP metric for the Medium and Hard splits of the Revisited Oxford and Paris datasots (Radenović et al., 2018a) separately.
B.2 Results with "Oracle" nearest neighbors
In Figure B we present results using an oracle version of TLDR, i.e. version that uses labels only keep pairs neighbors that come from the same landmark in the training set. As see, TLDR practically matches the oracle's performance.
B.3 Varying the projector architectures and the number of neighbors
Figure C studies the role of some of our parameters. On the left size of Figure C, we vary the architecture of the projector IG, an important module TLDR. We see that having hidden layers generally helps. As also noted in Zbontar et al. (2021), having high auxiliary dimension d' for computing the loss very important and highly impacts performance. On the right side of Figure show the surprisingly consistent performance of TLDR across wide range of numbers of neighbors k.
20

---

Published in Transactions on Machine Learning Research (06/2022)
ROxford5K-Medium
ROxford5K-Hal
0.65
0.6
0.35
mAp
mAP
0.3
0.55
TLDR,
TLDR TLDR; TLDRg
TLDR,
TLDR;
TLDB&
MSE
Contrastive
0.25
MSE Contrastive
0.5
PCA.
X
PCA
GeM-AP
32
64
128
256
512
2048
32
64
128
256
512
2048
d
RRaris6K-Medium
RParis6K-Hard
0.8
0.6
0.75
0.35
mAP
mAP
0.5
TLDR
TLDR1
TLDR
TLDR,
TLDR;
TLDHę
0.45
TLDR;
TLDRt
0.65
MSE
Contrastive
MSE
Contrastive
PCA
X GeM-AP
0.4
PCA
GeM-AP
32
64
128
256
512
2048
32
64
128
256
512
2048
d
Figure A: Image retrieval experiments. Mean average precision (mAP) on ROxford (top) and RParis (bottom), for the Medium (left) and Hard (right) test sets, as a function of the output dimensions We report TLDR with different encoders: linear (TLDR), factorized linear with 1 hidden layer (TLDR1), and MLP with hidden layer (TLDR;), the projector remains the same (MLP with 2 hidden layers). We compare with two baselines based TLDR, but which respectively train with reconstruction (MSF) and contrastive (Contrastive) loss. Our main baselines PCA with whitening, and the original 2048-dimentional features (GeM-AP Revaud et al. (2019)), i.e. before projection.
ROxford5K-Meddinn
ROxford5
0.65
0.4
0.6
0.35
mAP
TLDR
mAP
TLDR
Oracle
Oracle
0.55
ICA.
ICAs
PCA.
PCA.
GeM-AP
GeM-AP
128
512
32
256
2048
Figure Neighbor-supervised with oracle. Mean average precision (mAP) on ROxford5K (Radenović et al., 2018a) for the Medium (left) and Hard (right) test sets, as function of the output dimensions d. We compare TLDR with oracle version that labels to select, training pairs. We include as baselines both PCA and ICA with whitening, and the original 2048-dimension features (GeM-AP [32]), i.e. before projection.
21

---

Published in Transactions on Machine Learning Research (06/2022)
Varying the projector layers
Varying the number of neighbors
0.7
0.65
0.6
mAP
mAP
0.6
Rarii
0.5
ROxford
layidden-layers
L-hidden-layer
0.4
laydee
0.5
64
128
256
512
1024
2048 4096 8192 16384
d'
k
Figure C: Impact of TLDR hyper-parameters with linear encoder and d 128. Dashed (solid) lines are for RParis6K-Mean (ROxfordōK-Meann. (Left) Impact of the aurtharr dimension d' and the number of hidden layers in the projector. (Right) Impact of the number of neighbors k. We see how the algorithm is robust to the number of neighbors used.
ROxtord5K-Medium
ROxford5K-Hard
0.62
0.35
0.6
mAP
mAP
0.58
0.3
0.56
TLDR
TLDR
0.54
PCA
PCA
10
20
40
80
160
320
640
1280
10
20
40
80
160
320
640
1280
Number of images (in thousands)
Number of images (in thousands)
SharisK--Medimg
RParis6K -ard
0.77
0.56
0.76
mAP
mAP
0.75
0.54
0.74
TLDR
TLDR
0.62
0.73
PCA
PCA
10
20
40
80
160
320
640
1280
10
20
40
80
160
320
640
1280
Number of images (in thousands)
Number of images (in thousands)
Figure D: TLDR benefits from larger training sets. Impact of the size of the training set on performance. TLDR uses linear encoder and d 128.
B.4 Varying the size of the training set
In Figure D we show the impact of the size of the training set on TLDR's performance by randomly selecting subsets of images of increasing size from the Google Landmarks training set (Weyand et al., 2020). As we see, PCA outperforms TLDR for a reduced number of images, however, it does not benefit from adding more data, keeping the same performance across all training set sizes. In contrast, TLDR does benefit from adding more data; all plots suggest that larger training set could potentially boost the performance even further, increasing the gap with respect to PCA.
22

---

Published in Transactions on Machine Learning Research (06/2022)
ROxtord5K-MMciuu
ROxford5K-Hard
0.39
0.64
0.38
0.63
mA
0.37
0.62
0.36
0.61
TLDR
0.35
TLDR
0.6
0.34
128
256
512
1024
2048
128
256
512
1024
2048
Batch size
Batch size
Figure E: The surprising stability of TLDR across batch sizes. Impact of the size of the training mini-batch on performance. TLDR uses a linear encoder and d 128.
B.5 Batch size ablation
Finally, in Figure E, show results of TLDR varying the size of the training mini-batch. Surprisingly, we observe it is stable across wide range of values, allowing training TLDR under limited memory resources.
B.6 Training time comparisons when varyin the size of the training set
In Figure F' we report training time when learning linear dimensionality reduction encoder while varying the size of the training set. We randomly select subsets of images of increasing size from the Google Landmarks training set (Weyand et al., 2020) (1.5M images) and measure the training time and ROxford performance for TLDR and for a few manifold learning methods. All methods are run on the same servers. We used 16 CPUs and 300GB memory for all manifold learning methods, and one 32GB V100 GPU for TLDR. There is publicly available easy-to-use GPU implementation for these methods, so instead use the multi-core versions of scikit-learn.
The tables show that methods like LLE and LTSA scale exponentially with respect to both the time and memory needed. When sampling 5% of the data, i.e. 75k images, TLDR (and LPP) require 5 minutes to train while LLE and LTSA require over hours. The LLE and LTSA runs took more than (7) days when subsampling 8% (10%) of the data, while subsampling 15% and above led to oul-of-memory (OOM) related crashes despite 300GB of memory available. The LPP method is scaling better, but a) is 5x slower than TLDR when subsampling 10% of the dataset and b) leads to OOM when subsampling 15%. Obviously, reported times highly depend on their implementation. Here, we used the popular scikit-learn multi-core implementations for LLE and LTSA and the best implementation of LPP we could find
B.7 ResNet-101 101 features
We also experimented with features obtained from a larger pre-trained ResNet-101 model from Revaud et al. (2019)9, We see Figure that TLDR retains a signíficant gain over PCA and fact surpasses the highest. state-of-the-art numbers based on global features reported in Tolias al. (2020) for ROxford.
B.8 Additional results with DINO features
In Figure Ha present results ROxford using ResNet-50 DINO pretrained ImageNet; dimensionality reduction is learned on either ImageNet or GLD-v2. In Figure Hb we report performance on RParis using a VIT-S/16 DINO pretrained on ImageNet.No labels are used at any stage.
htttp::/ggthuu..cm//////////////
httpp:://gthhb..cc/aaave///gggg wttpc///aeeeeeeep.maaaeerrreaaaa
23

---

Published in Transactions on Machine Learning Research (06/2022)
300
TLDR(1 GPU)
PCA CPU)
LLE (16 CPU)
240
LTSA (16 CPU)
LPP (16 CPU)
mimtes
180
120
1.5k (1%)
3K
4.5K
7.5K
(5%)
12K
15K (10%)
22.5K
30K
45K (30%)
features (dataset percentage)
Figure F: Training time as training dataset size increases for d = 32, TLDR uses a linear encoder, a 2-hidden layer projector with d' 2048 and trained for 100 epochs.
ROxford5K-Medium
ROxford5K-Har
0.65
0.4
mAP
0.6
mAP
0.35
0.55
0.3
TLDR
PCA
0.25
PCAg
0.5
GeM-AP
GeM-AP
0.2
128
512
2048
32
256
512
2048
Figure G: ResNet-100 features. Mean average precision (mAP) on ROxford5K (Radenović el. al., 2018a) for different values of output dimensions d, using features obtained from the pre-trained ResNet-101 of Revaud et al. (2019).
24

---

Published in Transactions on Machine Learning Research (06/2022)
0.5
0.25
DINO
DINO
mAP
mAP
0.4
0.15
TLDR (GLD-v2)
PCA. (GLD-v2)
0.3
TLDR (GLD-v2)
PCA. (GLD-22
0.1
TLDR (ImNet)
PCA. (ImNet)
TLDR (ImNet)
PCA. (ImNet)
16
32
256
2048
16
32
64
256 384
d
d
(a) Results ROxford using DINO ResNet-50.
(b) Results on Rarri using DINO VIT-S/16.
Figure H: Self-supervised landmark retrieval performance using DINO backbones. Both plots report mAP on ROxford as a function of the output dimensions d. Left: Performance on ROxford using a ResNet-50 DINO pretrained on ImageNet; dimensionality reduction is learned on cither ImageNet or GLD-v2. Right: Performance on RParis using a ViT-S/16 DINO pretrained on ImageNet. No labels are used any stage.
Table A: Summary of tasks and datasets for the first stage document retrieval experiments presented in the Appendix.
Dataset
# Documents
# Queries
Avg positives per query Avg query length
Avg document length
Retrieval type
Question answering (pretraining only)
MSMarco
8.8M
6980
1.1
56
Asymmetric
Argument retrieval
ArguANA
8674
1406
167
Asymmetric Asymmetric
Wehhis Touché
380k
242
Duplicate question retrieval
Quora
523k
5000
1.6
10
11
Symmetric Symmetric
CQADupStuck
457k
13145
1.4
9
129
C Additional experiments: Document retrieval
In Section of the main paper we studied first stage document retrieval under the (ask of argument retrieval, where both dimensionality reduction and evaluation performed datasets designed for the same task. In this section, we extend this evaluation protocol introducing new task: duplicate query retrieval and now investigate not only dimensionality reduction for the same task, but also the case of dimensionality reduction transfer.
In the following paragraphs we first introduce the five datasets we use for first stage document retrieval, and then we discuss the additional experiments involving duplicate query datasets.
C.1 Tasks and dataset
A summary of dataset statistics is available in Table A and examples for each datasct are available in Table B.
MSMarco passages (Nguyen et al., 2016): question and answer dataset based on Bing queries. Very sparse anotation with a high number false negatives. Queries (Q) and Documents (D) are from different different domains due to size and content. Retrieval asymmetric, because if you input a D as Q, the answer will not be D. Used only for pretraining as it has of training pairs for contrastive learning, while our aim is to perform self-supervision only. For this goal, we have chosen the other four datasets, that do not have
25

---

Published in Transactions on Machine Learning Research (06/2022)
Table B: Examples of queries and documents from all the document retrieval datasets we use. Table extracted from Thakur al. (2021); Note the difference of length between query and document in some datasets.
Dataset
Query
Drrevann-Deeument
MSMARCO
what fruit native to anstralia
Perogruphh> Passifiona herhertions. rare passion fruit notive to Australia. Fruits are green-skiaed, white fleshed, with an unknown edible rating. Some sourees list the fruit edible, sweet and fasty, while others list the fruits being bister and inedàble. assiflora herbertiana. A rare passion fruit native to Austrolia...
ArguAna
Sexist arlvertieing subjective would he too difficull to codify. Elfective advertising appents to the ausisl, cultural, and personal values consurmers. Throughh the connection of values to products, survioes and ideas, advertising able to accomplish gool adopsion...
<Tiik, merlia modern culture television gender house would han sesist advertising <PPorggnpph>> Although there is claim that mexist advertisnng is to diflicull to cudify, such eodes have and being developed slandaeds speak to advertising which demetuns the atalus of woreen, ubjectiless them, and plays upon stereotypes ahout women which harm Eaclier the Council of Eurupe was mecatione, Demmark, Narway and Ausiralis as specifie examples corles standards for evalnating serist advortising which boen developed.
guide the advertising industry. These
and society gynemal.
Touche-2021
Should the government allow illegal immigrants to become citiseus!?
<Titic>> America should support blankett ammesty illegal immigrants.. <PPregraph>> Uadecumented warkens do not receive full Social Security benelits because they are not United States citisens nor should they until they sook cisizenship legally. Iesaal imunggrants aze legally obligated to pay taxes...
CQ Cuppuktt
Command to display first few and lines of file
< <Pitle> Combing head and tail single call vin pipe <Parograhhh On a regular basis, I am piping the outgut of some program to either 'heed' or "tail: Now, suppose that want see the first AND last 10 lines of piped ourpar, such that could something like /lotsoooutput heuitail...
Quora
How long does take to metinmpbetaniine out of your blood?
<Parograph>> How long does it take the body to get rid of meshammpartaaaiiee
& readily available set of training pairs (or triplets) for training, and thus self-supervision or unsupervised learning is required.
ArguAna (Wachsmuth et al., 2018): Counter-argument retrieval dataset. Queries and documents belong to the same domain, with some queries being part of the corpus, which makes it not suitable for training on this dataset. Queries and documents come from the same domain in both size and content, however associated query-document pairs have inverse context (Q defends a point, D is a rebuttal of Q), so input Q should not retricve Q if it is on the database. Retricval is asymmotric as a query should not retricve itsolf.
Webis-Toucht 2020 (Bondarenko et al., 2020; Wachsmuth et al., 2017): Argument retrieval dataset. Queries and documents are from different domains due to size and content, with queries being questions and documents being support arguments for the question. Retrieval is asymmetric as query should not retrieve itself.
CQADupStack (Hoogeveen et al., 2016): Duplicate question retrieval from StackExchange subforums, composed of 12 different subforums. Corpuses are concatenated during training and mean result over all corpuses is used for lesting (i.e. every corpus has equal weight even if the number of queries is different). Queries are titles of recent submissions, while documents are concatenation of titles and descriptions of existing ones. Queries and documents are from different domains due to size and content, with the query domain being part of the document one (Queries are contained in the documents). Retrieval is symmetric as query should return itself.
Quora: Duplicate question retrieval from the Quora platform. Queries are titles of recent submissions, while documents are titles of existing ones. Queries and documents are from the same domain concerning size and content. Retrieval is symmetric as a query should return itself.
C.2 Experimental results
We now test different combinations of the previously introduced datasets as dimensionality reduction and test datasets. The setup for all experiments is the same as the experiments in the main paper. In order to summarize our results (9 combinations of train/test datasets), we consider d = 64 (second highest we investigate) as the comparison mark between PCA and TLDR. If TLDR outperforms PCA for all d < 64, we consider that it performed better than PCA, and otherwise we consider that PCA performed better than TLDR. Note that in all cases the lower the dimension the better TLDR performed against PCA. We also
26

---

Published in Transactions on Machine Learning Research (06/2022)
Table C: Summary of the results on document retrieval. (L=0) and (L>0) indicate which version of TLDR had better perfomance (linear and factorized linear respectively). Note that arguana is suitable for training (not represented) and that we are not interested in using the same dataset for dimensionality reduction and test (thus the empty cells).
Test dataact
Argunent relzieval
Duplicate query
ArguAna
Webis- Touché
Quora
CQADupStack
Argument Retrieval
Webis: Touché
TLDR (L>0)
(L>0)
PCA (L=0))
Dimeusionality reduction
Quora
TLDR (L>0)
TLDR (L=0)
PCA (L=0)
Duplicate Question
CQADupStack
TLDR (L>0)
TLDR (1-0)
PCA (L>0)
100
X
90
40
Recall@ 100
NDCG@10
80
TLDR
TLDR
70
TLDR:
TLDR1
TLDR2
20
TLDRa
PCA
PCA
X ANCE
X ANCE
8
16
32
64
128
768
16
768
d
Figure Argument retrieval results on the ArguAna dataset using Webis-Touchs 2020 for dimensionality reduction for different values output dimensions d. On the left we present Recall@100 and the right we present NDCG@10.
report which version of TLDR performed better, > means that factorized linear is better than linear and L = 0 the opposite. We present summary the experimental results in Table C, and provide depictions of some experiments Figures I through L. From the results presented on the table, we derive two conclusions about TLDR:
Differences in retrieval from pretraining to dimensionality reduction impacts results Looking into evaluation on argument retrieval, TLDR outperforms PCA. On the other hand, looking into evaluations duplicate query retrieval, PCA is always able to outperform TLDR for d 2 64. We infer that this must be derived from the difference in retrieval condition, asymmetric retrieval TLDR is able to outperform PCA. Note that symmetric retrieval and same domain for document and queries differs from the original pretraining task, and we posit that PCA more robust to this type of change (which does not happen in our image retrieval experiments). Although we have this initial suspicion validated with datasets proper conclusion would need more tataset-pairss for experimentation, which leave for future work.
in all tests with
2. Choosing linear or factorized linear depends on the statistics of the dataset: Analyzing the results we are able to detect that the choice of which version of TLDR one should use depends on the length queries and documents of the original dataset. If both lengths are equal, factorized linear better (ArguANA and Quora), if not then linear the better choice (Webis-Touchh 2020 and CQADupStack). Even if by using ANCE representations we should not need deal with these differences (we only tackle embeddings of fixed size), the statistics of the resulting embedding is different enough that it is delected by the batch normalization layer that is added for factorized linear.
27

---

Published in Transactions on Machine Learning Research (06/2022)
30
&ecall@ 100
NDCG@10
20
20
15
TLDR
TLDR
10
10
TLDR2
TLDR2
PCA
5
PCA
X ANCE
ANCE
8
16
32
64
128
768
8
16
32
64
128
768
Figure J: Argument retrieval results on the Webis-Touché 2020 dataset using Quora for dimen- sionality reduction for different values of output dimensions On the left we present Recall@100 and on the right we present NDCG@10.
& 100
NDCG@10
40
20
TLDRo
TLDRo
TLDR2
10
TLDR2
PCA
PCA
X baseline
X baseline
0
8
16
32
64
128
768
8
16
32
64
128
768
d
d
Figure K: Duplicate question retrieval results the CQADupstack dataset using Quora for dimensionality reduction for different values of output dimensions d. On the left we present Recall@100 and on the right we present NDCG@10.
100
x
80
1ecall@1 00
90
NDCG@10
80
TLDR
60
TLDR
TLDR1
TLDR
TLDR2
TLDR2
40
60
PCA
PCA
X ANCE
X ANCE
20
8
16
32
64
128
768
8
16
32
64
128
768
d
Figure L: Duplicate question retrieval results on the Quora dataset using Webis-Toucht 2020 for dimensionality reduction for different values output dimensions d. On the left we present 10call@100 and on the right we present NDCG@10.
28

---

Published in Transactions on Machine Learning Research (06/2022)
TLDR
PCA
PCAm
TCA
Accuracy
UMAP
Isomap
32
64
128
256
512
784
Figure M: Results on the FashionMNIST dataset as a function of the output dimensions We compare TLDR with PCA, PCA with whitening, UMAP and Isomap and report accuracy after k'-NN classifiers (with k' 100) following (MeInnes et al., 2018). For TLDR and UMAP we set the number of neighbors 100. The performance of UMAP was very low for d' > 32.
D FashionMNIST: Learning from raw pixel data and visualization
Although beyond the scope of what TLDR is designed for (see also discussion at the end of Section 4), in this section we present some basic results when using it for learning from raw pixel data and for visualizations, e.e. when reducing the output dimension to only d' 2.
Learning from raw pixel data. In Figure M we present results when learning directly from raw pixel data. We use the predefined splits and, following related work (MeInnes et al., 2018), we measure and report accuracy after k-NN classifiers. We that TLDR retains its gains over any other manifold learning method lested. We have note however that these results have be taken with a pinch of salt, as a) the input pixel space is relatively simple compared to higher resolution natural images and b) to achieve such results we use the prior knowledge that we only have 10 classes and set high values for arper-paramcter k, i.e. k = 100 for all methods compared.
2D visualizations. Let us first clarify that TLDR was not created with 2D outputs mind; in fact, there are other excellent choices for visualization like t-SNE, UMAP (McInnes et al., 2018), TriMAP (Amid & Warmuth, 2019) or the recent Minimun-Distorien instead. In Figure N we show 2d visualizations when reducing the 60k training set of MNSH to d = dimensions. We present results for TLDR, t-SNE (Van der Maaten & Hinton, 2008), UMAP (Melnnes et al., 2018) and PyMDE (Agrawal et al., 2021). It is interesting how TLDR seems to be optimized for linear separability even for 2-dimensional outputs. For visualizations, we used the MDM library provided by the authors of (Agrawal et al., 2021).
Embedding (MDE) (Agrawal el. al., 2021) that we would
E Further discussions and related works
Graph diffusion for query expansion. For the task of retrieval, assuming access to the search (test) database, methods like (Iscen et al., 2017; 2018a; Liu et al., 2019) utilize manifold learning on the the k-NN graph of the database to facilitate query expansion. We note that while these methods have shown great empirical performance on the same image retrieval datasets we experiment on, we do not directly compare to them their methodology and goals greatly differs from ours. We aim at being ireariant to the torget dataset (thus not performing learning on them), differently from the aforementioned methods they need
10https://ymdde.rr/
29

---

Published in Transactions on Machine Learning Research (06/2022)
-00
1-SNE (Van der Maaten & Hinton, 2008)
MDE (Agrawal al., 2021)
3-
-10
UMAP (Melnnes et al., 2018) (k 100)
TLDR (ours) (k = 100)
Figure N: 2D visualizations of the training set of FashionMNIIST. From top to bottom and left to right: LSNE, MDE, UMAP and TLDR
access to the target dataset for learning, and to its k-NN graph during testing. TLDR is complementary to such graph diffusion techniques for query expansion.
Relation to knowledge distillation. Knowledge distillation (KD) (Hinton et al., 2015) aims at transferring knowledge from a pre-trained teacher network to student one, often for neural network compression. One way to perform KD is relational KD (RKD) (Park et al., 2019; Tian et al., 2020; Lin et al., 2020), which transfers knowledge using relations between samples such distance and angles. TLDR can be seen method for RKD. It enforces the student network (encoder) to reproducc a relational property (ncighborhood) found on the leacher (the input space). However, there are some main differences to traditional distillation methods: i) the application: self-super retrieval instead of supervised classification (Hinton et al., 2015), contrastive (Tian et al., 2020; Lin et al., 2020) or self-supervision for classification (Fang et al., 2021), ii) the definition of the relations: abstract (neighbors), instead of measurable ones (distance, angle), which avoids normalization problems due to the dimensionality difference between teacher and student, and iii) the link
30

---

Published in Transactions on Machine Learning Research (06/2022)
Algorithm 2: PyTorch-style pseudocode of TLDR.
X: training aet of D-dimemsional vectors input dimension
d':
projector's output dimension
d: encoder's 's output dimension
B: batch size
k: number of neighbours
graph: calculates the nearest neighboure each vector RandonBatchSampier: randonly sanples batch_size indices
Initialization
wwweiallze_nooteeo. nodel initialize encoder projector 8 returns matrix of size Mak
coapute_xnn.grapica x)
Training
for in nun aun_epochs:
for indices in KandonBatchhamploord (1en (x), batch_aize=B): BxD
[[indices]
X[randon, sarandon_sanples (M (Xindices), 1)] BxD
model(x) # Exd
y model(y)
loss BarionTviielossss, y)
bass.backaardl
nodel.update
addeardsscard www.ddidddpjjeecceeeeee projector() # discard projector
nodel Hxd
between teacher and student: in our case, the student becomes part of the teacher network at the end, instead of being a separate network.
Relation to node embedding. Node embedding methods aim at generating representations to graph nodes that are representative of the sample and its relations on the graph. that sense, TLDR could be seen as learning embeddings for nodes on graph. Compared to the traditional methods this space, such as LINE (Tang et al., 2015), Node2Vec (Grover & Leskovec, 2016), DeepWalk (Perozzi al., 2014), TLDR has three clear differences: does not rely the edge strength; ii) regularization of the space based the decorrelation of dimensions instead of L2-norm or orthogonality; and iii) only the 1-hop neighborhood information is used. More recent node embedding solutions are based on deep learning architectures that incorporate diffusion properties in the architecture like GCNs (Kipf & Welling, 2017; You et al., 2020), while TLDR achieves a similar effect via the Barlow Twins loss.
E.1 Limitations of our work
In the context of document retrieval we also tested TLDR on another task: duplicate question retricval. In duplicate question retrieval TLDR was only able to outperform PCA for the lower dimension values (1=8,16.32). We posit that TLDR does not achieve significant gains for the rest of the dimensions because duplicate task differs too much from the original protraining task (QA on MSMarco dataset) in that the duplicate retrieval is symmetric (the documents retrieved by query should also appear when we use the document query), while pretraining and argument. retrieval is assymetric. Tn order to verify this, performed ablations with different pairs of (dimensionality target dataset is a duplicate retrieval task TLDR is not able to outperform the compared method, but if we use duplicate retrieval only for dimensionality reduction and test on argument retrieval TLDR is able to outperform the compared methods. For full discussion and results ef. Section C.
ielluction,targee dataset) and confirm that if the
F Pseudocode of TLDR
In Algorithm 2 we show the pscudocode of TLDR, which includes initialization, training, and projection.
31


Reviews:
[{"review_id": "sSihYvuoM8", "reviewer": "Reviewer_hxnp", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The TLDR framework is intuitive and easy to understand.\n- This paper forges a link between self-supervised learning and manifold learning. I could see this being a fruitful connection, by applying methods in one area to the other.\n- The method and experiments are clearly presented.\n- Limitations of the method and where it is expected to do well are honestly discussed.\n- Experimental results of the method are fairly strong, particularly those that show an improvement of the low-dimensional TLDR representations against the learned representations of the original method.\n- Breadth of experiments across different tasks and datasets.\n- Ablations show which hyperparameters are important and which are not.\n\nWeaknesses:\n\n- Relative to other manifold learning methods, the lack of operations such as eigendecomposition and complicated optimization is pointed to as a benefit of TLDR. There is not much evidence to back up this claim, except for statements in §3.3 about the extent to which the dataset had to be subsampled for other manifold learning methods.\n- The motivation for specifically linear encoders could be better. If a simple MLP were to perform significantly better than a linear encoder, I doubt the extra complexity of the MLP would be a significant barrier to adoption.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a method for dimensionality reduction called TLDR based on the Barlow Twins technique for self-supervised learning (Zbontar et al., 2021). Contemporary self-supervised learning approaches for vision typically aim to learn representations that are consistent when the input image is transformed by e.g. cropping, flips, etc. TLDR uses a similar idea but replaces image transformations with nearest neighbors as computed by Euclidean distance in the input space. Thus representations should be consistent not when image transformations are applied (as in self-supervised learning) but rather when an input is replaced by one of its neighbors. \nExperiments demonstrate that TLDR produces useful low-dimensional representations for image and document retrieval.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "- [Not critical] Clearly demonstrate benefit of TLDR vs. baselines in terms of scalability. For the limiting resource (whether it be time, memory, etc.) how do baselines such as LLE, LTSA, etc. compare to TLDR as a function of dataset subsampling percentage and/or dimensionality?\n- [Not critical] Algorithm 1 is too high level and would benefit by taking a step closer to pseudocode (for example, it is not entirely how to create positive pairs by sampling).\n- [Not critical] Add a discussion about how the general idea of modifying self-supervised learning by replacing image transformations with neighbors on a graph could be more generally applicable. In particular, how important is the Barlow Twins method to this work? Could a similar dimensionality reduction technique be created by applying the general idea on top of SwAV, SimCLR, SimSiam, etc.? Also, what other applications involving a neighbor graph (beyond dimensionality reduction) could benefit from a similar approach?", "broader_impact_concerns": "I do not believe there to be significant ethical implications of this work. The primary value of this work is to speed up predictions for downstream tasks by preserving meaningful information in a low-dimensional representation. This method in general does not appear to enable predictions beyond what could have already been done with the original data given enough computational resources.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "jpmzTnXhDa", "reviewer": "Reviewer_j6W2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\nThe evidence supporting the claim that TLDR representations are more useful than PCA representations is strong. Figures in the main text cover a diverse set of tasks. It is consistent across figures that a flavor of TLDR outperforms PCA. Figures in the appendix show that TLDR is insensitive to its own algorithmic hyperparameters, making the method easy to use.\n\nThe paper also cites relevant benchmarks and adequately discusses whether alternative approaches could/should be applied to the tasks at hand (Section 4). \n\n# Weaknesses\nThere are some minor weaknesses which should not take long to address.\n\nThe foremost weakness is as follows. The paper makes the claim that TLDR is a scalable method (Section 1 says “This leads to a highly scalable method”). The evidence for this claim is not strong. For instance, the paper does not include discussions of runtime, especially in relation to the runtime of the main baseline, PCA. It is expected that TLDR takes more time than PCA; it would be good to know how much longer. \n\nThe other noticeable weakness is that the paper does not explain clearly how exactly PCA is done on the dimensionality reduction training set. Based on the xticks of Figure D from the appendix, sometimes the dimensionality reduction training set size can be as large as 1 million. As far as I know, standard singular value decomposition techniques for PCA cannot process such a large dataset in one pass. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a dimensionality reduction technique (twin learning for dimensionality reduction, or TLDR). The quality of the new representations is mainly measured by performance in retrieval tasks. The main bulk of experiments demonstrate that TLDR yields better representations than principal component analysis (PCA).", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "The following requested changes are necessary for securing my recommendation for acceptance.\n\n* Compare the runtime of TLDR with that of PCA.\n* Explain how PCA is done on the dimensionality reduction training set (since the dataset sizes are really big).\n* Explain how the error bars (like those in Figure 2) are generated.\n* Explain why some methods have error bars while some do not (in Figure D, and other figures, PCA does not have error bars while TLDR does). \n", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "VTwnNnEEFl", "reviewer": "Reviewer_FuBN", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n1. The paper is well-organized and the method itself is not hard to understand.\n2. Exhaustive experiments have been performed to validate the performance. The results look sound.\n3. Ablation studies on projector architecture, number of neighbors, and batch size are comprehensive.\n\nCons:\n1. Authors should further explain why they use euclidean distance. Let's say we shift an image to the right and bottom a lot. It is essentially the same image, but the euclidean distance would increase a lot. Though euclidean distance can be changed to other distances, if we have to choose different distances for different tasks, then the distance selection would become another hyper-parameter.\n2. Essentially the method is still looking for a certain invariance. The key is the distance defined in the input space. TLDR can be viewed as a variant of BYOL where the loss function is changed to Barlow Twins loss and the augmentation is replaced with k-NN. Further comparison and discussion with BYOL should be presented. Also, compared with the original Barlow Twins paper, this paper's novelty is incremental.\n3. The exact tasks should be better explained. Do you first pretrain the TLDR model without supervision and then perform supervised finetuning? How do you evaluate the information retrieval performance? Compared to the general supervised tasks, is the o\n4. What is the setup for contrastive loss and how do you make sure the comparison is fair? Should you also include BYOL for comparison?\n5. If you have to compute the correlation matrix for each pair (x, y), wouldn't the time cost be huge?\n6. Barlow Twins seems to be very closely related to CCA. Some related papers are worth discussing, DCCA[1], DCA[2], DAPC[3].\n\n[1] Andrew, Galen, et al. \"Deep canonical correlation analysis.\" International conference on machine learning. PMLR, 2013.\n[2] Clark, David, et al. \"Unsupervised discovery of temporal structure in noisy data with dynamical components analysis.\" Advances in Neural Information Processing Systems 32 (2019).\n[3] Bai, Junwen, et al. \"Representation Learning for Sequence Data with Deep Autoencoding Predictive Components.\" International Conference on Learning Representations. 2020.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes an unsupervised dimensionality reduction method called TLDR. The method first defines a distance function in the input space. Then k-NN is used to find k nearest inputs for any sample x. These k samples serve as the positive pairs for x. For each pair (x, y), both of them are first mapped to a low-dimensional latent space and then project back to a high-dimensional space. In this high-dimensional space, the authors compute a Barlow Twins loss as the objective function. Intuitively, the loss function forces the projected features to be highly correlated. This process requires no supervision. Once the training finishes, the encoder can be used for dimensionality reduction. Authors apply the method to image and document retrieval tasks. The results are better than existing methods.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "1. Better explanation/connections with existing works like Barlow Twins, CCA, BYOL, etc. \n2. How the encoder is used should be more clear. It's better to use some math notations to explain.\n3. It looks like you only attempted euclidean distance, could you also try some other distances?\n4. Is the euclidean distance complementary to the common augmentation techniques like distortion or shifting?", "broader_impact_concerns": "I don't have concerns on the ethical implications of this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Thanks for your submission to TMLR.

The reviewers are all in agreement that the paper is ready for publication and the concerns have been sufficiently addressed by the authors.  I'm happy to recommend acceptance as-is for this paper.   I appreciate both the thoroughness of the responses/edits, as well as the efficiency in responding.  
Decision: Accept (poster)