Self-supervised Representation Learning from Random Data Projectors
Authors: Yi Sui
Tongzi Wu
Jesse C. Cresswell
Ga Wu
George Stein
Xiao Shi Huang
Xiaochen Zhang
Maksims Volkovs
Venue: ICLR 2024 poster (2024)
Keywords: Representation learning
Self-supervised learning
random data projections
domain-agnostic representation learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We propose a new domain-agnostic self-supervised learning framework using random data projections

Abstract: Self-supervised representation learning (SSRL) has advanced considerably by exploiting the transformation invariance assumption under artificially designed data augmentations. While augmentation-based SSRL algorithms push the boundaries of performance in computer vision and natural language processing, they are often not directly applicable to other data modalities, and can conflict with application-specific data augmentation constraints. This paper presents an SSRL approach that can be applied to any data modality and network architecture because it does not rely on augmentations or masking. Specifically, we show that high-quality data representations can be learned by reconstructing random data projections. We evaluate the proposed approach on a wide range of representation learning tasks that span diverse modalities and real-world applications. We show that it outperforms multiple state-of-the-art SSRL baselines. 
Due to its wide applicability and strong empirical results, we argue that learning from randomness is a fruitful research direction worthy of attention and further study.

Published as a conference paper at ICLR 2024
SELF-SUPERVISED REPRESENTATION LEARNING FROM RANDOM DATA PROJECTORS
Yi Sui Layer Al amy@layer6.
Tongzi Wu Layer Al (angzi@layer6.cc
Jesse C. Cresswell Layer AI jessel ayer6.
Ga Wu Dalhousie University ga wu@dal.
George Stein
Xiao Shi Huang Layer 6 AI
Xiaochen Zhang Layer AI lisa ais@@layer6.a1
Maksims Volkovs Layer AI maks Layer6......
Layer 6 AI
george@layer6.at
garyelayer6.at
ABSTRACT
Self-supervised representation learning (SSRL) has advanced considerably by exploiting the transformation invariance assumption under artificially designed data augmentations. While augmentation-bins based SSRL algorithms push the boundaries of performance in computer vision and natural language processing, they are often not directly applicable to other data modalities, and can conflict with application- specific data augmentation constraints. This paper presents SSRL approach that can be applied to any data modality and network architecture because it does masking. Specifically,
not rely on augmentations data representations can be learned by reconstructing random data projections. We evaluate the proposed approach tasks that span diverse modalities and real-world applications. We show that it outperforms multiple state-of-the-art SSRL baselines. Due its wide applicability and strong empirical results, we argue that learning from randomness is a fruitful research direction worthy of attention and further study.
show that high-quality
wide range of representation learning
INTRODUCTION
Learning data representations a self-supervised manner is commonly associated with well-designed pretext tasks that can create heuristics for machine learning models identify and encode useful information from unlabelled data. While pretext tasks are often highly customized to specific applications, they are typically based on a handful of underlying assumptions. Pretext tasks utilizing the transformation invariance assumption across data augmentation views show leading performance in multiple research domains. For computer vision, image representations are commonly guided to remain identical after cropping, rotating, flipping, or corrupting, among others (Chen et al., 2020 Grill et al., 2020). Similarly, in natural language processing, sentences with similar words and semantic meaning are expected to have the same representation (Wei & Zou [019 Wu et al. 2019). In these and other domains transformation invariance is encouraged explicitly through contrastive or momentum objectives that aim to bring together representations before and after transformation.
Despite their strong performance in many domains, self-supervised representation learning (SSRL) algorithms that enforce transformation invariance are limited in that they do not support generic data types. Many well-known data augmentation methods are tailored to specific modalities, and are restricted in their generality across different domains. For instance, Gaussian noise corruption cannot be applied directly to textual data, while image rotation is not applicable in the tabular domain. Thus, augmentation based SSRL techniques are inherently constrained in their cross-domain applicability.
While these examples show modality limitations, a more subtle problem is that even standard augmentations can conflict with ipplication-speciife constraints. For example, pathology images of stained tissue samples have low color variation (Shen et al. 2022 Kang et al. 2003),, so naïve color- jittering produces unnatural augmentations (see Fig. DD. Such incorrect images may be inappropriate and unsafe for use in critical medical imaging applications (Elgendi et al.) (021). For time series with high periodicity, such as online monitoring data, random shifting augmentations can create

---

Published as a conference paper at ICLR 2024
identical augmented views that are not useful for pretext tasks (Eldele et al. 2021 Zhang & Ma 2022). In tabular settings where relatively few options for augmentations are available, random noise addition or random swapping of features between training examples can easily produce unrealistic examples. As a concrete example from particle physics, consider energies and mo- menta of interacting particles collected experi- mentally (Brehmer & Cranmer 2020). Since the incoming and outgoing energies are con- strained by the laws of physics, only certain combinations are possible to observe. Tabular augmentations would produce unphysical com- binations, harming the consistency of learned representations and leading unpredictable fail- ures on downstream tasks (Shorten & Khoshgof
H&E stained histopathology im- ages have characteristic appearance with blue tones indicating cell nuclei, while cytoplasm is stained pink (Chan 2014). Bottom: Color jitter with the standard settings of (Chen & He (02T) produces unrealistic augmentations with altered meanings. Choosing good augmentations requires domain knowledge (Shen et al., 2022).
Figure Top:
taar 2019).
Instead of relying on transformation invariance, many self-supervised learning techniques involve masking and reconstructing input data including masked image modelling (He et al. 2022) Xie et al. 2022) for computer vision and masked language modelling (Devlin et al. 2019 Raffel et al.] 2020) for natural language processing. However, these
methods often require specific backbone architectures like transformers (Vaswani et al. 2017) to
achieve good performance (Balestriero al. (023).
In this paper, we introduce a SSRL training scheme that requires neither domain domain-specific data augmentations nor particular architectures in masking approaches. Instead, the proposed learning method is based on surprising hypothesis that good data representations can be obtained by learning to simultaneously reconstruct multiple randomly generated data projection functions. The hypothesis comes from the conventional motivation of representation learning caapuring and extracting abstract and valuable concepts that can support a range of downstream predictive tasks (Bengio et al. 2013 Le- Ke-K et al., 22220). In particular, the downstream tasks could include arbitrary data projections. Formally, given random projection functions G {... (b(k) (x) ....) whose input domains are raw data features x E X. the motivation above suggests that, for a good representation z of data x, there another group of simple prediction functions H = {... h (k) (z)...) that can correctly predict the random functions' outputs. With this insight, the representation learning task can be construed as search for a combination representation and prediction functions that can reproduce random data projections. In short, we conduct SSRL by learning from randomness (LFR).
The primary advantage of LFR that random projection functions G can easily be created for any data modality. One straightforward instantiation is by taking neural network with suitable architecture for consuming the data, and randomly initializing its parameters. Hence, LFR applics to all subfields of SSRL. Also, data augmentations not used, so LFR avoids any concerns of unsafe, identical, or unrealistic augmentations as discussed above.
We empirically evaluate the effectiveness of LFR wide range of representation learning tasks that span diverse data types (including image, sequential, and tabular) and multiple real-world applications (banking, healthcare and natural sciences). The results show that LFR outperforms commonly used aggnsic SSRL algorithms. It is even competitive with many domain-specifti approaches that rely heavily on expert knowledge for their data augmentation designs. The remarkable performance demonstrates that learning high-quality data representations from randomness is feasible and plausible alternative when the transformation invariance assumption is hard establish or enforce in given application domain.
2 BACKGROUND AND RELATED WORK
Self-supervised representation learning (SSRL) methods enable the extraction of informative and compact representations from raw data without manual annotation or labelling. These methods rely on large amounts of unlabeled data and pretext tasks to implicitly model the observed distribution
2

---

Published as a conference paper at ICLR 2024
and optimize deep neural networks. In computer vision (CV) and natural language processing (NLP), SSRL has gained considerable attention due to an abundance of widely available unlabeled data.
Adopting the definitions outlined in (Balestriero et al. 2023). SSRL methods in the field of CV can be grouped into four main categories: deep metric learning CChnn et al., 2020), self self-distillation
(Grill et al. 2020 Chen et al.,, 2020), canonical correlation analysis (Zbontar al., 2021). and masked image modeling (He et al., 2022). Among these, the first three rely on creating positive views of given image to learn invariances. This is achieved by creating augmented versions of the same image and enforcing that the latent embeddings of these versions should be identical, with the underlying assumption that the semantic meaning of the original image invariant across different views. Recently, masked image modeling has emerged as a popular SSRL approach due to the success of Vision Transformers (Dosovitskiy et al., 202T These methods predict masked vision tokens (He et al. 2022) pixel patches (Doersch et al.] 2015), which have proven useful in learning representations. In NLP, masked language modelling is very prominent. The dominant models, including BERTs and GPTs, are trained to predict masked language tokens, which encourages the model to encode contextual information and reconstruct its inputs. Occasionally, other efficient language- inguuge-focused pretext tasks emerge in the literature, such maximizing the mutual information between a global sentence representation and n- gramss in the sentence (Kong et al.) 2020).
Despite the remarkable success in CV and NLP, the effectiveness of SSRL for other data modalities, such as tabular data, has been limited (Bahri et al. 2022 Balestriero et al. 2023). One challenge for SSRL methods relying on transformation invariance lies in identifying and designing appropriate augmentations. Augmentation strategies that work well for one modality may not directly translate to others due to inherent differences, and the choice of suitable augmentations can also be influenced by the specific application domain. For example, augmentations designed for natural images may not be suitable for medical images with low color variation, leading to unrealistic results and unsatisfactory performance (Kang et al., 2023). Appendix [A] proovdds further information on research efforts dedicated to designing effective modality- and application-speet While masking approaches offer general applicability to all data modalities, the most effective frameworks often rely on transformer- based backbones for optimal performance (Majmundar et al., 2022; He et al. 2022 Cheng et al. 2023). In this work, we focus on modcl-agnostic SSRL approaches. Classic autoencoder-base
augmentation techniques.
methods provide an alternative to SSRL without relying explicitly on transformation invariance (Hinton & Salakhutdinov [006 Vincent al.] [008 Zhhng jet al. 2017). However, these methods tend to prioritize low-level reconstruction over capturing high- level abstractions required for downstream tasks, resulting in suboptimal performance in practical applications (Liu et al. 202T).
The current landscape of SSRL research highlights the need for more versatile and effective approach capable of addressing wider range of modalities, applications, and architectures.
3 REPRESENTATION LEARNING FROM RANDOM DATA PROJECTORS
In this section, we present learning from randomness (LFR), an efficient and general SSRL algorithm. We recap the representation learning problem setting as the following: given observed raw data X == {....; ... }. where all data points share the same feature domain X', the representation learning task is to learn function (X) that produces a low-dimensional representation z, E for each raw data input x,, The representation z; should carry useful information about x; such that for an arbitrary downstream task g(X) is possible to learn a simple prediction function h2(2) that replicates g(x)) as his (fo(x;) for all x; E X.
3.1 PRETEXT TASK: MULTI-OBJECTIVE LEARNING FROM RANDOMNESS
As mentioned in the problem statement above, the ultimate purpose of representation learning is to support arbitrary downstream predictive tasks. In reality, there is usually small subset of downstream tasks which are considered important. It is not a priori clear that directly learning to predict purely random tasks would lead good representations for important tasks.
To demonstrate the possibility of learning from randomness, we propose the surprising pretext task shown in Figure?? 2 The pretext task contains three components, namely a representation model fo())), set of randomly generated data projection functions G (x)) ... and set of
3

---

Published as a conference paper at ICLR 2024
Targets
Random Projectors
X
Loss
Stop gradient
y
fo
Loss
Stop guadiect
Hommmeg
Loss
gradiant
y!A)
Figure 2: Our proposed architecture for learning from randomness. An input is encoded by fo into a useful representation z, while also being fed to random projection functions gd4), Simple, learnable predictor functions n try to match the outputs y(*) from the projectors g()), which is only possible when contains rich information about the input.
simple predictors He (-.--hh (z) function respectively. Formally, we propose optimizing the high-level objective:
(z).... .... that aim predict the outcome of each random projection
argmin Σ Σ
(k) (x,), (h) (fo(x.)) (0) (2
(1)
X.EX
where -1 is divergence metric measuring the similarity between its inputs, the (2())ss denote regularization terms, and the X's are the corresponding weights. To make this a non-trivial task, the predictors (/))) should have limited capacity, such as being linear functions, simple neural networks with few layers. This objective aligns with existing SSRL methods that use predictors (rril et al. 2020 Chen & He (021).
Objective is essentially lower-bound of the maximum likelihood estimation (MLE) objective; aim maximize the probability of observing data projections y! = (kx) (x,) given all datapoints,
Σ llogp(y x;) Σ log
(y:
2i p p2: x;) dz.
k
(2)
2 EΣ / p(z x,)
logp (x!h)
(k) zi dzz
ΣΣ logp (y!*) z: = fo(x)))
where (z; x;) is a Dirac delta distribution since the representation model is a deterministic function. As an example of the connection between Objectives (1) and (2), when the ply (k) 2.) are assumed
to be Gaussian distributions, the corresponding 'D in Objective (I) is the Euclidean distance. We discuss other options for D below.
A straightforward training strategy for Objective (I) is joint training where we treat the representation model fo and predictors Hę multi-objective autoencoder, updating all parameters in the same backpropagation pass. However, in preliminary experiments this naïve training strategy showed fluctuating progress which prevented the model from converging to satisfactory solutions.
We tackle the training instability issue by adopting the classic method. Considering the MLE lower-bound (Objective (2), optimizing the representation model fo is an E-step that repositions the posterior distribution of data p( ((2 given fixed log-likelihood
Dnteectation-Mxxiizzzinn (EM)
estimation modules logp(y; (A) zz). For the M-step, the representation distribution is fixed and we optimize the predictor heads Ha. Details of the derivation can be found in Appendix G Hence, in
this work we train the proposed SSRL model by alternating steps:
E-step: Optimize the representation model parameters 0 for one iteration
argmin ΣΣ (x:), 11 (1) (Jo(x.)) (O),
(3)
M-step: Optimize the predictor model parameters + for M iterations
argmin ΣΣ D (x,,) h (fo(x:)) + A2)
(4)
4

---

Published as a conference paper at ICLR 2024
Optimizing the predictor model for more iterations than the representation model brings the predictor closer to its optimal performance with the latest representation model fa. Previous studies have shown that optimizing the predictor to achieve optimality leads to improved performance (Chen & He 2021 Grill et al., 2020; Tian et al. 2021).
3.2 DIVERGENCE MEASURE: BATCH-WISE BARLOW TWINS
Now we return to options for the divergence D in Objective T While there are several common choices of divergence used in machine learning such as Mcan Squared Error (MSE), Cross En- tropy (CH), or the Contrastive (Chopra et al. 2005) and Triplet (Schroff' et al. 2015) losses, they often inadequate for enforcing identifications between subtly different points is crucial for representation learning tasks; MSE downweights the importance of small errors, CE is ill suited for regression tasks, while the Contrastive and Triplet losses introduce significant stochasticity.
The Barlow Twins loss (Zbontar et al. 2021 has garnered much interest in the SSRL literature as it disentangles learned representations through redundancy reduction. We also note its ability to scale to very -igh-dimensiona vectors (Zbontar et al. (202I). Thus, we introduce Batch-wise Barlow Twins (BBT), a variant that measures representation differences between data instances from two sources, the random projector g( and the predictor h/ rather than disentangling the representation encoding. We define the BBT loss as
LBTT =ΣΣ &
1 (1-c))
+
(k)2 ij
(5)
where the 6jj are the entries cosine similarity matrix,
(R)) T
(R)
9 (x:), (K) = h (fe(x:)),
(6)
and y; E Rd( Compared to the loss in (Zbontar al. 2021 Equation 5 has an extra summation over the ensemble k. The main difference comes from the definition of the cosine similarity matrix; our cosine similarity is an m m matrix with m the batch size, whereas in Barlow
Twins it d(k) x (k) matrix.
3.3 DIVERSITY ENCOURAGEMER ON RANDOM DATA PROJECTORS
Learning from randomness aims extract useful representations from random projection functions (x) E G which mimic arbitrary downstream tasks. In practice create multiple data projections by randomly initializing neural networks that reuse the architecture design of fe. but scaled down, which avoids the need to make domain-specific choices about the projectors. Functions generated this way can often capture similar information to each other when diversity is not specifically encouraged, which limits the generalization capabilities of the representations learned by fa. While increasing the number of random projection functions could mitigate the diversity problem by brute force, such an approach is computationally wasteful because would maintain many similar projectors.
We propose a solution that picks K diverse projectors from N >> K randomly generated candidates. The underlying hypothesis is that one sufficiently large batch of data can reveal the behavioral differences between candidate random projectors. Presuming there is batch of data X e Rxd for each of the N randomly generated projectors g ((k) (X') E G we produce the normalized outputs
(k) (x==g)))))))))) ( (k) E (
We then compute the cosine similarity over the batch of outputs for each projector as
A (h) = ()(()) (
mmx
(8)
By flattening the matrix (k) and again normalizing, we obtain a vector (k) E _ which acts as the signature of the k th projector with respect the batch. Finally, to select K target models from
5

---

Published as a conference paper at ICLR 2024
the N candidates, search for subset that maximizes the following binary constraint optimization problem involving matrices A made from K stacked vectors a (*),
argmax det(B)) s.t. = A
E [0, 1, Σ sk - K}. (9)
where the I's in the binary vector {0, 13 indicate the chosen projectors. While this problem is known to be NP-hard, approximate methods such as the Fast Determinantal Point Process (Chen et al. 2018) can find good solutions in reasonable time. It is worth noting that diversity encouragement solution does not involve gradient computations, and can be run once as a pre- pre-procsssiig step without occupying computation resources during the SSRL training phase. In addition, we also explored other diversity encouraging techniques through initialization in our experiments and provide analysis of the impact on performance Section 4.5 We summarize the full LFR algorithm in Appendix
4 EXPERIMENTS AND EVALUATION
4.1 DATASETS
We consider diverse data types to show the wide-ranging applicability of learning from randomness.
Time series: We utilized two standard time-series datasets, Human Activity Recognition (HAR) (An-
guita et al., 2013) and Epileptic Seizure Recognition (Epilepsy) (Andrzejak et al., 200T). Both datasets pre-processed using the same methods as in TS-TCC (Eldele al. [2021). As larger
scale test we also include the MIMIC-III dataset, a standard the medical domain for tasks involving
electronic health record data. We utilized the pre-processed version of the MIMIC-III Benchmark
dataset (Harutyunyan et al. 2019), and focused on the length-of-stay task (Yèche et al. (02T) which is framed 10-class classification problem, where each class represents different duration of stay.
Tabular: We used three tabular UCI datasets in our experiments: Adult Income (Income) (Kohavi 1996), First Order Theorem Proving (Theorem) (Bridge et al., 2014), and HEPMASS (Baldi et al. 2016). For Income, a binary classification problem, we followed the data preprocessing steps in (Ucar al. 202T). The Theorem dataset framed 6-class classification problem. The much larger HEPMASS dataset is another binary classification task which includes 7 million training and 3.5 million testing events, each with 27 features.
Computer vision: We tested on Kvasir (Pogorelov et al. 2017) medical image dataset consisting of 8,000 images of the gastrointestinal tract. There are eight balanced classes including seven disease types well as healthy images. We followed (Kalra et al.) 2023) to resize all images to 100x 80 pixels and split the data into 6,000 images for training and 2,000 for testing. We provide further results CIFARIO in Appendix D.6
4.2 IMPLEMENTATIONS
Evaluation: All the downstream tasks in our study treated as classification problems. To evaluate the quality of the pre-trained representations, we employed supervised classifiers that are specific to each dataset. For the MIMIC-III dataset we utilized a MLP classifier (Yèche et al. [202]). For tabular datasets, we used logistic regression, similar to the approach in STab (Hajiramezanali et al. (022). For the remaining datasets, linear classifier was employed. The classifiers were trained the frozen representations of the training set and evaluated on the test set, which is the most commonly used protocol. We also include finetuning results in Appendix 1D.4and transfer learning results on Time Series in Appendix D.I. Accuracy is our primary evaluation metric, except for MIMIC-III where we adopted linearly weighted Cohen's 's Kappa as in (Yeche et al., (02T). with higher values indicating better agreement. To ensure the robustness of our results, conducted multiple random runs and report the mean and standard deviation, using 5 runs for tabular datasets and 3 runs for others.
Model architectures: Regarding the model architectures, we adopted similar backbone encoders as previous works, For the HAR and Epilepsy datasets, we utilized the same 3-block convolutional layers as TS-TCC (Wang al., 2017). For the MIMIC-III dataset, Convolutional Network used by NCL (Yèche et al., 202T). For the Tabular datasets, we used 4-ayer MLPs, following the approach in SCARF (Bahri al. 2022. For Kvasír, employed the ResNet18
employed the Temporal

---

Published as a conference paper at ICLR 2024
Table 1: Baseline methods
Category
Method
Description
Autoenculer (Rumelhart et al.
1986
Encodenldevoder with low dimensional latents trained via the reconstruction loss.
Domain-gmostic
DACL (Vernii
Self supervisdd learning method that mix-up data augmentation across modalities. sule-vupee learning method that predicts the datum index as pretext lusk.
DIET [Salestner 2725
Domain speciilc
SimCLR Chen
2020
Contrustive learning method with both positive and negutive pairs.
augmentations
SimSiam Chen & HD
2021
Self-supervised learning with Stamese aetworks and only positive pairs.
Time series
TS-ICC (Bldelc et al
2021
Contrastive learning method that uses a corclltioo-based similarity capture temporal relationships time series augmentations to generate positive negative views.
SCARF (Bahri et al. 2022
Adaptation of SamCLR to tabular domains, using random corruption for dual views. augmentation--nn framework tabular self-supervised learning akin SimSiam. Positive pairs are created by different regularizations in the forward pass.
Tabular
Hajuramezannin
2022
Supervised
LogReg
Supervised
Supervised training with a classilication layer added to the cacoder used in other methods.
Supervised training with logistic regression.
Ablation
Random Init
As ablaion baseline we report the accuracy using a randomly initialized encoder (Edele ct
2021]
architecture (He et al. 2016). To avoid domain-specitte projector design, in each case the random
projectors reuse the encoder architecture, but are scaled down. Complete details are in Appendix C
Baseline methods: T summarizes all baselines used in our experiments. It is worth noting that while our proposed framework LFR is domain-agnostic, popular SSRL methods such SimCLR and SimSiam require domain-specine augmentations to achieve optimal performance. Specifically, the default augmentations used for view creation in SimCLR and SimSiam are designed for natural image classification, and may not be suitable for other modalities. In our experiments with tabular datasets, we compare our approach SCARF (Bahri et al., 2022), which a version of SimCLR adapted to tabular data that uses random corruptions as augmentations, For more detailed information on the implementations and augmentations, please refer to Appendix
4.3 PERFORMANCE ACROSS DATA MODALITIES
The performance of LFR and basclines across multiple modalities is shown in Table 2 Our ex- periments show that for modalities where there are no standardized augmentation pipelines such as medical images, time series, and tabular data, LFR had the strongest performance among the SSRL methods, outperforming other self-supervised learning methods in most cases including the domain-agnostic ones such DACL. For instance, on the HAR and Epilepsy datasets, LFR was the best performing method, beating the time-series specific self-supervised learning method TS- TCC. Similarly, for the Income and Theorem datasets, LFR outperformed the tabular data specific self-supervieed learning basclines SCARF and STab, Although on the HEPMASS dataset LFR was not the best, it still performed well, comparable to the autoencoder and SCARE. Interestingly, for the Income dataset, LFR even outperformed supervised training. For time series and tabular data, augmentation- based methods like SimSiam tend to underperform. For example, SimSiam worse than randomly initialized encoder in HAR and Income.
This experimental result reflects our hypothesis- it is feasible learn high-quality data representa- tions across all modalities tested by predicting random data projections. LFR shows comparatively good performance on domains where cemantic-preservve augmentations are difficult create.
4.4 PERFORMANCE ON MEDICAL APPLICATIONS
To evaluate the performance of LFR on specific application domains, we conducted performance comparison two medical datasets: Kvasir for medical images and MIMIC-III for medical time series (see shaded columns in Table D Our findings show that LFR performed the best on Kvssir and was highly comparable to other well-performing baselines on MIMIC-I11.
It is worth noting that the standard image augmentation pipeline used in other self-superriied methods is heavily tailored towards natural images and does not lead to superior performance medical images ( Similarly, the standard augmentations proposed for time series do not capture the
'ooee images in the Kvasir dataset have unnatural green annotation boxes in the bottom left corner which illustrate the configuration of the endoscope as it captured the image (Pogorelov et al. 2017), which may conflict with standard image augmentations. It would require domain knowledge craft suitable augmentations.
7

---

Published as a conference paper at ICLR 2024
Table 2:
Performance comparison across various data modalities and application domains. Results of the best self-supervised learning methods are in bold. Shaded columns denote medical applications.
Time series
Tabular
Image
HAR
Epilepsy MIMIC-III
Income
Theorem HEPMASS
Kvasir
Log Reg
57.5
80.9+NIA
47A
84.8+N/A
45.
90.7=NIA
Supervised
96.0+10.6
98.3+0.1
48.8+00
81.5+02
53.8=0.5
91.5+000
83.2
Random Init
80.7+2.3
89.1
42.4+111
83.1+02
44.9=0.8
84.3+11
28.945.7
Autoencoder
77.2
90.8+1..
44.940.
85.0+0.1
50.0=04
90.7:00.
72.4400.6
Self-supervised
DIET
88.6+1.3
96.8.10.3
33.815.
82.2.1.4
47.1
71.3
SimSiam
65. 10.8
97.4
41 41.0+1.9
79.2419
40.9
85.343.1
1..641.4
SimCLR
87.810.4
97.4
44.1
72. +0.3
SCARF
84.2....1
48.5
90.140.
STab
84.2.10.
50.7
83.6+1.7
TS-TCC
91,2
97.61m2
38.5
DACL
90.7
97.5
40.9
79.8
47.6=1.0
88.7
72.0
LFR (Ours)
93.1
97.9:0.22
46.6++0.3
85.2
51.6=007
90.1
74.9
unique characteristics of online monitoring data. In contrast, LFR does not require augmentations, making a promising approach for application- -pecciii datasets and tasks. Overall, our results suggest that LFR is an effective method for learning meaningful representations, even for specific applications that would otherwise require domain knowledge.
4.5 IMPACT OF RANDOM DATA PROJECTOR DIVERSITY
In this section, analyze the impact of the diversity of random data projectors on LFR using the Kvasir dataset. Projector diversity can be influenced at two stages: initialization and selection. For projector selection, we used the Determinantal Point Process illustrated in Section 3.3] Regarding projector initialization, we employ two techniques: Beta initialization and weight dropout.
Beta initialization: We drew inspiration from the Pre- witt operator (Prewitt 1970) and initialized the weights of 2D convolutional layers to create diverse targets that emphasize various edge features. To achieve this, we used a scaled version the Beta distribution with parameters = 0.5 and B 0.5 to initialize the convolutional layer weights to be close to -1 and
75
74
70
None
Selection
Initialization
Both
Weight dropout: We utilized DropConnect (Wan et al. 2013) to initialize the target networks. This involved ran- domly setting fraction of weights to with corrup- tion rate of 0.4. Unlike standard DropConnect used for network regularization, we froze the weights after initialization enhance diversity in the target representations.
Diversity Encouragement
Figure 3: Effect of target diversity
We compared the effectiveness of diversity encouragement at both the initialization and selection stages through ablation Kvasir. Our results, shown in Figure demonstrate that promoting diversity in the projector set at both initialization and selection helps with the downstream performance. These findings underscore the importance of diversity in the projector set and highlight the effectiveness of our proposed methods for promoting diversity in LFR.
Results:
4.6 ABLATION STUDY
this section, we investigate the impact of hyperparameters on the model's performance. To provide more focused analysis of the trends observed in ablation study, present the results for Kvasir, which is representative the other datasets. While we have observed similar trends other datasets, the significance may vary depending the specific characteristics of each.
Number of projectors: We evaluated the impact of the number of random projectors on linear evaluation accuracy using Kvasir data. We plotted the mean accuracy and standard deviation across
8

---

Published as a conference paper at ICLR 2024
76
75
74
73
72
71
71
71
70
70
70
Joint
Number projectors
Predictor epachs
Figure 4: Test accuracy with different hyperparameters on Kvasir. Left: Number of random projectors. Middle: Batch size. Right: Predictor training setting.
3 runs for K = 2, 4, and in Figure 4 The results indicate that the learned representation has higher linear probing accuracy the number of projectors increases until it reaches 6. Beyond K 6, 6, there is the possibility of reoccurring representations generated by the projectors, which introduce bias into the encoder's 's training process. This could happen the gradient descent optimization process might excessively favor redundant features, potentially leading to decreased performance. Based these findings, used K 6 for our experiments Kvasir.
Batch size: Because the selection of diverse projectors in LFR relies on one representative batch of data, examine the sensitivity of LFR to training batch sizes. We plotted the linear accuracy with batch sizes of 64, 128, 256, and 512, and reported the mean accuracy and standard deviation across runs in Figure Our results indicate that LFR has relatively stable accuracies across batch sizes, with no significant difference between them. However, the best performing batch size on the Kvasir dataset was 256, and thus we used this batch size for our subsequent experiments.
Predictor training epochs: Another option in LFR is how the encoder fe and predictors Ha are trained. We tested joint training, where all models are updated together, and alternating training with various numbers of epochs for the predictors between each encoder epoch. From Figure our findings on Kvasir indicate that updating the predictors for several epochs can improve performance compared to joint training, suggesting that more optimal predictors provide better learned representations.
conducted an analysis on effect of latent dimension on accuracy. Jigure[jdmmonsrraan that as the latent dimension increases, there is corresponding improvement in performance accuracy. However, when the latent dimension is more than 2048, the performance increase minimal. Therefore, we used 2048 for experiments.
Embedding dimension:
74
68
256
512
1024
2048
4096
Additional experimental results are provided in Appendix
Latent Dimensian
Figure 5: Test accuracy with differ- ent embedding dimensions.
5 CONCLUSION
This paper presents a novel self-supervised representation learning framework that is both modality- agnostic and upplication-agnostir Our proposed framework utilizes random projectors learn representations from unlabelled data, demonstrating excellent performance across various modalities and applications, particularly in situations where robust augmentation pipelines are not yet established.
The LFR technique is best suited to situations where the data cannot be reasonably augmented (due the lack of domain knowledge). Although surprising, this situation occurs frequently in critical application domains, such healthcare as we have highlighted. For general applications, however, if one knows the application domain well with adequate intuition around sensible data augmentations, using contrastive-leane Leaeree--eannnn-bbaaaaa based SSRL is still likely to outperform random projectors. Overall, treat LFR as a great complement to SSRL literature to fill the gap of data augmentation-Fin SSRL that satisfies the needs of many crucial applications.
While LFR encourages the use of SSRL across modalities and domains without the need for expert knowledge craft augmentations, the limited human input the learning process may increase the risk of sensitive features being misused, leading to privacy or faimess concerns. We recommend human oversight of self- superviee methods monitor for appropriate of data.

---

Published as a conference paper at ICLR 2024
Reproducibility Statement Towards the goal of reproducibilityy, we have provided our anonymized code repository as supplemantary material with this submission. The codebase includes instructions on how build the required environment, and how to our proposed method as well baseline methods. We also provide instructions for how to access the datasets, and run our pre-processing code. All relevant descriptions of the model architectures and hyperparameters are described in Section [4.2 and Appendix C Additionally, an algorithmic description of our proposed method in pseudocode is provided in Appendix statement on the computational resources spent given Appendix Additional notes on reproducibility are detailed in Appendix C.4
REFERENCES
Ralph G Andrzejak, Klaus Lehnertz, Florian Mormann, Christoph Rieke, Peter David, and Christian Elger. Indications of nonlinear deterministic and finite- dinte-iimensional structures in time series of brain electrical activity: Dependence on recording region and brain state. Physical Review E, 64 (6):061907, 2001.
D. Anguita, Alessandro Ghio, L. Oneto, Xavier Parra, and Jorge Luis Reyes-Ortiz. A public domain dataset for human activity recognition using smartphones. In The European Symposium on Artificial Neural Networks, 2013.
Dara Bahri, Heinrich Jiang, Yi Tay, and Donald Metzler. Scarf: Self-supervised contrastive learning using random feature corruption. In International Conference on Learning Representations, 2022.
Pierre Baldi, Kyle Cranmer, Taylor Faucett, Peter Sadowski, and Daniel Whiteson. Parameterized neural networks for high-energy physics. The European Physical Journal C, 76(5):235, Apr 2016. ISSN 1434-6052. doi: 10. 10/ee//sspss/ 10552001-4---004994944
Randall Balestriero. Unsupervised Learning on a DIET: Datum IndEx as Target Free of Self- Supervision, Reconstruction, Projector Head. urXiv:2302.10260 2023.
Randall Balestriero, Mark Ibrahim, Vlad Sobal, Ari Morcos, Shashank Shekhar, Tom Goldstein, Florian Bordes, Adrien Bardes, Gregoire Mialon, Yuandong Tian, al. A Cookbook of Self- Supervised Learning. arXiv:2304. 102 2023.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE Transactions Pattern Analysis and Machine Intelligence, 1888 2013.
Johann Brehmer and Kyle Cranmer. Flows for simultaneous manifold learning and density estimation. In Advances in Neural Information Processing Systems, volume 33, 2020.
James P. Bridge, S. Holden, and Lawrence Charles Paulson. Machine Learning for First- Order Theorem Proving Learning Select a Good Heuristic. J. Autom. Reason., 53:141-172, 2014.
John KC Chan. The wonderful colors of the hematoxylin eosin stain in diagnostic surgical pathology. International Journal of Surgical Pathology, 22(1): 2014.
Laming Chen, Guoxin Zhang, and Eric Zhou. Fast greedy map inference for determinantal point process to improve recommendation diversity. In Advances in Neural Information Processing Systems, volume 31, 2018
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning, volume 119, pp. 1597-1607, 13-18 Jul 2020.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15750-15758, 2021.
Joseph Y Cheng, Hanlin Goh, Kaan Dogrusoz, Oncel Tuzel, and Erdrin Azemi Subject-aware contrastive learning for biosignals. arXiv:2007.0487/ 2020.
Mingyue Cheng, Qi Liu, Zhiding Liu, Hao Zhang, Rujiao Zhang, and Enhong Chen. TimeMAE: Self-Superviser Representations of Time Series with Decoupled Masked Autoencoders. arXiv preprint ar arXiv:2303.003220 2023.
10

---

Published as a conference paper at ICLR 2024
S. Chopra, R. Hadsell, and Y. LeCun. Learning similarity metric discriminatively, with application to face verification. In JEEE Computer Society Conference on Computer Vision and Pattern Recognition, volume I, pp. 539-546, 2005. doi: (0.1109/CVPR.2205.222
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of NAACL-HLT, pp. 4171-4186, 2019.
Carl Doersch, Abhinav Gupta, and Alexei Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE International Conference Computer Vision, pp. 1422-1430, 2015.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition Scale. In International Conference on Learning Representations, 2021.
Emadeldeen Eldele, Mohamed Ragab, Zhenghua Chen, Min Wu, Chee Keong Kwoh, Xiaoli Li, and Cuntai Guan. Time-series representation learning via temporal and contextual contrasting. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, pp. 2352- 2359, 2021.
Mohamed Elgendi, Muhammad Umer Nasir, Qunfeng Tang, David Smith, John-Paul Grenier, Cather- ine Batte, Bradley Spieler, William Donald Leslie, Carlo Menon, Richard Ribbon Fletcher, Newton Howard, Rabab Ward, William Parker, and Savvas Nicolaou. The Effectiveness of Image Aug- mentation in Deep Learning Networks for Detecting COVID-19: A Geometrie Transformation Perspective. Frontiers in Medicine, 8:629134, 2021.
doi: 10.3389/fmee.2221..60 622.149134.
Linus Ericsson, Henry Gouk, and Timothy Hospedales. Why Do Self-Supervised Models Transfer? On the Impact of Invariance Downstream Tasks. In Proceedings of the 33rd British Machine Vision Conference 2022, 2022.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach supersueee learning. In Advances in Neural Information Processing Systems, volume 33, 2020.
Ehsan Hajiramezanali, Nathaniel Lee Diamant, Gabriele Scalia, and Max Shen. STab: Self- supervised Learning for Tabular Data. In NeurlPS 2022 First Table Representation Workshop, 2022. URL /tttpss:/ net/ wtr.m/orrmmmmmmm EIR5iiddEEEEB....oo crcl
Hrayr Harutyunyan, Hrant Khachatrian, David C Kale, Greg Ver Steeg, and Aram Galstyan. Multitask learning and benchmarking with clinical time series data. Scientific data, 6(1):96, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16000-16009, 2022.
Geoffrey E Hinton and Ruslan Salakhutdinov. Reducing dimensionality of data with neural networks. Science, 31 31355786)5504-007, 2006.
Brian Kenji Iwana and Seichi Uchida. An empirical survey of data augmentation for time series classification with neural networks. PLOS ONE, 16(7):1-32, 2021.
11

---

Published as a conference paper at ICLR 2024
Shivam Kalra, Junfeng Wen, Jesse C. Cresswell, Maksims Volkovs, and H. R. Tizhoosh. Decentralized federated learning through proxy model sharing. Nature Communications, 14(1):2899, May 2023. ISSN 2041-1723. doi: 10.1038//41447-023---333
Mingu Kang, Heon Song, Seonwook Park, Donggeun Yoo. and Sérgio Pereira. Benchmarking self- supervised learning on diverse pathology datasets. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3344-3354, June 2023.
Ron Kohavi. Scaling up the accuracy of Naive-Bayer classifiers: A decision-tree hybrid. In Second International Conference on Knowledge Discovery and Data Mining, volume 96, pp. 202-207, 1996.
Lingpeng Kong, Cyprien de Masson d' A'tuum, Lei Yu, Wang Ling, Zihang Dai, and Dani Yo- gatama. A mutual information maximization perspective of language representation learning, In International Conference on Learning Representations, 2020.
Phuc H Le-Khac, Graham Healy, and Alan F Smeaton. Contrastive representation learning: A framework and review. IEEE Access, 8: 8.199907-193934, 2020.
Xiao Liu, Fanjin Zhang, Zhenyu Hou, Li Mian, Zhaoyu Wang, Jing Zhang, and Jie Tang. Self- supervised learning: Generative or contrastive. IEEE Transactions on Knowledge and Data Engineering. 35(()8857-876, 2021.
Dongsheng Luo, Wei Cheng, Yingheng Wang, Dongkuan Xu, Jingchao Ni, Wenchao Yu, Xuchao Zhang, Yanchi Liu, Yuncong Chen, Haifeng Chen, and Xiang Zhang. Time series contrastive learning with information-aware augmentations. Proceedings of the AAAI Conference Artificial Intelligence, 37(4):4534-4542 2023. doi: 10. v1719/22....7
Kushal Majmundar, Sachin Goyal, Praneeth Netrapalli, and Prateck Jain. MET: Masked Encoding for Tabular Data. In NewrIPS 2022 First Table Representation Workshop, 2022.
Mostafa Neo Mohsenvand, Mohammad Rasool Izadi, and Pattie Maes. Contrastive representation learning for electroenccephalogran classification. In Machine Learning for Health, pp. 238-253.
PMLR, 2020.
Konstantin Pogorelov, Kristin Ranheim Randel, Carsten Griwodz, Sigrun Losada Eskeland, Thomas de Lange, Dag Johansen, Concetto Spampinato, Duc-Tien Dang-Nguyen, Mathias Lux, Peter The- lin Schmidt, et al. Kvasir: A multi-class image dataset for computer aided gastrointestinal disease detection. In Proceedings of the 8th ACM on Multimedia Systems Conference, pp. 164-169, 2017.
Judith Prewitt. Object enhancement and extraction. Picture processing and Psychopictorics, 10 (1):15-19, 1970.
Senthil Purushwalkam and Abhinav Gupta. Demystifying contrastive self-supervised learning: Invariances, augmentations and dataset biases. In Advances in Neural Information Processing Systems, volume 33, 2020.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21 21(140):1-67, 2020.
Mohamed Ragab, Zhenghua Chen, Min Wu, Haoliang Li, Chee-Keong Kwoh, Ruqiang Yan, and Xiaoli Li. Adversarial multiple-target domain adaptation for fault classification. IEEE Transactions on Instrumentation and Measurement, 70:1-11, 2020.
D. E. Rumelhart, G. E. Hinton, and R. J. Williams. Learning Internal Representations by Error Propagation, pp. 318-362. MIT Press, Cambridge, MA, USA, 1986. ISBN 026268053X.
Florian Schroff. Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 815- 823, 2015.
12

---

Published as a conference paper at ICLR 2024
Yiqing Shen, Yulin Luo, Dinggang Shen, and Jing Ke. RandStainNA: Learning Stain-Agnostic Features from Histology Slides by Bridging Stain Augmentation and Normalization. In Medical Image Computing and Computer Assisted Intervention, pp. 212-221, Springer, 2022.
Connor Shorten and Taghi M Khoshgoftaar. A survey on image data augmentation for deep learning. Journal of Big Data, 6(1):1-48, 2019.
Hari Sowrirajan, Jingbo Yang, Andrew Y. Ng. and Pranav Rajpurkar. MoCo Pretraining Improves Representation and Transferability of Chest X-ray Models. In Proceedings of the Fourth Conference on Medical Imaging with Deep Learning, volume 143, pp. 728-744, 2021.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? In Advances in neural information processing systems, volume 33, 2020.
Yuandong Tian, Xinlei Chen, and Surya Ganguli. Understanding self-supervised learning dynamics without contrastive pairs. In International Conference on Machine Learning, pp. 10268-10278)
2021.
Talip Ucar, Ehsan Hajiramezanali, and Lindsay Edwards. SubTab: Subsetting Features of Tabulaar Data for Self-Supervised Representation Learning. In Advances in Neural Information Processing Systems, volume 34, pp. 18853-18865 2021.
Terry T. Um, Franz M. J. Pfister, Daniel Pichler, Satoshi Endo, Muriel Lang, Sandra Hirche, Urban Fietzek, and Dana Kulić. Data Augmentation of Wcarable Sensor Data for Parkinson's 's Disease Monitoring Using Convolutional Neural Networks. In Proceedings of the 19th ACM International Conference Multimodal Interaction, pp. 216-220, 2017. doi: 10. 00.1145/3336555313366
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, 2017.
Vikas Verma, Thang Luong, Kenji Kawaguchi, Hieu Pham, and Quoe Le. Towards domain-agnostic contrastive learning. In Proceedings of the 38th International Conference on Machine Learning. volume 139, pp. 10530-10544 2021.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th International Conference on Machine learning, pp. 1096-1103, 2008. doi: 10.1 00.1145/1990560 13902944.
Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus. Regularization of neural networks using dropconnect. In Proceedings of the 30th International Conference on Machine Learning, 2013.
Zhiguang Wang, Weizhong Yan, and Tim Oates. Time series classification from scratch with deep neural networks: A strong baseline. In 2017 International Joint Conference Neural Networks, pp. 1578-1585. IEEE, 2017.
Jason Wei and Kai Zou. EDA: Easy data augmentation techniques for boosting performance on text classification tasks. In Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, 2019. doi: 10.18653/v1/
167-
Xing Wu, Shangwen Lv, Liangjun Zang, Jizhong Han, and Songlin Hu. Conditional bert contextual augmentation. In International Conference onComputational Science, pp. 84-95, 2019.
Tete Xiao, Xiaolong Wang, Alexei A Efros, and Trevor Darrell. What Should Not Be Contrastive in Contrastive Learning. In International Conference on Learning Representations, 2021.
Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. SimMIM: A Simple Framework for Masked Image Modeling. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9653-9663, 2022.
13

---

Published as a conference paper at ICLR 2024
Hugo Yěche, Gideon Dresdner, Francesco Locatello, Matthias Hüser, and Gumnar Rätsch. Neigh- borhood contrastive learning applied to online patient monitoring. In Proceedings of the 38th International Conference Machine Learning, volume 139, pp. 11964-11974, 2021.
Jinsung Yoon, Yao Zhang, James Jordon, and Mihaela van der Schaar. Vime: Extending the success of self-and semi-supervised learning to tabular domain. In Advances in Neural Information Processing Systems, volume 33. pp. 11033-1 11433, 2020.
Jure Zbontar, Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: learning via redundancy reduction. In International Conference on Machine Learning, pp. 12310- 12320. PMLR, 2021.
Self-supervised
Junbo Zhang and Kaisheng Ma. Rethinking the augmentation module in contrastive learning: Learning hierarchical augmentation invariance with expanded views. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16650-16659, 2022.
Richard Zhang, Phillip Isola, and Alexei A Efros. Split-brain autoencoders: Unsupervised learning by cross-channel prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1058-1067, 2017.
14

---

Published as a conference paper at ICLR 2024
A EXTENDED RELATED WORKS
Continuing our discussion of contrastive-eamming based pretext tasks, we summarize the most common augmentation techniques used in contrastive learning and compare them across domains.
Pretext tasks for contrastive learning
Contrastive learning methods rely heavily on augmentation generate positive views semantically similar examples which are optimized to have the same representation as the original datapoint. To ensure the quality of learned representations, augmentations should be semantic- prreettiiipeeeiingggee (Tian iet al. 2020; Balestriero et al. (202)). However, finding suitable augmentations for different application domains can be a challenging task, and researchers have invested considerable effort into this area to enhance downstream performance.
Computer vision Image datasets benefit from a wide range of semantically similar augmentations, including random cropping and resizing, horizontal flipping, color jittering, converting to grayscale, and Gaussian blurring (He et al. 2020, Chen et al.) 2020 Chen & He 202T). However, the best performing augmentations
often dataset-specife (Eriesson et al. 2022) and require domain knowledge to determine, For instance, enforcing color invariance by grayscale conversion may not be beneficial for flower datasets (Zhang & Ma 2022), but it can improve performance for ImageNet data (Chen et al. 2020). Furthermore, augmentations designed for natural images may not be suitable for medical domains (Elgendi al., 2021). For instance, MoCo-CXR (Sowrirajan al., 202I focuses on chest X-ray images and uses random rotation and horizontal flipping instead of random crops, Gaussian blur, and random grayscale, as the latter may alter disease labels or are not meaningful for grayscale X-ray images. In another study, the authors proposed specific color space transforms for pathology images (Kang et al., 2023), as naive color-jittering may produce unrealistic resulting images (Shen al. 2022). Finally, the choice of augmentations can even be task-specific (Xiao et al.) 2021p. As an example, aggressive cropping may be suitable for image classification, but may not be optimal for image recognition tasks (Purushwalkam & Gupta 2020
Time series Time series data often contains underlying patterns that are not easily identifiable by humans, unlike images with recognizable features (Luo al. 2023). Consequently, designing effective data augmentation methods for time series data poses significant challenges and often requires domain knowledge. For example, augmentations for wearable sensor signals include rotation to simulate different sensor placements and jittering to simulate sensor noise (Um et al. (017)) Other researchers have focused on bio-signals and introduced channel augmentations that preserve the semantic information in the context (Mohsenvand et al. 2020: Cheng et al., 2022)). Neighbourhood contrastive learning (Yeche et al. 2021) proposed leveraging patient identity information in online pa- tient monitoring and using near- wear-in-time data sequences of the same patient semantically equivalent pairs. However, these augmentations are often specifically designed for the dataset and downstream task (Zhang & Ma 2022). and their performance may deteriorate when applied to other time series data (Eldele et al. 2021). Therefore, identifying the optimal augmentation pipeline for each dataset and task requires extensive analysis (Iwana & Uchida 2021
Tabular SSRL methods are understudied in the tabular domain as designing effective semantic- preserving augmentations is particularly challenging for structured data (Yoon et al.) 2020). Like for time-series, it is often difficult for human to determine if two views should be considered semantically equivalent, and unlike computer vision small changes to individual features can drasti- cally change the content. SubTab (Ucar et al. 202T) proposed to generate positive views through different feature subsets. More recently, SCARF (Bahri et al. 2022) proposed to augment each record by corrupting a random subset of features. Finally, STab (Hajiramezanali et al. 2022] creates the contrastive views by imposing different regularization on the encoder for the same input.
15

---

Published as a conference paper at ICLR 2024
B ALGORITHM
We summarize the LFR algorithm in Algorithm which uses the subroutine in Algorithm
Algorithm LFR: Learning From Randomness
Require: Dataset D
(z:)²=1- number of random projectors K
Ensure: Encoder fo
1: Initialize encoder So
2: Initialize 10 K different random projectors and select K diverse projectors g" g using
DPP as introduced in Section 3.3
Initialize K predictors h1 h one for each projector.
4: for epochan in training epochs do
5: for each mini-batch do Train-Network(B3 (3, Ja. 1/. g*)
D. Train encoder
6:
Update parameters of fo with gradient descent using LEAT introduced in Section 3.2 8: end for
7:
9:
for epoch, in predictor epochs do for each mini-batch B do Train-Network(B, fa. h g²))
10:
Train predictor
12: end for
Update parameters of all hy with gradient descent using LBBT
13:
14: end for
15:
end for
Algorithm 2 Train-Network subroutine
1: procedure TRAIN-NETWORK(B, fe. h g²)
2: Compute representations: Z fo(B
3:
for k 1 to K do
4:
Compute output representations: 12)
5:
Compute representation from projector k: (Z)
6: Compute loss LBBT
7: end for
8:
end procedure
C IMPLEMENTATION DETAILS
C.1 DATASET SUMMARY
Table p a summary all datasets used in our experiments, along with the corresponding downstream tasks and evaluation metrics.
Table 3: Dataset description.
Datuset
Modality
Data Domain
Train Size
Test Size Downstream Task
Metric
HAR
Time series
Mohile sensors
7352
2947
Multi-class classification (6)
Accurncy
Epilepsy
EEG
9200
Binary classiication
Accuracy
MIMIC-I1 III
Time series
Patient Online Monitoring
2,568.619
$63,742
class-class classification (10)
Cohen's Kappa
Diagnosis (App
series
Motor current signals
8184
2728
Multi-class classification (3)
Accuracy
Income
Tabular
Census
30162
15060
Binary classification
Accurncy
Theorem
Tabolar
Logic Reasoning
3059
1530
Multi-class classification (6)
Accuracy
HEPMASS
Tabolar
Partiele Physics
7.000.000
3,500,000
Multi-olass classification (2)
Accuracy
Kvasir
Image
Medical Images
6000
2000
Multi-class classification
Accuracy
CIFARIO (App. D66
Image
Natural Images
50000
10000
Multi-class classification (10)
Accuracy
16

---

Published as a conference paper at ICLR 2024
Table 4: Details on LFR architectural parameters
Dataset
Projectors Projector Initialization
Encoder Architecture Projector Architecture
HAR//pilepsy//aul Diagnosis (App D.I
Pytorch Default
Three-blot CNN
Two-black CNN
Income/ Income/Theoren Theorem
Pytorch Default
Four-layer MLP
Two-layer MLP
HEPMASS
Pytorch Default
Four-layer MLP
Two-layer MLP
MIMIC-II III
10
Pytorch Default
Five block TCN
Three block TCN
Kvasir
8 Initializtion Weight Dropout ResNet18
Four-layer CNN
CIFARIO (App D.6
40
i Initializtion Weight Dropout ResNet18
Four layer CNN
C.2 NEURAL NETWORK ARCHITECTURES
HAR/Epiless/Fauut Diagnosis: For LFR, we used a three-block convolutional network from [Wang et al. 2017 Eldele et al. 2021) the representation model fa. For the predictors h
we used single linear layer. For the random projectors g g(b), we adopted a similar architecture the representation model but with slightly decreased complexity, two-block convolutional network with 16 and 32 channels, followed by two sequential linear layers with a hidden dimension of 256. For all other self-supervised methods, we used the same representation model for fair comparison. For SimCLR (Chen et al. 2020) and SimSiam (Chen et al. 2020), we used the same predictors as LFR, and a 3-layer ReI.U network of hidden dimension 512 as a projector. The first two linear layers are followed by batchnorm layer. To create the contrastive view for SimCLR (Chen et al. 2020) and SimSiam (Chen et al. 2020), we adopted the same augmentations as designed in TS-TCC (ldeec
al. (021).
MIMIC-III: For all methods, we followed the encoder structure from (Yèche al.) 2021 as the representaion model/encoder, with the exception that we used flattened temporal convolutional network (TCN) features followed by a linear layer, which produced the embedding size of 64. We also disabled the L2 normalization in the encoder. For the random projectors in LFR, we adopted three-block block TCN with kernel size of 2. followed by a linear layer with output channel size of 64 for each layer. The two-layer ReLU predictor is shared in LFR, SimCLR and SimSiam with a hidden dimension of 256. We used the same projector and augmentation as in HAR/Epilepsy for SimCLR and SimSiam.
Income/Theoremt For LFR, we followed the setup in (Hajiramezaaali et al. 2022 Bahri et al. 2022) and used a 4-layer ReLU network with a hidden dimension of 256 as the representation model, with single linear layer predictor. The random projector networks had similar architecture but were less complex, using 2-layer ReLU network with a hidden dimension of 256. For the contrastive baselines, we employed the same encoder and predictor for a fair comparison, and followed (Bahri et al. 2022) by using 2-layer ReLU network with hidden dimension of 256 as projectors. To generate the contrastive views, we used the SCARF (Bahri et al., 2022) augmentation technique to randomly corrupt features with values sampled from their empirical distribution, ensuring that our SimCLR baseline was identical to SCARE.
HEPMASS:
For the HEPMASS dataset, we used the same network architecture as for the In- come/Theorem datasets but with the output latent dimension of the encoder set to 16.
For LFR, we used Re (He al. 2016) the representation model, with an output dimension of 2048 for all datasets. For CIFAR TO we followed SimSiam (Chen et al. 2020) use the CIFAR variant of ResNet18. The predictors are 4-layer ReLU networks with a hidden dimension of 256. For the random projector networks, we adopted 4-layer CNN of channels [3, 8, 16, 32), each layer followed by a ReLU, and every two layers are followed by max- max-pooling. linear layer is used to map the output to 2048 dimensions. For the contrastive baselines, we adopted the same representation model for a fair comparison and used 3-layer ReLU networks with hidden dimen- sion 256 as projectors. To create the contrastive views, we employed the augmentation set adopted by SimSiam Chen et al. (2020) for CIFAR (RandomRes ietieesszpdrrpp. RandomHor! i z zont alFlip ColorJitter,
Kvasir/CIFAR10:
waadddGrrasscalan and added Gaussiand sissannnBur Bur for Kvasir. We applied the same set of augmentations to the implementation of DIET (Balestriero 2023). All other settings are the same as with the ResNet18 8 model.
17

---

Published as a conference paper at ICLR 2024
All supervised baselines the same representation model as the SSL methods, with the final layer being a linear classification layer.
We summarize all architectural related settings of LFR in Table 4
C.3 DETAILS OF TRAINING SETTINGS
LFR training settings: Table summarizes all the training settings used for LFR, while Table 6 outlines the evaluation settings used for downstream tasks. We used a Logistic Regression classifier used
for all tabular datasets including Income, Theorem, and HEPMASS, while for MIMIC-III MLP network to predict the length stay following (Yèche et al. 202I). For the remaining datasets, we followed prior works such as TS-TCC (Eldele et al. (021), SimSiam (Chen et al. (020), and BYOL (Grill et al. 2020 by using a linear classifier for classification tasks.
Table 5: Details on LFR Training Settings
Dataset
Optimizer Barch Size
Leuming Rate Optimizer Parameters
Epochs
HAR/EpillepyyFuu Diagnosis (App D.1
Adam
128
3e-4 5-(0.9, 0.999), wd-3e-4
Train epochs 200. Predictor epochs 5
Racome//heooren
Adam
128
le:3 3= 0.999), wd=0
Train epochs 100, Predicter epochs I
HEPMASS
Adam
512
f-(00.9 0.999), wd-0
Train epochs 20, Predictor spochs
Adam
5-10.9, 0.999) wd-5e-4
Train steps
Predictor steps
Kvasir CIFARIO (.6 D.6
SGD
256 Ie-4, vosine decay momestum=0.9, wd=Se-4
Train spochs 400, Predictor spochs
SGD
512 3--2, vosing decay momeatum=0.3. 999, wd=Se-1 Train epocht 400. Prodicter epochs =
Table 6: Details on Linear Evaluation Settings of SSL methods
Dataset
Optimizer Batch size
Learning Rate Optimizer Parameters
Epochs
HAR/Epilepsy
Adam
128
3e-4 8=(0.9, 0.999), wd=3e-4
100
MIMIC-III
Adam
4096
le-4 (=(0.,, 0.999), wd=5e-4
300
Kvasir
SGD
256
1e-3, cosine decay momentum=0.9, wd=0
100
CIFARIO IO (App D.6
SGD
512 0.2, cosine decay momentum=0.9, wd=0
100
Baseline training settings: All supe-super baselines adopt the same training setting as LFR unless stated otherwise. For DIET with CIFAR10, we followed the original paper to train 5000 epochs with 10 linear warmup epochs and then cosine decay. For DIET with MIMIC-III, we used batch size 512 and trained for 2000 steps with 10 warmup epochs. We reserved 5000 epochs for training the autoencoder on MIMIC-II with 500 warmup epochs. For other self-supervised methods with MIMIC-III, we also added 60 warmup epochs. We summarize the training settings of supervised baselines in Table 7
Table 7: Details on Supervised Training Settings
Dataset
Optimizer Batch size
Leaming Rate Optimizer Parameters
Epochs Augmentations
MAAR/Epilepsy/Fant Diagnosis (App D.i
Adam
128
3--4 (= 0.999), wd=le-4
500 None
Income/Theoaee
128
3=10.9, 0.999),
MIMIC-III
Adam
4096
5e-6 2-00. 0.999), wd-5e-4
10 Same as SimCLR and SimSiam
Kvasir
SGD
256
1e-2, vosine decay momentum=:09, wd=0
600 Same as SimCLR and SimSiam
CIFARJO D.6
SGD
512
3--2, cosine docay momentume099 wd=
800 Same SimCLR and SinSiam
C.4 REPRODUCIRILLT NOTES
TS-TCC: Our results for TS-TCC on the HAR and Epilepsy datasets had several discrepancies with the values reported in the original TS-TCC paper (Eldele et al. 2021). We discovered that in the official implementation of TS-TCC, the input data was augmented once and then kept the same throughout training, rather than being randomly augmented in each forward pass. We fixed this bug and were able to achieve better results. Additionally, we increased the number of training epochs for our supervised baseline, which also led to improved performance. Lastly, we noticed that in the original TS- TCC implementation, the random initialization ablation initialized linear classification head that was not trained, whereas we evaluated with a trained linear classification layer and saw significant increase in accuracy for this ablation.
evaluated using randomly
18

---

Published as a conference paper at ICLR 2024
Income
Theorem
latent dimension
latent dimension
16
16
0.55
0.82
32
32
Accuracy
256
Accuracy
256
0.80
0.50
0.78
0.45
0.76
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
Dropout Rate
Dropout Rate
Figure STab accuracy as function of dropout rate
STab: The original STab paper (Hajiramezanali et al. 2022) did not provide information about the random DropConnect ratio training hyperparameters used in their experiments. In our implementa- tion, we used the same training hyperparameters as other SSL methods and tested DropConnect ratios of 0.1, 0.2, 0.4, 0.6, and 0.8, with the results shown Figure 6 We selected the persorerroning ratio for each experiment and reported the corresponding results. We ended up selecting 0.1 for Income and 0.8 for Theorem.
D ADDITIONAL EXPERIMENTAL RESULTS
D.1 TRANSFER LEARNING EXPERIMENTS
We follow the settings in prior work (Elelee et al. 2021) and perform transfer learning experiments on four real-worke Fault Diagnosis datasets (Ragab et al., 2020). Each dataset was collected under different working condition and can be considered as separate domain with different characteristics. As shown in Table g LFR is able to learn features on the source task that transfer well to related tasks, outperforming the supervised approach (with the same model architecture) 10 out 12 dataset pairs, TS- TSC on pairs, and significantly better than all other approaches in terms of average performance. Hence,
that LFR is able transfer representations in tabular setting, at least when there some similarity in the domain of the datasets used.
Table Transfer learning experiments
(Enendder Arch.)
A-D
B->A
B-+D)
C-AA
C-B
C-+D)
D--A
D-
AVG
Supervised (Transt.)
34.28
44.94
34.57
52.93
63.67
99.82
$2.93
84.02
83,54
53.15
99.56
62.43
63,83
Supervised (CNN)
42.96
46.33
46.99
43.55
70.53
94.94
48.50
78.34
74.34
52.71
99.05
70.20
64.04
Rand Init (CNN)
79.55
68.80
79.95
78.96
81.34
70.60
80.02
80.10
80.13
57.77
75.05
TS-TCC (Transf.)
43.15
51.50
42.74
47.89
70.38
99.30
38.89
98.31
99.38
51.91
99.96
70.31
67.83
LFR (CNN)
90.51
81.63
93.40
75.99
75.48
88.64
69.32
80.54
88.34
78.92
87.90
75.26
82.16
D.2 EMBEDDING DIMENSIONS
As we discussed in Section 4.6 the dimension- ality of embeddings may have a strong effect on the richness of learned representations. To complement the image-based based results on Kvasir
Accuracy
presented in the main text, we also used the Theorem dataset to evaluate the performance of LFR and baseline SSRL approaches across latent dimension sizes. Figure shows that in- creasing the latent dimension improved the ac- curacy of each approach up about 256. LFR consistently outperformed all the other baselines across all the latent dimension settings.
SimSia Auttanceeeeee
Figure 7: Effect of embedding dimension on LFR and self-supervised learning baselines with the Theorem dataset.
19

---

Published as a conference paper at ICLR 2024
Table 9: Effect of sample size during random projector selection
Sample size
64
128
256
512
Accuracy
74.1 + 0.4
74.0 + 0.4
74.9 + 0.6
74.1 + 0.4
D.3 SAMPLE SIZE FOR PROJECTOR SELECTION
we use sufficiently large set data to select a diverse set of random projectors. In this section, we investigated the effect of the sample size for random projector selection with the Kvasir dataset. As shown in Table 9 the sample size does not heavily affect the accuracy (the fluctuation is less than 1.0%). Therefore, for the ease of implementation, use the same size of the batch size in training, i.e. 256 for K Kvasir.
As illustrated in Section 4.5
D.4 FINE-TUNING PERFORMANCE
In Section we evaluated SSRL methods on downstream tasks using linear evaluation. SSRL methods are sometimes finetuned to downstream task, so below we compare LFR's finetuning performance benchmark methods.
During the finetuning phase, we employed linear layer combined with labeled data to enhance the performance of the encoder. The architecture setup was consistent with that detailed in the main paper. However, recognizing the significant volume of data in larger datasets such MIMIC-III and HEPMASS, we chose semi-supervised learning approach in those cases. this strategy. we randomly selected 10% of labeled data from these datasets to facilitate the fine-tuning process. This decision was driven by the computational resources required when dealing with extensive data. Conversely, for smaller datasets, we conducted fine-tuning using the complete set of available labeled data, enabling us to evaluate the model's performance across the entirety of the datasets.
As shown in Table I0, through the fine-tuning process all methods exhibit more comparable perfor- mance across the datasets. LFR still achieved the best performance on a majority of the datasets we used, although with overlapping error bars to other methods in those cases.
Table 10: Performance comparison among the SSRL methods with finetuning.
Time series
Tabular
Image
HAR
Epilepsy
MIMIC-III
Income Theorem
HEPMASS
Kvasir
Supervised
96.0 + 0.6 98.3 0.1
48.8 0.00
81.5 + 02 53.8 + 0.5
91.5 + 0.0
83.2 0.2
Autoencoder
93.9 # 1.3 95.1 20
49.2 0.6
85.2 + 53.9 + 0.5
90.8 = 0.0
75.0 - 0.8
DIET
95.6 = 0,5 97.8 0.1
4
85.2 a0 52.4 0.9
74.4 + 0.3
SimSiam
93.4 = 97.9 0.2
49.4
85.2 52.5 + 0.8
90.7 = 0.0
74.5 + 0.6
SimCLR
93.7 97.8 0.2
48.6
74.5 0.6
SCARF
85.1 + 02 53.8
+ 0.8
90.9 + 0.0
STab
85.3 + 0.2 53.0
+ 0.7
91.1 - 0.0
LFR
94.7 + 1.4 98.2 + 0.2
49.6 0.1
85.3 + 0.1 54.3 0.4
90.8 + 0.0
75.5 + 0.7
D.5
LOSS FUNCTION ABLATIONS
In the main paper, we used the batch-wise Barlow Twins loss as a divergence measure, however this specific choice of loss function is not required for our method, and many other choices of contrastive loss could also work. As an ablation, we also ran LFR with the InfoNCE loss on the Kvasir dataset and observed slight decrease compared to the Barlow Twins loss (72.6 + 0.4 compared with 74.9 + 0.6).
D.6 PERFORMANCE ON CIFAR10
The primary focus of LER domain- domain-agnostic representation learning, mainly in cases where it is not clear how to usefully augment the data. As result, focused tabular, time-series, and medical
20

---

Published as a conference paper at ICLR 2024
Table 11: Performance on CIFAR10
Method
Supervised
Random Init
Autoencoder
DIET
SimSiam
SimCLR
DACL
LFR
Accuracy
91.4 11440.1 0.2
31.7 0.3
37.4 0.3
69.3 = 1.2
89.2 +
0.1
86.7 + 0.5
42.4 2.7
61.3
imaging applications in the main paper. Here, we explore the effectiveness of LFR on natural images with CIFARIO data. As shown in Table II] one might expect, SimSiam and SimCLR achieve the best performance out of our baselines thanks to their extensively engineered pipeline of image augmentations. Two domain-agnostic approaches that do not use image augmentations, DIET and our LFR, fall into second tier, outpacing the third tier of DACL, Autoencoder, and Random Init. This result in stark contrast what we found on Kvasir, where all methods ended up tight range. with LFR at the top. This suggests that the heavy augmentations of SimSiam and SimCLR really are specialized for natural images, and are not suitable for other domains within the image modality.
D.7 RANDOM PROJECTORS VISUALIZATION
To illustrate the behaviour of individual random projections, and the role of diversity selection, we trained LFR on a small image-based dataset, CIFARI and examined the nearest-neighbour representations for different projectors. First we selected a query image random, then for each of 100 randomly initialized projectors, we encoded all training images, and found the nearest encoded representations the query. Distances in embedding space are measured by cosine similarity. We then visually compared the nearest neighbours that were selected by pairs of projectors which were deemed most diverse or similar according to the DPP criterion from Equation (9). The pairs with the highest diversity (Diverse and Diverse 2) and the highest similarity (Similar and Similar 2) are
shown in Figure S
The results in Figure 8 show that our similarity measure is effective at selecting projectors that focus on diverse features. For example, in Figureßoon the frog query (top left) we see that the projector in the first represented white "edge" feature, since the nearest neighbours all share that feature from the query but are otherwise semantically different. The projector in the second row appears to focus on shape, and its nearest neighbours are very different from the first projector. On the other hand, looking at the bottom right set of images, the most similar projector pair selects very similar nearest neighbors with 2 out of 5 nearest neighbors being identical. This qualitative study suggests that promoting diversity in the projectors can lead to the capture of broader range of features. Then, diverse features will be available the representation model during training, potentially resulting in richer set of semantic knowledge and ultimately better performance on downstream tasks. This interpretation supported by the evidence Section 4.6 that encouraging projector diversity results in better performance on linear evaluation accuracy.
E COMPUTATIONAL RESOURCES AND TIME SPENT
The time series experiments with HAR and Epilepsy were conducted on Tesla V100 GPU with 32 GB of memory, except for TS-TCC which was conducted on a TITAN with 12 GB of memory. The experiments took total of 102 GPU hours, including all baseline experiments. The MIMIC-III experiments were conducted with an NVIDIA A100 GPU with 40GB of memory, except for TS-TCC which was again conducted on a TITAN with 12 GB of memory, and cost 608 GPU hours, including all baseline experiments. The Kvasir experiments were conducted using Tesla V100 GPU with 32 GB of memory, and they took a total of 1095 GPU hours, including all baseline experiments. The tabular dataset experiments with Income, Theorem, and HEPMASS were conducted on an NVIDIA TITAN GPU with 12 GB of memory. The experiments took a total 70 GPU hours, including all baseline experiments. The CIFAR experiments conducted cluster with single NVIDIA P100 GPU with 12 GB of memory per experiment, and they took a total of 3020 GPU hours, including all baseline experiments.
"CIFAR10 10 was chosen for this qualitative study rather than the other datasets we used because it has human-interprettblle features and visualizations.
21

---

Published as a conference paper at ICLR 2024
Qutty
Nearest Neiphicurs
Qudry
Wesrest Neightours
Dvess
Divens
2vess
Divens
Sm Arr
Sinilar
Sm 2r
Samitrr
Qurny
Querry
Velghoours
Dvers
Figure 8: Nearest neighbors identified by randomly initialized projector models. Each row shows the query image (left) and its five nearest neighbours identified by projector. The top two rows display the most dissimilar projectors, while the bottom two rows give the most similar projectors.
F NOTE ON COMPUTATIONAL EFFICIENCY OF LFR
Our experiments across diverse datasets consistently reveal that LFR's training time is often compara- ble to, or even notably shorter than the contrastive SSRL methods. For example, on Kvasir training SimCLR took around 9 hours, while LFR completed in just 2 hours the same machine. This assessment of training time considers both CPU and GPU usage.
The efficiency in LFR's training time be attributed to primary factors. First, existing SSRL methods rely on extensive augmentations that are performed on CPU and can lead to data loading bottlenecks. LFR has no such issue. CPU-based image augmentations account for a significant amount of SimCLR's s 9 hour training time on Kvasir. For datasets with less computationally intensive augmentations, training times tend to align more closely on the HAR dataset, LFR concluded training in approximately 3 hours while SCARF required around 3.3 hours.
Second, multiple passes through predictor heads in LFR is not overly burdensome. Unlike contrastive frameworks that use double augmentation and encoder passes, LFR requires fewer expensive oper- ations: one pass through each random projector before training, and one pass through the encoder followed by one pass through each small predictor during training. Given that the encoder is the largest network by far, LFR's training time often proves less than SimCLR which uses multiple encoder passes.
Compared to the cost of training an encoder, the diversity selection procedure using the Fast Deter- minantal Point Process (Chen et al. 2018) is almost negligible contribution to the total training time. To recap, to select K diverse projectors we initialize N small neural networks and for each we one forward pass size m² x K constructed from the network outputs. As a concrete example, on the Income dataset training LFR for 100 epochs took 849 our machine, while the DPP selection took 3
batch of training data of size Then, DPP is applied over matrices of
G MATHEMATICAL DERIVATION
Here provide more details the derivation of Equations through 4
22

---

Published as a conference paper at ICLR 2024
Consider the MLE objective in Equation 2 As the variable z; 's distribution could be intractable, represent it in a general form by the distribution g(z;) such that
(y! 2: p2z xx))
ΣΣ log [ (y. 2: p(z x1) dz, ΣΣ log / g(zi)
dz..
g(zi)
Using Jensen's ss inequality, the lower bound of the objective is therefore
8i
2; p (zi x;) dz ΣΣ
/ g(zi) log
(xt (R)
Zi (
\xs)
ΣΣ log g(z:)
dzi.
(Zi))
((2;)
Hence, maximize the variational lower-bound of the above equation with respect to the proposed distribution (Zz)), the optimal solution is simply to let equality hold (as in the classic EM algorithm),
log / g(z;)
(y:
Zi p (zi
xi) dz: ΣΣ ((z;) log
(y:
(k)
zi
(zi
xi)
dz;
((z,)
q(z,)
which, in turn, requires the following equation to hold,
p
yi
(k) Zi p
(zi
xx:
C,
q(2,)
where C' is a constant. Thus, the optimal solution for g(2;) is
p
Zi
((x))
(Zi x;)
(K)
p yi
x;
x:) delta distributions with probability conditioned respectively, the optimal solution of q(z:) is simply p(z, x,) given the conditions are satisfied. other words, we need p(z: xx) that can let y; (k) (h) (z.) for all Thus, the optimization is essentially EM algorithm where optimize
As both p(y?") (K)
2,) and ply (k)
deterministic functions modelled by (*t) and 11 (k)
0 and o alternatively to gradually increase the conditional likelihood of Equation 2
23


Reviews:
[{"review_id": "rIxFvmROOs", "reviewer": "Reviewer_ze2f", "summary": "This paper considers the problem of domain-agnostic self-supervised representation learning. The proposed method introduces multiple random projectors and corresponding predictors, and optimizes the batch-wise barlow twins loss, which constructs the Gram matrix instead of the empirical correlation matrix. To encourage the diversity, many projectors are initialized and then only 10% are subsampled for use. Experimental results on datasets from various domains show the effectiveness of the proposed method.", "questions": "Please address concerns in Weaknesses.\n\n> **post rebuttal**\n\nAfter discussion with authors, I feel that the experimental results are not sufficient to support the claim that they cover \"a wide range of representation learning tasks that span diverse modalities and real-world applications.\" Initially their experiments covered time series, medical image, and tabular domains, and the additional results in the natural image domain show that their method is not effective for natural images, compared to other baselines. **Authors are encouraged to explicitly limit the scope to the domains they experimented in the title/abstract/intro.**\n\nAlso, I am not sure if the comparison is fair (e.g., if they tuned hyperparameters for baselines properly), so experimental results are generally not convincing to me.\n\nThough I feel more confident on my rating, given that authors addressed all concerns from the other reviewers well and the proposed method is still interesting to me, I do not want to put too much weight to my rating.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Domain-agnostic representation learning is a timely topic.\n\n- The proposed idea is simple and the proposed method improves the performance in most cases.", "weaknesses": "- Discussion/comparison with other domain-agnostic methods seem to be not enough. For example, [Lee et al.] proposed a domain-agnostic augmentation strategy applied to image, speech, and tabular datasets, and [Wu et al.] proposed randomized quantization and experimented on image, point cloud, audio domains and the DABS benchmark. I suggest including discussion and experimental comparison with them. \n\n- It is good to see that experiments include various domains including time series, tabular, and image, but they seem to be relatively small and not commonly used for benchmarking machine learning models. For example, Kvasir is a medical image dataset, which is different from the widely used \"natural\" image datasets; it should be categorized differently from natural image datasets. Authors may want to refer to [Lee et al.] and [Wu et al.] to find commonly used datasets to provide the general applicability to various domains and scalability of the proposed method.\n\n- While the authors claim that the optimization strategy for the proposed method is EM, but it is not clear how the proposed alternating optimization is related with EM by looking at the formulation. I think the transition from Eq. (2) to Eq. (3--4) requires more explanation supported with some math.\n\n- The claim around batch-wise barlow twins that MSE is preferred over cross-entropy/contrastive/triplet losses is not justified. Isn't the batch-wise barlow twins loss just a kind of contrastive loss, in that it contrasts all samples within the batch? Note that the original contrastive loss (not the InfoNCE variation) also computes the MSE loss. An ablation study with different type of losses might also be helpful.\n\n- The criterion for diversity encouragement requires more intuition. It is hard to imagine what is going on when optimizing the proposed learning objective. Also, what is the computational cost for the NP-hard objective function?\n\n- The comparison might not be fair as the proposed method requires more computational cost to encode input with multiple random projectors and predictors compared to other baselines. The computational cost should be matched for a fair comparison and reported.\n\n[Lee et al.] i-Mix: A Domain-Agnostic Strategy for Contrastive Representation Learning. In ICLR, 2021.\n\n[Wu et al.] Randomized Quantization: A Generic Augmentation for Data Agnostic Self-supervised Learning. In CVPR, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ocya568X84", "reviewer": "Reviewer_gtio", "summary": "The authors of this paper propose a technique called Learning From Randomness (LFR), which allows the application of self supervised techniques in arbitrary data domains. The proposed method works by projecting the data into random representations, and then training a model to predict these random representations. The authors show that the resulting model can learn useful representations, even without domain knowledge for the datasets examined.", "questions": "I would be grateful if the authors could comment a bit on the choice of the number of projectors $K = 6$ and batch size $B = 256$. Intuitively, both of these values seem a bit small when trying to find diverse random projectors. The authors have tried going up to $K = 8$ and batch size $B = 512$, but I would like to know if they have tried higher values for these two hyperparameters (especially for the number of projectors $K$).", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is very clear and easy to understand. I did not have any issues in grasping the points the authors are trying to make.\n\n- The method proposed is novel, as far as I am aware. It is also very interesting, as learning from random data is not a very explored area of research. As the authors note, the proposed technique enables self supervised learning without the need for domain knowledge, in order to create good augmentations for the data. There is also a clear benefit from using LFR in the datasets examined, without having to rely on complicated techniques.\n\n- The authors perform ablations on the random projectors used, as well as the required diversity of the random representations and the training procedure for the model. I find it interesting that the authors perform an EM-based approach in learning the model, instead of simple optimizing all of its parts all at once. Similarly, I find equally interesting the preprocessing step that selects the best random projectors to predict during training.\n\nOverall, I find the proposed method insightful, with clear benefits over previous work in datasets that are not as explored as the usual ones (e.g. CIFAR-10/100, ImageNet).", "weaknesses": "- One of the issues I have with this paper is that despite the use of several datasets used to evaluate LFR, the commonly used ones such as CIFAR-10/100 are not among them. While I understand that LFR does not aim to improve performance on these datasets (since natural image augmentations already perform very well) it would be interesting to examine those to compare as well. It would be useful to know if LFR is better/worse than optimized augmentations, such as those used in SimCLR.\n\n- I think the paper could also be improved via further experiments on the following two subjects:\n\n  - I think it would be interesting to see some ablations on the distance metric used for training. Right now, the authors use Barlow Twins as the metric, but it would be interesting to perform ablations with e.g. MSE or Contrastive losses for this (although I must note that the authors do make an argument for this design decision in the paper).\n\n  - I think it would also be interesting to see the transferability of the trained models across different datasets. I would be interested in knowing if LFR leads to learning representations that are good for the particular dataset, or good for the chosen modality in general.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "05jKvzuYbR", "reviewer": "Reviewer_6gYm", "summary": "This paper proposed the method of self-supervised representation learning that can be used to learn useful representations from different modalities, e.g. images, text, time series, tabular data. Proposed method is based on recent approach where we have architectures with an additional projector (Guillotine regularization, removed for a downstream task) tries to predict multiple random projectors. The authors provide some analysis of hyper-parameter sensitivity, different initializations of random projectors, etc. Comparison with other methods on different modalities are proposed (time series, tabular, image).", "questions": "Q1. How the method perform without using heavy SSL (SimCLR) data augmentations?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "S1. The method is simple and very generic. Removes the prior of knowing what data aug. we should use for many current SSL contrastive-based methods.\nS2. Paper easy to follow and well-written.", "weaknesses": "W1. The main weakness for me is inability to compare with the existing method on well-known datasets in computer vision tasks. We have 3 x time series, 3 x tabular, and 1 x image - where Kvasir is not commonly used dataset. Not sure if the results are not picking the datasets that show good results. Why not presents the results on ImageNet, and if computationally not possible - cifar100. \n\nW2. There's no evaluation on the different downstream task for the learned representation, e.g. feature extractor trained with LFR on Kvasir and evaluated on the other dataset similar and disimilar one. We do not know how the learned representation generalize, and what if we only memorize patterns that then can be then useful for the lin. evaluation.\n\nW3. We don't know the final computational overhead of the initialization and comparison to any SSL method. In the appendix we have total time spent for a particular dataset (e.g. Kvasir V100 1095 GPU hours for all experiments). Would be better to know the comparison between SimCLR/SimSiam vs LFR. \n\nW4. 2 time series datasets (HAR, Epilepsy) and tabular Income&HEPMASS have already good accuracy on the randomized init. What is intresting, some methods are below that (HAR - Autoencoder). \n\nW5. Lack of more theoretical explanation why it should work? What random projectors can be used? etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ipMlcBygla", "reviewer": "Reviewer_Eryk", "summary": "This work proposes \"Learning from Randomness\" (LFR), which tackles the long-standing problem of removing augmentations from SSL. Instead, random projections of the representation are learnt via a bank of predictors, with the intuition that diverse random projections replicate a set of generic downstream tasks.\nAn objective like Barlow-Twins is used, as well as an iterative training procedure that updates the backbone and the projectors in separate steps. In order to have diverse random projections, the authors propose to sample several of them and select those that are more decorrelated.\n\nThe experiments show how the proposed method is a suitable option for time-series, tabular data and medical images datasets. Interestingly, no augmentation is used in all these settings, which I find interesting and novel. An insightful ablation study is also provided.", "questions": "I do not have a large amount of questions, since the paper is very well explained and motivated. However, I would greatly appreciate to discuss with the authors about the following:\n\n\n* The main intuition of this method is to create random downstream tasks that capture different aspects of $z$. Therefore, I would suggest adding a transfer learning experiment, in which an encoder $f_\\theta$ is learnt on a source dataset, and such features are transferred to other real downstream tasks. It would be really interesting to see a gain in such setting, wrt. other SOTA methods. I believe this would strongly reinforce the claim of the paper.\n\n* I wonder if random projections are enough to capture the diversity inherent in complex datasets. I think the paper would benefit from a discussion on the pros and cons of using random projections. Note that the datasets shown in the paper are considered small-scale in the community, which limit the understanding on how effective LFR is for _any_ dataset.   Larger, or more diverse, datasets would be extremely valuable. See following point.\n\n* I suggest evaluating LFR vs. computer vision methods in a different scenario than medical images. It is possible that SimCLR would perform better on CIFAR-10 or ImageNet, nevertheless that experiment would show how well LFR is performing against methods tailored for such scenarios. This would also help to understand how well random projections \"emulate\" the use of well-chosen augmentations. I think a natural images dataset is required for this paper for acceptance.\n  * In the Appendix, the authors provide an example of feature interpretability on CIFAR-10. Noting that the pipeline to train on natural images is already in place, I reinforce my observation that an evaluation on natural images should be part of the experimental section.\n\n* It would be great to have some discussion about how LFR could be applied to Transformers, given their wide use. Comparison with a masking approach would not be required, although of great interest.\n\n* A batch size of 256 is strongly detrimental for a contrastive approach such as SimCLR, for example. Small batch sizes harm performance for such methods. For the reader to fully grasp the scenarios where LFR is suitable, I suggest comparing with SimCLR in settings where the latter is known to excel.\n\n* I highly appreciated the breakdown of GPU hours required for this paper.\n\n-----\n\nOverall comment:\n\nThe method proposed is sound, and the goal of removing augmentations from SSL is a long-standing one. Learning from random projections seems a sensible way to tackle such problem. The overall architecture and objective is solid, I have no concerns in that sense. The manuscript is written elegantly, with the appropriate language. \n\nHowever, for the method to be rigorously evaluated, more experiments would be required. Otherwise, there is doubt about whether random projections are valuable for small datasets only, etc. \n\nNotably, I suggest the authors to:\n* Perform transfer learning experiments to support the main hypothesis of the work (random projections allow learning different aspects of $z$, thus making $z$ more generally applicable).\n* Evaluate in a setting where classic methods perform at their best, to understand the limitations of the proposed approach. I would suggest a natural images setting, using ImageNet ideally, or CIFAR-10/100 if GPU hours are a concern. \n\n* I think it would be interesting for the reader to see how the training behaves with joint training, I suggest adding some of these plots in the Appendix.\n\nI would be happy to increase my score after discussion and manuscript updates.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* The problem of removing augmentations in SSL is of great importance in the community.\n\n* The paper is very well written, organized and presented.\n\n* The authors will release code.", "weaknesses": "* The experimental section does not include any medium/large scale dataset, which obfuscates the benefit of the method for any SSL task.\n\n* I found a transfer learning experiment lacking in the results.\n\n* More discussion about the pros/cons of using random projections would be valuable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "q6rEn80KLB", "reviewer": "Reviewer_sQMY", "summary": "This work introduces a self-supervised representation learning scheme that can be applied to any data modality and network architectures. To this end, it proposed to learn the representations from random data projects of the input data. It is a scheme that learns from randomness, aiming to extract meaningful representations from randomness that mimic arbitrary downstream tasks.", "questions": "Please see Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1, this framework can accommodate various data modality\n\n2, extensive experimental results with good performance", "weaknesses": "1, Despite the good performance of the experiments, what has been learned from the latent space using the scheme introduced in this paper, even intuitively?\n\n2, It says in the paper that $g^{(k)}(x)$ uses the same architecture design of $f_{\\theta}$. Even with random initialization, it may still follow a certain distribution family. How would changing this architecture affect the learning?\n\n3, The paper only exams the accuracy on the classification task for frozen representations. Nonetheless, a good representation could used for various purposes, i.e., manipulation of each dimension in the latent space for generating new data, understanding the essential dynamics of the system in physical models and time series data. How could this strategy be applied to scenarios beyond classification?\n\n4, Although the fact that choosing good augmentations usually requires domain knowledge, in many of the application mentioned, such knowledge could be partially obtained using physical intuition. By comparing with models under random corruptions doesn't seem to be a fair comparison.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)