URLOST: Unsupervised Representation Learning without Stationarity or Topology

## Abstract

Abstract Unsupervised representation learning has seen tremendous progress but is constrained by its reliance on data modality-specific stationarity and topology, a limitation not found in biological intelligence systems. For instance, human vision processes visual signals derived from irregular and non-stationary sampling lattices yet accurately perceives the geometry of the world. We introduce a novel framework that learns from high-dimensional data lacking stationarity and topology. Our model combines a learnable self-organizing layer, density adjusted spectral clustering, and masked autoencoders. We evaluate its effectiveness on simulated biological vision data, neural recordings from the primary visual cortex, and gene expression datasets. Compared to state-of-the-art unsupervised learning methods like SimCLR and MAE, our model excels at learning meaningful representations across diverse modalities without depending on stationarity or topology. It also outperforms other methods not dependent on these factors, setting a new benchmark in the field. This work represents a step toward unsupervised learning methods that can generalize across diverse high-dimensional data modalities.

## 1 Introduction

Unsupervised representation learning, also known as self-supervised representation learning (SSL), aims to develop models that autonomously detect patterns in data and make these patterns readily apparent through a specific representation. There has been tremendous progress over the past few years in the unsupervised representation learning community. Popular methods like contrastive learning and masked autoencoders [; ; ; ] work relatively well on typical modalities such as images, videos, audio, time series, and point clouds. However, these methods make implicit assumptions about the data domain’s topology and stationarity.
Given an image, topology refers to the neighboring pixels of each pixel, or more generally, the grid structure in images, the temporal structure in time series and sequences, or the 3D structure in molecules and point clouds. Stationarity refers to the property that the low-level statistics of the signal remain consistent across its domain. For instance, pixels and patches in images exhibit similar low-level statistics (mean, variance, co-variance) regardless of their locations within the domain. The success of state-of-the-art self-supervised representation learning relies on knowing the prior topology and stationarity of the modalities. For example, joint-embedding SSL employs random-resized cropping augmentation [], and masked auto-encoding [] utilizes masked-image-patch augmentation. What if we possess high-dimensional signals without knowledge of their domain topology or stationarity? Can we still craft a high-quality representation? This is not only the situation that biological vision systems have to deal with but also a practical setting for many scientific data analysis problems. In this work, we introduce unsupervised representation learning without stationarity or topology (URLOST) and take a step in this direction.

As we mentioned earlier, typical modalities possess topology and stationarity prior information that can be utilized by unsupervised representation learning. Taking images as an example, digital cameras employ a consistent sensor grid that spans the entire visual field. However, biological visual systems have to deal with signals with less domain regularity. For instance, unlike camera sensors which have a uniform grid, the cones and rods in the retina distribute unevenly and non-uniformly. This results in a non-stationary raw signal.
Retinal ganglion cells connect to more photoreceptors in the fovea than in the periphery. The correlation of the visual signal between two different locations in the retina depends not only on the displacement between these locations but also on their absolute positions. Yet, biological visual systems can establish precise retinotopy from the retina to neurons based on spontaneous retinal activities and external stimuli [; ; ] and leverage retinotopic input to build unsupervised representation. This implies that we can potentially build unsupervised representation without relying on prior stationarity of the raw signal or topology of the input domain.

Figure: Figure 1: From left to right: the unsupervised representation learning through joint embedding and masked auto-encoding; the biological vision system that perceives via unstructured sensor and understands signal without stationarity or topology []; and many more such diverse high dimensional signal in natural science that our method supports while most existing unsupervised methods don’t. Data figures are borrowed from [; ; ].
Refer to caption: x1.png

In this work, we aim to build unsupervised representations for general high-dimensional vectors. Taking images as an example again, let’s assume we receive a set of images whose pixels are shuffled in the same order. How can we build representations in an unsupervised fashion without knowledge of the shuffling order? If possible, can we use such a method to build unsupervised representations for general high-dimensional data? Inspired by [], we use low-level statistics and spectral clustering to form clusters of the pixels, which recovers a coarse topology of the input domain. These clusters are analogous to image patches except that they are slightly irregularly shaped and different in size.
We mask a proportion of these “patches” and utilize a Vision Transformer [] to predict the masked “patches” based on the remaining unmasked ones. This “learning to predict masked tokens” approach is proposed in masked autoencoders (MAE) [] and has demonstrated effectiveness on typical modalities. Initially, we test the proposed method on the synthesized biological visual dataset, derived from CIFAR-10 [] using a foveated retinal sampling mechanism []. Then we generalize this method to two high-dimensional vector datasets: a primary visual cortex neural response decoding dataset [] and the TCGA miRNA-based cancer classification dataset [; ]. Across all these benchmarks, our proposed method outperforms existing SSL techniques, establishing its effectiveness in building unsupervised representations for signals lacking explicit stationarity or topology. Given the emergence of new modalities in deep learning from natural sciences [; ; ; ], such as chemistry, biology, and neuroscience, our method offers a promising approach in the effort to build unsupervised representations for high-dimensional data.

## 2 Method

### 2.1 Motivation and Overall framework

Our objective is to build robust, unsupervised representations for high-dimensional signals that lack explicit topology and stationarity. These learned representations are intended to enhance performance in downstream tasks, such as classification. To achieve this, we begin by using low-level statistics and clustering to approximate the signal’s topology. The clusters derived from the signal serve as input to a masked autoencoder. As depicted in Figure [1](#S1.F1), the masked autoencoder randomly masks out patches in an image and trains a Transformer-based autoencoder unsupervisedly to reconstruct the original image. After the unsupervised training, the autoencoder’s latent state yields high-quality representations. In our approach, signal clusters are input to the masked autoencoder.

Notably, the clusters differ from image patches in several key aspects due to the differences in the input signal: they are unaligned, exhibit varied sizes and shapes, and their clustering nodes are not confined to fixed 2D locations like pixels in image patches. To cope with these differences, we introduce a self-organizing layer responsible for aligning these clusters through learnable transformations. The parameters of this layer are jointly optimized with those of the masked autoencoder.
Our method is termed URLOST, an acronym for Unsupervised Representation Learning withOut Stationarity or Topology. Figure [2](#S2.F2) provides an overview of the framework. URLOST consists of three core components: density adjusted spectral clustering, self-organizing layer, and masked autoencoder. The functionalities of these components are detailed in the following subsections.

Figure: Figure 2: The overview framework of URLOST. The high-dimensional input signal undergoes clustering and self-organization before unsupervised learning using a masked autoencoder for signal reconstruction.
Refer to caption: x2.png

### 2.2 Density Adjusted Spectral clustering

Representation learning for high-dimensional signals without explicit topology is challenging. We propose to define a metric to measure inter-dimensional relationships. This metric effectively approximates a topology for the signal. Similar to [], where they use the absolute correlation values as the metric for pixels, we employ discrete mutual information (refer to Appendix [A.1](#A1.SS1)) as the metric.
Let affinity matrix $A_{ij}$ denote the mutual information between dimension $i$ and $j$, which approximates the manifold $\mathcal{M}$ that the signal lives on. We can define the discretized Laplacian operator based on $A$ and use the eigenvector of the Laplacian operator to perform spectral clustering, which segments the manifold. The detailed definition and the algorithm are left in Appendix [A.1](#A1.SS1). Finding the eigenvector of the Laplacian operator is a discretized approximation of the following optimization problem in function space:

$$ $\min_{||f||_{L^{2}(\mathcal{M})}}\int_{\mathcal{M}}||p(x)^{\frac{1}{2}}\nabla f (x)||^{2}$ (1) $$

where $f(x):\mathcal{M}\to[0,1]$ is the normalized signal defined on $\mathcal{M}$ and $p(x)$ is the density function. The integral is taken over standard measure on $\mathcal{M}$. Since spectral clustering heavily relies on the solution of equation [1](#S2.E1), the definition of the density function $p(x)$ affects the quality of the resulting clusters. Standard approaches often assume that nodes are uniformly distributed on the manifold, thereby treating $p(x)$ as a constant and excluding it from the optimization process. However, this assumption does not hold in our case involving non-stationary signals. To cope with non-stationary signals, our work introduces a variable density function $p(x)$ for each signal, making it a pivotal component in building good representations for the signal. This component is referred to as Density Adjusted Spectral Clustering. Empirical evidence supporting this design is provided through visualization and ablation studies in the experimental section.

### 2.3 Self-organizing layer

Transforming a high-dimensional signal into a sequence of clusters using the above method is not enough because it does not capture the internal structure within individual clusters.
To effectively perform unsupervised learning on these clusters, it is essential to align them in some manner.
Directly solving the exact alignment problem with low-level statistics of the signal is challenging. Thus, we propose a self-organizing layer with learnable parameters. Specifically, let vector $x^{(i)}$ denote the $i$th cluster. Each cluster $x^{(i)}$ is passed through a differentiable function $g(\cdot,w^{(i)})$ with parameter $w^{(i)}$, resulting in a sequence $z_{0}$:

$$ $z_{0}=[g(x^{(1)},w^{(1)}),\cdots g(x^{(M)},w^{(M)})]$ (2) $$

$z_{0}$ is comprised of projected and aligned representations for all clusters.
The weights of the proposed self-organizing layer, $\{w^{(1)},\cdots w^{(M)}\}$, are jointly optimized with the subsequent neural network introduced in the next subsection.

### 2.4 Masked autoencoder

After the self-organizing layer, $z_{0}$ is passed to a Transformer-based masked autoencoder (MAE) with an unsupervised learning objective. Masked autoencoder (MAE) consists of an encoder and a decoder which both consist of stacked Transformer blocks introduced in []. The objective function is introduced in []: masking random image patches in an image and training an autoencoder to reconstruct them, as illustrated in Figure [1](#S1.F1). In our case, randomly selected clusters in $z_{0}$ are masked out, and the autoencoder is trained to reconstruct these masked clusters. After training, the encoder’s output is treated as the learned representation of the input signal for downstream tasks. The masked prediction loss is computed as the mean square error (MSE) between the values of the masked clusters and their corresponding predictions.

## 3 Result

Since our method is inspired by the biological vision system, we first validate its ability on a synthetic biological vision dataset created from CIFAR-10. Then we evaluate the generalizability of URLOST on two high-dimensional natural datasets collected from diverse domains. Detailed information about each dataset and the corresponding experiments is presented in the following subsections. Across all tasks, URLOST consistently outperforms other strong unsupervised representation learning methods.

### 3.1 Synthetic biological vision dataset

As discussed in the introduction, the biological visual signal serves as an ideal dataset to validate the capability of URLOST. In contrast to digital images captured by a fixed array of sensors, the biological visual signal is acquired through irregularly positioned ganglion cells, inherently lacking explicit topology and stationarity. However, it is hard to collect real-world biological vision signals with high precision. Therefore, we employ a retinal sampling technique to modify the classic CIFAR-10 dataset and simulate imaging from the biological vision signal. The synthetic dataset is referred to as Foveated CIFAR-10. To make a comprehensive comparison, we also conduct experiments on the original CIFAR-10, and a Permuted CIFAR-10 dataset obtained by randomly permuting the image.

Permuted CIFAR-10. To remove the grid topology inherent in digital imaging, we simply permute all the pixels within the image, which effectively discards any information related to the grid structure of the original digital image. We applied such permutation to each image in the CIFAR-10 dataset to generate the Permuted CIFAR-10 dataset. Nevertheless, permuting pixels only removes an image’s topology, leaving its stationarity intact. To obtain the synthetic biological vision that has neither topology nor stationarity, we introduce the Foveated CIFAR-10.

Figure: Figure 3: Retina sampling (A) An image in CIFAR-10 dataset. (B) Retina sampling lattice. Each blue dot represents the center of a Gaussian kernel, which mimics a retinal ganglion cell. (C) Visualization of the car image’s signal sampled using the retina lattice. Each kernel’s sampled RGB value is displayed at its respective lattice location for visualization purposes. (D) density-adjusted spectral clustering results are shown. Each unique color represents a cluster, with each kernel colored according to its assigned cluster.
Refer to caption: extracted/5155611/figures/sampling_process.png

Foveated CIFAR-10.
Much like photosensors installed in a camera, retina ganglion cells within the primate biological visual system sample from visual stimuli and project images. However, unlike photosensors that have uniform receptive fields and adhere to a consistent sampling pattern, retinal ganglion cells at different locations of the retina vary in their receptive field size: smaller in the center (fovea) but larger in the peripheral of the retina. This distinctive retina sampling pattern results in foveated imaging []. It gives primates the ability to have both a high-resolution vision and a broad overall receptive field while consequently making visual signals sampled by the retina lack stationarity. The evidence is that responses of two ganglion cells separated by the same displacement are highly correlated in the retina but less correlated in the peripheral.
To mimic the foveated imaging with CIFAR-10, we adopt the retina sampling mechanism from []. Specifically, each retina ganglion cell is simplified and modeled using a Gaussian kernel. The response of each cell is determined by the dot product between pixel values and the Gaussian kernel.
Figure [3](#S3.F3) illustrates the sampling kernel locations. Applying this sampling grid and permuting the resulting pixels produces the foveated CIFAR-10.
In the natural retina, retinal ganglion cell density decreases linearly with eccentricity, which makes fovea much denser than the peripheral, compared to the simulated lattice in Figure [3](#S3.F3). However, considering the low resolution of the CIFAR-10 dataset, we reduce the simulated fovea’s density to prevent redundant sampling.

Experiments. We compare URLOST on both of the synthetic vision datasets as well as the original CIFAR-10 with popular unsupervised representation learning methods SimCLR [] and MAE []. All the models conducted unsupervised learning followed by linear probing for classification accuracy.
The evaluations are reported in Table [1](#S3.T1). SimCLR excels on CIFAR-10 but struggles badly with both synthetic datasets due to its inability to handle data without stationarity and topology. MAE gets close to SimCLR on CIFAR-10 with a $4\times 4$ patch size. However, the patch size no longer makes sense when data has no topology. So we additionally tested MAE masking pixels instead of image patches. It maintains the same performance on Permuted CIFAR-10 as on CIFAR-10, though poorly, invariant to the removal of topology as it should be. But It still drops greatly to $48.5\%$ on the Foveated CIFAR-10 when stationarity is also removed. In contrast, only URLOST is able to maintain consistently strong performances when there is no topology or stationarity, achieving $86.4\%$ on Permuted CIFAR-10 and $85.4\%$ on Foveated CIFAR-10 when the baselines completely fail.

**Table 1: Evaluation on computer vision and synthetic biological vision dataset. ViT (Patch) stands for the Vision Transformer backbone with image patches as inputs. ViT (Pixel) means pixels are treated as input units. ViT (Clusters) means clusters are treated as inputs instead of patches. The number of clusters is set to 64 for both Permuted CIFAR-10 and Foveated CIFAR-10 dataset.**
| Dataset | Method | Backbone | Eval Acc |
| --- | --- | --- | --- |
| CIFAR-10 | MAE | ViT (Patch) | 88.3 % |
|  | MAE | ViT (Pixel) | 56.7 % |
|  | SimCLR | ResNet-18 | 90.7 % |
| Permuted CIFAR-10 | URLOST MAE | ViT (Cluster) | 86.4 % |
| (no topology) | MAE | ViT (Pixel) | 56.7 % |
|  | SimCLR | ResNet-18 | 47.9 % |
| Foveated CIFAR-10 | URLOST MAE | ViT (Cluster) | 85.4 % |
| (no topology or stationarity) | MAE | ViT (Pixel) | 48.5 % |
|  | SimCLR | ResNet-18 | 38.0 % |

**Table 2: Evaluation on V1 response decoding and TCGA pan-cancer classification tasks. “Raw” indicates preprocessed (standardized and normalized) raw signals. Best $\beta$ values are used for $\beta$-VAE. For URLOST MAE, cluster sizes are 200 (V1) and 32 (TCGA).**
| Method | V1 Response Decoding Acc | TCGA Classification Acc |
| --- | --- | --- |
| Raw | 73.9% | 91.1% |
| $\beta$-VAE | 75.1% | 94.2% |
| MAE | 64.8% | 88.3% |
| URLOST MAE | 78.2% | 94.9% |

### 3.2 V1 neural response to natural image stimulus

After accessing URLOST’s performance on synthetic biological vision data, we take a step further to challenge its generalizability with high-dimensional natural datasets. The first task is decoding neural response recording in the primary visual area (V1) of mice.

V1 neural response dataset. The dataset, published by [], contains responses from over 10,000 V1 neurons captured via two-photon calcium imaging. These neurons responded to 2,800 unique images from ImageNet [], with each image presented twice to assess the consistency of the neural response. In the decoding task, a prediction is considered accurate if the neural response to a given stimulus in the first presentation closely matches the response to the same stimulus in the second presentation within the representation space.
This task presents greater challenges than the synthetic biological vision described in the prior section. For one, the data comes from real-world neural recordings rather than a curated dataset like CIFAR-10. For another, the geometric structure of the V1 area is substantially more intricate than that of the retina. To date, no precise mathematical model of the V1 neural response has been well established. The inherent topology and stationarity of the data still remain difficult to grasp [; ]. Nevertheless, evidence of Retinotopy  [; ] and findings from prior research [; ; ] suggest that the neuron population code in V1 are tiling a low dimensional manifold. This insight led us to treat the population neuron response as high-dimensional data and explore whether URLOST can effectively learn its representation.

Experiments. Following the approach in [] we apply standardization and normalization to the neural firing rate. The processed signals are high-dimensional vectors, and they can be directly used for the decoding task, which serves as the “raw” signal baseline in Table [2](#S3.T2). For representation learning methods,
URLOST is evaluated along with MAE and $\beta$-VAE []. Note that the baseline methods need to handle high-dimensional vector data without stationarity or topology, so SimCLR is no longer applicable. We use $\beta$-VAE instead. We first train the neural network with an unsupervised learning task, then use the latent state of the network as the representation for the neural responses in the decoding task. The results are presented in the table [2](#S3.T2). Our method surpasses the original neuron response and other methods, achieving the best performance.

### 3.3 Gene expression data

In this subsection, we further evaluate URLOST on high-dimensional natural science data from a completely different domain, the gene expression data.

Gene expression dataset.
The dataset comes from The Cancer Genome Atlas (TCGA) [; ], which is a project that catalogs the genetic mutations responsible for cancer using genome sequencing and bioinformatics. The project molecularly characterized over 20,000 primary cancers and matched normal samples spanning 33 cancer types. We focus on the pan-cancer classification task: diagnose and classify the type of cancer for a given patient based on his gene expression profile. The TCGA project collects the data of 11,000 patients and uses Micro-RNA (miRNA) as their gene expression profiles.
Like the V1 response, no explicit topology and stationarity are known and each data point is a high-dimensional vector.
Specifically, 1773 miRNA identifiers are used so that each data point is a 1773-dimensional vector. Types of cancer that each patient is diagnosed with serve as the classification labels.

Experiments. Similar to Section [3.2](#S3.SS2), URLOST is compared with the original signals, MAE, and $\beta$-VAE, which is the state-of-the-art unsupervised learning method on TCGA cancer classification [; ]. We also randomly partition the dataset do five-fold cross-validation and report the average performance in Table [2](#S3.T2). Again, our method learns meaningful representation from the original signal. The learned representation benefited the classification task and achieved the best performance, demonstrating URLOST’s ability to learn meaningful representation of data from diverse domains.

## 4 Ablation study

**Table 3: Ablation study on self-organizing layer. Linear probing accuracy with varying parameters, keeping others constant. For Locally-Permutated CIFAR-10, we use $4\times 4$ patch size. For Permutated CIFAR-10 and Foveated CIFAR-10, we set the number of clusters to 64 for the spectral clustering algorithm. We kept the hyperparameter of the backbone model the same as in table [1](#S3.T1).**
| Dataset | Projection | Eval Acc |
| --- | --- | --- |
| Locally-Permuted | shared | 81.4 % |
| CIFAR-10 | non-shared | 87.6 % |
| Permuted | shared | 80.7 % |
| CIFAR-10 | non-shared | 86.4 % |

### 4.1 Self-organizing Layer vs Shared Projection Layer

Conventional SSL models take a sequential input $x=[x^{(1)},\cdots x^{(M)}]$ and embed them into latent vectors with a linear transformation:

$$ $z_{0}=[Ex^{(1)},\cdots Ex^{(M)}]$ (3) $$

which is further processed by a neural network. The sequential inputs can be a list of language tokens [; ], pixel values [], image patches [], or overlapped image patches [; ; ]. $E$ can be considered as a projection layer that is shared among all elements in the input sequence.
The self-organizing layer $g(\cdot,w^{(i)})$ introduced in Section [2.3](#S2.SS3) can be considered as a non-shared projection layer.
We conducted an ablation study comparing the two designs to demonstrate the effectiveness of the self-organizing layers both quantitatively and qualitatively. To facilitate the ablation, we further synthesized another dataset.

Locally-permuted CIFAR-10.
To directly evaluate the performance of the non-shared projection approach, we designed an experiment involving intentionally misaligned clusters. In this experiment, we divide each image into patches and locally permute all the patches. The $i$-th image patch is denoted by $x^{(i)}$, and its permuted version, permutated by the permutation matrix $E^{(i)}$, is expressed as $E^{(i)}x^{(i)}$. We refer to this manipulated dataset as the Locally-Permuted CIFAR-10.
Our hypothesis posits that models using shared projections, as defined in Equation [3](#S4.E3), will struggle to adapt to random permutations, whereas self-organizing layers equipped with non-shared projections can autonomously adapt to each patch’s permutation, resulting in robust performance. This hypothesis is evaluated quantitatively and through the visualization of learned weights $w^{(i)}$.

Permuted CIFAR-10. Meanwhile, we also run the ablation study on the Permuted CIFAR-10. Unlike locally permuted CIFAR-10, a visualization check is not viable since the permutation is done globally. However, we can still quantitatively measure the performance of the task.

Quantitative results. Table [3](#S4.T3) confirms our hypothesis, demonstrating a significant performance decline in models employing shared projections when exposed to permuted data. In contrast, the non-shared projection model maintains stable performance.

Visual evidence. Using linear layers to parameterize the self-organizing layers, i.e. let $g(x,W^{(i)})=W^{(i)}x$, we expect that if the projection layer effectively aligns the input sequence, $E^{(i)T}W^{(i)}$ should exhibit visual similarities. That is, after applying the inverse permutation $E^{(i)T}$, the learned projection matrix $W^{(i)}$ at each location should appear consistent or similar. The proof of this statement is provided in Appendix [A.4](#A1.SS4). The model trained on Locally-Permuted CIFAR10 provides visual evidence supporting this claim. In Figure [4](#S4.F4), the weights show similar patterns after reversing the permutations.

Figure: Figure 4: Learnt weights of a self-organizing layer. (A) Image is cropped into patches, where each patch $x^{(i)}$ first undergoes a different permutation $E^{(i)}$, then the inverse permutation $E^{(i)T}$. (B) The learned weight of the linear self-organizing layer. The $12$th column of $W^{(i)}$ at all positions $i$ are reshaped into patches and visualized. When $W^{(i)}$ undergoes the inverse permutation $E^{(i)T}$, they show similar patterns. (C) Visualization of the $37$th column of $W^{(i)}$. Similar to (B).
Refer to caption: extracted/5155611/figures/filter.png

### 4.2 Density adjusted clustering vs Uniform Density Clustering

As explained in Section [2.2](#S2.SS2), the shape and size of each cluster depend on how the density function is defined. Let $q(i)$ represent the eccentricity, the distance from $i$th kernel to the center of the sampling lattice, and let $n(i)=\sum_{j}A_{ji}$ where $A$ is the affinity matrix, then the density is defined as:

$$ $p(i)=q(i)^{-\alpha}n(i)^{\beta}.$ (4) $$

By setting $\alpha$ and $\beta$ nonzero, the density function is eccentricity-dependent. Setting both $\alpha$ and $\beta$ to zero will make $n(i)$ constant which recovers the uniform density spectral clustering. We vary the parameters $\alpha$ and $\beta$ to generate different sets of clusters for the foveated CIFAR-10 dataset and run URLOST using each of these sets of clusters. Results in Table [4](#A1.T4) validate that the model performs better with density adjusted clustering. The intuitive explanation is that by adjusting the values of $\alpha$ and $\beta$, we can make each cluster carry similar amounts of information (refer to Appendix [A.3](#A1.SS3).). A balanced distribution of information across clusters enhances the model’s ability to learn meaningful representations. Without this balance, masking a low-information cluster makes the prediction task trivial, while masking a high-information cluster will make the prediction task too difficult. In either scenario, the model’s ability to learn effective representations is compromised.

## 5 Additional related works

Several interconnected pursuits are linked to this work, and we will briefly address them here:

Topology in biological visual signal.
2-D topology of natural images is strong prior that requires many bits to encode [; ]. Such 2-D topology is encoded in the natural image statistic [; ], which can be recovered []. Optic and neural circuits in the retina result in a more irregular 2-D topology than the natural image, which can still be simulated [; ; ; ; ; ]. This information is further processed by the primary visual cortex. Evidence of retinotopy suggests the low-dimensional geometry of visual input from retina is encoded by the neuron in primary visual cortex [; ; ; ; ; ]. These evidences suggest we can recover the topology using signal from retinal ganglion cell and V1 neurons.

Evidence of self-organizing mechanism in the brain.
In computational neuroscience, many works use the self-organizing maps (SOM) as a computational model for V1 functional organization: [; ; ; ; ; ]. In other words, this idea of self-organizing is likely a principle governing how the brain performs computations. Even though V1 functional organizations are present at birth, numerous studies also indicate that the brain’s self-organizing mechanisms continue after full development [; ; ].

Learning with signal on non-euclidean geometry.
In recent years, researchers from the machine learning community have made efforts to consider geometries and special structures beyond classic images, text, and feature vectors. [] treats an image as a set of points but depends on the 2D coordinates. The geometric deep learning community tries to generalize convolution neural networks beyond the Euclidean domain [; ; ; ]. Recent research also explores adapting the Transformer to domains beyond Euclidean spaces [; ]. However, none of them has tried to tackle the issue when the data has no explicit topology or stationarity, which is the focus of URLOST.

Self-supervised learning.
Self-supervised learning (SSL) has made substantial progress in recent years. Different SSL method is designed for each modality, for example: predicting the masked/next token in NLP[; ; ], solving pre-text tasks, predicting masked patches, or building contrastive image pairs in computer vision [; ; ; ; ; ]. These SSL methods have demonstrated descent scalability with a vast amount of unlabeled data and have shown their power by achieving performance on par with or even surpassing supervised methods. They have also exhibited huge potential in cross-modal learning, such as the CLIP by []. However, we argue that these SSL methods are all built upon specific modalities with explicit topology and stationarity which URLOST goes beyond.

## 6 Discussion

The success of most current state-of-the-art self-supervised representation learning methods relies on the assumption that the data has known stationarity and domain topology, such as the grid-like RGB images and time sequences. However, biological vision systems have evolved to deal with signals with less regularity. In this work, we explore unsupervised representation learning under a more general assumption, where the stationarity and topology of the data are unknown to the machine learning model and its designers. We argue that this is a general and realistic assumption for high-dimensional data in modalities of natural science. We propose a novel unsupervised representation learning method that works under this assumption and demonstrates our method’s effectiveness and generality on a synthetic biological vision dataset and two datasets from natural science that have diverse modalities.
We also perform a step-by-step ablation study to show the effectiveness of the novel components in our model.

During experiments, we found that density adjusted spectral clustering is crucial for the quality of representation learning. How to adjust the density and obtain a balanced clustering for any given data or even learning the clusters end-to-end with the representation via back-propagation is worth future investigation. Moreover, our current self-organizing layer is still simple though it shows effective performance. Extending it to a more sophisticated design and potentially incorporating it with various neural network architectures is also worth future exploration.

In summary, our method offers a handy and general unsupervised learning tool when dealing with high-dimensional data of arbitrary modality with unknown stationarity and topology, particularly common in the field of natural sciences, where many present strong unsupervised learning baselines cannot directly adapt. We hope it can provide inspiration for work in related fields.

## Appendix A Appendix

### A.1 Spectral clustering algorithm

Given a high dimensional dataset $S\in\mathbb{R}^{n\times m}$, Let $S_{i}$ be $i$th column of $S$, which represents the $i$th dimension of the signal. We create probability mass functions $P(S_{i})$ and $P(S_{j})$ and the joint distribution $P(S_{i},S_{j})$ for $S_{i}$ and $S_{j}$ using histogram. Let the number of bins be $K$. Then we measure the mutual information between $P(S_{i})$ and $P(S_{j})$ as:

$$ $I(S_{i};S_{j})=\sum_{l=1}^{K}\sum_{k=1}^{K}P(S_{i},S_{j})[l,k]\log_{2}\left( \frac{P(S_{i},S_{j})[l,k]}{P(S_{i})[l]P(S_{j})[k])}\right)$ $$

Let $A_{ij}=I(X_{i};X_{j})$ be the affinity matrix, $p(i)$ be the density function defined in [4](#S4.E4). We follow the steps from [] to perform spectral clustering with a modification to adjust the density:

- 1.
Define $D$ to be the diagonal matrix whose ($i$,$i$)-element is the sum of $A$’s $i$-th row, $P$ be the identity matrix where $P_{ii}=p(i)$. Construct the matrix $L=P^{\frac{1}{2}}D^{-\frac{1}{2}}AD^{-\frac{1}{2}}P^{\frac{1}{2}}.$
- 2.
Find $x_{1},x_{2},\cdots,x_{k}$, the $k$ largest eigenvectors of $L$, and form the matrix $X=[x_{1},x_{2},\cdots,x_{k}]\in\mathbb{R}^{n\times k}$ by stacking the eigenvectors in columns.
- 3.
Form the matrix Y from X by renormalizing each of $X$’s rows to have unit norms. (i.e. $Y_{ij}=X_{ij}/(\sum_{i}X_{ij}^{2})^{\frac{1}{2}}$)
- 4.
Treating each row of $Y$ as a point in $\mathbb{R}^{k}$, cluster them into $k$ clusters via K-means or other algorithms.

Some other interpretation of spectral embedding allows one to design a specific clustering algorithm in step 4. For example, [] interprets the eigenvector problem in [1](#S2.E1) as a relaxed continuous version of K-way normalized cuts problem, where they only allow $X$ to be binary, i.e. $X\in\{0,1\}^{N\times K}$. This is an NP-hard problem. Allowing $X$ to take on real value relaxed this problem but created a degeneracy solution. Given a solution $X^{*}$ and $Z=D^{-\frac{1}{2}}X^{*}$, for any orthonormal matrix $R$, $RZ$ is another solution to the optimization problem [1](#S2.E1). Thus, [] designed an algorithm to find the optimal orthonormal matrix $R$ that converts $X^{*}$ to discrete value in $\{0,1\}^{N\times K}$. From our experiment, [] is more consistent than K-means and other clustering algorithms, so we stick to using it for our model.

### A.2 Data synthesize process

We followed the retina sampling approach described in [] to achieve foveated imaging. Specifically, each retina ganglion cell is represented using a Gaussian kernel. The kernel is parameterized by its center, denoted as $\vec{x}_{i}$, and its scalar variance, $\sigma_{i}^{\prime 2}$, i.e. $\mathcal{N}(\vec{x}_{i},\sigma_{i}^{\prime 2}\mathbf{I})$, which is illustrated in Figure [5](#A1.F5).A. The response of each cell, denoted as $G[i]$, is computed by the dot product between the pixel value and the corresponding discrete Gaussian kernel. This can be formulated as:

$$ $G[i]=\sum_{n}^{N}\sum_{m}^{W}K(\vec{x}_{i},\sigma_{i}^{\prime})[n,m]I[n,m]$ $$

where $N$ and $W$ are dimensions of the image, and $I$ represents the image pixels.

For foveated CIFAR-10, since the image is very low resolution, we first upsample it 3 times from $32\times 32$ to $96\times 96$, then use in total of 1038 Gaussian kernels to sample from the upsampled image. The location of each kernel is illustrated in Figure [5](#A1.F5).B. The radius of the kernel scales proportionally to the eccentricity. Here, we use the distance from the kernel to the center to represent eccentricity. The relationship between the radius of the kernel and eccentricity is shown in Figure [5](#A1.F5).C. As mentioned in the main paper, in the natural retina, retinal ganglion cell density decreases linearly with eccentricity, which makes the fovea much denser than the peripheral, unlike the simulated lattice we created. The size of the kernel should scale linearly with respect to eccentricity as well. However, for the low-resolution CIFAR-10 dataset, we reduce the simulated fovea’s density to prevent redundant sampling. In this case, we pick the exponential scale for the relationship between the size of the kernel and eccentricity so the kernel visually covers the whole visual field. We also implemented a convolution version of the Gaussian sampling kernel to speed up data loading.

Figure: Figure 5: Foveated retinal sampling (A) Illustration of a Guassian kernel shown in []. Diagram of single kernel filter parameterized by a mean $\mu^{\prime}$ and variance $\sigma^{\prime}$. (B) the location of each Gaussian kernel is summarized as a point with 2D coordinate $\mu^{\prime}$. In total, the locations of 1038 Gaussian kernels are plotted. (C) The relationship between eccentricity (distance of the kernel to the center) and radius of the kernel is shown.
Refer to caption: extracted/5155611/figures/foveated_sampling.png

**Table 4: Evaluation on foveated CIFAR-10 with varying hyperparameter for density function. For each set of values of $\alpha$ and $\beta$, we perform density adjusted spectral clustering and run URLOST with the corresponding cluster. The evaluation of each trial is provided in the table.**
|  | beta = 0 | beta = 2 |
| --- | --- | --- |
| alpha = 0 | 82.74 % | 84.24 % |
| alpha = 0.5 | 84.52 % | 85.43 % |
| alpha = 1.0 | 83.83 % | 81.62 % |

### A.3 Density adjusted spectral clustering on foveated CIFAR10 dataset

We provide further intuition and visualization on why density adjusted spectral clustering allows the model to learn a better representation on the foveated CIFAR-10 dataset.

As shown in Figure [5](#A1.F5), the kernel at the center is much smaller in size than the kernel in the peripheral. This makes the kernel at the center more accurate but smaller, which means it summarizes less information. Spectral clustering with constant density will make each cluster have a similar number of elements in them. Since the kernel in the center is smaller, the cluster in the center will be visually smaller, than the cluster in the peripheral. The effect is shown in Figure [6](#A1.F6). Moreover, since we’re upsampling an already low-resolution image (CIFAR-10 image), even though the kernel at the center is more accurate, we’re not getting more information. There, to make sure each cluster has similar information, the clusters in the center need to have more elements than the clusters in the peripheral. In order to make the clusters at the center have more elements, we need to weight the clusters in the center more with the density function. Since the sampling kernels at the center have small eccentricity and are more correlated to their neighbor, increasing $\alpha$ and $\beta$ will make sampling kernels at the center have higher density, which makes the cluster at the center larger. This is why URLOST with density adjusted spectral clustering performs better than URLOST with constant density spectral clustering, which is shown in Table [4](#A1.T4). Meanwhile, setting $\alpha$ and $\beta$ too large will also hurt the model’s performance because it creates clusters that are too unbalanced.

Figure: Figure 6: Effect of density adjusted clustering. Eccentricity-based sampling lattice. The center of the sampling lattice has more pixels which means higher resolution compared to the peripheral. (A) Result of density adjusted spectral clustering ($\alpha=0.5,\beta=2$). Clusters in the center have more elements than clusters in the peripheral. But clusters look more visually similar in size than B. (B) Result of uniform density spectral clustering ($\alpha=0,\beta=0$). Each cluster has a similar number of elements in them but the clusters in the center are much smaller than the clusters in the periphery.
Refer to caption: extracted/5155611/figures/density_varying.png

### A.4 self-organizing layer learns inverse permutation

For locally-permuted CIFAR-10, we divide each image into patches and locally permute all the patches. The $i$-th image patch is denoted by $x^{(i)}$, and its permuted version, permuted by the permutation matrix $E^{(i)}$, is expressed as $E^{(i)}x^{(i)}$.
We use linear layers to parameterize the self-organizing layers. Let $g(x,W^{(i)})=W^{(i)}x$ denotes the $i$th element of the self-organizing layer. We’re providing the proof for the statement related to the visual evidence shown in Section [4.1](#S4.SS1)

Statement: If the self-organizing layer effectively aligns the input sequence, then $E^{(i)T}w^{(i)}$ should exhibit visual similarities.

Proof: we first need to formally define what it means for the self-organizing layer to effectively align the input sequence. Let ${\bf e}_{k}$ denote the $k$th natural basis (one-hot vector at position $k$), which represents the pixel basis at location $k$. Permutation matrix $E^{(i)}$ will send $k$th pixel to some location accordingly. Mathematically, if the projection layer effectively aligns the input sequence, it means $g(E^{(j)}e_{k},W^{(j)})=g(E^{(i)}e_{k},W^{(i)})$ for all $i,j,k$. We can further expand this property to get the following two equations:

$$ $\displaystyle g(E^{(i)}e_{k},W^{(i)})$ $\displaystyle=W^{(i)}E^{(i)}e_{k}$ $\displaystyle g(E^{(j)}e_{k},W^{(j)})$ $\displaystyle=W^{(j)}E^{(j)}e_{k}$ $$

for all $i,j,k$. Since the above equation holds for all $e_{k}$, by linearity and the property of permutation matrix, we have:

$$ $\displaystyle W^{(i)}E^{(i)}$ $\displaystyle=W^{(j)}E^{(j)}$ $\displaystyle E^{(i)T}W^{(i)}$ $\displaystyle=E^{(j)T}W^{(j)}$ $$

This implies $E^{(i)T}w^{(i)}$ should exhibit visual similarities for all $i$.

### A.5 Visualizing the weight of self-organizing

As explained in the previous section (Appendix [A.4](#A1.SS4)) and visualized in Figure [7](#A1.F7), we can visualize the weights of the learned self-organizing layer when trained on the locally-permuted CIFAR-10 dataset. If we apply the corresponding inverse permutation $E^{(i)T}$ to its learned filter $W^{(i)}$ at position $i$, the pattern should show similarity across all position $i$. This is because the model is trying to align all the input clusters. We have shown this is the case when the model converges to a good representation. On the other hand, what if we visualize the weight $E^{(i)T}W^{(i)}$ as training goes on? If the model learns to align the clusters as it is trained for the mask prediction task, $E^{(i)T}W^{(i)}$ should become more and more consistent as training goes on. We show this visualization in Figure [7](#A1.F7), which confirms our hypothesis. As training goes on, the pattern $E^{(i)T}W^{(i)}$ becomes more and more visually similar, which implies the model learns to gradually learn to align the input clusters.

Figure: Figure 7: Visualize the weight of the self-organizing layer after applying inverse permutation. A snapshot of $E^{(i)T}W^{(i)}$ is shown at different training epoch. The number of epochs is shown on the top row. Each figure shows one column of the weight of the self-organizing layer, at different positions, i.e. $W_{:,k}^{(1)}$, where $k$ is the column number and $i$ is the position index. In total, $9$ columns are shown.
Refer to caption: extracted/5155611/figures/vis_T.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - can authors elaborate on why a set of linear projectors (non-shared in this case, a different layer for each cluster) cannot replace the use of self-organising maps? seems like the point here is that operation should not be shared between clusters rather than the type of operation used.
- how were representations trained in table 3? standard MAE? 
- can authors provide numbers for 1) URLOST with CIFAR10 2) MAE (Patch) with Permuted CIFAR and Foveat CIFAR; 
- can authors explain by such a high number of epochs is needed (10,000), is this only necessary for URLOST, prior work recommend 800-1600 epochs for standard MAEs. 
- what is the dimensionality of the representation in the beta-VAE vs the MAE; how do the size of models compare? the VAE used seems very shallow. 
- what is the range of beta parameter that was considered? 
- is the MAE in table 2 also with patch size 4? 
- what is the masking ratio in the MAE and URLOST (75%), can authors provide a comparison with 0% masking to show a non-masking scenario with equivalent architecture ?
- how is k selected ? how variable are results _across datasets_ for varying k, alpha, and beta parameters ? 

Minor: 
- typo in line 55
- bold number in table 1 for CIFAR10 is wrong, should be SimCLR
soundness: 3
strengths: - relevance: this paper aims at extending existing SSL methods to novel modalities (notably modalities found in various scientific fields) for which current methods and their associated architecture of choice (e.g., MAE with ViT backbones) are not adapted because designed to suit standard data modalities (e.g., images) with known structure. This topic is relevant as SSL methods are often shown to be powerful but designed (both joint embedding methods and masked image modeling) to suit specific modalities which limit their applicability to a broader range of data. Even for standard modalities like natural images, approaches such as MAE rely on assumptions that might not generalize across datasets. 
- presentation: paper is well written and easy to follow, the overall state of the paper is good.
- experimental evidence: the experimental evidence to support the proposed method is quite convincing (see weaknesses below for information and experiments missing) as the authors conduct experiments on 3 real-world datasets on which results show superiority to the standard MAE baseline which is the most relevant related work with confidence intervals.
summary: The paper investigates the topic of self-supervised representation learning and how to extend current popular frameworks to data modalities beyond natural images or text for which these methods have been initially designed. This is particularly relevant for the processing of scientific data which at this point would be hard to integrate into frameworks like MAEs. The authors propose to add a preprocessing block before the classic MAE which allows to aggregate data into the correct format even in the absence of prior information about the data's structure. This preprocessing blocks involves spectral clustering with density adjustment followed by a self-organizing map embedding layer. The authors show that the approach is more effective when process high-dimensional unstructured real-world data (2 natural science datasets + a modified version of CIFAR10) than applying the standard MAE framework to this data.
weaknesses: - sensitivity of the method to choice of hyperparameters: the proposed method relies on spectral clustering for which a number of hyper parameters should be defined (like the number of clusters and hyperparameters linked to cluster density). These hyperparameters seem crucial in order to achieve high performance and are data-driven. 
- motivation: while it is clear that an aggregation/clustering of the input dimensions is necessary, the intuition behind aggregating dimensions that cluster together is missing; While the proposed method seems to work well, the paper would benefit from some additional intuition as to why one might want to combine elements that are similar in the same patch and compare this approach to the information found in standard image patches. There is little understanding of how MAE work, and most of their design choices are empirically driven, therefore it remains unclear whether MAE work better when pixels within a patch are identical. 
- missing experimental baseline numbers and details: multiple information regarding the experimental setup is missing thereby reducing the ability to judge the soundness of the experimental setup (see question below) and a couple of experiments seem to be missing in order to confidently conclude that the proposed method is an effective alternative to the standard MAE baseline (see questions below) 
- missing information about the work's limitations which might include the computational cost and scalability of the proposed method.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: No extra questions. See the above comments.
soundness: 2
strengths: The strengths of the paper lie in its introduction of URLOST, a new combined framework for unsupervised representation learning towadrs addressing the challenges of high-dimensional data with unknown stationarity and topology. By combining a learnable self-organizing layer, density-adjusted spectral clustering, and a masked autoencoder, URLOST can captures learning representations from various data modalities. Its demonstrated performance on synthetic biological vision data, neural recordings, and gene expressions shows that it outperforms baselines, enlightening its capability to handle complex and irregular structures in an unsupervised mode.

The work is nicely conceived and the presentation is tidy. Its bio-inspired sense makes the work basically intriguing.
summary: This work introduces URLOST, a new framework for unsupervised representation learning aiming to overcomes limitations in handling high-dimensional data with unknown stationarity and topology, towards going beyond traditional methods that rely on structured data assumptions, such as grid-like images or time sequences. The proposed method combines a learnable self-organizing layer, density-adjusted spectral clustering, and a masked autoencoder (MAE) to capture learning representations from various data modalities.

Demonstrated on synthetic biological vision data, neural recordings, and gene expressions, URLOST shows a good performance, outperforming existing baselines in capturing complex, irregular structures without prior domain knowledge. The paper highlights areas for future work, such as integrating clustering into the model's end-to-end learning process and enhancing the self-organizing layer.
weaknesses: Several aspects of the work can be improved:
1) While the paper primarily motivates unsupervised learning, the experiments predominantly focus on supervised tasks (classification). This disconnect diminishes the relevance of the initial claims made in the first half of the paper.
2) The use of foveated preprocessing is an intriguing aspect; however, there is a noticeable drop in performance (in Table 1), likely attributed to this process. This raises the question of whether foveated preprocessing detracts from overall performance.
3)  Although the MAE appears to perform worse on the foveated dataset, this could simply be due to its training on the original dataset. Thus, the comparison does not effectively support the goal of emulating the biological process of foveation for benefits.
4) While the Vision Transformer (ViT) serves as the backbone, the paper lacks clarity on how attention is utilized in the experiments. It would be beneficial for the community to understand the potential relationship between the foveated process and visual attention.
5) The comparison against the MAE seems somewhat rudimentary. For example, the reported performance on CIFAR-10 indicates that several top models significantly outperform the authors' results (Table 1):
	Rank	Model		Percentage correct
	1		ViT-H/14		99.5
	2		DINOv2 		99.5
	3		µ2Net		99.49
	4		ViT-L/16		99.42
It seems the claimed better performance over the baseline was quite embarrassed by the top runners above.
6) The meaning and algorithm behind the permutation of CIFAR-10 images are unclear. Providing examples of permuted images would clarify this aspect. The rationale for permuting pixels should also be addressed, as such transformations may hinder human recognition and contradict the stated bio-inspired motivation.
7) In Section 2, the proposed method appears to aggregate several state-of-the-art techniques with limited novel mathematical contributions. Incorporating theoretical exploration of its bio-inspired aspects would significantly enhance the paper's quality.

---

confidence: 4
contribution: 4
initial_rating: 6
presentation: 3
questions: - Why is SimCLR with the proposed clustering pipeline (and ViT architecture) not included as a baseline for the CIFAR10-based experiments?  Why not ViG with patches found by URLOST?  Unless I have misunderstood, the proposed method is ultimately about “patchifying” data for use with a downstream transformer, which means there’s nothing tying the paper to MAE.  Such a shift away from exclusively linking the method with MAE would broaden its potential impact.
- There is no need for explicit positional encoding information per cluster, correct?  Presumably the self-organizing layers can encode identifying information per cluster.
- Line 265: “Moreover, since CNN can only process signal with stationarity and topology” is an incomplete sentence and also incorrect, as CNNs can absolutely process signals without stationarity.  Overall, I felt the stationarity part of the motivation to be much less important than the topology. 
- Why is MAE bolded in Table 1 when it was outperformed by SimCLR?
soundness: 3
strengths: The premise to handle more diverse datasets than images with a ViT setup (and MAE pretraining, though it seems more general than a specific pretraining method) by clustering input components is strong.  The experiments cover a good range of complexity, starting with standard CIFAR10, permuting the pixels, and then remapping the visual field entirely.  Then the V1 and gene expression experiments represent actual use cases.  The comparisons are sufficient to demonstrate the utility of the method.
summary: The submission proposes a route to implementing masked autoencoders (MAE) when the data does not have a known structure and yet is too granular for each component of the input to be its own “patch”.  In other words, there’s prior knowledge about the dataset that its components could probably be meaningfully grouped, but no grouping is available.
The method clusters the components by mutual information and spectral clustering, with an additional density correction applied.  After components have been grouped, all groups must be mapped to a shared representation space so that the transformer can compare them; for this the authors use a learned projection for each group that they call a self-organizing layer.
There are a fairly comprehensive suite of comparisons: to MAE without grouping the components and to SimCLR with a CNN and ViG architecture, on a couple CIFAR10 variants, gene expression data, and on V1 neural recording data.
weaknesses: The clustering is the heart of the proposed method and comes with multiple design choices or parameters whose selection is not obvious.  While the experimental results demonstrate the method works, results supporting these design decisions are a bit weak.  As far as I can tell, the only results regarding the effect of the density adjusted spectral clustering are with respect to the foveated CIFAR10 data (Table 3b, Table 4)?  Why should we assume gene expression is best processed with the same parameters, considering how structured the synthesized foveated CIFAR10 is?  By contrast, the self-organizing layer is pretty intuitive, yet a large chunk of the manuscript and appendices are dedicated to it.

Related work could be strengthened.  Consider discussing the relation to FlexiViT (Beyer et al., CVPR 2023), where different patch sizes are mapped to the same representation space by learnable transformations, and PatchGT (Gao et al., LoG 2022), where nodes of a graph are clustered with spectral clustering, and the representation of each “patch” is a learned function of the subgraph.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 2
questions: see Weaknesses
soundness: 2
strengths: The paper is well motivated and the task tackled is important and relevant to the community.
summary: The paper proposes a method to learn useful representations of data without assuming the relationships between dimensions of topological structure, translation invariance, as in images/sounds.
weaknesses: A lot of work has gone in to the paper, but unfortunately it is quite difficult to follow; and the proposed method seems very expensive (computing entropy, eigenvalues, clustering, etc). On the first point, to reach a broad audience, specific effort should be made regarding aspects less familiar to the general ML community. I find the method and data transformations difficult to parse.

* Clarity
   * [036-041] Unclear references to topology and stationarity -  core aspects of the paper
      * Is the topology property that dimensions are ordered such that correlation (or similar) is expected between the signal at "nearby" dimensions? 
      * Stationarity seems very entwined with topology, does stationarity require topology? If not how? 
      * What assumptions *is* the model making?
   * [225] unclear how permuting pixels leaves stationarity intact
   * [259] unclear how reducing density prevents redundant sampling.
* [095] A fully connected NN doesn't assume topology/spatial invariance and seems a key benchmark, but is not considered/mentoined.
* [183] How does a vector denote a cluster? the cluster mean, indexes of cluster members?
* [337] Given $\beta$-VAE results in Table 2 are close on the real datasets, have optimal parameters/architecture been used? Why is this not compared to in Table 1? 
* The proposed method seems expensive but no mention of this is made.

Minor
* [031] unsupervised representation learning (UL) and self-supervised representation learning (SS) are not the same. SSL is a subset of UL, e.g. a $\beta$-VAE performs UL but not SSL, whereas SimCLR performs SSL (and so UL).
* Several typos
   * [037] "arised", [055] "singal" etc

---

confidence: 3
contribution: 2
initial_rating: 8
presentation: 2
questions: * if I permute all images in the same way, why would it ruin the topology? As far as I understand it, I will just transform it but the ordering is still preserved
soundness: 2
strengths: * great motivation: important problem of topological ordering and stationarity assumptions is raised. Biological inspiration is also interesting.
* novel ideas, especially, when it comes to clustering and self-organized layers instead of patching. This potentially allows to aggregate features on different scales, which could be highly beneficial in many cases (for example, action tracking could be composed of motions tracking)
* making the code available is great for reproducibility
summary: The paper suggests a novel way unsupervised representation learning pipeline, which consists of three stages: clustering, self-organising layers and masked autoencoder training. The authors try to tackle the assumptions of stationarity and topological ordering of the data for self-supervised learning.
weaknesses: * While the authors criticise graph neural networks for not being scalable and the paper is proposed as a new self-supervised approach, the scalability aspect is not addressed, both from the perspective of time (and spectral clustering can become slow) or performance . The datasets used in the paper are fairly small, while even for transcriptomics or neuroscience there are quite big datasets publicly available now.
* Topology is mentioned a lot, however, I do not fully understand, where exactly authors show that their method is not relying on topology. I would suggest authors to take a point cloud dataset (for example, 3D Point Cloud Classification on ModelNet40 ) and perform a comparison on it. The transcriptomics vectors are still ordered (like each position corresponds to a specific gene), which could be also thought as topological organisation (like a flattened picture). 
* The experiments are missing the error bars or confidence intervals, which makes it a bit hard to understand how robust is the method and how significant are the improvements. I would inspire the authors to add the error bars for all of the experiments
*  while the paper motivation is highly biologically grounded, the actual image dataset has very low signal-to-noise ratio, eg  a lot of natural scenes do not so nicely isolate a single object in the center of the image such as CIFAR. I would inspire the authors to use a more noisy multi-object classification or segmentation dataset
* while avoiding stationarity assumption is one of the main paper motivations, I felt like a textbook case of non-stationary data, eg time series, is missing. I would suggest to include some non-stationary timeseries classification example, for example, for epilepsy, to prove illustrate this point in a much more convincing manner.
* I find the biological motivation sometimes more confusing than helping, for instance, lines 471-476, mention self-organising maps, without giving a brief introduction on what they are.

Meta-review: This paper introduces a novel framework for learning from high-dimensional data without prior knowledge of stationarity or topology. The model integrates a learnable self-organizing layer, spectral clustering, and a masked autoencoder (MAE). Its effectiveness is demonstrated across three diverse data modalities: simulated biological vision data, neural recordings from the primary visual cortex, and gene expressions. This work presents an innovative unsupervised representation learning pipeline with sound motivation and convincing experimental results.

The rebuttal effectively addressed the major concerns raised by the reviewers. The paper received three acceptances, one borderline decision, and one rejection. Despite some disagreements during the discussion, the AC believes the authors adequately explained and justified their claims. The AC recommends accepting the paper and urges the authors to revise it to incorporate key points discussed during the rebuttal phase.

Decision: Accept (Poster)