URLOST: Unsupervised Representation Learning without Stationarity or Topology
Authors: Zeyu Yun
Juexiao Zhang
Yann LeCun
Yubei Chen
Venue: ICLR 2025 Poster (2025)
Keywords: Unsupervised learning
Self-Supervised Learning
NeuroAI
Multi-Modality
Human Vision
Biologically-inspired Models
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Unsupervised representation learning has seen tremendous progress. However, it is constrained by its reliance on domain specific stationarity and topology, a limitation not found in biological intelligence systems. For instance, unlike computer vision, human vision can process visual signals sampled from highly irregular and non-stationary sensors. We introduce a novel framework that learns from high-dimensional data without prior knowledge of stationarity and topology. Our model, abbreviated as URLOST, combines a learnable self-organizing layer, spectral clustering, and a masked autoencoder (MAE). We evaluate its effectiveness on three diverse data modalities including simulated biological vision data, neural recordings from the primary visual cortex, and gene expressions. Compared to state-of-the-art unsupervised learning methods like SimCLR and MAE, our model excels at learning meaningful representations across diverse modalities without knowing their stationarity or topology. It also outperforms other methods that are not dependent on these factors, setting a new benchmark in the field. We position this work as a step toward unsupervised learning methods capable of generalizing across diverse high-dimensional data modalities.

Published as a conference paper at ICLR 2025
URLOST: UNSUPERVISED REPRESENTATION LEARN- ING WITHOUT STATIONARITY OR TOPOLOGY
Zeyu Yun', Juexiao Zhang?, Yann Lecun 3.4, Yubei Chen 1 UC Berkeley, UC Davis, 3 New York University, 4 FFARR at Meta
ABSTRACT
Unsupervised representation learning has seen tremendous progress. However, it is constrained by its reliance on domain specific stationarity and topology, a limitation not found in biological intelligence systems. For instance, unlike com- puter vision, human vision can process visual signals sampled from highly ir- regular and non-stationary sensors. We introduce novel framework that learns from high-dimensional data without prior knowledge of stationarity and topology. Our model, abbreviated as URLOST, combines a learnable self-organizing layer, spectral clustering, and a masked autoencoder (MAE). We evaluate its effective- ness on three diverse data modalities including simulated biological vision data, neural recordings from the primary visual cortex, and gene expressions. Com- pared state- of-e-offthhe-rt unsupervised learning methods like SimCLR and MAE, our model excels at learning meaningful representations across diverse modalities without knowing their stationarity or topology. also outperforms other meth- ods that are not dependent
these factors, setting new benchmark in the field. We position this work as a step toward unsupervised learning methods capable generalizing across diverse high-dimensional data modalities. Code is available at this repository
INTRODUCTION
Unsupervised representation learning aims to develop models that autonomously detect patterns in data and make these patterns readily apparent through specific representations. Over the past few years, there has been tremendous progress in the unsupervised representation learning com- munity, especially self-supervised representation learning (SSL) method. Popular methods SSL. methods like contrastive learning and masked autoencoding [87] 12
work well on typical data modalities such as images, vidcos, time series, and point clouds. However, these methods make implicit assumptions about the data domain's topology and stationarity. Topology refers to the low-dimensional structure arisen from physical measurements, such as the pixel grids in images, the temporal structures in time series and text sequences, the 3D structures in molecules and point clouds 7 Stationarity refers to the characteristic that the statistical properties of the signal are invariant across its domain 6 For instance, the statisties of pixels and patches in images are in- variant to their spatial locations. A vase is still vase no matter placed at the corner or the center of the image.
33 90
The success of state-of-the-art self-supervised representation learning (SSL) methods largely de- pends on these two crucial assumptions. For example, in computer vision, popular SSL techniques, such as Masked Autoencoders [34] and joint-embedding methods like SimCLR [22 require the construction of image patches. Both approaches typically rely on convolutional neural networks (CNNs) or Vision Transformer (ViT) backbones (22]. These backbones consist of shared- weight convolutional filters or linear layers, which inherently exploit the stationarity and regular topology present in natural images. The geometric deep learning community has made significant efforts to extend machine learning to domains beyond those with regular topology. However, graph neural net- work (GNN)-based methods empirically struggle to scale with self-supervised learning objectives and large datasets (69 10: 9 Methods that do scale well with large data still assume a minimal level of stationarity and regular topology [32]
What if we have high- figh--imensional signals without prior knowledge of their domain topology or stationarity? Can still craft high-quality representation? This is not only the situation that

---

Published as a conference paper at ICLR 2025
biological intelligence systems have to deal with but also practical setting for many scientific data analysis problems. Taking images as an example, computer vision system takes in signal from digital cameras. Biological visual systems, regularity. Unlike the uniform grid in camera sensors, the cones and rods in the retina are distributed unevenly and non-uniformly. Yet, biological visual systems can establish a precise retinotopy map from the retina to neurons in visual cortex based on spontancous, pocally-rropagated retinal activities and external stimuli 47| and leverage retinotopic input build unsupervised representations. This motivates us to build unsupervised representations without relying on the prior stationarity of the raw signal or the topology of the input domain. The ability to build unsupervised representations without relying topology and stationarity has huge advantages. For example, it allows humans to utilize irregular sensors for both high resolution and broad coverage, which much more efficient than regular camera sensor for dynamic environments. Developing such model also allows us to create a powerful AI system that computes any high high-dimensional signal.
the other hand, have to deal with signals with less domain
w/ stationarity topology
Data w/o stationarity topology
Joint Embedding
MAE
Blological Vision System
New Paradigm
h
M
f
t
URLOST
hy
Enc
Enc
many more high-dimensional data in nature
t
M
M
Figure 1: From left right: the unsupervised representation learning through joint embedding and masked outo-encoding; the biological vision system that perceives via unstructured sensor and un- derstands signal without stationarity topology [6]] and many more such diverse high dimensional signal in natural science (57 89 that our method supports while most existing unsupervised meth- ods don't.
In this work, we aim to build unsupervised representations for general high-dimensional data and introduce unsupervised representation learning without stationarity or topology (URLOST). Tak- ing images as example again, let's assume we receive set of images whose pixels are shuffled in the same order. In this case, the original definition of topology of images is destroyed, i.c. cach pixel should no longer the neighbor of the pixels that's physically next to it. How representations in an unsupervised fashion without knowledge of the shuffling order? If possible, such method to build unsupervised representations for general high-dimensional data? Inspired by Roux et al. [67), we use low-level statistics and spectral clustering to form clusters of the pixels, which recovers coarse topology of the input domain. These clusters are analogous image patches except that they are slightly irregularly shaped and different size. We mask a proportion of these "patches" and utilize Vision Transformer [22) to predict the masked "patches" based on the remaining unmasked ones. This "learning to predict masked tokens" approach is proposed in masked autoencoders (MAE) 34 and has demonstrated effectiveness on typical modalities. Firstly, we test the proposed method on the synthesized biological visual dataset, derived from the CIFAR- 10 [4] using foveated retinal sampling mechanism [L4]. Then we generalize this method to two high-dimensional vector datasets: a primary visual cortex neural response decoding dataset 77 and the TCGA miRNA- based cancer classification dataset [79] 84). Across all these benchmarks, proposed method outperforms existing SSL methods, demonstrating its effectiveness in building unsupervised representations for signals lacking explicit stationarity or topology. Given the emer-
build
gence of new modalities in deep learning from natural sciences 75
31
62
46
85
17
2. such as
2

---

Published as a conference paper at ICLR 2025
Signal Clustering
Self-organizing Layer
Aligned Clusters
Masked Autoencoder
Re-organizing Layer
Reconstructed
Signal
Signal
0000
Enc
Dec
Figure 2:
The overview framework of URLOST. The nigh-dimensional input signal undergoes clustering and oelloorganizttion before unsupervised learning using a masked autoencoder for signal reconstruction.
chemistry, biology, and neuroscience, method offers promising approach in the effort to build unsupervised representations for nigh-dimensional data.
2
METHOD
2.1
MOTIVATION AND OVERALL FRAMEWORK
Our objective is to build robust unsupervised representations for high-dimensional signals without prior information explicit topology and stationarity. The learned representations are intended to enhance performance in downstream tasks such as classification. We begin by using low-level statistics and clustering approximate the topology of the signal domain. The clusters then serve as input to a masked autoencoder. As depicted in Figure [ the masked autoencoder randomly masks out patches in an image and trains Transformer-basse the original image. After unsupervised training, the latent state of the autoencoder yields high- quality representations. In our approach, the clusters derived from the raw signal input to the masked autoencoder.
autoencoder unsupervisedly to reconstruct
Taking images as an example data modality, clusters formed in images are bags of pixels, which differs from image patches in several key aspects. First, they are unaligned, exhibit varied sizes and shapes. Second, each pixel a cluster not confined to fixed 2D locations like pixels in image patches. To cope with these differences, we introduce a self-organizing layer responsible for aligning these clusters through learnable transformations. The parameters this layer are jointly optimized with those of the masked autoencoder. Our method is termed URLOST, acronym for Unsupervised Representation Learning withOut Stationarity or Topology. Figure 2 provides an overview of the framework. URLOST consists of three components: density adjusted spectral clustering, self-organizing layer, and masked autoencoder. The functionalities of these components detailed in the following subsections.
2.2 DENSITY ADJUSTED SPECTRAL CLUSTERING
Given dataset S E Rm where each row denote high dimensional data point. Let S Rex1 be ith column S, which represents the ith dimension of the signal. Since we want to process dimensions that share similar information together, dimensions. Let Aij I(S;; Sj) be the mutual information between dimension i and j. Consider each dimension node of graph and A the the affinity matrix of graph. I D graph Laplacian, where D is the diagonal matrix whose (i, -elemenn is the sum of A's i-th row. We can formulate the spectral embedding problem as finding such that minyyr - (tr(YLY') A clustering algorithm is then applied to the embedding Y as explained in appendix A.2 The size and shape of the clusters strongly affect the unsupervised learning performance. To adjust the size and shape, we apply a density adjustment matrix P to adjust L in the spectral embedding objective. The optimization problem becomes the following:
spectral clustering group these
is the
min Ir by(Yp1/21p1/22/) yyT=/
where P = diag(p(i)), p(i) is the unnormalized density function defined on each node i. We set p(i) = ((i)"n(i)-? where nii) Eje (Aze) Aşi and q(i) is the prior density which depends
3

---

Published as a conference paper at ICLR 2025
on specific dataset and is defined in the experiment section. a, 8 and K are ting a = and K = m will recover the normalized graph Laplacian. In appendix A.J. we provide a detailed interpretation and motivation of density adjustment. We further verify its effectiveness with ablation study in section 4.3 and appendix A.4
Set-
2.3 SELF-ORGANIZING LAYER
Transforming a high-dimensional signal into a sequence of clusters using the above method is not enough because it does not capture the internal structure within individual clusters. As an intuitive example, given an image, we divide it into a set of image patches of the same size. If we apply dif- ferent permutations these image patches, their inner product will no longer reflect their similarity properly. Clusters we obtained from section 2.2 are analogous to image patches, but elements in each cluster have arbitrary ordering. Thus, the inner product between two clusters is also arbitrary due to the ordering mismatch. In Transformers, since self-attention depends on the inner products between different "clusters," we need to align these clusters in space, where their inner products reflect their similarity, To align these clusters, we propose a self-organizing layer with learnable parameters. Specifically, let vector (*) denote the ith cluster. Each cluster (x1) is passed through a differentiable function 9(-, w()) with parameter w((),, resulting in a sequence zo:
= [g(2(") ) g0( (M))
(2)
20 is comprised of projected and aligned representations for all clusters. The weights of the proposed self-organizing layer, (w((),... ... (MM), are jointly optimized with the subsequent neural network
introduced in the next subsection.
2.4 MASKED AUTOENCODER
After the self-organizing layer, 2p is passed to a Transformer-based masked autoencoder (MAF) with unsupervised learning objective. Masked autoencoder (MAE) consists of an encoder and decoder which both consist of stacked Transformer blocks introduced in Vaswani et al. (82). The objective function is introduced in He et al. [34]: masking random image patches in an image and training an autoencoder to reconstruct them, as illustrated in Figure | In our case, randomly selected clusters in 2q are masked out, and the autoencoder trained to reconstruct these masked clusters. After training. the encoder's output is treated as the learned representation of the input signal for downstream tasks. The masked prediction loss computed as the mean square error (MSE) between the values of the masked clusters and their corresponding predictions.
RESULT
Since the biological vision system inspires method, first validate its ability synthetic bio- logical vision dataset created from CIFAR-10. Then we evaluate the generalizability of URLOST on two high-dimensional natural datasets collected from diverse domains. Detailed information about each dataset and the corresponding experiments is presented in the following subsections. Across all tasks, URLOST consistently outperforms other strong unsupervised representation learning meth- ods. For all the experiments in this work, we use linear layer to parametrize the self self-organizing layer, i.e. g(2, w())) ll z. Additionally, we provide training hyperparameters and experiments for effect of hyperparameters in Appendix A.6
3.1 SYNTHETIC BIOLOGICAL VISION DATASET
As discussed in the introduction, the biological visual signal serves as an ideal data modality vali- date the capability of URLOST. In contrast to digital images captured by fixed array of sensors, the biological visual signal is acquired through irregularly positioned ganglion cells, inherently lacking explicit topology and stationarity. However, it is hard to collect real-world biological vision signals with high precision and labels to evaluate our algorithm. Therefore, we employ a retinal sampling technique to modify the classic CIFAR-10 dataset and simulate imaging from the biological vision signal. The synthetic dataset referred to as Foveated CIFAR-IO. To make a comprehensive com- parison, also conduct experiments the original CIFAR-10, and a Permuted CIFAR-I dataset obtained by randomly permuting the image.
Permuted CIFAR-10. To remove the grid topology inherent in digital imaging, simply per- mute all the pixels within the image, which effectively discards any information related to the grid structure of the original digital image. We applied such permutation each image in the CI
4

---

Published as a conference paper at ICLR 2025
dataset to generate the Permuted CIFAR-10 dataset. Nevertheless, permuting pixels only removes an image's s topology, leaving its stationarity intact. To obtain the synthetic biological vision that has neither topology nor stationarity, we introduce the Foveated CIFAR-IO.
A
C
D
Figure 3: Retina sampling (A) An image in CIFAR-10 dataset. (B) Retina sampling lattice. Each blue dot represents the center of Gaussian kemel,, which mimics retinal ganglion cell. (C) Vi- sualization of the car image's s signal sampled using the retina lattice. Each kernel's sampled RGB value is displayed at its respective lattice location for visualization purposes.
(D) density-adjustr spectral clustering results are shown. Each unique color represents a cluster, with each kernel col- ored according its assigned cluster.
Foveated CIFAR-10 Unlike cameras that use uniform photosensors with consistent sampling pat- terns, the human retina features ganglion cells whose receptive field sizes vary smaller in the cen- tral fovea and larger in the periphery. This results foveated imaging [80]. cnabling primates to have both high-resolution central vision and wide overall receptive field. However, this distinctive sampling pattern causes visual signals in the retina to lack stationarity; the statistical properties differ between the center and periphery, with ganglion cell responses being highly correlated in the fovea but less correlated in the periphery. To mimic the foveated imaging with CIFAR-10, we adopt the retina sampling mechanism from Cheung et al. [14]. Like shown in Figure each dot represents retinal ganglion cell, which together form a non-stationary sampling lattice with irregular topology. Details on the implementation of foveation sampling is provided Appendix A.3
Experiments. We compare URLOST on both of the synthetic vision datasets as well as the original CIFAR-10 with popular unsupervised representation learning methods SimCLR [[2] and MAE (34). We also compared the VIT backbone used in URLOST to other backbones such convolutional neural network (CNN). Moreover, since CNN works better with stationary signals, we further com- pared our methods to SimCLR with graph neural network (GNN) backbone, which doesn' 't rely on the stationarity assumption the data. For the GNN baseline, we use Vision GNN (ViG), which state of the art graph neural network on images proposed in 332. After the model is trained without supervision, we use linear probing to evaluate it. This is achieved by training a linear classifier on top of the pre-trained model with the given labels. The evaluations are reported in Table SimCLR excels on CIFAR-10 10 but struggles with both synthetic datasets due to its inability to handle data without stationarity and topology. MAE gets close to SimCLR CIFAR-10 with x 4 patch size. However, the patch size no longer makes sense when data has no topology. So we additionally tested MAE masking pixels instead of image patches. The performance of pixel level MAE is invariant to the removal of topology. The performance on Permuted CIFAR-1O is the same as the performance on CIFAR- 10, though poorly. But it still drops greatly to 48.5% on the Foveated CIFAR- 10 when stationarity is also removed. In contrast, only URLOST is able to maintain consistently strong per- formances when there is no topology or stationarity, achieving 86.4% on Permuted CIFAR- 10 and 85.4% on Foveated CIFAR-IO when the baselines completely fail.
3.2 VI NEURAL RESPONSE TO NATURAL IMAGE STIMULUS
After testing URLOST's performance synthetic biological vision data, we take step further to challenge its generalizability with high-dimensional natural datasets. The first task decoding neural response recording in the primary visual area (V1) of mice.
V1 neural response dataset. The dataset, published by [57]. contains responses from over 10,000 VI neurons captured via two-photon calcium imaging. These neurons responded to 2,800 unique images from ImageNet [9),, with each image presented twice to assess the consistency the neural response. In the decoding task, prediction is considered accurate if the neural response to given
5

---

Published as a conference paper at ICLR 2025
Table 1: Evaluation on computer vision and synthetic biological vision dataset. ViT (Patch)
stands for the Vision Transformer backbone with image patches as inputs. VIT (Pixel) means pixels are treated as input units. ViT (Clusters) means clusters are treated as inputs instead of patches. The number of clusters is set to 64 for both Permuted CIFAR-1O and Foveated CIFAR-1O dataset. Eval Acc stands for linear probing evaluation accuracy.
Dataset CIFAR-10
Method
Backbone
Eval Ace
MAE
VIT (Patch)
88.3 %
MAE
VIT (Pixel)
56.7 %
SimCLR
ResNet-18
90.7 %
SimCLR
ViG
53.8 %
Permuted CIFAR-10
URLOST MAE
VIT (Cluster)
86.4
MAE
VIT (Pixel)
56.7 %
SimCLR
ResNet-18
47.9
SimCLR
ViG
40.0 %
Foveated CIFAR-10
URLOST MAE
VIT (Cluster)
85.4 %
MAE
VIT (Pixel)
48.5 %
SimCLR
ResNet-18
38.0 %
SimCLR
ViG
42.8
Table 2: Evaluation V1 response decoding and TCGA pan-cancer classification tasks. "Raw" indicates preprocessed (standardized and normalized) raw signals. Best 3 values are used for 3-VAE. For URLOST MAE, cluster sizes are 200 (V1) and 32 (TCGA). We pick 15 seeds ran- domly to repeat the training and evaluation for each method. For the MAE baseline, we randomly group dimensions instead of using spectral clusterings. We report the 95% confidence interval for all methods.
Method Raw MAE
VI Response Decoding Acc
TCGA Classification Ace 91.7 0.24% 90.6% + 0.63%
73.9% 0.00
70.6% + 0.22 %
3-VAE
75.64% + 0.11%
94.15% + 0.24%
URLOST MAE
78.75% + 0.18 %
94.90% + 0.25 %
stimulus in the first presentation closely matches the response to the same stimulus in the second presentation within the representation space. This task presents greater challenges than the synthetic biological vision described in the prior section. For one, the data comes from real-world neural recordings rather than curated dataset like CIFAR-I0. For another, the geometric structure of the VI area is substantially more intricate than that of the retina. To date, no precise mathematical model of the VI neural response has been well established. The inherent topology and stationarity
of the data still remain difficult to grasp [56] (55) Nevertheless, evidence of retinotopy [25) 26 and findings from prior research [54] 13
suggest that the neuron population code in VI
tiling a low dimensional manifold. This insight led us to treat the population neuron response as high-dimensional data and explore whether URLOST can effectively learn its representation.
Experiments. Following the approach in Pachitariu et al. [57] we apply standardization and nor- malization to the neural firing rate. The processed signals are high-dimensional vectors, and they can be directly used for the decoding task, which serves as the "raw" signal baseline in Table 2 For representation learning methods, URLOST is evaluated along with MAE and 3 V-V [35]. For MAE baseline, obtain patches by randomly selecting different dimensions from the signal. Note that the baseline methods need to handle high-dimensional vector data without stationarity or topology. Since SimCLR and other constrastive learning model leverage these properties make positive pair, they are no longer applicable. We use 3-VAE as a baseline instead. We first train the neural network with an unsupervised learning task, then use the latent state of the network the represen- lation for the neural responses in the decoding task. The results are presented in table Our method surpasses the original neuron response and other methods, achieving the best performance.

---

Published as a conference paper at ICLR 2025
Masking Unit
Permuted Cifar10
Foveated Cifar10
TCGA Gene
V1 Response
Clusters (URLOST)
86.4% %
85.4% %
94.9%
78.8%
Random patch Individual dimension
7%
51.1% %
91.7%
73.9%
56.7%
48.5%
88.3%
64.8%
Table Performance of different datasets using different masking units. "Clusters (URLOST)" refers to the clusters formed using pairwise mutual information and spectral clustering. "Random patch" refers to patches formed by aggregating random dimensions. "Individual dimension" refers to using individual dimensions masking units without any aggregation. The model is trained for the mask prediction task, and the probing accuracy is reported in the table.
3.3 GENE EXPRESSION DATA
In this subsection, we further evaluate URLOST on high-dimensional natural science data from a completely different domain, the gene expression data.
Gene expression dataset. The dataset comes from The Cancer Genome Atlas (TCGA) 779 84],
which is a project that catalogs the genetic mutations responsible for cancer using genome sequenc- ing and bioinformatics. The project molecularly characterized over 20,000 primary cancers and cancer types. We focus on the pan-cancer classification task: diagnose and classify the type of cancer for a given patient based on his gene expression profile. The TCGA project collects the data of 11,000 patients and uses Micro-RNY (miRNA) as their gene expression profiles. Like the VI response, no explicit topology and stationarity are known and each data point is a high-dimensional vector. Specifically, 1773 miRNA identifiers are used so that each data point is 7733-dimensional vector. Types of cancer that each patient is diagnosed with serve as the classification labels.
matched normal samples spanning
Experiments. Similar to Section 3.2 URLOST is compared with the original signals, MAE, and 3-VAE, which is the state-of-the-arr unsupervised learning method on TCGA cancer classification 9T 92 We also randomly partition the dataset do five-fold cross-validation and report the average performance in Table [ Again, our method learns meaningful representation from the original sig- nal. The learned representation benefited the classification task and achieved the best performance, demonstrating URLOST's ability to learn meaningful representation of data from diverse domains.
4
ABLATION STUDY
4.1 AGGREGATING SIMILAR DIMENSIONS FOR MASKING UNIT
Clustering is at the heart of the proposed method. Why should similar dimensions be aggregated to form masking unit? Intuitively, we assume that similar dimensions are used to sample similar re- gions of the underlying signals. Instead of masking each dimension, making the entire cluster force the model learn high-level structure from the signal, allowing the model to learn a rich represen- tation. For example, for the CIFAR10 example, similar pixels tend to sample similar regions, which together form image patches. For data, each neuron encodes a simple pattern like an oriented edge, but neurons together code patterns like shape and contours. We provide experiments for using different masking units, as shown in table aggregating similar dimensions significantly improves the performance.
4.2 SELF- ORGANIIGGN LAYER vs SHARED PROJECTION LAYER
Conventional SSL models take a sequential input x = [2(11) ....).......... (A() and embed them into latent vectors with linear transformation:
20 (1) ... (10)
which is further processed by a neural network. The sequential inputs can be list of language tokens
21 63), pixel values [ image patches [22). overlapped image patches [12 33 90]. E can be considered as a projection layer that is shared among all elements in the input sequence. The self- organizing layer g(,, w() introduced Section 2.3 can considered as a non-shared projection layer. We conducted an ablation study comparing the two designs to demonstrate the effectiveness of the orga-ongan layers both quantitatively and qualitatively. To facilitate the ablation, we further synthesized another dataset.
7

---

Published as a conference paper at ICLR 2025
Locally-permuted CIFAR-10. To directly evaluate the performance of the non-shared projection approach, we designed an experiment involving intentionally misaligned clusters. In this experi- ment, divide each image into patches and locally permute all the patches. The i-th image patch is denoted by x(1), and its permuted version, permutated by the permutation matrix E is expressed as (6) We refer to this synthetic dataset the Locally- sis posits that models using shared projections, as defined Equation 3 will struggle to adapt to random permutations, whereas self-organizing layers equipped with non-shared projections can au- tonomously adapt to each patch's 's permutation, resulting robust performance. This hypothesis evaluated quantitatively and through the visualization the learned weights w:(()
CIFAR- 10. Our hypothe-
Permuted CIFAR-10. Meanwhile, we also run the ablation study on the Permuted CIFAR-1O. Unlike locally permuted CIFAR-I0, a visualization check is not viable since the permutation is done globally. However, we can still quantitatively measure the performance of the task.
Quantitative results. Table confirms our hypothesis, demonstrating significant performance decline in models employing shared projections when exposed to permuted data. In contrast, the model with self-organizing layer maintains stable performance.
Visual evidence. Using linear layers to parameterize the self oaff-organizin layers, i.e. let g(z, W()) W x, we expect that if the projection layer effectively aligns the input sequence, ( (+()) should exhibit visual similarities. That is, after applying the inverse permutation (()) the learned projec- tion matrix W at each location should appear consistent similar. The proof of this statement Permuted CIFARIO provides visual evidence supporting this claim. In Figure 4 the weights show similar patterns after reversing the permutations. These observations demonstrate that URLOST can also be used as an unsupervised learning method recover topology and enforce stationary the signal. Note that Figure shows that the self-organizing layer learns to "undo" the permutation. Additionally, the self-organizing layer also does some extra regular transformation each patch. is likely that the self-organizing layer learns to encode position information as transformations of some simple group structures.
provided Appendix A.5 The model trained on Locally-Permuted
A
(11)
E (1)T
E
patchwise permute
patchwise unpermute
B W
ELYT
C
E
g
w:D
patchwise unpermute
patchwise unpermute
Figure 4: Learnt weights of self-organizing layer. (A) Image is cropped into patches, where each patch z00) first undergoes a different permutation E(0, then the inverse permutation E (B) The learned weight of the linear self-organizing layer. The 12th column of (i) at all positions i are reshaped into patches and visualized. When W undergoes the inverse permutation E(I) they show similar patterns. (C) Visualization of the 37th column of ( Similar to (B).
4.3 DENSITY ADJUSTED CLUSTERING VS UNIFORM DENSITY CLUSTERING
As explained in Section 2.2 the shape and size of each cluster depend how the density function p()) is defined as:
(()) (i))=
8

---

Published as a conference paper at ICLR 2025
where n(1) = E, (A,,) Aji. A is the affinity matrix, q(i) represent the eccentricity, the distance from äth kernel to the center of the sampling lattice. Setting a and [ nonzero, the density function is eccentriity-deeendeett Setting both a and 8 to zero will make nı(2) constant which recovers the uniform density spectral clustering. We vary the parameters and / generate different sets of clusters for the foveated CIFAR-10 dataset and run URLOST using each of these sets of clusters. Results in Table validate that the model performs better with density-adjusted clustering. The in- tuitive explanation is that by adjusting the values of a and 8, we can make each cluster carry similar amounts of information (refer to Appendix (A.4). ). A balanced distribution of information across clus- ters enhances the model's 's ability to learn meaningful representations. Without this balance, masking a now-infomaation cluster makes the prediction task trivial, while masking a ingfomnformtiion cluster will make the prediction task too difficult. In either scenario, the model's ability to learn effective representations is compromised.
Table 4: Ablation study on self-organizing layer. Linear probing accuracy with varying param- eters, keeping others constant. For Sncally-Permutaterr CIFAR-IO, we use patch size. For Permutated CIFAR-IO and Foveuted CIFAR-1O, we set the number of clusters to 64 for the spectral clustering algorithm. We kept the hyperparameter of the backbone model the same as table [
(a) Replacing the self-organizing layer with the shared projection layer entails a significant drop performance.
(b) "SC" denotes spectral clustering with uniform den- sity clustering and "DSC" denotes density adjusted spectral clustering. Using DSC to create clusters out- perform model without DSC.
Dataset
Projection
Eval Acc
Dataset
Cluster
Eval Acc
shared
81.4 %
SC
82.7 %
CIFAR-10
non-shared
87.6 %
Foveated CIFAR-10
Permuted
shared
80.7 %
DSC
85.4 %
CIFAR-10
non-shared
86.4 %
ADDITIONAL RELATED WORKS
Several interconnected pursuits are linked to this work, and we will briefly address them here:
Topology in biological visual signal. 2-D topology of natural images is a strong prior that requires many bits to encode (20] (.. Such 2-D topology encoded the natural image statistic 771 (7]. Optic and neural circuits in the retina result in a more irregular 2-D topology than the natural im- age, which can still be simulated 66 59
78 This information is further processed by the primary visual cortex. Evidence of retinotopy suggests the low-dimensional geometry of visual input from retina is encoded by the neuron in primary visual cortex [53] 27 36 26 83 61]. These study provide evidence that topology under retinal ganglion cell and VI neurons can be recovered. The theory and computational model of how visual system code encodes such 2-D is well-studied in computational neuroscience. The self- orgf-organi map (SOM) was proposed the first compu- tational model by Kohonen in 1982. The algorithm produces a low-dimensional representation of a dineer-dimensionall dataset while preserving the topological structure of the data 44 SOM is also motivated to solve the "unscramble" pixels problem by "descrambling" by mapping pixels into a 2D index set by leveraging the mutual information between pixels at different locations. More detail is in Appendix A. 67 tackles the same problem with manifold learning.
Evidence of selt-organizing mechanism in the brain. In neuroscience, many works use the self-
organizing maps (SOM) as computational model for V1 functional organization [3] 76 I 24 52]
43 In other words, this idea of self-organd self-organizing is a principle governing how the brain performs com- putations. Even though V1 functional organizations are present at birth, numerous studies indicate that the brain's 's self-organizing mechanisms continue after full development 30
Learning with signal on non-euclidean geometry. In recent years, researchers from the machine learning community have made efforts to consider geometries and special structures beyond classic images, text, and feature vectors. This is the key motivation for geometric deep learning and graph neural networks (GNN). Many works generalizes common operator for processing Euclidean data like 2d convolution and attention 7 49 18 28). Due to the natural of graph neural network, they often only work on limited data regime and do not scale to large data [69] 10, D. However, recent development in 32 shows this direction is prominent. They successfully scales GNN to ImageNet.
9

---

Published as a conference paper at ICLR 2025
We compared their proposed neural network architecture with the ViT backbone used in URLOST. Recent research also explores adapting the Transformer to domains beyond Euclidean spaces such as [66 15 29]. Ma et al. [4 treats an image set of points but relies 2D coordinates. UR- LOST employs a single mutual information graph to define the topology of the high-dimensional signal. Gao et al. [29]. on the other hand, is designed to handle graph data, where each data point corresponds to a distinct graph. It segments a graph into "subgraphs." processes them with a GNN, transformer. This approach is undeniably more flexible but requires all subgraphs to be globally aligned. Furthermore, the self- self-organizing layer in URLOST generalizes the "patch resizer" mechanism from Vie used in Beyer et al.
and then passes the output
4 Finally, the lines of works
follows the "perceiver" architecture is very related to URLOST 139 38 "Perceiver" also implic- itly aggregates similar dimensions of high dimensional data and processes the similar dimension with an attention mechanism. (88) is a follow-up work that combines the Perceiver architecture and masked-predicttoe objective. However, the key difference is that the "perceiver" computes similar- ity between dimensions on single example signal. For image data, this is essentially clustering pixels based on the pixel intensity of color in a signal image. On the other hand, URLOST computes similarity over statistics of the distribution of the signal. Additionally, "perceiver" implicitly aggre- gates similar dimensions, while URLOST explicitly aggregates similar dimensions and recovers the original topology of the signal.
Self-supervised learning. Self-supervised learning (SSL.) has made substantial progress in recent years. Different SSL method is designed for each modality, for example: predicting the masked/next token in NLP 63 81, solving pre-text tasks, predicting masked patches, or building contrastive
33 90] These SSL methods have demonstrated descent scalability with vast amount of unlabeled data and have shown their power by achieving performance on par with or even surpassing supervised methods. They have also exhibited huge potential in cross-modal learning, such as the CLIP by Radford et al. (64.
image pairs in computer vision 48
6 DISCUSSION, LIMITATIONS, AND FUTURE DIRECTION
The success of most current state-of -ofthe-----e-trtttt self-supervised representation learning methods relies on the assumption that the data has known stationarity and domain topology. In this work, we explore unsupervised representation learning under a more general assumption, where the stationarity and topology of the data are unknown to the machine learning model and its designers. We argue that this is general and realistic assumption for high- -igh-dimensional data in modalities of natural science. We propose a novel unsupervised representation learning method that works under this assumption and demonstrates our method's 's effectiveness and generality on a synthetic biological vision dataset and two datasets from natural science that have diverse modalities. We also perform a step-by-step ablation study to show the effectiveness of the novel components in our model.
Unlike the self-organizing layer, the procedure for aggregating similar dimensions together is sepa- rate from the training of MAE. Given the effectiveness of the self-organizing layer, this design could be sub-optimal and could be improved. Learning the clusters end-to-end with the representation via back propaaprpagttionn is worth investigating in the future. Additionally, the computational costs of computing pairwise mutual information and clustering both scales quadratically with the number of dimensions of the signal. Although each procedure only needs be performed once, the computa- tion could become too slow and infeasible for extremely high-dimensional data (d > 10, 000) with our current implementation as shown in Appendix A.9 In the Appendix, we provide benchmark of the runtime of our implementation. Nevertheless, we think this problem could be resolved by a GPU implementation for clustering and mutual information. Additionally, although natural science datasets such as TCGA and V1 calcium imaging are considered large datasets in their respective domains, they are still small compared to the datasets used in computer vision. We expect larger natural science datasets to emerge as measuring and imaging technology advances. Additionally, adapting URLOST to support contrastive learning objectives, such as SimCLR, presents another intriguing direction.
7 ACKNOWLEDGEMENT
We would like to specially thank Bruno Olshausen for suggesting key prior work computational models and biological evidence for recovering the 2D topology of natural images. We also appreci- ate the helpful suggestions from Surya Ganguli and Atsu Kotani.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
[1] Harry G Barrow, Alistair Bray, and Julian ML Budd. A self-organizing model of "color blob" formation. Neural Computation, 8(7):1427-1448, 1996.
[2] Joel Bauer, Troy W Margrie, and Claudia Clopath. Movie reconstruction from mouse visual cortex activity. bioRxiv, pp. 2024-06, 2024.
[3] Yoshua Bengio, Yann LcCun, et al. Scaling learning algorithms towards ai. Large-scale kernel machines, 34(5):1-41, 2007.
[4] Lucas Beyer, Pavel Izmailov, Alexander Kolesnikov, Mathilde Caron, Simon Kornblith, Xiao- hua Zhai, Matthias Minderer, Michael T'scannenn, Ibrahim Alabdulmohsin, and Filip Pavetic. Flexivit: One model for all patch sizes. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 14496-14506 2023.
[5] Hermine Biermé. Introduction to random fields and scale invariance. Stochastic Geometry:
Modern Research Frontiers, pp. 129-180, 2019.
[6] Hermine Biermé. Introduction random fields and scale invariance. Stochastic Geometry: Modern Research Frontiers, pp. 129-180, 2019.
[7] Michael M Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Vandergheynst. Geometric deep learning: going beyond euclidean data. IEEE Signal Processing Magazine, 34 2017.
[8] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language mod- els are few-shot learners. Advances neural information processing systems, 33:1877-1901, 2020.
[9] Chen Cai and Yusu Wang. note on over-smoothing for graph neural networks. arXiv preprint arXiv:2006.113 13318, 2020.
[10] Deli Chen, Yankai Lin, Wei Li, Peng Li, Jie Zhou, and Xu Sun. Measuring and relieving the over-smoothin problem for graph neural networks from the topological view. In Proceedings of the AAAI conference on artificial intelligence, 2020.
[11] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In International conference on machine learn- ing. pp. 1691-1703. PMLR, 2020.
[12] Ting Chen, Simon Komblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020.
[13] Yubei Chen, Dylan Paiton, and Bruno Olshausen. The sparse manifold transform. Advances in neural information processing systems, 31, 2018.
[14] Brian Cheung, Eric Weiss, and Bruno A. Olshausen. Emergence of foveal image sampling from learning to attend in visual scenes. In The Fifth International Conference on Learning Representations, 2016.
[15] Sungmin Cho. Raehyuk Jung, and Junseok Kwon. Spherical transformer. arXiv preprint arX1v:2202.04942, 2022.
[16] Simon Dahan, Logan ZJ Williams, Abdulah Fawaz, Daniel Rueckert, and Emma C Robinson. Surface analysis with vision transformers. arXiv preprint arXiv:2205. 2022.
[17] Saskia EJ de Vries, Jerome A Lecoq. Michael A Buice, Peter A Groblewski, Gabriel K Ocker, Michael Oliver, David Feng, Nicholas Cain, Peter Ledochowitsch, Daniel Millman, et al. A large-scale standardized physiological survey reveals functional organization of the mouse vi- sual cortex. Nature neuroscience, 23(1):138-1511 2020.

---

Published as a conference paper at ICLR 2025
[18] Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. Advances in neural information processing systems, 29, 2016.
[19] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. leee, 2009.
[20] J.S. Denker and Y. leCun. Natural versus "universal" probability, complexity, and entropy. In Workshop on Physics and Computation, pp. 122-127, 1992. doi: 10.1 10.1109/HYCMP.1199 1992. 615508.
[21] Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In North American Chapter of the Association for Computational Linguistics, 2019.
[22] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
[23] Richard Durbin and Graeme Mitchison. A dimension reduction framework for understanding cortical maps. Nature, 343(6255))6644-477, 1990. doi: 10.1038/343644a..
[24] Richard Durbin and Graeme Mitchison. A dimension reduction framework for understanding cortical maps. Nature, 343(6259);644-647,, 1990.
[25] Cameron T Ellis, Tristan Yates, Lena Skalaban, Vikranth R Bejjanki, Michael J Arcaro, and Nicholas B Turk-Browne. Retinotopic organization of visual cortex in human infants. Neuron, 109(16):26162260 2021.
[26] Stephen A Engel, Gary H Glover, and Brian A Wandell. Retinotopic organization in human visual cortex and the spatial precision of functional mri. Cerebral cortex (New York, NY: 1991), 122):181-192, 1997.
[27] Daniel Felleman and David C Van Essen. Distributed hierarchical processing in the primate cerebral cortex. Cerebral cortex (New York, NY: 1991), 1(1):1-47, 1991.
[28] Matthias Fey, Jan Eric Lenssen, Frank Weichert, and Heinrich Müller. Splinecnn: Fast ge- ometric deep learning with continuous b-spline kernels. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 869-877 2018.
[29] Han Gao, Xu Han, Jiaoyang Huang, Jian-Xun Wang, and Liping Liu. Patchgt: Transformer over non-trainable clusters for learning graph representations. In Learning on Graphs Confer- ence, pp. 27-1. PMLR, 2022.
[30] Charles D Gilbert and Wu Li. Adult visual cortical plasticity. Neuron, 75(2):250-264, 2012.
[31] Marzich Haghighi, Juan C. Caicedo, Beth A. Cimini, Anne E. Carpenter, and Shantanu Singh. High-dimensional gene expression and morphology profiles of cells across 28 00,000 genetic and chemical perturbations. Nature Methods, 19(12):1550-155) 2022. doi: 10.1038/
941592-022-01661
[32] Kai Han, Yunhe Wang, Jianyuan Guo, Yehui Tang, and Enhua Wu. Vision gnn: An image worth graph of nodes. Advances in neural information processing systems, 35:8291-8303, 2022.
[33] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9729-9738, 2020.
[34] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16000-16009, 2022.
12

---

Published as a conference paper at ICLR 2025
[35] Irina Higgins, Loic Matthey, Arka Pal. Christopher Burgess. Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual con- cepts with constrained variational framework. In International conference on learning rep- resentations, 2016.
[36] David H Hubel and Torsten Wiesel. Receptive fields, binocular interaction and functional architecture in the cat's visual cortex. The Journal of physiology. 160(1):106, 1962.
[37] Aapo Hyvärinen, Jarmo Hurri, and Patrick O Hoyer. Natural image statistics: A probabilistic approach to early computational vision., volume 39. Springer Science & Business Media, 2009.
[38] Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Catalin Tonescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, et al. Perceiver io: A general architecture for structured inputs & outputs. arXiv preprint arXiv:2107. 14795,
2021.
[39] Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Car- reira. Perceiver: General perception with iterative attention. In International conference machine learning. pp. 4651-4664. PMLR, 2021.
[40] Yaseen Jamal and Daniel Dilks. Rapid topographic reorganization in adult human primary visual cortex (vl) during noninvasive and reversible deprivation. Proceedings of the National Academy of Sciences, 1 117(2)): 1059-11067, 2020.
[41] Aditya Jonnalagadda, William Yang Wang, BS Manjunath, and Miguel Eckstein. Foveater: Foveated transformer for image classification. arXiv preprint arXiv: 105.7 2021.
[42] Michael Jordan, Zoubin Ghahramani, Tommi Jaakkola, and Lawrence K Saul. An intro- duction to variational methods for graphical models. Machine learning, 37:183-233, 1999.
[43] Teuvo Kohonen. Self-organized formation of topologically correct feature maps. Biological cybernetics, 43(1);59-69, 1982.
[44] Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. The cifar-10 10 dataset. online: http://www. cs. toronto. edu/kriz/cifar. html, 55(5), 2014.
[45] Xu Ma, Yugian Zhou, Huan Wang, Can Qin, Bin Sun, Chang Liu, and Yun Fu. Image as set of points. In The Eleventh International Conference on Learning Representations, 2023.
[46] Rahil Mehrizi, Arash Mehrjou, Maryana Alegro, Yi Zhao, Benedetta Carbone, Carl Fishwick, Johanna Vappiani, Jing Bi, Siobhan Sanford, Hakan Keles, et al. Multi-omics prediction from high-content cellular imaging with deep learning. arXiv preprint arXiv:2306.099 09391, 2023.
[47] Ronald Meyer. Tetrodotoxin inhibits the formation of refined retinotopography in goldfish. Developmental Brain Research, 6(3):293-299 1983.
[48] Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant rep- resentations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6707-6717, 2020.
[49] Federico Monti, Davide Boscaini, Jonathan Masci, Emanuele Rodola, Jan Svoboda, and Michael M Bronstein. Geometric deep learning on graphs and manifolds using mixture model cnns (2016). In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2016.
[50] David Mumford and Agnès Desolneux. Pattern theory: the stochastic analysis of real-world signals. CRC Press, 2010.
[51] Andrew Ng, Michael Jordan, and Yair Weiss. On spectral clustering: Analysis and an algo- rithm. Advances in neural information processing systems, 14, 2001.
[52] Klaus Obermayer, Helge Ritter, and Klaus Schulten. A principle for the formation of the spatial structure of cortical feature maps. Proceedings of the National Academy of Sciences, 87(21): 8345-8349, 1990.
13

---

Published as a conference paper at ICLR 2025
[53] Haluk Ogmen and Michael H Herzog. The geometry of visual perception: Retinotopic and nonretinotopic representations in the human visual system. Proceedings of the IEEE, 98(3): 479-492, 2010.
[54] Bruno Olshausen and David Field. Emergence of simple-cell receptive field properties by learning sparse code for natural images. Nature, 38 (6583))6077-609) 1996.
[55] Bruno A Olshausen and David Field. How close are we to understanding vi? Neural computation, 17(8):6665-169, 2005.
[56] Bruno A Olshausen and David J Field. What is the other 85 percent of v1 doing. L van Hemmen, & T. Sejnowski (Eds.), 23:182 18-21112,
211, 2006.
[57] Marius Pachitariu, Carsen Stringer, Sylvia Schröder, Mario Dipoppa, L Federico Rossi, Matteo Carandini, and Kenneth D Harris. Suite2p: beyond 10,000 neurons with standard two-photon microscopy. BioRxiv, pp. 06 061507, 2016.
[58] Eli Peli, Jian Yang, and Robert Goldstein. Image invariance with changes in size: The role of peripheral contrast thresholds. JOSA A, 1(11):162211774 1991.
[59] Jeffrey S Perry and Wilson S Geisler. Gaze-contingent real-time simulation of arbitrary visual fields. In Human vision and electronic imaging VII, volume 4662, pp. 57-69. SPIE, 2002.
[60] JS Pointer and RF Hess. The contrast sensitivity gradient across the human visual field: With emphasis on the low spatial frequency range. Vision research,
1989.
[61] Jonathan R Polimeni. Bruce Fischl, Douglas N Greve, and Lawrence L Wald. Laminar analysis of bold using an imposed spatial activation pattern in human vl. Neuroimage, 52(4):1334- 1346, 2010.
[62] Daniel Probst and Jean-Louis Reymond. Visualization of very large high-dimensional data sets as minimum spanning trees. Journal of Cheminformaticc, 12(1):12, 2020. doi: 10. 186/ 1-321-020-0416--1
[63] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018.
[64] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763, PMLR, 2021.
[65] Alexander G Ramm. Random fields estimation theory. Mathematical and Computer Mod- elling, 13(4):87-100, 1990.
[66] Austin Roorda and David R Williams. The arrangement of the three cone classes in the living human eye. Nature, 39767799:520-522, 1999.
[67] Nicolas Roux, Yoshua Bengio, Pascal Lamblin, Mare Joliveau, and Balázs Kégl. Learning the topology images. In Platt, D. Koller, Y. Singer, and Roweis (eds.), Advances in Neural Information Processing Systems, volume 20. Curran Associates, Inc., 2007.
[68] Rosanna P Sammons and Tara Keck. Adult plasticity and cortical reorganization after periph- eral lesions. Current Opinion in Neurobiology. 35:136-1411 2015.
[69] Han Shi, Jiahui Gao, Hang Xu, Xiaodan Liang, Zhenguo Li, Lingpeng Kong, Stephen Lee, and James T Kwok. Revisiting over-smoothing in bert from the perspective of graph. arXiv preprint arXiv:2202.000 08625, 2022.
[70] David Shuman, Sunil K Narang, Pascal Frossard, Antonio Ortega, and Pierre Vandergheynst. The emerging field of signal processing graphs: Extending high-dimensional data analysis to networks and other irregular domains. IEEE signal processing magazine, 30(3):83-98,
2013.
14

---

Published as a conference paper at ICLR 2025
[71] Eero P Simoncelli and Bruno A Olshausen. Natural image statistics and neural representation.
Annual review of neuroscience, 24(1):1193-1216, 2001.
[72] X Yu Stella and Jianbo Shi. Multiclass spectral clustering. In Computer Vision, IEEE Interna- tional Conference on, volume 2, pp. 313-313. IEEE Computer Society, 2003.
[73] Carsen Stringer, Marius Pachitariu, Mattco Carandini, and Kenneth Harris. Recordings of
10,000 neurons in visual cortex in response 2,800 natural images. Figshare Repos, 2018.
[74] Carsen Stringer, Marius Pachitariu, Nicholas Steinmetz, Matteo Carandini, and Kenneth D
Harris. High-dimensional geometry of population responses in visual cortex. Nature, 571 (7765):361-353 July 2019.
[75] Carsen Stringer, Michalis Michaelos, Dmitri Tsyboulski, Sarah E. Lindo, and Marius Pachi- tariu. High-precision coding in visual cortex. Cell, 184(10))777--778..15,, 2021. ISSN 0092-8674. doi: https://doi.orr/10..
[76] Nicholas V. Swindale and Hans-Ulrich Bauer. Application of kohonen's self-organizing fea- ture map algorithm to cortical maps of orientation and direction preference. Proceedings: Biological Sciences, 265 (555(1198:827--8800 838, 1998. ISSN 09628452.
[77] Michael Taylor. Random fields: stationarity, ergodicity, and spectral behavior. URL
http://www. unc. cdub/nati/aacullyyee//mmnoo pdf. 2018.
[78] Larry N Thibos. Acuity perimetry and the sampling theory of visual resolution. Optometry and vision science: official publication of the American Academy of Optometry, 75(6):399- 406, 1998.
[79] Katarzyna Tomezak, Patrycja Czerwinska, and Maciej Wiznerowicz. Review the can- cer genome atlas (tega): an immeasurable source of knowledge. Contemporary Oncol- ogy/Wspótezesna Onkologia, 2015(1):68-77, 2015.
[80] David Van Essen, Charles Anderson, and Daniel Felleman. Information processing in the primate visual system: an integrated systems perspective. Science, 255(5043):419-42, 1992.
[81] Erik Vanmarcke. Random fields: analysis and synthesis. World Scientific, 2010.
[82] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[83] Brian Wandell, Serge Dumoulin, and Alyssa A Brewer. Visual field maps in human cortex. Neuron, 56(2):366-33,, 2007.
[84] John N Weinstein, Eric A Collisson, Gordon B Mills, Kenna R Shaw, Brad A Ozenberger, Kyle Ellrott, Ilya Shmulevich, Chris Sander, and Joshua M Stuart. The cancer genome atlas pan-cancer analysis project. Nature genetics,
2013.
[85] Konstantin Willeke, Paul G Fahey, Mohammad Bashiri, Laura Pede, Max F Burg, Christoph Blessing, Santiago A Cadena, Zhiwei Ding, Konstantin Meastannti-KKemens Lurz, Kayla Ponder, et al. The sensorium competition on predicting large-scale mouse primary visual cortex activity. arXiv preprint arXiv:2206. 00600000066666... 2022.
[86) Rachel OL Wong. Retinal waves and visual system development. Annual review of neuro- science, 22(1):29 1999.
[87] Zhirong Wu, Yuanjun Xiong, Stella X. Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
[88] Johnathan Xie, Yoonho Lee, Annie Chen, and Chelsea Finn. Self-guided masked autoen- coders for domain-agnostie self-supervised learning. arXiv preprint arXiv:2402. 14789, 2024.
15

---

Published as a conference paper at ICLR 2025
[89] Youzheng Xu, Yixin Xu, Chun Wang, Baoguo Xia, Qingling Mu, Shaohong Luan, and Jun Fan. Mining tega database for gene expression in ovarian serous cystadenocarrcioma mi- croenvironment. PeerJ, 9:e 1375, 2021.
[90] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self- supervised learning via redundancy reduction. International Conference on Machine Learn- ing. pp. 123 12310-12320. PMLR, 2021.
[91] Xiaoyu Zhang, Jingqing Zhang, Kai Sun, Xian Yang, Chengliang Dai, and Yike Guo. Inte- grated multi-omies analysis using variational autoencoders: Application to pan-cancer classi- fication. In 20/9 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pp. 765-769, 2019. doi: 10.1 D0.1009/BBBM44725200090599 88832228
[92] Xiaoyu Zhang, Yuting Xing, Kai Sun, and Yike Guo. Omiembed: A unified multi-task deep learning framework for multi-omics data. Cancers, 13(12), 2021. ISSN 2072-6694. doi: 1..399//ancerrs11133000.
A APPENDIX
A.1 MOTIVATION OF DENSITY ADJUSTED SPECTRAL CLUSTERING
Using the terminologies in functional analysis, the mutual information graph defined in section 2.2 corresponds to a compact Riemannian manifold M and the Laplacian matrix discrete anal- ogous to the Laplace Beltrami operator L on M.. Minimizing the spectral embedding objective tr LY)) directly corresponds to the following optimization problem in function space:
/ ||V/]|2dx
min |lf
(5)
where f(x) : M - (0, 1] is the normalized signal defined on M. We particularly want to write out adapt it to non-stationary signals. To do so,
the continuous form of spectral embedding so we assume the measure is absolutely continuous with respect to standard measure μ. By apply the Radon Radon-Nikodym derivative equation 5
we get:
S. |IVSIPdA= /. |ISII
dA du
dy
dA is called the Radon-Nikodym, which is some form of density function. Let where the quantity dy p(x)= dy-
we can rewrite the optimization problem as the following:
min
[Il(=)
(6)
The density function p(x) on the manifold is analogous to the density adjustment matrix in equa- tion [ Standard approaches in equation 5 assumm that nodes are uniformly distributed on the man- ifold, thereby treating p(z:)
constant and excluding it from the optimization process. However, this assumption does not hold in our case involving non-stationary signals. Our work introduces a variable density function p(æ:) for each signal, making it pivotal component in building good rep- resentations for non-stationary signals. This component is referred to as Density Adjusted Spectral Clustering. Empirical evidence supporting this design is provided through visualization and ablation studies in the experimental section.
A.2 SPECTRAL CLUSTERING ALGORITHM
Given a high dimensional dataset S 7 Let S be ith column of which represents the ith dimension of the signal. We create probability mass functions P(S:) and P(Sş) and the joint distribution P(S,, for S; and S, using histogram. Let the number of bins be K. Then we measure the mutual information between P(S.) and P(S,) as:
K K
P(S,, S,) (I)
I[S;S)) = Σ Σ
P(S,, S,),, loga
16

---

Published as a conference paper at ICLR 2025
Let A. S,, be the affinity matrix, and let the density adjustment matrix be P defined in 2.2 Correlation is used instead of mutual information when the dimension is really high since computing mutual information is expensive. We follow the steps from [T to perform spectral clustering with a modification to adjust the density:
1. Define D to be the diagonal matrix whose (i.))element is the sum of A's i-th row. Con- struct the matrix L = P D AD- P!.
2. Find 21, :.. Ikk- the largest eigenvectors of L, and form the matrix X E Ruxk by stacking the eigenvectors in columns.
[1), ("2," ,3k]]
3. Form the matrix Y from X by renormalizing each of X's rows to have unit norms, (i.e.
Yo = xx,
4. Treating each row of Y as point in R² cluster them into & clusters via K-means or other algorithms.
Some other interpretation of spectral embedding allows one design specific clustering algorithm in step 4. For example, (72) interprets the eigenvector problem in olas relaxed continuous version of K- K-way normalized cuts problem, where they only allow X be binary, i.e. X {0, 1) N*K, This is an NP-hard problem. Allowing X to take on real value relaxed this problem but created degeneracy solution. Given solution X and Z D Xx*, for any orthonormal matrix R, RZ is another solution to the optimization problem Thus, [72 designed an algorithm to find the optimal orthonormal matrix R that converts X to discrete value in {0, N From our experiment, [72] is more consistent than K-means and other clustering algorithms, so we stick to using it for our model.
A.3 DATA SYNTHESIZE PROCESS
We followed the retina sampling approach described in 14 to achieve foveated imaging. Specifi- cally, each retinal ganglion cell represented using Gaussian kernel. The kernel parameterized by its center, denoted as Z. and its scalar variance, of i.e. Ni,, (22), which is illustrated in Figure A. The response of each cell, denoted as G[i),, is computed by the dot product between the pixel value and the corresponding discrete Gaussian kernel. This can be formulated as:
c ΣΣ K(T, or,, [ m]
where N and W are dimensions of the image, and / represents the image pixels.
For foveated CIFAR-10, since the image very low resolution, we first upsample it times from 32 32 to 96 96, then use in total of 1038 Gaussian kernels sample from the upsampled image. The location each kernel is illustrated Figure B. The radius of the kernel scales proportionally to the eccentricity. Here, use the distance from the kernel to the center to represent eccentricity. The relationship between the radius of the kernel and eccentricity is shown in Figure 5 As men- tioned in the main paper. in the natural retina, retinal ganglion cell density decreases linearly with eccentricity, which makes the fovea much denser than the peripheral, unlike the simulated lattice we created. The size of the kernel should scale linearly with respect to eccentricity as well. However, for the low-resolution CIFAR-10 dataset, we reduce the simulated fovea's 's density to prevent redun- dant sampling. this case, we pick the exponential scale for the relationship between the size the kernel and eccentricity so the kernel visually covers the whole visual field. We also implemented convolution version of the Gaussian sampling kernel to speed up data loading.
A.4 DENSITY ADJUSTED SPECTRAL CLUSTERING ON FOVEATED CIFARIO DATASET
We provide further intuition and visualization on why density-adjusted spectral clustering allows the model to learn a better representation of the foveated CIFAR- 10 dataset.
As shown in F the kernel at the center is much smaller in size than the kernel in the peripheral. This makes the kernel at the center more accurate but smaller, which means it summarizes less information. Spectral clustering with constant density will make each cluster have similar number
17

---

Published as a conference paper at ICLR 2025
A
A
B
Relationship between Eccentricity and Kernel Radius
Nim; 1 6)
Redius of the Kernel
Distance of the kernel to the center
Figure 5: Foveated retinal sampling (A) Illustration Guassian kernel shown in 14 Diagram of single kernel filter parameterized by mean μ' and variance a'. (B) the location of each Gaussian kernel summarized as a point with 2D coordinate μ'. In total, the locations of 1038 Gaussian kernels are plotted. (C) The relationship between eccentricity (distance of the kernel to the center)
and radius of the kernel is shown.
Table 5: Evaluation foveated CIFAR-10 with varying hyperparameter for density function. For each set of values of a and 8, we perform censity-adjusste spectral clustering and run URLOST with the corresponding cluster. The evaluation of each trial is provided the table.
beta 0
beta 2
alpha 0
82.74 %
84.24 %
alpha 0.5
84.52 %
85,43 %
alpha 1.0
83.83 %
81.62 %
of elements in them. Since the kernel the center is smaller, the cluster in the center will be visually smaller, than the cluster in the peripheral. The effect is shown in Figure Moreover, since we't re upsampling an already low-resolution image (CIFAR-10 10 image), even though the kernel at the center is more accurate, we're not getting more information. There, make sure each cluster has similar information, the clusters in the center need to have more elements than the clusters in the peripheral. In order make the clusters the center have more elements, we need weight the clusters in the center more with the density function. Since the sampling kernels at the center have small eccentricity and are more correlated their neighbor, increasing a and 3 will make sampling kernels at the center have higher density, which makes the cluster at the center larger. This is why URLOST with density-adjusted spectral clustering performs better than URLOST with constant density spectral clustering, which is shown in Table 5] Meanwhile, setting and / too large will also hurt the model's performance because it creates clusters that are too unbalanced.
A.5 SEL.-ORGAAIIING LAYER LEARNS INVERSE PERMUTATION
For locally-permuted CIFAR-1O, divide each image into patches and locally permute all the patches. The i-th image patch is denoted by (1),, and its permuted version, permuted by the permu- tation matrix E(i), is expressed (1)) We use linear layers to parameterize the self-organe self-organizing
layers. Let ((,, W W denotes the ith element of the self-organizing layer. We're provid- ing the proof for the statement related to the visual evidence shown in Section 4.2
Statement: the self organiiggn layer effectively aligns the input sequence, then E ( should exhibit visual similarities.
Proof: we first need formally define what it means for the self-organizing layer effectively align the input sequence. Let e₁ denote the kth natural basis (one-hot vector at position k), which represents the pixel basis at location k. Permutation matrix E will send kth pixel to some location accordingly. Mathematically, if the projection layer effectively aligns the input sequence, means g( ((FU)) ex. W o( (F(00g ex. W()) for all i, k. We can further expand this property to get the
18

---

Published as a conference paper at ICLR 2025
A
B
Figure 6: Efffct density adjusted clustering. Sccentricity-basee sampling lattice. The center of the sampling lattice has more pixels which means higher resolution compared to the peripheral. (A) Result of density-adjusted spectral clustering (a = 0.5, 8 = 2). Clusters in the center have more elements than clusters in the peripheral. But clusters look more visually similar in size than B. (B) Result of uniform density spectral clustering (a 0, B = 0). Each cluster has similar number of elements in them but the clusters in the center are much smaller than the clusters in the periphery.
following two equations:
(E W()) W() p:0 g(E eN, w()) W EUDe
for all i, k. Since the above equation holds for all ex. by linearity and the property of permutation matrix, we have:
1 = w()
goXWO = wG)
This implies EGTT w(() should exhibit visual similarities for all
A.6 TRAINING AND EVALUATION DETAILS
3-VAE. 3-VAE was trained for 1000 epochs and 300 epochs on the VI neura and TCGA gene expression respectively. We use the Adam optimizer with a learning rate of 0.001 and cosine an- nealing learning rate scheduler. The encoder composed a 2-layer MLP with batch normalization and LeakyReLU activation. We use hidden dimensions 2048 and 1024 for V1 and Gene datasets respectively. Then, two linear layers are applied get the mean and standard for reparameterization. The decoder also has 2-layer MIP, symmetric to the encoder but using standard ReL.U activation and no batch normalization. We tried out different hyperparameters and empirically found that this setting gave the best performance.
MAE. MAE follows the official implementation from the original paper. For CIFARIO, we ran model for 10,000 epochs. We use Adam optimizer with a learning rate of 0.00015 and cosine annealing. To fit in our tasks, we use 8-layer encoders and 4-layer decoders with hidden dimension 192. The ViT backbone (iT(lixel) means treating each pixel as a patch, so essentially the patch size is This is also used for the real-world high-dimensional dataset since concept of patch is defined in the signal space.
take different patch sizes and indicated them accordingly in Table
19

---

Published as a conference paper at ICLR 2025
For V1 neural recording and TCGA gene expression task, use 4 layers encoder and 2 layers. We use hidden dimension 1380 for 1000 epochs and hidden dimension 384 with 3000 epochs for V1 used
neural recording and TCGA dataset task. The hidden dimension and the number of epochs for MAE are greater than V-V.F. However, when we use the same parameters on B-VAE, we did not seem to find performance gain. Training Transformers usually requires large number of data. For example, the original transformer on vision is pretrained over 14M images.
URLOST MAE Many hyperparameters are associated with the spectral clustering algorithms. The performance of the model depends on the cluster, which thus depends on these hyperparameters. In general, URLOST not very sensitive these hyperparameters. Density parameters (how nodes are normalized) and the number of clusters affects the shape of the clusters, thus, affects the performance of the model to some degree.
Intuition on hyperparameter selection: Both these hyperparameters are related to the size of each cluster, which relates to how difficult the task is. If cluster is too big, then we need to predict a big missing area on the graph, making the unsupervised learning task too difficult. On the other hand, the cluster is too small, for example, the cluster shrinks down to one pixel for an image, then the prediction tasks become too easy. The solution is simply doing low-pass filter on the graph to fill in the missing pixel. So the model will not learn high-level or semantic representation. Both the number of clusters and density factors related to the cluster size. Increasing the number of clusters could make each cluster smaller. The density factor essentially defines how much we want to normalize each node the graph. For image dataset, clusters should contain semantic sub-structures of the image. like body parts or parts of objects. We use 64 clusters, which results in clusters of size roughly 4x4. In other words, the ratio between the size of clusters (part) and the number of dimensions (whole) is roughly 1:64, For other datasets (gene and V1), we roughly keep this ratio and perform grid search over these hyperparameters. Similarly, for the density factors, we also perform a grid search centered at the density factor equal to 1. Empirically, URLOST is not very sensitive too these hyperparameters as shown in Figure
We also provide the exact hyperparameters we used for the experiments in this paper: the parameter of URLOST MAE is the same as MAE except for the specific
in the method section. For CIFARIO, we use K 20, 0.5 and 3 2. We set the number of clusters to be 64. For VI neural recording, we use K = 15, a = and B = 1. We set the number of clusters to 200. For TCGA dataset, use K 10,
and B 1. We set the number of clusters to 32.
A.7 EFFECT OF SPECTRAL CLUSTERING HYPERPAR AMPEERS ON PERFORMANCE
We perform an experiment doing a grid search over each hyperparameter in spectral clustering that affects the performance of URLOST. The grid search experiment is performed over the TCGA gene classification dataset. As shown in Figure the model is not sensitive for most of the hyper- parameters. For a large range of hyperparameters, the model performance stays the same.
A
B
Accuracy vs Number of Clusters
Accuracy vs bots
Accuracy vs
100
100.0
297
Phoze Accusacy
P1b0- AcCurac
00.0
20.5
10
23
60
150
21
Rumber of Clusters
bets
Figure 7: Effect of spectral clustering hyperparameters on performance (A) Effect of number of cluster performance.
(B) Effect of normalization parameter for spectral clustering (3 in Eq. on performance. (C) Effect of numbers of neighbors from nearest neighbors on performance.
A.8 TOPOLOGY AND STATIONARITY
We provide formal definition of topology and stationarity used in the context of our work here:
20

---

Published as a conference paper at ICLR 2025
On topology of the domain of signals. A signal is function X(s) defined some domain S, where $ E S. Let's focus on S. While the definition of topology is a little abstract and relies on open sets, the intuition is that it defines generalized sense of nearness, whether two points are close to each other or far apart. A common way to give such a space the property of being near or far away is with an explicit distance function, objects-opor sets. These spaces (with an associated metric) are called metric spaces, For example, the topology of image space is induced by the L2 norm as the metric function. However, not every space will have a natural metric on it. This is why we need to use the term "topology".
metrie. The distance function generates class of
Topological spaces are generalizations of metric spaces. The definition of topology is any collec- tion of subsets of S that satisfies certain axioms: close under arbitrary union and finite intersection. In the practical setting, the domain S is always discretized as a set of indices, and we can assume that we always deal with a signal defined on a graph 77). Graphs can be embedded in R For a fi- nite graph, the graph topology is just the subspace topology inherited from the usual topology in IR3. Under this specific definition, the word "graph," "adjacency matrix," " and "topology" can be used interchangeably. In this work, we assume we are not provided the topology. Thus, using pairwise mutual information as the adjacency matrix is way to model the topology. Correlation is another way to model the topology, where we use only second-order pairwise statistics. This is slightly dif- ferent from the reviewer's understanding "Topology appears to correspond to "correlation" between dimensions." " More generally, graphs are topological spaces that are the simplicial 1 1-complexes or the I-dimensional CW complexes.
Example with stationarity but no topology: Permuted Cifarl0 is an example we had in the manuscript for this criterion (no topology but has stationarity). No topology refers the fact that n(i) is not defined after random permutation. "Has stationarity" means that the signal is stationary when the topology is recovered. Specifically, we use the mutual information graph to define neigh- bors n(i). Under this particular definition of ri))), there exists meaningful invariant statistic ()) for the signal. Thus, the signal is stationary. This because the topology recovered is very similar to the original 2d topology of images before permutation, and the sign before permutation is stationary. However, under an incorrect definition of nii)., the signal is no longer stationary. Example with no topology but no stationarity: One example would be foveated images simulated using convolutional kernels. One could make a different region of an image have a different reso- lution while preserving the 2d grid structure of the image by using Laplace pyramid. As result, the image is still represented as a 2d matrix, and the definition of nt() is the same as regular images. However, many invariant statistics before the foveation process are destroyed due to the imbalance resolution at different regions of the image.
On stationarity. In probability theory, random process is family of random variables. While the index was originally and frequently interpreted as time, it was generalized to random fields, where the index can be an arbitrary domain. That is, by modern definitions, a random field is generalization of stochastic process where the underlying parameter need no longer be real or integer valued "time" but can instead take values that are multidimensional vectors in I", points
65 5) Let's consider real-valued random field (s) defined on a homogeneous space S =s of points 8 equipped with a transitive transformation group G =g of mappings of S into itself, and having the property that the values of the statistical characteristics of this field do not change when elements of G are applied to their arguments. In convention, we call random field X(s) strict-sense stationary (or strict-sense homogeneous) random field if for all n = 1,2,... ... and 9 E G. the dinie--dimeniioal probability distribution of its values at any points $1,- Sn, coincides with that of its values gs1..... We call X (s) field wide-sense stationary (or wide-sense homogeneous) random field if X [2 inf and Ex = EX (gs), (x(s)X (s1) EX (gs) EX (gsl) for all 8, 81 E S and y E G. Or sense stationarity means the first and second-order statistics do not change w.r.t. the group action. Further, we can intuitively generalize the wide-sense stationarity to higher-order statistics if we need to define any nth-order stationarity between strict-sense and wide-sense.
on some manifolds and Lie Groups, graphs [8 50 77 42
g9Sn-
In intuitively, stationarity just means the statistics does not change w.r.t. transformations. Let's take the image as an example, it is signal X (() defined on S = Z And let's 's define the group action as translation. If we use a convolution neural network and learn a set of L x filters or use a VIT to learn set of shared features on Lx L patches. A hidden assumption is that the statistics of our image does not change for L region, or in order words the joint probability (1) X(L, L)) (( + a, 1 + b), ( + a). .................... X(L b)). where (a, b) E translation group T. While
21

---

Published as a conference paper at ICLR 2025
Table 6: Ratio between Parameters of Self-Organizing Layers and Parameters of the Entire Model. We report the model used for each dataset.
Dataset
Permuted Cifarl0
TCGA
V1
Ratio
6.7%
15.6%
5.9%
we have initially cited [8], it might be a good idea to provide a more thorough explanation in the appendix, hence, we have included another five references provide more comprehensive point of view.
A.9 COMPUTATIONAL COST
Computation cost. The computational cost and scalability of URLOST are nearly the same as MAE. The computational cost of MAE analyzed in the original MAE paper. Its efficient design of MAE speeds up the training and evaluation time around 4x compared to standard Transformer- based auto-encoder. The additional elements in URLOST are simply clustering and self-organizing layers. This effect is shown in figure 7 despite the encoder used in the experiment being times as the decoder, the runtime of the encoder still faster than the decoder. Moreover, the operation of clustering indexing and self organizing layer is very small relative to the overall runtime of the model. The mutual information and the clustering results are precomputed based on the dataset and thus do not introduce more cost for the model training. During training and inference, clustering is just a one-step indexing process that does not create significant overhead in additional to MAE. For all experiments performed in this paper, since clustering and mutual information are computed and saved for all datasets at once, their computational cost is negligible with respect to the training time of MAE. However, The runtime of these two procedure could be problem for extremely high dimensional data.
With the current implementation, we perform scalability benchmark as suggested by the reviewer. We resize an image dataset to different resolutions. At each resolution, we compute mutual informa- tion and clustering for a different number pixels (dimensions), and we provide plot of running time with respect to the number of pixels in Figure Both computational cost of computing pair- wise mutual information and clustering both scale quadratically with the number of dimensions of the signal. It takes aroudn 130 minutes to calculate mutual information for data with 10,000 dimen- sions. Clustering takes much shorter time (less than minute).
A
Runtime vs Number of Pixels for clustering
B
Runtime Number Pixels for computing pairwise
Jotal Runtime (minutes)
Total Runtime (mnnutes)
0:34
280
690
Number of Pixels
Number of Pixels
Figure 8: Runtime for clustering and computation mutual information. A. Benchmark of run- time for clustering for signal with different dimensions. B. Benchmark of runtime for computing pairwise mutual information with different dimensions. Both run time scale quadratically as the dimension of the signal.
Amount of parameters. The self-organizing layer introduces additional parameters compared to the original MAE model. We believe additional parameters reasonable trade-off for not knowing topology and stationarity, which are important inductive biases the data. However, it doesn t increase the parameters drastically, provide the ratio of the additional parameters in table
22

---

Published as a conference paper at ICLR 2025
Table 7: Runtime of each module during the forward operation. We report the runtime of each module during inference for single example. We use the model trained on permuted CIFARIO dataset. The experiment performed using a single RTX 2080TI and is averaged over 500 trials.
Module
Cluster Indexing
Self Organizing Layer Encoder
Decoder
Runtime
0.27 ms
1.11 ms
17.5 ms
20.4 ms
A. SCALABILITY
Scalability is not the primary focus of this work. In general, unsupervised representation learning benefits from scaling larger amount of data, and the MAE model, which method is built upon, demonstrates a level of scaling [34]. We believe URLOST inherits this potential for the fact that it shows stronger advantages simpler benchmarks. This suggests that scaling to larger and more complex datasets is a promising direction. Our study explores the novel settings of unsupervised representation learning without knowing the data topology and stationarity, demonstrating the effectiveness of our method across diverse data modalities with significant differences. This generalization is our key and positive contribution, highlighting the potential of our approach to inspire future work in this area.
more challenging tasks, as seen with CIFAR-10 compared to
A.11 NOISE ROBUSTNESS OF MUTUAL INFORMATION RECOVERED TOPOLOGY
We tested the robustness of the topology recovery algorithm used in URLOST. Specifically, we add gaussian noise to CIFAR10 to generate data with a low signal-to-noiss ratio.
We found out that while decreasing signal-to-noie ratio will lower the overall mutual informa- tion between pairs of pixels, the relationship between mutual information between pixels and im- age topology (distance between two pixels) does not change. We perform the following exper- iments: add iid gaussian noise to each pixel with fixed noise level, then measure the mutual information between pairs of pixels as M)((i,)), ()). For visualization, we aggregate all pairs by the distance between them and average their mutual information, i.e. let A(a) (i,1), (kt, (k., k + (j 1 1)2. We have MI(a) Ι;ंj EXJ.AEEA()) (k; (a) denotes the mutual information between pixels of different distances.
As shown in Figure 9 below, although the overall mutual information decreases lot the noise level increases, the order of MI(a) is still the same. In other words, the mutual information still the biggest when the two pixels nearby and decays when the mutual information decreases. This implies the topology recovered by pairwise mutual information is robust even when the signal has lower signal-to to-ise ratio. It only breaks when you add too much noise (o 0.9). For this amount of noise, the image is not distinguishable.
1.12 SELF-ORGANIZING MAP (SOM)
Kohonen in 1982 presents fascinating thought experiment [3: : "Suppose you woke up one day to find someone rewired your optic nerve (or you have been implanted with prosthetic retina). The signals from the retina to the brain are intact, but the wires are all mixed up, projecting to the wrong places. Can the brain learn to "descramble" the image?" Kohonen proposes the "self- organizing map" (SOM) algorithm to address this problem. SOM is an unsupervised machine learn- ing technique used to produce a low-dimensional (typically two-dimensional) representation of higher-dinee dimensional dataset while preserving the topological structure of the data. In the task of "de- scrambling" pixels, SOM maps each pixel into a 2D index set by leveraging the mutual information between pixels at different locations. The parameters of SOM include a set of neurons, where each neuron has location index. Let W. denote the weight vector of the uth neuron and D the input vector. f(u, v, s) is the interaction function that determines the influence between the uth and wth neurons based on their distance. The update equation for SOM is given by:
W.(s (8- + 1) = (.(s)) + 0(u,, v) as)) (D(t)= W.))))
In the task of "descrambling" pixels, the input consists pixel values. The indices will be two- dimensional, such (i,5), and W, Ws(s) will learn to represent set of "descrambled" pixels,
23

---

Published as a conference paper at ICLR 2025
Mutual Information Pixel Distance for Different Noise Levels
Noise Level
0.0
0.6
0.1
0.2
Average Mutual Information
0.5
0.3
0 = 0.4
0.5
0.4
0.9
0.2
0.0
0.0
2.5
5.0
7.5
10.0
12.5
15.0
17.5
20.0
Pixel Distance
Figure 9: The effect of additive noise pairwise mutual information. We plot averaged mutual information between pairs pixels. We aggregate all pairs with the same distance. Each curve represents specific noise level. Although the curve becomes lower noise increases, the decaying features of the curve remain the same.
where s represents the correct indices of the pixels. In other words, the index set defines the correct topology of the image as a 2D grid, and SOM maps the pixels onto this 2D topology by leveraging the mutual information between pixels at different locations. Other methods such [67] use mani- fold learning to uncovered topology 2-d topology from natural images. However, when the intrinsic dimension of the manifold is larger than 2d, it is difficult to integrate the "uncovered" topology with stateoof-the-al self-supervised learning algorithms.
A.13 VISUALIZING THE WEIGHTT OF SELF-ORGANIZING
As explained the previous section (Appendix A.5) and visualized Figure 10 we can visualize the weights of the leamed self orfaniigan layer when trained on the locally-permutt
CIFAR-10 dataset. If we apply the corresponding inverse permutation E to its learned filter W/(1) at position the pattern should show similarity across all position This is because the model is trying to align all the input clusters. We have shown this is the case when the model converges to a good representation. On the other hand, what if we visualize the weight E() (+) as training goes on? If the model learns to align the clusters as it is trained for the mask prediction task, g())T((> should become more and more consistent as training goes which confirms our hypothesis. As training goes the pattern EG)T (i) becomes more and more visually similar, which implies the model learns to gradually learn to align the input clusters.
We show this visualization in Figure IO]
24

---

Published as a conference paper at ICLR 2025
Figure 10: Visualize the weight of the self-organizing layer after applying inverse permutation. A snapshot of (O)) is shown at different training epoch. The number of epochs is shown the top row. Each figure shows one column of the weight of the self-organizing layer, at different positions, i.e. (!!)) where & is the column number and is the position index. In total, 9 columns are shown.
25


Reviews:
[{"review_id": "6vSjSqNyvb", "reviewer": "Reviewer_sHq9", "summary": "The paper investigates the topic of self-supervised representation learning and how to extend current popular frameworks to data modalities beyond natural images or text for which these methods have been initially designed. This is particularly relevant for the processing of scientific data which at this point would be hard to integrate into frameworks like MAEs. The authors propose to add a preprocessing block before the classic MAE which allows to aggregate data into the correct format even in the absence of prior information about the data's structure. This preprocessing blocks involves spectral clustering with density adjustment followed by a self-organizing map embedding layer. The authors show that the approach is more effective when process high-dimensional unstructured real-world data (2 natural science datasets + a modified version of CIFAR10) than applying the standard MAE framework to this data.", "questions": "- can authors elaborate on why a set of linear projectors (non-shared in this case, a different layer for each cluster) cannot replace the use of self-organising maps? seems like the point here is that operation should not be shared between clusters rather than the type of operation used.\n- how were representations trained in table 3? standard MAE? \n- can authors provide numbers for 1) URLOST with CIFAR10 2) MAE (Patch) with Permuted CIFAR and Foveat CIFAR; \n- can authors explain by such a high number of epochs is needed (10,000), is this only necessary for URLOST, prior work recommend 800-1600 epochs for standard MAEs. \n- what is the dimensionality of the representation in the beta-VAE vs the MAE; how do the size of models compare? the VAE used seems very shallow. \n- what is the range of beta parameter that was considered? \n- is the MAE in table 2 also with patch size 4? \n- what is the masking ratio in the MAE and URLOST (75%), can authors provide a comparison with 0% masking to show a non-masking scenario with equivalent architecture ?\n- how is k selected ? how variable are results _across datasets_ for varying k, alpha, and beta parameters ? \n\nMinor: \n- typo in line 55\n- bold number in table 1 for CIFAR10 is wrong, should be SimCLR", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- relevance: this paper aims at extending existing SSL methods to novel modalities (notably modalities found in various scientific fields) for which current methods and their associated architecture of choice (e.g., MAE with ViT backbones) are not adapted because designed to suit standard data modalities (e.g., images) with known structure. This topic is relevant as SSL methods are often shown to be powerful but designed (both joint embedding methods and masked image modeling) to suit specific modalities which limit their applicability to a broader range of data. Even for standard modalities like natural images, approaches such as MAE rely on assumptions that might not generalize across datasets. \n- presentation: paper is well written and easy to follow, the overall state of the paper is good.\n- experimental evidence: the experimental evidence to support the proposed method is quite convincing (see weaknesses below for information and experiments missing) as the authors conduct experiments on 3 real-world datasets on which results show superiority to the standard MAE baseline which is the most relevant related work with confidence intervals.", "weaknesses": "- sensitivity of the method to choice of hyperparameters: the proposed method relies on spectral clustering for which a number of hyper parameters should be defined (like the number of clusters and hyperparameters linked to cluster density). These hyperparameters seem crucial in order to achieve high performance and are data-driven. \n- motivation: while it is clear that an aggregation/clustering of the input dimensions is necessary, the intuition behind aggregating dimensions that cluster together is missing; While the proposed method seems to work well, the paper would benefit from some additional intuition as to why one might want to combine elements that are similar in the same patch and compare this approach to the information found in standard image patches. There is little understanding of how MAE work, and most of their design choices are empirically driven, therefore it remains unclear whether MAE work better when pixels within a patch are identical. \n- missing experimental baseline numbers and details: multiple information regarding the experimental setup is missing thereby reducing the ability to judge the soundness of the experimental setup (see question below) and a couple of experiments seem to be missing in order to confidently conclude that the proposed method is an effective alternative to the standard MAE baseline (see questions below) \n- missing information about the work's limitations which might include the computational cost and scalability of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AzmPT7rRN4", "reviewer": "Reviewer_3Rzb", "summary": "This work introduces URLOST, a new framework for unsupervised representation learning aiming to overcomes limitations in handling high-dimensional data with unknown stationarity and topology, towards going beyond traditional methods that rely on structured data assumptions, such as grid-like images or time sequences. The proposed method combines a learnable self-organizing layer, density-adjusted spectral clustering, and a masked autoencoder (MAE) to capture learning representations from various data modalities.\n\nDemonstrated on synthetic biological vision data, neural recordings, and gene expressions, URLOST shows a good performance, outperforming existing baselines in capturing complex, irregular structures without prior domain knowledge. The paper highlights areas for future work, such as integrating clustering into the model's end-to-end learning process and enhancing the self-organizing layer.", "questions": "No extra questions. See the above comments.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The strengths of the paper lie in its introduction of URLOST, a new combined framework for unsupervised representation learning towadrs addressing the challenges of high-dimensional data with unknown stationarity and topology. By combining a learnable self-organizing layer, density-adjusted spectral clustering, and a masked autoencoder, URLOST can captures learning representations from various data modalities. Its demonstrated performance on synthetic biological vision data, neural recordings, and gene expressions shows that it outperforms baselines, enlightening its capability to handle complex and irregular structures in an unsupervised mode.\n\nThe work is nicely conceived and the presentation is tidy. Its bio-inspired sense makes the work basically intriguing.", "weaknesses": "Several aspects of the work can be improved:\n1) While the paper primarily motivates unsupervised learning, the experiments predominantly focus on supervised tasks (classification). This disconnect diminishes the relevance of the initial claims made in the first half of the paper.\n2) The use of foveated preprocessing is an intriguing aspect; however, there is a noticeable drop in performance (in Table 1), likely attributed to this process. This raises the question of whether foveated preprocessing detracts from overall performance.\n3)  Although the MAE appears to perform worse on the foveated dataset, this could simply be due to its training on the original dataset. Thus, the comparison does not effectively support the goal of emulating the biological process of foveation for benefits.\n4) While the Vision Transformer (ViT) serves as the backbone, the paper lacks clarity on how attention is utilized in the experiments. It would be beneficial for the community to understand the potential relationship between the foveated process and visual attention.\n5) The comparison against the MAE seems somewhat rudimentary. For example, the reported performance on CIFAR-10 indicates that several top models significantly outperform the authors' results (Table 1):\n\tRank\tModel\t\tPercentage correct\n\t1\t\tViT-H/14\t\t99.5\n\t2\t\tDINOv2 \t\t99.5\n\t3\t\tµ2Net\t\t99.49\n\t4\t\tViT-L/16\t\t99.42\nIt seems the claimed better performance over the baseline was quite embarrassed by the top runners above.\n6) The meaning and algorithm behind the permutation of CIFAR-10 images are unclear. Providing examples of permuted images would clarify this aspect. The rationale for permuting pixels should also be addressed, as such transformations may hinder human recognition and contradict the stated bio-inspired motivation.\n7) In Section 2, the proposed method appears to aggregate several state-of-the-art techniques with limited novel mathematical contributions. Incorporating theoretical exploration of its bio-inspired aspects would significantly enhance the paper's quality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nhhfGYluVb", "reviewer": "Reviewer_o7jg", "summary": "The submission proposes a route to implementing masked autoencoders (MAE) when the data does not have a known structure and yet is too granular for each component of the input to be its own “patch”.  In other words, there’s prior knowledge about the dataset that its components could probably be meaningfully grouped, but no grouping is available.\nThe method clusters the components by mutual information and spectral clustering, with an additional density correction applied.  After components have been grouped, all groups must be mapped to a shared representation space so that the transformer can compare them; for this the authors use a learned projection for each group that they call a self-organizing layer.\nThere are a fairly comprehensive suite of comparisons: to MAE without grouping the components and to SimCLR with a CNN and ViG architecture, on a couple CIFAR10 variants, gene expression data, and on V1 neural recording data.", "questions": "- Why is SimCLR with the proposed clustering pipeline (and ViT architecture) not included as a baseline for the CIFAR10-based experiments?  Why not ViG with patches found by URLOST?  Unless I have misunderstood, the proposed method is ultimately about “patchifying” data for use with a downstream transformer, which means there’s nothing tying the paper to MAE.  Such a shift away from exclusively linking the method with MAE would broaden its potential impact.\n- There is no need for explicit positional encoding information per cluster, correct?  Presumably the self-organizing layers can encode identifying information per cluster.\n- Line 265: “Moreover, since CNN can only process signal with stationarity and topology” is an incomplete sentence and also incorrect, as CNNs can absolutely process signals without stationarity.  Overall, I felt the stationarity part of the motivation to be much less important than the topology. \n- Why is MAE bolded in Table 1 when it was outperformed by SimCLR?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The premise to handle more diverse datasets than images with a ViT setup (and MAE pretraining, though it seems more general than a specific pretraining method) by clustering input components is strong.  The experiments cover a good range of complexity, starting with standard CIFAR10, permuting the pixels, and then remapping the visual field entirely.  Then the V1 and gene expression experiments represent actual use cases.  The comparisons are sufficient to demonstrate the utility of the method.", "weaknesses": "The clustering is the heart of the proposed method and comes with multiple design choices or parameters whose selection is not obvious.  While the experimental results demonstrate the method works, results supporting these design decisions are a bit weak.  As far as I can tell, the only results regarding the effect of the density adjusted spectral clustering are with respect to the foveated CIFAR10 data (Table 3b, Table 4)?  Why should we assume gene expression is best processed with the same parameters, considering how structured the synthesized foveated CIFAR10 is?  By contrast, the self-organizing layer is pretty intuitive, yet a large chunk of the manuscript and appendices are dedicated to it.\n\nRelated work could be strengthened.  Consider discussing the relation to FlexiViT (Beyer et al., CVPR 2023), where different patch sizes are mapped to the same representation space by learnable transformations, and PatchGT (Gao et al., LoG 2022), where nodes of a graph are clustered with spectral clustering, and the representation of each “patch” is a learned function of the subgraph.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "e4c9qgrmK2", "reviewer": "Reviewer_u6uR", "summary": "The paper proposes a method to learn useful representations of data without assuming the relationships between dimensions of topological structure, translation invariance, as in images/sounds.", "questions": "see Weaknesses", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper is well motivated and the task tackled is important and relevant to the community.", "weaknesses": "A lot of work has gone in to the paper, but unfortunately it is quite difficult to follow; and the proposed method seems very expensive (computing entropy, eigenvalues, clustering, etc). On the first point, to reach a broad audience, specific effort should be made regarding aspects less familiar to the general ML community. I find the method and data transformations difficult to parse.\n\n* Clarity\n   * [036-041] Unclear references to topology and stationarity -  core aspects of the paper\n      * Is the topology property that dimensions are ordered such that correlation (or similar) is expected between the signal at \"nearby\" dimensions? \n      * Stationarity seems very entwined with topology, does stationarity require topology? If not how? \n      * What assumptions *is* the model making?\n   * [225] unclear how permuting pixels leaves stationarity intact\n   * [259] unclear how reducing density prevents redundant sampling.\n* [095] A fully connected NN doesn't assume topology/spatial invariance and seems a key benchmark, but is not considered/mentoined.\n* [183] How does a vector denote a cluster? the cluster mean, indexes of cluster members?\n* [337] Given $\\beta$-VAE results in Table 2 are close on the real datasets, have optimal parameters/architecture been used? Why is this not compared to in Table 1? \n* The proposed method seems expensive but no mention of this is made.\n\nMinor\n* [031] unsupervised representation learning (UL) and self-supervised representation learning (SS) are not the same. SSL is a subset of UL, e.g. a $\\beta$-VAE performs UL but not SSL, whereas SimCLR performs SSL (and so UL).\n* Several typos\n   * [037] \"arised\", [055] \"singal\" etc", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BDasnLGlze", "reviewer": "Reviewer_t33q", "summary": "The paper suggests a novel way unsupervised representation learning pipeline, which consists of three stages: clustering, self-organising layers and masked autoencoder training. The authors try to tackle the assumptions of stationarity and topological ordering of the data for self-supervised learning.", "questions": "* if I permute all images in the same way, why would it ruin the topology? As far as I understand it, I will just transform it but the ordering is still preserved", "limitations": "", "rating": 8, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* great motivation: important problem of topological ordering and stationarity assumptions is raised. Biological inspiration is also interesting.\n* novel ideas, especially, when it comes to clustering and self-organized layers instead of patching. This potentially allows to aggregate features on different scales, which could be highly beneficial in many cases (for example, action tracking could be composed of motions tracking)\n* making the code available is great for reproducibility", "weaknesses": "* While the authors criticise graph neural networks for not being scalable and the paper is proposed as a new self-supervised approach, the scalability aspect is not addressed, both from the perspective of time (and spectral clustering can become slow) or performance . The datasets used in the paper are fairly small, while even for transcriptomics or neuroscience there are quite big datasets publicly available now.\n* Topology is mentioned a lot, however, I do not fully understand, where exactly authors show that their method is not relying on topology. I would suggest authors to take a point cloud dataset (for example, 3D Point Cloud Classification on ModelNet40 ) and perform a comparison on it. The transcriptomics vectors are still ordered (like each position corresponds to a specific gene), which could be also thought as topological organisation (like a flattened picture). \n* The experiments are missing the error bars or confidence intervals, which makes it a bit hard to understand how robust is the method and how significant are the improvements. I would inspire the authors to add the error bars for all of the experiments\n*  while the paper motivation is highly biologically grounded, the actual image dataset has very low signal-to-noise ratio, eg  a lot of natural scenes do not so nicely isolate a single object in the center of the image such as CIFAR. I would inspire the authors to use a more noisy multi-object classification or segmentation dataset\n* while avoiding stationarity assumption is one of the main paper motivations, I felt like a textbook case of non-stationary data, eg time series, is missing. I would suggest to include some non-stationary timeseries classification example, for example, for epilepsy, to prove illustrate this point in a much more convincing manner.\n* I find the biological motivation sometimes more confusing than helping, for instance, lines 471-476, mention self-organising maps, without giving a brief introduction on what they are.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)