Dual Advancement of Representation Learning and Clustering for Sparse and Noisy Images

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 03:48:16

Abstract: Sparse and noisy images (SNIs), like those  in spatial gene expression data, pose significant challenges for effective representation learning and clustering, which are essential for thorough data analysis and interpretation. In response to these challenges, we propose $\textbf{D}$ual $\textbf{A}$dvancement of $\textbf{R}$epresentation $\textbf{L}$earning and $\textbf{C}$lustering ($\textit{\textbf{DARLC}}$), an innovative framework that leverages contrastive learning to enhance the representations derived from masked image modeling. Simultaneously, $\textit{DARLC}$ integrates cluster assignments in a cohesive, end-to-end approach. This integrated clustering strategy addresses the ``class collision problem'' inherent in contrastive learning, thus improving the quality of the resulting representations. To generate more plausible positive views for contrastive learning, we employ a graph attention network-based technique that produces denoised images as augmented data. As such, our framework offers a comprehensive approach that improves the learning of representations by enhancing their local perceptibility, distinctiveness, and the understanding of relational semantics. Furthermore, we utilize a Student's t mixture model to achieve more robust and adaptable clustering of SNIs. Extensive evaluation on 12 real-world datasets of SNIs, representing spatial gene expressions, demonstrat $\textit{DARLC}$'s superiority over current state-of-the-art methods in both image clustering and generating representations that accurately reflect biosemantics content and gene interactions.

Main paper: 
 related works 2.1 self-supervised representation learning for images: 
 Most related SSL studies include CL and MIM methods. In CL, an input instance forms positive pairs with its augmented views, while forms negative pairs with other instances. Paradigmatic CL methods aim to learn instance discriminative representations by maximizing the similarity between positive pairs while minimizing it between negative pairs in a latent space [8, 20] . To address the class collision problem due to false negative samples, several studies [12, 24, 45] regularize CL with clustering, while others [5, 6, 9, 17] bypass the using of negative samples altogether. In contrast, MIM methods focus on learning local context-aware features by restoring raw pixel values from masked image patches [2, 16, 19, 42] . Several researchers are realizing the advantages of integrating these methodologies and endeavoring to utilize CL to refine MIM-generated representations [21, 42] . For instance, iBOT [47] contrasts between the reconstructed tokens of masked and unmasked image patches. Yet, to the best of our knowledge, DARLC is the first method that learns image representations from all aspects of discriminability, local perceptability, and relational semantic structures.Related deep image clustering studies include deep autoencoderbased methods [32] , which couple representation learning with deep embedded clustering in an end-to-end manner, as exemplified by methods like DEC [18, 23, 41] . Subsequent improvements to DEC focus on strategies like overweighing reliable samples (e.g., IDCEC [29] ), and replacing Euclidean distance-based clustering with deep subspace clustering (e.g., EDESC [3] ) or GMM-based clustering [38, 49] . Recent studies, including CC [25] , DCP [27] , CVCL [7] , and CCES [43] , directly integrate CL into the clustering process by contrasting at both instance and cluster levels across views, generating a soft cluster assignment matrix as deep embeddings for iterative refinement. Compared to these methods, DARLC offers a more comprehensive and potent mechanism for learning deep embeddings, a more robust and adaptable clustering algorithm, and a warm-up representation learning phase for accelerating clustering convergence. 
methodology 3.1 overview: 
 The framework of DARLC, as illustrated in Figure 2 and Algorithm 1, comprises two modules: a self-supervised representation learning and a deep clustering. The self-supervised representation learning module unifies CL and MIM, encompassing three encoders: an online encoder and a target momentum encoder for the contrastive branch, and a masked encoder for the MIM branch. All three encoders adopt the identical vision transformer (ViT) [14] architecture, with shared parameters between the online encoder and masked encoder. The parameters of the target momentum encoder are updated using exponential moving average (EMA), as suggested by BYOL [17] . The initial phase of the unified representation learning involves a warm-up pretraining to generate preliminary embeddings, which are then normalized by a nonlinear projection head. This normalization aligns the embeddings more closely with tdistributions, setting the stage for subsequent t-distribution-based clustering. The deep clustering module utilizes a SMM to cluster the normalized embeddings, generating soft cluster assignment scores involved in the calculation of various loss functions. There are two types of loss functions: L 1 , an epoch-level loss function for maximizing the empirical likelihood of observed instances, and L 2 , a batch-level loss function for discriminatively boosted clustering optimization [41] . The two loss functions work in tandem, enabling the joint refinement of image embeddings and cluster assignments in a self-paced and end-to-end manner.Learning (Module I)3.2.1 Denoising-based Data Augmentation. We train a graph attention autoencoder G to generate smoothed images, serving as augmented positive instances [13, 37] . Initially, for each image, we construct an undirected and unweighted graph by treating pixels as nodes connected to their k-nearest neighbors. Specifically, for a given image X ∈ R 𝐶 ×𝐻 ×𝑊 , where 𝐶 is the number of channels, 𝐻 and 𝑊 are the height and width of the image, respectively. Let 𝑁 𝑝𝑖𝑥 = 𝐻 × 𝑊 denote the number of pixels, v 𝜄 ∈ R 𝐶 denote the pixel vector at location 𝜄, ∀𝜄 ∈ {1, 2, ..., 𝑁 𝑝𝑖𝑥 }. The encoder in G comprises 𝐿 layers. For each layer 𝑡 ∈ {1, 2, ...𝐿 -1}, with the initial value hEQUATIONwhere W (𝑡 ) represents the trainable weights of the 𝑡-th autoencoder layer, 𝑆 𝜄 the set of node 𝜄's neighbors within a pre-specified radius 𝑟 . The attention score, att 𝜄𝜈 , between nodes 𝜄 and 𝜈 are computed as follows:EQUATIONEQUATIONwhere w(𝑡 )𝑎𝑡𝑡 represents the trainable attention weights. The decoder of G mirrors the encoder with tied weights. The total loss function is defined as:EQUATIONwhere h(0) 𝜄 denotes the reconstructed v 𝜄 output by the decoder. Once trained, G is applied to any given image X 𝑖 , to generate a smoothed image X 𝑖 for a given image X 𝑖 as: forEQUATIONX 𝑏 , X 𝑏 in X, X do 4:Compute X 𝑏 by Eq. ( 6) → L 𝑟𝑒𝑐 (X 𝑏 , X 𝑏 ) by Eq. ( 7).5:Compute e 𝑏 , e 𝑏 by Eq. ( 9) → L 𝑐𝑙𝑟 (e 𝑏 , e 𝑏 ) by Eq. ( 10).6:L 𝑠𝑠𝑙 = UWL(L 𝑟𝑒𝑐 , L 𝑐𝑙𝑟 ). 7:Update Π(M) using L 𝑠𝑠𝑙 .8:Update Π(C) with EMA(Π(M)).end for 10: end while 11: while (not converged) & (𝑒𝑝𝑜𝑐ℎ < 𝐸 𝑚𝑎𝑥 ) do Compute X by Eq. ( 6) → L 𝑟𝑒𝑐 (X, X) by Eq. ( 7). Compute Z by Eqs. ( 9), ( 12) → L ℓ𝑎𝑝 (Z, S) by Eq. ( 19). 22), (20) . 16 : Update Π(M), Π(C), Π(H ) using L 1 .L 1 = 𝜂L ℓ𝑎𝑝 + (1 -𝜂)UWL(-L ℓℓ , -L 𝑠𝑖𝑧𝑒 , L 𝑟𝑒𝑐 ).for X 𝑏 , X 𝑏 in X, X do Compute 𝑃 by Eq. ( 24) → L 𝑘𝑙 (𝑃, 𝑄) by Eq. ( 23).Compute X 𝑏 by Eq. ( 6) → L 𝑟𝑒𝑐 (X 𝑏 , X 𝑏 ) by Eq. ( 7).Compute e 𝑏 , e 𝑏 by Eq. ( 9) → L 𝑐𝑙𝑟 (e 𝑏 , e 𝑏 ) by Eq. (10) .L 2 = UWL(L 𝑘𝑙 , L 𝑟𝑒𝑐 , L 𝑐𝑙𝑟 ).Update Θ and Π(M), Π(C), Π(H ) using L 2 .end for 25: end while 26: return Z, 𝑄 is substituted with a fully-connected linear decoder M 𝐷 . For any given image X 𝑖 , the regenrated image X 𝑖 is as follows:EQUATIONThe MIM branch loss for the current batch are:EQUATIONwhere 𝑁 𝑏 is the batch size, W 𝐸 and W 𝐷 represent the parameters of the encoder and decoder, respectively. 𝑁 𝑚𝑎𝑠𝑘𝑒𝑑 denotes the number of masked patches, 𝑆 𝑚𝑎𝑠𝑘𝑒𝑑 the set of masked patches. p 𝑖,𝑗 and p 𝑖,𝑗 represent the original and regenerated 𝑗-th image patch of X 𝑖 , respectively.For the contrastive branch C, the 𝑁 𝑏 raw images form positive pairs with their respective smoothed images, and negative pairs with the other 2𝑁 𝑏 -2 images in the same batch. C is structured around a pseudo-siamese network with two encoders: an online encoder C 𝑂 and a target momentum encoder C 𝑇 . Both encoders share the identical network architecture as P 𝐸 , with C 𝑂 and P 𝐸 having tied parameters. The parameters of C 𝑇 are updated using EMA. Concretely, let W 𝐸 and W 𝐸 denote the parameters of the C 𝑂 and C 𝑇 , respectively. Then we have:EQUATIONHere, 𝑚 represents the momentum, fixed at 0.999. For each image X 𝑖 and its augmented counterpart X 𝑖 , their respective embedding vectors, e 𝑖 and e 𝑖 ∈ R 𝐷 , are obtained as:EQUATIONwhere g and ḡ are linear mapping functions with trainable weights, and the weights of ḡ are updated using EMA as well. The contrastive loss L 𝑐𝑙𝑟,𝑖 is computed as : L 𝑐𝑙𝑟,𝑖 = -log s(where s(•, •) = exp(cos(•, •)/𝜏), and 𝜏 is a temperature coefficient, defaulting to 0.5. Consequently, the loss function L 𝑐𝑙𝑟 is defined asL 𝑐𝑙𝑟 = 1 𝑁 𝑏 𝑁 𝑏𝑖=1 L 𝑐𝑙𝑟,𝑖 . Finally, L 𝑟𝑒𝑐 and L 𝑐𝑙𝑟 are dynamically integrated into the total SSL loss function L 𝑠𝑠𝑙 using the uncertain weights loss (UWL) function [26] :EQUATIONwhere 𝜎 1 and 𝜎 2 are trainable noise parameters.3.3.1 Student's t mixture model. Let e 𝑖 denote the Module I -generated embedding vector for the 𝑖-th original image. We first map e 𝑖 to z 𝑖 ∈ R 𝐷 in a latent space wherein it is more conformed to a tdistribution. This mapping is achieved through a nonlinear projection head with batch normalization and scaled exponential linear unit (SELU) activation function:EQUATIONThe set of these vectors, Z = {z 𝑖 } 𝑁 𝑖=1 ∈ R 𝑁 ×𝐷 , is modeled using an SMM, whose components correspond to image clusters. Since extreme values are downweighed by SMM during parameter inference, this clustering is more robust to outliers and variances. The SMM is parameterized by Θ = {𝜃 𝑘 :𝜋 𝑘 , 𝜇 𝑘 , Σ 𝑘 , 𝑣 𝑘 , ∀𝑘 ∈ 𝐾 }, where 𝐾 represents the number of components and is assumed to be known or can be automatically inferred (see Section 4.1). Here, 𝜋 𝑘 , 𝜇 𝑘 , Σ 𝑘 , 𝑣 𝑘 denote the weight, mean, covariance matrix, and degree of freedom of the 𝑘-th component, respectively. The density function of z 𝑖 is expressed as:EQUATIONFor robust model inference, we use the maximum a posteriorexpectation maximization (MAP-EM) algorithm by applying a conjugate Dirichlet prior on Π = {𝜋 𝑘 , ∀𝑘 ∈ [1, 𝐾]} and a normal inverse Wishart (NIW) prior on 𝜇 𝑘 , Σ 𝑘 :EQUATIONEQUATIONTo simplify the inference, we rewrite the Student's t density function 𝜙 as a Gaussian scale mixture by introducing an "artificial" hidden variable 𝜁 𝑖,𝑘 , ∀𝑖 ∈ [1, 𝑁 ] , ∀𝑘 ∈ [1, 𝐾] that follows a Gamma distribution parameterized by 𝑣 𝑘 :EQUATIONWe further introduce a missing variable 𝜉 𝑖 to represent the component membership of z 𝑖 . The posterior complete data log likelihood is then expressed as:EQUATIONIn the 𝑡-th iteration of the E-step, the expected sufficient statisticsEQUATIONand 𝜁 𝑖,𝑘(𝑡 )are derived based on Θ (𝑡 -1) . In the subsequent M-step, Θ (𝑡 -1) is updated to Θ (𝑡 ) by maximizing the auxiliary function 𝑄 (Θ, Θ (𝑡 -1) ) = 𝐸 (ℓ 𝑐 (Θ)|Θ (𝑡 -1) ). These two steps are alternated until either convergence is reached or a predefined maximum number of iterations is attained. Refer to Supplementary 1.1 for details of the model inference.Embeddings and Cluster Assignments. Two loss functions, L 1 and L 2 , are calculated based on clustering results for updating parameters of both Module I and the SMM through loss gradient backpropagation. This iterative process progressively improves the clustering-oriented image embeddings and clustering results. Upon completing the inference of SMM parameters Θ in each epoch, an epoch-level loss L 1 is calculated for updating parameters of Module I :EQUATIONHere, L ℓ𝑎𝑝 is a Laplacian regularization term that promotes the similarities among image embeddings Z to be consistent with a seeding image-image similarity matrix S, informing the initial training phase. The derivation of S is detailed in Supplementary 1.2. L ℓ𝑎𝑝 is defined as follows:EQUATION)where D is the degree matrix of S, and 𝜂, initially set at 0.5, decays over the training course so that the influence of S is gradually reduced. L ℓℓ represents the log likelihood of the embeddings given the estimated SMM parameters Θ:EQUATIONEQUATIONL 𝑠𝑖𝑧𝑒 penalizes empty and tiny clusters, while exempting those whose size exceeds a predefined threshold 𝜐 so that image assignments is not overly uniform:EQUATIONL 𝑟𝑒𝑐 , defined in Equation 7, aims to enhance the local-context awareness of embeddings. Subsequently, within the same epoch, a batch-level loss L 2 = UWL(L 𝑘𝑙 , L 𝑟𝑒𝑐 , L 𝑐𝑙𝑟 ) is utilized to update Module I and SMM parameters across successive batches. Here, L 𝑟𝑒𝑐 and L 𝑐𝑙𝑟 remains same as in Equations 7 and ?? except being calculated on the batch-level. L 𝑘𝑙 boosts high-confidence images, incrementally grouping similar instances while separating dissimilar ones:EQUATIONwhere q 𝑖,𝑘 = 𝑞 𝑖,𝑘 𝑐 𝑞 𝑖,𝑐 , p 𝑖,𝑘 = q 2 𝑖,𝑘 / 𝑖 q 𝑖,𝑐 𝑐 q 2 𝑖,𝑐 / 𝑖 q 𝑖,𝑐Here, 𝑞 𝑖,𝑘 is same as in Equation 21, q 𝑖,𝑘 represents the probability of assigning 𝑖-th image to the 𝑘-th SMM component, and p 𝑖,𝑘 an auxiliary target distribution that boosts up high-confidence images. After this joint optimization, the training progresses to the next epoch, iterating until the end of the training process. The mathematical derivations of gradients of L 1 and L 2 with respect to W 𝐸 , W 𝐷 and Θ are detailed in Supplementary 1.3. 
experiments 4.1 experimental settings: 
 4.1.1 Datasets. To comprehensively evaluate DARLC, we utilize 12 real ST datasets derived from different tissue slices of the human dorsolateral prefrontal cortex (hDLPFC). These datasets display spatial expression levels of the whole genome across the hDLPFC tissue (approximately 18700 75x75 SGEP images per dataset). These datasets are named by tissue slice numbers (151507-151510, 151669-151676), and accessible through the spatialLIBD package [31] at http://spatial.libd.org/spatialLIBD.4.1.2 Data quality control and preprocessing. We conform to the conventional procedure for preprocessing ST data, as implemented in the SCANPY package [40] . Specifically, we first remove mitochondrial and External RNA Controls Consortium spike-in genes.Then, genes detected in fewer than 10 spots are excluded. To preserve the spatial data integrity, we do not perform quality control on spatial spots. Finally, the gene expression counts are normalized by library size, followed by log-transformation. Given the number of image clusters is not known a priori, DARLC can estimate this number using a seeding similarity matrix S ∈ R 𝑁 ×𝑁 , where 𝑁 represents the number of images [44] . S is transformed to a graph Laplacian matrix (refer to Supplementary 1.2), L ∈ R 𝑁 ×𝑁 as follows:EQUATIONEQUATION)Here, D ∈ R 𝑁 ×𝑁 represents the degree matrix of S, and S ′ aims to enhance the similarity structure. The eigenvalues of L are then ranked as𝜆 (1) ≤ 𝜆 (2) ≤ • • • ≤ 𝜆 (𝑁 ). The number of clusters, denoted as 𝐾, is inferred as: [41] and DAGMM [49] , as well as five SOTA methods that either improve DEC (e.g.,EDESC [3] and IDCEC [29] ) or incorporate CL (e.g., CC [25] , DCP [27] , and CVCL [7] ). In addition, we include a benchmark method consisting of iBOT [47] , which integrates CL and MIM for representation learning, and a boosted GMM for clustering, denoted as iBOT-C.EQUATIONThe GAT for data augmentation adopts an encoder including a single attention head with C-512-30 network structure, and a symmetric decoder. In Module I, the shared encoder structure is a ViT comprising four transformer blocks, each having four attention heads, for processing 75 × 75 input images segmented into 4 × 4 patches in our case. The MIM decoder follows a D-128-256-512-1024-75*75 residual network. The Module I is pretrained for 50 epochs with a learning rate of 0.001. The nonlinear projection head that bridges Module I and II is a two-layer MLP for normalizing image representations to a dimension size of 32. The iterative joint optimization of representation learning and clustering continues for 50 epochs using Adam optimizer. Given the absence of ground truth in gene cluster labels, we heuristically determine the number of gene clusters for our experiments to achieve an average cluster size of 30 genes, approximating the typical size of a gene pathway [46] . 4.1.6 Evaluation Metrics. Without ground truth cluster labels, we evaluate the clustering results using the Davies-Bouldin index (DBI) metric [10] :EQUATIONwhere 𝐾 is the number of clusters, 𝐶 𝑖 the samples in cluster 𝑖, 𝛿 𝑖,𝑗 the distance between instances 𝑖 and 𝑗, 𝑐 𝑖 the centroid of cluster 𝑖. Cluster width 𝑑 𝑖 is the mean intra-cluster distance to 𝑐 𝑖 , and 𝑑 (𝑖,𝑗 ) = 𝛿 𝑐 𝑖 ,𝑐 𝑗 measures the distance between clusters 𝑖 and 𝑗. DBI quantifies the clustering efficiency by measuring the ratio of intracluster compactness to inter-cluster separation, with lower scores indicating better clustering. To evaluate clustering from different perspectives, we use two DBI metrics, DBIP and DBIE, based on Pearson and Euclidean distances, respectively [34] . in DBIP across all datasets. Finally, DARLC clusters are divided into high and medium-quality groups using spectral angle mapper (SAM) metric scores [22] , with lower SAM indicating greater intra-cluster similarity. To provide a visual illustration of DARLC's clustering performance, we randomly select one cluster from both the high-and medium-quality groups within each of the 151675 and 151676 datasets. From each of the selected clusters, we then randomly select five genes to be displayed in Figure 3 . Figure 3 clearly demonstrates that DARLC effectively groups images with similar expression patterns into the same cluster.In this section, we present a comprehensive evaluation of gene image representations produced by the fully implemented DARLC model (denoted as DARLC-full). First, we assess whether representations generated by DARLC-full capture corresponding biosemantics, particularly through pathway enrichment analysis, compared to original gene expressions. Furthermore, we extend our investigation to specific critical downstream tasks: predicting interactions between genes and clustering genes based on spatial cofunctionality. These evaluations are conducted across six distinct datasets (151671-151676). Additionally, gene image embeddings generated by iBOT and a variant of DARLC (denoted as DARLC-C1), which is deprived of Module II, serve as baselines.We employ an MLP-based classifier for predicting gene pair interactions. This is achieved by linear probing using gene image representations generated by DARLC and baseline methods (see Supplementary 1.4). We follow the methodology in [15] , which is based on the Gene Ontology, to acquire the gene-gene interaction ground truth. Theoretically, gene image representations with richer semantic meanings should yield more accurate predictions. As shown by the accuracy scores in Table 2 , the classifier yields the most accurate predictions (77.62%±1.85%) using embeddings generated by DARLC-full, and the second most accurate predictions (75.18%±2.17%) using embeddings generated by DARLC-C1, followed by the predictions using iBOT-generated embeddings (67.54%±1.03%).Cofunctional Gene Clustering. Spatially cofunctional genes are those belonging to the same gene family and exhibit similar spatial expression patterns [11] . Their family identities can serve as labels to evaluate the quality of gene image embeddings via clustering. Specifically, our evaluation involves five spatially cofunctional genes from each of the HLA, GABR, RPL, and MT gene families (see Supplementary Figure 1 ). The Leiden algorithm [36] is used to cluster gene image embeddings generated by DARLC-full and the baseline methods. The clustering results, evaluated using the normalized mutual information (NMI) and adjusted rand index (ARI) scores, as shown in Table 3 , demonstrate that Leiden yields the most accurate clustering with DARLC-full and the second most accurate with DARLC-C1. In summary, these results collectively highlight the effectiveness of the joint clustering (i.e., DARLC-full surpasses DARLC-C1) and GAT-based data augmentation (i.e., DARLC-C1 surpasses iBOT) in enhancing the quality of gene image representations.Here, we conduct a series of ablation studies on six ST datasets (151671-151676) to investigate the contributions of DARLC's components in image clustering. The results, detailed in DBIE and DBIP scores, are presented in Table 4 . Notably, DARLC's performance declines most with the CL branch removal ("w/o CLR"), followed by the elimination of Module II ("w/o SMM") and the substitution of the robust SMM with an outlier-sensitive GMM ("SMM → GMM"). We also observed that employing traditional image smoothing methods such as Gaussian Kernel Smoothing (GKS) instead of GAT ("GAT → GKS") on SNIs leads to a decline in DARLC's performance. Additionally, removing either L ℓ𝑎𝑝 ("w/o L ℓ𝑎𝑝 ") or L 𝑠𝑖𝑧𝑒 ("w/o L 𝑠𝑖𝑧𝑒 ") from the optimization decreases DARLC's performance, indicated Overall, all key components of DARLC have proven effective, particularly the CL module. This is due to: a) The discontinuous and sparse nature of SNIs, which greatly benefits from the enhanced discriminability provided by CL. b) CL's use of smoothed images as augmented data, which serve as effective priors for guiding DARLC to discern the primary visual patterns of SNIs. 
conclusion: 
 In this study, we introduce DARLC, a novel algorithm for joint learning of representations and cluster assignments for SNIs. DARLC features in its enhanced data augmentation technique, comprehensive and potent representation learning approach that integrates MIM, CL and clustering, as well as robust and adaptable clustering algorithm. These features collectively contribute to DARLC's superiority in both image representation learning and clustering, as evidenced by our extensive benchmarks over multiple real datasets and comprehensive ablation studies.  
: 
 complicate the discerning of its underlying major gene expression pattern.Image clustering can group unlabelled images into distinct clusters, facilitating the exploration of image-implied semantics or functions. For example, clustering SGEP images offers a cost-effective means to identify groups of cofunctional genes and infer gene functions [34] . To obtain informative clustering results, it is essential to learn meaningful image representations, for which self-supervised learning (SSL) is the predominant approach in general scenarios. These include contrastive learning (CL) methods, exemplified by MoCo [20] , and masked image modeling (MIM) such as MAE [19] . These methods offer distinct learning perspectives: MIM methods tend to learn local context-aware, holistic features for reconstruction tasks [19] , while CL methods focus on learning instance-wise discriminative features [21] . Acknowledging the synergistic advantages of these methodologies, some researchers are endeavoring to utilize CL to refine the representations acquired through MIM [21, 47] . Moreover, many efforts [25, 32, 41] are directed towards guiding the process of learning representations through clustering tasks. This is achieved by jointly learning representations and executing clustering in an integrated and end-to-end manner, yielding image representations that are well-suited for clustering tasks.However, for SNIs, both representation learning and clustering present significant challenges. Firstly, the widespread presence of uninformative voids or background areas, along with elevated noise levels and extremely fragmented visual patterns, substantially impedes the extraction of semantically meaningful visual features. This challenge has been highlighted in prior studies [30] and is further corroborated by our experiments (see Supplementary Table 1). Secondly, the inherent random noise across pixels induces considerable variability in visual patterns, even among images of the same category [34] , exposing clustering algorithms to a high overfitting risk [1] .Inspired by the aforementioned works, in order to better analyze SNIs, we propose a novel and unified framework, named Joint Learning of Representations and Cluster Assignments (DARLC). This framework not only leverages CL to boost the representation learned by MIM but also jointly learns cluster assignments in a self-paced and end-to-end manner, further refining the representation. Nonetheless, our experiments (see Supplementary Table 2) showcase that conventional data augmentation techniques (e.g., cropping and rotating) are ineffective for SNIs, as the augmented images often contain substantial void regions and noise, hampering the extraction of informative visual features. To overcome this limitation, we introduce a data augmentation method based on a graph attention network (GAT) [13, 37] that aggregates information from neighboring pixels to enhance visual patterns, generating smoothed images that act as more plausible positive views so as to improve the effectiveness of contrastive learning.Additionally, we observed that the clustering algorithms used in many deep clustering methods are either sensitive to outliers, as demonstrated by the Gaussian mixture model (GMM) in DAGMM [49] and manifold clustering in EDESC [3] , or lack the flexibility to different data distributions, such as the inflexible Cauchy kernel-based method in DEC [41] . In response, DARLC employs a specialized nonlinear projection head to normalize image embeddings, aligning them more closely with a t-distribution. This is followed by modeling with a Student's t mixture model (SMM) for soft clustering. SMM provides a more robust solution by down-weighing extreme values and is more adaptable by altering the degrees of freedom, making it particularly suitable for clustering in the context of SNIs. Furthermore, the clustering loss in DARLC also serves to regularize CL, alleviating the "class collision problem" that stems from false negative pairs in CL [4] . Unlike existing regularized CL methods [12, 24, 45] , which directly integrate clustering into the CL throughout training, this clustering follows the "warm-up" representation learning, significantly expediting training convergence and enhancing clustering accuracy, as demonstrated in our ablation study. All these features collectively contribute to the finding that DARLC-generated image representations not only enhance clustering performance but also exhibit improvements in other semantic distance-based tasks, such as the discovery of functionally interactive genes. In summary, our main contributions are:• We propose DARLC, a novel unified framework for joint representation learning and clustering of SNIs. DARLC marks the first endeavor in integrating contrastive learning, MIM and deep clustering into a cohesive process for representation learning. The resultant representations not only enhance image clustering performance but also benefit other semantic distance-based tasks.• DARLC has developed a data augmentation method more suitable for SNIs, using a GAT to generate smoothed images as plausible positive views for CL. • An SMM-based method is designed to cluster SNIs in a more robust and adaptable manner. Additional features of this clustering method include a novel Laplacian loss for guiding the initial phase of clustering, and a differentiable cross-entropy hinge loss for controlling cluster sizes. This clustering also addresses the class collision problem by pulling close related instances. • Extensive experiments have been conducted across 12 real SGEP datasets [28] . Our results show that DARLC surpasses the state-ofthe-art (SOTA) methods in both image clustering and generating image representations that accurately capture gene interactions. 
 
 Appendix:  


Decision: All reviewers agree that this paper has its own contribution to the field, although there are some issues in writing and detailed representation. We recommend the authors taking all comments from reviewers to further improve this paper.

Reviews: review: paper_summary: paper_summary: The paper introduces DARLC, a novel framework for joint representation learning and clustering of sparse and noisy images (SNIs). DARLC integrates contrastive learning (CL), masked image modeling, and deep clustering into a cohesive process, enhancing image clustering performance and benefiting semantic distance-based tasks. It addresses the challenges posed by SNIs, such as uninformative regions and high noise levels, by introducing a GAT-based data augmentation method to generate smoothed images that improve contrastive learning. Furthermore, DARLC employs a Student’s t mixture model for robust and adaptable soft clustering, which down-weights extreme values and adjusts degrees of freedom to suit SNIs. The framework also addresses the class collision problem in CL by using the clustering loss to regularize it.
strength_weakness: strength_weakness: - This paper proposes a novel algorithm for joint learning of representations and cluster assignments for SNIs and verifies it on spatial transcriptome data.
- In the experiment, the method proposed in the paper significantly outperforms the baseline methods.
- The paper is well-written.
limitations: limitations: ### Weakness:
- The DARLC proposed in this paper uses the t distribution for subsequent analysis. There is nothing wrong with the technical process, but there seems to be no good explanation and description of why the t distribution was chosen. In other words, can other tractable distributions and corresponding mixture models be chosen? The answer may be yes, in which case the author needs to explain the rationality and advantages of the t distribution modeling.
- Please explain the intuition behind Eq(25) and Eq(27), and why use the degree matrix of $S$ to normalize $S'$ in Eq(26)? What does the distribution of $\lambda_{i} - \lambda_{i-1} $ look like?
- How do you choose $k$ in line 284?

### Minor points:
- Extra '.' in the caption of section 3.2.
paper_summary: paper_summary: Sparse and noisy images (SNIs) pose significant challenges to effective representation learning and clustering, which are essential for thorough data analysis and interpretation.
This paper proposed the Dual Advancement of Representation Learning and Clustering (DARLC), an innovative framework that leverages contrastive learning to enhance the representations derived from masked image modeling. 
The proposed framework offers a comprehensive approach that improves the learning of representations by enhancing their local perceptibility, distinctiveness, and understanding of relational semantics.
Experiment results demonstrate DARLC’s superiority over current state-of-the-art methods in both image clustering and generating representations that accurately reflect biosemantics content and gene interactions.
strength_weakness: strength_weakness: 1. The authors have done well in the demonstration of the studied problem and the overall framework. Such as the Figure 1.

2. I agree with the authors that Sparse and noisy images (SNIs) are interesting and important to work on.

3. The experiment evaluations are comprehensive and reasonable, the authors evaluated their method on 12 real-world datasets and compared their method with multiple baselines.

3. Although I can't fully examine the correctness of the proof in the Appendix. I think the authors have done well in deriving of the gradients, which shows their certain abilities in statistics and optimization.
limitations: limitations: 1. The first apparent weakness is the quality of the writing of the abstract. Note that I think this is relatively minor and easy to fix, but the grammar and vocabulary should be improved. For example "demonstrat" should be "demonstrate". 

2. I believe the authors should provide a detailed explanation of the reasons for choosing techniques such as CL, MIM, and SMM. It will be good to explain in detail why these techniques are more effective for SNIs. The current demonstration is more like a combination of techniques or a technical report rather than a unified paper. One example is in section 3.2.2 "specifically designed for generating image patch embeddings in the context of SNIs." I think a brief explanation for why this "adapted version of MAE" is more effective for SNIs is required.

3. The authors employ a lot of symbols, I think a summary of all used symbols will help readers to understand this work better.

4. In section 3.3.2, there is a reference error, "Here, L𝑟𝑒𝑐 and L𝑐𝑙𝑟 remain the same as in Equations 7 and ?? except being calculated on the batch-level." 

5. I suggest authors add a summary sentence before each section, especially the methodology section, which can help readers figure out the structure of this section. 

6. I think a more detailed introduction of the used datasets will be useful.

7. There are multiple reference errors in the Appendix 1.4. For example, "which is a basic MLP-based network as described in Du et al., 2019 [?]". I suggest authors inspect all these errors carefully.

Overall, please understand that I agree with the authors that Sparse and noisy images (SNIs) are important and difficult to work on. However, I think authors should put more effort into the clarity of the writing, especially in the methodology section.
paper_summary: paper_summary: This paper proposes the DARLC framework, which enhances image representation and clustering quality through contrastive learning and a graph attention network, demonstrating superior performance in handling sparse and noisy images.
strength_weakness: strength_weakness: 1. The paper proposes an effective method to solve image representation learning and clustering. 

2. The paper is well-written.
limitations: limitations: 1. The proposed method seems to be a stack of existing techniques, including contrastive learning, graph attention networks, and the Student’s t mixture model. While these techniques each have their strengths, merely stacking them does not necessarily lead to significant innovation. The authors need to clearly articulate the unique technical and theoretical contributions of their approach.

2. The paper does not provide a detailed discussion of the computational complexity of the proposed method. Complexity is a crucial consideration when handling large-scale sparse and noisy images. The authors should include a complexity analysis, covering both time and space complexity, to assess the feasibility and efficiency of their method in practical applications.

3. Although the paper evaluates the method on 12 real-world SNI datasets, the scale and diversity of these datasets might not be sufficient to comprehensively validate the effectiveness of the method. It is recommended to conduct experiments on larger and more diverse datasets to better demonstrate the robustness and general applicability of the method.
paper_summary: paper_summary: In this paper, the authors propose a clustering framework for sparse and noisy images, which integrating contrastive learning, masked image modeling and deep clustering into a cohesive process for representation learning. Extensive experiments are conducted on 12 real ST datasets to verify the effectiveness of the proposed method.
strength_weakness: strength_weakness: 1. The paper is well-written and the structure is well organized.
2. The experiments are numerous and plentiful, and well demonstrate the validity of the methodology.
3. The derivation The proofs of some formulas are detailed and theoretically complete.
limitations: limitations: In this paper, the authors propose a clustering framework for sparse and noisy images, which integrating contrastive learning, masked image modeling and deep clustering into a cohesive process for representation learning. Extensive experiments are conducted on 12 real ST datasets to verify the effectiveness of the proposed method. I have some concerns as follows:
1. As the authors state in their contributions, the authors combine the techniques of contrast learning, self-paced learning, and deep clustering for learning the representations of SNIs, which appears to be an incremental effort. Therefore, what are the authors' motivations for using these techniques, and what particular problems in representation learning of  SNIs can these techniques address? 
2. Many components are involved in the proposed method, such as the data generation based on GAT and self-paced learning for clustering etc. Therefore, the time complexity of the algorithm will be high, and the authors should give a detailed time complexity analysis and runtime comparison.
3. The authors learn that good representations of SNIs are used for clustering, and as far as I know, the common evaluation metrics for clustering are ACC, NMI, ARI, etc., and I am more concerned in how the proposed algorithm performs on mainstream metrics versus comparative algorithms in Table I.