Title: A versatile informative diffusion model for single-cell ATAC-seq data generation and analysis
Venue: NeurIPS 2024
Accepted: yes
Keywords: scATAC-seq

diffusion model

generative model
Abstract: The rapid advancement of single-cell ATAC sequencing (scATAC-seq) technologies holds great promise for investigating the heterogeneity of epigenetic landscapes at the cellular level. The amplification process in scATAC-seq experiments often introduces noise due to dropout events, which results in extreme sparsity that hinders accurate analysis. Consequently, there is a significant demand for the generation of high-quality scATAC-seq data in silico. Furthermore, current methodologies are typically task-specific, lacking a versatile framework capable of handling multiple tasks within a single model. In this work, we propose ATAC-Diff, a versatile framework, which is based on a diffusion model conditioned on the latent auxiliary variables to adapt for various tasks. ATAC-Diff is the first diffusion model for the scATAC-seq data generation and analysis, composed of auxiliary modules encoding the latent high-level variables to enable the model to learn the semantic information to sample high-quality data. Gaussian Mixture Model (GMM) as the latent prior and auxiliary decoder, the yield variables reserve the refined genomic information beneficial for downstream analyses. Another innovation is the incorporation of mutual information between observed and hidden variables as a regularization term to prevent the model from decoupling from latent variables. Through extensive experiments, we demonstrate that ATAC-Diff achieves high performance in both generation and analysis tasks, outperforming state-of-the-art models.

A versatile informative diffusion model for single-cell ATAC-seq data generation and analysis

Abstract

The rapid advancement of single-cell ATAC sequencing (scATAC-seq) technologies holds great promise for investigating the heterogeneity of epigenetic landscapes at the cellular level. The amplification process in scATAC-seq experiments often introduces noise due to dropout events, which results in extreme sparsity that hinders accurate analysis. Consequently, there is a significant demand for the generation of high-quality scATAC-seq data in silico . Furthermore, current methodologies are typically task-specific, lacking a versatile framework capable of handling multiple tasks within a single model. In this work, we propose ATAC-Diff, a versatile framework, which is based on a latent diffusion model conditioned on the latent auxiliary variables to adapt for various tasks. ATAC-Diff is the first diffusion model for the scATAC-seq data generation and analysis, composed of auxiliary modules encoding the latent high-level variables to enable the model to learn the semantic information to sample high-quality data. Gaussian Mixture Model (GMM) as the latent prior and auxiliary decoder, the yield variables reserve the refined genomic information beneficial for downstream analyses. Another innovation is the incorporation of mutual information between observed and hidden variables as a regularization term to prevent the model from decoupling from latent variables. Through extensive experiments, we demonstrate that ATAC-Diff achieves high performance in both generation and analysis tasks, outperforming state-of-the-art models.

1 Introduction

Assay for Transposase Accessible Chromatin with sequencing (ATAC-seq) data has shed light on genome research to dissect gene regulatory landscapes and cellular heterogeneity. Particularly, singlecell ATAC-seq (scATAC-seq) can be harnessed to probe the chromatin accessibility profile at singlecell level to reserve the diversity of cell types in a heterogeneous tissue, which can reveal important genomic elements for transcription factor binding and regulating downstream gene expression, particularly at distal non-coding regulatory regions[1, 2]. However, the analysis of such data poses challenges due to the high levels of noise, sparsity, and data scale encountered. Furthermore, the complexity of biological systems, with their intricate intercellular communications and numerous

molecules involved, presents another challenge in analyzing them, especially when limited by small sample sizes, which can hinder reproducibility in biomedical research[3].

Recent advances in machine learning models especially generative models have driven rapid development in analyzing single-cell sequencing data to understand fundamental mechanisms of biology. These models achieved state-of-the-art (SOTA) performances across a range of single-cell tasks[4, 5, 6, 7, 8], including clustering analysis, batch correction, data denoising and imputation, and in silico data generation[3]. However, most methods focused on single-cell RNA-seq data and there is little work focusing on scATAC-seq data which is more sparse and high dimensional. Furthermore, these models treat each task with different frameworks, ignoring the potential correlations across these tasks.

To tackle these challenges, it's essential to create generative models that can not only produce highquality scATAC-seq data but also maintain the integrity of biological representations for subsequent analysis. Recently, diffusion models [9, 10, 11] have emerged as a promising tool, demonstrating impressive results on realistic imaging generation [12, 13, 14], audio synthesis [15, 16], and molecular ligand design [17, 18]. Nevertheless, the latent variables of these diffusion models lack semantic meaning and are not interpretable since they aim to learn the diffused noise, which is not suitable for scATAC-seq data representation learning to uncover the underlying biological patterns (i.e., cellular heterogeneity within a tissue or organism). Since single-cell sequencing data is typically represented as bag-of-words, where there is no upper limit to the value of peak calling per cell and the order of peaks does not affect the cell, the diffusion model, which is designed for continuous data, is less suited for processing discrete data. To address this, we can employ a latent diffusion model[13] that utilizes a pretrained autoencoder to transform scATAC-seq samples into a continuous latent space. This transformation facilitates the effective training of the diffusion model. Additionally, it provides access to an efficient and compact latent space where high-frequency and imperceptible details are abstracted away, allowing the model to focus more on the important and variable genomic fragments.

In this study, we propose ATAC-Diff, a versatile informative latent diffusion model, to analyze and generate scATAC-seq data by introducing semantically meaningful latent space within a unified framework. Inspired by diffusion autoencoder[19] and InfoVAE[20], we introduce an auxiliary encoder equipped with Gaussian Mixture Model (GMM) as the prior of the latent space to learn the cell representation. The low dimensional cell embeddings could enhance the diffusion model to capture intrinsic high-level factors of variation present in the heterogeneous scATAC-seq data. By optimizing the mutual information between the cell embeddings and real data points, ATAC-Diff could avoid ignoring the latent variables as the conditional information when utilizing ELBO as the objectivity[21]. To the best of our knowledge, we are the first to utilize the diffusion model for scATAC-seq data generation and analysis. We conduct comprehensive experiments to demonstrate that ATAC-diff achieves SOTA or comparable performances across a range of tasks, including realistic and conditional generation, denoising, imputation, and subgroup clustering, compared to the existing models specifically designed for these tasks.

We summarize the contributions of this work as follows:

We propose a versatile framework ATAC-Diff which can generate in silico scATAC-seq data with conditional information and reverse the genomic latent representation to reveal the underlying principles.

We are the first study to harness diffusion based models for scATAC-seq data generation and analysis.

We equip the diffusion backbone module with an auxiliary encoder which utilizes GMM as the latent feature extractor and mutual information to regulate the variational objectivities.

We provide evidence that ATAC-Diff exhibits the capability to yield high-quality scATACseq data and cell latent embeddings, surpassing or achieving performance levels comparable to SOTA models.

2 Background

The diffusion model is formulated as two Markov chains: diffusion process and reverse process. Given a variance preserve schedule, the diffusion process transforms the real data x 0 to a latent variable distribution p ( x 0: T ) and finally the data is diffused into predefined Gaussian noise x T

through the time step setting t = 1 ...T . The transform distribution is formulated as a fixed Markov chain which gradually adds Gaussian noise to the data with a user-defined variance schedule:

where x t is the state of x at the time step t which is obtained by adding Gaussian noise to the former state x t -1 and β t controls the degree of the noise. The distribution delineates a Gaussian distribution centered on the incrementally corrupted data state x t . We could obtain a closed form from the input data x 0 to x T in a tractable way. The posterior distribution could be factorized as:

Let ¯ α t = ∏ t s =1 1 -β s , we could sample x t at any arbitrary time step t in a closed form by utilizing the reparameterization trick[9]:

If the time step is sufficiently large, the final distribution will approach that of a standard Gaussian distribution. The reverse process aims to reconstruct the original data x 0 from the diffused data x T , sampled from Gaussian distribution N (0 , I ) , which is accomplished through the diffusion process. The reverse process can also be factorized as a Markov chain:

We aim to investigate the iterative reverse process p ( x t -1 | x ) . However, Estimating the distribution p ( x t -1 | x t ) is hard to estimate unless the gap between t -1 and t is infinitesimally small ( T = ∞ ) [10]. Therefore, a learned Gaussian transitions p θ ( x t -1 | x t ) is devised to approximate p ( x t -1 | x t ) at every time step:

Following previous work ([9]), µ θ ( x t , t ) can be modeled as:

where ϵ θ is a neural network w.r.t trainable parameters θ . Having formulated the reverse process, we could maximize the likelihood of the training data as our object. Since directly calculating the likelihood is intractable, we adopt the evidence lower bound (ELBO) [9] to optimize.

3 Method

In this section, we formally present our proposed model ATAC-Diff for scATAC-seq data analysis and generation. As illustrated in Figure 1, ATAC-Diff is based on the conditional latent diffusion model equipped with the auxiliary module which provides the additional latent variables. Specifically, we use fragment counts to represent the scATAC-seq data. Then we compress the scATAC-seq data (fragment counts) into lower dimensional latent space. Here, we adopt the autoencoder (AE) framework, where the encoder EncAE encodes raw data x raw into latent domain x 0 = EncAE ( x raw ) and the decoder DecAE learns to decode x 0 back to raw data x raw. This framework can be trained by minimizing the reconstruction objective. The auxiliary encoder z = Enc ϕ ( x 0 ) learns to map

an input ATAC-seq data to a semantically meaningful representation z . By incorporating a latent variable of this nature, the diffusion model can prioritize high-level semantic information during the generation process, thereby producing data of superior quality. Additionally, this approach enables the utilization of meaningful representations at the hidden layer for downstream task analysis. Our work is inspired by the recent success of Diffusion Autoencoders[19], but learning latent variables for the sparse scATAC-seq data is however challenging. We address this by introducing GMM to extract the distinct features from the latent space and utilize mutual information to maximize the shared information between latent variables and real scATAC-seq data. We elaborate on the auxiliary module in Section 3.1 and the conditional diffusion model in Section 3.2. Finally, we briefly describe the training and sampling scheme in section 3.3.

X

!"#

X

!

X

$

X

%

Auxiliary

Encoder

Auxiliary

Decoder

𝒛

…

Mutual information

Maximization

$

, 𝑧)

Latent generator

(VAE)

GMM

Conditional LDM

𝑝

&

𝑿

!"#

𝑿

!

, 𝑧, 𝑦

𝑞 𝑿

!

𝑿

!"#

, 𝑿

$

Clustering

Cells

×Fragment Counts

Sample

3.1 Informative auxiliary module

3.1.1 Semantic encoder

The goal of the informative auxiliary module is two-fold. The first is to summarize an input scATACseq data into a semantic representation z = Enc ϕ ( x 0 ) which contains fine-grained information to assist the diffusion decoder p θ ( x t -1 | x t , z ) denoise and predict the output scATAC-seq data. The second one is that the meaningful representation yielded by the auxiliary module could be utilised for downstream analysis such as cell relationships visualization and cell heterogeneity identification.

The latent variables z , unlike the latent variables in the diffusion process, are flexible and can represent a low-dimensional vector of latent factors of variation which could be either continuous or discrete. In order to infer z , we design the approximate posterior q θ ( z | x 0 ) which could be any architecture for the encoder. In our experiments, the encoder shares the same architecture in the diffusion backbone as the transformer encoder, which is formulated as:

where FFL is the feed-forward layer, MHA is the multi-head self-attention layer.

3.1.2 Semantic prior

To learn the latent distribution of the input data and enable the unconditional sampling of x 0 in the diffusion model, we model the semantic prior p ( z ) over data features. Usually, the prior over the latent variables is commonly an isotropic Gaussian[22]. However, utilizing a Gaussian distribution as

the prior probability distribution p ( z ) restricts the latent representation to a single mode while the raw features of the scATAC-seq data are multi-modal with different cell types [23]. Thus, we apply GMMas the posterior to learn multi-modal latent representations[8, 24]. We predefined categorical c ∈ { 1 , ...K } where K is a predefined number of components in the mixture, e.g., the number of cell types. The joint probability p ( x 0 , z, c ) could be modeled as:

where p ( z | c ) is mixture of Gaussian distribution parametrized by a series of µ c and σ c . Then we define each factorized probability as:

We employ Kullback-Leibeler (KL) divergence to calculate the regularization term to enforce the latent variable z to the GMM manifold: R GMM = D KL ( q ϕ ( z , c | x 0 ) ∥ p ( z , c ))

3.1.3 Consistent latent variables with mutual information maximization as regularizer

Diffusion models can be considered as a special realisation of hierarchical V AE[25] which the encoder contains no learnable parameters. The iterative decoder model a complex decoding distribution p θ ( x 0 | x 1: T , z ) but suffers from the ignoring of low-dimensional latent variables z [20] since highdimensional x 0 does not depend on z , leading to degrading to unconditional generation. We tackle this issue by maximizing the mutual information (MI) between z and x 0 , assuming meaningful semantic variables reserve high MI with the real data point. We define MI of x 0 and z as:

where H ( z ) and H ( z | x 0 ) are the marginal information entropy and conditional information entropy, and q ϕ ( z ) is the parameterized posterior of mixture of Gaussian distribution. Maximization of the mutual information enables the model to generate x 0 which can infer z , thus avoiding the ignorance issue.

Finally, we employ an auxiliary decoder to reconstruct the latent variables to recover the real data point. The auxiliary decoder could help the auxiliary module to consistently generate semantic variables z by maximizing the likelihood of p ( x 0 | z ) , which can be considered as complementary information to mutual information I ( x 0 ; z ) . Furthermore, this could prevent the latent variable z from generating into the pure mixture of Gaussians, which in turn interferes with the generation of x 0 . We define the reconstruction loss as:

3.2 Conditional diffusion decoder

Having formulated the latent auxiliary module, we conditioned the diffusion models on the latent variables z and other conditional information y such as cell types, tissue, and other omics data (e.g. scRNA-seq data). Then the learning objectivity is converted to p θ ( x t -1 | x t , z, y ) . Hence, Eq. 4 and Eq. 5 becomes the conditional format:

Since the diffusion model predefined the diffusion process by the user-defined variance schedules, we could encode x 0 to the stochastic x t by running the deterministic process in Eq. 3:

where ϵ is sampled from N (0 , I ) . Similar to the auxiliary module, we also utilize transformer to model µ θ ( x t , t, z, y ) . For the conditional information y embedding, we also utilize MLPs to convert the raw representation of the features to the final conditional set: ϕ ( y ) = { MLP i ( y ) } . where i ∈ 1 , ...L .

Then we utilize the cross-attention mechanism to enable the prediction of x 0 to be conditioned on the specific attribute information y and z .

where W Q , W K , W V are parameterized linear layers. For the categorical condition information such as cell type, we train a classifier on the latent embeddings z , and then select the corresponding the latent embeddings for certain cell type.

3.3 Training process

Having formulated the informative auxiliary and the conditional diffusion decoder, the training of the reverse process is performed by optimizing the evidence lower bound (ELBO) on negative loglikelihood (conditional form of Eq. 7) with the auxiliary module regulation. We train ATAC-Diff by the following form:

The full derivation can be found in the Appendix. The third term and fourth term can be considered as the ELBO objectivity of VAE model, which are feasible to calculate. However, the last term is intractable to calculate since we cannot evaluate q ϕ ( z ) . Following [20], we could sample z ∼ q ϕ ( z | x 0 ) which x 0 ∼ q ( x 0 ) and then optimize it by likelihood free optimization techniques [26]. Empirically, the ELBO of diffusion models could be simplified as:

In fact, we can directly model µ θ by utilizing x θ instead of ϵ θ since Eq.6 can be rewritten as:

In practice, we find that predicting x 0 will enable the model to converge faster. We speculate that the reason for this phenomenon is that the scATAC-seq data is highly sparse with noise, predicting ϵ from x t which is close to the white noise is very hard, leading to a decrease in sampling quality. For the reconstruction loss of the AE, we adopt L2 norm loss.

3.4 Sampling process

ATAC-Diff is different from vanilla Diffusion models which is conditioned on the latent variables. In order to sample from ATAC-Diff model, we need to design a sampling strategy to sample z from the latent distribution. Since the prior distribution of latent variables is the mixture of Gaussians, it is hard to sample z for unconditional generation. Therefore, we could train any arbitrary generative model to sample z . In this study, we employ a vanilla V AE to sample the latent variables. For data denoising and imputation, we just use the same data as the inputs for both diffusion model and the auxiliary module. In general, we could calculate the mean of the reverse Gaussian transitions u θ by Eq. 18. To sample the scATAC-seq data, we first sample the chaotic state x T from N (0 , I ) or use the desired noised data as inputs. z is sampled by the generative model or has the same value of x T . The next less chaotic state x t -1 is generated by Eq. 5. The final state x 0 is iteratively sampled x t -1 for T times.

4 Experiments

In this section, we evaluate our proposed model ATAC-Diff across a range of experiments on three benchmark datasets, utilizing metrics which span generation quality, denoising and imputation effects, and latent space representation analysis. The extensive experimental results suggest that ATAC-Diff achieves higher or competitive performances compared with SOTA models which are designed for individual tasks.

4.1 Datasets

We adopt three datasets to benchmark our model and baseline models: Forebrain [27], Hematopoiesis [28], and PBMC10k 3 . Forebrain dataset is derived from P56 mouse forebrain. Hematopoiesis dataset includes 2,000 cells during hematopoietic differentiation through FACS. PBMC10k dataset comprises peripheral blood mononuclear cells isolated from a healthy donor, with granulocytes selectively removed through cell sorting, resulting in a cell population of approximately 10,000 cells for detailed analysis.

4.2 Metrics

For the latent representation analysis, we extract the latent representation of cells. We evaluate the clustering results of latent representations by Normalised Mutual Information (NMI), Adjusted Rand Index (ARI), Homogeneity score (Homo) and Average Silhouette Width(ASW). To evaluate the similarities between the generated and real cells, we adopt Spearman Correlation Coefficient (SCC) and Pearson Correlation Coefficient (PCC) for performance comparison.

4.3 Latent representation analysis

4.3.1 Experiment setting

We explore the quality of the latent representations by clustering to examine if it could be used to identify cell types. For performance comparison, we apply Highly Variable Peak (HVP) method from SCANPY [29], cisTopic[7], SCALE[8], and PeakVI[5] to obtain the corresponding dimensionality reduction features. Specifically, we leverage K-Means to obtain the clusters. We evaluate the clustering performance based on NMI, ARI, Homo, and ASW.

4.3.2 Experimental results

The clustering results of ATAC-Diff and the baseline models on 3 benchmark datasets are presented in Table 8. The first, second and third highest values are colored by red , green and purple . The higher the value if the metrics, the better the clustering performance performance. Among the 3 datasets based on the four metrics, ATAC-Diff yields 8 best scores and comparable scores compared with the baseline models. For instance, ATAC-Diff outperforms all the baseline models across all the metrics on Hematopoiesis dataset, defeats SOTA models by at least 1.25% in NMI, 7.66% in ARI, 0.91% in Homo, and 10.16% in ASW. Furthermore, we observe that ATAC-Diff and SCALE achieve similar performance on Forebrain dataset but ATAC-Diff surpasses it and achieves competitive performances on PBMC10k dataset.

HVP

PCA

peakVI

cisTopic

SCALE

ATAC-Diff

Forebrain

Hematopoiesis

16

10

14

8

UMAP dim 2

1

wip dwwn

UMAP dim 2

2 wp dwwn

EX1

AC

EX2

1

2

2

0

4

1

2 -

UMAP dim 1

7

9

10

6

UMAP dim 1

8

10

12

14

16

4

UMAP dim 1

8

10

12

14

-2

0

2

UMAP dim 1

4

6

-4

2

UMAP dim 1

-2

0

UMAP dim 1

2

6

10

z wp dvwn

10

12

z wp dvwn

12

10

0 J

z wip dvwn

s

15.0 1

12.5

10.0

5.0

2.5

00

12

10

8

2

UMAP dim 2

12.5

10.0

2.5 *

2.5

0.0

501

15

CMP

GMP

HSC

MEP

MPP

UNK

mono

pDC

-24

23

UMAP dim 1

21

20

s?-

2.55.07.510.0

UMAP dim 1

5

0

UMAP dim 1

5

10

15

UMAP dim 1

5

10

2.50.0

2.55.0 7.5

UMAP dim 1

10.012.5

0

5

UMAP dim 1

10

i5

To determine whether the methods effectively separate the cell types in the latent space, we employ UMAP to visualize the extracted features (Figure 2). Our analysis reveals that ATAC-Diff exhibits superior performance in distinguishing the cell types, whereas HVP fails to differentiate between the cell types. Furthermore, some of the other methods show overlapping results for certain cell types. In addition, our findings demonstrate that ATAC-Diff has the ability to unveil the distance between distinct cell subpopulations and their developmental trajectories. For instance, we observed that the three excitatory neuron cell types (EC1, EC2, EC3) from the Forebrain dataset exhibit close proximity to each other in the latent space of ATAC-Diff. Conversely, the distances between different cell types are significantly larger, indicating distinct cellular identities. In the Hematopoiesis dataset, we observed that the Multipotent Progenitor Cells (MPPs), which are downstream progenitor cells derived from Hematopoietic Stem Cells (HSCs), exhibit close proximity to each other in the latent space of ATAC-Diff. This finding aligns with their known differentiation relationship, further supporting the accuracy and biological relevance of our analysis.

4.4 Generation quality measurement

4.4.1 Experimental settings

In this section, we have devised two strategies to evaluate the generation capabilities of ATAC-Diff. The first strategy involves unconditional generation, where we assess whether the scATAC-seq data generated by ATAC-Diff exhibits realistic characteristics. If our model excels in capturing the data distribution, it can be leveraged to create simulated data for augmentation purposes rather than sequencing additional cells, thus conserving time and resources. Furthermore, visualizing the synthetic data generated can enhance researchers' comprehension of the distribution patterns and fundamental structure of scATAC-seq data. In this case, we anticipate that ATAC-Diff will be able to generate cells that possess the specific subpopulation features associated with the provided conditional information. Since we are the first to generate scATAC-seq data from scratch, there are no baseline models for comparison. Therefore, we have made modifications to SCALE and PeakVI, which are VAE-based models, to enable unconditional generation by utilizing random noise as inputs. For conditional generation, we have replaced the VAE module of SCALE with a conditional VAE module 4 . For the unconditional generation, we average all single cells as the ground truth. For the conditional generation, we average all single cells of the same biological cell type as the ground truth. To address the requirement of ATAC-Diff for latent variables as auxiliary information, we employ a vanilla V AE to sample the latent variables. We evaluate the generation performance based on the SCC and PCC between the mean values of generated samples and the two ground truth data.

4.4.2 Experimental results

We report the results of unconditional generation in Table 6. We observe that ATAC-Diff outperforms other methods in generating scATAC-seq data from scratch, yielding more realistic results. Specifically, ATAC-Diff achieves the highest SCC of 0.892 and PCC of 0.969 on Forebrain dataset, SCC of 0.822 and PCC of 0.973 on Hematopoiesis dataset, and PCC of 0.983 on PBMC10k dataset. Table 6 also displays the mean correlation values of each cell type in the context of conditional generation. We observe that ATAC-Diff outperforms conditional SCALE among all the metrics. In addition, the performance of ATAC-Diff without conditional information degraded, indicating that ATAC-Diff can fuse the conditional cell type information to generate high-quality scATAC-seq data of specific cell types. Overall, we prove that ATAC-Diff is capable of generating realistic scATAC-seq data.

4.5 Data denoising and imputation

4.5.1 Experimental settings

The scATAC-seq data always contains both noised and a large number of missing values due to dropout events[30]. We design two approaches to test the ability of the method for denoising and recovering missing values (imputation) to address the issues under real scenarios. For scATACseq data denoising, analyzing real datasets poses a challenge due to the lack of ground truth data. Following previous work[8], we mitigate this challenge to average all single cells within the same cell type, resulting in a meta-cell that serves as a good approximation of those individual cells. For scATAC-seq data imputation, previous works chose to create a corrupted matrix by randomly dropping out some non-zero entries. In our case, we follow the setting used in DCA [31] to simulate dropout events by masking 10% of the non-zero counts and setting them to zero. The probability of masking a particular non-zero count follows an exponential distribution which peaks with lower expression levels have a higher likelihood of dropout compared to genes with higher expression levels[32]. For comparison, we choose SCALE and peakVI as baseline models and SCC and PCC as metrics.

4.5.2 Experimental Results

The results of scATAC-seq data denoising and imputation of ATAC-Diff and baseline models are presented in Table 7 where the best result is highlighted in bold. For the data denoising evaluation, ATAC-Diff achieves the highest correlation of the denoised single cells with the corresponding meta-cells of each cell type on PBMC10k dataset and competitive results on the other two datasets. Although the other generative models PeakVI and SCALE perform well on some of the datasets, their performances are not robust. One potential reason is that the auxiliary latent variables enable ATAC-Diff to capture the high-level semantics information. Similarly, ATAC-Diff performs stable on scATAC-seq data imputation, where the generated data is closely related to the meta-cells.

5 Conclusion

In this work, we propose a versatile framework ATAC-Diff which is based on a diffusion model conditioned on the latent variables for scATAC-seq data analysis and generation. ATAC-Diff could utilize the proposed auxiliary module to yield the latent variables which contain high-level meaningful

information; it could be incorporated to assist the data sampling during diffusion. Additionally, these variables could be examined for downstream analyses such as cell type annotation. Based on the formulation of the mutual information, we prevent the model from ignoring the low-dimensional latent variables. Furthermore, we employ a Gaussian Mixture Model (GMM) as the latent prior and auxiliary decoder to enhance the recovery of real data, allowing the latent variables to learn the genomic semantics.

We conducted extensive experiments to evaluate the performance of ATAC-Diff on three datasets and compared it with SOTA models, indicating that the ATAC-Diff model can generate high-quality data in silico scATAC-seq data while effectively disentangling the cell embeddings. An intriguing potential direction for ATAC-Diff involves exploring various conditional generation scenarios for genomic discovery, such as identifying the connections between scRNA-seq data and perturbation prediction. We anticipate that ATAC-Diff can contribute to the advancement of genomic analysis in the near future of personalized medicine at the single-cell level.

6 Acknowledgment

The work was supported in part by NIH grants R01 AG081017-01, R01 AG067151, and HG01200901. The work was partially supported by the grant from the Research Grants Council of the Hong Kong Special Administrative Region [CityU 11203723]. The work described in this paper was partially supported by the grants from City University of Hong Kong (2021SIRG036, CityU 9667265, CityU 11203221) and Innovation and Technology Commission (ITB/FBL/9037/22/S).

References

Jason D Buenrostro, Beijing Wu, Ulrike M Litzenburger, Dave Ruff, Michael L Gonzales, Michael P Snyder, Howard Y Chang, and William J Greenleaf. Single-cell chromatin accessibility reveals principles of regulatory variation. Nature , 523(7561):486-490, 2015.

Sandy L Klemm, Zohar Shipony, and William J Greenleaf. Chromatin accessibility and the regulatory epigenome. Nature Reviews Genetics , 20(4):207-220, 2019.

Mohamed Marouf, Pierre Machart, Vikas Bansal, Christoph Kilian, Daniel S Magruder, Christian F Krebs, and Stefan Bonn. Realistic in silico generation and augmentation of single-cell rna-seq data using generative adversarial networks. Nature communications , 11(1):166, 2020.

Romain Lopez, Jeffrey Regier, Michael B Cole, Michael I Jordan, and Nir Yosef. Deep generative modeling for single-cell transcriptomics. Nature methods , 15(12):1053-1058, 2018.

Tal Ashuach, Daniel A Reidenbach, Adam Gayoso, and Nir Yosef. Peakvi: A deep generative model for single-cell chromatin accessibility analysis. Cell reports methods , 2(3), 2022.

Han Yuan and David R Kelley. scbasset: sequence-based modeling of single-cell atac-seq using convolutional neural networks. Nature Methods , 19(9):1088-1096, 2022.

Carmen Bravo González-Blas, Liesbeth Minnoye, Dafni Papasokrati, Sara Aibar, Gert Hulselmans, Valerie Christiaens, Kristofer Davie, Jasper Wouters, and Stein Aerts. cistopic: cisregulatory topic modeling on single-cell atac-seq data. Nature methods , 16(5):397-400, 2019.

Lei Xiong, Kui Xu, Kang Tian, Yanqiu Shao, Lei Tang, Ge Gao, Michael Zhang, Tao Jiang, and Qiangfeng Cliff Zhang. Scale method for single-cell atac-seq analysis via latent feature extraction. Nature communications , 10(1):4576, 2019.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems , 33:6840-6851, 2020.

Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning , pages 2256-2265. PMLR, 2015.

Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems , 32, 2019.

Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning , pages 8162-8171. PMLR, 2021.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10684-10695, 2022.

William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 4195-4205, 2023.

Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. arXiv preprint arXiv:2009.09761 , 2020.

Rongjie Huang, Jiawei Huang, Dongchao Yang, Yi Ren, Luping Liu, Mingze Li, Zhenhui Ye, Jinglin Liu, Xiang Yin, and Zhou Zhao. Make-an-audio: Text-to-audio generation with prompt-enhanced diffusion models. arXiv preprint arXiv:2301.12661 , 2023.

Lei Huang, Hengtong Zhang, Tingyang Xu, and Ka-Chun Wong. Mdm: Molecular diffusion model for 3d molecule generation. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 5105-5112, 2023.

Emiel Hoogeboom, Vıctor Garcia Satorras, Clément Vignac, and Max Welling. Equivariant diffusion for molecule generation in 3d. In International conference on machine learning , pages 8867-8887. PMLR, 2022.

Konpat Preechakul, Nattanat Chatthee, Suttisak Wizadwongsa, and Supasorn Suwajanakorn. Diffusion autoencoders: Toward a meaningful and decodable representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10619-10629, 2022.

Shengjia Zhao, Jiaming Song, and Stefano Ermon. Infovae: Information maximizing variational autoencoders. arXiv preprint arXiv:1706.02262 , 2017.

Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Improved variational inference with inverse autoregressive flow. Advances in neural information processing systems , 29, 2016.

Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 , 2013.

Christopher Heje Grønbech, Maximillian Fornitz Vording, Pascal N Timshel, Casper Kaae Sønderby, Tune H Pers, and Ole Winther. scvae: variational auto-encoders for single-cell gene expression data. Bioinformatics , 36(16):4415-4422, 2020.

Nat Dilokthanakul, Pedro AM Mediano, Marta Garnelo, Matthew CH Lee, Hugh Salimbeni, Kai Arulkumaran, and Murray Shanahan. Deep unsupervised clustering with gaussian mixture variational autoencoders. arXiv preprint arXiv:1611.02648 , 2016.

Arash Vahdat and Jan Kautz. Nvae: A deep hierarchical variational autoencoder. Advances in neural information processing systems , 33:19667-19679, 2020.

Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference on machine learning , pages 214-223. PMLR, 2017.

Sebastian Preissl, Rongxin Fang, Hui Huang, Yuan Zhao, Ramya Raviram, David U Gorkin, Yanxiao Zhang, Brandon C Sos, Veena Afzal, Diane E Dickel, et al. Single-nucleus analysis of accessible chromatin in developing mouse forebrain reveals cell-type-specific transcriptional regulation. Nature neuroscience , 21(3):432-439, 2018.

Jason D Buenrostro, M Ryan Corces, Caleb A Lareau, Beijing Wu, Alicia N Schep, Martin J Aryee, Ravindra Majeti, Howard Y Chang, and William J Greenleaf. Integrated single-cell analysis maps the continuous regulatory landscape of human hematopoietic differentiation. Cell , 173(6):1535-1548, 2018.

F Alexander Wolf, Philipp Angerer, and Fabian J Theis. SCANPY: large-scale single-cell gene expression data analysis. Genome Biol. , 19(1):15, February 2018.

David Van Dijk, Roshan Sharma, Juozas Nainys, Kristina Yim, Pooja Kathail, Ambrose J Carr, Cassandra Burdziak, Kevin R Moon, Christine L Chaffer, Diwakar Pattabiraman, et al. Recovering gene interactions from single-cell data using data diffusion. Cell , 174(3):716-729, 2018.

Gökcen Eraslan, Lukas M Simon, Maria Mircea, Nikola S Mueller, and Fabian J Theis. Singlecell rna-seq denoising using a deep count autoencoder. Nature communications , 10(1):390, 2019.

Luke Zappia, Belinda Phipson, and Alicia Oshlack. Splatter: simulation of single-cell rna sequencing data. Genome biology , 18(1):174, 2017.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 , 2017.

A Proof of the diffusion model

We provide proofs for the derivation of several properties in ATAC-Diff.

A.1 Marginal distribution of the diffusion process

In the diffusion process, we have the marginal distribution of the data at any arbitrary time step t in a closed form: q ( x t | x 0 ) = N ( x t ; √ ¯ α t x 0 , (1 -¯ α t ) I ) .

Proof : recall the posterior q ( x t | x t -1 ) in Eq. 3, we can obtain x t using the reparameterization trick. A property of the Gaussian distribution is that if we add N ( 0 , σ 2 1 I ) and N ( 0 , σ 2 2 I ) , the new distribution is N ( 0 , ( σ 2 1 + σ 2 2 ) I )

where α t = 1 -β t , ϵ and ˆ ϵ are sampled from independent standard Gaussian distributions. Then we could derive Eq. 3

A.2 The derivation of parameterized mean µ θ

A learned Gaussian transitions p θ ( x t -1 | x t ) is devised to approximate the q ( x t -1 | x t ) of every time step: p θ ( x t -1 | x t ) = N ( x t -1 ; µ θ ( x t , t ) , σ 2 t I ) . µ θ is parameterized as follows:

Proof : the distribution q ( x t -1 | x t ) can be expanded by Bayes' rule:

where C ( x t , x 0 ) is a constant. We can find that q ( x t -1 | x t ) is also a Gaussian distribution. We assume that:

Then we could parameterize µ θ ( x t , t ) = √ α t (1 -¯ α t -1 ) 1 -¯ α t x t + √ ¯ α t -1 β t 1 -¯ α t x θ , which is presentned in Eq. 18. From Eq. 19, we have x t == √ ¯ α t x 0 + √ 1 -¯ α t ¯ ϵ . We take this into ˜ µ :

µ θ is designed to model ˜ µ . Therefore, µ θ has the same formulation as ˜ µ but parameterizes ϵ : µ θ ( x t , t ) = 1 √ α t ( x t -β t √ 1 -¯ α t ϵ θ ( x t , t ) ) .

A.3 The derivation of loss function

It is hard to directly calculate the conditional log-likelihood of the data. Instead, we can derive its ELBO objective for optimizing. For simplicity, the c in GMM is omitted

Then we further derive the conditional ELBO objective:

A.4 Proof of the ATAC-Diff Objectivity

For simplicity, we utilize the last term log p θ ( z ) q ϕ ( z | x 0 ) and the regular term of GMM and mutual information to obtain the final form of the optimizing objectivity in Eq. 7 (We also omit c in GMM for simplicity).

Finally, we could derive the objectivity as:

B Experiments details

B.1 Dataset and implementation

We summarize the statistic information of all processed datasets in Table 4.

For all three datastes, ATAC-Diff is trained by AdamW [33] optimizer for 10K iterations with a batch size of 256 and a learning rate of 0.0001 on one NVIDIA V100 GPU card. We set α as 0.1 and λ as 1.1001.

B.2 Ablation study

In this study, we have removed the GMM and MI modules to conduct the ablation study for each task.

B.3 Euclidean distances of different cell types

We present a similarity matrix of the latent embedding based on Euclidean distances. Specifically, we average the latent embeddings within each cell type population and calculate the Euclidean distances across cell types.

Our results show that ATAC-Diff can effectively capture the relationships between distinct cell subpopulations and their developmental trajectories. For example, in the Forebrain dataset, the three excitatory neuron subtypes (EC1, EC2, EC3), which are biologically similar, cluster closely in ATAC-Diff's latent space based on the correlation of their embeddings. In contrast, more biologically distant cell types display greater separation. Similarly, in the Hematopoiesis dataset, Multipotent Progenitor Cells (MPPs) and Hematopoietic Stem Cells (HSCs) exhibit proximity, reflecting their known differentiation path and reinforcing the biological relevance of our findings.

C Additional figures

C.1 The UMAP visualization of the methods on PBMC10k

We have visualized the extracted features by different methods on PBMC10k dataset through UMAP, which is shown in Figure 3

HVP

PCA

peakVI

cisTopic

SCALE

ATAC-Diff

PBMC10k

N

UMAP dim 1

0

2

wp dvwn

N

10

-5

0

UMAP dim 1

5

10

15

20

UMAP dim

N

15

10

20

5

0

0

UMAP dim 1

5

10

15

Z wp dvwn

10

51

UMAP dim 1

5

10

15

Z wp dvwn

10

15

5

口

0

UMAP dim 1

5

10

15

2 wp dVwn

10

15

UMAP dim 1

5

10

15

20

CD4 Naive

CD4 TCM

9fT

POC

CD16 Msto

CD4 TEM

CD8 TEM_1

CD8 TEM_2

HSPC

Intermediate B

MAIT

Memory B

NK

Naive B

Plasmi

69L

cDC

NeurIPS Paper Checklist

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS paper checklist" ,

Keep the checklist subsection headings, questions/answers and guidelines below.

Do not modify the questions and only use the provided macros for your answers .

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification:

[TODO]

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification:

[TODO]

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [No]

Justification: This paper does not include theotrtical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification:

[TODO]

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: We will release the full source code soon. The link is https://github.com/LayneHuang/ATAC-Diff.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification:

[TODO]

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification:

[TODO]

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification:

[TODO]

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification:

[TODO]

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification:

[TODO]

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [Yes]

Justification: This paper poses no such risk.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: [TODO]

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification:

[TODO]

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [No]

Justification: This paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [No]

Justification: This paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a new diffusion model ATAC-diff for scATAC-seq data generation and analysis. ATAC-diff using a latent diffusion model which is conditioned on latent auxiliary models to encode latent variables, and integrate GMM as the latent prior to capture genetic information. This paper introduces a mutual information regularization to maintain the connection between observed and latent variables. Extensive experiments show that ATAC-diff outperforms state-of-the-art models in both data generation and analysis tasks.\n\nSTRENGTHS:\nThis paper introduces a uniform model that is applicable for multiple tasks in scATAC-seq analysis. The method section is clear and easy to follow, providing a clear explanation on auxiliary module, integration of GMM and mutual information regularization. \nThe paper provides an in-depth theoretical analysis, helps the audience to better understand the method. \nThe experiment covers a wide range of scATAC-seq analysis, including clustering, generation, denoising and imputation. The results are comprehensive and shows advantage of the new model.\n\nWEAKNESSES:\nMultiple baseline performance on the benchmark dataset does not match what was reported in previous publications. For example, [1] shows much higher PCA performance on PBMC10k dataset. This discrepancy makes me concern about the reliability of the experiment results. \nIf would be great to include a few benchmark dataset to help cross validate the model performance, for example Buenrostro2018. \nThere are top baselines that are not included in the experiment, please consider add the comparison on the latest baseline as well, such as scBasset. \n[1] Yuan H, Kelley D R. scBasset: sequence-based modeling of single-cell ATAC-seq using convolutional neural networks[J]. Nature Methods, 2022, 19(9): 1088-1096.\n\nQUESTIONS:\n1. Explain the discrepancy on baseline performance and cross validation with previous publications. \n2. Include more benchmark dataset for better cross validation of the model performance, such as Buenrostro2018.\n3. Add top baselines, such as scBasset into comparison.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis submission introduces ATAC-Diff, a conditional latent diffusion model for scATAC-seq data. ATAC-Diff incorporates a few components to a basic diffusion model, including a Gaussian Mixture Model as a semantic prior over the latent variables and mutual information as a regularizer. ATAC-Diff is benchmarked on several tasks: latent representation clustering, (contitional) data generation, and denoising/imputation. ATAC-Diff is shown to do better than PeakVI and SCALE (among other methods for clustering).\n\n\n[UPDATE 8/9]: Adjusting score from Reject to Accept due to clarifications.\n\nSTRENGTHS:\n- It seems to be the first application of a diffusion model to atac-seq data represented as cellsxpeaks\n- ATAC-Diff is benchmaked against other generative models based on VAEs, including SCALE and PeakVI\n- Performance of ATAC-Diff seems to be better than previous methods on 3 tasks (i.e., cell type clustering, data generation, and denoising). \n- ATAC-Diff can generate unconditionally and conditionally with assistance from a VAE\n\nWEAKNESSES:\n- presentation of ATAC-seq data is not clear. The cell x peak representation is a processed version that doesn't necessarily encapsulate the full ATAC-seq data. Clarity that they are not generating ATAC-seq data but rather processed ATAC-seq data in the form of cell x binary peaks is necessary.\n- Example: other approaches to analyze ATAC-seq data incorporate sequences, eg. chromBPnet and AI-TAC.\n- Weak evaluations. The evaluations of cell clustering are not clear. It is not clear how ground truth was determined, given it is likely given by another computational method. The metrics are not clear what they are and their strengths and limitations. The umap visualization is not reliable. The authors state that they observe EC1, EC2, EC3 from the Forebrain dataset are close proximity in latent space. But latent space can be highly warped, rendering any Euclidean distances in umap space not meaningful. \n- Generation quality task is confusing. \n\t- When is unconditional generation needed? Why is this a meaningful task?\n\t- Not clear how the metrics are calculated. What is the ground truth here?\n\t- Why are generated cells averaged? Shouldn't generation be assessed at the single-cell level? \n\t- The ATAC-Diff w.o con seems quite high. This affirms my concern that the evaluations might be weak. \n- The proposed components GMM and MI regularizer are not evaluated via an ablation study. Are they even needed?\n- This study treats ATAC-seq data as a cell x (binary) peaks representation, instead of fragment counts, which improves scATAC-seq analysis (see Martens et al, Nature Methods, 2023).\n\nQUESTIONS:\nQuestions are integrated within Weaknesses (above).\n\nLIMITATIONS:\nThis study seems interesting and their method ATAC-Diff may be an advance, but it is difficult to assess given the poor presentation. There are several weaknesses such as questionable ground truth within the benchmarks. Moreover, the task itself seems to be dated, focusing on cells x binary peaks. Moreover, the components of the proposed approach are not tested in any ablation study, bringing their purpose questionable. This approach seems to be a new method, but one of many in the growing universe of scATAC-seq data and it's not clear what impact this work will have due to these limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nGenerating simulated scATAC-seq data is important for developing new methods and gaining a deeper understanding of the data. However, the simulation is challenging due to dropout and high noise in the data. Authors proposed a diffusion + VAE type of method to solve the problem. The general idea is first to use VAE to project original scATAC data to a lower embedding space. Then impose a diffusion process in the embedding space. The lower embedding space is a GMM rather than the classical isotropic Normal, which is the novelty part of the method. This configuration makes biological sense as cells can be grouped as different cell types. Due to the introduction of GMM distribution in the hidden GMM space, there exists complications in generalising the diffusion loss function. The authors have shown nice and solid derivations in the appendix. The method is then applied to three datasets on three different tasks and achieved comparable  performance as SOTA methods. It seems that authors have provided a convincing solution to the research question. While this paper is clearly written and the general idea is relatively easy to follow, it will be nice if the author could help to answer the following questions.\n1. In Eq. 11, what is the parametric form of q_\\phi(z|x_0)\n2. In Eq. 14, what is the actual meaning of conditional information y, could you list the parameters\n3. Could you provide a concrete network architecture of your network in a supplementary figure, i.e. including the tensors and their dimensions?\n4. In sec 4.4.1, what is the dropout rate distribution in your simulated data, are they similar to the real data?\nDepending on the answers, I may change my ratings in the future.\n\nSTRENGTHS:\nGenerating simulated scATAC-seq data is important for developing new methods and gaining a deeper understanding of the data. However, the simulation is challenging due to dropout and high noise in the data. Authors proposed a diffusion + VAE type of method to solve the problem. The general idea is first to use VAE to project original scATAC data to a lower embedding space. Then impose a diffusion process in the embedding space. The lower embedding space is a GMM rather than the classical isotropic Normal, which is the novelty part of the method. This configuration makes biological sense as cells can be grouped as different cell types. Due to the introduction of GMM distribution in the hidden GMM space, there exists complications in generalising the diffusion loss function. The authors have shown nice and solid derivations in the appendix. The method is then applied to three datasets on three different tasks and achieved comparable  performance as SOTA methods. It seems that authors have provided a convincing solution to the research question. While this paper is clearly written and the general idea is relatively easy to follow, it will be nice if the author could help to answer the following questions.\n1. In Eq. 11, what is the parametric form of q_\\phi(z|x_0)\n2. In Eq. 14, what is the actual meaning of conditional information y, could you list the parameters\n3. Could you provide a concrete network architecture of your network in a supplementary figure, i.e. including the tensors and their dimensions?\n4. In sec 4.4.1, what is the dropout rate distribution in your simulated data, are they similar to the real data?\nDepending on the answers, I may change my ratings in the future.\n\nWEAKNESSES:\nGenerating simulated scATAC-seq data is important for developing new methods and gaining a deeper understanding of the data. However, the simulation is challenging due to dropout and high noise in the data. Authors proposed a diffusion + VAE type of method to solve the problem. The general idea is first to use VAE to project original scATAC data to a lower embedding space. Then impose a diffusion process in the embedding space. The lower embedding space is a GMM rather than the classical isotropic Normal, which is the novelty part of the method. This configuration makes biological sense as cells can be grouped as different cell types. Due to the introduction of GMM distribution in the hidden GMM space, there exists complications in generalising the diffusion loss function. The authors have shown nice and solid derivations in the appendix. The method is then applied to three datasets on three different tasks and achieved comparable  performance as SOTA methods. It seems that authors have provided a convincing solution to the research question. While this paper is clearly written and the general idea is relatively easy to follow, it will be nice if the author could help to answer the following questions.\n1. In Eq. 11, what is the parametric form of q_\\phi(z|x_0)\n2. In Eq. 14, what is the actual meaning of conditional information y, could you list the parameters\n3. Could you provide a concrete network architecture of your network in a supplementary figure, i.e. including the tensors and their dimensions?\n4. In sec 4.4.1, what is the dropout rate distribution in your simulated data, are they similar to the real data?\nDepending on the answers, I may change my ratings in the future.\n\nQUESTIONS:\nGenerating simulated scATAC-seq data is important for developing new methods and gaining a deeper understanding of the data. However, the simulation is challenging due to dropout and high noise in the data. Authors proposed a diffusion + VAE type of method to solve the problem. The general idea is first to use VAE to project original scATAC data to a lower embedding space. Then impose a diffusion process in the embedding space. The lower embedding space is a GMM rather than the classical isotropic Normal, which is the novelty part of the method. This configuration makes biological sense as cells can be grouped as different cell types. Due to the introduction of GMM distribution in the hidden GMM space, there exists complications in generalising the diffusion loss function. The authors have shown nice and solid derivations in the appendix. The method is then applied to three datasets on three different tasks and achieved comparable  performance as SOTA methods. It seems that authors have provided a convincing solution to the research question. While this paper is clearly written and the general idea is relatively easy to follow, it will be nice if the author could help to answer the following questions.\n1. In Eq. 11, what is the parametric form of q_\\phi(z|x_0)\n2. In Eq. 14, what is the actual meaning of conditional information y, could you list the parameters\n3. Could you provide a concrete network architecture of your network in a supplementary figure, i.e. including the tensors and their dimensions?\n4. In sec 4.4.1, what is the dropout rate distribution in your simulated data, are they similar to the real data?\nDepending on the answers, I may change my ratings in the future.\n\nLIMITATIONS:\nGenerating simulated scATAC-seq data is important for developing new methods and gaining a deeper understanding of the data. However, the simulation is challenging due to dropout and high noise in the data. Authors proposed a diffusion + VAE type of method to solve the problem. The general idea is first to use VAE to project original scATAC data to a lower embedding space. Then impose a diffusion process in the embedding space. The lower embedding space is a GMM rather than the classical isotropic Normal, which is the novelty part of the method. This configuration makes biological sense as cells can be grouped as different cell types. Due to the introduction of GMM distribution in the hidden GMM space, there exists complications in generalising the diffusion loss function. The authors have shown nice and solid derivations in the appendix. The method is then applied to three datasets on three different tasks and achieved comparable  performance as SOTA methods. It seems that authors have provided a convincing solution to the research question. While this paper is clearly written and the general idea is relatively easy to follow, it will be nice if the author could help to answer the following questions.\n1. In Eq. 11, what is the parametric form of q_\\phi(z|x_0)\n2. In Eq. 14, what is the actual meaning of conditional information y, could you list the parameters\n3. Could you provide a concrete network architecture of your network in a supplementary figure, i.e. including the tensors and their dimensions?\n4. In sec 4.4.1, what is the dropout rate distribution in your simulated data, are they similar to the real data?\nDepending on the answers, I may change my ratings in the future.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}