Title: ChromFound: Towards A Universal Foundation Model for Single-Cell Chromatin Accessibiltiy Data
Venue: NeurIPS 2025
Accepted: yes
Keywords: computational biology

scATAC-seq

cell representation
Abstract: The advent of single-cell Assay for Transposase-Accessible Chromatin using sequencing (scATAC-seq) offers an innovative perspective for deciphering regulatory mechanisms by assembling a vast repository of single-cell chromatin accessibility data. While foundation models have achieved significant success in single-cell transcriptomics, there is currently no foundation model for scATAC-seq that supports zero-shot high-quality cell identification and comprehensive multi-omics analysis simultaneously. Key challenges lie in the high dimensionality and sparsity of scATAC-seq data, as well as the lack of a standardized schema for representing open chromatin regions (OCRs). Here, we present ChromFound, a foundation model tailored for scATAC-seq. ChromFound utilizes a hybrid architecture and genome-aware tokenization to effectively capture genome-wide long contexts and regulatory signals from dynamic chromatin landscapes. Pretrained on 1.97 million cells from 30 tissues and 6 disease conditions, ChromFound demonstrates broad applicability across 6 diverse tasks. Notably, it achieves robust zero-shot performance in generating universal cell representations and exhibits excellent transferability in cell type annotation and cross-omics prediction. By uncovering enhancer-gene links undetected by existing computational methods, ChromFound offers a promising framework for understanding disease risk variants in the noncoding genome. The implementation of ChromFound is available via https://github.com/JohnsonKlose/ChromFound.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce a foundation model specifically designed for single-cell chromatin accessibility (scATAC-seq) data. The authors propose a hybrid encoder architecture that integrates Mamba blocks with local windowed self-attention and introduces a genome-aware tokenization scheme for encoding open chromatin regions (OCRs). Trained on 1.97 million cells from over 30 tissues and 6 disease conditions, ChromFound is evaluated on a broad set of downstream tasks: zero-shot cell representation, cell type annotation, cross-omics prediction, and enhancer-gene link inference.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: $\\bullet$ This is a pioneer foundation model explicitly for scATAC-seq. $\\bullet$ In order to address the high dimension issue of scATAC, the authors propose a novel architectural design combining Mamba and windowed attention. $\\bullet$ It demonstrates strong performance across multiple tasks (representation, annotation, prediction).\n\nWeakness: \n$\\bullet$ The novelty of the model architecture is limited. The difference between Chromefound and other single-cell foundation models is that it adopts Mamba to process long sequential data. $\\bullet$ The training loss is not really suitable for the scATAC-seq data since the contextual signals of other regions are not significant with the dropout events. scATAC data is more sparse than scRNA data, however, the foundation model on scRNA does not really achieve robust performance when comparing to single linear regression. $\\bullet$ Unlike scRNA data which the gene set is already defined, scATAC fragment sets are not always the same among different datasets by different depths. The authors do not claim how they unify the formats of different datasets to employ Chromfound. $\\bullet$ As far as I know, there is another scATAC foundation model Epifoundation(https://www.biorxiv.org/content/10.1101/2025.02.05.636688v1) before Chromefound. $\\bullet$  Missing reference like Epifoundation, ATAC-Diff (https://arxiv.org/abs/2408.14801), and GET (https://www.nature.com/articles/s41586-024-08391-z). $\\bullet$ In the ablation study, the performance of Chromfound is close with much less training set. I am wondering if less OCR would decrease the performance.\n\nQUESTIONS:\n1. The authors should compare Chromfound with linear regression in cell type prediction and cross-omics prediction\n2. How do the authors unify the peaks to adapt Chromfound for different datasets?\n3. Does Chromfound achieve different performance across different tissues for all tasks?\n4. Do the authors compare the performance and computational complexity of WPSA and linear attention?\n4. Statistical Significance: Did you do any random seed runs or significance tests for the modest performance gain?\n\nLIMITATIONS:\n1. The novelty of model architecture is limited. There is no well-designed module for scATAC-seq data or tabular data, except for the position embeddings and mamba adoption.\n2. The training loss is not really suitable for the scATAC-seq data since the contextual signals of other regions are not significant with the dropout events.\n3. The downstream tasks are not comprehensive. The authors should conduct more research on perturbation prediction and temporal scATAC-seq prediction.\n4. More ablation studies should be conducted.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors proposed ChromFound, an encoder-decoder framework to build foundation model for scATAC-seq data. Genomic coordinate information and chromatin accessibility profile were used to represent open chromatin regions (OCR). The authors proposed window partition self-attention (WPSA) module to capture local dependencies among OCRs, and Mamba layer to handle ultra-long OCRs. The decoder is MLP layer.   \nChromFound shows improved performance on cell representation accommodating low count and batch effect, which are popular in scATAC-seq data. ChromFound also improves cell type annotation and cross -omics data prediction.Application of ChromFound on enhancer-gene link prediction and perturbation response prediction shows the results make biological sense.\n\nSTRENGTHS AND WEAKNESSES:\nThe submission is technically sound.  The experimental results are well supporting the authors' claim. The methods used are appropriate. The work is almost complete piece of work. The authors mentioned strengths of their work carefully and honestly.\n\nThe submission is clearly written and well organized. It adequately inform the reader. No source code was provided.\n\nThe authors propose for the first time foundation model for scATAC-seq and achieve promising results compared with existing approaches.  It is very likely other researchers will utilize or extend the approach to address more challenges in scATAC-seq data analysis. \n\nThe authors offer a novel combination of existing techniques, and the reasoning behind this combination is well-articulated. The work addresses challenges in generating foundation model in scATAC-seq with suitable techniques and adaption of existing technique when needed. The authors clearly distinguish their contributions with previous contributions.\n\nQUESTIONS:\n1. it is interesting to see how will it perform on gene regulatory network prediction?\n2. A nature way is to incorporate DNA sequence  and or scRNA-seq information into the model, since DNA sequence is already there and scRNA-seq data is more widely available.\n3. minor comments: \n   reference to Table 4.4 should be Table 4\n   In figure 6, the order of figure in Harmony panel is reversed compared with other methods.\n\nLIMITATIONS:\n1. The authors forget to include limitations, such as the model is only on human data, which can be extend to include mouse and other species", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents ChromFound, the first foundation model developed specifically for single-cell chromatin accessibility (scATAC-seq) data. The authors propose a novel hybrid architecture that leverages a Mamba block for long-range genomic context and a windowed self-attention (WPSA) module for local regulatory interactions. A core contribution is the genome-aware tokenization scheme that encodes genomic coordinates and continuous accessibility values, addressing the challenge of heterogeneous data inputs. After pre-training on a large-scale corpus of 1.97 million human cells, the model is comprehensively evaluated across six downstream tasks, where it demonstrates state-of-the-art performance and notable zero-shot capabilities.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n* The work successfully establishes the first foundation model for a challenging and important biological data modality. This is a novel and impactful contribution that charts a new direction for the field.\n* The scale of the pre-training is impressive and a key factor in the model's strong performance. The subsequent evaluation is both rigorous and comprehensive, providing convincing evidence for the model's capabilities across a wide array of relevant tasks.\n* The hybrid Mamba-WPSA architecture is not merely a concatenation of popular methods; it's a well-justified design choice that reflects the multi-scale nature of genomic regulation, which is a commendable feature.\n\n**Weaknesses**\n\nWhile the work is strong overall, there are several areas where a deeper analysis would strengthen the paper's conclusions and provide a more complete picture for the research community. I've ordered these from minor points to more substantial considerations.\n\n* On a minor note, while key hyperparameters are provided, the paper would benefit from a brief discussion on the model's sensitivity to these choices (e.g., the `temp` value in positional embeddings or the Mamba projection dimension). This would provide readers with a better sense of the model's robustness and tuning requirements.\n* A more significant point is that the ablation study, while useful, could be more comprehensive. The current study primarily demonstrates that removing major architectural blocks is detrimental. However, it leaves open important questions about the sources of performance gain. For instance, a comparison to simpler, strong baselines (like a pure Mamba or a pure Transformer architecture) trained on the same massive dataset would be invaluable. Such an analysis would help disentangle the benefits of the proposed hybrid design from the undeniable power of the large-scale pre-training data.\n* Finally, the most substantial area for improvement relates to the model's generalizability and its dependency on upstream analytical choices. The model's performance is intrinsically linked to the input set of Open Chromatin Regions (OCRs), which can vary significantly based on the peak-calling algorithm used. A sensitivity analysis or at least a discussion on this crucial variable would be necessary to fully substantiate the model's claim of universality. Similarly, the model's impressive performance may not fully generalize to disease contexts (e.g., autoimmune disorders) that are not well-represented in the pre-training corpus, a key consideration for its application as a foundational tool for discovery.\n\nQUESTIONS:\n1.  Could you elaborate on the design choice to project to a lower-dimensional space ($D_{low}=32$) for the Mamba block? What were the primary trade-offs you considered between performance and computational efficiency (e.g., parameter count, inference speed)?\n2.  The zero-shot performance for cell representation is quite strong, yet fine-tuning yields considerable gains for tasks like cell type annotation. What is your interpretation of this? Does it suggest that the pre-trained model captures general cell state well, but that cell-type-specific regulatory grammars are only fully resolved during task-specific fine-tuning?\n3.  Regarding the positional embeddings, could you comment on the model's sensitivity to the `temp` hyperparameter? Is performance stable across a range of values, or was this a carefully tuned parameter?\n4.  The fixed window for WPSA is well-justified for typical enhancer interactions. Have you explored its potential limitations for capturing regulatory phenomena over larger genomic distances, and is the Mamba block intended as the primary mechanism for these cases?\n5.  Given the pre-training corpus, what are your expectations for ChromFound's zero-shot or few-shot performance on cell types from disease contexts not well-represented in the training data, such as autoimmune disorders?\n\nLIMITATIONS:\nyes\n\nThe authors appropriately acknowledge the model's human-centric focus as a primary limitation. For a more comprehensive discussion, I would suggest also including a brief commentary on two other practical points: (1) the model's implicit reliance on the quality of user-provided peak calls, which is an important contextual factor for anyone applying the model, and (2) the practical compute resources required for fine-tuning, which is relevant for assessing the model's accessibility to the broader research community.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nChromFound is a foundation model for single-cell ATAC-seq data pretrained on ~2 million cells that aims to learn batch corrected representations of cells that (a) cluster by cell type, (b) can be used to predict cell types for new cells, and (c) can be used to predict other modalities (such as RNA-seq). Across all three aforementioned tasks, the authors show that ChromFound outperforms baseline methods that train on specific datasets and are not pre-trained.\n\nThe model is pre-trained in a self-supervised fashion to predict masked accessibility values. Crucially, the model parametrizes open chromatin regions (OCRs) by their start and end location, as opposed to having a trainable embedding for a specific OCR, allowing the model to work on OCRs from new datasets, which will not exactly match OCRs from the pre-training datasets.\n\nSTRENGTHS AND WEAKNESSES:\nTo my knowledge, ChromFound is the first foundation model for scATAC-seq data. In my opinion, the paper's primary contribution is the demonstration that pre-training on millions of scATAC-seq cells enables zero-shot representation learning for completely new datasets, without requiring additional training or fine-tuning. (I do not think the model itself minus the pre-training is a general advance given the results in Fig. 3 and Table 3, which show that baselines are competitive with a non-pretrained ChromFound model.) This type of pre-training has mostly not been done before because different datasets have different peak/OCR boundaries and harmonizing peak sets is not straightforward. The authors cleverly solve this problem by parametrizing OCRs by a function of their start and end positions, as opposed to using an embedding table. This I find to be their primary computational contribution.\n\nHowever, there are other methods of harmonizing peak sets that would enable other architectures (such as a standard VAE) to be used for pre-training that the authors do not compare to. Computational biologists routinely (i) aggregate fragment files from multiple datasets to re-call a unified set of peaks or (ii) adopt the peak set of a large reference atlas. If fragment files are available for the new dataset, then peak counts can be called specifically against the reference peak set. But even if fragment files are not available and only a count matrix is, counts from the new peak set can be transferred to the reference peak set by examining peak overlaps. Demonstrating that ChromFound outperforms (a) both itself with a different harmonization strategy and (b) other methods pre-trained using one of the aforementioned harmonization strategies would substantially strengthen the paper.\n\nThe other notable weakness of the paper is the benchmarking to baselines. The authors omit several standardly used tools, do not systematically compare against the same tools for all tasks even though most methods are generally applicable, and crucially do not detail how they implement the baselines. For cell clustering and batch-effect correction, Signac, peakVI, and SnapATAC2 are the most commonly used tools and should be compared against.  For cross-omics predictions, Signac and multiVI should also be compared against. Additional recommendations are listed in the \"Questions\" section.\n\nQUESTIONS:\nMajor suggestions/questions:\n1. Please compare to non-pretrained ChromFound for all experiments, not just Fig. 3, Table 3, and Table 4. \n2. Benchmark against Signac (TF-IDF + LSI), peakVI/poissonVI, and SnapATAC2 for cell clustering tasks, as these are the methods most commonly used by practioners. For each, the appendix should indicate the exact pre-processing done before applying the method (e.g. are peaks filtered by their observation frequency before passing them into peakVI?). \n3. Do not use scVI/scANVI for batch-effect correction, as they are designed for RNA-seq. Use peakVI/poissonVI instead.\n4. What input representations do you pass to Harmony for batch-effect correction? PCA or LSI or neither?\n5. What l2 regularization parameter do you use for scBasset for batch-effect correction? How do you choose this regularization parameter?\n6. A small linear layer on top of low-dimensional representations learned by Signac, peakVI, and SnapATAC2 can also be used for cell type annotation. Please compare to those baselines.\n \n\nMinor suggestions/questions:\n1. At test time, when cell type labels are unavailable, how does OCR filtering (as described on lines 611-612) work?\n2. Why do you use an MSE loss after log transformation instead of a Poisson loss on the raw counts, which likely better fits the generative process. \n3. In Fig. 4, are you using ABC predictions as ground-truth? This seems suspect as ABC itself is far from perfect and has several biases (i.e. overly discounting distal enhancers). \n4. Question 3 in the ablations asks if long-context modeling using Mamba layers helps, but I'm not sure if the experiment performed actually answers that question. My reading of Table 4 is that half and three-fourths of the OCRs are just removed from the input data, but that doesn't test the utility of the Mamba layers directly. A better test would be to use all OCRs, but pool after the WPSA layer without having any Mamba layers.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-07"}