Generating Synthetic Genotypes using Diffusion Models

Authors: Philip Kenneweg, Raghuram Dandinasivara Rangaram, Xiao Luo, Barbara Hammer, Alexander Schoenhuth
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
Keywords: Diffusion, Genome, DNA, 1000 Genome, ALS

Abstract:
In this paper, we introduce the first diffusion model designed to generate complete synthetic human genotypes, which, by standard protocols, one can straightforwardly expand into full-length, DNA-level genomes.
The synthetic genotypes mimic real human genotypes without just reproducing known genotypes, in terms of approved metrics. When training biomedically relevant classifiers with synthetic genotypes, accuracy is near-identical to the accuracy achieved when training classifiers with real data. We further demonstrate that augmenting small amounts of real with synthetically generated genotypes drastically improves performance rates. This addresses a significant challenge in translational human genetics: real human genotypes, although emerging in large volumes from genome wide association studies, are sensitive private data, which limits their public availability. Therefore, the integration of additional, insensitive data when striving for rapid sharing of biomedical knowledge of public interest appears imperative.

Decision: reject
Reviewer scores: 10, 8, 5, 1, 5

Reviews and discussion:
<thread>
creation date:
1730424055258

thread_type:
official_review

<thread_object>
creator: reviewer_ydes

<content>
summary:
In this paper, the authors use diffusion models to synthesize genomic data, allowing researchers to access it at a low cost. They expanded the synthesis scale to encompass the entire human genome. To handle long sequences, the authors employed a method similar to Latent-Space Diffusion Model: first splitting the genome into parts. For each part, they used PCA as an encoder/decoder to reduce the dimensionality to 8. The authors applied various methods to test the model's generation accuracy, generalization performance, and efficiency in downstream tasks. Notably, models trained on synthetic data performed almost identically to those trained on real data.

strengths:
As the authors claim, they propose the first model that augments human genomic data at full scale, which is larger compared to previous works. Their experiments demonstrate that their generated data can be used as effectively as real data to train downstream models.

weaknesses:
The paper's most significant weakness is its clarity and writing quality. In the section on previous works, there is only a table without main text, making it difficult to understand how the authors' approach compares to prior research. What are the limitations of previous works beyond their smaller scale? How does the authors' approach relate to existing methods? 

The authors' claim of "using diffusion model" as a novelty is questionable. Diffusion models aren't fundamentally different from other generative models. While using a diffusion model might be justified based on performance, it doesn't constitute a novel contribution in itself.

Lines 92 to 94 are confusing: they state that DDIM is beneficial, but DDPM is also necessary, without propose any solutions. In fact, DDPM is a special case of DDIM with specific alpha and sigma values.

On line 95, the authors mention using classifier-free guidance, claiming it can improve quality. However, classifier-free guidance was developed for conditional generation, not specifically for quality improvement. The authors should clarify their reasoning for this claim.

The experiment section lacks details about the data generation process. Although the model was trained with classifier-free guidance, it's unclear how the data is generated with conditions. The authors should clarify how they generate the data and how they select the conditions for generation.

Several other details also require improvement. The UMAP experiment is distracting and should be removed, as it's highly subjective and doesn't provide useful information. In Figure 2, the horizontal lines are not truly horizontal, and the markers are misplaced. The notations in this figure are unclear, making it difficult to determine which parts they refer to. Figure 3 appears to be a screenshot from Weights & Biases, which is unprofessional. Figure 5 is challenging to read due to its poor layout and formatting.

Regarding the model design, the use of CNNs is questionable due to the lack of translational symmetry. The same vector can represent entirely different information in the 18,279 or 26,624 distinct latent spaces created by PCA.

questions:
My biggest question concerns how exactly you augment the data. There are no details about how you utilize the conditional generating capabilities. Given a small amount of data, how do you select the conditions? What is the relationship between your "seed" training data and the generated data?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730655115751

thread_type:
official_review

<thread_object>
creator: reviewer_iupa

<content>
summary:
In this work, the authors propose a technique for generating complete human genomes. Motivated by the challenges of obtaining individual genomes due to sequencing costs and data privacy concerns, they argue that generating synthetic genomes will improve the performance of predictive models. To address the long-context problem faced by machine learning models processing full genomes, they employ a previously introduced technique to embed genomes in a low-dimensional latent space. A diffusion model is then trained within this space. The authors train their model using classifier guidance on two datasets: one for generating genomes with or without ALS, and another for generating genomes representing 26 populations from the 1000 Genomes Project. To validate their diffusion model, they try to demonstrate (1) that classifiers trained on the generated data achieve comparable accuracy to those trained on real data, and (2) that the generated data distribution is both diverse and sufficiently distinct from the real data distribution.

strengths:
Most current deep learning models in genomics operate at the sequence level, preventing them from processing complete genomes due to their immense length. Additionally, most models rely on the reference genome, hindering their ability to accurately capture intra-individual variants. These limitations, well-documented in the literature, are likely central to the challenges faced in improving these models. Therefore, I appreciate this paper's focus on working at the complete genome level to represent full individuals. Employing an embedding technique to represent full genomes in a low-dimensional space and utilizing latent diffusion within this space is a promising approach, and I am happy to see research in this area. 

Furthermore, the choice of datasets for training the model is well-justified.

weaknesses:
However, the paper presents several limitations:

- The most compelling aspect, in my view, is the embedding of complete genomes into a low-dimensional space and subsequent model training. However, the employed technique is not a novel contribution and lacks sufficient discussion. The authors also fail to adequately address the extent to which complete genomes can be reconstructed from this latent space and the impact on the final sequence. While reconstruction is briefly mentioned, a detailed explanation of the process and precise reconstruction metrics at the sequence level are missing.

- The literature review is incomplete. The authors should include references to D3 [1] and DNADiffusion [2]. While these works have slightly different motivations and are not directly comparable, they warrant mention. Given the discussion of long-range limitations in current models, referencing Enformer [3] and Borzoi [4] would also be beneficial as they represent significant advancements in addressing this issue.

- The paper lacks crucial technical details, hindering a comprehensive understanding of the proposed method. Certain sections require rewriting for improved clarity. Specifically, a dedicated section should detail genome processing and embedding, and another should explain the diffusion model training process, including duration. The model architecture section is difficult to follow, and the ablations lack proper introductions. The section introducing the classifier accuracy recovery metric is overly long and imprecise; it could be condensed into a few concise sentences. Section 3.4.2, while critical, is poorly detailed. The introduced metrics lack explanations, particularly the notation 'd' (presumably representing distance), which is not defined, nor is its measurement space specified. Additionally, the mathematical notations in the equations are incorrect. Measuring the quality and diversity of generated data is crucial in generative modeling and has been a key research area in computer vision. I strongly encourage the authors to refine this aspect, provide thorough introductions to their metrics, and better reference existing literature.

- Employing a convolution-based architecture might be suitable, but the authors should reference existing architectures, use them as a starting point, and discuss necessary modifications for their specific context. Inspiration can be drawn from architectures used in genomics, such as those in D3, Borzoi, BPNet, and Enformer. The 1D U-NET architecture from SegmentNT [5] could also be relevant. Additionally, a vast body of literature exists on 1D convolution and 1D U-NET in signal processing.

- The figures are of poor quality. The cartoons are unclear and lack information, and the font sizes are too small. The evolution plots are difficult to read, and using screenshots from run monitoring software is unacceptable. The authors should regenerate these plots using a plotting library like matplotlib.

- The paper's structure is unconventional. The authors underutilize the allotted space and employ numerous small sub-paragraphs. A denser presentation is expected for a conference like ICLR. The authors should utilize the available space to provide a detailed description of the method and experimental setup.

- The experimental section is also unconvincing. The experimental setup lacks a proper introduction; for instance, the classifier's input and architecture are not specified. As previously mentioned, the privacy metrics are inadequately introduced, making it difficult to assess the reported performance. Lastly, the experiment investigating the use of synthetic data for improving predictors in data scarcity regimes is flawed. When training the classifier on k% of the dataset, the diffusion model should also be trained on k% to accurately reflect the intended use case. In reality, with only k% of the data available, training a diffusion model on 100% is not feasible.

Overall, this paper falls short of the expected quality and contribution standards for ICLR, and I confidently recommend strong rejection.

[1] https://www.biorxiv.org/content/10.1101/2024.05.23.595630v1.abstract

[2] https://www.biorxiv.org/content/10.1101/2024.02.01.578352v1

[3] https://www.nature.com/articles/s41592-021-01252-x

[4] https://www.biorxiv.org/content/10.1101/2023.08.30.555582v1

[5] https://www.biorxiv.org/content/10.1101/2024.03.14.584712v1.full.pdf

questions:
See my points above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730692188481

thread_type:
official_review

<thread_object>
creator: reviewer_nelf

<content>
summary:
The authors develop diffusion models to generate realistic human genotypes. Their claimed contribution is the first models to produce full-length human genotypes. They develop CNN-, CNN+MLP- and Transformer- based models optimized for reconstruction loss. They evaluate on ALS and 1000 Genomes data. They evaluate the ability of the models to augment classifier training and also test the privacy loss associated with generating genotypes with the models.

strengths:
The paper is relatively clear although the language and grammar could use polishing. The claimed contribution is clear and the authors discuss prior work clearly. For the most part the paper is easy to follow. The problem of synthetic data generation is a significant one and the development of synthetic human genomes is surely of interest to some in the research community, although I do not work in this area.

weaknesses:
Although the authors discuss prior work, they do not seem to compare to modeling approaches used in it. It is not clear if this is because of technical limitations or an oversight. The motivation for generating synthetic genotypes to improve ML models is clear, but the authors do not spend enough time discussing the possible negative ethical implications of generating human genotypes. I appreciated the evaluation of privacy loss but the implications were not fleshed out. 

Specific comments:

- All of the figure texts are too small to read.

- change " to `` for front quotes

- " that do not only" -> "that not only"

questions:
I would like a more thorough ethical discussion including, but not limited to, privacy concerns. What would a bad actor do with this technology?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730706937466

thread_type:
official_review

<thread_object>
creator: reviewer_q7fw

<content>
summary:
The authors present a novel approach to generating full-length synthetic human genotypes with diffusion models. They empirically show that the synthetic genotype closely resembles the actual data. They show that the performance of disease and population classifiers was maintained when synthetically generated data was used for training those classifiers. The study also demonstrates that augmenting the training data with synthetically generated data improves the performance of the classifiers. This approach can significantly improve data availability and access and preserve data privacy.

strengths:
1. The paper is well-written and presents a technically sound and well-structured approach.
2. Using diffusion models to generate full-length genotypes is a novel approach. The rationale and motivation for this approach are clearly stated. 
 2. The paper presents a significant contribution to addressing the challenges with data access restrictions and privacy in genome data. 
 3. The paper presents empirical evidence for comparable performance between the models trained with synthetic and real data.

weaknesses:
1. Authors should discuss potential limitations of their current evaluation approach in more detail.
2. Could authors discuss any limitations in generalizing their approach to other genomic datasets?

questions:
1. Can the authors explain why they opted for variable PCA component usage and did not use a consistent number, such as 8, for all genes?
2. Given its relevance for assessing distributional similarity, why was the Frechet Distance not used for holdout evaluation?
3. Could authors clarify their data splitting procedure and discuss any potential implications for the validity of their results?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730743088265

thread_type:
official_review

<thread_object>
creator: reviewer_dfcy

<content>
summary:
This article presents a method for generating full synthetic human genomes using diffusion. Four diffusion architectures are compared, and the quality of the generated genomes is analyzed. The article demonstrates that this method can generate useful synthetic data for geneticists.

strengths:
This article is very exciting. It is incredibly well-written with a clear and enjoyable prose. It tackles an important issue, human genome analysis, with state of the art methods, diffusion models. It presents an interesting comparative study for deep learning in a data regime not often considered, especially in the diffusion literature, but of great import and scientific interest. The insights from the architecture comparison study are useful for a machine learning readership beyond the biomedical application; an understanding of diffusion methods applied to different data types, and how generated data can be measured and made useful, is relevant to ICLR. The demonstration of generating full human genomes is both an impressive showcase of the capabilities of diffusion and an important step in the study of the human genome.

weaknesses:
It would have been helpful to contextualize the results in comparison to other methods, like the work of Szatkownik et al, or HyenaDNA. It is unclear to me how feasible the application of these methods to the full human genome is, but this could have been discussed in the results. However, it should be possible to generate smaller sequences with the proposed diffusion method in order to compare to previous works. The experimental results, as they are, demonstrate the viability of diffusion models on the full human genome (and provide a useful architecture comparison). However, they do not give an indication of the relative quality of the generated data over the data generated by other works.

Other than that, I identified no major weaknesses. Rather, here are some minor quibbles.

+ the plots are not vectors and have small fonts, making them blurry in standard sized readers. pdf plots with larger fonts would help with clarity.
+ line 451: "We draw two conclusions from this:" this is an incomplete sentence, or the next paragraph shouldn't be presented as a new paragraph.
+ eq 2 could be defined inline
+ lines 453 and lines 483 begin paragraphs that say roughly the same thing. Those two paragraphs could be merged or each trimmed.
+ there's an unneeded comma in line 492

questions:
The MLP architecture seems very shallow, especially compared to the CNN, and the first downscale layer, from 147k to 1024, seems very abrupt. What motivated these choices? Were deeper architectures tried?

What is the explanation for the MLP loss not moving? line 363: "even though only small drops in loss during training can be observed for
the MLP, the reconstruction error keeps improving." Is the MLP learning the data distribution? Is it acting as more than a regularization term in the MLP+CNN architecture?

It seems odd that a CNN, albeit with 1D kernels, would outperform a transformer on such long sequences. What are the intuitions behind the transformer architecture not working as well, especially on recovery rates of data coming from the transformer? This seems odd. I agree with the comments on lines 760/761, but am curious if there was more analysis here.

A greater exploration of DDIM could have been nice. Was DDIM tried, and if so, at what timestep reduction? If DDIM is used just for inference but not for training, is there a large decrease in quality?
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents a diffusion based approach to generating synthetic genotype data. The idea is interesting and presents a new direction for the field that merits further exploration. However, as pointed out by reviewers iupA and ydeS, the current manuscript suffers from many issues, most notably immature experimental evaluation that hints at potential utility but does not unambiguously demonstrate it. The paper in its current form is more a proof-of-principle, and an imperfect one at that.