Beyond the Alphabet: Deep Signal Embedding for Enhanced DNA Clustering

## Abstract

Abstract The emerging field of DNA storage employs strands of DNA bases (A/T/C/G) as a storage medium for digital information to enable massive density and durability. The DNA storage pipeline includes: (1) encoding the raw data into sequences of DNA bases; (2) synthesizing the sequences as DNA strands that are stored over time as an unordered set; (3) sequencing the DNA strands to generate DNA reads ; and (4) deducing the original data. The DNA synthesis and sequencing stages each generate several independent error-prone duplicates of each strand which are then utilized in the final stage to reconstruct the best estimate for the original strand. Specifically, the reads are first clustered into groups likely originating from the same strand (based on their similarity to each other), and then each group approximates the strand that led to the reads of that group. This work improves the DNA clustering stage by embedding it as part of the DNA sequencing. Traditional DNA storage solutions begin after the DNA sequencing process generates discrete DNA reads (A/T/C/G), yet we identify that there is untapped potential in using the raw signals generated by the Nanopore DNA sequencing machine before they are discretized into bases, a process known as basecalling , which is done using a deep neural network. We propose a deep neural network that clusters these signals directly, demonstrating superior accuracy, and reduced computation times compared to current approaches that cluster after basecalling.

## 1 Introduction

The rapid growth of digital data, projected to reach $180$ zettabytes by 2025, is causing a data storage crisis that cannot be addressed by existing storage technologies [rydning2022worldwide]. In response, deoxyribonucleic acid (DNA) is emerging as a promising alternative storage medium due to its incredible density and durability. The DNA storage process includes four stages illustrated in Figure 1: (1) an “encoding” stage in which binary data files are encoded into DNA strands (design files) using error-correcting code (ECC) [koblitz2000state] schemes that may also overcome errors, (2) a “synthesis” stage, which produces artificial DNA strands of each design strand and are then stored in a storage container [leproust2010synthesis], (3) a “sequencing” stage [anavy2019data, erlich2017dna, organick2018random, yazdi2017portable] which translates a DNA strand into a digital sequence known as a “read”, and (4) a “retrieval” stage where reads are decoded back to binary data files while correcting any errors using the chosen coding methods. Despite the vast potential of DNA storage, current DNA sequencers are yet to overcome challenges such as low throughput and high costs compared to the traditional alternatives [alliance2021preserving, shomorony2022information, yazdi2015dna].

Figure: Figure 1: The stages in the DNA storage pipeline.
Refer to caption: extracted/5911208/figforpaper/fig2new.png

The emerging Nanopore technology offers real-time sequencing of DNA strands with drastically lower costs and portability compared to traditional Illumina sequencing machines  [jain2016oxford, kono2019nanopore]. Despite having higher error rates compared to other sequencing technologies such as Illumina, Nanopore sequencing is gaining significant attention due to its lower cost, portability, and capability to sequence longer strands of DNA. Nanopore sequencing directly reads the nucleic acids by passing them through a nanoscale pore, called a nanopore, embedded in a membrane. As the nucleic acid strand moves through the pore, it impacts the electrical current in a fashion dependent on the current bases in the pore (between four and six at any given time) [mao2018models, 10409278]. The current produced by the DNA molecules passing through the pore is converted into raw analog signals. In Nanopore sequencing, basecalling entails converting these raw signals into nucleotide sequences composed of the bases $\{A,T,C,G\}$  [10409278]. Basecalling is done using deep neural networks.

While Nanopore sequencing benefits from a drastic reduction in costs, it also poses several challenges due to significantly higher sequencing error rates (approximately $10\%$ of bases suffer from edit, insertion or deletion errors as compared to only $1\%$ in Illumina) [chandak2020overcoming]. This paper overcomes these errors by exploiting the raw Nanopore signals which contain significantly more information as compared to the discrete post-basecaller read. Unfortunately, direct utilization of the raw signal is challenging since, to the best of our knowledge, no previous work in DNA storage has directly exploited the signals for processes such as clustering, reconstruction, and error correction [chandak2020overcoming].

A major stage in the DNA storage pipeline involves clustering billions of strings based on the edit distance metric in order to identify which groups of reads were likely duplicated from the same source DNA strand. The edit distance between two strings $x$,$y$ denoted as $d_{e}(x,y)$ is the minimum number of edits, deletions, or insertions required to transform $x$ to $y$ [ristad1998learning]. The complexity of this procedure is caused by several factors: (1) the variability and probabilistic nature of the cluster size due to varying duplicate counts; (2) the number of original clusters in the design files; (3) the protocols used in the experiments, for example, PCR errors pre-sequencing procedure and sampling errors that occur during nanopore sequencing machine; and (4) the fact that the computation of a distance matrix based on the edit distance between each pair of reads is not scalable for typical DNA storage datasets that are very big.

Existing clustering algorithms often suffer from high computational time and compromised accuracy [qu2022clover, zorita2015starcode]. While approximation algorithms and tools such as metric embeddings and locality-sensitive hashing (LSH) offer promising avenues by sacrificing some accuracy for substantial reductions in runtime, their applicability to edit distances remains poorly understood. For the retrieval of a given file, the sequencing and retrieval stages take approximately a number of days to compute; therefore, any reduction in computation time is a major step forward.

At the heart of the challenges of clustering DNA strands for DNA storage lies the impracticality of computing the edit distance between every pair of input strands to create a distance matrix. Due to the sheer volume of data, this is exacerbated by the quadratic time complexity of edit-distance algorithms. Our work addresses this concern by proposing a novel approach: searching for similarity directly in the raw analog DNA signals rather than the discrete DNA reads. Our work introduces a paradigm shift in the field of DNA data storage retrieval by leveraging deep neural network embedding directly on the raw signals instead of the post-basecaller strands. By embedding the raw signals directly in an architecture composed of convolutional neural network (CNN) layers, long-short-term memory layers (LSTM), and linear layers, we bypass the limitations associated with clustering based on the A/T/C/G alphabet, allowing us to cluster raw signals before passing them through the basecaller. This step can significantly reduce the errors introduced during the sequencing phase and lead to a more accurate reconstruction of the original design file.

The key contributions to our work are as follows:

- •
We are the first to perform similarity groupings of DNA strands directly on the raw electrical signals.
- •
The proposed approach significantly reduces sample exclusion because it uses the continuous dimension of electrical signals, which reduces the loss of information during the conversion to the discrete dimension after basecalling.
- •
The proposed approach demonstrates the ability to achieve between 1 to 3 order-of-magnitude faster computation times while producing clusters with high enough quality to allow data reconstruction.

The rest of this paper is organized as follows: Section 2 defines the problem, and Section 3 reviews related work. Section  LABEL:datasets introduces the uniqueness of the datasets and their creation, Section 4 describes the proposed deep learning model, and Section 5 presents an evaluation of the trained ML models across different experiments. Section 6 uses the trained ML models for applications in the DNA storage pipeline, and Section 8 concludes the paper.

## 2 Problem Statement

The main objective of this work is to develop a distance function for raw DNA signals that effectively captures the similarity between them. Thus, the goal is to ensure that raw signals that originate from the same cluster are close together in the metric space, while those that are not are far apart. We formulate the problem as a supervised learning problem, where signals belonging to the same cluster should have small distances between them in the transformed feature space, while signals from different clusters should have large distances between them in the transformed feature space.
More specifically, let, without loss of generality, $C=\{1,\ldots,m\}$ be the set of $m$ labels originating from the design file and let $S=\{s_{1},s_{2},\ldots,s_{n}\}$ be the set of raw DNA signals, each belonging to one of the $m$ labels (we denote the label of $s_{i}$ by $c_{i}$). Our goal is to find a distance function $D$ that is targeted to achieve the following properties:

- 1.
Similarity Preservation: If two samples $s_{i}$ and $s_{j}$ belong to the same class ($c_{i}=c_{j}$), the distance $D(s_{i},s_{j})$ should be minimized.
- 2.
Dissimilarity Preservation: If two samples $s_{i}$ and $s_{j}$ belong to different classes ($c_{i}\neq c_{j}$), the distance $D(s_{i},s_{j})$ should be maximized.
- 3.
Robustness to Noise: The distance function should be robust to noise and variability inherent in raw DNA signal data.
- 4.
Computational Efficiency: The metric should be computationally feasible for large datasets, where $D(s_{i},s_{j})$ computation has to be done in $O(1)$.

## 3 Related Work

The clustering phase of the DNA storage pipeline has been explored solely in the context of clustering the discrete DNA reads. The number of clusters in the clustering phase of the DNA storage pipeline is typically between $1,000$ and $100,000$, while each cluster contains between $10$ and $1,000$ samples. To cluster DNA strands, the similarity between pairs of reads is computed – typically using the edit-distance method. This presents two main challenges: (1) the computational complexity of pairwise comparisons being $\mathcal{O}(n^{2})$ where $n$ is the number of reads, and (2) the $\mathcal{O}(\ell^{2})$ complexity of the edit-distance computation, where $\ell$ represents the reads length. DNA clustering algorithms address these challenges by employing (1) filtering techniques to reduce the number of pairwise comparisons and (2) approximation methods to expedite edit-distance calculations. Both approaches aim to decrease the time complexity with minimal accuracy loss.

Several DNA clustering algorithms in bioinformatics and metagenomics, such as UCLUST [leproust2010synthesis], CD-HIT [fu2012cd], and USEARCH  [srinivasavaradhan2021trellis] (which UCLUST is based on), use different methods to deal with the two challenges. One approach involves using greedy methods for filtering, such as identifying common short sub-sequences as a preliminary step before edit-distance computation. Location Sensitive Hashing (LSH), which is used in [antkowiak2020low, rashtchian2017clustering, ben2023gradhc], is another emerging method that limits the edit-distance calculation to strands that share the same LSH value to make filtering more efficient.

To address the latter challenge, the edit-distance computation, several methods have been proposed. First, one can restrict the computation depth of the edit-distance, as demonstrated in [bao2011seed], to mitigate the complexity. Second, the edit-distance can be calculated based on fixed-sized short sub-strings, as implemented in Starcode [zorita2015starcode] using short DNA bar-codes. Another approach leverages heuristic similarity instead of edit-distance to expedite processing times, as demonstrated in Clover [qu2022clover] utilizing a tree structure.

We acknowledge several works  [ji2021dnabert, li2021bioseq] that have been done in the context of DNA embeddings. Those works differ from ours since they involve correcting strands to align with a protein or gene. It is a process that is irrelevant in our case because our data is synthetic, and we aim to reconstruct our original data without, for example, capturing protein structure.

In the signal domain, a common method for comparing signals is Dynamic Time Warping (DTW) [senin2008dynamic]. DTW is a technique designed to compare two time series data sequences with varying lengths whose data points are not aligned, much like strands in the discrete domain, allowing for insertions, deletions, and substitutions. Similar to edit-distance, DTW’s computational complexity is $\mathcal{O}(m^{2})$, where $m$ represents the signal length. Given that signal lengths are typically ten times longer than DNA strands, DTW computation becomes impractical for large datasets that are used in DNA storage applications. The focus of this work is in the signal domain; therefore, it is imperative to first develop an accurate solution to the second challenge.

## 4 Proposed Deep Learning Scheme

Public DNA storage datasets do not share the raw signals, recall that during during the DNA storage pipeline (Figure 1), the mapping between the DNA strands and their corresponding reads involves an intermediate stage where the DNA strands are converted to raw signals. Therefore, it is necessary to create our own datasets from real experiments that maps raw signals to their corresponding DNA reads.

Each experiment in the DNA storage domain requires choosing an error-correcting code as part of the “encoding” stage for file reconstruction. In addition, a design file is generated for each experiment. The end-to-end experiment process involves DNA strand “synthesis” stage, followed by a “sequencing” stage, the DNA sequencer process approximately lasts $72$ hours. Then basecalling is performed on the DNA reads, which takes approximately a week. Lastly, creating the ground truth for the dataset lasts between $6-30$ days.

Table 1 shows three design files and their corresponding Nanopore sequencing experiments (^1^11The \sayMicrosoft Experiment design file is taken from [srinivasavaradhan2021trellis] and the sequenced data is from [sabary2024reconstruction]. The \sayDeep DNA test and \sayDeep DNA pilot design file and sequenced data are from [bar2021deep].). The table outlines three different experiments that vary the number of clusters, the length of the DNA strands, and the average edit-distance between strands in the design files, therefore resulting in high variability in key parameters for the proposed datasets.

The design files of the different experiments encompass encoded data, index information for orderly reconstruction, and a consistent primer shared among all reads, positioned at the beginning and end of each strand within the file. Subsequently, the files were synthesized by \sayTwist Bioscience, resulting in \sayfast5 [payne2019bulkvis] file formats. The raw signal data is encapsulated in \sayfast5 file formats, which are then processed by a basecaller to convert it into \sayfastq format. The \sayfastq format represents the sequenced data using the following symbols: $\{A,T,C,G\}$. Each \sayfast5 file is linked to a \sayfastq file, with a \sayread-id serving as the connecting identifier. Given that the standard signal length in DNA storage datasets typically reaches up to $3000$ units, for $95\%$ of the cases, we uniformly reshaped all of the signals to this length using common techniques.

The ground truth for clustering is obtained by brute-force calculating the edit distance between each DNA read produced by the basecaller and the original strands from the respective design file. We then assign each DNA read to the cluster corresponding to the closest DNA strand from the design file. It is not practical or scalable to perform a full edit-distance computation due to its quadratic time complexity between the millions of reads per experiment, and their corresponding design files. To solve this, we impose a $k=20$ limit constraint, where $k$ is the maximum distance between any two strands, and all possible edit values between two strands will be within the range of $[0,k]$. This is a common approach used in extensive edit-distance computations. Table 1 shows that for each of the experiments, $k=20$ ensures each strand will be categorized to the correct cluster with high probability since the majority of the reads are closely aligned to their intended design.

**Table 1: Experiments differences**
| Experiment | No. Clusters | Strand Length | Avg Edit Dist. | Avg No. Copies | Avg Strand Length post Basecalling | Time Length Creation |
| --- | --- | --- | --- | --- | --- | --- |
| Microsoft Experiment | $10,000$ | $110$ | $52.6376$ | $300$ | $211$ | $18$ days |
| Deep DNA Pilot | $1,000$ | $140$ | $64.4932$ | $1,300$ | $240$ | $6$ days |
| Deep DNA Test | $110,000$ | $140$ | $4.0328$ | 10 | $243$ | $30$ days |

We propose a deep-learning-based algorithm for computing similarity measures between the raw signals of DNA reads. This is done by leveraging the Dorado basecaller  [dorado2024] pre-trained weights. The architecture extends the Dorado model’s feature extractor, coupled with the ArcFace loss function. In the following section, we explain our solution in detail.

The neural network (NN) architecture begins with the Dorado basecaller  [dorado2024], a key component from the Oxford Nanopore Technologies (ONT) product suite, designed for DNA sequencing via nanopore technology. ONT has developed several basecalling models that leverage deep learning (DL) to translate the raw electrical signals from nanopore sequencing into nucleotide sequences. These DL-based basecallers have significantly improved the accuracy and speed of nanopore sequencing data analysis.

The Conditional Random Field (CRF) layer in the Dorado model is employed as a feature extractor for encoding raw signals within the NN architecture. The CRF architecture includes several components, each with its specific role in processing the sequencing data. Initially, convolutional layers are deployed to extract salient features from the raw input signals through a sequence of convolutional operations, capturing critical patterns essential for the basecalling task. Following are recurrent layers (LSTM) that capture the sequential nature of the DNA sequence. They process the outputs from the convolutional layers, taking into account the order and dependencies between the nucleotides. The data is then passed through linear layers, which perform a linear transformation to align the processed features with the final output space. This is followed by a Clmap layer that ensures that the scores used for predicting the DNA sequence are within a specified range. This entire sequence of operation is called the CRFModel.

Dorado is an open-source basecaller for Oxford Nanopore reads  [dorado2024]. We utilize the initial pre-trained layers of this model, truncating its architecture after the CRF layer, which outputs a size of $1024$. Subsequently, we append a linear layer with dimensions $1024\times 500$, where $500$ represents the number of classes. The training process is coupled with the ArcFace loss function  [deng2019arcface]. This function is used in facial recognition and other DL tasks that involve distinguishing between different classes. It is designed to enhance the discriminative power of the feature embeddings produced by a NN. ArcFace loss achieves this by enforcing a margin between the features of different classes in the angular space, making the decision boundary more distinct. Specifically, it adds an angular margin penalty to the target angles in the softmax function, which forces the NN to learn more separable features  [khan2024improvised]. The loss uses a cosine similarity in its implementation. This results in a more compact clustering of features from the same class and a wider separation between features of different classes. This loss function assists deeply in quantifying the similarity between the different signals.

Figure: Figure 2: Deep signal embedding scheme, using a trimmed version of the Dorado model with an appended linear layer, using the ArcFace Loss function
Refer to caption: x1.png

During training, we employed a fine-tuning approach by initializing our model weights with pre-trained weights of the Dorado model and then iteratively updating them during training. For training purpose, $500$ clusters consisting of $205,690$ samples were selected arbitrarily from the \sayMicrosoft Experiment dataset whose size is in the magnitude of millions of samples. The samples are divided into three sets. The first contains $70\%$ of the samples, and it is used for training. The second contains $15\%$ of the samples, and it is used for validation, and the remaining $15\%$ are used for testing purposes, and to visualize the test set embedding. The ML models are trained with a batch size of 256 samples using the Adam optimizer [kingma2014adam]. During the first $50$ epochs of training, the training was conducted on a subset of 100 clusters, from the initial selected $500$. For the remaining $50$ epochs, training was conducted on all the clusters. When training on a large amount of data, using the ArcFace Loss, a common approach is to divide the training epochs, by gradually increasing the size of the training set  [khan2024improvised]. The performance is measured on a 2x AMD EPYC 7513 (32 core) server with 1TB RAM and four NVIDIA A6000 GPUs.

To evaluate the model, t-Distributed Stochastic Neighbor Embedding (tSNE) is used on the output’s embeddings of the model, on the test set. tSNE is a dimensionality reduction technique utilized for visualizing high-dimensional data in lower-dimensional spaces, with an emphasis on preserving the local and global structure of the data points. Figure 3 illustrates the visualization of tSNE on the test set, demonstrating a distinct separation between the different clusters.

Figure: Figure 3: The test set is visualized using tSNE, with each color representing a distinct cluster. The presence of 500 clusters makes it difficult to distinguish similar colors visually. Each dot-like shape in the Figure represents a single cluster that is primarily well-separated from the other clusters.
Refer to caption: x2.png

## 5 Results

For each of the three datasets created in this work, we have the raw signals and the sequenced bases for each DNA read. Each of these datasets contains $50$ clusters. The \sayDeep DNA Pilot dataset’s size is $63,849$ samples, the \sayDeep DNA Test dataset’s size is $739$ samples, and the \sayMicrosoft Experiment dataset’s size is $16,109$ samples that were not used during the training phase. These datasets ensure a diverse evaluation. For each of the three experiments, a binary ground truth square matrix was constructed with a set of $50$ clusters. In this matrix, the $[i,j]$ entry equals to $1$ only if the $i$ and $j$ signals belong to the same cluster.

The results present a comparison between raw signals’ embeddings using a cosine similarity, and DNA strands using the edit-distance similarity. Recall that when using edit-distance, the possible values are amongst $\{0,1,\ldots,k\}$, where $k$ is the upper limit of insertion, deletion or substitution. For each different $k$ value, an edit-distance similarity matrix is calculated, where a higher score entails a higher dissimilarity. The $k$ values chosen are $10$, $20$, $50$, and $200$; For example, for $k=10$, the tested thresholds set is $\{0,1,\ldots,10\}$, and for each threshold a prediction matrix is calculated. These values exemplify the typical approach taken by approximation-based DNA clustering algorithms to achieve faster computation times with minimal loss of accuracy. Such algorithms often use constraints or reduction techniques in edit-distance calculations (Starcode [zorita2015starcode], Meshclust [james2018meshclust], SEED [bao2011seed] and Microsoft algorithm [rashtchian2017clustering]).

Every non-heuristic-based DNA clustering algorithm that measures similarity without direct computation relies on an edit-distance constraint. The ArcFace Loss  [deng2019arcface] uses a cosine similarity for separating between inputs, therefore, a cosine similarity matrix is created to compare every pair of signals. For cosine similarity, the thresholds are continuous and not discrete as when using edit-distance, thus, containing much more data points, ranging between $-1$ and $1$, where $1$ is identical. Using these matrices, receiver operating characteristic (ROC) are generated. The ROC curve illustrates the trade-off between the true positive (TP) and false positive (FP) values of an machine learning model.

Figure: (a) \sayMicrosoft Experiment
Refer to caption: x3.png

Figure 4 shows the ROC curves for the proposed signal-model and the three different baselines that use edit-distance similarity of the DNA strands with varying values of $k$. We expect that models with better classification performance will have ROC curves closer to the top-left corner since this indicates high true positive rate with a small false positive. The curve of a random ML model is expected to be close to the diagonal identity line. We quantitatively measure this performance according to the area under the ROC curve (AUC) metric, normalized to $[0,1]$, where 1 indicates perfect classification.

Figure 4 presents the ROC curve results for the three different experiments. The figure shows that the model that uses raw signals as inputs (signal-model) outperforms the models that use DNA strands as inputs (strand-$k=xx$) as higher TP rates are achieved for the same FP rates. Quantitatively, for the \sayMicrosoft Experiment, we find that the signal-model achieves the best trade-off between TP and FP with an AUC of $0.99$ as compared to $0.81$, $0.80$, $0.57$, and $0.51$ for the strand-$k=200$, the strand-$k=50$, the strand-$k=20$, and the strand-$k=10$ models, respectively. The sharp turn and linear increase across all the models, after the rapid rise, is attributed to read pairs with distance above $k$, thus, they are all treated as equally distant. The strand-$k=200$ is the only model that is comparable performance wise to the signal-model, though its complexity is much higher. Recall from Table 1, the average strand’s length is approximately $200$, therefore using an edit-distance with a $k=200$ value, can be treated as an unbounded edit-distance calculation. Similar trends can be seen for \sayDeep DNA Pilot and \sayDeep DNA Test (refer to Figures  4(4(b)) and  4(4(c)), respectively). Notice that the superior performance in the \sayMicrosoft Experiment compared to the other experiments is due to the training data originating from the same dataset. Even though the clusters that are used for evaluation were not present during the training process, the training process was conducted on the same dataset and thus the signals are taken from the same data distribution.

Figure: Figure 5: Compute times for three different experiments. For the signal-model it includes both the embeddings and the cosine similarity computation times; for the different strand-$k=xx$-models it includes edit-distance similarity matrix computation time. The red cross indicates the omitted entries in which the running time is larger than 30 hours
Refer to caption: x6.png

An important property of our similarity scheme is its execution time, as shown in Figure 5 on a logarithmic scale. It shows the computation time for the similarity matrices for each model, across all experiments. For the strand-$k=xx$ models, this entails the edit-distance similarity matrix computation time, and for the signal-model it entails the feed-forward that generates the embedding, and the cosine similarity calculation times. For the \sayDeep DNA Pilot dataset, the computation time of the strand-$k=200$-model is marked using a red cross, because its computation time is larger than 30 hours.

It is evident that the running time of strand-$k=xx$ models is between $1$ and $3$ orders of magnitude larger than that of signal-model. It is also evident that strand-$k=xx$ models have a linear increase between the different $k$ values across the different experiments. The variation between the datasets is due to is due to the varying dataset sizes applied to the same model.

## 6 Applications

Dealing directly with the raw signals has the potential to revolutionize the pipeline for analyzing Nanopore sequencing, especially in the context of DNA storage. Two critical aspects of the DNA storage pipeline are the clustering and reconstruction phases, which could benefit significantly from this approach.

### 6.1 Clustering Phase

To show the potential of the signal-model in the clustering phase, we used a hierarchical clustering algorithm [murtagh2012algorithms] that is compatible with cosine similarity. We conduct the evaluation on the three datasets introduced earlier. Unlike modern DNA clustering algorithms which are commonly used, such as Clover [qu2022clover] and Microsoft’s [rashtchian2017clustering], the signal-model computation time is considerably faster. Additionally, we include all the raw signals samples, unlike for example Clover, which excludes $3\%$ of the strands samples. We compare the proposed signal-model to Clover, since Clover’s execution time is the only one that is comparable in its execution time, however Clover’s accuracy is significantly lower than the signal model’s. The number of clusters that Clover returns is different from the actual number, $50$, and the number of samples in each cluster greatly differs from the ground truth. Microsoft’s algorithm [rashtchian2017clustering] is orders of magnitude slower than Clover; see e.g. [ben2023gradhc, qu2022clover], it also discards a notable number of reads, and similar to Clover, the number of clusters it produces does not align with the original cluster count, though its accuracy is higher. Both, Clover and Microsoft’s algorithm, output a subset of highly distinguished clusters, while excluding many samples. Unlike them, Signal-model outputs clusters which are distinguishable enough so reconstruction can be done, across all clusters.

**Table 2: Clustering scores for the three experiments using raw DNA signals**
| Dataset | Time [s] | Rand | Homogeneity | Completeness | V measure |
| --- | --- | --- | --- | --- | --- |
| Microsoft Experiment | $58$ | $0.971$ | $0.790$ | $0.959$ | $0.866$ |
| Deep DNA Pilot | $970.82$ | $0.887$ | $0.528$ | $0.859$ | $0.654$ |
| Deep DNA Test | $0.12$ | $0.978$ | $0.821$ | $0.857$ | $0.839$ |

Table 2 presents time and clustering scores for the three out-of-sample datasets introduced earlier, for the signal-model. It should be noted that unlike the traditional DNA strands clustering, the signal-model achieves its results on the complete set of original clusters and not a partial set. Recall that the datasets differ in size in orders of magnitude, where the \sayDeep DNA pilot is the largest dataset, followed by the \sayMicrosoft dataset, and lastly, the \sayDeep DNA Test.

The Table lists four different scoring measures: (1) The Rand score measures the similarity between two clusterings, which are different ways of grouping a set of data points into clusters, by evaluating the proportion of samples pairs that are either placed in the same cluster or placed in different clusters. It ranges between 0 0 and $1$, where a score of $1$ indicates that the clusters are exactly identical, and a score close to 0 0 suggests almost no agreement between the clusters. As the table reports, across all three datasets signal-model achieves a very high accuracy above 88%; (2) The Homogeneity score assesses clustering quality by determining if each cluster exclusively contains members of a single class. It ranges between 0 0 and $1$, where a score of $1$ indicates perfect homogeneity with each cluster composed entirely of samples from a single class, whereas a score closer to 0 0 indicates that clusters contain a mix of different classes. It is evident from the table that using a larger dataset, it hinders the performance; (3) The Completeness score evaluates clustering quality by checking if all samples of a given class are assigned to the same cluster. It ranges between 0 0 and $1$, with a score of $1$ indicating perfect completeness, meaning that all data points belonging to a given class are entirely within a single cluster. Across the three datasets, the performance of the signal-model is consistently high and comparable; and (4) the V measure score, which is a harmonic mean of homogeneity and completeness scores. It provides a single measure to assess the quality of a clustering output. It ranges between 0 0 and $1$, where $1$ indicates perfect clustering with maximum homogeneity and completeness. Signal’s model performance is very high for \sayDeep DNA Test and Microsoft datasets, while for \sayDeep DNA pilot the performance is worse.

Figure 6 shows the tSNE visualizations of the signal-model across the three datasets. Figure 6(6(a)) shows that the \sayMicrosoft Experiment has the best separation among the clusters, as explained earlier. Figure 6(6(b)) exhibits higher density with larger clusters, while Figure 6(6(c)) appears sparse with smaller clusters, as outlined in Table 1.

Figure: (a) \sayMicrosoft Experiment
Refer to caption: x7.png

### 6.2 Reconstruction Phase

The reconstruction phase is done by taking the output clusters produced by the models or algorithms  [pan2022rewritable, sabary2024reconstruction] and it computes the error rate within each cluster. Then, the algorithms selectively remove reads to achieve a specific error rate conducive to successful reconstruction of the design file. In contrast, since signal-model directly uses the raw signals, the clustering algorithm will be given more data for the reconstruction algorithms. This is because signal-model’s approach significantly reduces the sample exclusions, but still outputs well enough distinguishable clusters. This enables the removal of reads as necessary, potentially yielding equivalent or improved results. When certain clustering algorithms fail to produce all clusters, signal-model ensures that all clusters are available for reconstruction.

## 7 Limitations

## 8 Conclusion

Directly utilizing raw DNA signals for analyzing Nanopore sequencing data offers a transformative approach to the DNA storage pipeline, particularly in the clustering and reconstruction phases. The signal-based approach enhances both clustering and reconstruction, showing superior computational efficiency and accuracy over traditional DNA strand-based clusterings. Experiments reveal that the signal-model outperforms conventional clustering algorithms like Clover and Microsoft’s, providing faster and more accurate results. Additionally, the signal-model demonstrates significantly lower computation times, by orders of magnitude, highlighting its efficiency and scalability for large-scale DNA storage tasks. This paper advocates for a shift towards using raw DNA signal data, improving current methodologies, and paving the way for future advancements in DNA storage and analysis technologies.

This paper ignites research on using raw DNA signals directly, prompting the development of a clustering algorithm suitable for raw DNA signals. Future work can extend the approach of using the raw signals to enhance the reconstruction algorithms as well as the decoding procedure of error-correcting codes in the DNA storage pipeline. Additionally, further extensions of this methodology could apply to DNA and RNA sequencing across various fields, including bioinformatics, chemistry, and biology.

Limitations: The approach is tailored for Nanopore sequencing, but any modifications to the Nanopore machine may necessitate retraining the ML model, increasing the complexity of its practical application. While effective with up to $500$ clusters, the algorithm has not yet been scaled for thousands of clusters, which could pose additional challenges. The implementation relies on the open-source Dorado model  [dorado2024] weights, introducing dependencies on external updates. Additionally, further adjustments may be needed to adapt the algorithm for biological tasks, particularly signal comparison, due to challenges like varied-length reads, which differ from the DNA storage context.

## NeurIPS Paper Checklist

- 1.
Claims
- Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
- Answer: [Yes]
- Justification: The claims made match experimental results, and reflect how much the results can be expected to generalize to other settings.
- 2.
Limitations
- Question: Does the paper discuss the limitations of the work performed by the authors?
- Answer: [Yes]
- Justification:
•
Section 5 explains about the limitations of our approach to diverse performance across different datasets.
•
Section 5 discusses the computational efficiency of the proposed algorithms, specifically Figure 5.
- 3.
Theory Assumptions and Proofs
- Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
- Answer: [N/A]
- Justification: the paper does not include theoretical results, nor does it include any proofs.
- 4.
Experimental Result Reproducibility
- Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
- Answer: [Yes]
- Justification: Sections 4 and Section 5 describes all the procedures needed to reproduce the experimental results.
- 5.
Open access to data and code
- Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
- Answer: [No]
- Justification: We reserve the rights to keep our new algorithm for future work and in addition to enlisting a patent about it. If requested, supplementary material and code, can be available upon demand in private.
- 6.
Experimental Setting/Details
- Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
- Answer: [Yes]
- Justification: Section 4 provides an explanation of the trained datasets, and how the datasets were created. Section 5 provides an explanation of the out-of-sample datasets that the model is being tested on.
- 7.
Experiment Statistical Significance
- Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
- Answer: [Yes]
- Justification: In Section 5 Figure 4 discusses the trade-off between the accuracy and the errors.
- 8.
Experiments Compute Resources
- Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
- Answer: [Yes]
- Justification: Section  4 provides computer resources specifications.
- 9.
Code Of Ethics
- Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics [https://neurips.cc/public/EthicsGuidelines](https://neurips.cc/public/EthicsGuidelines)?
- Answer: [Yes]
- Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
- 10.
Broader Impacts
- Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
- Answer: [Yes]
- Justification: Section 8 discusses future applications of our proposed algorithm for other social domains.
- 11.
Safeguards
- Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
- Answer: [N/A]
- Justification: the paper poses no such risks.
- 12.
Licenses for existing assets
- Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
- Answer: [Yes]
- Justification: We provide explanations about the public dataset and pre trained weights that are in use in the paper.
- 13.
New Assets
- Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
- Answer: [N/A]
- Justification: The paper does not release new assets.
- 14.
Crowdsourcing and Research with Human Subjects
- Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
- Answer: [N/A]
- Justification: the paper does not involve crowdsourcing nor research with human subjects.
- 15.
Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
- Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
- Answer: [N/A]
- Justification: the paper does not involve crowdsourcing nor research with human subjects.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: * The claims about previous clustering algorithms do not seem very accurate or thorough. For example, the authors say "Unlike modern DNA clustering algorithms which are commonly used, such as Clover (Qu et al., 2022) and Microsoft’s (Rashtchian et al., 2017), the signal-model computation time is considerably faster." I am not sure how this is verified. For example, one inaccuracy is in Section 7.1 -- the authors say "Microsoft’s algorithm (Rashtchian et al., 2017) is orders of magnitude slower than Clover" but the cited paper GradHC (Ben Shabat, 2023) points out that the Microsoft algorithm is very fast, except when the clusters are very small and it does not converge (which may not be the case for the datasets in this paper, since the number of clusters is 50 in Section 6, and the number of reads is much larger). Also why not compare to the GradHC algorithm, which is shown to be quite fast?

* I am confused about "ground truth for clustering" and in particular the statement "Table 1 shows that for each of the experiments, k = 20 ensures each strand will be categorized to the correct cluster with high probability since the majority of the reads are closely aligned to their intended design" In Table 1, it seems that for the first two datasets, the average edit distance is between 52 and 63. So what happens for the strands with edit distance more than k = 20? Will they be assigned to a different cluster? 

* Section 7.2 seems to be unfinished. There are no experiments to back up the claims. Also this sentence in 7.2 does not make sense as written: "In contrast, since signal-model directly uses the raw signals, the clustering algorithm will be given more data for the reconstruction algorithms." The clustering algorithm will operate on the reads, while the reconstruction algorithm will operate on the clusters. I don't know what it means for one algorithm to be given more data for another algorithm.

* One of the more interesting aspects to explore is whether other parts of the DNA storage system can be replaced with deep learning. For example, is it possible to train a model to go directly from raw signals to the final prediction for the stored data? Perhaps the network can learn to implicitly cluster then reconstruct then error correct, without needing separate algorithms for each of these steps.

* See also the questions at the end of each of the weaknesses above.
soundness: 1
strengths: This paper studies an important problem in DNA storage. It has long been an open question whether systems can utilize information in the raw signals to better improve the analysis and downstream use of Nanopore data.

The authors show the effectiveness of taking a pre-trained model (Dorado) for Nanopore basecalling and repurposing it for a downstream application of the raw signals. This is pretty interesting because it is good to know the pre-trained model can be useful, and researchers do not have to start from scratch and train a model on Nanopore data themselves. 

The authors also motivate their work by saying that the computation time can be much faster than basecalling with an existing algorithm. This is indeed an important aspect to improve upon. The methods in this paper could take the signal processing time from days/weeks down to under a single day, which would make DNA data storage more feasible for recovering stored data.
summary: This paper tackles the clustering problem of DNA storage by using raw Nanopore signals as opposed to first translating the signals to strings (basecalling) and then clustering on the text strings. The motivation here is clear because the process to go from raw signals to strings can be very slow (weeks to months) and hence operating directly on the signals can be more efficient. Therefore, the authors design a deep learning method that operates on the data before it is discretized into bases. This paper presents a deep neural network that clusters these signals directly with reduced computation times compared to current approaches that cluster after basecalling.

From a technical point of view, the authors start with a pre-trained model and then do fine-tuning on additional data. The base model is Dorado, which is the standard basecaller for Nanopore reads. After truncating to get an embedding, they append a linear layer with output dimension equal to the number of classes (number of clusters). Then the training also uses the ArcFace loss function to increase separation between the clusters based on cosine similarity.
weaknesses: * A major weakness of the approach seems to be that the model needs to be trained with an output dimension equal to the number of clusters. However, the number of clusters in real settings can be very, very large. E.g., for a file that is stored that is 100MB, there could easily be over 1M clusters. I don't think network training will scale well in this case. So the experiments on 500 clusters are not representative of a real setting. Concretely, can the authors address how their approach might scale to much larger numbers of clusters (e.g. 1M+) that would be encountered in real-world applications. Please discuss potential solutions or modifications to their method to handle this challenge.

* In general, the experiments in Section 6 seem quite ad-hoc. The number of clusters is restricted without much discussion. One interesting outcome of this research could be how to train/design a good Nanopore signal clustering algorithm. But there are very few ablations, and not much details about what the authors learned or why their method works/fails. It feels as if the authors did one set of experiments and quickly wrote them up. This is below the bar for a ML paper at a top conference. There should be more tangible findings, and more justification for the design decisions. Specifically, can the authors investigate the impact of varying the number of clusters, explore different network architectures, and analyze how different components of their method contribute to its performance.

* The experimental results in Section 7 are missing comparisons to baselines. For example, the conclusion states that " Experiments reveal that the signal-model outperforms conventional clustering algorithms like Clover and Microsoft’s, providing faster and more accurate results." However, I don't see these experiments in the paper. For example, Table 2 and Figure 6 only seem to present results for the authors' algorithm, not the baselines (e.g., Clover or Microsoft's algorithms for clustering). It seems necessary to include direct comparisons with Clover, Microsoft's algorithm, and other relevant baselines in Table 2 and Figure 6, using the same datasets and evaluation metrics. This would provide a clearer picture of the proposed method's performance relative to existing approaches.

* At a high level, the related work is oddly about clustering, but the paper is about more than clustering. I am not familiar with things like how the Dorado method was trained, what competing methods there are, etc. Also have people really not tried to train deep learning algorithms on Nanopore data for other applications? This seems very surprising to me. For example, a quick search shows some other papers that use nanopore signals for applications:

  * HycDemux: a hybrid unsupervised approach for accurate barcoded sample demultiplexing in nanopore sequencing
Renmin Han, Junhai Qi, Yang Xue, Xiujuan Sun, Fa Zhang, Xin Gao & Guojun Li 
Genome Biology volume 24, Article number: 222 (2023) 

  * Kovaka S, Fan Y, Ni B, Timp W, Schatz MC. Targeted nanopore sequencing by real-time mapping of raw electrical signal with UNCALLED. Nat Biotechnol. 2021;39(4):431–41.

  * Overall, can the authors expand their discussion of related work to include: A more comprehensive overview of Nanopore signal processing methods, including Dorado and its competitors; A broader discussion of deep learning applications on Nanopore data beyond clustering; An analysis of how their approach compares to or builds upon these, as well as other relevant work in the field.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: Generalizability of the Dataset: 
How do you plan to address the potential lack of generalizability due to the customized dataset? Have you considered testing the method on natural genomic data to assess its applicability in real-world scenarios?

Edit Distance Threshold: 
Could you elaborate on the decision to set the edit distance threshold at 20? How do you plan to ensure accuracy in clustering when dealing with reads that might have more variations? Can you provide an ablation study on the impact of choosing different edit distance thresholds on clustering accuracy?

Similarity Comparisons: 
Can you provide a rationale for using different similarity measures for raw signals and DNA strands? How might you align these metrics to ensure consistency in comparison?

Model Architecture and Novelty: 
Are there any future plans to innovate the model architecture or training strategies beyond using raw signals? What potential modifications could enhance the novelty of your approach?
soundness: 3
strengths: Originality: 
The paper addresses a novel problem by introducing raw signal clustering in DNA storage, which could set a new direction for handling sequencing data.

Relevance: 
With data storage demands growing exponentially, improving DNA storage efficiency and accuracy is highly relevant.

Computational Efficiency: 
The use of deep embeddings and cosine similarity yields significant computational improvements, making the method scalable and suitable for high-throughput sequencing.

Clarity in Experimental Validation: 
Experimental results across multiple synthetic datasets support the authors’ claims, with performance metrics like AUC, ROC curves, and computation times well-documented.
summary: This paper proposes a novel approach to clustering in DNA data storage by focusing on clustering raw signals from Nanopore sequencing, rather than traditional basecalled DNA sequences (A/T/C/G). Traditional DNA storage pipelines typically perform clustering after basecalling, which introduces errors and computational inefficiencies. The authors aim to improve clustering accuracy and efficiency by directly embedding the raw signals produced during sequencing. The proposed deep learning model integrates the Dorado basecaller architecture and an ArcFace loss function, allowing for similarity-based grouping of raw signals. This approach is demonstrated to enhance clustering accuracy while reducing computational time by one to three orders of magnitude compared to traditional sequence-based methods. Results are validated on multiple datasets, and the model shows potential for enhancing both the clustering and reconstruction phases of DNA data storage systems.
weaknesses: Limited Generalizability Due to Custom Dataset: 
The dataset is highly customized, relying on specific design files and synthetic DNA sequences generated by a particular synthesis provider (Twist Bioscience). Since real-world DNA samples often include much higher variability, especially in natural genomic data, the results from this dataset may not generalize well to other applications or to DNA data with biological origins rather than synthetic sources.

Fixed Threshold for Edit Distance (k=20):
The authors limit their edit distance calculations to a maximum of 20 differences, which helps with computational efficiency but may introduce inaccuracies. For reads that are more distant from their original design strands, this constraint could lead to incorrect cluster assignments, particularly if the dataset scales up to include a broader diversity of sequences with higher variances.

Different similarity comparisons:
The comparison between raw signal embedding using cosine similarity and DNA strands using edit distance may not be entirely fair due to their application to different data types—continuous versus discrete. Cosine similarity captures vector orientation in high-dimensional spaces, while edit distance measures specific sequence transformations. This discrepancy can lead to inconsistencies in capturing variations and information. Ensuring a fair comparison might require using a consistent metric or providing a rationale for the chosen methods.

May lack of novelty:
Aside from making and utilizing raw signals, the paper lacks additional innovation in either its model architecture, training strategies, or data processing. The use of dorado CRF layers and standard methods doesn't introduce novel techniques. To enhance innovation, further exploration in these areas would be beneficial.

---

confidence: 4
contribution: 4
initial_rating: 8
presentation: 3
questions: - Raw sequencing signals can vary considerably due to noise and machine-specific artifacts. How does the model handle this variability, and what measures are taken to ensure robust clustering across varying signal qualities?
soundness: 4
strengths: - The core contribution is a model, based on the model in the Dorado basecaller, that performs clustering on these raw signals before basecalling. To my knowledge, this signal-based approach is novel and promising in the field of DNA storage.
- The authors evaluate their method on several DNA datasets. Their results show that signal-based clustering outperforms existing methods in terms of both time and accuracy.
summary: The paper presents a deep learning-based approach for clustering DNA reads using raw electrical signals generated during nanopore sequencing. Traditionally, DNA clustering occurs after basecalling, where raw signals are converted into nucleotide sequences. The authors propose to bypass this step by clustering directly on the raw signals, which retain more information than the processed sequences.

The paper proposes the use of deep signal embedding to improve the clustering stage of DNA storage pipelines, bypassing traditional methods that rely on basecalled DNA reads. Experiments indicate that the proposed method outperforms state-of-the-art approaches.
weaknesses: Major:
- An important point that the authors should address is that it is uncertain whether the clustering gains translate into significant improvements in the final data retrieval phase.

Minor:
- Line 40: The description 'a "retrieval" stage where reads are decoded back to binary data files while correcting any errors using the chosen coding methods' is inconsistent with Figure 1, where the decoding is shown to occur after the retrieval stage.
- Line 80: "edits" should be replaced by "substitutions"
- Line 104: "contributions to" -> "contributions of"
- Line 156: How can UCLUST (published 2020) be based on USEARCH (published 2021)?
- Line 173: There are also algorithms such as GCTW (Zhou and De la Torre, 2016) for sequence alignment with sub-quadratic complexity. This should be acknowledged.
- Line 183: "during during" -> "during"
- Line 186: "maps" -> "map"
- Lines 189-193: This paragraph seems rather disorganized.
- Line 214 (footnote): "Deep DNA test" -> "Deep DNA Test", "Deep DNA pilot" -> "Deep DNA Pilot"
- Lines 199-206: This paragraph needs improvement. The text says that the synthesis will produce '"fast5" file formats'. Firstly, the synthesis yields oligonucleotides. Secondly, nanopore sequencing would yield raw signal data in the form of files in the FAST5 format.
- Line 223: I do not understand how to deduce from Table 1 that the strands are correctly clustered with the k=20 maximum distance constraint.
- Line 247: "Clmap" -> "Clamp"
- Line 250: The authors should be clearer about what is meant by the number of classes. It only becomes clear later that this refers to the number of (expected) clusters.
- Figure 2: The figure could be improved. On the left, it is not clear that the input files are used to synthesize the DNA. Also, according to the figure, it seems that the ArcFace loss is an input to the CNN.
- Line 409: "due to is due to" -> "due to"

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: N/A
soundness: 2
strengths: The approach of commencing the analysis from raw Nanopore signals, rather than relying on pre-processed discrete DNA sequences, represents a novel direction in the field. 

Sequence clustering is a challenging problem, especially when dealing with a large number of sequences. The proposed method may have significant impact on the DNA storage community.
summary: The manuscript introduced a DL-based approach for clustering sequences retrieved from the DNA sequencing pipeline, utilizing raw signals from Nanopore sequencing method.  The proposed model is derived through fine-tuning an open-source signal-to-base model known as Dorado, under a classification task.
weaknesses: There are several weaknesses that the authors may need to address. 

0. The authors should pay more attention to the typos, grammars, etc. e.g line125 ", while those that are not are for apart. "

1. There are existing works that employs deep learning techniques to assess the similarity between sequences (DNA sequences), which are very closely related to this manuscript but omitted in this work. e.g. 
```
1. "Convolutional embedding for edit distance."  SIGIR 2020
2. "Neural distance embeddings for biological sequences."  NeurIPS 2021
3. "Deep Squared Euclidean Approximation to the Levenshtein Distance for DNA Storage." ICML 2022
```

2. To the best of the reviewer’s knowledge, none of the three datasets utilized in this study are available in an open-access format as raw signals. This limitation may impede the reproducibility and validation of the proposed method. 

3. The author(s) have chosen to employ the ArcFace loss function as a substitute for the cross-entropy loss in the classification task. However, the manuscript lacks an analysis or ablation study that would substantiate the necessity of introducing this more complex loss function. 

4. The contribution of the proposed method to the machine learning and representation learning communities, in terms of novelty, appears to be limited. Most of the involved methods in this work are mature DL techs. The primary novelty of this work lies in utilizing the untapped potential information from the raw signals. Given this focus, the manuscript would be more suited for  a bioinformatics-specific paper. 

5. It is not demonstrated that whether the untapped potential information from the raw signals helps in clustering the sequences, which is claimed as the main contribution as in the Abstract. There should be a comparision between the raw-signal/DNA-sequence as inputs to support this claim. 

6. The reviewer also concerns about the effectiveness of employing supervised classification tasks to derive embeddings, 
particularly since approximating distance metrics is a relatively straightforward task with the datasets used in this study, which is also used in the existing works presented in item 1. 

7. The author(s) may want to improve the writting. e.g.  the reviewer did not find any table or figure which is refered by "The results" in line 313.