Blind Biological Sequence Denoising with Self-Supervised Set Learning
Authors: Nathan Hoyen Ng
Ji Won Park
Jae Hyeon Lee
Ryan Lewis Kelly
Stephen Ra
Kyunghyun Cho
Venue: Accepted by TMLR ()

Abstract: Biological sequence analysis relies on the ability to denoise the imprecise output of sequencing platforms. We consider a common setting where a short sequence is read out repeatedly using a high-throughput long-read platform to generate multiple subreads, or noisy obser- vations of the same sequence. Denoising these subreads with alignment-based approaches often fails when too few subreads are available or error rates are too high. In this paper, we propose a novel method for blindly denoising sets of sequences without directly observing clean source sequence labels. Our method, Self-Supervised Set Learning (SSSL), gathers subreads together in an embedding space and estimates a single set embedding as the mid- point of the subreads in both the latent and sequence spaces. This set embedding represents the “average” of the subreads and can be decoded into a prediction of the clean sequence. In experiments on simulated long-read DNA data, SSSL methods denoise small reads of ≤ 6 subreads with 17% fewer errors and large reads of > 6 subreads with 8% fewer errors compared to the best baseline. On a real dataset of antibody sequences, SSSL improves over baselines on two self-supervised metrics, with a significant improvement on difficult small reads that comprise over 60% of the test set. By accurately denoising these reads, SSSL promises to better realize the potential of high-throughput DNA sequencing data for downstream scientific applications.

Blind Biological Sequence Denoising with Self-Supervised Set Learning
Nathan Ng Ng..23, 1223.
Won Park"
Jae Hyeon Lee*
Ryan Lewis Kelly*
Stephen Ra
Kyunghyun Cho 4,5,6
nathanngomit. edu,
(park.ji_war leejj22,, (aleej222,kkkllrr ra. steppenn)ggnn..com kyunghyun. cho@nyu. edu
Universily of Toronto
Veclor Institule
MIT Prrescient Design, Genentech
NNew York University
6 CIFAR Fellow
Abstract
Biological sequence analysis relies on the ability denoise the imprecise output of sequenc- ing platforms. We consider common setting where short sequence is read out repeatedly using high-throughput long-read platform to generate multiple subreads, or noisy obser- vations of the same sequence. Denoising these subreads with alignment-based approaches often fails when too few subreads are available or error rates are too high. In this paper, we propose a novel method for blindly denoising sets of sequences without directly observing clean source sequence labels. Our method, Self-Supervised Set Learning (SSSL), gathers subreads logether in an embedding space and estimates single set embedding as the mid- point of the subreads in both the latent and sequence spaces. This set embedding represents the "average" of the subreads and can be decoded into prediction of the clean sequence. In experiments on simulated long-read DNA data, SSSL methods denoise small reads of S 6 subreads with 17% fewer errors and large reads of > subreads with 8% fewer errors compared to the best baseline. On a real dataset of antibody sequences, SSSL improves over basclines on two supervuud metrics, with significant improvement on difficult small reads that comprise over 60% of the test set. By accurately denoising these reads, SSSL promises to better realize the potential of high-throughput DNA sequencing data for downstream scientific applications.
1 Introduction
Denoising discrete-valued sequences is a task shared by a variety of applications such as spelling correction (Angluin & Csürös 1997 Damerau & Mays] 1989: Mays et al.] 1991) and hidden Markov Model state estimation (Ephraim & Merhay 2002). Recently, this task has become central part of biological sequence
analysis (Tabus et al. 2002
2003 Lee al.) (017) as sequencing hardware becomes faster and more cost- efficient. Short-read DNA/RNA sequencing platforms which can process sequences of up 200 base pairs (bps) with low noise levels (0.1-0.5%) utilize traditional algorithms involving k-mers Mannkar & Sathe
2018 Yang et al.) [2010] Greenfield et al.) 2014 Nikolenko et al.
2013
Medvedev et al.
2011 Lim et al,
2014 statistical error models (Schulz et al.) 2014 Mcacham ot al.
2011
Yin ct al. 2013). or multi-sqquence
aligument (MSA) (Kao et al. 2011 Salmela & Schröder] [201 Bragg el. al. 2012: Gao el al.
2020 to achieve high quality results. However, long-read sequencing platforms such as the Oxford Nanopore Technology (ONT), which can quickly process sequences of thousands to millions of base pairs at the cost of much higher error rates (5-20%), require more efficient and accurate denoising methods.
When these platforms are used read genome- or transcriptome-level sequences, the high coverage and overlap allow methods such as IsONCorrect (Sahlin & Medvedey 2021 and Canu (Koren et al. 2017
Work done during internship with Prescient Design.

---

reduce errors to manageable rates. Increasingly, however, these platforms are instead being applied to much shorter (1-5kbps) sequences, such as full single-chain variable fragments (scFv) of antibodies (Goodwin et al. 2016). In this sequencing paradigm, practitioners generate a long read consisting of multiple noisy repeated observations, or subreads, of the same source sequence which are then denoised together. Denoising in this setting presents unique challenges for existing methods based on fitting statistical models or training a model on a small number of ground-truth sequences.
First, ground-truth source sequences are usually resourve-intensive to obtain, although supervised models (Figure Tbp have found some success in limited settings (Baid et al.) 2022). Second, priors of sclf-similarity and smoothness that enable denoising in other domains such as images (Fan et al. 2019) often do not transfer to discrete, variable-length data. Third, assumptions on the independence of the noise process that underpin statistical methods are violated as the errors introduced during sequencing tend to be context-dependent (Abnizova et al. 2012; Ma et al., 2019). Owing to these challenges, liggmeent-based algorithms (Figure la) such as MAFFT remain the most commonly used denoising method. These algorithms align the subreads and then collectively denoise them by identifying a consensus nucleotide base for each position (e.g. Kao cot al. 2011). MSA-basod denoising tends to be unreliable when very few subreads are available or high error rates lead to poor alignment as is often the case with long-read platforms. Even when alignment does not fail, it may not always be possible to break a tie among the subreads for given position and identify an unambiguous "consensus" nucleotide.
In this paper, we propose self-supervised set learning (SSSL)) blind denoising method that is trained without ground-truth source sequences and provides strong alternative when MSA produces poor results. SSSL, illustrated in Figure Icl aims to Iearn an embedding space in which subreads cluster around their associated source sequence. Since we do not have access to the source sequence, we estimate its embedding as the midpoint between subread embeddings in both the latent space and the sequence space. This "average" set embedding can then be decoded to generate the denoised sequence. Our SSSL model consists of three components: an encoder, decoder, and set aggregator. We formulate a self-supervised objective that trains the encoder and decoder to reconstruct masked subreads, trains the set aggregator to find their midpoint, and regularizes the latent space to ensure that aggregated embeddings can be decoded properly. Because SSSL observes multiple sets of subreads that may be generated from the same source sequence, it can more accurately denoise in few-subread settings.
We evaluate SSSL on two datasets a simulated antibody sequence dataset for which we have ground-truth sequences and proprietary dataset of real light chain scFv antibody ONT reads for which we do not-and compare the performance against three different MSA algorithms and a median string algorithm. We focus on the antibody light chains in this work since their ground truth sequences exhibit fewer variations and thus more similarity between reads, making self-supervision a natural choice. In settings without source sequences, we propose two metrics to evaluate and compare the quality of SSSL denoising to other baselines. Our primary metric, leave-one-out (LO0) edit distance, provides a local similarity metric that measures an algorithm's 's ability to denoise unseen sequences. Our secondary metric, fractal entropy, measures the global complexity of a denoised sequence relative to its subreads. In experiments on simulated antibody data, SSSL methods improve source sequence edit distance on small reads with < 6 subreads by an average of 6 bps, or a 17% reduction in errors over the best baseline methods. On larger reads with > 6 subreads, SSSL methods reduce source sequence edit distance by bps, or an 8% reduction in errors over the best baseline methods. In experiments on real selFy antibody data, SSSL outperforms all baselines on both LOO edit and fractal entropy on smaller reads and matches their performance on larger reads. Importantly, SSSL is able to produce high-quality denoised sequences on the smallest subreads, even when only one or two subreads are available. Denoising these reads enables their use for downstream analysis, helping to realize the full potential of long-read DNA platforms.
2 Background and Related Work
We first establish the notation and terminology used throughout the paper. Given an alphabet A, we consider a ground-truth source sequence = (s).-.. ST.) with length T, of tokens s; E A. We define a noisy read R as a set of m subread sequences {r.,..., } with varying lengths (Tr..... .Trm). Each subread sequence r; is a noisy observation of the source sequence generated by stochastic corruption process q(r We
2

---

CTZ GEAGTCANATO
Decoising Modal
1ttt
C A
R
1 13
R
12
T G
(a) Multi-Sequence Alignment Denoising
(b) Supervised Neural Denoising
r CTTGTCRAATO
T C
C CATTCTAAATE
(c) Denoising with Self- Self-Ssue Set Learning
Figure 1: Given a set of subreads R = (ri, r2, ...... (a) commonly used denoising methods identify a per-position consensus from multi-sequence alignment of the subreads. They are prone to failing when few subreads are available or error rates are high. (b) Supervised denoising methods train a neural network directly predict the source sequence. During training, they rely on access ground-truth sequences, which can be prohibitively expensive to obtain. (c) Our proposed self-supervised set. learning framework denoises individually embed subreads,
without access source sequences by learning a latent space in which aggregate them, and then decode the aggregated embedding into a prediction of the clean sequence.
refer to the average percentage of tokens corrupted by q as the error rate. In the case of DNA sequencing, this corruption process consists of base pair insertions, deletions, and substitutions. These errors are often context-dependent and carry long-range correlations (Ma et al. 2019). The error rates and profiles can also vary widely depending on the specific sequencing platform used (Abnizova et al. 2012
2.1
DNA Sequence Denoising
The goal of DNA sequence denoising is to produce the underlying source sequence given noisy read R. Traditional denoising methods are based k-mer analysis, statistical error modeling, or MSA (Yang et al.
2012
Lachnemann et al.] 2015). k-mer based methods (Manekar & Sathe 2018 Yang et al. 2010 Greenfield
ct al. 2014 Nikolenko et al. 2013: Medvedev et al. 2011 Lim et al. 2014 Koren et al. 2017 build a library of aggregate k-mer coverage and consensus information across reads and use this to correct untrusted k-mers in sequences, but fail without large coverage of the same identifiable sections
the genome. Statistical error model-based mothods (Schulz et al.) [2144] Mcacham al.] [0011 Yin al.] (013)) build an empirical model of the error generation process during sequencing using existing dalasets, but require strict modeling assumptions and computationally expensive fitting procedures {Lee et al.
2017)- MSA-based methods (Kao
et al.] 2011 Salmela & Schröder 2011 Bragg et al.) 2012 Gao et al. 2020 align the subreads within a read and then aggregate the information by identifying a consensus nucleotide for each position, but perform poorly when few subreads are available or the subreads are noisy. Although combining these met may alleviate some of these issues, current denoising methods are reliable only in specific settings.
Recent work has leveraged large neural networks to perform denoising. A model can be trained to directly denoise reads in fully supervised manner. Specifically, given datasct D = ((R(), {())] consisting of reads (() generated from associated source sequences g(t), & neural denoising model learns to generate the source imput ((t) given subread sequences in ( and any other associated features. Models like DeepConsen- sus (Baid et al. 2022) have shown that with a sufficiently large datasct of reads and ground-truth sequences, model can be trained to outperform traditional denoising methods. However, ground-trute sequences are often prohibitively expensive to obtain, limiting the use cases for fully supervised neural methods.
3

---

2.2 Blind Denoising
In settings where we do not have access to ground-truth sequences, we need to perform blind denoising. Specifically, given a dataset that consists of only noisy reads, D = (R we train a model that can generate the associated source sequences g(t) without ever observing them during training validation. Existing work in blind denoising focuses on natural images, for which strong priors can be used to train models with self-supervised objectives. The smoothness prior (i.e., pixel intensity varies smoothly) motivates local averaging mothods using convolution operators, such as a Gaussian filter, that blur out local detail. Another common assumption is self-similarity; natural images are composed of local patches with recurring motifs. Convolutional neural networks (CNNs) are often the architectures of choice for image denoising as they encode related inductive biases, namely scale and translational invariance (LeCun et al., 2010 In the case that the noise process is independently Gaussian, a denoising neural network can be trained on Stein's 's
unbiased risk estimator (Ulyanov et. al. 2018 Zhussip et al. 2019 Metzler et al. 2018 Raphan & Simoncelli, 2011). An extensive line of work ("Noise2X")
significantly relaxes the assumption on noise so that it need only be statistically independent across pixcls, allowing for a greater variety of noise processes to be modeled
[Tehtinen et. al. 2018 Batson & Royer 2019
Laine et al. 2019 Krull el al. 2020).
Discrete-valued sequences with variable lengths do not lend themselves naturally to assumptions of smooth- ness, selffsimillrrty, Gaussianity, or even statistical independence under the Noise2X framework all assump- tions explicitly defined in the measurement space, Moreover, we must model more complex noise processes for correcting DNA sequencing errors, which tend to depend on the local context and carry long-range cor- relations (Abnizova al.) 2012 Ma al.) 2019). We thus require more flexibility in our modeling than are afforded by common statistical error models that assume independent corruption at each position (Weissman)
et al. 2005 Lee et al. 2017).
2.3 Learning Representations of Biological Sequences
A separate line of work altempts to learn high-dimensional representations of sequences that can be used to approximate useful metrics in sequence space. Typically, encoder is trained to approximate edit distances and closest string retrievals with various emebedding geometries (Jaccard, cosine, Euclidean, hyperbolic) and
architectures (CNN, CSM, GRU, transformer) (Zheng et al. 2018 Chen et al. 2020 DAI et al. 2020) Gómez lt al. 2017 Corso et al. 2021). These models require supervision from ground-truth pairwise edit distances or closest strings, which are prohibitively expensive to generate at the scale of our datasets (~800,0000 sequences and 640B pairwise distances). Most similar to our work is NeuroSEED (Corso et al. (021), which learns an approximation of edit distance by modeling embeddings in a representation space with hyperbolic geometry. NeuroSEED considers the additional task of MSA by training an autoencoder with embedding noise and decoding from median in representation space. As distinguished from NeuroSEED, we require no supervision, learn the aggregation operation directly, and operate on variable-length sequences of latent vectors which enables a richer and more complex representation space.
3
Blind Denoising with Supervuue Set Learning
3.1 Motivation
Suppose we are given a dataset D = (R)) of noisy reads, each containing m(() variable-length subread pm sequences (r,) J==1 with correspond to noisy observations of a ground-truth source sequence g( Assume that individual subreads and source sequences can be represented on a smooth, lower dimensional manifold M Chapplle et al. 2006). Because we do not have direct access to s(i), we cannot directly find its embedding in M. However, since subreads from a given read are derived from a shared source sequence (() and are more similar to s(() than to one another, we expect the representations of the subreads to cluster around the representation of their associated source sequence in M, shown in Figure Ic In such an embedding space, we can estimate the embedding of g(t) at the midpoint between subread embeddings in both the latent space and the sequence space. This embedding represents an "average" of the noisy subreads, and can then be decoded to generate 8((), our prediction of the true source sequence. We propose self-supervised set learning (SSSL.) to learn the manifold space M accommodating all of these tasks: encoding individual soquences, finding a plausible midpoint by aggregating their embeddings, and decoding the aggregated embedding into denoised prediction.
4

---

ge($)
sequence mid
ETTGTCARAZE
Dencising Generation
Aggregator
(xp())
Lattent mid
R
(-)
(-)
R
H
Mm
Transformer
Transforner Decoder
Encuder
Lautoencode
Our self-supervised set learning model architecture. All subread sequences in given read first pass through transformer encoder, which yields set of embeddings. These embeddings are decoded by transformer decoder and an autocncoding loss minimizes the reconstruction error. The set of embeddings are also combined by a set aggregator to produce single set embedding, from which we can decode the denoised sequence using the transformer decoder. This set embedding is regularized to be the midpoint of the subreads in both the latent space and the sequence space.
Figure 2:
3.2 Model Framework
Our proposed SSSL model architecture is shown in Figure It consists of three components:
An encoder network fo parameterized by é that takes as input a subread sequence of T, tokens and outpuls sequence of d-dimensional embeddings, h E RddT,
A set aggregator network as parameterized by + that takes input set of variable-length beddings H (h,,... hmm) with h, E Rd+Ir, and outputs set embedding h e Rd+T" of length T'.
An autoregressive decoder network go parameterized by that takes as input an embedding E Rd=I" and a sequence of previously predicted output tokens , and outputs a distribution over the next token conditioned on the previous tokens 11.
To denoise a given read R = {ri, ...,.... Tmm , we pass each subread through the encoder to generate associated embeddings H (h1.-.. hm). This of verighle-length embeddings passed the set aggregator yield set embedding h. The decoder then takes this set embedding h, representing the whole read, and generates denoised sequence 8. We emphasize that at no point during training or validation does model observe the true source sequence 8.
In our experiments, we parameterize our encoder and decoder as transformer encoder and decoder, re- spectively, with an additional projection head on top of the encoder network. Since the inputs to our set aggregator network are of varying lengths, we first transform the subread embeddings common length by applying a monotonic location-based attention mechanism (Kim et al., 2021 Shu et al. 2019). The single transformed length 7" for given read is calculated the average of the subread lengths. The scale parameter is learned via linear layer that takes input feature vector consisting of the d-dimensional average of the embeddings in H, along with the sequence lengths {Tr, .T.. Once trans- formed to the same length T', the subrcad embeddings are passed to a set aggregation mechanism that combines the length-transformee embeddings at each position across subreads to produce a single IRd em- bedding for each position and a sequence embedding h e IRd""", In our experiments, we consider a simple mean pooling operation as well as set transformer (Lec al. 2019), a. learned permutation-invarian atten- tion mechanism. We call models using these mechanisms SSan-Mean and SSSL-Set respectively. Specific model and training hyperparameters provided in Appendix
the read to a
B

---

3.3 Training Objective
We formulate self-supervised training objective to construct embedding space in which we embed, aggregate, and decode sequences. The encoder Sa and decoder do are trained to map sequences to the embedding space and back to the sequence space. We optimize fo and ge with a simple autoencoding objective that attempts to minimize the negative log probability of reconstructing a given subread Ty, i.c encoding it with (T;) and decoding via go:
m
Lautoencode
(R) = log go (r(()))
Σ
(1)
We regularize the latent. space learned by So by applying a small Gaussian noise to the embeddings produced by fo and randomly masking the input subreads during training. In addition, we apply L2 decay on the embeddings to force them into small ball around the origin. We call this rogularization embodding decay:
m
Rambed (R) = Σ - 1lfa(r))a122 j=1
The combination of these techniques allows the decoder to decode properly not only from the embeddings of observed subreads but also from the aggregate set embedding at their center.
The set aggregator produces single set embedding given a set of variable-length input embeddings. For convenience, we define 0g (R) ((((((())))); as the set embedding produced by the combination of the encoder and set aggregator. In our experiments we consider both learned and static set aggregation mech- anisms. In order to estimate the true source sequence embodding, to produce an embedding at the midpoint of the subreads in both the latent space and sequence space. To find a midpoint in our latent space we require a distance metric d between variable-length embeddings. We consider the sequence of emboddings as a set of samples drawn from an underlying distribution defined for each sequence and propose the use of kernelized maximum mean discrepancy (MMD) (Gretton et al. 2012
train the encoder and set aggregator
Tx
Lank. Σ ú,j=
Ly
1
(x =
k <(Wi x,))
n(x,, yı) + L
Σ 1,j=1
x(y, Us)
(3)
L
La Ly
1-j=
for a choice of kernel k. In experiments we use the Gaussian kernel k(æ;, y) = exp( 20 We considered other kernels such as cosine similarity and dot product but found the models failed to converge. Intuitively, each sequence while
this metric aims to reduce pairwise distances between individual token embeddings also preventing embedding collapse within each sequence embedding. To find midpoint in sequence space minimize the negative log likelihood of decoding each of the individual subreads given the set embedding. Combining these sequence and latent midpoint losses gives us the loss used to train our set aggregator:
m
Luidpoint RR)
log posgoor,laad (ao, + (as (R),
(4)
Cengueee
Putting all the components together, the objective used to jointly train the encoder, decoder, and set aggregator is
arg min $.0.0
(R. Latooanodaa (R()) + ( mindppone (R
+ XRanbad (R (R(),,
(5)
(
where 77 and À control the strength of the midpoint loss and regularization respectively. Importantly, rather than average losses first over the subreads in each read and then over each batch of reads, we average over the total number of subreads present in batch in order to ensure every subread in the batch is weighted equally. This weighting scheme has the effect of upweighting higher signal-to-noise reads with more subreads.
6

---

4 Evaluation Metrics
The ultimate goal for a given denoising method f()) is to produce a sequence f(R) = å from a given read R = fri..... F with the smallest edit distance deda s) to the associated source sequence s. Since we do not observe 8, we require an estimator or proxy metric of dean (8, 8) such that differences in our metric correlate strongly with differences in dedit (8, 8). In this work we propose the use of two metrics, leave-one-out edit distance and fractal entropy.
4.1 Leave-One- Out Edit Distance
Existing work typically uses the average subread edit distance, or average distance between the denoised sequence and each subread:
SE(R, f) = 1 m
Σ deaat (f(R), r:))
(6)
However, since subread edit distance measures the edit distance to a sequence that the algorithm has already seen, it biases our estimate downwards for algorithms that explicitly minimize this distance, such median string algorithms. In order to more closely mimic the actual test environment where we do not see the sequence we compare to, as well as provide metric that is similarly biased for wide range of algorithms, we propose leave-one-oul (LO0) edil. dislance. For each subread in the read, we denoise the read with the subread removed, R_;; then measure the distance to the removed sequence:
LOO(R, 1
deai (f(R.))
(7)
This metric allows us to capture an algorithm's ability to generate consistent quality denoised sequences.
4.2 Fractal Entropy
Since edit distance-based metrics measure similarity only on individual positions, we propose an additional metric that complements them by measuring motif similarity globally across all subreads. Specifically, we propose a method of measuring the entropy of a denoised sequence relative to its subreads. Intuitively, a denoised sequence should have lower relative entropy when more of its k-mers are consistent with those present in the subreads, and higher relative entropy when the k-mers in the denoised sequence are rare or not present in the subreads. Since entropy is calculated the k-mer rather than sequence level, applicable for much longer sequences for which calculating pairwise edit distances is computationally expensive. Entropy has been applied in genomics for analyzing full genomes Tenreiro Machado, 2012 Schmitt & Herzel 1997)
and detecting exon and introns (Li et al., 2019 Koslicki 2011), and in molecular analysis for generating molecular descriptors (Delgado-Son Soler et al. 2009) and characterizing entanglement (Tubman & McMinii
2012). However, directly calculating entropy for a small set of sequences with ambiguous tokens is difficult and often unreliable (Schmitt & Herzel 1997).
Our metrie, fractal entropy, is based the KL divergence of the probability densities of two sets of sequences in universal sequence mapping (USM) space estimated with a fractal block kernel that respects suffix boundaries. Intuitively, fractal entropy mcasures differences in k-mer frequencics between sequences at different scales without the sparsity issues of traditional k-mer based measures. For a given sequence r = (21,72, ... ... ,TT,) of length T, and an alphabet A with size |A], a USM (Almeida & Vinga, 2002) maps each subsequence (r), to a point 8; the unit hypercube of dimension log |A] such that subsequences that share suffixes are encoded closely together. Details on the USM encoding process are provided in Appendix
C The USM space allows to analyze the homology between two sequences independently of scale
fixed-memory context.
To estimate the probability density of the USM points generated by sequence, we use Parzen window
estimation with a fractal block kernel (Vinga & Almeida 2007 Almeida & Vinga 2006) that computes a weighted k-mer suffix similarity of two sequences s; and Sy:
Σ/ olA 3 $ Ln,s, (si)
61, 8,)
(8)
Σ/. 2 8+

---

where Ik,e;(s)) is an indicator function that is 1 when the sequences corresponding to s, and 8j share the same length k suffix. The kernel is defined by two parameters: I, which sets the maximum k-mer size resolution, and 3, which controls the how strongly weighted longer k-mers are. For a given subsequence (ri,".,, encoded by the point 8, the probability density can then be computed as:
PL. 1 1/T, Ek=1 A k))
(9)
Σx=0 8h
where rz[: k] is the sullix of length k, (ri-k+1,"* rk), and c(-) is count function that reduces the task of summing the indicator function values to counting occurrences of particular k-mer in (Almeida & Vinga, 2006). When counting k-mers for MSA-based denoised sequences for which some S k ambiguous base pairs may be present, we add 1/|4|" to the counts of each potential k-mer. For example, the k-mer A7G adds 4/4 to the counts of k-mers [AAG, ATG, ACG, AGG]. To calculate the density for a set of sequences R, we can simply aggregate the k-mer counts and total lengths T₁ in Equation 9 over all sequences.
Finally, given set of subreads R with density QL.3 and denoised sequence f(R) with density PL,3; we can calculate the fractal entropy of (RR as the KL divergence between the two distributions:
(L.s(f())
R) = DKL(PLL (L,B) =
1
Σ SE(A)"
PL, log
(L.,(()) qr,,s(s))
(10)
[A
where we calculate the integral in the USM space sum all possible k-mers of size L, which we denote by {A)². We report results fractal entropy using a kernel with parameters L 9 and B 8. Further discussion of specific kernel parameters and calculation is provided in Appendix
5
Experimental Setup
5.1 Data
We begin by investigating the ability of our model to de- noise simulated antibody-like sequences for which the ground-truth source sequences are known. We generate a set of 10,000 source se- quences S using a procedure that mimics V-J recombination, a pro- cess by which the V-gene and J-gene segments of the autibody light chain recombine to form diverse sequences. Details provided in Appendix A To generate the simulated reads dataset D (R()) y we first select a sequence s(6) from S at random, then sample a num- ber of subreads to generate, m from a beta distribution with shape parameters a = 1.3, 3 = 3 and scale parameter 20. Finally, using the PBSTM2 $Ono el al. 2020) long read simulator with an error profile mimicking the R9.5 Oxford Nanopore Flow Cell, we generate a read from s()) with (+) subreads. We split these reads into training, validation, and test set with 99%%/5% 55%/5% split, respectively. On average, subreads have error rate of 18% and reads from given sequence ( are seen 9 times in the training set.
PBSIM:
100
Commlativ % of Docaset
80
80
40
Subreads per Dead
Figure 3: Cumulative distribution of subreads per read in the scfF anti- body test set. Over 60% of reads con- tain 6 or fewer subreads.
scFv Antibody: To investigate our model's ability to denoise real data, we use a proprictary experimental scFv antibody library soquenced with ONT. This dataset contains a total of 592,773 reads of antibody sequences, each consisting of a single light and heavy chain sequence. The scFv library was experimentally designed with more variation in heavy chain ground truth sequences compared to light chain ground truth sequences. In this paper focus on the light chain sequences since the higher similarity between reads lends itsclf well to the self-supervised method we propose. Since the lack of ground truth sequences means we cannot explicitly measure edit distance, we use our proposed LOO edit and fractal entropy metrics to compare the quality of denoised sequences instead. Each read contains 2 to 101 subreads with an average of 8 subreads per read, and an average light chain subread length of 327 base pairs. The cumulative distribution of subreads per reads is shown in Figure 3. No reads of size 1 are included since we only include reads for which valid consensus can be generated. As before, we split our data randomly into training, validation,
8

---

SSSL Se:
Avenge Sourre Sequecioo
50
MUSCLS
Malan
T.Caffe MAPPT
400
Median
Mcan
T:Comee
Editt Distanoe
60
300
MJSCCLE
Raindam
40
200
100
Subreads Read
10 14 16
188
20
Source Sequence Edit Distance
40
60
80
100
Figure 4: Median edit distances for reads of varying sizes. Error bars represent first and third quantiles. On reads with < subreads, SSSL-Mean outper- forms the best. baseline by ~66 bps on average. On reads with 6 subreads, SSSL-Set outperforms the best baseline by ~2 bps on average.
Figure 5: Histograms of source edit distances on simulated data. SSSL-Mean achieves lower vari- ance edit distances and is the only method that does not exhibit long tail of poorly denoised reads.
and test by randomly sampling 00%/5%/55 of the data respectively. Although additional experiments public datasets of similar size heavy chain data would bolster results, to our knowledge no real long read sequencing datasets exist at the scale of our scFv antibody dataset.
5.2 Baselines
We compare our method against baseline methods that only take the raw subread sequences as input. Our key baselines are MSA-based consensus methods, which generate a per-position alignment of the sub- reads. Specifically, we consider MAFFT (Katoh & Standley 2013), MUSCLE (Edgar 2004), and T-Coffee Di Tommaso et al. 2011). After alignment, we generate consensus by selecting the relative majority base pair at each position, where positions with ties assigned an "ambiguous" base pair. We also consider a direct median string approximation algorithm and naive algorithm that selects a random subread. We do not consider k-mer based methods as baselines as they require access to genome alignments. We also do not consider methods based on statistical error modeling, which require extensive domain knowledge about the error-generating process.
6 Results
6.1 PBSIM
We begin by examining the behavior of SSSL methods on simulated data. We first analyze the median edit distances for reads of given size in Figure 4 over the best bascline by average of over bps, 17% decrease in error rate. On larger reads with more than 6 subreads, SSSL-Set improves over the best bascline by an average of 2 bps, a 8% decrease in error rate. Tn addition, SSSL methods are able to denoise reads with only one or two subreads, which baseline methods cannot since no consensus or median can be formed. Examining the distributions of edit distances in Figure shows that most methods exhibit a long tail of reads that are denoised particularly poorly, with some even worse than our random algorithm. In contrast, SSSL-Mean achieves strong results without this long tail, exhibiting much lower variance in results. SSSL-Set displays tail similar to the best performing baseline while also achieving the lowest edit distances on individual reads. SSSL-Set's strong performance on large subreads indicates that a learned aggregation mechanism is crucial when combining information from many subreads which may be difficult. to align. In contrast, SSST-Mean's strong performance on smaller subreads indicates that this learned mechanism becomes more difficult to learn when few subreads are available, while simple mean pooling operation can still find high-quality midpoints.
On small reads with 6 or fewer subreads, SSSL-Mean improves
Since we have ground-truth source sequences for cach read in this simulated datasct, we also investigate the quality of our LOO edit and fractal entropy metrics. We calculate both metrics for all denoised sequences and measure their Pearson R correlation (Freedman al. 2007 with source sequence edit distances across

---

SSSL Set
SS5I-Mren
T-Calie
MUSCLE
LOO EiGH
MAFFT
# Sularends per Read
1.2
SSSL Coffee
Enttogg
MUSCLE
MAPFT
0.6
0.4
10
13
14
17
# Subreade per Resod
Figure 7: Results on scFv Antibody data. Top: median LOO edit distance. Bottom: median fractal entropy. Error bars represent first and third quantiles.
all denoising methods. Results are shown in Figure 6 We find that LOO edit achieves strong correlation for reads with fewer subreads, but decreases quickly as read size increases. This due to edit distance's bias towards local consistency which cannot maintained for a large number of subreads. In contrast, fractal entropy's correlation lower for smaller reads, but degrades slowly as read size increases and overtakes LOO edit for large reads. We report both metries in experiments real scFv antibody data they complement each other.
6.2 scFv Antibody
Next examine the results of our SSSL methods on real antibody sequencing data. Since we do not have ground truth source sequences for the reads in this dataset, distance and fractal entropy metrics instead. Given the observed behavior of our metrics and SSSL methods on simulated data, we consider "small" reads with I 6 subreads and "large" reads with > 6 subreads separately. Small reads comprise over 60% of the dataset. Results are shown Figure For large reads, all methods perform similarly IO0 edit. distance, with SSST, methods and median achieving slightly lower fractal entropy. For small reads, observe much larger differences in performance. Similar to the results on simulated data, SSSL-Mean achieves significantly lower LOO edit distance (~ 11 bps) and fractal entropy. SSSL-Set also beats bascline methods on these smaller reads but does not perform as well as SSSL-Mean. Since small reads comprise majority (> 60%) of the dataset, these results indicate that SSSL methods provide distinct advantage over standard MSA based algorithms.
analyze our proposed LOO edit
0.8
Pranson R
0.6
0.2
Entegg
Subcoods per Road
Figure 6: Pearson correlation of our evaluation metrics and source edit dis- tance. LOO edit achieves strong cor- relation on reads with few subreads, and fractal entropy achieves strong correlation on reads with many sub- reads.
Next, we qualitatively study the types of errors that SSSL corrects
better than baseline methods. The ONT platform used sequence antibodies is specifically prone introducing random insertions, deletions, and substitutions well as erroneous repetitions of the same base
10

---

S3S1
tttge
aaagtggggt
acatto.d
tttgo
asagtggggt
MAFFT
tttge
#azagtgg9gt
S58L
tacttagcotggt accagcag..................a..... ....999..999 acagac
scaffold
tacttagectggt accegeag... .999.999999 acagec
r001
tttgc
-eag----a---- -tagt
MAFFT
tecttagentgg Bccegnag... ..9. 99982cages
r002
Sttgo--aaaatggan
titgrsaaasggt9959
X001
tacttagectad gcag. ..a............................. adega
r004
tttge tttgo--aaa4499950 as999999 ttoge=-anaag9gg55
bacttagcetggtacceaat ................................ gacceaagge...gogggoo..ooo
r005
r003
tacttagcetgg tacageag. gsgttopgze
(a) Homopolymer insertions that throw off the align- ment and present difficulties for MAFFT are de- noised properly by SSSL.
(b) MAFFT outputs ambiguous predictions sec-
tions with large numbers of insertions and deletions, whereas SSSL remains robust.
SSSL
qaaat
scaffold
MAFET
r001
r002
cgte
r003
r004
gaantsstatssgacccaae caggeaceetgtatthet
(c) Both methods are capable of preserving true genetic variations the source sequence that differentiates it from the library scaffold.
Figure 8: Curated examples from the scFv dataset. SSSL fixes many errors commonly made by MSA-based algorithms such as MAFFT.
pair, known homopolymer insertions. An ideal denoising algorithm must be able to remove all these errors. We curate examples of reads denoised with SSSL and MAFFT that demonstrate each type of sequencing error in Figure S Here, "scaffold" refers to the sequence used at the library construction time to generate the read. While it is similar to the true source sequence, it may differ from it by some genetic variation.
In particularly noisy regions with homopolymers (Figure Sal SSSL generates the correct number of base pairs while MAFFT either generates too many or too few. In regions with many insertions and deletions (Figure (b). SSSL properly removes inserted base pairs while MAFFT cannot identify consensus nucleotide and often outputs ambiguous base pairs. When real genetic variations are present all the subreads (Figure Bch, both MAFFT and SSST produce the correct base pair, ignoring other reads it may have seen with different base pairs at those positions.
7 Analysis and Discussion
In this section analyze how different data and model settings affect the downstream success of denoising method. Since trained and evaluated on the PBSIM dataset described in Section 5.1
only control data parameters precisely on simulation data, all models are
7.1 Aggregation Method and MMD Kernel
In architecture design, consider multiple choices for aggregating the subread embeddings well the MMD kernel. For the aggregation scheme, experimented with sum pooling, max pooling, mean pooling, and a learned set transformer mechanism. Sun pooling and max pooling failed to converge to any reasonable performance. Since mean pooling and a learned set transformer work well in different settings as shown in Figure we also consider learning a weighted sum between the two operations. However, we found this combined aggregation to perform worse than the individual aggregation models. For our MMD kernel, in addition to the Gaussian kernel also considered dot product and cosine similarity kernel. Both models failed to converge.
7.2 Regularization
How important is cach regularization component of our SSSL objective? SSSL has key regularization components: sequence masking, embedding decay, embedding noise, sequence midpoint loss, latent midpoint loss, and 7) loss weighting. To investigate the individual contribution of each component, remove them
11

---

60
SSSL-Set
Average Source Distatic
Average Souree Dditt Distrnnce
No Embed Desay
No Masking
30
20
200
10
SSSL-Set
SSSL-MMan
0
# Repeats per Read
# Reads Seen Training
Figure 9: All the regularization techniques used to train SSSL models are important, and remov- ing any individual component reduces the effec- tiveness of our method.
Figure 10: SSSL methods achieve consistent per- formance in blind denoising across sampled scaf- folds, even when reads generated from them are only seen a single time during training.
and retrain the model for a fixed number of steps on the simulated antibody dataset. We perform ablations on the SSSL-Set variant of our model. Results are shown Figure g We find that all of our regularization components important in improving the success of our model. The latent midpoint and sequence midpoint Josses are the most important; removing them causes the model to diverge and the loss to explode, we do not display these results. Input sequence masking is the next most important, and removing it increases the source edit distance on average by ~4.3 bps. Removing the embedding decay and setting the loss weighting term to increase the edit distance by ~227 bps, and removing the embodding noise increases the edit. distance by ~1.2 bps.
7.3 Number of Reads Seen During Training
How does our model perform when it sees only few reads of a particular scaffold during training? We analyze the edit distances of reads whose corresponding scaffolds were seen varying times during training. Our results are shown in Figure I0 We find that SSSL methods achieve consistent performance across all scaffolds, regardless of how many times reads generated from them were seen during training. This demonstrates SSSL's ability to learn a well-conditioned sequence embedding space that does not simply rely on memorizing commonly seen sequences.
8 Conclusion
We propose self-supervised set learning (SSSL.), method for denoising set of noisy sequences without observing the associated ground-truth source sequence during training. SSSL learns an embedding space in which the noisy sequences cluster together and estimates source sequence representation as their midpoint. This set embedding, which represents an "average" of the noisy sequences, can then be decoded to generate a prediction of the clean sequence. We apply SSSL the task of denoising long DNA sequence reads, for which current denoising methods perform poorly when source sequences are unavailable or error rates are high. To evaluate method real data with ground truth, we propose two self-supervised metrics: leave-one-out (LOO) edit distance and fractal entropy. On simulated dataset of antibody- sequences, SSSL methods consistently achieves lower source sequence edit distances compared to basclines, reducing error rates by 17% on small reads and 8% on large reads. On an experimental dataset of antibody sequences, SSSL significantly improves both proposed metrics over baselines on small reads which comprise over 60% of the datasct, and matches bascline performance on the remaining larger reads. By denoising these difficult smaller reads, SSSL enables their use in downstream analysis and scientific applications, more fully realizing the potential of long-read DNA sequencing platforms.
12

---

References
Trina Abnizova, Steven Leonard, Tom Skelly, Andy Brown, David Jackson, Marina Gourtovaia, Guoying Qi, Rene Te Boekhorst, Nadeem Faruque, Kevin Lewis, et al. Analysis of context-dependent errors for illumina sequencing. Journal of Bioinformatics and Computational Biology, 10(00))1241005, 2012.
Jonas S. Almeida and Susana Vinga. Universal sequence map (usm) of arbitrary discrete sequences. BMC Bioinformatices, 3(1):6, Feb 2002. ISSN 1471-2105. doi: 10.1186/1471-2105--1 URL. https://dii.orrr org/ 10. 111111661111111-222
Jonas S. Almeida and Susana Vinga. Computing distribution of scale independent motifs in biologi- cal sequences. Algorithms for Molecular Biology, 1(1):18, Oct 2006. ISSN 1748-7188. doi: 10.1186/
1748-7188-1-18. URL https: http:////ooo.o//...oo org/10. 1111://1178--11---11--
Dana Angluin and Miklós Csürös. Learning Markov chains with variable length memory from noisy output. Tn Proc. 10th Annu. Conf. on Comput. Learning Theory, pp. 298 308. ACM Press, New York, NY, 1997.
Gunjan Baid, Daniel E. Cook, Kishwar Shafin, Taedong Yun, Felipe Llinares-López, Quentin Berthet, Anas- tasiya Belyaeva, Armin Töpfer, Aaron M. Wenger, William J. Rowell, Howard Yang, Alexey Kolesnikov, Walced Ammar, Jcan-Philippe Vert, Ashish Vaswani, Cory Y. McLean, Maria Nattostad, Pi-Chuan Chang, and Andrew Carroll. Deepconsensus improves the accuracy of sequences with a gap-aware sequence transformer. Nature Biotechnology, Sep 2022. ISSN 1546-1696. doi: 10.1038//11587-022001133 URL
https://aai..or/10...2 1ttp:/000.000//11180000000333330
Joshua Batson and Loic Royer. Noisc2self: Blind denoising by self-supervision. In International Conference on Machine Learning, pp. 534-533. PMLR, 2019.
Lauren Bragg, Glenn Stone, Michael Imelfort, Philip Hugenholtz, and Gene W. Tyson. Fast, accurate error-correction of amplicon pyrosequences using acacia. Nalure Melhods, 9(5):425 426, May 2012. ISSN 1548-7105. doi: 10.1038/ 10.1038/mmeth. wwwth. 1990. URL https://doi.
1038/nmeth, 1990
Olivier Chapelle, Bernhard Schölkopf, and Alexander Zien. Semi-Supervised Learning (Adaptive Computation and Machine Learning). The MIT Press, 2006. ISBN 0262033585.
Jian Chen, Le Yang, Lu Li, and Yijun Sun. Predicting alignment distances via contimnous sequence matching. bioRaiv, 2020. doi: 10. 10.1101//0 /2t.//222.....113333... 09/01/2020.00 05. 24. 113852
URL hattps://wwe h/wp.///rxwwwwiiiv/oooooo wwt.t/ottee///errrlrl/oaoaaoo/// 2020/
Gabriele Corso, Rex Ying, Michal Pándy, Petar Veličković, Jure Leskovec, and Pietro Liò. Neural distance embeddings for biological sequences. Advances in Neural Information Processing Systems, 2021.
Xinyan DAI, Xiao Yan, Kaiwen Zhou, Yuxuan Wang, Han Yang, and James Cheng. Convolutional embedding for edit distance. In Proceedings of the 43rd International ACM SIGIR Conference on Research and
Development in Information Retrieval, SIGIR '20 pp. 599-608, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781450380164. doi: 10. 10.1145/3397271..3B2 10. N0.1145/33972711 3401045
URL https:://oi.ooooo org/
Fred J. Damerau and Eric Mays. An examination of undetected typing errors. Information Processing & Management, 6566) 664, 1989. ISSN 0306-4573. doi: URL URRpanppp///rr..ccccccaaarrrrrr wtt.a/cccc....tt...ooo/o/oooo//a
(0306-45773
Laura Delgado-Soler, Raul Toral, M. Santos Tomás, and Jaime Rubio-Martinez. Red: A set of molecular
descriptors based rényi entropy. Journal of Chemical Information and Modeling, 49(11):2457 24412224477-4468 2009. doi: 10.1021//1900255w URL https://doi.onn org/ 1021///19902
PMID: 19842710.
Paolo Di Tommaso, Sebastien Moretti, Ioannis Xenarios, Miquel Orobitg, Alberto Montanyola, Jia-Ming Chang, Jean-François Taly, and Cedric Notredame. T-Coffec: a web server for the multiple sequence alignment of protein and RNA sequences using structural information and homology extension. Nucleic Acids Res,, 39 Web Server issue):W13 7, July 2011.
13

---

Robert C. Edgar. Muscle: multiple sequence alignment method with reduced time and space complexity. BMC Bioinformalies, 5(1):113, Aug 2004. ISSN 1471-2105. doi: 10.1186/1471-2005-55511 URL https:
//doi.org/ org/10. 0100://01111//14172200---0---5
Y. Ephraim and N. Merhav. Hidden markov processes. IEEE Trensactions on Information Theory, 48(6):
1518 1569, 2002. doi: 101109/TTT.2002110000
Linwei Fan, Fan Zhang, Hui Fan, and Caiming Zhang. Brief review of image denoising techniques. Visual Computing for Industry, Biomedicine, and Arl, 2(1):1-12, 2019.
David Freedman, Robert Pisani, and Roger Purves. Statisties (international student edition). Pisani, R. Purves, 4th edn. WW Norton & Company, New York, 2007.
Yan Gao, Yongzhuang Liu, Yanmei Ma, Bo Liu, Yadong Wang, and Yi Xing. abPOA: an SIMD-based
C library for fast partial order alignment using adaptive band. Bioinformaties, (2(15):20092221 2020. ISSN 1367-4803. doi: 10.1093/bio phooinformatiis//aaa000
URL https://doi org/10.1093
bioinformaticc//taaage
Sara Goodwin, John D. McPherson, and W. Richard McCombie. Coming of age: len years of next-gemeration sequencing technologies. Nature Reviews Genetics, 17(3):333 351, Jun 2016. ISSN 1471-0064. doi: 10.
1038/nrg:2016.20m URL htt. htt.///........///g.g...... wtt.rrg..06...0 2016.49
Paul Greenfield, Konsta Duesing, Alexie Papanicolaou, and Denis C. Bauer. Blue: correcting sequencing
errors using consensus and context. Bioinformalicss 30(19):2723 2732, 06 2014. ISSN 1367-4803. doi:
htta////booioooattte//bbeeeeee btt366.. URL httt::://ooi..rrr//...00///oooo0
Arthur Gretton, Karsten M. Borgwardt, Malte Rasch, Bernhard Schölkopf, and Alexander Smola. kernel two-sample test. Journal of Machine Learning Research, 13(25):723 773, 2012. URL http://jmlr. org/
happrs//v3//rettonnlt html
Lluís Gómez, Marçal Rusiñol, and Dimosthenis Karalzas. Lsde: Levenshtein space deep embedding for query-by-string word spotting. In 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR), volume 01, pp. 504, 2017. doi: 10.1 2.1100//CDARR.201...5
Sergey loffe and Christian Szegedy, Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference International Conference on Machine Learning Volume 37, (,,, pp. 448-456. JMLR.org, 2015.
Wei-Chun Kao, Andrew H Chan, and Yun S Song. Echo: a reference-free short-read error correction algo- rithm. Genome research, 21(7):1181-119) 2011.
Kazutaka Katoh and Daron M Standley. MAFFT multiple sequence alignment software version 7: improve- ments in performance and usability. Mol. Biol, Evol., 30(4):772 780, April 2013.
Yekyung Kim, Seohyeong Jeong, and Kyunghyun Cho. LINDA: Unsupervised Learning to Interpolate in Natural Language Processing. ar e-prints, art. arXiv:2112.13969, December 2021.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arv preprint arXiv:1412 198600 2014.
Sergey Koren, Brian P Walenz, Konstantin Berlin, Jason R Miller, Nicholas H Bergman, and Adam M Phillippy. Canu: scalable and accurate long-read assembly adaptive k-mer weighting and repeat separation. Genome Res., 27(5):722 736, May 2017.
David Koslicki. Topological entropy of DNA soquences. Bioinformatics, 1060110061 02 2011. ISSN 1367- bito///fooraaaics///.rrr. 100::000//b/iiiiraiaaieeee URL
4803. doi: 10.1093
Alexander Krull, Tomáš Vičar, Mangal Prakash, Manan Lalit, and Florian Jug. Probabilistic noise2void: Unsupervised content-aware denoising. Frontiers in Computer Science, 2:5, 2020.
14

---

David Lachnemann, Arndt Borkhardt, and Alice Carolyn McHardy. Denoising DNA deep sequencing data-highte high-throughput sequencing errors and their correction. Briefings in Bioinformalliss, 1:54- 179, 05 2015. ISSN 1467-5463. doi: ww..099//ib/bbritt
URL
Samuli Laine, Tero Karras, Jaakko Lehtinen, and Timo Aila. High-quality self-supervised deep image de- noising. Advances in Neural Information Processing Systems, 32, 2019.
Yann LeCun, Koray Kavukcuoghu, and Clément Farabet. Convolutional networks and applications in vision. In Proceedings of 2010 IEEE international symposium on circuils and systems, pp. 253 256. IEFE, 2010.
Byunghan Lee, Taesup Moon, Sungroh Yoon, and Teachy Weissman. Dude-seq: Fast, flexible, and robust denoising for targeted amplicon sequencing, PLoS ONE, 12(())e0181463, 2017. doi: 10.1371/journal.por pone. 0181463.
Juho Lee, Yoonho Lee, Jungtack Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set trans- former: A framework for attention-based permulation-invarian neural networks In Proceedings of the 36th International Conference on Machine Learning, pp.
2019.
Jaakko Lehtinen, Jacob Munkberg, Jon Hasselgren, Samuli Laine, Tero Karras, Mika Aittala, and Timo Aila. Noisc2noise: Learning image restoration without clean data. arXiv preprint Xiv:11803. Xxx:1888.04410, 2018.
Junyi Li, Li Zhang, Huinian Li, Yuan Ping, Qingzhe Xu, Rongjie Wang, Renjie Tan, Zhen Wang, Bo Liu, and Yadong Wang. Integrated entropy-based approach for analyzing exons and introns in dna sequences. BMC Bioinformatics, 20(8):283, Jun 2019. ISSN 1471-2105. doi: 10.1 (0.118//122500-019-7777
URL
https://doi.. 0rgg11..10////2285-01199999
Eun-Cheon Lim, Jonas Müller, Jörg Hagmann, Stefan R. Henz, Sang-Tac Tae Kim, and Detlef Weigel. Trowel: a fast and accurate error correction module for Illumina sequencing reads. Bioinformatics, 30622):3261- 3265, 07 2014. ISSN 1367-4803. doi: 10.1093/bioiie bitp//oooooiiiis/////// /tt5133. URL https://dol orgg/0..0000 bioinformaticc//buut
Xiaotu Ma, Ying Shao, Liqing Tian, Diane A Flasch, Heather L Mulder, Michael N Edmonson, Yu Liu, Xiang Chen, Scott Newman, Joy Nakitandwe, et al. Analysis of error profiles in deep next-generation sequencing data. Genome biology, 20(1):1-15, 2019.
Swati C Manekar and Shailesh R Sathe. A benchmark study of k-mer counting methods for soquencing. GigaScience, 7(12), 10 2018. ISSN 2047-217X. doi: 10.1093/ gigascienco//iig 12y URL https:
7/doi org/ioor1111..0 1093/ http://ggaaciicceggggeeo g2y125 giy
Eric Mays, Fred J. Damerau, and Robert L. Mercer. Context based spelling correction. Information Process- ing & Management, 27(5): 522, 1991. ISSN 0306-4573. doi: http:://tb..org//1.111//0/0 000-000066000004433900 90066-U. URL
Frazer Mcacham, Dario Boffelli, Joseph Dhahbi, David IK Martin, Mcromit Singer, and Lior Pachter. Iden- tification and correction of systematic error in high-throughput sequence data. BMC Bioinformalics, 12 (1):451, Nov 2011. ISSN 1471-2105. doi: 10.1186 11.1186/4471-1055 111-1-2100-12-45.1 URL https://doi. http::////ooooo/1....o 1186/
12-1-2105-124551
Paul Medvedev, Eric Scott, Boyko Kakaradov, and Pavel Pevzner. Error correction of high-throughput sequencing datasets with non-uniform coverage. Bioinformalics, 27(13):1137 i141, 06 2011. ISSN 1367- 4803. doi: 2.1093/bioinformaate
URL https: org/10..00111. http:///...///biiooooott..tt./// /btr208
Christopher A Metzler, Ali Mousavi, Reinhard Heckel, and Richard G Baraniuk. Unsupervised learning with stein's unbiased risk estimator. arXiv preprint 105.:1105 2018.
Sergey Nikolenko, Anton I. Korobeynikov, and Max A. Alckscyev. Baycshammer: Bayesian clustering for error correction in single-cell sequencing. BMC Genomics, 14(1):87, Jan 2013. ISSN 1471-2164. doi: 10.116//447-22664444-11 URL https:://ool http:///.....r....o....111111111
15

---

Yukiteru Ono, Kiyoshi Asai, and Michiaki Hamada. PBSIM2: simulator for long-read sequencers with
novel generative model of quality scores. Bioinformaties, 37(5):589 595, 09 2020. ISSN 1367-4803. doi: hiai000//iiinfoomaatices//// htt.ooornnnatcc//baa5555555 btaa835. URL https::/dbi..org/// http:://......0....ooooooo/oaaaa
Martin Raphan and Eero P. Simoncelli. Least squares estimation without priors supervision. Newral
Comput., 23(2):374-22) feb 2011. ISSN 0899-7667. doi: 10.1162/NECO a 00076. URL WtLLhtts:://dd
105//00.1662//B/ 00076
Kristoffer Sahlin and Paul Medvedev. Error correction enables use of oxford nanopore technology for reference-free transcriplome analysis. Nalure Communiicalions, 12(1):2, Jan 2021. ISSN 2041-1723. doi:
10.1038/441146--0022003383 URL http:///0//////////////////////0
Leena Salmela and Jan Schröder. Correcting errois in short reads by multiple aliguments. Bioinformatics, 27(11):1555-1661 04 2011. ISSN 1367-4803. doi: 10.1093 10.1093/bioinfone bit.n/ooooooiccs/////////o /bt1170. URL https://doi.
A O Schmitt and H Herzel. Estimating the entropy of DNA sequences. J. Theor. Biol., 188(3):369 377, October 1997.
Marcel H. Schulz, David Weese, Manuel Holtgrewe, Viktoria Dimitrova, Sijia Niu, Knut Reinert, and Hugues Richard. Fiona: parallel and automatic strategy for read error correction. Bioinformatics, 30(17):3356- 1363, 08 2014. ISSN 1367-4803. doi: 10.1093 10.1099/bionfoommatiss btwo//oormatics/bbuteeee /bu4440. URL https: http:/:///oo.oo/1.....o bioinformattics/buuuee
Raphael Shu, Jason Lee, Hidcki Nakayama, and Kyunghyun Cho. Latent-variable non-antoregreessive neural machine translation with deterministic inference using a delta posterior. CoRR, abs/ 1908.07181, 2019.
URL /
Ioan Tabus, Jorma Rissanen, and Jaakko Astola. Normalized Mazimum Likelihood Models for Boolean Regression with Application to Prediction and Classification in Genomics, pp. 173-189. Springer US, Boston, MA, 2002. ISBN 978-0-306-17825-3, 10. 1007/0-306 01177/00-306-7725-0000 47825-0010
doi: 10.1007 141100700-30--17825-0 10. URL https://doi.or//
Ioan Tabus, Jorma Rissanen, and Jaakko Astola. Classification and feature gene selection using the nor- malized maximum likelihood model for discrete regression. Signal Processing, 83(4):713 727, 2003. ISSN 0165-1684. doi: https://doi..rr 1tt.:///..r11111111161111 http://0wc///rrccco////////66660
URL https://ww.scieeneeiieee www.cccieeee.er.ct.t... Genomic Signal Processing.
J. A. Tenreiro Machado. Shannon entropy analysis of the genome code. Mathematical Problems in
Engineering, 2012:132620 May 2012. ISSN 1024-123X. doi: 10.1155/2012/132625, URL https:
//doi wtt.oo/.//......0....... 11111/11//2221162222
Norm M. Tubman and Jeremy McMinis. Renyi Entanglement Entropy of Molecules: Interaction Effects and Signatures of Bonding. arXiv e-prints, art. arXiv:1204.4331, April 2012.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Deep image prior. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 9446 9446-9454, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL
neurips. /carrips.c//apprr//////////
Papiaa--ppr..... Pfff
Susana Vinga and Jonas S Almeida. Local renyi entropic profiles of DNA sequences. BMC Bioinformatics, 8(1):393, October 2007.
Tsachy Weissman, Erik Ordentlich, Gadiel Scroussi, Sergio Verdú, and Marcelo Weinberger. Universal discrete denoising: Known channel. IEFF Transactions on Information Theory, (1): 28, 2005.
16

---

Xiao Yang, Karin S. Dorman, and Srinivas Aluru. Reptile: representative tiling for short read error correction. Bioinformalices, 26(20):2526 2533, 08 2010. ISSN 1367-4803. doi: 10.1093/ 10.:10/////iionnfoono httas//ooooooomss//ittteseee URL
Xiao Yang, Sriram P. Chockalingam, and Srinivas Aluru. A survey of error-correction methods for next- generation sequencing. Briefings in Bioinformatics, 14(1):56-66, 04 2012. ISSN 1467-5463. doi: 10.1093/
bib/bbs015. URL httppp///aab.oog//....000////bb
Xin Yin, Zhao Song, Karin Dorman, and Aditya Ramamoorthy. Premier turbo: Probabilistic error-correction using markov inference in errored reads using the turbo principle. In 2013 IEEE Global Conference on Signal and Information Processing, pp. 73-76, 2013. doi:
Wei Zheng, Le Yang, Robert J Genco, Jean Wactawski-Wende, Michael Buck, and Yijun Sun. SENSE: Siamese neural network for sequence embedding and alignment-free comparison. Bioinformatics, 3511 1820-1828, 10 2018. ISSN 1367-4803. doi: 10.1093/bioan http://oooooancccbbeeeeeeee URL sttpps://oi... org/10.
1093/biooinfommaiic////bb//
Magauiya Zhussip, Shakarim Soltanayev, and Se Young Chun. Training deep learning based image denoisers from undersampled measurements without ground truth and without image prior. In Proceedings of the IEEE/CVP Conference on Compuler Vision and Pollern Recognition, pp. 10255 10264, 2019.
A Simulated Data Generation
In this section we describe our data simulation process. First, we randomly generate template V-gene and J- gene sequences by sampling sequence length then randomly sampling base pair from {A,T,C, C,G) uniformly and independently for each position. V-gene sequence lengths are sampled from normal distribution with = 300 and o, = 6. J-gene sequence lengths are sampled from a normal distribution with μj = 33 and = Given the resulting sets of V-gene and J-gene Lemplates, V and J, we then generale a set of source sequences S by concatenating each template sequence in V with each template sequence J. For our dataset we gencrate 100 V soquences and 100 J sequences for a total of 10,000 sequences.
Model and Training Hyperparameters
In this section we describe model and training hyperparameters. We preprocoss data by tokenizing sequences using codon vocabulary of all 1-, 2-, and 3-mers. We learn token and position embedding with dimension 64. Our sequence encoder and decoder are 4-layer transformers (Vaswani et al. 2017 with 8 attention heads and hidden dimension of size 64. Our set transformer also uses a hidden dimension of size with attention heads. On top of the base encoder we apply an additional 3-layer projection hoad all with dimension and BatchNorm (Tolle & Szegedy 2015) layers between each linear layer. Decoding is performed via beam search with beam size 32. All models are trained with the Adam optimizer (Kingma & Ba 2014) with learning rate of 0.001 and batch size of 8 reads, although the total number of subreads present varies from batch to batch. We apply loss weighting values = 10 and À 0.0001, and apply independent Gaussian noise to embeddings with standard deviation of 0.01. Models and hyperparameters are selected based on validation LOO edit distance (Section
C Fractal Entropy
C.1
USM Encoding
Given an alphabet A (for DNA, A = (A,T,C,C)). we definc the universal sequence mapping (USM) space (Almeida & Vinga 2002) in the d = log, |A| 2 dimensional unit hypercube where each corner corresponds to character in our alphabet. For a given sequence r = (r1, T2, ,r,,,, with length T,, we (8), .2. st., generated by a chaos game that randomly se- lects Sy On, 1)d then calculates s; 1/2(s;-1 1 b,) where b, is the corner of the hypercube corresponding to the character position
compute a sequence of USM coordinates
17

---

16
Pearson It of Entropy and Source Sequence Edit Distance
Pvarson R. of Eatropp and Souree Sequence Edit Distance
-0.5
-0.5
2
10
12
14
16
18
20
2
6
10
12
14
16
18
20
# Subreads per Read
# Subreads per Read
Figure 11: Ablations on fractal kernel parameters and / which control kernel resolution and smoothness respectively. Increasing L and B increases correlation on all read sizes but reaches plateau with minimal improvements for the additional computational costs. In our experiments we use kernel with parameters L L= 9 and B 16.
C.2 L and 3 Ablations
In this section analyze the behavior of fractal entropy as our kernel parameters, L and 8 change. In- tuitively, L controls the resolution of the kernel, and the smoothing parameter 8 control the weighting among k-mers of different lengths. Values 3 < 1/1 ^a correspond to higher weighting of shorter k-mers, and 3 > 1/AA correspond to higher weighting of longer k-mers (Almeida & Vinga 2006). As we desire consistency between longer k-mers, we consider only values of 1/ A). As increases, the correlation of fractal entropy increases as well, although large values of perform similarly. Since increasing the resolution of the kernel increases computation costs, select value of
as reasonably high resolution that corresponds to group of 3 codons. We observe a similar behavior when increasing values of B. Since the correlation values with 8 = 16 are marginally higher, we select this value in our experiments.
18


Reviews:
[{"review_id": "AKPyeoSqMY", "reviewer": "Reviewer_swPp", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n* Overall, I think this paper is well written, proposes an interesting architecture, and shows reasonable improvements on the datasets chosen.\n* The problem is an important one, and lowering error rate even slightly can have significant impact on many scientists' work\n* The set-aggregator architecture is a novel and interesting contribution, and is a good fit to the problem space. The authors motivate the design well and for the most part explain the architecture clearly.\n* The two datasets chosen demonstrate the utility of the method well.\n\n## Weaknesses\n* I find it slightly difficult to follow the architecture definition. Figure 2 is very helpful, but is missing lengths of various components. For example, I didn't quite see how the length of the consensus sequence was decided. T' is the average of subread lengths. Is this also the length of the consensus sequence that is decoded?\n* The datasets chosen for demonstration are good, but also limited. One is a simulated dataset. This leaves an evaluation on only one real dataset. Furthermore, the authors propose new metrics for this dataset. The authors also note they expect their method to be especially effective on this dataset (\"We focus on the light chain sequences in this\npaper since the unobserved ground truth sequences should exhibit fewer variations from a set of known\nscaffolds, increasing the effectiveness of our model\").", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "* Introduce a new set-autoencoder based method of denoising sequencing reads in a setting where no ground truth alignments are available\n* Propose two new metrics for evaluating decoding when a ground truth sequence is not available.\n* Show improved performance on simulated data and real data (scFv antibody dataset)", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "## Critical\n* Improve clarity of architecture description. One suggestion is to add lengths to Fig. 2, but if the authors have a better idea they need not follow that exactly.\n* Clarify how the length of the consensus sequence is chosen (it is possible I just missed this). If it is just T', this should be said explicitly somewhere.\n\n## Improvements\n* I don't think further experiments are critical for publication. I do think testing on 1-2 additional datasets, including some that are not necessarily advantageous for the method would significantly improve the impact of the paper.\n* It would be good to see a comparison of the proposed metrics (LOO and Fractal Entropy) against edit distance to ground truth sequence in a case where that edit distance is known.", "broader_impact_concerns": "I don't believe there are any significant ethical concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "tfVYVhJmNP", "reviewer": "Reviewer_Lwhc", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I would like to preface this with the statement that I am not an expert in sequencing analysis, consensus building methods, or long read sequencing. That being said, I do have substantial experience in machine learning on sequencing data data and the antibody domain. \n\nStrengths:\n\n- Timely given long read sequencing is the nature methods method of the year in 2022. Denoising long reads is an important task, even as error rates continue to improve over time.\n- Novel algorithm that seems to work well for light chain antibody sequence denoising against baselines.\n- The method is overall is quite sensible and applies known techniques in deep learning for this novel application.\n- The proposed evaluation metrics seem reasonable and could be useful for future benchmarks in the same direction, although the name “fractal entropy” is a bit opaque and does not give me a great understanding of what it is measuring.\n\nWeaknesses:\n\n- Applicability: While this method is called blind biological sequence denoising, it is only applied on denoising antibody light chains on a single dataset. It is therefore unclear from the experiments when SSSL can be applied successfully. I am particularly curious about the setup, and when the “subread” setup, where it is known which subreads correspond to which antibody, is applicable. I would not expect that this correspondence is known in general.  In addition,\n    \n    > We focus on the light chain sequences in this paper since the unobserved ground truth sequences should exhibit fewer variations from a set of known scaffolds, increasing the effectiveness of our model.\n    > \n    \n    While it is useful to show off the usefulness of SSSL in this application, it somewhat brings into question the wider applicability of SSSL if results are only shown on this portion of the data.\n    \n- The baselines are somewhat weak, and all from prior to advent of efficient long-read sequencing (2004, 2011, 2013) where long read sequencing has only really become useful in the last 3-5 years. For example, Zhang et al. 2020 benchmarks error correction methods in long read sequencing. Could the authors discuss whether or not these are applicable here? I am personally not familiar with methods used for antibody sequence consensus building from long read sequencing in practice today. What is used in practice for antibody consensus building? Is it one of these three methods?\n\nMinor:\n\n> In addition, SSSL methods are able to denoise reads with only one or two subreads, which the baseline methods cannot since no consensus or median can be formed.\n\nI think this is a bit of an exaggeration. The consensus / median absolutely can be formed for one or two subreads.\n\n\nZhang, H., Jain, C., & Aluru, S. (2020). A comprehensive evaluation of long read error correction methods. *BMC genomics*, *21*(Suppl 6), 889. https://doi.org/10.1186/s12864-020-07227-0", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work presents a new algorithm called self-supervised set learning for the task of blind DNA sequence denoising. Two metrics are proposed, leave-one-out edit distance and a “fractal entropy” metric to measure denoising capability without a ground truth. Compared to existing baselines, SSSL is able to better denoise antibody sequences, which becomes particularly apparent in the few subread scenario.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would ask the authors consider a more specific framing of title, introduction, and conclusion, as the experiments are only on antibody sequence denoising, and it is not clear that the SSSL setup of subreads with known correspondence to antibodies is applicable to general long-read sequencing problems or biological sequences in general. I think either the paper must be framed as an antibody sequence denoising method, or substantially more experiments and discussion is needed to back up the claim that SSSL can be used for long-read and / or biological sequences in general over simpler MSA-based methods.\n\nWhile long read sequencing is used here, I believe these are relatively short long reads, and really at the upper edge of what I would consider a short read (please correct me if this is incorrect). From my understanding long reads can be between thousands and hundreds of thousands of base pairs. I suspect different methods may be needed to scale to read sizes of these lengths, even if just in the transformer architecture used. Could the authors add discussion of how the length of read affects the applicability of SSSL?\n\nCould the authors provide more details on the datasets used? Specifically\n\n- Are the datasets public and under what license? If not, will they be made public?\n- What preprocessing / preprocessing is done to the raw reads before applying SSSL?\n- How are subreads assigned to antibodies, and what might the error rate on this be? Is this using some sort of random tagging?\n\nCould the authors provide more detail on how hyperparameters were picked? \n\n- “Models and hyperparameters are selected based on validation LOO edit distance”.\n- How specifically is this done?\n\nThe error rate for long-read sequencing has substantially decreased in the past few years. Could the authors discuss the error rate for this length of sequence? Could the authors discuss the effects of error rate on this algorithm? Especially with known ground truth, how does error rate affect the performance of SSSL as compared to more traditional MSA consensus methods.\n\nOverall I do believe this is useful and novel work, but has only been demonstrated on a narrow subproblem in long read sequencing, specifically, antibody light chain denoising, and either needs stronger empirical results to back up the claim of general biological sequence denoising, or needs to be written as a more application specific paper. Both ways in my opinion would be acceptable at TMLR.\n\nSome potential ways to strengthen the claims empirically:\n\n- More datasets on diverse denoising tasks\n- Results denoising the heavy-chain sequence of antibodies\n- Stronger and more recent baselines of consensus methods", "broader_impact_concerns": "N / A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "4QxOljQvbs", "reviewer": "Reviewer_6eRu", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "_[**Disclaimer**: I have reviewed this work for TMLR before. I will comment mainly on how well authors have addressed my comments:]_\n\nI have suggested the authors to change their title, to be more explicit about the biological domain. They have indeed done so.\n\nI have suggested adding new baselines, some of which parametric. The authors have broadly extended the set of baselines, which I am very happy with.\n\nOn performing ablations, the authors ablate against one additional aggregation function (mean). This is a welcome addition, but more work can be done. For example, I would have expected at least max to also be considered. Further, the authors haven't taken into consideration either of the two references I recommended here: \n> The authors may wish to consult either the set representation paper \"On the Limitations of Representing Functions on Sets\" (Wagstaff, Fuchs, Engelcke, et al., ICML'19) or Principal Neighbourhood Aggregation (Corso, Cavalleri et al., NeurIPS'20) when deciding on various aggregation schemes to use.\n\nand I think taking these works into account would be valuable.\n\nI have suggested to compare and contrast against NeuroSEED. While I believe the authors could have empirically compared to NeuroSEED -- if nothing else, to have a \"supervised\" baseline -- I do think the comparison is now crisp enough that there are no doubts about the present work's contributions.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a new set-based self-supervised embedding method for denoising reads (mainly of biological sequence data, such as DNA).\n\nThe key idea is to embed individual sub-reads in a shared embedding space in a self-supervised manner, aggregate them together, and then denoise by decoding from the aggregated embedding.\n\nThe method appears to improve performance compared to traditional MSA-based approaches on biologically-relevant reconstruction datasets---both simulated and real-world.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would encourage the authors to provide a more detailed ablation study of the various set aggregation choices in the minimum, and ideally  to go beyond ablating just the aggregator to encompass other aspects of their proposed model. Otherwise, I feel this was a solid revision, and the paper is nearly ready for acceptance.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Requested changes:
1. Please discuss the limitations of your empirical evaluation, including the focus on light chains. Also, mention this focus in the introduction.
2. In section 6.1, it is claimed that "fractal entropy’s correlation increases as read size increases", while Figure 6 shows that fractal entropy correlation starts decreasing after a certain point. Please make the description consistent with the figure.
Decision: Accept (poster)