Self-GenomeNet: Self-supervised Learning with Reverse-Complement Context Prediction for Nucleotide-level Genomics Data
Authors: Hüseyin Anil Gündüz
Martin Binder
Xiao-Yin To
René Mreches
Philipp C. Münch
Alice C McHardy
Bernd Bischl
Mina Rezaei
Venue: ICLR 2022 Submitted (2022)
Keywords: Genome Sequence Analysis
Self-supervised Learning
Representation Learning
Application in Computational Biology

Abstract: We introduce Self-GenomeNet, a novel contrastive self-supervised learning method for nucleotide-level genomic data, which substantially improves the quality of the learned representations and performance compared to the current state-of-the-art deep learning frameworks. To the best of our knowledge, Self-GenomeNet is the first self-supervised framework that learns a representation of nucleotide-level genome data, using domain-specific characteristics. Our proposed method learns and parametrizes the latent space by leveraging the reverse-complement of genomic sequences. During the training procedure, we force our framework to capture semantic representations with a novel context network on top of intermediate features extracted by an encoder network. The network is trained with an unsupervised contrastive loss. Extensive experiments show that our method with self-supervised and semi-supervised settings is able to considerably outperform previous deep learning methods on different datasets and a public bioinformatics benchmark. Moreover, the learned representations generalize well when transferred to new datasets and tasks. The source code of the method and all the experiments are available at supplementary.

Under review as a conference paper at ICLR 2022
SELF-GENOMENET SELF-SUPERVISED LEARNING WITH REVERSE-COMPLEMENT CONTEXT PREDICTION FOR NUCLEOTIDE-LEVEL GENOMICS DATA
Anonymous authors Paper under double-bling blind review
ABSTRACT
In this paper, we introduce novel contrastive self-supervised method, namely Self-GeenmeeNet, for nucleotide genome representation learning. To the best our knowledge, Solf-GenomeNer is the first self-supervised framework that learns a representation of nucleotide genomic data, using Nomain-speciff characteris- ties. Our proposed method learns and parameterizes the latent space by leverag- ing the reverse-complee complemet of genomic sequences. During the training proce- dure, we force our framework to capture semantic representations with a novel context network on top of features extracted by an encoder network. The network is trained with an unsupervised contrastive loss. Extensive experiments with dif- ferent datasets show that our method with self-supervised and semi-supervised settings outperforms state-of-the-at deep learning methods. Furthermore, we show that the learned representations generalize well and can be transferred to new datasets and tasks. The source code of Beff-GenomeNet and all experiments are provided as supplementary material.
INTRODUCTION
In bioinformatics, learning from unlabeled data can reduce development costs in many deep learning applications, since reduces the number of required annotated samples that often require manual experimental validation, such as the functional annotation of genes (Gligorijević chromatin effects of single nucleotide polymorphisms (SNPs) (Zhou and Troyanskayal 2015). Self- supervised learning is amongst the most promising approaches for learning from limited amounts of labeled data and is an attractive approach in bioinformatics due to the availability of large quantities of unlabeled sequence data. contrast to supervised methods, these techniques learn represen- tation of the data without relying on human annotation. Early self-supervised methods focused on solving a pretext task (Devlin et al. 2018) that allows utilizing the data itself to generate labels, and use supervised methods to solve unsupervised problems. The representations learned by per- forming this task can be used as a starting point for downstream supervised tasks such as taxonomic prediction gene annotation. Several different self-supervised representation learning methods were proposed e.g. in the fields of natural language processing (NIP) and computer vision (CV). However, self-supervised learning has not yet seen such widespread adoption in bioinformatics and remains an important and challenging endeavor for the representation of genome data.
al.] 202T) or
Existing methods for representation learning for omics-data tends use methods from NLP CV with slight architecture adjustments and additional preprocessing (e.g. Rives et al. 2021 Lu et al., 2020 Ciortan and Defrance (202T) However, these methods far do not take many biological char- acteristics into account. In particular, the genome sequence embodied by one strand on the DNA molecule always accompanied by another strand going in the opposite direction with comple- (RC)
mentary nucleotides. This means that both the forward and the so-called reverse-complentr sequences are valid DNA sequences and equally correspond to several properties of the genome, such regulatory and taxonomical characteristics. Therefore, machine learning models developed to distinguish different aspects of genomic sequences ideally take the RC into account, for example by being equivariant with respect to it (Mallet and Veri, 2021 Zhou et al., 2021). Our proposed Self-Genomen uses a context network that learns representations that also take advantage of the RC.

---

Under review as a conference paper at ICLR 2022
Inpurt Sequenie
Depressttatioo
Embedding: Prediction
St
fo
On
(xt)
Phodisiional
Cnn 7++ (r 2..2")
(Gp))
Sj
Co
4
Encoder natwork
Contest network
Prediction natwork
Figure 1: Illustration our proposed Seff-GenomeNet. Conceptually, our method splits a sequence S into two subsequences Si, 8j and uses the reverse- everse-complementt 8j of the second subsequence. The subsequences are fed into a representation block composed of the encoder network Se. followed by the context network Co to produce the embeddings 24 Z5. Each of the two embeddings predicts the other one, against embeddings of other sequences the same batch, using an unsupervised contrastive loss.
We propose simple contrastive self-supervised algorithm for learning representations of nucleotide-level genomic sequences (Figure [T]. Our method divides a sequence into two non- intersecting subsequences, where one of them transformed its RC. Then, the contextual rep- resentations of these subsequences are required to distinguish each other from the representations of other, random sequences. We exploit the inherent symmetry in the genome sequence implied by the RC to achieve a natural simplification over possible methods that do not take it into account, all while encoding RC-awareness our architecture. Moreover, we train our network with varying- length subsequences both for the input and the target subsequence. Because, genome sequences can have semantic structure at varying ranges of length, representations that respect both local and global structures are preferable. For this, the context network makes use of recurrent neural net- work layer, which makes the computationally efficient training based on all used length scales within single step possible. Our self-supervised method utilizes contrastive loss to maximize the simi- larity between representations of subsequence and its RC while minimizing the similarity between representations derived from unrelated sequences.
Contributions:
We present the first self-supervised framework tailored specifically to DNA se- quence data. Our main findings and contributions duce a contrastive self-supervised method that is specialized for nucleotide-level genomics data. (2) We propose context-aware network that makes use of inherent symmetries in genomics data in the form of the reverse-come comeleecommlement and improves the quality of learning representation. (3) Self- GenomeNet is evaluated in self- self-supe and cumi--seervieed settings on bioinformatic bench- marks and datasets and achieved superior performance in several tasks. (4) We show the learned representations generalize well when transferred to a new dataset and task.
be summarized follows: (1) We intro-
2 RELATED WORK AND BACKGROUND
Self-supervised learning Deep learning models based on self-supervised learning have become standard method of choice in natural language processing (NLP). Powerful models such as GPT- 3 (Brown et al. 2020) and BERT (Devlin al. 2018 are pre-training in self-supervised manner on large text corpus. They are then often fine-tuned for various tasks using smaller task-specific datasets, where they show performance that is superior to purely supervised methods. Recently, self- supervised visual representation learning (Chen et al., (020) outperformed supervised methods and achieved state-of-the-ar performance on ImageNet (Deng et al. 2009). Self-supervised contrastive learning is one of the most successful methods where it maximizes the agreement of learned repre- sentations between differently augmented views of the same example (Chen et al. 2020 Oord al.) 2018). Motivated by this, propose novel contrastive seff-supervised model similar (Oord et al. 2018) but differing in network architecture, representation learning strategy, and with applica- tions specifically in bioinformatics.
Self-supervised learning in bioinformatics Existing self-supervised learning techniques applied on sequential RNA and protein data adopted from other application fields of deep learning. (Rives)
2

---

Under review as a conference paper at ICLR 2022
et al. 2021) adopted BERT (Devlin et al. 2018 pre-trainee model for predicting masked amino acids and achieved superior performance on sequence prediction task. Contrastive-se (Ciortan and Defrance 202T is method adapted from computer vision for cell clustering based single-cell RNA sequencing data. Contrastive-se creates two copies of sequences with randomly masked nu- cleotides and trained the network using a contrastive loss. As downstream task, self-supervised embeddings are used for clustering with K-Means algorithm (Hartigan and Wong 1979). All these methods for self-supervision described so far have common that they rely on randomly mask- ing part of their input to create pretext task. This differs from Contrastive Predictive Coding (CPC) (Oord et al.] 2018), which splits the input into parts such as image patches or subsequences, and predicts the representation of some of the parts, contrastively, from the representation of other parts. CPCProt (Lu et al. 2020) adapted the CPC model for protein data. Similarl,,,,, eeff-eenoeeeet is trained by the contrastive loss in which the positive pairs include the input sequence and a reverse- complement of the sequence. Unlike previous works, our network is trained with varying-lenghh -ength targets to explore both local and global structures of sequences.
Deep learning methods for genome sequences In genomics, many deep learning architectures use convolutional layers to capture local patterns in data, whereas recurrent layers learn long-range dependencies in the sequences (Eraslan et al.] 2019). Zhou and Troyanskaya (2015) created the DeepSEA dataset for predicting chromatin effects of sequence alterations, and used it train model consisting of multiple convolutional layers. Later, DanQ (Quang and Xiel 2016) improved the deep learning framework of DeepSEA using a bidirectional LSTM (Hochreiter and Schmidhu- ber 1997) unit on top of a convolutional layer. iDeepS (Pan et al. 2018), DeepBind (Alipanahi et al.,
2015), and DeeperBind (Hassanzadeh and Wang, (2016) use a sequence CNNs and RNNs predict RNA binding proteins sites. DeepDNA (Wang et al. 2019) uses an LSTM layer on top of a CNN layer to compress human mitochondrial genome data. Instead of recurrent layers, it is also possible to use dilated convolutional layers to model long-range dependencies in genome data, such as done Basenji (Kelley et al. 2018), which is used to predict cell-type-specine epigenetic and transcrip- tional profiles in large mammalian genomes. Trabelsi al. (2019) performed a comprchensive study on network architecture where the ECBLSTM network, composed of an embedding layer. CNNs, and LSTM layer outperformed other networks on several genomic tasks. Here, also used the DeepSEA dataset to compared the performance of our method with several other self-supervision techniques. The architecture of Self Gelf-Genomeeet requires recurrent layer for it create repre- sentations of varying-length subsequences, and is therefore based off the DanQ-architettue (Quang and Xie 2016), which has shown good performance on the DeepSEA task.
Reverse Complement DNA molecules in living organisms always occur as a double strand, where both strands have opposite directions and consist of nucleotides that are complementary to each other. The sequence on one strand is therefore accompanied by its reverse complement (RC) on the other: The sequence in reverse, and with the A, C, G. nucleotides replaced by T, G, C, A respec- tively. Since both forward and reverse-com creese-commlementt sequences are found in the structure, they are equally responsible for several properties of the genome, such as regulatory and taxonomical char- acteristics. However, next-generation sequencing techniques (NGS) output sub-sequences randomly drawn from both strands, since it usually involves breaking the genome into a collection of small DNA fragments that sequenced individually. Therefore, both the original sequence data and the RC of this sequence are valid inputs to the machine learning models developed to work on NGS datasets (Shrikumar al., 2017 Zhou al. [021]. In this paper, we exploit this genome-specific feature in contrastive self-supervised learning framework helps to improve the learning representa- tion in several genomic tasks.
3 METHOD
We first describe the motivation of our proposed method before explaining its details in Section 3.2
3.1
MOTIVATION
Most successful self-supervised learning approaches are built for computer vision or NLP and are therefore not optimal for bioinformatics data. For example, the self-supervision task of CPC (Oord et al. 2018) generative language models (Dai and Le 2015) are based on predicting the repre- sentation of a short, constant length subsequence or a single token, which could be an N-gram or a single nucleotide. We hypothesize single-length short sequence targets hinder the model from learn-
3

---

Under review as a conference paper at ICLR 2022
Prediction Network
9,
Prediction Network
Embeddlegs
7
Z,
Z
Z
Z
Z
Enbcddiigg
Cortodt Networt
Co
Co
C+
C
C
C+
ConTiNt
Entoder Networt
fe
fo
fe
fe
fo
fe
S
Pis) AICGAIAA
Reverse Complement Best laput Sequenic
IAGOGCI
AIAATIAG
S
S
S
p(1) AOGGGICA ICA GICACGIG OGTGCIAC
e(6) TTATCGAT
Sequeece
AGCSCTAA
Input Sequence
Ixput Sequence
A ACGOTTAATTCTTTAACCCCTTTTCCCCCCCC
Ieput Sequance
Figure 2: Implementation of Self- NetfmGenoomNet. Multiple representations z; of subsequences of dif- ferent lengths are created in a single step using the sequence-output of the recurrent context net- work C.(+) which follows the encoder network fo(.). The same process happens to the reverse- complement of the given sequence, creating Z;- Different prediction layers Jm are used to predict representations that different number of patches apart.
ing global semantics effectively. We therefore propose training representations that are based on target subsequences whose lengths are varying. While predicting future observations of constant se- quence length from input sequences which are changing in length is common for methods like CPC and single character models, predicting representations of varying length sequences is a novelty in this area. Similar to Oord et al. (2018), we use recurrent context network on top of an encoding network that encodes to constant-size patches. However, we use the output the context network to predict the output of the same context network applied to other patches, opposed to predicting only the output of the encoding network. The fact that the RC of an input sequence is a valid genome sequence well simplifies this approach: instead of having fit both one forward and reverse context network that predict each other, only single context-network is trained that both applied the forward as well to the RC sequence.
3.2 SELF-GENOMENET
The Geff-GenomeNet architecture learns representations of varying-length genome sequences by encoding them in both convolutional and recurrent neural network layers, which are trained using a contrastive loss. The architecture enforces the symmetry implied by the RC while also making training more efficient. As depicted in Fig. our proposed method initially creates two subsequen- ces from an input sequence. The subsequences are taken by representation network composed of the encoder network (fe) and the context network (C&). Later, on top of the embedding repre- sentation, the linear prediction layer (gm) estimates the embedding of the other subsequence using contrastive loss against random other subsequences. More specifically, given
unlabeled genome sequence of nucleotides SI.N = [81, 82. .8.. and its RC SN = [aN, w..,....... T where E [A, G, T} and उ7 gives the complementary nucleotide, e.g. A = T. We aim to pro- duce an embedding of subsequence S1.; that contrastively predicts the embedding of the RC of the remaining subsequence, N for some l' >> L. Predicting 3 N i.e. the RC of Sr.N, instead of just the reverse Sp.N. has the benefit that it is possible to use the same embedding and context network for both the first and the second subsequence, as S is a valid genome sequence, whereas SN:e' is not. We use recurrent layer build the representations of both and S, so representations of different length subsequences are calculated at once by using the hidden state of the recurrent unit at different points the sequence. This makes it possible to contrastively predict subsequences of varying lengths against each other efficiently in a single training step. This has advantage over using two forward sequences SI.t and SpEN, where using different starting point values t' in one step would be much more computationally demanding, since the values of recurrent units would have to be calculated fresh for every t'. The entire procedure symmetric since our model learns sequence and the RC of sequence. This symmetry provides the possibility
context network well
4

---

Under review as a conference paper at ICLR 2022
as the prediction networks with shared weights both when reading the subsequence in forward di- rection as in the left subsequence in Fig. 2 as well as when reading the reverse-complent of the sequence the right. This is visualized Fig. El in Appendix. (i) The same recurrent context net- work is used for predicting and target subsequences, (ii) the target subsequence can also predict the predicting subsequence, (iii) the same prediction network(s) used for predictions of the target sub- sequences from the predicting subsequences can also be used for the predictions of the predicting subsequences from the target subsequences. In other words, with the help of the symmetry induced by RC, substantial weight sharing is provided.
To produce the representation, the sequence is first divided into P overlapping patches Spor with p(j) (j a + 1):(j- 1), where I is the patch-length and the patch stride length, and where a < / for overlapping patches. Patches are first encoded with a convolutional neural network fa()). The resulting sequence of vectors [So(S)())), fo(p())) fo(Sp(P-)))) is fed into a recurrent context network Co(+), giving rise to embeddings 2 = Co ((Jo(Sp(u)) Patches of the RC, Spg) = [sj.ath.... sy.++]] are equally fed into first fe then Co. giving rise to Z, = (()n z, is then representation of (1:((-1))++)) likewise, z represents 3 (:(ia+1))
for S P.
Training the encoder and context networks consists of predicting Zi+m from Zg, with m > 0. con- trastively against corresponding embeddings from other, negative example sequences 5(k)-, i.c. against (//) (note: m is the stride length between the patches). For this, a linear prediction layer Qm is used, and the Noise Contrastive Estimation or InfoNCE loss (Gutmann and Hyvärinen] (2010). which maximizes the mutual information shared between forward sequence and its matching RC sequence, is used:
exp 4 Im (zi)) exp mm (zi)) + Σn exp (z(4)m)) qm (m(()))
Lim - log
(1)
Negative samples (k)- are efficiently produced by comparing against representations of other quences loaded in the same mini-batch. Note that every sequence in the minibatch produces two
negative examples for other sequences, giving 2(B - 1) negative samples when using minibatch size B: The embedding of the RC of the last (P- - i m) patches starting from path (i + m) i.e., the analog 74+m coming from other sequences and the embedding of the first iP-i-mi m) patches of the normal of the training method with respect to the RC. Embeddings are, however, always only contrasted with embeddings of sequences that have the same length as the positive example. This is to prevent the network from learning to encode the represented length directly to gain an advantage, which would not be an intrinsically interesting feature for downstream tasks.
so the analog to p--i- This is because of the invariance
One loss term is introduced for each index i, denoting the number of patches represented by z;- Furthermore, because it is possible to skip different number of patches between the represen- tations being matched, one can sum over multiple values of m, each with its individual weights for predictor Qm- Because of the symmetry of the setup, the model is both predicting Zi+m from z; as described above, as well as z; from Zi+m The corresponding loss Li,.. then uses negative examples with the same length as z, The ultimate loss for each individual sequence S is therefore:
L= -= (Lim + Lim))
(2)
was proven by (Oord et al. 2018), optimizing a contrastive loss by predicting future patches Sp maximizes the lower bound of mutual information between the patches and the embeddings. The theoretical proof regarding how we maximize the lower bound of mutual information between the embeddings 2/ and varying-length reverse-complement targets { for several values of j is mnaight-forward by exchanging (3 (ppm)) huzj and (p) (they use &&+t as notation) in the equations of the proof derived by Oord al. (2018). By adopting the equations accordingly, we derive,
((5po)) 21) 2 log(N) T.
(3)
5

---

Under review as a conference paper at ICLR 2022
where [(15ptu) p(u) hugs, 24) is the mutual information between the learned representation 2; and the reverse-oomplemen of the consecutive sequence (Sp(w))ozj- N is the number of random samples in the contrastive pretraining and L is the loss function given in the 2q Note that in our method, differently from CPC (Oord et al. 2018), as the value of changes, the number patches in the set and therefore the total length of the targeted patches vary. By doing this, we maximize the mutual information between varying-length targets and the learned representations.
4 EXPERIMENT SETUP
Datasets and Tasks We evaluate our method with the following datasets:
(1) DeepSEA dataset: introduced by Zhou and Troyanskaya (2013). contains around 5 million hu- man genome sequences, where each data sample contains 1000 nucleotides as input and a label vector for 919 binary chromatin features. We evaluated our model with the DeepSEA dataset be- cause it is a dataset often used by genomic deep learning methods for evaluation (e.g. Zhou and Troyanskaya 2015 Quang and Xie, 2016 Kelley et al. (2018). It represents the task of identifying the class of certain regions within a genome sequence.
(2) Virus dataset: We downloaded all publicly available viral genomes from GenBank (Sayers et al., 202T]. We allocated the FASTA files into training. validation, and test sets by approximate ratios of 70%, 20%, and 10%. We divided the dataset into two taxonomical classes task: bacteriophages vs. viruses that are not bacteriophages, based on provided annotations. The bacteriophage class contained approximately 1.0 billion nucleotides, the non-phage virus dataset approximately 0.5 billion nucleotides. The Virus dataset is a machine learning task that is comple- mentary to it: instead of regions of a genome, the class of the whole sequence should be determined, based on a given subsequence of fixed length. Species classification tasks often arise in metage- nomics, where NGS is used to analyse DNA found in environmental samples (Pust and Tümmler 2021]. NGS typically produces 150nt samples (Quail et al. 2012), therefore a sequence length of 150nt is used with this dataset, which is drawn from the sequences at random. The length of 150nt would therefore allow direct application of our model in metagenomics.
classification
(3) Bacteria dataset: We downloaded all publicly available bacteria genomes from GenBank, com- prising approximately 83 billion nucleotides, similarly to how we collected the Virus dataset. This dataset was only used for unsupervised pre-training for transfer learning, so no supervised task was defined. The sequence length was set to 1000 nt, to compare the capabilities of the methods for different sequence lengths.
(4) T6SS effector protein dataset: We performed experiments on 76SS dataset that publicly avail- able bacteria dataset (SecReT6, Li et al. 2015) to demonstrate that our method works well for a real-world dataset with actual label scarcity. Here the task is the identification of effector proteins. We used T6SS effector proteins as the positive samples to identify, whereas T6SS immunity pro- teins, T6SS regulators, and T6SS accessory proteins are negative samples. We divided the training, validation, and test set by approximate ratios of 60%, 20%, and 20%. The sequence length is 1000nt in all experiments. We provide more details for pre-processing and motivation of the datasets in Appendix B
Network architecture As mentioned in Section 3 Self-GenomeNee composed of representation block followed by an embedding layer and a prediction network. The representation block includes encoder network fo(-) and context network C.(-). For the self-supervised training on the Virus dataset, the encoder network is a convolution layer with 1024 filters, kernel size of 24, and a stride of 6. Only a single CNN-layer is used, so the kernel parameters directly correspond to patch parameters described above, i.e. = 24 and 6. The context network is an LSTM layer with hidden layer size of 512. For the DeepSEA benchmark, we followed the DanQ network architecture by Quang and Xie (2016). The encoder network has an additional max-pooling layer followed by dropout layer.
For each value of m, different fully connected layer is used the prediction layer 4m In any self- supervised pretraining, the default choice is to have only one prediction network 91 due to longer pretraining time when multiple prediction layers are used. Therefore the default is estimating the neighbor subsequence of the predicting subsequence without any skipped nucleotide between them. However, additionally test pretraining with multiple prediction networks and therefore estimate

---

Under review as a conference paper at ICLR 2022
even further subsequences for the DeepSEA dataset. The reported results are the best performance of Self- Net.meeno.eet. We provide more information in Appendix A
Optimization We used Adam optimizer Kingmma and Ba 2014 with 31 0.9, 32 0.999 and learning rate of 0.0001 for all experiments except the DeepSEA dataset that we used the RMSprop optimizer. The dropout rate of 0.2 and 0.5 are used respectively for the dropout layers as in the DanQ model QQuang and Xie 2016). For all datasets, we used the default Keras weight initialization, which the Glorot uniform initialization (Glorot and Bengio 2010). During the self-supervised pre-training, we used the largest fitting minibatch size (power of 2) that worked in a Geforce RTX 2080 Ti, which was 512 for the virus dataset and 128 for the Bacteria and DeepSEA datasets.
For the baselines experiments, we used the same hyperparameter settings for all our experiments such Adam hyperparameters or learning rate. When not applicable, we followed the recommen- dations in the papers of the baseline methods regarding the hyperparameter selection. We provided more detail on optimization in Appendix B
Evaluation For all experiments, we report the performance in terms of F1 score. However, We use Receiver Operating Characteristics (ROC) Area Under Curve (AUC) and Precision-Recnt (PR) AUC for DeepSEA dataset for all 919 binary outputs, as these metrics are used by Quang and Xie (2016). In this paper, it was also shown that the sparsity of positive binary targets(~> 2%) in the dataset inflated the ROC AUC, and that PR AUC is a better indicator of performance since it does not take the number of true negatives into account. We macro-avcraged recall Recally (Sokolova) and Lapalme 2009)), which is equivalent to balanced accuracy, and F1 score for dataset evaluations on Virus.
5 RESULTS
We compare our proposed Colf-GenomeNet with CPC (Oord et al., 2018 as the CPC is the clos- est self-supervision method to our method which can also be applicd in different application fields of deep learning. Moreover, provide comparison with Contrastive-sc (Ciortan and Defrance 2021) and generative language model since these methods originally proposed for computer vision or natural language processing respectively but applied in bioinformatics several times. Both single nucleotides, denoted as (single nt), and 3-gram targets are predicted for pretraining using a language model (See Fig. 4.. We follow the standard evaluation protocol for self-supervised learning and eval- uate the unsupervised learned representation with a linear classification and semi- cumi-supervised tasks, as well as using transfer learning to different datasets and different tasks (Henaff) 2020).
Linear Evaluation The common evaluation protocol for self-supervised learning is based on freez- ing the representation networks (both base-encoder and context network) after unsupervised pre- training and then train a supervised linear classifier on top of them. The linear classifier is fully connected layer followed by softmax-classificatee
using cross-entropy loss, which plugged on top of the context network after removing the prediction layer. Table [ shows the comparison of our model against the baselines under the linear evaluation on Virus and DeepSEA dataset.
Dataset
DeepSEA
Virus
Metric
ROC AUC
PR AUC
Recall
Fi
CPC
0.734
0.094
58.7
0.607
Language Model (single nt) Language Model (3-gram)
0.699
0.069
58.8
0.598
61.8
0.626
Contrastive-sc
0.682
0.056
54.9
0.462
Getf-GenomeNet
0.757
0.120
69.1
0.709
Table 1: Mean values of ROC AUC and PR AUC metrics for the 919 outputs defined for the DeepSEA dataset and Macro-averaged recall (in %) and F, for Virus under linear evaluation.
Semi-supervised Learning We evaluate the performance of our models on a semi-supervised learn- ing task. In this task, we pretrain our model on unlabeled datasets and fine-tune a classification or prediction model using a subset of datasets with labels. Differently from the linear evaluation protocol, the pretrained base network is also fine-tuned. We follow the semi-supervised protocol of (Henaff) 2020)) and use the same fixed splits of respectively 1 1% and 10% of labeled training data in DeepSHA and Virus datasets.
7

---

Under review as a conference paper at ICLR 2022
Dataset
DeepSEA
Virus
Labeled
1%
10%
1%
10%
Metrie
ROC
PR
ROC
PR
Recollar
Fi
Recoll
Fi
Supervised
0.776
0.119
0.865
0.239
69.3
0.694
72.4
0.732
CPC
0.780
0.125
0.863
0.236
69.4
0,700
74.4
0.753
Language Model (single nt)
0.774
0.116
0.859
0.232
71.5
0.727
75.5
0.760
Language Model (3-gram)
72.0
0.732
76.3
0.766
Contrastive-sc
0.764
0.105
0.848
0.222
70.3
0.706
73.2
0.742
Self-GenomeNet
0.795
0.138
0.865
0.235
75.3
0.751
78.2
0.785
Table 2: Performance of the self-supervision techniques under semi-supervised learning protocol using 1% and 10% the labels DeepSEA and Virus datasets. Supervised training performance without any pretraining also reported as baseline. Reported metrics are in line with Table 1
Transfer to Other Tasks We further assess the generalization capacity of the learned representa- tion on viral detection by pre-training Self-GenomeNet the Bacteria datasets. Our motivation for transfer learning is to take ad advantage of abundant bacterial data, which are much better described and studied than their viral counterparts, and optimize the model for tasks on viruses(such as viral identi- fication or taxonomic classifications). We also used the 76SS dataset to demonstrate that our method works well for real-world dataset with actual label scarcity. To this end, the Seff-GenomeNer is trained in self-supervised fashion on the bacteria dataset and then we train in supervised manner us- ing the phage and non-phage labels on the virus dataset and evaluate the performance on this dataset. Table 3 provides comparison of transfer learning performance of our self-supervised approach for the task of classification of viruses.
Dataset
Virus
T6SS
Base Network
Fine-tuned
Fixed
Fine-tuned
Fixed
Metric
Recalls
Fi
Recally
Fi
Recally
Fi
Recolla
Pi
Supervised
92.0
0.938
58.2
0.624
CPC
95.5
0.966
76.1
0.812
81.8
0,855
72.3
0.725
LM (single nt)
92.6
0.944
67.9
0.735
79.1
0.825
72.5
0.752
Contrastive-sc
93.4
0.950
56.4
0.578
74.3
0.775
62.0
0.627
Self-GenomeNeet
96.7
0.974
82.1
0.861
83.4
0.857
79.3
0.796
Table 3: The achieved performance terms of Macro-averaged recall (in %) and Fi score under transfer learning evaluation on virus dataset (Column 2-5) and T6SS (Column 6-9). We evaluated the results when base networks are fine-tuned and fixed. Here, the base network is trained in self- supervised way on the big bacteria dataset. Supervised training performance without any pretraining is also reported baseline.
ABLATION STUDIES AND DISCUSSIONS
We present multiple ablation studies on Self-CienomeNet to give an intuition of its behavior and performance: (I) the impact of reverse-complement targets against other potential targets; (2) pre- dicting varying-length targets against single-length targets; (3) robustness of our algorithm in the low-data regime. All the ablation study is performed on the virus dataset under linear evaluation setting.
Learning from the reverse complement of sequences Self- Gelf-GenomeNet uses the embedding rep- resentation of a context network for downstream tasks which learns by predicting the reverse com- plement of neighbor subsequence SN. as a target for its predictions. We investigated the effect of using target embeddings derived by RC of neighbor subsequences against two other potential ways of creating embeddings derived from varying-length targets. To this end, besides our method, we performed two other experiments where the recurrent context network reads the "target" subsequen- ces in different ways: in forward and reverse directions. In forward setting, the context network reads the target subsequence St:N in the same direction whereas reverse direction refers to reading the target sequence other way around, as Syze. Table compares the performance of our trained network in these ways.
reads the predicting subsequence S145
8

---

Under review as a conference paper at ICLR 2022
Forward
Reverse
Reverse- Reverse-Commplement
0. 0.7048+00.022 0.6952=00.003
0.7089 0.0008
Table 4: The achieved performance in terms of F1 score for virus dataset under linear evaluation setting with fixed pre-trained weights.
Predicting varying-length targets We investigate the effect of having varying-length target sub- sequences by comparing method against the constant-length target subsequences. As mentioned, having non-varying length targets is indeed a common feature for self-supervision techniques such as CPC language models. Nevertheless, we would like investigate the effect of varying-length targets, when other features of Self-GenomeNet is unchanged. To create nonvarying-lenge target subsequences we divide the sequences into two nearly equal-in-sizz subsequences. Specifically, we split the sequences of 150 into 72 and 78, so that subsequences with length 72 and 78 predicts the embeddings derived from the reverse-complement of each other. In the Self-GienomeNet however, for the given architecture, subsequences with length E {24, 30, 36, 126} predicts subsequence with length 150 x. 72 and 78 was chosen because they are the two closest values that can inves- tigated using our previous architecture without any padding (splitting into 2 subsequence with 75 is not possible due to stride value of 6). We observed for linear evaluation setting on the virus dataset the performance dropped from 69.1% to 64.9% (macro-averagee averaged recall), when only same-size tar- gets are predicted. A straight-forwant forward explanation to this is that maximizing the mutual information between the learned embeddings and parying-length target subsequences (both short and long) helps the machine learning model to learn more comprehensive range of semantics and therefore boosts the performance in the downstream supervised tasks.
Data efficient We study and compare the performance of Self-GenomeNet with other methods in the low-data regime. We created label-scareity artificially on the Virus and DeepSEA datasets by subsetting the number of labeled training samples down to 0.1%. We train the models shown in Table 5 using 0.1% of the dataset after pretraining on the whole dataset in superviue way. The reported results are when the base networks are fine-tuned. Based on the results reported in Table our method outperforms other methods by substantial margin in the low-data regime.
Dataset
DeepSEA
Virus
Metric
ROC AUC
PR AUC
Recalla
Supervised
0.715
0.074
56.4
0.505
CPC
0.732
0.089
59.2
0.621
Language Model (single nt)
0.729
0.081
61.8
0.645
Language Model (3-gram)
62.9
0.634
Contrastive-sc
0.726
0.073
61.8
0.623
Self-GenomeNet
0.753
0.115
67.2
0.700
Table 5: Comparison the methods in performance when they are pretrained using the whole dataset in self-supervised manner, followed by supervised training 0.1 1% of the dataset. Super- vised baseline is not pretrained.
CONCLUSION
In this paper. we presented Self- GetfmGenomeet, a framework for contrastive learning of nucleotide- level genomic data. Belf-GenomeNer composed of an encoding network followed by context network and uses a prediction layer for training. The network extracts compact latent representations by predicting representations of the reverse complement of future observations. Gelf-GenomeNet is optimized end-to-end using unsupervised contrastive loss. We tested the learned representations by linear evaluation, semi-supervised learning, and transfer to other tasks and achieved better perfor- mance on the four public bioinformatics benchmarks compared to other self supervueevii methods. Moreover, we showed the ability of our method for training in the low-data regime, tested the ef- fect of predicting varying-length target subsequences, and the effect of estimating the output of the context network which reads the predicted subsequence in forward or reverse direction compared to reading reverse-complement of the subsequences.
Reproducibility Statement: The source code is available in the supplementary material.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Babak Alipanahi, Andrew Delong, Matthew T Weirauch, and Brendan J Frey. Predicting the se- quence specificities of dna-and rna-binding proteins by deep learning. Nature biotechnology. 33 (8):831-838) 2015.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14163 2020.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pages 1597-1607. PMLR, 2020.
Madalina Ciortan and Matthicu Defrance. Contrastive self-supervised clustering of serna-seq data. BMC bioinformatics, 22(1))1-27, 2021.
Andrew M Dai and Quoc V Le. Semi- somersupee sequence learning. Advances in neural information processing systems, 28:3079-3033 2015.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pages 248-255. Iece, 2009.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 04805, 2018.
Gökcen Eraslan, Ziga Avsec, Julien Gagneur, and Fabian J Theis. Deep learning: computational modelling techniques for genomics. Nature Reviews Genetics, 20(7):389-403, 2019.
Vladimir Gligorijević, P Douglas Renfrew, Tomasz Kosciolek, Julia Koehler Leman, Daniel Beren- berg, Tommi Vatanen, Chris Chandler, Bryn C Taylo,, Ian M Fisk, Hera Vlamakis, et al. Structure- based protein function prediction using graph convolutional networks. Nature communications, 12(1):1-14, 2021.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249-256. JMLR Workshop and Conference Proceedings, 2010.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: new estimation principle for unnormalized statistical models. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 297-304. JMLR Workshop and Conference Proceed- ings, 2010.
John Hartigan and Manchek A Wong. Algorithm 136: k-means clustering algorithm. Journal of the royal statistical society. series (applied statistics), 28(1):100-100 1979.
Hamid Reza Hassanzadeh and May D Wang. Deeperbind: Enhaneing prediction of sequence speci- ficities of dna binding proteins. In 2016 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pages 178-183. IEEE, 2016.
Olivier Henaff. Data-efficient image recognition with contrastive predictive coding. In International Conference Machine Learning, pages 4182-4192. PMLR, 2020.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8):
1735-1780, 1997.
David R Kelley, Yakir A Reshef, Maxwell Bileschi, David Belanger, Cory Y McLean, and Jasper Snock. Sequential regulatory activity prediction across chromosomes with convolutional neural networks. Genome research, 28(5):73--750, 2018.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.60 14/2.6980, 2014.
10

---

Under review as a conference paper at ICLR 2022
Jun Li. Yufeng Yao, H Howard Xu, Limin Hao, Zixin Deng, Kumar Rajakumar, and Hong- Yu Ou. Secret6: web-based resource for type vi secretion systems found in bacteria. Environmental microbiology, 7777:2196-22000 2015.
Amy X Lu, Haoran Zhang, Marzyeh Ghassemi, and Alan M Moses. Self-supervised contrastive learning of protein representations by mutual information maximization. BioRxiv, 2020.
Vincent Mallet and Jean-Philippe Vert. Reverse-complement equivariant networks for dna se- quences. bioRxiv, 2021.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint 1rXiv:1807.03345. 2018.
Xiaoyong Pan, Peter Rijnbeek, Junchi Yan, and Hong-Bin Shen. Prediction of rna-protein sequence and structure binding preferences using deep convolutional and recurrent neural networks. BMC genomics, 19(1):1-11, 2018.
Marie-Madlen Pust and Burkhard Tümmler. Identification core and rare species in metagenome samples based on shotgun metagenomic sequencing, fourier transforms and spectral comparisons. ISME Communications, 1(1):1-4, 2021.
Michael A Quail, Miriam Smith, Paul Coupland, Thomas Otto, Simon R Harris, Thomas R Con- nor, Anna Bertoni, Harold P Swerdlow, and Yong Gu. A tale of three next generation sequencing platforms: comparison of ion torrent, pacific biosciences and illumina miseq sequencers. BMC genomics, 13(1):1-13, 2012.
Daniel Quang and Xiaohui Xie. Danq: a hybrid convolutional and recurrent deep neural network for quantifying the function of dna sequences. Nucleic acids research, 44(1 1):e107-e100 e077 2016.
Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ou, Lawrence Zitnick, Jerry Ma, et al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proceedings of the National Academy of Sciences, 118(15), 2021.
Eric W Sayers, Mark Cavanaugh, Karen Clark, Kim D Pruitt, Conrad L Schoch, Stephen T Sherry, and llene Karsch-Mizrachi. Genbank. Nucleic acids research, 49(D1):D92-D96, 2021.
Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Reverse-complement parameter sharing improves deep learning models for genomics. bioRxiv, page 103663, 2017.
Marina Sokolova and Guy Lapalme. A systematic analysis of performance measures for classifica- tion tasks. Information processing & managemens, 45(4):427-437, 2009.
Ameni Trabelsi, Mohamed Chaabane, and Asa Ben-Hur. Comprehensive evaluation of deep learning architectures for prediction of dna/rna sequence binding specificities. Bioinformatics, 35(14): 1269-i277, 2019.
Rongjie Wang, Tianyi Zang, and Yadong Wang. Human mitochondrial genome compression using machine learning techniques. Human genomics, 13(1):1-8, 2019.
Hannah Zhou, Avanti Shrikumar, and Anshul Kundaje. Towards better understanding of reverse- complement equivariance for deep learning models in regulatory genomics. bioRxiv, pages 2020- 11, 2021.
Jian Zhou and Olga G Troyanskaya. Predicting effects of noncoding variants with deep learning- based sequence model. Nature methods, (1100)93119344 2015.
11

---

Under review as a conference paper at ICLR 2022
APPENDIX A PREDICTION NETWORK
Single prediction network multiple prediction networks: For the pretraining in DeepSEA dataset, we trained the network with only one prediction network (4m where m = 1) as well as with multiple prediction networks m {1, 4, 7, 10, 13, 16} for DeepSEA. As mentioned in the manuscript, multiple prediction networks are used when further subsequences are predicted, in addition the closest subsequence the predicting subsequence. We report the results of Self- GenomeNet when only one single network is used in the tables
g
Only single prediction
network is used during the pretraining stage of the other datasets.
Dataset
DeepSEA
Metric
ROC AUC
PR AUC
CPC
0.734
0.094
Language Model
0.699
0.069
Contrastive-sc
0.682
0.056
Self- Gelf-GenomeNet- {1}
0.757
0.116
Self-GenomeNet-- m [1, 13, 16)
0.757
0.120
Table 6: Mean values of ROC AUC and PR AUC metrics for the 919 outputs defined for the DeepSEA dataset under linear evaluation protocol.
Dataset
DeepSEA
Labeled
1%
10%
Metric
ROC AUC
PR AUC
ROC
PR AUC
Supervised
0.776
0.119
0.865
0.239
CPC
0.780
0.125
0.863
0.236
Language Model
0.774
0.116
0.859
0.232
Contrastive-sc
0.764
0.105
0.848
0.222
Self GenomeNet m (1)
0.795
0.135
0.862
0.234
Self Ger-meeooeee--te m {1. 7, 10, 16)
0.795
0.138
0.865
0.235
Table 7: Performance of the self-supervision techniques under semi-supervised learning protocol using 1 1% and 10% of the labels in DeepSEA dataset. Supervised baseline without any pretraining is also reported baseline. Reported metrics are in line with Table T
Dataset
DeepSEA
Metrie
ROC AUC
PR AUC
Supervised
0.715
0.074
CPC
0.732
0.089
Language Model
0.729
0.081
Contrastive-sc
0.726
0.073
Self- Self-GenomeNet-ne m (1)
0.750
0.103
Self-GeeomeNett- m E [1,4, 1,, 16)
0.753
0.115
Table 8: Comparison of the methods in performance when they pretrained using the whole dataset in self-supervised manner, followed by supervised training on 0.1 % of the dataset. Super- vised baseline is pretrained.
Here we also report the pretraining time in hours in Table 9 for method and the baselines on DeepSEA dataset using a GeForce RTX 2080 Ti. One can immediately see, the pretraining time for the simplest version of Self-GenomeNet with prediction network (m 1) is shorter than or comparable to the pretraining time the other self-supervision methods, whereas the performance superior.
12

---

Under review as a conference paper at ICLR 2022
Dataset
DeepSEA
Method
Pretraining Time (hours)
Supervised
0
CPC
73
Language Model
30
Contrastive-se
14
Self-GenomeNet m E {1}
25
Self-GenomeNet-.. E {1,4, 10, 13, 16)
158
Table 9: Comparison of the methods performance when they are pretrained using the whole dataset in self-suuervised manner, followed by supervised training on 0.1% of the dataset. Super- vised baseline not pretrained.
APPENDIX B PROCESSING OF DATASETS AND FILES
For virus-bacteria dataset, data samples are created by reading consecutive samples taken from ran- dom regions of FASTA files. Compared to division of FASTA files into same-length non-intersecting sequences, this technique increases number samples that can be created out of FASTA files. In order provide randomness, either maximum 90% and 10% of the file is processed for virus files and bacteria files respectively, or maximum 64 data samples are created from file, before next FASTA file is processed.
For phage/non-phaeee classification on virus dataset, we set the sequence length to 150. The mo- tivation limit the imput length to 150 in some of our evaluations to be close potential application, since this is typical read length that gets produced during Next Generation Sequencing which is frequently used. The length of 150 nt, therefore, allows direct application of the sequenc- ing output to perform model predictions that can be built using pre-trained models as presented in our work. We also evaluate some experiments with a sequence length of 1,000 nt, because this is a meaningful length of the output of genome assemblers, where these 150 nt long reads are assembled together using overlap information or mapping to reference sequences to produce longer fragments.
The short sequence length be justified to build neural networks directly the read output of sequencer (150 nt), rather than on the output of the assembling process since the latter comes with several limitations. requires (i) computational resources and often (ii) the presence of reference genomes, which are often absent when an environmental sample is sequenced; furthermore (iii) assembling Softwares perform poorly on repetitive regions of bacterial genomes that are of particular interest to study since these often belong to the bacterial defense system. Therefore, traditional bioinformatics software is taking raw read input rather than preprocessed contigs.
APPENDIX C OPTIMIZATION OF SELF-SUPERVISED METHODS
For all architectures and all our experiments, we used the same encoder and context models for fair comparison. Moreover, we took most hyperparameters as given in the original method.
As depicted in Fig 5 we adopted the Henati (2020) implementation of CPC for 1-dimensional data and use an RNN (LSM)) and single fully connected layer on top of LSTM layer for prediction as Oord al. (018). It is shown by (Oord et 2018) that the performance for the baseline models decreases sigmificanlly when the span of predicted future observations is too short, however decreases slightly when they are too long. To this extend, the predicted span for the patches are selected to be not too short well as not too long compared to the full length of the sequence. For pretraining on the virus dataset, every second feature from 4t 10th derived by encoder network predicted with different linear layers. Thus, the representations derived from the following 60 nu- cleotides are predicted, when the full sequence length is 150. For bacteria pretraining, the following 2nd to 10 features predicted, making the prediction span of 200 nucleotides for the sequence length 1000 nucleotides. For the DeepSEA dataset, every second feature from 3rd to 20th features predicted, making the span up to 260 nucleotides for the sequence length of 1000.
13

---

Under review as a conference paper at ICLR 2022
Prediction Harwwrk
9,
Fabedding:
2
Z
2z
Z
Z
Frsheidings
Connect Network
Co
Co
Co
Co
fiatwork
fa
fa
fe
fe
fe
Encoder Network
$
S
S
p(4j TTAGOGCT
P(5)
B(6) ATOGATAA
of
ATAATTAG
Bast Impur Sequance
S
S
p(1)
#12)
2(4)
6f5)
Input Sequance
ACGGGICA
GICACGIG CGIGCIAL
AGCOCTAA
CTAATTAT
TTATOGAT
Resz Sequence
Irpst Sequence
LOGETCACCTCTAACCCCCTTTTTCCC
Hepurt Sequennce
Figure 3: How is Self- Seefmeennmeeet inherently symmetrical? The context network and the prediction network with the shared weights are used both for predicting the subsequence on the right from left one well as the left subsequence from the the right one.
For Contrastive-sc (Ciortan and Defrance 2021 (Fig. 5). mended by the paper and did use additional linear layer on top of the final outputs of LSTM.
we used the masking rate of 0.9 as recom-
For the language modelling (Fig. 4 we followed different strategy for different datasets. For the virus and bacteria dataset, the model predicts the following 60 and 200 nucleotides respectively, which are taken from outside of the sequence. For DeepSEA, the nucleotides outside the sequence not provided within the dataset. Thus, the last 260 nucleotides are estimated from the first 740 nucleotides during pretraining. Please note that the span of predicted observations is the same for the CPC and the language modelling.
In the supervised training of virus datasets, we firstly train only the final linear layer while the pretrained base model is frozen. Once this layer is optimized, starting from this point, we tune the whole network. For DeepSEA dataset, we directly optimize the whole model for both freezed and unfreezed base-networks.
C.I ADDITIONAL EXPERIMENTAL RESULTS
In this section we further compare our proposed Gelf-GenomeNet with the baseline methods for lin-
evaluation (Fig. (6, semi-supervised learning (Fig. Fig. 9), transfer learning (Fig. TO Fig.
and data efficiency (Fig. 7).
14

---

Under review as a conference paper at ICLR 2022
Prediction Network
2
2
Contest Network
Co
C
C
Encoder
fe
fo
fe
Prediction Network
9,
000
9
4N
S
S
P(O)
p(1)
P(2)
Input Sequence
ACGGGTCA
Input Sequence
ACCGTTAACTCCCAAAACCCATTTTCCCCCC
Inpul Sequience
(a)
Prediction
Embeddings
Z
Z
Context Network
Co
C
C+
fe
fe
fe
Prediction Network
9,
92
0000000
9u
4N
S
S
P(0)
p(1)
p(2)
Input Sequence
ACGGGTCA
GTCACGTG CGTGCTAC
Input Sequence
ACCOTTACCTCCAAAACCCCTTTTACCCCA
Input Sequence
(b)
Figure 4: Illustration of Language Model using 3-gram as targets (on the top), followed by Language Model using single nucleotides as targets (in the bottom); In language modeling. future nucleotides are predicted using cross-entropy loss. In (a) the nucleotides are one-hot encoded as a group of nucleotides into vector of 64, whereas in (b) each nucleotide one-hot encoded seperately vector of 4.
15

---

Under review as a conference paper at ICLR 2022
Prediction Netuoik
brleedings
Z
Z
A
A
Contect Network
Co
C
C
Z,
Zg
Encoders
Friheddilngs
Enooder Network
fe
fe
fe
fe
fe
fe
S
S
S
H(O)
#/21
p(6)
Seqpaence
ACOOGICA GICACGIG C6TGCIAC
AGCGCTAA
CTAATTAT
TIATCGAT
input
A
(a)
tntedding
Z
Entedding
Contest Netwodi
C+
C
C
C
C
C.
Co
fo
fo
fo
fo
fn
fe
fo
Loodde Retwork
S
S
S
Aprommnnt
B(D)
A(1)
of2
p(3)
p44)
A(5)
Randors Musking
COGOSOOC
DOCDO660
OGTOZDAO
034000001
aCIRDIDA
D0300T0O
Kardoon Masidnn
C GGGGGIN A TAT.CAAGGGTTCCCCCCCCCCCCCCCCCCCC C
Ieput Sequence
Ingust Sequencc
Enckeiding
Erekeoldigg
Contect Belsook
Co
C
C
C
C+
C.
Co
Cacode Retwork
fg
fu
fe
fo
fe
fg
E800d21
S plo)
S
S
S
P(1)
#(2)
p(3)
p(t) 000G00AA
AlS)
e65)
Musking
ADOSS0 0D
SDIACOIG COOECCIOO
CCA00000
COLOTOOO
OTOOCOOT
Sandom Masiing
copy boquanoee
XCCOTTAGIIGTACCGGGGTATTAACC
Copy of
(b)
Figure 5: Illustration of CPC (on the top) and Contrastive-sc (in the bottom); CPC predicts the out- put representations obtained from the encoder network for the future patches using the contrastive loss, where the negative samples are representations of other patches. Contrastive-se takes se- quence and copy of sequence and mask the input sequence randomly. Then, the agreement between embedding of masked copies are maximized using the contrastive loss, where negative samples are embeddings of other masked sequences.
16

---

Under review as a conference paper at ICLR 2022
Linear Evaluation Virus Dataset
Macro Averaged Recal on Valtation Set
6.70
Method
Contustiv--oc
CPC
Language gramm)
6-6-
Sett:GenomeNet
2500
5000
7500
10002
10500
Number herations
Figure 6: Linear evaluation
Data Efficiency Training with 0.1% of Data on the Virus Dataset
Fixed Base Network
Fine-tuned Base Network
Macro Averaged Recall Validation Set
0.70
0.70
Method
0.65
0.65
Contrastive-so
Language Model (3-gram)
Language Model (single nt)
0.60
0.60
Self-GenomeNet
Supervised
0.55-
0.55
500
1000
1500
500
750
Number of iterations
Figure 7
17

---

Under review as a conference paper at ICLR 2022
Semi-supervised Learning Training with 1% of Data on the Virus Dataset
Fixed Base Network
Fine-Juned Base Network
Macro Averaged Recall Validation Set
0.75
0.75
Method
Contasive-se
0.70
0.70
GPC
Language Model (3-gram)
Language Model (single ng
0.65
0.65
Supervised
2500
5000
7500
10000
12500
2500
5000
7500
10000
12500
Number of Ittrations
Figure 8: Semi-supervised learning
Training with 10% of Data on the Virus Dataset
Fixed Base Network
Fine -uned Base Network
0.80
0.80
Macro Averaged Recall Validation Set
0.75
0.75
Method
Contrastive-sc
0.70
0.70
Language Model (3 (3-gram)
Language Model (single
Self-GenomeNet
0.65
Supervised
0.60
0.60
2500
5000
7500
10000
12500
2500
5000
7500
10000
12500
Number of iterations
Figure 9
18

---

Under review as a conference paper at ICLR 2022
Transter Leaming Pretraining the Bacteria Dataset, Supervised hing Virus Dataset
Micro Aveeeveeee seeeededd Feed on Valtaton Sel
Method
Langsoge Mudel bingret)
2509
5050
12000
1o0co
.5009
29000
Number tcrations
Figure 10
Trensfer Leaming Pretraining on the Batteria Datasel, Supervised Traring on Ihe Tess Dataser
Eaequags Mentel parple
104
200
373
350
200
Figure 11: Transfer Learning
19


Reviews:
[{"review_id": "tPGY_VtJPN", "reviewer": "Reviewer_ERw2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "-\tWhile the paper has its own merits, my biggest concern is that the connection between motivation and the proposed method is not crystal clear. I agree that pursuing the representation equivariance of two reverse-complementary sequences is an important research direction. However, the proposed method deals with representations of two subsequences. Even if they are originally from a single sequence, RC of one subsequence does not make it reverse-complementary to the other subsequence. I’m not sure why the prediction of S^bar_(N:t’) is more helpful than the prediction of S_(N:t’) or S^bar_(1:N).\n-\tOn page2-line3, the authors stated that “our method divides a sequence into two complementary subsequences..” I don’t think the authors meant that they are composed of complementary nucleotides (A^bar = T), but it might be misleading for some readers. \n-\tOn page3-line10, the authors stated that “positive pairs include the input sequence and a reverse-complement of the sequence.” However, this is not entirely correct. The proposed method used one subsequence and the RC of the other subsequence as the positive pairs. \n-\tOn the page3-last sentence of the background, the authors stated that “exploit this genome-specific feature to tighten the lower bound between mutual information between learned representations.” However, I could not find theoretical or empirical grounds to claim that the proposed method tightens the mutual information lower bound. Please clarify the point.\n-\tIn Sec3.2-line4, the authors stated that the proposed architecture makes training more efficient. Can you explain why it is more efficient? \n-\tPlease provide some missing experiment setups. (1) which value was used for “m” in the proposed loss term. (2) length statistics of the three datasets. (3) Training time and learning curve for the proposed method.\n-\tAccording to the network architecture section, it seems using the DanQ model architecture for the DeepSEA dataset causes some issues that only 3.6% of parameters are trained in the self-supervised stage. Then, why didn’t you use the model architecture used for the Virus dataset? \n-\tTo be self-contained, please provide more algorithmic details for the compared baselines. In addition, can you clarify whether the baselines use the N-gram or single nucleotide for the pre-training? If it is the latter, can you also provide the results for using the N-gram?\n-\tDo you have any specific reasons for using the recall as the evaluation metric? Can you also compare them in terms of the F1 score?\n-\tIn Table7, the authors compared the autoregressive models reading the target subsequences in forward, reverse, and RC directions. What is the difference between the forward model and the CPC model? In my view, the relatively small performance difference between the forward and RC models might indicate that the effectiveness of the proposed method based on the RC-equivariance is not significant.\n-\tWhile the proposed method is largely motivated by the RC equivariance, the paper does not seem to provide sufficient theoretical or empirical analyses concerning the point. (1) Since the RC-equivariance is not guaranteed within the model, can you show how well the learned representations satisfy the RC equivariance? (2) Can you compare its performance with those of previous RC-equivariance techniques on some NGS datasets?\n", "summary_of_the_paper": "Recently several works have proposed semi-supervised learning methods to leverage unlabeled biological sequences for learning their general-purpose representations. In this work, the authors proposed the Self-GenomeNet, a novel contrastive learning method for nucleotides based on the reverse-complement (RC) context prediction. First, given a sequence, they divide it into two subsequences and transform one into its RC. Then, the model is trained to distinguish their representations from those of other random nucleotide sequences. The authors claimed that the proposed method considerably outperforms previous self-supervised baseline models on three benchmark datasets in both self-supervised and semi-supervised evaluation ", "summary_of_the_review": "While the paper has its own merits, it has several issues to be addressed regarding (1) the connection between motivation and the proposed method, (2) theoretical or empirical analyses concerning the RC-equivariance, and (3) some questionable experiment setups. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "mnACuTbydIE", "reviewer": "Reviewer_ifj3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n+ The idea to use self-supervised contrastive learning for genomic sequences is interesting and well-motivated\n+ Proposed changes to the existing model are domain-specific and relevant for the application of the method for genomic tasks\n+ The improvement in classification performance over supervised models is promising, especially for 0.1% and 1% labels cases, is promising. \n+ The results indicate that learning robust representations via contrastive loss-based self-learning of the sequences can lead to better performances for supervised and semi-supervised learning in genomics.\n\nWeaknesses:\n\n- The paper's central claim is that self-learning-based methods can help with genomics tasks with limited data. However, the datasets used in the paper do not reflect those tasks and have an ample amount of labeled information. While the label scarcity is simulated in some results, it would be helpful to demonstrate the applicability of Self-GenomeNet for classification for genomic tasks with limited labels. \n- The rationale for the choice of the datasets, models, and baselines is not entirely clear. For example, there are more recent methods (like Basenji) that perform genome sequence classification. So why was DanQ chosen as the model of choice? \n- Similarly, how was the model architecture for classifying viral genomes selected? Are there existing deep learning models performing this task? \n- Data imbalance is an issue for both Virus and DeepSEA datasets. Why were different evaluation metrics chosen for the classification tasks on the two datasets?\n- It is unclear how the training for the baseline models like CPC was performed? Was sufficient hyperparameter tuning performed for all the models?\n- I am not entirely sure how the model can perform transfer learning from bacteria to virus datasets. Was the model trained using bacteria sequence (or a part of it) re-trained on viral genomes? \n- The ablation analysis needs to be more comprehensive. It is also unclear if the variable sequence length was explored thoroughly. It seems that only 2 settings were tried - 150 and 72/78. \n\nMinor points:\n- The result section could be arranged better for making it easier to follow\n- Typo in Table 6 caption: \"din\"\n- Table 5 results are repeated in Table 8\n- Are the ablation study differences for forward and reverse-compliment strands significant?\n- The related work section for genomic sequence predictions could include more recent works than DeepSEA and DanQ\n- The input length of 150 bp might be limiting the model's performance as studies have shown that longer DNA sequences can better capture the neighboring context of the signal being predicted. \n", "summary_of_the_paper": "This paper presents a self-supervised learning approach using contrastive loss for representation learning of genomic sequences. The contrastive loss has been used for self-supervised learning in the NLP and computer vision domains and the paper presents its application for genomics. Self-supervised contrastive learning tries to maximize the agreement between augmented views of a sample. Therefore, Self-GenomeNet splits a sequence into two subsequences, and the learned representation of subsequence 1 is compared to subsequence 2 and its revers compliment (positive samples) as well as other sequences (negative samples). The described method has two considerations specific to the genomics tasks - (1) it handles variable sequences (2) it incorporates reverse complement information of the sequences. The method is applied on two prediction tasks and one transfer learning task using sequences from viral, bacterial, and human genomes. Its performance is compared to the supervised model, generative language model, and self-supervised learning models - CPC and Contrastive-sc. The results show improved classification performance over the baseline for both supervised retraining and semi-supervised training settings. ", "summary_of_the_review": "Overall the ideas presented in the paper are interesting and potentially useful. However, limited experiments and results fail to support the main claims of the paper. The results could include tasks with label scarcity and some of the choices for experimental settings are unclear. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "mKWWF-qbwSY", "reviewer": "Reviewer_4DkC", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The question of self-supervision for networks predicting from DNA is\ntimely, given the success encountered by this paradigm in other\napplications of learning. The proposed contribution is novel, and does\nseem to improve the prediction performance on a variety of tasks. The\nevaluation is thorough, including large datasets from different\ndomains of the tree of life, relevant baselines and assessments on\nfine-tuning, semi-supervised and transfer learning settings. It also\nincludes an ablation study confirming in particular the contribution\nof (i) training on variable sequence sizes and (ii)\nreverse-complementing the piece of the sequence to be predicted.\n\nMy main concern is on the clarity of the manuscript, in particular:\n\n- The terms \"encoder\" and \"context\" networks are used without being\n  clearly defined. It could help to say earlier to which architectures\n  (respectively, a CNN and an RNN) they refer specifically.\n\n- I found the use of macro-averaged recall confusing: my understanding\n  is that in this context (binary classification) it is equivalent to\n  balanced accuracy, but it first seems like only recall is considered\n  (as opposed to eg precision or anything else quantifying false\n  positives).\n\n- The claims on RC-invariance should be clarified. For example in 3.2\nit is claimed that \"The architecture enforces the symmetry implied by\nthe RC\". Is it really the case? It seems like the network could learn\nto be RC-invariant, but that the invariance is not built in.\n\n- It was not immediately clear to me why the self-supervision task\nmade sense (predicting the end of a sequence from its start,\ncontrastively to other sequences). It made more sense when I saw the\ntask of predicting a class of sequence (prophage vs non-prophage\nviruses. Maybe this could be justified earlier. Maybe more\nimportantly, is the relevance of this self-sueprvision universal, or could it be\nless efficient on some other tasks?\n\nThe writting could also be improved, eg:\n\n- \"One likely explanation is that, because we maximized the mutual\ninformation between the learned embeddings and varying-length target\nsequences with both short and long inputs.\"\n\n- \"One loss term is introduced for each index i, denoting\n  the number of patches represented by z i\"\n\n- \"We investigated the effect of the change in length of target subsequen-\nces by comparing our method against the single-length or longer target subsequences.\"\n\n- \"Self-GenomeNet uses the embedding representation of a context\nnetwork for downstream tasks which learns from reverse complement as a\ntarget for its predictions.\"\n\n- In Section 3 (motivation) I also found the formulation unclear: the\ntext suggests that CPC doesn't deal with multiple sizes, and then says\nthe opposite.\n", "summary_of_the_paper": "This submission introduces self-genomenet, a self-supervised training\nmethod for learning from DNA sequences. The self-supervision is done\nby predicting the end of a sequence from its start (both broken into\nsmaller subsequences), through a contrastive loss against other random\nsequences. The predicted part is also reverse-complemented (RC),\nmaking the network learn the expected reverse-complement invariance of\nthe prediction function. The method is extensively tested on several\nlearning tasks, where it shows good performances.", "summary_of_the_review": "The contribution is novel, timely and useful to the community. Its assessment is thorough. The clarity of the manuscript could be improved.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "6DXbdFqW0eT", "reviewer": "Reviewer_9TG8", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n* The proposed method is simple yet effective. The domain-specific data augmentation (e.g., reverse complement) improves the representation power considerably. \n* Unlike CPC or skip-gram methods that use a fixed-length short subsequence, the proposed method trains models based on varying length sequences. \n* The improvement compared to baselines CPC, Language model, and even supervised learning method is considerable.\n\n\n**Weaknesses**\n* The proposed method is constructed by existing methods and beyond the genome data, the benefits of the proposed method are limited.\n* If the authors provide more explanation about the difference between the proposed method and baseline algorithms such as contrastive-sc, it would be better to evaluate the main contributions of this manuscript.", "summary_of_the_paper": "The authors proposed a self-supervised learning method for nucleotide-level genomic data utilizing reverse-complement of genomic sequences. The proposed method achieved a considerable performance improvement. In addition, the authors proposed an architecture called Self-GenomeNet that handles varying-length genome sequences.", "summary_of_the_review": "The proposed method is a simple and effective contrastive learning method and shows great representation power. The proposed method showed the best performance in many different settings such as linear evaluation and semi-supervised learning. The reverse-complement based data augmentation seems simple and effective. However, the proposed framework is overall a simple combination of existing methods and beyond genome datasets, the impact of this proposed method is questionable.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Based on the contrastive learning loss wildly used in the NLP and computer vision domains, this paper presents Self-GenomeNet, a contrastive learning method for representation learning of genomic sequences.
As shown in the experiment section, the improvement compared to baselines CPC, Language model, and even supervised learning method is considerable, on three benchmark datasets in both self-supervised and semi-supervised evaluation.

Even after the discussion phase, there exists disagreement among the reviewers.
AC considered all reviews, author responses, and the discussions, as well as read the paper.
While the paper has some merit such as an effective Self-GenomeNet model for the particular problem setup, reviewers still have several reservations to directly accepting it:
+ Questionable impact. The proposed framework is overall a simple combination of existing methods and beyond genome datasets, the impact of this proposed method is questionable.
+ Limited inspiration. The proposed method is mainly constructed on the previously-proposed contrastive learning loss wildly-used in the NLP and computer vision domains, the benefits of the proposed method may be limited on the genome data (especially the domain-specific data augmentation e.g., reverse complement). How can the insights foster future research?
+ Lack of justification. The innovations introduced by the paper seem ad-hoc, and the reasons for the large observed improvement are not entirely intuitive.
Meanwhile, even with the provided response from the authors, the connection between motivation and the proposed method is still not crystal clear.

Given the above reservations, AC could not accept the paper for now but encourage the authors to fully revise the paper and strengthen their work.
Decision: Reject