Benchmarking Machine Learning Robustness in Covid-19 Spike Sequence Classification

The rapid spread of the COVID-19 pandemic has resulted in an unprecedented amount of sequence data of the SARS-CoV-2 viral genome — millions of sequences and counting . This amount of data , while being orders of magnitude beyond the capacity of traditional approaches to understanding the diversity , dynamics and evolution of viruses , is nonetheless a rich resource for machine learning ( ML ) and deep learning ( DL ) approaches as alternatives for extracting such important information from these data . It is of hence utmost importance to design a framework for testing and benchmarking the robustness of these ML and DL approaches . This paper the first ( to our knowledge ) to explore such a framework . In this paper , we introduce several ways to perturb SARS-CoV-2 spike protein sequences in ways that mimic the error profiles of common sequencing platforms such as Illumina and PacBio . We show from experiments on a wide array of ML approaches from naive Bayes to logistic regression , that DL approaches are more robust ( and accurate ) to such adverarial attacks to the input sequences , while k-mer based feature vector representations are more robust than the baseline one-hot embedding . Our benchmarking framework may developers of futher ML and DL techniques to properly assess their approaches towards understanding the behaviour of the SARS-CoV-2 virus , or towards avoiding possible future pandemics . 1 INTRODUCTION . A novel ( RNA ) coronavirus was identified in January 2020 , which began the COVID-19 pandemic that is still ongoing today . With the help of sequencing technology and phylogenetic analysis , the scientific community disclosed that this novel coronavirus has 50 % similarity with the Middle-Eastern Repiratory Syndrome Coronavirus ( MERS-CoV ) , 79 % sequencing similarity to Severe Acute Respiratory Syndrome Coronavirus ( SARS-CoV ) — also known simply as “ SARS ” — and more than 85 % similarity with coronavirusus found in bats . Further studies confirmed that bats are the likely reservoir of these coronaviruses ; however , the ecological separation of bats from humans indicates that some other organisms may have acted as intermediate hosts . Considering all scientific evidence , the International Committee on Taxonomy of Viruses named the novel RNA virus SARS-CoV-2 ( Wu et al. , 2020 ; Park , 2020 ; Zhang & Holmes , 2020 ) . RNA viruses generally introduce errors during replication , the resulting mutations incorporated into the viral genome after repeated replication within a single host , generating a heterogenous population of viral quasispecies . However , SARS-CoV-2 has an excellent proofreading mechanism that encodes a nonstructural protein 14 ( nsp14 ) allowing it to have a 10-fold lower mutation rate than the typical RNA viruses . Epidemiologists estimate that SARS-CoV-2 has 33 genomic mutations per year on average . Some of these mutations are advantageous , leading to the more infections variants of SARS-CoV-2 that continue to emerge ( Nelson , 2021 ) . Because of this relatively slow process of accumlating mutations , and the fact that each major variant can be characterized or differentiated by a handfull of mutations ( SARS-CoV-2 Variant Classifications and Definitions , 2021 ) , small perturbations ( errors ) in the sequence should not lead to mistaking one variant from another . Moreover , most of these changes occur in the S gene — the segment of the genome which encodes the surface , or spike , protein ( see Figure 1 ) — hence , characterizing variants using ( the resulting tran- scribed ) spike proteins is sufficient for the classification task ( SARS-CoV-2 Variant Classifications and Definitions , 2021 ; Kuzmin et al. , 2020 ) . The diminishing cost of next-generation sequencing ( NGS ) technology has aided scientists from different parts of the world to generate SARS-CoV-2 whole-genome sequencing ( WGS ) . The Centers for Disease Control and Prevention ( CDC ) of the United States has also provided a wealth of information on resources , tools , and protocols for SARS-CoV-2 WGS from different sequencing platforms such as Illumina , PacBio , and Ion Torrent . Finally , the Global Initiative on Sharing All Influenza Data ( GISAID ) hosts the largest SARS-CoV-2 genome sequencing dataset to date — the largest of any virus in history , with millions of sequences . This unprecedented amount of genomic data generation and easy availability allowed researchers to explore the molecular mechanism , genetic variability , evolutionary progress , and capability of development and spread of novel variants of the virus . On the other hand , this amount of data exceeds the capacity of methods such as Nextstrain ( Hadfield et al. , 2018 ) or even the more recent IQTREE2 ( Minh et al. , 2020 ) by several orders of magnitude — a Big Data challenge . As a result , recent alternative approaches based on clustering and classification of sequences , e.g. , to identify major variants , have appeared in the literature ( Melnyk et al. , 2020 ; Ali et al. , 2021a ; b ; Ali & Patterson , 2021 ) , with promising accuracy and scalability properties . Many issues still remain , however , such as sequencing errors being mistaken for mutations in different analyses , when studying the evolutionary and transmission pattern of the SARS-CoV-2 ( GISAID History , 2021 ; Arons et al. , 2020 ) , or other viruses . Incorporation of error in NGS sequences due to contamination in sample preparation , sequencing technology , or genome assembly methodology are other confounding factors . Generally , computational biologists filter those sequences having errors or mask those sequence fragments having errors . For example , each GISAID ( GISAID Website , 2021 ) sequence is a consensus sequence from the intra-host viral population sampled from the patient , averaging out the minor variations which exist in this population . While such a consensus sequence is a good representative of this population , i.e. , it is still precise enough to capture the SARS-CoV-2 variant harboured by the infected individual , it comes at the cost of losing this important information , such as these minor variations . Such minor variations , when given enough time to evolve , e.g. , within an immunocompromized individual , can become dominant — one of the theories theory behind the emergence of the Alpha variant ( Frampton et al. , 2021 ) . What this means is that many of machine learning approaches towards clustering and classification of sequences ( Ali et al. , 2021a ; b ; Ali & Patterson , 2021 ) have been operating under rather idealized conditions of virtually error-free consensus sequences . Moveover , these methods rely on a k-mer based feature vector representation — an approach that does not even rely on alignment of the sequences , something which can also introduce bias ( Golubchik et al. , 2007 ) . Such a framework should easily cope with errors as well — something machine learning approaches can do very naturally ( Du et al. , 2021 ) . There is hence a great need for some way to reliably benchmark such methods for robustness to errors , which is what we carry out in this paper . We highlight the main contributions of this paper as follows : • We propose several ways of introducing errors into spike sequences which reflect the error profiles of modern NGS technologies such as Illumina and PacBio ; • We demonstrate that the k-mer based approach is more robust to such errors when compared to the baseline ( one-hot encoding ) ; and • We show that deep learning is generally more robust in handling these errors than machine learning models . Moreover , we extend our error testing procedure as a framework for bechmarking the performance of different ML methods in terms of classification accuracy and robustness to different types of simulated random errors in the sequences . The two types of errors that we introduce are “ consecutive ” and “ random ” errors ( see Sec . 3.4 ) . Random errors are just point mutations , which happen uniformly at random along the protein sequence , simulating closely the behaviour of Illumina sequenceing technolgies ( Stoler & Nekrutenko , 2021 ) . Consecutive errors , on the other hand , are small subsequences of consective errors , which can model insertion-delection ( indel ) errors which are common in third generation long-reads technologies such as Pacific Biosciences ( PacBio ) SMRT sequencing ( Dohm et al. , 2020 ) . This paper is structured as follows . In Sec . 2 we discuss related work . In Sec . 3 we discuss some approaches we benchmark , and then how we benchmark : the type of adversarial attacks we use . Sce . 4 details the experiments , and Sec . 5 gives the results . Finally , we conclude this paper in Sec . 6 . 2 RELATED WORK . Assessing and benchmarking the robustness of ML or DL approaches by a series of adversarial attacks is popular in the image classification domain ( Hendrycks & Dietterich , 2019 ) , but there are others that are closer to the domain of molecular data . In ( Schwalbe-Koda et al. , 2021 ) , the authors provide a series of realistic adversarial attacks to benchmark methods which predict chemical properties from atomistic simulations e.g. , molecular conformation , reactions and phase transitions . Even closer to the subject of our paper — protein sequences – the authors of ( Jha et al. , 2021 ) show that methods , such as AlphaFold ( Jumper et al. , 2021 ) and RoseTTAFold ( Baek et al. , 2021 ) which employ deep neural networks to predict protein conformation are not robust : producing drastically different protein structures as a result of very small biologically meaningful perturbations in the protein sequence . Our approach is similar , albeit with the different goal of classification : namely , to explore how a small number of point mutations ( simulating the error introduced certain types of NGS technologies ) can affect the downstream classification of different machine learning and deep learning approaches . After getting the numerical representation , a popular approach is to get the kernel matrix and give that matrix as input to traditional machine learning classifiers like support vector machines ( SVM ) ( Leslie et al. , 2003 ; Farhan et al. , 2017 ; Kuksa et al. , 2012 ) . However , these methods are expensive in terms of space complexity . Authors in ( Ali & Patterson , 2021 ; Kuzmin et al. , 2020 ) proposes an efficient embedding method for classification and clustering of spike sequences . However , their approaches are either not scalable or perform poorly on bigger datasets . 3 PROPOSED APPROACH . In this section , we start by explaining the baseline model for spike sequence classification . After that , we will explain our deep learning model in detail . 3.1 ONE-HOT ENCODING ( OHE ) BASED EMBEDDING . Authors in ( Kuzmin et al. , 2020 ) propose that classification of viral hosts of the coronavirus can be done by using spike sequences only . For this purpose , a fixed-length one-hot encoding based feature vector is generated for the spike sequences . In spike sequence , we have 21 unique characters ( amino acids ) that are “ ACDEFGHIKLMNPQRSTVWXY ” . Also , note that the length of each spike sequence is 1273 with the stopping character ‘ * ’ at the 1274th position . When we design the OHE based numerical vector for the spike sequence , the length of numerical vector will be 21 × 1273 = 26733 . This high dimensionality could create the problem of “ Curse of Dimensionality ( CoD ) ” . To solve CoD problem , any dimensionality reduction method can be used such as Principal Component Analysis ( Abdi & Williams , 2010 ) . After reducing the dimensions of the feature vectors , classical Machine Learning ( ML ) algorithms can be applied to classify the spike sequences . One major problem with such OHE based representation is that it does not preserve the order of the amino acids very efficiently ( Ali et al. , 2021a ) . If we compute the pair-wise euclidean distance between any two OHE based vectors , the overall distance will not be effected if a random pair of amino acids are swapped for those two feature vectors . Since the order of amino acids is important in case of sequential data , OHE fails to give us efficient results ( Ali et al. , 2021a ) . In this paper , we use OHE as baseline embedding method .

Summary: This paper presents a framework to test the accuracy and robustness of different machine learning algorithms in classifying the COVID-19 spike sequences. It benchmarks Naive Bayes, Logistic regression, Ridge regression, and fully-connected neural network architectures. It also explores k-mer and one hot encoding representation of the sequences as inputs into these models. For benchmarking robustness, the paper proposes 2 ways to introduce errors in the spike sequence (reflecting sequencing errors produced by state-of-the-art sequencing technologies).