Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: In this paper, the authors propose a method to identify sequenced SARS-CoV-2 genomes with sequencing errors. They explore several feature encodings and machine learning methods to identify such erroneous sequences.
main_review: While I appreciate the time and energy the authors have put into this paper, I have some major concerns with this paper:

- My major issue with this paper is the sequencing errors are not modeled realistically. Most Illumina sequencers have error rates of less than 0.5% of nucleotides [PMID: 33817639], the authors introduce at least 5% error rates in the amino acids (each amino acid is encoded by three nucleotides). NB. Interestingly, this is the citation that the authors use to justify their “simulated” error rates. The authors mention PacBio sequencing errors, but I see no evidence of that in the text.
- Although spike protein is 1,273 amino acids long; however, there are around 200 amino acids in the receptor-binding domain (RBD) of the spike protein that is of special interest [PMID: 34075212]. A more realistic analysis would focus mostly on the RBD. The community is increasingly generating more high-throughput datasets assaying different aspects of the RBD and ACE2 binding, which can help researchers train more accurate models, e.g., [PMID: 32841599]. 
- The concepts introduced in the paper are not novel and have been studied in detail: encoding of amino acids, _k_-mer-based models, and simple MLPs have been thoroughly studied in computational genomics.
- The text is hard to follow and has quite a few grammatical errors/missing words starting from the abstract. A thorough review and polish of the text would improve the paper.

summary_of_the_review: While the paper has a couple of good ideas, it is hard to follow, does not introduce and validate novel concepts and methods, and models the errors in an unrealistic manner.
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: Taking sequences of the spike portion of COVID-19 genome sequences, this paper builds classifiers for different labeled variants. Two approaches for generating errors in test sequences are introduced and evaluations of the models robustness to these errors measured.
main_review: Strengths
1. The paper is clear about its contributions and provides a clear introduction to the problem domain.
2. The motivation for the research of using this large dataset to explore and characterise errors in sequencing platforms is reasonable.

Weaknesses:
1. Why is a fixed length sequence of 1273 amino acids considered for the spike protein? Insertions or deletions mean this could be a variable length.
2. The approach does not take into account any domain knowledge from biology. For example, considering the secondary protein structure among the sequences such as the alpha-helices or beta-sheets would potentially be more predictive than a one-hot encoding or k-mer representation. How much structure is conserved among spike proteins? If there is strong conservation this should be used to perform feature selection on the input data.
3. PCA is mentioned but it is unclear whether the input dimension is reduced or to what level.
4. It is unclear how many labels are used in the classification problem. Is it the number of variants (22) or the number of labeled variants (6)? What about the unlabelled variants? Related to this, justification for the topology of the neural network is unclear. Did one hidden layer of 9261 nodes perform better than a deeper model?   
5. It would be far superior to statistically model the error profiles from Illumina and PacBio technologies. The proposed approach is simplistic.
6. Why was only 1% of the data set used for training? Such a small proportion of the data in this very imbalanced problem risks not adequately representing all classes.
7. The proportion of labels in table 1 do not match the plot in fig 1. Gamma seems over represented compared to alpha and delta.
8. Use of the performance metrics is unclear and inappropriate for this imbalanced problem. For example, average accuracy will overemphasise classification it the major class at the expense of the smaller classes. Similarly, high weighted F1 values overemphasise the large classes. The macro F1 shows that overall the classification of the minor classes seems to be worse. How was ROC-AUC computed for this multi-class problem?    
9. Error bars or confidence should be generated for algorithms so that they can better be compared.
summary_of_the_review: The paper takes a very simplistic approach to classification of these sequences not taking into account any biological domain knowledge. There are concerns about justification for the proposed model, the experimental methodology and evaluation.
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: This paper presents a framework to test the accuracy and robustness of different machine learning algorithms in classifying the COVID-19 spike sequences. It benchmarks Naive Bayes, Logistic regression, Ridge regression, and fully-connected neural network architectures. It also explores k-mer and one hot encoding representation of the sequences as inputs into these models. For benchmarking robustness, the paper proposes 2 ways to introduce errors in the spike sequence (reflecting sequencing errors produced by state-of-the-art sequencing technologies). 

main_review: Major comments:

-- The related work lacks previous papers that have tried to tackle the COVID-19 sequence classification task as well as details about what existing methods have been explored for it. 

-- One hot encoding representation of sequences has been used as inputs into CNN-based deep learning models. The convolution filters allow the model to extract relevant motif/patterns to make accurate predictions. The deep learning model explored in the paper is just a fully connected network that may not be able to leverage the One hot encoding effectively. 

-- The machine learning methods chosen for benchmarking are all simple linear models. K-mer representation has been shown to perform well with SVMs (in the works cited in the paper). Why were these non-linear methods and others like random forests not explored for this study?

-- The k-mer representation presented in this paper is a simple approach and does not account for mismatches or gaps. While such representations have been shown to give better results, they have not been explored in the paper.

-- Each representation has its strengths and weaknesses combined with a relevant model. This aspect needs to be investigated further to support the claims of the paper

-- The paper is missing details on how the hyperparameter tuning was performed for all the models 

-- It is unclear why a training/test split of 1% and 99% were chosen for this study. Usually one picks a larger percentage of training samples.

-- The description of the results and the figures are hard to parse. 

-- The paper requires thorough proofreading to address the grammatical errors and issues with clarity

summary_of_the_review: While the paper aims to investigate a relevant problem, the ICLR venue may not be the best place to present the benchmarking work. It lacks new contributions towards the method or application domains. The paper also requires major revisions and additional results to improve its clarity and to support its claims.
review_title: null

Meta-review: This paper presents a framework to test the accuracy and robustness of different machine learning algorithms in classifying the COVID-19 spike sequences. After reading the paper and taking into consideration the reviewing process, here are my comments:
- The work is aligned to the efforts on understanding the COVID-19 pandemic.
- Many concepts are not novel.
- Sequences errors are not modeled in a realistic way.
- The benchmark is very limited and nonlinear machine learning approaches are presented. 
- Many typos are presented.
From the above, the paper is not suitable for aacceptance in ICLR.
Decision: Reject