Over the course of the pandemic, genetic surveillance focus shifted from the analysis of consensus genomes alone to include more frequent use of the underlying sequence data. While consensus genomes reduce the complexity and size of the data, recent cases of recombinants [26,27], co-infection [28,29], and monitoring for drug resistance has made clear the importance of working beyond the consensus genome. If only the consensus sequence is provided, contamination that may have occurred during sample collection, laboratory preparation, sequencing, or bioinformatics processing may be interpreted as a coinfection. In absence of corresponding raw data, these potentially confounding factors cannot be easily tracked and corrected. However, as consensus genomes are usually the starting material for many common analyses, we aimed here to identify factors which influence the accuracy and consistency of variant calling results.

The importance of primer trimming, shown in Figure 1, highlights the need for comprehensive metadata when sharing data publicly, most notably information on which primer scheme was used. While consensus between platforms is possible, as illustrated by Figure 5, the potential challenges highlight the need for transparency regarding how the original sequencing was performed, even if the submitted sequence is a consensus sequence; specifically, information on what instrument was used for sequencing, along with the method and parameters used for consensus construction. Additionally, the difficulty in finding consensus with low-quality data suggests a need for quality standards for SARS-CoV-2 NGS data. One such approach would be to identify a minimum reference coverage and a minimum average depth of coverage, such that data below the minimum threshold cannot yield consistent-enough results to be used for common downstream analysis, such as lineage assignment or phylogenetic reconstruction.

Given the number of discrepancies found even when only considering a minimum within-sample allele frequency of 15% and the parameter setting found necessary to ensure inter-workflow compatibility, the use of minor alleles should be considered carefully. In support of this suggestion, prior work on respiratory syncytial virus sequencing found that, below allele frequencies of 25%, variant calling across a range of coverages, variant callers, and a range of simulated error profiles resulted in a precision of less than 50%, approaching 0% at allele frequencies below 5% [30]. Additionally, sequencing analysis on influenza samples has found that minor alleles cannot be confidently called at frequencies below 5% [31], and that without correction to the methods, minor variants were overestimated by as much as 10-fold [32]. These studies also identified several methods to improve confidence in variant calls, including generating technical replicates for each sample and adjusting allele-frequency thresholds based on Polymerase Chain Reaction (PCR) cycle threshold values, both of which underscore the need for rich metadata to support thorough and accurate use of public sequence data and the need to consider consensus sequences carefully in the absence of such metadata. Furthermore, these suggestions highlight a topic not examined in this work, which is the need to carefully consider wet-lab controls to ensure quality data are generated prior to bioinformatic analyses. Together with the results presented here, this suggests that more research is needed into the conditions under which minor variants (<15% allele frequency) can be determined most reliably.

Pharmaceutical and biotechnology companies are required to comply with Good Practice (GxP) guidelines and obtain system validation when developing computer software to ensure that medical devices, drugs, and other life science products are safe and effective [33]. At present, it is unclear what level of computer system validation is required by regulatory agencies, including the FDA for NGS analysis workflows, and regulatory agencies typically perform their own independent analyses and compare them to the submitted analysis from sponsors. Given the numerous sample preparation methods and sequencing platforms, as well as the various analysis workflows (many of which are proprietary), this presents challenges for regulatory review of deep sequencing data [34]. In the future, the FDA anticipates the development of standardized analysis workflows that will provide reproducible data analysis sufficient for generating consistent and robust results from NGS data [34]. However, we have shown that standardized deep sequencing analysis workflows are currently not used across academia, government, and industry. Further collaborative efforts, like the work from ACTIV TRACE shown here, are needed to achieve the standardization of workflows and reproducibility of results that are needed in the face of pandemic threats. This collaboration representing industry, government, and academic partners has moved us closer to the standardized analysis that will be required for deep sequencing, suggesting that regulatory agencies should implement these robust recommendations in their current guidelines.

------------------------------------------------------------
translate above into plain Chinese