Peer review of: Detecting sample swaps in diverse NGS data types using linkage disequilibrium (2020)

Reviewers' comments: 

Reviewer #1 (Remarks to the Author): 

In this manuscript, Javed and the authors developed a new algorithm and its implementation 
Crosscheck that detect sample swaps in NGS cohorts. Unlike several previous tools for the same 
purpose, Crosscheck uses linkage disequilibrium information to increase accuracy especially in 
samples with different coverage. In the comparison with NGSCheckMate, the authors showed that 
Crosscheck can detect sample swaps with a lower error rate in low depth sequencing data 
(subsampled from original data). Using Crosscheck, followed by manual inspection, the authors 
detected three types of mislabeling errors in the ENCODE dataset, and reported that about 1% of 
the data could be mislabeled. Overall, the suggested algorithm adopted new strategies to improve 
the performance in the given problem, but several concerns remain in the assumption, formality of 
the method, and practical utility, which should be addressed. 

Major comments: 
1. It is a good idea to use LD information to increase the coverage of SNPs between two or more 
samples, especially if SNPs that are covered and genotyped in one sample, but not in other 
samples. However, SNPs in same LD block are not always in perfect LD. Also SNPs in different LD 
block are not always independent if their physical distance in chromosome is close. Even though it 
may not be critical in the actual performance, the description in the introduction is clearly wrong 
and should be corrected. In the method, the authors used the same simplification including perfect 
LD in same block and perfect independence in different block. The authors should explicitly clarify 
their assumption and simplification strategies for implementation, and they are reasonably robust. 
2. The formal definition of the likelihood in the Method is a strong point of this manuscript, 
whereas other tools, to my knowledge, mostly tried to separate mismatches from matches based 
on heuristic and empirical cutoffs measured from SNP concordance. However, at the later part of 
the formal quantification, the authors dropped the prior probability term p(s=0)/p(s=1), due to 
the difficulties in estimation. I understand that the prior is always hard to estimate. But, if prior is 
not used, the defined posterior odd is not formal anymore, seriously compromising the strong 
point of the method. Like in other problems such as variant calling, prior probability should be 
estimated even if it is not perfectly realistic, at least empirically or parameterized for users. And 
this might affect the final call. For example, let’s assume that p(s=0) is 100 times higher than 
p(s=1) (it is easily assumed that sample swap is a rare event). In that assumption, the final LOD 
will be shifted toward positive by 2. This means that the current LOD is more favorable for no-
swap. Also the ‘flagged’ and ‘inconclusive’ samples in the ENCODE analysis can be changed. 
3. I have a few more questions about the general utility of the tool: 
3-a. Crosscheck showed better performances in low sequencing depth, compared to 
NGSCheckMate, when subsampled by the factor of <15% of the original data of ENCODE (the 
actual average read depth is not provided in the manuscript). But, I wonder how frequently 
sequencing data are generated in this depth. In general WGS, even though the depth is generally 
low (usually around 30x), the number of SNPs should be more than sufficient due to the large 
observable area. In typical WES/RNA-seq and targeted panels, sequencing depth is usually much 
higher (usually >100x). The authors should elaborate more on convincing the practical utility and 
how frequently Crosscheck can be beneficial in general. 
3-b. Crosscheck may be useful in the set of samples, each of which targets disjoint genomic areas 
such as ChIP-seq and RNA-seq, where conventional tools do not work. Proving this is one way to 
emphasize the utility. 
3-c. The authors provided potential mismatches in the ENCODE dataset. Can these mismatches be 
found by other tools? If so, how much of them are only detectable by Crosscheck? 
4. The author compared the performance only to NGSCheckMate. Is there any specific reason not 
to include others? There are at least four more tools including Conpair (Bergmann et al, 2016), 
BAM-matcher (Wang et al, 2016), HYSIS (Schroder et al, 2017), and BamixChecker (Chun et al, 
2019). It would be more convincing if the authors provide more comparison results, especially with 

recent tools. 
5. It would be great to show performance comparisons on more difficult dataset, such as familial 
data, which have been tested in previous tools such as NGSCheckMate and BamixChecker. 

Minor comments: 
1. The authors should provide the average read depth after subsampling where Crosscheck starts 
to outperform other tools. 
2. In page 3, Supplementary Fig 1d is not given in the manuscript (probably Supplementary Fig 
1c?) 

Reviewer #2 (Remarks to the Author): 

The authors describe a new method/tool called Crosscheck that is able to assess sample 
relatedness from NGS data by using a Bayesian approach applied to LD-blocks. The tool is widely 
available as part of the Picard suite and can be readily used to find identity mismatches between 
different sets of NGS data. This approach generates an LOD score, and using a subset of the data 
from the ENCODE project the authors find different useful thresholds which are then used on the 
entire ENCODE dataset, finding about 1% of the samples have potential samples 
swaps/mismatches that are then studied in detail. The model used is statistically sound and 
represents a novel and useful approach to the problem of sample swapping and the manuscript 
merits to be published contingent on addressing some concerns, which I am including below: 

Major concerns: 

1) In page 3), second paragraph, the authors used a simplified approach (that dramatically saves 
time) to test their approach on the ENCODE dataset by "comparing each dataset to a set of three 
representative datasets from its annotated donor". This strategy seems sound, however I believe 
the choice of the number of representative datasets (three) could have been tested using the 
dataset where they did the pilot (279 sets from page 2 where they perform all to all comparisons). 
Using this pilot dataset they can show that their strategy of just using 3 representatives does not 
alter significantly FMR and/or FMR while saving a lot of computation time. 

2) On page 3, on the last paragraph, they identify some mismatched-cases where cell types are 
derived from multiple donors. How does the LOD score looks like in those cases and are those 
score distribution different that just simple sample swaps? Is the LOD score affected by the 
proportion of a particular sample involved in the mixture? 

3) In the methods section the authors refer to a report that outlines in more detail the statistical 
approach followed. Such report is authored by Jon Bloom and Yossi Farjoun. Yossi is included in 
the list of authors in this manuscript, however Jon Bloom is not. Is there a reason for that? It 
seems the manuscript benefits greatly from such method/approach so it would seem natural to 
include both of the authors. 

Minor comment: 

4) Although not the main objective of this manuscript, the authors perhaps should comment on 
how this method is affected by sample coming from related individuals. Also perhaps some 
comments on what happens when sample contamination of other samples occur.