Trust, But Verify: On the Importance of Chemical Structure Curation in Cheminformatics and QSAR Modeling Research
(2010)

With the recent advent of high-throughput technologies for both compound synthesis and biological screening, there is no shortage of publicly or commercially available data sets and databases that can be used for computational drug discovery applications (reviewed recently in Williams et al.). Rapid growth of large, publicly available databases (such as PubChem or ChemSpider containing more than 20 million molecular records each) enabled by experimental projects such as NIH’s Molecular Libraries and Imaging Initiative provides new opportunities for the development of cheminformatics methodologies and their application to knowledge discovery in molecular databases. A fundamental assumption of any cheminformatics study is the correctness of the input data generated by experimental scientists and available in various data sets. Nevertheless, a recent study showed that on average there are two errors per each medicinal chemistry publication with an overall error rate for compounds indexed in the WOMBAT database as high as 8%. In another recent study, the authors investigated several public and commercial databases to calculate their error rates: the latter were ranging from 0.1 to 3.4% depending on the database. How significant is the problem of accurate structure representation (given that the error rates in current databases may appear relatively low) since it concerns exploratory cheminformatics and molecular modeling research? Recent investigations by a large group of collaborators from six laboratories have clearly demonstrated that the type of chemical descriptors has much greater influence on the prediction performance of QSAR models than the nature of model optimization techniques. These findings suggest that having erroneous structures represented by erroneous descriptors should have a detrimental effect on model performance. Indeed, a recent seminal publication clearly pointed out the importance of chemical data curation in the context of QSAR modeling. The authors have discussed the error rates in several known databases and evaluated the consequences of both random and systematic errors with respect to the prediction performances of the derivative QSAR models. They also presented several illustrative examples of incorrect structures generated from either correct or incorrect SMILES. The main conclusions of the study were that small structural errors within a data set could lead to significant losses of predictive ability of QSAR models. The authors further demonstrated that manual curation of structural data leads to substantial increase in the model predictivity. This conclusion becomes especially important in light of the aforementioned study of Oprea et al. that cited a significant error rate in medicinal chemistry literature. Alarmed by these conclusions, we have examined several popular public databases of bioactive molecules to assess possible error rates in structure representation. For instance, the NCI AIDS Antiviral Screen (the paper describing this screen has been cited 57 times in PubMed Central only) comprises 42687 chemical records with their associated activity. Even quick analysis of this data sets revealed that 4202 (i.e., ca. 10%) compounds should be either treated with care or removed before any cheminformatics investigation: 3350 compounds were mixtures and salts, and we detected 741 pairs of exact duplicates and stereoisomers, possessing different or opposite reported activities. Similar observations can be made for several well-known public databases such as NCI Human Tumor Cell Line and PubChem, as well as for smaller data sets studied and reported in published literature. For instance, in a study already mentioned above, six research teams each specializing in QSAR modeling (including the authors of this paper!) collaborated on the analysis of the Tetrahymena pyriformis aquatic toxicity data set comprising 1093 compounds. Later this exact data set was used by the organizers of CADASTER toxicity challenge. However, our re-examination of this data set showed the presence of six pairs of duplicates among 1093 compounds because of the presence of different metal cations in salts (with different aquatic toxicities measured by pIGC50 ranging from ∼0.1 to 1 logarithmic unit. Furthermore, in the new external set compiled by the organizers of the CADASTER challenge to evaluate the comparative performance of competing models, eight out of 120 compounds were found to be structurally identical to modeling set compounds but with different toxicity values (∼0.3 pIGC50). Such disappointing observations may, at least in part, explain why QSAR models may sometimes fail, which is * To whom correspondence should be addressed. E-mail: alex_tropsha@ unc.edu. † University of North Carolina at Chapel Hill. ‡ A.V. Bogatsky Physical-Chemical Institute NAS of Ukraine. J. Chem. Inf. Model. XXXX, xxx, 000 A