determined that, if the semantic group of one of the asserted codes were consistent with the dominant semantic group of the CDE, then the permissible value would pass the check. There are some examples in table 2 that did not pass the check and were marked as ‘multiple codes’ to indicate their inconsistency. For the longer run, we would advocate that a formal post-coordi- nation model such as that described in the SNOMED CT Tech- nical Implementation Guide24 should be adopted.25 The structure described in this model identiﬁes the focus concept(s) of the expression, and only the semantic groups that correspond to those concepts would be used for the CDE consistency checking. That said, the generation and comparison of post-coordinated concepts is a sufﬁciently complex task that the added contri- bution to interoperability may still be limited. The remaining inconsistencies consisted of the errors in the assignment of value meanings to permissible values, where one of the terms associated with the assigned concept is a homonym for the real target term (eg, ‘X-Ray’, the physical phenomenon, vs ‘X-Ray’, the diagnostic procedure). As mentioned above, the caDSR is part of the NCI caCORE infrastructure, and uses caCORE resources to support data standardization in clinical research cancer studies. Speciﬁcally, the CDE Curation Tool in the caDSR used the NCI EVS vocabulary services for annotating the permissible values of a CDE. The regular term search based on the vocabulary services is usually used to help assign the NCI concept codes (ie, the value meaning) to the permissible values.26 Errors are prone to occur if curators just see the surface meaning based on term strings. Although careful reading of the textual deﬁnition (if available) of a concept code would be helpful for determining a correct concept code, an automatic alerting functionality would be more desirable. For example, once an incorrect code is assigned, the curation system will raise an alerting ﬂag for quality checking. Note that we are aware that some of the errors may be caused by other factors. These factors include SN classiﬁcation error on the part of the NCIt (eg, the semantic type of T185 on NO described in ﬁgure 1). We also uncovered a small portion of terminological annota- tions by the NCIt concept codes that were no longer available, as they had been retired or merged in the latest version of NCIt.27 From the perspective of quality control, we consider that a versioning control mechanism for the terminological annota- tions is an important aspect to ensure the quality of value set components of the CDEs. We also note that about 51.3% of permissible values in the caDSR still do not have any termino- logical annotations asserted at all. This reﬂects a very important aspect of challenges for the meta-data standard (such as ISO/IEC 11179)dthat is, it is not an easy task to author the terminological annotations for a large set of permissible values. Without having the terminological annotations asserted, it would be hard to ensure the quality of the value set components of a CDE for its downstream use. Although the automatic approach using the natural language processing-based ontology annotation tools may partially help resolve the problem, we consider the recent advance in value set deﬁnition services (eg, one of the CTS2 services2) in the community of terminological services would also be useful for dealing with the challenge. While the potential causes of the semantic inconsistencies discovered in our study varied, we would assert that all of them could have a negative impact on data integration and interop- erability across disparate clinical research datasets. Residual context codes have already been well documented by Cimino23 when it comes to meaning drift, but the circumstances in the CDEs render it even more problematic in that the code by itself does not indicate ‘a process/disease/organism/etc not elsewhere J Am Med Inform Assoc 2012;19:e129ee136. doi:10.1136/amiajnl-2011-000739 Research and applications classiﬁed’dinstead providing one big category stating that ‘something’ was reported that we were unable to name. The juxtaposition of multiple codes in an attempt to create a post- coordinated grammar presents many problems, including isolating the actual head code and mapping between the expressions and local data. Errors due to homonyms and false similarity have the potential to result in incorrect information (eg, classifying a particular surgical procedure as a profession), confusion, and wholesale errors in the content of data itself. We should also note that the loss of information due to reorgani- zation of ontology content has the potential to render depen- dent information useless. We also observe in conclusion that the semantic web frame- work we used in this study has been demonstrated to be very useful for achieving our goal. Similarly, semantic web technol- ogies have been leveraged to audit the large-scale biomedical terminologies such as NCIt and SNOMED CT for the purpose of quality assurance.28 29 In this study, we utilized the XML Semantics Reuse technologyda XML2RDF converter, the RDF store and the standard SPARQL query services. With the capacity of the RDF store, we were able to integrate multiple, large-scale, heterogeneous CDE data and ontology resources easily in an agile manner. The underlying RDF model encoding of knowledge in the form of triples plays a key role in this, as the RDF can be used as a schema-less data representation format. This ensures the ﬂexibility and scalability of our system and enables a high-throughput mechanism for quality evaluation. Using the powerful SPARQL query language, we were able to map the CDE-permissible values, their annotations, semantic types and groups across different graph models for the purpose of designed evaluation. In addition, we will be able to build a community-based collaborative framework for quality assur- ance of clinical study CDE authoring, as we have explored in our previous studies.30 31 Finally, there are several limitations in this study. First, the evaluation was not focused on the clinical validity of the value sets, although we believe that the original value set creation process may possibly be a contributing factor to the semantic inconsistency of the terminological annotations downstream. In addition, it would be interesting to explore in future whether there are clinical research cases where a value set for a CDE would have multiple concepts from multiple semantic groups and that might be ideal for that CDE. Second, in this study, we only used the NCI caDSR CDE repository as an example to develop a quality evaluation mechanism. We believe that the proposed approaches could be generalized to audit the CDE repository developed in individual institutes. CONCLUSION In conclusion, we have developed a novel SPARQL-enabled approach for quality evaluation of terminological annotations on the value set components (ie, the permissible values of the enumerated value domain) of a CDE using UMLS semantic types and groups. We believe that our approach produces a useful quality assurance mechanism for a clinical study CDE repository. Acknowledgments The authors thank Ms Ann Ihrke, a lead copy editor/proofreader at Mayo Clinic, who provided copy editing services. Contributors GJ and HRS conceived and designed the evaluation; GJ analyzed the data; GJ, HRS and CGC performed the evaluation; GJ and HRS wrote the paper; CGC reviewed the paper. Funding This study is supported in part by NCI caBIG Vocabulary Knowledge Center Solicitation S07-03 (Contract Number 28XS192). e135