NIST GCR 11-938

Quality Reporting Data Architecture (QRDA) Suitability

Analysis

Lantana Consulting Group 
NIST GCR 11-938

Quality Reporting Data Architecture (QRDA) Suitability

Analysis

Prepared for 

 National Institute of Standards and Technology

Gaithersburg Md 20899-8202

By

Lantana Consulting Group

May 2011 

U.S. Department of Commerce 

John E. Bryson, Secretary

National Institute of Standards and Technology 

Patrick D. Gallagher, Under Secretary for Standards and Technology and Director

This publication was produced as part of contract SB1341-10-SE-0911 with the National Institute of Standards and

Technology. The contents of this publication do not necessarily reflect the views or policies of the National Institute of

Standards and Technology or the US Government.
Quality Reporting Data Architecture (QRDA)

Suitability Analysis

May 2011

Prepared by Lantana Consulting Group

for

National Institute of Standards and Technology

© 2011 Lantana Consulting Group

All rights reserved.
Lantana Consulting Group

PO Box 177

East Thetford, VT 05043

www.lantanagroup.com

Liora Alschuler

Chief Executive Officer

liora.alschuler@lantanagroup.com

Bob Dolin, MD, FACP

President and Chief Medical Officer

bob.dolin@lantanagroup.com

Rick Geimer

Chief Technology Officer

rick.geimer@lantanagroup.com

Lauren Wood, Ph.D.

Senior Project Manager

lauren.wood@lantanagroup.com

Gaye Dolin, RN, MSN

Senior Nurse Informaticist

gaye.dolin@lantanagroup.com

Jingdong Li

Chief Architect

jingdong.li@lantanagroup.com

Yan Heras

Senior Medical Informaticist

yan.heras@lantanagroup.com

Bob Yencha

Principal Consultant

bob.yencha@lantanagroup.com

Kate Hamilton

Senior Information Analyst

kate.hamilton@lantanagroup.com

Susan Hardy

Senior Technical Editor

susan.hardy@lantangroup.com

Lantana Consulting Group QRDA Suitability Analysis May 2011

Prepared for NIST Page 4

 2011, all rights reserved
Acknowledgments

This analysis was produced and developed by Lantana Consulting Groupfor the National Institute

of Standards and Technology (NIST) under contract number SB134110SE0911.

We appreciate the information and insights provided by several healthcare experts who answered

our e-mail questionnaire or took part in phone interviews. To comply with regulations, fewer than

ten private-sector experts took part in discussions about each standard, although the total number

of experts we talked to exceeded ten. They were Anthony LaRocca, Sage Software; Ben Hamlin,

National Committee for Quality Assurance (NCQA); Dave Perry, Lovelace Clinic Foundation;

David Dobbs, Science Applications International Corporation (SAIC)/Biosense; Carl Dvorak,

Epic; George Cole, Allscripts; David Kibbe, American Academy of Family Physicians (AAFP);

Austin Kreisler, SAIC; Mark Stine, Medplus; Paul Klinker, Harris; Dan Pollock, Centers for

Disease Control and Prevention (CDC); Rick Moore, NCQA; Tone Southerland, Greenway; Steve

Waldren, AAFP; and Thanos Tsiolis, Epic.

Any mention of commercial products or organizations in this report is for information only; it

does not imply recommendation or endorsement by Lantana or NIST.

Lantana Consulting Group QRDA Suitability Analysis May 2011

Prepared for NIST Page 5

 2011, all rights reserved
Table of Contents

EXECUTIVE SUMMARY................................................................................................ 6

INTRODUCTION............................................................................................................. 7

QRDA Summary of Purpose....................................................................................................... 7

QRDA and Meaningful Use........................................................................................................ 8

STAKEHOLDER INTERVIEWS...................................................................................... 9

QRDA SUITABILITY ANALYSIS................................................................................. 11

Is the standard based on a stable, well-vetted data model?........................................................ 11

Does the standard have a clear, robust vocabulary-binding syntax?.......................................... 11

Does the standard support reusable modules, such as templates or data types?.........................12

Does the standard have a well-defined constraint mechanism?................................................. 12

Does the standard have a well-defined extensibility mechanism?............................................. 12

Are there unambiguous definitions of what is testable?............................................................ 13

Are there automated test tools and test suites?.......................................................................... 14

Are there reference implementations?....................................................................................... 14

Is there documented existence of errors, including estimates of the severity?...........................14

Is there a defined and effective process for handling errors?.....................................................15

Do industry associations endorse the standard?.........................................................................15

Has the standard been implemented by a range of vendors?.....................................................15

Is the standard used in more than one country?......................................................................... 15

Is certification available for developers and architects?............................................................ 16

QRDA SUITABILITY SUMMARY................................................................................ 17

MEANINGFUL USE ANALYSIS................................................................................... 19

QRDA and eMeasure................................................................................................................ 19

QDM Building-Block Approach to eMeasures..........................................................................21

QRDA and CCD........................................................................................................................23

QRDA and PQRI Registry XML Specification......................................................................... 23

QRDA Ballot Consideration......................................................................................................24

Lantana Consulting Group QRDA Suitability Analysis May 2011

Prepared for NIST Page 6

 2011, all rights reserved
CONCLUSIONS..............................................................................................................25

REFERENCES................................................................................................................ 26

ACRONYMS AND ABBREVIATIONS......................................................................... 28

List of Figures

FIGURE 1: MEANINGFUL USE AND QRDA PURPOSES............................................ 8

FIGURE 2: QUALITY FRAMEWORK.......................................................................... 19

FIGURE 3: EMEASURE AND QRDA........................................................................... 20

FIGURE 4: QUALITY DATA MODEL...........................................................................21

FIGURE 5: BUILDING-BLOCK APPROACH TO EMEASURES................................22

List of Tables

 TABLE 1: QRDA STANDARD-SPECIFIC INTERVIEW QUESTIONS........................9

TABLE 2: QRDA CRITERIA MATRIX.......................................................................... 17

Lantana Consulting Group QRDA Suitability Analysis May 2011

Prepared for NIST Page 7

 2011, all rights reserved
Executive Summary

This document analyzes the suitability of Quality Reporting Data Architecture (QRDA) to meet

its purpose and to support the US Department of Health and Human Services’ (HHS) Meaningful

Use. It is part of the Healthcare Information Technology (HIT) Standards Analysis Project carried

out by Lantana Consulting Group for the National Institute of Standards and Technology (NIST).

We applied the methodology described in the “Healthcare Information Technology Standards:

General Suitability Analysis,” also prepared for NIST. In addition, we focused on whether QRDA

meets the current and projected data requirements of Meaningful Use. 

QRDA specifies the framework for quality reporting. QRDA, together with Clinical Document

Architecture (CDA) templates, can standardize quality reporting as well as measure-defined data

elements.

QRDA is a relatively new standard that is not fully specified; however, it is well suited for its

purpose. CDA Release 2 is the base standard for QRDA, hence it is suitable for transmitting

clinical quality measure data for current and future Meaningful Use requirements. CDA

templates, implemented in QRDA, complement the National Quality Forum (NQF)’s Quality

Data Model (QDM) building-block approach for creating eMeasures and make possible a tight

coupling between the QRDA and Health Quality Measures Format (HQMF) standards. 

While we found that QRDA is suitable, we identified a number of areas for improvement: QRDA

requirements should be accompanied by corresponding Schematron rules to support testing and

validation, unless other test methods are widely available. The specification itself requires further

development and balloting to support aggregate-level reporting. Inconsistencies between QRDA

and HQMF must be reconciled, and clarification must be provided for QRDA’s reuse of templates

from other CDA implementations. Improvements in testability and error handling along with the

creation of a certification program and a reference implementation should encourage a wide range

of vendors to implement the standard.

Lantana Consulting Group QRDA Suitability Analysis May 2011

Prepared for NIST Page 8

 2011, all rights reserved
Introduction

This document assesses the suitability of the Health Level Seven (HL7) Quality Reporting Data

Architecture (QRDA) specification1 to support US Department of Health and Human Services’

(HHS) Meaningful Use.2 It is part of the Healthcare Information Technology (HIT) Standards

Analysis Project carried out by Lantana Consulting Group for the National Institute of Standards

and Technology (NIST). For that project, we assessed four standards for suitability: QRDA, HL7

Continuity of Care Document (CCD), ASTM Continuity of Care Record (CCR), and HL7 Version

2 Biosurveillance. 

Suitability analysis can be thought of as a “fitness for purpose” study. Before we could assess the

fitness of QRDA, we defined the goals of QRDA and then aligned those with Meaningful Use. 

We assessed QRDA by applying each criterion defined in the “Healthcare Information

Technology Standards: General Suitability Analysis”3. We also assessed QRDA for its suitability

in reporting the Stage 1 Meaningful Use clinical quality measures. 

QRDA Summary of Purpose

QRDA is a relatively new HL7 standard, released in 2009. It specifies a framework for quality

reporting and standardizes the representation of measure-defined data elements for

interoperability between organizations. 

QRDA defines three types of quality measure reports:

• QRDA Category I is a patient-level report; it contains raw data for one patient and for one

or more quality measures. There is no assertion about the status of quality compliance. 

• QRDA Category II is a multi-patient report; each report contains data for a set of patients

for one or more quality measures.

• QRDA Category III Calculated Report is an aggregate report. A Category III report

contains only calculated population data, whereas Category I and II reports contain data

for an individual patient.

The QRDA Category I report is a HL7 Draft Standard for Trial Use (DSTU), while QRDA

Category II and Category III are currently published for comments only. 

Clinical Document Architecture Release 2 (CDA R2)4 is the base standard for QRDA. QRDA

further constrains CDA R2 and reuses CDA templates, including those for CCD wherever

possible.

1 HL7 Implementation Guide for CDA Release 2 Quality Reporting Document Architecture (QRDA) Draft

Standard for Trial Use, March 2009. 

2

http://healthit.hhs.gov/portal/server.pt?open=512&objID=2996&mode=2 (accessed March 2011)

3 Lantana Consulting Group. Healthcare Information Technology Standards: General Suitability Analysis,

May 2011. Related document prepared for NIST.

4

 Dolin et al. 2005. See References for complete information.

Lantana Consulting Group QRDA Suitability Analysis May 2011

Prepared for NIST Page 9

 2011, all rights reserved
QRDA and Meaningful Use

The ultimate goal of Meaningful Use is to achieve significant improvements in care. The

objectives of Meaningful Use are to (1) improve quality, efficiency, safety, and reduce health

disparities; (2) engage patients and families; (3) improve care coordination; (4) improve

population and public health; and (5) ensure adequate privacy and security protections for

personal health information. 

QRDA’s goal is to support quality reporting, which is aligned with the Meaningful Use goals of

improving quality and reducing health disparities, and improving population and public health

(see Figure 1: Meaningful Use and QRDA purposes). 

QRDA standardizes the representation of measure-defined data elements to enable

interoperability between all of the stakeholder organizations. QRDA supports the exchange of (1)

patient-level quality data from a provider system to a quality data measurement and reporting

facility and (2) population-level quality data from a measurement and reporting facility to quality

data recipients.

Figure 1: Meaningful Use and QRDA purposes

Lantana Consulting Group QRDA Suitability Analysis May 2011

Prepared for NIST Page 10

 2011, all rights reserved
Stakeholder Interviews

The “Healthcare Information Technology Standards: General Suitability Analysis” describes our

interviews with experts in HIT standards development, quality reporting, and standards

implementation to evaluate the maturity, robustness, and suitability of QRDA. 

We asked the following questions specific to QRDA during stakeholder interviews. 

 Table 1: QRDA Standard-Specific Interview Questions

Question

block ID

Interview Questions

QRDA • Please briefly describe your experience with QRDA to help us understand the basis

for your answers to the following questions. 

• How well is QRDA meeting its primary purpose?

• Are there errors or ambiguities in QRDA that mean it’s harder than it should be to

implement and use? Do you have examples?

• Consider the primary purpose of QRDA:

o What are some measurable criteria that should be assessed to ensure it is

meeting this purpose?

o In what ways does it not meet its purpose?

o What makes QRDA testable?

• Where is “suitability” or “fitness for purpose” lacking or present in QRDA? For

instance, here are some potential areas:

o Conveyance of all data needed by a processing entity to compute a

provider’s or organization’s score on one or more quality measures

o Data reuse, whereby data having been originally communicated in a CCD

(or other CDA document) is analyzed against a quality criterion, and is

repackaged as part of a QRDA report

o Conveyance of other aggregation data not directly part of the quality

measure (metadata)

o Ability to improve the quality feedback loop

• Please discuss the one feature about QRDA that most supports the Meaningful Use

ultimate goal of achieving significant improvements in health care.

The interviews included senior developers as well as government-level directors. We received

good, general information based on their experience implementing standards and their wideranging knowledge about standards quality. Unfortunately, we received no information on QRDA

specifically as the interviewees are not familiar with the standard.

Lantana Consulting Group QRDA Suitability Analysis May 2011

Prepared for NIST Page 11

 2011, all rights reserved
QRDA Suitability Analysis

We applied the questions defined in the “Healthcare Information Technology Standards: General

Suitability Analysis” to QRDA.

Is the standard based on a stable, well-vetted data model?

QRDA is an implementation of CDA R2, a standard approved by the American National

Standards Institute (ANSI) in May 2005. Since its release, CDA R2 has been widely implemented

internationally and it is a foundational standard selected by, amongst others, the Healthcare

Information Technology Standards Panel (HITSP). It was recognized by the Secretary of Health

and Human Services, and was designated as a key standard within the Final Rule for electronic

health record (EHR) technology5. Numerous HL7 implementation guides and Integrating the

Healthcare Enterprise (IHE) profiles define the use of CDA in healthcare exchange scenarios.

Following HL7 practice, QRDA is defined in an implementation guide that declares constraints

on the CDA-base standard for quality reporting purposes. Like all CDA documents, QRDA

documents derive their machine-processable meaning from the HL7 Reference Information

Model (RIM) and use the HL7 Version 3 (V3) Data Types. The RIM was developed in a

collaborative process that engaged a broad range of experts, both clinical and technical, within

HL7. The RIM is a robust, implementable standard (as demonstrated by the many implementation

guides and profiles) and it applies to a large number of clinical data exchange use cases within the

healthcare domain. The RIM has demonstrated stability over other models by its longevity

(approximately 15 years).6 The RIM and HL7 V3 foundation components, such as data types and

vocabulary, are documented in a clear and understandable way that passes muster with the experts

who reviewed it. 

Given the global adoption by both vendors and healthcare providers of both the RIM and CDA

R2, we conclude that QRDA is based on a stable, well-vetted data model. 

Does the standard have a clear, robust vocabulary-binding syntax?

Vocabulary binding is essential to a standard's success. The correct interpretation of an exchanged

message relies upon correct message syntax and correct data semantics. While syntactic

correctness is defined by the standard format, semantics are defined by vocabulary binding. Clear,

robust vocabulary binding defines unambiguous links between a data field and medical

vocabulary systems. A data field can be valued only with one specific code or one selected from a

“value set” of codes in the specified vocabulary system.

5 Department of Health and Human Services. Health Information Technology: Initial Set of Standards,

Implementation Specifications, and Certification Criteria for Electronic Health Record Technology; Final

Rule, 45 CFR Part 170. http://edocket.access.gpo.gov/2010/pdf/2010-17210.pdf

6 National Cancer Institute Center for Biomedical Informatics and Information Technology (CBIIT). CBIIT

SAIF Implementation Guide, March 10, 2011 Working Draft. (accessed March 2011)

https://wiki.nci.nih.gov/display/SAIF/3.1+-+Overview+of+CBIIT+SAIF+Information+Framework?

focusedCommentId=39297400

Lantana Consulting Group QRDA Suitability Analysis May 2011

Prepared for NIST Page 12

 2011, all rights reserved
The vocabulary-binding syntax defined by the HL7 vocabulary work group has been through

extensive review and has been improved through numerous rounds of balloting; it is widely used

in the RIM and CDA documents throughout the world. This clear and robust binding syntax is a

foundation for expressing the complex requirements of the QRDA.

Does the standard support reusable modules, such as templates or data

types?

QRDA specifies a framework for quality measure reporting. The measure data that can go into a

QRDA report are potentially limitless. It is not within the scope of QRDA to define how every

possible measure data element should be represented; however, QRDA clearly states that measure

data should reuse CCD and other CDA implementation-guide templates where possible.

Templated CDA, which we discuss in detail in “Templated CDA: Key Concept for

Interoperability”7 prepared for NIST, is an example of a strictly defined architecture for

compatible reuse.

As EHR vendors become familiar with structured documents and the CCD as part of the Office of

National Coordinator (ONC) certification process, they will be able to reuse CDA and CCD

templates for rapid development and implementation of QRDA systems. 

Does the standard have a well-defined constraint mechanism?

The HL7 V3 standard has a well-defined constraint mechanism supported by CDA. The standard

defines constraints on the RIM, data types, vocabularies, implementation guides, et cetera. Along

with extensibility, constraints provide a clear set of rules for producing local variants to meet

realm-specific requirements while preserving the global applicability of the V3 standard.8

As a US-realm implementation guide based on the CDA R2 base standard, QRDA Release 1 uses

the HL7 V3 standard constraint mechanism. It meets the needs of quality reporting requirements

in the US.

Does the standard have a well-defined extensibility mechanism?

The base CDA R2 standard has a well-defined extensibility mechanism. Implementations may

use namespace extensions to include additional Extensible Markup Language (XML) elements

and attributes that are not in the CDA schema. These extensions cannot change the meaning of

any of the standard data items, and document recipients must be able to faithfully render the CDA

document while ignoring extensions.

For vocabulary binding to a domain, the HL7 V3 standard allows an Extensibility Qualifier to be

associated with the coded entry. The Extensibility Qualifier has two possible values: CNE (coded

no extensions), and CWE (coded with extensions). CWE allows the code set to be expanded to

meet local implementation needs. When a coded attribute is sent in a message, local concepts or

7 Lantana Consulting Group. Templated CDA: Key Concept for Interoperability. May 2011. Related

document prepared for NIST.

8 HL7 International Localization Task Force Report Version 2b. http://www.eosiris.it/documenti/s_HL7/HL7IntlLocalizationReport_v2b.doc (accessed March 2011)

Lantana Consulting Group QRDA Suitability Analysis May 2011

Prepared for NIST Page 13

 2011, all rights reserved
free text may be sent in place of a standard code if the desired concept is not represented in the

standard vocabulary domain. 

An additional extensibility feature of the CDA standard is the inclusion of generic classes, such as

act and participant. The act class can be used if no more specific class is available for the use

case. “Teach cast care” is an example. While “teach cast care” is defined in some code systems

as a procedure, it does not fit the HL7 definition of procedure: “an Act whose immediate and

primary outcome (post-condition) is the alteration of the physical condition of the subject”;9 nor

is teaching an observation. The act class can represent this in CDA such that it is semantically

interpretable across systems. The generic participant can represent any participants not explicitly

mentioned by other classes that were involved with the patient or the situation being documented.

For example, there is no “Next of Kin” participant, but the generic participant can use standard

modeling and codes to represent detailed information about such a participant.

These well-documented extensibility mechanisms support unanticipated use cases within the

quality reporting domain and provide flexibility until specific use cases can be brought to HL7

and incorporated into a future release.

Are there unambiguous definitions of what is testable?

Unambiguous definitions of what is testable directly relate to the implementability of a standard.

As a CDA R2 implementation guide, a QRDA instance can be validated for its structural

correctness by the CDA R2 XML schema. 

Conformance statements define the enforceable aspects of an HL7 specification. Conformance

statements are constructed from common language statements and keywords to ensure semantic

interoperability across standards and wire formats. These statements guide the coding and

information content of a given template. CDA implementation guides contain different types of

conformance statements such as parent-to-child constraints, document-tree conditional

constraints, and template-specific vocabulary constraints. The “CCD Coverage Report”10

describes the types of constraints and their testability. 

Not every CDA and QRDA conformance statement is testable; ambiguity and inconsistency in the

application of some statements create “fuzziness” about what is testable. The ONC’s Standards

and Interoperability (S&I) CDA Consolidation project11 is clarifying and documenting those

conformance statements that are fully machine testable, those that require human intervention,

and those that cannot be evaluated by a machine-driven process. The “CCD Coverage Report”

analyzes the CCD specification from the perspective of validation, looking at the limits of

automated testing and assessing various approaches to it. The conclusions and recommendations

in that report provide valuable information for developing strategies that will improve testability

of CCD and other CDA standards, including QRDA.

9

 HL7 V3 RIM Definitions http://www.hl7.org/v3ballot/html/welcome/environment/index.html Normative

Vocabulary for the RIM, actClass, Procedure (PROC). Note: Access Requires download of the V3 Ballot

10 Lantana Consulting Group. CCD Coverage Report, May 2011. Related document prepared for NIST.

11 http://jira.siframework.org/wiki/display/SIF/CDA+Consolidation+Project

Lantana Consulting Group QRDA Suitability Analysis May 2011

Prepared for NIST Page 14

 2011, all rights reserved
Are there automated test tools and test suites?

NIST provides a test package that contains the Schematron rules for validating a document

against the guidelines specified in QRDA. Any off-the-shelf XML tool, such as XML Spy, can

apply the Schematron rules to validate QRDA instances. QRDA Schematron rules are not

currently available. 

Are there reference implementations?

The “Healthcare Information Technology Standards: General Suitability Analysis” describes two

types of implementations: reference implementations and real-world implementations. A

reference implementation is a fully instantiated software solution that is a reference for software

developers. There is no reference implementation for QRDA yet. 

QRDA has not yet been widely implemented. The Centers for Medicare and Medicaid Services

(CMS) has the only real-world implementation that we are aware of. In 2009, CMS contracted

Iowa Foundation for Medical Care (IFMC) to implement QRDA for the Physician Quality

Reporting System (PQRS).12 This implementation used the QRDA Category I (patient-level)

report and reused several CCD templates, such as Problems, Procedures, and Payers. The CMS

“Alternative Reporting Mechanisms: Physician Qualtiy Reporting System” web page lists several

related documents.13

Is there documented existence of errors, including estimates of the severity?

Errors exist in any standard; the question is whether the important ones have been found and

fixed. A good standard has lists of errors that contain only minor and no severe, known errors.

The lists of errors should include change requests and error logs, as well as errors documented in

published errata. A lack of a list of known errors may indicate that the standard has not been

widely enough implemented, or that errors discovered during implementation were not made

public.

HL7 maintains a log of errors and comments on all DSTUs, including QRDA14. The log contains

open issues known at the time of publication and comments logged since then by NIST on

conformance test requirements.

The published CMS and IFMC implementation of QRDA did not mention any specific errors. It

did suggest two additional components and features for QRDA: (1) include Schematron files with

the QRDA publication for easier adoption of the standard by the user community; and (2)

generate OIDs for providers.

12 Velamuri S. QRDA-technology overview and lessons learned. J Healthc Inf Manag. 2010 Summer;

24(3):41-8. Note: PQRS was originally called the Physician Quality Reporting Initiative (PQRI), but it is

not the same as the PQRI Registry XML Specification; throughout this document, we use PQRS to refer to

CMS’ reporting system and PQRI for the XML specification.

13

 https://www.cms.gov/PQRI/20_AlternativeReportingMechanisms.asp (accessed March 2011)

14 http://www.hl7.org/dstucomments/showdetail.cfm?dstuid=39

Lantana Consulting Group QRDA Suitability Analysis May 2011

Prepared for NIST Page 15

 2011, all rights reserved
QRDA’s base CDA R2 standard does have documented errors, all of which are minor. The CDA

R2 Errata page on the HL7 wiki15 (last modified April 14, 2009) records sixteen errors. Seven of

the errors are related to the sample CDA instance released with the standard, and are not related to

the CDA R2 standard itself. The other nine errors are minor, such as the need for more

clarification or improved readability. The errors are easy to address and do not affect the overall

quality of the standard.

Is there a defined and effective process for handling errors?

HL7 maintains a log of errors and comments on all DSTUs, including QRDA. HL7 itself has a

well-defined error-reporting process. The QRDA standard recommends reusing CDA templates

wherever possible. The HL7 Structured Document Working Group (SDWG) maintains publicly

accessible errata wiki pages for the CDA R2 standard and for CCD. 

Do industry associations endorse the standard?

QRDA was adopted by the Healthcare Information Technology Standards Panel (HISTP). HITSP

C105 constrains QRDA to support the communication of patient-level quality data for analysis

and measurement.16 IHE includes QRDA in the IHE Performance Quality Report profile

development effort.17 In addition to the HITSP and IHE adoption, CMS’s definition of eMeasure

encompasses both the Health Quality Measure Format (HQMF)18 used in electronic specification

of the measure and the corresponding QRDA for the measure. The CMS Physician Quality

Reporting System (PQRS)19 accepts quality reports in QRDA Category I format.

Has the standard been implemented by a range of vendors?

As previously noted (“Are there reference implementations?”), the CMS/IFMC implementation

for the PQRS program is the only published real-world implementation of QRDA that we are

aware of. Even though QRDA is not yet implemented by a range of vendors, it was tested

successfully as part of the HITSP Interoperability Demonstration during the Integrating the

Healthcare Enterprise (IHE) Connectathon and Healthcare Information and Management Systems

Society (HIMSS) Interoperability Showcase.20 The reuse of CDA and CCD templates in QRDA

should lead to rapid implementation and development of QRDA.

15

 http://wiki.hl7.org/index.php?title=CDA_R2_Errata. (accessed March 2011)

16 HITSP. Patient Level Quality Data Component Using HL7 Quality Reporting Document Architecture

(QRDA Component. Version 1.0 (HITSP/C105). http://www.hitsp.org/ConstructSet_Details.aspx?

&PrefixAlpha=4&PrefixNumeric=105 (accessed March 2011).

17 http://wiki.ihe.net/index.php?title=Performance_Quality_Report

18 HL7 Version 3 Standard: Representation of the Health Quality Measures Format (eMeasure), Release 1,

Draft Standard for Trial Use, March 2010. http://www.hl7.org/v3ballot/html/domains/uvqm/uvqm.html

19 CMS Alternative Reporting Mechanisms: Physician Quality Reporting System Web site.

https://www.cms.gov/PQRI/20_AlternativeReportingMechanisms.asp (accessed March 2011)

20 http://www.ahima.org/downloads/pdfs/advocacy/AHIMAStandardsCertIFRResponseLetter_Final.pdf

Lantana Consulting Group QRDA Suitability Analysis May 2011

Prepared for NIST Page 16

 2011, all rights reserved
Is the standard used in more than one country?

QRDA was developed to meet the quality reporting use cases in the United States. Even though

QRDA itself has not been used in other countries, its parent standard, CDA R2, has been widely

adopted and implemented in many countries. As noted in the section on constraint mechanisms,

QRDA preserves the global applicability of the HL7 V3 standard. QRDA specifies a quality

reporting framework and uses the templated CDA strategy. Similar quality reporting

implementation guides for other realms could be developed following the same approach to meet

quality reporting needs in different countries. 

Is certification available for developers and architects?

If certification for a standard is available for developers and architects, it indicates that the

standard is relatively mature and that there is a proven, repeatable process for implementers to

successfully implement the standard. It also indicates that the certifying body considers the

standard important and stable enough to justify the expense of developing the certification

criteria.

HL7 has not offered certification for any of the specific CDA R2 implementation guides, nor is

any specific QRDA certification available; however, HL7 has offered a “Certified CDA

Specialist” certification test since January 2007. The number of certified HIT professionals each

year has been climbing, with close to 300 HIT professionals receiving CDA certification in the

past four years. 

The CDA Academy21 offers hands-on, weeklong CDA training in the US.

21

 http://www.cdaacademy.com

Lantana Consulting Group QRDA Suitability Analysis May 2011

Prepared for NIST Page 17

 2011, all rights reserved
QRDA Suitability Summary

The following matrix summarizes the results of applying the criteria to the QRDA standard. One

criterion is not applicable to QRDA; the standard meets most other criteria, with caveats for

testability and test tools. QRDA lacks a specific error-handling process, certification program,

and reference implementation.

Table 2: QRDA Criteria Matrix

Criteria Results Notes

Is the standard based on a stable, well-vetted

data model?

Yes

Does the standard have a clear, robust

vocabulary-binding syntax?

Yes

Does the standard support reusable modules,

such as templates or data types?

Yes

Does the standard have a well-defined

constraint mechanism?

Yes

Does the standard have a well-defined

extensibility mechanism?

Yes

Are there unambiguous definitions of what is

testable?

Yes Testable conformance statements are

needed. Current projects will further

clarify and document testability.

Are there automated test tools and test suites? Yes General test tools are available, but

QRDA Schematron or other rule sets are

required.

Are there reference implementations? No

Is there documented existence of errors,

including estimates of the severity?

No Open issues known at the time of QRDA

publication and NIST comments on

conformance test requirements are

documented. Minor errors are

documented for the parent standard CDA

R2. 

Is there a defined and effective process for

handling errors?

Yes HL7 has a well-defined process, and

there is a publicly accessible HL7 DSTU

comments page for QRDA.

Do industry associations endorse the

standard?

Yes Endorsed by HISTP and IHE. Uptake by

CMS.

Has the standard been implemented by a

range of vendors?

No To date, there is only one implementation

for QRDA Category I, but the

widespread use of its parent standard

CDA R2 should enable rapid

development.

Is the standard used in more than one

country?

NA The parent standard, CDA R2, is used in

more than one country.

Lantana Consulting Group QRDA Suitability Analysis May 2011

Prepared for NIST Page 18

 2011, all rights reserved
Criteria Results Notes

Is certification available for developers and

architects

No Certification is available for the parent

standard CDA R2.

Lantana Consulting Group QRDA Suitability Analysis May 2011

Prepared for NIST Page 19

 2011, all rights reserved
Meaningful Use Analysis

The QRDA standard specifies the framework for quality reporting. In the following sections, we

examine QRDA’s suitability for CMS-required data as defined in Stage 1 Meaningful Use

measure specifications. 

Stage 1 Meaningful Use requires that eligible professionals report on six total measures: three

required core measures (substituting alternate core measures where necessary) and another three

selected from 38 additional clinical quality measures (CQMs). All of the Stage 1 Meaningful Use

CQMs are endorsed by NQF. NQF has converted these measures from a paper-based format to an

electronic format through a “retooling” process using the HQMF standard (see below).

QRDA and eMeasure

HQMF is an HL7 standard for representing a quality measure in an electronic format22. Measures

that are specified using HQMF are called eMeasures. Both HQMF (eMeasure) and quality

reporting are components of a larger quality framework, as shown in the Quality Framework

figure below.Ideally, in this end-to-end framework, providers can push a button and import these

eMeasures into their EHRs. The eMeasures will then query the EHRs’ data repositories and

generate reports for internal use or for external reporting to quality organizations such as CMS.

The generated reports will be the corresponding QRDA reports for a measure. The eMeasure

and QRDA figure shows that HQMF (eMeasure) specifies the measures that will query EHRs;

QRDA specifies how to report the queried results from EHRs. HQMF is not a CDA R2

implementation guide, but, rather, has a peer-to-peer relationship with CDA. 

eMeasures provide the rules for determining if a particular patient is included in a population,

such as initial patient population (IPP), denominator population (DENOM), or numerator

population (NUM). QRDA contains sufficient data elements to determine if the patient meets IPP,

DENOM, or NUM criteria, so a quality organization can do its own aggregation. QRDA lets sites

send quality reports that have individual patient-level or aggregate-level data. For a given

eMeasure, it is possible to generate different QRDA reports based on different reporting goals.

The base CDA R2 standard for QRDA provides a wide scope of coupling mechanisms between

eMeasures and QRDA. 

22 HL7 Version 3 Standard: Representation of the Health Quality Measures Format (eMeasure), Release 1,

Draft Standard for Trial Use, March 2010. http://www.hl7.org/v3ballot/html/domains/uvqm/uvqm.html

Lantana Consulting Group QRDA Suitability Analysis May 2011

Prepared for NIST Page 20

 2011, all rights reserved
Figure 2: Quality framework

Figure 3: eMeasure and QRDA

Lantana Consulting Group QRDA Suitability Analysis May 2011

Prepared for NIST Page 21

 2011, all rights reserved
QDM Building-Block Approach to eMeasures

In 2010, Lantana worked with NQF to develop a building-block approach to consistently

construct eMeasures. 

Reusable building blocks are derived from the Quality Data Model (QDM, formerly known as

Quality Data Set) defined by the Health Information Technology Expert Panel (HITEP). The

QDM clearly defines concepts used in quality measures and clinical care; it automates EHR use.23

QDM Version 2.124 describes the data elements and their context in four levels of information:

standard elements, quality data types, quality data elements, and data flow attributes. A standard

element such as diagnosis of diabetes takes on additional meaning when used with a quality data

type, such as active diagnosis, to form a quality data element active diabetes diagnosis (see the

Quality Data Model figure).

Figure 4: Quality Data Model

QDM Version 2.1 lists about 80 quality data types and quality data attributes for measure

development. These quality data types include diagnoses, procedures, findings, medications

ordered, care plans, et cetera. We converted each quality data type and quality data attribute in the

QDM into an XML pattern modeled on the HL7 V3 RIM. Coupled with a code list, the quality

data type becomes a quality data element. We assembled data criteria (using Boolean and other

logical and numeric operators) into population criteria, thereby creating a formal and computerprocessable representation of a quality measure. See the Building-block approach to eMeasures

figure below. 

We used this building block approach to develop all of the Stage 1 Meaningful Use clinical

quality measures—such as NQF 0421 Adult Weight Screening and Follow Up and NQF 0028

Tobacco Use Assessment and Tobacco Cessation Intervention. As an example, active diagnosis of

pregnancy is part of the exclusion criteria in NQF 0421. To represent it in eMeasure, we bound

the XML pattern that was developed for the quality data type of active diagnosis to the SNOMED

code for pregnancy, and turned the pattern into a quality data element for active diagnosis:

23 http://www.qualityforum.org/Projects/h/QDS_Model/Quality_Data_Model.aspx (accessed

March 2011). Note: this discussion is based on QDM Version 2.1; NQF has published Version 3.0, with a

comment period from April 20, 2011 to May 26, 2011. Version 3.0 changes some of the terminology for

elements and data types, but the concepts remain the same. Page 5 of the Version 3.0 overview document

(avaliable from the above link) lists the changes from 2.1. 

24 National Quality Forum, Quality Data Set Model, Version 2.1, September 2010. Available at

http://www.qualityforum.org/Projects/h/QDS_Model/Quality_Data_Model.aspx

Lantana Consulting Group QRDA Suitability Analysis May 2011

Prepared for NIST Page 22

 2011, all rights reserved
pregnancy. We then assembled the quality data element active diagnosis: pregnancy with other

data criteria to form the exclusion criteria. 

Figure 5: Building-block approach to eMeasures

We can continue to use this QDM-based building–block approach to transform any future

Meaningful Use measures into eMeasures. On occasion, we may need to create additional

patterns when we encounter new quality data types for future measures. Currently, the pattern

library we developed along with NQF is able to support the 113 NQF retooled measures. 

All of the XML patterns in the library are based on the HL7 V3 RIM. Because of this, we assert

that each XML pattern that constructs eMeasures can be automatically converted to a

corresponding reusable CDA template, and each of the quality data elements can be automatically

converted to a CDA template instance data. EHRs process queries based on eMeasure criteria and

generate measure data. CDA templates can be derived from the corresponding patterns for the

eMeasure data criteria, and these CDA templates can represent generated measure data. The CDA

template instance data can then be sent in a QRDA report. 

The patterns in the library support construction of NQF’s 113 retooled measures to eMeasures.

Thus, we should be able to generate CDA templates based on these patterns. We can use these

CDA templates to represent all of the required data for the Stage 1 Meaningful Use clinical

quality measures and insert them in QRDA. These reusable CDA templates will also support

future CMS clinical quality reporting requirements for Meaningful Use. As we develop new

patterns to support future measures, new reusable CDA templates can be generated.

QRDA was developed before the HQMF standard and the QDM-based building-block approach

to eMeasure. As a result, there are inconsistencies in the representation of data elements between

a QRDA and a corresponding eMeasure. These need to be reconciled.

Lantana Consulting Group QRDA Suitability Analysis May 2011

Prepared for NIST Page 23

 2011, all rights reserved
QRDA and CCD

CCD contains some, but not necessarily all, of the data needed to determine whether a particular

patient meets the population criteria within a particular measure. QRDA Category I carries quality

data tailored to a specific measure or measure set. As such, QRDA and CCD partially overlap in

data content. QRDA reuses CCD templates wherever possible. The QRDA specification does not

currently specify how QRDA aligns with CCD or any other clinical document types; this should

be included in future releases.

QRDA and PQRI Registry XML Specification

The Physician Quality Reporting Initiative (PQRI) 2009 Registry XML Specification25 is the

Stage 1 Meaningful Use standard for quality reporting defined by the final rules26. Unlike QRDA,

which is derived from CDA R2 under the HL7 V3 paradigms, PQRI is a government-unique

standard—it is not a voluntary consensus standard. 

Both the PQRI Registry XML Specification and QRDA Category III support aggregate,

summary-level quality reporting. The PQRI specification does not support patient-level measure

data reporting; it is not the same as the CMS Physician Quality Reporting System (PQRS,

formerly known as the PQRI program)27.. The PQRS program accepts both aggregated quality

reporting through the PQRI specification and individual patient-level quality reporting through

QRDA Category I format. 

Even though Stage 1 Meaningful Use requires only aggregate data, it is very likely that future

Meaningful Use measures will require patient-level data. QRDA supports both patient-level and

aggregate-level quality reports; the PQRI specification will not be able to meet this future need

without significant improvements. As mentioned earlier, the Meaningful Use measures have been

retooled and expressed in HQMF eMeasures. The ultimate goal is to turn these eMeasures into

automatic queries on an EHRs’ data repositories and to generate quality reports for the measures.

The PQRI specification is not compatible with the HQMF standard, which will make it more

difficult to generate reports directly based on eMeasures. The building-block approach to

eMeasures and the coupling between the XML patterns and CDA templates provides significant

advantages to QRDA over PQRI for patient-level quality reporting. An EHR that is certified for

Stage 1 Meaningful Use can generate an electronic summary report in CCD format; it would be a

familiar task for the same EHR to generate a QRDA report based on the CDA templates.

The PQRI Registry XML Specification and QRDA Category III are not comparable, although

they both provide summary-level data. If we applied our general suitability criteria to the PQRI

Registry XML Specification, it would not meet most of the criteria. For example, it was not

25 PQRI 2009 Registry XML Specifications

http://www.cms.gov/PQRS//downloads/PQRI2009RegistryXMLSpecsFinal508.pdf

26 Department of Health and Human Services. Health Information Technology: Initial Set of Standards,

Implementation Specifications, and Certification Criteria for Electronic Health Record Technology; Final

Rule, 45 CFR Part 170. http://edocket.access.gpo.gov/2010/pdf/2010-17210.pdf

27 CMS Alternative Reporting Mechanisms: Physician Quality Reporting System Web site.

https://www.cms.gov/PQRI/20_AlternativeReportingMechanisms.asp (accessed March 2011)

Lantana Consulting Group QRDA Suitability Analysis May 2011

Prepared for NIST Page 24

 2011, all rights reserved
developed based on a stable and well-vetted data model; it is based on a simple XML schema.

The data elements use simple data types, such as number or character. It does not have a clearly

defined, robust vocabulary binding syntax. 

In summary, the PQRI specification is analogous to QRDA Category III only on the surface. The

PQRI specification does not support patient-level quality reporting and does not align well with

the eMeasure specification. QRDA couples with the eMeasure specification nicely and supports

both patient-level and aggregate summary-level quality reporting, although QRDA Category III is

currently in comments-only status and needs to be fully specified and balloted through the HL7

balloting process. 

QRDA Ballot Consideration

QRDA Category I is an HL7 DSTU standard; Category II and Category III are in HL7 commentsonly status. QRDA Category II and III are not fully specified and have not yet been balloted, so

there is no way to implement them consistently, test conformance, or constrain them formally. 

A ballot for QRDA Category II and Category III is much needed so that we can define and

specify them fully and obtain and address feedback from the healthcare and healthcare IT

communities. Many people are working toward the goal of Stage 1 Meaningful Use reporting, so

we expect that we will have a much better understanding of the use cases for Category II and III.

As more institutions and vendors gain experience in reporting aggregate data, new requirements

and new use cases will surface and they must be carefully reviewed through the ballot process. 

Lantana Consulting Group QRDA Suitability Analysis May 2011

Prepared for NIST Page 25

 2011, all rights reserved
Conclusions

The QRDA standard is relatively new; none of our interviewees has experience implementing it.

Even though QRDA does not fully meet some of our criteria, it fits its purpose of reporting

quality measures. QRDA is robust because of the underlying, well-accepted HL7 CDA standard

and because it can represent elements in a consistent manner through vetted templates. QRDA is

also well suited for current and future data requirements of Meaningful Use. This suitability is a

result of the tight coupling between eMeasures and QRDA reports and the QDM-based buildingblock approach of eMeasures. The building blocks—the XML patterns—can be automatically

converted to their corresponding reusable CDA templates. The coherent framework created by the

coupling of eMeasures and QRDA could lead to automatic queries of EHRs and generation of

reports.

Enhancements to QRDA; improvements in testability, testing and validation, and error handling;

and the creation of a certification program and reference implementation should encourage a wide

range of vendors to implement the standard. QRDA Category I, which handles patient-level

reporting, needs Schematron rules for testing. For aggregate-level quality reporting, QRDA

Categories II and III must be fully specified and balloted to become a standard for real-life

implementation. Inconsistencies in data representation in some QRDA and corresponding

eMeasures must be reconciled. Descriptions of how QRDA aligns with CCD are also needed.

In summary, QRDA suits its purpose and satisfies the Meaningful Use criteria. With

improvements in the areas we identified, QRDA holds great potential to become a single standard

that supports all patient- and aggregate-level quality reporting. 

Lantana Consulting Group QRDA Suitability Analysis May 2011

Prepared for NIST Page 26

 2011, all rights reserved
References

• AHIMA comment letter on ONC’s Initial Set of Standards, Implementation

Specifications, and Certification Criteria for Electronic Health Record Technology

Interim Final Rule (IFR), as published in the January 13, 2010 Federal Register (45 CFR

Part 170).

http://www.ahima.org/downloads/pdfs/advocacy/AHIMAStandardsCertIFRResponseLett

er_Final.pdf

• CDA Academy. http://www.cdaacademy.com

• CDA R2 Errata – HL7 Wiki Web site. http://wiki.hl7.org/index.php?

title=CDA_R2_Errata. (accessed March 2011)

• CMS Alternative Reporting Mechanisms: Physician Quality Reporting System Web site.

https://www.cms.gov/PQRI/20_AlternativeReportingMechanisms.asp (accessed March

2011)

• Department of Health and Human Services. Health Information Technology: Initial Set of

Standards, Implementation Specifications, and Certification Criteria for Electronic

Health Record Technology; Final Rule, 45 CFR Part 170.

http://edocket.access.gpo.gov/2010/pdf/2010-17210.pdf

• Dolin RH, Alschuler L, Boyer S, Beebe C, Behlen FM, Biron PV, Shabo A, (Editors).

HL7 Clinical Document Architecture, Release 2.0. ANSI-approved HL7 Standard; May

2005. Ann Arbor, Mich.: Health Level Seven, Inc. Available through HL7

(http://www.hl7.org) or if an HL7 member with the following link:

http://www.hl7.org/documentcenter/private/standards/cda/r2/cda_r2_normativewebeditio

n.zip.

• HITSP. Patient Level Quality Data Component Using HL7 Quality Reporting Document

Architecture (QRDA Component. Version 1.0 (HITSP/C105).

http://www.hitsp.org/ConstructSet_Details.aspx?&PrefixAlpha=4&PrefixNumeric=105

(accessed March 2011).

• HL7 CDA R2 Errata wiki page. http://wiki.hl7.org/index.php?title=CDA_R2_Errata

• HL7 Implementation Guide for CDA Release 2 Quality Reporting Document Architecture

(QRDA) Draft Standard for Trial Use March 2009. Available at:

http://www.hl7.org/documentcenter/ballots/2008sep/downloads/CDAR2_QRDA_R1_DS

TU_2009APR.zip

• HL7 International Localization Task Force Report Version 2b. http://www.eosiris.it/documenti/s_HL7/HL7IntlLocalizationReport_v2b.doc (accessed March 2011)

• HL7 V3 RIM Definitions: Normative Vocabulary for the RIM, actClass, Procedure

(PROC). http://www.hl7.org/v3ballot/html/welcome/environment/index.html

Note: Access requires download of the V3 Ballot.

• HL7 Version 3 Publishing Facilitator’s Guide.

http://www.hl7.org/v3ballot/html/help/pfg/pfg.html

Lantana Consulting Group QRDA Suitability Analysis May 2011

Prepared for NIST Page 27

 2011, all rights reserved
• HL7 Version 3 Standard: Representation of the Health Quality Measures Format

(eMeasure), Release 1, Draft Standard for Trial Use, March 2010.

http://www.hl7.org/v3ballot/html/domains/uvqm/uvqm.html

• IHE Performance Quality Report Wiki Web site. http://wiki.ihe.net/index.php?

title=Performance_Quality_Report

• Lantana Consulting Group. CCD Coverage Report, May 2011. Related document

prepared for NIST.

• Lantana Consulting Group. Healthcare Information Technology Standards: General

Suitability Analysis, May 2011. Related document prepared for NIST.

• Lantana Consulting Group. Templated CDA: Key Concept for Interoperability, May

2011. Related document prepared for NIST.

• Meaningful Use Announcement Website. http://healthit.hhs.gov/portal/server.pt?

open=512&objID=2996&mode=2 (accessed March 2011)

• National Cancer Institute Center for Biomedical Informatics and Information Technology

(NCI CBIIT). CBIIT Sevice-Aware Interoperability Framework (SAIF) Implementation

Guide, March 10, 2011 Working Draft. https://wiki.nci.nih.gov/display/SAIF/3.1+-

+Overview+of+CBIIT+SAIF+Information+Framework?focusedCommentId=39297400

(accessed March 2011)

• National Quality Forum, Quality Data Model Web site.

http://www.qualityforum.org/Projects/h/QDS_Model/Quality_Data_Model.aspx

(accessed March 2011).

• National Quality Forum, Quality Data Set Model, Version 2.1, Septmeber 2010.

http://www.qualityforum.org/Projects/h/QDS_Model/Quality_Data_Model.aspx

• Office of the National Coodinator Standards and Interoperability CDA Consolidation

Project. http://jira.siframework.org/wiki/display/SIF/CDA+Consolidation+Project

• PQRI 2009 Registry XML Specifications

http://www.cms.gov/PQRS//downloads/PQRI2009RegistryXMLSpecsFinal508.pdf

• Velamuri S. QRDA-technology overview and lessons learned. J Healthc Inf Manag. 2010

Summer; 24(3):41-8.

Lantana Consulting Group QRDA Suitability Analysis May 2011

Prepared for NIST Page 28

 2011, all rights reserved
Acronyms and Abbreviations

AAFP American Academy of Family Physicians

ANSI American National Standards Institute

CBIIT (National Cancer Institute) Center for Biomedical Informatics and Information

Technology

CCD Continuity of Care Document

CCR Continuity of Care Record 

CDA Clinical Document Architecture

CDC Centers for Disease Control and Prevention

CMS Centers for Medicare and Medicaid Services

CNE coded no extensions

CQM clinical quality measure

CWE coded with extensions

DENOM denominator population

DSTU Draft Standard for Trial Use

EHR Electronic Health Record

HHS US Department of Health and Human Services

HIMSS Healthcare Information and Management Systems Society

HIT healthcare information technology

HITEP Health Information Technology Expert Panel

HITSP Healthcare Information Technology Standards Panel

HL7 Health Level Seven

HQMF Health Quality Measure Format

IFMC Iowa Foundation for Medical Care

IHE Integrating the Healthcare Enterprise

IPP initial patient population

NCI CBIIT National Cancer Institute Center for Biomedical Informatics and Information

Technology

NCQA National Committee for Quality Assurance

Lantana Consulting Group QRDA Suitability Analysis May 2011

Prepared for NIST Page 29

 2011, all rights reserved
NIST National Institute of Standards and Technology

NQF National Quality Forum

NUM numerator population

ONC Office of National Coordinator

PQRI Physician Quality Reporting Initiative

PQRS Physician Quality Reporting System (CMS system that accepts both QRDA and PQRI

Registry XML input; it was originally called the Physician Quality Reporting

Initiative but is not the same as the PQRI Registry XML Specification)

QDM Quality Data Model (QDM, formerly known as Quality Data Set)

QRDA Quality Reporting Document Architecture

R2 Release 2

RIM Reference Information Model

S&I Standards and Interoperability 

SAIC Science Applications International Corporation

SAIF Service-Aware Interoperability Framework

SDWG Structured Document Working Group

V3 Version 3

XML Extensible Markup Language

Lantana Consulting Group QRDA Suitability Analysis May 2011

Prepared for NIST Page 30

 2011, all rights reserved