NIST GCR 11-939 

Quality Reporting Data Architecture (QRDA) Standards

Action Plan

Lantana Consulting Group 
NIST GCR 11-939

Quality Reporting Data Architecture (QRDA) Standards

Action Plan

Prepared for 

 National Institute of Standards and Technology

Gaithersburg Md 20899-8202

By

Lantana Consulting Group

June 2011 

U.S. Department of Commerce 

John E. Bryson, Secretary

National Institute of Standards and Technology 

Patrick D. Gallagher, Under Secretary for Standards and Technology and Director

This publication was produced as part of contract SB1341-10-SE-0911 with the National Institute of Standards and

Technology. The contents of this publication do not necessarily reflect the views or policies of the National Institute of

Standards and Technology or the US Government.
Quality Reporting Data Architecture (QRDA) 

Standards Action Plan

June 2011

Prepared by Lantana Consulting Group

for

National Institute of Standards and Technology

© 2011 Lantana Consulting Group

All rights reserved.
Lantana Consulting Group

PO Box 177

East Thetford, VT 05043

www.lantanagroup.com

Liora Alschuler

Chief Executive Officer

liora.alschuler@lantanagroup.com

Bob Dolin, MD, FACP

President and Chief Medical Officer

bob.dolin@lantanagroup.com

Lauren Wood, Ph.D.

Senior Project Manager

lauren.wood@lantanagroup.com

Gaye Dolin, RN, MSN

Senior Nurse Informaticist

gaye.dolin@lantanagroup.com

Jingdong Li, MD

Chief Architect

jingdong.li@lantanagroup.com

Yan Heras

Senior Medical Informaticist

yan.heras@lantanagroup.com

Bob Yencha

Principal Consultant

bob.yencha@lantanagroup.com

Susan Hardy

Senior Technical Editor

susan.hardy@lantangroup.com

Lantana Consulting Group QRDA Standards Action Plan June 2011

Prepared for NIST FINAL Page 4

 2011, all rights reserved
Table of Contents

EXECUTIVE SUMMARY................................................................................................ 4

INTRODUCTION............................................................................................................. 5

RECOMMENDED ACTIONS.......................................................................................... 6

Enhancements to the Standard.....................................................................................................6

QRDA and eMeasures ....................................................................................................... 6

QRDA Category II and Category III...................................................................................7

QRDA Category I............................................................................................................... 7

Testability.................................................................................................................................... 8

Testing and Validation................................................................................................................. 9

Error Handling.......................................................................................................................... 10

Certification.............................................................................................................................. 10

CONCLUSIONS..............................................................................................................11

REFERENCES................................................................................................................ 12

ACRONYMS AND ABBREVIATIONS......................................................................... 13

Lantana Consulting Group QRDA Standards Action Plan June 2011

Prepared for NIST FINAL Page 5

 2011, all rights reserved
Executive Summary

This document is the action plan to improve the suitability of Health Level Seven (HL7)

Implementation Guide for CDA Release 2 – Quality Reporting Document Architecture (QRDA)

Release 1 for quality reporting. Lantana Consulting Group prepared this report for the National

Institute of Standards and Technology (NIST) to accompany the related “QRDA Suitability

Analysis”.

While we believe that QRDA is suitable for Meaningful Use (MU) quality reporting, we

identified areas in which the standard should be improved. This report discusses the actions to be

taken to improve QRDA in these areas: enhancements to the standard itself; improvements in

testability, testing, and validation, and error handling; and the creation of a certification program

and a reference implementation. QRDA needs to be more fully aligned with the National Quality

Forum’s Quality Data Model and HL7’s Health Quality Measures Format (eMeasures) standard.

QRDA Category II and III must be fully specified and balloted. Better definitions of testable

conformance statements are needed. Specific Schematron rules and a reference implementation

can improve testing and validation. A QRDA-specific certification will further encourage the

adoption of the QRDA standards. 

If acted upon, these recommendations will enhance QRDA’s potential to become a single standard

that supports all quality reporting use cases.

Lantana Consulting Group QRDA Standards Action Plan June 2011

Prepared for NIST FINAL Page 6

 2011, all rights reserved
Introduction

This document is the action plan to enhance the suitability of Quality Reporting Document

Architecture (QRDA) for quality measure reporting as required by the Department of Health and

Human Services’ Stage 1 Meaningful Use. Lantana Consulting Group prepared this report for the

National Institute of Standards and Technology (NIST) to accompany the related “QRDA

Suitability Analysis”1.

This plan recommends actions for all improvements identified in the “QRDA Suitability

Analysis”. These actions will solve the issues discussed in the suitability analysis and they may

also encourage more rapid adoption of the QRDA standard.

We do not duplicate our analysis in this document, but do refer to it wherever applicable. It must

be decided who will manage the action plan and which (if any) of the items should be carried out

and by whom.

Any mention of commercial products or organizations in this report is for information only; it

does not imply recommendation or endorsement by Lantana or NIST.

1

 Both QRDA reports were prepared under contract number SB134110SE0911.

Lantana Consulting Group QRDA Standards Action Plan June 2011

Prepared for NIST FINAL Page 7

 2011, all rights reserved
Recommended Actions

Our recommendations for the QRDA standard fall into five general categories: enhancements to

the standard, testability, testing and validation, error handling, and certification.

Enhancements to the Standard

Enhancements to QRDA will better support its goals and align it with the Continuity of Care

Document (CCD)2, Quality Data Model (QDM)3, and the HL7 Health Quality Measures Format

(HQMF) eMeasures standard4. We recommend the following actions:

QRDA and eMeasures 

Improve the QRDA implementation guide to be more prescriptive and describe exactly how it

aligns with CCD, QDM, and eMeasure.

The QRDA and eMeasure standards are two important components of a larger quality

framework. The eMeasure specification formalizes the representation of the quality

measure, and QRDA formalizes the output of patient quality data. QRDA was developed

before the eMeasure specification and the QDM-based building-block approach to

eMeasure. Hence, there are some inconsistencies that need to be reconciled including: 1)

misalignment between the representation of patient data in a QRDA vs. the representation

of corresponding criteria in an eMeasure; and 2) misalignment between the representation

of patient data in QRDA vs. CCD documents. 

Build eMeasures and QRDAs in parallel and to be consistent with the capabilities inherent in

Meaningful Use-certified EHRs. This notion is consistent with the perspective from Centers for

Medicare and Medicaid Services (CMS). The upcoming version of the “CMS Measures and

Measure Management Systems Blueprint” has an eMeasure Specifications chapter5. The

eMeasure Specifications contain both the specification of a measure and its corresponding

transmitting formats, including QRDA Category I. 

Refine QRDA and use the HL7 balloting process to make the QRDA and HQMF implementation

guides as consistent as possible. An HL7 HQMF US Realm Implementation Guide is currently

under development; it describes the QDM-based building-block approach for developing an

eMeasure in detail. We need to align QRDA while balloting both guides through HL7.

2 HL7 Implementation Guide: CDA Release 2 – Continuity of Care Document (CCD): A CDA

implementation of ASTM E2369-05 Standard Specification for Continuity of Care Record© (CCR) April

01, 2007 available through HL7: http://www.hl7.org

3

 National Quality Forum, Quality Data Model Web site.

http://www.qualityforum.org/Projects/h/QDS_Model/Quality_Data_Model.aspx

4 HL7 Version 3 Standard: Representation of the Health Quality Measures Format (eMeasure), Release 1,

Draft Standard for Trial Use, March 2010.

http://www.hl7.org/v3ballot/html/domains/uvqm/uvqm.html

5

 CMS Measures and Measure Management Systems Blueprint, Version 8.0. To be released in fall, 2011.

Lantana Consulting Group QRDA Standards Action Plan June 2011

Prepared for NIST FINAL Page 8

 2011, all rights reserved
QRDA Category II and Category III

Push the standard development effort for QRDA Category II and Category III to provide a single

standard that meets all quality reporting use cases.

Ballot QRDA Category II and Category III in a timely manner for potential adoption by the Stage

2 Meaningful Use.

QRDA Category I is an HL7 Draft Standard for Trial Use (DSTU), but Category II and

Category III are in HL7 comments-only status. The CMS Physician Quality Reporting

System (PQRS) currently accepts submission of both summary- and patient-level data

through both the Physician Quality Reporting Initiative (PQRI) Registry XML

Specification and QRDA Category I format, respectively. It would be better if all input

comes through a single standard: QRDA; different standards within the same quality

program will cause unnecessary overhead to the quality programs and participants. 

As stakeholder interests in QRDA continue to grow with the Stage 1 Meaningful Use

quality measure reporting requirements, we anticipate that new requirements and new use

cases will surface, which should be carefully reviewed through the ballot process. 

QRDA Category I

Update the QRDA Category I specification to align with the templates developed through the

Clinical Document Architecture (CDA) consolidation project6 for consistency across all CDA

implementation guides. 

Describe how to define a QRDA Category I report using the same QDM-based building-block

approach for developing eMeasures. Defining both QRDA reports and eMeasures based on the

QDM maximizes the coupling between the two specifications. 

Distinguish QRDA Category I from CCD/Continuity of Care Record (CCR). Both CCD and CCR

carry single-patient data for transition of care. As a result, CCD and CCR contain some, but not

all, of the data needed to determine whether or not a particular patient meets the population

criteria within a particular measure. For instance, the Meaningful Use Emergency Department

measures assess wait time in the Emergency Department (ED) for those patients who are

subsequently admitted to the hospital. Typical summary documents include a list of

hospitalizations, possibly with dates for each hospitalization, but would not contain details on ED

arrival time, decision to admit time, etc. On the other hand, QRDA Category I carries quality data

tailored to a specific measure or measure set. As such, QRDA and CCD/CCR overlap

considerably in data content. As more quality measures are retooled to become eMeasures,

quality reporting will require data elements that are with more variety and broader extent. QRDA

Category I has the capability of supporting such growth, while CCD/CCR will be limited by their

intended purposes. 

Promote the adoption of quality reporting tools such as popHealth—an open source Meaningful

Use Clinical Quality Measure (CQM) reporting tool7. popHealth accepts patient data via either

6 CDA Consolidation Project. http://jira.siframework.org/wiki/display/SIF/CDA+Consolidation+Project

7popHealth – An open source quality measure reference implementation. http://projectpophealth.org

Lantana Consulting Group QRDA Standards Action Plan June 2011

Prepared for NIST FINAL Page 9

 2011, all rights reserved
the Healthcare Information Technology Standards Panel (HITSP) C32 or CCR XML standards. It

calculates whether the patient meets numerator or denominator criteria for 44 Meaningful Use

outpatient ambulatory measures. With simple modification, popHealth will import QRDA

Category I instances, make calculations, apply appropriate validations, and submit aggregate

quality reports using QRDA Category II or Category III. Tools such as popHealth allow CMS to

collect aggregate rather than individual data and let CMS verify that the EHR is correctly

computing populations. 

Testability

QRDA specifies the framework for quality reporting and reuses the CDA templates where

possible. QRDA’s conformance statements constrain the CDA R2 standard and define the QRDA

payload. Testability can be improved by refining these conformance statements as defined in the

“CCD Coverage Report”8and creating a template library for QRDA data elements as defined in

“Templated CDA: Key Concept for Interoperability”

9

. 

We recommend the following actions:

Re-write QRDA conformance statements into multiple, discrete conformance statements with a

consistent style, each of which are individually testable.

Develop a standard style of prose guidance when discrete testable conformance statements

require additional guidance for untestable real-world requirements.

Create a template library for QRDA data elements; the template library for QRDA should be part

of the centralized template library for all CDA templates.

The CDA Consolidation Project10 is addressing conformance and testability. Should future

iterations of QRDA adopt this consistent approach, it will simplify the definition of what is

testable and the requirements for validation and run-time validation versus certification.

Templated CDA is a key technology underlying QRDA and other HL7 CDA implementation

guides. Such template libraries (databases) support both standards development and

implementation. A template library will improve quality assurance, consistency, and Schematron

testing. 

8 Lantana Consulting Group. CCD Coverage Report, May 2011. Related document prepared for NIST.

9

 Lantana Consulting Group. Templated CDA: Key Concept for Interoperability, May 2011. Related

document prepared for NIST.

10 http://jira.siframework.org/wiki/display/SIF/CDA+Consolidation+Project

Lantana Consulting Group QRDA Standards Action Plan June 2011

Prepared for NIST FINAL Page 10

 2011, all rights reserved
Testing and Validation

Schematron provides syntax validation at several levels of syntactic correctness. Implementers

rely on Schematron to determine the correctness of QRDA instance generation. A better review

process for Schematron rule development and creation of a reference implementation will

improve testing and validation. We recommend the following actions:

Develop and publish Schematron rules along with the QRDA standard. Many off-the-shelf XML

tools can then apply these Schematron rules to validate QRDA instances. 

Establish a Healthcare Information Technology (HIT) community review and consensus process

as part of the development of Schematron rules so that they can be publicly vetted. 

Consider using other testing technologies to improve instance testing as identified in the “CCD

Coverage Report”, such as model-driven validation and RELAX NG. 

Create one or more QRDA reference implementations. Reference implementations could be built

to QRDA Category I, Category II, and Category III specifications, when we further enhance the

Category I and fully specify QRDA Category II and Category III. Ideally, the reference

implementations should show how to implement QRDA-based quality reporting for measures that

are specified as HQMF eMeasures. 

Create a pilot real-world implementation for QRDA Category II and Category III, once they are

fully specified. CMS has implemented Category I for its reporting program: Physician Quality

Reporting System (PQRS). Lessons learned and experiences gained from the pilot real-world

implementation will improve the QRDA standard itself and spur the growth of adoption and

implementation rates of QRDA by a range of vendors.

Improve vocabulary validation. Terminology validation is an important piece of QRDA

validation. When a system receives a QRDA instance, it must validate the structural correctness

of the instance and whether a correct code is received for a coded data element. Codes from

dynamic vocabulary bindings are difficult to validate and require interface with terminology

services. Health care often uses dynamic value sets to support new knowledge and to supply

missing terms A single go-to terminology service based on the HL7 Version 3 Standard: Common

Terminology Services, Release 2 (CTS 2)11 would mitigate this testing problem. See the “CCD

Coverage Report” for a full discussion of this issue.

11 HL7 Version 3 Standard: Common Terminology Services, Release 2

http://www.hl7.org/documentcenter/public/standards/dstu/2009may/V3_CTS_R2_DSTU_2009OCT.pdf

Lantana Consulting Group QRDA Standards Action Plan June 2011

Prepared for NIST FINAL Page 11

 2011, all rights reserved
Error Handling

HL7 has a well-defined error-handling system. It maintains a log of errors and comments on all

DSTUs, including QRDA. After aligning QRDA with the QDM, the eMeasure standard, and

CCD, and then going through further HL7 ballots, a centralized errata page will be valuable. 

Certification

HL7 provides certification for the CDA standard, but not in varying depths, and there is no

certification for QRDA specifically. We recommend the following actions:

Develop a certification program for proficiency in both eMeasures and QRDA. Due to the tight

coupling between eMeasures and QRDA, a certification program for proficiency in both would be

very valuable. 

Create several depths to the certification program so that it can be tailored to different types of

audiences, such as measure developers and eMeasure/QRDA implementers. 

Lantana Consulting Group QRDA Standards Action Plan June 2011

Prepared for NIST FINAL Page 12

 2011, all rights reserved
Conclusions

QRDA suits its purpose for quality reporting. QRDA is robust because of the underlying, wellaccepted HL7 CDA standard and the ability to represent elements in a consistent manner through

vetted templates. The areas identified in this action plan, if acted upon, will better align QRDA

Category I with CCD, QDM, and eMeasure; fully specify Category II and Category III for

aggregate-level quality reporting; increase the rate of implementation; and prepare it for the

future. The recommended actions also call for improvement in testability, testing and validation,

and certification. 

In summary, QRDA holds great potential to become a single standard that supports all quality

reporting use cases—patient-level and aggregate-level—with the capability to grow. The QDMbased building-block approach bridges eMeasures and QRDA. The tight coupling between the

two specifications provides a coherent quality reporting framework, which could enable

automatic querying of EHRs based on eMeasures and generating and submitting reports in the

near future.

Lantana Consulting Group QRDA Standards Action Plan June 2011

Prepared for NIST FINAL Page 13

 2011, all rights reserved
References

CDA Consolidation Project.

http://jira.siframework.org/wiki/display/SIF/CDA+Consolidation+Project

Common Terminology Services, Release 2

http://www.hl7.org/documentcenter/public/standards/dstu/2009may/V3_CTS_R2_DSTU_2009O

CT.pdf

CMS Measures and Measure Management Systems Blueprint, Version 8.0. To be released in fall,

2011.

HL7 Implementation Guide: CDA Release 2 – Continuity of Care Document (CCD): A CDA

implementation of ASTM E2369-05 Standard Specification for Continuity of Care Record©

(CCR) April 01, 2007 available through HL7: http://www.hl7.org

HL7 Version 3 Standard: Representation of the Health Quality Measures Format (eMeasure),

Release 1, Draft Standard for Trial Use, March 2010.

http://www.hl7.org/v3ballot/html/domains/uvqm/uvqm.html 

Lantana Consulting Group. CCD Coverage Report, May 2011. Related document prepared for

NIST.

Lantana Consulting Group. Templated CDA: Key Concept for Interoperability, May 2011.

Related document prepared for NIST.

Lantana Consulting Group. QRDA Suitability Analysis, May 2011. Related document prepared

for NIST.

National Quality Forum, Quality Data Model Web site.

http://www.qualityforum.org/Projects/h/QDS_Model/Quality_Data_Model.aspx

popHealth – An open source quality measure reference implementation.

http://projectpophealth.org

Lantana Consulting Group QRDA Standards Action Plan June 2011

Prepared for NIST FINAL Page 14

 2011, all rights reserved
Acronyms and Abbreviations

CCD Continuity of Care Document

CCR Continuity of Care Record 

CDA Clinical Document Architecture

CMS Centers for Medicare and Medicaid Services

CQM clinical quality measure

HITSP Healthcare Information Technology Standards Panel

HL7 Health Level Seven

HQMF Health Quality Measures Format

IT Information Technology

MU Meaningful Use

NIST National Institute of Standards and Technology

PQRS Physician Quality Reporting System

QDM Quality Data Model (formerly known as Quality Data Set)

QRDA Quality Reporting Document Architecture

Lantana Consulting Group QRDA Standards Action Plan June 2011

Prepared for NIST FINAL Page 15

 2011, all rights reserved