Federal Register / Vol. 83, No. 165 / Friday, August 24, 2018 / Notices 42913 

permit evaluation of possible sources of 

conflicts of interest. 

A nomination package should include 

the following information for each 

nominee: 

(1) A letter of nomination from an 

employer, a colleague, or a professional 

organization stating the name, 

affiliation, and contact information for 

the nominee, the basis for the 

nomination (i.e., what specific 

attributes, perspectives, and/or skills 

does the individual possess that would 

benefit the workings of the NACNEP, 

and the nominee’s field(s) of expertise); 

(2) A letter of interest from the 

nominee stating the reasons they would 

like to serve on the NACNEP; 

(3) A biographical sketch of the 

nominee, a copy of his/her curriculum 

vitae, and his/her contact information 

(address, daytime telephone number, 

and email address); and 

(4) The name, address, daytime 

telephone number, and email address at 

which the nominator can be contacted. 

Nominations will be considered as 

vacancies occur on the NACNEP. If you 

submitted a nomination more than four 

years ago, please resubmit an updated 

nomination to be considered for council 

vacancies. 

HHS strives to ensure that the 

membership of HHS federal advisory 

committees are balanced in terms of 

points of view represented and the 

committee’s function. The Department 

encourages nominations of qualified 

candidates from all groups and 

locations. Appointment to the NACNEP 

shall be made without discrimination 

on the basis of age, race, ethnicity, 

gender, sexual orientation, disability, 

and cultural, religious, or 

socioeconomic status. 

Amy P. McNulty, 

Acting Director, Division of Executive 

Secretariat. 

[FR Doc. 2018–18344 Filed 8–23–18; 8:45 am] 

BILLING CODE 4165–15–P 

DEPARTMENT OF HEALTH AND 

HUMAN SERVICES 

Request for Information Regarding the 

21st Century Cures Act Electronic 

Health Record Reporting Program 

AGENCY: Office of the National 

Coordinator for Health Information 

Technology (ONC), HHS. 

ACTION: Request for information. 

SUMMARY: This request for information 

(RFI) seeks input from the public 

regarding the Electronic Health Record 

(EHR) Reporting Program established as 

Section 4002 of the 21st Century Cures 

Act (Cures Act) codified Section 3009A 

in Title XXX of the Public Health 

Service Act (PHSA). This RFI is a first 

step toward implementing the statute. 

Its responses will be used to inform 

subsequent discussions among 

stakeholders and future work toward the 

development of reporting criteria under 

the EHR Reporting Program. 

DATES: To be assured consideration, 

written or electronic comments must be 

received at one of the addresses 

provided below, no later than 5 p.m. on 

October 17, 2018. 

ADDRESSES: The public should address 

written comments on the proposed 

system of records to http://

www.regulations.gov or to the HHS 

Office of Security and Strategic 

Information (OSSI), 200 Independence 

Avenue SW, Washington, DC 20201. 

• Federal eRulemaking Portal: Follow 

the instructions for submitting 

comments. Attachments should be in 

Microsoft Word, Microsoft Excel, or 

Adobe PDF; however, we prefer 

Microsoft Word. 

• Regular, Express, or Overnight Mail: 

Department of Health and Human 

Services, Office of the National 

Coordinator for Health Information 

Technology, Attention: EHR Reporting 

Program Request for Information, Mary 

E. Switzer Building, Mail Stop: 7033A, 

330 C Street SW, Washington, DC 

20201. Please submit one original and 

two copies. 

• Hand Delivery or Courier: Office of 

the National Coordinator for Health 

Information Technology, Attention: EHR 

Reporting Program Request for 

Information, Mary E. Switzer Building, 

Mail Stop: 7033A, 330 C Street SW, 

Washington, DC 20201. Please submit 

one original and two copies. (Because 

access to the interior of the Mary E. 

Switzer Building is not readily available 

to persons without federal government 

identification, commenters are 

encouraged to leave their comments in 

the mail drop slots located in the main 

lobby of the building.) 

Enhancing the Public Comment 

Experience: To facilitate public 

comment on this RFI, a copy will be 

made available in Microsoft Word 

format on ONC’s website (http://

www.healthit.gov). 

Inspection of Public Comments: All 

comments received before the close of 

the comment period will be available for 

public inspection, including any 

personally identifiable or confidential 

business information that is included in 

a comment. Please do not include 

anything in your comment submission 

that you do not wish to share with the 

general public. Such information 

includes, but is not limited to: A 

person’s social security number; date of 

birth; driver’s license number; state 

identification number or foreign country 

equivalent; passport number; financial 

account number; credit or debit card 

number; any personal health 

information; or any business 

information that could be considered 

proprietary. We will post all comments 

that are received before the close of the 

comment period at http://

www.regulations.gov. 

Comments received timely will also 

be available for public inspection, 

generally beginning approximately 3 

weeks after publication of a document at 

Office of the National Coordinator for 

Health Information Technology, 330 C 

Street SW, Room 7033A, Washington, 

DC 20201. Contact Michael Wittie, 

listed below, to arrange for inspection. 

Docket: For access to the docket to 

read background documents or 

comments received, go to http://

www.regulations.gov or the Department 

of Health and Human Services, Office of 

the National Coordinator for Health 

Information Technology, Mary E. 

Switzer Building, Mail Stop: 7033A, 330 

C Street SW, Washington, DC 20201 

(call ahead to the contact listed below 

to arrange for inspection). 

FOR FURTHER INFORMATION CONTACT: 

Michael Wittie, Office of Policy, Office 

of the National Coordinator for Health 

Information Technology, 202–690–7151, 

Michael.Wittie@hhs.gov or Lauren 

Richie, Office of Policy, Office of the 

National Coordinator for Health 

Information Technology, 202–690–7151, 

Lauren.Richie@hhs.gov. 

SUPPLEMENTARY INFORMATION: 

I. Background 

The Secretary has delegated authority 

to the Office of the National Coordinator 

for Health Information Technology 

(ONC) to carry out the provisions of 

sections 4002(a) and 4002(c) of the 

Cures Act. Section 4002(a) creates PHSA 

section 3001(c)(5)(D) and instructs the 

Secretary to ‘‘require, as a condition of 

certification and maintenance of 

certification’’ that health IT developers 

satisfy certain requirements, including 

submitting ‘‘reporting criteria in 

accordance with section 3009A(b).’’ 

Section 4002(c) creates PHSA Section 

3009A and requires the Secretary to 

develop an ‘‘Electronic Health Record 

Reporting Program’’ (EHR Reporting 

Program or Program). Section 3009A 

also calls on the Secretary to lead a 

public, transparent process to establish 

the ‘‘reporting criteria’’ associated with 

the EHR Reporting Program. Section 

3009A directs the Secretary to award 


daltland on DSKBBV9HB2PROD with NOTICES
42914 Federal Register / Vol. 83, No. 165 / Friday, August 24, 2018 / Notices 

1For further discussion, see the DEFINITIONS 

FOR CERTIFIED HEALTH IT AND CEHRT section 

of the 2016 Report on the Feasibility of Mechanisms 

to Assist Providers in Comparing and Selecting 

Certified EHR Technology Products (https://

www.healthit.gov/sites/default/files/macraehrpct_

final_4-2016.pdf). 

2Understanding Certified Health IT. https://

www.healthit.gov/sites/default/files/understandingcertified-;health-it-2.pdf. 

3https://dashboard.healthit.gov/apps/healthinformation-technology-data-summaries.php?state=

National&cat9=all+data&cat1=ehr+adoption. 

4 In this RFI, all references to acquisition of 

certified health IT include purchasing, licensing, 

and other methods of obtaining technology. 

grants, contracts, or agreements to 

independent entities to support the EHR 

Reporting Program. For the purposes of 

this RFI and the Program, the term 

‘‘certified health IT’’ includes the full 

range of potential technologies, 

functions, and systems for which HHS 

has adopted standards, implementation 

specifications, and certification criteria 

under the ONC Health IT Certification 

Program.1 ONC will engage a contractor 

to convene stakeholders and use the 

responses to this RFI to inform 

stakeholder discussion in order to 

formally develop these criteria. 

The Cures Act requires the EHR 

Reporting Program’s reporting criteria to 

address the following five categories: 

Security; interoperability; usability and 

user-centered design; conformance to 

certification testing; and other 

categories, as appropriate to measure the 

performance of certified EHR 

technology. The Cures Act also suggests 

several other categories for 

consideration, including, but not 

limited to: Enabling users to order and 

view results of laboratory tests, imaging 

tests, and other diagnostic tests; 

exchanging data with clinical registries; 

accessing and exchanging data from 

medical devices, health information 

exchanges, and other health care 

providers; accessing and exchanging 

data held by federal, state, and local 

agencies. 

For the purposes of this RFI, we have 

focused our questions on the five 

mandatory categories from the Cures 

Act. However, the public is welcome to 

comment on any of the additional 

categories noted by the Cures Act 

(please consult section 3009A(a)(3)(B)). 

The ONC Health IT Certification 

Program 

The ONC Health IT Certification 

Program provides a process to support 

certifying health information technology 

(health IT) to the appropriate standards, 

implementation specifications, and 

certification criteria that have been 

adopted by the Secretary.2 As a result, 

since 2015, nearly all hospitals and 

most physicians used health IT certified 

under the ONC Health IT Certification 

Program.3 The 2015 Edition certification 

criteria is the most recent edition of 

certification criteria adopted by the 

Secretary for use in the ONC Health IT 

Certification Program. 

II. Solicitation of Comments 

This RFI includes two main sections 

for public comment: 

• Cross-cutting: Requests input on 

priorities on the intersection of health 

IT product-related reporting criteria and 

healthcare provider reporting criteria; 

and 

• Categories: Requests input on 

specific focus areas, including the 

reporting criteria categories required by 

the Cures Act. 

In reviewing the RFI questions, 

commenters should consider existing 

sources of information about health IT 

products. Commenters should also 

consider how reporting criteria for 

different stakeholders could be 

constructed based on their differing 

perspectives, especially, for example, 

since health IT developers will be 

required to respond to reporting criteria 

for their product(s) in order to maintain 

the product’s certification. To prevent 

duplication of efforts, commenters 

should consider what other information 

is lacking from the existing sources 

about health IT products and what the 

reporting criteria under the EHR 

Reporting Program could uniquely 

contribute. 

Overall, we seek input about reporting 

criteria that will be used to: 

• Show distinct, measurable 

differences between products; 

• Describe the functionalities of 

health IT products varying by the setting 

where implemented (e.g., primary 

versus specialty care); 

• Provide timely and reliable 

information in ways not unduly 

burdensome to users or to small and 

start-up developers; 

• Comparatively inform acquisition,4

upgrade, and customization decisions 

that best support end users’ needs 

beyond currently available information; 

and 

• Support analysis for industry trends 

with respect to interoperability and 

other types of user experiences. 

ONC is especially interested in 

feedback targeting users in ambulatory 

and small practice settings, where 

providers typically do not have 

substantial time and resources to 

conduct broad market research. To 

reduce data collection burden, ONC also 

seeks input on the availability and 

applicability of existing data sources 

that could be used to report on this 

information (e.g., the reporting criteria). 

Finally, ONC seeks input on the most 

efficient processes to minimize 

stakeholder burden for collecting and 

reporting the information. 

III. Cross-Cutting Topics 

Existing Data Sources: 

In 2016, ONC released the Report to 

Congress on the Feasibility of 

Mechanisms to Assist Providers in 

Comparing and Selecting Certified EHR 

Technology Products (EHR Compare 

Report). The report was based on market 

analysis and insight from subject matter 

experts including the ONC Certified 

Technology Comparison Task Force of 

the Joint Health IT Policy and Health IT 

Standards Committees. It described 

mechanisms for improving the health 

care community’s ability to compare 

and select certified health IT. The report 

identified and described existing 

sources of health IT comparison data, as 

well as gaps in the information available 

with possible mechanisms to address 

those gaps. The sources identified in the 

report are listed in Appendix A. 

ONC is interested in stakeholders’ 

input on currently existing sources of 

health IT comparison data as well as 

gaps in such information since the EHR 

Compare Report release. 

Questions: 

• Please identify any sources of 

health IT comparison information that 

were not in the EHR Compare Report 

that would be helpful as potential 

reporting criteria are considered. In 

addition, please comment on whether 

any of the sources of health IT 

comparison information that were 

available at the time of the EHR 

Compare Report have changed notably 

or are no longer available. 

• Which, if any, of these sources are 

particularly relevant or should be 

considered as they relate to certified 

health IT for ambulatory and small 

practice settings? 

Given the wide range of data that is 

reported to HHS and other agencies, we 

seek to avoid duplicate reporting 

through the EHR Reporting Program. We 

are interested in stakeholders’ input on 

information already available from 

health IT acquisition decision makers 

and users who report to Federal 

programs that could be re-used and 

factored into the EHR Reporting 

Program. We are particularly interested 

in any data reported by providers 

participating in Centers for Medicare & 

Medicaid Services (CMS) programs 

since they can be considered verified 

users of certified health IT. 

Questions: 


daltland on DSKBBV9HB2PROD with NOTICES
Federal Register / Vol. 83, No. 165 / Friday, August 24, 2018 / Notices 42915 

5Section 4003 of the Cures Act provides that 

interoperability: (1) Enables secure exchange and 

use of electronic health information without special 

effort on the part of the user; (2) allows for complete 

access, exchange, and use of all electronically 

accessible health information for authorized use; 

and (C) does not constitute information blocking. 

• What, if any, types of information 

reported by providers as part of their 

participation in HHS programs would 

be useful for the EHR Reporting Program 

(e.g., to inform health IT acquisition, 

upgrade, or customization decisions)? 

• What data reported to State 

agencies (e.g., Medicaid EHR Incentive 

Program data), if available nationally, 

would be useful for the EHR Reporting 

Program? 

Data Reported by Health IT 

Developers versus End-Users: 

User-reported data can help assess 

interoperability,5 the usability of 

information that is exchanged, and the 

accessibility of that information to end 

users. There may also be areas where it 

would be useful to obtain both 

qualitative end user experiences as well 

as qualitative information from the 

developers on the same aspect of a 

particular EHR Reporting Program 

criterion, such as interoperability. Such 

information may provide insights into 

how well a certified health IT product 

is performing from both perspectives. 

However, there may be criteria where 

developers, as opposed to acquisition 

decision makers and end users, would 

serve as the primary source of 

information. 

Questions: 

• What types of reporting criteria 

should developers of certified health IT 

report about their certified health IT 

products: 

Æ That would be important to use in 

identifying trends, assessing 

interoperability and successful 

exchange of health care information, 

and supporting assessment of user 

experiences? 

Æ That would be valuable to those 

acquiring health IT in making health IT 

acquisition, upgrade, or customization 

decisions that best support end users’ 

needs? 

• What types of reporting criteria for 

health care providers, patients, and 

other users of certified health IT 

products would be most useful in 

making technology acquisition, upgrade, 

or customization decisions to best 

support end users’ needs? 

• What kinds of user-reported 

information are health IT acquisition 

decision makers using now; how are 

they used in comparing systems; and do 

they remain relevant today? 

• What types of reporting criteria 

would be useful to obtain from both 

developers and end users to inform 

health IT comparisons? What about 

these types of reporting criteria makes 

them particularly amenable to reporting 

from both the developer and end user 

perspective? 

User-Reported Criteria: 

The Cures Act calls for collecting EHR 

Reporting Program reporting criteria 

information from health care providers, 

patients, and other users of certified 

EHR technology, as well as from 

developers. As addressed in the EHR 

Compare Report, there are currently 

private sector resources where users can 

provide and view reviews of health IT 

products. However, the resources may 

be improved upon because they are not 

comprehensive, reflective of verified 

users’ views, nor accessible and 

affordable to all. 

ONC is interested in input about what 

user-submitted information would make 

the EHR Reporting Program a valuable 

addition to the existing landscape of 

market research and analysis. ONC is 

also interested in feedback on what 

factors might influence end users’ 

decisions to report more easily. 

Questions: 

• How can data be collected without 

creating or increasing burden on 

providers? 

• What recommendations do 

stakeholders have to improve the 

timeliness of the data so there are not 

significant lags between its collection 

and publication? 

• Describe the value, if any, in an 

EHR Reporting Program function that 

would display reviews from existing 

sources, or provided a current list with 

hyperlinks to access them. 

• Discuss the benefits and limitations 

of requiring users be verified before 

submitting reviews. What should be 

required for such verification? 

• Which reporting criteria are 

applicable generally across all 

providers? What reporting criteria 

would require customization across 

different provider types and specialties, 

including small practices and those in 

underserved areas? 

• For what settings (e.g., hospitals, 

primary care physicians, or specialties) 

would comparable information on 

certified health IT be most helpful? If 

naming several settings, please list in 

your order of priority. 

• How helpful are qualitative user 

reviews (such as ‘star ratings’ or Likert 

scales) compared to objective reports 

(e.g., that a system works as expected 

with quantifiable measures)? Which 

specific types of information are better 

reflected in one of these formats or 

another? 

• How could HHS encourage 

clinicians, patients, and other users to 

share their experiences with certified 

health IT? 

• Which particular reporting 

mechanisms, if any, should be avoided? 

Health IT Developer-Reported 

Criteria: 

The Cures Act requires that health IT 

developers report information on 

certified health IT as a condition of 

certification and maintenance of 

certification under the ONC Health IT 

Certification Program. A common set of 

criteria reported by health IT developers 

could help acquisition decision makers 

compare across products to make more 

informed decisions that best support 

end users’ needs. Such reporting criteria 

could also be used to establish a 

consistent set of metrics to provide a 

baseline and identify trends over time in 

key focus areas associated with health 

IT use and interoperability. 

However, there may be information 

that uniform reporting criteria may not 

adequately reflect, particularly in health 

IT targeted towards smaller or 

specialized settings with specific needs. 

A mixed approach that blends common 

and optional sets of reporting criteria 

may better address the needs of 

providers and developers of varying 

sizes and settings. 

Questions: 

• If you have used the certified health 

IT product data available on the ONC 

Certified Health IT Products List (CHPL) 

to compare products (e.g., to inform 

acquisition, upgrade, or customization 

decisions), what information was most 

helpful and what was missing? If 

providing a brief list of the information, 

please prioritize the information from 

most helpful to least helpful also 

considering their grouping into 

categories in Section IV. 

• Would a common set of criteria 

reported on by all developers of 

certified health IT, or a mixed approach 

blending common and optional sets of 

criteria, be more effective as we 

implement the EHR Reporting Program? 

• What developer-reported criteria 

are particularly relevant, or not relevant, 

to health IT users and acquisition 

decision makers in the ambulatory and 

small practice settings? 

• Which criteria topics might be 

especially burdensome or difficult for a 

small or new developer to report on? 

• What types of criteria might 

introduce bias (e.g., unfair advantage) in 

favor of larger, established developers or 

in favor of small or new developers? 

• In what ways can different health IT 

deployment architectures be 

accommodated? For instance, are there 

certain types of criteria that cloud-based 


daltland on DSKBBV9HB2PROD with NOTICES
42916 Federal Register / Vol. 83, No. 165 / Friday, August 24, 2018 / Notices 

6 ISO 9241–11; https://www.iso.org/obp/ui/ 

#iso:std:iso:ts:20282:-2:ed-2:v1:en. 

7Friedberg M, et. al. Factors Affecting Physician 

Professional Satisfaction and Their Implications for 

Patient Care, Health Systems, and Health Policy. 

RAND Corporation, 2013. https://www.rand.org/ 

content/dam/rand/pubs/research_reports/RR400/ 

RR439/RAND_RR439.pdf. 

8NISTIR 7804–1, https://nvlpubs.nist.gov/ 

nistpubs/ir/2015/NIST.IR.7804-1.pdf. 

9ONC Health IT Playbook. https://

www.healthit.gov/playbook/electronic-healthrecords/#section-1-4. 

10Sinsky C, Colligan L, Li L, Prgomet M, 

Reynolds S, Goeders L, Westbrook J, Tutty M, Blike 

G. Allocation of Physician Time in Ambulatory 

Practice: A Time and Motion Study in 4 Specialties. 

Ann Intern Med. 2016 Dec 6;165(11):753–760. 

11Friedberg MW, Chen PG, Van Busum KR, 

Aunon F, Pham C, Caloyeras J, Mattke S, Pitchforth 

E, Quigley DD, Brook RH, Crosson FJ, Tutty M. 

Factors Affecting Physician Professional 

Satisfaction and Their Implications for Patient Care, 

Health Systems, and Health Policy. Rand Health Q. 

2014 Dec 1;3(4):1. 

12Certified Health IT Product List. https://

chpl.healthit.gov/#/search. 

13Medstar National Center for Human Factors in 

Healthcare. EHR User-Centered Design Evaluation 

Framework. https://www.medicalhumanfactors.net/ 

ehr-vendor-framework/. 

14Arndt BG, Beasley JW, Watkinson MD, Temte 

JL, Tuan WJ, Sinsky CA, Gilchrist VJ. Tethered to 

the EHR: Primary Care Physician Workload 

Assessment Using EHR Event Log Data and TimeMotion Observations. Ann Fam Med. 2017 

Sep;15(5):419–426. 

15Tai-Seale M, Olson CW, Li J, Chan AS, 

Morikawa C, Durbin M, Wang W, Luft HS. 

Electronic Health Record Logs Indicate That 

Physicians Split Time Evenly Between Seeing 

Patients And Desktop Medicine. Health Aff 

(Millwood). 2017 Apr 1;36(4):655–662.A. 

16Robert Wachter. The Digital Doctor. New York, 

NY: McGraw-Hill Education; 2015. 

certified health IT developers would be 

better able to report on versus those who 

are not cloud-based? How might this 

affect generating and reporting 

information on criteria? 

IV. Categories for the EHR Reporting 

Program 

The Cures Act requires the following 

categories to be addressed when it 

comes to EHR Reporting Program 

reporting criteria. Please consult the end 

of this RFI section for specific questions 

on the many other reporting criteria 

categories suggested by the Cures Act. 

• Security; 

• Usability and user-centered design; 

• Interoperability; 

• Conformance to certification 

testing; and 

• Other categories, as appropriate to 

measure the performance of certified 

EHR technology. 

Questions: 

• What categories of reporting criteria 

are end users most interested in (e.g., 

security, usability and user-centered 

design, interoperability, conformance to 

certification testing)? Please list by 

priority. 

Security: 

The ONC Health IT Certification 

Program supports the privacy and 

security of electronic health information 

by establishing a detailed set of 

requirements that health IT developers 

must meet for their products to be 

certified to the Privacy and Security 

certification criteria. Implementation of 

these capabilities can also help certified 

health IT users meet certain Health 

Insurance Portability and 

Accountability Act (HIPAA) compliance 

requirements. 

Questions: 

• What reporting criteria could 

provide information on meaningful 

differences between products in the ease 

and effectiveness that they enable end 

users to meet their security and privacy 

needs? 

• Describe other useful security and 

privacy features or functions that a 

certified health IT product may offer 

beyond those required by HIPAA and 

the ONC Health IT Certification 

Program, such as functions related to 

requirements under 42 CFR part 2. 

What information about a certified 

health IT product’s security and privacy 

capabilities and performance have 

acquisition decision makers used to 

inform decisions about acquisitions, 

upgrades, or use to best support end 

users’ needs? How has that information 

helped inform decision-making? What 

other information would be useful in 

comparing certified health IT products 

on security and privacy (e.g., 

compatibility with newer security 

technologies such as biometrics)? 

Usability and User-Centered Design: 

Usability centers on the extent to 

which a system supports a user to 

efficiently and effectively achieve their 

desired goals.6 Poor usability of health 

IT systems can contribute to clinician 

burden and physician burnout,7 and 

problems with usability may lead to 

risks to patient safety and end user 

error.8 9 Traditionally, usability 

assessment involves analyses of 

clinician workflow, including error rates 

and time spent on specific tasks. 

Usability assessments use methods that 

include conducting time-motion 

studies 10 and other qualitative 

measures that gather end users’ input 

about their experiences using the 

system.11

User-Centered Design (UCD) is a type 

of development process that can 

improve system usability. UCD 

considers users’ needs during each stage 

of system design and development, and 

is designed to lead to more usable end 

products. To have their products 

certified, health IT developers must 

attest that they employed a UCD process 

and report the results of usability testing 

on certain technical functions. The 

results—including measures such as 

time to perform certain tasks, the 

number of individuals used in the 

testing, and Likert scale scores that rate 

usability of the technical functions—are 

available on the CHPL.12 The UCD 

process implemented will likely vary 

based on the health IT functionality 

certified.13 Thus, it may be difficult to 

use certification results to assess and 

compare effective use of UCD across all 

certified health IT products. 

An important method for evaluating 

the usability of health IT products is 

through an analysis of information from 

users’ experiences and data in realworld settings. Recent studies14 15 have 

used audit logs to examine physicians’ 

time spent on specific tasks, and some 

cloud-based health IT systems have the 

capability to monitor this to optimize 

workflow.16 However, given that tasks 

and workflows may vary by specialty, 

setting, and other factors, it may be 

difficult to compare the results across 

systems. 

With qualitative assessments being a 

key part of assessing usability, 

subjective user assessments are 

complementary to quantitative 

measures, such as time to perform tasks. 

Information about the source of reviews, 

such as if a reviewer has actually used 

the system, their setting, specialty, and 

background (e.g., as a clinician, practice 

manager, etc.), may affect the value of 

the reviews to health IT acquisition 

decision makers. As noted in the EHR 

Compare Report, resources exist that 

provide user reviews, though these may 

be outdated. 

Questions: 

• How can the usability results 

currently available in the CHPL best be 

used to assist in comparisons between 

certified health IT products? 

• Describe the availability and 

feasibility of common frameworks or 

standard scores from established 

usability assessment tools that would 

allow acquisition decision makers to 

compare usability of systems. 

• Discuss the merits and risks of 

seeking a common set of measures for 

the purpose of real world testing that 

health IT developers could use to 

compare usability of systems. What 

specific types of data from current users 

would reflect how well the certified 

health IT product: 

Æ Supports the cognitive work of 

clinical users (e.g., displays relevant 

information in useful formats at relevant 

points in workflow)? 

Æ Reflects the ability of implementers 

to make customization and 


daltland on DSKBBV9HB2PROD with NOTICES
Federal Register / Vol. 83, No. 165 / Friday, August 24, 2018 / Notices 42917 

17https://www.qualityforum.org/Publications/ 

2017/09/Interoperability_2016-2017_Final_

Report.aspx. 

18Quality Payment Program. https://qpp.cms.gov/ 

mips/advancing-care-information. 

19Certified Health IT Product List. https://

chpl.healthit.gov/#/search. 

20 2015 Edition Test Method. https://

beta.healthit.gov/topic/certification-ehrs/2015- 

edition-test-method. 

21Program Guidance #17–02: ONC Exercises 

Enforcement Discretion With Respect to 

Implementation of Randomized Surveillance. 

Retrieved from https://www.healthit.gov/sites/ 

default/files/ONC_Enforcement_Discretion_

Randomized_Surveillance_8-30-17.pdf. 

22Certified Health IT Product List (CHPL) Public 

User Guide. Retrieved from https://

www.healthit.gov/sites/default/files/policy/chpl_

public_user_guide.pdf. 

23Products: Corrective Action Status. https://

chpl.healthit.gov/#/collections/correctiveAction. 

implementation decisions in a usercentered manner? 

• What usability assessment data, if 

available, are less resource intensive 

than traditional measures (e.g., time 

motion studies)? 

• Comment on the feasibility and 

applicability of usability measures 

created from audit log data. How would 

health IT acquisition decision makers 

use this information to improve their 

system acquisition, upgrade, and 

customization decisions to best support 

end users’ needs? 

Æ Who should report audit log data 

and by what mechanism? 

• How feasible would it be to 

implement usage monitoring tools (e.g., 

for time spent on specific tasks)? 

Interoperability: 

The Cures Act defines interoperability 

as: ‘‘(A) enables the secure exchange of 

electronic health information with, and 

use of electronic health information 

from, other health information 

technology without special effort on the 

part of the user; (B) allows for complete 

access, exchange, and use of all 

electronically accessible health 

information for authorized use under 

applicable State or Federal law; and (C) 

does not constitute information 

blocking.’’ 

The EHR Compare Report identified 

product integration as a potential means 

to assess interoperability and proposed 

federal and private sector strategies to 

address it. The National Quality 

Forum’s Measurement Framework to 

Assess Nationwide Progress Related to 

Interoperable Health Information 

Exchange to Support the National 

Quality Strategy also specified various 

domains of interoperability that might 

be useful to measure per the health IT 

consumers’ perspective.17 Applicable 

domains include the exchange of 

electronic health information (referring 

to the availability of electronic health 

information, method of exchange, and 

quality of data content), and the 

usability of the exchanged electronic 

health information (referring to issues 

related to relevance, accessibility and 

comprehensibility of the information 

that is exchanged). 

Two existing data sources with many 

Medicare providers are the Inpatient 

Hospital Promoting Interoperability 

Program and the Merit-based Incentive 

Payment System (MIPS), which include 

measures related to health information 

exchange and interoperability.18 The 

exchange-related measures within this 

domain primarily focus on a summary 

of care record exchange (e.g., send and 

receipt/acceptance transaction) and 

clinical information reconciliation. The 

measures would provide insights on 

health IT product performance related 

to summary of care record exchange. For 

additional information, we encourage 

reviewers to reference the 2019 IPPS 

Final Rule and the 2019 PFS which 

includes the Quality Payment Program 

(QPP) NPRM for proposed changes that 

may impact what information is 

submitted to Medicare. 

Industry reports (such as those listed 

in Appendix 3 of the EHR Compare 

Report), which typically involve 

provider surveys, serve as another data 

source to assess interoperability. 

However, the applicability of these 

reports may be limited to larger 

providers and health systems and may 

not necessarily reflect the experiences 

or needs of small practices and all 

settings (e.g., behavioral health). In 

addition, industry reports may not be 

affordable to all users. 

Questions: 

• Please comment on the usefulness 

of product integration as a primary 

means of assessing interoperability (as 

proposed in the EHR Compare Report). 

• What other domains of 

interoperability (beyond those already 

identified and referenced above) would 

be useful for comparative purposes? 

• Of the data sources described in 

this RFI, which data sources would be 

useful for measuring the interoperability 

performance of certified health IT 

products? 

Æ Comment on whether State 

Medicaid agencies would be able to 

share detailed attestation-level data for 

the purpose of developing reports at a 

more detailed level, such as by health IT 

product. If so, how would this 

information be useful to compare 

performance on interoperability across 

health IT products? 

Æ How helpful would CMS program 

data (e.g., Quality Payment Program 

MIPS Promoting Interoperability 

Category, Inpatient Hospital Promoting 

Interoperability Program, Medicaid 

Promoting Interoperability Programs) 

related to exchange and interoperability 

be for comparative purposes? What 

measures should be selected for this 

purpose? Given that some of these data 

may be reported across providers rather 

than at the individual clinical level, 

how would this affect reporting of 

performance by health IT product? 

• What other data sources and 

measures could be used to compare 

performance on interoperability across 

certified health IT products? 

Conformance to Certification Testing: 

Health IT that has been submitted by 

a health IT developer for certification 

and successfully tested and certified is 

listed in the CHPL,19 which is an online, 

openly available resource. This data 

ranges from the user-centered design 

and transparency disclosures made by 

health IT developers to the certification 

criteria to which health IT has been 

certified. However, user experiences, 

product performance, and 

interoperability-oriented metrics from 

health IT developers and healthcare 

providers are not reported in a 

consistent way across all products 

certified through the ONC Health IT 

Certification Program. 

ONC-prepared materials to support 

certification testing, such as the 2015 

Edition Test Method, are intended to be 

read and understood with the express 

purpose of evaluating the health IT’s 

functional capabilities that have been 

submitted for certification in a 

controlled environment, but are not 

determinative of the full scope of the 

health IT’s capabilities, such as in a 

production environment,20

Nevertheless, testing results for health 

IT are available on the CHPL, and much 

of it in a structured format that makes 

the reported test results accessible for 

analysis and comparison. 

As part of maintaining certification 

and ensuring ongoing conformance to 

certification testing, ONC-Authorized 

Certification Bodies (ONC–ACB) 

perform surveillance of health IT 

products and verify that the certified 

health IT also conforms in the 

production environment (i.e., ‘‘in the 

field’’). This includes reviewing any 

complaints about or potential issues 

with certified health IT products 

brought to their attention (i.e., reactive 

surveillance). They may also elect to 

conduct randomized surveillance.21 22

If a certified health IT product does 

not demonstrate the functionality 

required by its certification, the certified 

health IT product is considered nonconforming and then listed and detailed 

as non-conforming on the CHPL.23 A list 


daltland on DSKBBV9HB2PROD with NOTICES
42918 Federal Register / Vol. 83, No. 165 / Friday, August 24, 2018 / Notices 

24 Developers Under Certification Ban. https://

chpl.healthit.gov/#/collections/developers. 

25 Decertified Products. https://chpl.healthit.gov/ 

#/collections/products. 

of banned developers 24 is maintained 

on the CHPL in addition to a list of 

decertified health IT25 where 

certification was withdrawn by the 

developer’s ONC–ACB, by the developer 

under surveillance/review, or 

terminated by ONC. The available 

surveillance information about nonconformant health IT and developers 

provides an avenue that can help 

potential consumers evaluate and 

compare how certified health IT 

performs in real-world settings. In 

addition, developers must post 

mandatory disclosures on types of 

additional costs and limitations for their 

certified health IT as part of the ONC 

Health IT Certification Program 

requirements. 

Question: 

• What additional information about 

certified health IT’s conformance to the 

certification testing (beyond what is 

currently available on the CHPL) would 

be useful for comparison purposes? 

What mechanisms or approaches could 

be considered to obtain such data? What 

barriers might exist for developers and/ 

or end users in reporting on such data? 

Other Categories for Consideration: 

The Cures Act lists other possible 

categories for the EHR Reporting 

Program related to certified health IT 

product performance, including: 

• Enabling the user to order and view 

the results of laboratory tests, imaging 

tests, and other diagnostic tests; 

• Submitting, editing, and retrieving 

data from registries, such as clinicianled clinical data registries; 

• Accessing and exchanging 

information and data from and through 

health information exchanges; 

• Accessing and exchanging 

information and data from medical 

devices; 

• Accessing and exchanging 

information and data held by Federal, 

State, and local agencies and other 

applicable entities useful to a health 

care provider or other applicable user in 

the furtherance of patient care; 

• Accessing and exchanging 

information from other health care 

providers or applicable users; 

• Accessing and exchanging patient 

generated information; 

• Providing the patient or an 

authorized designee with a complete 

copy of their health information from an 

electronic health record in a computable 

format; and 

• Providing accurate patient 

information for the correct patient, 

including exchanging such information, 

and avoiding the duplication of patients 

records. 

Questions: 

• How should the above categories be 

prioritized for inclusion/exclusion in 

the EHR Reporting Program, and why? 

What other criteria would be helpful for 

comparative purposes to best support 

end users’ needs (e.g., to inform health 

IT acquisition, upgrade, and 

implementation decisions)? 

• What data sources could be used to 

compare performance on these 

categories across certified health IT 

products? 

• Please comment on different types 

of information, or measures, in this area 

that would be useful to acquisition, 

upgrade, and customization decisions in 

the ambulatory setting as opposed to 

inpatient settings? 

In addition to the other categories 

listed in the Cures Act, the EHR 

Compare Report identified gaps in 

information related to the domains of 

cost transparency, quality metrics, and 

population health. The cost 

transparency domain includes base, 

subscription, and transaction costs, as 

well as peer reviews regarding price 

expectations, which could also allow 

acquisition decision makers to make 

more informed acquisition, upgrade, 

and customization decisions to best 

support end users’ needs. 

Questions: 

• Please comment on the usefulness 

and feasibility of including criteria on 

quality reporting and population health 

in the EHR Reporting Program. What 

criteria should be considered to assess 

health IT performance in generating 

quality measures, reporting quality 

measures, and the functions required for 

supporting population health analytics 

(e.g., bulk data export)? 

• What data sources, if any, are 

available to assess certified health IT 

product capabilities and performance in 

collecting, generating, and reporting on 

quality measures, and the ability to 

export multiple records for population 

health analytics? Are these data sources 

publicly available? 

• Please comment on other categories, 

if any, besides those listed in this RFI 

that should be considered to be 

included in the EHR Reporting Program. 

Why should these be included, and 

what data sources exist to report on 

performance for the suggested 

categories? 

Hospitals and Health Systems: 

The focus of this RFI is on the 

information needs of health IT end users 

in ambulatory and small practice 

settings, as these groups report 

challenges accessing relevant 

information at affordable costs to help 

them compare certified health IT. 

However, ONC is aware that there are 

also gaps in the availability of 

information that hospitals and health 

systems need. 

Questions: 

• Please describe the types of 

comparative information about certified 

health IT hospitals and health systems 

currently use (e.g., to inform health IT 

acquisition, upgrade, and customization 

decisions). What are the sources of this 

information? What information would 

be useful but is currently unavailable? 

• What types of comparative 

information about certified health IT, if 

any, are specifically useful to hospitals 

and health systems, as opposed to 

ambulatory or small practices? What 

types of information could be collected 

or reported that would be helpful to 

both hospitals and health systems and 

to ambulatory and smaller providers? 

• Please comment on how an EHR 

Reporting Program could best reflect the 

information needed for hospitals and 

health systems, ambulatory and smaller 

provider settings, and overlapping 

information in developing summary 

reports or comparison tools. 

V. Collection of Information 

Requirements 

This document does not impose 

information collection requirements, 

that is, reporting, recordkeeping or 

third-party disclosure requirements. 

Consequently, there is no need for 

review by the Office of Management and 

Budget under the authority of the 

Paperwork Reduction Act of 1995 (44 

U.S.C. 3501 et seq.). 

VI. Response to Comments 

ONC typically receives a large public 

response to its published Federal 

Register documents. ONC will consider 

all comments received by the date and 

time specified in the DATES section of 

this document, but will not be able to 

acknowledge or respond individually to 

public comments. 

Dated: August 17, 2018. 

Donald Rucker, 

National Coordinator, Office of the National 

Coordinator for Health Information 

Technology. 


daltland on DSKBBV9HB2PROD with NOTICES
Federal Register / Vol. 83, No. 165 / Friday, August 24, 2018 / Notices 42919 

APPENDIX A—CERTIFIED HEALTH IT COMPARISON TOOLS IDENTIFIED THROUGH ONC MARKET RESEARCH

Comparison tool Company website 

4Med+ Marketplace .............................. www.4medapproved.com/wizard/marketplace. 

AmericanEHR ....................................... www.americanehr.com. 

Blackbook ............................................. www.blackbookrankings.com/healthcare. 

California Healthcare Foundation ......... www.chcf.org/publications/2007/10/ehr-selection-toolkit-for-community-health-centers. 

CHPL 4.0 .............................................. www.healthit.gov/chpl. 

Consumer Affairs .................................. www.consumeraffairs.com/emr-software. 

EHR Compare ...................................... www.ehrcompare.com. 

EHR in Practice .................................... www.ehrinpractice.com/ehr-product-comparison.html. 

Gartner ................................................. www.gartner.com. 

HealthRecord.US .................................. www.healthrecord.us. 

IDC Health Insights .............................. www.idc.com. 

KLAS .................................................... www.klasresearch.com. 

LeadingAge .......................................... www.leadingage.org/ehr/search.aspx. 

NCQA ................................................... www.ncqa.org/Programs/Recognition/practices/PatientCenteredMedicalHomePCMH/ 

PCMHPrevalidationProgram/VendorList.aspx. 

Software Advice ................................... www.softwareadvice.com. 

Software Insider ................................... www.ehr.softwareinsider.com. 

Technology Advice ............................... www.technologyadvice.com/medical/ehr-emr/smart-advisor. 

Texas Medical Association (TMA) ....... www.texmed.org/EHRTool. 

[FR Doc. 2018–18297 Filed 8–23–18; 8:45 am] 

BILLING CODE 4150–45–P 

DEPARTMENT OF HEALTH AND 

HUMAN SERVICES 

National Institutes of Health 

Government-Owned Inventions; 

Availability for Licensing 

AGENCY: National Institutes of Health, 

HHS. 

ACTION: Notice. 

SUMMARY: The invention listed below is 

owned by an agency of the U.S. 

Government and is available for 

licensing to achieve expeditious 

commercialization of results of 

federally-funded research and 

development. Foreign patent 

applications are filed on selected 

inventions to extend market coverage 

for companies and may also be available 

for licensing. 

FOR FURTHER INFORMATION CONTACT: 

Vince Contreras, Ph.D., 240–669–2823; 

vince.contreras@nih.gov. Licensing 

information and copies of the U.S. 

patent application listed below may be 

obtained by communicating with the 

indicated licensing contact at the 

Technology Transfer and Intellectual 

Property Office, National Institute of 

Allergy and Infectious Diseases, 5601 

Fishers Lane, Rockville, MD, 20852; tel. 

301–496–2644. A signed Confidential 

Disclosure Agreement will be required 

to receive copies of unpublished patent 

applications. 

SUPPLEMENTARY INFORMATION: 

Technology description follows. 

Prefusion HPIV F Immunogens and 

Their Use 

Description of Technology: Human 

parainfluenza virus (hPIV) is an RNAbased paramyxovirus that causes 

respiratory infections in children and 

adults. There are four serotypes that can 

result in a myriad of diseases of the 

respiratory tract including croup, 

bronchitis, and pneumonia (Mao et al., 

2012). hPIV is a leading cause of 

respiratory tract infection and 

hospitalization among children under 5, 

only surpassed by the respiratory 

syncytial virus (RSV). Currently, there 

are limited treatment options and no 

approved vaccines. Recently, studies 

showed that a large proportion of 

neutralizing antibodies preferentially 

recognize exposed epitopes in the 

prefusion conformation of the RSV F 

protein, which together with other 

evidence suggests that creation of 

stabilized prefusion F protein 

immunogens might be a universal 

strategy to develop vaccine candidates 

for inducing protective immune 

responses in RSV and other related 

viruses, such as hPIV. 

Researchers at the Vaccine Research 

Center (VRC) of the National Institute of 

Allergy and Infectious Diseases created 

immunogenic PIV fusion (F) 

glycoproteins for types 1,2,3 and 4 

(hPIV1, hPIV2, hPIV3 and hPIV4) that 

have been modified to stabilize the 

prefusion conformation. 

These stabilized prefusion F 

immunogens, especially hPIV3, induced 

high titer neutralizing responses in mice 

and rhesus macaques, and should thus 

serve as promising candidates for the 

prevention of PIV infection in humans. 

This technology is available for 

licensing for commercial development 

in accordance with 35 U.S.C. 209 and 37 

CFR part 404. 

Potential Commercial Applications: 

• hPIV vaccines for people of all ages; 

• Specific focus on the elderly and 

young children. 

Competitive Advantages: 

• Use as a multivalent hPIV vaccine; 

• Use in combination with influenza 

or RSV vaccine compositions; 

• hPIV3 neutralizing titers induced in 

both mice and rhesus macaques were 

substantially higher than the highest 

PIV3 neutralizing titers observed in a 

cohort of over 100 humans. 

Development Stage: 

• In vivo testing (primates and mice). 

Inventors: Peter Kwong (NIAID), GwoYu Chuang (NIAID), Kai Xu 

(NIAID),Tongqing Zhou (NIAID), 

Yaroslav Tsybovsky (Leidos Biomedical 

Research, Inc), Aliaksandr Druz 

(NIAID), Antonio Lanzavecchia 

(Institute for Research in Biomedicine), 

Davide Corti (Institute for Research in 

Biomedicine), Guillaume BE StewartJones (NIAID), Baoshan Zhang (NIAID), 

Yongping Yang (NIAID), Paul Thomas 

(NIAID), John Mascola (NIAID), Li Ou 

(NIAID), Wing-pui Kong (NIAID). 

Intellectual Property: HHS Reference 

Number E–215–2016 includes U.S. 

Provisional Patent Application Number 

62/412,699 filed 10/25/2016 and PCT 

Application Number PCT/US2017/ 

058322 filed 10/25/2017 (pending). 

Related Intellectual Property: HHS 

Reference Number: E–064–2016. 

Licensing Contact: Vince Contreras, 

Ph.D., 240–669–2823; vince.contreras@

nih.gov. 


daltland on DSKBBV9HB2PROD with NOTICES