APPENDICES

FY 2014 Annual Performance Report and FY 2016 Annual Performance Plan—U.S. Department of Education 87

Appendix A: Data Validity and Verification

The Government Performance and Results Act Modernization Act of 2010 requires 

agencies to describe the accuracy and reliability of data presented. During 2014, the

Department continued to strengthen its approach to data verification and validation. 

Specifically, the Department improved its documentation of the processes used to 

collect, verify, and validate data. The Department also reassessed the quality and 

limitations of the data. Details of how the Department assesses the completeness 

and reliability of the data reported are presented as part of this Appendix. Known 

limitations of the data are also included. 

To my knowledge, the data verification and validation process and the data sources 

used provide, to the extent possible, complete and reliable performance data 

pertaining to goals and objectives in our FY 2014–18 Strategic Plan. 

Through a process of continuous improvement, the Department continues to assess 

its validation process and welcomes input from stakeholders. 

/s/ 

Arne Duncan 

February 2, 2015 
APPENDICES

DATA VALIDITY AND VERIFICATION 

FY 2014 Annual Performance Report and FY 2016 Annual Performance Plan—U.S. Department of Education 88

The GPRA Modernization Act of 2010 requires agencies to prepare information on the reliability 

of data presented. OMB guidance indicates: 

Agencies may develop a single data verification and validation appendix used to 

communicate the agency’s approaches, and/or may also choose to provide information 

about data quality wherever the performance information is communicated (e.g., 

websites). Agencies should discuss their verification and validation techniques with their 

respective OMB Resource Management Office, if necessary. The transmittal letter 

included in Annual Performance Reports must contain an assessment by the agency 

head of the completeness and reliability of the performance data presented and a 

description of agency plans to improve completeness, reliability, and quality, where 

needed.

29

The data presented in the Department’s FY 2013 Annual Performance Report and FY 2015 

Annual Performance Plan were described as being obtained from five basic sources (i.e., 

statistical data, program and enforcement data collections, monitoring and grant applications, 

management information systems or business operations, and external (nonstatistical) data 

sources). Although these types of data may still apply to the performance metrics, in this year’s 

report the metrics are presented by goal instead of being categorized by data source. 

Below is a list of metrics with information on data quality, limitations, and improvements. 

 

29 OMB Circular A-11, Preparation, Submission, and Execution of the Budget, Part 6, Section 260.9, 2014. 
APPENDICES

DATA VALIDITY AND VERIFICATION 

FY 2014 Annual Performance Report and FY 2016 Annual Performance Plan—U.S. Department of Education 89

FY 2014–18 Strategic Plan Public-Facing Metrics

GOAL 1: POSTSECONDARY EDUCATION, CAREER AND TECHNICAL EDUCATION, AND ADULT EDUCATION.

Increase college access, affordability, quality, and completion by improving postsecondary education 

and lifelong learning opportunities for youths and adults.

# Metric Data Source Data Quality, Limitations, and Improvements

1.1.A Rate of increase in net price of

public four-year institutions

Integrated 

Postsecondary 

Education Data System 

(IPEDS) Data Center

Average net price is calculated only on the universe of students who 

receive financial aid. It does not include students who are attending 

college without financial aid. Thus, the metric represents the average 

net price for a subset of students and is not representative of the cost 

of college for all students.

Data quality and limitations are identified in IPEDS First Look 

Publications, “Data Collection Procedures,” and IPEDS methodology 

available at http://nces.ed.gov/pubs2014/2014067.pdf. 

Graduation Rates for Selected Cohorts, 2005–10; and Student 

Financial Aid in Postsecondary Institutions, Academic Year 2012–13: 

First Look (Provisional Data) NCES 2014—NCES Number: 2014105

Release Date: November 20, 2014 available at: 

http://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2014105.

Postsecondary Institutions and Price of Attendance in 2013–14; 

Degrees and Other Awards Conferred: 2012–13; and 12-Month 

Enrollment: 2012–13: First Look (Provisional Data), available at: 

http://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2014066rev.

1.1.B Rate of increase in net price of

public two-year institutions

IPEDS Data Center

1.1.C Percentage of high school seniors 

filing a FAFSA

The denominator is the 

number of graduating 

seniors according to the 

most recent projection 

by NCES. The 

numerator is from 

Federal Student Aid’s 

(FSA’s) Central 

Processing System and 

is based on the number 

of applications during 

the first nine months of 

Since the FAFSA does not ask the year of high school graduation, 

several assumptions and criteria are made (such as age of applicant) 

to identify those likely to be high school seniors. These assumptions 

and criteria are applied consistently across all baseline year and future 

calculations.

These calculations also restrict the application period to the first nine 

months of the application cycle (the close of the fiscal year), rather 

than the entire 18 months. Because most applicants, including high 

school seniors, file their FAFSA prior to the start of the upcoming 

academic year (usually before fiscal year end), this decision better 

aligns the performance metric with the fiscal year where most of the 
APPENDICES

DATA VALIDITY AND VERIFICATION

FY 2014 Annual Performance Report and FY 2016 Annual Performance Plan—U.S. Department of Education 90

# Metric Data Source Data Quality, Limitations, and Improvements

the application cycle that 

are—as of September 

30 of the first year of the 

application cycle—

complete (not rejected); 

first-time filers; incoming 

freshmen, with or 

without previous college 

attendance; age 18 or 

less as of June 30 of the 

first year of the 

application cycle; 

reporting high school 

diploma attainment; and 

attended a high school 

in the fifty states and 

Washington, DC.

performance occurred. The alternative is waiting for the close of the 

18-month cycle, where a performance metric would mostly reflect 

performance from an earlier fiscal year.

1.1.D Index of national annual aggregate 

earnings of VR consumers (based 

on the number of competitive 

employment outcomes, hours 

worked, and hourly wages of VR 

consumers)

Rehabilitation Services 

Administration (RSA)-

911

Instructions for submitting the RSA-911 data are provided to agencies. 

Agency questions for coding particular data elements are provided by 

Data Unit staff on a case-by-case basis. 

RSA provides the agencies with the edit program that we use and are 

encouraged to run their data using the program often, at least 

quarterly, during the year. In addition, agencies have edits in their own 

systems which run additional checks.

1.1.E Index of national annual aggregate 

earnings of Transition-Age Youth 

(based on the number of 

competitive employment outcomes, 

hours worked, and hourly wages of 

VR Transition-Age Youth)

RSA-911

1.1.F Number of peer-reviewed 

publications resulting from NIDRRsupported grantee projects

NIDRR—Annual 

Performance Report 

(APR) Accomplishments 

Database

The data are self-reported by grantees in the Department’s APR 

system. Included in the baseline are all manuscripts generated by 

NIDRR grantees that are accepted for or published in a peer reviewed 

journal by the end of the evaluation period.

The verification occurs at the grantee level, the project officer level, 

and the contractor level. The validation occurs as part of the strategic 

planning with NIDRR’s data group in consultation with the contractor.

The Workforce Improvement and Opportunity Act (WIOA) transfers the 
APPENDICES

DATA VALIDITY AND VERIFICATION

FY 2014 Annual Performance Report and FY 2016 Annual Performance Plan—U.S. Department of Education 91

# Metric Data Source Data Quality, Limitations, and Improvements

NIDRR from the Department to Health and Human Services. For that 

reason, the Department is considering removing this metric.

1.1.G Number of VR state directors and 

other state VR personnel who 

express knowledge of NIDRR 

grantee research

New VR state director 

survey

This data collection effort is planned to be an annual survey item. 

However, the baseline data are not yet collected as the contract and 

process for collecting the data are not finalized and may not be.

The Workforce Improvement and Opportunity Act (WIOA) transfers the 

NIDRR from the Department to Health and Human Services (HHS).

The Department does not know if HHS will choose to collect these 

data. For that reason, the Department is considering removing this 

metric.

1.2.A Number of low-performing 

institutions with high loan default 

rates and low graduation rates

FSA Cohort Default Rate 

(CDR) Report, 

September 2014

IPEDS Data Center

The number of low-performing Institutions of Higher Education (IHEs) 

is calculated by first identifying institutions with a cohort default rate of 

30% or higher (the threshold that jeopardizes an institution’s access to 

federal financial aid if that level is sustained for three consecutive 

years). The graduation rate within 150% of normal time for each of 

those IHEs is then extracted from IPEDS, along with the transfer-out 

rate, if applicable, and the two rates are combined to create a 

“completion rate.” If the completion rate is below the average 

completion rate for similar types of U.S. Title-IV eligible IHEs—

separately categorized as four-year, two-year, and less-than-twoyear—then the IHE is included in the number of low-performing 

institutions. Community colleges that offer a limited number of 

bachelor’s degrees, which are categorized as four-year (primarily 

associate’s degree-granting) institutions in IPEDS, are treated as twoyear IHEs in terms of calculating graduation rates and the comparable 

average completion rate. For four-year IHEs, the graduation rate is 

based on the degree- or certificate-seeking cohort, not the bachelor’sseeking cohort, since many four-year IHEs offer certificates and 

associate’s degrees in addition to bachelor’s degrees, and thus the 

broader cohort is more indicative of their performance. Institutions 

which do not have a graduation rate cohort (degree-seeking, first-time, 

full-time students) and thus no comparable graduation/completion rate 

are not included in the count. The calculation includes an assumption 

that the cohort default rates for all institutions for the baseline year will 

not change. In actuality, institutions can appeal their rates after 

publication and, if justified, those rates may be changed, thus changing 

the record from one year to the next. This means the baseline number, 

or the number in any given year, may not be replicable using revised 
APPENDICES

DATA VALIDITY AND VERIFICATION

FY 2014 Annual Performance Report and FY 2016 Annual Performance Plan—U.S. Department of Education 92

# Metric Data Source Data Quality, Limitations, and Improvements

data in subsequent reports.

Graduation rate data quality and limitations are identified in IPEDS 

First Look Publications, “Data Collection Procedures,” and IPEDS 

methodology available at http://nces.ed.gov/pubs2014/2014067.pdf.

Data are checked by NCES for consistency.

CDR data quality and limitations are available at 

http://ifap.ed.gov/DefaultManagement/finalcdrg.html.

The cohort default rate data, which serves as the basis of the 

spreadsheet, is verified by FSA prior to publication. Because the 

process for manually entering the completion rate data is so laborintensive, there is not a process for Independent Verification and 

Validation (IV&V) verifying the accuracy of the process for extracting 

the data from IPEDS or the accuracy of the data that has been entered 

into the spreadsheet.

1.3.A Degree attainment among 25–34-

year-old age cohort

NCES Digest of 

Education Statistics, 

Table 104.30 

(http://nces.ed.gov/progr

ams/digest/d13/tables/dt

13_104.30.asp), Number 

of persons age 18 and 

over, by highest level of 

educational attainment, 

sex, race/ethnicity, and 

age: 2013. Tabulated 

from Current Population 

Survey data, U.S. 

Census

Data quality and limitations are documented in 

http://www.census.gov/cps/files/Source%20and%20Accuracy.pdf. 

NCES tabulates the data, which is verified prior to publication 

according to NCES guidelines. 

1.3.B Retention rate of first-time degreeseeking undergraduates; Full-timeIPEDS Data Center The retention rate is calculated based on degree-seeking, first-time 

undergraduates at degree-granting U.S. institutions that participate in 

Title IV.

Institutions employ a variety of methods for determining degreeseeking status to determine which students to include in the cohort, but 

none of these methods is foolproof. Furthermore, a student who 

transfers to another institution in the second year will be considered as 

not retained, suppressing the aggregate rate. (Retention in this context 

1.3.C Retention rate of first-time degreeseeking undergraduates; Part-timeIPEDS Data Center
APPENDICES

DATA VALIDITY AND VERIFICATION

FY 2014 Annual Performance Report and FY 2016 Annual Performance Plan—U.S. Department of Education 93

# Metric Data Source Data Quality, Limitations, and Improvements

is at the reporting institution so transfer-outs are not retained at the 

institution.) Consequently, the retention rate is only an approximation 

of true retention. 

Data quality and limitations are identified in IPEDS First Look 

Publications, “Data Collection Procedures,” and IPEDS methodology 

available at http://nces.ed.gov/pubs2014/2014067.pdf. Data are 

checked by NCES for consistency.

1.4.A Number of STEM postsecondary 

credentials awarded

IPEDS Data Center IPEDS collects completions by Classification of Instructional Programs 

(CIP) Code which may be aggregated into STEM and non-STEM 

counts. The Department of Education does not currently have a single 

definition for which CIP codes are STEM.

Data quality and limitations are identified in IPEDS First Look 

Publications, “Data Collection Procedures,” and IPEDS methodology 

available at http://nces.ed.gov/pubs2014/2014067.pdf. Data are 

checked by NCES for consistency.
APPENDICES

DATA VALIDITY AND VERIFICATION

FY 2014 Annual Performance Report and FY 2016 Annual Performance Plan—U.S. Department of Education 94

GOAL 2: ELEMENTARY AND SECONDARY EDUCATION. Improve the elementary and secondary education 

system’s ability to consistently deliver excellent instruction aligned with rigorous academic standards 

while providing effective support services to close achievement and opportunity gaps, and ensure all 

students graduate high school college- and career-ready.

# Metric Data Source Data Quality, Limitations, and Improvements

2.1.A Number of states that have adopted 

college- and career-ready 

standards

Elementary and 

Secondary Education 

Act (ESEA) 

Flexibility Monitoring

The Office of Elementary and Secondary Education’s (OESE) new 

(and aforementioned) Office of State Support, which replaces and 

enhances services previously provided by the Office of Student 

Achievement and School Accountability (SASA), the Office of School 

Turnaround, and the Implementation and Support Unit (ISU), will count 

the number of states that either (a) have a memorandum of 

understanding in place to implement the Common Core or (b) have a 

letter from an Institute for Higher Education in their state certifying that 

their state has college- and career-ready standards. There are no 

known data limitations.

2.1.B Number of states that are 

implementing next-generation 

reading and mathematics 

assessments, aligned with collegeand career-ready standards

ESEA Flexibility 

Monitoring

The Office of Elementary and Secondary Education’s (OESE) OSS,

which replaces and enhances services previously provided by the 

SASA, the Office of School Turnaround, and the ISU, monitors states 

that receive ESEA Flexibility waivers in three phases over the threeyear waiver. Monitoring includes desk monitoring and on-site 

monitoring. OSS state leads use a monitoring protocol and rubric to 

ensure that monitoring is consistent across all states. OSS state leads 

work with their Group Leader to finalize monitoring reports. All reports 

are reviewed by both the Group Leader for Monitoring and Technical

Assistance and the OSS Director for consistencies across states. 

States have an opportunity to review the draft report before the final 

report is issued. There are no known data limitations.
APPENDICES

DATA VALIDITY AND VERIFICATION

FY 2014 Annual Performance Report and FY 2016 Annual Performance Plan—U.S. Department of Education 95

# Metric Data Source Data Quality, Limitations, and Improvements

2.2.A Number of states that have fully 

implemented teacher and principal 

evaluation and support systems that 

consider multiple measures of 

effectiveness, with student growth 

as a significant factor

ESEA Flexibility 

Applications and 

Monitoring

The Office of Elementary and Secondary Education’s (OESE) OSS, 

which replaces and enhances services previously provided by the 

SASA, the Office of School Turnaround, and the ISU, monitors states 

that receive ESEA Flexibility waivers. Monitoring includes desk 

monitoring and on-site monitoring. OSS state leads use a monitoring 

protocol and rubric to ensure that monitoring is consistent across all 

states. OSS state leads work with their Group Leader to finalize 

monitoring reports. All reports are reviewed by both the Group Leader 

for Monitoring and Technical Assistance and the OSS Director for 

consistencies across states. States have an opportunity to review the 

draft report before the final report is issued.

Additionally, Chief State School Officers must sign each state’s ESEA 

Flexibility request before it is approved. There are no known data 

limitations.

2.3.A Disparity in the rates of out-ofschool suspensions for students 

with disabilities and youth of color 

(youth of color metric)

Civil Rights Data 

Collection (CRDC)

The Office for Civil Rights (OCR) strives to ensure CRDC data are an 

accurate and comprehensive depiction of student access to 

educational opportunities in school districts. The submission system 

includes a series of embedded edit checks to ensure significant data 

errors are corrected before the district submits its data. Additionally, 

each district is required to certify the accuracy of its submission. Only a 

district superintendent, or the superintendent’s designee, may certify 

the CRDC submission. Ultimately, the quality of the CRDC data 

depends on accurate collection and reporting by the participating 

districts.

2.3.B Disparity in the rates of out-ofschool suspensions for students 

with disabilities and youth of color 

(SWD, IDEA only metric)

CRDC
APPENDICES

DATA VALIDITY AND VERIFICATION

FY 2014 Annual Performance Report and FY 2016 Annual Performance Plan—U.S. Department of Education 96

# Metric Data Source Data Quality, Limitations, and Improvements

2.4.A Number of persistently low 

graduation rate high schools

EDFacts EDFacts works with the data stewards (in this case OESE) to 

determine the appropriate business rule checks for these data. The 

checks that were done on these data include:

 File validation and format checks: Identifies file submissions that 

don’t conform with the file format (unable to be processed), the data 

universe (school/LEA/state directory), or the reporting requirements 

(mandatory reporting fields).

 SAS Data Quality Reviews: Post submission checks to determine 

completeness, consistency, and comparability.

 DataFlux Data Quality Rules: Post submission checks to determine 

completeness, consistency, and comparability.

 Submission Edit Business Rules: While not utilized for the 2012–13 

data collection, several of the post-submission business rule checks 

performed in SAS and DataFlux for 2012–13 have been 

implemented against the 2013–14 submission (and beyond).

At the end of a collection period these rules are run against the 

submitted data and presented to the program office. It is the 

responsibility of the program office (OESE in this case), in consultation 

with the EDFacts staff, to determine which errors should be escalated 

to the state for further review. Many of these errors were sent along to 

the state for remediation (data explanation/update/correction). OESE 

should be considered the point-of-contact for identifying which rules 

were escalated and the result of those escalations.

Data concerns from 2011–12 were documented in the NCES data 

release report 

(http://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2014391). The 

data limitations for the 2012–13 are not yet complete and neither 

NCES nor OESE are prepared, at this time, to identify the data quality 

concerns with the 2012–13 submission. If these data are once again 

released by NCES, then work will be done to impute any missing data 

for states and to produce a national estimate. The imputation 

methodology and data limitations will be documented at that time as 

was done for the 2011–12 data release. This work is still pending and 

any discussion of the data limitations and/or concerns at this time 

would be premature.
APPENDICES

DATA VALIDITY AND VERIFICATION

FY 2014 Annual Performance Report and FY 2016 Annual Performance Plan—U.S. Department of Education 97

# Metric Data Source Data Quality, Limitations, and Improvements

Path to public release and national rates: For each collection cycle 

states report their calculated cohort counts and ACGR graduation 

rates. These counts and rates are tested against a number of business 

rules for format, consistency, completeness, and comparability. Those 

business rule checks are delivered to the Office of Elementary and 

Secondary Education (OESE) for follow-up correspondence with the 

state(s). The program office, in consultation with NCES EDFacts staff 

and other stakeholders, meet to determine which issues identified by 

the business rules should be raised with the state for explanation, 

update, or correction. Following that review, data are then aggregated 

to the national level for the purpose of calculating and publishing a 

national rate. NCES processes these data to force conformity of 

reporting categories (mapping reported “Major Racial/Ethnic Groups” 

to the traditional 5 racial/ethnic groups) and imputes any missing data 

(3 states were imputed for 2011–12, 1 will be for 2012–13, no 

imputations are expected to be necessary for 2013–14 forward). After 

imputations are made, NCES produces a weighted national average 

rate for the country as a whole; a rate which is representative of every 

state. Once produced, NCES documents any remaining issues with 

these data, and the aggregation and imputation methodology in a 

release report. That report, documentation, and the associated data 

tables are put through several stages of review including independent 

reviews at the division (Administrative Data), center (NCES), and 

agency (IES) levels prior to public dissemination.

2.4.B Percent of Cohort 1 priority schools 

that have met the state exit criteria 

and exited priority school status

EDFacts ESEA Flexibility plans do not allow for one standard methodology to 

determine whether or not a school “met the state exit criteria.” This will 

need to be looked at manually, state-by-state, once the list of schools 
APPENDICES

DATA VALIDITY AND VERIFICATION

FY 2014 Annual Performance Report and FY 2016 Annual Performance Plan—U.S. Department of Education 98

# Metric Data Source Data Quality, Limitations, and Improvements

2.4.C Percent of Cohort 1 focus schools 

that have met the state exit criteria 

and exited focus school status

EDFacts exiting priority status has been identified.

EDFacts works with the data stewards (in this case OESE) to 

determine the appropriate business rule checks for these data. The 

checks that were done on these data include:

 File validation and format checks: Identifies file submissions that 

don’t conform with the file format (unable to be processed), the 

data universe (school/LEA/state directory), or the reporting 

requirements (mandatory reporting fields).

 SAS Data Quality Reviews: Post submission checks to determine 

completeness, consistency, and comparability.

 DataFlux Data Quality Rules: Post submission checks to determine 

completeness, consistency, and comparability.

 Submission Edit Business Rules: While not utilized for the 2012–

13 data collection, several of the post-submission business rule 

checks performed in SAS and DataFlux for 2012–13 have been 

implemented against the 2013–14 submission (and beyond).

At the end of a collection period these rules are run against the 

submitted data and presented to the program office. It is the 

responsibility of the program office (OESE in this case), in consultation 

with the EDFacts staff, to determine which errors should be escalated 

to the state for further review. Many of these errors were sent along to 

the state for remediation (data explanation/update/correction). OESE 

should be considered the point-of-contact for identifying which rules 

were escalated and the result of those escalations.

Data concerns from 2011–12 were documented in the NCES data 

release report 

(http://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2014391). The 

data limitations for the 2012–13 are not yet complete and neither 

NCES nor OESE are prepared, at this time, to identify the data quality 

concerns with the 2012–13 submission. If these data are once again 

released by NCES, then work will be done to impute any missing data 

for states and to produce a national estimate. The imputation 

methodology and data limitations will be documented at that time as 
APPENDICES

DATA VALIDITY AND VERIFICATION

FY 2014 Annual Performance Report and FY 2016 Annual Performance Plan—U.S. Department of Education 99

# Metric Data Source Data Quality, Limitations, and Improvements

was done for the 2011–12 data release. This work is still pending and 

any discussion of the data limitations and/or concerns at this time 

would be premature.

Path to public release and national rates: For each collection cycle 

states report their calculated cohort counts and ACGR graduation 

rates. These counts and rates are tested against a number of business 

rules for format, consistency, completeness, and comparability. Those 

business rule checks are delivered to the Office of Elementary and 

Secondary Education (OESE) for follow-up correspondence with the 

state(s). The program office, in consultation with NCES EDFacts staff 

and other stakeholders, meet to determine which issues identified by 

the business rules should be raised with the state for explanation, 

update, or correction. Following that review, data are then aggregated 

to the national level for the purpose of calculating and publishing a 

national rate. NCES processes these data to force conformity of 

reporting categories (mapping reported “Major Racial/Ethnic Groups” 

to the traditional 5 racial/ethnic groups) and imputes any missing data 

(3 states were imputed for 2011–12, 1 will be for 2012–13, no 

imputations are expected to be necessary for 2013–14 forward). After 

imputations are made, NCES produces a weighted national average 

rate for the country as a whole; a rate which is representative of every 

state. Once produced, NCES documents any remaining issues with 

these data, and the aggregation and imputation methodology in a 

release report. That report, documentation, and the associated data 

tables are put through several stages of review including independent 

reviews at the division (Administrative Data), center (NCES), and 

agency (IES) levels prior to public dissemination.

2.5.A Percentage of high school and 

middle school teachers who teach 

STEM as their main assignment 

who hold a corresponding 

undergraduate degree

Schools and Staffing 

Survey (SASS), NCES

The methods report for the 2011–12 SASS is not yet released. Study 

documentation from the 2007–08 survey is available at 

http://nces.ed.gov/surveys/sass/methods0708.asp.
APPENDICES

DATA VALIDITY AND VERIFICATION

FY 2014 Annual Performance Report and FY 2016 Annual Performance Plan—U.S. Department of Education 100 

# Metric Data Source Data Quality, Limitations, and Improvements

2.5.B Number of public high school 

graduates who have taken at least 

one STEM AP exam

College Board/AP 

administrative records

The College Board does not collect or report race/ethnicity based on 

the federal guidelines. Examinees are asked to select one of the 

options noted in the data. The College Board Public School List is 

updated annually by state educational agencies; thus small changes to 

the list over time are to be expected as schools open, close, and/or 

merge. Students are assigned to graduating cohorts based on selfreported information (i.e., grade level and/or graduation year) provided 

at the time of registration (in the case of SAT) or test administration (in 

the case of AP and PSAT). The College Board matches students’ data 

across programs to identify the most recent valid value when assigning 

students to cohorts.
APPENDICES

DATA VALIDITY AND VERIFICATION

FY 2014 Annual Performance Report and FY 2016 Annual Performance Plan—U.S. Department of Education 101 

GOAL 3: EARLY LEARNING. Improve the health, social-emotional, and cognitive outcomes for all 

children from birth through 3rd grade, so that all children, particularly those with high needs, are on 

track for graduating from high school college- and career-ready.

# Metric Data Source Data Quality, Limitations, and Improvements

3.1.A Number of states with Quality 

Rating and Improvement Systems 

(QRIS) that meet high quality 

benchmarks for child care and other 

early childhood programs

Child Care 

Development Fund 

(CCDF) Report of 

States Plans with 

annual updates from 

states and territories 

(HHS/Office of 

Childcare)

The data are self-reported by the states in their CCDF state plans. In 

addition, the data do not take into account the participation or coverage 

of the QRIS systems.

3.2.A Number of states and territories with 

professional development systems 

that include core knowledge and 

competencies, career pathways, 

professional development capacity 

assessments, accessible 

professional development 

opportunities, and financial support 

for childcare providers

Child Care 

Development Fund 

(CCDF) Report of State 

Plans (HHS/Office of 

Childcare)

The data are self-reported by the states in their CCDF state plans. In 

addition, the data do not take into account the participation or coverage 

of the professional development systems.

3.3.A Number of states collecting and 

reporting disaggregated data on the 

status of children at kindergarten 

entry using a common measure

Race to the Top (RTT)-

Early Learning 

Challenge Technical 

Assistance Center (ELC 

TA)

The data are limited to the 20 RTT-ELC states. 

Department staff has requested KEA data be collected in the 2015–16 

academic year as part of the National Center for Education Statistics 

(NCES) State of Preschool data collection and has requested to add 

KEA data reporting in EDFacts. In addition, the new Preschool 

Development Grants will provide data on additional states.
APPENDICES

DATA VALIDITY AND VERIFICATION

FY 2014 Annual Performance Report and FY 2016 Annual Performance Plan—U.S. Department of Education 102 

GOAL 4: EQUITY. Increase educational opportunities for underserved students and reduce 

discrimination so that all students are well-positioned to succeed.

# Metric Data Source Data Quality, Limitations, and Improvements

4.1.A National high school graduation rate EDFacts EDFacts works with the data stewards (in this case OESE) to 

determine the appropriate business rule checks for these data. The 

checks that were done on these data include:

 File validation and format checks: Identifies file submissions that 

don’t conform with the file format (unable to be processed), the 

data universe (school/LEA/state directory), or the reporting 

requirements (mandatory reporting fields).

 SAS Data Quality Reviews: Post submission checks to determine 

completeness, consistency, and comparability.

 DataFlux Data Quality Rules: Post submission checks to 

determine completeness, consistency, and comparability.

 Submission Edit Business Rules: While not utilized for the 2012–

13 data collection, several of the post-submission business rule 

checks performed in SAS and DataFlux for 2012–13 have been 

implemented against the 2013–14 submission (and beyond).

At the end of a collection period these rules are run against the 

submitted data and presented to the program office. It is the 

responsibility of the program office (OESE in this case), in consultation 

with the EDFacts staff, to determine which errors should be escalated 

to the state for further review. Many of these errors were sent along to

the state for remediation (data explanation/update/correction). OESE 

should be considered the point-of-contact for identifying which rules 

were escalated and the result of those escalations.

Data concerns from 2011–12 were documented in the NCES data 

release report 

(http://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2014391). The 

data limitations for the 2012–13 are not yet complete and neither 

NCES nor OESE are prepared, at this time, to identify the data quality 

concerns with the 2012–13 submission. If these data are once again

released by NCES, then work will be done to impute any missing data 
APPENDICES

DATA VALIDITY AND VERIFICATION

FY 2014 Annual Performance Report and FY 2016 Annual Performance Plan—U.S. Department of Education 103 

# Metric Data Source Data Quality, Limitations, and Improvements

for states and to produce a national estimate. The imputation 

methodology and data limitations will be documented at that time as 

was done for the 2011–12 data release. This work is still pending and 

any discussion of the data limitations and/or concerns at this time 

would be premature.

Path to public release and national rates: For each collection cycle 

states report their calculated cohort counts and ACGR graduation 

rates. These counts and rates are tested against a number of business 

rules for format, consistency, completeness, and comparability. Those 

business rule checks are delivered to OESE for follow-up 

correspondence with the state(s). The program office, in consultation 

with NCES EDFacts staff and other stakeholders, meet to determine 

which issues identified by the business rules should be raised with the 

state for explanation, update, or correction. Following that review, data 

are then aggregated to the national level for the purpose of calculating 

and publishing a national rate. NCES processes these data to force 

conformity of reporting categories (mapping reported “Major 

Racial/Ethnic Groups” to the traditional 5 racial/ethnic groups) and 

imputes any missing data (3 states were imputed for 2011–12, 1 will 

be for 2012–13, no imputations are expected to be necessary for 

2013–14 forward). After imputations are made, NCES produces a 

weighted national average rate for the country as a whole; a rate 

which is representative of every state. Once produced, NCES 

documents any remaining issues with these data, and the aggregation 

and imputation methodology in a release report. That report, 

documentation, and the associated data tables are put through several 

stages of review including independent reviews at the division 

(Administrative Data), center (NCES), and agency (IES) levels prior to 

public dissemination.

4.2.A Percentage of proactive civil rights 

investigations launched annually 

that address areas of concentration 

in civil rights enforcement

Office for Civil Rights’ 

(OCR) Case 

Management System 

(CMS) and Document 

Management (DM) 

systems

The Contracts and Acquisition Management/Case Management 

database utilized to collect data for this metric has built-in validation 

checks (such as requiring certain data elements and valid date 

entries), as well as automated entries based on other information 

entered into the system by staff, to reduce errors in data entry. OCR 

also employs additional safeguards to ensure data accuracy such as 
APPENDICES

DATA VALIDITY AND VERIFICATION

FY 2014 Annual Performance Report and FY 2016 Annual Performance Plan—U.S. Department of Education 104 

# Metric Data Source Data Quality, Limitations, and Improvements

4.2.B Percentage of proactive civil rights 

investigations resolved annually 

that address areas of concentration 

in civil rights enforcement

Office for Civil Rights’ 

(OCR) Case 

Management System 

(CMS) and Document 

Management (DM) 

systems

(1) periodic monthly checks to address missing or inconsistent entries, 

(2) publication of guidance materials including data entry 

requirements, codes, definitions, checklist and protocol for staff 

responsible for entering data; and (3) additional training and support 

for primary users inputting the data.
APPENDICES

DATA VALIDITY AND VERIFICATION

FY 2014 Annual Performance Report and FY 2016 Annual Performance Plan—U.S. Department of Education 105 

GOAL 5: CONTINUOUS IMPROVEMENT OF THE U.S. EDUCATION SYSTEM. Enhance the education 

system’s ability to continuously improve through better and more widespread use of data, research and 

evaluation, evidence, transparency, innovation, and technology.

# Metric Data Source Data Quality, Limitations, and Improvements

5.1.A Number of public data sets included in ED 

Data Inventory and thus linked to Data.gov or 

ED.gov websites

Data Strategy Team Data 

Inventory and the public 

ED Data Inventory at 

http://datainventory.ed.gov

The data are validated with a crosswalk between Inventory 

entries and the listing of public Department datasets, 

ensuring that the data described in the ED Data Inventory 

is publicly available at the identified web address.

5.1.B Number of states linking K-12 and 

postsecondary data with workforce data

State Longitudinal Data 

Systems (SLDS) grant 

monitoring (monthly 

updates from states, 

annual performance 

reports, final performance 

reports, and site visits)

Data are collected through monitoring of states with active 

SLDS grants. The data are limited to this population. While 

47 states and territories have received these grants over 

time, by June 2015, there will be fewer than 25 states with 

active grants, which leads to incomplete and not up-to-date 

data from states that either have not receive grants or that 

do not have active grants.
APPENDICES

DATA VALIDITY AND VERIFICATION

FY 2014 Annual Performance Report and FY 2016 Annual Performance Plan—U.S. Department of Education 106 

# Metric Data Source Data Quality, Limitations, and Improvements

5.1.C Number of states linking K-12 with early 

childhood data

SLDS grant monitoring 

(monthly updates from 

states, annual 

performance reports, final 

performance reports, and 

site visits)

States project teams report data based on current grants. If 

the state is focusing on an early childhood data system, the 

state’s reporting might reflect only those development 

efforts, and not statewide P20W (preschool to college 

workforce) development efforts. There are additional 

sources for information about state data linkages in the 

field. For example the Data Quality Campaign (DQC) 

conducts surveys of state capacity to collect, store, link, 

and use data. Because the Department does not use the 

same definitions as DQC, our figures tend to be lower than 

theirs. For example, we require that a state possess the 

capacity to follow its own students in order for us to report 

that the state has the linkage in place. Similarly, we enable 

states to report on whether particular linkages are planned, 

in progress, or complete, and report that a state has a 

linkage when the state reports that the project is complete; 

DQC might give a state credit for an “in progress” or pilotstage linkage.

A survey administered to the universe of states and 

territories would enable more systematically collect data 

about all states’ capacity for data linkages and data use.

There is a concern, however, that if those data were to be 

used for public reporting, states might begin to overstate 

their capacities, particularly on data that are also publicly 

reported by organizations such as DQC. Currently, data 

from monitoring is used in an iterative, formative approach 

to program improvement; our technical assistance program 

is designed to support states’ efforts to improve their 

systems. This relies on states being honest about their own 

internal capacities.
APPENDICES

DATA VALIDITY AND VERIFICATION

FY 2014 Annual Performance Report and FY 2016 Annual Performance Plan—U.S. Department of Education 107 

# Metric Data Source Data Quality, Limitations, and Improvements

5.2.A Average time to close “cases” (PTAC + 

FPCO)

Case Tracking System 

(CTS)

The term “case” refers to requests for quick, informal 

responses to routine questions related to student privacy.

These requests are received via email, the Family Policy 

Compliance Office (FPCO)/Privacy Technical Assistance 

Center (PTAC) resource website, or by telephone and 

subsequently entered into the Case Tracking System 

(CTS).

The preliminary data for this metric are reviewed at least 

weekly for verification. If anomalous data are identified in 

the periodic reviews or when anomalies are suspected, 

individual cases are examined individually to identify if they 

were properly closed or if their status was entered 

incorrectly. When appropriate, corrections are made. Staff 

responsible for entering data into the CTS will continue to 

be trained on policies and procedures. Quarterly monitoring 

of data entered will be conducted to assure completeness 

and reliability of data and to recommend any improvements 

to the CTS or modifications to the standard operating 

procedures.

5.3.A Percentage of select new (non-continuation) 

discretionary grant dollars that reward 

evidence

Department calculations 

based upon multiple 

Department-controlled 

data sources, including 

G5

In determining which discretionary grants are considered 

“evidence-based” (i.e., the numerator when calculating the 

percentage), the Department includes all programs that 

rewarded applicants with supporting evidence of promise or 

better (per the EDGAR evidence framework). This could be 

done through a competitive preference or absolute priority, 

an eligibility requirement, or a selection factor. Only the 

amounts of the grants awarded for those projects were 

counted. In determining what counts as discretionary 

funding (i.e., the denominator when calculating the 

percentage), the Department includes all programs for 

which the EDGAR evidence framework could conceivably 

work. Specifically, we determined that NIDRR and RSA are

unlikely to find the evidence framework useful, and did not 

count funding for those programs in the denominator.
APPENDICES

DATA VALIDITY AND VERIFICATION

FY 2014 Annual Performance Report and FY 2016 Annual Performance Plan—U.S. Department of Education 108 

# Metric Data Source Data Quality, Limitations, and Improvements

5.3.B Number of peer-reviewed, full-text resources 

in the Education Resources Information 

Center (ERIC)

Education Resources 

Information Center (ERIC)

To be counted as a full-text, peer-reviewed article in ERIC, 

the document had to be indexed in ERIC and designated 

with a “peer-reviewed” and “full-text” flag, both of which are 

available on the public ERIC website.

The ERIC contractor uses specified quality assurance 

procedures. In addition, the IES program officer pays close 

attention to the metric in their review of deliverables.

5.3.C Number of reviewed studies in the What 

Works Clearinghouse (WWC) database

What Works 

Clearinghouse (WWC)

To be counted as a reviewed study, the study had to be 

listed in the What Works Clearinghouse’s publicly available 

Database of Reviewed Studies. 

The What Works Clearinghouse contractors use specified 

quality assurance procedures. In addition, the IES program

officers pay close attention to the metric in their review of 

deliverables.

5.4.A Percentage of schools in the country that 

have actual internet bandwidth speeds of at 

least 100 Mbps

Education Superhighway, 

CoSN/AASA E-rate 

Infrastructure Survey

The Department uses an external data source for this 

metric and relies on the external, third party’s verification 

and validation methodology. Based on the information 

provided (http://cosn.org/cosns-second-annual-e-rate-andinfrastructure-survey), the response rate for this survey 

may not be sufficient to ensure that the data are 

representative of all districts in the country. The 

Department is exploring the feasibility of collecting data on 

access to and use of education technology from a 

representative sample of schools and districts across the 

country. In the meantime, we believe these data are the 

best currently available and provide useful information to 

gauge progress on this metric.
APPENDICES

DATA VALIDITY AND VERIFICATION

FY 2014 Annual Performance Report and FY 2016 Annual Performance Plan—U.S. Department of Education 109 

GOAL 6: U.S. DEPARTMENT OF EDUCATION CAPACITY. Improve the organizational capacities of 

the Department to implement this strategic plan.

# Metric Data Source Data Quality, Limitations, and Improvements

6.1.A Staffing gaps percentage Mission Critical 

Occupation (MCO) 

Staffing Gap Report

The Department’s Budget Service obtains the staffing gap 

data from the Department’s Federal Personnel and Payroll 

System (FPPS) Datamart roster and separations reports. 

As FPPS is a user-driven system, the data used for the 

Mission Critical Occupation (MCO) Staffing Gap Report are 

only as reliable as the data that are entered into FPPS. 

Human Capital and Client Services (HCCS) intends to 

improve data in FPPS by updating standard operating 

procedures, implementing process maps, and training 

customers and HCCS staff to follow these new processes 

when entering data into the system.

6.1.B EVS Engagement Index Office of Personnel 

Management (OPM) 

Federal Employee 

Viewpoint Survey (FEVS) 

results

Any questionable FEVS data points are brought to the 

attention of OPM. In turn, the OPM point of contact 

responds to the inquiry. Further, if there are questions 

regarding the FEVS data, the Department works with the 

OPM point of contact to obtain clarity.

The 2013 appendix A indicated that the Department would 

work to increase the participation rate to reduce nonresponse bias in the FEVS data. However, in 2014 the 

participation rate decreased from 68.9% in 2013 to 63.3% 

in 2014.
APPENDICES

DATA VALIDITY AND VERIFICATION

FY 2014 Annual Performance Report and FY 2016 Annual Performance Plan—U.S. Department of Education 110 

# Metric Data Source Data Quality, Limitations, and Improvements

6.1.C Time to hire Workforce Transformation 

Tracking System (WTTS) 

and Entrance on Duty 

System (EOS)

Data are pulled biweekly and tested. Results are tracked 

and analyzed for consistency and monitored for patterns or 

trends; anomalies are identified and explored to determine 

root cause and then corrected (or work-around developed 

and noted pending further analysis/correction). The data 

are entered into the systems by Department customers and 

Human Capital and Client Services (HCCS) staff, so quality 

of data, and in-turn the quality of the final calculations, are 

only as reliable as the information entered into the system.

As HCCS continues to reduce the shortage of trained 

human capital practitioners and staff become more 

proficient leveraging systems’ capabilities and streamlining 

processes, we expect data quantity and quality will 

increase which will negate the need to have an order of 

precedence for substitute/back up data when measured 

data points are missing.

Even though the 90 Day Model’s time to hire methodology 

was successfully advanced from initial concept to a stable, 

sustainable agency-wide process in less than a year, it has 

not undergone peer review or independent verification and 

validation to validate its rigor. Facilitated peer review is the 

recommended next step to ensure continued Departmentwide application of this metric as a meaningful measure. 

6.1.D Effective Communication Index Office of Personnel 

Management (OPM) 

Federal Employee 

Viewpoint Survey (FEVS) 

results

Any questionable FEVS data points are brought to the 

attention of OPM. In turn, the OPM point of contact 

responds to the inquiry. Further, if there are questions 

regarding the FEVS data, the Department works with the 

OPM point of contact to obtain clarity.

The 2013 appendix A indicated that the Department would 

work to increase the participation rate to reduce nonresponse bias in the FEVS data. However, in 2014 the 

participation rate decreased from 68.9% in 2013 to 63.3% 

in 2014. 
APPENDICES

DATA VALIDITY AND VERIFICATION

FY 2014 Annual Performance Report and FY 2016 Annual Performance Plan—U.S. Department of Education 111 

# Metric Data Source Data Quality, Limitations, and Improvements

6.2.A Percentage of A-133 Single Audits Overdue 

for resolution

Office of the Chief 

Financial Officer’s (OCFO) 

Audit Accountability & 

Resolution Tracking 

System (AARTS)

Calculations for this metric are determined by dividing the 

total number of audits that are overdue at the end of the 

Department’s fiscal year by the total number of audits in the

Department’s inventory. 

Access rights to the AARTS database are managed by 

Office of the Chief Financial Officer (OCFO) and all users 

receive annual IT security training to help ensure data 

integrity. A Quality Control reviewer initiates a weekly 

upload of A-133 audit data to AARTS through a file 

submission directly from the Federal Audit Clearinghouse

(FAC). An AARTS administrator (separate from the Quality 

Control reviewer) must verify the uploaded data with the 

actual audits. Data for individual programs are verified by 

Principal Offices (POs) through periodic review by 

Responsible Managers and Audit Liaison Officials. In 

addition, the specific data for this metric are verified by POs 

each month as part of monthly Dashboard reports.

Data are validated by OCFO monthly. Staff work to 

reconcile data reported on the Dashboards with any 

discrepancies reported by the POs.

6.2.B Compliance rate of contractor evaluation 

performance reports

Past Performance 

Information Retrieval 

System, www.ppirs.gov, 

“PPIRS Compliance 

Report”

Compliance rates of contractor performance evaluations 

are set by OMB and are calculated by use of a Government 

wide reporting tool available in the PPIRS (www.ppirs.gov).

Office of the Chief Financial Officer (OCFO) staff analyzes 

the PPIRS report weekly to ensure that each contractor 

performance evaluation reflected on that report should 

appear on that report, and to rectify any errors on the 

report.
APPENDICES

DATA VALIDITY AND VERIFICATION

FY 2014 Annual Performance Report and FY 2016 Annual Performance Plan—U.S. Department of Education 112 

# Metric Data Source Data Quality, Limitations, and Improvements

6.3.A Percentage of states who annually rate the 

Department’s technical assistance as helping 

build state capacity to implement education 

reforms.

The Reform Support 

Network Annual State and 

CoP Leads Survey 

(administered Q3 2014)

The source for the data is the Race to the Top (RTT)

Annual Leads Survey, which was sent to all 19 Race to the 

Top Phase 1, 2 and 3 states. Because RTT is a 

discretionary program, we only surveyed grantee states 

who received direct technical assistance.

The data collection process includes online survey software 

with questions in which respondents click on the 

appropriate response choice and data is then downloaded 

directly to analysis software. By configuring the online 

software and downloading the data directly, we reduce the 

opportunities for human error in data entry. Additionally, we 

utilize data quality assurance procedures which include 

having analysts review the data by hand upon download 

and run basic descriptive statistics to illustrate downloading 

issues, sample size concerns, variable integrity, data types, 

and other potential data concerns. We then merge with 

existing data files using established syntax which provides 

a further check to identify potential data errors. Merged 

data is examined again for potential data concerns. These 

procedures are used across a number of federal projects 

and are well-established and effective for verifying data 

integrity.

6.4.A Number of ED IT security incidents Operational Vulnerability

Management Solution 

(OVMS) System

Office of the Chief Information Officer (OCIO) has standard 

operating procedures (SOP) to verify and validate the data: 

1) OVMS SOP, 2) Incident Response Tracking SOP, and 

3) OVMS Checklist. This process is executed on a weekly 

basis.

6.4.B EVS Results-Based Performance Culture 

Index

Office of Personnel 

Management (OPM) 

Federal Employee 

Viewpoint Survey (FEVS) 

results

Any questionable FEVS data points are brought to the 

attention of OPM. In turn, the OPM point of contact 

responds to the inquiry. Further, if there are questions 

regarding the FEVS data, the Department works with the 

OPM point of contact to obtain clarity.

The 2013 appendix A indicated that the Department would 

work to increase the participation rate to reduce nonresponse bias in the FEVS data. However, in 2014 the 

participation rate decreased from 68.9% in 2013 to 63.3% 

in 2014.

6.4.C EVS Leadership and Knowledge 

Management Index

Office of Personnel 

Management (OPM) 

Federal Employee 

Viewpoint Survey (FEVS) 

results
APPENDICES

DATA VALIDITY AND VERIFICATION

FY 2014 Annual Performance Report and FY 2016 Annual Performance Plan—U.S. Department of Education 113 

# Metric Data Source Data Quality, Limitations, and Improvements

6.4.D Total usable square footage Department’s Master 

Space Management Plan

Because usable square footage relates directly to rent 

costs, the Department uses the same data verification and 

validation procedures. Specifically, the data are collected 

directly from Occupancy Agreements and rent bills per 

building.

The data are derived from historic examples and relevant 

experience. Department leadership has agreed to a set of 

assumptions by which the data are based. Leadership has 

reached out to subject matter experts to broaden the scope 

of the data set, and lower risks of missing contingencies 

that may affect the data. At each step, the data are 

reviewed independently to double check the work of each 

team member and provide quality control. These processes 

help ensure the data’s completeness and reliability. 

For the baseline data, the Department made the following 

assumptions: 

1) All leased buildings: 2% is applied for anticipation of CPI 

(Consumer Price Index) annual increases on the 

anniversary date of the active lease/occupancy agreement 

(OA); and 2.5% is applied for anticipation of annual tax 

increases. 

2) All federal buildings: 2.5% is applied for operating cost 

escalations on the anniversary date of the active OA. 

3) 20% is applied to all federal buildings after an OA has 

expired and a new OA is unavailable. (Projected increase 

on the appraisal.) 

4) 40% is applied to all leased buildings after an OA has 

expired and a new OA is unavailable. (Projected increase 

on the market rent.) 

5) If a new OA is unavailable, 3-months early rent is applied 

to all buildings that are relocating due to possible 

Department delays. Example: Changes made to the 

designs after Scope of Work (SOW) is completed. 
APPENDICES

DATA VALIDITY AND VERIFICATION

FY 2014 Annual Performance Report and FY 2016 Annual Performance Plan—U.S. Department of Education 114 

# Metric Data Source Data Quality, Limitations, and Improvements

6) 3-months late rent is applied to all buildings that are 

relocating due to possible Department delays. For example, 

delays in returning space back to a rentable condition. 

The Department reconciles its rent bills per building on a 

monthly basis.

6.4.E Rent cost Department’s Master 

Space Management Plan

The rent is based on the above assumptions. Data are 

collected directly from Occupancy Agreements and rent 

bills per building. The actual rent may vary significantly if 

the Department relocates to a new leased building and/or 

signs short lease extensions. The Department is leveraging 

the examples and experience of the mobility labs and 

building consolidations programs.

The Department reconciles its rent bills per building on a 

monthly basis. Every six months, leadership will re-evaluate 

the data, the assumptions on which it is based, and 

incorporate actual costs and project schedules. These 

steps will become part of our quality assurance program 

and procedures. Leadership looks to improve 

completeness, reliability, and quality of the data at these 

milestones.