FY 2015 Annual Performance Report and FY 2017 Annual Performance Plan—U.S. Department of Education 119 

Appendix A: Data Validity and Verification

The Government Performance and Results Act Modernization Act of 2010 requires 

agencies to describe the accuracy and reliability of data presented. During 2015, the 

Department continued to strengthen its approach to data verification and validation. 

Details of how the Department assesses the completeness and reliability of the data 

reported are presented as part of this Appendix, and known limitations of the data 

are included also. 

To the best of my knowledge, the data verification and validation process and the 

data sources used provide, to the extent possible, complete and reliable 

performance data pertaining to goals and objectives in our FY 2014–18 Strategic 

Plan. Through a process of continuous improvement, the Department continues to 

assess its validation process and welcomes input from stakeholders. 

/s/ 

John B. King, Jr. 

Acting Secretary 

February 9, 2016

 
APPENDICES

DATA VALIDITY AND VERIFICATION

FY 2015 Annual Performance Report and FY 2017 Annual Performance Plan—U.S. Department of Education 120 

GOAL 1: POSTSECONDARY EDUCATION, CAREER AND TECHNICAL EDUCATION, AND ADULT EDUCATION. Increase college access, affordability, quality, 

and completion by improving postsecondary education and lifelong learning opportunities for youths and adults.

Metric No. Metric (Indicator) Data Source Data Quality, Limitations and Improvements

1.1.A Rate of increase in net price for public 

4-year IHEs

Integrated Postsecondary Education Data 

System (IPEDS) Data Center

Average net price is calculated only on the universe of students who receive financial aid. 

It does not include students who are attending college without financial aid. Thus, the 

metric represents the average net price for a subset of students and is not representative 

of the cost of college for all students.

Data quality and limitations are identified in IPEDS First Look Publications, “Data 

Collection Procedures,” and IPEDS methodology available at 

http://nces.ed.gov/pubs2014/2014067.pdf.

Graduation Rates for Selected Cohorts, 2005–10; and Student Financial Aid in 

Postsecondary Institutions, Academic Year 2012–13: First Look (Provisional Data) NCES 

2014 - NCES Number: 2014105 Release Date: November 20, 2014, available at: 

http://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2014105

Postsecondary Institutions and Price of Attendance in 2013–14; Degrees and Other 

Awards Conferred: 2012–13; and 12-Month Enrollment: 2012–13: First Look (Provisional 

Data), available at: http://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2014066rev

1.1.B Rate of increase in net price for public 

2-year IHEs

1.1.C Percent of High School Seniors Filing a 

FAFSA

The denominator is the number of 

graduating seniors according to the most 

recent projection by NCES. The numerator 

is from Federal Student Aid’s (FSA’s) 

Central Processing System and is based on 

the number of applications during the first 

nine months of the application cycle that 

are—as of September 30 of the first year of 

the application cycle—complete (not 

rejected); first-time filers; incoming 

freshmen, with or without previous college 

attendance; age 18 or less as of June 30 of 

the first year of the application cycle; 

reporting high school diploma attainment; 

and attended a high school in the fifty states 

and Washington, DC.

Since year of high school graduation is not asked on the FAFSA, several assumptions are 

made and specific criteria are used (such as age of applicant) to identify those likely to be 

high school seniors. These assumptions and criteria are applied consistently across all 

baseline year and future calculations.

These calculations also restrict the application period to the first nine months of the 

application cycle (the close of the fiscal year), rather than the entire 18 months. Because 

most applicants, including high school seniors, file their FAFSA prior to the start of the 

upcoming academic year (usually before fiscal year end), this decision better aligns the 

performance metric with the fiscal year where most of the performance occurred. The 

alternative is waiting for the close of the18-month cycle, where a performance metric 

would mostly reflect performance from an earlier fiscal year.

1.1.D Index of national annual aggregate 

earnings of VR consumers (based on 

the number of competitive integrated 

employment outcomes, hours worked, 

and hourly wages of VR Consumers)

Rehabilitation Services Administration 

(RSA)- 911

Instructions for submitting the RSA-911 data are provided to agencies. Agency questions 

for coding particular data elements are provided by Data Unit staff on a case-by-case 

basis. 

RSA provides the agencies with the edit program that we use and are encouraged to run 

their data using the program often, at least quarterly, during the year. In addition, 

agencies have edits in their own systems which run additional checks.

1.1.E Index of national annual aggregate 

earnings of Transition-Age Youth 

(based on the number of competitive 

integrated employment outcomes, 

hours worked, and hourly wages of VR 

Transition-Age Youth)
APPENDICES

DATA VALIDITY AND VERIFICATION

FY 2015 Annual Performance Report and FY 2017 Annual Performance Plan—U.S. Department of Education 121 

Metric No. Metric (Indicator) Data Source Data Quality, Limitations and Improvements

1.2.A Number of low-performing institutions 

with high loan default rates and low 

graduation rates

FSA Cohort Default Rate (CDR) Report, 

September 2015

IPEDS Data Center

The number of low-performing Institutions of Higher Education (IHEs) is calculated by first 

identifying institutions with a cohort default rate of 30 percent or higher (the threshold that 

jeopardizes an institution’s access to federal financial aid if that level is sustained for three 

consecutive years). The graduation rate within 150 percent of normal time for each of 

those IHEs is then extracted from IPEDS, along with the transfer-out rate, if applicable, 

and the two rates are combined to create a “completion rate.” If the completion rate is 

below the average completion rate for similar types of U.S. Title-IV eligible IHEs—

separately categorized as four-year, two-year, and less-than-two-year—then the IHE is 

included in the number of low-performing institutions. Community colleges that offer a 

limited number of bachelor’s degrees, which are categorized as four-year (primarily 

associate’s degree-granting) institutions in IPEDS, are treated as two-year IHEs in terms 

of calculating graduation rates and the comparable average completion rate. For fouryear IHEs, the graduation rate is based on the degree- or certificate-seeking cohort, not 

the bachelor’s-seeking cohort, since many four-year IHEs offer certificates and 

associate’s degrees in addition to bachelor’s degrees, and thus the broader cohort is 

more indicative of their performance. Institutions which do not have a graduation rate 

cohort (degree-seeking, first-time, full-time students) and thus no comparable 

graduation/completion rate are not included in the count, as they are institutions that 

cannot be found in the IPEDS database. The calculation includes an assumption that the 

cohort default rates for all institutions for the baseline year will not change. In actuality, 

institutions can appeal their rates after publication and, if justified, those rates may be 

changed, thus changing the record from one year to the next. This means the baseline 

number, or the number in any given year, may not be replicable using revised data in 

subsequent reports.

Graduation rate data quality and limitations are identified in IPEDS First Look 

Publications, “Data Collection Procedures,” and IPEDS methodology available at 

http://nces.ed.gov/pubs2014/2014067.pdf. Data are checked by NCES for consistency.

CDR data quality and limitations are available at 

http://ifap.ed.gov/DefaultManagement/finalcdrg.html.

The cohort default rate data, which serves as the basis of the spreadsheet, is verified by 

FSA prior to publication. Because the process for manually entering the completion rate 

data is so labor-intensive, there is not a process for Independent Verification and 

Validation (IV&V) verifying the accuracy of the process for extracting the data from IPEDS 

or the accuracy of the data that has been entered into the spreadsheet.

1.3.A Degree attainment among 25–34-yearold age cohort

NCES Digest of Education Statistics, Table 

104.30 

(https://nces.ed.gov/programs/digest/d14/tab

les/dt14_104.30.asp), Number of persons 

age 18 and over, by highest level of 

educational attainment, sex, race/ethnicity, 

and age: 2014. Tabulated from Current 

Population Survey data, U.S. Census

Data quality and limitations are documented in http://www.census.gov/prod/2006pubs/tp66.pdf. NCES tabulates the data, which is verified prior to publication according to NCES 

guidelines.
APPENDICES

DATA VALIDITY AND VERIFICATION

FY 2015 Annual Performance Report and FY 2017 Annual Performance Plan—U.S. Department of Education 122 

Metric No. Metric (Indicator) Data Source Data Quality, Limitations and Improvements

1.3.B Retention rate of first-time degreeseeking undergraduates; Full-time

IPEDS Data Center The retention rate is calculated based on degree-seeking, first-time undergraduates at 

degree-granting U.S. institutions that participate in Title IV.

Institutions employ a variety of methods for determining degree-seeking status to 

determine which students to include in the cohort, but none of these methods is foolproof. 

Furthermore, a student who transfers to another institution in the second year will be 

considered as not retained, suppressing the aggregate rate. (Retention in this context is 

at the reporting institution so transfer-outs are not retained at the institution.) 

Consequently, the retention rate is only an approximation of true retention. 

Data quality and limitations are identified in IPEDS First Look Publications, “Data 

Collection Procedures,” and IPEDS methodology available at 

http://nces.ed.gov/pubs2014/2014067.pdf. Data are checked by NCES for consistency.

IPEDS collects completions by Classification of Instructional Programs (CIP) Code which 

may be aggregated into STEM and non-STEM counts. The Department of Education 

does not currently have a single definition for which CIP codes are STEM, but for the 

purpose here the NCES definition is used: http://nces.ed.gov/pubs2011/2011226.pdf.

1.3.C Retention rate of first-time degreeseeking undergraduates; Part-time

1.4.A Number of STEM postsecondary 

credentials awarded
APPENDICES

DATA VALIDITY AND VERIFICATION

FY 2015 Annual Performance Report and FY 2017 Annual Performance Plan—U.S. Department of Education 123 

GOAL 2: ELEMENTARY AND SECONDARY EDUCATION. Improve the elementary and secondary education system’s ability to consistently deliver excellent 

instruction aligned with rigorous academic standards while providing effective support services to close achievement and opportunity gaps, and ensure all students 

graduate high school college- and career-ready.

Metric No. Metric (Indicator) Data Source Data Quality, Limitations and Improvements

2.1.A Number of states/territories that have 

adopted college- and career-ready 

standards

Elementary and Secondary Education Act

(ESEA) Flexibility Requests and Monitoring

The Office of Elementary and Secondary Education’s (OESE) Student Achievement and 

School Accountability (SASA) office will count the number of states that either (a) have a 

memorandum of understanding in place to implement the Common Core or (b) have a 

letter from an Institute for Higher Education in their state certifying that their state has 

college- and career-ready standards. There are no known data limitations.

The Office of Elementary and Secondary Education’s (OESE) Student Achievement and 

School Accountability (SASA) office monitors states that receive ESEA Flexibility waivers 

in three phases over the three-year waiver. Monitoring includes desk monitoring and onsite monitoring. SASA state leads use a monitoring protocol and rubric to ensure that 

monitoring is consistent across all states. SASA state leads work with their Group Leader 

to finalize monitoring reports. All reports are reviewed by both the Group Leader for 

Monitoring and Technical Assistance and the Director of SASA for consistencies across 

states. States have an opportunity to review the draft report before the final report is 

issued. There are no known data limitations.

2.1.B Number of states/territories that are 

implementing next-generation reading 

and mathematics assessments, aligned 

with college- and career-ready standards

2.2.A Number of states that have fully 

implemented teacher and principal 

evaluation and support systems that 

consider multiple measures of 

effectiveness, with student growth as a 

significant factor

ESEA Flexibility Requests and Monitoring The Office of Elementary and Secondary Education’s (OESE) Office of State Support 

(OSS) office monitors states that receive ESEA Flexibility waivers. Monitoring includes 

desk monitoring and on-site monitoring. OSS state leads use a monitoring protocol and 

rubric to ensure that monitoring is consistent across all states. OSS state leads work with 

their Group Leader to finalize monitoring reports. All reports are reviewed by both the 

Group Leader for Monitoring and Technical Assistance and the OSS Director for 

consistencies across states. States have an opportunity to review the draft report before 

the final report is issued.

Additionally, Chief State School Officers must sign each state’s ESEA Flexibility request 

before it is approved. There are no known data limitations.

2.3.A Disparity in the rates of out-of-school 

suspensions for students with disabilities 

and youth of color (youth of color metric)

Civil Rights Data Collection (CRDC) The Office for Civil Rights (OCR) strives to ensure CRDC data are an accurate and 

comprehensive depiction of student access to educational opportunities in school 

districts. The submission system includes a series of embedded edit checks to ensure 

significant data errors are corrected before the district submits its data. Additionally, each 

district is required to certify the accuracy of its submission. Only a district superintendent, 

or the superintendent’s designee, may certify the CRDC submission. Ultimately, the 

quality of the CRDC data depends on accurate collection and reporting by the 

participating districts.

2.3.B Disparity in the rates of out-of-school 

suspensions for students with disabilities 

and youth of color (SWDs, IDEA only 

metric)
APPENDICES

DATA VALIDITY AND VERIFICATION

FY 2015 Annual Performance Report and FY 2017 Annual Performance Plan—U.S. Department of Education 124 

Metric No. Metric (Indicator) Data Source Data Quality, Limitations and Improvements

2.4.A Number of persistently low graduation 

rate high schools

EDFacts EDFacts works with the data stewards to determine the appropriate business rule checks 

for these data. The checks that were done on these data include:

 File validation and format checks: Identifies file submissions that don’t conform 

to the file format (unable to be processed), the data universe (school/LEA/state 

directory), or the reporting requirements (mandatory reporting fields).

 Submission Edit Business Rules: Data consistency checks, which produce 

warning messages for states to double-check their submitted data, are 

programmed into the EDFacts Submission System.

 SAS Data Quality Reviews: Post submission checks to determine 

completeness, consistency, and comparability.

 At the end of a collection period these rules are run against the submitted data 

and presented to the program office. It is the responsibility of the program 

office, in consultation with the EDFacts staff, to determine which errors should 

be escalated to the state for further review. Many of these errors were sent 

along to the state for remediation (data explanation/update/correction). OESE 

should be considered the point-of-contact for identifying which rules were 

escalated and the result of those escalations.

Data concerns for state, district and school level 2013–14 data submissions are 

documented and available in two places:

a) State Notes to submitted Adjusted Cohort Graduation Rate (ACGR) data 

are available within the Department’s ED Data Express Tool

b) Identified Data Anomalies are documented at the state level in appendix B 

to the file documentation released along with school and district level data 

files at http://www2.ed.gov/about/inits/ed/edfacts/data-files/index.html.

Additional concerns, if any are identified, related to the calculated national rate or work 

done prior to 2013–14 utilizing an imputation methodology to address states that were not 

yet calculating ACGR, will be available in future releases of the NCES report “Public High 

School Four-Year On-Time Graduation Rates and Event Dropout Rates: School Year 

####–##.” The most recent report covers 2010–11 and 2011–12. Reports covering more 

recent years of data are being prepared for release at this time and will be available at 

http://nces.ed.gov/ccd/pub_dropouts.asp in the coming months.

Path to public release and national rates: For each collection cycle states report their 

calculated cohort counts and ACGR graduation rates. These counts and rates are tested 

against a number of business rules for format, consistency, completeness, and 

comparability. Those business rule checks are delivered to the Office of Elementary and 

Secondary Education (OESE) for follow-up correspondence with the state(s). The 

program office, in consultation with NCES EDFacts staff and other stakeholders, meet to 

determine which issues identified by the business rules should be raised with the state for 

explanation, update, or correction. Following that review, data are then aggregated to the 

national level for the purpose of calculating and publishing a national rate. NCES 

processes these data to force conformity of reporting categories (mapping reported 

“Major Racial/Ethnic Groups” to the traditional 5 racial/ethnic groups) and imputes any 

missing data (3 states were imputed for 2011–12, 1 state was imputed for 2012–13, no 

imputations were necessary for 2013–14 forward). After imputations are made, NCES 

produces a national rate for the country as a whole; a rate which is representative of 

every state. Once produced, NCES documents any remaining issues with these data, and 

the aggregation and imputation methodology in a public report. That report, 

documentation, and the associated data tables are put through several stages of review 

including independent reviews at the division (Administrative Data), center (NCES), and 

POC (IES) levels prior to public dissemination.
APPENDICES

DATA VALIDITY AND VERIFICATION

FY 2015 Annual Performance Report and FY 2017 Annual Performance Plan—U.S. Department of Education 125 

Metric No. Metric (Indicator) Data Source Data Quality, Limitations and Improvements

2.4.B Percent of Cohort 1 priority schools that 

have met the state exit criteria and 

exited priority school status

ESEA Flexibility plans do not allow for one standard methodology to determine whether or 

not a school “met the state exit criteria.” This will need to be looked at manually, state-bystate, once the list of schools exiting priority status has been identified. 

2.4.C Percent of Cohort 1 focus schools that 

have met the state exit criteria and 

exited focus school status

2.5.A Percentage of high school and middle 

school teachers who teach STEM as 

their main assignment who hold a 

corresponding undergraduate degree

Schools and Staffing Survey (SASS), 

NCES

The methods report for the 2011–12 SASS is not yet released. Study documentation from 

the 2007–08 survey is available at http://nces.ed.gov/surveys/sass/methods0708.asp.

2.5.B Number of high school graduates who 

have taken at least one STEM AP exam

College Board/AP administrative records College Board Public School List is updated annually by state DOEs; thus small changes 

to the list over time are to be expected as schools open, close, and/or merge. Students 

are assigned to graduating cohorts based on self-reported information (i.e., grade level 

and/or graduation year) provided at the time of registration (in the case of SAT) or test 

administration (in the case of AP and PSAT). The College Board matches students’ data 

across programs to identify the most recent valid value when assigning students to 

cohorts.
APPENDICES

DATA VALIDITY AND VERIFICATION

FY 2015 Annual Performance Report and FY 2017 Annual Performance Plan—U.S. Department of Education 126 

GOAL 3: EARLY LEARNING. Improve the health, social-emotional, and cognitive outcomes for all children from birth through 3rd grade, so that all children, 

particularly those with high needs, are on track for graduating from high school college- and career-ready.

Metric No. Metric (Indicator) Data Source Data Quality, Limitations and Improvements

3.1.A Number of states with Quality Rating 

and Improvement Systems (QRIS) that 

meet high quality benchmarks for child 

care and other early childhood programs

Biennial Child Care Development Fund 

(CCDF) Report of States Plans with annual 

updates from states and territories 

(HHS/Office of Childcare)

The data are self-reported by the states in their CCDF state plans. In addition, the data do 

not take into account the participation or coverage of the QRIS systems.

3.2.A Number of states and territories with 

professional development systems that 

include core knowledge and 

competencies, career pathways, 

professional development capacity 

assessments, accessible professional 

development opportunities, and financial 

support for childcare providers

Biennial Child Care Development Fund 

(CCDF) Report of State Plans (HHS/Office 

of Childcare)

The data are self-reported by the states in their CCDF state plans. In addition, the data do 

not take into account the participation or coverage of the professional development 

systems.

3.3.A Number of states collecting and 

reporting disaggregated data on the 

status of children at kindergarten entry 

using a common measure

Race to the Top (RTT)-Early Learning 

Challenge (ELC) Technical Assistance 

Center

The data are limited to the 20 Race to the Top-Early Learning Challenge (ELC) states. 

Department staff has requested Kindergarten Entry Assessment (KEA) data be collected 

in the 2015–16 academic year as part of the State of Preschool data collection and has 

requested to add KEA data reporting in EDFacts. In addition, the new Preschool 

Development Grants will provide data on additional states.
APPENDICES

DATA VALIDITY AND VERIFICATION

FY 2015 Annual Performance Report and FY 2017 Annual Performance Plan—U.S. Department of Education 127 

GOAL 4: EQUITY. Increase educational opportunities for underserved students and reduce discrimination so that all students are well-positioned to succeed. 

Metric No. Metric (Indicator) Data Source Data Quality, Limitations and Improvements

4.1.A National high school graduation rate EDFacts EDFacts works with the data stewards to determine the appropriate business rule checks 

for these data. The checks that were done on these data include:

 File validation and format checks: Identifies file submissions that don’t conform 

to the file format (unable to be processed), the data universe (school/LEA/state 

directory), or the reporting requirements (mandatory reporting fields).

 Submission Edit Business Rules: Data consistency checks, which produce 

warning messages for states to double-check their submitted data, are 

programmed into the EDFacts Submission System.

 SAS Data Quality Reviews: Post submission checks to determine 

completeness, consistency, and comparability.

 At the end of a collection period these rules are run against the submitted data 

and presented to the program office. It is the responsibility of the program 

office, in consultation with the EDFacts staff, to determine which errors should 

be escalated to the state for further review. Many of these errors were sent 

along to the state for remediation (data explanation/update/correction). OESE 

should be considered the point-of-contact for identifying which rules were 

escalated and the result of those escalations.

Data concerns for state, district and school level 2013–14 data submissions are 

documented and available in two places:

a) State Notes to submitted Adjusted Cohort Graduation Rate (ACGR) data 

are available within the Department’s ED Data Express Tool

b) Identified Data Anomalies are documented at the state level in appendix B 

to the file documentation released along with school and district level data 

files at http://www2.ed.gov/about/inits/ed/edfacts/data-files/index.html.

Additional concerns, if any are identified, related to the calculated national rate or work 

done prior to 2013–14 utilizing an imputation methodology to address states that were not 

yet calculating ACGR, will be available in future releases of the NCES report “Public High 

School Four-Year On-Time Graduation Rates and Event Dropout Rates: School Year 

####–##.” The most recent report covers 2010–11 and 2011–12. Reports covering more 

recent years of data are being prepared for release at this time and will be available at 

http://nces.ed.gov/ccd/pub_dropouts.asp in the coming months.
APPENDICES

DATA VALIDITY AND VERIFICATION

FY 2015 Annual Performance Report and FY 2017 Annual Performance Plan—U.S. Department of Education 128 

Metric No. Metric (Indicator) Data Source Data Quality, Limitations and Improvements

Path to public release and national rates: For each collection cycle states report their 

calculated cohort counts and ACGR graduation rates. These counts and rates are tested 

against a number of business rules for format, consistency, completeness, and 

comparability. Those business rule checks are delivered to the Office of Elementary and 

Secondary Education (OESE) for follow-up correspondence with the state(s). The 

program office, in consultation with NCES EDFacts staff and other stakeholders, meet to 

determine which issues identified by the business rules should be raised with the state for 

explanation, update, or correction. Following that review, data are then aggregated to the 

national level for the purpose of calculating and publishing a national rate. NCES 

processes these data to force conformity of reporting categories (mapping reported 

“Major Racial/Ethnic Groups” to the traditional 5 racial/ethnic groups) and imputes any 

missing data (3 states were imputed for 2011–12, 1 state was imputed for 2012–13, no 

imputations were necessary for 2013–14 forward). After imputations are made, NCES 

produces a national rate for the country as a whole; a rate which is representative of 

every state. Once produced, NCES documents any remaining issues with these data, and 

the aggregation and imputation methodology in a public report. That report, 

documentation, and the associated data tables are put through several stages of review 

including independent reviews at the division (Administrative Data), center (NCES), and 

POC (IES) levels prior to public dissemination.

4.2.A Percentage of proactive civil rights 

investigations launched annually that 

address areas of concentration in civil 

rights enforcement

Office for Civil Rights’ (OCR) Case 

Management System (CMS) and 

Document Management (DM) systems

The Contracts and Acquisition Management/Case Management database utilized to 

collect data for this metric has built-in validation checks (such as requiring certain data 

elements and valid date entries), as well as automated entries based on other information 

entered into the system by staff, to reduce errors in data entry. OCR also employs 

additional safeguards to ensure data accuracy such as (1) periodic monthly checks to 

address missing or inconsistent entries, (2) publication of guidance materials including 

data entry requirements, codes, definitions, checklist and protocol for staff responsible for 

entering data; and (3) additional training and support for primary users inputting the data.

4.2.B Percentage of proactive civil rights 

investigations resolved annually that 

address areas of concentration in civil 

rights enforcement
APPENDICES

DATA VALIDITY AND VERIFICATION

FY 2015 Annual Performance Report and FY 2017 Annual Performance Plan—U.S. Department of Education 129 

GOAL 5: CONTINUOUS IMPROVEMENT OF THE U.S. EDUCATION SYSTEM. Enhance the education system’s ability to continuously improve through better 

and more widespread use of data, research and evaluation, evidence, transparency, innovation, and technology.

Metric No. Metric (Indicator) Data Source Data Quality, Limitations and Improvements

5.1.A Number of public data sets included in 

ED Data Inventory and thus linked to 

Data.gov or ED.gov websites

Data Strategy Team Data Inventory and 

the public ED Data Inventory at 

http://datainventory.ed.gov

The data are validated with a crosswalk between Inventory entries and the listing of public 

Department datasets, ensuring that the data described in the ED Data Inventory is 

publicly available at the identified web address.

5.1.B Number of states linking K-12 and 

postsecondary data with workforce data

State Longitudinal Data Systems (SLDS) 

grant monitoring (monthly updates from 

states, annual performance reports, final 

performance reports, and site visits)

Data are collected through monitoring of states with active SLDS grants. The data are 

limited to this population. While 47 states and territories have received these grants over 

time, by June 2015, there will be fewer than 25 states with active grants, which leads to 

incomplete and not up-to-date data from states that either have not receive grants or that 

do not have active grants.

If the state is focusing on an early childhood data system, the state’s reporting might 

reflect only those development efforts, and not statewide P20W (preschool to college 

workforce) development efforts. There are additional sources for information about state 

data linkages in the field. For example the Data Quality Campaign (DQC) conducts 

surveys of state capacity to collect, store, link, and use data. Because the Department 

does not use the same definitions as DQC, our figures tend to be lower than theirs. For 

example, we require that a state possess the capacity to follow its own students in order 

for us to report that the state has the linkage in place. Similarly, we enable states to report 

on whether particular linkages are planned, in progress, or complete, and report that a 

state has a linkage when the state reports that the project is complete; DQC might give a 

state credit for an ‘in progress’ or pilot-stage linkage.

A survey administered to the universe of states and territories would enable more 

systematically collect data about all states’ capacity for data linkages and data use. There 

is a concern, however, that if those data were to be used for public reporting, states might 

begin to overstate their capacities, particularly on data that are also publicly reported by 

organizations such as DQC. Currently, data from monitoring is used in an iterative, 

formative approach to program improvement; our technical assistance program is 

designed to support states’ efforts to improve their systems. This relies on states being 

honest about their own internal capacities.

5.1.C Number of states linking K-12 with early 

childhood data
APPENDICES

DATA VALIDITY AND VERIFICATION

FY 2015 Annual Performance Report and FY 2017 Annual Performance Plan—U.S. Department of Education 130 

Metric No. Metric (Indicator) Data Source Data Quality, Limitations and Improvements

5.2.A Average time to close “cases” (PTAC + 

FPCO)

Case Tracking System (CTS) 

Monthly metric reports

The term “case” refers to requests for quick, informal responses to routine questions 

related to student privacy. These requests are received via e-mail, the Family Policy 

Compliance Office (FPCO) / Privacy Technical Assistance Center (PTAC) resource 

website, or by telephone and subsequently entered into the Case Tracking System (CTS).

In contrast, “correspondence and complaints” refers to written complaints of alleged 

failures to comply with Family Educational Rights and Privacy Act (FERPA) / Protection of 

Pupil Rights Amendment (PPRA) filed with FPCO; requests for formal written 

guidance/interpretation laws administered by FPCO; and, to the reporting of instances of 

data breaches by educational agencies and institutions. These inquiries are logged into 

the Correspondence Control Manager (CCM) System, given a tracking control number, 

and assigned to FPCO staff.

The preliminary data for this metric are reviewed at least weekly for verification. If 

anomalous data are identified in the periodic reviews or when anomalies are suspected, 

individual cases are examined individually to identify if they were properly closed or if their 

status was entered incorrectly. When appropriate, corrections are made. Staff responsible 

for entering data into the CTS will continue to be trained on policies and procedures.

The monthly metric reports are scrutinized by the Director of FPCO, the Contracting 

Officer Representative for the PTAC contractors, and the Department's Chief Privacy 

Officer, to assure completeness and reliability of data and to recommend any 

improvements to the CTS or modifications to the standard operating procedures. The 

quarter entry represents the fiscal year to date average days to close as of the end of that 

quarter taken from the corresponding monthly report.

5.3.A Percentage of select new 

(noncontinuation) discretionary grant 

dollars that reward evidence

Forecast Report issued by the Office of the 

Chief Financial Officer (OCFO) and final 

Funding Reports from relevant programs

In determining which discretionary grants are considered “evidence-based” (i.e., the 

numerator when calculating the percentage), the Department includes all programs that 

rewarded applicants with supporting evidence of promise or better (per the EDGAR 

evidence framework). This could be done through a competitive preference or absolute 

priority, an eligibility requirement, or a selection factor. Only the amounts of the grants 

awarded for those projects were counted. In determining what counts as discretionary 

funding (i.e., the denominator when calculating the percentage), the Department includes 

all programs for which the EDGAR evidence framework could conceivably work. In Fiscal 

Year 2015, the Department counted all discretionary grant programs except for those 

programs run through the Institute of Education Sciences (IES), which are already 

evidence-based and would not be candidates for the EDGAR evidence framework).

5.3.B Number of peer-reviewed, full-text 

resources in the Education Resources 

Information Center

Education Resources Information Center 

(ERIC)

To be counted as a full-text, peer-reviewed article in ERIC, the document had to be 

indexed in ERIC and designated with a “peer-reviewed” and “full-text” flag, both of which 

are available on the public ERIC website

The ERIC contractor uses specified quality assurance procedures. In addition, the IES 

program officer pays close attention to the metric in their review of deliverables.
APPENDICES

DATA VALIDITY AND VERIFICATION

FY 2015 Annual Performance Report and FY 2017 Annual Performance Plan—U.S. Department of Education 131 

Metric No. Metric (Indicator) Data Source Data Quality, Limitations and Improvements

5.3.C Number of reviewed studies in the What 

Works Clearinghouse (WWC) database

What Works Clearinghouse (WWC) To be counted as a reviewed study, the study had to be listed in the What Works 

Clearinghouse’s publicly available Database of Reviewed Studies. 

The What Works Clearinghouse contractors use specified quality assurance procedures. 

In addition, the IES program officers pay close attention to the metric in their review of 

deliverables.

5.4.A Percentage of schools in the country that 

have actual internet bandwidth speeds 

of at least 100 Mbps

Education Superhighway (for baseline), 

Consortium for School Networking 

(CoSN)/AASA E-rate Infrastructure Survey

The Department uses an external data source for this metric and relies on the external, 

third party’s verification and validation methodology. Based on the information provided, 

the response rate for this survey may not be sufficient to ensure that the data are 

representative of all districts in the country. The Department is exploring the feasibility of 

collecting data on access to and use of education technology from a representative 

sample of schools and districts across the country. In the meantime, we believe these 

data are the best currently available and provide useful information to gauge progress on 

this metric.
APPENDICES

DATA VALIDITY AND VERIFICATION

FY 2015 Annual Performance Report and FY 2017 Annual Performance Plan—U.S. Department of Education 132 

GOAL 6: U.S. DEPARTMENT of EDUCATION CAPACITY. Improve the organizational capacities of the Department to implement this strategic plan.

Metric No. Metric (Indicator) Data Source Data Quality, Limitations and Improvements

6.1.A Staffing gaps percentage Mission Critical Occupation (MCO) Staffing 

Gap Report

The Department’s Budget Service obtains the staffing gap data from the Department’s 

Federal Personnel and Payroll System (FPPS) Datamart roster and separations reports. 

As FPPS is a user-driven system, the data used for the Mission Critical Occupation 

(MCO) Staffing Gap Report are only as reliable as the data that are entered into FPPS. 

The Department’s Office of Human Resources (OHR) intends to improve data in FPPS by 

updating standard operating procedures, implementing process maps, and training 

customers and HCCS staff to follow these new processes when entering data into the 

system.

6.1.B EVS Engagement Index Office of Personnel Management (OPM) 

Federal Employee Viewpoint Survey 

(FEVS)

Any questionable data points from the Federal Employee Viewpoint Survey (FEVS) are 

brought to the attention of the U.S. Office of Personnel Management (OPM). In turn, the 

OPM point of contact responds to the inquiry. Further, if there are questions regarding the 

FEVS data, the Department works with the OPM point of contact to obtain clarity.

The Engagement index score is calculated by OPM by first determining the percent 

positive for each of the 15 items in the sub-indices (i.e., Leaders Lead, Supervisors, and 

Intrinsic Work Experiences). Then the unrounded percent positive scores are averaged 

across the items in the index to get the index score. The overall index score is then 

rounded for reporting purposes.

6.1.C Time to Hire Federal Personnel/Payroll System (FPPS) 

Datamart

Data reflecting actions input into the Workforce Transformation and Tracking System 

(WTTS) are exported from FPPS Datamart and validated via Excel formulas for reporting 

purposes. Actions with negative or missing hiring information are excluded from time to 

hire calculations.

Based on established data handling protocols, criteria established for the 90-Day Hiring 

Model metric, OM is confident the data presented is reasonably accurate and consistent.

Data are pulled biweekly and tested. Results are tracked and analyzed for consistency 

and monitored for patterns or trends; anomalies are identified and explored to determine 

root cause and then corrected (or work-around developed and noted pending further 

analysis/correction). The data are entered into the systems by Department customers and

the Department’s Office of Human Resources (OHR) staff, so quality of data, and in turn 

the quality of the final calculations, are only as reliable as the information entered into the 

system.

As OHR continues to reduce the shortage of trained human capital practitioners and staff 

become more proficient leveraging systems’ capabilities and streamlining processes, we 

expect data quantity and quality will increase which will negate the need to have an order 

of precedence for substitute/back up data when measured data points are missing.

Even though the 90 Day Model’s time to hire methodology was successfully advanced 

from initial concept to a stable, sustainable agencywide process in less than a year, it has 

not undergone peer review or independent verification and validation to validate its rigor.

Facilitated peer review is the recommended next step to ensure continued 

Departmentwide application of this metric as a meaningful measure.

6.1.D Effective Communication Index Office of Personnel Management (OPM) 

Federal Employee Viewpoint Survey 

(FEVS)

Any questionable FEVS data points are brought to the attention of OPM. In turn, the OPM 

point of contact responds to the inquiry. Further, if there are questions regarding the 

FEVS data, the Department works with the OPM point of contact to obtain clarity.

This index score is calculated by the Department by averaging the percent positive scores 

from OPM FEVS questions 53, 58, and 64.
APPENDICES

DATA VALIDITY AND VERIFICATION

FY 2015 Annual Performance Report and FY 2017 Annual Performance Plan—U.S. Department of Education 133 

Metric No. Metric (Indicator) Data Source Data Quality, Limitations and Improvements

6.2.A Percentage of A-133 Single Audits 

Overdue for resolution

Office of the Chief Financial Officer’s 

(OCFO) Audit Accountability & Resolution 

Tracking System (AARTS)

Calculations for this metric are determined by dividing the total number of audits that are 

overdue at the end of the Department’s fiscal year by the total number of audits in the 

Department’s inventory. 

Access rights to the AARTS database are managed by Office of the Chief Financial 

Officer (OCFO) and all users receive annual IT security training to help ensure data 

integrity. A Quality Control reviewer initiates a weekly upload of A-133 audit data to 

AARTS through a file submission directly from the Federal Audit Clearinghouse (FAC). An 

AARTS administrator (separate from the Quality Control reviewer) must verify the 

uploaded data with the actual audits. Data for individual programs are verified by Principal 

Offices (POs) through periodic review by Responsible Managers and Audit Liaison 

Officials. In addition, the specific data for this metric are verified by POs each month as 

part of monthly Dashboard reports.

Data are validated by OCFO monthly. Staff work to reconcile data reported on the 

Dashboards with any discrepancies reported by the POs.

6.2.B Compliance rate of contractor evaluation 

performance reports

Past Performance Information Retrieval 

System, www.ppirs.gov, “PPIRS 

Compliance Report”

Compliance rates of contractor performance evaluations are set by OMB and are 

calculated by use of a Government wide reporting tool available in the PPIRS 

(www.ppirs.gov).

Office of the Chief Financial Officer (OCFO) and Federal Student Aid (FSA) Acquisitions 

staff actively monitor the PPIRS report to ensure that each contractor performance 

evaluation reflected on that report should appear on that report, and to rectify any errors 

on the report.

6.3.A Percentage of states who annually rate 

the Department’s technical assistance 

as helping build state capacity to 

implement education reforms.

Baseline is from the Race to the Top State 

Lead Survey, n=19. Future data will come 

from the Annual Grantee Satisfaction 

Survey

The source for the data is the Race to the Top (RTT) Annual Leads Survey, which was 

sent to all 19 Race to the Top Phase 1, 2 and 3 states. Because RTT is a discretionary 

program, we only surveyed grantee states who received direct technical assistance.

The data collection process includes online survey software with questions in which 

respondents click on the appropriate response choice and data is then downloaded 

directly to analysis software. By configuring the online software and downloading the data 

directly, we reduce the opportunities for human error in data entry. Additionally, we utilize 

data quality assurance procedures which include having analysts review the data by hand 

upon download and run basic descriptive statistics to illustrate downloading issues, 

sample size concerns, variable integrity, data types, and other potential data concerns. 

We then merge with existing data files using established syntax which provides a further 

check to identify potential data errors. Merged data is examined again for potential data 

concerns. These procedures are used across a number of federal projects and are wellestablished and effective for verifying data integrity.

6.4.A Number of Department IT security 

incidents

Operational Vulnerability Management 

Solution (OVMS) System

Office of the Chief Information Officer (OCIO) has standard operating procedures (SOP) 

to verify and validate the data: 1) OVMS SOP, 2) Incident Response Tracking SOP and 3) 

OVMS Checklist. This process is executed on a weekly basis.

6.4.B EVS Results-Based Performance 

Culture Index

Office of Personnel Management (OPM) 

Federal Employee Viewpoint Survey 

(FEVS)

Data verification and validation process is integrated into the OPM FEVS survey results 

validation process. Any questionable FEVS results would be brought to the attention of 

OPM; the Department would then work with the OPM point of contact to obtain clarity.

Data verification and validation process is integrated into the OPM FEVS survey results 

validation process. The Results-Oriented Performance Culture Index score is calculated 

by OPM by first determining the percent positive for each of the 13 items in the index.

Then the unrounded percent positive scores are averaged across all index items to get 

the index score. The index score is then rounded for reporting purposes.

6.4.C EVS Leadership and Knowledge 

Management Index
APPENDICES

DATA VALIDITY AND VERIFICATION

FY 2015 Annual Performance Report and FY 2017 Annual Performance Plan—U.S. Department of Education 134 

Metric No. Metric (Indicator) Data Source Data Quality, Limitations and Improvements

6.4.D Total usable square footage Department’s Master Space Management 

Plan

The Department reconciles its rent bills per building on a monthly basis. Every six months 

the usable square footage is verified with GSA

Because usable square footage relates directly to rent costs, the Department uses the 

same data verification and validation procedures. Specifically, the data are collected 

directly from Occupancy Agreements and rent bills per building.

The data are derived from historic examples and relevant experience. Department 

leadership has agreed to a set of assumptions by which the data are based. Leadership 

has reached out to subject matter experts to broaden the scope of the data set, and lower 

risks of missing contingencies that may affect the data. At each step, the data are 

reviewed independently to double check the work of each team member and provide 

quality control. These processes help ensure the data’s completeness and reliability.

For the baseline data, the Department made the following assumptions:

1) All leased buildings: 2% is applied for anticipation of CPI (Consumer Price Index) 

annual increases on the anniversary date of the active lease/occupancy agreement (OA); 

and 2.5% is applied for anticipation of annual tax increases. 

2) All federal buildings: 2.5% is applied for operating cost escalations on the anniversary 

date of the active OA.

3) 20% is applied to all federal buildings after an OA has expired and a new OA is 

unavailable. (Projected increase on the appraisal.)

4) 40% is applied to all leased buildings after an OA has expired and a new OA is 

unavailable. (Projected increase on the market rent.)

5) If a new OA is unavailable, 3-months early rent is applied to all buildings that are 

relocating due to possible Department delays. Example: Changes made to the designs 

after Scope of Work (SOW) is completed.

6) 3-months late rent is applied to all buildings that are relocating due to possible 

Department delays. For example, delays in returning space back to a rentable condition.

The Department reconciles its rent bills per building on a monthly basis.

6.4.E Rent cost Department’s Master Space Management 

Plan

The Department reconciles its rent bills per building on a monthly basis. 

Data are collected directly from Occupancy Agreements and rent bills per building. The 

actual rent may vary significantly if the Department relocates to a new leased building 

and/or signs short lease extensions. The Department is leveraging the examples and 

experience of the mobility labs and building consolidations programs.

The Department reconciles its rent bills per building on a monthly basis. Every six 

months, leadership will re-evaluate the data, the assumptions on which it is based, and 

incorporate actual costs and project schedules. These steps will become part of our

quality assurance program and procedures. Leadership looks to improve completeness, 

reliability, and quality of the data at these milestones.

Total number of the Department’s 2015 external (public-facing) metrics (indicators): 46