1

Measurement Strategies for Identifying 

Holders of Certificates and Certifications 

Sharon A. Boivin and Isaiah L. O’Rear 

National Center for Education Statistics 

 1990 K Street NW 

Washington, DC 20006 

sharon.boivin@ed.gov 

isaiah.l.o’rear@ed.gov 

Introduction 

In February of 2009, President Obama in his State of the Nation address called for the U.S. to become once again 

―first in the world‖ in postsecondary educational attainment by the year 2020. As a means to this goal, he asked each 

American ―...to commit to at least one year or more of higher education or career training. This can be a community 

college or a four-year school, vocational training or an apprenticeship. But whatever the training may be, every 

American will need to get more than a high school diploma.‖ Concurrently, the deepening recession was leading to 

an increased research and policy interest in the role of industry-conferred and -recognized credentials in preparing 

adults for specific, living-wage jobs. 

In this context, the Council of Economic Advisors and the Office of Management and Budget partnered with the 

Office of the Under Secretary in the U.S. Department of Education to convene senior staff from the Census Bureau, 

the Bureau of Labor Statistics (BLS), and the National Center for Education Statistics (NCES) to discuss ways to 

improve the data on educational certificates and industry-recognized certifications in the federal statistical system in 

order to meet research and policy data needs. After a series of initial meetings outlining the goals and resources of 

the project, the Interagency Working Group on Certificates and Certifications (IWG) began meeting monthly in 

December of 2009.1 

This paper describes a development process that sought to identify a short set of survey items that would count the 

number of educational certificates and industry-recognized certifications in the U.S. adult population. It includes a 

summary analysis of a pilot study designed to evaluate item validity by comparing its results against population 

estimates from other sources and by assessing key characteristics of known credentials. It discusses the tradeoffs 

between using more specificity to define the construct ―certification/license‖ possibly resulting in underreporting of 

the prevalence of these credentials and using less specificity possibly resulting in overreporting. Finally it presents 

recommendations from the IWG for using the new items and next steps in development and deployment in the

federal statistical system. These recommendations and their associated measurement discussions may help survey 

developers decide how best to test and adapt or adopt these items in their studies of education, training, occupational 

credentials, and outcomes of out-of-school youth and adults. 

The IWG guided the item development work, providing detailed feedback at each stage, reviewing table results, 

discussing findings, and developing recommendations. The members of the IWG have also worked within their 

respective agencies to find opportunities to further develop, test, and use the questionnaire items found valid for 

their purpose in this project. NCES managed, staffed, and funded the work described in this paper and is responsible 

for its contents. 

The full results of this development effort, including reports from the focus groups and cognitive interviews and 

detailed tables from the pilot study results, will be available from NCES 2012-190, ―The Adult Training and 

Education Survey (ATES) Pilot Study‖ technical report forthcoming in early Spring 2012. This paper extracts key 

 

1

 See Appendix A for a list of IWG members. 
2

findings and lessons from that report and includes some additional reflections on the tradeoffs involved in how 

narrowly or broadly federal surveys operationally define certifications and licenses. 

Background 

The measurement of educational attainment has been a staple in social and economic research for decades, and 

federal data collections have long collected reliable, valid data on the attainment of postsecondary academic degrees, 

including associate’s, bachelor’s, and graduate and professional degrees. This ability to measure degree-based 

educational credentials has played a significant role in helping researchers learn about the relationship between 

educational attainment and employment outcomes and in shaping public policy aimed at increasing Americans’ 

access to education. In recent years, the role of credentials other than academic degrees in helping adults obtain jobs 

and advance in careers has become a more prominent part of policy discussions. 

However, there is no federal data collection that measures the prevalence of industry-recognized certifications, state 

and local government issued licenses, and subbaccalaureate educational certificates among the US adult population. 

The prevalence and utility of these credentials cannot be assessed through institution-based data collection efforts, 

such as the National Center for Education Statistics’ (NCES) Integrated Postsecondary Education Data System 

(IPEDS). IPEDS collects information on certificates awarded, but is limited to counting awards based on creditbearing instruction in institutions of higher education that participate in federal student financial aid programs. 

Students receive educational certificates from institutions that do not participate in federal student financial aid 

programs, and they receive certifications and licenses from a wide variety of institutional bodies other than 

postsecondary institutions such as trade associations, private companies, state governments, and employers. As a 

result, no institutional sampling frame is available for enumerating these credentials at the national level. 

Additionally, information about certifications, licenses, and certificates is not adequately covered in federal data 

collections that survey adults directly. Without the ability to count certifications, licenses, and certificates, it is 

difficult to accurately assess their effects on social and economic outcomes and difficult to evaluate public policies 

designed to help Americans obtain and benefit from these credentials. 

After its formation in late 2009, the IWG undertook the development of a short set of survey items to measure the 

prevalence of certifications, licenses, and educational certificates. This development effort culminated in the Adult 

Training and Education Survey (ATES) Pilot Study, a national household survey of noninstitutionalized adults ages 

18 and over. The pilot was conducted from September 2010 to January 2011, and the results are presented in this 

summary. 

Item Development Process 

The first step for the IWG was to review the recent research literature on education and training credentials and 

develop a list of existing survey items on certifications, licenses, and educational certificates. Based on this 

background research and discussions with experts, certifications and licenses were found to be distinct from 

educational certificates in that they are a job or occupational requirement meant to demonstrate that the holder has 

the authority or has met certain qualifications necessary to perform specific job duties; educational certificates are 

awarded to show completion of a program of study—which may help in the performance of a specific job, but is not 

necessarily required or considered proof of qualification for work. This review led to the following working 

definitions of these credentials for the purposes of pilot testing: 

Certification: A credential awarded by a certification body based on an individual demonstrating, through 

an examination process, that he or she has acquired the designated knowledge, skills, and abilities to 

perform a specific job. The examination can be written, oral, or performance-based. Certification is a timelimited credential that is renewed through a recertification process. 

License: A credential awarded by a licensing agency based on pre-determined criteria. The criteria may

include some combination of degree attainment, certifications, certificates, assessment, apprenticeship 

programs, or work experience. Licenses are time limited and must be renewed periodically. 
3

Educational certificate: A credential awarded by a training provider or educational institution based on 

completion of all requirements for a program of study, including coursework and tests or other performance 

evaluations. 

 

 

Certificates, as an academic award, are not time limited and do not need to be renewed. Most educational certificates 

are awarded at the subbaccalaureate level, but a small number are awarded after the completion of a postsecondary 

degree. Certificates of attendance or participation are not in the definitional scope for educational certificates. 

As the definitions illustrate, although these credentials have specific characteristics and definitions, they are not 

mutually exclusive. Individuals may enroll in a certificate program to gain the knowledge needed to attain a 

certification which may be required prior to applying for a license. The interrelated nature of these credentials was 

important to consider when developing and evaluating potential survey measures. 

The next step for the IWG was to engage in a series of focus groups and cognitive interviews designed to yield a 

pilot study questionnaire to test for validity a short series of measures designed to enumerate adults with 

certifications, licenses, or certificates. 

Four major findings from the focus groups and cognitive interviews influenced key decisions during the design of 

the pilot study questionnaire. First, the focus groups showed that the terms ―certification‖ and ―license‖ are not 

always distinguishable to respondents. As noted above, certification is often the process by which someone gains the 

prerequisite qualifications needed to be awarded a license, and several focus group participants did not make a 

distinction between the process and the outcome. Consequently, the decision was made to ask about certification and 

licensure jointly. 

Second, focus groups confirmed that certification and license holders do not view their certifications or licenses as 

educational credentials. When asked where they would place a certification on their resume, respondents indicated it 

would be placed under a header such as Professional Qualifications or Professional Certifications. Consequently, 

certifications/licenses were not asked about within the questionnaire section on educational attainment. 

Third, the cognitive interviews showed that respondents also had trouble distinguishing between the terms 

―certification‖ and ―certificate.‖ These two terms sound similar and, in fact, are shown as synonyms in the 

Merriam-Webster dictionary. The cognitive interviews revealed that some respondents who have a certification will 

answer that they have a ―certificate‖ if the questionnaire asked about certificates before certifications. Therefore, the 

questionnaire was designed to specifically ask about certifications and licenses first and certificates second. 

Fourth, respondents’ interpretation of the term ―certificate‖ was varied and broad when given little or no definitional 

context. Accordingly, examples of appropriate certificate providers were included in the lead item asking whether or 

not a respondent has a certificate. 

Based on these findings the following items were used as gatekeepers into a set of questions on each type of 

credential: 

Main Certification/License Item (CN1) 

Now I’d like to ask you about professional certification and licensure. Do you have a 

professional certification or a state or industry license? 

Probe: A professional certification or license shows you are qualified to perform a 

specific job and includes things like Licensed Realtor, Certified Medical Assistant, 

Certified Construction Manager, a Project Management Professional or PMP 

certification, or an IT certification. 

Main Certificate Item (CT1) 

Some people decide to enroll at a college, university, community college, or trade school 

to earn a certificate rather than a degree. Have you ever earned this type of certificate? 
4

Probe: An educational certificate is typically earned by completing a program of study 

offered by a college or university, a community college, or a trade school, but it does not 

lead to an associate’s, bachelor’s, or graduate degree. Sometimes these are also called 

vocational diplomas, for example, a cosmetology or mechanics diploma, which differs 

from a high school diploma. 

The focus groups and cognitive interviews helped to identify key characteristics associated with certifications, 

licenses, and certificates. These key characteristics formed the operational constructs for the remainder of the 

questionnaire items. They included the level of effort to obtain the credential, examination and performance 

assessment requirements, type of provider or awarding body, industry recognition, occupational specificity, 

requirement for employment, suspension or revocation potential, continuing education and recertification 

requirements, and perceived labor-market value. Survey items measuring these key characteristics were developed 

for the pilot study questionnaire and asked of respondents for their most recent certification/license and certificate. 

The pilot study questionnaire also collected information about respondents’ demographic characteristics, such as 

age, gender, employment status, race/ethnicity, and income. These questions were derived from the NCES National 

Household Education Survey (NHES) and the Current Population Survey (CPS). 

ATES Pilot Study Design 

The ATES Pilot Study was administered from September 2010 through January 2011. It is important to note that the 

pilot study was not conducted for the purpose of collecting data to make population estimates (although survey 

weights were developed to examine the reasonableness of population estimates generated from the survey

measures). The primary purpose of the pilot study was to examine the empirical properties of potential survey 

measures. 

The ATES Pilot Study was conducted by mail and telephone using an address-based sample covering the 50 states 

and the District of Columbia. The study was sponsored by NCES and designed and implemented by the American 

Institutes for Research (AIR) and Westat. The ATES Pilot Study data file contains a nationally-representative 

sample of 3,730 adults, of whom 1,230 reported certifications/licenses and 530 reported certificates (220 reported 

both).2 In addition to the nationally-representative sample, the ATES Pilot Study data file contains a convenience, or 

―seeded‖, sample of 340 adults who were voluntarily reported by three community colleges to have been awarded a

certificate and by five credentialing bodies to have received a certification or license.3 The seeded sample contains 

state license holders in public health, electrical, pharmaceutical, plumbing, real estate, and energy-related fields; 

certifications/licenses in lab technology, radiation, and other health-related fields; and community college 

certificates covering a variety of programs. The seeded sample is not a representative sample and was used primarily 

to assess underreporting of credentials. Although care was taken to find current and reliable administrative records 

for the seeded sample by requesting records directly from awarding bodies, these records have unknown 

measurement error, and therefore, at least some portion of the reporting error discussed in the findings of this paper 

may be attributed to errors in the seeded sample frame. 

Finally, in a subsample of approximately 1,000 households with two or more adults, the responding adult was also 

asked to report on certifications, licenses, and certificates held by another adult in the household. This was done in 

order to examine measurement error for responses reported by proxy because federal surveys that may be candidates 

for the inclusion of these new items may use proxy respondents when they are unable to interview the sampled adult 

in the household. 

The first-stage mail survey response rate was 52 percent. The second-stage telephone interview response rate was 44 

percent (unweighted) and 42 percent (weighted). It should be noted that limited refusal conversion was undertaken 

at the telephone stage in order to maintain timeliness of the data collection. Timeliness and sample yield were a 

higher priority for the pilot study than were response rates. 

 

2

 The pilot study data are not available for secondary analysis due to their preliminary nature, their methodological rather than 

reporting purpose, and disclosure concerns. 

3

 It is not possible to identify the specific credentialing bodies for disclosure reasons. 
5

The data file used for analysis underwent limited data editing. Imputation was performed on variables needed for 

weighting and on the overall certification/license and certificate items. Weights and variables to calculate standard 

errors were developed and used to examine the reasonableness of population estimates generated from the survey 

items. 

ATES Pilot Study Results: Weighted Estimates and Comparisons to Other Nationally-Representative Data 

Responses from the ATES sample were weighted to represent all U.S. adults ages 18 and older. As shown in table 1, 

an estimated 38 percent of adults, or approximately 82 million adults, reported having at least one of the credentials 

of interest. A certification or license was more prevalent than a certificate; 30 percent of adults reported having a 

certification or license, compared to 14 percent who reported having a certificate.4 

To determine whether the ATES Pilot Study estimates were consistent with past research, the ATES estimates were 

compared to estimates derived from the Princeton Data Improvement Initiative (PDII) for certifications/licenses and 

to the Survey of Income and Program Participation (SIPP) for certificates. 

The PDII survey was conducted in 2008 by Westat on behalf of Princeton University and covered the topics of 

worker perceptions of the ―offshoreability‖ of their jobs, occupational licensing, and lifetime work experience.5

Participants were adults in the labor force, currently employed or unemployed and looking for work. After 

restricting the ATES and PDII samples to adults who were currently employed, the percentage of adults reporting a 

certification/license was 38 percent based on the ATES Pilot Study data and 34 percent based on the PDII data (table 

2). Although the ATES estimate was measurably different from the PDII estimate, the difference is small enough to 

be considered reasonable, given dissimilarities in the data sets in terms of timing, item wording, and item order. 

When broken out by educational attainment level, the rate of certifications/licenses was not measurably different for 

adults whose highest level of education was below a bachelor’s degree, but was measurably higher in ATES than in 

the PDII for those at the bachelor’s degree level and above (by around 7 to 8 percentage points). 

To examine the ATES estimates on certificates, the ATES data were reanalyzed and compared to 2008 data from the 

U.S. Census Bureau’s SIPP, using adults ages 23–64 who reported an income or reported working for earnings in 

the past year.6 The overall rate of certificates based on ATES was 15 percent, compared to 18 percent for SIPP. 

Again, the difference was statistically significant, but reasonable given the dissimilarities between the two surveys 

(e.g. timing, mode of interview). Significant differences also existed between the two surveys for adults with 

educational attainment levels at or below high school level, and at the graduate or professional degree level 

 

4Although the primary interest of the ATES Pilot Study was on subbaccalaureate certificates, the survey instrument did not 

distinguish the level of the certificate, so respondents may have reported postbaccalaureate certificates. 

5

 The PDII data are available at http://irs.princeton.edu/Conferences/PDII/PDII_RDD_Survey.dta. 

6

 The SIPP 2008 data are available at http://www.bls.census.gov/sipp_ftp.html. 

Characteristic Number Percent

Reported credential

 Either credential reported 82,348,000 37.5

 Professional certification(s) or license(s) 65,324,000 29.8

 Educational certificate(s) 29,901,000 13.6

 No credential reported 137,076,000 62.5

SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Training and Education 

Survey (ATES), 2010-2011 Pilot Test. 

NOTE: Figures represent weighted estimates. Respondents were able to select multiple credentials; therefore,

details may not sum to totals. 

Table 1. Number and percentage of US adults who report holding certification(s)/license(s) or educational 

certificate(s): 2010 
6

(differences in the range of 2 to 6 percentage points). There were no measurable differences in the estimates for 

adults with some college, an associate’s degree, or a bachelor’s degree. 

ATES Pilot Study Results: Unweighted Analysis of Underreporting of Credentials in the ATES Seeded 

Sample 

Reverse record checks, like the one employed in the ATES Pilot Study, are designed to measure underreporting of 

known characteristics. The seeded sample in ATES can be used to identify characteristics of respondents and/or 

credentials that could help explain underreporting (false negatives). This analysis examined differential 

underreporting rates by type of credential, age, and employment industry. Some portion of underreports could be 

due to errors in the administrative records or data collection, such as reaching the wrong individual. 

One possible explanation for underreporting is that respondents may misidentify a certification/license as a 

certificate (or vice versa). Differential underreporting rates can be used to evaluate this by determining whether a 

certificate was also reported by the respondent (and whether that certificate was from someplace other than a college 

or other educational program, as this would be an indication that the credential was more likely a 

certification/license than a certificate). Similarly, the evaluation of seeded certificates looked for differential 

underreporting rates by whether a certification/license was also reported by the respondent and whether those 

certifications/licenses were part of a degree or other educational program. Finally, the evaluation of seeded 

certifications/licenses assessed differential underreporting rates by credential type (certification or license). These 

additional analyses (reported in the full NCES technical report) can provide evidence of credential misidentification 

at an aggregate level. 

A record was classified (at the person level) as a certification/license underreport if a respondent who was sampled 

from the seeded certification/license frame answered ―no,‖ ―don’t know,‖ or ―refused‖ to the main 

Table 2. 

Respondent characteristic PDII1 ATES SIPP:082 ATES

Total 34.4 37.8 * 18.1 15.1 *

Highest level of education

Less than high school diploma 18.5 27.2 7.3 11.7 *

High school diploma 25.7 27.8 22.2 15.8 *

Some college 33.9 38.1 24.8 23.2

Associate's degree † † 34.3 24.3

Bachelor's degree 35.1 42.7 * 10.3 9.4

Graduate or professional degree 50.3 56.8 * 6.4 8.8 *

† Not available. 

Table 2. Comparison of estimates of percentage of US adults who report holding 

certification(s)/license(s) and educational certificates, by educational attainment

SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Training and 

Education Survey (ATES), 2010-2011 Pilot Test. U.S. Department of Commerce, Census Bureau, Survey of 

Income and Program Participation (SIPP), 2008; Princeton Data Improvement Initiative (Kleiner and Kruger 

2009). 

NOTE: Figures represent weighted estimates. Data for certifications/licenses limited to those currently 

employed or looking for work. Data for certificates limited to those who reported an income or earnings in the 

previous year. 

1

 Princeton Data Improvement Initiative, 2009 

2 Survey of Income and Program Participation, 2008 

* observed difference is statistically significant at .05 level

Certifications/licenses Certificates
7

certification/license survey question (item CN1: Do you have a professional certification or a state or industry 

license?). Similarly, a record was classified (at the person level) as a certificate underreport if a respondent who was 

sampled from the certificate frame answered ―no,‖ ―don’t know,‖ or ―refused‖ to the main certificate survey 

question (item CT1: Some people decide to enroll at a college, university, community college, or trade school to 

earn a certificate rather than a degree. Have you ever earned this type of certificate?). Proxy responses were not 

included in this analysis to avoid confounding the possibility of limited proxy knowledge with actual underreports. 

The overall underreporting rate for seeded certifications/licenses was 15 percent, and the overall underreporting rate 

for seeded certificates was 50 percent (table 3). Few of the differences in underreporting rates by subgroup were 

statistically significant. However, the results for age and employment status suggest that saliency may be a factor. 

Those in the oldest age group (65 years and over) and those not employed were most likely to underreport these 

credentials. 

Respondents selected from one of the community colleges in the certificate sample frame were removed after 

finding that they had an unusually high certificate underreporting rate (85 percent); further investigation revealed 

that there was a problem with the sample frame itself. The school had encoded students who achieved a transfer 

sequence of 18 to 60 units as ―certificates.‖ These students represented about 78 percent of the records in the 

sampling frame and 86 percent of the survey respondents for this school. The sampled respondents with problematic 

records from this school had an underreporting rate of 93 percent, while the sampled respondents with valid records 

had an underreporting rate of 57 percent. However, while the rate for the latter group is more in line with the other 

certificate samples, the sample size was too small to be included in the analyses (n <10). 

Percent of respondents 

underreporting 

certifications/licenses 

Percent of respondents 

underreporting certificates

Total 15 50

Type of certification

Certification 17.1 † 

License 13.7 † 

Age

18 to 34 years 19 48.4

35 to 64 years 11.6 48.4

65 years and over 29.2 80

Employment industry 

Private company 15.4 46.2

Government, non-profit, or self-employed 6.9 30.0 to 50.0 

Not employed 45.8 62.5

† Not available 

NOTE: Figures represent unweighted estimates. Observations with missing data for a given variable are excluded. 

Includes self-reported data only; data completed by proxy respondents were excluded from the analysis. Detail may 

not sum totals because of rounding. 

Underreporting Rates

SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Training and Education 

Survey (ATES), 2010-2011 Pilot Test.

Table 3. Percent of seeded sample respondents underreporting of credentials in the ATES seeded sample: 

2010
8

The identified inaccuracies in the certificate seeded sampling frame may reflect similar (though less extensive) 

issues in data supplied by the other community colleges. Therefore, the high underreporting rate found for 

certificates may be partially due to characteristics of the frame. This possibility will be investigated by NCES in 

upcoming work on certificate item development and testing. 

ATES Pilot Study Results: Unweighted Analysis of Key Characteristics of Certifications/Licenses 

Based on the focus groups and cognitive interviews, several factors were expected to covary with holding a 

certification/license or certificate. First, certification/license holders were expected to report that their credentials 

were in some way job relevant; for example, respondents might report that their certifications/licenses were required 

for their current job. Respondents were also expected to report taking courses or training to earn the 

certification/license and that they had been required to demonstrate their knowledge or skills in order to be 

certified/licensed. Finally, respondents were expected to report being certified or licensed by a state, industry, 

company, or professional association and that their certification/license could be revoked or suspended. Factors 

expected to be related to certificates reflected the fact that this type of credential is obtained through an educational 

program. Therefore, certificate holders were expected to report that their credentials were awarded by an educational 

institution (e.g., a community college). 

Respondents were asked some items, such as the name and year the credential was obtained, for all 

certifications/licenses or certificates held (up to five of each), whereas the majority of credential-specific items were 

only asked for the most recent certificate earned and the most recent work-related certification/license held.7 

Because information was collected on potentially multiple credentials per respondent, items were analyzed at the 

credential level rather than the respondent level and are reported as the percentage of certifications/licenses or 

certificates held by respondents. In addition, these analyses are not weighted because, unlike comparisons of 

estimates, they are methodological analyses of sample characteristics and are not intended to show population 

characteristics. 

In fact, the national estimates from the ATES Pilot Study did find relationships between holding a 

certification/license or certificate and other factors as expected, based on ATES developmental work and the 

definitions of the credentials. As shown in the NCES technical report, most reported certifications/licenses were 

pursued for mainly work-related reasons (90 percent), and most work-related certifications/licenses required the 

respondent to pass a test or exam (91 percent) and take continuing education units to maintain them (64 percent). 

The majority of the most recent certifications/licenses could be revoked or suspended (76 percent). Similarly, the 

most of the most recent certificates were conferred by an educational institution (57 to 73 percent) and required 21 

or more credit hours to complete (57 percent for those reporting in credit hours). Instances where these expected 

relationships were not found might indicate misreporting by respondents. 

ATES Pilot Study Results: Unweighted Analysis of Overreporting of Certifications/Licenses in the ATES 

National Sample 

Although the seeded sample is not representative of all credential holders, it does allow us to examine the 

measurement properties of the survey items in two important ways. First, we can compare information reported by 

the sampled adult (self-reported survey data) to the frame information provided from the administrative records of 

the credentialing body to examine reporting accuracy. This analysis is referred to as ―matching.‖ Matched cases are 

considered true credentials because they are a report of a credential that is confirmed both in the administrative 

records data and by the survey data. Second, the characteristics of true credentials can then provide a rule-of-thumb 

for identifying reported credentials in the nationally-representative sample that do not reflect the characteristics of a 

true credential. These cases are classified as credential overreports. 

Because of the high underreporting rate and the small number of respondents with known certificates in the seeded 

sample, the analysis of overreporting was limited to matched certifications/licenses. 

 

7

 If respondents did not report any work-related certifications/licenses, they were asked follow-up questions about the most recent 

credential they obtained for personal interest. 
9

Characteristics of these matched credentials in the seeded sample can serve as checks on the accuracy of reporting in 

the national sample. The pilot study identified three key characteristics most often associated with having a known 

certification/license: 

Was the credential earned for mainly work related reasons? (Item CN6) 

Does the credential require a demonstration of skills while on the job or passing a test or exam? (Item 

CN10a/b) 

Can the credential be used to get a job with any employer in that field? (Item CN15b) 

Similar to the results reported for the national sample above, a high percentage of certifications and licenses that 

matched the seeded sample frame were obtained for mainly work-related reasons—95 percent of certifications and 

98 percent of licenses (details in the NCES technical report); 100 percent of the most recent matching certifications 

and 80 percent of the most recent licenses required respondents to pass a test or exam; 71 percent of certifications 

and 62 percent of licenses required a demonstration of skills while on the job; and 96 percent of the certifications 

and 100 percent of the licenses could be used to get a job with any employer in the field. Nearly all certifications and 

licenses had at least one of these requirements. 

This analysis considers whether these key characteristics identified using the seeded sample can be used to spot 

possible instances of overreporting of credentials in the national sample (false positives), by determining the extent 

to which respondents report a certification/license that does not have the characteristics of typical known credentials. 

Using this approach, uncommon responses (e.g., reporting a certification/license, but not reporting that the 

certification/license was work-related) are used to flag possible instances of overreporting. 

Table 4 shows the percentage of matching seeded sample certifications/licenses that would be miscategorized as 

overreports in the seeded sample—and possibly in the national sample—based on responses to various combinations 

of the three related factors measured by items CN6 (credential earned for work related reasons), CN10a/b (test or 

demonstration of skills required), and CN15b (credential can be used to get a job with any employer in the field).

The analysis defined a ―true‖ certification/license as one with all three of these characteristics (based on the analysis 

of the seeded sample). Applying this definition to the national sample, a respondent who reports having a 

certification/license and all three characteristics would be considered a true positive. If the credential does not have 

all three key characteristics, it could possibly be a false positive, or overreport. As shown in the second column of 

Table 4, different combinations of these three characteristics result in different percentages of overreports. If the 

certification/license has only two of the key characteristics, the overreporting rate ranges from 1.8% to 5.6% 

depending on the combination of items. If the certification/license has only one of the key characteristics, the 

overreporting rate ranges from 6.9% to 9.1%. Finally, if the respondent reports that they have a certification or 

license in the main certification item (CN1), this analysis projects a 13.3% overreporting rate in the national sample. 
10

If surveys define ―true‖ certifications/licenses using the main certification item and all three key characteristics, 

therefore, overreporting is minimized. But, as the first column in Table 4 shows, using such a restrictive definition 

underestimates the percentage of matched true certifications/licenses in the seeded sample. Applying this most 

restrictive definition using the main item and all three key characteristics to the seeded sample resulted in 6.8% of 

known, matched certifications/licenses being misclassified as overreports. Using fewer key characteristics results in 

fewer miscategorized overreports. 

Figure 1 illustrates the tradeoffs between having more or less restrictive definitions on possible over- and 

underreporting. Using all three key characteristics reduces overreporting to zero (by definition) but it results in 

possible underreporting among known credential holders because they failed to answer one of the three items in the 

expected way. Using none of the three key characteristics results in relatively high overreporting of credentials, but 

minimizes underreporting by not eliminating known credential holders because they failed to answer one of the 

three items in the expected way. 

Table 4. Overreporting of certifications/licenses in the ATES national sample: 2010 

Key Characteristics

 Percent of 

certifications/licenses 

miscategorized as 

overreports in seeded 

sample 

Percent of 

certifications/licenses 

overreported in 

national sample 

All Three Items 

CN6: CN was work related AND

CN10a/b: CN required demonstration of skills while on the job 

or passing a test or exam AND

CN15b: CN could be used to get a job with any employer in 

that field. 6.8 0.0 (by definition) 

Two Items

CN6 AND CN10a/b 6.8 1.8

CN6 AND CN15b ‡ 2.0

CN10a/b AND CN15b 5.1 5.7

Single Item

CN6 ‡ 6.9

CN10a/b 5.1 8.8

CN15b 0.0 9.1

Only CN1: Report Certification/License N/A 13.3

‡Data suppressed because reporting standards were not met for disclosure. 

NOTE: Figures represent unweighted estimates. ―CN‖ refers to certifications and licenses. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Training and Education 

Survey (ATES), 2010-2011 Pilot Test.
11

The results (discussed in more depth in the technical report) imply that all combinations perform fairly well at 

identifying true positives (while minimizing overreporting) and that any one of the key characteristics could be used 

independently to eliminate overreporting. Federal surveys that elect to use these items will need to weigh the 

tradeoffs between overreporting and underreporting taking into consideration the analytic needs of their customers 

and the amount of survey space they can devote to this topic. An in-depth set of items on certifications/licenses 

would ideally include the main item and all three key characteristics, enabling analysts to apply their own definition 

to the construct. 

IWG Conclusions and Recommendations 

The ATES Pilot Study findings have implications for determining a parsimonious set of items that could be included 

in other federal household surveys of adults—either in the near future for those items requiring no further 

development or after additional testing has been conducted to address the measurement issues identified in this 

report. It is important to note that the items developed for the pilot test were limited in their scope. They were 

designed specifically to enumerate adults with alternative credentials, and their utility as survey measures outside 

the scope of this Pilot Study has not been tested. Therefore, although results of the pilot are used in this section to 

recommend items for researchers with various goals; the properties of the items may differ in other survey 

administration contexts. 

With that caveat, this section summarizes the recommendations of the IWG for items to use in other household 

surveys trying to achieve the following research goals: counting adults with a certification or license; counting the 

certifications or licenses held, and distinguishing between certifications and licenses. 

To count the number of adults in the United States who have a certification or license, the IWG recommends the 

main certification/license item CN1. This recommended item is an aggregate, broad measurement item that could be 

administered alone or in conjunction with other credential-specific items to provide counts of adults with workrelevant credentials. To reduce measurement error, survey sponsors may consider adding to this item one to three of 

the items that measure key defining characteristics of certifications/licenses. These items include whether or not the 

certification/license is work-related (CN6); if the individual had to pass a test or exam (CN10b), and; if the 

certification/license is transferable (CN15b). 

Figure 1. Trade-offs using key characteristics

SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Training and Education 

Survey (ATES), 2010-2011 Pilot Test.

7 7

5 5

0 0 0

2

6

9 9

13

0

5

10

15

CN6 & 

CN10a/b & 

CN15b

CN6 & 

CN10a/b

CN10a/b & 

CN15b

CN10a/b CN15b Only CN1

Percent

Underreporting Overreporting
12

The main certificate item was found to have a high underreporting rate among the seeded sample respondents in the 

ATES Pilot Study and is not recommended for use at this time. The main certificate item and the series of items 

tested in the certificate questionnaire section require additional testing before they should be used.

The results of the pilot have several implications for measurement. First, the items in the ATES were tested in a 

short survey focused on certifications, licenses, and certificates. It is possible that the items may perform differently 

in different survey contexts, particularly in surveys where other topics are covered in more depth. Second, cognitive 

testing for the ATES showed an order effect when asking about certifications and certificates. If certification holders 

were asked about certificates before certifications, they would report a certification as a certificate. Also, cognitive 

testing confirmed that respondents do not think of certifications and licenses as educational attainment, leading the 

ATES to ask about them in separate sections. Researchers should carefully consider the placement of the items 

within a survey. 

The results of the pilot also demonstrated that the salience of the credential (as characterized by recency and work 

relevance) affects the accuracy of reporting. If respondents are reporting their most salient credentials, results may 

be biased toward the credentials held and used by those currently employed. It is also important to note that 

respondents who were not employed had greater reporting error compared to employed respondents. 

The IWG member agencies are planning several data collection efforts to provide counts of adults with certifications 

or licenses. The IWG is also planning to conduct additional developmental work to better measure educational 

certificates. These plans are summarized below. 

Upcoming work on certification/license measurement 

IWG members have looked within their own agencies for opportunities to incorporate the tested items into ongoing 

or planned studies of out-of-school youth and adults. In addition, they have sought ways to test these items further to 

explore their performance in different survey contexts. These plans are described below. 

NCES. NCES is incorporating a set of questions on certifications/licenses into its upcoming Baccalaureate and 

Beyond (B&B) longitudinal study of 2008 Bachelor’s degree recipients. Currently-active secondary longitudinal 

studies (the High School Longitudinal Study of 2009 freshmen and the Education Longitudinal Study of 2002 

sophomores) will incorporate questions on certification for out-of-school youth and young adults. There are shortterm plans to develop an adult education and training household study that will feature items on certifications and 

licenses and long-term plans to collect background data that would permit linkages between credentials and literacy. 

Census. The Census Bureau has recognized an opportunity for further testing of certification items in the field test 

of the redesigned Survey of Income and Program Participation (SIPP). The inclusion of these items in a different 

survey context will test the robustness of the item across platforms. The Bureau is also planning to collect weighted 

national estimates of the number of people with certifications and will field a topical module with the main 

certification item as the gateway question for the final existing SIPP panel (2008 panel). 

Upcoming work on certificate measurement 

Due to the inconclusive results of the Pilot Study analysis of data on certificates, the IWG has recommended 

additional developmental work on the survey items related to certificate attainment. In addition, NCES plans to 

respond to increased interest in subbaccalaureate educational certificates by improving the ability of current data 

collections to measure these credentials. The four projects described below will improve and expand the ability of 

NCES studies to answer important policy questions about the prevalence and economic impact of certificate 

education. 

Developing a Valid Survey Item to Enumerate Certificates in Federal Surveys 

In April of 2012, the IWG will begin development and testing of a new survey item designed to measure the 

prevalence of educational certificates in the US adult population. This effort will be designed to improve on the 

measurement of certificates that was attempted in ATES, including a sharper focus on these measures, and use of a 

larger and cleaner sample of known certificate holders. The study will begin with focus groups to determine how 

certificate holders talk and think about their credential. Next, questionnaire items will be developed and tested in 
13

cognitive interviews. A new pilot study will feature a larger sample of known certificate holders that is more 

representative both geographically and by field of study. Paired with a new national sample, this new seeded sample 

will provide a good test of the validity of the new certificate items. 

Improving Institutional Data Collection on Certificates 

NCES currently collects annual counts of the number of certificates awarded through its Integrated Postsecondary 

Education Data System (IPEDS). IPEDS gathers information from every college, university, and technical and 

vocational institution that participates in federal student aid programs—over 7,000 postsecondary institutions in all. 

The IPEDS data collection is guided by the National Postsecondary Education Cooperative (NPEC), which oversees 

IPEDS research and development. 

NPEC has recently created a working group to look at how institutions report certificate completions to IPEDS in 

order to improve the quality, comparability, and usefulness of these data. To support this work, a study of certificate 

reporting practices is being carried out. The working group began meeting in October 2011; a final report is 

scheduled for September 2012. 

Oversampling Certificate Holders in BPS:12 

Starting in 1996, NCES has conducted a periodic longitudinal study of students who have recently matriculated into 

postsecondary education, the Beginning Postsecondary Students Study (BPS). The BPS is conducted about every 

eight years, with the next study scheduled to start in 2012. BPS is the largest federal survey that regularly examines 

the economic outcomes of students in subbaccalaureate programs. However, previous rounds of BPS have lacked 

the capacity to conduct in-depth analyses of the economic returns to educational certificates in specific fields of 

study. NCES plans to drastically increase the sample size for beginning students who enter postsecondary education 

with the goal of attaining an educational certificate. This oversample will allow finer-grained analysis of the reasons 

students select educational programs leading to a certificate, the characteristics of students in such programs, their 

persistence and attainment, and their occupational and economic outcomes. 

Fielding a Household Survey on Adult Education and Training 

NCES is planning a household study to document adults’ acquisition of education and training that is oriented 

towards work. This new study will collect in-depth information on the education required to attain various levels of 

qualifications, the relationship between industry-recognized certifications and employment, the prevalence of 

industry-recognized certifications among adults with formal education credentials, the career pathways of adults 

seeking certificates and certifications, and the supports and barriers faced by adults seeking such credentials. 

NCES previously had a survey of adult education, the National Household Education Study (NHES) Adult 

Education Survey, but it was not focused on industry-recognized credentials. Moreover, that survey was halted in 

2007 because of low response rates. NCES has redesigned the NHES to collect data on early childhood education. 

The redesigned NHES will use a mail administration process rather than random digit dialing as had been done in 

the past—a change that yielded improved response rates in field testing. To see if response rates would be similarly 

improved with the redesigned procedures when asking adults about themselves, rather than their children, NCES 

will conduct a response rate test for a survey on adult education and training in 2012. The response rate test will 

consist of a nationally-representative sample of households and adults to evaluate response rates for adults with key 

characteristics related to age, educational attainment, and employment status. NCES is also planning an in-depth 

nonresponse bias study with personal interviews of a subset of nonrespondents as part of this project. If response 

rates are sufficiently high or if nonresponse bias is deemed to be sufficiently negligible, NCES could field a full 

scale adult education and training survey as soon as funding becomes available. 
14

Appendix A. Members of the Interagency Working Group (IWG) 

Beginning in December 2009, the federal interagency working group (IWG) convened monthly to develop new 

survey measures of certifications, licenses, and certificates. The IWG represents a broad range of federal agencies 

committed to improving the measurement of work-related credentials. Current members of the IWG (as of March 

2012), as well as past IWG members from each participating agency include: 

Bureau of Labor Statistics: Harley Frazis 

Tom Nardone 

Census Bureau: Stephanie Ewert 

Bob Kominski 

Council of Economic Advisors: Elizabeth Ananat (past) 

Sarena Goodman (past) 

Jessie Rothstein (past) 

Chuck Pierret (past) 

Department of Education, Office of the Undersecretary: Jon O’Bergh 

National Center for Education Statistics: Sharon Boivin 

Sarah Crissey (originally at Census Bureau) 

Lisa Hudson 

Kashka Kubzdela 

Isaiah O’Rear 

Matthew Soldner 

Tom Weko 

National Science Foundation: John Finamore 

Daniel Foley 

Office of Management and Budget: Shelly Martinez