Docket: ED-2014-ICCD-0071

2015 National Household Education Survey (NHES 2015) Full Scale Data Collection

Comment On: ED-2014-ICCD-0071-0005

Document: ED-2014-ICCD-0071-0008

Comment on FR Doc # 2014-15935

COMMENT

I'm happy to see that there is a second chance for those of us who hoped to comment on the first 

docket due in July but let it fall through on our summer to-do lists! I hope I'm not the only academic 

sitting on the sidelines eagerly waiting for data from this survey, because this is a truly important project 

that will fill a massive gap in our understanding of the extent and effectiveness of this country's 

educational and credentialing system If successful, it will be a gold mine for sociologists and other social 

scientists studying a range of topics, from membership associations to educational achievement to 

inequality.

However, I worry that the proposed CWS might be a missed opportunity to learn more about the 

characteristics of certifications themselves. The draft instrument is a very good start, but we need all the 

data we can get. NHES is not a great survey for learning about the demographic and labor market 

correlates of certification (and licensure) because it lacks the extensive and well-developed core items 

found in such surveys as the SIPP and ELS. Perhaps, then, it makes sense to emphasize NHES as a means 

of getting a broad picture of the certification landscape and characteristics of certifications across 

occupations. It is difficult to prioritize in such a space-limited environment as the NHES, but I would like 

to suggest a few points that would greatly enhance our understanding of this phenomenon that the 

NHES would be a uniquely suitable place to inquire about:

 Why did you choose this credential over others in this occupation?

 How many hours did you spend preparing for this credential?

 How much does this certification cost, in dollars?

 Is it a generalist certification covering most aspects of your occupation, or more specialized?

 How would you rate the difficulty of earning this credential?

 How closely related was the curriculum of the certification program related to your day-to-day 

work?

 Is this certification issued by a non-profit association?

 If so, are you a member of that association?

Some further inspiration for items about certification characteristics might be found in the 2002 follow 

up to the 1992 Baccalaureate and Beyond, which seems to be the only major federal survey that asked 

about professional and trade association certifications before the current working group was 

established.

I hope that GEMEnA and/or DoEd will consider expanding on some of these themes in future revisions, 

but even in its current form this survey will be tremendously helpful for researchers and policymakers 

alike. I eagerly await the results!

Respectively submitted,

Kyle Albert

Ph.D. Candidate

Department of Sociology, Cornell University

kwa23@cornell.edu
NCES RESPONSE

First, we are happy to hear that the proposed data collection will be of use to researchers. We look 

forward to seeing what researchers will learn from the survey and how this information might be used 

by policymakers, credentialing bodies, and others interested in education and skill development.

Second, we agree with the commenter that the NHES, while useful in many ways, is limited in the 

amount of corollary information it can collect on adult’s labor market background, educational 

background, and other characteristics. For this reason, GEMEnA is working with both the Census Bureau 

and the Bureau of Labor Statistics (BLS) to add core certification and licensure questions to their key 

surveys. At this time, these core questions have been added to the Census Bureau’s Survey of Income 

and Program Participation, and are in the process of being added to BLS’s Current Population Survey. 

Those surveys will provide more detailed background on the characteristics of credential-holders than 

NHES can, while NHES will provide more detail on the credentials obtained (getting extensive details on 

all of these topics would be prohibitively expensive and time-consuming in a sample survey, but might 

eventually be possible in a unit-record data collection system, which some states are starting to build).

Thank you for suggesting a number of interesting additional questions for inclusion in the NHES data 

collection. As explained below, we had considered some of these questions in our development process, 

but decided to drop them in an effort to minimize respondent burden and maximize data quality. The 

proposed questions, our plans for future testing, and reasons for not including them in the current 

instrument are listed below.

Why did you choose this credential over others in this occupation?

This is a very interesting question, but “why” questions are notoriously difficult in survey data 

collections. People are often unclear of their motivations, and social desirability effects tend to be strong 

when people are asked to explain their motivations (i.e., people tend to give the answer they think is 

most socially appropriate). Because of this, we do not believe it would be prudent to add this question 

without first cognitively testing it. As 2015 will be the first administration of the Adult Training and 

Education Survey, we expect that some revisions might be needed after 2015, and we thus hope to 

conduct a set of cognitive interviews in 2016. We will test this question during those interviews.

How many hours did you spend preparing for this credential? How much does this certification cost, in 

dollars?

We agree that these are useful questions, and we originally tried to ask them. Unfortunately, our 

cognitive testing showed that these were difficult questions for people to answer, as what to count in 

the time and cost calculations was often unclear. For example, if an educational certificate or degree 

provided the training necessary to earn the credential, should that time and cost be counted? What 

about the cost of equipment (e.g., cosmetology supplies, mechanics tools)? Or the time spent in an 

internship? Recall was also an issue, as many people earned their credential many years ago. Even 

asking about renewal costs was difficult, as renewals differ in the time they cover (one year, two years, 

etc.) and in who pays for the renewal (e.g., the adult or the employer). So we dropped these questions 

from our initial instrument. But we do plan to keep thinking about how we might best get at these 

concepts, as the “investment costs” to credentialing are an important issue.

Is it a generalist certification covering most aspects of your occupation, or more specialized?

This is another interesting question. We did see both types of certifications during our cognitive 

interviews, with adults often first getting a generalist certification and then adding specializations. But it 

is not clear how this question should be asked in a survey. Based on our work to-date, we are not sure 
that the terms “generalist” and “specialized” will work for all respondents. So, again, we would need to 

do cognitive testing before adding this to the instrument. We would also need to weigh the value of this 

information against other information in the survey, given burden issues. Meanwhile, we encourage 

researchers to investigate whether the write-in responses on the question asking for the name of the 

certification can reliably provide this information. This might be one way to get this information.

How closely related was the curriculum of the certification program to your day-to-day work?

Although this is an interesting question, it is subjective in nature and probably not as useful for research 

and policy purposes as the other questions currently on the survey (and the other questions raised for 

consideration here). Also, some adults will be many years out of their certification program and are 

likely to have recall issues, as well as issues related to the relevancy of programs taken 10, 15, or 20 

years ago. We will mention this as a question to consider by our NCES colleagues who do longitudinal 

studies of postsecondary students, where the question would be more appropriate.

Is this certification issued by a non-profit association? If so, are you a member of that association?

We originally tried to capture information on the type of organization that issued the certification, but 

we found that adults had trouble answering this question; they often were not clear on who the issuing 

agency was. We did not drop this issue entirely, because it is important to distinguish licenses (issued by 

the government) from certifications (issued by non-governmental organizations). So we do ask if the 

issuing agency was the government. Fortunately, adults do seem better at identifying the government as 

an issuing agency than at identifying other issuers—although even here, there will inevitably be some 

measurement error. We hope that eventually all the states will have unit-record data collection systems 

that include certification and licensure records, which would be the most reliable source for this 

information (admittedly, that solution is likely in a more distant future).

With regards to the 2002 Baccalaureate and Beyond (B&B) survey instrument, it included a number of 

questions focusing on the relationship between education and licensure/certification, but these 

questions are more relevant for B&B (which samples recent graduates) than for NHES (which samples all 

adults). The more recent B&B included fewer questions on certification and licensure, but we are 

working with our NCES colleagues to re-visit this section of the survey in order to increase the 

information collected on the certification and licensure topics.

Thank you very much for your interest in our surveys.

With regards,

Sharon A. Boivin, Ph.D.

Chair, Interagency Committee on Expanded Measures of Enrollment and Attainment (GEMEnA)

National Center for Education Statistics