1

Appendix 4

Graduate Employment and Earnings Survey

Cognitive Interviews, Round 1 - Results and Recommendations

1. Recruitment and Participation

Cognitive testing was conducted from January 26 to February 11, 2015, in three of RTI’s U.S. office 

locations – Research Triangle Park, NC; Washington, DC; and Chicago. To recruit graduates, RTI posted 

web advertisements on Craigslist and used direct outreach though advertisement postings at area 

businesses. A total of 42 individuals responded to the posted advertisements; of these, 19 were ineligible, 

15 were eligible and participated, and 8 were eligible but did not participate due to cancellation, difficulty 

scheduling a time to participate, or not coming to their scheduled appointment. Participants were recent 

graduates (up to 4 years following completion of the credential) from institutions in the vicinity of RTI’s 

offices, including online program graduates residing in the same geographic areas. Of those who 

participated, nine were female and six male. Table 1 provides a distribution of the final number of 

interview participants, by control and level of institution.

Table 1: Number of respondents, by institution control and level

Participants were given the short paper survey and asked to complete it on their own. Upon completion, 

they were asked to think aloud retrospectively in order to identify potential problems related to question 

comprehension, recall, information editing due to sensitivity or social desirability and response mapping. 

On average, the interviews took 32 minutes to complete.

2. Results and Recommendations

Introduction

The purpose of the Graduate Employment and Earnings Survey is to collect information about 

the average earnings of graduates from programs whose students receive federal financial aid 

and are subject to regulations on gainful employment (34 CFR Parts 600 and 668).

The information on this survey will be used to report average earnings for program graduates to 

the U.S. Department of Education. Your individual earnings will never be disclosed or used in 

any other manner unless required by law. 

Institution control Institution level Credential type(s)

Number of 

respondents

Private for-profit Less-than-2-year Vocational certificates 3

2-year Vocational certificates and 

associates 2

4-year All 2

Private not-forprofit

Less-than-2-year and 2-

year

Vocational certificates and 

associates 1

Public Less-than-2-year and 2-

year

Vocational certificates and 

associates 7
2

While most interviewees read the introduction, three appeared only to skim the introduction, and two 

indicated that they had not read it at all. Of those who read the introduction, respondents generally agreed 

that the introduction was an informative and adequately succinct summarization, clearly communicating 

the purpose of the survey. Two participants thought the survey applied to people who received federal 

financial aid and a few commented that the gainful employment legislation part of the introduction was 

confusing.

Recommendation:

Remove reference to gainful employment legislation. Revise first sentence of the introduction to read:

The purpose of the Graduate Employment and Earnings Survey is to collect information about 

the average earnings of graduates from programs whose students receive federal financial aid 

and are subject to regulations on gainful employment (34 CFR Parts 600 and 668).

Survey Questions

1. Please verify that you are _______________________________________. Is this 

correct? If you have legally changed your name, but once went by 

______________________, please answer “Yes” below.

Yes

No  Please stop filling out the survey. It is important that you still return the survey in 

the envelope provided so that we will not contact you again.

Most interviewees noted that if there were a minor spelling error in their name, they would make the 

correction and continue filling out the survey. Several respondents noted that if there was an error in their 

name, they might return the survey if provided a postage-paid return envelope. However, six interviewees 

mentioned that, were the listed name completely incorrect, they would throw away the survey rather than 

return it.

Recommendation:

Shorten the instructions associated with the “No” response option to increase the chances of people 

reading them. Revise to read:

Thank you. Please return the survey in the attached envelope, so that we can remove you from 

our list.

2. According to our information, you completed the _______________________ 

program at _____________________________ between _______________ and 

___________________. Is that correct?

Nearly all of the interviewees mentioned that they would have no difficulty recalling the year they began 

and ended their program; however, some reported that recalling the months might be more difficult.

Several interviewees noted their ease in recalling the date was due to the recentness or significance of 

program completion. 
3

Recommendation:

None.

3. Did you work at a job or business at any time during 2014?

Seven interviewees interpreted this question as asking about a job or business related to their degree 

program, despite the question not referencing this in particular. Their confusion may be due to the 

question’s proximity to the prior question which references the student’s degree program completion. 

The remainder of respondents generally saw no difference between “job” or “business,” and interpreted 

the question as “Did you have anywhere you earned income in 2014?” These respondents generally 

mentioned interpreting this question as excluding any unpaid work (e.g. unpaid internships or volunteer 

work).

Recommendation:

Revise to read: During 2014, did you work for pay at a job or business? (Source: modified Q29 of the 

American Community Survey—PAPI version).

4. Did you receive any wages or salary in 2014? 

Respondents had no difficulty with “calendar year” and all interpreted as meaning January 1, 2014 –

December 31, 2014. However, interpretations of the terms “salary” and “wages” were inconsistent. Seven 

interviewees interpreted “salary” as referring to a fixed yearly rate and wages as hourly pay, seven noted 

that they saw no difference between the two terms, and one interviewee interpreted salary as hourly pay 

and wages as a fixed yearly rate. In addition, some participants with more than one job (for example, a 

salaried job and part-time jobs, such as baby sitting, tutoring, or participating in studies) did not consider 

their part time jobs when answering this question.

Recommendation:

Start the series of questions on income with a common lead-in: “The next set of questions are about the 

different types of income you might have received in 2014.”

Additionally, include instructions to the question: “Please include both full-time jobs and part-time 

jobs.” Alternatively, revise the question to read: “In 2014, did you receive any wages or salary from fulltime or part-time jobs?”

4a. How much did you receive in wages and salary from ALL jobs before taxes and 

other deductions in 2014? Be sure to include wages and salary from ALL jobs in 2014

Nearly all interviewees expressed having difficulty with this question. Interviewees were generally 

confused as to what to include and what to exclude in this amount, especially after encountering the 

remaining questions (5 – 6). Three respondents revised their initial answers to this question once they 

attempted to answer the following questions (5 – 6), mentioning that they thought first 4a included 

everything, but were then unsure. Several interviewees noted that they excluded part-time jobs or 

irregular pay (e.g. payment for babysitting, participation in surveys, gambling winnings, tips, bonuses). 
4

Most interviewees mentioned that the amount they provided was a rough estimate; however, two said they 

had simply reported what was on their W-2, Wage and Tax Statement, which they had recently received.

Two interviewees arrived at an amount by first calculating approximate monthly earnings before taxes 

and then multiplying by 12 months. 

A majority of respondents noted that the amount which they provided excluded taxes and deductions, and 

had no trouble interpreting the term “deductions.” Two interviewees expressed confusion over how to 

enter the numeric amount given the “.00” at the end of the blank field.

All but one of the interviewees noted that they had no reservations providing income information, and did 

not feel the questions infringed on their privacy.

Recommendation:

Restructuring the questionnaire to include all income questions as subparts of one question, using a 

common introduction (see recommendation to question 4) should eliminate double counting as it will 

become obvious to respondents that questions 4a-6a are subcategories that should sum up to their total 

annual income. 

To further ensure tips and bonuses are not double counted, follow the approach of the American 

Community Survey and combine questions 4a and 5a, asking respondents to report on “wages, salary, 

commissions, bonuses, or tips from all jobs.” Include instructions that all jobs should be considered 

whether or not they are related to the program of study.

In addition, use a box, rather than a blank field, where the requested dollar amount can be entered.

5. Did you receive any additional tips, overtime pay, bonuses, or commissions? 

Three respondents indicated “No” on this question because they had already reported overtime and 

commission pay in the previous question (4a). Two interviewees treated this question as a catch-all 

category and interpreted it as asking about money made in addition to salary and wages, including “under 

the table” pay or for money made in small temporary jobs. One interviewee indicated “No” because she 

did not report her tips for tax purposes, and one mentioned excluding tips and commissions due to their

job not being related to their degree program.

Recommendation:

Include tips, bonuses, overtime pay, and commissions as part of the overall income from wages and 

salaries in questions 4a. 

5a. How much did you receive in tips, overtime pay, bonuses, or commissions from ALL 

jobs before taxes and other deductions in 2014? Be sure to include ALL tips, overtime 

pay, bonuses, or commissions in 2014.

Similarly to the previous question, the interpretation of this question and its relationship to question 4a 

was puzzling to most respondents. Five of the eight eligible to answer this question noted that they had 

already included tips, overtime pay, bonuses and commissions in their response to question 4a. Several 

interviewees mentioned that this figure would be hard to come up with; some of the recall strategies 

mentioned were estimation based on a minimum yearly figure, and estimation based on a monthly figure. 
5

A self-employed person was confused whether to include his commission under 4a, or 6a, but never 

considered it in response to this question, that explicitly states commissions.

Recommendation:

Include instructions in question 4a to include this figure in the overall income from wages and salaries. 

6. Did you receive any self-employment income? Report income from own business (farm 

or non-farm) including proprietorships and partnerships.

Interviewees generally had no issues determining how to answer this question. One interviewee who 

holds a part-time job in a salon, was unsure how to answer this question, as she rents space in the salon 

and pays for her own supplies. Several interviewees who were not self-employed, expressed difficulty 

determining the difference between proprietorships and partnerships when probed on the meaning by 

interviewers.

Recommendation:

None.

6a. What was the amount you received in self-employment income before taxes and other 

deductions in 2014? Report net income after operating expenses. If net income was a 

loss, mark the “Loss” box to the right of the dollar amount.

Four interviewees expressed confusion over what to enter in this field because they had already reported 

their self-employment income in question 4a.

While most of the interviewees had no issue identifying what net income after operating expenses was 

conceptually, two noted that they had reported the amount before operating expenses.

Three interviewees noted confusion over what “loss” would mean, with one believing that expenses for 

equipment would be considered a loss, and one thinking that “loss” pertained only to those dependent on 

the stock market for their income. 

Recommendation:

Restructure the questionnaire to include all income questions as subparts of one question, using a 

common introduction (see recommendation for question 4). Graphically (using a different font or 

underlining) emphasize that the requested amount should be reported after business expenses. Remove 

the instructions for “loss” from the question stem, as they seem to confuse rather than help respondents.

Overall Impressions

Interviewees were generally pleased by the brevity and straightforwardness of the survey. Most noted that 

they had no privacy concerns about any of the questions and would leave none blank, but several 

emphasized that they would participate only if the survey looked official and was clearly from their prior 

institution or the U.S. Department of Education. Four interviewees emphasized their confusion about 

where to report what income, and what should be included and excluded in each question.
6
7

Graduate Employment and Earnings Survey

Cognitive Interviews, Round 2 - Results and Recommendations

1. Recruitment and Participation

Cognitive testing was conducted from March 17 to April 23, 2015, in three of RTI’s U.S. office 

locations – Research Triangle Park, NC; Washington, DC; and Chicago, IL. To recruit graduates, 

RTI posted web advertisements on Craigslist, Backpage.com, and Ebay Classifieds, and used 

direct outreach though advertisement postings at area businesses. A total of 41 individuals 

responded to the posted advertisements; of these, seven were ineligible, 15 were eligible and 

participated, and eight were eligible but did not participate due to cancellation, difficulty

scheduling a time to participate, or not coming to their scheduled appointment. Participants were 

recent graduates (up to 4 years following completion of the credential) from institutions in the 

vicinity of RTI’s offices, including online program graduates residing in the same geographic 

areas. Of those who participated, 11 were female and four male. Table 2 provides a distribution 

of the final number of interview participants, by control and level of institution.

Table 2: Number of respondents, by institution control and level

Participants were given the short paper survey and asked to complete it on their own. Upon 

completion, they were asked to think aloud retrospectively in order to identify potential problems 

related to question comprehension, recall, information editing due to sensitivity or social 

desirability and response mapping. On average, the interviews took 22 minutes to complete.

2. Results and Recommendations

Introduction

The purpose of the Graduate Employment and Earnings Survey is to collect information about 

the average earnings of graduates from programs whose students receive federal financial aid 

and are subject to regulations on gainful employment.

The information you provide on this survey will be grouped with information from other 

graduates of your program. Average earnings for all graduates of the program will be calculated 

Institution control Institution level Credential type(s)

Number of 

respondents

Private for-profit Less-than-2-year Vocational certificates 5

2-year Vocational certificates and 

associates 3

4-year All 2

Private not-forprofit

Less-than-2-year and 2-

year

Vocational certificates and 

associates 1

Public Less-than-2-year and 2-

year

Vocational certificates and 

associates 4
8

and reported to the U.S. Department of Education. Your individual earnings will not ever be 

reported, disclosed or used in any other manner unless required by law. 

About half of the cognitive interview participants actually read the introductory statement. Participants 

seemed to have a good understanding about the purpose of the survey. Only one participant commented 

that she did not quite understand the introduction as it contained a lot of information; the rest of the 

respondents did not seem to have problems with it. 

Recommendation:

No changes.

Survey Questions

1. Please verify that you are <First-Name Last-Name>. Is this correct? If you have 

legally changed your name, but once went by <First-Name Last-Name>, please answer 

Yes below.

☐ No Thank you. Please return the survey in the enclosed envelope so that we can 

remove you from our list and you are not contacted again.

☐ Yes

Most interviewees reported that if the name was completely wrong, they would likely return the survey 

rather than proceeding to fill it out. Four said they would likely discard the interview, as it would be 

easier than mailing back. Two noted that if there was a telephone number to contact, they would call if the 

name was incorrect. Two interviewees expressed some difficulty with the legally changed name portion 

of this question.

Recommendation:

No changes to the question.

We would encourage institutions to designate a telephone number when launching the survey and include 

that number in all communications with sample members.

2. Did you receive any wages, salary, tips, overtime pay, bonuses or commissions 

from an employer in 2013? 

☐ No Skip to question 3.

☐ Yes

Fourteen interviewees confirmed that they were recalling information from the 2013 year, and that recall 

was not difficult. Several respondents noted that they used key events (e.g., being pregnant) in 2013 to 

help aid in recall; while others mentioned that they recently reviewed their 2014 taxes while preparing 

their 2015 taxes. One respondent thought that the question was asking about the current year.

When answering this question, respondents reported including income from full-time jobs, selfemployment, casual employment (e.g. babysitting, nanny work, geriatric care), part-time employment, 

and tips in their consideration of this question. Two respondents excluded student financial aid. One 

respondent was unsure of whether to include child support income, work-study income, or grant money. 
9

One respondent excluded overtime pay. One respondent expressed hesitance to include “off the books” 

income. 

Recommendation:

No changes. 

2a. Think about the job you held the longest during 2013. In total, how much did you 

earn from that employer, before taxes and other deductions, in 2013?

$_____________

2b. Now think about all OTHER jobs you held during 2013. In total, how much did you 

receive from all other employers, before taxes and other deductions, in 2013? If you 

had no additional jobs, please enter a zero.

$_____________

At least half of the participants had only one job in 2013. For those with more than one, the question 

about the job held the longest did not seem to present a problem. One respondent interpreted the question 

to mean her “primary” job (i.e, the job where she gained the larger portion of her income).

One respondent commented that initially, he thought question 2b should include any miscellaneous 

income, but later saw question 4, so concluded that question 2b referred to regular pay only. Another 

respondent, who had only one job, entered his income amount here and left question 2a blank. Jobs 

reported under question 2b included part-time jobs such as baby-sitting, CNA for a private person, or 

other “part time odd jobs” that were not reported in respondents’ taxes. Three respondents incorrectly 

reported self-employment.

Recommendation:

Combine questions 2a and 2b into one question, asking about wages, salary, tips, overtime pay, bonuses 

or commissions from all employers in 2013. Since several participants commented that they reviewed

their previous year’s taxes before doing their current taxes, reporting income from all jobs might be an 

easier cognitive task. The initial intention of the longest job question was to help recall, but respondents 

seemed to use personal landmark events instead. Furthermore, this question will not be relevant for 

respondents with only one job and might unnecessarily confuse them on where to report their income. 

The example of types of income reported in question 2b suggests that there might be an overlap with what 

is reported in question 4 (see below for some evidence). In order to avoid double counting, we 

recommend the following overarching structure:

1. Earnings from all employers

2. Earnings from self-employment 

3. Earnings from other jobs, such as freelancing, consulting, etc.

To follow this structure, we recommend combining questions 2a and 2b into the following one question:
10

In total, how much did you earn from all employers, before taxes and other deductions, in 

2013?

3. Excluding any income mentioned above, did you receive any self-employment 

income, including income from your own business? 

☐ No Skip to question 4.

☐ Yes

3a. What was the amount you received in net self-employment income in 2013? 

Please report income minus expenses. If this amount is negative, please enter 

zero.

$_____________ 

All but one participant correctly defined self-employment as “working for yourself” and “running/owning

your business.” Two respondents suggested that “1099” be included in the question as an example of 

self-employment. Four participants seemed to treat this question as a catch-all category, and noted that 

they would include tips, any other on-the-side income, or income earned outside of the traditional 40 hour 

work week.

One respondent was confused by what “net self-employment” income meant and she guessed it was 

income before taxes. One respondent missed the routing instructions in 3, therefore the question did not 

make sense. The respondent suggested that the skip instruction and arrow should be more prominent.

Six out of the eight respondents who answered question 3a did not have problems with the “income minus 

expenses” concept and were able to provide their own examples.

Recommendation:

No changes.

4. People sometimes earn extra money doing work outside of their regular jobs, such as 

freelancing, consulting, moonlighting, or doing other casual jobs. In 2013, did you earn 

any other money from work you did that you have not already reported on this 

survey?

☐ No Thank you. Please return the survey in the envelope provided.

☐ Yes

4a. What was the amount of your additional earnings from work in 2013 that are 

not already reported above? 

$_____________ 

Three respondents found this question overlapping with the self-employment question (question 3), as 

they considered jobs such as doing hair and baby-sitting. Most participants interpreted the question to 

mean “side gigs,” participating in clinical trials, lawn mowing, or other ways of making extra money and 

being paid in cash. Two respondents double-counted by considering income already reported in 2b. One 
11

respondent who was previously confused about whether she should include grant money, internship 

funding, and child support noted that she would likely have included that income here in 4. This 

respondent, however, was still not sure if they should include Social Security Income, veterans’ benefits, 

or Social Security benefits. Two participants noted they were unsure of the meaning of “moonlighting” 

and “freelancing,” and one respondent expressed that they were confused about the term “casual jobs.” 

One individual commented that this question was an invasion of his privacy.

Recommendation:

No changes to the question. The issue with occasional double counting between this question and 

questions 2b and 3 will be resolved with the new structure, proposed in the recommendations for 

questions 2a and 2b.

Overall Impressions

Interviewees were generally pleased by the brevity and straightforwardness of the survey. Most noted that 

they had no privacy concerns about any of the questions. Six said that they would be willing to complete 

the survey if they received it. However four said they would not likely complete the survey. Among 

reasons for not wanting to complete the survey, respondents noted that they already receive “too many 

things in the mail”; they would want more information about the study and how the information would be 

used; there were too many questions about income and did not feel comfortable providing such 

information; concerned that the questions might be used in an IRS audit. One interviewee noted that some 

questions (e.g. question 2) look like a question from a tax form, and therefore people might be suspicious 

that the information would be matched or compared directly to tax information. This respondent 

suggested including a confidentiality note explaining that the information provided would not affect your 

income for the year or your eligibility for grants and loans. 

Two interviewees emphasized their confusion about where to report what income, and what should be 

included and excluded in each question. One commented that the layout was difficult to navigate and the 

font was too small to read, and that the interview needed more use of highlighting and bolding to 

emphasize important portions. Another respondent noted that some of the wording needs to be simplified, 

but did not provide any specific examples. One participant expressed a concern over the sensitivity of the 

questions, and noted that they would want to seal and tape the envelope to ensure the information was 

secure. Another respondent noted that the survey would need to look official and include a letter with 

signature and toll-free phone number for questions.

A revised survey, incorporating the recommended revisions, is provided below.
12

Recent Graduates Employment and Earnings Survey

The purpose of the Recent Graduates Employment and Earnings Survey is to collect information about the average 

earnings of graduates from programs whose students receive federal financial aid and are subject to regulations on 

gainful employment.

The information you provide on this survey will be grouped with information from other graduates of your program. 

Average earnings for all graduates of the program will be calculated and reported to the U.S. Department of 

Education. Your individual earnings will not ever be reported, disclosed or used in any other manner. 

1. Please verify that you are <First-Name Last-Name>. Is this correct? If you have legally changed your 

name, but once went by <First-Name Last-Name>, please answer Yes below.

☐ No Thank you. Please return the survey in the enclosed envelope so that we can remove you from our 

list and you are not contacted again.

☐ Yes

The next few questions are about income during the 2013 calendar year, NOT the most recent 2014 calendar 

year.

This survey asks about wages and self-employment income separately. In some cases, it may not be clear to you 

where income you received should be reported. Please just use your best judgment about this, but please do NOT 

report the same income more than once. Remember to include income from ALL jobs.

2. Did you receive any wages, salary, tips, overtime pay, bonuses or commissions from an employer in 

2013?

☐ No Skip to question 3.

☐ Yes

2a. In total, how much did you earn from all employers, before taxes and other deductions, in 2013?

$_____________

3. Excluding any income mentioned above, did you receive any self-employment income, including 

income from your own business? 

☐ No Skip to question 4.

☐ Yes

3a. What was the amount you received in net self-employment income in 2013? Please report 

income minus expenses. If this amount is negative, please enter zero. 

$_____________ 

 

4. People sometimes earn extra money doing work outside of their regular jobs, such as freelancing, 

consulting, moonlighting, or doing other casual jobs. In 2013, did you earn any other money from work 

you did that you have not already reported on this survey?

☐ No Thank you. Please return the survey in the envelope provided.

☐ Yes
13

4a. What was the amount of your additional earnings from work in 2013 that are not already 

reported above? 

$_____________ 

 

Thank you for your participation.

Please return the survey in the envelope provided.