1 

### Ap **pendix 4 Graduate**  **Employ ment and Earnings Survey**  **Cognitive Interview s**  **, Round 1 -**  **Results and Recommendations 1.**  **Recruitment and P**  **articipation** Cognitive testing was conducted from January 26 to February 11, 2015, in three of RTI’s U.S. office locations

– Research Triangle Park, NC; Washington, DC; and Chicago. To recruit graduates, RTI posted web advertisements on Craigslist and used direct outre ach though advertisement postings at area businesses. A total of 42 individuals responded to the posted advertisements; of these, 19 were ineligible, 15 were eligible and participated, and 8 were eligible but did not participate due to cancellation, diffic ulty scheduling a time to participate, or not coming to their scheduled appointment . Participants were recent graduates (up to 4 years following completion of the credential) from institutions in the vicinity of RTI’s offices, including online program grad uates residing in the same geographic areas. Of those who participated, nine were female and six male. Table 1 provides a distribution of the final number of interview participants, by control and level of institution. **Table 1: Number of respondents, by i**  **nstitution control and level**

Participants were given the short paper survey and asked to complete it on their own. Upon completion, they were asked to think aloud retrospectively in order to identify potential problems related to question comprehension, recall, information editing du e to sensitivity or social desirability and response mapping. On average, the interviews took 32 minutes to complete. **2. Results and Recommendations**  **Intro duction**  The purpose of the Graduate Employment and Earnings Survey is to collect information about th e average earnings of graduates from programs whose students receive federal financial aid and are subject to regulations on gainful employment (34 CFR Parts 600 and 668). The information on this survey will be used to report average earnings for program graduates to the U.S. Department of Education. Your individual earnings will never be disclosed or used in any other manner unless required by law. **Institution control**  **Institution level Credential**  **ty pe**  **(s) Number**  **of respondents**  **Private for -**  **profit** Less
- than
- 2
- year Vocational certificates 3

2
- year Vocational certificates and associates 2 4
- year All 2 **Private not**  **- for**  **** **profit** Less
- than
- 2
- year and 2
-

ye ar Vocational certificates and associates 1 **Public** Less
- than
- 2
- year and 2
-

ye ar Vocational certificates and associates 72

While most interviewees read the introduction, three appeared only to skim the introduction, and two indicated that they had not read it at all. Of those who read the introduction, respondents generally agreed that the introduction was an informative and adequately succinct summarization, clearly communicating the purpose of the survey. Two participants thought the survey applied to people who received federal financial aid and a few commented that the gainful employment legislation part of the introducti on was confusing. **Recomme n**  **dation :**  Remove reference to gainful employment legislation. Revise first sentence of the introduction to read : The purpose of the Graduate Employment and Earnings Survey is to collect information about the average earnings of graduates from programs whose students receive federal financial aid and are subject to regulations on gainful employment (34 CFR Parts 600 and 668)

. **Survey Questions** 
**1. Please verify** 

**that y ou are _______________________________________. Is this**  **correct?** If you have legally changed your name, but once went by ______________________, please answer “Yes” below. Yes No  Please stop filling out the survey. It is important that you st ill return the survey in the envelope provided so that we will not contact you again. Most interviewees noted that if there were a minor spelling error in their name, they would make the correction and continue filling out the survey. Several respondents noted that if there was an error in their name, they might return the survey if provided a postage

-paid return envelope. However, six interviewees mentioned that, were the listed name completely incorrect, they would throw away the survey rather than return it. **Recommendation :**  Shorten the instructions associated with the “N o” response option to increase the chances of people reading them. Revise to read : Thank you. Please return the survey in the attached envelope, so that we can remove you from our list.

**2. Ac**  **cording to our information, y ou completed the**  **_______________________ program at _____________________________ between _______________ and**  **___________________. Is that correct?** Nearly all of the interviewees mentioned that they would have no difficulty recalling the year they began and ended their p rogram ; however , some reported that recalling the month s might be more difficult. Several interviewees noted their ease in recalling the date was due t o the recentness or significance of program completion.3

**Recommendation:** None.

**3. Did y**  **ou w ork at a job**  **or business at any time during 2014?**  Seven interviewees interpreted this question as asking about a job or business related to their degree program , despite the question not referencing this in particular. Their confusion may be due to the question’s prox imity to the prior question which references the student’s degree program completion. The remainder of resp ondents generally saw no difference between “job” or “business,” and interpreted the question as “Did you have anywhere you earned income in 2014?” These respondents generally mentioned interpreting this question as excluding any unpaid work (e.g. unpaid internships or volunteer work). **Recommendation:** Revise to read: During 2014, did you work for pay at a job or business ? ( Source: modified Q29 of the American Community Survey

—PAPI version)
.
**4. Did y**  **ou receive any w**  **ages or salary**

**in 2014?** Respondents had no difficulty with “calendar year” and all interpreted as meaning January 1, 2014

– December 31, 2014. However, interpretations of the terms “sa lary” and “wages” were inconsistent. Seven interviewees interpreted “salary” as referring to a fixed yearly rate and wages as hourly pay, seven noted that they saw no difference between the two terms, and one interviewee interpreted salary as hourly pay and wages as a fixed yearly rate. In addition, some participants with more than one job (for example, a salaried job and part

- time jobs, such as baby sitting, tutoring, or participating in studies) did not consider their part time jobs when answering this question. **Recommendation:** Start the series of questions on income with a common lead

- in: “ The next set of questions are about the different types of income you might have received in 2014. ” Additionally, i nclude instr uctions to the question: “Please include both full

- time jobs and part

- time jobs
. ” Alternatively, revise the question to read: “ In 2014, did you receive any wages or salary from full

time or part

- time jobs? ”

**4 a**  **. How** **much did y ou receive in wages and**  **salary from A LL jobs before taxes and**  **other deductions in 2014**  ? Be sure to include wages and salary from ALL jobs in 2014 Nearly all interviewees expressed having difficulty with this question. Interviewees were generally confused as to what to include and what to exclude in this amount, especially after encountering the remaining questions (5

– 6). Three respondents revised their initial answers to this question once they attempted to answer the following questions ( 5

– 6 ) , mentioning that they thought first 4 a included everything , but were then unsure. Several interviewees noted that they excluded part

- time jobs or irregular pay (e.g. payment for babysitting, participation in surveys, gambling w innings, tips, bonuses).4

Most interviewees mentioned that the amount they provided was a rough estimate ; however , two said they had simply reported what was on their W

- 2, Wage and Tax Statement, which they had recently received. Two interviewees arrived at an amount by first calculating approximate monthly earnings before taxes and then multiplying by 12 months. A majority of respondents noted that the amount which they provided excluded taxes and deductions , and had no trouble interpreting the term “de ductions
. ” Two interviewee s expressed confusion over how to enter the numeric amount given the “.00” at the end of the blank field. All but one of the interviewees noted that they had no reservations providing income information, and did not feel the questions infringed on their privacy. **Recommendation:** Restructuring the questionnaire to include all income questions as subparts of one question, using a common introduction ( see recommendation to question 4) should eliminate double counting as it will b ecome obvious to respondents that questions 4a

- 6 a are subcategories that should sum up to their total annual income. To further ensure tips and bonuses are not double counted, follow the approach of the American Community Survey and combine questions 4a and 5a, asking respondents to report on “wages, salary, commissions, bonuses, or tips from all jobs.” Include instructions that all jobs should be considered whether or not they are related to the program of study. In addition, use a box, rather than a blank field, where the requested dollar amount can be entered.

**5. Did y**  **ou receive any additional tips, overtime pay**  **, bonuses, or commissions?** Three respondents indicated “No” on this question because they had already reported overtime and commission pay in the previous question ( 4 a)
. Two interviewees treated this question as a catch

- all category and interpreted it as asking about money made in addition to salary and wages, including “under the table” pay or for money made in small temporary jobs. One interviewee indicated “No” because she did not report her tips for tax purposes, and o ne mentioned excluding tips and commissions due to their job not being related to their degree program. **Recommendation:** Include tips, bonuses, overtime pay, and commissions as part of the overall income from wages and salaries in questions 4a.
**5a. Ho w much did y**  **ou receive in tips, overtime pay , bonuses, or commissions from A**  **LL jobs before taxes and other deductions in 2014?**  Be sure to include ALL tips, ove rtime pay, bonuses, or commissions in 2014. Similarly to the previous question, the interpretation of this question and its relationship to question 4a was puzzling to most respondents. Five of the eight eligible to answer this question noted that they ha d already included tips, overtime pay, bonuses and commissions in their response to question 4a. Several interviewees mentioned that this figure would be hard to come up with; some of the recall strategies mentioned were estimation based on a minimum yearly figure, and estimation based on a monthly figure.5

A self

- em ployed person was confused whether to include his commission under 4a, or 6a, but never considered it in response to this question, that explicitly states commissions. **Recommendation:** Include instructions in question 4a to include this figure in the overall income from wages and salaries.

**6. Did y**  **ou receive any self**  **- employment income?**  Report income from own business (farm or non - farm) including proprietorships and partnerships. Interviewees generally had no issues determining how to answer this question. One interviewee who holds a part

- time job in a salon, was unsure how to answer this question, as she rents space in the salon and pays for her own supplies. Several interviewees who were not self

- em ployed, expressed difficulty determining the difference between proprietorships and partnerships when probed on the meaning by interviewers
. **Recommendation:** None.
**6a .**  **What was the amount y ou received in self** **- employ**  **ment income before taxes and other deductions in 2014?**  Report net income after operating expenses. If net income was a loss, mark the “Loss”

box to the right of the dollar amount. Four interviewees expressed confus ion over what to enter in this field because they had already reported their self

- em ployment income in question 4 a. While most of the interviewees had no issue identifying what net income after operating expenses was conceptually, two noted that th ey had reported the amount before operating expenses. Three interviewees noted confusion over what “ l oss” would mean, with one believing that expenses for equipment would be considered a loss, and one thinking that “l oss” pertained only to those dependent on the stock market for their income. **Recommendation:** Restructure the questionnaire to include all income questions as subparts of one question, using a common introduction (see recommendation for question 4 ). Graphically (using a different font or underlining) emphasize that the requested amount should be reported after business expenses. Remove the instructions for “loss” from the question stem, as they seem to confuse rather than help respondents.
**Overall Impressions**  Interviewees were generally pleased by the brevity and straightforwardness of the survey. Most noted that they had no privacy concerns about any o f the questions and would leave none blank , but several emphasized that they would participate only if the survey looked official and was clearly from their prior institution or the U.S. Department of Education
. Four interviewees emphasized their confusion about where to report what income , and wh at should be included and excluded in each question.67



### Graduate Employ **ment and Earnings Survey Cognitive Interview**  **s, Round 2 -**  **Results and Recommendations 1.**  **Recruitment and P**  **articipation** Cognitive testing was conducted from March 17 to April 23, 2015, in three of RTI’s U.S. office locations

– Research Triangle Park, NC; Washington, DC; and Chicago, IL. To recruit graduates, RTIposted web advertisements on Craigslist, Backpage.com, and Ebay Classifieds, and used direct outreach though advertisement posting s at area businesses. A total of 41 individuals responded to the posted advertisements; of these, seven were ineligible, 15 were eligible and participated, and eight were eligible but did not participate due to cancellation, difficulty scheduling a time to participate, or not coming to their scheduled appointment. Participants were recent graduates (up to 4 years following completion of the credential) from institutions in the vicinity of RTI’s offices, including online program graduates residing in the sam e geographic areas. Of those who participated, 11 we re female and four male. Table 2 provides a distribution of the final number of interview participants, b y control and level of institution.
**Table 2 : Number of respondents, by**  **institution control and lev el** 



### Participants were given the short paper survey and asked to complete it on their own. Upon completion, they

were asked to think aloud retrospectivel y in order to identify potential problems related to question comprehension, recall, information editing du e to sensitivity or social desirability

and response mapping. On average, the interviews took 2 2 minutes to complete.
**2. Results and Recommendations**  **Intro duction**  The purpose of the Graduate Employment and Earnings Survey is to collect information about the average earnings of graduates from programs whose students receive federal financial aid and are subject to regulations on gainful employment. The information you provide on this survey will be grouped with information from other graduates of your program. Average earnings for all graduates of the program will be calculated **Institution control**  **Institution level Credential**  **ty pe**  **(s) Number**  **of respondents**  **Private for -**  **profit** Less
- than
- 2
- year Vocational certificates 5

2
- year Vocational certificates and associates 3 4
- year All 2 **Private not -** **for**   ** profit** Less
- than
- 2
- year and 2
-

ye ar Vocational certificates and associates 1 **Public** Less
- than
- 2
- year and 2
-

ye ar Vocational certificates and associates 48

and reported to the U.S. Department of Education. Your individual earnings will not ever be reported, disclosed or used in any other manner unless required by law

. About half of the cognitive interview participants actually read the introductory statement. Participants seemed to have a good understanding about the purpose of the survey. Only one participant commented that she did not quite understand the introduction as it contained a lot of information; the rest of the respondents did not seem to have problems with it. **Recommendation:** No changes. **Survey Questions**  1. **Please verify**

**that y ou are <First**  **- Name Last**  **- Name>. Is this correct?**  If you have legally changed your name, but once went by <First
- Name Last
- Name>, please answer Yes below. ☐ No  Thank you. Please return the survey in the enclosed envelope so that we can remove you from our list and you are not contacted again. ☐ Ye s Most interviewees reported that if the name was completely wrong, they would likely return the survey rather than proceeding to fill it out. Four said they would likely discard the interview, as it would be easier than mailing back. Two noted that if there was a telephone number to contact, they would call if the name was incorrect. Two interviewees expressed some difficulty with the legally changed name portion of this question

. **Recommendation:** No changes to the question
. We would encourage institutions to designate a telephone number when launching the survey and include that number in all communications with sample members.

**2. Did y**  **ou receive any w**  **ages, salary , tips, overtime pay**  **, bonuses or commissions from an employ** **er in 2013?** ☐ No  Skip to question 3

. ☐ Yes  Fourteen interviewees confirmed that they were recalling information from the 2013 year, and that recall was not difficult. Several respondents noted that they used key events (e.g., being pregnant) in 2013 to help aid in recall ; while others mentioned that they recently reviewed their 2014 taxes while preparing their 2015 taxes
. One respondent thought that the question was asking about the current year. When answering this question, r espondents reported including income from full

- time jobs, self

employment, casual employment (e.g. babysitting, nanny work, geriatric care), part

- time employment, and tips in their consideration of this question. Two respondents excluded student financial aid. One respondent wa s unsure of whether to include child support income, work

- study income, or grant money.9

One respondent excluded overtime pay. One respondent expressed hesitance to include “off the books” income. **Recommendation:** No changes.

**2a.** Think about **the job y ou held the longest**  during 2013. **In total** , how much did you earn **from that employ er,**  before taxes and other deductions, in **2013** ? $_____________

**2b.** Now think about **all OTHER jobs** you held during 2013. **In total** , how much did you receive **from all other employ ers,**  before taxes and other deductions, in **2013** ? If you had no additional jobs, please enter a zero. $_____________ At least half of the participants had only one job in 2013. For those with more than one, the question about the job held the longest di d not seem to present a problem . One respondent interpreted the question to mean her “primary” job (i.e, the job where she gained the larger portion of her income). One respondent commented that initially, he thought question 2b shoul d include any miscellaneous income, but later saw question 4, so concluded that question 2b referred to regular pay only. Another respondent, who had only one job, entered his income amount her e and left question 2a blank. Jobs reported under question 2b included part

- time jobs such as baby

- sitting, CNA for a private person, or other “part time odd jobs” that were not reported in respondents’ taxes. T hree respondents incorrectly reported self

- employment. **Recommendation:** Combine questions 2a and 2b into one question, asking about wages, salary, tips, overtime pay, bonuses or commissions from all employers in 2013. Since several participants commented that they reviewed their previous year ’s taxes before doing their current taxes, reporting income from all jobs might be an easier cognitive task. The initial intention of the longest job question was to help recall, but respondents seemed to use personal landmark events instead. Furthermore, this question will not be relevant for respondents with only one job and might unnecessarily confuse them on where to report their income. The example of types of income reported in question 2b suggests that there might be an overlap with what is reported in question 4 (see below for so me evidence) . In order to avoid double counting, we recommend the following overarching structure: 1. Earnings from all employers 2. Earnings from self

- em ployment 3. Earnings from other jobs, such as freelancing, consulting, etc. To follow this structure, we re commend combining questions 2a and 2b into the following one question :10

In total, h ow much did you earn from **all employ ers**  , before taxes and other deductions , in 2013 ?
**3. Excluding any** 

**income mentioned above, did you receive any self**  **- employment**  **income, including income from y our ow**  **n business?** ☐ No  Skip to question 4. ☐ Yes  **3a.** Wh at was the amount you received in net self
- employment income in **2013** ? Please report income minus expenses. If this amount is negative, please enter zero. $_____________ All but one p articipant correctly defined self

- em ployment as “working for yourself” and “ running/owning your business
. ” Two respondents suggested that “1099” be in cluded in the question as an example of self

- em ployment. Four participants seemed to treat this question as a catch

- all category, and noted that they would include tips, any other on

- the

- side income, or income earned outside of the traditional 40 hour wo rk week. One respondent was confused by what “net self

- em ployment” income meant and she guessed it was income before taxes
. One respondent missed the routing instructions in 3, therefore the question did not make sense. The respondent suggested that the skip instruction and arrow should be more prominent. Six out of the eight respondents who answered question 3a did not have problems with the “income minus expenses” concept and were able to provide their own examples. **Recommendation:** No changes.
**4.** People sometimes earn extra money doing work outside of their regular jobs, such as freelancing, consulting, moonlighting, or doing other casual jobs. **In 2013, did y ou earn**  **any other money** 

**from work y ou did that y**  **ou have not already reported on this**  **survey ?**  ☐ No  Thank you. Please return the survey in the envelope provided. ☐ Yes  **4a.** Wh at was the amount of your additional earnings from work in **2013** that are not already reported above? $_____________ Three respondents found this question ov erlapping with the self

- em ployment question (question 3), as they considered jobs such as doing hair and baby

- sitting. Most participants interpreted the question to mean “side gigs , ” participating in clinical trials, lawn mowing, or other ways of making extra money and being paid in cash. Two respondents double

- counted by considering income already reported in 2b. One11

respondent who was previously confused about whether she should i nclude grant money, internship funding, and child support noted that she would likely have included that income here in 4. This respondent, however, was still not sure if they should include Social Security Income, veterans’ benefits, or S ocial S ecurity be nefits. Two participants noted they were unsure of the meaning of “moonlighting” and “freelancing,” and one respondent expressed that they were confused about the term “casual jobs.” One individual commented that this question was an invasion of his privac y. **Recommendation:** No changes to the question. The issue with occasional double counting between this question and questions 2b and 3 will be resolved with the new structure, proposed in the recommendations for questions 2a and 2b.
**Overall Impressions** Interviewees were generally pleased by the brevity and straightforwardness of the survey. Most noted that they had no privacy concerns about any of the questions. Six said that they would be willing to complete the survey if they received it. However four said they would not likely complete the survey. Among reasons for not wanting to complete the survey, respondents noted that they already receive “too many things in the mail”; they would want more information about the study and how the information woul d be used; there were too many questions about income and did not feel comfortable providing such information; concerned that the questions might be used in an IRS audit. One interviewee noted that some questions (e.g. question 2) look like a question from a tax form, and therefore people might be suspicious that the information would be matched or compared directly to tax information. This respondent suggested including a confidentiality note explaining that the information provided would not affect your i ncome for the year or your eligibility for grants and loans. Two interviewees emphasized their confusion about where to report what income, and what should be included and excluded in each question. One commented that the layout was difficult to navigate and the font was too small to read, and that the interview needed more use of highlighting and bolding to emphasize important portions. Another respondent noted that some of the wording need s to be simplified, but did not provide any specific examples. On e participant expressed a concern over the sensitivity of the questions, and noted that they would want to seal and tape the envelope to ensure the information was secure. Another respondent noted that the survey would need to look official and include a l etter with signature and toll

- free phone number for questions. A revised survey, incorporating the recommended revisions, is provided below.12 

### Recent Graduates Employ **ment and Earnings Survey** The purpose of the Recent Graduate s Employment and Earnings Survey is to collect information about the average earnings of graduates from programs whose students receive federal financial aid and are subject to regulations on gainful employment. The information you provide on this survey will be grouped with informatio n from other graduates of your program. Average earnings for all graduates of the program will be calculated and reported to the U.S. Department of Education. Your individual earnings will not ever be reported, disclosed or used in any other manner

. 1. **Please verify**

**that yo**  **u are <First -**  **Name Last -**  **Name>. Is this correct?** If you have legally changed your name, but once went by <First
- Name Last
- Name>, please answer Yes below. ☐ No  Thank you. Please return the survey in the enclosed envelope so that we ca n remove you from our list and you are not contacted again. ☐ Yes The next few questions are about income **during the 2013 calendar y ear, NOT the most recent 2014 calendar**  **ye ar**  . This survey asks about wages and self
- employment income separately. In some cases, it may not be clear to you where income you received should be reported. Please just use your best judgment about this, but please **do NOT report the same income more than o**  **nce** . Remember to include income **from ALL jobs**

.
**2. Did y** **ou receive an y**  **wages, salary , tips, overtime pay** **, bonuses or commis sions from an employ**  **er in 2013?**  ☐ No  Skip to question 3

. ☐ Yes  **2a.** In total , how much did you earn **from all**  **employ er** **s ,**  before taxes and other deductions, in **2013** ? $_____________

**3. Excluding any** 

**income mentioned above, did y ou receive any**  **self -**  **employ ment income, including**  **income from y our own business?**  ☐ No  Skip to question 4. ☐ Yes  **3a.** Wh at was the amount you received in net self
- employment income in **2013** ? Please report income minus expenses. If this amount is negative, please enter zero. $_____________

**4.** People sometimes earn extra money doing work outside of their regular j obs, such as freelancing, consulting, moonlighting, or doing other casual jobs. **In 2013, did y ou earn any**  **other money**

**from work you did that**  **yo u have not already**  **reported on this survey?** ☐ No  Thank you. Please return the survey in the envelope provided. ☐ Yes 13 **4 a.**  Wh at was the amount of your additional earnings from work in **2013** that are not already reported above? $_____________

Thank you for your participation. Please return the survey in the envelope provided.