Influence of Importance Statements and Box Size on Response Rate and Response Quality of Open-Ended Questions in Web/mail Mixed-Mode Surveys *

To understand the thinking behind respondents' answers, survey designers sometimes use open-ended questions in paper and web surveys. According to Dillman, Smyth, and Christian (2014), open-ended questions can be put into three categories: 1) numerical entry (e.g., count, date, frequency), 2) list of items (e.g., name of persons to whom you provided advice, places visited in the last week), and 3) descriptive open-ended questions (e.g., suggestions for ways to improve service delivery). Among all three, descriptive open-ended questions are often used by researchers because respondents can report rich and detailed information about the topic of interest (Tourangeau, Rips, and Rasinski 2000). Open-ended questions have the potential to generate thick, rich and descriptive responses (Israel 2010; Smyth et al. 2009) but getting a quality response and a high item response rate to these questions can be challenging (Dillman 2007; Israel and Lamm 2012; Reja et al. 2003; Smyth et al. 2009). Placement of open-ended questions in the questionnaire can also be problematic, as space on the page(s) is a constraint for mail surveys (Israel 2010). Even after getting high-quality responses to open-ended questions, data cleaning, coding, and analysis can be costly and time consuming (Denscombe 2008; Dillman 2007; Israel 2014; Smyth et al. 2009). All these issues make openended questions used less frequently by many researchers (Israel 2014). Previous research provides considerable evidence for the influence of verbal and visual design elements (symbols, words, graphics and numbers) on response behavior for closed-ended questions (Christian, Dillman, and Smyth 2007; Dillman et al. 2014; Toepoel and Dillman 2010). Yet research on the effect of visual design elements on responses to open-ended questions is limited. Christian and Dillman (2004) found that the larger of two answer spaces in a mail survey of college students resulted in more words and themes. The findings of Christian and Dillman (2004) were corroborated by Stern, Dillman, and Smyth (2007), where they also found more words were written in the larger box of an open-ended question in a mail survey. On the other hand, Smyth et al., (2009) found that a larger box size for open-ended questions produced more words than did the smaller box for late respondents and, in comparison to mail surveys, web surveys produced more words with a larger box size. Recently, Israel (2010) experimented with a series of answer boxes ranging from 0.28 to 1.68 inches high for two open-ended questions in a mail survey and found a linear increase in the number of words was associated with an increase in the box size. These studies provide a clear indication that a larger answer space for an open-ended question acts as a visual prompt for respondents about the amount of information expected by survey researchers.Besides box size, a motivating statement included with the stem of an openended question can improve response quality. In their web survey of college students, Smyth et al. (2009) found that an open-ended question stem that emphasized its importance increased the response length (as measured by the number of words), number of themes and the likelihood of elaboration in the answer, more so for later respondents. Likewise, Israel (2014) found that including an importance statement resulted in more words for both mail and web modes.Beyond response length, several researchers studied the effect of an importance prompt and box size on item response rates (Israel 2010; 2014; Smyth et al. 2009; Stern et al. 2007; Zuell, Menhold, and Korber 2015). Israel (2010) found that box size did not affect item non-response to open-ended questions for the sizes included in the experiment in a mail survey. On the other hand, Israel (2014) found that an importance prompt improved the item response rate for open-ended questions and especially for mail respondents in mixed-mode surveys. Recently, Zuell et al. …