Mitigating Satisficing in Cognitively Demanding Grid Questions: Evidence from Two Web-Based Experiments

Satisficing often has been assumed to be a hazard to response quality in web surveys because interview supervision is limited in the absence of a human interviewer. Therefore, devising methods that help to mitigate satisficing poses an important challenge to survey methodology. The present article examines whether splitting up cognitively demanding grid questions into single items can be an effective means to reduce measurement error and nonresponse resulting from survey satisficing. Furthermore, we investigate whether modifying the question design decreases the adverse effects of low ability and motivation of the respondents on response quality. The statistical analyses in our study relied on data from two web-based experiments with respondents from an opt-in and a probability-based online panel. Our results showed that using single items increased the response times compared to the use of standard grid questions, which might indicate a greater response burden. However, the use of single items significantly reduced the amount of response nondifferentiation and nonsubstantive responses. Our results further showed that the impact of respondent ability and motivation on the likelihood of satisficing was moderated by the question design.

Related papers:

title: Designing Effective Web Surveys by Mick P. Couper
abstract: Designing Effective Web Surveys is a practical guide to designing Web surveys, based on empirical evidence and grounded in scientific research and theory. It is designed to guide survey practitioners in the art and science of developing and deploying successful Web surveys. It is not intended as a “cookbook”; rather it is meant to get the reader to think about why we need to pay attention to design. It is intended for academic, government, and market researchers who design and conduct Web surveys.

title: Response strategies for coping with the cognitive demands of attitude measures in surveys
abstract: This paper proposes that when optimally answering a survey question would require substantial cognitive effort, some repondents simply provide a satisfactory answer instead. This behaviour, called satisficing, can take the form of either (1) incomplete or biased information retrieval and/or information integration, or (2) no information retrieval or integration at all. Satisficing may lead respondents to employ a variety of response strategies, including choosing the first response alternative that seems to constitute a reasonable answer, agreeing with an assertion made by a question, endorsing the status quo instead of endorsing social change, failing to differentiate among a set of diverse objects in ratings, saying ‘don't know’ instead of reporting an opinion, and randomly choosing among the response alternatives offered. This paper specifies a wide range of factors that are likely to encourage satisficing, and reviews relevant evidence evaluating these speculations. Many useful directions for future research are suggested.

title: The Design of Grids in Web Surveys
abstract: Grid or matrix questions are associated with a number of problems in web surveys. In this article, we present results from two experiments testing the design of grid questions to reduce breakoffs, missing data, and satisficing. The first examines dynamic elements to help guide respondent through the grid, and on splitting a larger grid into component pieces. The second manipulates the visual complexity of the grid and on simplifying the grid. We find that using dynamic feedback to guide respondents through a multiquestion grid helps reduce missing data. Splitting the grids into component questions further reduces missing data and motivated underreporting. The visual complexity of the grid appeared to have little effect on performance.

title: Speeding in Web Surveys: The tendency to answer very fast and its association with straightlining
abstract: Web surveys can be programmed to capture a variety of paradata regarding how respondents answer questions. These paradata provide great opportunities for researchers to assess response quality, specifically whether respondents engage in satisficing – not spending enough effort to provide accurate responses. In particular, speeding (i.e., giving answers very quickly) has increasingly been used as an indicator for satisficing and low response quality. However, few studies have examined whether speeding actually leads to compromised response quality. To address this gap in the literature, the current study investigates speeding behaviors among Web respondents from a probability-based panel. We first identify and characterize respondents who speed more frequently than others over the entire questionnaire. To explore the impact of speeding on response quality, we then examine whether respondents who speed more frequently also straightline in more grid questions. The results show that the tendency to speed is related to several respondent characteristics, particularly age (younger respondents are more likely to speed). This study also reveals that more speeding seems to be universally related to more straightlining, and this relationship is particularly strong among the less educated respondents.

title: Spacing, Position, and Order Interpretive Heuristics for Visual Features of Survey Questions
abstract: We present the results of six experiments that demonstrate the impact of visual features of survey questions on the responses they elicit, the response process they initiate, or both. All six experiments were embedded in Web surveys. Experiments 1 and 2 investigate the effects of the placement of nonsubstantive response options (for example, No opinion and Don't know answer options) in relation to the substantive options. The results suggest that when these options are not differentiated visually (by a line or a space) from the substantive options, respondents may be misled about the midpoint of the scale; respondents seemed to use the visual rather than the conceptual midpoint of the scale as a reference point for responding. Experiment 3, which varied the spacing of the substantive options, showed a similar result. Responses were pushed in the direction of the visual midpoint when it fell to one side of the conceptual midpoint of the response scale. Experiment 4 examined the effects of varying whether the response options, which were arrayed vertically, followed a logical progression from top to bottom. Respondents answered more quickly when the options followed a logical order. Experiment 5 examined the effects of the placement of an unfamiliar item among a series of similar items. For example, one set of items asked respondents to say whether several makes and models of cars were expensive or not. The answers for the unfamiliar items depended on the items that were nearby on the list. Our last experiment varied whether a battery of related items was administered on a single screen, across two screens, or with each item on its own screen. The intercorrelations among the items were highest when they were all on the same screen. Respondents seem to apply interpretive heuristics in assigning meaning to visual cues in questionnaires. They see the visual midpoint of a scale as representing the typical or middle response; they expect options to be arrayed in a pro

title: Item-by-item Versus Matrix Questions
abstract: While the choice of matrix versus item-by-item questions has received considerable attention in the literature, it is still unclear in what situation one is better than the other. Building upon the previous findings, this study expands this line of research by examining whether the difference between the two question types is moderated by the number of response options. Through a web survey experiment, this study compares matrix and item-by-item questions with 2, 3, 4, 5, 7, 9, and 11 response options. Additionally, we also investigate the impact of the device used to complete the survey on data quality. The results show that straight lining and response time are similar between the two question types across all response lengths, but item nonresponse tends to be higher for matrix than item-by-item question, especially among mobile respondents. Also measurement models reveal measurement equivalence between the two question types when there are fewer than seven response options. For matrices with 9 or 11 response options, analyses reveal substantial differences compared to item-by-item questions.

title: The Psychology of Survey Response
abstract: 1.  Respondents' understanding of survey questions 3. The role of memory in survey responding 4. Answering questions about date and durations 5.  Factual judgments and numerical estimates 7. Attitude judgments and context effects 8. Mapping and formatting 9. Survey reporting of sensitive topics 10. Mode of data collection 11. Impact of the application of cognitive models to survey measurement.

title: A TEST OF THE FORM-RESISTANT CORRELATION HYPOTHESIS RATINGS, RANKINGS, AND THE MEASUREMENT OF VALUES
abstract: A number of researchers have argued that ranking techniques are more appropriate than rating methods for the mea- surement of values in surveys. The form-resistant correlation hy- pothesis proposes that observed associations among values and between values and other variables should remain invariant across measurement methods. However, some recent research on parental values for child qualities suggests that ratings and rankings produce different correlational results. The present study tests the hypothesis that discrepancies between rating and ranking results are due to the fact that, when responding to rating questions, some respondents avoid making difficult choices be- tween valued qualities by rating all the qualities as highly and equally desirable. Consistent with this hypothesis, when nondif- ferentiating respondents are removed from the analyzed sample, the substantive results of analyses of rating data resemble the results typically obtained using ranking data. This suggests that ranking may be the superior method for measuring values.

title: Satisficing in surveys: Initial evidence
abstract: A new theoretical perspective proposes that various survey response patterns occur partly because respondents shortcut the cognitive processes necessary for generating optimal answers and that these shortcuts are directed by cues in the questions.

title: What are the Links in a Web Survey Among Response Time, Quality, and Auto-Evaluation of the Efforts Done?
abstract: Evaluating the quality of the data is a key preoccupation for researchers to be confident in their results. When web surveys are used, it seems even more crucial since the researchers have less control on the data collection process. However, they also have the possibility to collect some paradata that may help evaluating the quality. Using this paradata, it was noticed that some respondents of web panels are spending much less time than expected to complete the surveys. This creates worries about the quality of the data obtained. Nevertheless, not much is known about the link between response times (RTs) and quality. Therefore, the goal of this study is to look at the link between the RTs of respondents in an online survey and other more usual indicators of quality used in the literature: properly following an instructional manipulation check, coherence and precision of answers, absence of straight-lining, and so on. Besides, we are also interested in the link of RT and the quality indicators with respondents’ auto-evaluation of the efforts they did to answer the survey. Using a structural equation modeling approach that allows separating the structural and the measurement models and controlling for potential spurious effects, we find a significant relationship between RT and quality in the three countries studied. We also find a significant, but lower, relationship between RT and auto-evaluation. However, we did not find a significant link between auto-evaluation and quality.

title: Satisficing Among Reluctant Respondents in a Cross-National Context
abstract: Higher response rates are commonly assumed to be associated with higher survey quality. Nevertheless, increasing response rates without attention to response quality may result in an increase of measurement error if it is associated with reluctance to participate in surveys. This article exam- ines the nature of the relationship between reluctance and response quality. The question is studied in a cross-national context using data from the Eu- ropean Social Survey, Round 3, conducted in 2006-2007. Hypotheses are tested separately and compared across four countries: Belgium, the Nether- lands, Norway, and Sweden. A measure of response quality, satisficing, is developed through latent class analysis using the indicators of ''don't know,'' straight-lining, inconsistent, and ''extreme þ middle'' responses. To explore the relationship between the latent variables of reluctance and satisficing, before and after controlling for cognitive ability, structural equation modeling is used. The findings suggest that the relationship be- tween reluctance and response quality is present but spurious, being com- pletely explained by cognitive ability. The observed effects are not only consistent but also invariant in their magnitudes across the countries. The implications of this study for future data collection are discussed.

title: Straightlining in Web survey panels over time
abstract: Straightlining, an indicator of satisficing, refers to giving the same answer in a series of questions arranged on a grid. We investigated whether straightlining changes with respondents’ panel experience in two open-access Internet panels in the Netherlands: the LISS and Dutch Immigrant panels. Specifically, we considered straightlining on 10 grid questions in LISS core modules (7 waves) and on a grid of evaluation questions in both the LISS panel (150+ waves) and the Dutch immigrant panel (50+ waves). For both core modules and evaluation questions we found that straightlining increases with respondents’ panel experience for at least three years. Straightlining is also associated with younger age and non-western 1st generation immigrants. Where straightlining was a plausible set of answers, prevalence of straightlining was much larger (15-40%) than where straightlining was implausible (

title: The Effects of Questionnaire Completion Using Mobile Devices on Data Quality. Evidence from a Probability-based General Population Panel
abstract: The use of mobile devices such as smartphones and tablets for survey completion is growing rapidly, raising concerns regarding data quality in general, and nonresponse and measurement error in particular. We use the data from six online waves of the GESIS Panel, a probability-based mixed-mode panel representative of the German population to study whether the responses provided using tablets or smartphones differ on indicators of measurement and nonresponse errors from responses provided via personal computers or laptops. We follow an approach chosen by Lugtig and Toepoel (2015), using the following indicators of nonresponse error: item nonresponse, providing an answer to an open question; and the following indicators of measurement error: straightlining, number of characters in open questions, choice of left-aligned options in horizontal scales, and survey duration. Moreover, we extend the scope of past research by exploring whether data quality is a function of device-type or respondent-type characteristics using multilevel models. Overall, we find that responding with mobile devices is associated with a higher likelihood of measurement discrepancies compared to PC/laptop survey completion. For smartphone survey completion, the indicators of measurement and nonresponse error tend to be higher than for tablet completion. We find that most indicators of nonresponse and measurement error used in our analysis cannot be attributed to the respondent characteristics but are rather effects of mobile devices.

title: Web Survey Design Paging versus Scrolling
abstract: A key choice in the design of Web surveys is whether to place the survey questions in a multitude of short pages or in long scrollable pages. There are advantages and disadvantages of each approach, but little empirical evidence to guide the choice. In 2003 we conducted a survey of over 21,000 undergraduate students. Ten per- cent of the 10,000 respondents were directed to the scrollable version of the survey, containing a single form for each of the major sections. The balance was assigned to the paging version, in which questions were presented to be visible without scrolling. The instrument con- tained a maximum of 268 possible questions, including topics that varied in sensitivity and desirability. The survey also permitted comparison of the effect of skip patterns by implementing skip instructions and hyperlinks in the scrollable design, and also recorded time at the end of each of the five topical sections. Differences between designs are evaluated in terms of various forms of non- response, univariate and bivariate measurement properties, and proxies for respondent burden.

title: Design of Web Questionnaires: The Effects of the Number of Items per Screen
abstract: This article analyzes the effects of an experimental manipulation of the number of items per screen in a Web survey with forty questions aimed at measuring arousal. The authors consider effects on survey answers, item nonresponse, interview length, and the respondents' evaluation of several aspects of the survey (such as layout). Four different formats are used, with one, four, ten, and forty items and headers on a screen. The authors find no effect of format on the arousal index, but nonresponse increases with the number of items appearing on a single screen. Having multiple items on a screen shortens the duration of the interview but negatively influences the respondent's evaluation of the questionnaire layout. Grouping effects are generally similar for different demographic groups, though there are some differences in magnitude and significance level.

title: Mobile Web Survey Design: Scrolling versus Paging, SMS versus E-mail Invitations
abstract: There is some evidence that questionnaire design (scrolling or paging) and invitation mode (SMS or e-mail) have an impact on response rates in web surveys completed on personal computers (PCs). This paper examines whether these findings can be generalized to mobile web surveys. First, we explore the effect of scrolling versus paging design on the breakoff rate, item nonresponse, and completion time in mobile web surveys. Second, we investigate which type of invitation and reminder mode (SMS or e-mail) is more effective in terms of producing higher participation rates and maximizing the percentage of respondents who complete the survey via a mobile device rather than a PC. The paper summarizes the results of an experiment conducted among members of a volunteer online access panel in Russia, who were asked to complete the survey using a mobile device. We find that the scrolling design leads to significantly faster completion times, lower (though not significantly lower) breakoff rates, fewer technical problems, and higher subjective ratings of the questionnaire. We also find that SMS invitations are more effective than e-mail invitations in mobile web surveys.

title: Exploring the Effects of Removing “Too Fast” Responses and Respondents from Web Surveys
abstract: This paper addresses speeding, that is, “too fast” responses, in web surveys. Relying on the response process model, we argue that very short response times indicate low data quality, stemming from a lack of attention on the part of respondents. To identify speeding, prior research employed case-wise procedures. Using data from nine online surveys, we demonstrate that response behavior of individual respondents varies considerably during a survey. Thus, we use case- and page-wise procedures to capture speeding behavior that taps different, although related, phenomena. Moreover, page-specific speeding measures capture aspects of data quality that traditional quality measures do not cover. Employing both page-specific and case-wise speeding measures, we examine whether removing speeders makes a difference in substantive findings. The evidence indicates that removing “too fast” responses does not alter marginal distributions, irrespective of which speeder-correction technique is employed. Moreover, explanatory models yield, by and large, negligible coefficient differences (on average about one standard error). Only in exceptional cases differences exceed two standard errors. Our findings suggest that speeding primarily adds some random noise to the data and attenuate correlations, if it makes a difference at all. The paper concludes by discussing implications and limitations.

title: Data quality in telephone surveys and the effect of questionnaire length: a cross- national experiment
abstract: Respondents in long telephone survey interviews may adopt satisficing strategies as they approach the end of the questionnaire (Holbrook, Green and Krosnick, 2003). However, there is inconsistency regarding the relationship between questionnaire length and different forms of satisficing. We investigate whether long questionnaires are associated with a reduction in response quality using data from a cross-national survey experiment. Sample members were randomly assigned to interviews of 60, 45 or 30 minutes. We compare responses to attitudinal measures from a module on happiness and well-being, which was asked at different points in the interview in each of the three groups.

title: Computing Response Metrics for Online Panels
abstract: As more researchers use online panels for studies, the need for standardized rates to evaluate these studies becomes paramount. There are currently many different ways and conflicting terminology used to compute various metrics for online panels. This paper discusses the sparse literature on how to compute response, refusal, and other rates and proposes a set of formulas and a standardized terminology that can be used to calculate and interpret these metrics for online panel studies. A description of and distinction between probability-based and volunteer opt-in panels is made since not all metrics apply to both types. A review of the existing discussion and recommendations, mostly from international organizations, is presented for background and context. In order to propose response and other metrics, the different stages involved in building an online panel are delineated. Metrics associated with these stages contribute to cumulative response rate formulas that can be used to evaluate studies using online probability-based panels. Only completion rates can be calculated with opt-in panels.) We conclude with a discussion of the meaning of the different metrics proposed and what we think should be reported for which type of panel.

title: A test of the hypothesis that cronbach's alpha or kuder-richardson coefficent twenty is the same for two tests
abstract: An approximate statistical test is derived for the hypothesis that the reliability coefficients (Cronbach's α) associated with two measurement procedures are equal. Control of Type I error is investigated by comparing empirical sampling distributions of the test statistic with the theoretical model derived for it. The effect of platykurtosis in the test-score distribution on the test statistic is also considered.

title: Testing the Equality of Two Alpha Coefficients
abstract: This paper reviews and illustrates statistical tests of the equality of two values of Cronbach's alpha reliability coefficient. Tests for independent and dependent coefficients are considered. Possible pitfalls in determining the critical value for these tests or in computing the p value associated with the tests' outcome are discussed.

title: Response Order Effects in Dichotomous Categorical Questions Presented Orally The Impact of Question and Respondent Attributes
abstract: Using data from 548 experiments in telephone surveys conducted by the Gallup Organization, we explored how attributes of questions and respondents moderate response order effects in dichotomous categorical questions. These effects were predominantly recency effects and occurred most in questions that were more difficult to comprehend (especially among respondents with the least education), with response choices that were more difficult to comprehend (because they were complete sentences instead of words or phrases and because they were not mutually exclusive), and that were asked after many prior questions. Recency effects were also more common in questions that explicitly or implicitly encouraged respondents to wait until they had heard all the answer choices before formulating a judgment than in questions that induced respondents to begin formulating a judgment before all the answer choices had been read (especially among the least educated respondents). A study of interviewer behavior revealed patterns of pausing between and within sentences that help to explain why some types of questions are especially prone to recency effects and others are not.

title: Response Heaping in Interviewer-Administered Surveys Is It Really a Form of Satisficing?
abstract: Response heaping (also referred to as rounding or digit preference) occurs when respondents show a preference for rounded numbers (often those divisible by five or 10). Conventional wisdom is that this is the result of taking cognitive shortcuts to make question answering easier, and as such, that it may be a form of survey satisficing. In four studies, we test this conventional wisdom for the first time by exploring whether response heaping occurs for five types of survey questions (behavioral frequency questions, questions that ask about an individual�s personal characteristics, questions that ask about an individual�s age at the time of an event, questions that ask the respondent to report a percentage, and feeling-thermometer attitude reports) under the conditions thought to foster survey satisficing (e.g., among respondents lower in ability and motivation, when the task of question-answering is difficult, and later in a long questionnaire) and whether heaped responses show effects of survey satisficing (e.g., shorter response latencies, less accuracy, and lowered predictive validity). We also examine the prevalence of response heaping and the extent to which heaping is associated across questions. Heaping above chance levels was found for most types of questions (although the prevalence of heaping varied systematically across different types of questions), but we found little evidence that heaping for most types of questions is more common under conditions thought to foster satisficing. In fact, heaping for some questions may actually reflect more thoughtful processes and result in higher data quality.

title: Metrics and Design Tool for Building and Evaluating Probability-Based Online Panels
abstract: Probability-based online panels are beginning to replace traditional survey modes for existing established surveys in Europe and the United States. In light of this, current standards for panel response rate calculations are herein reviewed. To populate these panels cost-effectively, more diverse recruitment methods, such as, mail, telephone, and recruitment modules added to existing surveys are being used, either alone or in combinations. This results in panel member cohorts from different modes complicating panel response rate calculations. Also, as a panel ages with inevitable attrition, multiple cohorts result from panel refreshment and growth strategies. Formulas are presented to illustrate how to handle multiple cohorts for panel metrics. Additionally, drawing on relevant metrics used for a panel response rate, we further demonstrate a computational tool to assist planners in building a probability-based panel. This provides a means to estimate the recruitment effort required to build a panel of a predetermined size.

title: Effects of Questionnaire Length on Participation and Indicators of Response Quality in a Web Survey
abstract: This paper investigates how expected and actual questionnaire length affects cooperation rates and a variety of indicators of data quality in web surveys. We hypothesized that the expected length of a web-based questionnaire is negatively related to the initial willingness to participate. Moreover, the serial position of questions was predicted to influence four indicators of data quality. We hypothesized that questions asked later in a web-based questionnaire will, compared to those asked earlier, be associated with (a) shorter response times, (b) higher item-nonresponse rates, (c) shorter answers to open-ended questions, and (d) less variability to items arranged in grids. To test these assumptions, we manipulated the stated length (10, 20, and 30 minutes) and the position of questions in an online questionnaire consisting of randomly ordered blocks of thematically related questions. As expected, the longer the stated length, the fewer respondents started and completed the questionnaire. In addition, answers to questions positioned later in the questionnaire were faster, shorter, and more uniform than answers to questions positioned near the beginning.