Testing the Consistency of Preferences in Discrete Choice Experiments: An Eye Tracking Study

Although past research has tried to refine stated preferences elicited under discrete choice experiments (DCEs) by using methods to mitigate the hypothetical bias problem, little has been done to explore the consistency of preferences in repeated choice experiments. A key assumption of most DCEs is that individual’s preferences are stable across choice sets and remain unchanged throughout the experiment. However, it is possible that even little details in the experimental design, like changing the position of the alternatives within the same choice set, can hold a significant effect on choices. Using a within-subject experiment with eye tracking, we assess the influence of the position of the alternatives on the consistency of individuals’ choices. Specifically, we recruited 101 nonstudent participants from the general population in the Southern United States to participate in a 30-minute session. Participants received a $10 participation fee. Subjects ranged in age from 19 to 69, with an average of 28 years old and average income of $45,000. The experiment consisted of an ABA design that included three conditions and two “distraction tasks” between each treatment (Figure 1). The first condition was the baseline control, which entailed a standard DCE containing 12 hypothetical choice sets for vegetable products. In each choice set, subjects were asked to choose between three vegetable products and a “no-purchase” option. Each alternative was presented in four possible positions on the computer screen: 1) upper-left corner, 2) upper-right corner, 3) lower-left corner, and 4) lower-right corner. In the second condition, the “ position change treatment” , the same DCE was implemented; however, the position of the alternatives was randomized for each choice set. The third condition, referred to as the baseline treatment, replicated the original 12 choice sets in the baseline control (with the same positions for each alternative). In order to avoid subjects trying to memorize their choices in the baseline control and deliberately trying to match them in the baseline treatment, the order of the choice sets was randomized. Furthermore, two “distraction tasks” were included between the conditions in order to measure choice preferences after the subject’s attention was diverted (by manipulating the focus of attention). The first distraction task was a short socio-demographic survey presented between the baseline control and the position change treatment. The second distraction task was a cognitive function test commonly used to measure fluid intelligence. Figure 1 . Experimental Procedure. In this application, an orthogonal D-efficient fractional factorial design with no priors was generated using NGENE 1.1.2. Five artichoke vegetable attributes with three levels each were used: 1) size ( small, medium, large ), 2) color ( green, purple, mixed ), 3) production method ( conventional, organic, pesticide-free ), 4) presentation form ( fresh, canned, glass ), and 5) price ($1/unit, $2/unit, $3/unit ). In order to ensure that the subject was familiar with the attributes, a review of the definitions of each product attribute and attribute levels was presented prior to the baseline control condition. Theoretically the order of the choice sets and the position of the alternatives should not alter the subject’s preferences; however, we report evidence that both the position in which the alternatives are presented and the choice sets’ order influence which attributes the participants pay more attention to and ultimately, their choices. Overall, subjects searched for their preferred alternative in a “Z” motion, going from left to right and top to bottom, and the time they spent evaluating the different choice sets quickly decays as they progress through the experiment. Moreover, it appears that the alternatives located in the upper positions, especially the upper right position, received the highest attention in terms of how often subjects see those alternatives and how long they spend looking at them. This result ties into the relationship between the position of the alternatives and the frequency of choices. That is, subjects tend to choose the products located at the upper positions more often, with a higher inclination towards the alternative in the upper right position. The eye tracking results are further explored by estimating willingness-to-pay (WTP) space parameters. The distributions of WTP estimates differed significantly for nearly all attributes after changing the order of the choice sets and the position of the alternatives. Recall that the alternatives and choice sets are identical. However, after changing the position of the alternatives in the choice set, subjects selected the same alternative only 69% of the time. Furthermore, after reverting back to the original positions (baseline treatment), subjects consistently selected the same alternative only 67% of the time. The latter consistency level is a bit concerning considering that subjects were facing identical choice sets to those in the baseline control, only in different order. The results presented here highlight the importance of the position of the alternatives in the experimental design. The knowledge that even slight changes in the experimental design could significantly affect individuals stated preferences warrants more attention to detail when designing DCEs to elicit individuals’ valuations. That being said, the position of the alternatives, and the order of the choice sets should be taken as part of the experimental design in order to obtain more stable preference parameter estimates.