# An Empirical Assessment of Stimulus Presentation Mode Bias in Conjoint Analysis

Debi Mishra, Junhong Min, M. Deniz Dalman

<figure><img src="image_1.png" /></figure>

## To cite this version:

Debi Mishra, Junhong Min, M. Deniz Dalman. An Empirical Assessment of Stimulus Presentation Mode Bias in Conjoint Analysis. Innovative Marketing, 2012, 7 (4), 108-121 p. hal-04325786

HAL Id: hal-04325786

https://hal.science/hal-04325786v1

Submitted on 13 Dec 2023

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
"An empirical assessment of stimulus presentation mode bias in conjoint analysis"

**AUTHORS**
Debi P. Mishra
Junhong Min
M. Deniz Dalman

**ARTICLE INFO**
Debi P. Mishra, Junhong Min and M. Deniz Dalman (2011). An empirical assessment of stimulus presentation mode bias in conjoint analysis. *Innovative Marketing*, 7(4)

**RELEASED ON**
Wednesday, 29 February 2012

**JOURNAL**
"Innovative Marketing"

**FOUNDER**
LLC "Consulting Publishing Company "Business Perspectives"

<figure><img src="image_3.png" /></figure>

NUMBER OF REFERENCES
0

<figure><img src="image_4.png" /></figure>

NUMBER OF FIGURES
0

<figure><img src="image_5.png" /></figure>

NUMBER OF TABLES
0

© The author(s) 2023. This publication is an open access article.
Debi P. Mishra (USA, Australia), Junhong Min (USA), M. Deniz Dalman (USA)

# An empirical assessment of stimulus presentation mode bias in conjoint analysis

## Abstract

Conjoint analysis, which aims to uncover the optimal combination of attributes influencing customer choice, is widely used by marketers to predict the success of new product and service introductions. In recent years, researchers have incorporated considerable mathematical sophistication into conjoint models and extended its domain to diverse areas such as pricing, market share, profitability, product positioning, distribution channels, and advertising. Despite these advances, the predictive power of conjoint applications is often compromised by response biases and measurement errors. The purpose of this research is to isolate and investigate the impact of one such bias that arises from the manner in which stimuli are presented to respondents. Based upon an appraisal of over four decades of conjoint studies in the major marketing journals, the authors make a case for the possible existence of two types of biases, i.e.: (1) stimulus joint presentation bias, when concept cards are shown simultaneously (side by side) to respondents, and (2) stimulus separate presentation bias, where cards are presented separately (one at a time). Two conjoint experiments were designed to investigate the effects of these biases on respondent choices. Results indicate that bias manifests itself in conjoint designs when there is a mismatch between presentation mode and respondents' cognitive (evaluable) burden. Left unaddressed, stimulus presentation mode bias may: (1) have a deleterious effect on respondents' choice behavior; and (2) compromize the predictive accuracy of conjoint models. The authors discuss several approaches that can account for and mitigate the negative impact of presentation mode biases on conjoint outcomes.

**Keywords:** conjoint analysis, stimulus presentation mode bias, and evaluability theory.

## Introduction

Conjoint analysis is a widely popular and powerful analytical tool used by marketers to uncover optimal combinations of attributes influencing customer preference and choice. Till date, the most significant applications in marketing have been conducted in the area of new products, i.e., concept development, concept testing, and new product introduction. For example, using the conjoint methodology, marketers can know what features a proposed new product should possess, and how it should be priced.

In recent years, the basic analytical framework of conjoint analysis has been extended to investigate complex business problems involving other elements of the marketing mix such as pricing, advertising, and distribution. Furthermore, conjoint studies have also shed light on how firms might pursue optimal marketing strategy in areas like market segmentation, market share, profitability, and positioning (Green, Krieger, and Wind, 2001; Wittink and Cattin, 1989). Finally, to study diverse business problems, researchers have used a variety of increasingly sophisticated and mathematically intricate models such as choice based conjoint experiments (Louviere and Woodworth, 1983), hierarchical Bayesian approaches (Allenby and Ginter, 1995), latent class models (Ramaswamy and Cohen, 2000), and incentive-aligned mechanisms (Ding, 2007).

Although prior conjoint studies have enhanced our understanding of customer behavior and business phenomena in unique ways, the overriding focus of recent research has been on the development and application of sophisticated mathematical models and techniques. Absent from this research is a systematic consideration of customers' cognitive processes and biases that may influence choice decisions (Bradlow, Hu, and Ho, 2004; Ding, 2007; Nowlis and Simonson, 1997). This lack of attention to thought processes is rather surprising because a number of researchers have indicated that cognitive biases may have a deleterious effect on the predictive power and efficiency of conjoint models. For example, Sethuraman, Kerin, and Cron (2005) found that the mode of stimuli administration (web-based vs. mail-based) differentially affected respondents' mental attention to various choice scenarios. Likewise, Segal (1982), and Bradlow, Hu, and Ho (2004) have emphasized the impact of respondents' cognitive effort and information processing styles on conjoint studies. In sum, cognitive biases, which are an inherent and inseparable aspect of conjoint studies, have been somewhat understudied in the extant literature.

In this research, we address the gap in our understanding of cognition processes by investigating how one particular type of response bias introduced via the differential effect of stimulus presentation modes affects respondents' outcomes. In particular, we examine how response bias affects the outcome of conjoint studies when attributes used to
create conjoint stimuli and stimuli presentation modes are mismatched. The presentation mode involves: (1) *stimulus separate presentation* (conjoint stimulus is presented individually, one at a time); and (2) *stimulus joint presentation* (conjoint stimulus is presented simultaneously, side by side). It may be noted that prior research in conjoint evaluability suggests that consumers who find it difficult to evaluate attribute combinations in separate presentation, might find it easy to evaluate the same attributes in joint presentation and vice versa (Hsee, Blount, Loewenstein, and Bazerman, 1999; Nowlis and Simonson, 1997; Schmeltzer, Caverni, and Warglien, 2004; Willemsen and Keren, 2004). However, a systematic consideration of the impact of stimulus presentation bias on conjoint outcomes is not forthcoming in the literature.

The basic purpose of this research is to empirically examine the impact of stimulus presentation mode bias on the outcome of conjoint studies. Since bias is susceptible to the type of product/service features used in stimuli, we also consider the relative importance of comparable (e.g., price) and enriched (e.g., brand and quality) attributes in influencing conjoint outcomes.

This paper is organized in the following manner. First, we explain stimulus presentation mode bias by reviewing the evaluability literature. Second, to gain a deeper understanding of presentation bias, i.e., its nature, boundary conditions, and effects, we conduct a qualitative reappraisal of prior studies that have utilized trade-off, full-profile, and choice based conjoint models. Third, to empirically investigate the impact of presentation mode bias, we conduct two conjoint experiments, each one in the product (toaster) and service (hotel) categories respectively. Finally, we discuss our findings, specify implications of our study, and outline the scope for future research.

# 1. Literature review

## 1.1. Stimulus presentation mode bias.
While evaluating products and services, customers typically make more accurate preference appraisals when the focal product is evaluated in the context of comparative benchmarks or a *joint* set of stimuli. For example, while assessing the quality of sound emitted by a speaker, most non-experts may find the task easier when speakers are evaluated side by side with other speakers, as opposed to a *separate* evaluation method.

Likewise, when consumers evaluate a particular high definition television (HDTV) model, its dramatically superior quality may not be readily apparent to them. However, when consumers compare this television to a traditional model, they are more likely to appreciate the relatively superior attributes of HDTV like vivid colors, contrast, sound, and overall picture quality. In other words, in a conjoint study, the mode of stimuli presentation (separate or joint) might introduce a systematic bias in the quality of respondents' evaluations.

The concept of presentation bias has been directly addressed in the evaluability literature (Hsee, Blount, Loewenstein, and Bazerman, 1999; Schmeltzer, Caverni, and Warglien, 2004; Yeung and Soman, 2005). Specifically, researchers have argued that the mode of presentation (separate or joint) is the main antecedent of consumer's preference reversal because some attributes which are used to describe a product may be cognitively difficult to appraise separately but far easier to evaluate when presented jointly. In this research, we define stimulus presentation mode bias as a systematic response bias introduced by mismatched conditions between stimuli and the mode of presentation, i.e., stimulus separate presentation (SSP) and stimulus joint presentation (SJP).

Examples of stimulus separate presentation and stimulus joint presentation are shown in Appendix A and Appendix B respectively. As depicted in Appendix A, respondents in stimulus separate presentation mode were asked to evaluate 16 different hotels across a range of attributes. Notice that respondents evaluated these concepts separately from one another. In contrast, respondents in the stimulus joint presentation mode were instructed to provide a rank ordering of the same 16 concepts considered jointly (see Appendix B). Our central premise is that predictive accuracy of the conjoint experiment will be higher given the matched condition between stimuli and presentation modes.

## 1.2. Impact of stimulus presentation mode bias.

To understand when stimulus presentation considerations become salient and engender bias, we present and briefly describe the main steps in a typical conjoint study. While there is no consensus among researchers regarding the ideal conjoint procedure, most users incorporate the five-step procedure depicted in Figure 1 and described below.
<figure><img src="image_6.png" /><figcaption>Fig. 1. Flowchart of conjoint analysis procedure</figcaption></figure>

Marketers use conjoint analysis while dealing with new product concept development, price sensitivity, repositioning, and competitive analysis (Green and Srinivasan, 1990). In these studies, the *first* step often involves problem formulation, where researchers make a relative assessment of the suitability of conjoint analysis in relation to competing approaches such as qualitative studies, survey methodologies, and multivariate techniques. In the *second* step, four significant design decisions are usually made, i.e.: (1) model choice (e.g., choice-based conjoint, full profile conjoint, or hybrid conjoint); (2) selection of attributes, levels, and stimulus format such as picture, text, or multimedia; (3) consideration of appropriate response metrics such as choice, rating, or ranking scales; and (4) selection of the number of concept cards in conjunction with

a particular approach, e.g., full factorial, fractional, or orthogonal design.

Presentation mode bias becomes germane during the *third* or data collection step of conjoint analysis. At this stage, researchers focus on stimulus presentation method (trade-off matrix, profile, or pair wise comparison), data collection format (paper-pencil, online, or computer assisted), and survey administration issues (personal interview, self-administrated mail survey, phone survey, or online survey). Notice that stimulus presentation *method* is distinct from the *mode of presentation*. Specifically, method issues involve decisions regarding the number and type of concept cards, whereas presentation mode indicates whether subjects evaluate concept cards separately or jointly (in tandem). In the *fourth* or data analysis step, important decisions
pertaining to estimation (e.g., MANOVA, Logit, Probit, or OLS) and validation (e.g., holdout sample validation, R-square evaluation, or hit-ratio test) methods are made. Finally, while reporting results, conjoint researchers have to adopt the most effective manner of communicating results with relevant audiences. At a minimum, a conjoint report should be diagnostic, describe the relative importance of attributes, provide sensitivity analyses, and discuss appropriate simulations.

## 2. Study 1: appraisal of presentation mode biases in published studies

Although the concept of presentation mode bias is straightforward, there is a paucity of systematic knowledge about its pervasiveness, boundary conditions, and effects. As such, very little practical guidance is available to marketers for conceptualizing presentation bias and mitigating its deleterious effects on the predictive accuracy of conjoint models.

To better understand presentation mode bias, we conducted a thorough examination of past conjoint studies in marketing. This exercise provided us with sufficient background, inputs and guidance for conducting a set of experiments to empirically assess presentation bias. Our summary finding from the directed review is that different conjoint models use different presentation modes, creating the potential for biased outcomes.

We identified conjoint articles that appeared in the major marketing journals (Journal of Marketing, Journal of Marketing Research, Journal of Consumer Research, and Marketing Science), over a four decade period (1970-2010). To keep our analysis tractable, we selected studies that have used one or more of the most popular approaches, i.e., trade-off conjoint, full-profile conjoint, and choice-based conjoint. Our specific focus was on the methodology and the appendix section of each article where stimuli and data collection procedures are usually discussed.

To identify the mode of stimulus presentation in each study, we focused on the description of stimuli and data collection procedures. As an example, a statement such as "full profile descriptions of houses one at a time were used" (Oppewal and Klabbers, 2003, p. 303) was judged to be of the separate presentation mode type. In contrast, stimulus joint presentation was gauged from statements such as "each set was then printed on a deck of 3 x 5 cards" (Srinivasan, Flachbart, Dajani, and Hartley, 1981, p. 162), or "subjects were asked to rank the order of the blender models" (Heeler, Okechuku, and Reid, 1979, p. 61). In sum, as discussed

below (1) stimulus joint presentation mode is associated with trade-off and choice based models, while (2) stimulus separate and joint modes are typical of full-profile models.

Our a priori expectation is that the mode of stimulus joint presentation may be appropriate for trade-off conjoint models where respondents compare pairs of attributes (e.g., price and brand, size and weight) until rank ordered preference is completed (Johnson, 1974). This type of presentation is best illustrated by the studies of Johnson (1974) and Segal (1982). In particular, in the Johnson (1974) study, each respondent received a booklet depicting all pairs of attributes and associated trade-off matrices (Johnson, 1974, p. 122). Next, subjects were asked to examine all pairs of attributes together (or side by side) and provide rank ordered preferences.

Like trade-off models, choice-based conjoint analysis also involves joint presentation of stimuli. Here, respondents receive a choice task containing several profiles. Next, they are instructed to examine all concept cards in a choice task set and choose only one card within each set. Respondents repeat this procedure until they exhaust the last choice set. Notice that respondents end up selecting one concept only after simultaneously comparing all concepts within a choice set. This procedure is best illustrated by Louviere and Woodworth (1983, p. 353), and Toubia, Hauser, and Simester (2004, p. 117).

In contrast to trade-off and choice-based models, in full-profile designs, some studies use separate presentation, while others use the individual mode (Green and DeSarbo, 1979; Jain, Acito, Malhotra, and Mahajan, 1979). In this method, respondents usually receive a booklet containing general survey questions (e.g., demographic variables or usage and awareness questions) and concept cards (Green and DeSarbo, 1979, p. 87) followed by a conjoint stimulus. Upon evaluation, respondents provide preference ratings of concepts. This practice of presenting concepts individually represents stimulus separate presentation. Note that the separate presentation mode is a natural fit for the self-explication conjoint method where users are instructed to rate attributes and levels of a concept individually with the goal of determining the most important levels for different attributes (Green, Goldberg, and Montemayor, 1981).

In addition to separate presentation, some full-profile studies utilize the joint mode. As an example, consider the presentation mode employed by Ostrom and Iacobucci (1995, p. 27). In this study, respondents first received a booklet containing general survey questions and concept cards. Next, they
were instructed to sort all concepts into ordered categories (i.e., excellent, good, fair, bad, etc.) and rank order concept cards within each category until all category rankings were exhausted.

The results of our appraisal are summarized in Table 1. While we cannot quantitatively estimate the magnitude of presentation mode bias, it is possible to make informed judgments about its source and effects. To begin with, by examining Table 1, we can qualitatively assess the degree of fit between respondents' cognitive burden and the mode of presentation employed. Greater the degree of natural fit, or match between cognitive burden and presentation mode, less is the possibility of bias.

First, consider the self-explication and joint designs and how respondents' natural cognitive thought processes may influence the choice task. Recall that the objective of the self-explication approach is for respondents to choose the most preferred level within each attribute. Since no joint concept comparisons are needed, subjects' cognitive burden is simplified when concepts are evaluated separately. In other words, if a self-explication study were to use a joint presentation mode, respondents may be distracted and their cognitive burden would increase. In these situations, joint stimuli would engender bias by diluting fit and creating a mismatch between respondents' cognitive burden and the mode of presentation.

In contrast to the self-explication mode, the trade-off approach requires respondents to consider a set of concepts simultaneously. Hence, the most appropriate cognitive outcomes are expected to result from a joint presence of stimuli. On the other hand, if a separate mode is used, respondents' cognitive burden may increase as they have to access information from their memory banks to construct preference outcomes. In summary, mismatched condi-

tions can create biased outcomes and lower the predictive accuracy of conjoint models.

Presentation modes (separate or joint) appear to be naturally matched to respondents' cognitive burden in the self-explication and trade-off methods. However, the potential for bias arises in hybrid and full profile models where some studies have used joint modes while others have used separate approach. Since respondents align their cognitive apparatus with an ideal mode of stimulus presentation, it is difficult to imagine that they will be indifferent to the mode of presentation. As it was noted earlier, in theory, a sound speaker can be evaluated by respondents either separately or jointly in conjunction with other speakers. In both cases, preference data can be obtained and analyzed further. However, for respondents only one presentation mode is cognitively optimal (ideal). Given the use of both types of presentation in the hybrid and full-profile approaches, we suspect that results of some studies in this area may be biased. Specifically, differential presentations are expected to create mismatches with respondents' cognitive burden and lead to potentially biased outcomes.

Biased outcomes can be inferred from the degree to which respondents reverse their original preference in response to changes in the presentation mode. In academic parlance, this phenomenon is formally known as *preference reversal* (Nowlis and Simonson, 1997).

While we have reasons to suspect that presentation mode biases might systematically influence the outcome of conjoint experiments, it is impossible to empirically estimate the effect of this bias on conjoint outcomes by merely inspecting conjoint studies. Hence, two conjoint experiments were conducted to empirically investigate the presence and effect of presentation mode bias.

<table><caption>Table 1. Different stimulus presentation modes and biases</caption><thead><tr><th>Conjoint model</th><th>Stimulus presentation mode</th><th>Match between cognitive burden and choice task</th><th>Representative literature</th></tr></thead><tbody><tr><td>Self explication conjoint</td><td>Separate</td><td>Yes</td><td>Srinivasan (1988), Srinivasan and Park (1997)</td></tr><tr><td>Trade-off conjoint</td><td>Joint</td><td>Yes</td><td>Johnson (1974), Segal (1982)</td></tr><tr><td rowspan="2">Full profile conjoint</td><td>Separate</td><td>?<sup>a</sup></td><td>Green and DeSarbo (1979), Oppewal and Klabbers (2003)</td></tr><tr><td>Joint</td><td>?</td><td>Jain, Acito, Malhotra, and Mahajan (1979), Ostrom and Lacobucci (1995)</td></tr><tr><td>Hybrid conjoint / ACA<sup>b</sup></td><td>Hybrid: separate and joint</td><td>?</td><td>Green (1984), Mehta, Moore, and Pavia (1992)</td></tr><tr><td>Choice-based conjoint</td><td>Joint</td><td>Yes</td><td>Louviere, Hensher, and Swait (2000), Toubia, Hauser, and Simester (2004)</td></tr></tbody></table>

Notes: <sup>a</sup>Denotes possible presentation mode bias; <sup>b</sup>denotes adaptive conjoint analysis.

### 3. Study 2: conjoint experiment 1

The purpose of the first conjoint experiment was to investigate whether a change in presentation mode results in respondents' preference reversal for the price and brand attributes. The second conjoint ex-

periment involved a more rigorous design and investigated additional aspects pertaining to boundary conditions and the effects of presentation bias.

**3.1. Subjects.** Sixty-six business undergraduate students were randomly assigned to two conditions. In
the stimulus separate presentation condition, subjects (n = 32) received a booklet containing four toaster concept cards and evaluated them one at a time. In the stimulus joint presentation condition, subjects (n = 34) evaluated all four toaster concept cards simultaneously.

**3.2. Design.** Toasters were selected because student subjects are familiar with the product category. As shown in Table 2, each toaster had two levels each of price ($19.99 and $26.99) and brand (K-Mart and Black & Decker). The choice of levels and attributes was influenced by information most commonly displayed by major retailers (e.g., Best Buy) on their website and in print advertisements.

<table><caption>Table 2. Features and levels used in toaster concept card</caption><thead><tr><td>Feature</td><td>Level 1</td><td>Level 2</td></tr></thead><tbody><tr><td>Price</td><td>$19.99</td><td>$26.99</td></tr><tr><td>Brand</td><td>K-Mart</td><td>Black & Decker</td></tr></tbody></table>

Although obtaining relative preference data about price and brand was the main focus, additional attributes were used to describe the concept and make it more realistic and relevant for respondents. Realism, in turn, was expected to keep respondents mentally engaged with the evaluation and ranking task. A full-factorial (2 levels x 2 attributes) design resulted in four toaster concepts. An example of a toaster concept card and instructions is provided in Appendix C. After receiving the booklet, subjects in the first condition evaluated toasters *separately* (one at a time) and then immediately rated the likelihood of purchase for each toaster using a twenty point scale ("1" means very unlikely to buy, "20" refers to very likely to buy). For subjects in the second condition, they were first asked to examine all toasters *jointly* and rate the likelihood of purchase using the same scale.

**3.3. Relative importance measure.** The relative importance weights of price and brand were computed using the Sawtooth SMRT software (2008). First, following the guidelines of Jain, Acito, Malhotra, and Mahajan (1979), we obtained individual utilities as follows:

$$U(X) = \sum_{i=1}^{n} \sum_{j=1}^{m_i} a_{ij} x_{jj},$$

where $U(X)$ is the overall utility, $a_{ij}$ is the part-worth estimated by OLS with two levels ($j, j = 1, 2$) and two attributes ($i, i = 1, 2$). Next, we computed relative importance using the formula below.

$$w_i = \left\{ \left( \max_j - \min_j(a_{ij}) \right) \right\}, \text{ for each } i.$$

where $a_{ij}$ is the utility of $j^{th}$ level ($j, j = 1, 2$) of the $i^{th}$ attribute ($i, i = 1, 2$).

**3.4. Prediction.** We predict that respondents' relative importance ratings for price and brand are susceptible to bias when there is a mismatch between cognitive evaluation burden and stimulus presentation mode. Specifically, in evaluating an objective attribute such as price across multiple concepts, subjects don't have to recall information from memory when stimuli are presented jointly. In contrast, when stimuli are presented individually, respondents' cognitive burden increases because price data has to be accessed from memory. In contrast to price, while evaluating a subjective attribute like a brand, respondents might consider separate presentation to be cognitively more efficient even if they recall information from memory. In particular, a brand is an abstraction of a product, is conditioned by history, and may have symbolic and utilitarian meanings for customers. Typically, brands possess unique, holistic meanings and are best evaluated in isolation. If brand information is presented in conjunction with other brands, respondents may become cognitively distracted, leading to suboptimal evaluation of the focal brand.

In summary, respondents may find it easier (cognitively less burdensome) to compare price when concepts are presented jointly, rather than separately. Conversely, the brand attribute will be more easily evaluated when concepts are presented separately as opposed to jointly. These predications are generally congruent with the evaluability literature (Hsee, Blount, Loewensten, and Bazerman, 1999; Schmeltzer, Caverni, and Warglien, 2004; Yeung and Soman, 2005) which posits that respondents' ex-ante preference for a concept may be reversed by a new mode of stimuli presentation that causes an increase in cognitive burden.

**3.5. Results.** Our goal has been to empirically examine whether consumers' relative preference for attributes (price and brand) changes when presentation modes are altered. Consistent with our prediction, we find evidence of preference reversal. As depicted in Figure 2, the relative importance of price decreases from 65% under stimulus separate presentation to 46% when stimulus joint presentation is used. Likewise, the relative importance of brand increases from 35% under stimulus separate presentation, to 54% in the joint mode.

Our results provide general support for the existence of presentation bias and its effect on preference reversal. However, two potential shortcomings should be noted. First, the results are merely indicative of preference reversal since no test of statistical significance has been conducted. Second, a somewhat skeletal concept description involving price and brand attributes is at variance with real-life product evaluation decisions.
<figure><img src="image_7.png" /></figure>

Note: Y axis represents relative importance score (min = 0, max = 100).

<caption>Fig. 2. Relative importance of price and brand under stimulus separate presentation (SSP) and stimulus joint presentation (SJP)</caption>

Notice that the purpose of this research has been to investigate whether a change in presentation mode creates biased outcomes as measured by respondents' preference reversal. Whether relative importance ratings for price and brand would be higher when matched with appropriate presentation modes is a different research question and outside the scope of this study. For example, brand importance would depend on several factors such as category importance, usage context, presence of competitive brands, country of origin, etc. (Fischer, Volckner, and Sattler, 2010).

Given the above mentioned shortcomings of the present study and to further understand the nature and boundary conditions of presentation bias, we conducted a second conjoint experiment.

## 4. Study 3: conjoint experiment 2

The purpose of the second conjoint experiment was threefold. The first objective was to design a study that permitted estimation of statistical significance for the preference reversal effect. Second, our focus was on developing and testing more realistic conjoint concepts involving comparable (price) and enriched (quality) attributes. Finally, to aid generalizability, we incorporated a service concept (hotel), given the modern view in marketing that products are essentially service bundles (Vargo and Lusch, 2004).

**4.1. Subjects.** One hundred twenty business undergraduate students were randomly grouped into each of two conditions. The first group (n = 55) received a booklet containing sixteen hotel concept stimuli and

evaluated them in the separate presentation mode (see Appendix A). The second group (n = 65) received the same booklet and evaluated concepts presented via the joint presentation method (see Appendix B).

**4.2. Design.** Recall that the first conjoint experiment used toasters and only measured the relative importance of the price and brand attributes. Although the second conjoint experiment is similar in spirit, there are a number of differences. First, to aid generalizability, a service class, e.g., a hotel was used. Second, we controlled for bias in order of administration by distributing different versions of the questionnaire. Third, we incorporated additional control variables (perceived importance of scenario, ease of evaluation, consideration of attributes). Finally, we computed tests of statistical significance for preference reversal and gauged validity by computing the hit ratio of classification.

The hotel concept was used since it was expected to be familiar to undergraduate students. To sustain respondents' interest and create personal relevance, they were tasked with the responsibility of booking a hotel for an important client. Hotels were described in terms of two levels each of four attributes. As shown in Table 3, each concept contained two levels of price ($70.00 and $120), two levels of quality (reasonable quality, really good quality), two levels of staff courtesy (not too friendly, friendly), and two levels of service customization (standardized, customized). These combinations resulted in a total of 16 (2 x 2 x 2 x 2) different concepts.
<table><caption>Table 3. Attributes and levels used in hotel concept card</caption><thead><tr><th>Attributes</th><th>Level 1</th><th>Level 2</th><th>Attribute type</th></tr></thead><tbody><tr><td>Price</td><td>$70</td><td>$120</td><td>Comparable attribute</td></tr><tr><td>Quality</td><td>Reasonable quality</td><td>Really good quality</td><td>Enriched attribute</td></tr><tr><td>Staff courtesy</td><td>Not too friendly</td><td>Friendly</td><td>Enriched attribute</td></tr><tr><td>Service customization</td><td>Standardized</td><td>Customized</td><td>Enriched attribute</td></tr></tbody></table>

Note: Levels and attributes were chosen based upon Ostrom and Iacobucci (1995).

To control for bias, the sixteen different concepts were randomly ordered and administered to respondents using two distinct questionnaire versions and two different presentation modes (separate and joint). Under the separate presentation condition, subjects evaluated concepts one at a time and immediately rated the likelihood of booking using a seven point scale. In the joint condition, subjects were asked to first examine all hotels simultaneously and then provide preference ratings. After completing the choice task, subjects rated three control items (perceived importance of scenario, ease of hotel evaluation, consideration of all attributes during evaluation) using appropriate Likert scales.

**4.3. Relative importance measure.** Similar to the first experiment, we obtained individual utilities and relative importance of the four attributes using the Sawtooth SMRT software (2008). In addition, for conducting validation checks, respondents were asked to answer a ‘choice’ question. The preference data from this question was compared with choice data predicted by conjoint analysis. A good match between actual and predicted choice is indicated by a high value of the hit ratio (>= 0.8). In the present study, the high hit ratio of 0.82 (Table 4) suggests that there is congruency between the different choice measures and that our study is robust to validity concerns.

<table><caption>Table 4. Hit ratio statistics for conjoint analysis validation</caption><thead><tr><th rowspan="2"></th><th rowspan="2"></th><th colspan="2">Actual choice</th></tr><tr><th>Concept A (n = 13)</th><th>Concept B (n = 107)</th></tr></thead><tbody><tr><td rowspan="3">Predicted choice (conjoint analysis <sup>a</sup>)</td><td>Correct prediction</td><td>39.82<sup>b</sup> (5)<sup>c</sup></td><td>13.36 (14)</td></tr><tr><td>Incorrect prediction</td><td>60.18 (8)</td><td>86.65 (93)</td></tr><tr><td>Overall prediction success</td><td colspan="2">82 %</td></tr></tbody></table>

Notes: <sup>a</sup> Computed using randomized first choice rule. <sup>b</sup> Relative importance score (min = 0, max = 100). <sup>c</sup> Actual sample size.

**4.4. Measurement of presentation bias.** Recall from our previous discussion that presentation bias effects are likely to occur when there is a mismatch between attribute characteristics and the mode of stimulus presentation. As per Nowlis and Simonson (1997), respondents can evaluate enriched or subjective attributes (e.g., brand and quality) with more cognitive ease in the separate presentation mode, while comparable or objective attributes (e.g., price and size) can be easily gauged in separate presentation. On the other hand, when a mismatched condition exists (e.g., enriched attributes and joint presentation), respondents need to expend additional complex cognitive effort to complete the choice task. Such mismatches can lead to further bias because subjects may attempt to simplify complex tasks by employing heuristic decision-making strategies.

Mismatches may also suppress respondents’ de-biasing tendency and ability to make logical decisions (Chatterjee, Heath, and Min, 2009). For example, consider a choice task which requires customers to rank order concepts described by comparable attributes such as size or price. When these concepts are presented in separate mode, respon-

dents cannot easily compare across concepts since they have to recall information from memory. This additional cognitive challenge can become a source of bias and lead to sub-optimal choice outcomes. In contrast, when this presentation mode is changed to reflect a matched situation, we should expect presentation bias to be reversed because of a corresponding increase or decrease in cognitive burden. Hence, respondents’ ex-ante preference for a concept administered through a particular mode may be reversed when a new format of presentation is used. This change in preference or the preference reversal measure is used in the present study to gauge presentation bias.

**4.5. Results.** Table 5 provides evidence of preference reversal for price, while for quality, courtesy, and customization the effects are non-significant. With respect to price, as the mode of stimulus presentation changes from separate to joint, respondents reverse their preference. Under the separate presentation condition, 21.79% of respondents consider price to be important, while this proportion drops to 15.48% when the mode changes, suggesting a preference reversal effect. Furthermore, the associated change sta-
tistic is significant ($F = 5.683, df = 1, 118, p = .019$), which rejects the null hypothesis ($H_0$) of no preference reversal effects. Notice that we do not find any statistically significant preference reversal effects for the other variables included in the study. In particular, as shown in Table 5, no significant preference reversal effects are observed for quality ($F = .762, df = 1, 118, p = .385$) and the three control variables, i.e., (1) perceived importance of scenario ($F = .715, df = 1, 118, p = .400$), (2) ease of evaluation ($F = .558, df = 1, 118, p = .456$), and (3) consideration of all attributes ($F = .403, df = 1, 118, p = .527$).

Note that we could not establish statistical significance of the reversal effect for enriched attributes (quality and brand). However, our finding of preference reversal effect for a comparable attribute such as price is consistent with the results of the first experiment. In addition, we demonstrate that the reversal effect for price is statistically significant, and robust to validity and generalizability concerns. In summary, we have been able to provide empirical evidence regarding the existence of presentation mode bias for price. Note that price is one of the most widely used attributes in conjoint studies. Hence, marketers should proceed with caution and explicitly test for the existence of such bias before implementing decisions based upon the results of a conjoint study. In the concluding section, we discuss several approaches that marketers can use to tackle presentation bias.

<table><caption>Table 5. ANOVA for preference reversal effects</caption><thead><tr><th>Attributes</th><th>Overall sample<br>(n = 120)</th><th>Stimulus separate presentation group<br>(n = 55)</th><th>Stimulus joint presentation group<br>(n = 65)</th><th>df<br>(between, within)</th><th>F</th><th>Sig.</th></tr></thead><tbody><tr><td>Price</td><td>18.37<sup>a</sup></td><td>21.79</td><td>15.48</td><td>1, 118</td><td>5.683</td><td>.019**</td></tr><tr><td>Quality</td><td>23.23</td><td>22.10</td><td>24.20</td><td>1, 118</td><td>.762</td><td>.385</td></tr><tr><td>Courtesy staff</td><td>43.72</td><td>42.63</td><td>44.64</td><td>1, 118</td><td>.457</td><td>.500</td></tr><tr><td>Service customization</td><td>14.67</td><td>13.48</td><td>15.69</td><td>1, 118</td><td>1.434</td><td>.233</td></tr></tbody></table>

Note: <sup>a</sup> Relative importance score (min = 0, max = 100). ** p < 0.05.

## Conclusion, implications, limitations, and scope for future research

Conjoint studies remain widely popular in marketing, and all trends suggest that its central role in influencing new product decisions will continue well into the future (Ding, Park, and Bradlow; 2009; Green, Krieger, and Wind, 2001; Green and Srinivasan, 1990). Not surprisingly, both academics and practitioners have paid considerable attention in recent years to advancing our knowledge of the conjoint method. However, most current research been rather narrowly confined to technical issues involving computational and statistical considerations (Ding, 2007). Absent from these studies is a systematic focus on cognitive biases and errors that, in turn, can have a deleterious effect on conjoint outcomes. Given that conjoint experiments often provide significant input into firms' product launch decisions, monetary expenditures are inextricably linked to the results of these studies. To ensure efficient utilization of resources, users should therefore ensure that conjoint studies are designed to minimize cognitive biases and enhance predictive accuracy.

The main goal of this paper has been to understand how a common type of cognitive bias might arise by the manner in which stimuli are presented to respondents in a conjoint study. At its heart, conjoint studies rely on customer choice data collected in response to stimuli. Hence, studying how the mode of stimulus administration (separate or joint) endorses bias is a worthwhile research endeavor.

The results of a comprehensive review of extant conjoint studies indicate that possible bias might exist given rather inconsistent manner in which stimuli is presented to respondents. Subsequently, based upon the results of two experiments, we find overwhelming evidence for the existence of presentation bias, or preference reversal, for attributes such as price, that are relatively objective or comparable (Nowlis and Simonson, 1997). Based upon the evaluable literature (Hsee, Blount, Loewenstein, and Batzerman, 1999; Schmeltzer, Caverni, and Warglien, 2004; Willemsen and Keren, 2004; Yeung and Soman, 2005), we argued that the fundamental source of bias is the degree of mismatch between attribute characteristics and the mode of stimulus presentation. In the presence of such mismatch, respondents typically experience heightened cognitive burden, which negatively affects their evaluation task and results in preference reversal. Our results appear robust to validity and generalizability considerations. We discuss the broad implications of our study below.

**Implications.** Several important implications directly follow from our finding. First, irrespective of the conjoint method employed (trade-off, full profile, hybrid) users should pay careful attention to the mode of stimuli presentation. Instead of selecting modes (separate or joint) arbitrarily, the overriding
objective should be to create matched conditions between attribute characteristics and presentation formats. A good starting point in this regard is to incorporate logical and theoretical considerations when selecting a particular approach. For example, Nowlis and Simonson (1997) suggest that assessing attribute characteristics (comparable or enriched) might be an important first step in creating matched conditions. From a cognitive standpoint, enriched attributes such as brands and quality are better matched with a separate presentation mode. In contrast, for comparable attributes such as price or size, the joint presentation mode is more appropriate.

In situations where there is ambiguity, or when clear-cut theoretical guidelines for selecting the mode of presentation are unavailable, researchers should determine the magnitude of preference reversal and use it as a selection criterion. For instance, if a particular design calls for the use of both presentation modes, the degree of preference reversal should be used to select the optimal mode. If the degree of preference reversal is minimal, the original mode should be preserved.

**Limitations and scope for further research.** The results of this paper have to be considered in light of some limitations. First, a limitation in scope and its consequent impact upon generalizability should be noted. Although we used two product classes, our results will probably not generalize to all situations. For example, additional cognitive variables such as product involvement, attachment, or self-image might impact the degree of cognitive effort for complex concepts like automobiles. Hence, a more systematic consideration of different product categories would aid generalizability. Second, the number of levels and attributes might also influence the degree of bias because cognitive burden on respondents increases with more levels and attributes.

In addition to pursuing research that can aid generalizability, future research efforts can investigate the phenomena of bias when other stimuli types such as verbal or pictorial concepts are used. Finally, future research should focus on uncovering better measurement metrics for assessing presentation bias.

## References

1. Allenby, Greg M. and James Ginter (1995). "Using extremes to design productsand segment markets", *Journal of Marketing Research*, 32 (4), pp. 152-162.

2. Bradlow, Eric T., Ye Hu, and Teck-Hua Ho (2004). "Modeling Behavioral Regularities of Consumer Learning in Conjoint Analysis", *Journal of Marketing Research*, 41 (November), pp. 392-396.

3. Chatterjee, Subimal, Timothy B. Heath, and Junhong Min (2009). "The Susceptibility of Mental Accounting Principles, to Evaluation Mode Effects," *Journal of Behavioral Decision Making*, 22, pp. 120-137.

4. Ding, Min (2007). "An Incentive-Aligned Mechanism for Conjoint Analysis," *Journal of Marketing Research*, 44 (May), pp. 214-223.

5. Ding, Min, Young-Hoon Park, and Eric T. Bradlow (2009). "Barter Markets for Conjoint Analysis," *Management Science*, 55 (6), pp. 1003-1017.

6. Fischer, Marc, Franziska Volckner, and Henrik Sattler (2010). "How Important Are Brands? A Cross-Category, Cross-Country Study," *Journal of Marketing Research*, 47 (October), pp. 823-839.

7. Green, Paul E. (1984). "Hybrid models for conjoint analysis: An expository review," *Journal of Marketing Research*, 21 (May), pp. 155-159.

8. Green, Paul E., and Wayne S. DeSarbo (1979). "Componential Segmentation in the Analysis of Consumer Trade-Offs," *Journal of Marketing*, 43 (Fall), pp. 83-91.

9. Green, Paul E., Abba M. Krieger, and Yoram Wind (2001). "Thirty Years of Conjoint Analysis: Reflections and Prospects," *Interface*, 31 (3), pp. S56-S73.

10. Green, Paul E., Stephen M. Goldberg, and Mila Montemayor (1981). "A Hybrid Utility Estimation Model for Conjoint Analysis," *Journal of Marketing*, 45 (Winter), pp. 33-41.

11. Green, Paul E., and V. Srinivasan (1990). "Conjoint Analysis in Marketing: New Developments with Implications for Research and Practice," *Journal of Marketing*, 54 (October), pp. 3-19.

12. Heeler, Roger M., Chike Okechuku, and Stan Reid (1979). "Attribute Importance: Contrasting Measurements," *Journal of Marketing Research*, 41 (February), pp. 60-63.

13. Hsee, Christopher, Sally Blount, George F. Loewenstein, and Max H. Bazerman (1999). "Preference Reversals between Joint and Separate Evaluations of Options: A Review and Theoretical Analysis," *Psychological Bulletin*, 125 (5), pp. 576-590.

14. Jain, Arun K., Franklin Acito, Naresh K. Malhotra, and Vijay Mahajan (1979). "A Comparison of the Internal Validity of Alternative Parameter Estimation Methods in Decompositional Multiattribute Preference Models," *Journal of Marketing Research*, 16 (August), pp. 313-322.

15. Johnson, Richard M. (1974). "Trade-off Analysis of Consumer Values," *Journal of Marketing Research*, 11 (May), pp. 121-127.

16. Louviere, Jordan J., David A. Hensher, and Joffre D. Swait (2000). *Stated Choice Methods: Analysis and Applications*, Cambridge: University Press.
17. Louviere, Jordan J., and George Woodworth (1983). "Design and Analysis of Simulated Consumer Choice or Allocation Experiments: A Method Based on Aggregate Data," *Journal of Marketing Research*, 20 (November), pp. 350-367.

18. Metha, Raj, William L. Moore, and Teresa M. Pavia (1992). "An Examination of the Use of Unacceptable levels in Conjoint Analysis," *Journal of Consumer Research*, 19 (December), pp. 470-476.

19. Nowlis, Stephen M. and Itamar Simonson (1997). "Attribute-Task Compatibility as a Determinant of Consumer Preference Reversals," *Journal of Marketing Research*, 34 (May), pp. 205-218.

20. Oppewal, Harmen and Martijn Klabbers (2003). "Compromising between Information Completeness and Task Simplicity: A Comparison of Self-Explicated, Hierarchical Information Integration, and Full-profile Conjoint Methods," *Advances in Consumer Research*, 30, pp. 298-304.

21. Ostrom, Amy and Dawn Iacobucci (1995). "Consumer Trade-Offs and the Evaluation of Services," *Journal of Marketing*, 59 (January), pp. 17-28.

22. Ramaswamy, Venkatram and Steven H. Cohen (2000). "Latent class models for conjoint analysis," in *Conjoint Measurement: Methods and Applications*, eds. A. Gustafsson, A. Hermann, and F. Huber, Springer-Verlag, Berlin, Germany.

23. Sawtooth Software (2008). *Sequim, WA*.

24. Schmeltzer, Christophe, Jean-Paul Caverni, and Masimo Warglien (2004). "How Does Preference Reversal Appear and Disappear? Effects of the Evaluation Mode," *Journal of Behavioral Decision Making*, 17, pp. 395-408.

25. Segal, Madhav N. (1982). "Reliability of Conjoint Analysis: Contrasting Data Collection Procedures," *Journal of Marketing Research*, 19 (February), pp. 139-143.

26. Sethuraman, Raj, Roger A. Kerin, and William L. Cron (2005). "A field study comparing online and offline data collection methods for identifying product attribute preferences using conjoint analysis," *Journal of Business Research*, 58, pp. 602-610.

27. Srinivasan V. (1988). "A Conjunctive-Compensatory Approach to the Self-Explication of Multiattributed Preferences," *Decision Sciences*, 19 (Spring), pp. 295-305.

28. Srinivasan, V., Peter G. Flachbart, Jarir S. Dajani, and Rolfe G. Hartley (1981). "Forecasting the Effectiveness of Work-Trip Gasoline Conservation Policies Through Conjoint Analysis," *Journal of Marketing*, 45 (Summer), pp. 157-172.

29. Srinivansan V. and Chan Su Park (1997). "Surprising Robustness of the Self-Explicated Approach to Customer Preference Structure Measurement," *Journal of Marketing Research*, 34 (2), pp. 286-291.

30. Toubia, Olivier, John R. Hauser, and Duncan I. Simester (2004). "Polyhedral Methods for Adaptive Choice-Based Conjoint Analysis," *Journal of Marketing Research*, 41 (February) pp. 116-131.

31. Vargo, Stephen L. and Robert F. Lusch (2004). "Evolving to a New Dominant Logic for Marketing," *Journal of Marketing*, 68 (January), pp. 1-17.

32. Willemsen, Martijn C. and Gideon Keren (2004). "The Role of Negative Features in Joint and Separate Evaluation," *Journal of Behavioral Decision Making*, 17, pp. 313-329.

33. Wittink, Dick R. and Philippe Cattin (1989). "Commercial Use of Conjoint Analysis: An Update," *Journal of Marketing*, 53 (July), pp. 91-96.

34. Yeung, Catherine W.M. and Dilip Soman (2005). "Attribute Evaluability and the Range Effect," *Journal of Consumer Research*, 32 (December), pp. 363-369.

## Appendix A. Example of stimulus separate presentation

Imagine that you need to book a hotel room to impress a very important client who is coming into town to possible "deal".

There will be a total of sixteen different hotels. Please rate these hotels using the 7 point scale, where "1" means very unlikely to book the hotel, "4" means neither unlikely or likely to book the hotel, and "7" means very likely to book the hotel.

<table><caption>Table 1A. Q1: Likelihood of booking the hotel below ( )</caption><thead><tr><th>Very unlikely to book</th><th>Neither unlikely or likely to book</th><th>Very likely to book</th></tr></thead><tbody><tr><td>1-----------2-----------3-----------4-----------5-----------6-----------7</td><td></td><td></td></tr></tbody></table>

<table><caption>Table 2A. Q2: Likelihood of booking the hotel below ( )</caption><tr><td>Price</td><td>$120</td></tr><tr><td>Quality</td><td>Really good quality</td></tr><tr><td>Staff courtesy</td><td>Friendly</td></tr><tr><td>Service customization</td><td>Customized</td></tr></table>

<table><tr><td>Price</td><td>$120</td></tr><tr><td>Quality</td><td>Really good quality</td></tr><tr><td>Staff courtesy</td><td>Not too friendly</td></tr><tr><td>Service customization</td><td>Standardized</td></tr></table>
<table><caption>Table 3A. Q3: Likelihood of booking the hotel below ( )</caption><tr><td>Price</td><td>$120</td></tr><tr><td>Quality</td><td>Reasonable quality</td></tr><tr><td>Staff courtesy</td><td>Not too friendly</td></tr><tr><td>Service customization</td><td>Customized</td></tr></table>

<table><caption>Table 4A. Q4: Likelihood of booking the hotel below ( )</caption><tr><td>Price</td><td>$70</td></tr><tr><td>Quality</td><td>Really good quality</td></tr><tr><td>Staff courtesy</td><td>Not too friendly</td></tr><tr><td>Service customization</td><td>Standardized</td></tr></table>
<table><caption>Appendix B. Example of stimulus joint presentation</caption><tbody><tr><td>Price</td><td>Hotel 2<br>$120<br>Really good quality<br>Not too friendly<br>Standardized<br>Rate ( )</td><td>Hotel 8<br>$70<br>Really good quality<br>Not too friendly<br>Customized<br>Rate ( )</td><td>Hotel 14<br>$70<br>Reasonable quality<br>Not too friendly<br>Customized<br>Rate ( )</td><td>Hotel 1<br>$120<br>Really good quality<br>Friendly<br>Customized<br>Rate ( )</td></tr><tr><td>Quality</td><td>Hotel 7<br>$120<br>Reasonable quality<br>Friendly<br>Standardized<br>Rate ( )</td><td>Hotel 13<br>$70<br>Really good quality<br>Friendly<br>Customized<br>Rate ( )</td><td>Hotel 6<br>$70<br>Really good quality<br>Friendly<br>Standardized<br>Rate ( )</td><td>Hotel 12<br>$70<br>Really good quality<br>Not too friendly<br>Standardized<br>Rate ( )</td></tr><tr><td>Staff courtesy</td><td>Hotel 5<br>$70<br>Reasonable quality<br>Not too friendly<br>Standardized<br>Rate ( )</td><td>Hotel 11<br>$120<br>Reasonable quality<br>Not too friendly<br>Standardized<br>Rate ( )</td><td>Hotel 4<br>$70<br>Reasonable quality<br>Friendly<br>Customized<br>Rate ( )</td><td>Hotel 10<br>$120<br>Really good quality<br>Friendly<br>Standardized<br>Rate ( )</td></tr><tr><td>Price</td><td>Hotel 16<br>$70<br>Reasonable quality<br>Friendly<br>Standardized<br>Rate ( )</td><td>Hotel 3<br>$120<br>Reasonable quality<br>Not too friendly<br>Customized<br>Rate ( )</td><td>Hotel 9<br>$120<br>Reasonable quality<br>Friendly<br>Customized<br>Rate ( )</td><td>Hotel 15<br>$120<br>Really good quality<br>Not too friendly<br>Customized<br>Rate ( )</td></tr><tr><td>Quality</td><td></td><td></td><td></td><td></td></tr><tr><td>Staff courtesy</td><td></td><td></td><td></td><td></td></tr></tbody></table>

<caption>Fig. 1A. Example of stimulus joint presentation</caption>
## Appendix C. Example of toaster concept card

Assume that you need to buy a toaster soon. Please carefully evaluate the product description below. If the toaster is available in the market, how likely is it that you would buy it? Please indicate your preference using the following scale.

<figure><img src="image_8.png" /><figcaption>Table 5A. Example of toaster concept card</figcaption></figure>

<table><tr><td>Brand</td><td>K-Mart</td></tr><tr><td>Features<sup>a</sup></td><td>2 wide slots<br>Mechanism centers bread in slot<br>Automatically pops the toast up when done<br>Removable crumb tray</td></tr><tr><td>Price</td><td>$19.99</td></tr></table>

Note: <sup>a</sup> Common features were added to all concept cards for realism.