Abstract:
Measurement errors arising from presentation aspects of a survey are detected and can be corrected, based upon responses from a selected set of respondents to different survey presentations. The survey presentations are different not in their information content, but in their presentation, such as using a different order of questions. An alternative presentation or presentations for the survey can be derived such that the measurement error is reduced. Accordingly, an “unbiased” survey can be designed in the pre-deployment phase, or measurement errors can be detected and corrected during the deployment phase.

Description:
FIELD OF THE INVENTION  
       [0001]     The present invention relates to assessing survey presentations, such as marketing surveys and the like.  
       BACKGROUND  
       [0002]     Surveys are one of the most popular methods of obtaining business intelligence. For example, customers&#39; preferences, pain points and future intent are some of the common forms of business intelligence gathered using surveys. Gathering business intelligence using surveys typically consists of several steps, sometimes referred to as: “Goal”, “Who”, “What”, “How” and “Analysis”.  
         [0003]     The “Goal” step defines the goals of the survey (what is to be learnt from the survey exercise), the “Who” step defines who is going to be asked to provide feedback, the “What” step involves creating a set of questions, the “How” step defines the presentation and administration of the survey. The “Analysis” step defines what would be done to the responses to obtain business intelligence.  
         [0004]     Measurement errors in surveys are deviations of the respondents&#39; answers from their true value on the measure. For “online” surveys, and more particularly those offered or administered via the World Wide Web, there are two main sources of measurement errors, which stem from the Web questionnaire itself.  
         [0005]     The first source of error is the wording of the questions or the flow of the questionnaire that can both have an effect on the quality of respondents&#39; answers. The second source of error is the questionnaire form, (that is, the visual layout of the questionnaire), which is of particular importance in self-administered surveys. These aspects have been studied in detail in the socio-psychological field, with the emphasis on developing certain guidelines or heuristics that can be used to design “better” survey presentations, that is, presentations that lead to a smaller measurement error. A need clearly exists for an improved method of assessing survey presentations.  
       SUMMARY  
       [0006]     Measurement errors in surveys are deviations of the respondents&#39; answers from their true value on the measure. Errors result, in general, from inaccurate responses that may stem from poor questionnaire design, biased question flow, and poor visual layout, amongst other factors.  
         [0007]     The techniques described herein relate to detecting and also correcting measurement errors in a survey, based upon responses from respondents to different survey presentations. First, different surveys are presented to different respondents. The surveys are different not in their information content, but in their presentation (for example, surveys with a different order of questions). These different surveys are presented to a carefully sampled set of respondents. Second, the responses received from respondents are used to detect, estimate and correct the measurement error in the surveys.  
         [0008]     Optionally, an alternate presentation or presentations for the survey can be generated such that the measurement error is reduced. Thus the method can be used either in the pre-deployment phase to design an “unbiased” survey, that is, a survey with minimal measurement error, or can be used in the deployment phase to detect and correct the measurement error by analyzing the responses of a given set of surveys offered in different presentations. 
     
    
     DESCRIPTION OF DRAWINGS  
       [0009]      FIG. 1  is a flow chart of steps involved in the technique for refining survey presentations.  
         [0010]      FIG. 2  is a graph of four probability distribution functions corresponding to responses to a specific question for a specific respondent profile in four different survey presentations.  
         [0011]      FIG. 3  is a graph of a probability distribution function corresponding to the actual true response and three different estimates of the probability distribution function corresponding to the true response obtained using the observed responses shown in  FIG. 2 .  
         [0012]      FIG. 4  is a schematic representation of a computer system suitable for performing the techniques described herein. 
     
    
     DETAILED DESCRIPTION  
       [0013]      FIG. 1  flow charts, in overview, steps involved in refining survey presentation. First, a number of different survey presentations are generated for a particular survey in step  110 . Then, these various survey presentations are presented to particular selections of a population in step  120 . An unbiased response to the survey is estimated, based on an analysis of the survey responses, in step  130 . Based upon this estimated unbiased response, estimation errors are assigned to each of the survey presentations in step  140 . The estimation error can be used to compensate survey findings, and the process can be iterated to design survey presentations that have lower measurement errors.  
         [0000]     Survey Presentation Design  
         [0014]     The survey presentation design component designs alternative presentations of the survey. Note that it does not design the information content of a survey, a survey designer designs a survey, that is, completes the “Goal”, “Who” and “What” steps of the survey. The survey presentation design component then defines “How” the survey is presented to the user.  
         [0015]     Different features of a presentation may include the following: 
        1. Question Ordering: The ordering of questions has an impact on the survey responses. For example there are well known psychological heuristics that suggest that negative questions should not be asked in the beginning of a survey. Thus the questions can be ordered in different ways in different presentations of the survey.     2. Word Phrasing: A question could have a different word phrasing, each phrasing delivering the same information content but maybe with different connotations. Thus, selecting alternate phrasing for each question allows for alternate presentations.     3. Color: The background (or foreground) color could affect the survey responses, for example, use of particular colors may in some circumstances bias the response towards a corporation that uses a similar color in its branding. Thus different presentation can use different colors in presenting a question.     4. Display Type: There are different display types for the given questions constituting the survey, for example, options for a Select question can be shown in the form of radio buttons or drop-down list box, or options for a Select multiple questions can be shown as Check-boxes or 2-buckets with add/remove options. Each of these displays affect the survey response maybe because the default response option that is visible may be different. Thus different display types can be used to design different presentations.        
 
         [0020]     One may perhaps anticipate, based on relevant experience or reasoning, the likely effects of some of the above presentation factors on the survey response. For example, one may expect more respondents to select the default displayed option in a drop-down list box than to select the default option in a radio button list. This is because all the options are not visible in a drop-down list box, and time and effort is required on the respondents&#39; part to view the entire list of options.  
         [0021]     The effect of more complex presentation designs are more difficult to anticipate, as various presentation factors may interact.  
         [0022]     The survey designer could for each question, list the possibilities of the question&#39;s position (with respect to other questions), a list of word phrasings, a list of colors, and a list of display types. Multiple presentations of surveys are then generated using a combination of these features. As described in further detail below, these presentations are then offered to a set of respondents, and the resulting responses analyzed to detect measurement error. One can then compensate for such measurement errors.  
         [0023]     If the goal is to find a survey presentation that has a minimum measurement error, the respondent data collected in previous iterations can be used in suggesting an alternate presentation of the survey that has less measurement error.  
         [0024]     Each survey presentation can be mapped onto a point in the feature space defined by the various presentation features, for example, color, word phrasing, and so on. The initial set of survey presentations can be selected from the set of valid presentation designs in the feature space by taking presentations that are pair-wise distant in order to cover the feature space well.  
         [0025]     Distances in the feature space can be defined as follows. For question ordering, the distance between the presentations can be taken as the sum of the distances between the questions, where the distance of a question in the presentations is defined as the number of positions that they are apart by. For other features like display types, colors and word-phrasings, indicator function can be taken as the distance metric, that is, different values can be taken as unit distance apart. The combined distance of multiple presentation features (question ordering, color, and so on) can be taken by adding the individual distances after appropriate scaling.  
         [0026]     If applied iteratively, the presentations for further iterations, that is, points in the space, can be selected based on any local search heuristics techniques such as Hill Climbing, Genetic Algorithms, Simulated Annealing, Tabu search, Ant colony optimization, and so on, where the goal is to find a presentation (that is, a point in the feature space) with minimum measurement error. The measurement error is computed for each presentation, as described in further detail below.  
         [0000]     Selection of Respondents  
         [0027]     Respondents are desirably selected as a fair representation of the generic population. A constraint on the sampled population, however, is that the analysis method (described below) can create groups of the sampled population with similar “profiles”, with each group containing sufficient number of respondents.  
         [0028]     Some examples of attributes that could constitute profiles are age, sex, country, political affiliation, income group, and so on. If the profile information is not available beforehand, the first few questions of the survey can be used to gather the profile information. Moreover, the respondents can be matched by profile across the different survey presentations, that is, different respondents of each of the profiles are desirably offered all of the different presentations.  
         [0029]     The reason for selecting similar respondents is that one expects respondents of similar profile to respond similarly to questions in an unbiased survey, that is, a survey with no measurement error. Moreover, if a set of surveys with different presentations is offered to similar respondents then in most of the presentations, most of the respondents are not biased in their response. That is, there may be a few presentations in which most of the respondents are biased in their response, but in most presentations few respondents are biased in their response.  
         [0000]     Analysis of Responses  
         [0030]     Survey responses are analyzed to detect, and in some cases correct, measurement errors. As described above, different presentations of the survey are offered to respondents, who can be grouped according to their profiles. Multiple responses are analyzed to detect and correct the measurement error and optionally generate a survey presentation design with minimum measurement error, if required.  
         [0031]     One can determine if there is a significant bias (measurement error) in any of the survey presentations in the offered set of presentations. Detection requires an estimation of the unbiased response across the set of offered survey presentations. The measurement error is the measure of the deviation of the observed set of responses from this estimated unbiased response. If this is significant then the presence of bias in the set of survey presentations is indicated.  
         [0032]     Given the multiple responses, any standard estimation technique (for example, Least Mean Square Estimation, Max A Priori Estimation) can be used to estimate the unbiased response {circumflex over (f)}. To improve the estimate, robust estimation techniques and/or penalty regularization techniques, as described below, can be used.  
         [0033]     As stated above, if a set of surveys with different presentations is offered to similar respondents then, in most of the presentations most of the respondents are not biased in their response. Therefore, the few outliers (that is, the survey presentations that bias a large fraction of the respondents) can be removed by using robust estimation techniques.  
         [0034]     One example of a robust estimation technique is summarized in the pseudocode of Table 1 below.  
                                                             TABLE 1                           For each question                For each profile                To eliminate the m (some threshold) survey presentations           that have the           most deviating responses, repeat the following steps:                Find the average of the response distributions of           the remaining           set of survey presentations.           Mark and remove the survey presentation for which the           response distribution deviates most from the average           response           distribution.                End            End                  
 
         [0035]     Additional prior information can aid the estimation. Compared to more conventional techniques, such as MAP estimation, penalty regularization provides increased flexibility in selecting the penalty: explicit knowledge of a statistical prior is not required. This is an appreciable advantage, because: (i) priors are often not available; and (ii) priors may be difficult to describe analytically. Penalty regularization is defined as in Equation [1] below.  
               f   ^     =     arg   ⁢       min   f     ⁢     [         -   ln     ⁢           ⁢     p   ⁡     (     y   ⁢     |     ⁢   f     )         +     λΦ   ⁡     (   f   )         ]                 [   1   ]             
 
         [0036]     Equation [1] above defines y as the observed variable, f as the variable to be estimated, and Φ(f) as the penalty or regularization function, which takes into account a priori knowledge about f, and {circumflex over (f)} the estimate variable.  
         [0037]     An example of a penalty regularization function for estimating unbiased survey response uses the reasonable assumption that in an unbiased survey, similar respondents answer a question in a similar way. Mathematically, this implies that the probability distribution function (pdf) of the responses (for a question) is such that its entropy is low, that is, Φ(f) measures the entropy of the pdf. The unbiased response {circumflex over (f)} is then estimated as per Equation [2] below.  
               f   ^     =     arg   ⁢       min   f     ⁢       ∑   questions     ⁢       ∑   profiles     ⁢     [         -   ln     ⁢           ⁢     p   ⁡     (     y   ⁢     |     ⁢   f     )         +     λΦ   ⁡     (   f   )         ]                     [   2   ]             
 
         [0038]     A final step is to compute the degree of bias in a given survey presentation. This depends upon the deviation between observed response (y), and estimated unbiased response ({circumflex over (f)}). This deviation can be calculated using suitable divergence measure between probability distribution such as KL Divergence/JS Divergence. This deviation is denoted herein as E.  
         [0039]      FIG. 2  and  FIG. 3  jointly illustrate an example of how responses can be analyzed to detect the measurement error and estimate the true response (unbiased). In this example the survey is provided in four different presentations. The responses for a particular question are collected for all four presentations. The pdf for each presentation of the responses is displayed in  FIG. 2 . The responses are from respondents of a similar profile.  
         [0040]      FIG. 3  displays the pdf of the actual “true” response (for the question under consideration), and three different estimates of the pdf of the true response. The first estimate (“LMSE with Outlier”) uses a least mean square estimator on all four observed response pdfs (from  FIG. 2 ). The second estimate (“LMSE without Outlier”) uses a least mean square estimator after removing the outliers of the observed responses, using a robust estimator. The third estimate (“Penalized LMSE”) uses least mean square estimator after removing the outliers of the observed responses and uses entropy as the penalty function. Thus, this estimation prefers a pdf that has lower entropy.  
         [0041]     The compensated or corrected response is the estimated unbiased response ({circumflex over (f)}). The corrected response can be reported for each question of the survey individually, for each profile (in the set of profiles of respondents), or for an entire survey presentation.  
         [0000]     Optimal Survey Presentation  
         [0042]     Alternate survey presentations can be designed to minimize measurement error, based on the analysis of responses to the offered survey presentations. This can be done by modeling the measured error as a cost function of the presentation factors, and then searching for the factors that minimize the cost function. The cost function can be formulated using standard methods such as regression analysis. Domain specific knowledge can be used to refine the cost functions. The modeled objective function can be minimized using search algorithms such as, randomized search, Genetic algorithms, Tabu search or other machine learning algorithms.  
         [0043]     As an example, let q 1 , q 2 , . . . , q n  be a list of n questions. In a given survey presentation design, the presentation factors that affect the measurement error are, for each question, as in Table 2 below.  
                   TABLE 2                           1.   The effect of the position of the question           in the survey presentation       2.   The effect of the position of other questions           in the survey presentation (cross-effects)       3.   The effect of the attributes of the question,           that is, color, display type, word phrasings           in the survey presentation design                  
 
         [0044]     A presentation design D is represented by the matrix pair ([v], [w]) where v is a 2-dimensional matrix of size n×n, where n is the number of questions, such that 
        v qp =1 if the question q is placed in the location p in D, and zero otherwise and w is a d-dimensional matrix of size n×m 1 ×m 2 × . . . x m d  where there are d attributes a 1 , a 2 , . . . , a d  (for example, color, display type, and so on) that can be presented in m 1 , m 2 , . . . , m d  different ways in a presentation design, respectively, such that w qx     1     x     2     , . . . x     d   =1 if for question q the presentation of attribute a i  takes the value x i  in D for all i (for example, attribute a i  is color, and value x i  is “red”).        
 
         [0046]     The total measurement error for a survey presentation design represented by the matrix pair ([v], [w]) can be modeled as Equation [3] below.  
             E   =         ∑       q   i     ,     p   i         ⁢       v       q   i     ⁢     p   i         ⁢       g   1     ⁡     (       p   i     ;     θ   i       )           +       ∑       q   i     ,     q   j     ,     p   i     ,     p   j         ⁢       v     q   i       ⁢     v     q   j       ⁢       g   2     ⁡     (       p   i     ,       p   j     ;     θ   ij         )           +       ∑       q   i     ,     q   j     ,     q   k     ,     p   i     ,     p   j     ,     p   k         ⁢       v     q   i       ⁢     v     q   j       ⁢     v     q   k       ⁢       g   3     ⁡     (       p   i     ,     p   j     ,       p   k     ;     θ   ijk         )           +   …   +       ∑       q   i     ,     a   i         ⁢       w       q   i     ⁢     a   i         ⁢     h   ⁡     (       a   i     ;     p   i       )                     [   3   ]             
 
         [0047]     In Equation [3] above, 
        g 1 (.) captures the effect of the position p i  of the question q i  in the survey presentation design on the measurement error. Θ i  is a set of parameters controlling the nature of the function g 1 (.) for the question q i . The function g 1 (.) is represented as a parametric function.     g 2 (.), g 3 (.), . . . capture the second order and higher order effects of positions of other questions with respect to question q i . Θ ij , Θ ijk , . . . are the parameters controlling the functions g 2 (.), g 3 (.), . . . that is, relative positions of the questions q i  and q j , relative positions of the questions q i , q j , and q k  are represented as parametric functions.     h(.) captures the effect of the attribute a i  of the question q i  in the survey presentation design on the measurement error. ρ i  is a set of parameters controlling the nature of the function h(.) for the question q i . The function h(.) is represented as a parametric function.        
 
         [0051]     The optimal survey presentation design with respect to the total measurement error E can be obtained by minimizing the expression E with respect to ([v], [w]) subject to the constraints that are outlined as Equation [4] below.  
                     ∑   p     ⁢     v   qp       =     1   ⁢     ∀   q         ,     
     ⁢         ∑   q     ⁢     v   qp       =     1   ⁢     ∀   p         ,     
     ⁢         ∑     a   i       ⁢     w       qa   1     ⁢   …   ⁢           ⁢     a   i     ⁢   …   ⁢           ⁢     a   d           =     1   ⁢     ∀   q         ,     a   1     ,   …   ⁢           ,     a     i   -   1       ,     a     i   +   1       ,   …   ⁢           ,     a   d       ⁢     
     ⁢         for   ⁢           ⁢   all   ⁢           ⁢   i     =   1     ,   …   ⁢           ,   d             [   4   ]             
 
         [0052]     The first two constraints correspond to the restrictions that a question cannot appear in more than one location and that one location cannot be occupied by more than one question. The last set of constraints corresponds to the restriction that exactly one of the attribute values must be selected for each attribute in each question.  
         [0053]     The survey presentation design parameters θ i , θ ij , θ ijk , . . . , ρ i  can be estimated from the observed measurement error as computed from the obtained set of responses for the different survey presentations designs (that is, for every offered survey design presentation) the matrix pair ([v], [w]) and the actual measurement error E is known.  
         [0054]     Once the parameter set is known the expression for E can be optimized by various methods of stochastic and deterministic optimization (such as linear or quadratic programming, neural networks, simulated annealing, mean field annealing, genetic algorithms, randomized search algorithms).  
         [0000]     Computer Hardware  
         [0055]      FIG. 4  is a schematic representation of a computer system  400  of a type that is suitable for executing computer software for assessing survey presentations. Computer software executes under a suitable operating system installed on the computer system  400 , and may be thought of as comprising various software code means for achieving particular steps.  
         [0056]     The components of the computer system  400  include a computer  420 , a keyboard  410  and mouse  415 , and a video display  490 . The computer  420  includes a processor  440 , a memory  450 , input/output (I/O) interfaces  460 ,  465 , a video interface  445 , and a storage device  455 .  
         [0057]     The processor  440  is a central processing unit (CPU) that executes the operating system and the computer software executing under the operating system. The memory  450  includes random access memory (RAM) and read-only memory (ROM), and is used under direction of the processor  440 .  
         [0058]     The video interface  445  is connected to video display  490  and provides video signals for display on the video display  490 . User input to operate the computer  420  is provided from the keyboard  410  and mouse  415 . The storage device  455  can include a disk drive or any other suitable storage medium.  
         [0059]     Each of the components of the computer  420  is connected to an internal bus  430  that includes data, address, and control buses, to allow components of the computer  420  to communicate with each other via the bus  430 .  
         [0060]     The computer system  400  can be connected to one or more other similar computers via a input/output (I/O) interface  465  using a communication channel  485  to a network, represented as the Internet  480 .  
         [0061]     The computer software may be recorded on a portable storage medium, in which case, the computer software program is accessed by the computer system  400  from the storage device  455 . Alternatively, the computer software can be accessed directly from the Internet  480  by the computer  420 . In either case, a user can interact with the computer system  400  using the keyboard  410  and mouse  415  to operate the programmed computer software executing on the computer  420 .  
         [0062]     Other configurations or types of computer systems can be equally well used to execute computer software that assists in implementing the techniques described herein.  
         [0000]     Conclusion  
         [0063]     Various alterations and modifications can be made to the techniques and arrangements described herein, as would be apparent to one skilled in the relevant art.