Respondent Perceptions of Integrity and Personality Measures: Does Response Format Make a Difference?

Two studies are reported that examined the effect on respondent reactions of modifying the number of points on a response scale. Results indicate that for an integrity test, respondents who completed a measure using a 5-point Likert scale perceived themselves to have performed better on the test and perceived the measure to be significantly more fair and more face valid than respondents who completed an identical scale using a dichotomous response format. No such differences, however, emerged between dichotomous, 5-point, and 9-point versions of a personality inventory, indicating that although respondent reactions may be affected by the number of points in a response scale, some types of personnel selection measures may be more affected by number of response points than others. In the past two decades, much research has emerged on applicant reactions to a variety of employee selection methods (Anderson, 2004). One area that has received relatively little attention is the impact methodological decisions may have on applicant reactions (Chan & Schmitt, 2004) and the extent to which the impact of those methodological factors may depend on the content of the test. Chan and Schmitt (1997) examined test medium as a potential factor, but to our knowledge, specific methodological factors other than those related to technology (e.g., Bauer, Truxillo, Paronto, Weekley, & Campion, 2004; Chapman, Uggerslev, & Webster, 2003) have not generally been focal to research on applicant reactions. However, other micro-level decisions made during the construction of an assessment tool may also affect respondent reactions. In the current study, we sought to understand whether reactions to different types of tests differs on the basis of whether responses are made on a dichotomous (true/false) scale or on a scale with more than two response options. Study One: Integrity Despite significant relationships between integrity scores and work behaviors (e.g., Ones, Viswesvaran, & Schmidt, 1993; Sackett & Wanek, 1996), test takers often perceive integrity tests as being unrelated to the job (lacking face validity) and unfair (Rosse, Ringer, & Miller, 1996; Rynes & Connerley, 1993; Steiner & Gilliland, 1996). Although researchers hypothesized that these negative perceptions of integrity tests were the result of the invasiveness or offensiveness of the personal questions, little or no support has been found for this claim (Neuman & Baydoun, 1998). Therefore, the items themselves may not be causing negative reactions, but the response format of the test might. Research indicates that negative reactions to testing may partially result from the response format of the test, and that the available range of response can influence assessment fairness perceptions. Specifically, participants responding to attitudinal scales reported feeling unable to give an accurate answer when being assessed with a scale with fewer than 5 points (Chicchetti, Showalter, & Tyrer, 1985), preferring scales with 5-9 points, and feeling that scales with 2-3 points provided inaccurate assessments (Cox, 1980). That the response format of integrity tests is often dichotomous (yes/no, true/false; Rudner, 1992) suggests a potential problem. If integrity items with dichotomous response formats lead to perceived inaccurate assessment, this may lead to lower perceptions of likely performance on the test, which have been shown to be related to perceptions of face validity (Chan, Schmitt, Jennings, Clause, & Delbridge, 1998; Macan, Avedon, Paese, & Smith, 1994; Rynes & Connerly, 1993). Perceptions of low performance may in turn result in perceptions of low face validity. Research has demonstrated that perceptions of face validity influence perceptions of instrument fairness (Smither, Millsap, Stoffey, Reilly, & Pearlman, 1996). Hence, we hypothesized that individuals who received an integrity test with a 5-point response scale would perceive the test to be significantly more fair and face valid and would perceive themselves to have performed better than individuals who received a content-identical scale with a dichotomous response format. …