Matched Samples Analysis for Item Responses

A statistical technique for testing treatment differences for binary responses from matched samples is discussed and applied to item responses. Examples are cited in which the technique is applicable, and a numerical example is given in which the technique is applied to a new data set concerning the states of happiness of people. G. David Faulkenberry is Associate Professor of Statistics and Project Director, Survey Research Center, and Robert G. Mason is Professor of Sociology and Project Director, Survey Research Center, Oregon State University. POQ 40 (1976) 256-260 This content downloaded from 157.55.39.55 on Tue, 23 Aug 2016 04:38:10 UTC All use subject to http://about.jstor.org/terms MATCHED SAMPLES ANALYSIS 257 T = total item responses m = number of items being compared n = number of respondents The approximate distribution of Q under the null hypothesis is chi square with m 1 degrees of freedom. An example of recent research in which it would be appropriate to apply this statistic is the work of Kirkpatrick concerning the analysis of panel data to determine changes in attitudinal consistency on several issues.3 Kirkpatrick used survey data at two points in time, and at each point he classified the respondent as being consistent or inconsistent in certain attitudes. He did not use statistical tests, but if his hypothesis was that the population as a whole is remaining stable (i.e., that the proportion of consistents in the population is the same at the two or more points in time), then the Q statistic would be appropriate. More recently, Schuman has interpreted the percent cuts from three types of employment discrimination items to show that attitudinal inconsistencies occur among responses to the three items.4 For each item one response condones, in different circumstances, job discrimination. The author presents his ideas without benefit of statistical tests and gives a clear discussion of the results. However, the Q statistic is clearly appropriate as a first step in a statistical analysis and this would add a degree of conciseness to his presentation. Weinstein employed a reduced (i.e., one degree of freedom) form of the statistic to demonstrate that improved prediction from attitudes to behavior is possible if attitudes toward action also are assessed.5 An overall test, using the Q statistic, would have been appropriate prior to the single degree of freedom tests. An opportunity to employ the statistic directly occurred in a recent effort to test a hypothesized interrelationship among responses to items about the overall state of happiness of respondents. The items were asked of a 1974 pretest sample of 100 Oregon residents, the sample having been drawn by area probability methods. The wording and percent cuts for the items are shown in Table 1. The percent cuts suggest an interesting anomaly: viz., that individuals believe most people were happier 25 years ago than they are today, yet, they would prefer to live under today's conditions than live under those presumably happier ones of a generation earlier. Moreover, they believe that people generally will be just as happy or happier 10 years in the future as they are today. Erskine reported similar percent cuts with items of 3 Samuel A. Kirkpatrick, "Political Attitude Structure and Component Change," Public Opinion Quarterly, Vol. 34, 1970, pp. 403-407. 4 Howard Schuman, "Attitudes vs. Actions versus Attitudes vs. Attitudes," Public Opinion Quarterly, Vol. 36, 1972, pp. 347-354. 5 Alan G. Weinstein, "Predicting Behavior from Attitudes," Public Opinion Quarterly; Vol. 36, 1972, pp. 355-360. This content downloaded from 157.55.39.55 on Tue, 23 Aug 2016 04:38:10 UTC All use subject to http://about.jstor.org/terms 258 FAULKENBERRY AND MASON Table 1. State of Happiness (N = 100) XResnse Happier Happier Same, Item 25 Years Ago Now DK 1. Is it your impression that Americans were happier and more content 25 years ago or are they happier and more contented now? 62% 35% 3% 2. Now, please compare your 25 Years Present DK impression of living conditions 25 Ago years ago with those of today. You would be the same age and you wouldn't be living any of your life over again. Nothing would change but the time period. Under these circumstances, do you wish that you were living in those days 25 years ago, or at the present time? 23% 77% 0 3. Ten years from now, would you More Less Same, guess that Americans will generally Happy Happy DK be more or less happy than they are now? 46% 27% 27% the same type.6 Now, to investigate the consistency of the attitude of "happier times in the past," let A denote the class of people in the population who think Americans were happier and more contented 25 years ago, B denote the class of people in the population who wish they were living in those days 25 years ago, and C denote the class of people in the population who think Americans will be less happy in 10 years. Also let A, B and C denote the complements and P(A), P(B), and P(C), the respective proportions in the population or the probability of getting a person in that class on a random selection. The main question we are considering is how to test the null hypothesis Ho: P(A) = P(B) = P(C), and it is not sufficient to have only the sample proportions; we need the intersections associated with the classes. Table 2 gives the sample information in a form we need, where a "1" indicates a response of type A, B, or C, respectively, for the three items, and "O" indicates a response of 6 Hazel Erskine, "The Polls: Some Thoughts about Life and People," Public Opinion Quarterly, Vol. 28, 1964, pp. 517-528. This apparent anomaly was employed, in part, to define conceptually the term "life satisfaction." See Robert Mason, G. David Faulkenberry, and Alexander Seidler, The Quality of Life as Oregonians See It, Corvallis, Oregon State University, 1975. This content downloaded from 157.55.39.55 on Tue, 23 Aug 2016 04:38:10 UTC All use subject to http://about.jstor.org/terms MATCHED SAMPLES ANALYSIS 259 Table 2. Response Patterns to State of Happiness Items Response Observed Pattern Item I Item 2 Item 3 Frequency 1 1 1 1 3 2 1 1 0 15 3 1 0 1 12 4 0 1 1 0 5 1 0 0 32 6 0 1 0 5 7 0 0 1 12 8 0 0 0 21