How Sample Size and Strength of Association Affect the Ability to Detect Group Differences in Cross-Classification Analysis

It is well-known in the statistical li terature that when sample size is large, any small departure from the “true” model would yield significant test statistics. This issue is further compounded by the fact that researchers often face the problem of analyzing groups of unequal sizes and as a result our abili ty to detect group differences may be hampered. Even though this topic has not been investigated systematically, researchers conducting statistical analysis in multi-way cross-classification tables often adopt various ad hoc standardization procedures to address this particular problem by inflating and deflating group sizes to comparable values. Using Monte Carlo simulations, this study offers an empirical investigation on the effect of (a) varying sizes across groups, (b) differences in the strength of association between groups, and (c) the ordering of (a) and (b) on our abili ty to detect differences in association between groups and whether the standardization procedure offers a viable solution. The findings confirm that while it is true that differences in group size and strength of association affect the likelihood-ratio test statistics significantly when the specified models are incorrect, their influence on the “true” model and its over-parameterized counterparts is negligible. In other words, unlike statistical models such as those in covariance structure analysis, the goodness-of-fit statistics in log-linear models are independent of sample size when the models specified are “true. ” On the other hand, two common problems are associated with the standardization procedure: the problem of underdispersion and the possibili ty of committing type II error in some occasions. As a result, standardization does not appear to be a viable strategy and researchers should instead rely on conventional nested chi-squared tests for detecting group differences. Finally, the study provides strategies to examine whether the preferred models indeed are the proper ones. The calculation of the likelihood-ratio chi-squared statistic (L2) is the following: L = -2 log = 2 nij log (nij/m̂ij) where is the Wilk statistic, nij and m̂ij are observed and expected frequencies under a particular model (Agresti 1990). Introduction Most empirical investigations by social science researchers naturally involve comparisons of different social groups (such as gender, racial/ethnic groups, countries, organizations, and industries) or temporal trend (such as birth cohorts, time series data, and longitudinal data). They include the study of racial and ethnic differences in the relationship between education and occupation, country-specific variation in occupational mobility, and temporal changes in the relationship between education and religiosity, political identification, or attitudes toward abortion. Furthermore, it is not unusual that the sizes of these groups are unequal, due either to population distribution or sampling design. In multiway contingency table analysis, it is generally believed that varying group sizes, especially when the discrepancy is large, may play a pivotal role in influencing our ability to detect group differences or temporal trend. Such belief is further reinforced by the well-known fact in statistical literature that when the sample size is large, any small departure from the “true” model would yield significant test statistics and therefore researchers run considerable risks of identifying trivial differences when in reality these group differences are negligible. As noted by Gelman and Rubin (1995), “it is possible to have so much data that a test will reject every parsimonious model that is proposed” (p. 166). To date, there is little methodological investigation into this particular subject matter (see Fitzmaurice and Goldthorpe 1997; Fitmaurize, Heath, and Cox 1997; and Wong 1994 for some exceptions). Meanwhile, several 1-df tests and other complex statistical models have been developed for cross-classification analysis that partially address this particular problem (Goodman