Empirical Bayes Estimates of Parameters from the Logistic Regression Model. ACT Research Report Series 97-6.

Maximum likelihood and least-squares estimates of parameters from the logistic regression model are derived from an iteratively reweighted linear regression algorithm. Empirical Bayes estimates are derived using an m-group regression model to regress the within-group estimates toward common values. The m-group regression model assumes that the parameter vectors from m groups are independent, and identically distributed, observations from a multivariate normal "prior" distribution. Based on asymptotic normality of maximum likelihood estimates, the posterior distributions are multivariate normal. Under the assumption that the parameter vectors from the m groups are exchangeable, the hyperparameters of the common prior distribution are estimated using the EM algorithm. Results from an empirical study of the relative stability of the empirical Bayes and maximum likelihood estimates are consistent with those reported previously for the m-group regression model. Estimators that use collateral information from exchangeable groups to regress within-group parameter estimates toward a common value are more stable than estimators calculated exclusively from within-group data. EMPIRICAL BAYES ESTIMATES OF PARAMETERS FROM THE LOGISTIC REGRESSION MODEL Logistic regression is used in many areas of substantive interest in the social and biological sciences to model the conditional expectation (probability) of a binary dependent variable as a function of an observed (or latent) vector of covariates. In some applications, the parameters of the logistic regression function must be estimated from small samples, which can result in parameter estimates with large sampling variability. For example, ACT uses logistic regression to model the probability of success within specific college courses. Using current estimation procedures, a minimum sample size of 45 within-course observations is required. Further, because the estimation algorithm for the parameters of the logistic regression model is iterative, parameter estimates based on small samples may fail to converge, or converge to local, rather than global, stationary points. One way to stabilize parameter estimates is to use collateral information from "exchangeable" groups (Lindley, 1971) to refine the within-group estimates. Bayesian m-group regression models have been shown to increase the prediction accuracy and stability of the parameter estimates, relative to estimates that ignore collateral information. In m-group regression models, the parameter vectors within each of m exchangeable groups are assumed to be independent and identically distributed observations with a common probability density function. When this common distribution is treated as a "prior" distribution, posterior distributions and associated inferences follow from standard Bayesian theory (DeGroot, 1970). The effect of the prior distribution is to regress the within-group maximum likelihood estimates toward common values. The extent of the regression effect is inversely related to the precision of the within-group estimates; as the precision of the within-group estimate decreases, the empirical Bayes estimate moves closer to the prior mean. As the precision increases, the maximum likelihood and empirical Bayes estimates converge. Estimators that regress withingroup parameter estimates toward common values (often referred to as "regressed" or "shrinkage" estimators) are also found in classical theory (James & Stein, 1961; Evans & Stark, 1996). The parameters of the prior distribution are often referred to as "hyperparameters", to distinguish them from the parameters of the logistic regression function. There are different methods for estimating the hyperparameters. A fully Bayesian analysis requires a prior density for the hyperparameters (Novick, Jackson, Thayer, and Cole, 1972). The posterior density of the parameter vector is found by integrating the joint posterior density of the parameters and hyperparameters with respect to the hyperparameters. Since these integrals are seldom expressible in a closed form, numerical integration procedures are often required. Markov chain sampling schemes, such as the Gibbs sampler (Tanner, 1993), are currently under investigation as an alternative to the numerical methods used previously. Empirical Bayesian approaches (Braun, Jones, Rubin, & Thayer, 1983; Houston & Sawyer, 1988; Rubin, 1980), in contrast, derive maximum likelihood estimates of the hyperparameters using the EM algorithm (Dempster, Laird, & Rubin; 1977). The estimated hyperparameters are used in calculating the prior and posterior densities for the within-group parameters. The first section of the paper is concerned with parameter estimation and model fit for the logistic regression model within a single group. An iteratively reweighted linear regression algorithm is used to derive maximum likelihood and least-squares estimates of the parameter vector of the logistic regression function. A chi-square test of the hypothesis of model fit is derived, based on the mean squared error (MSE) from the weighted linear regression model used 2 in the estimation algorithm. The use of a linear regression algorithm makes many of the diagnostic procedures developed for the linear model applicable, with a modification of MSE, to the logistic model, as well. In the next section, collateral information across m exchangeable groups is used to refine the within-group maximum likelihood estimates. Empirical Bayes estimates are derived from a Bayesian m-group regression model. In this model, the parameter vectors of the logistic regression function are assumed to be independent and identically distributed observations from a multivariate normal distribution. In the final section of the paper, an application involving college course placement data (grades in specific college courses and standardized achievement test scores) is presented. An empirical study uses these data to compare the stability of the empirical Bayes and maximum likelihood estimates. Model Specification Let Yj denote a binary random variable, and let Xj denote a (p x 1) vector of covariates, for subject i (i = 1 to n). The logistic regression model may be expressed as The logistic regression function, p ^S ^), has range 0 to 1, with the parameter vector 0 and covariate vector as arguments. It is assumed that the (i = 1 to n) are stochastically The Logistic Regression Model Y i = P i(e ’x i) + £ i> (I)