An Inductive Approach to Production-function Modeling: A Comparison of Group Method of Data Handling (GMDH) and Other Neural Network Methods

This study addresses the susceptibility of the traditional theoretical framework of the educational production-function to political influence and personal bias. The politics of the production-function are becoming increasingly obvious in educational research as the battle over whether money matters in improving educational productivity continues (Hanushek, 19891997; Hedges et. al., 1994,1996). One common thread of the studies on both sides of the money issue is the predominantly deductive approach that is used. Through the purely deductive approach, researchers are driven to test the statistical significance of preconceived notions, political or personal in origin, about how the educational system works. As an alternative approach, inductive, non-linear pattern recognition algorithms are applied to school productivity data from the state of Vermont. These algorithms, known as Neural Networks, yield optimal solutions that differ both in functional form and parameter significance from many of our preconceived notions. Introduction DOES MONEY MATTER? or is this really even the question? And if it’s not, then why do we keep asking it, over and over (Hanushek, 1989-’97; Hedges, Laine and Greenwald, 1994, 1996)? For his most recent meta-analysis of this question, Hanushek (1997) combines the results of 90 studies regarding the relationship between schooling inputs and student outcomes. Betts (1996) uses approximately 60 related studies similarly in his assessment of the relationship Politics of the Production-function Draft as of: 01/17/99 Page 3 of 34 between school spending, spending-related factors and student earnings. Yet, while these meta-analyses provide us with an insightful overview of existing educational productivity research, and a relative balance of political perspectives, we seem to be covering little new ground with regard to understanding the system of schooling as a whole. Structurally, most studies of educational productivity consist of some statistical test of the effects of a selected set of schooling inputs (S) and student inputs (X) on student outcomes (Q). These tests are most often performed within a standard, deductive, hypothesis-testing framework. In addition to typical production-function analysis, there are a number of other methods for which the end result is quite the same. Among these methods are rate of return analyses (Betts, 1996) and cost-function analyses (Duncombe and Miner, 1996). At the most basic level, each of the methods rely on the determination of some mathematical relationship between spending and learning or spending and earning. When it comes to whether money matters, it is quite easy to see how there can be both political and personal incentives for making arguments toward one side or the other of the issue. While it is expected that the cumulative outcomes of our biases will result in a balanced pool of useful information, eventually revealing a middle-ground, the tendency has been for the pool of results to become more polarized over time. How do we go about breaking the gridlock on Politics of the Production-function Draft as of: 01/17/99 Page 4 of 34 this question? And, more importantly, should our efforts continue to be so narrowly focussed on the many permutations of this question? Perhaps our problems lie more deeply in the processes of educational research itself. Perhaps the methods by which we generate and subsequently test the “important questions of today", themselves, are too restrictive. Potential Pitfalls of Exclusively Deductive Processes As researchers in the social sciences, we find ourselves tightly bound to testing the rationality of existing and proposed theories. A vast amount of our research consists of statistical hypothesis tests based on some theoretical framework. Research lacking such a framework is unlikely to be granted credibility in the educational research community. Ultimately, in quantitative analyses, these theoretical frameworks are operationalized into formal mathematical models, equation types, and even functional forms. But, where do these theories and models come from? The research process is designed such that new theories are intended to evolve from novelties in previous findings, that these new theories will be validated in subsequent studies that may themselves reveal new novelties, and so on. Two distinct problems may inhibit this process. First, ongoing research on a 1 Median Voter Model, Political Support Maximization Model etc. 2 Production-Function etc. Politics of the Production-function Draft as of: 01/17/99 Page 5 of 34 given theoretical framework may at some point become stagnant, revealing little novelty in its findings, yet persisting nonetheless, “spinning the wheels of the research process. ” Second, the selection and combination of previous findings toward the generation of novel ideas must occur solely within the mind of the researcher(s). Statistical Methods: Technical Limitations and Potential Political Influence Beyond the formation of the research question is a limited set of statistical methods. Linear correlation and multiple linear regression analysis are among the methods that lend themselves to traditional statistical hypothesis testing. The basic objective, in either case, is to determine whether the prescribed inputs and outcomes share a statistical relationship (generally linear) other than “0. ” Yet, for statistical hypothesis testing to occur, two key a priori decisions must be made: 1) the parameters to be tested, and 2) the shape of the frame (functional form) to be applied to the relationship. While these issues are seemingly technical, both parameter selection and determination of functional form are susceptible to the influence of personal and/or political biases. The question of specific parameters to be tested is an applied extension of the research question. Among the important considerations are: 1) how to 3 Linear, Logarithmic, Exponential, Multiplicative 4 with the usual acceptable probability of being wrong set at 5%. Politics of the Production-function Draft as of: 01/17/99 Page 6 of 34 measure the constructs in question and 2) how to organize the parameters into an equation structure that will provide us with the appropriate test statistics to identify their significance. Verstegen (1997) points out that even "order of entry" can play a significant role in discriminating between the effects of school and non-school inputs. While it is quite easy to see how parameter selection can be used to promote political biases regarding spending, the political undertones of functional forms are somewhat more elusive. Cohn & Geske (1990) note: “A linear relationship between X inputs and the Q outputs would be empirically valid to the extent that the curvature of the total output function is only mildly violated by employing a linear approximation. ” A common economic assertion with regard to shape of the “actual curve” is that “the education production, like all wellbehaved production-functions, is subject to diminishing returns”(Betts, 1996). This behavior is generally well captured (e.g. displays a significant, positive coefficient less than 1.0) by applying a log-log specification of wages relative to per pupil spending (Betts, 1996). Others have replaced the log of spending with a 5 p. 166 6 It is somewhat ironic that we consider a production to be "well behaved" based on the extent to which it conforms to our theoretical expectations and functional forms as this assessment assumes that we truly know how a production-function is supposed to behave. 7 p. 163 Politics of the Production-function Draft as of: 01/17/99 Page 7 of 34 quadratic function, achieving a similar interpretation (Johnson & Stafford, 1973). While a significant parameter under either of these circumstances signifies a statistically adequate fit, it does not necessarily signify the best, or even most appropriate fit. Nor are we likely to find the best fit by these methods. In particular, where patterns of diminishing returns are expected, placing a rigid frame (predefined functional form) over a data set is unlikely to reveal local optima, or important decision points. In rate of return studies, where determinations are made along a log-log production-function curve, results at different levels of inputs can be nearly as misleading as the linear specification. The differential rates of response of outcomes to changes in inputs in a priori nonlinear specifications can have serious political implications. For example: At what point along the production-function curve does spending more yield negligible performance returns? The answer to this question varies depending on the shape of the curve which will vary depending on the functional form selected. New Approaches to Modeling To synthesize the issues presented in the previous section, the key problem 8 It is important to note that differential treatment of inputs and outcomes in the same model may compromise the actual relationship between the variables. 9 At its most basic level, the significant parameter simply signifies that the fit is better than “nothing. ” Mathematically, the only difference that exists in applying the log-log specification is that we are forcing a slightly bent rigid frame onto the data rather than an entirely Politics of the Production-function Draft as of: 01/17/99 Page 8 of 34 with using purely deductive processes for studying systems as complex as our education system is that we must assume that we actually understand how the system works, by way of existing theory, before we even begin our analyses. In light of the lack of novelty of our recent findings, and continued indecision regarding the role of fiscal resources, perhaps we should humbly begin to recognize our ignorance, take a step back, and ask the more basic question: ”What makes schools tick? ” This, however, is a dauntingly inductive question and one for which we have not necessarily devel