A Measurement Model for Synthesizing Multiple Comparative Indicators: The Case of Judicial Independence

We present a new cross-national measure of de facto judicial independence, which is available for 200 countries from 1948-2012. To do so, we introduce a statistical measurement model for uncovering latent concepts commonly encountered in time series, cross-sectional analyses in comparative politics and international relations. Our approach addresses unique challenges that arise in these data: temporal dependence in the observed and unobserved variables, conceptual boundedness in the latent quantity, and substantial missing data and measurement error in the observable indicators. The resulting measures match a common conceptual definition of independence with greater reliability than existing alternatives. The model is extensible to many concepts in comparative politics and international relations. Introduction The concept of judicial independence has been invoked as a key causal variable across comparative political science for outcomes ranging from regime stability to economic development and the protection of human rights (e.g., Gibler and Randazzo, 2011; La Porta et al., 2004; North and Weingast, 1989). Researchers have also sought to understand the determinants of judicial independence (e.g., Ginsburg, 2003; Helmke, 2005; Vanberg, 2005; Hayo and Voigt, 2007; Clark, 2010; Carrubba, Gabel and Hankla, 2008); and, in light of its importance to various aspects of human welfare, the international community spends considerable resources each year promoting judicial reform and tracking its success (Carothers, 2006). Despite this attention, researchers have continued to face vexing measurement challenges that complicate efforts to answer even very basic questions. The central obstacle is that judicial independence—like many concepts in comparative politics and international relations—is not directly observable. Whether we care to explain judicial independence, track its evolution over time, or evaluate its effects on human welfare or state choices in the international system, we must do a better job measuring it. In this paper, we propose a solution to the measurement problem using a statistical measurement model that systematically combines information from multiple observable indicators into a single unified measure. We introduce a dynamic bounded graded response item response theory (IRT) model for time series, cross-sectional (TSCS) data that is applicable to a wide variety of latent concepts in the study of international relations and comparative politics—e.g., level of democracy, degree of corruption, or military resolve. As when seeking to estimate any latent variable, we appeal to features of the world that we can observe to infer characteristics we cannot (Treier and Jackman, 2008, 203). The term graded response in the context of item response theory refers to models for outcome variables that are measured at the ordinal level (Johnson and Albert, 1999, ch. 7). 2 There are core differences between the assumptions underlying traditional IRT models, and the process that generates indicators of latent concepts in comparative TSCS research, which together necessitate a new approach. Since consecutive years’ observations are not independent, our model constrains the latent variable to trend smoothly within countries over time. The model is sensitive enough to uncover both gradual changes and abrupt shifts in the latent characteristic from year to year. It seamlessly accommodates indicators with varying amounts of data missingness. Our model further permits the calibration of extant indicators, so that a researcher who wishes to use any one of them—rather than our scale—will be able to compare and align each indicator’s measurement levels to any of the others. Finally, it constrains estimates of the latent variable to lay on a bounded interval. This not only reflects the idea that many comparative concepts themselves have natural boundaries; but also that countries at the extremes of the scale should be the easiest ones to measure. Traditional IRT models that assume an unbounded scale place the greatest estimation uncertainty around countries at the ends of the scale, which we consider unrealistic in a comparative setting. The model that we introduce will enable researchers to uncover the common element in any series of TSCS indicators of a latent variable of interest, assess which of these factors are more or less informative, and characterize the uncertainty in the resulting inferences. Yet as we emphasize, the successful use of our model requires that researchers first present a theoretically valid conceptualization of how the latent variable is expected to manifest, and why. As such, our model not only does not replace careful theoretical argumentation—but in fact it depends upon it. We apply our model to construct a unified measure of judicial independence that is available for 200 countries over 65 years from 1948 to 2012. The latent variable estimates draw upon a series of direct and indirect indicators of judicial independence collected by Feld and Voigt (2003); Howard and Carey (2004); Cingranelli and Richards (2010); Marshall and Jaggers (2010); Keith (2012); The PRS Group (2013); Ŕıos-Figueroa and Staton (2013) and Johnson, Souva and Smith (2013). We also provide estimates of uncertainty for each 3 latent value. In a supplementary appendix, we describe some current and potential research uses for the new measure, and illustrate the application of our model to the measurement of regime type, which highlights advantages of our approach in comparison to the scaling model proposed by Pemstein, Meserve and Melton (2010). Measuring Judicial Independence Globally As with many concepts in the social sciences, there is disagreement over the precise definition of judicial independence (see discussions in Keith, 2012; Burbank and Friedman, 2002). Yet the notion is far from “essentially contested,” as in the case of the rule of law (Waldron, 2002). Scholars generally draw two broad sets of conceptual distinctions. The first relates to the difference between independent judging in practice (de facto independence), and the existence of a set of formal institutions—such as fixed budgets or cumbersome removal procedures—which are thought to provide incentives or constraints on independent judging (de jure independence). The second distinction directly concerns the independence concept itself. In one common approach, judicial independence is conceived of as “autonomy,” where a judge is thought to be independent to the extent that her decisions only reflect her sincere evaluation of the legal record, i.e., the decision process is free from undue external influence, especially from government (Kornhauser, 2002; Rosenn, 1987). Another approach requires that an independent judge not only be autonomous, but that she can expect her decisions to be implemented properly, especially by sitting governments. In this second sense, judicial independence is conceptualized as “power”—judges are not only autonomous, they are influential (Cameron, 2002). Conceptual debates are not restricted to these considerations. For example, we might attempt to distinguish among judges who are independent from external actors, but not from actors within the judiciary, or vice versa. There is also a question of whether it is possible to conceive of judges as independent even though the judiciary, considered as a 4 We seek to measure the power concept of de facto independence. The power concept is of considerable interest on its own, but there are, in addition, theoretical reasons to question whether it is feasible to measure autonomy without simultaneously measuring power. Expectations about the compliance process can influence decision-making, in the sense that an inability to fully control the implementation of orders can undermine decisional autonomy (e.g., Epstein and Knight, 1998; Vanberg, 2005; Carrubba and Zorn, 2010). Whether independence is conceived of as autonomy or power, the fact that neither are directly observable poses significant challenges for accurate measurement. One methodological response has been to look for specific instances of judicial behavior (e.g., the choice of a peak court to invalidate a law) that under the right conditions, and under a particular theoretical logic, can indicate a form of independence. Although plausible for a limited set of states, there is nothing close to a representative sample of judicial decisions for all countries, much whole, is dependent (Ferejohn, 1998). Our focus is on the judiciary as a whole, with respect to external forms of independence. That said, the core measurement modeling arguments we develop, and specifically the point that judicial independence is latent, would apply to alternative conceptions as well. For example, Ŕıos-Figueroa (2007) uses decisions against the PRI in Mexico to evaluate the “effectiveness” of the Mexican Supreme Court. To be effective, a court must first be willing to challenge rulers, which depends on having an “independent” view of the record. Similarly, Helmke (2005) uses Supreme Court decisions against the sitting government in Argentina to evaluate the extent to which judges’ decisions are disconnected from government preferences (i.e. independent) over case outcomes. Of course, simply observing a court strike down a law does not imply that the court is acting “independently. ” Whittington (2005) notes that leaders can prefer to use their courts to strike down policies that they themselves did not wish to pass in the first place. This does not imply, however, that decisions say nothing about independence. Armed with a strong theoretical argument that identifies conditions under which judges ought to confront strong pressures that undermine independence, observing decisions can be informative. 5 less over time and for many courts in a state’s system. Nor does the approach generalize across countries in systematic and consistent manner. Scholars in need of