LOCATING TDs IN POLICY SPACES: WORDSCORING DÁIL SPEECHES

This paper adapts a new technique for the computerized analysis of political texts (Laver, Benoit and Garry, 2002), previously used to analyse party manifestos, to the analysis of speeches made in a legislature. The benefits of computerized text analysis come from the ability to analyse for the first time complex and daunting electronic sources of text, such as the parliamentary record. This allows the systematic estimation of the policy positions of individual political actors, with huge benefits both for theory development and empirical analysis. In this paper, the technique is used to analyse all 58 English language speeches made in the October 1991 confidence debate on the future of the incumbent Fianna Fáil – PD coalition. The task was to use the words spoken in the debate to locate every one of the individual speakers on a “proversus anti-government” dimension. The purpose was, first, to examine the validity of computerised text analysis when applied to legislative speeches and, second, to answer substantively interesting positions about the positions of individual Irish legislators in 1991. The results vindicate the use of computerised analysis in the context of legislative speeches and locate all speakers in the 1991 debate in a substantively interesting policy space. * Michael Laver’s work on this paper was carried out while he was a Government of Ireland Senior Research Fellow in Political Science. Locating TDs in policy spaces / 2 INTRODUCTION New developments in computational text analysis within political science have been made possible by recent huge improvements in computing power. These take political science content analysis well beyond the traditional very labour intensive hand coding of political texts, as conducted for example by the influential Manifesto Research Group (MRG), now the Comparative Manifestos Project (CMP) (Budge et al., 1987; Laver and Budge 1982, Klingeman et al, (1994). While grounded in a very specific “saliency theory” of party competition that has not found widespread support within the profession, the CMP data have been widely used by many who have sought time-series data on party policy positions in postwar western Europe. Until recently this has in large part been because of the phenomenal effort that would have been needed to recode all of the documents involved in a manner more suitable to the application at hand. Computer coded content analysis, however, now offers the prospect of fast and effective coding and recoding of documents according to the research needs of a specific analyst, with no need to resort to an existing dataset simply because of the huge costs involved in doing otherwise. Successful implementations of computerised text analysis, replicating completely independent data sources, have recently been published by a number of authors (Laver and Garry, 2000; Kleinnijenhuis and Pennings, 2001; Garry, 2001; de Vries et al. 2001; Bara, 2001). Nearly all published work on the computer coding of political texts has focused on the analysis of party manifestos, for several important reasons. First, for all the reasons that motivate the CMP project, party manifestos are considered important substantive statements of the policy positions of political parties, and are therefore of great research value to political scientists. Second, because the enterprise of coding these manifestos by hand is extremely resource-intensive, a successful method for computerised coding of manifestos promises enormous gains simply from a practical standpoint. Finally, because computerised methods for analyzing political texts are relatively new, it has made sense to assess the validity of the Locating TDs in policy spaces / 3 new techniques by comparing the results with those obtained using more traditional methods of scoring the policy positions of texts and the parties that issued them. As the profession becomes increasingly confident and experienced in the methodology of computer coding, however, it becomes possible to apply computerised coding to other forms of political texts, and therefore to tackle new substantive problems—including ones that would be very difficult to approach without having access to some form of fast, cheap, effective and reliable text analysis. In this paper we present an application of computerised analysis of political texts that goes beyond the scoring of election manifestos issued political parties. Here our focus is on texts generated by individual legislators in the form of speeches made in the legislative debates. We do this using a new probabilistic “word-scoring” method for computerised text analysis that has been developed and found effective by Laver, Benoit and Garry (2002), applying this to the analysis of the speeches made by Irish TDs during a long and acrimonious debate of confidence, held in October 1991, on the future of the incumbent Fianna Fáil-PD coalition government. Our aim is to estimate the positions of individual Irish legislators in a common policy space. Methodologically, this allows us to evaluate the use of computer coding in a context where it has the potential to generate huge payoffs. Substantively, it allows us to explore interand intra-party differences in Ireland at the level of the individual legislator, and specifically to looks for potential splits within both the coalition government and the opposition. In what follows we first outline the word-scoring technique we use and discuss issues arising from applying this to legislative speeches rather than party manifestos. Next we briefly describe the texts we analyse. We then present and discuss the results of our analysis both methodologically and substantively, concluding by drawing lessons for future uses of computerised text analysis in investigating the policy positions contained in legislative speeches. Locating TDs in policy spaces / 4 THE WORD-SCORING APPROACH TO COMPUTERISED TEXT ANALYSIS “Traditional” techniques of computerised text analysis essentially count the frequencies of words found in predefined coding “dictionaries”. These dictionaries are lists of key words deemed a priori by the analyst, as a matter of subjective judgement guided by empirical exploration, to be associated with particular policy positions. The relative frequencies of words observed to fall into particular categories are then subjected to some form of scaling technique in order to derive estimates of the policy positions of the texts under analysis. A recent successful application of this approach to estimating the economic and social policy positions of party manifestos in Britain and Ireland is described by Laver and Garry (2000), and has subsequently been implemented for German and Norwegian party manifestos by Garry (2001) and for Dutch and Italian manifestos, as well as Irish government declarations, by de Vries et al. (2001). An alternative dictionary-based approach, computer coding the CMP data and applying this to the European parliament, can be found in Pennings (2002). For recent essays in this area by the CMP itself, see Budge et al. (2001). While this technique works well it has two paradoxical disadvantages. First, despite the fact that it is a computerised technique it remains labour-intensive in that very considerable time and effort must be applied to developing an appropriate coding dictionary upon which to ground the analysis, in a situation in which changes in the political lexicon across time and context my render any given coding dictionary inappropriate. Second, this highly numerical technique remains ultimately subjective in the sense that the analyst typically has considerable freedom in the construction of the word lists that comprise the computer coding dictionary. Addressing these problems in an attempt to realise the full benefits of computer coding, Laver, Benoit and Garry (2002) have developed from first principles a probabilistic technique for coding political text that does not use predefined dictionaries and uses no subjective judgement calls by the researcher. This technique is described fully in Laver et. al (2002) but essentially involves the following. First, there is a preliminary analysis of a set of “reference” texts with well-known positions on the policy dimensions in which the analyst is interested. Locating TDs in policy spaces / 5 For example, Laver et al. (2002) use British party manifestos in 1992 as reference texts for an analysis of the policy positions of British party manifestos in 1997, and Irish party manifestos in 1992 as reference texts for an analysis of Irish party manifestos in 1997. The technique requires that there be independent estimates of the policy positions of the reference texts on the policy dimensions under investigation. Laver et al. take these independent estimates from expert surveys, but any independent estimates in which the analyst is confident – for example mass survey data or even prior hand-coded content analysis – would fulfil the same role. The computer analysis of the reference texts provides no new substantive information, but is used to calculate the matrix of “word scores” that replaces traditional coding dictionaries in the computerised analysis of new “virgin” texts in which the analyst is interested. This preliminary analysis of reference texts observes the relative frequencies of all words used in each text, allowing the calculation of the key quantity in the word-scoring approach. This is the conditional probability Pwr that the analyst is reading reference text r, given that s/he is reading word w. Using these conditional probabilities and the known positions of the set of reference texts on policy dimension d, it is possible to assign a “score” Swd on dimension d to every word w in the word universe of the reference texts. This score is in effect a conditional estimate of the position of any text on dimension d, given that the analyst is reading word w. Given the power of modern computers, the