More than Votes? Voting and Language based Partisanship in the US Supreme Court
Authors: Biaoyan Fang
Trevor Cohn
Timothy Baldwin
Lea Frermann
Venue: EMNLP 2023 Findings (2023)
Keywords: Fairness and Bias; Sociolinguistic; Cultural Analysis; Partisanship Analysis
TL;DR: We propose a framework to investigate language partisanship of justices based on spoken text in court and show a significant correlation between language partisanship and voting ideology.

Abstract: Understanding the prevalence and dynamics of justice partisanship and ideology in the US Supreme Court is critical in studying jurisdiction. Most research quantifies partisanship based on voting behavior, and oral arguments in the courtroom --- the last essential procedure before the final case outcome --- have not been well studied for this purpose. To address this gap, we present a framework for analyzing the language of justices in the courtroom for partisan signals, and study how partisanship in speech aligns with voting patterns. 
Our results show that the affiliated party of justices can be predicted reliably from their oral contributions. We further show a strong correlation between language partisanship and voting ideology.

More than Votes? Voting and Language based Partisanship in the US Supreme Court
Biaoyan Fang
Trevor Cohn
Timothy Baldwin
Lea Frermann"
The University of Melbourne, Australia MBZUAI
CSIRO Data61
(biaoyan, tcohn, tbaldwin, (arrrnrrnneremmmmnnnnnmmmmo edu. au
Abstract
1.00
1.0
Understanding prevalence and dynamics of justice partisanship and ideology in the US
Languing particonchip
0.75
Supreme Court is critical in studying jurisdic- tion. Most research quantifies partisanship based on voting behavior, and oral arguments the courtroom - the last essential proce- dure before the final case outcome have been well studied for this purpose. To address this gap, we present framework for analyzing the language of justices in the courtroom for partisan signals, and study how partisanship in speech aligns with voting patterns. Our results show that the affiliated party of justices be predicted reliably from their oral contributions. We further show a strong correlation between language partisanship and voting ideology."
MO scorss
0.50
-0.5
Liberal
sartisarchip
1960
1970
1980
1590
2000
2010
2020
Figure 1: The US Surpreme Court overall partisanship between 1955 and 2019 as approximated by voting- based measures (MQ scores; Martin and Quinn (2007)) and our language- based framework.
beliefs and partisanship (Gentzkow et al., 2019; Demszky et al., 2019; Thomas et al., 2006; Bergam et al.. 2022; Vafa et al., 2020).
1 Introduction
The study of partisanship and ideology has been an important topic understanding the US legal system (Jacobi and Sag, 2018; Devins and Baum, 2017; Bonica and Sen, 2021; Doerfler and Moyn, 2021). Most research has focused on justice voting patterns (Bonica and Sen, 2021; Martin and Quinn, 2002, 2007; Epstein et al., 2007a; Bailey, 2013) and behavior in court, e.g. the frequency of inter- ruptions (Epstein et al., 2010) questions (Epstein and Weinshall, 2021). Despite their core role in legal determination, the content in terms of whether oral arguments portray partisan values has received less attention (Bergam et al., 2022).
In this paper, we ask to what extent do oral ar- guments reveal justice partisanship. We propose framework to analyze partisanship in justices' oral arguments in the Supreme Court of the United States (SCOTUS), We classify justices as Democrat or Republican based the political party of their nominating president, following common practice in the law literature (Devins and Baum. 2017: Yalof, 2001). We cast language-based partisanship pre- diction as classification task and show that our models can reliably predict partisanship. suggest- ing that justices project their affiliations in court arguments.
In political discourse in particular, word choice is nuanced to convey specific messages (Lakoff, 2010; Jarvis, 2004; Robinson et al., 2017; Jensen al., 2012; Dutta et al., 2022). For instance, Re- publicans tend to prefer the baby over fetus to emphasize their belief that human rights begin at conception (Simon and Jerit, 2007), reflecting their
We go on to derive language-based partisan- ship scores from our models, and ask How does language-based partisanship align with estab- lished measures of -bseg ideology? We show that our scores correlate well with established yoting-based measures (Figure Equipped with this layer of validation, we move on to more nu- anced analyses of language-bart based partisanship: (a) of the overall court, (b) of individual justices, and
"Now at Google DeepMind, 'The dataset and code are available from: http::/ghhh...cmmmiioa///SSSSSSS

---

Model
Train
Dev
Test
Cases
6,663
Random
0.50 +: 0.00
0.50 +: 0.00
0.50 - 0.00
Justices
35
Majority
0.37 1 0.00
0.37 0.00
0.37 1 0,00
Turns
756,018
BERT
0.87 + 0.02
0.83 0.01
0.83 = 0.01
Averuge #words per turn
27.24
BERT (+-ear)
0.91 0.01
0.85 + 0.00
0.85 = 0.01
Instances
104,498
Republican
62,497
(59.8%)
Table 2: Macro-F1 scores for affiliated party pre-
Democratic
42,001
(40.2%)
diction for Random and Majority baselines (major- ity=Republican, cf. Appendix B), as well as fined- tuned BERT classifiers based on justice instances only (BERT), or an additional year tag (BERT year))
Table 1: Core dataset statistics, where "Turns" are in- dividual justice contributions (of arbitrary length) and "Instances" are the individual data points presented to the models, i.e., 510-token segments with sliding 255 token window from all turns of one justice in one case.
Model
Train
Dev
Test
Random 0.49
0.47
0.50
Majority 0.40
0.42
0.34
(c) of landmark cases over time.
BERT 0.88 + 0.02
0.80 1 0.01
0.70 + 0.01
2 Methodology
Table 3: Macro-FI scores for affiliated party prediction based chronological data split. We show the averaged results of BERT over three runs with different random seeds.
Data We use a subset of the Super-SCOTUS dataset (Fang et al., 2023b), which contains tran- scripts of SCOTUS oral arguments from 1955- 2019 (Chang al., 2020).2 We filter out cases where hearings stretch over more than one year. We additionally remove non-linguistic indicators, e-g., [laughter], and mask all person names with a BERT NER model (Devlin et al., 2019) in order to focus our models on linguistic indicators rather than mentions of individuals with potential party affiliations.*
Classification To understand to what extent their language reveals justice's ' party affiliation (Demo- crat vs. Republican), we formulate partisanship pre- diction as binary classification task, and fine-tune the BERT-base mode! (Devlin et al., 2019) to pre- diet the affiliated party of a justice from their oral contributions in a single court hearing. For each iteration of cross validatio,,, we fine-tune BERT on the training fold for 30 epochs, with batch size of 16, and dropout rate of 0.3, and select the best model based on the Macro-F1 score on the development fold.
We concatenate all turns from individual jus- tice in single court hearing, to derive multiple instances with a maximum of 510 tokens and a sliding window of 255 tokens. To retain only in- formative instances, we further remove those that have less than 50 tokens. We finally randomly separate the resulting instances into 10 folds for cross- vooss-valiotion.. Partisanship and polarization are dynamic phenomena, responding to shifts in topics and political landscapes. We equip our models with a notion of time by adding a special [year] tag to each instance, flagging its year of origin.
3 Affiliated Party Prediction
We first evaluate our framework intrinsically, ask- ing To what extent do oral arguments reveal jus- tices' party affiliation? by predicting justices' affil- iated party from their court contributions.
Table 2 shows affiliated party prediction perfor- mance of our classifiers." The large boost for the BERT models over random and majority baselines, as well as the high overall performance, shows that the wording of the oral arguments reveals justices' party affiliation as a proxy for their partisanship. Furthermore, models benefit from temporal tags (+year), indicating that relevant linguistic signals drift over time.
We obtain the reference party affiliation for each justice as the party (Republican or Democrat) of the President who nominated the justice, which correlates strongly with the self-reported party of the justice and has been shown to be associated with the interest of the nomination party (Devins and Baum, 2017: Shipan, 2008; Yalof, 2001). The statistics of the resulting dataset are shown in Ta- ble 1.
To further investigate the impact of temporal lan-
hhups://convoki..com/// wwm.eel..duuoocmmenaatinnnnunnuu "htps://hugginggace http://gggaac..ccsiiiioi.mmm biidddiimmeet-base--ee 'See Appendix A for data construction details.
'htts://hugginnffee.cooooo wwwpeeee..bbbbbbbbeecoee
"Distributions of gold vs. predicted labels over one fold are provided in Appendix B.

---

LR
SVM
scores established measures of ideologg obtained from justices' voting behavior (Martin and Quinn, 2002, 2007; Bonica and Sen, 2021). While par- tisanship and ideology are not identical, a strong correlation exists, particularly in the two-party sys- tem of the United States (Baum and Devins, 2019; Devins and Baum, 2017; Lupton et al., 2020).
BERT (off-the-shelf)
0.45 + 0.01
0.43 + 0.01
BERT (+ year)
0.29 T 0.06
0.32 1 0.06
Table 4: Speaker identification performance (Macro-F1) of logistic regression (LR) and SVM classifiers, taking as input off-the-shelf BERT embeddings or embeddings from our best-performing party classification model.
Having shown that our BERT models capture partisanship from court arguments (Section 3) and with grounded assumption that partisanship is reflected in (reasonably) local linguistic choice (Jarvis, 2004; Haddow and Klassen, 2006), we ob- tain a language partisanship score of a justice by averaging the predicted party probability for each of the justices' instances (Section 2). For example, to obtain the oral partisanship score of a given jus- tice in given year, we average the predicted party probability of all related instances from the given justice in the given year. Similarly, the language partisanship score for a case (year) is calculated by averaging the predicted party probability of all instances from all justices in that case (year), re- spectively.
guage shift (Hu et al., 2019; Ding al., 2023) on our oral partisanship prediction task, we train and test our model with chronological data split. We split the data into non-overlapping temporal spans, with the training covering all cases from 1955-
2000 (N = 5336), development set (2001-2009,
N = 671) and test set (2010-2019, N = 656). The results in Table 3 show that, compared to the random split (Table 2), BERT performs worse on the development and test sets. In line with prior work, this suggests that characteristics of partisan language change over time. Incorporating language drift into predictive models is a fertile arca for fu- ture work.
A natural question is whether our model indeed captures partisanship, or rather language idiosyn- cracies of individual justices. To test this, we use our trained models to predict speaker identities. Specifically, we extract the final layer embedding of the (CLS) token and train logistic regression (LR) and support vector machine (SVM) models for speaker prediction (i.e., a 35-way justice clas- sification task). We compare against off-the-shelf BERT-base-cased embeddings, with no fine-tuning, We use instances with year tags as input, based on the best-performing setup in Table 2.7 Our fine-tuned model performs worse than off-the-shelf BERT embeddings on the task of speaker predic- tion (Table 4). As such, we conclude that our fine- tuned model representations abstract away from in- dividual speaker characteristics to representations that indeed capture properties indicative of party affiliation.
We use MQ scores (Martin and Quinn, 2002), widely used and validated measure of the idcol- ogy of a court or individual justices, derived from voting outcomes. MQ scores are estimated with a dynamic Bayesian item response model (West and Harrison, 2006), which infers the latent 'ideal point' of justice, i.c., their idcological standpoint on unidimensional scale (Liberal - Conserva- tive) based on their observed voting behavior and prior encouraging smooth change in ideology over time.
Court-level partisanship Figure I compares the overall SCOTUS ideology based voting behav- ior (MQ scores, orange) and language (blue) over time, from 1955 to 2019. We observe a strong cor- relation of both measures across time (Pearson's 's r = 0.611, p = 6.5c-8), indicating that partisan- ship in oral arguments reflects the voting behavior in court. Similar observations have been made for more overtly-partisal domains such as political dis- course and votes in the US Congress (Gentzkow et al., 2019; Diermeier al., 2012).
4 Justices' language reflects their voting
We next ask How does language-based partisan- ship align with established measures of voting- based ideology? To do so, we derive a partisanship score from our predicted party affiliation probabili- ties. We then relate our language-baur based ideology
Justice-level partisanship We further investi- gate language partisanship and voting ideology at the individual justice level. Justices in recent years have shown clearer alignment with party affilia-
'Appendix C. shows that patterns are consistent for inputs without year tags.
"http://mqscorrssu www.ss.............m..........ea wtt.aacccccccc/aaarr...o cdu/mcasurres.phpppp

---

Conservative
Kistett
Republican Democrat
JGRoberts
0.8
Language partisanship
0.7
1 Vi
Language partisanship
Grrsouthh
0.6
Skibans
N MO scores
0.5
142
EMKaaaaaaugh
EKagen
0.4
Ginsburgg
0.3
5Sotomayyr
0.2
Liberal
22-
Sd
1990
1995
2000
2005
2010
2015
2020
Figure 2: Comparison of voting-based (MQ: dashed lines) and language partisanship measures (LP: solid lines) of individual justices over their tenure. Different colors denote different justices. The bold-dashed line indicates the neutral point.
MQ scores
Figure 3: Drift of justice partisanship comparing their first year (small dot) and full tenure (large dot). Blue dots correspond to Democrat- and red dots to Republican-nominated nominated justices. The error bars on large dots indicate the standard deviation of justice partisan- ship. i.c., MQ scores and language partisanship in corre- sponding axes, across their full tenure.
tions in terms of their voting patterns (Devins and Baum, 2017), and we ask whether this is reflected in their language, too. We present case study of the eight most recently appointed justices in our data set."
for language partisanship, we compare in Figure 3 how our eight most recent justices' voting ideology and language partisanship have drifted between their first year of tenure (small dots) vs. their entire tenure to date (large dots). The error bars indicate the standard deviation of partisanship scores (MQ on the x- and language- years of service. For most justices, the overall trend exceeds the SD interval, suggesting that it goes be- yond the typical year-to-year fluctuation. As before, MQ and language scores align well (justices clus- tering in the bottom left and top right quadrants), Compared to their first year, apppontt- justices (bluc) statistically significantly tend to be- come more liberal in both MQ scores and language partisanship over their tenure. The relative shift in language partisanship for Republican-appoin justices (red) is less pronounced, in terms of both language and MQ scores.
Figure 2 shows that Republican- afflliated/conservatiee justices (Roberts. Saalito, Gorsuch, Kavanaugh) are consistently separated by the neutral (thick, dashed) line from the Demo- crat/liberal justices (Ginsburg, Breyer, Kagan, Sotomayor). This holds for both language-based partisanship scores (solid lines) and MQ scores (dashed lines). For Democratic justices, MQ scores and language partisanship are aligned in tendency over their tenure, with a minimum of Pearson's 's r = 0.6 across justices. We do not observe such strong evidence in Republican justices, indicating that conservative justices project their values less directly in their speech. Language-based partisanship estimates tend to be more extreme than voting ideology. Notable examples are Roberts (Republican) and Breyer (Democrat), who is known to be a pragmatist whose decisions are often guided by real-life consequences regardless of pirty-lines." 10 possibly explaining the disparity between partisanship and MQ scores.
on the y-axis) over all
Case-level partisanship In high-profile cases, voting behaviors have more distinctively lined up along party lines in the more recent years (Devins and Baum, 2017). Correspondingly, we might ex- pect the gap between the average language partisan- ship in Democrat (blue) vs. Republican-affilian (red) justices to increase over time. We test this by looking at 14 high-profile cases between 1962 and 2014 (Richard Wolf, 2015)." !1 As shown in Fig-
Epstein al. (2007b) observed that, over their tenure, drift away from their first-year pref- erences, but with no certainty in what direction they will move. To study the same observation holds
We excluded the most-recently appointed justice Clarence Thomay due to data sparsity. The same comparison for all justices is provided in Appendix D. g_brreer
Full details of the cases are provided in Appendix E.

---

1.0
Conservative
guage partisanship aligns with voting ideology. We derived uunuuag--based partisanship scores from our validated models and compared them with MQ scores, an established ideology measure derived from voting patterns. We showed a strong corre- lation between language partisanship and voting ideology at the overall court, individual justice, and important case level over time (Section 4), indicat- ing that the oral arguments of justices do encode their political leanings. Moreover, our work re- veals nuanced differences in linguistic and voting behaviors, c.g., their respective tendency shift over justices' tenure (Figure 3), which further demon- strates the importance of studying SCOTUS from various perspectives.
Language particanship
0.8
0.6-
Republican Democrat
0.4
0.2
Liberal
G. 1962
NTTC 5, 1,193
1965
Lv. 1964.
US NL193
RIUC 1979
B G. 2000
Lv. 1,2002
Dc v. H 1,2007
cu v FEC 2008
NFI0 v s 5,2011
5 H. 2012222-
us v. W 201220222
v.v.1.000 2014
Figure Language Partisanship on important cases. We average the language partisanship of justices that were nominated by the same party, i.e. Republican (red) Democrat (blue). Dashed lines denote the estimated linear regression for the corresponding party.
More broadly, test the extent which lan- guage representations from large language mod- els capture nuanced nocio-cultural phenomena, by comparing predictions against corresponding be- havioral data sets. Here we focus on partisanship. building on related approaches for stance (Bergam et al., 2022) and counterfactual approaches to in- vestigate the effect of social speaker attributes like gender and seniority language use in the court- room (Fang et al., 2023a).
ure 4, language partisanship indeed becomes more polarized. Particularly, the language of Democrats, which in the 1960s occasionally crossed the neu- tral line to the Conservative side, has become more liberal over time. Additionally, the gap in language partisanship between the two groups of justices has increased, again confirming that justices' final voting behavior is reflected in their spoken court arguments.
Our proposed framework enriches the partisan- ship and idcology analysis with an additional di- mension to voting behavior, the spoken text, where we show a strong correlation between voting and spoken language. We hope that this could further spur analyses from other dimensions, such as parti- sanship of advocates (Patton and Smith, 2017), and amicus curiae (Sim et al., 2016), and take impor- tant parts of the legal process, e.g., further opinion- writing (Clark, 2009), into consideration.
5 Discussion
Partisanship affiliation along the liberal- conservative political spectrum is fundamental axis the political discourse. Automatically iden- tifying partisan language through NLP techniques enables large-scale analysis of political and public discourse and a better understanding of divisions and polarization. While prior work has mostly focused on explicitly partisan text, e.g., congress speeches (Jensen et al., 2012) or news outlets (Dutta et al., 2022), we study supreme court lan- guage, which, by definition, should be politically neutral, and ask does the language of justices in court reveal their party affiliation? We proposed an analysis framework showing that BERT-based classifiers can reliably predict the affiliated party of justices based on linguistic signals in their SCO- TUS arguments.
Limitations
Our work focuses on the Supreme Court of the United States, due a wealth of available re- sources and prior research to ground or results. Future work should extend our framework other political corpora, e.g., congressional records (Gentzkow al., 2019) and the federal eircuit," 12 as well as languages and their associated politi- cal/legal systems.
We did not exhaustively search for the best lan- guage representation, and other language models, e.g. RoBERTa (Liu et al., 2019), XLNNet (Yang et al., 2019), BART (Lewis et al., 2020) may
In line with research which has shown cor- relation between conversational content and final voting in political discourse (Lupton et al., 2020; Vafa et al., 2020; Bonica and Sen, 2021; Bergam et al., 2022), we further asked to what extent lan-
hhtp::///aauuuuuottts/////////aa oral-arguments/

---

lead to improved performance. Also, experiments with additional domain-specinn language models, e.g., LEGAL-BERT (Chalkidis et al., 2020), and prompt engineering (Trautmann et al., 2022) may achieve a further increase in performance.
It has been shown that pretrained language mod- els are biased (Delobelle et al., 2022: Nadeem al., 2021). Although our evaluation validates our re- sults against external sources, we acknowledge a possible impact of biases in language representa- tions (e.g., resulting from prevalence of liberal or conservative sources in the pre-training corpora).
The temporal drift of language itself (e.g. dif- ferent talking styles in court over time), as well as focus of topies (e.g. digital security only emerged in the past few decades), could be confounders in our partisanship analysis. Although we mitigate their impact by randomly splitting instances per year across folds, we acknowledge that these con- founders are worthy of further exploration. Our use case suggests itself as a testbed for causal model- ing approaches that control for confounders explic- itly (Feder et al., 2022).
References
Michael Bailey. 2013. today's court the most con- servative in sixty years? Challenges and opportuni- ties in measuring judicial preferences. The Journal of Politics, 75(3):221-88
Lawrence Baum and Ncal Devins, 2019. The com- pany they keep: How partisan divisions came to the Supreme Court. Oxford University Press.
We acknowledge that it's hard to obtain reliable features, i.e., important words that impact predic- tion results, out of large language models. Our analysis framework could be further enhanced by inspecting the highly-associatte words for the par- ties.
Noah Bergam, Emily Allaway, and Kathleen Mckeown. 2022. Legal and political stance detection of SCO- TUS language. In Proceedings of the Natural Legal Language Processing Workshop 2022, pages 265- 275, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
Adam Bonica and Maya Sen. 2021. Estimating judi- cial ideology. Journal of Economic Perspectives, 35(1):97-118.
Ethics Statement
Our models predict party affiliation and explicitly not voting outcomes. We neither attempt nor are able to do voting prediction based on our analy- sis framework, noting the severe ethical concerns it would raise (including, but not limited to, as- sociating individuals with predicted professional behavior of potentially low quality, that may reflect detrimentally on their reputation).
Hias Chalkidis, Manos Fergadiotis, Prodromos Malaka- siotis, Nikolaos Aletras, and Ion Androutsopoulos. 2020. LEGAL-BERTT The muppets straight of law school. In Findings of the Association for Com- putational Linguistics: EMNLP 2020, pages 2898- 2904, Online. Association for Computational Lin- guistics.
Jonathan P. Chang, Caleb Chiam, Liye Fu, An- drew Wang, Justine Zhang, and Cristian Danescu- Niculescu-Mizil. 2020. ConvoKit: A toolkit for the analysis of conversations. In Proceedings of the 21th Annual Meeting of the Special Interest Group on Dis- course and Dialogue, pages 57-60. 1st virtual meet- ing. Association for Computational Linguistics.
Our analysis framework aims to understand how oral court arguments reveal the affiliated party of justices in the Supreme Court of the US. Although our study covers the analysis of individual justices, we make no presumptions of the values or beliefs of justices beyond what is in the public domain, nor do we target individual justices. All our analyses aggregate predictions over several cases per justice and we do not aim to predict or overly rely on indi- vidual votes or contributions of individuals. This research aids in understanding the US legal system, and we strongly advise against over-interprette of the results in terms of behaviors of individual justices.
Tom S Clark. 2009. Measuring ideological polariza- tion on the United States Supreme Court. Political Research Quarterly, 62(1 6261)114
Pieter Delobelle, Ewoenam Tokpo, Toon Calders, and Bettina Berendt. 2022. Measuring fairness with bi- ased rulers: A comparative study on bias metrics for pre- trained language models. Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies. pages 1693-1706. Seattle, United States. Association for Computational Linguisties.
Our case study focuses on a subset of historical oral arguments from SCOTUS. Although it covers most publicly-avillall cases, this does not reflect the full history of SCOTUS, nor represent the cur- rent state of the court.
Dorottya Demszky, Nikhil Garg, Rob Voigt, James Zou, Jesse Shapiro, Matthew Gentzkow, and Dan Juraf- sky. 2019. Analyzing polarization in social media:

---

Method and application to tweets on 21 mass shoot- ings. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technolo- gies, Volume 1 (Long and Short Papers), pages 2970- 3005, Minneapolis, Minnesota, Association for Com- putational I. Linguistics.
Biaoyan Fang, Trevor Cohn, Timothy Baldwin, and Lea Frermann. 2023a. It's not only What You Say, It's also Who It's Said to: Counterfactual Analysis of In- teractive Behavior in the Courtroom. In Proceedings of The 1 13th International Joint Conference on Natu- ral Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Com- putational Linguistics, Nusa Dua, Bali. Association for Computational Linguistics.
Neal Devins and Lawrence Baum. 2017. Split definitive: How party polarization turned the Supreme Court into a partisan court. The Supreme Court Review, 2016(1):301365.
Biaoyan Fang. Trevor Cohn, Timothy Baldwin, and Lea Frermann. 2023b. Super-SCOTUS: A multi-sourced dataset for the Supreme Court of the US, In Pro- ceedings of the Natural Legal Language Processing Workshop 2023, Singapore. Association for Compu- tational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 1 (Long and Short Papers), pages 4171-4186 Minneapolis, Minnesota. Association for Computational Linguistics.
Amir Feder, Katherine A. Keith, Emaad Manzoor, Reid Pryzant, Dhanya Sridhar, Zach Wood-Dought Ja- cob Eisenstein, Justin Grimmer, Roi Reichart, Mar- garet E. Roberts, Brandon M. Stewart, Victor Veitch, and Diyi Yang. 2022. Causal inference in natural lan- guage processing: Estimation, prediction, interpreta- tion and beyond. Transactions of the Association for Computational Linguistics, 10:1 138 1158.
Daniel Diermeier, dean-François Godbout, Bei Yu, and Stefan Kaufmann. 2012. Language and ideology Congress. British Journal of Politicul Science, 42(1))31-55
Matthew Gentzkow, Jesse M Shapiro, and Matt Taddy, 2019, Mcasuring group differences in high- dimensional choices: method and application to con- gressional speech. Econometrica, 87(4):
Xiaohan Ding, Michael Horning, and Eugenia H Rho. 2023. Same Words, Different Meanings: Semantic Polarization in Broadcast Media Language Forecasts Polarity in Online Public Discourse. In Proceedings of the International AAAI Conference on Web and Social Media, volume 17, pages 161-172.
Rodney S Haddow and Thomas Richard Klassen. 2006. Partisanship, globalization, and Canadian labour market policy: four provinces in comparative per- spective, volume 25. University of Toronto Press.
Renfen Hu, Shen Li, and Shichen Liang. 2019. Di- achronic sense modeling with deep contextualized word embeddings: An ecological view. In Proceed- ings of the Annual Meeting of the Association for Computational Linguistics, pages 3899-3908, Flo- rence, Italy. Association for Computational Linguis- tics.
Ryan D Doerfler and Samuel Moyn. 2021. Democratiz- ing the Supreme Court. Cal. L. Rev., 109:1703.
Sujan Dutta, Beibei Li, Daniel S Nagin, and Ashiqur R KhudaBukhsh. 2022. A Murder and Protests, the Capitol Riot, and the Chauvin Trial: Estimating Dis- parate News Media Stance. In Proceedings of the Thirty-First International Joint Conference Artifi- cial Intelligence, IJCAI-22, pages 5059-5065.
Tonja Jacobi and Matthew Sag. 2018. The new oral argument: justices as advocates. Notre Dame L. Rev. 94:1161.
Lee Epstein, William M Landes, and Richard Posner. 2010. Inferring the winning party in the Supreme Court from the pattern of questioning at oral argu- ment. The Journal of Legal Studies, 393244333-467
Sharon E Jarvis. 2004. Partisan patterns in presidential campaign speeches, 1948-2000. Communication Quarterly, 52(4):403 419.
Jacob Jensen, Suresh Naidu, Ethan Kaplan, Laurence Vilss-Samson,, David Gergen. Michael Zuckerman, and Arthur Spirling. 2012. Political polarization and the dynamics of political language: Evidence from 130 years of partisan speech [with comments and discussion). Brookings Papers on Economic Activity, pages 1-81.
Lee Epstein, Andrew D Martin, Kevin M Quinn, and Jeffrey A Segal. 2007a. Ideological drift among Supreme Court justices: Who, when, and how impor- tant. Nw. UL Rev, 101:1483.
Lee Epstein, Kevin Quinn, Andrew D Martin, and Jef- frey A Segal. 2007b. On the perils drawing infer- ences about Supreme Court justices from their first few years of service. Judicature, 91:168.
George Lakoff. 2010. Moral politics: How liberals and conservatives think. University of Chicago Press.
Lee Epstein and Keren Weinshall. 2021. The Strate- gic Analysis of Judicial Behavior: Comparative Perspective. Cambridge University Press.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy. Veselin Stoyanov, and Luke Zettlemoyer. 2020.

---

BART: Denoising sequence-tor sequeneee pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computa- tional Linguistics.
Adam F Simon and Jennifer Jerit. 2007. Toward a theory relating political discourse, media, and public opinion. Journal of Communication, 57(2)2254-271
Matt Thomas, Bo Pang. and Lillian Lec. 2006. Get out the vote: Determining support opposition from congressional floor-dehate transcripts. In Proceed- ings of the 2006 Conference on Empirical Methods in Natural Language Processing, pages 327-335 Sydney, Australia. Association for Computational Linguistics.
Yinhan Liu, Myle Oit, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:
Dietrich Trautmann, Alina Petrova, and Frank Schilder. 2022. Legal prompt engineering for multilin- gual legal judgement prediction. arXiv preprint arXiv:2212.02199.
Robert N Lupton, Steven M Smallpage. and Adam M Enders. 2020. Values and political predispositions in the age of polarization: Examining the relation- ship between partisanship and ideology in the United States, 1988-2012. British Journal of Political Sci- ence, 2011):241-260
Keyon Vafa, Suresh Naidu, and David Blei. 2020. Text- based ideal points. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guisties, pages 5345-5357, Online. Association for Computational Linguistics.
Andrew Martin and Kevin M Quinn. 2002. Dynamic ideal point estimation via Markov chain Monte Carlo for the US Supreme Court, 1953-1999. Political Analysis, 1342)334-1533
Mike West and Jeff Harrison. 2006. Bayesian fore- casting and dynamic models. Springer Science & Business Media.
Andrew D Martin and Kevin M Quinn. 2007. Assess- ing preference change on the US Supreme Court. The Journal of Law, Economics, & Organization, 23(2):365-33 3855.
David Alistair Yalof. 2001. Pursuit of justices: Presi- dential politics and the selection of Supreme Court nominees. University of Chicago Press.
Moin Nadeem, Anna Bethke, and Siva Reddy, 2021. StereoSet: Measuring stereotypical bias in pretrained language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). pages 5356-5371, Online. Association for Computational Linguistics.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Car- bonell, Russ Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for lan- guage understanding. Advances in Neural Informa- tion Processing Systems, 32.
A Data Construction
Dana Patton and Joseph I. Smith. 2017. Lawyer, in- terrupted: Gender bias in oral arguments at the US Supreme Court. Journal of Law and Courts, 5(2):337-361.
We removed all nonlinguistic information from the transcripts, including indicators of cross-talk (e.g. [voice overlap), (interruption)), nonverbal expres- sions (c.g. [laughter], [sighs], [applause]), and pro- cedural markers (e.g. [luncheon], [recess]). The full list is made available as part of the code reposi- tory.
USA TODAY Richard Wolf. 2015. The 21 most famous Supreme Court decisions. Accessed on June 17th, 2023.
Michael D Robinson, Ryan L Boyd, Adam K Fetterman, and Michelle Persich. 2017. The mind versus the body in political (and nonpolitical) discourse: Linguistic evidence for ideological signature in US politics. Journal of Language and Social Psychology. 36(4):438-461.
B Distribution of Gold and Predicted Affiliated Party
Figures shows the label distribution of gold vs. predicted for affiliated party prediction over one randomly-selected fold.
Charles Shipan. 2008. Partisanship, ideology, and Senate voting on Supreme Court nominees. Journal of Empirical Legal Studies, 5(1):55-76.
C Detailed Experiment on Justice Identification Prediction
Yanchuan Sim, Bryan Routledge, and Noah A. Smith. 2016. Friends with motives: Using text to infer in- fluence on SCOTUS. In Proceedings of the 2016 Conference on Empirical Methods in Natural Lan- guage Processing, pages 1724-1733, Austin, Texas. Association for Computational Linguistics.
Table provides a detailed experiment on justice identification prediction with and without year tags on instances.

---

Rep-bitcn Democrat
(a) Gold labels.
(b) Predicted labels.
Figure 5: Label distribution of gold predicted for affiliated party prediction in data (as exemplified by one representative test fold).
LR
SVM
Instances year
BERT (Vanilla)
0.44 0.01
0.43 + 0.01
BERT (Finc-tuned)
0.32 0.10
0.35 + 0.10
Instances year
BERT (Vanilla)
0.45 0.01
0.43 + 0.01
BERT (Fine-tuned)
0.29 = 0.06
0.32 + 0.06
Table 5: Macro-F1 scores for justice identity prediction over instances with year tags on 10-fold cross-validation. LR and SVM were trained the training fold and performances were reported on the test folds.
D Comparison of Voting-based and Language-based Partisanship at Individual Justice Level
Figure 6 shows the comparison of voting-based (MQ scores; Martin and Quinn (2007) and lan- guage partisanship measures of all individual jus- tices over their tenure.
Figure shows the Comparison of all justice partisanship in their first year and over their tenure.
E
High Profile SCOTUS Cases
Table 6 lists the details of the high-profile SCOTUS cases analyzed in Section 4, Figure 4.

---

stalley
22
Pata
Byron Kirll
arthar gonibeg
Xhe Bitas
Hurgoos manball
2.0
5.0
Liiry Marrein
1.0
Mr
0.0
0.0
0.2
10
Figure 6: Comparison of voting- based (MQ scores (Martin and Quinn, 2007> (orange lines) and language partisanship measures (blue lines) of individual justices their tenure. The dashed line notes the standard derivation of language-bal linuaage-based partisanship measures.
Democratic
Language partisanship
score
Figure 7: Comparison of justice partisanship in their first year (small dots) and over their tenure (big dots). Blue and Red dots denote justices that are affiliated with the Democrat and Republican partics, respectively.

---

Year
Title
Issue Area
Abbreviated Label
1962
Gideon v. Wainwright
Criminal Procedure
G.v. W W_1962
1963
New York Times Company Sullivan
First Amendment
NYTC v. $_1963
1965
Miranda v. Arizona
Criminal Procedure
M A_1965
1966
Loving v. Virginia
Civil Rights
V. V_1966
1973
United States v. Nixon
Criminal Procedure
US v. 1973
1979
Regents of the University of California v. Bakke
Civil Rights
RTUC v. B 1979
2000
Bush v. Gore
Civil Rights
v. G 2000
2002
Lawrence v. Texas
Privacy
Lv. T 2002
2007
District of Columbia v. Heller
Criminal Procedure
DC v. H H020
2008
Citizens United v. Federal Election Commission
First Amendment
CU v. FEC_2008
2011
National Federation of Independent Business v. Sebelius
Federalism
NFIB S_2011
2012
Shelby County v. Holder
Civil Rights
S H_2012
2012
United States Windsor
Due Process
US W 2012
2014
Obergefell Hodges
Due Process
O v. 2014
Table 6: List of high-profike cases in SCOTUS (Richard Wolf, 2015).


Reviews:
[{"review_id": "jhfOD6RObi", "reviewer": "Reviewer_9L3i", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback.  \n\n1. **Temporal language shift.** We agree with your concern about language shift over the time period of our data, and we discuss this in the Limitations section. Our work shows that it is possible to capture partisan language in this data set without an explicit model of time: we split our data into 10 folds ensuring coverage of the full period in each fold. An explicit representation of language shift may well improve performance and is a great avenue for future work. In preliminary experiments, we quantified the impact of language drift on performance by experimenting with a chronological data split (the test set covering strictly later years than the training set). We will add this analysis to the paper.   \n\n2. **Benchmark models.** This paper asks, “To what extent do oral arguments reveal justice partisanship”. We use the BERT models as a benchmark to explore if it is possible to infer justice partisanship based on their oral arguments, and present positive results. In our experiments, we include random/majority prediction baselines, and we observe a large improvement with BERT.   Experiments with additional domain-specific language models may very well show a slight increase of performance but this is not the focus of this (short) paper, and not necessary to answer our research question.  \n\n3. **Input length.** We evaded the input limitation of the BERT models by using overlapping sliding windows with shifts of 255 tokens. In Section 4, we consider the average partisanship value across all data points (windows) per justice in each year. We effectively assume that partisanship is reflected in (reasonably) local linguistic choice. This assumption builds on prior work which showed that partisanship is reflected in word choice (see e.g., Jarvis (2004); Robinson et al (2007), cited in the paper). We will clarify our assumptions. \n\n4. We will add the suggested references to the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "**Clear Structure**\n\nThe overall methodology and research pipeline of the paper are well-defined. Firstly, the research motivation logically stems from prior studies, which primarily quantify partisanship based on voting behavior and oral arguments in the courtroom. The research method predominantly employs BERT as the foundational model for classification, adeptly capturing the binary classification results. Lastly, to substantiate the significance of the results, an analysis of how the Justices' language reflects their voting was undertaken.\n\n**Effective Correlation Analysis**\n\nFrom the focal point of the research, this paper emphasizes *\"How does language-based partisanship align with established measures of voting-based ideology\"*. The paper employs enough graphs and tables to substantiate the correlation between voting behavior and spoken language. Moreover, the approach of comparing voting-based and language partisanship measures of individual justices throughout their tenure is methodologically sound.\n\n**Insightful Perspective**\n\nThe authors' choice to analyze the alignment from a temporal standpoint is commendable and holds significant value. This perspective offers fresh insights and paves the way for future studies that converge political systems with linguistic inquiries.", "reasons_to_reject": "**Research Integrity:**\n\nFrom a data perspective, the SCOTUS dataset spans from 1955 to 2019. Undoubtedly, this represents a significant time frame, which implies that the authors aim to classify from a more comprehensive temporal standpoint. However, this extended time span also introduces variations in linguistic styles. For a binary classification (Democrat vs. Republican), the linguistic evolution within each party over several decades can pose challenges. Specifically, the choice of vocabulary and linguistic expressions for both the Democrats and Republicans have likely evolved over this period, which implies that achieving a thorough understanding and classification can be challenging.\n\n *Therefore, the lack of analysis of the temporal dimension in relation to linguistic features/patterns is not sufficient to demonstrate the accuracy and significance of the classification model.*\n\n1. The study could venture into examining the linguistic evolution within each party. Taking the Democrats as an example, the authors could use Sentence-BERT or other large language models to extract sentence embeddings. They could then calculate the similarity between embeddings across years (e.g., using cosine similarity, etc.) to gauge if there has been an \"evolutionary\" shift in the Democrats' linguistic patterns from 1955 to 2019.\n2. Building on this, the similarity between embeddings could be incorporated as an additional feature in the classification process, bolstering the robustness of their analysis.\n\n**Baseline Work:**\n\nIn this study, the authors fine-tuned the BERT model, experimented with adding temporal tags [year], and tried embedding the last layer of BERT into traditional ML models. While these represent innovative attempts, it's better to benchmark against more baseline models or those from relevant domains. This not only showcases the novelty of the authors' approach but also positions their contributions in the larger context. Models like LegalBERT, CaseLawBERT, or LegalRoBERTa could serve as relevant benchmarks. Furthermore, I recommend that the authors consider models like the Longformer. Given its maximum input sequence length of 2,048 tokens, it might enhance the model's ability to perceive broader contextual relationships. \n\n*Therefore, the lack of sufficient and comprehensive model comparisons, and the limitation of the length of the text inputs, affect the comprehensiveness of the results and the creativity of the author's methodology.*", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper addresses a significant gap in the understanding of partisanship and ideology in the US Supreme Court. Building on the many previous studies that have quantified partisanship based on judicial voting patterns and certain courtroom behaviors, the authors present a comprehensive approach to address the existing research gaps concerning the content of oral arguments in courtrooms. Specifically, this paper seeks to understand:\n\n- To what extent do the oral arguments made by justices in the US Supreme Court reflect their partisan or ideological beliefs?\n\n- Can linguistic patterns in these oral arguments be used to predict the affiliated party or ideology of the justices?\n\n- How does language-based partisanship align with established measures of voting-based ideology?\n\n**Contributions**\n\n*Highlighting the Significance of Oral Arguments*. The paper underscores the importance of oral arguments in the courtroom as a potential measure of partisanship. It emphasizes that while voting patterns are a direct measure, the nuances in the language during oral arguments can offer deeper insights into their beliefs and partisanship.\n\n*Bridging NLP technology and oral argument in SCOTUS*. The paper leverages the BERT-based classifiers which can be employed effectively to predict the affiliated party of justices. And the authors' work indicates a successful marriage of legal studies/political corpora and advanced computational techniques, showcasing the potential of NLP for a wider range of research areas.\n\n*Correlation Analysis of Behavior and Linguistic Patterns*. The authors correlate language-based ideology scores with well-established ideology measures obtained from justices' voting patterns. By building on previous research, which primarily quantifies partisanship through voting behavior, this paper further corroborates and analyzes the connection between voting and spoken language.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "1. **Consider the comprehensiveness and comparison of the models.**\n\nI recommend: \n\nTrautmann, D., Petrova, A., & Schilder, F. (2022). Legal prompt engineering for multilingual legal judgement prediction. arXiv preprint arXiv:2212.02199.\n\n2. **Consider diachronic word embeddings and semantic shifts to make the research more comprehensive**\n\nI recommend:\n\nDing, X., Horning, M., & Rho, E. H. (2023, June). Same Words, Different Meanings: Semantic Polarization in Broadcast Media Language Forecasts Polarity in Online Public Discourse. In Proceedings of the International AAAI Conference on Web and Social Media (Vol. 17, pp. 161-172).\n\nHu, R., Li, S., & Liang, S. (2019, July). Diachronic sense modeling with deep contextualized word embeddings: An ecological view. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (pp. 3899-3908)."}}, {"review_id": "Nd8NbHpkHA", "reviewer": "Reviewer_HTaT", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback.  \n\nWe agree that prior work has demonstrated that ideology can be predicted from texts that are expected to capture author stance (such as congress speeches in Jensen et al., (2012) or practically party-affiliated US news channels in Dutta et al., (2021)). We study supreme court arguments which, by definition, should be politically neutral. Consequently, the fact that we can reliably predict party affiliation from court language is a noteworthy finding. The fact that our language-based predictions correlate overall with established voting-based measures of justice partisanship is reassuring and validates our method. Moreover, our method enables deeper insights, for instance by studying the varying degree of discrepancy between words (language) and action (voting) for individual justices (Fig. 2). We will clarify this point in the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Analyzing the language-based partisanship of supreme court justices is interesting. The paper compares the affiliated party prediction model's performance on speaker identity detection against an off-the-shelf model to conclude that the model is not capturing the idiosyncracies of individual justices. Then it correlates the partisanship score with the voting ideology (MQ score). The work also conducts deeper analysis at the individual justice level and case level. Overall, this work broadens the understanding of partisan language in oral arguments used by the justices.", "reasons_to_reject": "The findings of this paper are somewhat expected. Existing work has studied linguistic differences among (groups of) people with different political ideologies.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a framework for analyzing the language of justices (during oral arguments) to identify partisan signals, demonstrating the predictability of justices' affiliated parties based on their oral contributions. The authors found a strong correlation between the justices' language partisanship and their voting behavior.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "I found no typos, and the presentation style is good."}, "extra_text": {"Missing_References": "This is one of the earlier works that study partisan language in political discourse.\n\n1. Jensen, Jacob, et al. \"Political polarization and the dynamics of political language: Evidence from 130 years of partisan speech [with comments and discussion].\" Brookings Papers on Economic Activity (2012): 1-81.\n\nThe following paper showed that it is possible to predict the source of a text in a particular domain based on the partisan linguistic signals present in the news transcripts.\n\n2. Dutta, Sujan, et al. \"A Murder and Protests, the Capitol Riot, and the Chauvin Trial: Estimating Disparate News Media Stance.\" Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22. 2022.\n\n\n\n"}}, {"review_id": "tvv6lmFTPX", "reviewer": "Reviewer_2Gwe", "summary": "", "questions": "A. Why is partisanship measure important to political science research?\nB. You noted word choice conveys specific messages in the introduction. Wouldn't model interpretability be a vital aspect of this approach?   If so, have you analyzed the salient words or phrases?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments.   \n\n**Significance of contribution.** The importance of the partisanship analysis in our paper can be summarized into two folds: (1) We link the language in oral arguments to justice voting behavior in the SCOTUS. Our analysis shows for the first time that voting partisanship is reflected in court language. Moreover, our work reveals nuanced differences in linguistic and voting behaviors e.g. the tendency shift in Fig 3 (First year vs. Tenure), which further demonstrates the importance of studying SCOTUS from various perspectives.  (2) More fundamentally, partisanship as affiliation along the liberal---conservative political spectrum is a fundamental axis in the (western) political discourse. Automatically identifying partisan language through NLP techniques enables large-scale analysis of political and public discourse, and better understanding of divisions and polarization. Our use case is one example of this. We will add this clarification to the paper. \n\n**Linguistic features.** We address the issue of feature analysis in the Limitations section. While we agree that analyzing partisanship-indicators is a valuable direction for future work, it is outside the scope of this paper. We acknowledge the difficulty of obtaining reliable linguistic features from large language models --- particularly given the substantial time span covered in our data. Obtaining salient words from oral arguments is a promising future venture to enhance the analysis of oral partisanship.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper provides a novel proxy for partisanship based on speech instead of voting records. The paper's main strengths include: \n- Well-documented data prep and model training procedures.\n- Analysis to ensure the model isn't focused on speaker identification\n- Temporal analysis of language partnership and MQ scores.", "reasons_to_reject": "The paper must clearly state why the partisanship/ideology measure is essential or how this measure impacts political science research.   The approach would be better if it provided insights into linguistic features that indicate partisanship in the utterances.", "questions_to_authors": "A. Why is partisanship measure important to political science research?\nB. You noted word choice conveys specific messages in the introduction. Wouldn't model interpretability be a vital aspect of this approach?   If so, have you analyzed the salient words or phrases?", "paper_topic_and_main_contributions": "The paper examines partisan language in SCOTUS oral arguments. The authors attempt to answer two research questions: 1.) Does a justice's speech in oral arguments indicate their political affiliation? 2.) How well do oral arguments correlate to other partisanship measures?\n\nThe main contribution of this paper is a speech-based alternative to voting-based partisanship measures, specifically, a classification model to predict party affiliation based on utterances. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "eZ94toiDFZ", "reviewer": "Reviewer_NGhd", "summary": "", "questions": "A:  A sentence or two on how you derive \"a partisanship score from our predicted party affiliation probabilities\" would be helpful.  In particular, how that works per speaker per year?  I think this is mentioned in:  \"we obtain a language partisanship score of a justice by averaging the predicted party probability for each of the justices’ instances.\"   Is the model predicting party or party probability?  (I think this is a familiar caution about interpreting logistic-style regressions as likelihoods).\n\nB: How does the model deal with different volumes of text per speaker?  Some justices ask more questions.  \n\nC: Are there any insights into the language from the model that are interesting?  The simpler text-regressions have the advantage of being easy to interpret  (\"baby\" versus \"fetus\" is mentioned in the text).  Can this be captured here?\n\nD:  This is beyond the scope of the paper, but perhaps worth asking:  Does the questions/text in oral argument predict the voting outcome?\n\nE: Figure 3. It is interesting to compare first year to the whole tenure.  However, it would helpful to have a notion of \"standard errors\" here.  Is the movement from year one \"big\" or is this just movement year to year?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback and questions.  \n\n**Significance of contribution.** The importance of the partisanship analysis in our paper can be summarized into three areas: (1) Prior work on partisan language classification in NLP has mostly focused on explicitly partisan text (such as congress speeches or news outlets), while prior work on bias in SCOTUS has largely ignored the court arguments. We study supreme court language, which, by definition, should be politically neutral. Consequently, the fact that we can reliably predict party affiliation from court language is a novel and noteworthy finding in itself. (2) We link the language in oral arguments to justice voting behavior in the SCOTUS. Our analysis shows for the first time that voting partisanship is reflected in court language. And (3) our work reveals nuanced differences in linguistic and voting behaviors e.g. the tendency shift in Fig 3 (First year vs. Tenure), which further demonstrates the importance of studying SCOTUS from various perspectives.  We will clarify these points in the paper. \n\n\nTo answer your questions: \n\nQuestion A. For each data point (i.e., contribution by a given justice in a given year), the model predicts the probability p(y=Democrat). We average the predicted probability e.g., to obtain a justice-level score.  In preliminary experiments, we also experimented with aggregating class predictions (rather than probabilities), observing similar results. We will clarify this in the paper.  \n\n \n\nQuestion B. To ensure sufficient data for each justice in our analysis, we excluded justices with fewer than five data points for every single year in our data set. Additionally, by using a sliding window we leverage the full available data per justice.  \n\n \n\nQuestions C. This is a great suggestion, but as discussed in the Limitations section, we emphasize the difficulty in obtaining reliable salient words from large language models. A concerted attempt is beyond the scope of this (short) paper. \n\n \n\nQuestion D. Thanks for the suggestion. First, we would like to point out the ethical issues raised by applications that predict voting behavior based on language. That said, we can confirm from preliminary experiments that voting prediction based on our representation was poor. One hypothesis is that the spoken language does not give away the voting information as justices often make up their mind only after the full argument (in fact, we would hope this is the norm). So, the justice language in the court room might reflect their (static) party affiliation, but the oral argument itself does not reveal their voting behaviour. Other official documents, such as Court Opinions, capture this stage. \n\n \n\nQuestion E. We will add the analysis of “standard errors”, i.e., average year-to-year movement for justices, to Fig 3.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper is clear, straightforward, and insightful.  Using the text from oral arguments is a good idea and generates interesting results.\n- The analysis of individual justices (over time; relative to the first year on the court) is insightful", "reasons_to_reject": "- This is a topic that has been heavily studied.  (And, to be clear, the paper notes this with thoughtful and concise citations).  So generating novel results here is more difficult.  What do we learn here that is new?\n", "questions_to_authors": "A:  A sentence or two on how you derive \"a partisanship score from our predicted party affiliation probabilities\" would be helpful.  In particular, how that works per speaker per year?  I think this is mentioned in:  \"we obtain a language partisanship score of a justice by averaging the predicted party probability for each of the justices’ instances.\"   Is the model predicting party or party probability?  (I think this is a familiar caution about interpreting logistic-style regressions as likelihoods).\n\nB: How does the model deal with different volumes of text per speaker?  Some justices ask more questions.  \n\nC: Are there any insights into the language from the model that are interesting?  The simpler text-regressions have the advantage of being easy to interpret  (\"baby\" versus \"fetus\" is mentioned in the text).  Can this be captured here?\n\nD:  This is beyond the scope of the paper, but perhaps worth asking:  Does the questions/text in oral argument predict the voting outcome?\n\nE: Figure 3. It is interesting to compare first year to the whole tenure.  However, it would helpful to have a notion of \"standard errors\" here.  Is the movement from year one \"big\" or is this just movement year to year?", "paper_topic_and_main_contributions": "- The paper measures partisanship based on the text supreme court justices say during oral argument (and not the text in the written opinions).  This is interesting since it measures the text of individual justice and not the written opinions (that are joint statements).  The paper uses the existing idea that a model that regresses party affiliation (known) on text is informative.  The coefficients of that regression tell you what words correlate with party affiliation.  Here it is not literally a text regression but using the BERT structure.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Nicely written paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Three reviewers evaluated the paper’s soundness with 4 (strong) and one with 3 (good) – overall, indicating that the paper on the most part provides sufficient support for all of its claims/arguments.  All four reviewers, however, were ambivalent about the paper’s excitement (3) – noting some ambiguity in the novelty of the contribution and potential limitations. Reviewer 9L3i  raised concerns about the limitation of not considering the diachronic aspects and more comprehensive model comparisons. Reviewer 2Gwe requested the authors to clarify the importance of their partisanship measure. 2Gwe also noted, the paper would be strengthened by providing key insights into linguistic features that indicate partisanship in the justice languages. Reviewer NGhD also notes that this is a topic that has been heavily studied and argues that the paper would benefit from highlighting new contributions. Relatedly, Reviewer HTaT notes that existing work has studied linguistic differences among (groups of) people with different political ideologies. An additional analysis examining the partisan differences in the language of justices would address such reviewers’ concerns. While the authors addressed some limitations with relevant citations and provided clarification to the reviewers’ questions, the paper will still benefit from additional analyses suggested by 2Gwe and 9L3i.
Decision: Accept-Findings