Debatrix: Multi-dimensional Debate Judge with Iterative Chronological Analysis Based on LLM (2024)

Abstract: How can we construct an automated debate judge to evaluate an extensive, vibrant, multiturn debate? This task is challenging, as judging a debate involves grappling with lengthy texts, intricate argument relationships, and multi-dimensional assessments. At the same time, current research mainly focuses on short dialogues, rarely touching upon the evaluation of an entire debate. In this paper, by leveraging Large Language Models (LLMs), we propose Debatrix, which makes the analysis and assessment of multi-turn debates more aligned with majority preferences. Specifically, Debatrix features a vertical, iterative chronological analysis and a horizontal, multi-dimensional evaluation collaboration. To align with real-world debate scenarios, we introduced the PanelBench benchmark, comparing our system’s performance to actual debate outcomes. The findings indicate a notable enhancement over directly using LLMs for debate evaluation. Source code and benchmark data are available at https: //github.com/ljcleo/debatrix.

text: Findings of the Association for Computational Linguistics ACL 2024, pages 14575–14595 August 11-16, 2024 ©2024 Association for Computational Linguistics
How can we construct an automated debate judge to evaluate an extensive, vibrant, multiturn debate? This task is challenging, as judging a debate involves grappling with lengthy texts, intricate argument relationships, and multi-dimensional assessments. At the same time, current research mainly focuses on short dialogues, rarely touching upon the evaluation of an entire debate. In this paper, by leveraging Large Language Models (LLMs), we propose Debatrix, which makes the analysis and assessment of multi-turn debates more aligned with majority preferences. Specifically, Debatrix features a vertical, iterative chronological analysis and a horizontal, multi-dimensional evaluation collaboration. To align with real-world debate scenarios, we introduced the PanelBench benchmark, comparing our system’s performance to actual debate outcomes. The findings indicate a notable enhancement over directly using LLMs for debate evaluation. Source code and benchmark data are available at https: //github.com/ljcleo/debatrix.

heading: 1 Introduction
text: Debating is the formal process of gaining consensus among groups with different opinions. In many cases, such as in competitive debates, only the policy proposed by the winning side will be accepted (Zhang et al., 2016). Debaters must apply various strategies to convince the audience to support their side in these debates. While systems like Project Debater (Slonim et al., 2021) have enabled automatic debating in competitive debates, judging these debates still relies on human annotation. Automating debate assessment is helpful to improve debate quality in political, commercial, or educational scenarios and can also accelerate the evolution of debate automatons.
Recently, large language models (LLM) such as ChatGPT and GPT-4 (OpenAI, 2023) have
∗Corresponding author.
shown a solid ability to evaluate text quality (Liu et al., 2023; Chiang and Lee, 2023). In this task, LLMs can provide results more aligned with human preference than traditional metrics Zheng et al. (2023). Additionally, LLM judges’ verdicts are more straightforward to interpret than computational metrics or audience votes since they usually come with generated explanations.
However, judging debates with LLMs incorporates several issues to be considered, as illustrated in Figure 1. First, evaluating long, multi-turn debates remains challenging, as most current research focuses on short text such as open-question answers and user-request responses (Zhong et al., 2022; Wu et al., 2023). Second, debate judging requires a deep understanding of how arguments are organized and refuted across speeches; not only does it require more professional knowledge, but it also demands a thorough, precise analysis of how argu-
14575
ments work in the debate. Finally, speech quality is affected by various factors, such as argument strength, evidence reliability, and language style, requiring systematic evaluation across dimensions.
To this end, we propose Debatrix, a fine-grained framework to assist LLMs in handling these challenges by breaking down debate judging along both chronological and dimensional axes. 1) Iterative Chronological Analysis: We instruct the LLM to analyze the debate speech by speech, maintaining speech and analysis streams with a memory system and providing content analysis of previous speeches when analyzing new speeches. After reviewing all speeches, the LLM makes decisions based on all analyses. This iterative approach lets the LLM concentrate on one speech at a time and understand the context more effectively. It also produces feedback or decisions for each speech, each debater, and the final winner. 2) Multi-Dimensional Collaboration: Debatrix also allows LLMs to focus on a specific judging dimension, such as argument, language, or clash, during the speech analyzing process. Each LLM agent can make comments on these specific aspects. For the overall judgment, all these individual analyses are combined into one summary, providing a systematic judgment across multiple dimensions.
Furthermore, we introduce PanelBench, a novel benchmark for evaluating automatic debate judging systems. PanelBench consists of two collections of debates with judgments, namely DebateArt and BP-Competition. DebateArt sources from online platforms that follow competitive debate formats; debates vary in speech count and length and have dimensional voting results. BP-Competition includes debates transcribed from world-class competitive debate competitions; these debates follow the British Parliamentary (BP) format involving four teams (two on each side), enriching PanelBench with long, complex, high-quality samples. On PanelBench, Debatrix increased winner prediction accuracy compared to the baseline of directly prompting the LLM with raw speeches; perdimension judging is also improved. Furthermore, the results have also proved that iterative speechby-speech analysis and splitting dimensions help generate a more accurate final verdict.
Our contributions are as follows:
1. We propose Debatrix, a fine-grained automatic debate judging framework based on LLM, but performs analysis iteratively on mul-
tiple dimensions before summarizing and generating the final verdict.
2. We propose PanelBench, a benchmark for automatic debate judging, including multidimensional and multi-debater settings, which differ from simple 1v1 debate assessment.
3. We investigate how well LLMs can judge debates directly or using Debatrix, enabling chronological and/or dimensional analysis.

heading: 2 Related Work
text: Argumentation persuasion assessment is the foundation of automatic debate systems, as they must be persuasive enough to argue effectively. Previous works have focused on the persuasiveness of arguments, including empirical studies (Thomas et al., 2017, 2019a), machine learning models (Persing and Ng, 2015; Zhang et al., 2016; Gleize et al., 2019) and work covering both (Al Khatib et al., 2020; Donadello et al., 2022). Some argumentbased chatbots also take persuasiveness as a motivating factor (Rosenfeld and Kraus, 2016; Thomas et al., 2019b). Few works have extended these methods to judging whole debates (Potash and Rumshisky, 2017), including competitive debates (Ruiz-Dolz et al., 2023).
While these works mainly involve empirical laws or delicate models, within the bigger context of text evaluation, large language models (LLMs) have become a new and powerful tool to tackle this task. Multiple exploitation methods have been proposed, such as conditional probability (Fu et al., 2023), score prompting (Wang et al., 2023a; Liu et al., 2023; Zhu et al., 2023) and pairwise comparison (Wang et al., 2023b). Several works, such as Bai et al. (2023), utilize multiple of them and introduce various methods to stabilize the results.
Regarding the challenges of judging debates with LLMs, some pioneering works have investigated some of them. For instance, Chang (2023), Li et al. (2023) and Chan et al. (2023) focus on multi-dimensional assessment and propose different strategies to improve accuracy, such as dialetics, peer review and group chat. Meanwhile, de Wynter and Yuan (2023) and Chen et al. (2023) have explored LLM’s ability to handle argumentation tasks, measuring its capability of argumentation reasoning. These works are partially in line with our work yet do not cover all 3 issues mentioned in Section 1. Finally, Li et al. (2019) proposed a
dialog evaluation framework that works in a multiturn manner, similar to our chronological design; however, this framework is designed for human annotators instead of LLM judges.

heading: 3 Debatrix
text: In this section, we provide a detailed overview of Debatrix, our fine-grained debate judging framework.

heading: 3.1 Structure and Components
text: The overall structure of Debatrix is illustrated in Figure 2. Debatrix contains a collection of iterative chronological analyses like matrix columns, each capable of evaluating debates under a specific preference or dimension. Each of them processes speeches one by one, using past analysis for new ones and generating judgments for every speech, including a score and a comment. When all speeches are analyzed, past analyses are summarized and converted into debater judgments (similar to speech judgments but for individual debaters) and the winner verdict (including the winner and a comment). Multiple analyses can collaborate like a matrix, producing systematic judgments that cover multiple dimensions.
There are two components in Debatrix: memories and judges; they work together to analyze the debate thoroughly. One iterative chronological analysis utilizes one memory and a set of judges.
Memory Memories provide long-term storage during the debate judging process. There are two
types of memory: context memory records incoming speech context, and analysis memory stores intermediate analyses. Every incoming speech is added to the context memory before being analyzed by the speech judge. At any time, judges can fetch or query contents from both memories and add new analyses to the analysis memory.
Judge Judges are the core components to analyze and judge the debate, including the speech judge, the debater judge, and the winner judge. The speech judge analyzes the stream of speeches, utilizing memories to understand them; the analysis is added to the memory and can be used to generate judgments. The debater and winner judges work after all speeches are processed; they use past analyses by the speech judges to generate debater judgments and winner verdicts, respectively.

heading: 3.2 Iterative Chronological Analysis
text: LLMs have shown a strong capability to evaluate single passages and short conversations. To elaborate on this power for long multi-turn debates, we propose a dedicated design to analyze the debate iteratively in chronological order, which can already be seen in Figure 2. Figure 3 illustrates a more detailed version; the complete algorithm is listed in Appendix A.
The key point of the design is the iterative speech analysis process. Speech analysis focuses on decomposing the content of the speech, such as how arguments interact with each other, what evidence is introduced to back arguments, and what language
style the speech has shown. The preference or dimension controls what is included in the analysis, forcing the speech judge to generate corresponding speech judgments.
Each speech analysis is added to the analysis memory as part of the content analysis, acting as a digested version of the current speech. When analyzing the subsequent speech, we fetch both the new speech content and the content analysis of all previous speeches from the memory and input them to the LLM. We can also include relevant speech contexts from previous speeches in the input. This iterative approach reduces the complexity of the context and results in a more precise analysis.
At the end of the debate, the debater and the winner judges must take advantage of the entire list of content analyses to judge a specific debater or compare between debaters. This is achieved by an extra debate analyzer, which converts all content analysis into a debater-directed debate analysis. Finally, the debater judges and the winner judge generate debater judgments and the winner’s verdict based on the debate analysis.

heading: 3.3 Dimensional Collaboration
text: One way to obtain the overall judgment of a debate — a judgment covering all concerned dimensions — is to configure a single iterative chronological analysis to consider multiple dimensions when analyzing speeches. Yet, in Debatrix, we can also split and distribute dimensions to multiple analyses. Each of them focuses on a specific dimension and can provide a more detailed insight regarding

heading: Source

heading: Src. Dim. Debate

heading: Sources

heading: Debate Analysis Summarizer

heading: Debate Analysis
text: the dimension, which is useful when we demand dimensional comments on debates.
To synthesize an overall judgment from dimensional ones, Debatrix can combine debate analyses from multiple iterative chronological analyses into one systematic analysis. Figure 4 demonstrates the combination process for debater assessment and winner judging, summarizing debate analysis from various dimensions; the same process can also be applied to speech analyses for speech judgments. With improved dimensional analyses, Debatrix can generate more accurate overall judgments.

heading: 4 PanelBench
text: To assess LLMs and our proposed Debatrix framework with real competitive debates, we introduce PanelBench, a novel benchmark for automatic debate judging covering various styles, formations, and judging dimensions. We include two debate collections in PanelBench: DebateArt for 1v1 debates with dimensional judgments and BPCompetition for high-quality debates with multiple debaters on each side.

heading: 4.1 DebateArt
text: DebateArt debates are collected from DebateArt1, an online debate platform that provides 1v1 debate arenas. The formation setting makes the platform different from many other debate forums that do
1https://debateart.org
not restrict the speaking order, as debates on this platform are closer to formal competitive ones.
In DebateArt, users vote to decide debate winners. Besides the common winner selection system, the platform also provides a categorical point assignment system, where voters must consider and vote on four metrics for comparative performance insights: Argument, source, legibility, and conduct.2 This voting system provides judgments under separate dimensions: pro-winning, con-winning, or a tie. Moreover, voters must provide detailed explanations of their decisions, and their votes are supervised by experienced moderators, enhancing their quality. The weighted average of dimensional votes decides the debate’s overall winner. Appendix B.1 details how DebateArt debates are run.
PanelBench includes 100 valid debates with valid votes from DebateArt; we include our data collection process and filtering criteria in Appendix B.2. Table 1 lists the statistics of these debates. To align with oral debates that are not formatted, we merged two dimensions — legibility and conduct — into a single language dimension, representing the language style. We averaged their votes in these two dimensions for each vote and
2https://info.debateart.com/terms-of-s ervice/voting-policy#casting-votes
converted them into a single vote (pro/tie/con). Table 2 lists the winner distribution of all debates: While voters tend to give ties to source and language, most debates have a specific winning side, largely because of the argument dimension.

heading: 4.2 BP-Competition
text: To extend PanelBench with high-quality formal debates, we furthermore collected debates from recent world-class competitive debate competitions, including the World Universities Debating Championship (WUDC), the European Universities Debating Championship (EUDC), and the North American Debating Championship (NAUDC). All these competitions follow the British Parliamentary (BP) format (World Universities Debating Council, 2023); therefore, we name them BPCompetition debates. In this format, four teams (OG, OO, CG, and CO) are divided into two sides of a motion but compete against all three other teams in the debate; hence, instead of predicting the winning side, PanelBench requires judging which of the four teams is the best. Please refer to Appendix B.3 for more details about the BP format.
We transcribed debate videos from knockout rounds of famous competitive debate competitions to obtain high-quality BP debates; Please refer to Appendix B.4 for data collection details. After filtering incomplete or damaged transcriptions, we obtained 22 debates with full transcriptions and final verdicts. Table 3 lists statistics of these debates; they are significantly longer than DebateArt ones.
It is worth noting that we were only able to collect the winning teams of BP-Competition debates, which is not necessarily unique: In the semifinals and quarter-finals, two of them (they can even be mutual opponents) can win and proceed. Among all BP-Competition debates, 6 have only one winner, and 16 have two; Table 4 lists the distribution of all winners. To unify them, PanelBench treats predicting any winning teams as correct.

heading: 5 Experiments
text: We conducted experiments on PanelBench to evaluate the debate-judging performance of LLMs. We also compare our Debatrix framework with judging directly with LLMs.

heading: 5.1 Model & Framework Configuration
text: We utilize the latest GPT family as our target LLMs, including ChatGPT and GPT-43; our experiments mainly focus on ChatGPT to test Debatrix under a limited context window (16, 385 tokens). We set the temperature to 0 and repeated all experiments 3 times, measuring the average performance.
We provide judge preferences in the system prompt to control the dimension of the judgments. To shorten the input, we do not include relevant context from previous speeches, only the content analysis. We ask the judges to output comments only and then call the LLM again to generate their corresponding scores (integers from 1 to 10) or winner verdict to diminish mismatches. All prompt templates are listed in Appendix C.
As mentioned in Section 4, we introduce 3 judging dimensions for DebateArt debates: argument, source, and language. In our experiments for these debates, we stick to these dimensions and their weights (argument: 3; source: 2; language: 2) to measure dimensional judging performance and utilize them to generate systematic verdicts. BPCompetition debates do not have dimensional judgments, yet we continue splitting dimensions for these debates. We adopt the dimensions defined for DebateArt and add a clash dimension, which originally belongs to the argument dimension, so that all dimensions are balanced (and thus, all dimensions have equal weight).

heading: 5.2 Baseline Models
text: We compare Debatrix with these baseline models:
3For ChatGPT, we use gpt-3.5-turbo-0125; for GPT-4, we use gpt-4-0125-preview.
• ChatGPT and GPT-4 reads the entire debate in one input and generates an overall verdict, with an all-in-one judge preference covering all dimensions, backbone LLMs being ChatGPT and GPT-4 respectively;
• Chronological introduces iterative chronological analysis but does not split dimensions, instead using the all-in-one judge preference.
• Dimensional introduces dimensional collaboration, assessing debates under every dimension and summarizing dimensional judgments, but inputs the debate as a whole;
• NonIterative is almost identical to Debatrix, but the analysis is not iterative: we fetch the raw content instead of the content analysis of all previous speeches when analyzing new speeches, so the speech analysis is solely based on the original text.
We use ChatGPT, GPT-4, Debatrix, etc. to refer to the models below, distinguishing them from backbone LLMs and frameworks.

heading: 5.3 Evaluation Configuration
text: We generate winner verdict predictions for both debate collections according to the model output and compare them with true verdicts. We experiment with two methods of winner prediction for DebateArt debates: score comparison (SC) compares the score of each debater generated by debater judges and chooses the debater with a higher score (or announces a tie when the scores are equal); direct prediction (DP) predicts the winner directly according to the verdicts by the winner judges. Specifically, for the source and language dimensions, we treat score differences within ±3 as ties, as human users tend to neglect small differences in them (as can be seen in Table 2). We only utilize direct prediction for BP-Competition debates since they do not allow ties and score comparison often fails to produce a single winner.
When evaluating model performance on DebateArt debates, we measure the root mean square error (RMSE) between the model prediction and the true verdict due to the existence of ties. More specifically, we assign the values of pro, tie, and con verdicts to 0, 0.5, and 1, respectively; then, we match each true verdict to its corresponding model prediction and calculate the RMSE. Meanwhile, we directly measure the completion rate and prediction
accuracy for BP-Competition debates since some models fail to generate complete verdicts because of a limited context window.

heading: 5.4 Main Results
text: Table 5 lists the main experiment results across models and debate collections. Appendix D.1 provides significance tests of the results. Debatrix consistently outperforms all baselines on both debate collections, including GPT-4.
Iterative chronological analysis is crucial for ChatGPT, with a limited context window, to handle very long debates: only Chronological and Debatrix judged all 22 BP-Competition debates in all trials successfully, while ChatGPT only completed very few judgments and none of them are correct; NonIterative failed to judge some BP-Competition debates because raw speeches are much longer than content analyses.
Meanwhile, dimensional collaboration is also beneficial in shorter debates, as seen in the DebateArt collection. When debates are short enough to fit in the context window, Dimensional significantly outperforms ChatGPT and is on par with GPT-4. On BP-Competition debates, the completion rate also increases since judging a single dimension costs fewer tokens.
Nevertheless, combining both of them gives the best performance. NonIterative is already aligned with GPT-4, especially superseding it on the BP-Competition collection while not even completing all trials. However, using previous content analysis iteratively, Debatrix overcomes the context window issue and shows significant improvements compared to NonIterative.

heading: 6 Analysis

heading: 6.1 Multi-Turn & Long Debates
text: As demonstrated in Table 1, DebateArt debates vary in speech count and length: among all 100 debates, 34 have no fewer than 8 speeches, and 51 contain at least 4, 000 tokens. Figure 5 compares the performance of different models on debates with different numbers of speeches or tokens.
On one hand, some baseline models show partial advantage but fail to cover all scenarios: GPT-4 handles long debates well but degrades when judging debates with many turns; Dimensional achieves low RMSE on debates with few turns or tokens and NonIterative on ones with many turns or tokens, but both become rather worse when switching to the other half of the debates.
On the other hand, Debatrix maintains a relatively low RMSE compared to other models, regardless of the number of speeches or tokens. This shows that Debatrix effectively assists the LLM in evaluating long, multi-turn debates while extending its advantage to short debates.

heading: 6.2 Performance on Argument Dimension
text: Among all the dimensions we introduced, the argument dimension is the major one affecting debaters’ persuasiveness, as shown in Table 2. Table 6 lists model performance on DebateArt debates under this dimension. We only include models with dimensional collaboration; we also modify Dimensional to use GPT-4 to examine whether upgrading the LLM is beneficial.
We can see in Table 6 that using a more powerful LLM such as GPT-4 does not improve Dimensional much on this dimension; instead, chronological analysis brings a significant improve-
ment. Iterative analysis when analyzing speeches is also beneficial, as illustrated by the difference between Debatrix and NonIterative. All these results prove that Debatrix can help the LLM understand arguments better without resorting to larger models.

heading: 6.3 Dimensional Collaboration for Systematic Judgment
text: Table 5 also clearly demonstrates the advantage of dimensional collaboration. Models that split dimensions (Dimensional and Debatrix) consistently outperform their duets without splitting dimensions (ChatGPT and Chronological). This shows that splitting dimensions improves the analysis quality for each dimension, leading to a more precise systematic judgment at the end. Specifically, Dimensional significantly outperforms Chronological on DebateArt debates, indicating that the quality of intermediate content analysis is crucial for the final judgment.

heading: 6.4 Position Bias on GPT-4
text: From Table 5, we can find that GPT-4, while having a context window large enough to handle all BP-Competition debates, does not achieve a high accuracy. To investigate what prevents GPT-4 from better performance, Table 7 summarizes the
winner prediction distribution of various models. Surprisingly, GPT-4 always predicts the closing half (CG and CO), which speaks after the opening half (OG and OO); in most cases, it selects CO, who speaks the last. Meanwhile, ChatGPT-based Debatrix gives relatively balanced predictions roughly matching the expectation. The case study in Appendix D.4 also reveals this phenomenon.
We conjecture that position bias (Ko et al., 2020; Wang et al., 2023b) could be a significant factor that causes GPT-4 to fail in judging BP debates. When the arguments of all debaters are similarly strong, LLM may prefer the last speaker who can refute other debaters without being refuted by others, thus seemingly more convincing.

heading: 7 Conclusion
text: In this paper, we propose a fine-grained debate judging framework based on LLM, Debatrix. We decompose the debate judging task into an iterative chronological analysis to tackle multi-turn, long debates and elaborate multiple dimensions to generate systematic judgments. We introduce a novel debate judging benchmark, PanelBench, to assess our framework and other automatic debate
judging approaches. The benchmark covers multidimensional and multi-debater scenarios. Under both settings, Debatrix significantly improves ChatGPT, aiding it in judging long debates that exceed the context window and outperforming GPT-4.

heading: Limitations
text: Despite the above results, this paper has a few limitations for which we appreciate future studies. First, while the iterative chronological analysis in Debatrix already shows competence, it can be further studied and polished. Second, the position bias issue on GPT-4 for BP-Competition debates needs further investigation, and more powerful tools are needed to rectify this phenomenon.
Ethical Impact
As a debate judging framework based on LLM, Debatrix may be exposed to ethical risks if the backbone LLM is biased beforehand, especially when judging debates concerning political, social, or cultural motions. In our experiments, however, using LLMs with strong ethical constraints like ChatGPT and GPT-4 has resulted in no significant ethical bias within the generated verdicts. Furthermore, as demonstrated in Appendix D.4, Debatrix guides LLMs to focus on the content instead of basing their judgments on a priori knowledge or belief, alleviating potential ethical impact.

heading: Acknowledgements
text: This work is supported by National Natural Science Foundation of China (No. 62176058) and National Key R & D Program of China (2023YFF1204800). The project’s computational resources are supported by CFFF platform of Fudan University.

heading: A Debatrix Algorithm
text: A.1 Iterative Chronological Analysis Algorithm
Algorithm 1 demonstrates the complete algorithm of iterative chronological analysis in Debatrix. Here, we do not include dimensional collaboration. Note that our experiments do not query and include relevant contexts from previous speeches.

heading: A.2 Computational Cost Analysis
text: Here, we provide an approximate analysis of Debatrix’s computational cost based on the number of input tokens. We do not consider the number of generated tokens because, first, they are much fewer than the input tokens (especially for long debates) and have much less impact; second, extra outputs from Debatrix can be further utilized, for example, when generating speech verdicts.
Assume we have a debate with N speeches, each speech having x tokens. Furthermore, assume each speech analysis has y tokens and debate analysis z tokens. We follow our experiment settings in Section 5 for all estimations.
First, we consider the single-dimension case. ChatGPT or GPT-4 consumes Nx input tokens and generates z output tokens. Debatrix consumes y · N(N + 1)/2 more input tokens and generates Ny more output tokens because we input the speech analysis for all later speeches. For long debates, for example BP-Competition where N = 8 and y ≈ x/6, the number of input tokens
is only 1.75 times more than ChatGPT/GPT-4. Even for debates with shorter speeches, like DebateArt where y ≈ x/2, the ratio is less than 4.
Now we turn to dimensional collaboration with k dimensions. In this case, the total cost is multiplied by k, with an extra kz → z step to generate the summarized judgment. In our experiments, k = 3 (DebateArt) or k = 4 (BP-Competition) and z ≈ y. This results in a final ratio lower than 20 — the price ratio between GPT-4 and ChatGPT4.
In Section 5, we show that ChatGPT-based Debatrix consistently outperforms GPT-4; the above estimation demonstrates that our computational cost is also lower.

heading: B Benchmark Data Details
text: B.1 DebateArt Debate Procedure5
Users instigate debates in DebateArt. The instigator needs to provide a debate topic and set debate configurations such as character limit, time limit (12 hours to 2 weeks), and the number of rounds (up to 5); they can also include a description with pertinent details like definitions, expanded resolution, special rules, and scope limitations.
The instigator may elect to be pro or con, leaving the other position to the contender; the contender can be any community user willing to accept the challenge or another user requested directly by the
4https://openai.com/pricing 5This section mainly refers to https://info.debat
eart.com/help/debates.
instigator. No matter which case, once the contender enters, the debate starts, and both sides publish their arguments. If any side fails to propose an argument within the time limit, they will automatically forfeit the round; in our work, debates with forfeited turns are treated as incomplete.
When all arguments have been published, the community or the appointed judges select the debate’s winner by voting. Voters need to follow the specified voting system and give fair verdicts. The debate is finished when the winner has been selected according to the votes.

heading: B.2 DebateArt Debate Collection
text: To collect debates with valid content and votes, we first crawled the list of finished debates on DebateArt and then filtered out debates that had no valid votes or were interrupted (not using all preset rounds). Next, we crawled the debate details of the remaining debates, including topic (motion), debaters, description (info slide), arguments (speeches), and votes. The raw arguments are in HTML format; we use markdownify6 to convert them into Markdown documents.
We further filtered out debates that do not fit our benchmark, including debates that do not use the categorical point assignment system, are not formal, and contain very short speeches, which may indicate a forfeit. Although many are close to being professionals, we also excluded some very long debates to ensure that inputs do not exceed the LLM’s context window during the experiment.

heading: B.3 BP Debates
text: The British Parliamentary (BP) format is a widely accepted competitive debate style, followed by famous competitions like WUDC, EUDC, and NAUDC (World Universities Debating Council, 2023). Each BP debate contains four teams, with a total of eight debaters. There are two teams on each side of the debate: on one side are Opening Government (OG) and Closing Government (CG); on the other side are Opening Opposition (OO) and Closing Opposition (CO). They follow the order specified below to give speeches:
• OG Speaker 1 (“Prime Minister”);
• OO Speaker 1 (“Leader of Opposition”);
• OG Speaker 2 (“Deputy Prime Minister”); 6https://github.com/matthewwithanm/py
thon-markdownify
• OO Speaker 2 (“Deputy Leader of Opposition”);
• CG Speaker 1 (“Government Member”);
• CO Speaker 1 (“Opposition Member”);
• CG Speaker 2 (“Government Whip”);
• CO Speaker 2 (“Opposition Whip”).
Each speech lasts for 7 minutes, with limited tolerance for timeouts. In general, OG should define the motion, propose arguments, and refute arguments from OO; OO should rebut OG’s case and propose constructive arguments for their side; CG and CO should provide further supplementary analysis in favor of their side, respectively.
Points of Information (POI) is a special feature of BP debates. A POI is a formalized interjection from any debater on the opposite side to the current speaker. The current speaker has the right to decide whether the POI is accepted or rejected; once accepted, the debater offering the POI can make an argument or ask a question within 15 seconds, and the current speaker should respond properly before continuing their speech. We mark POI conversations as quoting blocks and prompt LLMs to pay attention to them, as engaging in POIs may contribute to the debaters’ overall performance.

heading: B.4 Competition Debate Collection
text: Many debate competitions, including WUDC, EUDC, and NAUDC, only provide essential information about debates, such as motions, info slides, teams, and winners. They do not have official transcriptions; unofficial ones are often incomplete. Fortunately, in recent years, many of these competitions have provided official video recordings of debates not long before the finals.
We selected debates starting from the quarterfinals from WUDC (2020-2023), EUDC (2019- 2022), and NAUDC (2021 and 2023) and downloaded their video recordings. Next, we extracted audio files from the recordings and used Whisper (Radford et al., 2023) to recognize the speeches. We manually checked and formatted the output results into valid transcriptions.
Not every debate was available for transcription due to missing, damaged, or incomplete recordings; we only kept debates whose transcription was complete. Finally, we merged their transcriptions with debate information to produce the final data.

heading: C Prompt Template
text: Tables 8, 9 and 10 demonstrates the prompt templates in our experiments. For BP-Competition debates, we insert an extra section describing BP debate characteristics in the system message right after the debate information section, as shown in Table 11. Dimensional preferences of each stage are available in the PanelBench release.

heading: D Additional Experiments

heading: D.1 Statistical Significance
text: We conducted paired t-test (for DebateArt debates) and McNemar’s test (for BP-Competiton debates) to measure the statistical significance of Table 5. Table 12 demonstrates the results. We can see that the improvements by Debatrix are significant in most cases. Moreover, Debatrix successfully judged all BP-Competition debates without showing severe bias, as can be seen in Tables 5 and 7, while other baselines could hardly achieve this.

heading: D.2 Dimensional Performance on DebateArt
text: Table 13 extends Table 6 to all dimensions for DebateArt debates. Compared with Dimensional, Debatrix improves both the argument and language dimensions and the source dimension when predicting the winner directly. NonIterative is especially good at judging the language dimension, probably because it is strongly related to the speech rather than its content.
D.3 VivesDebate
As mentioned in Section 2, Ruiz-Dolz et al. (2023) proposed an automatic debate evaluation model that utilized argumentation semantics related to persuasiveness. They conducted experiments on VivesDebate (Ruiz-Dolz et al., 2021), a dataset containing 29 debates from a Catalan debate tournament. The debate structure of VivesDebate debates is similar to DebateArt ones, except that the final turn is inverted so that the first speaker speaks the last. Each debate is judged under 3 dimensions, and scores are available for both sides. However, all debates from VivesDebate share the same motion, hence lacking diversity; moreover, the speeches are in Catalan rather than English7. For these reasons,
7The English translation mentioned in Ruiz-Dolz et al. (2021) is not released to the public; the Argumentative Discourse Units (ADUs) are available in English, yet they cannot form complete speeches.
we proposed PanelBench, which covers competitive debates in English with various motions and structures, and we conducted experiments on it.
Nevertheless, as a (loose) comparison between their model and ours, we applied Debatrix and other baseline models to the translation (obtained with Google Translate) of VivesDebate debates. Among all 29 debates, we selected 25 of them whose transcriptions and dimensional scores are complete. We followed the dimension and weight settings in VivesDebate.
Table 14 lists the winner prediction performance on VivesDebate debates. Debatrix consistently performed better than ChatGPT; GPT-4 is exceptionally good in this case, possibly because the speaking order switch in the last turn has alleviated positional bias. Note that Ruiz-Dolz et al. (2023) split the debates into train and test sets, where the test set only contains 6 debates; therefore, their results cannot be directly compared with ours. Furthermore, their model heavily relies on human annotation on the speeches, while Debatrix can judge the debate end-to-end.

heading: D.4 Case Study
text: Table 15 demonstrates the debate analyses generated by Dimensional and Debatrix on the argument dimension of the DebateArt debate “The existence of God is impossible” between Type1 and Barney. Dimensional failed to notice the drawbacks of Type1’s arguments, which eventually caused it to give the wrong verdict. In contrast, Debatrix pointed out that Type1’s arguments lack clarity in backing premises and examples and awarded Barney as the winner, matching human votes. This can be seen in the content analyses Debatrix generated, listed in Table 16. As mentioned in Section 3, these content analyses can not only improve the final verdict but also be used to assess individual speeches.
Table 17 shows another case where the motion and winner’s policy are the opposite of the one in Table 15, this time comparing NonIterative and Debatrix. Human votes indicate that both debaters perform well, matching both models’ verdicts. However, Debatrix successfully predicts the correct winner (Brutal) by introducing a standard closer to the debate setting. These two cases also show that Debatrix focuses more on the content instead of basing their judgments on a priori knowledge or belief (whether God exists or not),

heading: System:
text: As an AI with expertise in competitive debating, you’re serving as a judge on a panel. The debate motion is: <debate motion> Pro side debaters are: <pro debaters> Con side debaters are: <con debaters> The debate consists of <speech count> speeches. Now, <new speech speaker> gives Speech <new speech id>. You are given the debate info slide. Also, you have analyzed all previous speeches made in the debate. <content analysis dimensional preference> Please think critically before responding.
User:

heading: System:

heading: User:

heading: System:
text: As an AI with expertise in competitive debating, you’re serving as the main judge on a panel. The debate motion is: <debate motion> Pro side debaters are: <pro debaters> Con side debaters are: <con debaters> The debate consists of <speech count> speeches. Now the debate ends. Your team of specialized judges, each assigned to a dimension, have given their judgments according to their assigned dimension respectively. You are given the dimensions and corresponding judgments from your team. Please summarize the dimensional judgments according to their weights into a general judgment. Rank the <debater count> debaters (<debaters>) individually; award one and only one individual debater that performs the best. Please think critically before responding.

heading: User:
text: alleviating potential ethical bias within LLMs. Tables 18 and 19 demonstrate debate analysis by GPT-4 (using all-in-one judge preference) and Debatrix (using dimensional collaboration) of the BP-Competition debate “This House Would transfer management of all water resources to an international body.” Although GPT-4 generated a much more detailed response, it leaned towards CG and CO throughout the analysis and eventually chose CO as the winner. Debatrix, in this case, correctly selected OG and CG as the best debaters in the debate, matching the actual result.
Debate Motion: The existence of God is impossible Pro Debater: Type1 Con Debater: Barney Argument Winner: Barney (0 : 2)
Dimensional:
In this debate, Type1 argues for the Pro side, affirming that the existence of God is impossible. Type1’s main argument revolves around the causality problem, stating that for God to exist, it would have to come from nothing and possess knowledge without any prior information, which is logically impossible. Type1 also dismisses the argument that a simulation could be considered as God, emphasizing the specific definition of God as an all-powerful being. Type1’s arguments are logically structured, clear, and supported by reasoning. However, Type1’s tone can be perceived as confrontational and dismissive towards the opponent’s arguments. On the Con side, Barney argues against the motion, claiming that the existence of God is not impossible. Barney counters Type1’s causality argument by suggesting that not understanding the cause of something does not make it impossible. Barney also brings up the point that a significant portion of the world’s population believes in God, indicating that the concept of God’s existence is not universally deemed impossible. Barney also challenges the idea that a simulation could not be considered as God, highlighting the ambiguity in defining God. Barney’s arguments are supported by references and attempt to show the plausibility of God’s existence through various perspectives. In terms of argumentation, Type1 presents a strong case based on logical reasoning and the laws of causality, while Barney focuses on challenging the notion of impossibility and highlighting the subjective nature of defining God. Type1’s arguments are more focused and directly address the core of the motion, while Barney’s arguments are more varied and attempt to introduce different angles to the debate. Overall, in this debate, Type1’s arguments appear more coherent, logically sound, and directly related to the motion at hand. While Barney brings up valid points regarding belief systems and perspectives, Type1’s arguments regarding causality and the impossibility of God’s existence are more compelling within the context of the debate. Therefore, in this debate, Type1’s arguments hold more weight, and they effectively affirm the resolution that the existence of God is impossible. Debatrix:
Type1 consistently argues for the impossibility of God’s existence based on causality, emphasizing the logical constraints that make it impossible for a sentient and all-knowing being to exist without a causal basis. Type1 also challenges Barney’s points by critiquing appeals to authority and broad definitions of existence, maintaining a focus on the specific definition of God within the debate context. However, Type1’s arguments lack clarity in backing premises and could benefit from more detailed explanations and examples to strengthen their reasoning. Barney, on the other hand, effectively counters Type1’s arguments by challenging the notion of impossibility and expanding the discussion to include different perspectives on reality and existence. Barney introduces strong counterarguments that broaden the scope of the debate, such as the simulation hypothesis and the impact of beliefs on individuals and societies. Barney also utilizes logical fallacy identification and appeals to authority to add credibility to their points. In terms of clashes and relative strength, Type1’s emphasis on the impossibility of God’s existence based on causality clashes with Barney’s arguments for the possibility of God’s existence within alternative frameworks and the impact of beliefs on reality. The clash between certainty and possibility, as well as the distinction between physical and conceptual existence, forms the core of the debate. While Type1 maintains a strong logical stance, Barney’s broader perspectives and counterarguments challenge the rigid impossibility claim effectively. Considering the depth of arguments, logical coherence, and the ability to counter and broaden the debate effectively, Barney presents a stronger case in negating the resolution. Barney’s ability to challenge Type1’s arguments, introduce alternative perspectives, and provide logical critiques showcases a more comprehensive approach to the debate. Therefore, in this context, Barney would be awarded as the debater who negates the resolution the best.
Table 15: Debate analysis by Dimensional and Debatrix on the argument dimension of DebateArt debate “The existence of God is impossible” between Type1 and Barney.
Debate Motion: The existence of God is impossible Pro Debater: Type1 Con Debater: Barney Argument Winner: Barney (0 : 2)

heading: Analysis of Speech 1 by Type1:
text: Type1’s argument revolves around the idea that the existence of God is causally impossible because God would have to spontaneously appear from nothing as a fully developed sentient and all-knowing entity. The argument suggests that there would be nothing prior to God’s existence to create God, and nothing for God to know since nothing else would exist initially. The argument also touches on the concept of a primordial quantum loop or "possibility cloud" from which God supposedly spawned, but dismisses this as equally impossible due to the contradictory nature of everything both infinitely existing and not existing at the same time. The argument presented by Type1 seems to be on topic, as it directly addresses the motion that the existence of God is impossible. However, the argument lacks clarity in its backing premises and logic. While it raises valid points about the challenges of explaining the existence of God from nothing, it does not delve into the nuances of theological or philosophical arguments for the existence of God. Additionally, the argument could benefit from providing more detailed explanations or examples to strengthen its reasoning. The reference provided by Type1 in the speech is a link to an external source discussing the impossibility of God’s existence. While it is good to see the debater engaging with external sources, it is important to incorporate the arguments and evidence from these sources into the speech itself rather than just referencing them. This would enhance the credibility and depth of the argument presented. Overall, Type1’s speech introduces an interesting perspective on the impossibility of God’s existence, but could be further developed with clearer premises, logical connections, and incorporation of external sources within the speech itself.

heading: Analysis of Speech 2 by Barney:
text: Debate Motion: The Claim That The Christian God Exists Pro Debater: Mopac Con Debater: BrutalTruth Argument Winner: BrutalTruth (3 : 2)
NonIterative:
In summarizing the arguments presented by each debater in all their speeches: **BrutalTruth** consistently challenges the basis of the claim that the Christian God exists by critiquing the lack of empirical evidence, questioning the reliance on dictionary definitions, and emphasizing the burden of proof on the Pro side. They also highlight the logical fallacies in their opponent’s arguments and stress the need for tangible evidence to support extraordinary claims. **Mopac** consistently argues that the Christian God exists as The Supreme and Ultimate Reality, equating God with The Truth itself. They draw from Christian theology, biblical references, and dictionary definitions to support their claim. Mopac emphasizes the self-evident nature of The Truth and counters demands for empirical proof by asserting the inherent existence of The Ultimate Reality. In analyzing the relative strength of arguments and impacts against each other: **BrutalTruth** effectively challenges Mopac’s assertions by pointing out logical fallacies, lack of empirical evidence, and the need for tangible proof of the Christian God’s existence. They consistently emphasize the burden of proof and the importance of empirical evidence in supporting extraordinary claims. However, their arguments primarily focus on questioning the validity of their opponent’s claims rather than providing a strong affirmative case against the existence of the Christian God. **Mopac** maintains a strong theological stance by equating the Christian God with The Truth and The Ultimate Reality. Their arguments are consistent, coherent, and directly address the core of the debate motion. Mopac effectively counters challenges regarding empirical evidence by asserting the self-evident nature of The Truth and The Ultimate Reality. They provide a clear theological framework for understanding the Christian God that goes beyond empirical observations. In terms of clashes between both sides’ argument sets: - **Definition of the Christian God**: Mopac defines the Christian God as The Supreme and Ultimate Reality, while BrutalTruth insists on defining the Christian God based on the Christian Bible. This clash revolves around the source of authority for defining the Christian God. - **Evidence and Proof**: Mopac’s argument relies on the concept of the Christian God being The Truth, while BrutalTruth challenges the lack of substantial evidence or proof to support this claim. The clash highlights the need for tangible evidence to support assertions about the Christian God’s existence. - **Nature of Truth**: BrutalTruth questions the nature of truth as an intellectual idea, while Mopac asserts that The Truth is self-evident and equates it with the Christian God. This clash revolves around the conceptualization of truth and its relationship to the existence of the Christian God. Considering the arguments presented and their impacts on the debate, **Mopac** emerges as the stronger debater in affirming the resolution that the Christian God exists. Their consistent theological framework, supported by biblical references and dictionary definitions, provides a coherent and logical basis for asserting the existence of the Christian God as The Supreme and Ultimate Reality. While BrutalTruth raises valid points regarding the burden of proof and empirical evidence, Mopac’s theological perspective effectively counters these challenges and maintains a strong affirmative stance throughout the debate. Debatrix:
In analyzing the debate between Mopac and BrutalTruth on the claim that the Christian God exists, both debaters presented strong arguments rooted in theological and epistemic reasoning. BrutalTruth’s arguments primarily focused on the need for empirical evidence to justify belief in the existence of the Christian God. They challenged Mopac’s reliance on textual sources and definitions, emphasizing the importance of empirical verification. BrutalTruth’s use of logical fallacies, such as Argumentum ad Dictionarium, effectively critiqued Mopac’s approach and set a higher evidentiary standard for proving the existence of God. On the other hand, Mopac’s arguments centered around the concept of The Ultimate Reality as synonymous with the Christian God. They clarified the nature of The Truth and The Ultimate Reality, emphasizing God’s existence as inherent in these concepts. Mopac’s responses to challenges from BrutalTruth focused on reinforcing the theological understanding of God as The Supreme and Ultimate Reality, transcending human comprehension. In terms of argument strength, both debaters presented coherent and logically structured points. BrutalTruth’s emphasis on empirical evidence and logical fallacies provided a critical analysis of Mopac’s arguments, challenging the basis of belief in the Christian God. Mopac, on the other hand, effectively linked the concept of The Ultimate Reality to the existence of God, presenting a theological perspective on God’s nature and attributes. In evaluating the clashes between the debaters, BrutalTruth’s focus on empirical evidence clashed with Mopac’s theological arguments based on The Ultimate Reality. The clash between empirical verification and theological interpretation highlighted the different epistemic approaches to proving the existence of the Christian God. Ultimately, in this debate, the strength of arguments lies in the ability to meet the burden of proof set by the resolution. While both debaters presented compelling arguments, the emphasis on empirical evidence and logical reasoning by BrutalTruth aligns more closely with the standards of proof required in a debate setting. Therefore, in this context, the debate would be awarded to BrutalTruth for effectively challenging Mopac’s arguments and setting a higher evidentiary standard for proving the existence of the Christian God.
Table 17: Debate analysis by NonIterative and Debatrix on the argument dimension of DebateArt debate “The Claim That The Christian God Exists” between Mopac and BrutalTruth. Note that the motion is the opposite of the one in Table 15, and so is the winner’s policy.
Debate Motion: This House Would transfer management of all water resources to an international body. Pro Debater: OG, CG Con Debater: OO, CO Overall Winner: OG, CG
GPT-4:
Debatrix:
Based on the assessments across different dimensions, here is the ranking of the debaters from the strongest performance to the weakest: 1. **OG (Opening Government):** OG presented a comprehensive and coherent argument framework in favor of transferring water management to an international body. They effectively engaged with the opposition’s concerns, provided strong empirical evidence, and had a clear and logical structure to their arguments. Their language was clear and precise, and they effectively supported their case with a good balance of logical mechanisms and moral intuitions. 2. **CG (Closing Government):** CG built on OG’s arguments and provided further analysis supporting the transfer of water management to an international body. They effectively engaged with the opposition’s points, presented a well-developed argument framework, and had a clear and assertive language style. While they lacked direct empirical evidence, their logical reasoning and moral intuitions were strong, contributing to a compelling case. 3. **OO (Opening Opposition):** OO presented a well-structured and coherent argument framework opposing the transfer of water management internationally. They raised valid concerns about sovereignty and property rights, engaged effectively with the government’s arguments, and had a good command of language. However, they lacked empirical evidence to strengthen their case, which impacted the overall credibility and sufficiency of their arguments. 4. **CO (Closing Opposition):** CO provided a comprehensive and coherent argument framework against transferring water management internationally. They effectively engaged with the government’s points, presented logical mechanisms, and appealed to moral intuitions. However, they also lacked direct empirical evidence to support their arguments, which could have enhanced the credibility and sufficiency of their case. In summary, OG demonstrated the strongest performance in the debate, excelling in arguments, clash, language, and support. CG followed closely behind with a strong case and effective engagement with the opposition. OO and CO also presented compelling arguments but could have strengthened their positions with more empirical evidence and clearer engagement with the opposing side.
Table 19: Debate analysis by Debatrix of the debate in Table 18, summarized from dimensional debate analyses.

heading: Future Work
text: Despite the above results, this paper has a few limitations for which we appreciate future studies. First, while the iterative chronological analysis in Debatrix already shows competence, it can be further studied and polished. Second, the position bias issue on GPT-4 for BP-Competition debates needs further investigation, and more powerful tools are needed to rectify this phenomenon. Ethical Impact
As a debate judging framework based on LLM, Debatrix may be exposed to ethical risks if the backbone LLM is biased beforehand, especially when judging debates concerning political, social, or cultural motions. In our experiments, however, using LLMs with strong ethical constraints like ChatGPT and GPT-4 has resulted in no significant ethical bias within the generated verdicts. Furthermore, as demonstrated in Appendix D. 4, Debatrix guides LLMs to focus on the content instead of basing their judgments on a priori knowledge or belief, alleviating potential ethical impact