Classification of Arguments according to Walton Schemes: Zero-Shot versus Means-to-End Prompting
Authors: Maximilian Heinrich, Khalid Al Khatib, Benno Stein


Reviewer paper summaries: The paper proposes solving the argument classification with LLMs as a series of questions using Visser's decision tree. Authors test their approach on three datasets: Araucaria, ReCAP, and US2016WALTON. Three LLMs take part in the evaluation: GPT-3.5, Gemini-1.0-Pro, and LLaMA-3.

The findings show that the algorithm-driven approach works best with the dataset in which the annotations are based on the same decision tree.
Summary of strengths: The proposed method integrates a human-curated decision tree and LLM prompting.
Summary of weaknesses: This paper explores the application of LLMs to a previously proposed argument classification algorithm (Visser's DT). While the concept holds merit, the execution requires significant revision. The key issues lie in the description of the experiments and the data processing choices.

1. The paper would be strengthened by a clearer explanation of the Reed-06, Dumani-21, and Visser-21 datasets. It is important to understand how these modified versions relate to the original datasets mentioned elsewhere by their original names.

2. In the Related Work section at [269-270], the statement "This idea has also been demonstrated before." raises concerns. Firstly, it lacks any references to support the claim. Secondly, it contradicts the description in the Abstract where the algorithm-driven prompting is presented as "probably the most original contribution" of the paper. This inconsistency needs to be addressed.

3. The paper differentiates between _zero-shot_ prompting and the proposed _algorithm-based_ prompting. However, since both involve no ground truth question-answer examples, technically they are both zero-shot approaches. To ensure consistent terminology, the authors could revise either the naming of their method or clarify the distinction between these prompting techniques.

4. Lines [282-289] require further explanation.

     a. Firstly, it mentions matching ADUs with original sentences. However, the paper previously stated there are no source texts in these corpora. Clarification is needed on the origin of the original sentences.

     b. Secondly, the use of embeddings for matching seems unnecessary since ADUs are the exact sentence fragments. A direct matching approach would likely be more efficient. 

     c. Does the entire text where ADUs are mentioned function as an argument in their setting?

5. A potential issue arises with the proposed method of grouping general classes _after_ filtering. Figure 1 reveals that this approach can group classes from entirely different branches of Visser's tree (e.g., 4 and 1, 34 and 13). This is evident in the merging of classes 13 and 34 ("Alternatives"), which neighbor subtrees representing distinct concepts: "Cases" (13:6-7) and "Consequences of action" (34:36). This merging likely contributes to the high number of misclassifications where the model confuses "Consequences of action" with "Alternatives," as shown in the confusion matrices.

6. The paper focuses on precision metrics, and F1 evaluations, which could be useful for comparing different approaches, are not reported. It would be beneficial to understand the reasoning behind this choice.

7. [484-485] The authors leverage the existing decision tree proposed by Visser et al., rather than introducing a novel one.
Comments, suggestions and typos: 1. Captions for Tables 1-13 should be placed below the tables for better readability.

2. Citations throughout the paper (lines [056-057], [158-159], [190-191], and [440-441]) require formatting revision. Possible revisions are "existing debate systems, such as those proposed by Rakshit et al. (2019), Le et al. (2018), and Slonim et al. (2021)" or the in-text citation style "existing debate systems (Rakshit et al., 2019; Le et al., 2018; Slonim et al., 2021)".

3. The statement "(see Appendix)" on line [073] should explicitly reference the relevant table (e.g., "see Table 7 in the Appendix").

4. The caption for Figure 1 should include a cross-reference to Table 7 in the Appendix for clarity.

5. The relevance of the Newell and Simon (1995) citation on line [116] needs to be reevaluated. Consider removing it if it doesn't directly connect to the topic.

6. The Moens et al. citation on line [195] seems redundant if the dataset and the previously listed papers already include genre information.

7. [241-243] This text should be revised and elaborated.

8. Tables 2 and others that use a 0.xx format for precision and recall could be improved by presenting the data in F1 score (%) format for easier comprehension. This applies similarly to figures with confusion matrices annotated with 0.x0 values.
Overall assessment scores: 2