Node Placement in Argument Maps: Modeling Unidirectional Relations in High & Low-Resource Scenarios
Year: 2023

Abstract:
Argument maps structure discourse into nodes in a tree with each node being an argument that supports or opposes its parent argument. This format is more comprehensible and less redundant compared to an unstructured one. Exploring those maps and maintaining their structure by placing new arguments under suitable parents is more challenging for users with huge maps that are typical in online discussions. To support those users, we introduce the task of node placement: suggesting candidate nodes as parents for a new contribution. We establish an upper-bound of human performance, and conduct experiments with models of various sizes and training strategies. We experiment with a selection of maps from Kialo, drawn from a heterogeneous set of domains. Based on an annotation study, we highlight the ambiguity of the task that makes it challenging for both humans and models. We examine the unidirectional relation between tree nodes and show that encoding a node into different embeddings for each of the parent and child cases improves performance. We further show the few-shot effectiveness of our approach.

heading: 1 Introduction
text: Online discussions can have huge numbers of contributors and contributions, making the discussion hard to follow for new users. Getting an overview of a discussion and finding points of interest for a new user might be hard in such an unstructured format which is also prone to redundancy. Argument maps, in their simplest form, structure arguments into a tree with each node being a pro or contra argument for its parent node (also an argument, see Figure 1). Relying on the structure of the map, users can dive deeper into specific aspects of an argument and collectively add more arguments to support or oppose it: this improves the overall quality of the discourse and at the same time, triggers
the generation of new ideas and the continued discussion of existing ones. For an argument map to serve its purpose, it is essential to keep a somewhat clean structure, but this can be challenging for larger maps since finding where to add a new contribution can become a tedious task, and where the user initially decides to add their contribution based on their limited exploration of the map might be a sub-optimal choice.
To support in creating argument maps, we propose node placement as a new task defined as finding suitable candidates from an argument tree to be the parent of an argument. Deciding if an argument is pro or contra its parent is not a focus here as it does not constitute a bottleneck when adding a new contribution (binary decision vs. choosing the suitable parent from possibly hundreds of nodes). A number of nodes could be suitable as parents at varying or more similar degrees making the task inherently ambiguous (nodes n3, n8 are equally
5854
suitable in Figure 1). The effect of this could be loosened by presenting the user with top-n recommendations (visualized using color shades similar to Figure 1). The task then could be employed to support users in two ways: 1) exploration: the user enters a short keywords-argument and based on its suggested node placement, finds the most relevant places in the map to explore; 2) optimization: after a user is done writing the argument, node placement suggestions are used to better place the final new contribution (example in Figure A.5). The task could be also seen as a first step to automatically and incrementally build argument maps from unstructured discussions or to enrich existing argument maps based on those discussions.
We use publicly available argument maps from Kialo1 (where users manually & collectively maintain discussions in maps) and conduct an annotation study on a sample of nodes with 10 challenging candidates per node, in order to gain insights about the task and estimate human performance. We highlight the challenging nature of the task even with this low number of candidates. We formulate the task as a ranking problem and conduct modeling experiments using sentencetransformers with large and small models and a variety of intermediate-task training. We show that more intermediate-task training yields better results, and that the performance of the large trained models is on par with humans on our annotated samples. We highlight the unidirectional nature of the relationship between child and parent nodes showing that it is beneficial to decouple the parent and child representations of the same node. To address this, we propose using different textual templates for the child vs. parent representation of a node (see embeddings in Figure 1), and show a boost in performance as a result (∼4, 3 points for top1, top5 accuracy of the large model). We further examine the data-efficiency of our training strategies in low-resource scenarios where the number of maps and/or the size of maps are small. We show that the zero-shot performance is still relatively good and is consistently improved with fewshot training even with a small number of samples and that using templates is especially beneficial for a smaller number of samples.
Our main contributions are summarized as: 1) Defining a new task, node placement in argument maps; 2) Estimating human performance on
1https://www.kialo.com
the task through an annotation study; 3) Conducting modeling experiments, proposing a simple approach to tackle unidirectional relations between text pairs, and employing this to improve the performance of the proposed task; 4) Demonstrating the effectiveness of our training in low-resource scenarios. A detailed analysis of the results is also conducted to gain insights into our task and method. Our code is made publicly available2.

heading: 2 Related Work
text: Node Placement in Argument Maps The task is related to two widely explored tasks in Argument Mining: argument retrieval and modeling argument relationships. Argument retrieval can be viewed as a more general form of node placement, in which a system should provide relevant arguments given a controversial question or topic (Stab et al., 2018; Reimers et al., 2019; Bondarenko et al., 2021) or a suitable counter-argument given an input argument (Wachsmuth et al., 2018). With respect to general argument retrieval, our task tackles a finergrained problem: finding suitable positions in the argument tree. Regarding the modeling of the relationship between arguments (Stab and Gurevych, 2017), e.g. support/attack, only few works consider it in the context of a full, structured debate. To automatically construct argument graphs, Lenz et al. (2020) use structured debates to classify relationship between argumentative units. Agarwal et al. (2022) model the relationship between arguments as a polarity prediction task using the tree-structure, and exploiting the ancestors of a node to classify support/attack relationships. Retrieval & Ranking: Cross-encoders like BERT (Devlin et al., 2019) can be used to score pairs of sentences, but this does not scale well for large number of candidates in retrieval & ranking tasks. Siamese networks (Bromley et al., 1993) (also called bi-encoders) have long been used to create embeddings for efficiently tackling those tasks using contrastive learning. sentence-transformers (Reimers and Gurevych, 2019) employs this by using BERT or other Transformer models and utilizing labeled data while ConSERT (Yan et al., 2021) and SimCSE (Gao et al., 2021) also utilize unlabeled data. The original BERT can be used to encode each sample (of a pair) into a vector for more efficiency, but the resulting embeddings have high
2https://github.com/imanjundi/ argument-relations
similarity in general which BERT-flow (Li et al., 2020) and WhiteningBERT (Huang et al., 2021) tackle using normalization. We utilize bi-encoders and pay attention to the high similarity issue with a thorough analysis of the effect our approach has on the embedding space. Templates & low-resource: Templates were recently heavily used in prompts to tap into the knowledge encoded in large PLMs and to make use of their few-shot capabilities by using a taskinformation template (Petroni et al., 2019; Brown et al., 2020). They were also used to fine-tune PLMs in a few-shot setup (Schick and Schütze, 2021; Tam et al., 2021; Liu et al., 2022) while others attempted to do away with them (Logan IV et al., 2022; Karimi Mahabadi et al., 2022; Tunstall et al., 2022), but they were mainly employed to directly solve NLP tasks and not to learn embeddings. Prompts were used for the latter more recently (Jiang et al., 2022) with contrastive learning. We simply utilize templates with no language modeling training or inference and show they are beneficial with contrastive learning to learn embeddings in high and low-resource.

heading: 3 Data
text: We use argument maps from Kialo, an online platform on which people engage in discussions on specific topics or statements. In a discussion about a controversial thesis topic, the thesis acts as a root node under which further and increasingly more specific arguments for or against this point of view can be added. An example of how the original data looks like is shown in Figure 2 (taken from Kialo 3). It shows how the tree of arguments evolves for the root node or thesis “Video game storytelling should portray gender equality.” Users can navigate through the tree to find aspects of the discussion that they are interested into or to find a good node to attach their new input to. Each new argument can in turn be attacked or supported with a variety of different arguments. Thus every debate in Kialo represents a unidirected tree, where each edge represents a support or attack relation (henceforth, pro and con).
We rely on data from Agarwal et al. (2022) and use a total of 1,378 maps covering a wide variety of topics: politics, technology, ethics, etc. (overview in Appendix Figure A.2). The majority contain up to 200 nodes, but a quarter of the data are large-
3https://www.kialo.com
scale discussions (up to 6k nodes) for which an automatic support is especially beneficial (complete analysis in Section A.1).

heading: 4 Annotation Study
text: To have a better understanding of the task and data, generalize a baseline of human performance, and, estimate the difficulty and cost humans encounter with such a task, we conduct an annotation study. Design: We employed 3 annotators with a background in NLP and Social Sciences (details in Section A.2. The annotators were presented a specific contribution to a discussion4 – the child – and 10 candidate parents selected from the discussion’s argument map to which the child could attach. The annotators were tasked to classify each of the candidates with one of the following labels: BEST PARENT (count 1), SUITABLE PARENT (max. 4), or LESS SUITABLE PARENT. The annotation guidelines and an example are provided in Appendix Figures A.6 and A.7.
In order to control for an appropriate variety of candidates that a user might encounter, the candidates consisted of the actual parent, 6 candidates closely related in the tree to the child (with a maximum path distance of 3), and 3 randomly selected candidates from the full tree. In total, the annotated dataset consists of 200 child instances. The instances selected were evenly split between pro relations with its parent and con. The nodes were sampled from small and large-scale maps (90 to 2500 nodes). The topics of the maps are environment, economy, gender, politics and immigration.
To better understand the annotators’ approach to the task, we asked them to provide their confidence scores for each annotation, as well as short-answer motivation for a subset of 100 annotations. Annotation Results: We measure the annotator agreement using weighted Kappa (κw) (Cohen, 1960) as we would like to account for the seriousness of the disagreements, i.e. disagreeing about BEST and SUITABLE should be penalized less than BEST and LESS SUITABLE. The annotators have a fair to moderate agreement of 0.387. While we can conclude that the participants generally agreed, κw in the lower range of agreement is an indication of the difficult and subjective nature of this task, despite the clear guidelines and training.
4The topic of the discussion was also provided to the annotators for context in each instance.
To calculate the performance, we convert the human annotated labels for each child into a score and thus obtain a ranked list of all candidates5. Table 1 shows the aggregated performance of the annotators in being able to select the best parents for each child (metrics in Section A.3). The average top1 is just under 50%, meaning the participants were not necessarily able to select the best parent among the 10 candidates with ease. On the other hand, the average top5 is quite high. This contrast suggests that the best parent is often ambiguous; while distinguishing between a set of those that might be the best parent (conflating BEST and SUITABLE) and those that are certainly not (LESS SUITABLE) is less ambiguous and easier to define for the participants.
We find that for the participants it was easier to determine the best parent in cases of a con relationship between parent and child. Contributions to a discussion of this class are generally presented as negations to specific points in the parent comment, and most likely contain high lexical overlap. For example, the child The boys referred to appear
5Each label corresponds to a number (BEST:1, SUITABLE:2, LESS SUITABLE:3) and the score for a candidate is the average of all three numbers. The lower the score, the higher the candidate will be ranked.
to be having fun, rather than trying to hurt each other, in response to the parent It shows the harm boys can do when people allow bad behaviour because “boys will be boys.” Contributions that are instead of a pro relationship to the parent are more often an elaboration or extension of the argument made in the parent comment, likely resulting in less pronounced links between parent and child. For example, the child There is poor cooperation between the Commission and national financial regulators, in response to the parent The Emissions Trading System is susceptible to fraud.
Our analysis of the motivation behind the annotators’ choice shows it can be divided into 3 categories, in decreasing frequency: (1) Process of elimination, described often as “most obvious”, “best fit”, or “makes most sense”; (2) Linguistic overlap, reported as similar use of terminology or structure; and (3) Logical connection, in which participants found a direct child/parent relationship, such as an elaboration or offering examples.
Note that the task given to the annotators is rather simple in comparison to a real-world application where dozens, if not hundreds, of options across the full argument map would need to be considered. That said, the three annotators averaged a total of 31 hours to complete the 200 annotations. Clearly, this cost can be greatly reduced for users or moderators of the argument maps with a filtered shortlist of candidates provided by a model trained on our proposed task: node placement.

heading: 5 Main Modeling Experiments
text: The task can be formulated as a ranking problem where a score is predicted for each candidate node and used to rank all candidate nodes. We use a biencoder to scale to the huge number of nodes that each map might contain (up to 6k nodes, c.f. Section 3). Using a cross-encoder that scores each pair of nodes is not feasible to apply on all nodes, but could still be used to refine the ranking of the top-n candidates based on the scores from the biencoder. This re-ranking step is out-of-scope given the noisy data available and the ambiguous nature of the task, as seen in the annotation study, which makes judging the final ranking not feasible. Unidirectional Relation Between Nodes Common sentence or document embedding methods assume a bi-directional relation. For example, in the case of semantic similarity: if sentence1 x1 is similar to sentence2 x2 then sentence2 is also similar to sentence1 so: F (x1, x2) = y ↔ F (x2, x1) = y This is clearly not the case when representing parent/child relation so: F (x1, x2) = y ↛ F (x2, x1) = y This makes encoding the tree nodes into an embedding space challenging since the training should make the child nodes (c1,...,cn) closer to their parent node (p), but when p is considered as a child node with the aim of predicting its parent, it should still be closer to its parent (g) than to (c1,..,cn). The latter should be in this case regarded as negative training examples. The recursive structure of the tree might increase the effect of this issue since all nodes are eventually connected to the root.

heading: 5.1 Templates
text: Motivated by the successful use of prompts in related work, we use templates to better represent the unidirectional relation, exploit the stance label and utilize the knowledge encoded in the model. By encoding those signals textually through templates, they are passed through the model which allows for effective interaction with other features while keeping the approach simple. Our templates are: parent/child: differentiate the parent vs. child by using parent:"text" when considering the node as a candidate parent for another node vs. child:"text" when considering the node as the child. This allows us to have two different embeddings for each node. The resulting training data has the same size as the original.
pro/con: represents pro & con child nodes using pro:"text" & contra:"text" which we add to parent/child template samples for training, and use parent/child template (main template) for evaluation. Using only pro/con templates would otherwise complicate evaluation since it results in two different rankings of the candidates: one when considering the node as pro for a candidate and one for con. The resulting training data is 2x the original size. all: includes pro/con templates and 3 templates that use similar keywords while combining child & parent text during training e.g. pro:"text" parent: "text" (see Appendix Table A.2 for all templates). The resulting training data is 5x the original size.

heading: 5.2 Experimental Setup
text: Each argument in a map is encoded into an embedding using sentence-transformers6 (Reimers and Gurevych, 2019). Cosine similarity is used between the embeddings of a node and all possible candidates to calculate the scores. Experiments use models with varying size and intermediatetask training (Pruksachatkun et al., 2020) based on a large model, MPNet (Song et al., 2020) and a smaller one, MiniLM (Wang et al., 2020)7: mpnet without intermediate training, nli-mpnet with intermediate training on MNLI and SNLI, paraphrasempnet with additional paraphrase data, all-mpnet with additional QA and other data, and finally allmini with similar training but based on MiniLM (models overview in Appendix Table A.1).
The argument maps in the dataset are split into 80% train, 20% test (1102 and 276 maps each). No hyperparameter search was done and no validation set was used to avoid influencing the few-shot performance by knowing hyperparameter values based on extra validation data that isn’t available in few-shot (see Section 6.1). The maps from which the items of the annotation studies were sampled are part of the test set. 5 different train/test random splits are used and the average performance on the various test sets is reported in the main results. Each node with its actual parent constitute a training sample resulting in ∼211k training pairs. The models are trained using a batch contrastive loss where the actual parent of a node is considered a positive sample and all other parents in a random
6https://github.com/UKPLab/ sentence-transformers
7https://www.sbert.net/docs/pretrained_models. html
batch are considered negatives8. The models are trained for 1 epoch and then evaluated on the test set by calculating the metrics for each leaf child node and averaging over all those nodes in the map. We report the average for all maps. Evaluation is also done on the annotated samples to compare to human performance. We do not attempt to optimize hyperparameters to avoid influencing our few-shot experiments (Section 6.1). To have a more detailed estimation of the task difficulty and modeling performance, we report the average of a variety of metrics: top1, top5 accuracy and MRR (metrics description in Section A.3).

heading: 5.3 Results & Analysis
text: Table 2 shows the results for MNPet without the use of any templates. The performance improves with more intermediate-task training for all metrics. Best performance is achieved using all-mpnet (more generic and larger training data), which we use in all following experiments.
Table 3 shows that training improves on the zeroshot performance and performs comparable to or better than the human performance on the 200 annotated samples. The task, however, remains challenging in general mostly because of its inherent ambiguity (as was shown in the annotation study) and because of the noisy data that is available. Using the parent/child template further boosts the performance by ∼4, 3 points for top1, top5 respectively. Adding pro/con templates improves the performance only slightly. This might be because the signal about the type of relation is not as important in solving the task or that this signal is not utilized properly. Using more templates in all also does not improve the performance. It is hard to estimate how much more improvement is still possible since the human performance is estimated in a controlled setup and the highest performance here is already on bar with it or exceeds it. Similar observations can be made for the smaller model all-mini in Appendix Table A.4, except that the boost from using parent/child is smaller and the best performance
8MultipleNegativesRankingLoss in sentence-transformers
still lags behind that of humans especially for top1. Based on those findings, we focus on parent/child template in the following analysis of the results.
Agreement between model predictions and human annotations is moderate (κw=0.459) for zero-shot and is somewhat increased with training (0.491) meaning the model has a general agreement with humans about the ranking independent of what the actual parent is in the original Kialo data.
Embedding Space (Parent vs. Child) To analyse the effect of parent/child templates on the embedding space, we visualize in Figure 3 the embedding space for the nodes when using parent/child template. The visualization of the parent (orange) vs. child (blue) is more distinct for the
nodes that are closer to the root (darker color) in contrast to the deeper nodes which their parent & child visualizations are overlapping (light gray). This shows that representations for child vs. parent is more distinct for the more generic nodes at the top of the tree. This difference is not that significant as shown in Figure 4, where we visualize cosine similarity between the parent and child embedding of the same node averaged over all nodes in maps from the test set. As expected, the similarity is still high in general at ∼0.9 and the top nodes in the tree have slightly lower similarity (0.01) which might still be important in improving the performance.
Table 4 shows the average cosine similarity between all nodes for zero-shot and no template where we can see that the training decreases the similarity on average. When the similarity is calculated for the various combination in the case of training using parent/child templates, the training seems to have a different effect: the similarity stays higher when comparing arguments using only child template (child, child), but the parent embeddings are more distinct and less similar to each other (parent, parent) and to child embeddings (child, parent) of all other nodes (not specifically actual children).
Finally, we compute cosine similarity between the embeddings of each child c and its actual parent p using (child:"c", parent:"p") getting an average of .5496, whereas that of (child:"p", parent:"c") is much lower at .4951. This shows that using parent/child template indeed leads to a better representation of the unidirectional relation
c → p and p ↛ c .
Pro vs. Con Performance: Table 5 shows the detailed results of one train/test split according the type of relation between the argument its parent. As expected, zero-shot performance is better for pro compared to con since the pro relation is similar to entailment and other relations used to construct positive samples in tasks the model was trained on. This changes after training (no template) where con performance improves more than pro (+.11 vs. +.05), which can be partially due to more data available for con vs. pro (≈ 162k vs. 120k or 57% vs. 43%). Using templates gives a similar small boost for both. We see a similar pattern for top1 (Appendix Table A.5) except here con performance is similar to pro for zero-shot but after training, the performance of con is again better than pro. The pattern is similar after training to that of human performance on pro and con for both top1 and top5.

heading: 6 Few-shot Experiments
text: Our approach could be used for similar tasks, for which obtaining the scale of data that we used here is not feasible. Motivated by this, we investigate the data efficiency of our approach, analyze the results, and where to better invest resources.

heading: 6.1 Few-shot Experimental Setup
text: Random samples from the training set are used with varying numbers of maps (#maps) and numbers of nodes used from each map (#nodes) in (8, 16, 32, 64) where the final number of samples used for
training (#samples=#node×#maps) varies between 64 to 4096. 5 random samples for each combination of (#nodes×#maps) are used and the average performance is reported. Various templates are again investigated as their effect is expected to be different for low-resource.
True Few-shot
We refrain from using extra unlabeled data or extra samples as dev set to report true few-shot performance (Perez et al., 2021). We use default hyperparameters (Appendix section A.4) and batch size=8 (smallest number of nodes available for training per map).

heading: 6.2 Few-shot Results & Analysis
text: We see in Table 6 that using the same training paradigm proves to be effective for low-resource. Few-shot training improves on zero-shot in all cases with and without templates and no degradation in performance is observed due to overfitting even with a small number of samples (where fewshot is more prone to overfitting). Training with parent/child template improves the performance, especially for a lower number of #samples. For 64x64 the templates still improve the performance on the test but don’t improve on the annotated samples, however, the performance there is already close to human performance. The pro/contra template also helps boost the performance and the best performance is achieved when using a combination of various templates (all). When comparing for each #samples the performance when trained with no templates vs. all, we see that using tem-
plates helps narrow the gap between low-resource and high-resource (full dataset) with a boost that is larger for smaller #samples. Similar findings can be seen for the smaller model (Appendix Table A.11) except using more templates is not as effective there, especially with larger #samples.
The pro/con and all templates are more helpful here than when training on the full dataset (Table 3). This might be due to an augmentation effect since each sample is used in a parent/child template as well as other templates resulting in a training size that is 2x and 5x the original size (for pro/con, all respectively). Such augmentation would be more beneficial in more low-resource cases. To verify how this compares to the model seeing the samples more often, we train the model for double the amount (2 instead of 1 epoch) without any templates and those results (Appendix Table A.6) are comparable to no template with 1 epoch and worse than pro/con. The same is seen for all vs. 5 epochs which also holds when training with a parent/child template for 5 epochs in which case the performance of all is still better although to a lesser degree (Appendix Table A.7, A.8). Those initial results demonstrate the usefulness of templates with the potential to further improve the performance with template engineering or template search which were out of scope here.
This shows that the use of templates with contrastive learning is an effective approach in lowresource: the parent/child signal can be effectively exploited even at a low #samples and incorporating more templates in the training is a promising direction to bridge the low to high-resource gap.
Template Semantics: Motivated by research done on the effect of prompt semantics (Le Scao and Rush, 2021; Webson and Pavlick, 2022), we employ templates with no semantic meaning (foo/bar) using foo:"text" for child and bar:"text" for parent. Table 7 shows comparable results for foo/bar vs. parent/child (the same is seen when training with the full dataset Appendix Table A.9), and a similar effect is seen when using various templates in Appendix Table A.10. This is in line with findings about prompt-based fine-tuning (Webson and Pavlick, 2022) that is shown to yield good performance with irrelevant and misleading prompts. Number of Maps vs. Number of Nodes We investigate here where resources are more useful either when annotating and creating a dataset or when limiting training size and computing resources. For the same #samples (e.g. 128), different #maps and #nodes per map can be used (e.g. 16×8 or 8×16). We show in Table 8 a comparable combination of #node×#maps to investigate the effect each has on the performance and whether it is more beneficial to have few big maps or many small maps for training. We see better performance with more #nodes per map compared to more #maps with fewer nodes. This is probably because the more #nodes are available, the better negative samples are possible for better training.

heading: 7 Conclusion & Contributions
text: We propose and evaluate a solution to support in creating argument maps, contributing: 1) At the methodological level, we define the new task of node placement in argument maps, and conduct an annotation study to establish the human performance on the task gaining insights about factors that affect the choice of suitable parents for a node. 2) At the experimental level, we present modeling results with different training setups and base models, showing that templates can be used to improve the representations and are beneficial in high and low-resource scenarios. 3) At the level of application potential, the task could be adapted using top-n candidates by highlighting the nodes based on their predicted score similar to Figure 1. This allows for a more intuitive user interaction and loosens the effect of the ambiguity inherent in the task.

heading: 8 Limitations
text: • Our work focuses on data from one platform, Kialo, which contains cleaner and higher quality arguments from a diverse range of topics and domains. How our approach performs on data from other platforms or more specialized domains (e.g. deliberations about policy) has to be investigated in the future.
• The vast majority of data available is English which makes conducting and evaluating multilingual experiments not feasible even with language transfer (see Appendix Section A.1).
• The dataset used in the training and evaluation has only one correct position although there might be multiple suitable parents. Given the large scale of the data and the huge number of nodes per tree, annotating all suitable parents would’ve require a very-large-scale unfeasible annotation. This could be investigated in future-work with the support of our models.
• The design of our annotation study does not take into consideration the structure of the tree. This might have made the task more challenging for the annotators. Reconstructing or representing the tree structure without revealing the actual parent (since the majority of the candidates are close relatives) is challenging when limiting the candidate parents to 10. Further refinement of the annotation study is left for future work along with the inclusion of the structure in the modeling.
• Although small models are shown to perform relatively well and are recommended to use when computation resources are limited, the models that perform, in our experiments, on par with humans are large models that are costly to train. Employing parameter efficient fine-tuning methods might be of interest here. • We use only manually designed templates as a simple approach that required no extra training or engineering. How the results compare to using automated template/prompt engineering methods is also left for future work. Including prompt-based fine-tuning might be also of interest to investigate in combination with contrastive training although language modeling training would require more computational resources. • Our task definition excludes the prediction of pro/con relation as less important, but the pro/con template information might be useful for this. More evaluation and analysis is needed to verify that. • Extra analysis that was out-of-scope to include in this paper might be of interest: e.g. the effect of topic, the degree of a node, and semantic similarity to siblings on model or human performance.

heading: 9 Ethics Statement
text: We use available data from previous research. Automated tools to support in the exploration and creation of argument maps might be biased to favor arguments that are explored more often or that have more prominent styles as they are seen more often in the data as parents. This might lead to decreased suggestions as parents of those arguments that have underrepresented styles or using jargon/slang. This in turn leads to those arguments being less discussed and explored as they have less number of contribution. It’s important to take this into consideration and investigate any such effects before and after employing such models in real-world applications.

heading: Acknowledgements
text: We acknowledge funding by the Bundesministerium für Bildung und Forschung (BMBF) through the project E-DELIB. We thank Michael Roth, Sebastian Padó, and Sean Papay for providing feedback about the paper.

heading: A Appendix
text: A.1 Data Details The following section gives more details about the argument maps we use in this work.
The snapshot from Agarwal et al. (2022) contains a total of 1,560 argument maps. Using an automatic language-detection tool on a sample of the content of the map we assign a language to each map. The vast majority is English with very few other languages: 21 German, 6 Spanish, 5 French and 4 Italian. This makes conducting multilingual experiments even for mere evaluation challenging. As a result, we filter out all maps that are not-English and all with less than 19 nodes. Figure A.1 shows the distribution of maps with different amounts of nodes, the smallest having 19 nodes and the largest 6,252 nodes. Most
argument maps is associated with a number of topic tags, which can be selected by the user creating a new argument tree on a specific thesis. We merge similar tags into more coarse-grained topics such that every map can be associated with one specific topic. Figure A.2 depicts the number of maps per general topic, showing that the data covers a variety of different domains but also that more specific topics occur less frequent (e.g. animals).
Figure A.3 gives an idea about how many actual parent nodes are available. Most parent nodes have between 1 and 3 children with some exceptions having a very large amount of children (e.g. one node has 411 direct children).
Figure A.4 compares the distribution of nodes that act as a pro vs. nodes that act as a con. The distributions do not completely overlap as the majority of the data is slightly biased towards con.
A.2 Annotation Study The 4 authors annotated 20 samples while developing the guideline. We further recruited 3 student assistant as annotators, who have been paid 12,87 Euro per hour. The student assistants were Master Students of Computational Linguistics and Digital Humanities and have all participated in an Argument Mining course. Two annotators were female, one male. All have a very high level of English proficiency (one native speaker). Countries of origin: Canada, Pakistan, Germany. The annotators were aware that the data from the annotation study was used for the research purposes of our project.
A.3 Metrics
For one sample: top1: accuracy at rank 1. 1 if actual answer is at rank 1, 0 otherwise top5: accuracy at rank 5. 1 if actual answer is in the top-5 ranked, 0 otherwise MRR: the mathematical inverse of the rank of the actual answer in the ranked predictions The metrics are averaged for all samples so MRR for Q samples:
MRR = 1
Q
∑
i=1
1
ranki
A.4 Models & Training Details
Software: We use sentence-transformers11 for our experiments. Our code is made publicly available12. Hardware: NVIDIA RTX A6000 with 48G memory is used for training and inference. Average runtime: training for 1 epoch using the full training dataset takes around (in minutes):
for MPNet 0:22 with no template or parent/child template 0:44 with pro/con template (double data size) 1:44 with all templates
for MiniLM 0:08 with no template or parent/child template 0:14 with pro/con template (double data size) 0:34 with all templates Hyperparameters: Default hyperparameters are used to avoid influencing few-shot results which also kept computational cost minimal. The hyperparameters used are following: batch size = 64, learning rate = 2e-5 with 10% of training steps as warm-up steps.
A.5 Templates
Other templates with a more expressive form yielded similar results: This sentence: "child text" is child
11https://github.com/UKPLab/sentence-transformers 12https://github.com/imanjundi/argument-relations
"child text" is a child of "parent text"
A.6 Supplementary Results
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
8
3 A2. Did you discuss any potential risks of your work? 9
3 A3. Do the abstract and introduction summarize the paper’s main claims? 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? 3, 5.2
3 B1. Did you cite the creators of artifacts you used? 3, 5.2
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? A1
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. 5.2, 6.1
C 3 Did you run computational experiments? 5, 6
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? A.4
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? A.4
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 5, 6
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? A.4
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? 4
3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? A.2 (figure A.7, A.8)
3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? 4, A.2
3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? A.2
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? A.2

heading: Future Work
text: deliberations about policy) has to be investigated in the future. • The vast majority of data available is English which makes conducting and evaluating multilingual experiments not feasible even with language transfer (see Appendix Section A. 1). • The dataset used in the training and evaluation has only one correct position although there might be multiple suitable parents. Given the large scale of the data and the huge number of nodes per tree, annotating all suitable parents would’ve require a very-large-scale unfeasible annotation. This could be investigated in future-work with the support of our models. • The design of our annotation study does not take into consideration the structure of the tree. This might have made the task more challenging for the annotators. Reconstructing or representing the tree structure without revealing the actual parent (since the majority of the candidates are close relatives) is challenging when limiting the candidate parents to 10. Further refinement of the annotation study is left for future work along with the inclusion of the structure in the modeling. • Although small models are shown to perform relatively well and are recommended to use when computation resources are limited, the models that perform, in our experiments, on par with humans are large models that are costly to train. Employing parameter efficient fine-tuning methods might be of interest here. • We use only manually designed templates as a simple approach that required no extra training or engineering. How the results compare to using automated template/prompt engineering methods is also left for future work. Including prompt-based fine-tuning might be also of interest to investigate in combination with contrastive training although language modeling training would require more computational resources. • Our task definition excludes the prediction of pro/con relation as less important, but the pro/con template information might be useful for this. More evaluation and analysis is needed to verify that. • Extra analysis that was out-of-scope to include in this paper might be of interest: e. g. the effect of topic, the degree of a node, and semantic similarity to siblings on model or human performance