Sövereign at PerpectiveArg2024: Using LLMs with Argument Mining (2024)

Abstract: This paper presents the Sövereign submission for the shared task on perspective argument retrieval for the Argument Mining Workshop 2024 (Falk et al., 2024). To address the challenge, we apply open-access Large Language Models (Mistral-8x7b) in a zero-shot fashion for re-ranking and explicit similarity scoring. Additionally, we combine different features in an ensemble setup using logistic regression. Our system ranks second in the competition for all test set rounds on average for the logistic regression approach using LLM similarity scores as a feature. We also make the code publicly available1.

heading: 1 Introduction
text: Although the World Wide Web is full of content, search engines nowadays still lack support for extracting information regarding argument retrieval (Bondarenko et al., 2022). Argument retrieval addresses the issue of retrieving relevant arguments from a corpus based on a specific query (Falk et al., 2024). Further issues arise for particular perspectives, i.e., an argument might only be relevant in a special situation with certain restrictions. The shared task on “Perspective Argument Retrieval” (Falk et al., 2024), investigates these challenges by including sociocultural properties/factors (e.g. political interests, occupation, age, and gender) in a multilingual setup (see Figure 1 for illustration). The data includes documents in German, French, and Italian.
Motivated by the abilities of large language models (LLMs; cf. Zhao et al., 2023), we investigate methods to leverage them for this task. We consider two approaches: implicitly re-ranking the argument candidates, and explicitly computing rele-
*Equal contribution. 1https://github.com/uhh-lt/
sovereign-perspectiveArg24 2https://translate.google.com
vance scores for candidate arguments. First, we use the cosine similarity of Sentence BERT (Reimers and Gurevych (2019) between the encoded arguments and the query to retrieve the nearest neighbors (arguments) as candidates similar to the baseline approach by Falk et al. (2024). Note that the given arguments also contain topic labels and socio-cultural factors (e.g. in Scenario 2, Section 3), which we also benefit from. We then supply the query and the retrieved candidates to an LLM and ask it to re-rank the arguments. In our second method, the LLM is asked to produce a score for a given query-argument pair. We further train a logistic regression classifier using several initial similarity scores as features and use the computed feature weights in an ensemble fashion to compute a final relevance score.
150
Our LLM scoring based method shows improved performance for Scenario 1 and 2, while the LLM re-ranking performs competitively in Scenario 3, on the development set. Therefore, we submit the results obtained with the logistic regression using the LLM scoring as the final solution to the competition. The name of our team in the leaderboard of the organizers is “Sövereign”.
The contributions of this paper are as follows:
• We investigate the ability of LLMs in argument mining with socio-cultural factors, experimenting with two approaches in a zeroshot setup: ranking by LLM directly and predicting relevance scores using LLM.
• We present a runner-up model, ranked as the second-best system, in the shared task in 2024.

heading: 2 Related Work
text: In this section, we briefly describe the existing studies that we take into consideration while developing our proposed approach.
Argument Retrieval Apart from the current Perspective Argument Mining shared task (Falk et al., 2024), there exists a series of scientific events and shared tasks on computational argumentation and causality which named Touché (Bondarenko et al., 2022, 2023). Traditionally, the shared task is related to the specific topics, e.g., Retrieval for Comparatives / Controversy (Bondarenko et al., 2022, 2023), Image Retrieval (Bondarenko et al., 2022, 2023), etc. For a detailed overview of the Argument Mining field, we refer the reader to the papers by Lawrence and Reed (2019) and Bondarenko et al. (2023).
LLMs for Ranking According to Qin et al. (2023), LLMs in zero-shot ranking tasks can be categorized into pointwise, listwise, pairwise, and setwise. Our approach applies the listwise method by Sun et al. (2023). The authors propose RankGPT, a generative LLM (here ChatGPT and GPT-4) for passage relevance ranking in information retrieval (IR) settings. Despite the fear of data contamination, they eventually concluded, that properly instructed LLMs can deliver competitive performance compared to supervised IR methods and can rank unknown knowledge.

heading: 3 Task Description
text: In this shared task, the goal is to retrieve multilingual arguments gathered from the voting recommendation platform3. For a description of the dataset, we refer to Falk et al. (2024). The key challenge here is to consider socio-cultural factors during retrieval. For the shared task 2024 we submit systems for all the three competition scenarios:
• Scenario 1: Default retrieval ranks argument candidates from a given corpus for a specific query ignoring any social-cultural attributes.
• Scenario 2: Explicit perspectivism adds socio-cultural information to the query and the arguments, which limits relevant arguments that match the corresponding socio-cultural factors.
• Scenario 3: Implicit perspectivism adds socio-cultural information only to the query, while it is not provided for the arguments.
For each evaluation round, the data consists of a set of queries and a set of candidates/arguments. The set of queries includes the query text, and for Scenarios 2 and 3 it also contains an explicitly given socio-cultural/demographic attribute. The set of candidates contains the argument text, a “stance” parameter (“favor” or “against”), and a “topic” parameter. The retrieval performance is measured using: a) relevance: NDCG@k and Pr@k (precision @ k), and b) diversity: αNDCG@k and klDiv@k (Kullback-Leibler Divergence @ k), where k is the rank of retrieved arguments.

heading: 4 Methodology
text: In this section, we present two approaches for the argument mining task. The first method applies the LLM directly to rank the arguments, the second integrates LLM scores as a feature for a logistic regression model. In the next subsection, we introduce scores utilized in both approaches.

heading: 4.1 Feature Scores
text: In both approaches, the LLM re-ranking and the logistic regression re-ranking, we employ three different scores. We describe each score in details below. In the LLM re-ranking approach presented in Section 4.2 below, scores are summed up and
3https://www.smartvote.ch/
the arguments are re-ranked accordingly before being sent to the LLM for re-ranking. We use each score as a feature to train a logistic regression classifier which then produces the final relevance score (Section 4.3). We demonstrate an example input for logistic regression in Figure 2 for better understanding of each score.
SBERT Cosine Similarity Score We encode queries and arguments using SBERT and utilize the cosine similarity of their embeddings to rank arguments given a query. This strategy replicates the baseline approach by the organizers.
Demography Matching Score For Scenario 2, the explicitly given socio-cultural attributes allow hard filtering of the arguments that do not match the socio-cultural attributes of the query. We assign a score of 1 to all arguments that match the given socio-cultural attribute parameter and a score of 0 to all other arguments.
Topic Frequency Score We predict the relevance of each topic to a query as the frequency of that topic among the 50 highest-ranked arguments based on SBERT cosine similarity. For each query-argument pair we use only the relevance of the given topic to the query.

heading: 4.2 LLM Ranking
text: As the first approach, we prompt the LLM with the current query and a list of the 50 highest-ranked arguments based on our SBERT scores. For Scenarios 1 and 3, these scores are the sum of the similarities and the topic scores, and for Scenario 2 they include the scores based on socio-cultural attributes, as well. We then ask the model to return an ordered list of the arguments based on their relevance to the query. The template prompt that we use is presented in Appendix A in Example 1.
Despite producing the machine-readable lists, the LLM output barely includes all 50 argument
IDs submitted as input. We interpret all missing IDs as irrelevant to the query and rank them lower at the end of the list.
For Scenario 3, the model re-ranks the arguments according to the socio-cultural attribute from the query and the implicit socio-cultural backgrounds for each argument in Appendix A in Example 2.

heading: 4.3 LLM as Score Predictor
text: For the second approach, we provide the LLM with a list of the 50 highest-ranked arguments (based on the summed feature scores of SBERT, cf. Section 4.1) and prompting it to assign a relevance score between 0 and 1 for each candidate. The prompt for Scenarios 1 and 2 is presented in Appendix A in Example 3.
The expected result is supposed to render a Python dictionary, where keys are sentence IDs, and values are the assigned relevance scores. This approach is also limited by omitting argument IDs in the LLM output. In such cases, we score the missed argument IDs as 0.
For Scenario 3, we all ask the model to predict the relevance score between the given sociocultural attribute from the query and the implicit socio-cultural backgrounds for each argument with the prompt present in Appendix A in Example 4.

heading: 4.4 Ensemble Learning
text: The previously computed scores are aggregated as features for a logistic regression classifier. More specifically, the feature set is comprised of the LLM relevance score, SBERT cosine similarity between query and argument, the topic frequency score, and the demography matching score based on sociocultural attributes, as shown in Figure 2. We train a logistic regression classifier for each scenario separately; the goal is to predict whether an argument is relevant to a query or not (label 0 or 1). To train the model, we use the top-100 (Scenario 2) or top500 (Scenarios 1 and 3) highest-ranked candidate
arguments for each query from the training set. The resulting weights from the logistic regression are presented in Table 2 in Appendix A. We interpret those scores as importance weights to rebalance the individual features of the candidate arguments. We additionally normalize them to sum up to 1. The weighted sum of the features is then used for re-ranking previously retrieved arguments.

heading: 5 Experimental Setup
text: We use the Mixtral-8x7B-Instruct-v0.14 LLM model by mistralai5 with the default parameters using HuggingChat6. This model comes with a lenient license and offers a good balance between performance and model size7. By using the HuggingChat framework, we explicitly make the model exchangeable, and we expect increased performance by using larger models. We refrain from the model fine-tuning and apply it as a zero-shot.
Regarding SBERT, we use the pre-trained model paraphrase-multilingual-mpnet-basev28, likewise the baseline from the organizers (Falk et al., 2024). We trained the logistic regression classifier using the scikit-learn9 framework on the training dataset with the default parameters.
To choose the solution for the final evaluation round, we test our approaches on the development set and submit the test set ranking using the bestperforming algorithm.
Scenario 1: Default Argument Retrieval Here, SBERT is already a very strong baseline. Logistic regression achieves better scores for NDCG, Pr,
4https://huggingface.co/mistralai/ Mixtral-8x7B-Instruct-v0.1
5https://mistral.ai/ 6We use HuggingChat version v0.8.4: https:// huggingface.co/chat/ 7Measured by personal experience. 8https://huggingface.co/sentence-transformers/ paraphrase-multilingual-mpnet-base-v2 9https://scikit-learn.org
and αNDCG and k > 4 (cf. Table 4, in Appendix). Thus, for Scenario 1 we submit the results achieved with logistic regression.
Scenario 2: Explicit Perspectivism Results are shown in Appendix Table 3. LLM re-ranking performs well as compared to the SBERT baseline, however, the logistic regression ensemble achieves the best scores. For this scenario, we also submit the results achieved with logistic regression.
Scenario 3: Implicit Perspectivism In this scenario, both approaches perform almost on par; the LLM re-ranking methods perform better than other approaches, as shown in Table 5 in the Appendix. However, we still decided to submit the logistic regression approach, as we consider learned weights to be more fair for the unseen data.

heading: 6 Results on the Test Sets
text: In this section, we present the test results of our approach from logistic regression. These results are evaluated and shared by task organizers. Table 1 presents the average results for all scenarios and test rounds. Additionally, we show the average results of our approaches across different test rounds and scenarios in Tables 6, 7 and 8 in Appendix A. We achieve competitive results for all scenarios on test set 1 and test set 2, however, our predictions for test set 3 fall short of first place quite significantly. In Scenario 1 test 3 “Sövereign” underperforms even the SBERT baseline. We believe this happens because of the topic scores, included in the final logistic regression. If the SBERT baseline predicts relevant arguments that match the expected topic, this will improve the results by increasing the final scores for those arguments, that match the expected topic. Otherwise, this will impair the results by increasing the final scores for the arguments that do not match the topic. For test set 1 (Precision@20 = 0.978)
and test set 2 (Precision@20 = 0.824) the topic scores are predominantly valuable, hence our results are significantly higher than the baseline. On test set 3 (Precision@20 = 0.565) the topic scores are deceptive for a significant amount of queries: the usage of this feature puts the irrelevant arguments higher. The reason for the difference across different test sets needs to be further investigated.

heading: 7 Conclusion
text: We present Sövereign, an LLM re-ranking approach for perspective argument retrieval. We show an investigation of two LLM utilizations, a) implicit re-ranking, and b) explicit relevance scoring. The explicit relevance scoring methods achieve better scores for explicit perspectivism when used in an ensemble with other similarity features, i.e., SBERT, topic, and socio-cultural (if applicable). In Scenario 3, implicit perspectivism, LLM reranking performs better than the LLM scoring. We believe this might be due to the formulations of the prompts in Scenario 3: ranking prompt emphasizes socio-cultural property, directly. In future work, we would like to explore more utilization methods of LLMs in this task, e.g., trying different prompts. The data additionally contains “stance” attributes, which we omitted to use for our submission, but might be an important feature. We also plan to try other LLM models and improve the results for test set 3 by classifying the topic from the query and matching it with the topics from the arguments.
Limitations
Nowadays, dozens of large pre-trained generative models exist and we report results only on mistralai/Mixtral-8x7B-Instruct-v0.1. It might be that some other foundation models could further push the results, however, our main goal was to investigate the ability of LLMs to re-rank arguments given socio-cultural factors.
As we use HuggingChat API10, it could produce every time different responses, which might slightly affect the results if reproducing the approach from scratch. However, we have saved the model output used for the final score submission, therefore, they can be used to reproduce the results.
10https://huggingface.co/chat/

heading: A Appendix
text: Here is the template prompt for the Scenario 1 or 2 performed with the LLM re-ranking:
(1) 
Answer with a python list containing all ranked argument ids
 
The following are passages related to question <query text> 

[0] <1st argument text> ... [49] <50th argument text>

Rank these passages based on their relevance to the question.

Here is the template prompt for the Scenario 3 performed with the LLM re-ranking:
(2) 
Answer with a python list containing all ranked argument ids
 
The task is to rank arguments, if they fit the sociocultural property: <query demographic property>.

[0] <1st argument text> ... [49] <50th argument text>

Rank these passages based on their relevance to the sociocultural property.

Here is the template prompt for Scenario 1 or 2 performed with similarity scores as a feature for Logistic Regression:
(3) 
Answer with a python dictionary containing a score between 0 and 1 for each argument id
 
Given the question <query text> and a list of arguments with IDs. The task is to rank the arguments according to the question. The higher the score the more relevant it is to the question

[0] <1st argument text> ... [49] <50th argument text>

Return a python dict with every single argument id and the scores only! No text!!! e.g. 1: 0.9, 2: 0.3

Here is the template prompt for Scenario 3 performed with similarity scores as a feature for Logistic Regression:
(4) 
Answer with a python dictionary containing a score between 0 and 1 for each argument id
 
The task is to rank arguments, if they fit the sociocultural property: <query demographic property>

[0] <1st argument text> ... [49] <50th argument text>

Return a python dict with all argument IDs between 0 and 49 and a score between 0 if the argument does not fit the demographic and 1 if it fits very well.


heading: Future Work
text: In future work, we would like to explore more utilization methods of LLMs in this task, e. g. , trying different prompts. The data additionally contains “stance” attributes, which we omitted to use for our submission, but might be an important feature. We also plan to try other LLM models and improve the results for test set 3 by classifying the topic from the query and matching it with the topics from the arguments. Limitations
Nowadays, dozens of large pre-trained generative models exist and we report results only on mistralai/Mixtral-8x7B-Instruct-v0. 1. It might be that some other foundation models could further push the results, however, our main goal was to investigate the ability of LLMs to re-rank arguments given socio-cultural factors. As we use HuggingChat API10, it could produce every time different responses, which might slightly affect the results if reproducing the approach from scratch. However, we have saved the model output used for the final score submission, therefore, they can be used to reproduce the results. 10https://huggingface. co/chat/