ArgMining 2023: Stance-Aware Re-Ranking for Non-factual Comparative Queries
Channel: webis
Date: 2023-12-13

Title: Stance-Aware Re-Ranking for Non-factual Comparative Queries
Authors: Jan Heinrich Reimer, Alexander Bondarenko, Maik Fröbe, Matthias Hagen
Paper: https://aclanthology.org/2023.argmining-1.5
GitHub: https://github.com/webis-de/argmining23-stance-aware-re-ranking

Abstract:
We propose a re-ranking approach to improve the retrieval effectiveness for non-factual comparative queries like ‘Which city is better, London or Paris?’ based on whether the results express a stance towards the comparison objects (London vs. Paris) or not. Applied to the 26 runs submitted to the Touché 2022 task on comparative argument retrieval, our stance-aware re-ranking significantly improves the retrieval effectiveness for all runs when perfect oracle-style stance labels are available. With our most effective practical stance detector based on GPT-3.5 (F₁ of 0.49 on four stance classes), our re-ranking still improves the effectiveness for all runs but only six improvements are significant. Artificially “deteriorating” the oracle-style labels, we further find that an F₁ of 0.90 for stance detection is necessary to significantly improve the retrieval effectiveness for the best run via stance-aware re-ranking.