Argument Mining in BioMedicine: Zero-Shot, In-Context Learning and Fine-tuning with LLMs
Authors: Jérémie Cabessa, Hugo Hernault, Umer Mushtaq (University of Versailles Saint-Quentin, University of Paris-Saclay, Czech Academy of Sciences, Playtika Ltd., University of La Rochelle, 2024)
Topic: Language Models

Abstract
Argument Mining (AM) aims to extract the complex argumentative structure of a text and Argument Type Classification (ATC) is an essential sub-task of AM. Large Language Models (LLMs) have shown impressive capabilities in most NLP tasks and beyond. However, fine-tuning LLMs can be challenging. In-Context Learning (ICL) has been suggested as a bridging paradigm between training-free and fine-tuning settings for LLMs. In ICL, an LLM is conditioned to solve tasks using a few solved demonstration examples included in its prompt. We focuse on AM in the biomedical AbstRCT dataset. We address ATC using quantized and unquantized LLaMA-3 models through zero-shot learning, in-context learning, and fine-tuning approaches. We introduce a novel ICL strategy that combines kNN-based example selection with majority vote ensembling, along with a well-designed fine-tuning strategy for ATC. In zero-shot setting, we show that LLaMA-3 fails to achieve acceptable classification results, suggesting the need for additional training modalities. However, in our ICL training-free setting, LLaMA-3 can leverage relevant information from only a few demonstration examples to achieve very competitive results. Finally, in our fine-tuning setting, LLaMA-3 achieves state-of-the-art performance on ATC task in AbstRCT dataset.

Introduction
Argument Mining (AM) focuses on extracting the underlying argumentative and discursive structure from raw text AUTHOR. Argument Type Classification (ATC), which involves classifying argumentative units in text according to their argumentative roles, is the crucial sub-task in AM. Research has shown that the argumentative role of a unit cannot be inferred solely for its text: additional structural and contextual information is needed AUTHOR. This additional information can be incorporated via feature engineering AUTHOR, memory-enabled neural architectures AUTHOR or LLM-based hybrid methods AUTHOR. Large Language Models (LLMs) have become ubiquitous in deep learning and have shown impressive capabilities in most NLP tasks AUTHOR. %Pre-trained in unsupervised fashion on increasingly massive data corpus to learn general language modalities, LLMs have been used for various downstream Natural Language Processing (NLP) tasks such as text classification, text summarization and sentiment analysis. In the main, LLMs are used in two distinct settings: (i) training-free, where the pre-trained LLM is used for inference without any parameter adjustment, and (ii) fine-tuning, where the parameters of the LLM are updated through supervised training to enable transfer learning on a downstream task. Zero-shot learning refers to the training-free approach where a pre-trained LLM is prompted to solve tasks on completely unseen data samples. Recently, In-Context Learning (ICL) has been proposed as a bridging pa\-ra\-digm between the training-free and fine-tuning settings. ICL is a prompt engineering technique whereby an LLM is conditioned to solve tasks by means of a few solved demonstration examples included as part of its input prompt AUTHOR. Generally, the input prompt includes task instructions, the current input sample to be solved as well as several solved input-output pair examples. In this way, ICL maintains the training-free posture (parameters frozen) of the LLM while at the same time providing it with some supervision through demonstration examples. It also enables direct incorporation of selected features inside the prompt template, thereby obviating the need for architecture customization. Creative ICL strategies combining kNN-based examples selection, generated chain-of-thought (CoT) prompting, and majority vote ensembling have been proposed and shown to outperform fine-tuning approaches AUTHOR. In the main, kNN-based examples selection optimizes the process of learning from few examples and ensembling increases the robustness of the predictions AUTHOR. This work focuses on AM in the biomedical AbstRCT dataset AUTHOR. More specifically, we address the ATC task using quantized and unquantized LLaMA-3 models, among the most capable openly available LLMs (\href{https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard}{cf.~leaderboard}), through zero-shot learning, in-context learning, and fine-tuning approaches. Our contributions are as follows: \item In zero-shot learning setting, we show that LLaMA-3 fails to achieve acceptable classification results, suggesting the need for implementing additional training modalities. \item We introduce a novel ICL strategy that combines kNN-based example selection with majority vote ensembling. In this training-free setting, LLaMA-3 can leverage relevant information from only a few demonstration examples to achieve very competitive results. \item We further experiment with fine-tuning strategy for LLaMA-3. In this setting, we achieve state-of-the-art performance on the ATC task for AbstRCT dataset. \noindent Our code is freely available on \href{https://github.com/mohammadoumar/BioMedicalAMwithLLaMA3}{GitHub}.

Conclusion
In this work, we address argument type classification (ATC) in the biomedical AbstRTC dataset with openly available LLaMA-3 from the three-fold perspective of zero-shot learning (ZSL), in-context learning (ICL) and fine-tuning (FT). We show that ZSL fails to achieve acceptable performance, ICL significantly improves the results, and FT reaches state-of-the-art performance. These results support the fact that ATC task cannot be solved in a zero-shot setting by relying solely on general-purpose language modalities acquired during pre-training. Additional learning is essential, either in the form of solved demonstration examples (ICL) or via parameters' updates (FT). We conjecture that the sequential flow of arguments within a text is a corpus-specific feature that cannot be inferred through zero-shot methods. Previous works demonstrated that the text of argument components alone do not suffice to infer their argumentative roles AUTHOR. Additional contextual, structural and syntactic features are necessary. In our ICL and FT settings, comprehensive contextual and structural information is incorporated through task-specific information and complete abstract text provided in the prompt. This information enables the model to discern the sequence of arguments, their associated markers, and other characteristics closely associated with their argumentative roles. For future work, the design and implementation of a full AM pipeline using LLMs represents a major milestone. In this scenario, the LLM would take raw texts as input and produce a detailed map of the argumentative structure as output. We believe that LLMs will substantially transform the landscape of AM and its practical applications. This work benefited from access to the computing resources of the L3i laboratory, operated and hosted by the University of La Rochelle. It is financed by the French government and the Region Nouvelle-Acquitaine. This research also benefited from institutional support RVO: 67985807 and partially supported by the grant of the Czech Science Foundation No. GA22-02067S. Finally, we are grateful to Playtika Ltd. for their support for this research. \pagebreak \clearpage \appendix