<figure><img src="image_1.png" /></figure>

HAL
open science

# Assessing Argument-based Natural Language Explanations in Medical Text

Benjamin Molinet, Elena Cabrio, Serena Villata

► To cite this version:

Benjamin Molinet, Elena Cabrio, Serena Villata. Assessing Argument-based Natural Language Explanations in Medical Text. The 40th ACM/SIGAPP Symposium On Applied Computing (SAC 2025), Mar 2025, Catania, Sicily, Italy. hal-05051047

HAL Id: hal-05051047

https://hal.science/hal-05051047v1

Submitted on 29 Apr 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Assessing Argument-based Natural Language
Explanations in Medical Text

Benjamin Molinet¹, Elena Cabrio¹ and Serena Villata¹

¹Université Côte d'Azur, Inria, CNRS, I3S, France
(benjamin.molinet, elena.cabrio, serena.villata)@univ-cotedazur.fr

Abstract

Natural language explanations are attracting increasing attention in the eXplainable AI context given their ability to elucidate human and artificial decision making processes. These kinds of explanations are particularly relevant in sensitive scenarios like medicine, law and defence. However, these explanations may be constructed in different ways, depending on the overall purpose they are employed for. In this paper, we propose a novel approach to characterize natural language explanations from an argumentation perspective. The proposed architecture is composed of a first module addressing an end-to-end argument mining pipeline to identify argument components and relations, then, a second module analyzing the resulting argumentation graphs to identify patterns which characterize explanations from an argumentation viewpoint. The experimental setting relies on the Casimedicos dataset of clinical cases. Obtained results show that the proposed approach is a promising solution towards applying AI in education with the aim to demonstrate to medical residents how to improve their explanations for a given diagnosis so as to increase patients' understanding of clinical deliberations.

Introduction

As eXplainable Artificial Intelligence (XAI) gained traction in recent years due to the “black box” nature of deep learning methods, and given the need to make explanations intelligible to humans, different approaches have been proposed in the literature to generate explanations in natural language. Explanation generation is particularly relevant in sensitive scenarios like medicine [1], law and defence. Natural language explanations, being the most user-interpretable [2] and transparent forms of expressing explanations, fit this need. One of the research directions in this domain consists in constructing argument-based natural language explanations, given the justification ability of argumentation in decision making [3, 4]. In this paper, we focus on the argumentative structure of natural language explanations written by humans to justify a medical decision. More precisely, we propose a number of criteria to characterise the argumentation structure of these explanations. The proposed framework, named ABEXA and shown in Figure 1, automatically tags the explanations with respect to these criteria, based on standard argumentation patterns in the literature [5, 6], and the empirical analysis of medical explanations in clinical case question-answering (QA). The proposed model first extracts the argumentation structure of the natural language explanation, and second, it automatically detects argumentation patterns in the extracted argumentation graphs. More specifically, we propose an approach based on pre-trained transformers adapted from the ACTA tool [7] to extract

argumentative structures, finetuned on annotations
of clinical cases from the Casimedicos dataset [8].

Explanation Assessment and
patterns

The evaluation of explanations remains an open chal-
lenge, with recent literature highlighting the impor-
tance of user-centeredness and multidimensional-
ity [9, 10]. Existing works employ metrics such as
Accuracy, Precision, Recall, F1, Exact Match, and
word embedding based measures like BLEU or BERT-
score, often in settings like multi-hop question an-
swering [11] and interpretable reading comprehen-
sion [12, 13]. Although embedding-based metrics
correlate better with human judgments than word-
overlap measures, their performance still falls short
of capturing the quality of human-rated explana-
tions [14]. In this contribution, we propose a new
modular system to assess explanations from the argu-
mentation structure viewpoint. These explanations
are automatically extracted with our argument min-
ing pipeline. Our goal is not to classify explanations
as being good and bad, but proposing different crite-
ria to aid users in semi-automatic assessment of argu-
mentative explanations, according to their needs and
use cases. We suggest a set of graph-based criteria
for argumentation structure assessment. We develop
and test the criteria on the Casimedicos dataset which
contains QA documents with both the Question, the
potential Answers and the Explanation written by an
external doctor. Then, established argumentative ex-
planation tags and their corresponding descriptions
<figure><img src="image_3.png" /><figcaption>**Figure 1:** Overview of our proposed pipeline. Argumentation patterns are detailed in Section 2.</figcaption></figure>

are used to assess the structure of explanations. The criteria we defined are the following:

**Inconsistent (Inc)** we found out that some of the argumentation graphs show inconsistent argumentation patterns. To detect this kind of patterns, we created a set of graph-based rules that triggers the tag, if one component is both (indirectly) attacked and (indirectly) supported by another argument component.

**No Explanation (NoE)** This tag aims to detect whether an explanation is empty or if it contains one single component, reflecting a poor argumentation.

**No Question (NoQ)** This tag focuses on cases where elements provided in the Question are not used (no relations) in the formulation of the Explanation.

**No Premise (NoP)** The Explanans plays a central role in the explanation. Therefore, when an explanation does not use Premises mainly present in the Question, we tag that document.

**No Answer Positive (NAP)** This tag is assigned when a claim or claims from the correct answer is not linked to any other components in the document.

**No Answer Negative (NAN)** Similar to the NAP pattern, this criterium aims to show the cases where at least one incorrect answer is not linked to any other components in the explanation.

**Not/Too Long (NoL-ToL)** These two tags compute the length of the Explanation in characters. We address an empirical analysis to define the default criteria in terms of length of the Explanation.

**Implicit (Imp)** Finally, we detect an overuse of components introduced in the explanation. This criterion relies on the fact that the argumentation scheme is based on the explainer's knowledge rather than the Premises or the Question of the QA document.

## Conclusion and Discussion

In this paper, we presented a full pipeline to assess and tag Natural Language Explanations within the

context of medical Question Answering. This approach tackles the assessment from the argumentation prism, providing an Argument Mining module to retrieve the argumentative structure from the raw text. Our system returns a list of tags associated to the QA document, highlighting certain argumentation-based characteristics of the explanations. With our module we achieved state-of-the-art results for argument mining on randomized clinical trials accounting for 0.65 of macro f1 and on clinical cases with 0.51 in our end-to-end configuration. Our analysis on the extracted graphs shows a comparable proportion of detected patterns, highlighting that our approach can successfully detect in an automatic wat argumentation patterns characterizing argument-based explanations in medicine.

## Acknowledgements

This work has been supported by the French government, through the 3IA Côte d'Azur Investments in the Future project managed by the National Research Agency (ANR) with the reference number ANR-19-P3IA-0002. This work has also been supported by the CHIST-ERA grant of the Call XAI 2019 of the ANR with the grant number Project-ANR-21-CHR4-0002.

## References

[1] Julia Amann et al. "To explain or not to explain?—Artificial intelligence explainability in clinical decision support systems". In: *PLOS Digital Health* 1.2 (2022), e0000016.

[2] Erico Tjoa and Cuntai Guan. "A survey on explainable artificial intelligence (xai): Toward medical xai". In: *IEEE transactions on neural networks and learning systems* 32.11 (2020), pp. 4793–4813.

[3] Kristijonas Cyras et al. "Argumentative XAI: A Survey". In: *ArXiv abs/2105.11266* (2021).
[4] Alexandros Vassiliades, Nick Bassiliades, and Theodore Patkos. "Argumentation and explainable artificial intelligence: a survey". In: *The Knowledge Engineering Review* 36 (2021), e5. DOI: 10.1017/S0269888921000011.

[5] Stephen E Toulmin. *The uses of argument*. Cambridge university press, 2003.

[6] DN Walton. *Argumentation schemes*. Cambridge University Press, 2008.

[7] Cristian Cardellino et al. "ANTIDOTE: ArgumeNtaTIon-Driven explainable artificial intelligence fOr digiTal mEdicine". In: *ECAI-24-Demos Proceedings*. Santiago de Compostela, Spain, Oct. 2024. URL: https://hal.science/hal-04673974.

[8] Rodrigo Agerri et al. "Hitz@ antidote: Argumentation-driven explainable artificial intelligence for digital medicine". In: *arXiv preprint arXiv:2306.06029* (2023).

[9] Tim Miller. "Explanation in artificial intelligence: Insights from the social sciences". In: *Artificial intelligence* 267 (2019), pp. 1-38.

[10] Hendrik Schuff et al. "Challenges in explanation quality evaluation". In: *arXiv preprint arXiv:2210.07126* (2022).

[11] Sewon Min et al. "Multi-hop reading comprehension through question decomposition and rescoring". In: *arXiv preprint arXiv:1906.02916* (2019).

[12] Kosuke Nishida et al. "Towards interpretable and reliable reading comprehension: A pipeline model with unanswerability prediction". In: *IJCNN Proceedings 2021*. IEEE. 2021, pp. 1-8.

[13] Ming Tu et al. "Select, answer and explain: Interpretable multi-hop reading comprehension over multiple documents". In: *Proceedings of the AAAI conference on artificial intelligence*. Vol. 34. 05. 2020, pp. 9073-9080.

[14] Miruna Clinciu, Arash Eshghi, and Helen Hastie. "A study of automatic metrics for the evaluation of natural language explanations". In: *arXiv preprint arXiv:2103.08545* (2021).