# An Automatic Scoring Method for Responses to the Hayling Test

Insaf Hamdi, Joseph Lam Weil, Émilie Martz, Ines Abdeljaoued-Tej, Antonio Capobianco

<figure><img src="image_1.png" /></figure>

## To cite this version:

Insaf Hamdi, Joseph Lam Weil, Émilie Martz, Ines Abdeljaoued-Tej, Antonio Capobianco. An Automatic Scoring Method for Responses to the Hayling Test. 2025 IEEE 37th International Conference on Tools with Artificial Intelligence (ICTAI), Michail Alexiou, Nov 2025, Athens, Greece, Greece. 10.1109/ICTAI66417.2025.00181.1.1

hal-05355459

HAL Id: hal-05355459

https://hal.science/hal-05355459v1

Submitted on 8 Nov 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# An Automatic Scoring Method for Responses to the Hayling Test

Insaf Hamdi

ICube Laboratory and University of Carthage (ESSAI)

Strasbourg, France / Tunis, Tunisia

insaf.hamdi@essai.ucar.tn

Emilie Martz, PhD, Psychologist

Cognitive Psychology Laboratory, University of Strasbourg

Strasbourg, France

emilie.martz@unistra.fr

Joseph Lam Weil, PhD

ICube Laboratory, CNRS, and University of Strasbourg

Strasbourg, France

jlam@unistra.fr

Ines Abdeljaoued-Tej, PhD

University of Carthage (ESSAI)

University Tunis-El-Manar, LR16IPT09

Tunis, Tunisia

ines.tej@essai.ucar.tn

Antonio Capobianco, PhD

ICube Laboratory, HMI and 3D VR Group, CNRS and University of Strasbourg

Strasbourg, France

a.capobianco@unistra.fr

**Abstract**—Timely and objective assessment of executive functions is critical for diagnosing and monitoring cognitive disorders, yet conventional scoring of Hayling Test responses remains subjective and prone to inter-rater variability. To address this challenge, we introduce an end-to-end automated scoring pipeline using state-of-the-art natural language processing (NLP). We systematically benchmark both static embeddings FastText, a range of transformer-based encoders, and recent large language models (LLMs) including Phi-3-mini, Zephyr-7B, XGLM-7.5B, Qwen1.5-7B, DeepSeek-V2, Llama-2-7B, and others on a corpus of 2,414 manually annotated French responses. Each response is classified into three penalty levels: 0 (correct inhibition), 1 (partial inhibition error), and 3 (failure to inhibit). After fine-tuning, the cross-lingual encoder LaBSE achieves the best overall performance, with a Macro-F₁ of 0.80, Balanced Accuracy of 0.79, and overall Accuracy of 0.83, outperforming all baselines and LLMs. Among LLMs, Phi-3-mini achieves the highest scores (Macro-F₁ 0.57, ACC 0.63), yet remains below supervised approaches. Our automated approach standardizes test scoring, reduces subjective bias, and enables near-instantaneous, scalable neuropsychological assessments paving the way for more reliable and efficient clinical and research workflows. The code supporting this work is publicly available at: https://github.com/insafhamdi/AI-hayling.

**Index Terms**—Cognitive Assessment, Hayling Test, Natural Language Processing, Automation, Executive Functions, Clinical Neuropsychology, Large Language Models, Static Embedding, Contextual Encoders, Finetuning

## I. INTRODUCTION

Accurate assessment of executive functions is crucial for diagnosing and monitoring cognitive and neuropsychiatric disorders such as Alzheimer's disease, traumatic brain injury, Parkinson's disease, tobacco dependence, and depression [1]–[4]. The Hayling Test is widely used to probe response inhibition and cognitive flexibility [5]–[8]. It has been validated in various populations, including older adults [8], patients

with brain injury [9], and those with mood or movement disorders [2], [4]. Recent multimodal and longitudinal studies further highlight the relevance of inhibition measures in neurological and psychiatric contexts [1], [10].

Manual scoring of Hayling responses, however, remains time-consuming and subjective, leading to significant inter-rater variability [6], [8]. To address these challenges, recent research has explored automated scoring of neuropsychological and clinical free-text data, leveraging advances in natural language processing (NLP) and word embeddings [11]–[13]. Similar directions have also been pursued within the ICTAI community, where NLP methods were applied to challenging language and speaker-related tasks [14]. Embedding methods such as Word2Vec [15], GloVe [16], and FastText [17], as well as transformer-based models like BERT [18], SBERT [19], LaBSE [20], and CamemBERT [21], have proven effective for a variety of classification and semantic tasks in clinical and healthcare settings [11].

Recent advances in large language models (LLMs) such as Phi-3-mini [22], Zephyr-7B [23], Mistral-7B [24], Qwen1.5-7B [25], DeepSeek-V2 [26], XGLM-7.5B [27], Llama-2-7B [28], Falcon-7B [29], Gemma-7B [30], and BLOOMZ-7B1 [31] have enabled generalist text understanding and classification across languages, even with minimal supervision. Nevertheless, automated scoring for Hayling-style inhibition tasks in French, especially with modern contextual models, remains largely unexplored [32]. To ensure full reproducibility, we release the source code accompanying this work at https://github.com/insafhamdi/AI-hayling.

In this work, we present a reproducible baseline for automated scoring of French Hayling inhibition responses. We benchmark both static and contextual embedding pipelines on a dedicated corpus, with rule-based checks for forbidden
responses and proper nouns.

The rest of this paper is organized as follows: Section II describes the assessment protocol and data; Section III discusses workflow challenges; Section IV covers key concepts; Section V details our model pipelines; Section VI presents results; and Section VII offers a critical discussion and future directions.

## II. EXPERIMENTAL PROCEDURES

Participants completed the Hayling Test as part of a broader cognitive assessment protocol. Ethical approval was obtained from the Comité de Protection des Personnes Nord-Ouest IV (favorable opinion issued on December 18, 2024). The final dataset contained 2,414 manually annotated inhibition responses.

The Hayling Test is composed of 30 sentence stems, divided into two conditions: 15 in the automatic completion condition and 15 in the inhibition condition [6]. For each item, sentence stems were presented in French, and participants provided verbal responses according to the instructions of each condition.

In the *automatic condition*, participants completed the sentence with the most logical word (e.g., for the stem *The firefighters put out the __*, the expected response was *fire*). This condition assesses verbal initiation and semantic retrieval. In the *inhibition condition*, participants were instructed to provide a semantically unrelated word while suppressing the prepotent response. For example, given the stem *To protect himself from the rain, he opened his __*, a correct response would be an unrelated word like *book* rather than the expected *umbrella*. Responses were scored using three penalty levels:

0. Correct inhibition: completely unrelated word, e.g. *cucumber*;

1. Partial inhibition: semantically related word, e.g. *notebook*, *sunshade*, or *coat*;

3. Failure to inhibit: prepotent response, e.g. *umbrella*.

The distribution of penalty labels was as follows: 1,582 responses scored as 0, 827 responses as 1, and only 5 responses as 3. All responses were transcribed and annotated by expert raters. The test was administered in French, and each response was time-stamped for subsequent analysis.

## III. CLINICAL WORKFLOW CHALLENGES

Despite its clinical relevance, the current workflow for administering and scoring the Hayling Test still presents several limitations that impact efficiency and consistency in real-world settings.

### A. Scoring Time and Manual Burden

Although the administration of the Hayling Test itself takes only about 10 minutes, the scoring process remains time-consuming. For each of the 30 items (15 automatic and 15 inhibition trials), clinicians must transcribe responses, check for forbidden answers, and assign a penalty score based on semantic appropriateness. This manual process typically takes 10 to 18 minutes per participant and adds to the workload of practitioners.

### B. Inter-Rater Variability and Subjectivity

Manual scoring of Hayling responses is inherently subjective. Previous studies have reported notable variability between human raters. For example, Andrés and Van der Linden [33] found only 76.5% agreement between two expert annotators on 1,425 responses, highlighting the difficulty of maintaining consistent scoring criteria.

### C. Risk of Human Error

Under time constraints or fatigue, clinicians may introduce scoring errors—such as mistyping words, misclassifying responses, or overlooking deviations from expected inhibition behavior. These errors compromise diagnostic accuracy and delay result reporting.

### D. Need for Standardized and Reproducible Ratings

To address these challenges, we propose an automated scoring pipeline based on NLP techniques. Unlike manual methods, this approach ensures consistent application of the scoring rules, reduces processing time, and minimizes inter-rater variability. All inhibition trials in our corpus (2,414 responses) were annotated by a single expert clinician based on standardized Hayling scoring criteria [6]. While this avoids inter-rater conflict, it still requires considerable time and introduces potential for inconsistency. Automating this process allows for fast, repeatable, and objective ratings—facilitating broader adoption in clinical and research settings.

## IV. TECHNICAL BACKGROUND AND TERMINOLOGY

In this section, we introduce key concepts relevant to our methodology, including *word embeddings*, the distinction between *static* and *contextual* representations, and the notion of *fine-tuning*. Rather than simply listing existing models, we motivate the architectural choices made in our study and explain the evaluation metrics used. We also present the main equations involved in our analysis, including cosine similarity, the $F_β$ score, and Balanced Accuracy.

### A. Word Embeddings

To classify inhibition errors based on verbal responses, we first need to convert words into numerical representations so called *embeddings*. Embeddings map discrete tokens (e.g., words) to continuous vectors that capture semantic and syntactic similarities. In this work, we evaluate two families of embeddings: *static* and *contextual*.

**Static embeddings** such as Word2Vec [15], GloVe [16], and FastText [17] assign each word a fixed vector, independent of context. Despite their limitations particularly their inability to disambiguate meaning based on sentence context they remain widely used as baselines due to their computational efficiency and ease of use. Notably, FastText incorporates subword information (e.g., character n-grams), which makes it more robust to rare or out-of-vocabulary words a useful property when scoring free-form patient responses.

**Contextual embeddings** address the limitations of static models by generating word vectors that depend on the full
sentence. Models such as BERT [18] and its French variant CamemBERT [21] rely on transformer architectures to learn contextualized representations—for instance, the word bark is embedded differently in “the dog barked” versus “the bark of the tree.” Because Hayling scoring hinges on the fine-grained semantic relation between a stem and a candidate response, contextual models could capture nuances that static embeddings ignore. We therefore hypothesize that they will outperform static vectors, a claim we test empirically in Section VI.

## B. Fine-Tuning

Pretrained transformers like CamemBERT can be adapted to specific tasks via fine-tuning, a process that updates the model's parameters using annotated data. In this work, we consider two types of fine-tuning. First, we apply task-specific fine-tuning by appending a classification layer to a pretrained model and retraining it on our own labeled dataset for inhibition scoring. This enables the model to specialize in distinguishing error types in the context of the Hayling Test. Second, we evaluate models that have been fine-tuned by their original authors on other supervised tasks, such as natural language inference (e.g., BART-MNLI [34], XLM-R-MNLI). Although these models are not adapted to our data, they encode semantic relations between sentence pairs that can be repurposed for inhibition classification.

We compare both categories of models to simpler cosine-based scoring pipelines in order to assess the added value of supervision and contextual adaptation.

## C. Evaluation Metrics and Key Equations

To evaluate model performance, we use Cosine Similarity (1), the F<sub>β</sub> score (2–4), Accuracy (ACC), and Balanced Accuracy (BAcc).

1) Cosine Similarity: To measure the similarity between two embedding vectors **u**, **v** ∈ ℝ<sup>d</sup>, we use the cosine similarity:

$$ \cos(\mathbf{u}, \mathbf{v}) = \frac{\mathbf{u} \cdot \mathbf{v}}{\|\mathbf{u}\| \|\mathbf{v}\|} \qquad (1) $$

where **u** · **v** = Σ<sub>i=1</sub><sup>d</sup> u<sub>i</sub>v<sub>i</sub> and ||**u**|| = √Σ<sub>i=1</sub><sup>d</sup> u<sub>i</sub><sup>2</sup>.

2) F<sub>β</sub> Score: Precision and recall are defined as:

$$ \text{Precision} = \frac{\text{TP}}{\text{TP} + \text{FP}}, \qquad (2) $$

$$ \text{Recall} = \frac{\text{TP}}{\text{TP} + \text{FN}}. \qquad (3) $$

The general F<sub>β</sub> score combines them:

$$ F_{\beta} = (1 + \beta^2) \frac{\text{Precision} \times \text{Recall}}{(\beta^2 \times \text{Precision}) + \text{Recall}}. \qquad (4) $$

Here, β > 0 is a tunable parameter that balances precision and recall in the F<sub>β</sub> score: values β > 1 emphasize recall, while values β < 1 emphasize precision. The special case β = 1 yields the F<sub>1</sub> score, equally weighting precision and recall.

3) Macro-F<sub>1</sub>: To account for class imbalance, we report the macro-averaged F<sub>1</sub>:

$$ \text{Macro-F}_1 = \frac{1}{C} \sum_{c=1}^{C} F_1^{(c)} \qquad (5) $$

where C is the number of classes and F<sub>1</sub><sup>(c)</sup> the F<sub>1</sub> for class c.

4) Balanced Accuracy (BAcc): Balanced Accuracy averages sensitivity and specificity:

$$ \text{BAcc} = \frac{1}{2} \left( \frac{\text{TP}}{\text{TP} + \text{FN}} + \frac{\text{TN}}{\text{TN} + \text{FP}} \right). \qquad (6) $$

Both Macro-F<sub>1</sub> (5) and BAcc (6) are particularly appropriate for clinical NLP tasks like inhibition scoring, where class imbalance is frequent and minority-class errors are of primary interest.

## V. MODEL PIPELINES

Each family of models follows a distinct data-processing pipeline. All models process French-language responses from the Hayling Test, which is particularly important given the language-specific nuances in semantic inhibition tasks. We describe the general steps here and refer to the equations above when needed.

### A. Static Embedding Pipeline

Fig. 1 illustrates our static-embedding pipeline, which proceeds in seven sequential steps.

First, each response is normalized to standardize case and remove accents or punctuation (SE.1). Next, the system retrieves the corresponding pretrained French word vector using either Word2Vec (frWac) [15], our own GloVe-FR trained on French Wikipedia, or FastText (CC-300 fr) [17] (SE.2). The cosine similarity between the response vector and the vector of the forbidden word is then computed (SE.3).

In parallel, we apply a set of fixed rules to flag responses as inhibition errors independently of their vector similarity. These rules are based on surface-level lexical or semantic cues (SE.4). Any remaining responses are then classified as errors if their cosine similarity exceeds a decision threshold τ (SE.5).

To determine the optimal value of τ, we perform a grid search over the interval [0, 1] in 0.01 increments on a stratified validation set. The value that maximizes BAcc on the validation set is retained (SE.6). Finally, Accuracy, BAcc, Precision, Recall, and Macro-F<sub>1</sub> are computed on the held-out test set using this optimized threshold (SE.7).

This simple, token-level pipeline provides a fast, fully reproducible baseline before moving on to contextual encoders and fine-tuned models.

### B. Contextual Embedding Pipeline

Fig. 2 depicts our contextual embedding pipeline, which parallels the static one but operates on sentence-level embeddings derived from a concatenation of the “stem” (i.e., the word to inhibit) and the participant’s response.
<figure><img src="image_3.png" /><figcaption>Fig. 1. Overview of the static embedding pipeline used for Hayling response classification</figcaption></figure>

<figure><img src="image_4.png" /><figcaption>Fig. 2. Overview of the contextual embedding pipeline used for Hayling response classification</figcaption></figure>

In our dataset, each entry contains the stem and the partici-
pant's reply. Both are first normalized (converted to lowercase,
with accents and punctuation removed), then concatenated into
a single input sequence of the form:

[CLS] stem [SEP] response [SEP]

where `[CLS]` marks the beginning of the sequence and
`[SEP]` separates its segments.

The resulting input is passed through a pretrained trans-
former (e.g., SBERT, CamemBERT, or LaBSE), and we ex-
tract the hidden state corresponding to the `[CLS]` token. This
vector often referred to as the pooled embedding is a fixed-
size representation designed to capture the global semantics
of the entire input sequence. In our case, it encodes the joint

meaning of the stem-response pair and serves as the contextual
embedding used for scoring.

We apply the same rule-based override as in the static
pipeline: any response violating predefined conditions (e.g.,
proper noun, profanity, semantic intrusion, or repetition of a
prior answer) is directly labeled as an inhibition error. For all
remaining cases, we compute the cosine similarity between the
pooled embedding and the static embedding of the forbidden
word. Responses with similarity greater than or equal to a
threshold τ are labeled as errors.

The threshold $\tau$ is selected by sweeping over the interval $[0, 1]$ in steps of 0.01 on a stratified validation split, retaining the value that maximizes Balanced Accuracy (BAcc). Final evaluation metrics Accuracy, BAcc, and Macro-F₁ are reported on the test set using the selected $\tau$.

We benchmarked three principal types of contextual
models in our study. First, transformer models such as
SBERT-MiniLM, LaBSE, Flaubert-small [35], XLM-R,
and CamemBERT were used as bi-encoders: these models
generate a single vector representation for the concatenated
pair of forbidden word and participant response, which is then
used for classification. Second, we evaluated models fine-tuned
for natural language inference (NLI), such as BART-MNLI
and XLM-R-MNLI; these models process the pair of inputs
jointly and output a probability or "entailment" score reflecting
how likely the response is to be related to the forbidden word.
We convert this score to a class label using a fixed threshold
of 0.5. Third, we tested a zero-shot sequence-to-sequence
(seq2seq) model (FLAN-T5-base [36]) by prompting it in
French with explicit instructions about the inhibition task, so
that its generated output (yes/no or error type) can be directly
mapped to our scoring labels<fnref n="1" />.

C. Fine-Tuning Pipeline

Fig. 3 diagrams our fine-tuning pipeline. In the text be-
low, "FT.n" labels correspond to the numbered steps in the
flowchart.

First (FT.1), each input pair is normalized (lowercase,
remove accents/punctuation). Second (FT.2), we concatenate
the stem and response into a single input with special tokens
[CLS] and [SEP]. Third (FT.3), the resulting sequence is
tokenized and encoded by the transformer. Fourth (FT.4),
we fine-tune the model end-to-end by adding a dense clas-
sification layer atop the pooled [CLS] embedding. Fifth
(FT.5), stratified 5-fold cross-validation is applied to ensure
full training/validation separation. Sixth (FT.6), we sweep
the classification threshold $\tau \in [0, 1]$ in 0.01 increments to
optimize BAcc on each fold. Seventh (FT.7), we report the
mean $\pm$ standard deviation of Accuracy, BAcc and Macro-F₁
across folds.

<footnote n="1">¹*Natural Language Inference* (NLI) is a task where a model is trained to determine the logical relationship between a premise and a hypothesis (e.g., entailment, contradiction, or neutrality). Here we adapt NLI to measure semantic relatedness between the forbidden word and participant response.</footnote>
<figure><img src="image_5.png" /><figcaption>Fig. 3. Overview of the fine-tuning pipeline used for Hayling response classification</figcaption></figure>

To reduce the risk that our performance estimates are driven by chance or sampling bias, we apply stratified 5-fold cross-validation on the full 2,414-sample corpus. To further mitigate data leakage, we verified for each fold that the set of unique response-stem pairs in the validation set was fully disjoint from those in the training set. Specifically, we ensured:

$$ \forall k, \quad |\text{Train}_k \cap \text{Valid}_k| = 0 \qquad (7) $$

where the intersection in Eq. (7) refers to the set of unique response-stem pairs, not just indices.

## D. Large Language Model (LLM) Pipeline

Fig. 4 outlines our automated pipeline for benchmarking large language models (LLMs) on the Hayling Test dataset. Each step is explained below (LL.1–LL.7):

<figure><img src="image_6.png" /><figcaption>Fig. 4. LLM evaluation pipeline (LL.1–LL.7) for Hayling response classification</figcaption></figure>

First (LL.1), we load the expert-annotated Hayling dataset, focusing on inhibition trials. Second (LL.2), we filter for classes 0 and 1, remove missing values, and normalize text responses. Third (LL.3), for each input, we construct a natural language prompt that includes scoring rules and few-shot examples to guide the LLM’s output. Fourth (LL.4), the prompt is fed into a HuggingFace LLM (Mistral, Llama-2, Falcon, etc.) via the Transformers pipeline, generating a numerical score (0, 1, or 3). Fifth (LL.5), we parse the model’s output, map it to the appropriate class, and mask class 3 responses if not part of the main evaluation. Sixth (LL.6), we compute evaluation metrics Accuracy, Balanced Accuracy, and Macro-F₁ on the remaining data. Seventh (LL.7), all results are aggregated, saved to CSV, and visualized as bar plots for comparison across models.

This pipeline enables reproducible, scalable evaluation of different LLMs on neuropsychological scoring tasks.

# VI. EXPERIMENTAL RESULTS

To evaluate our scoring pipelines, we divided the dataset into stratified training, validation, and test sets. All models were evaluated on the test set after selecting the threshold $τ$ that maximized BAcc on the validation split.

## A. Static Embeddings

We evaluated three widely-used static embedding models: Word2Vec, GloVe, and FastText. Table I reports the best test set metrics for each static embedding at its optimal threshold $τ_{opt}$ (the value that maximises BAcc). The best result in each column is highlighted in bold. FastText clearly outperforms Word2Vec and GloVe across all metrics.

<table><caption>TABLE I
PERFORMANCE OF STATIC EMBEDDINGS</caption><thead><tr><th>Embedding</th><th>τ<sub>opt</sub></th><th>BAcc</th><th>ACC</th><th>Macro-F<sub>1</sub></th></tr></thead><tbody><tr><td>FastText</td><td>0.22</td><td><strong>0.69</strong></td><td><strong>0.73</strong></td><td><strong>0.61</strong></td></tr><tr><td>GloVe</td><td>0.18</td><td>0.57</td><td>0.53</td><td>0.53</td></tr><tr><td>Word2Vec</td><td>0.22</td><td>0.57</td><td>0.52</td><td>0.52</td></tr></tbody></table>

## B. Contextual Encoders

We evaluated eight contextual models, including sentence encoders and cross-encoders: SBERT-MiniLM, LaBSE, Flaubert-small, XLM-R-base [37], Camembert, BART-MNLI, XLM-R-MNLI [38], and FLAN-T5-base. Table II presents the main metrics for each contextual encoder. All values are reported at the threshold maximizing Balanced Accuracy (BAcc).

Among all contextual models, LaBSE and XLM-R-MNLI achieve the highest Balanced Accuracy (0.56 and 0.57, respectively), with XLM-R-MNLI also reaching the top Macro-F₁ score (0.56). Notably, FLAN-T5-base attains the highest Accuracy (0.66), but its Macro-F₁ remains modest (0.40), indicating a strong bias towards the majority class. These results reinforce the importance of using Macro-F₁ and BAcc—rather
<table><caption>TABLE II
PERFORMANCE OF CONTEXTUAL ENCODERS BEFORE FINE-TUNING</caption><thead><tr><th>Model</th><th>BAcc</th><th>ACC</th><th>Macro-F<sub>1</sub></th></tr></thead><tbody><tr><td>XLM-R-MNLI</td><td><strong>0.57</strong></td><td>0.58</td><td><strong>0.56</strong></td></tr><tr><td>LaBSE</td><td><strong>0.56</strong></td><td>0.56</td><td>0.54</td></tr><tr><td>XLM-R-base</td><td>0.53</td><td>0.55</td><td>0.53</td></tr><tr><td>SBERT-MiniLM</td><td>0.53</td><td>0.50</td><td>0.50</td></tr><tr><td>BART-MNLI</td><td>0.48</td><td>0.42</td><td>0.42</td></tr><tr><td>Flaubert-small</td><td>0.52</td><td>0.39</td><td>0.35</td></tr><tr><td>FLAN-T5-base</td><td>0.50</td><td><strong>0.66</strong></td><td>0.40</td></tr><tr><td>CamemBERT</td><td>0.51</td><td>0.37</td><td>0.31</td></tr></tbody></table>

than Accuracy alone—to robustly evaluate models in imbalanced clinical classification settings.

Overall, the comparative analysis highlights the effectiveness of cross-lingual and NLI-based encoders for nuanced inhibition error detection, while also demonstrating the risk of overestimating performance when relying solely on Accuracy.

### C. Fine-Tuned Contextual Encoders

Fine-tuning led to marked improvements for most contextual models. Table III summarizes post-fine-tuning metrics for all evaluated encoders, and Fig. 5 illustrates the gains: dashed bars show performance before fine-tuning, while solid bars show performance after fine-tuning. LaBSE achieves the highest scores across all metrics, with a Balanced Accuracy of 0.79, Accuracy of 0.83, and Macro-F₁ of 0.73. Both XLM-R-MNLI and BART-MNLI also deliver strong, balanced performance, confirming the effectiveness of NLI-based and cross-lingual encoders for this inhibition scoring task.

<table><caption>TABLE III
PERFORMANCE OF CONTEXTUAL ENCODERS AFTER FINE-TUNING</caption><thead><tr><th>Model</th><th>BAcc</th><th>ACC</th><th>Macro-F<sub>1</sub></th></tr></thead><tbody><tr><td>LaBSE</td><td><strong>0.79</strong></td><td><strong>0.83</strong></td><td><strong>0.80</strong></td></tr><tr><td>XLM-R-MNLI</td><td>0.77</td><td>0.82</td><td>0.79</td></tr><tr><td>SBERT-MiniLM</td><td>0.71</td><td>0.76</td><td>0.72</td></tr><tr><td>BART-MNLI</td><td>0.69</td><td>0.75</td><td>0.70</td></tr><tr><td>Flaubert-small</td><td>0.64</td><td>0.72</td><td>0.64</td></tr><tr><td>XLM-R-base</td><td>0.50</td><td>0.63</td><td>0.39</td></tr><tr><td>CamemBERT</td><td>0.50</td><td>0.63</td><td>0.39</td></tr></tbody></table>

In contrast, models such as XLM-R-base and CamemBERT exhibit persistently lower Macro-F₁ scores (0.39), indicating ongoing challenges with class imbalance despite fine-tuning. This reinforces the importance of reporting macro-averaged metrics to properly evaluate minority class detection and to avoid misleading conclusions from Accuracy alone.

### D. Statistical Significance: Wilcoxon Test

We compared models pairwise with Wilcoxon signed-rank tests on Balanced Accuracy across the 5 folds. In our reporting, $p < 0.05$ is taken as a statistically significant difference between the two models, and $p > 0.05$ as not significant. In Fig. 6, darker blue cells correspond to smaller $p$-values, meaning stronger evidence that the compared models differ.

<figure><img src="image_7.png" /><figcaption>Fig. 5. Impact of Fine-Tuning on Contextual Model Performance</figcaption></figure>

Warmer colors indicate larger $p$-values, suggesting little or no evidence of a difference.

In our results, no off-diagonal comparison is below 0.05. For example, LaBSE vs. CamemBERT has $p = 0.062$ and LaBSE vs. XLM-R-base has $p = 0.062$ (both non-significant), and Flaubert-small vs. BART-MNLI has $p = 0.812$ (clearly non-significant).<fnref n="2" />

<figure><img src="image_8.png" /><figcaption>Fig. 6. Pairwise Wilcoxon signed-rank test $p$-values for Balanced Accuracy across models</figcaption></figure>

### E. Large Language Models (LLMs)

We evaluated the following large language models in a few-shot inhibition response classification setting: Phi-3-mini, Zephyr-7B, XGLM-7.5B, Mistral-7B, Qwen1.5-7B, DeepSeek-V2, Llama-2-7B, Falcon-7B, Gemma-7B, and BLOOMZ-7B1. Table IV reports the Balanced Accuracy, overall Accuracy, and Macro-F₁ for each model.

Among all tested models, Phi-3-mini stands out by achieving the best results across all metrics: Balanced Accuracy (0.57), Accuracy (0.63), and Macro-F₁ (0.57). Zephyr-7B shows similar but slightly lower scores, while

<footnote n="2">²A smaller $p$-value does *not* mean a model is “better”; it only indicates stronger evidence that the two models’ performances are different. Which model performs better must be judged by the metrics themselves (e.g., BAcc, Macro-F₁, ACC).</footnote>
<table><caption>TABLE IV
LLM RESULTS ON THE HAYLING TEST</caption><thead><tr><th>Model</th><th>BAcc</th><th>ACC</th><th>Macro-F<sub>1</sub></th></tr></thead><tbody><tr><td>Phi-3-mini</td><td><strong>0.57</strong></td><td><strong>0.63</strong></td><td><strong>0.57</strong></td></tr><tr><td>Zephyr-7B</td><td>0.56</td><td>0.63</td><td>0.56</td></tr><tr><td>XGLM-7.5B</td><td>0.54</td><td>0.45</td><td>0.45</td></tr><tr><td>Mistral-7B</td><td>0.54</td><td>0.62</td><td>0.53</td></tr><tr><td>Qwen1.5-7B</td><td>0.50</td><td>0.63</td><td>0.47</td></tr><tr><td>DeepSeek-V2</td><td>0.50</td><td>0.62</td><td>0.48</td></tr><tr><td>Llama-2-7B</td><td>0.50</td><td>0.33</td><td>0.29</td></tr><tr><td>Falcon-7B</td><td>0.50</td><td>0.35</td><td>0.26</td></tr><tr><td>Gemma-7B</td><td>0.50</td><td>0.66</td><td>0.40</td></tr><tr><td>BLOOMZ-7B1</td><td>0.49</td><td>0.37</td><td>0.27</td></tr></tbody></table>

models like Mistral-7B, Qwen1.5-7B, DeepSeek-V2, and XGLM-7.5B obtain moderate results. Several popular LLMs, including Llama-2-7B, Falcon-7B, Gemma-7B, and BLOOMZ-7B1, underperform on this task, suggesting limitations for clinical French inhibition scoring in few-shot settings. Overall, even the strongest LLMs remain below the performance of supervised, fine-tuned models, highlighting the challenges of direct clinical application without adaptation.

## VII. DISCUSSION

This work highlights key methodological choices and their implications for automated scoring of the French Hayling Test.

### A. Evaluation Metrics and Class Imbalance

A central decision in our evaluation was to adopt both Balanced Accuracy (BAcc) and Macro-F₁ as primary metrics for model selection and threshold optimization. In imbalanced binary classification, the standard F₁-score computed for the minority class (here, inhibition errors) overlooks the majority class. Macro-F₁ remedies this by averaging F₁-scores across both classes, offering a fairer and more comprehensive performance measure. Balanced Accuracy similarly balances sensitivity and specificity, which is crucial in clinical contexts. Reporting both metrics ensures a robust and clinically meaningful evaluation. This methodological emphasis resonates with prior ICTAI contributions, where rigorous evaluation protocols were central to benchmarking NLP and AI systems in healthcare and language-related tasks [14].

### B. Avoiding Resampling and LLM-Based Augmentation

We deliberately chose not to apply resampling techniques such as SMOTE or random oversampling [39]. These methods are tailored for structured numeric features and often degrade performance on high-dimensional or contextualized textual embeddings [40]. Furthermore, SMOTE interpolates between data points in feature space, which does not preserve semantic or grammatical validity in text-based inputs [41].

Likewise, we avoided data augmentation using large language models (LLMs), since several of these models were benchmarked in our evaluation. Generating training data with the same model family being tested would risk circularity and inflated results. Instead, we advocate for lightweight augmentation approaches such as synonym replacement or

embedding-based substitution (e.g., using FastText neighbors) to preserve interpretability and reduce bias. These strategies are particularly relevant for future experiments on emotionally charged stimuli, where labeled data are scarce.

### C. Toward Emotionally-Informed Variants

The classical Hayling Test uses 15 standardized cloze-style sentences and is widely validated for assessing verbal inhibition across diverse populations. However, this structure also offers semantic flexibility, enabling task adaptation. In collaboration with clinicians, we are exploring a variant that integrates emotionally evocative sentence stems to examine the interaction between affect and inhibitory control. This direction aligns with previous work demonstrating how emotional interference impacts executive function, particularly in conditions such as borderline personality disorder, depression, or trauma-related disorders.

By extending the Hayling paradigm with emotional content, this variant could provide more sensitive markers of dysregulated inhibition in affective disorders. Its clinical relevance is twofold: first, it enables targeted investigation of emotion-cognition interactions; second, it remains compatible with automated NLP-based scoring pipelines, which are well-suited for adapting to new stimuli without compromising standardization.

### D. Toward Inclusive, Real-Time Interfaces

To facilitate practical adoption, we developed and released an open-source graphical application (available in the repository mentioned above), supporting both keyboard and speech input. This interface enables real-time use in clinical or research contexts while minimizing manual effort.

Looking ahead, we envision extending this tool to incorporate automatic sign language input. Such a feature would broaden accessibility for deaf patients, especially in clinical settings where practitioners may lack sign language training. Beyond increasing equity, this extension reinforces our goal of scalable, inclusive, and automated cognitive assessment.

## VIII. CONCLUSION

We introduced a fully automated scoring pipeline for the French Hayling Test, benchmarking static embeddings, contextual encoders, and large language models under real-world class imbalance. Fine-tuned LaBSE achieved the best overall performance, and Phi-3-mini led among LLMs, though still below supervised baselines. Balanced Accuracy and Macro-F₁ ensured robust evaluation across penalty classes.

To maintain methodological consistency, we avoided LLM-generated data and resampling techniques like SMOTE. Future work will explore controlled data augmentation using synonym substitution, as well as adaptation to emotional variants of the Hayling Test and sign language interfaces for broader accessibility.

This work contributes to faster, reproducible, and fair digital assessment of executive function in neuropsychology.
## ACKNOWLEDGMENT

This work of the Interdisciplinary Thematic Institute HealthTech, as part of the ITI 2021–2028 program of the University of Strasbourg, CNRS and Inserm, was supported by IdEx Unistra (ANR-10-IDEX-0002) and SFRI (STRAT’US project, ANR-20-SFRI-0012) under the framework of the French Investments for the Future Program.

## REFERENCES

[1] B. C. P. Ghosh, R. H. S. Carpenter, and J. B. Rowe, "A longitudinal study of motor, oculomotor and cognitive function in progressive supranuclear palsy," *PLoS One*, vol. 8, no. 9, p. e74486, 2013.

[2] A. Siquier and P. Andrés, "Cognitive and behavioral inhibition deficits in parkinson's disease: The hayling test as a reliable marker," *Frontiers in Aging Neuroscience*, vol. 12, p. 621603, 2021.

[3] V. Flaudias, M.-C. Picot, J.-C. Lopez-Castroman, P.-M. Llorca, A. Schmitt, J. Perriot, V. Georgescu, P. Courtet, X. Quantin, and S. Guillaume, "Executive functions in tobacco dependence: Importance of inhibitory capacities," *PLoS One*, vol. 11, no. 3, p. e0150940, 2016.

[4] D. Jan, J. López-Pigüe, I. Padrón, and M. de Vega, "Semantic inhibition impairment in college students with depressive states as evidenced by eeg and pupillometry during the hayling task," *Scientific Reports*, vol. 14, no. 1, p. 24089, 2024.

[5] P. W. Burgess and T. Shallice, "Response suppression, initiation and strategy use following frontal lobe lesions," *Neuropsychologia*, vol. 34, no. 4, pp. 263–272, 1996.

[6] S. Bayard, M.-C. Gély-Nargeot, S. Raffard, S. Guerdoux-Ninot, E. Kamara, F. Gros-Balthazard, J.-P. Jacus, C. Moroni, and C.-L. Roussillon, "French version of the hayling sentence completion test, part i: normative data and guidelines for error scoring," *Archives of Clinical Neuropsychology*, vol. 32, no. 5, pp. 585–591, 2017.

[7] T. Cervera-Crespo and J. González-Alvarez, "Age and semantic inhibition measured by the hayling task: A meta-analysis," *Archives of Clinical Neuropsychology*, vol. 32, no. 2, pp. 198–214, 2017.

[8] A. A. M. Bielak, L. Mansueti, E. Strauss, and R. A. Dixon, "Performance on the hayling and brixton tests in older adults: Norms and correlates," *Archives of Clinical Neuropsychology*, vol. 21, no. 2, pp. 141–149, 2006.

[9] M. J. Kmiecik, B. N. Rodgers, D. M. Martinez, S. B. Chapman, and D. C. Krawczyk, "A method for characterizing semantic and lexical properties of sentence completions in traumatic brain injury," *Psychological Assessment*, vol. 30, no. 5, pp. 645–655, 2018.

[10] J. C. Vásquez-Correa, T. Arias-Vergara, J. R. Orozco-Arroyave, B. Eskofier, J. Klucken, and E. Nöth, "Multimodal assessment of parkinson's disease: A deep learning approach," *IEEE Journal of Biomedical and Health Informatics*, vol. 23, no. 4, pp. 1618–1630, 2018.

[11] E. Dynamant, R. Lelong, B. Dahamna, C. Massonnaud, G. Kerdelhé, J. Grosjean, S. Canu, S. J. Darmoni et al., "Word embedding for the french natural language in health care: comparative study," *JMIR Medical Informatics*, vol. 7, no. 3, p. e12310, 2019.

[12] T. Schick and H. Schütze, "Exploiting cloze questions for few-shot text classification and natural language inference," in *Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics (EACL)*, 2021, pp. 255–269.

[13] H. Zhou, F. Liu, B. Gu, X. Zou, J. Huang, J. Wu, Y. Li, S. S. Chen, P. Zhou, J. Liu et al., "A survey of large language models in medicine: Progress, application, and challenge," 2023.

[14] C. Gong, P. Wu, and J. D. Choi, "Aligning speakers: Evaluating and visualizing text-based speaker diarization using efficient multiple sequence alignment," in *Proceedings of the 35th IEEE International Conference on Tools with Artificial Intelligence (ICTAI)*, 2023, pp. 778–783.

[15] T. Mikolov, K. Chen, G. Corrado, and J. Dean, "Efficient estimation of word representations in vector space," 2013.

[16] J. Pennington, R. Socher, and C. D. Manning, "Glove: Global vectors for word representation," in *Proceedings of EMNLP*, 2014, pp. 1532–1543.

[17] P. Bojanowski, E. Grave, A. Joulin, and T. Mikolov, "Enriching word vectors with subword information," *Transactions of the Association for Computational Linguistics*, vol. 5, pp. 135–146, 2017.

[18] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, "Bert: Pre-training of deep bidirectional transformers for language understanding," in *Proceedings of NAACL-HLT*, 2019, pp. 4171–4186.

[19] N. Reimers and I. Gurevych, "Sentence-bert: Sentence embeddings using siamese bert-networks," in *Proceedings of EMNLP*, 2019, pp. 3982–3992.

[20] F. Feng, Y. Yang, D. Cer, N. Arivazhagan, and W. Wang, "Lase: Language-agnostic bert sentence embedding," in *Proceedings of EMNLP*, 2020, pp. 883–895.

[21] L. Martin et al., "Camembert: a tasty french language model," in *Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*, 2020.

[22] M. I. Abdin et al., "Phi-3 technical report: A highly capable language model locally on your phone," 2024, coRR preprint.

[23] L. Tunstall, S. von Werra, S. Ruder et al., "Zephyr: Direct preference optimization for open llms," *HuggingFace Blog*, 2023. [Online]. Available: https://huggingface.co/HuggingFaceH4/zephyr-7b-beta

[24] M. Siino, "Mcrock at semeval-2024 task 4: Mistral 7b for multilingual detection of persuasion techniques in memes," in *Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024)*, 2024, pp. 53–59.

[25] Owen Team, "Qwen1.5: A family of open multilingual language models," Online, 2024. [Online]. Available: https://huggingface.co/Qwen/Qwen1.5-7B-Chat

[26] DeepSeek Team, "Deepseek llm," Online, 2024. [Online]. Available: https://huggingface.co/deepseek-ai/deepseek-llm-7b-chat

[27] X. Li, Y. Zhu, T. Du et al., "Xglm: An explanation dataset for cross-lingual language models," in *Advances in Neural Information Processing Systems (NeurIPS)*, 2021.

[28] H. Touvron, L. Martin, K. Stone, P. Almazrouei, J. B. Taylor, A. Babaei, Y. Bashlykov et al., "Llama 2: Open foundation and fine-tuned chat models," 2023.

[29] E. Almazrouei et al., "The falcon series of open language models," 2023, coRR preprint.

[30] Google, "Gemma: Open models built from the same research and technology used to create gemini models," Online, 2024. [Online]. Available: https://ai.google.dev/gemma

[31] N. Muennighoff et al., "Crosslingual generalization through multitask finetuning," in *Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics*, 2023.

[32] S. Bayard, C. Madiouni, C. Niel Carlos, C. Cornu, A. Moulin, and S. Raffard, "French adult normative data for a standard executive neuropsychological test battery," *Archives of Clinical Neuropsychology*, vol. 40, no. 3, pp. 486–497, 2025.

[33] P. Andrés and M. Van der Linden, "Age-related differences in supervisory attentional system functions," *The Journals of Gerontology Series B: Psychological Sciences and Social Sciences*, vol. 55, no. 6, pp. P373–P380, 2000.

[34] M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer, "Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension," in *Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*, 2020, pp. 7871–7880.

[35] G. Le, A. Mansour, L. Martin, and J. Deleu, "Flaubert: Unsupervised language model pre-training for french," in *Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing*, 2020, pp. 119–127.

[36] J. Wei, X. Wang, D. Schuurmans, M. Bosma, E. H. Chi, Q. V. Le, and D. Zhou, "Finetuned language models are zero-shot learners," 2021.

[37] A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Morin, M. Ghazvininejad, L. Zettlemoyer, and V. Stoyanov, "Unsupervised cross-lingual representation learning at scale," in *Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics*, 2020, pp. 8440–8451.

[38] A. Williams, N. Nangia, and S. R. Bowman, "A broad-coverage challenge corpus for sentence understanding through inference," in *Proceedings of NAACL-HLT*, 2018, pp. 1112–1122.

[39] N. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer, "Smote: synthetic minority over-sampling technique," *Journal of Artificial Intelligence Research*, vol. 16, pp. 321–357, 2002.

[40] R. Blagus and L. Lusa, "Smote for high-dimensional class-imbalanced data," *BMC Bioinformatics*, vol. 14, no. 1, p. 106, 2013.

[41] S. F. Taskiran, B. Turkoglu, E. Kaya et al., "A comprehensive evaluation of oversampling techniques for enhancing text classification performance," *Scientific Reports*, vol. 15, p. 21631, 2025.