<figure><img src="image_1.png" /></figure>

HAL
open science

# Assessing learning materials: hybrid vs Large Language Model-based generation of grammar exercises

Lucas Poirot, Yannick Parmentier

► To cite this version:

Lucas Poirot, Yannick Parmentier. Assessing learning materials: hybrid vs Large Language Model-based generation of grammar exercises. EvalLAC'25: 2nd Workshop on Automatic Evaluation of Learning and Assessment Content, Jul 2025, Palermo, Italy. hal-05138160

HAL Id: hal-05138160

https://hal.science/hal-05138160v1

Submitted on 1 Jul 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Assessing learning materials: hybrid vs Large Language Model-based generation of grammar exercises

Lucas Poirot¹, Yannick Parmentier¹,²

¹Université de Lorraine, CNRS, LORIA, F-54000 Nancy, France

²Université d'Orléans, LIFO, F-45000 Orléans, France

**Abstract**

In this paper, we compare two methods for automatic generation of grammar exercises from authentic texts: (i) a hybrid method where texts are first fed to a neural syntactic parser in order to get morpho-syntactic annotations, from which grammar concepts (e.g. direct objects of verbs) are automatically extracted using handcrafted rules (and then turned into grammar questions), and (ii) a Large Language Model (LLM) -based method where grammar concepts are extracted directly from texts using few shot learning. Concept extraction is evaluated against manually annotated data for French coming from the SEQUOIA treebank, using grammar concepts from French elementary and middle school curricula. Preliminary results show that hybrid approaches offer a viable alternative to pre-trained LLMs in the context of grammar exercise generation.

**Keywords**

Computer Assisted Language Learning, Grammar exercises, Large Language Model, Dependency trees

## 1. Introduction

Since the advent of robust deep machine learning models, especially pre-trained Large Language Models (LLM), and their spread among the public a few years ago, the field of education has changed drastically. The linguistic capabilities of LLM have been seen by a wide range of actors in education, including learners and teachers, as a game-changing opportunity [1]. The utility of LLM in educational contexts have been demonstrated empirically on tasks including generation of comprehension exercises [2], language adaptation [3, 4], Automated Essay Scoring [5], item difficulty prediction [6] or Intelligent Tutoring Systems [7] to name a few. While exhibiting promising linguistic capabilities, these deep models suffer from a limited explainability, which makes their use hazardous in contexts where precision is as important as robustness (if not more). The design of learning materials belongs to such contexts. Indeed, presenting ill-formed learning materials can lead to a lost of trust and commitment to the learning activities from learners. Still, designing learning materials, especially exercises is a tedious and time-consuming task. Several approaches can be used to reduce the workload of material creation using manual (see e.g. [8]) or automatic (see e.g. [9, 10, 11]) processes.

In this work, we are interested in assessing the quality of learning materials automatically generated from authentic texts in the context of grammar learning. We consider the specific task of annotating *grammar concepts* (such as *subject, verb simple past*, etc.) in texts (which is, in our case, followed by a transformation into typical grammar questions such as *Fill-in-the-Blanks, Multiple-Choice or Mark-the-Words*). We compare two approaches described below: hybrid vs fully neural models.

## 2. Models

**Hybrid model** Our hybrid model consists of handcrafted rules which are applied on texts pre-annotated by a neural linguistic analysis engine such as Stanza [12]. The syntactic annotation scheme used by such engines corresponds to Universal Dependencies (UD) [13] and the annotation format to CoNLL-U [14]. Basically annotations encapsulate dependency trees, while our handcrafted rules are expressions of a tree description language inspired by the Grew treebank query language [15], e.g.:

EvalLAC'25: 2nd Workshop on Automatic Evaluation of Learning and Assessment Content, July 26, 2025, Palermo, Italy

<figure><img src="image_3.png" /></figure>

lucas.poirot@loria.fr (L. Poirot); yannick.parmentier@loria.fr (Y. Parmentier)

<figure><img src="image_4.png" /></figure>

© 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
[Tense=Pres] aux > [Tense=Past&VerbForm=Part]

This expression selects pairs of words such that these are labelled with the morphological features Tense=Pres and Tense=Past&VerbForm=Part respectively, and such that there exists a dependency relation of type aux between them. This rule is used to annotate the *verb in simple past* concept.

**Large-Language Models** We selected open-source models from the DeepSeek-R1 family [16]. These are so-called reasoning models. On top of achieving state-of-the-art results on various benchmarks such as MMLU for language understanding [17], they have been trained specifically to realize complex tasks including document analysis, pattern or precise information extraction [18], which makes them good candidates for grammar concepts annotation. We experimented with two models, a medium- and a large-size one (both having a context window of 128,000 tokens): **DeepSeek-R1-Distill-Llama-70B** and **DeepSeek-R1**. We used *few shot learning*: we fed a small number of annotated examples to the models using structured prompting prior to asking them to infer grammar annotations on new texts (an example of a prompt used for few-shot learning is given in Appendix A.1.1). For each grammar concept we trained the models with 25 examples (which did not belong to the evaluation data).

## 3. Experimental setup

**Dataset** For our experiment, we selected the following grammar concepts from French middle and elementary school curricula [19]: adjectives, adverbs, coordinating conjunction, determiner, direct object, noun, preposition, subject, subject attribute, subordinating conjunction, and verb.

To evaluate the capabilities of our models, we created a gold-standard dataset. We used the **SEQUOIA** treebank for French [20], which contains handcrafted deep syntactic annotations in the Penn TreeBank constituency format. From these annotated structures, grammar concepts were extracted using regular expressions parameterized by PTB tags and structural information (namely the list of ancestor nodes). The quality of the output gold-standard was evaluated manually on randomly selected annotated sentences. The resulting dataset contains 2,231 sentences, 76422 tokens (≈ words), 272,626 characters.

**Evaluation metrics** Our annotation task is somehow similar to Named Entity Recognition (NER) where text spans are labelled according to the entity they represent. We thus used the evaluation metrics from NER [21] (see details in Appendix). Using these metrics in a so-called *strict* setting, we computed the classical precision (number of predicted labelled spans that line up exactly with spans in the gold-standard), recall (number of grammar concepts in the gold standard that appear at exactly the same location in the predictions) and f-measure (harmonic mean of precision and recall).

**Preliminary results** The results of our experiments are summarized in table 1 below, which contains the average scores over all target grammar concepts for each model.

<table><caption>**Table 1**
Summary of grammar concept annotation (average scores over all concepts).</caption><thead><tr><th>Model</th><th>#Sentences</th><th>#COR</th><th>#INC</th><th>#PAR</th><th>#MIS</th><th>#SPU</th><th>Prec.</th><th>Recall</th><th>F-score</th></tr></thead><tbody><tr><td>Hybrid</td><td>2,231</td><td>62,316</td><td>391</td><td>0</td><td>7,905</td><td>3,436</td><td><b>0.94</b></td><td><b>0.88</b></td><td><b>0.91</b></td></tr><tr><td>DeepSeek-R1-Llama-70B</td><td>48</td><td>908</td><td>372</td><td>0</td><td>555</td><td>1,297</td><td>0.35</td><td>0.49</td><td>0.41</td></tr><tr><td>DeepSeek-R1</td><td>48</td><td>1,217</td><td>277</td><td>0</td><td>365</td><td>1,178</td><td>0.46</td><td>0.65</td><td>0.54</td></tr></tbody></table>

The hybrid model outperforms the neural ones in terms of both precision and recall. The scaling law of neural models can also be seen as the large-size model exhibits a significant improvement (+0.13 in f-measure) over the medium-size one. The positive bias of LLMs can also be observed by the relatively large number of spurious annotations. These first results need to be mitigated notably due to the limitations in the use of the LLMs (see *Limitation* section in Appendix).
# Acknowledgments

The authors acknowledge the support of CNRS Innovation, under grant CG-06-22 (project Gramex).

# Declaration on Generative AI

The authors have not employed any Generative AI tools.

# References

[1] R. Arum, M. C. Leon, X. Li, J. Lopes, Chatgpt early adoption in higher education: Variation in student usage, instructional support, and educational equity, AERA Open 11 (2025) 23328584251331956. URL: https://doi.org/10.1177/23328584251331956. doi:10.1177/23328584251331956. arXiv:https://doi.org/10.1177/23328584251331956.

[2] C. Xiao, S. X. Xu, K. Zhang, Y. Wang, L. Xia, Evaluating reading comprehension exercises generated by LLMs: A showcase of ChatGPT in education applications, in: E. Kochmar, J. Burstein, A. Horbach, R. Laarmann-Quante, N. Madnani, A. Tack, V. Yaneva, Z. Yuan, T. Zesch (Eds.), Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023), Association for Computational Linguistics, Toronto, Canada, 2023, pp. 610–625. URL: https://aclanthology.org/2023.bea-1.52/. doi:10.18653/v1/2023.bea-1.52.

[3] D. Rooein, P. Röttger, A. Shaitarova, D. Hovy, Beyond flesch-kincaid: Prompt-based metrics improve difficulty classification of educational texts, in: E. Kochmar, M. Bexte, J. Burstein, A. Horbach, R. Laarmann-Quante, A. Tack, V. Yaneva, Z. Yuan (Eds.), Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024), Association for Computational Linguistics, Mexico City, Mexico, 2024, pp. 54–67. URL: https://aclanthology.org/2024.bea-1.5/.

[4] C.-Y. Huang, J. Wei, T.-H. K. Huang, Generating educational materials with different levels of readability using llms, 2024. URL: https://arxiv.org/abs/2406.12787. arXiv:2406.12787.

[5] S. Banno, H. K. Vydana, K. Knill, M. Gales, Can GPT-4 do L2 analytic assessment?, in: E. Kochmar, M. Bexte, J. Burstein, A. Horbach, R. Laarmann-Quante, A. Tack, V. Yaneva, Z. Yuan (Eds.), Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024), Association for Computational Linguistics, Mexico City, Mexico, 2024, pp. 149–164. URL: https://aclanthology.org/2024.bea-1.14/.

[6] A.-C. Rogoz, R. T. Ionescu, UnibucLLM: Harnessing LLMs for automated prediction of item difficulty and response time for multiple-choice questions, in: E. Kochmar, M. Bexte, J. Burstein, A. Horbach, R. Laarmann-Quante, A. Tack, V. Yaneva, Z. Yuan (Eds.), Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024), Association for Computational Linguistics, Mexico City, Mexico, 2024, pp. 493–502. URL: https://aclanthology.org/2024.bea-1.41/.

[7] E. G. Gado, T. Martorella, L. Zunino, P. Mejia-Domenzain, V. Swamy, J. Frej, T. KÃßer, Student answer forecasting: Transformer-driven answer choice prediction for language learning, in: B. PaaÄyen, C. D. Epp (Eds.), Proceedings of the 17th International Conference on Educational Data Mining, International Educational Data Mining Society, Atlanta, Georgia, USA, 2024, pp. 634–642. doi:10.5281/zenodo.12729904.

[8] X. Hou, O. Hafez, Y. Zhang, T. Nagy, S. Khalid, Z. Kazmi, Simpledoc: Crowdsourcing learning materials from learners, in: Proceedings of the 26th ACM Conference on Innovation and Technology in Computer Science Education V. 2, ITiCSE '21, Association for Computing Machinery, New York, NY, USA, 2021, p. 648. URL: https://doi.org/10.1145/3456565.3460069. doi:10.1145/3456565.3460069.

[9] T. Heck, D. Meurers, Using learning analytics for adaptive exercise generation, in: E. Kochmar, J. Burstein, A. Horbach, R. Laarmann-Quante, N. Madnani, A. Tack, V. Yaneva, Z. Yuan, T. Zesch
(Eds.), Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023), Association for Computational Linguistics, Toronto, Canada, 2023, pp. 44–56. URL: https://aclanthology.org/2023.bea-1.4/. doi:10.18653/v1/2023.bea-1.4.

[10] M. Uto, Y. Tomikawa, A. Suzuki, Difficulty-controllable neural question generation for reading comprehension using item response theory, in: E. Kochmar, J. Burstein, A. Horbach, R. Laarmann-Quante, N. Madnani, A. Tack, V. Yaneva, Z. Yuan, T. Zesch (Eds.), Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023), Association for Computational Linguistics, Toronto, Canada, 2023, pp. 119–129. URL: https://aclanthology.org/2023.bea-1.10/. doi:10.18653/v1/2023.bea-1.10.

[11] B. Paddags, D. Hershcovich, V. Savage, Automated sentence generation for a spaced repetition software, in: E. Kochmar, M. Bexte, J. Burstein, A. Horbach, R. Laarmann-Quante, A. Tack, V. Yaneva, Z. Yuan (Eds.), Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024), Association for Computational Linguistics, Mexico City, Mexico, 2024, pp. 351–364. URL: https://aclanthology.org/2024.bea-1.29/.

[12] P. Qi, Y. Zhang, Y. Zhang, J. Bolton, C. D. Manning, Stanza: A Python natural language processing toolkit for many human languages, in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, 2020. URL: https://nlp.stanford.edu/pubs/qi2020stanza.pdf.

[13] M.-C. de Marneffe, C. D. Manning, J. Nivre, D. Zeman, Universal Dependencies, Computational Linguistics 47 (2021) 255–308. URL: https://aclanthology.org/2021.cl-2.11/. doi:10.1162/coli_a_00402.

[14] S. Buchholz, E. Marsi, CoNLL-X shared task on multilingual dependency parsing, in: L. Màrquez, D. Klein (Eds.), Proceedings of the Tenth Conference on Computational Natural Language Learning (CoNLL-X), Association for Computational Linguistics, New York City, 2006, pp. 149–164. URL: https://aclanthology.org/W06-2920/.

[15] B. Guillaume, Graph matching and graph rewriting: GREW tools for corpus exploration, maintenance and conversion, in: D. Gkatzia, D. Seddah (Eds.), Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, Association for Computational Linguistics, Online, 2021, pp. 168–175. URL: https://aclanthology.org/2021.eacl-demos.21/. doi:10.18653/v1/2021.eacl-demos.21.

[16] DeepSeek-AI, D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, X. Zhang, X. Yu, Y. Wu, Z. F. Wu, Z. Gou, Z. Shao, Z. Li, Z. Gao, A. Liu, B. Xue, B. Wang, B. Wu, B. Feng, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan, D. Dai, D. Chen, D. Ji, E. Li, F. Lin, F. Dai, F. Luo, G. Hao, G. Chen, G. Li, H. Zhang, H. Bao, H. Xu, H. Wang, H. Ding, H. Xin, H. Gao, H. Qu, H. Li, J. Guo, J. Li, J. Wang, J. Chen, J. Yuan, J. Qiu, J. Li, J. L. Cai, J. Ni, J. Liang, J. Chen, K. Dong, K. Hu, K. Gao, K. Guan, K. Huang, K. Yu, L. Wang, L. Zhang, L. Zhao, L. Wang, L. Zhang, L. Xu, L. Xia, M. Zhang, M. Zhang, M. Tang, M. Li, M. Wang, M. Li, N. Tian, P. Huang, P. Zhang, Q. Wang, Q. Chen, Q. Du, R. Ge, R. Zhang, R. Pan, R. Wang, R. J. Chen, R. L. Jin, R. Chen, S. Lu, S. Zhou, S. Chen, S. Ye, S. Wang, S. Yu, S. Zhou, S. Pan, S. S. Li, S. Zhou, S. Wu, S. Ye, T. Yun, T. Pei, T. Sun, T. Wang, W. Zeng, W. Zhao, W. Liu, W. Liang, W. Gao, W. Yu, W. Zhang, W. L. Xiao, W. An, X. Liu, X. Wang, X. Chen, X. Nie, X. Cheng, X. Liu, X. Xie, X. Liu, X. Yang, X. Li, X. Su, X. Lin, X. Q. Li, X. Jin, X. Shen, X. Chen, X. Sun, X. Wang, X. Song, X. Zhou, X. Wang, X. Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Y. Zhang, Y. Xu, Y. Li, Y. Zhao, Y. Sun, Y. Wang, Y. Yu, Y. Zhang, Y. Shi, Y. Xiong, Y. He, Y. Piao, Y. Wang, Y. Tan, Y. Ma, Y. Liu, Y. Guo, Y. Ou, Y. Wang, Y. Gong, Y. Zou, Y. He, Y. Xiong, Y. Luo, Y. You, Y. Liu, Y. Zhou, Y. X. Zhu, Y. Xu, Y. Huang, Y. Li, Y. Zheng, Y. Zhu, Y. Ma, Y. Tang, Y. Zha, Y. Yan, Z. Z. Ren, Z. Ren, Z. Sha, Z. Fu, Z. Xu, Z. Xie, Z. Zhang, Z. Hao, Z. Ma, Z. Yan, Z. Wu, Z. Gu, Z. Zhu, Z. Liu, Z. Li, Z. Xie, Z. Song, Z. Pan, Z. Huang, Z. Xu, Z. Zhang, Z. Zhang, Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025. URL: https://arxiv.org/abs/2501.12948. arXiv:2501.12948.

[17] D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, J. Steinhardt, Measuring massive multitask language understanding., in: ICLR, 2021. URL: http://dblp.uni-trier.de/db/conf/iclr/iclr2021.html#HendrycksBBZMSS21.
[18] M. Besta, J. Barth, E. Schreiber, A. Kubicek, A. Catarino, R. Gerstenberger, P. Nyczyk, P. Iff, Y. Li, S. Houliston, T. Sternal, M. Copik, G. Kwaśniewski, J. Müller, Łukasz Flis, H. Eberhard, H. Niewiadomski, T. Hoefler, Reasoning language models: A blueprint, 2025. URL: https://arxiv.org/abs/2501.11223. arXiv:2501.11223.

[19] P. Monneret, F. Poli, La grammaire du français. Terminologie grammaticale, EDUSCOL, 2020. URL: https://hal.science/hal-03959089.

[20] M. Candito, G. Perrier, B. Guillaume, C. Ribeyre, K. Fort, D. Seddah, É. Villemonte de La Clergerie, Deep Syntax Annotation of the Sequoia French Treebank, in: International Conference on Language Resources and Evaluation (LREC), Reykjavik, Iceland, 2014. URL: https://inria.hal.science/hal-00969191.

[21] I. Segura-Bedmar, P. Martínez, M. Herrero-Zazo, SemEval-2013 task 9: Extraction of drug-drug interactions from biomedical texts (DDIExtraction 2013), in: S. Manandhar, D. Yuret (Eds.), Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013), Association for Computational Linguistics, Atlanta, Georgia, USA, 2013, pp. 341–350. URL: https://aclanthology.org/S13-2056/.

[22] G. Hinton, O. Vinyals, J. Dean, Distilling the knowledge in a neural network, 2015. URL: https://arxiv.org/abs/1503.02531. arXiv:1503.02531.

[23] M. Sclar, Y. Choi, Y. Tsvetkov, A. Suhr, Quantifying language models’ sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting, 2024. URL: https://arxiv.org/abs/2310.11324. arXiv:2310.11324.

[24] A. Razavi, M. Soltangheis, N. Arabzadeh, S. Salamat, M. Zihayat, E. Bagheri, Benchmarking prompt sensitivity in large language models, 2025. URL: https://arxiv.org/abs/2502.06065. arXiv:2502.06065.

# A. Details about the experimental setup

## A.1. Large Language Models

We used the following models of the DeepSeek R1 family:<fnref n="1" />

* DeepSeek-R1-Distill-Llama-70B is a medium-size model having 70 billion parameters. It was trained using distillation [22], a knowledge compression technique used to reduce model size. During this process, Llama-3.3-70B-Instruct has been used as a base model fine-tuned on data generated by DeepSeek-R1.

* DeepSeek-R1 is a large-size model of 685 billions of parameters, which has been trained using reinforcement learning techniques.

### A.1.1. Example of a prompt used for few-shot learning

Based on the examples below, parse the input sentence into its treebank representation using bracketed French labels as shown. Return each word in the same order as the sentence input. Examples:

(nom:(déterminant:Vingt-six) (nom:tableaux) (préposition:(préposition:pour)
(verbe:(verbe:(verbe:expliquer))) (complément d'objet direct:(pronom:ce)
(attribut du sujet:(pronom:qu')) (verbe:(verbe:est)) (sujet:(déterminant:une)
(nom:éclipse) (préposition:(préposition:de) (nom:(nom:soleil))))))
(préposition:(préposition:avec) (nom:(déterminant:une) (nom:quinzaine)
(préposition:(préposition:de) (nom:(nom:définitions)
(préposition:(préposition:des) (nom:(nom:mots-clefs) (adjectif:tels)

<footnote n="1">¹https://huggingface.co/deepseek-ai/DeepSeek-R1</footnote>
(préposition::) (nom:(nom:amas) (adjectif:globulaire)) (préposition:.)
(nom:(nom:amas) (adjectif:ouvert)) (préposition:.) (nom:(nom:comètes))
(préposition:.) (nom:(nom:constellations)) (préposition:.)
(nom:(nom:galaxie)) (préposition:.) (nom:(nom:mama) (préposition:-LRB-))
(nom:(nom:machine) (adjectif:automatique) (préposition:(préposition:à))
(verbe:(verbe:(verbe:mesurer)) (complément d'objet direct:(déterminant:1'))
(nom:astronomie)))) (préposition:-RRB-))))))))) (préposition:.)
(sujet:(déterminant:Les) (nom:associations) (préposition:(préposition:du))
(nom:(nom:village)))) (verbe:(verbe:ont) (verbe:vu)) (complément d'objet
direct:(déterminant:leur) (nom:contribution)) (verbe:(verbe:revue))
(préposition:(préposition:à) (nom:(déterminant:la) (nom:baisse)))) (préposition:.)
...

## A.2. Evaluation

Automatically annotated texts were compared with the gold-standard annotations token by token, in order to compute the following metrics:<fnref n="2" />

* Correct (COR) – both annotations (prediction and goldstandard) are the same,

* Incorrect (INC) – annotations do not match,

* Partial (PAR) – model and golden annotations overlap,

* Missing (MIS) – a golden annotation is not captured by the model,

* Spurious (SPU) – a prediction does not exist in the golden annotation.

The full SEQUOIA dataset<fnref n="3" /> was used when evaluating our hybrid model. Due to rate limits enforced by deep models' providers, we could not use the full dataset when evaluating neural models. We had to strongly limit our dataset to 48 randomly chosen sentences (1648 tokens, 5832 characters). In other words, these were evaluated on twice as many predictions as examples given during few-shot training.

## A.3. Detailed results

If we give a closer look at the results per concept, which are given in table 2, we can see that these are relatively stable across concepts. The hybrid model systematically gets the higher scores on the considered concepts.

<caption>**Table 2**</caption>

Annotation scores of selected grammar concepts.

<table><thead><tr><th>Concept</th><th>Model</th><th>#COR</th><th>#INC</th><th>#PAR</th><th>#MIS</th><th>#SPU</th><th colspan="3">Prec. Recall F-score</th></tr></thead><tbody><tr><td rowspan="3">Subject</td><td>Hybrid</td><td>8,666</td><td>56</td><td>0</td><td>157</td><td>380</td><td><strong>0.95</strong></td><td><strong>0.98</strong></td><td><strong>0.96</strong></td></tr><tr><td>DeepSeek-R1-Llama-70B</td><td>122</td><td>47</td><td>0</td><td>60</td><td>603</td><td>0.19</td><td>0.53</td><td>0.24</td></tr><tr><td>DeepSeek-R1</td><td>173</td><td>43</td><td>0</td><td>29</td><td>600</td><td>0.21</td><td>0.71</td><td>0.33</td></tr><tr><td rowspan="3">Direct object</td><td>Hybrid</td><td>13,393</td><td>193</td><td>0</td><td>1,349</td><td>1,755</td><td><strong>0.87</strong></td><td><strong>0.90</strong></td><td><strong>0.88</strong></td></tr><tr><td>DeepSeek-R1-Llama-70B</td><td>126</td><td>10</td><td>0</td><td>220</td><td>30</td><td>0.76</td><td>0.35</td><td>0.48</td></tr><tr><td>DeepSeek-R1</td><td>245</td><td>2</td><td>0</td><td>119</td><td>37</td><td>0.86</td><td>0.67</td><td>0.75</td></tr><tr><td rowspan="3">Verb</td><td>Hybrid</td><td>5,778</td><td>23</td><td>0</td><td>45</td><td>112</td><td><strong>0.98</strong></td><td><strong>0.98</strong></td><td><strong>0.98</strong></td></tr><tr><td>DeepSeek-R1-Llama-70B</td><td>98</td><td>29</td><td>0</td><td>22</td><td>234</td><td>0.27</td><td>0.66</td><td>0.38</td></tr><tr><td>DeepSeek-R1</td><td>104</td><td>29</td><td>0</td><td>21</td><td>197</td><td>0.32</td><td>0.68</td><td>0.43</td></tr></tbody></table>

<footnote n="2">²Scoring was computed by the NERvaluate Python package https://github.com/MantisAI/nervvaluate.</footnote>

<footnote n="3">³https://deep-sequoia.inria.fr</footnote>
<figure><img src="image_5.png" /><figcaption>**Figure 1:** Evaluation of grammar concept annotation by the hybrid model using precision, recall and f-score metrics.</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>**Figure 2:** Evaluation of grammar concept annotation by DeepSeek-R1-Distill-Llama-70B using precision, recall and f-score metrics.</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>**Figure 3:** Evaluation of grammar concept annotation by DeepSeek-R1 using precision, recall and f-score metrics.</figcaption></figure>
## B. Limitations

Our results need to be mitigated. Indeed, as mentioned above, the size of the models' evaluation datasets is very unbalanced. Neural models are evaluated against tiny datasets compared with that of the rule-based model. This is somehow compensated by the fact that the robustness of symbolic rules is challenged by the size of the dataset.

Furthermore, the choice of the target grammar concepts may have a large impact on the results since these are closely related to the syntactic annotations used by the rule-based model. Some concepts are correctly annotated due to the fact that the parser already marked the target concept (i.e., a mere alignment between tags is needed).

Finally, general limitations apply here: these results are sensitive to the considered language (recall the models were asked to process French texts) since LLM's linguistic capabilities are. LLMs are known to be sensitive to the prompt design [23], further work would be required to check whether our experiments suffered from this sensitivity. We could for instance build on [24].

## C. Online Resources

Source code used in this work (including evaluation dataset and scripts) are available at https://github.com/Sacus1/Compare.