# Fine grain semantic analysis by LLMs: real-world technical reports vs simple synthetic sentences

Mariame Maarouf, Ludovic Tanguy

► To cite this version:

Mariame Maarouf, Ludovic Tanguy. Fine grain semantic analysis by LLMs: real-world technical reports vs simple synthetic sentences. "LLM fails" workshop, IDS (Leibniz Institute for the German Language), Apr 2025, Mannheim, Germany. hal-05386025

HAL Id: hal-05386025

https://hal.science/hal-05386025v1

Submitted on 3 Dec 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Fine grain semantic analysis by LLMs: real-world technical reports vs simple synthetic sentences

## Abstract

This study explores the use of an LLM for fine-grain semantic analysis. It focuses on prompting an LLM to for a sequence labeling task on multiple French written datasets (real-word, technical and synthetic data) according to a specific "Hindering" frame. The study shows that the model successfully extracts the information on synthetic data which present simpler sentence structure and vocabulary, but struggles when confronted to web and technical data.

## 1 Introduction

The aim of this study is to perform a fine-grained semantic analysis of industrial anomaly reports. These reports are produced in a critical safety environment. As such, operators are asked to report every anomaly encountered systematically. The corpora at our disposal consist of about 35,000 French written minimalist descriptions of the anomaly (average length of 19.3 words per report), with highly technical vocabulary and a noisy text. The goal of this work is to obtain a formalized description of these issues and, to do so, to identify the specific elements in the text. This identification entails performing an annotation task of the corpora. Such a task can be costly in numerous aspects, especially when working with specialized corpora and experts whose time availability may vary. On the other hand, generative LLMs have opened the door to the automation of a number of tasks, including automatic annotation without a prior manual annotation phase (Ettinger, Hwang, Pyatkin, Bhagavatula, & Choi, 2023) (Cui & Swayamdipta, 2024). In this article, we aim to study the performances of an LLM as an annotator on this kind of data. As can be expected, the task itself is difficult and is made even more so by the technicality and noise of the target text. In order to measure the impact of these characteristics on the performance, we tested the same LLM for the same task but on a different panel of text data. We developed different test sets consisting of a normalized version of the target text, a similar text extracted from a non-technical data source, and finally different versions of synthetic data, i.e. text generated by an LLM in order to produce similar sentences. Section 2 presents a short overview of related studies; Section 3 gives a description of the semantic annotation task; Section 4 presents the datasets and finally Section 5 gives the results for each configuration.
## 2 Related and previous work

Working with specialized domain texts can be a challenge in NLP (Natural Language Processing) given the inadequacy of current methods and tools conceived for (and from) standardized texts. As such, the domain adaptation of these methods requires the use of an expert, especially in manual annotation tasks (Brundage, Sexton, Hodkiewicz, Dima, & Lukens, 2021), (Dima, Lukens, Hodkiewicz, Sexton, & Brundage, 2021). However, the surge of LLMs as able to master linguistic competency (Mahowald et al., 2024) opened the way to their use as assistant in linguistic tasks (Torrent, Hoffmann, Almeida, & Turner, 2024) such as annotation (Ettinger et al., 2023), (Cui & Swayamdipta, 2024). Some studies have also been exploring the use of such models in order to correct the noise, especially in post-OCR treatments (Thomas, Gaizauskas, & Lu, 2024; Zhang, Haverals, Naydan, & Kernighan, 2024).

The work presented here takes place in a longer term project that aims at a deep analysis of the technical problems in an industrial context. Previous work has been performed to automatically classify target reports according to a twelve-class typology of problems. These classes correspond to categories of issues encountered ("malfunction", "leakage", "missing components"...) and are based on lexical markers ("not working", "leak", "absent"...). A pretrained *BERT* based model was fine-tuned on a sequence labeling task, using a set of manually annotated reports, in order to classify these reports. The results attained 0.74 macro-averaged F-score. Among the different class, the "Hindering/Obstacle" one seemed appropriate for a more fine-grain analysis given its regularity in expression structures. In the sequence labeling task prior, this class has been identified with an 0.67 average F1-score.

## 3 How and what to annotate ?

Beyond this classification, obtaining a formalized description of the problems requires further identification of the different elements expressed in the text. We base the selection of the elements on Fillmore’s frame semantic theory (Fillmore & Baker, 2009) and the *Framenet* resource (Ruppenhofer, Ellsworth, Petruck, Johnson, & Scheffczyk, 2016) which offers a formal description of a large selection of frames and their frame elements, and an annotation scheme. The frame semantics theory is based on the hypothesis that to fully understand a word, it requires taking into account the event and situation context in which it occurs.

In this experiment, we focused on an extended *Hindering* frame<fnref n="1" />. This frame normally includes three "Core" elements: *Protagonist*, *Hindrance* and *Action*. It is defined in *Framenet* as: "In this frame a *Hindrance* makes it more difficult for a *Protagonist* to complete their intended *Action* or a *Hindrance* directly affects an *Action* where the *Action* can be a natural act or the actions of many individuals in a collective endeavor. Crucially, these lexical units do not entail that the *Protagonist* was unable to complete the *Action* or that the *Action* did not occur. The *Hindrance* may indicate a lack of

<footnote n="1">¹https://framenet.icsi.berkeley.edu/fnReports/data/frameIndex.xml?frame=Hindering</footnote>
enablement or an obstacle on the natural progression of the *Protagonist or Action*'.
The *Protagonist* is never expressed in the anomaly reports, so was considered irrelevant
in this study. The *Action* is what the *Protagonist* wants to achieve, but the *Hindrance*
makes it difficult. The *Hindrance* refers to the force that prevents or make more difficult
the achievement of the *Action*. We decided to enrich the frame by adding a new element,
*Target*, which corresponds to the recipient of the *Action* which is made more difficult by
the *Hindrance*. For example, in the sentence "Magnesium reduces stress by preventing
the rise of cortisol", the *Hindrance* is "Magnesium", *Target* is "cortisol" and the *Action*
is "rise". Contrary to the target dataset of anomaly reports, this example consists in a
*Hindering* situation which is desirable. However, the frame in question, activated by
the lexical unit "preventing", is indeed the "Hindering" one.

As already stated in Section 2, an annotation of a dataset such as technical reports
requires the work of experts in the field and a significant time cost. To avoid this issue,
we explored using an LLM (Meta-Llama-3-8B (Grattafiori et al., 2024)) to automatically
 annotate technical reports according to the modified *Hindering* frame. To obtain an
overview of the LLM capacity to execute this task, the annotation was performed on
five different datasets, from real-word sentences to synthetic data (see Section 4). The
same prompt was used for the different datasets. In it was included the full definition
of the *Hindering* frame, the three target frame elements with their name, a definition
and an example for each one. It was also specified that all frame elements might not
necessarily be present in the text.

4 Datasets overview

We selected 25 random original tech reports from our corpus, based on the automatic
classification but with an additional verification that they belong to the "hindrance"
category of problem (dataset 1).

In addition, four additional different datasets were used to compare the efficiency of the
automatic annotation, all related to a hindering situation: the result of an automated
cleaning procedure of dataset 1 by an LLM (dataset 2), 25 related sentences selected
from a generic corpus (dataset 3), 25 generic LLM-generated sentences (dataset 4), 25
LLM-generated sentences prompted to reproduce the technical reports from examples
 dataset 5). Table 1 gives a sample item from each dataset, with a translation in English.
As mentioned, the technical reports (1) are full of specialized lexicon and noise phe-
nomena. Among these, we can find: the entirety of the text in uppercase, no accents,
acronyms (as can be seen in example 1), but also some typing errors and punctuation
and spaces are not always respected.

In an attempt to clean these texts without losing information, a "correction" by a small
sized LLM (Mistral-7B-OpenOrca) has been performed on the same 25 sentences (2).
This model was selected after tests carried out on multiple models (Anonymous, 2025).
In the prompt, it was asked to put back the lowercase and the accents as needed, correct
the punctuation issues but to preserve as much of the vocabulary as possible in order
to avoid the modification of acronyms and other technical words.
The third dataset contains sentences from the *frWaC* corpus (Baroni, Bernardini, Ferraresi, & Zanchetta, 2009), a large collection of text extracted from French websites (with the *fr* domain). We extracted full sentences using generic cue phrases from our target reports. The 25 sentences were selected after manually verifying that they are related to a hindering situation. This provided us with real-life, ecological data.

In the fourth dataset, there are 25 sentences automatically generated by an LLM. It was prompted to generate sentences corresponding to the *Hindering* frame. Framenet's definition of the frame and a generic example were given in the prompt.

The last dataset also consists of 25 automatically generated sentences. However, in this case, the prompted instructions were to generate sentences similar to the technical reports we are working with. Three real technical reports were given as example. The goal was to obtain sentences with similar structures (parenthesis, quotes...), but with a mundane vocabulary.

For these two datasets (LLM-generated), the results of the generation are mitigated. The LLM produced sentences with a more accessible vocabulary and related to the hindering frame as asked. On the other hand, the structures and the lexicon were redundant, lacked diversity, across the corpus.

<table><caption>Table 1: Examples from each dataset</caption><thead><tr><td>#</td><td>Dataset</td><td>Sentence</td><td>Our translation</td></tr></thead><tbody><tr><td>1</td><td>Technical reports</td><td>CABLE BERB CLIENT DANS CADRE MCT EMPECHE LA MISE EN PLACE DE LA POE.</td><td>Customer cable BERB in MCT frame prevents the setting of the POE.</td></tr><tr><td>2</td><td>Corrected reports</td><td>Le cheminement du tube de ventilation le long du flex d'alimentation SVE empeche la mise en place conformément au plan de la PT.</td><td>The path of the ventilation tube along the alimentation flex SVE prevents the installation according to the PT plan.</td></tr><tr><td>3</td><td>FrWac</td><td>Le magnésium réduit le stress en empêchant la montée du cortisol.</td><td>Magnesium reduces stress by preventing the rise of cortisol.</td></tr><tr><td>4</td><td>LLM-generated</td><td>Le développement du pays est empêché par les sanctions internationales.</td><td>The country's development is prevented by international sanctions.</td></tr><tr><td>5</td><td>LLM-generated reports</td><td>Le montage du tableau périodique est impossible (la colonne est bloquée) due à la mauvaise position du support.</td><td>The assembly of the periodic table is impossible (the pillar is blocked) due to the support's wrong position.</td></tr></tbody></table>

## 5 Performances of the automatic extraction

The automatic extraction was performed on each item individually and repeated five times for each data to account for non-determinism (temperature=0.8). The extracted answer was evaluated by comparison to a gold standard produced by the authors and verified by an expert. Table 2 gives a sample sentence along with expected results (gold standard) and a sample output.

An answer for a given frame element is considered correct if the extracted text segment overlaps with the gold standard sequence by at least a non-stop word. To further interpret this evaluation, the *Dice coefficient* was used to calculate the overlap
rate (number of characters). If the overlap is high, there is a high correspondence between the annotated segment and the gold. If it is low, either the extracted segment is a mismatch, or it is too wide compared to the gold.

<table><caption>**Table 2: Automatic extraction scores**</caption><thead><tr><th>#</th><th>Datasets</th><th>Recall</th><th>Precision</th><th>F-score</th><th>Overlap</th></tr></thead><tbody><tr><td colspan="6" style="text-align:center; font-weight:bold;">Hindrance</td></tr><tr><td>1</td><td>Technical reports</td><td>0.57</td><td>0.87</td><td>0.69</td><td>0.33</td></tr><tr><td>2</td><td>Corrected reports</td><td>0.57</td><td>0.86</td><td>0.68</td><td>0.30</td></tr><tr><td>3</td><td>FrWac</td><td>0.43</td><td>0.83</td><td>0.56</td><td>0.32</td></tr><tr><td>4</td><td>LLM-generated</td><td>0.98</td><td>0.92</td><td>0.95</td><td>0.68</td></tr><tr><td>5</td><td>LLM-generated reports</td><td>0.98</td><td>1</td><td>0.99</td><td>0.46</td></tr><tr><td colspan="6" style="text-align:center; font-weight:bold;">Action</td></tr><tr><td>1</td><td>Technical reports</td><td>0.84</td><td>0.81</td><td>0.83</td><td>0.35</td></tr><tr><td>2</td><td>Corrected reports</td><td>0.69</td><td>0.78</td><td>0.73</td><td>0.30</td></tr><tr><td>3</td><td>FrWac</td><td>0.68</td><td>0.59</td><td>0.63</td><td>0.28</td></tr><tr><td>4</td><td>LLM-generated</td><td>0.96</td><td>0.92</td><td>0.94</td><td>0.55</td></tr><tr><td>5</td><td>LLM-generated reports</td><td>0.98</td><td>1</td><td>0.99</td><td>0.71</td></tr><tr><td colspan="6" style="text-align:center; font-weight:bold;">Target</td></tr><tr><td>1</td><td>Technical reports</td><td>0.56</td><td>0.93</td><td>0.70</td><td>0.36</td></tr><tr><td>2</td><td>Corrected reports</td><td>0.45</td><td>0.91</td><td>0.60</td><td>0.28</td></tr><tr><td>3</td><td>FrWac</td><td>0.73</td><td>0.95</td><td>0.82</td><td>0.48</td></tr><tr><td>4</td><td>LLM-generated</td><td>0.82</td><td>1</td><td>0.90</td><td>0.73</td></tr><tr><td>5</td><td>LLM-generated reports</td><td>0.98</td><td>1</td><td>0.99</td><td>0.60</td></tr><tr><td colspan="6" style="text-align:center; font-weight:bold;">All frame elements - average</td></tr><tr><td>1</td><td>Technical reports</td><td>0.66</td><td>0.87</td><td>0.74</td><td>0.35</td></tr><tr><td>2</td><td>Corrected reports</td><td>0.61</td><td>0.85</td><td>0.67</td><td>0.29</td></tr><tr><td>3</td><td>FrWac</td><td>0.61</td><td>0.79</td><td>0.60</td><td>0.36</td></tr><tr><td>4</td><td>LLM-generated</td><td>0.92</td><td>0.95</td><td>0.93</td><td>0.65</td></tr><tr><td>5</td><td>LLM-generated reports</td><td>0.98</td><td>1</td><td>0.97</td><td>0.59</td></tr></tbody></table>

As shown in the results in Table 2, the annotation scores for the LLM-generated datasets are above 0.9 (except the recall of "Target" for the LLM-generated sentences). The segment overlap for these two datasets also shows good scores (around 0.6). The real data (technical reports and Web extracted data) show lower but still acceptable score (between 0.56 and 0.82 F-score depending on the frame element). However, the segment overlap is low, about 0.3 except for the frame element *Target* with 0.48, demonstrating a difficulty for the LLM to identify the segment. A qualitative analysis shows that the extracted segments are too long and, as such, not accurately corresponding to the frame element. For example, in the example Table 3, the extracted segment for the frame element *Action* includes the word of the gold (i.e. "passer"/"to pass"), so it is considered correct when computing the scores. However, the extracted segment is far too long for the answer to actually be considered as correct, with a segment overlap of 0.16. The LLM is not able to correctly delimit the span of the frame element, and thus to provide an exploitable result.
Automatic correction of the original reports does not improve the results, on the contrary all scores are decreased. Similarly, the segment overlap of the corrected reports also decreased by a few points compared to the original reports scores.

<table><caption>Table 3: Example of annotation output</caption><tr><td>Report</td><td colspan="3">LA PRESENCE DU DOME RLH2 EMPECHE DE PASSER LA PERCEUSE POUR CHASSER LES RIVETS POUR AMENAGER LES LOGEMENTS DES BUSES DE PASSAGE DE CLOISON CONE CASE. (tr. "The presence of the RLH2 dome prevents inserting the drill to drive out the rivets to fit the CONE CASE bulkheads nozzle housings."<sup>*</sup></td></tr><tr><td></td><th>Hindrance</th><th>Action</th><th>Target</th></tr><tr><td>Gold standard</td><td>DOME</td><td>PASSER</td><td>PERCEUSE</td></tr><tr><td>LLM labeling</td><td>la présence du dome RLH2</td><td>passer la perceuse pour chasser les rivets pour aménager les logements des bus de passage de cloison cone case</td><td>les bus de passage de cloison cone case</td></tr></table>

## 6 Conclusion

The outcome of this study indicates that the ability of an LLM to correctly annotate real-world data is limited. When the sentence structure differs from a simple, canonical one, the model does not know how to precisely identify and delimitate the target segment. Sentence structure seems to have an important impact in the LLMs performance. Wróblewska (2024) has shown the difficulty for an LLM to apprehend semantic information when confronted to noisy and non-canonical texts. After a closer qualitative analysis of our results, our hypothesis is that, when faced with complex structure sentences and noisy text, the model relies heavily on the structural elements (punctuation for example) to delimitate the segments. The presence of non-canonical structures and additional elements in the real-word and, especially, the technical data act as disrupting actors for the LLM.

In addition, the LLM shows erratic behaviors that tamper with the results such as answering in English despite the instructions to answer in French, modifying unknown words (for example, the acronym "FE" becomes "feu" (tr. "fire")) or using inflected variants of the extracted segments instead of keeping the original form. These issues are critical when considered in the broader context of a downstream task (such as using or feeding a knowledge base). This also raises questions about the definition of the segment extraction task and how it is evaluated. To our knowledge, calculating the coordinates of the target segments is a completely different task for a generative LLM.

To further explore this experiment, a variation on the prompt could of course be performed, especially indicating the span of the text to annotate. In this task, the span is reduced due to the target task: to identify the different elements of the problem. As such, this information could be provided in the prompt. More examples could also be provided in an attempt to vary the examples and to show the kind of annotation expected from the LLM.
## References

Baroni, M., Bernardini, S., Ferraresi, A., & Zanchetta, E. (2009). The WaCky Wide Web: A collection of very large linguistically processed web-crawled corpora. *Language Resources and Evaluation, 43*, 209-226. doi: 10.1007/s10579-009-9081-4

Brundage, M. P., Sexton, T., Hodkiewicz, M., Dima, A., & Lukens, S. (2021). Technical language processing: Unlocking maintenance knowledge. *Manufacturing Letters, 27*, 42-46. doi: 10.1016/j.mfglet.2020.11.001

Cui, X., & Swayamdipta, S. (2024). Annotating FrameNet via structure-conditioned language generation. In L.-W. Ku, A. Martins, & V. Srikumar (Eds.), *Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 2: Short papers)* (pp. 681–692). Bangkok, Thailand: Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-short.63

Dima, A., Lukens, S., Hodkiewicz, M., Sexton, T., & Brundage, M. P. (2021). Adapting natural language processing for technical text. *Applied AI Letters, 2*(3), e33. doi: 10.1002/ail2.33

Ettinger, A., Hwang, J., Pyatkin, V., Bhagavatula, C., & Choi, Y. (2023). “you are an expert linguistic annotator”: Limits of LLMs as analyzers of Abstract Meaning Representation. In H. Bouamor, J. Pino, & K. Bali (Eds.), *Findings of the association for computational linguistics: Emnlp 2023* (pp. 8250–8263). Singapore: Association for Computational Linguistics. doi: 10.18653/v1/2023.findings-emnlp.553

Fillmore, C. J., & Baker, C. (2009). 313 a frames approach to semantic analysis. In *The oxford handbook of linguistic analysis*. Oxford University Press. doi: 10.1093/oxfordhb/9780199544004.013.0013

Grattafiory, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., ... Ma, Z. (2024). *The llama 3 herd of models*. arXiv:2407.21783.

Mahowald, K., Ivanova, A. A., Blank, I. A., Kanwisher, N., Tenenbaum, J. B., & Fedorenko, E. (2024). Dissociating language and thought in large language models. *Trends in Cognitive Sciences, 28*(6), 517-540. doi: https://doi.org/10.1016/j.tics.2024.01.011

Ruppenhofer, J., Ellsworth, M., Petruck, M., Johnson, C., & Scheffczyk, J. (2016). *FrameNet II: Extended Theory and Practice*. http://framenet.icsi.berkeley.edu.

Thomas, A., Gaizauskas, R., & Lu, H. (2024). Leveraging LLMs for post-OCR correction of historical newspapers. In R. Sprugnoli & M. Passarotti (Eds.), *Proceedings of the third workshop on language technologies for historical and ancient languages (lt4hala) @ lrec-coling-2024* (pp. 116–121). Torino, Italia: ELRA and ICCL.

Torrent, T. T., Hoffmann, T., Almeida, A. L., & Turner, M. (2024). *Copilots for Linguists: AI, Constructions, and Frames*. Cambridge University Press.

Wróblewska, A. (2024). Investigating large language models for their competence in extracting grammatically sound sentences from transcribed noisy utterances. In *Proceedings of the 28th conference on computational natural language learning*
(pp. 10–23).

Zhang, J., Haverals, W., Naydan, M., & Kernighan, B. W. (2024). Post-OCR correction with openai's gpt models on challenging english prosody texts. In *Proceedings of the acm symposium on document engineering 2024*. New York, NY, USA: Association for Computing Machinery. doi: 10.1145/3685650.3685669