# Studying Model Design Biases in LLMs for Multilingual Historical Newspaper Extraction; The Messina Earthquake Case Study

Sarah Oberbichler, Johanna Mauermann, The Trung Tran, Carlos-Emiliano

González-Gallardo

► To cite this version:

Sarah Oberbichler, Johanna Mauermann, The Trung Tran, Carlos-Emiliano González-Gallardo. Studying Model Design Biases in LLMs for Multilingual Historical Newspaper Extraction; The Messina Earthquake Case Study. The 29th International Conference on Theory and Practice of Digital Libraries, Sep 2025, Tempere, Finland. hal-05166022

HAL Id: hal-05166022

https://univ-tours.hal.science/hal-05166022v1

Submitted on 16 Jul 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Studying Model Design Biases in LLMs for
Multilingual Historical Newspaper Extraction;
The Messina Earthquake Case Study

Sarah Oberbichler¹[0000-0002-1031-2759], Johanna
Mauermann²[0009-0000-4790-8196], The Trung Tran³[0009-0005-1261-950X], and
Carlos-Emiliano González-Gallardo⁴[0000-0002-0787-2990]

¹ Leibniz Institute of European History

oberbichler@ieg-mainz.de

² University of Mainz

jmauerma@students.uni-mainz.de

³ University of Science and Technology of Hanoi

trungtt.bi12-452@st.usth.edu.vn

⁴ Université de Tours

gonzalezgallardo@univ-tours.fr

**Abstract.** Large language models offer new opportunities for processing historical documents, yet their application raises questions of reliability. We present the first comprehensive and explainability-driven framework for evaluating model design bias in multilingual historical news article extraction, using newspaper coverage of the 1908 Messina earthquake as our test case across German, English, and French sources. Through systematic analysis of six state-of-the-art models, we uncover three critical bias patterns that, in addition to data quality, compromise extraction quality: contextual integration bias, overconfidence bias, and preference bias. Our evaluation reveals that these biases stem from alignment procedures rather than training data limitations, findings that establish methodological foundations for responsible AI deployment in digital humanities.

**Keywords:** article extraction · bias evaluation · historical newspapers.

# 1 Introduction

Critical assessment of historical sources, datasets, and tools is essential, as neither data nor computational methods are neutral [13, 2]. Bias can emerge at multiple stages—from archival curation to datafication and analytical tools—impacting representativity and quality [2, 3, 4]. These biases risk distorting interpretations, making systematic quality assessment crucial. As deep learning techniques of artificial intelligence (AI) become more integrated into historical research, evaluating both data and tools is necessary.

This study investigates the application of instruct models powered by large language models (LLMs)<fnref n="5" /> for multilingual historical newspaper article extrac-

<footnote n="5">⁵ All models used in this study are instruct models; however, we keep the acronym LLM in the text to facilitate the lecture.</footnote>
tion (NAE), using newspaper coverage of the 1908 Messina earthquake as a case study. We focus on LLMs because of their accessibility to a broader range of researchers in digital history and humanities compared to traditional information retrieval methods. While we assess model performance in NAE, our primary objective extends beyond output evaluation. We aim to establish systematic methods for understanding LLMs' behavior in historical content processing, focusing on bias detection and analysis via explainability. Research on explainability seeks to clarify the function of alignment fine-tuning, investigate the reasons behind hallucinations, and measure the uncertainty associated with predictions made by language models [28].

With NAE, we developed a novel approach to historical document processing by enabling targeted, topic-specific extraction of articles from OCR-processed newspaper pages. Our method explores semantic-context-based extraction, bypassing traditional reliance on layout analysis and visual cues. The novelty of this strategy introduced distinct methodological challenges, as no pre-existing frameworks, benchmark datasets, or performance metrics existed for evaluation. To address this gap, we developed new datasets and designed evaluation and explainability frameworks that should help to understand the potential as well as the limitations and biases of this approach (Figure 1). We made all our code and prompts accessible in a GitHub repository<fnref n="6" />.

<figure><img src="image_2.png" /><figcaption>Fig. 1: Quality assessment and bias evaluation for a novel approach on NAE.</figcaption></figure>

Drawing on [15], who defines bias as “a consistent pattern of error within a dataset, or within a method of data processing and analysis, that skews findings and interpretation,” we examine three critical bias dimensions: (1) data selection bias, (2) model algorithmic bias, (3) and training/alignment bias [8, 9].

While research on bias in AI predominantly focuses on social biases [1], our work addresses critical model design biases that can significantly distort outputs in historical document processing. Although social biases are inherent in historical sources, the successful extraction of original content from multilingual cultural heritage data depends on three fundamental technical components: (1) comprehensive representation of diverse language and domain sources, (2) sophisticated semantic and contextual understanding capabilities, and (3) precise alignment optimized for extraction tasks. By applying rigorous source-critical approaches and systematic bias evaluation methods, researchers

<footnote n="6">⁶ https://github.com/johannamauermann/LLM_Biases_Article_Extraction</footnote>
can identify limitations within these components and make informed decisions when deploying LLMs for historical document extraction. The reader is invited to consult our ethics statement in Appendix E.

# 2 Article Extraction and Classification Dataset

To conduct our study and analyze the behavior of different LLMs regarding biases, we first created a topic-specific evaluation dataset containing historical newspaper pages in German, English, and French, along with a manually curated ground truth covering the 1908 Messina earthquake in Italy.

## 2.1 The 1908 Messina Earthquake in Historical Newspapers

In the early hours of December 28, 1908, a massive earthquake struck Southern Italy, leaving widespread destruction in its wake. Messina, a city on the island of Sicily, was hit hardest by the earthquake, followed by a devastating tsunami and fires raging for days. Entire neighborhoods were leveled, and over half of the city's residents lost their lives. The devastation extended across the Strait of Messina to Reggio Calabria and nearby villages, with over 200,000 individuals killed, injured, or left without shelter [3, 1, 24, 19].

This disaster, the deadliest earthquake ever recorded in Europe, captured global attention through extensive news reports. This extensive coverage makes the Messina earthquake an excellent case study for building a multilingual dataset, but it also presents significant challenges. Articles reporting on the Messina earthquake are often exceptionally long, and the event's complexity demands substantial contextual knowledge in order to identify relevant articles.

## 2.2 Dataset Description

We focused on data from three prominent newspapers of the time: *The Dresden Daily* (English), *Le Petit Parisien* (French), and *Norddeutsche Allgemeine Zeitung* (German). We collected the data using APIs from the German Digital Library (DDB)<fnref n="7" /> and the French National Library (BNF)<fnref n="8" />. We utilized topic-specific keywords (*tremblement*, *messine*, *séisme*, *earthquake*, *messina*, *Erdbeben*, *Messina*) to extract newspaper issues that potentially contain relevant articles, and we retained newspaper issues from the day of the earthquake until the end of 1909. While it is important to note that the list of keywords is not exhaustive and that the dataset is not representative of the case study in rigorous historical terms, it should serve as a suitable foundation for evaluating the capabilities of LLMs in processing and understanding historical texts in different languages. We further manually created a ground truth (four eyes principle) as the benchmark for evaluation, which involved meticulously reviewing the newspaper issues and identifying and extracting relevant articles.

<footnote n="7">⁷ https://www.deutsche-digitale-bibliothek.de</footnote>

<footnote n="8">⁸ https://www.bnf.fr</footnote>
<figure><img src="image_3.png" /><figcaption>Fig. 2: Cumulative percentage and counts of context windows with relevant and irrelevant articles by language.</figcaption></figure>

The final dataset comprises 875 entries containing context windows (1000 characters before topic-specific keywords and 2000 after) drawn from 528 distinct newspaper issues. Figure 2 illustrates the proportion of context windows with relevant articles related to the 1908 Messina earthquake. The final dataset also encompasses multiple layers of information, including detailed metadata and human-annotated ground truth extractions. It is designed to support a three-faceted evaluation of LLMs: (1) the capacity to classify content as relevant or not relevant to a specific topic (e.g., the 1908 Messina earthquake), (2) the accuracy in extracting complete relevant articles from the broader newspaper text, and (3) the ability to correctly delineate the beginning and end of an article, especially when several articles appear within the same newspaper issue.

By providing human-annotated ground truth in combination with prompts, including explainability and human-in-the-loop approaches, our dataset and evaluation framework allow for systematic assessment of how well LLMs can understand historical text, maintain contextual relevance, and perform precise information extraction. This framework helps to evaluate LLMs' effectiveness in handling real-world historical document processing tasks and how model design bias can affect the quality of the results. The dataset is publicly available in HuggingFace<fnref n="9" />.

## 3 News Article Extraction (NAE) Approach and Evaluation Framework

Approaches to article separation in historical newspapers predominantly rely on layout information [26, 14] or a combination of layout and semantic understanding [14, 12, 22, 11]. While layout information is beneficial to delimit columns employing visual cues, it is also a bottleneck when facsimiles are unavailable or when they present poor quality. Our approach proposes a fundamentally different methodology leveraging purely textual content from OCR output, deliberately excluding layout information, to achieve efficient article extraction through

<footnote n="9">⁹ https://huggingface.co/datasets/oberbics/Multilingual_Topic-Specific_Article-Extraction_and_Classification</footnote>
prompt engineering. This text-centric methodology offers precise targeting of relevant content and enhanced processing efficiency by avoiding image analysis techniques. In addition, the flexibility of prompt-based article definition enables dynamic adaptation to varying scholarly requirements and historical contexts without necessitating modifications to the underlying extraction architecture. Given the novel nature of this semantic-only prompt-based approach to historical newspaper processing, no standardized evaluation frameworks exist.

Evaluation frameworks for layout-based article separation tasks traditionally rely on word counts or transcription metadata to compare model outputs with ground truth. However, when working with text-only sources, no positional metadata is available. Furthermore, traditional methods did not include explainability elements (as described in Section 3.1), which provide insights for identifying biases and risks. Consequently, we developed a comprehensive evaluation methodology to assess the efficacy of our text-based extraction paradigm, following a strategy grounded in explainability.

Explainability is fundamental when working with LLMs, given the need to understand the model's behavior in human-understandable terms [6]. This understanding enables the building of trust with the end user by clarifying the generation process in an understandable manner without the need for strong technical knowledge. Further, explainability is a critical tool for detecting and analyzing model design biases that might affect extraction quality and historical interpretation. Our explainability-based methodological strategy addresses a significant gap in historical document processing, establishing new protocols for evaluating prompt-based article extraction systems that can reveal both performance limitations and potential sources of bias.

## 3.1 Prompt Design and Parameters Settings

To systematically optimize our prompt engineering approach, we developed a controlled and resource-friendly test environment using a strategically sampled subset of the dataset described in Section 2. This subset comprised 15 newspaper pages from German, French, and English language sources, specifically curated to include boundary conditions such as articles with extreme length characteristics and structural discontinuities. Notably, we deliberately abstained from incorporating domain-specific knowledge about the 1908 Messina earthquake into the prompt design. This methodological choice allowed us to evaluate the model's autonomous contextual reasoning capabilities and historical event interpretation without explicit priming. Our iterative optimization process leveraged this sampled dataset to maximize development efficiency through (1) systematic performance analysis, where we conducted comprehensive testing of prompt variants against our sampled dataset, (2) failure mode analysis, where we performed detailed error analysis, and (3) iterative optimization of prompts until reaching convergence. This empirically driven approach ensured optimal prompt design while avoiding overfitting to the test corpus.

For our bias study, we selected instruction-based prompts as our simple prompting strategy and the chain-of-thought (CoT) [25] technique as our com-
plex prompting strategy<fnref n="10" />. By prompting the model to articulate its reasoning, we aimed to better understand the stage at which potential biases emerged. This allowed us to trace reasoning patterns more effectively and gain insights into the model’s behavior. We experimented with a simple user prompt in either English, French, and German (Simple_{EN|FR|DE}); and one complex user prompt (Complex_{EN|FR|DE}). For the German and French datasets, we applied the prompts in both the native language and English.

Concerning our baseline configuration, we set the temperature to *t* = 0.0 and seed to 42, to produce deterministic outputs by consistently choosing the highest probability token. We additionally implemented temperature *t* = 0.2 as a comparative condition to examine how introducing minimal randomness affects response quality and variability across all models. Although optimal parameter configuration typically varies between models, we deliberately applied identical settings across all systems to enable direct comparative analysis.

For the explainability component of our framework, we enhanced the prompts with three elements: (1) self-verification scoring mechanisms, (2) human verification notifications, and (3) model parameter adjustments to access underlying logits. An example of an LLM answer is shown in Example A.1 from Appendix A.

## 3.2 Article Comparison and Success Score

The article comparison employs a fuzzy string-matching algorithm designed to ensure that correct articles are accurately matched within a given context window when multiple articles are present. This process initiates with two primary inputs: ground truth articles and articles extracted using an LLM, both of which follow a preprocessing normalization stage to allow for consistent comparisons. The system applies character-level comparisons guided by a configurable similarity threshold, facilitating an effective bidirectional matching strategy that accommodates both one-to-many and many-to-one relationships between the ground truth and LLM outputs. This strategy also enables the identification of partial matches. During each comparison, the system utilizes sliding windows of *n* characters (with a default size of 50) to delineate matching boundaries between texts. This methodology assures a robust comparison, even when texts exhibit minor discrepancies or contain OCR artifacts, thereby allowing for flexible and precise matching between the source materials and the extracted content.

We considered several key parameters to calculate a *success score* (*SSCr*) ranging from 0 to 1 to quantify the alignment quality. The description of each parameter can be consulted in Table 1 from Appendix D. The *SSCr* is defined differently based on the alignment category:

- *SSCr* = 0 for **false positives** and **false negatives**, as these represent entirely incorrect alignments or missing matches;

- *SSCr* = 1 for **true negatives**, since these indicate correct non-matches;

<footnote n="10">¹⁰ Both prompt strategies can be found in the GitHub repository.</footnote>
- SSCr = textsim<sub>g,a</sub> for ground truth (g) and an extracted **aligned article** (a). The score is computed based on the degree of textual similarity between aligned segments, using the Equation 1.

The textual similarity $SSCr = textsim_{g,a}$ for aligned articles is set to 1 and is reduced based on the normalized Levenshtein distance $d_{lev}^{g,a}$, the proportion of omitted characters $d_{omit}^{g,a}$, and the proportion of added characters $d_{add}^{g,a}$. Two penalties, $\alpha$ and $\beta$, weight omitted and added characters, respectively. A tolerance threshold $\tau$ is applied to $d_{lev}^{g,a}$ to account for minor OCR-related adjustments. Additionally, a formatting error penalty $\delta$ is considered. The final text similarity score is computed as

$$textsim_{g,a} = \max(0, 1 - (\max(0, d_{lev}^{g,a} - \tau) + \alpha \cdot d_{omit}^{g,a} + \beta \cdot d_{add}^{g,a}) - \delta), \quad (1)$$

where $\tau = 0.1$ and $\delta = 0.2$ if a formatting error is present (otherwise $\delta = 0$). $\alpha$ and $\beta$ are set to 1.5 and 1 given that omitted characters are penalized more heavily, as even small omissions may significantly distort meaning.

This comprehensive scoring mechanism ensures that minor discrepancies do not result in disproportionate penalties, while more significant misalignments and omissions are accurately represented in the scoring. It allows for a nuanced evaluation that goes beyond a binary framework, facilitating the differentiation not only between correct and incorrect alignments but also among the aligned articles themselves, thereby highlighting cases where content was omitted, added, or otherwise inadequately extracted.

# 4 Model Performance Analysis for Bias Detection

Detecting and measuring bias in LLMs involves combining quantitative and qualitative analyses [20]. By following evaluation approaches that can reveal biases from different origins as outlined in Table 2 from Appendix D, we compare model evaluation metrics against close reading to get a deeper understanding of the limitations that skew model outputs. We distinguish between data quality issues (such as OCR errors) that represent external factors degrading performance across all models, and model bias that manifests as systematic patterns persisting independent of input quality—such as consistently exhibiting overconfidence.

We evaluate six models representing different paradigms: three open-weight models (Llama 3 70B from Meta, Nemotron 3.1 70B from Nvidia, and Qwen 2.5 72B from Alibaba Cloud) accessed via Together.ai<fnref n="11" />, and three proprietary models (GPT-4o<fnref n="12" />, Claude 3.5 Sonnet<fnref n="13" />, and DeepSeek V3¹⁴) accessed through their respective APIs. The models were evaluated based on their ability to accurately identify content, align text correctly, and manage language-specific variations.

<footnote n="11">¹¹ https://api.together.ai</footnote>

<footnote n="12">¹² https://openai.com/api/</footnote>

<footnote n="13">¹³ https://www.anthropic.com/api</footnote>

<footnote n="14">¹⁴ https://api-docs.deepseek.com/api/deepseek-api</footnote>
## 4.1 Temperature Response Performance

We established a default temperature of $t = 0.0$ for model evaluation. To assess the models' sensitivity to sampling parameters, we conducted supplementary experiments with the temperature adjusted to $t = 0.2$. Both temperature settings fall within the low range [0, 0.3] to facilitate an in-depth examination of the stability of each model's internal representations and decision boundaries.

<figure><img src="image_4.png" /><figcaption>Fig. 3: Proportion of found relevant vs irrelevant articles with model temperature set to 0.0 and 0.2 aggregated across all languages.</figcaption></figure>

As seen in Figure 3, increasing the temperature parameter from 0.0 to 0.2 reveals significant alignment challenges across models, particularly in maintaining balanced performance between content correctly marked as irrelevant (precision) and relevant articles fully or partially found (recall). Deepseek V3 exhibits the most sensitivity, with its recall for relevant articles dramatically improving (from overall 0.63 to overall 0.96) while its precision in correctly identifying irrelevant content decreased to around 0.80 for most prompts (with an outlier for the complex English prompt). In contrast, Claude 3.5 gets overall slightly better results in correctly marking irrelevant articles with the temperature set to 0.2, and Nemotron 3.1 shows overall better results with the temperature set to 0.2. GPT-4o follows a similar pattern as Deepseek V3 towards better finding relevant articles with the higher temperature, however, just like Claude 3.5 and Qwen 2.5, it consistently prioritizes precision over recall across temperature settings. The different responses to temperature changes across models reveal fundamental differences in how these models handle uncertainty and make decisions.
Qualitative error pattern evaluation provides insights into the decision boundaries and the models' approach to uncertainty. With temperature *t* = 0.0, some models strongly focus on primary criteria: place (Messina), date (28 of December), and earthquake context, while ignoring locations related to the Messina earthquake but not Messina itself and disregarding aftershocks (despite explicit instructions in the prompts to include aftershock information). The too narrow focus also limited what qualified as an article (while we kept the definition broad on purpose). We show concrete examples in Examples A.2 and A.3 from Appendix A.

The error pattern analysis also showed that, despite demonstrating strong historical knowledge and geographic understanding in isolation, models can fail to synthesize these separate knowledge domains to reach accurate conclusions, especially when the temperature is set to *t* = 0.0. Nevertheless, setting a temperature *t* ≠ 0.0 implies variability in the generated text, which limits reproducibility and makes evaluation difficult.

## 4.2 Language-specific Performance

To analyze language-specific performance, we calculated total average success scores (*SSCr*) and conducted cross-lingual comparisons. Our evaluation, shown in Figure 4, revealed notable variation across languages. French achieved the highest overall performance with an average accuracy of 0.78, followed by English (0.72) and German (0.70).

<figure><img src="image_5.png" /><figcaption>Fig. 4: Total average model accuracy (percentage of correct predictions) for the French, German, and English datasets.</figcaption></figure>

However, these results do not necessarily indicate substantial superiority among source languages in the article extraction task. A more nuanced interpretation requires examining the composition of the dataset (Figure 2), the differential performance of the models between correctly classifying irrelevant content versus correctly identifying relevant articles (Figures 3a and 3b), and OCR impact (Figure 13b). The apparently higher performance metrics in French can be attributed mainly to the distinct distribution of relevant versus irrelevant articles across these language corpora. All evaluated models with temperature
$t = 0.0$ demonstrated stronger capabilities in correctly identifying irrelevant content than in detecting relevant articles. Consequently, languages with different proportions of irrelevant examples, as seen in the French and, to a lesser extent, in the German datasets, naturally yield different overall performance scores without necessarily reflecting superior language processing capacities. Additionally, the French dataset contained more OCR errors than other language datasets, negatively affecting model accuracy (see Section 5 and Appendix B).

Similarly, the language of the prompts had minimal impact on model performance. For example, for the French dataset, English prompts scored 0.75 versus 0.78 for native French prompts. On the German dataset, the difference between German and English prompts was only 0.03 as well.

## 4.3 Instruction Following Performance

Analysis of model output consistency further revealed variations in instruction adherence across the evaluated models. We evaluated how well the models maintain the requested output format (extracting original articles without summarizing or truncating), ensuring content completeness, and preserving original text. This analysis is a combination of quantitative and qualitative approaches. As a quantitative approach, we measured the article under-extraction frequency where models captured only portions of relevant content (Figure 5), and structural parsing errors, where models failed to create separate XML tags for distinct articles within a context window (Figure 6).

<figure><img src="image_6.png" /><figcaption>Fig. 5: Proportion of under-extracted articles across models and aggregated across all languages.</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>Fig. 6: Proportion of incorrectly parsed articles across models and aggregated across all languages.</figcaption></figure>

Our qualitative analysis of results with partial article extractions revealed distinct error patterns across models. Despite their differing capacities, both the mid-size Llama 3 and the substantially more capable Claude 3.5 frequently failed to process very long articles by either claiming they exceeded length limitations
or simply appending phrases like “full content truncated for brevity” and truncating the content. These models disregarded explicit instructions against such behavior, which consequently lowered their performance on NAE.

Error patterns also varied in the article parsing. While Claude 3.5 demonstrated good parsing accuracy with minimal structural errors, GPT-4o exhibited considerable parsing difficulties when using simple prompts. Also noteworthy is Nemotron 3.1's systematic formatting deviation, not only in parsing articles, but also favoring markdown formatting conventions, mixing markdown and XML.

## 4.4 Confidence Elicitation and Logits Correlation

LLMs tend to be overconfident in their output and struggle to identify the factual knowledge boundary precisely [21]. For these reasons, it is relevant to quantify the confidence of the model during the generation process. Different techniques oriented to measure the fluctuations of the model's weights can be used in white-box environments in which access to the architecture and weights of the models is possible. However, when working in a black-box environment, model confidence must be assessed through alternative means: either by analyzing the model's prediction logits (when available) or through confidence elicitation techniques. Confidence elicitation is the process of extracting LLMs' uncertainty judgments through specific prompting strategies, without requiring model fine-tuning or accessing internal information [27]. We implement a dual approach where we analyze prediction logits to quantify statistical confidence while also employing elicitation by explicitly asking models to indicate when human verification is needed and requesting self-confidence scores for their responses.

<figure><img src="image_8.png" /><figcaption>Fig. 7: Proportion of predictions where models requested human verification aggregated across all languages.</figcaption></figure>

Analysis of human verification needed patterns, as shown in Figure 7, reveals distinct behavioral characteristics across models and prompts. Among all models tested, GPT-4o and Claude 3.5 show the least frequency of requesting human verification, indicating model alignment with low uncertainty thresholds for decision-making. Llama 3 and Qwen 2.5 show opposite behavior, demonstrating an unnecessarily high tendency toward verification and uncertainty expression.
Nemotron 3.1 and Deepseek V3 have shown the best balance in communicating uncertainty. Apart from analyzing responses using the `<human_verification_needed>` label, we also ask the model to rate its answers through a `<confidence_score>` ranging from 0 to 1. This score provides a more precise, quantitative measure of the model's confidence estimation, enabling deeper analysis of its certainty in various scenarios. Figures 8 and 9 illustrate the mean and standard deviation of the self-assessed confidence scores of each model for combined German, English, and French datasets. Claude 3.5 consistently demonstrates the highest self-confidence scores with a mean value of 0.93 ± 0.06. GPT-4o revealed a slightly lower but more stable mean confidence score of 0.86 ± 0.04. Llama 3 obtains a mean value of 0.73 ± 0.22, showing the lowest confidence score, while the most significant fluctuations. Figures 14a and 14b in Appendix B show the detailed values for each language.

<figure><img src="image_9.png" /><figcaption>Fig. 8: Mean self-confidence scores across models.</figcaption></figure>

<figure><img src="image_10.png" /><figcaption>Fig. 9: Standard deviation of self-confidence scores.</figcaption></figure>

Upon these analyses, we could also make a direct comparison between these scores and the logits from the last layer of the LLMs themselves to get more insights into the differences between the self-evaluated scores and the probability value for each token in the generated answer. Figures 10 and 11 show the mean value and standard deviation for the token probability of each model for the combined German, English, and French datasets. Claude 3.5 is excluded from this analysis, given that its API does not provide logits or full probabilities. Overall, the token probability is much more consistent compared to the confidence score, which is understandable given that it reflects the probability of each individual token with a temperature $t = 0.0$. Most tokens usually exhibit near-perfect probabilities, leading to lower variance in this metric. DeepSeek V3 stands out with a perfect mean token probability and no variability, which is possibly due to the use of a greedy decoding algorithm for the final layer of this model. Further experiments with temperature $t = 0.2$ reflected variations in the mean value token probability similar to GPT-4o, which shows the lowest and most variable token probability. This is opposite to what we have analyzed above on the confidence score and could be due to a more diverse and probabilistic token generation from GPT-4o. Other models show quite similar values, indicating a balanced behavior between confidence and variability. Figures 15a and 15b in Appendix B show the detailed values for each language.
<figure><img src="image_11.png" /><figcaption>Fig. 10: Mean value of token probability with different setups.</figcaption></figure>

<figure><img src="image_12.png" /><figcaption>Fig. 11: Standard deviation value of token probability with different setups.</figcaption></figure>

To more effectively visualize the relationship between confidence scores and token probabilities, we computed their linear correlation. However, as our analysis suggests, using all tokens may not be ideal, as it often leads to redundant or uniform results. Therefore, we selected more distinctive tokens within each response that better reflect the model's confidence. Specifically, we utilized the token True/False from the human_verifyation_needed tag. When this token is True, indicating that the model lacks complete confidence and human verification may be required, we computed the adjusted probability as 1 - probability, since a high token probability for True signifies uncertainty rather than confidence.

<figure><img src="image_13.png" /><figcaption>Fig. 12: Correlation between confidence score and Probability in human verification token across languages.</figcaption></figure>

The resulting correlations are presented in Figure 12. GPT-4o and Llama 3 both exhibit strong positive correlations under English complex prompts, suggesting that their confidence scores reliably reflect output quality in these contexts. In contrast, GPT-4o shows near-zero or even negative correlations for simple prompts. Similarly, Qwen 2.5 demonstrates positive correlations with complex prompts but weaker or negative correlations with simpler ones. Lastly, DeepSeek V3 and Nemotron 3.1 maintain moderately positive correlations across most scenarios. Overall, model confidence scores tend to be more reliable indicators of output quality under complex prompting conditions, while this reliability diminishes for simpler prompts.
5 Impact of OCR Errors

To systematically evaluate the impact of OCR transcription errors, we created a control ground truth sample comprising 88 manually corrected context transcriptions (30 in French, 29 in German, and 29 in English) with a similar distribution of relevant and irrelevant content as the original dataset. Overall, the mean Character Error Rate (CER) across this control sample was 0.1388, indicating a substantial level of OCR-induced noise in the original data. However, the CER varied by language: French exhibited the highest CER (0.2008), followed by German (0.1377), while English had the lowest (0.0759).

<figure><img src="image_14.png" /><figcaption>Fig. 13: Percentage of found relevant vs irrelevant articles on control sample aggregated across all languages.</figcaption></figure>

Our quantitative analysis highlights the significant impact of OCR quality on LLM output. As illustrated in Figure 13, the average model accuracy on the control sample notably improved following OCR correction. The results indicate a strong causal relationship, with all models exhibiting consistent performance trends across both the control sample and the full dataset. After OCR correction, models demonstrated markedly superior performance in identifying relevant articles. The open-weight models Llama 3, Nemotron 3.1, and Qwen 2.5 showed remarkable improvements, while their accuracy in marking articles as irrelevant remained consistent. GPT-4o and Claude 3.5, on the other hand, did not indicate significant OCR-related changes in performance. These findings underscore the importance of OCR postprocessing. Further details can be consulted in Figures 16 to 18 from Appendix C.
# 6 Overall Discussion and Conclusion

*Temperature variations expose contextual integration bias in LLMs.* Our comparison between temperature settings reveals how models respond to parameter changes. With temperature set to *t* = 0.0, we observed some models exhibiting stronger contextual integration bias for articles lacking explicit identifying markers compared to temperature *t* = 0.2, resulting in failure to integrate related contextual information. This indicates that deterministic processing hampers models' ability to synthesize contextual relationships. This limitation further suggests that incorporating richer historical domain knowledge or knowledge graphs [18] might enable models to better synthesize temporal, cultural, and causal relationships into a cohesive narrative.

*Overall results show that all models deal well with multilingual sources and prompts.* While it is often assumed that the higher representation of English in LLM training data would lead to superior performance, our findings did not reveal significant differences across the evaluated language datasets (English, German, and French), the models used, or the language of the prompt, despite observing different accuracy levels for each language dataset. Considering data composition, better results for precision, as well as the impact of OCR errors, these differences become statistically insignificant. This suggests that German and French are sufficiently well-represented in the training data.

*Bias through format sensitivity and built-in safety constraints has a notable impact on the quality of the results.* Our analysis indicates that some models exhibit a preference toward truncation, summarization, or rigid output formatting despite explicit prompting—likely as a result of built-in safety constraints or format-sensitive decoding mechanisms during the reinforcement learning with human feedback fine-tuning. While these features are designed to prevent the generation of inappropriate content and maintain stylistic consistency, they can inadvertently lead to the loss of critical contextual details. This behavior diminishes the models' capacity for information extraction, especially when creating historical research corpora.

*Overconfidence bias limits effective human oversight.* Our evaluation revealed that overconfidence bias in certain LLMs significantly limits the opportunity for human intervention. When a model exhibits high confidence in its outputs—even in ambiguous cases—it is less likely to signal uncertainty or request human verification. This overconfidence hampers effective human involvement, as users receive outputs without the necessary prompts to critically review or correct errors. Our analysis of the correlation between token probability and self-verification further supports findings that over-confidence is mainly introduced by alignment procedures, rather than being an inherent feature of the pre-trained base model [16].

**Disclosure of Interests.** The authors have no competing interests to declare that are relevant to the content of this article.
Bibliography

[1] Argnani, A., Alessandro Pino, N.: The 1908 messina straits earthquake: Cornerstones and the need to step forward. Seismological Research Letters 94(2A), 557-561 (2023). https://doi.org/10.1785/0220220355

[2] Beelen, K., Lawrence, J., Wilson, D.C.S., Beavan, D.: Bias and representativeness in digitized newspaper collections: Introducing the environmental scan. Digital Scholarship in the Humanities 38(1), 1-22 (2022). https://doi.org/10.1093/llc/fqac037

[3] Carcione, J., Kozák, J.: The messina-reggio earthquake of december 28, 1908. Studia Geophysica et Geodaetica 52, 661-672 (2008). https://doi.org/10.1007/s11200-008-0043-x

[4] Carter, R.G.S.: Of things said and unsaid: Power, archival silences, and power in silence. Archivaria 61, 215-233 (2006), https://archivaria.ca/index.php/archivaria/article/view/12541

[5] Dai, S., Xu, C., Xu, S., Pang, L., Dong, Z., Xu, J.: Bias and unfairness in information retrieval systems: New challenges in the llm era. In: Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. p. 6437-6447. KDD '24, Association for Computing Machinery, New York, USA (2024). https://doi.org/10.1145/3637528.3671458

[6] Doshi-Velez, F., Kim, B.: Towards a rigorous science of interpretable machine learning (2017), https://arxiv.org/abs/1702.08608

[7] European Union Agency for Fundamental Rights: Bias in algorithms – Artificial intelligence and discrimination. Publications Office of the European Union (2022). https://doi.org/doi/10.2811/25847

[8] Ferrara, E.: Fairness and bias in artificial intelligence: A brief survey of sources, impacts, and mitigation strategies. Sci 6(1) (2024). https://doi.org/10.3390/sci6010003

[9] Ferrer, X., Nuenen, T.v., Such, J.M., Cote, M., Criado, N.: Bias and discrimination in ai: A cross-disciplinary perspective. IEEE Technology and Society Magazine 40(2), 72-80 (2021). https://doi.org/10.1109/mts.2021.3056293

[10] Gallegos, I.O., Rossi, R.A., Barrow, J., Tanjim, M.M., Kim, S., Dernon-court, F., Yu, T., Zhang, R., Ahmed, N.K.: Bias and fairness in large language models: A survey. Computational Linguistics 50(3), 1097-1179 (2024). https://doi.org/10.1162/coli_a_00524

[11] Girdhar, N., Coustaty, M., Doucet, A.: Benchmarking nas for article separation in historical newspapers. In: Leveraging Generative Intelligence in Digital Libraries: Towards Human-Machine Collaboration. p. 76-88. Springer-Verlag, Berlin, Heidelberg (2023). https://doi.org/10.1007/978-981-99-8085-7_7

[12] Gu, Z., Meng, C., Wang, K., Lan, J., Wang, W., Gu, M., Zhang, L.: Xylayoutlm: Towards layout-aware multimodal networks for visually-rich
document understanding. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022. pp. 4573-4582. IEEE (2022). https://doi.org/10.1109/CVPR52688.2022.00454

[13] Herrmann, J.B., Bories, A.S., Frontini, F., Jacquot, C., Pielström, S., Rebora, S., Rockwell, G., Sinclair, S.: Tool criticism in practice. on methods, tools and aims of computational literary studies. Digital Humanities Quarterly 17(2) (2023)

[14] Huang, Y., Lv, T., Cui, L., Lu, Y., Wei, F.: Layoutlmv3: Pre-training for document ai with unified text and image masking. In: Proceedings of the 30th ACM International Conference on Multimedia. p. 4083-4091. MM '22, Association for Computing Machinery, New York (2022). https://doi.org/10.1145/3503161.3548112

[15] Kitchin, R.: Critical Data Studies: An A to Z Guide to Concepts and Methods. John Wiley & Sons (2024)

[16] Leng, J., Huang, C., Zhu, B., Huang, J.: Taming overconfidence in llms: Reward calibration in rlhf (2025), https://arxiv.org/abs/2410.09724

[17] Navigli, R., Conia, S., Ross, B.: Biases in large language models: Origins, inventory, and discussion. J. Data and Information Quality 15(2) (Jun 2023). https://doi.org/10.1145/3597307

[18] Pan, S., Luo, L., Wang, Y., Chen, C., Wang, J., Wu, X.: Unifying large language models and knowledge graphs: A roadmap. IEEE Transactions on Knowledge and Data Engineering 36(7), 3580-3599 (Jul 2024). https://doi.org/10.1109/tkde.2024.3352100, http://dx.doi.org/10.1109/TKDE.2024.3352100

[19] Parrinello, G.: Chapter 1. The 1908 Messina Earthquake. In: Earthquakes and Urbanism in Modern Italy, pp. 21-47. Berghahn Books, New York, Oxford (2015). https://doi.org/doi:10.1515/9781782389514-005

[20] Ranjan, R., Gupta, S., Singh, S.N.: A comprehensive survey of bias in llms: Current landscape and future directions (2024), https://arxiv.org/abs/2409.16430

[21] Ren, R., Wang, Y., Qu, Y., Zhao, W.X., Liu, J., Wu, H., Wen, J.R., Wang, H.: Investigating the factual knowledge boundary of large language models with retrieval augmentation. In: Rambow, O., Wanner, L., Apidianaki, M., Al-Khalifa, H., Eugenio, B.D., Schockaert, S. (eds.) Proceedings of the 31st International Conference on Computational Linguistics. pp. 3697-3715. Association for Computational Linguistics, Abu Dhabi, UAE (Jan 2025), https://aclanthology.org/2025.coling-main.250/

[22] Sun, W., Tran, H.T.H., González-Gallardo, C.E., Coustaty, M., Doucet, A.: LIAS: Layout Information-Based Article Separation in Historical Newspapers. In: Antonacopoulos, A., Hinze, A., Piwowarski, B., Coustaty, M., Di Nunzio, G.M., Gelati, F., Vanderschantz, N. (eds.) Linking Theory and Practice of Digital Libraries. Lecture Notes in Computer Science, vol. 15177, pp. 256-272. Springer (2024). https://doi.org/10.1007/978-3-031-72437-4_15
[23] Traub, M.C.: Measuring Tool Bias & Improving Data Quality for Digital Humanities Research. Ph.D. thesis, Utrecht University (2020)

[24] Vermiglio, C., Lombardi, R., Zarone, V.: The 1908 earthquake of messina: An accounting perspective on the city's reconstruction between faith and fascism. Accounting History 29(1), 15–39 (2024). https://doi.org/10.1177/10323732221141412

[25] Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E.H., Le, Q.V., Zhou, D.: Chain-of-thought prompting elicits reasoning in large language models. In: Proceedings of the 36th International Conference on Neural Information Processing Systems. NIPS '22, Curran Associates Inc., Red Hook, NY, USA (2022)

[26] Weidemann, M., Michael, J., Laasch, B.: Citlabrostock/citlab-article-separation: Article separation (Oct 2019). https://doi.org/10.5281/zenodo.3520703, https://doi.org/10.5281/zenodo.3520703

[27] Xiong, M., Hu, Z., Lu, X., Li, Y., Fu, J., He, J., Hooi, B.: Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs (Oct 2023), https://openreview.net/forum?id=gjeQKFxFpZ

[28] Zhao, H., Chen, H., Yang, F., Liu, N., Deng, H., Cai, H., Wang, S., Yin, D., Du, M.: Explainability for Large Language Models: A Survey. ACM Transactions on Intelligent Systems and Technology 15(2), 1–38 (2024). https://doi.org/10.1145/3639372
# A Model Output Examples

## A.1 Example Model Output (Deepseek V3, Complex English Prompt, English Dataset, Temperature 0.0)

### <think>

The first part of the text discusses Denmark's national security and defenses, which is irrelevant to the Messina earthquake. The second part reports shipping disasters in New Zealand and Brest, which are also irrelevant. The third part, "TERRIBLE WEATHER IN CALABRIA," mentions Reggio, a town affected by the Messina earthquake, and describes a violent storm and an earthquake shock. This is relevant as it discusses the consequences of the earthquake. The fourth part mentions an official decree regarding duty-free access for goods destined for earthquake sufferers, which is also relevant as it pertains to aid for the victims of the Messina earthquake. Both relevant parts are complete semantic units and are clearly distinguished from the preceding and following content. Confidence is high (0.9) because the relevant articles are clearly identifiable and complete. Human verification is not needed as the segmentation and relevance assessment are clear. <think>

### <article> TERRIBLE WEATHER IN CALABRIA. Reggio, February 12. A violent storm accompanied by showers of hail and heavy rain swept the ruined town last night, while this morning an earthquake shock was felt. <article>

### <article> Rome, February 12. An official decree announces that all consign-

ments of goods from abroad destined for sufferers from the earthquake, including provisions, transportable houses, and materials for building, will be allowed ac- cess duty free until June 30th. <article>

### <verification> Is the unit coherent? Yes. Is the subject present? Yes. Is the article complete? Yes. Have all articles been found? Yes. <verification>

### <human_verify needed> False <human_verify needed>

### <confidence> 0.9 <confidence>

### <confidenceExplanation> The relevant articles are clearly identifiable, complete, and directly related to the Messina earthquake and its consequences. <confidenceExplanation>
## A.2 Example 1: Uncertainty About Right Location

This example illustrates how the model exhibits uncertainty when classifying an earthquake event as relevant or not (e.g., mention of damage in Calabria as consequence of the Messina earthquake December 1908).

**Ground Truth Article**

EARTHQUAKE IN ITALY. Monte leone, December 28. Extensive damage was done here and at Mileto, Cantazaro, San Giorgio, and other Calabrian districts by a violent earthquake which occured at 5.50 o'clock this morning. [...]

**Deepseek V3 Answer (complex-EN, temperature 0.0)**

High confidence (1.0) that this article is not about the Messina earthquake, as the date and location are clearly different. No ambiguity exists in this case.

**Deepseek V3 Answer (complex-EN, temperature 0.2)**

The second section titled "EARTHQUAKE IN ITALY" explicitly reports on a December 28 earthquake affecting Calabrian districts (Monteleone, Mileto, Cantazaro, San Giorgio)[...]. It is a direct mention of Calabrian districts impacted by the 1908 Messina earthquake.

**Nemotron 3.1 (complex-EN, temperature 0.0)**

The confidence is relatively low because, although an article about an earthquake in Italy was found, it does not specifically mention the Messina earthquake of December 1908 or its consequences, which are the primary criteria for relevance. The article's relevance to the specified topic is partial at best, focusing on a different earthquake in Calabria.

**Nemotron 3.1 (complex-EN, temperature 0.2)**

The article about the earthquake in Italy is relevant, but it does not specifically mention the Messina earthquake of December 1908. However, since it is an earthquake in Italy and occurred in December, it might be related to the Messina earthquake. Further verification is not needed in this case.

**Claude 3.5 Answer (simple-EN, temperature 0.0)**

High confidence that this is not the 1908 Messina earthquake because: 1) Different locations are mentioned (none include Messina), 2) The scale of damage reported is much smaller than the catastrophic Messina earthquake, 3) While the month is December, no year is specified, but the context and reporting style suggest this is a different even

**Claude 3.5 Answer (simple-EN, temperature 0.2)**

While this article does discuss an earthquake in Italy in December 1908, it appears to be reporting on a different seismic event that occurred in Calabria [...]. The Messina earthquake affected Sicily and Calabria and was much more devastating, killing over 80,000 people.
### A.3 Example 2: Classification Uncertainty

This example illustrates how the models exhibit uncertainty when determining whether a text qualifies as an article according to our criteria (own semantic unit).

**Ground Truth Article**

Lieutenant-Commander Reginald Belknap, American Naval Attache in Berlin, is temporarily on active duty in Italy. After accompanying a section of the American Fleet to Gibraltar, he was put in charge of the thousands of shelter-sheds erected in Italy for the protection of those rendered homeless by the late terrible earthquake at Messina.

**Deepseek V3 Answer (complex-EN, Temperature 0.0)**

The provided text does not contain any clearly distinguished semantic units with their own headlines. It appears to be a collection of unrelated snippets or news items without clear segmentation into articles.

**Deepseek V3 Answer (complex-EN, Temperature 0.2)**

The semantic unit is coherent and complete and there is an explicit mention of Messina earthquake consequences.

**Claude 3.5 Answer (Complex-EN, Temperature 0.0)**

The text appears to be a society/social news column from Berlin, with only a brief mention of Lieutenant-Commander Reginald Belknap's involvement in shelter construction after the Messina earthquake. This mention is too brief and incidental to constitute a relevant article about the Messina earthquake according to the given criteria.

**Claude 3.5 Answer (simple-EN, Temperature 0.2)**

I found one brief mention of the 1908 Messina earthquake within a larger article, but it's only a passing reference in the context of reporting on a naval officer's activities. Here is the relevant excerpt in XML format:

**Nemotron 3.1 (complex-EN, Temperature 0.0)**

High confidence due to the clear mention of the Messina earthquake and its consequences (homelessness and shelter-sheds), making the article's relevance unequivocal. The article's coherence and completeness are also evident, as it forms a self-contained unit within the provided text.

**Nemotron 3.1 (complex-EN, Temperature 0.2)**

The paragraph forms a coherent unit discussing Lieutenant-Commander Reginald Belknap's temporary duty in Italy, specifically in relation to the aftermath of the Messina earthquake.
## B Confidence Score and Token Probability

<figure><img src="image_15.png" /><figcaption>Fig. 14: Mean and standard deviation confidence score across languages</figcaption></figure>

<figure><img src="image_16.png" /><figcaption>Fig. 15: Mean and standard deviation of token probability across languages</figcaption></figure>
## C OCR impact

<figure><img src="image_17.png" /><figcaption>Fig. 16: Total average model accuracy (percentage of correct predictions) across French, German, and English control samples.</figcaption></figure>

<figure><img src="image_18.png" /><figcaption>Fig. 17: Mean confidence score on control sample.</figcaption></figure>
<figure><img src="image_19.png" /></figure>

(a) control sample with OCR errors

<figure><img src="image_20.png" /></figure>

(b) corrected control sample

<caption>Fig. 18: Mean token probability on control sample.</caption>

## D Parameter Setting, and Evaluation Approaches

<table><caption>Table 1: Parameters of the article boundary comparison and quality assessment frame-work, showing the key components used in the evaluation process.</caption><thead><tr><th>Component</th><th>Parameter</th><th>Description</th></tr></thead><tbody><tr><td rowspan="4">Article Boundary Comparison</td><td>Window Size</td><td>50 characters for start/end detection</td></tr><tr><td>Similarity Threshold</td><td>65% minimum match requirement</td></tr><tr><td>Fuzzy Matching</td><td>Character-level comparison for boundaries</td></tr><tr><td>Bidirectional Search</td><td>Handles one-to-many and many-to-one matches</td></tr><tr><td rowspan="4">Quality Metrics</td><td>OCR Tolerance</td><td>0.1 threshold for character variations</td></tr><tr><td>Omission Weight</td><td>1.5x penalty for missing characters</td></tr><tr><td>Format Error</td><td>20% reduction in success score</td></tr><tr><td>Levenshtein Distance</td><td>Normalized character-level differences</td></tr><tr><td rowspan="3">Error Thresholds</td><td>Over-Extraction</td><td>&gt;4% added characters</td></tr><tr><td>Under-Extraction</td><td>&gt;4% omitted characters</td></tr><tr><td>Content Mismatch</td><td>&gt;20% Levenshtein distance</td></tr><tr><td rowspan="4">Classification</td><td>True Positive</td><td>No significant errors</td></tr><tr><td>False Positive</td><td>Incorrect match detection</td></tr><tr><td>False Negative</td><td>Missed match detection</td></tr><tr><td>True Negative</td><td>Correct null detection</td></tr></tbody></table>
<table><caption>Table 2: Evaluation approaches for measuring LLM article extraction performance and associated biases.</caption><thead><tr><td>Evaluation</td><td>Quantitative</td><td>Qualitative</td><td>Source of Potential Bias</td></tr></thead><tbody><tr><td>Temperature response</td><td>Comparing extraction accuracy at temperature 0.0 vs 0.2</td><td>Error pattern analysis</td><td>Model architecture and preference alignment</td></tr><tr><td>Language performance</td><td>Cross-lingual accuracy comparison</td><td></td><td>Training data</td></tr><tr><td>Instruction following</td><td>Format adherence rates and instruction compliance metrics</td><td>Error pattern analysis</td><td>Supervised training</td></tr><tr><td>Confidence elicitation and human verification</td><td>Correlation between model prediction logits, self-expressed confidence, and actual performance</td><td>Analysis of confidence explanation</td><td>Preference alignment</td></tr></tbody></table>

## E Ethics Statement

This study examines biases in LLMs applied to historical text processing, emphasizing the risks of misrepresentation, overconfidence, and contextual misalignment. While our findings highlight challenges in LLM reasoning, they also inform strategies for bias mitigation and responsible AI use in digital humanities. We ensure transparency by making our dataset and evaluation framework publicly available. No personally identifiable or sensitive data was used in this research.