# Annotating Scientific Uncertainty: A comprehensive model using linguistic patterns and comparison with existing approaches

Panggih Kusuma Ningrum, Philipp Mayr, Nina Smirnova, Iana Atanassova

► To cite this version:

Panggih Kusuma Ningrum, Philipp Mayr, Nina Smirnova, Iana Atanassova. Annotating Scientific Uncertainty: A comprehensive model using linguistic patterns and comparison with existing approaches. Journal of Infometrics, 2025, 19 (2), 10.1016/j.joi.2025.101661. hal-05043196

HAL Id: hal-05043196

https://hal.science/hal-05043196v1

Submitted on 23 Apr 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# Annotating Scientific Uncertainty: A comprehensive model using linguistic patterns and comparison with existing approaches

Panggih Kusuma Ningrum*<fnref n="1" />, Philipp Mayr², Nina Smirnova², and Iana Atanassova<fnref n="1" />,³

¹Université Marie et Louis Pasteur, CRIT, F-25000 Besançon, France

²GESIS — Leibniz Institute for the Social Sciences, Cologne, Germany

³Institut Universitaire de France (IUF), France

## Abstract

We present UnScientify<fnref n="1" />, a system designed to detect scientific uncertainty in scholarly full text. The system utilizes a weakly supervised technique to identify verbally expressed uncertainty in scientific texts and their authorial references. The core methodology of UnScientify is based on a multi-faceted pipeline that integrates span pattern matching, complex sentence analysis and author reference checking. This approach streamlines the labeling and annotation processes essential for identifying scientific uncertainty, covering a variety of uncertainty expression types to support diverse applications including information retrieval, text mining and scientific document processing. The evaluation results highlight the trade-offs between modern large language models (LLMs) and the UnScientify system. UnScientify, which employs more traditional techniques, achieved superior performance in the scientific uncertainty detection task, attaining an accuracy score of 0.808. This finding underscores the continued relevance and efficiency of UnScientify's simple rule-based and pattern matching strategy for this specific application. The results demonstrate that in scenarios where resource efficiency, interpretability, and domain-specific adaptability are critical, traditional methods can still offer significant advantages.

**Keywords:** *rule-based, pattern matching, research article, uncertainty, LLM*

## 1 Introduction

Uncertainty is an inherent part of scientific research, as the very nature of scientific inquiry involves posing questions, developing hypotheses, and testing them using empirical evidence. Despite the best efforts of scientists to control for extraneous variables

<footnote>*Corresponding author: panggih_kusuma.ningrum@univfcomte.fr</footnote>

<footnote n="1">¹https://bit.ly/unscientify-demo</footnote>
and obtain accurate measurements, there is always a certain degree of uncertainty associated with any scientific findings. This uncertainty can arise from a variety of sources, such as measurement error, sampling bias, or limitations in experimental design. Consequently, researchers resort to various strategies to manage and mitigate uncertainty when presenting their findings in academic articles. These may include using language that is overly definitive or hedging their claims with qualifiers such as "presumably" or "possible" (Hyland, 1996).

Uncertainty is a complex concept with various definitions (see, e.g. Walker et al. (2003); Refsgaard et al. (2007); Ascough et al. (2008)). For instance, Sigel et al. (2010) defined uncertainty in the scope of water management as "Lack of confidence about knowledge relating to a specific question", while Lehmann and Rillig (2014) interpreted uncertainty in environmental research as a "Measure of unexplained variation". The literature offers a wide range of meanings and interpretations of the term such as lack of knowledge, doubt, inconsistency, ambiguity, vagueness, subjectivity, error, risk, ignorance, and contradiction (Popper, 2002; Ruhrmann et al., 2013; Schwandt, 2015; Klir and Folger, 1988). The terminology used in describing scientific uncertainty varies depending on its nature and characteristics, disciplines, sources, and elements related to the individuals involved in scientific activity resulting plethora of meanings and connotations. When one uses the term, the context shapes the specific meaning and connotation used.

Journal articles are a common medium used by scientists to communicate their ideas and research results. As such, the textual content of these articles is a valuable source for studying scientific uncertainty. They are subjected to extensive independent peer review, which allows to guarantee the validity of the research methods and results that are reported. Most importantly, journal articles play an important role in disseminating knowledge to a wider audience. They not only describe the structural thinking of the author(s), depict the author's persona, and explain the research and analysis process (Candlin and Hyland, 2014; Hyland, 1996; Candlin, 2000; Hyland, 2000). Furthermore, journal articles invite to discuss ideas and research results, which are not always unequivocally clear but may involve some uncertainties that give space for further improvement and exploration.

Given the wide variety of definitions and scope used to describe uncertainty, it is not surprising that significant efforts have been made to detect and identify uncertainty in science. Previous research has mainly focused on identifying a specific set of uncertainty cues or markers, and text segments that express uncertainty in scholarly articles, using the abstract (Vincze et al., 2008) or the full text (Medlock and Briscoe, 2007; Atanassova et al., 2018; Riccioni et al., 2021). These studies have helped expand the vocabulary and lexicon associated with uncertainty. However, their practical application is often ineffective because of the intricate nature of natural language.

The task of identifying scientific uncertainties remains challenging due to several factors. Firstly, there is a scarcity of extensively annotated corpora that cover the full spectrum of scientific uncertainty. Existing datasets, such as the BioScope corpus (Szarvas et al., 2008), focus narrowly on negation and speculation in biological abstracts, while the FACTBANK corpus (Saurí and Pustejovsky, 2009) targets the veracity of event mentions. Similarly, the Genia Event corpus (Kim et al., 2008) is limited to annotating biological events with negation. These corpora, along with others focusing on hedging, modality, hypothetical markers, or weasel words (Riccioni et al., 2021;
Medlock and Briscoe, 2007; Ganter and Strube, 2009; Szarvas et al., 2012; Styler IV et al., 2014), are restricted to specific types of general uncertainty and fail to capture the diversity of scientific uncertainty expressions across domains.

Scientific uncertainty, however, encompasses a broader range of expressions, including research limitations, variability in findings, and ungeneralizable conclusions, which may not rely on traditional cues like hedging or modality (Ningrum and Atanassova, 2023). These expressions are often overlooked in existing datasets, limiting their applicability to the broader scope of scientific uncertainty addressed in this study. Consequently, there is an urgent need for more diverse corpora, covering a wider range of uncertainty types and domains, to enable a more comprehensive understanding of uncertainty in natural language processing (NLP). This gap reinforces the importance of developing a robust framework and annotated dataset to advance the detection and analysis of scientific uncertainty.

Secondly, identifying expressions of uncertainty is inherently challenging due to the ambiguity of linguistic cues and the diverse ways uncertainty can be expressed, depending on the writing style or stance of the scientist. A significant obstacle lies in the polysemy of cues and markers—words or phrases that carry multiple meanings depending on their context. This polysemy complicates uncertainty detection, as the same cues can convey entirely different meanings in different contexts. For instance, Suhadi (2017) explored the phenomenon of "two sides of the same coin modality" highlighting how the same modality word can imply distinct meanings, not all of which convey genuine uncertainty.

Another challenge concerns scientists’ discourse in scientific writing. A typical scientific text contains various statements and information that discuss not only the present study but also former studies (Stocking and Holstein, 1993). Scientists can use uncertainty claims from other studies as a rhetorical tool to persuade or describe and organize the state of knowledge. As a result, distinguishing the reference of the uncertainty feature – whether the statement demonstrates uncertainty in the current study or former studies, is a crucial factor in better understanding the context of scientific uncertainty. A study by Bongelli et al. (2019) is one of the few that was aware of this concern. In more detail, this study only focused on the certainty and uncertainty expressed by the speakers/writers in the here-and-now of communication and excluded those that were expressed by the other party.

## 1.1 Related Works

A wide range of techniques has been explored to detect and analyze uncertainty or related phenomena such as hedging, modality, and factuality in text. These techniques generally focus on two main objectives: treating uncertainty detection as a sentence-level classification task, where the text is categorized as either expressing uncertainty or not, and identifying specific linguistic cues or markers of uncertainty. The approaches used in previous studies can be grouped into three categories: rule-based methods, machine learning approaches, and deep learning or transformer-based methods. Each offers unique advantages and faces distinct limitations.
1.1.1 Rule-Based Approaches

Rule-based techniques represent some of the earliest efforts to detect uncertainty in
text, relying on predefined cues, markers, or linguistic patterns. These approaches
have typically employed handcrafted lexicons to identify linguistic indicators of uncer-
tainty. For example, early works by Mur-Dueñas (2021) employed such lexicons to
detect hedges in English and Spanish research articles, while more recent studies, such
as Yao et al. (2023), leveraged Mur-Dueñas’ academic hedging lexicon to map hedging
cues in scientific articles, analyzing the diachronic development of uncertainty expres-
sions. However, as noted by Ningrum and Atanassova (2023), cues alone are not always
reliable indicators of uncertainty, as their contextual meaning can vary significantly,
making simple cue matching insufficient.

To address these limitations, researchers have advanced rule-based methods by in-
corporating syntactic patterns and regular expressions (RegEx) to capture more com-
plex linguistic structures. For instance, Chapman et al. (2007) introduced ConText, a
system that combines cue detection with "pseudo-triggers" to disambiguate uncertain
assertions. Similarly, Atanassova et al. (2018) analyzed hedge verbs and an extended
vocabulary of hedging cues to study uncertainty in biomedicine and physics, while Rey
(2022) explored uncertainty in climate change discussions, organizing uncertainties into
reasoning types (abductive, inductive, deductive) and linking them to quantitative ref-
erences. Despite these advancements, the rigid nature of regular expressions limit these
approaches in capturing the flexibility and variability inherent in natural language.
Additionally, these methods often remain confined to specific domains and contexts,
reducing their generalizability.

Further refinements in rule-based methods include leveraging linguistic insights to tag nuanced and complex expressions. Omero et al. (2020) implemented a system to identify if-clauses expressing uncertainty in biomedical scientific articles. Although their approach demonstrated the value of linguistic features, it struggled with capturing if-clauses involving more intricate patterns.

Similarly, Müller et al. (2021) employed a rule-based approach incorporating man-
ual annotation, frame-linguistic modeling, and corpus-linguistic operationalization to
detect uncertainty expressions in British and German newspaper articles about the
coronavirus pandemic. They combined manual annotation with pattern-based searches,
known as the “annotation-by-query” technique, to identify semantic concepts of social
uncertainty. Building on this study, Müller (2023) developed the application Sketch
to detect expressions of ignorance and uncertainty specifically in political science texts
and conspiracy theory texts. Müller (2023) applied this technique to annotate the
Darmstadt International Relations Corpus (Müller et al., 2020), the conspiracy theory
subcorpus of LOCO (Miani et al., 2022), and a reference corpus consisting of randomly
selected articles from US national newspapers. The results indicate that this technique
offers the advantage of being applicable to much larger datasets than purely qualitative
annotation approaches, while also allowing efficient control over textual appearances
and potential ambiguity through accessible concordances. However, a significant limi-
tation of this approach lies in its ability to address the nuanced meaning of uncertainty
expressions within complex sentences. Such sentences often contain uncertainty cues or
markers along with elements of rebuttal such as neutrality, negation, or confirmation,
which can shift the overall meaning of the sentence (see Section 2.4 in the Complex
Sentence Checking part for a detailed explanation). This complexity poses a challenge for the existing pattern-based and rule-based detection methods, as they often fail to account for the broader context necessary to accurately identify uncertainty expressions. As a result, these methods risk misinterpreting or overlooking genuine uncertainty, particularly in sentences where multiple linguistic elements interact to alter the intended meaning. This limitation underscores the need for more advanced approaches that incorporate contextual and semantic analysis to improve the accuracy of uncertainty detection in scientific texts.

### 1.1.2 Machine Learning Approaches

Machine learning (ML) methods have significantly advanced uncertainty detection by automating token classification and sequence labeling tasks. These methods were prominent in initiatives such as the CoNLL-2010 Shared Task (Morante and Daelemans, 2009; Clausen, 2010; Rei and Briscoe, 2010; Fernandes et al., 2010; Mamani Sánchez et al., 2010; Li et al., 2010; Tang et al., 2010; Zhang et al., 2010), where models identified and labeled tokens expressing uncertainty. ML-based approaches rely on feature-rich representations that include surface-level attributes, part-of-speech (POS) tags, and syntactic relations. Some studies extended these feature sets to incorporate dependency relations derived from uncertainty cues, as demonstrated by Rei and Briscoe (2010), Mamani Sánchez et al. (2010), and Zhang et al. (2010).

Some studies also employed weakly supervised methods to further refine their machine learning models by reducing reliance on extensive labeled datasets. For instance, Medlock and Briscoe (2007) and Szarvas et al. (2008) annotated six papers to create a test set and randomly selected 300,000 sentences as training data for iterative bootstrapping. These techniques used minimal seed annotations to iteratively expand training datasets, effectively leveraging linguistic features such as span detectors to capture uncertainty patterns.

Despite these advancements, ML methods are not without limitations. Findings from the CoNLL-2010 Shared Task (Farkas et al., 2010) revealed that even with the inclusion of complex linguistic feature sets, performance improvements were minimal. These results underscore the critical need for effective cue disambiguation and emphasize the importance of further research into domain adaptation and the development of models capable of handling uncertainty across diverse contexts.

One major limitation lies in the reliance on handcrafted features and predefined cues, which often fail to capture the nuanced semantics and syntactic intricacies of uncertainty expressions. Narrowly focused datasets, as seen in prior studies targeting phenomena like hedging or speculative assertions, constrain models and impede their ability to generalize to different domains or capture broader types of uncertainty. For example, Medlock and Briscoe's (2007) model struggled to identify assertive statements of knowledge paucity, as these statements did not rely on explicit cues included in the training set.

Moreover, Omero et al. (2020) highlighted the difficulty ML models face in detecting complex uncertainty assertions, particularly those embedded in syntactically intricate constructions like if-clauses. This challenge may stem from limitations in how features are encoded during training. The reliance on fixed cue spans restricts ML models from addressing long-distance dependencies and intricate word interactions, both of which
are crucial for capturing complex uncertainty expressions. These constraints highlight the need for methods that integrate deeper semantic and syntactic understanding to improve the detection of nuanced uncertainty patterns.

### 1.1.3 Large Language Models and Transformer-Based Approaches

The advent of deep learning and transformer-based models has revolutionized text classification including speculation assertion and negation detection by enabling the integration of contextual embeddings and word representations. Models like BERT, XLNet, and RoBERTa have demonstrated significant advancements in detecting uncertainty markers. Khandelwal and Britto (2020) evaluated these models on datasets such as the BioScope Corpus (Szarvas et al., 2008) and the SFU Review Corpus (Kohatkar et al., 2020) for the detection of speculation and negation, leveraging contextual embeddings for greater precision. They proposed a joint training approach for cue detection and scope resolution, which outperformed separately trained versions of the same transformer-based architectures. This method achieved new state-of-the-art results, with the Combined Early Stopping variant delivering the best performance, highlighting the effectiveness of these models in capturing complex speculation and negation expressions.

The models' ability to leverage contextual embeddings and word representations marks a significant leap in speculation and negation detection, therefore offer exciting potential for further exploration, particularly in addressing diverse and nuanced types of scientific uncertainty. However, the conceptual boundaries of their training datasets, often based on simplified classifications, present an opportunity to expand their applicability. By experimenting with richer and more varied annotations, these models may potentially capture the complexity of scientific uncertainty expressions more effectively.

## 1.2 Research Objectives

Based on the rationales and research problems discussed in previous sections, the primary objective of our study is to develop a system for automatically annotating sentences that express scientific uncertainty in scholarly articles. The system goes beyond basic cue detection by incorporating deeper context and semantic analysis, ensuring the system not only identifies uncertainty but also captures its contextual and semantic nuances.

To achieve this, our study builds upon the uncertainty annotation framework presented in Ningrum and Atanassova (2024). We introduce UnScientify, a weakly supervised system that combines a fine-grained annotation framework, pattern matching, linguistic features, and complex sentence analysis to detect scientific uncertainty in the full text of articles across diverse domains. UnScientify also extracts deeper contextual information, such as authorial reference, distinguishing between uncertainty expressed by the authors and uncertainty reported from cited studies. This focus on authorial reference addresses a gap in prior research, offering a nuanced understanding of how uncertainty is framed and attributed in scientific discourse. By prioritizing interpretability and minimizing the need for fully annotated datasets, UnScientify provides a comprehensive and efficient solution for analyzing scientific uncertainty in scholarly articles.
To summarise, we focus on two main research objectives:

* **Develop an Automated System for Scientific Uncertainty Detection:** Build a system to detect and annotate scientific uncertainty in scholarly articles across various domains with interpretable results.

* **Provide a Publicly Available Dataset:** Create and release a gold-standard annotated corpus for scientific uncertainty, serving as a benchmark for future research and model evaluation.

The current study represents a substantially extended version of the article presented at the Joint Workshop of the 4th Extraction and Evaluation of Knowledge Entities from Scientific Documents and the 3rd AI + Informetrics (EEKE-AII2023) (Ningrum et al., 2023), and the authors appreciate the insights gained from the workshop that contributed to the refinement of the research.

# 2 Materials and Methods

## 2.1 Conceptual Definition of Scientific Uncertainty

In order to set a clear scope for the current study, we will focus on the uncertainty expressed in the scientific articles. After studying the different definitions of uncertainty in the previous works cited above, we provide a conceptual definition of scientific uncertainty that will be used in the current study. In particular, we need to be able to distinguish the sentences in articles (or other text spans) that express uncertainty from those that do not express uncertainty. We define them as follows:

> A sentence expresses uncertainty if it (explicitly) expresses a state of absence of knowledge, insufficient knowledge, or lack of precision in information about a concept or subject.

## 2.2 Data

The present study employs three annotated corpora. These corpora consist of 59 journals from four different disciplines: Medicine; Biochemistry, Genetics & Molecular Biology; Interdisciplinary & miscellaneous; and Empirical Social Science<fnref n="2" /> which represent Science, Technology, and Medicine (STM) as well as Social Sciences and Humanities (SSH). The corpora consist of 975 randomly selected English sentences from 312 articles across 59 journals. These sentences were annotated to identify uncertainty expressions and authorial references. Using multiple corpora from different disciplines, this study aims to capture a diverse range of uncertainty expressions and improve the generalizability of the results. Table 1 illustrates the distribution of the data in the corpora and Table 2 shows the sample of annotated sentences.

Two sampling methods were implemented to produce the dataset: cues mapping and manual searching. The first method used lists of uncertainty cues and markers

<footnote n="2">²All social science articles are from SSOAR (https://www.ssoar.info/); we selected articles from 53 social science journals indexed in SSOAR.</footnote>
<table><caption>Table 1: Corpora description</caption><thead><tr><th>Discipline</th><th>Journal</th><th>Articles</th><th>Sentences</th></tr></thead><tbody><tr><td rowspan="2">Medicine</td><td>a. BMC Med</td><td>51</td><td>95</td></tr><tr><td>b. Cell Mol Gastroenterol Hepatol</td><td>25</td><td>36</td></tr><tr><td rowspan="2">Biochemistry, Genetics & Molecular Biology</td><td>a. Nucleic Acids Res</td><td>52</td><td>63</td></tr><tr><td>b. Cell Rep Med</td><td>22</td><td>48</td></tr><tr><td rowspan="2">Interdisciplinary & miscellaneous</td><td>a. Nature</td><td>34</td><td>57</td></tr><tr><td>b. PLoS One</td><td>42</td><td>55</td></tr><tr><td>Empirical Social Science</td><td>SSOAR (53 journals)</td><td>86</td><td>621</td></tr><tr><td>Total</td><td></td><td>312</td><td>975</td></tr></tbody></table>

<table><caption>Table 2: Sample of annotated sentences</caption><thead><tr><th>Sentence</th><th>Uncertainty</th><th>Authorial Ref.</th></tr></thead><tbody><tr><td>It is possible that corticosteroids prevent some acute gastrointestinal complications.</td><td>Yes</td><td>Author(s)</td></tr><tr><td>In this test, a likelihood ratio test statistic is calculated for the two tree versus one tree models, and compared to a null distribution generated by non-parametric bootstrapping (see Methods).</td><td>No</td><td>-</td></tr><tr><td>Previous meta-analyses have shown a significant benefit for NaHCO3 in comparison to normal saline (NS) infusion [6,7], although they highlighted the possibility of publication bias.</td><td>Yes</td><td>Former/Prev. Study(s)</td></tr></tbody></table>

from Hyland (1996), Chen et al. (2018), and Bongelli et al. (2019) (see Appendix C). These cues were mapped through our corpora using Regular Expression, and from the sentences detected with these cues, we randomly selected 200 sentences from each journal in Medicine; Biochemistry, Genetics & Molecular Biology; and Interdisciplinary & miscellaneous disciplines, and up to 900 sentences from the Empirical Social Science corpus, resulting in 2100 sentences. After removing duplicates, 880 sentences remained.

The second method involved randomly selecting two articles from each journal in Medicine; Biochemistry, Genetics & Molecular Biology; and Interdisciplinary & miscellaneous disciplines. This manual search accounted for the flexibility of natural language, recognizing that uncertainty expressions may vary and not strictly follow predetermined cues. This step ensured the inclusivity of potential variations, enhancing the robustness of our analysis. From this method, 95 data points were obtained, resulting in a final dataset of 975.

Manual annotation was then conducted on the dataset. First, each sentence was labeled for the presence of a Scientific Uncertainty (SU) expression (Yes/No). For sentences marked "Yes," the annotators identified the SU markers (span) and the authorial reference, labeling it as 1: author(s), 2: former study(ies), or 3: both author(s) & former study(ies). Two annotators, provided with explicit guidelines (see Appendix D) and reference text data, were trained and tested together to ensure consistency. They
then independently labeled the dataset, resolving any discrepancies through discussion and consensus. In rare cases of disagreement, a third annotator made the final decision. This corpora construction process adheres to the framework outlined by Ningrum and Atanassova (2024). The annotated dataset used in this study is publicly accessible on the Zenodo platform<fnref n="3" /> (Ningrum and Atanassova, 2025).

## 2.3 Approach

Identifying scientific uncertainty in academic texts is a complex task due to various reasons. Previous research indicates that relying solely on cues or markers such as hedging words or modal verbs may not accurately detect and identify scientific uncertainty (Ningrum and Atanassova, 2023). The natural language and writing styles used by scientists, along with variations in domain-specific terminology, add to the complexity of identifying uncertainty in scientific text. To address these limitations, our research proposes a fine-grained annotation scheme for identifying uncertainty in scientific texts and their authorial reference.

### 2.3.1 Fine-grained SU annotation scheme and patterns formulation

The current study adopts a span-based method to identify scientific uncertainty in academic texts, building on prior research indicating that word-long phrases enhance the detection of hedging and uncertainty (Szarvas et al., 2008; Ningrum and Atanassova, 2023). This approach goes beyond mere linguistic cues, incorporating analysis of linguistic patterns and features such as Part-of-speech (POS) tags, morphology, and dependency within the spans.

A list of annotated spans has been created and classified into twelve groups of scientific uncertainty (SU) patterns based on their semantic meaning and characteristics. The groups include conditional expressions, hypotheses, predictions, and subjectivity, among others. In other words, the classification is based on the types of expressions used to convey uncertainty and the context in which they are used.

Additionally, the scheme considers spans of text that signal disagreement statements as one of SU groups, despite ongoing debate regarding whether disagreement expressions should be considered as such. The justification for this approach is rooted in the idea that uncertainty in research can stem from conflicting information or data, where multiple sources provide contradictory knowledge (Zimmermann, 2000). This type of uncertainty cannot be reduced by increasing the amount of information.

Once the annotated spans are classified, Scientific Uncertainty Span Patterns (SUSP) are formulated based on the word patterns of each span and its linguistic features. Figure 1 shows examples of the output from the spans annotation process. Each span is assigned a label corresponding to its SU pattern group. It should be noted that a sentence can have multiple labels assigned to different SU pattern groups, as seen in the second example, where labels for both conditional expression and modality are present. This feature of our annotation scheme enables the identification of complex expressions of uncertainty in scientific text. Table Appendix A shows more details about the list of

<footnote n="3">³AURORA-MESS: Annotated Uncertainty and Reference in Open Research Articles for Multidisciplinary and Empirical Social Science: https://zenodo.org/records/15001250</footnote>
SU pattern groups and samples from each group. More information about the pattern formulation process can be seen in Appendix E.

<figure><img src="image_2.png" /><figcaption>Figure 1: Two annotated sentences with SU expressions. Samples of output from the span annotation process are shown in different colours based on their SU Pattern Group.</figcaption></figure>

## 2.3.2 Authorial Reference Checking

Authorial reference is crucial in scientific writing to provide context, especially when identifying scientific uncertainty. It helps to indicate the authorship of the argument and distinguish between the claims of the author and those of others. This can be achieved through various styles of authorial reference, such as in-text citations, reference or co-reference (Powley and Dale, 2007). Additionally, there are disciplinary variations in both the frequency and use of personal and impersonal authorial references (Khedri and Kritsis, 2020). Proper attribution of uncertain claims is important to determine their origin and evaluate the credibility of the argument. For instance, when stating a hypothesis, it is essential to indicate whether it is the author's hypothesis or cited from another source. This helps the reader to assess the level of uncertainty associated with the statement.

In the present study, the authorial reference of each sentence was annotated based on the citation and co-citation patterns, and the use of personal and impersonal authorial references. Furthermore, sentences were labeled into three groups including 1) author(s) of the present article, or 2) author(s) of previous research. The last group, 3) both, is intended to accommodate complex sentences in which uncertainty may refer to both the author(s) and the previous study(s). Here, we present some examples of typical authorial reference mentions in context:

1. <I/We/Our study...>< present(s)/introduce(s)/contribute(s) to> ...

2. <Author/The former study...>< text >

3. (Author)(Year) <Text>

4. <Text> (Author1, Year1; Author2, Year2...)

5. <Text> [Ref - No1, Ref - No2...]
## 2.4 UnScientify Workflow

The UnScientify demo system for identifying scientific uncertainty expressions in articles is built using the spaCy framework. We decided to use the spaCy python library for its robust linguistic analysis capabilities, including extraction of features such as part-of-speech (POS) tags, lemmas, morphology, and syntactic dependencies, along with its efficient processing pipeline. Its extensible pattern matcher framework enables seamless integration of expert-driven rules, aligning well with our weakly supervised approach. These features make spaCy a practical choice for rule-based methods and reproducible research. While we acknowledge that spaCy's default models may lack the deep contextual representations, such as those of transformer-based models, its lightweight architecture, focus on linguistic features, and compatibility with our task-specific requirements proved highly advantageous.

Figure 2 shows the workflow of UnScientify. It consists of four main components: 1) Text Pre-processing, 2) Pattern Matching, 3) Complex Sentence Checking, and 4) Authorial Reference Checking.

<figure><img src="image_3.png" /><figcaption>Figure 2: Scientific Uncertainty (SU) expression identification workflow</figcaption></figure>

The first step, Text Preprocessing, is designed to ensure the integrity and clarity of the data prior to analysis. This critical phase includes data cleansing, which entails removing irrelevant elements from text that might obscure or distort the analysis of scientific uncertainty. Furthermore, linguistic features such as part-of-speech (POS) tags, morphology, and dependency information are extracted from the input text. In this step, the *en_core_web_trf* from spaCy is used in UnScientify system (see Appendix F). The selection of the *en_core_web_trf* pipeline was driven by its superior performance across key linguistic tasks essential for our system, outperforming other spaCy pipelines. Unlike the tok2vec-based pipelines in *en_core_web_sm*, *en_core_web_md*, and *en_core_web_lg*, the *en_core_web_trf* integrates a shared transformer layer that enhances contextual understanding and task consistency, significantly improving the precision of linguistic feature extraction. These advantages align closely with our system's requirements, ensuring better detection and analysis of uncertainty. Detailed performance comparisons between available spaCy pipelines can be found in spaCy documentation<fnref n="4" />. These benchmarks further validate the pipeline's suitability for our use case.

<footnote n="4">⁴https://spacy.io/models/en#en_core_web_trf</footnote>
Additionally, the implementation of in-text citation standardization employs regular expressions (Regex). This process is crucial because there exist various variations of the in-text citation, such as "[1, 2, 5]", "(see Max & Betty, 2002a; Marshal & Mansell, 2001)", or "James et al. (2005)". During this process, the system converts all detected in-text citations to "@CITATION". Following this process, co-reference resolution is applied to maintain the narrative's continuity and coherence. This stage includes the identification and normalization of citation blocks, simplifying the task of distinguishing between the author(s)' arguments and those from former studies. The output now produces the clean text, ready for further processing.

The second step, Pattern Matching, employs a list of patterns derived from 12 SU pattern groups (see Table Appendix A) with up to 89 patterns in total and a Scientific Uncertainty (SU) Lexicon that have been constructed during the pattern formulation process Appendix E. The input sentence is matched against the specified patterns. In the event that no match is found, the sentence is labeled 'Label 0' (Output 2), which means 'Claim'. Conversely, if a match is identified, a list of SU span candidates is generated, and the sentence is then moved to the next step. In order to optimize the matching process in this step, we have customized a spaCy rule-based matcher that takes into account keywords, span matches, sentence patterns, and linguistic features.

The third step, Complex Sentence Checking, determines whether any rebuttal, neutral, or confirmation statements are present in the sentence. The occurrences of these statements, along with the spans previously identified, eliminate the uncertainty in the sentence. If any of these patterns are detected, the sentence is labeled as 'Label 0' (Output 2), which is 'Claim', while if the system does not detect any such statements, the sentence is passed on to the final author reference checking step. Similar to the pattern matching step, this step uses a rule-based matcher from spaCy and patterns that capture the rebuttal or confirmation statement, as in the following examples:

"...no evidence to support this hypothesis..."

"In order to test whether X has a contribution to Y, statistical analysis was employed...."

"The high correlations scores confirm hypothesis H3"

Lastly, Authorial Reference Checking is conducted to determine the authorship of
uncertainty expressions, distinguishing whether they originate from the current authors,
prior studies, or a combination of both. In addition to utilizing the spaCy Matcher and
SU Lexicon, Named Entity Recognition (NER) is used in this step to accurately identify
direct mentions of author names from previous studies within the text. The outcome
of this step is the annotation of sentences with authorial references. Once the process
is complete, three outputs will be generated: sentence with 'Label 1' (Output 3), which
is sentence with 'Uncertainty'; a list of final SU spans (Output 4), which is required to
provide the interpretation of the annotation; and authorial reference label (Output 5).

Appendix B provides an overview of how UnScientify functions and its interface. The demonstration shows that out of the five sentences in the input text in the example, four express uncertainty. Two of these sentences discuss results or statements from previous studies. The detailed explanation provides information on which sentence elements support the outcome, as well as descriptive information on why the sentence
is considered an SU expression. In the demonstration, the output identifies the first sentence as an SU expression due to the occurrence of the 'Explicit Scientific Uncertainty group' represented by the phrase 'remains unexplained' in the text. UnScientify also checks for authorial references and labels the first instance as 'Author(s)', indicating that the sentence originates from the author rather than being cited from other sources or previous studies. This provides more contextual and interpretable results. To test UnScientify, please visit the demo app at https://bit.ly/unscientific-demo.

# 3 UnScientify Performance Evaluation

In order to assess the performance of the UnScientify in detecting and analyzing uncertainty in scientific articles, we evaluated it on the corpora of 975 sentences described above. This section outlines the evaluation framework, as well as the comparative experimentation with Large Language Models (LLMs).

To contextualize the performance of the proposed system, comparative experiments were conducted with various LLMs: 1) Fine-tuning BERT-based language representation models, 2) Few-shot learning, and 3) Zero-shot scenario, using the same corpora of 975 annotated sentences. The Fine-tuning, Few-shot learning strategies, and Zero-shot scenario with prompting were designed to optimise the application of LLMs for this dataset and task. Table 4 shows the models used in the experiments.

<table><caption>Table 4: Models used in the experiments</caption><thead><tr><th>Experiment</th><th>Method</th><th>Model</th></tr></thead><tbody><tr><td>1</td><td>Fine-tuning BERT-based language representation models</td><td>a. SSciBERT (Shen et al., 2023)<br>b. SciBERT (Beltagy et al., 2019)<br>c. DistilBERT(Sanh et al., 2019)<br>d. RoBERTa (Liu et al., 2019)</td></tr><tr><td>2</td><td>Few-shot learning</td><td>a. TARS (Halder et al., 2020)<br>b. SetFit (Tunstall et al., 2022)</td></tr><tr><td>3</td><td>Zero-shot learning</td><td>a. comprehend-it-multilingual-base *<br>b. GPT4 (OpenAI et al., 2023)<br>c. Llama2 (Touvron et al., 2023)<br>d. Mistral (Jiang et al., 2023)</td></tr><tr><td>4</td><td>weakly supervised with patterns matching</td><td>UnScientify (Ningrum et al., 2023)</td></tr></tbody></table>

## 3.1 Fine-tuning BERT-based language representation models

All 4 BERT-based models were fine-tuned using the Transformers library (Wolf et al., 2019). The BERT model generally showed high performance in text classification (Devlin et al., 2018). At the same time, RoBERTa outperforms BERT in many benchmark tasks and DistilBERT is a distilled version of BERT retaining BERT's performance (Liu et al., 2019; Sanh et al., 2019). SciBERT was pre-trained on a large corpus of

<footnote>*https://huggingface.co/knowledgator/comprehend_it-multilingual-t5-base</footnote>
<table><caption>Table 5: Distribution of datasets in each fold for Experiment 1</caption><thead><tr><th>Dataset</th><th>Sentences with uncertainty</th><th>Sentences without uncertainty</th><th>Total number of sentences</th></tr></thead><tbody><tr><td>Training</td><td>~391</td><td>~486</td><td>~877</td></tr><tr><td>Validation</td><td>~43</td><td>~55</td><td>~98</td></tr><tr><td>Total</td><td>434</td><td>541</td><td>975</td></tr></tbody></table>

scientific articles (Beltagy et al., 2019) while SSciBERT was pre-trained on abstracts of scientific articles from the social sciences (Shen et al., 2023).

The dataset used in this experiment consisted of 975 sentences, out of which 434 were labeled as containing uncertainty and 541 were labeled as containing non-uncertainty. To ensure a robust evaluation and a fair comparison among models, we implemented a stratified 10-fold cross-validation approach. This strategy preserved the class distribution within each fold, with 90 % (~877 sentences) of the data allocated for training and 10 % (~98 sentences) for validation. Table 5 shows the distribution of the data for the training and validation datasets. Each fold used identical training and validation splits across all models, ensuring consistency and comparability.

Training parameters were set as follows: learning rate = $5e^{-5}$, the evaluation is done (and logged) every evaluation step, the number of evaluation steps (number of update steps between two evaluations) is set to 40, the batch size per GPU for training and evaluation was set to 8, the total number of training epochs to perform was limited to 10.

## 3.2 Few-shot learning

Two approaches were used for the few-shot learning. Task-Aware Representation of Sentences (TARS) is a transformer-based approach for zero-shot or few-shot text classification (Halder et al., 2020). TARS model considers semantic information of the class label, therefore we defined the labels in natural language. We used three sets of labels for this experiment, as Table 6 shows. We used the label "claim" for sentences that do not contain uncertainty, given that claim is one of the antonyms of uncertainty.

<table><caption>Table 6: Label sets used for the TARS fine-tuning</caption><thead><tr><th>Set</th><th>Label 1</th><th>Label 2</th></tr></thead><tbody><tr><td>1</td><td>uncertainty</td><td>claim</td></tr><tr><td>2</td><td>sentence containing uncertainty</td><td>sentence containing claim</td></tr><tr><td>3</td><td>Sentences containing uncertainty are sentences in which the author(s) express (explicitly) some lack of knowledge, or a lack of precision in the information, on a subject or a concept that is clearly identified</td><td>Sentences that do not contain uncertainty are sentences in which the authors do not express any uncertainty or lack of knowledge on the subject or concept that is treated.</td></tr></tbody></table>
Sentence Transformers Fine-tuning (SetFit) is a framework designed to fine-tune sentence transformer models (Tunstall et al., 2022), enabling impressive performance with very limited training data. We used this framework to fine-tune a sentence transformer model (Reimers and Gurevych, 2019), taking advantage of its ability to deliver effective results with as few as 8 labeled sentences. This makes SetFit particularly suitable for tasks with small datasets.

To improve the model's performance and capture a wider variety of linguistic patterns, we decided to use more training data than the minimum few-shot requirement. By splitting the dataset as shown in Table 7, we provided the model with a broader range of examples, ensuring it was exposed to different contexts and nuances of uncertainty.

## 3.3 Zero-shot learning

We tested different approaches for the zero-shot learning scenario on the whole dataset of 975 sentences. Google's mT5 model<fnref n="5" /> (Xue et al., 2021) serves as the foundation for the comprehend-it-multilang-base model, enabling its application in zero-shot learning. This model supports zero-shot learning through three different approaches: (1) employing entailment-based classification or Natural Language Inference (NLI) based classification with predefined labels, (2) performing text classification with Pre-defined Hypothesis, and (3) conducting NLI-based classification in Question-Answering Setup. In this study, the comprehend-it-multilang-base model was evaluated using these three approaches. First, it was tested with the same set of labels defined in natural language as in Experiment 2 (few-shot learning). Second, the model underwent text classification using an instruction tuning strategy, incorporating three prompts as shown in Table 8 to provide additional context to form the instructions. Finally, the model was tested using the multiple choice question-answering approach, with the prompts used as questions.

<table><caption>Table 7: Number of sentences in the training data in Experiment 2 (SetFit)</caption><thead><tr><th>Dataset</th><th>Sentences containing uncertainty</th><th>Sentences without uncertainty</th><th>Total number of sentences</th></tr></thead><tbody><tr><td>Train</td><td>274</td><td>326</td><td>600</td></tr><tr><td>Test</td><td>81</td><td>94</td><td>175</td></tr><tr><td>Validation</td><td>79</td><td>121</td><td>200</td></tr><tr><td>Total</td><td>434</td><td>541</td><td>975</td></tr></tbody></table>

In the second round of experiments, generative AI was examined: GPT4 OpenAI et al. (2023), Llama2 Touvron et al. (2023) and Mistral Jiang et al. (2023). All models were tested with a set of three prompts, as Table 8 shows. Generally, prompts for conversational AI (specifically the GPT model) should contain instructions, context, input data and output indicator Giray (2023); Hu et al. (2024). The prompts utilized in this study were built using these recommendations. Prompts 1 and 2 adhere to

<footnote n="5">⁵https://huggingface.co/knowledgator/comprehend_it-multilingual-t5-base</footnote>
identical structures, but differ in wording, as we wanted to explore the potential impact of lexical choices on model accuracy. Prompt 3 is a condensed version of the previous prompts excluding the definition of uncertainty, which enables an examination of whether excessive instructions can affect the model's performance. GPT4 was tested with two temperature settings: 0.2 and 1, while Llama2 and Mistral used the default temperature setting in Ollama's framework<fnref n="6" />.

<table><caption>Table 8: Prompts used for the generative AI testing</caption><thead><tr><td>No.</td><td>Prompt</td></tr></thead><tbody><tr><td>1</td><td>As an academic researcher, I would like to distinguish sentences that express scientific uncertainty in publications. Sentences containing uncertainty are sentences in which the author(s) express (explicitly) some lack of knowledge, or a lack of precision in the information, on a subject or a concept that is clearly identified. Does the following sentence contain scientific uncertainty? Respond in the form of yes or no.</td></tr><tr><td>2</td><td>As a scientist, I want to distinguish sentences that express scientific uncertainty in scientific articles. Sentences containing uncertainty are sentences in which the author(s) express (explicitly) some lack of knowledge, or a lack of precision in the information, on a subject or a concept that is clearly identified. Does the following text contain scientific uncertainty? Answer with yes or no.</td></tr><tr><td>3</td><td>As an academic researcher, I would like to distinguish sentences that express scientific uncertainty in publications. Does the following sentence contain scientific uncertainty? Respond in the form of yes or no.</td></tr></tbody></table>

# 4 Results

## 4.1 Models' performance comparison

The experiments that we performed include three distinct experiments involving large language models (LLMs) and compare their performance with UnScientify. The results are detailed in Table 9.

In the first experiment, RoBERTa emerged as the top-performing fine-tuned BERT-based model, achieving an accuracy of 0.799. This result highlights RoBERTa's superior ability to grasp the nuances of scientific language, along with its strong contextual understanding and adaptability.

The second experiment explored few-shot learning strategies, involving multiple configurations of Task-Aware Representation of Sentences (TARS) and Sentence Transformers Fine-tuning (SetFit) across 1 and 10 epochs. The analysis revealed that both SetFit configurations, regardless of the epoch count, attained the highest accuracy, each scoring 0.771. This result suggests a significant potential of the SetFit strategy in adapting to the task with limited labeled examples.

<footnote n="6">⁶https://python.langchain.com/api_reference/ollama/chat_models/langchain_ollama chat_models.ChatterBot.html</footnote>
<table><caption>Table 9: Performances Evaluation</caption><thead><tr><th>Experiment</th><th>Model</th><th>Label</th><th>Precision</th><th>Recall</th><th>F1-Score</th><th>Support</th><th>Accuracy</th></tr></thead><tbody><tr><td rowspan="6">1</td><td rowspan="2">SciBERT</td><td>uncertainty</td><td>0.641</td><td>0.6131</td><td>0.623</td><td>434</td><td>0.676</td></tr><tr><td>claim</td><td>0.705</td><td>0.726</td><td>0.714</td><td>541</td><td></td></tr><tr><td rowspan="2">SsciBERT</td><td>uncertainty</td><td>0.789</td><td>0.765</td><td>0.771</td><td>434</td><td>0.798</td></tr><tr><td>claim</td><td>0.816</td><td>0.824</td><td>0.818</td><td>541</td><td></td></tr><tr><td rowspan="2">DistilBERT</td><td>uncertainty</td><td>0.757</td><td>0.765</td><td>0.760</td><td>434</td><td>0.784</td></tr><tr><td>claim</td><td>0.810</td><td>0.799</td><td>0.802</td><td>541</td><td></td></tr><tr><td rowspan="2">RoBERTa</td><td>uncertainty</td><td>0.762</td><td>0.811</td><td>0.780</td><td>434</td><td><strong>0.799</strong></td></tr><tr><td>claim</td><td>0.847</td><td>0.789</td><td>0.813</td><td>541</td><td></td></tr><tr><td rowspan="6">2</td><td rowspan="2">TARS1</td><td>uncertainty</td><td>0.773</td><td>0.630</td><td>0.694</td><td>81</td><td>0.720</td></tr><tr><td>claim</td><td>0.755</td><td>0.819</td><td>0.786</td><td>94</td><td></td></tr><tr><td rowspan="2">TARS2</td><td>uncertainty</td><td>0.767</td><td>0.691</td><td>0.727</td><td>81</td><td>0.731</td></tr><tr><td>claim</td><td>0.763</td><td>0.787</td><td>0.775</td><td>94</td><td></td></tr><tr><td rowspan="2">TARS3</td><td>uncertainty</td><td>0.744</td><td>0.753</td><td>0.749</td><td>81</td><td>0.703</td></tr><tr><td>claim</td><td>0.772</td><td>0.755</td><td>0.763</td><td>94</td><td></td></tr><tr><td rowspan="2">SetFit (1 epoch)</td><td>uncertainty</td><td>0.730</td><td>0.802</td><td>0.765</td><td>81</td><td><strong>0.771</strong></td></tr><tr><td>claim</td><td>0.814</td><td>0.745</td><td>0.778</td><td>94</td><td></td></tr><tr><td rowspan="6">3*</td><td rowspan="2">SetFit (10 epoch)</td><td>uncertainty</td><td>0.738</td><td>0.765</td><td>0.752</td><td>81</td><td>0.766</td></tr><tr><td>claim</td><td>0.791</td><td>0.766</td><td>0.778</td><td>94</td><td></td></tr><tr><td rowspan="2">GPT4 temp 0.2 (Prompt 1)</td><td>uncertainty</td><td>0.690</td><td>0.724</td><td>0.706</td><td>434</td><td>0.732</td></tr><tr><td>claim</td><td>0.769</td><td>0.740</td><td>0.754</td><td>541</td><td></td></tr><tr><td rowspan="2">GPT4 temp 0.2 (Prompt 2)</td><td>uncertainty</td><td>0.649</td><td>0.783</td><td>0.710</td><td>434</td><td>0.715</td></tr><tr><td>claim</td><td>0.792</td><td>0.660</td><td>0.720</td><td>541</td><td></td></tr><tr><td rowspan="2">GPT4 temp 0.2 (Prompt 3)</td><td>uncertainty</td><td>0.647</td><td>0.850</td><td>0.735</td><td>434</td><td>0.727</td></tr><tr><td>claim</td><td>0.840</td><td>0.628</td><td>0.719</td><td>541</td><td></td></tr><tr><td rowspan="2">GPT4 temp 1 (Prompt 1)</td><td>uncertainty</td><td>0.695</td><td>0.735</td><td>0.714</td><td>434</td><td><strong>0.738</strong></td></tr><tr><td>claim</td><td>0.777</td><td>0.741</td><td>0.759</td><td>541</td><td></td></tr><tr><td rowspan="2">GPT4 temp 1 (Prompt 2)</td><td>uncertainty</td><td>0.649</td><td>0.793</td><td>0.714</td><td>434</td><td>0.717</td></tr><tr><td>claim</td><td>0.798</td><td>0.656</td><td>0.720</td><td>541</td><td></td></tr><tr><td rowspan="2">GPT4 temp 1 (Prompt 3)</td><td>uncertainty</td><td>0.647</td><td>0.850</td><td>0.735</td><td>434</td><td>0.727</td></tr><tr><td>claim</td><td>0.840</td><td>0.628</td><td>0.719</td><td>541</td><td></td></tr><tr><td>knowledgator Approach1 label1</td><td></td><td>uncertainty</td><td>0.510</td><td>0.707</td><td>0.593</td><td>434</td><td>0.567</td></tr></tbody></table>

Continued on next page
<table><caption>Table 9 continued from previous page</caption><thead><tr><th>Experiment</th><th>Model</th><th>Label</th><th>Precision</th><th>Recall</th><th>F1-Score</th><th>Support</th><th>Accuracy</th></tr></thead><tbody><tr><td></td><td></td><td>claim</td><td>0.660</td><td>0.455</td><td>0.538</td><td>541</td><td></td></tr><tr><td></td><td>knowledgator Approach1 label2</td><td>uncertainty</td><td>0.503</td><td>0.553</td><td>0.527</td><td>434</td><td>0.558</td></tr><tr><td></td><td></td><td>claim</td><td>0.610</td><td>0.562</td><td>0.585</td><td>541</td><td></td></tr><tr><td></td><td>knowledgator Approach1 label3</td><td>uncertainty</td><td>0.498</td><td>0.276</td><td>0.356</td><td>434</td><td>0.554</td></tr><tr><td></td><td></td><td>claim</td><td>0.610</td><td>0.562</td><td>0.585</td><td>541</td><td></td></tr><tr><td></td><td>knowledgator Approach2 (Instruct-Prompt1)</td><td>uncertainty</td><td>0.458</td><td>0.956</td><td>0.619</td><td>434</td><td>0.477</td></tr><tr><td></td><td></td><td>claim</td><td>0.725</td><td>0.092</td><td>0.164</td><td>541</td><td></td></tr><tr><td></td><td>knowledgator Approach2 (Instruct-Prompt2)</td><td>uncertainty</td><td>0.467</td><td>0.919</td><td>0.619</td><td>434</td><td>0.496</td></tr><tr><td></td><td></td><td>claim</td><td>0.708</td><td>0.157</td><td>0.257</td><td>541</td><td></td></tr><tr><td></td><td>knowledgator Approach2 (Instruct-Prompt3)</td><td>uncertainty</td><td>0.483</td><td>0.371</td><td>0.420</td><td>434</td><td>0.544</td></tr><tr><td></td><td></td><td>claim</td><td>0.575</td><td>0.682</td><td>0.624</td><td>541</td><td></td></tr><tr><td></td><td>knowledgator Approach3 (QA-Prompt1)</td><td>uncertainty</td><td>0.382</td><td>0.509</td><td>0.436</td><td>434</td><td>0.414</td></tr><tr><td></td><td></td><td>claim</td><td>0.462</td><td>0.338</td><td>0.391</td><td>541</td><td></td></tr><tr><td></td><td>knowledgator Approach3 (QA-Prompt2)</td><td>uncertainty</td><td>0.429</td><td>0.885</td><td>0.577</td><td>434</td><td>0.425</td></tr><tr><td></td><td></td><td>claim</td><td>0.375</td><td>0.055</td><td>0.097</td><td>541</td><td></td></tr><tr><td></td><td>knowledgator Approach3 (QA-Prompt3)</td><td>uncertainty</td><td>0.331</td><td>0.249</td><td>0.284</td><td>434</td><td>0.442</td></tr><tr><td></td><td></td><td>claim</td><td>0.498</td><td>0.597</td><td>0.543</td><td>541</td><td></td></tr><tr><td></td><td>Llama2 7B Prompt 1</td><td>uncertainty</td><td>0.446</td><td>1.000</td><td>0.617</td><td>434</td><td>0.446</td></tr><tr><td></td><td></td><td>claim</td><td>1.000</td><td>0.001</td><td>0.003</td><td>541</td><td></td></tr><tr><td></td><td>Llama2 7B Prompt 2</td><td>uncertainty</td><td>0.449</td><td>0.998</td><td>0.619</td><td>434</td><td>0.454</td></tr><tr><td></td><td></td><td>claim</td><td>0.909</td><td>0.018</td><td>0.036</td><td>541</td><td></td></tr><tr><td></td><td>Llama2 7B Prompt 3</td><td>uncertainty</td><td>0.446</td><td>1.000</td><td>0.617</td><td>434</td><td>0.448</td></tr><tr><td></td><td></td><td>claim</td><td>1.000</td><td>0.006</td><td>0.01</td><td>541</td><td></td></tr><tr><td></td><td>Mistral 7B Prompt 1</td><td>uncertainty</td><td>0.630</td><td>0.597</td><td>0.613</td><td>434</td><td>0.665</td></tr><tr><td></td><td></td><td>claim</td><td>0.690</td><td>0.719</td><td>0.704</td><td>541</td><td></td></tr><tr><td></td><td>Mistral 7B Prompt 2</td><td>uncertainty</td><td>0.660</td><td>0.518</td><td>0.581</td><td>434</td><td>0.667</td></tr><tr><td></td><td></td><td>claim</td><td>0.670</td><td>0.786</td><td>0.723</td><td>541</td><td></td></tr><tr><td></td><td>Mistral 7B Prompt 3</td><td>uncertainty</td><td>0.635</td><td>0.657</td><td>0.646</td><td>434</td><td>0.679</td></tr><tr><td></td><td></td><td>claim</td><td>0.717</td><td>0.697</td><td>0.707</td><td>541</td><td></td></tr><tr><td>4</td><td>UnScientify</td><td>uncertainty</td><td>0.712</td><td>0.956</td><td>0.816</td><td>434</td><td><strong>0.808</strong></td></tr><tr><td></td><td></td><td>claim</td><td>0.952</td><td>0.689</td><td>0.800</td><td>541</td><td></td></tr></tbody></table>

<footnote>*Note: Considering the probabilistic nature of the generative LLMs, which produce outcomes with multiple variations, each experiment setting in this group was executed three times for each model. The results presented in this table were derived from the majority of answers across the three runs.</footnote>
The third experiment assessed the performance of generative LLMs in a zero-shot setting by utilizing various prompts with GPT-4 at temperatures of 0.2 and 1, alongside LLaMA 2 7B, and Mistral 7B. While for the model from the Knowledgator, three different zero-shot approaches were conducted, including the experiment using a set of labels from experiment 2, performing text classification with instruction tuning using prompts, and conducting question-answering approach using prompts. Considering the probabilistic nature of the generative LLMs which produce outcomes with multiple variations, each experiment setting in this group was executed three times for each model and setting. The results presented in Table 9 were derived from the majority of answers across the three runs.

The results from the third experiment show that GPT-4 with temperature settings of 1 (Prompt 1) achieved the highest accuracy of 0.738. This demonstrates the effectiveness of carefully crafted prompts and the influence of temperature settings on generative models' performance in zero-shot learning tasks. Meanwhile, the model from knowledgator also yields interesting results. As shown by Table 9, the implementation of different zero-shot approaches to this model produced different results even though identical prompts (for Approach 2 and 3) were used during the process.

Next, UnScientify was evaluated using the full dataset. We processed all 975 sentences and compared the system's annotations to the ones in the dataset. The results show that our method for identifying uncertainty outperforms all other methods and the above-mentioned models, achieving an accuracy of 0.808. The system shows a very high precision for 'claim' labels (0.952) and recall for 'uncertainty' labels (0.956), which indicated that our approach is relevant for the processing of complex concepts such as scientific uncertainty.

Furthermore, we evaluated the performance of all tested models using the Friedman test. The normality of the data was examined using the Shapiro-Wilk test, and the homogeneity of variance was evaluated with Levene's test. The p-values of both tests are below the 0.05 threshold, indicating that there exists a significant difference between the variances and that the data is not normally distributed. Consequently, we opted for the Friedman test over other alternative statistical methods such as ANOVA (Demšar, 2006).

<table><caption>Table 9: Results of the Friedman test</caption><thead><tr><td>Friedman chi-squared</td><td>Degrees of Freedom</td><td>p-value</td></tr></thead><tbody><tr><td>1,160</td><td>30</td><td>&lt; 2.2e-16</td></tr></tbody></table>

Zero-shot approaches were run three times each, therefore, the median prediction from the three outputs was used as the final prediction for evaluation. Additionally, fine-tuned models were evaluated using 10-fold cross-validation. Consequently, we combined the results across all folds. As Table 9 demonstrates, the p-value of the Friedman test for classifiers is less than 0.05 indicating a statistically significant difference in the performance of the classifiers.

To determine which classifiers exhibited statistically significant differences in performance, we conducted pairwise comparisons using the Pairwise Wilcoxon Rank Sum Test. To account for the increased risk of Type I error, the p-values were adjusted using the Bonferroni correction method for multiple comparisons. The test results are
displayed in Figure 3.

In the Pairwise Wilcoxon Rank Sum Test, the null hypothesis posits that there is no significant difference between the two groups, with a significance level set at 0.05. As Table 10 shows there is a statistically significant difference between the performance of UnScientify and all classifiers except TARS3, SetFit, GPT 4 prompts 2 and 3, knowledge Approach1 label1, and knowledge Approach3 (QA-Prompt1).

<figure><img src="image_4.png" /><figcaption>Figure 3: P-values of Pairwise Wilcoxon Rank Sum Test for classifier comparisons. P-values below the threshold of 0.05 indicate a statistically significant difference in the performance of classifiers.</figcaption></figure>

## 4.2 Error Analysis

Error analysis is a crucial component in evaluating the performance and robustness of language models. In this section, we conduct an error analysis to assess the accuracy and reliability of the model's predictions.
<table><caption>Table 10: P-values of Pairwise Wilcoxon Rank Sum Test for the comparison of UnScientify and other models.</caption><thead><tr><th>Model</th><th>p-value</th></tr></thead><tbody><tr><td>SciBERT</td><td>1.777101e-14</td></tr><tr><td>SsciBERT</td><td>1.804134e-19</td></tr><tr><td>DistilBERT</td><td>1.947215e-15</td></tr><tr><td>RoBERTa</td><td>7.117186e-11</td></tr><tr><td>TARS1</td><td>2.850555e-04</td></tr><tr><td>TARS2</td><td>0.016</td></tr><tr><td>TARS3</td><td>0.748</td></tr><tr><td>SetFit (1 epoch)</td><td>1</td></tr><tr><td>SetFit (10 epoch)</td><td>1</td></tr><tr><td>GPT4 temp 0.2 (Prompt 1)</td><td>4.701133e-11</td></tr><tr><td>GPT4 temp 0.2 (Prompt 2)</td><td>0.253</td></tr><tr><td>GPT4 temp 0.2 (Prompt 3)</td><td>1</td></tr><tr><td>GPT4 temp 1 (Prompt 1)</td><td>1.851902e-10</td></tr><tr><td>GPT4 temp 1 (Prompt 2)</td><td>0.979</td></tr><tr><td>GPT4 temp 1 (Prompt 3)</td><td>1</td></tr><tr><td>knowledgator Approach1 label1</td><td>1</td></tr><tr><td>knowledgator Approach1 label2</td><td>2.998742e-04</td></tr><tr><td>knowledgator Approach1 label3</td><td>1.579781e-49</td></tr><tr><td>knowledgator Approach2 (Instruct-Prompt1)</td><td>2.605883e-57</td></tr><tr><td>knowledgator Approach2 (Instruct-Prompt2)</td><td>2.802330e-40</td></tr><tr><td>knowledgator Approach2 (Instruct-Prompt3)</td><td>5.656876e-28</td></tr><tr><td>knowledgator Approach3 (QA-Prompt1)</td><td>1</td></tr><tr><td>knowledgator Approach3 (QA-Prompt2)</td><td>8.182360e-48</td></tr><tr><td>knowledgator Approach3 (QA-Prompt3)</td><td>5.845618e-24</td></tr><tr><td>Llama2 7B Prompt 1</td><td>2.343425e-84</td></tr><tr><td>Llama2 7B Prompt 2</td><td>2.563024e-81</td></tr><tr><td>Llama2 7B Prompt 3</td><td>1.731503e-83</td></tr><tr><td>Mistral 7B Prompt 1</td><td>1.166394e-15</td></tr><tr><td>Mistral 7B Prompt 2</td><td>5.954896e-30</td></tr><tr><td>Mistral 7B Prompt 3</td><td>8.774242e-10</td></tr></tbody></table>

The implementation of LLMs for text annotation, particularly in identifying scientific uncertainty from scientific articles, presents several challenges and inconsistencies. A significant concern is the inherent "black-box" nature of LLMs, which makes it difficult to trust the outputs produced by these models. While fine-tuned LLMs such as SciBERT appear promising as tools for annotating scientific uncertainty, the rationale behind its predictions remains opaque.

Generative LLMs such as GPT-4, the model from Knowledgator, Llama2, and Mistral also exhibit questionable results regarding the reasoning behind their predictions. When we re-ran the experiments three times for each model in this group, the results indicated inconsistency. Table 11 details the occurrences of these inconsistencies across three runs for each model in each experimental setting. Furthermore, Figure 4 illustrates some examples of inconsistent results from the best performer in this experiment group: GPT-4 with temperature 1 using Prompt 1.
<table><caption>Table 11: Inconsistencies in LLM labeling following three labeling processes</caption><thead><tr><td>Model</td><td>Inconsistency (n)</td></tr></thead><tbody><tr><td>GPT4 temp 1 (Prompt 1)</td><td>98</td></tr><tr><td>GPT4 temp 1 (Prompt 2)</td><td>113</td></tr><tr><td>GPT4 temp 1 (Prompt 3)</td><td>82</td></tr><tr><td>GPT4 temp 0.2 (Prompt 1)</td><td>49</td></tr><tr><td>GPT4 temp 0.2 (Prompt 2)</td><td>65</td></tr><tr><td>GPT4 temp 0.2 (Prompt 3)</td><td>35</td></tr><tr><td>knowledgator (All Approaches)</td><td>0</td></tr><tr><td>Llama2 7B Prompt 1</td><td>2</td></tr><tr><td>Llama2 7B Prompt 2</td><td>6</td></tr><tr><td>Llama2 7B Prompt 3</td><td>19</td></tr><tr><td>Mistral 7B Prompt 1</td><td>83</td></tr><tr><td>Mistral 7B Prompt 2</td><td>54</td></tr><tr><td>Mistral 7B Prompt 3</td><td>60</td></tr></tbody></table>

<figure><img src="image_5.png" /><figcaption>Figure 4: Examples of result inconsistency from LLM (using identical model, setting & prompt)</figcaption></figure>

As illustrated in Figure 4, GPT-4 model at temperature 1 and Prompt 1 resulted in inconsistent results. We use the text from Data point 14 as an example:

> "This motivates a new hypothesis, that sensory memories can act offline (indirectly) on sensorimotor performance via spontaneous activity."

In the first and second runs, GPT4 annotated this text as 'claim', and in the third run, it labeled the text as 'uncertainty'. These results make it difficult for us to identify the correct label and the cause of error in this model. This behavior also occurred in other generative LLMs, except for the model from knowledgator.

Nevertheless, the model from knowledgator revealed another kind of questionable results. The model showed inconsistent results when different zero-shot approaches were implemented as depicted by Figure 5. For example, identical prompts used in Approach 2 (zero-shot instruction) and Approach 3 (zero-shot question-answering) yielded different predictions for some data points. Moreover, the results from those two approaches showed different results to the results from approach 1 (zero-shot NLI-based classification with predefined labels) as shown in Figure 5. The inconsistent labels (shown by the colors) happen in the same data point because of the different zero-shot approaches used.

Conversely, the UnScientify did not exhibit any inconsistency issues. Its predictions remained stable across multiple runs. However, errors still emerged, primarily due to
<figure><img src="image_6.png" /><figcaption>Figure 5: Distribution of labels (claim/uncertainty) across three different zero-shot approaches of the knowledgator/comprehend_it-base1 model (top 50 data points)</figcaption></figure>

the absence of certain scientific uncertainty expression patterns in the SU Patterns Groups. Below is the sample cases that UnScientify inaccurately annotated.

*"The study needs to be replicated in different settings using a larger sample size to ensure generalizability."*

*"Nonetheless, only a subset of alcohol consumers develops CRC."*

These errors were typically associated with uncommon indirect expressions in the text which the model had never seen previously. To deal with these errors, we only need to add new patterns to the SU Patterns Group or adjust the available patterns to make UnScientify able to accommodate these writings and capture these expressions.

# 5 Discussion

## 5.1 Comparative Insights from Alternative Approaches

The comparative analysis of the results underscores the distinct challenges and advantages of each approach in the context of uncertainty detection in scientific texts. The inclination towards employing LLMs for tasks such as uncertainty detection is understandable, given their remarkable success in various NLP tasks. However, the deployment of LLMs comes with its own set of challenges (Sarkar et al., 2024), including substantial computational requirements and the need for significant customization to adapt to specific tasks like uncertainty detection.

The statistical analysis of the results revealed a significant difference in performance among the tested approaches. However, no statistically significant differences were observed between the fine-tuned models, which indicates that observed differences may occurred by chance. Notably, the performance of the fine-tuned models was significantly different to that of most zero-shot approaches, including generative models such as GPT, Llama, and Mistral.

The fine-tuning scenario with BERT-based models highlights that while domain-specific pre-training can be advantageous, its effectiveness depends on how closely the task aligns with the nuances of the pre-training corpus. Among the four models, RoBERTa achieved the highest accuracy (0.799), followed closely by SsciBERT (0.798), while DistilBERT (0.784) and SciBERT (0.675) lagged behind. Despite both SciBERT and SsciBERT being pre-trained on scientific texts, SsciBERT's strong performance suggests its social science pre-training effectively captured patterns relevant
to uncertainty detection. In contrast, SciBERT may have overfit to its pre-training data, limiting its adaptability to the task.

RoBERTa's superior performance stems from its advanced pre-training strategies, including larger datasets in the pre-training process, dynamic masking, and removal of Next Sentence Prediction (NSP), which enhance its ability to handle nuanced linguistic tasks. DistilBERT, while efficient and lightweight, showed weaker performance, likely due to its reduced model complexity, which limits its capacity to capture subtle cues. These results reveal that general-purpose models like RoBERTa excel in tasks requiring complex linguistic reasoning, while domain-specific models like SSciBERT perform well when their pre-training aligns closely with the task, and lighter models like DistilBERT prioritize efficiency over depth.

Few-shot learning strategies, particularly SetFit, demonstrated promising adaptability with limited data, suggesting a potential pathway for tasks constrained by sparse labeled examples. Nevertheless, the dependency on the quality and representativeness of few-shot examples could limit the approach's robustness across broader applications. These methods still face the challenge of transferability, where the model's ability to generalize from a few examples to unseen data can vary significantly based on the task complexity and the specificity of the examples provided. The performance of few-shot learning significantly depends on the choice of examples (Perez et al., 2021). Poorly selected examples can lead to suboptimal model understanding and performance. Additionally, crafting effective prompts can require substantial domain expertise and manual effort. Therefore, employing these strategies requires a balance between leveraging pre-trained knowledge and adapting to the specifics of the task at hand, often necessitating fine-tuning and experimentation. The weakly supervised approach that we propose, in contrast, leverages existing patterns and rules that can be easily adjusted or expanded upon by domain experts without the need for extensive datasets or complex model tuning.

The zero-shot learning scenario revealed the power of generative LLMs to adapt to tasks without direct training, albeit with the caveat of prompt sensitivity and the need for temperature adjustments to optimize performance. While promising, the variability in outcomes based on prompt design and model configurations underscores the importance of experimentation to identify optimal setups. Moreover, the interpretability of the decisions made by generative LLMs in a zero-shot context remains a challenge, as the reasoning behind their outputs can be opaque because of the "black box" nature of these models (Singh et al., 2024), making it difficult to trust and verify their assessments in critical applications such as scientific research. This lack of transparency can be a significant drawback in scientific and technical applications where understanding the "why" behind a prediction is as important as the prediction itself. In the case of UnScientify, the weakly supervised method, with its transparent and interpretable framework, offers clarity and ease of adjustment that generative LLMs currently struggle to match.

The Wilcoxon Rank Sum Test further revealed statistically significant differences between UnScientify's performance and that of most classifiers, except for certain zero-shot approaches such as TARS3, SetFit, GPT-4 prompts 2 and 3, and specific configurations of Knowledgator (Approach1 label1 and Approach3 QA-Prompt1). In cases where no significant difference was observed, this outcome may be partially attributed to the limited size of the testing dataset, which can reduce statistical power and obscure meaningful performance distinctions. Despite this, UnScientify demonstrates a
distinct advantage in its ability to efficiently handle limited labeled datasets through
its rule-based, weakly supervised approach. Unlike data-intensive machine learning or
zero-shot LLM methods, UnScientify's reliance on interpretable linguistic patterns and
minimal annotated data allows for quick adaptation and effective uncertainty detection
without extensive retraining or resource demands.

## 5.2 Performance and Insights from the UnScientify System

UnScientify, the weakly supervised system introduced in this study, exemplifies a bal-
anced and effective approach to detecting uncertainty in scientific texts. By combining
a rule-based pattern matching strategy with a mechanism for complex sentence analy-
sis, the system achieves an accuracy of 0.808, with a good balance between precision,
recall, and F1 scores. This performance underscores UnScientify's ability to navigate
the complex linguistic structures often found in scientific discourse. At the same time,
this approach is lightweight, and it does not require large labeled datasets or intensive
computational resources.

5.2.1 UnScientify vs. Transformer-Based Language Models

Transformer-based language models excel in semantic and contextual understanding by leveraging token embeddings (derived from word or subword units) and, in some cases, sentence embeddings to represent entire sentences as single vectors. These embedding strategies have proven highly effective in numerous NLP tasks, offering state-of-the-art performance in areas like text classification, named entity recognition, and semantic similarity. However, these approaches face significant challenges in tasks like phrase detection, similar to our study, which heavily depend on accurately identifying uncertainty assertion spans within sentences.

Word or token embeddings, as utilized in Transformer models, benefit from seman-
tic representation and attention mechanisms, allowing them to capture relationships
between tokens and spans effectively. However, these embeddings are still limited in
explicitly isolating and representing critical spans within sentences (Joshi et al., 2020),
particularly in tasks that require a precise focus on specific spans amid noisy or domain-
specific contexts. While the attention mechanism aids in contextualizing each token,
the resulting embeddings may diffuse focus across the sentence, making it challenging
to zero in on key spans essential for certain applications, such as identifying uncertainty
expressions. On the other hand, sentence embeddings, like those used in SetFit models,
condense an entire sentence into a single dense vector. This approach is highly effective
for tasks that benefit from a holistic sentence-level understanding, but it poses signifi-
cant limitations for applications that require precise retrieval of sub-sentence phrases.
By encoding the entire sentence as a single unit and representing a complete sentence
with a single dense vector, sentence embeddings risk losing the nuanced relationships
and interdependencies within smaller spans, making them less suitable for span-specific
tasks. This limitation has been highlighted by Orbach et al. (2024), who observed that
in real-world phrase retrieval applications, dense sentence representations struggle when
target phrases reside within noisy contexts. Similarly, in our study, these challenges im-
pede the ability of transformer-based models to precisely focus on uncertainty assertion
spans, particularly when surrounded by domain-specific or extraneous information, and
when the training data are limited. For example:

> "The correlation is often unclear between recovery speed from Covid-19 and diet consumption"

The sentence embedding-based representations might tend to dilute the signal in the sentence. Text classifiers can overemphasize common, non-discriminatory phrases such as "recovery speed from Covid-19 and diet consumption" obscuring the core uncertainty span "The correlation is often unclear" particularly in complex or verbose sentences.

Our UnScientify system, in contrast, focuses on linguistic patterns and features, which allows us to detect uncertainty expressions consistently regardless of the surrounding content or domain. By operating independently of domain-specific training, UnScientify is more efficient in capturing the important spans of scientific uncertainty expression from texts, providing a practical solution for uncertainty detection without the computational and data demands of training or fine-tuning large models.

Another key strength of our approach lies in its ability to provide interpretable outputs. Unlike black-box models, UnScientify can provide the reasons why specific sentences are flagged as uncertain, identifying and highlighting the linguistic elements from the evaluated sentences that contribute to this categorization. This transparency enhances trust in the system's outputs and makes it a valuable tool for researchers who seek not only results but also an understanding of the reasoning behind them.

### 5.2.2 UnScientify vs. Former Rule-Based Systems

UnScientify's integration of linguistic features makes it more flexible and effective than former rule-based systems that mostly utilized token level identification (Omero et al., 2020) or regex-based system (Chapman et al., 2001; Harkema et al., 2009; Atanassova et al., 2018). Token-level detection approaches struggle to accurately identify scientific uncertainty expressions because they fail to capture broader syntactic and semantic relationships within a sentence. Similarly, regex-based systems rely solely on character-based pattern matching, lacking the ability to parse grammatical dependencies, which further limits their effectiveness in detecting linguistic nuances.

In contrast, UnScientify is capable of processing complex sentence structures involving dependency relationships. This includes the identification of uncertainty expressions in embedded clauses or across non-contiguous spans. As was found by Peng et al. (2017), using dependency to detect negation or uncertainty expression in radiology reports made the detection 9.5 % higher in precision and 5.1 % higher in F1-score compared to using a regex-based approach.

Another feature of UnScientify is its ability to annotate authorial references, distinguishing whether expressions of uncertainty stem from the current author(s), previous studies, or a combination of sources. While this functionality has not yet been formally evaluated, it represents a significant step forward in uncertainty analysis, with the possibility of examining the consensus or the divergences within specific research domains.
# 6 Limitations

While UnScientify presents a viable and effective solution for detecting and analyzing uncertainty in scientific articles, several limitations should be noted. The system relies on pattern matching, which, although effective, may not fully capture the complexity of all linguistic expressions of uncertainty. However, its design offers a significant advantage: UnScientify is very easy to maintain and improve. Unlike machine learning models or LLMs that require computationally intensive retraining, enhancing UnScientify's performance only involves adding new patterns to its system. This flexibility allows it to adapt quickly to new requirements or domains without the need for large datasets or retraining.

Furthermore, while UnScientify is inherently domain-agnostic and leverages linguistic patterns, increasing the size and diversity of the dataset could introduce valuable variations to enrich its pattern library. The current dataset, which covers Medicine, Biochemistry, Interdisciplinary and Empirical Social Sciences, provides a strong foundation as it represents Science, Technology and Medicine (STM), Social Sciences and Humanities (SSH), and two journals (Nature and PLoS One) representing both broad and interdisciplinary research. However, we recognize that an extension to other fields such as engineering or arts and humanities could further improve the generalization of the system.

Although optimized for English scientific texts, UnScientify's rule-based approach is not easily transferable to other languages or tasks without significant adaptation. Despite these limitations, its evaluation scores highlight its superior capability to identify uncertainty expressions compared to other models, while also providing interpretable results. This makes UnScientify a practical, efficient, and reliable tool for uncertainty detection, offering an easy-to-maintain alternative to computationally intensive LLMs. Additionally, while GPT-4 was included in comparisons, the reproducibility of such experiments is not guaranteed due to potential modifications to GPT-4 or discontinuation of access to specific versions.

# 7 Conclusion

## 7.1 Key Findings

UnScientify offers a first comprehensive approach to identifying uncertainty expressions in scientific text. By utilizing pattern matching, complex sentence checking, and authorial reference checking, it provides clear and interpretable output that explains why a sentence is annotated as expressing uncertainty, addresses the element of SU expression, and verifies authorship reference.

The results from the evaluation show that UnScientify outperforms all other tested models for the task of identifying scientific uncertainty in sentences. These results show that our method presents a potential for enhancing information retrieval, text mining, and broader scientific article processing. Moreover, it lays the groundwork for further research on scientific uncertainty. To further enhance the UnScientify system, we intend to extend the annotation to more dimensions of scientific uncertainty including its nature, context, timeline, and communication characteristics. In future work, we also aim to explore hybrid approaches that combine spaCy's linguistic analysis with
the semantic depth of large language models (LLMs), which may further enhance the
system's performance and adaptability.

## 7.2 Contributions

The key contributions of this study are as follows:

* **UnScientify System:** A weakly supervised system combining linguistic patterns, contextual analysis, and interpretable outputs for detecting scientific uncertainty.

* **Gold-Standard Dataset:** AURORA-MESS<sup>7</sup>, a publicly available annotated corpus to support reproducibility and further research.

<footnote n="7"><sup>7</sup>https://zenodo.org/records/15001250</footnote>
Acknowledgment

This research was funded by the French ANR InSciM Project (2021-2025)<fnref n="8" /> under grant number ANR-21-CE38-0003-01, and the Chrysalide Mobilité Internationale des Doctorants (MID) mobility grant from the University of Bourgogne Franche-Comté, France. Additionally, our appreciation extends to the GESIS – Leibniz Institute for the Social Sciences for providing the dataset and invaluable support. Nina Smirnova received funding from the Deutsche Forschungsgemeinschaft (DFG) under grant number: MA 3964/7-3 (POLLUX project). Philipp Mayr received additional funding by the European Union under the Horizon Europe grant OMINO – Overcoming Multilevel Information Overload<fnref n="9" /> under grant number 101086321 (Hołyst et al., 2024).

<footnote n="8">⁸https://project-inscim.github.io/</footnote>

<footnote n="9">⁹http://ominoproject.eu</footnote>
References

J.C. Ascough, H.R. Maier, J.K. Ravalico, and M.W. Strudley. Future research challenges for incorporation of uncertainty in environmental and ecological decision-making. *Ecological Modelling*, 219(3):383-399, 2008. ISSN 0304-3800. doi: https://doi.org/10.1016/j.ecolmodel.2008.07.015. URL https://www.sciencedirect.com/science/article/pii/S0304380008003554. The Importance of Uncertainty and Sensitivity Analysis in Process-based Models of Carbon and Nitrogen Cycling in Terrestrial Ecosystems with Particular Emphasis on Forest Ecosystems.

Iana Atanassova, François-Claude Rey, and Marc Bertin. Studying Uncertainty in Science: a distributional analysis through the IMRaD structure. 2018. URL http://rgai.inf.u-szeged.hu/con112010st/tasks.html.

Iz Beltagy, Kyle Lo, and Arman Cohan. SciBERT: A pretrained language model for scientific text. In *Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)*, pages 3613-3618. Association for Computational Linguistics, 2019. doi: 10.18653/v1/D19-1371. URL https://www.aclweb.org/anthology/D19-1371.

Ramona Bongelli, Ilaria Riccioni, Roberto Burro, and Andrzej Zuczkowski. Writers' uncertainty in scientific and popular biomedical articles. A comparative analysis of the British Medical Journal and Discover Magazine. *PLoS ONE*, 14(9):e0221933, September 2019. ISSN 1932-6203. doi: 10.1371/journal.pone.0221933. URL https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6728051/.

Christopher N Candlin and Ken Hyland. *Writing: Texts, processes and practices*. Routledge, 2014.

Fiona Candlin. Practice based doctorates and questions of academic legitimacy. *Journal of Art & Design Education*, 19, 02 2000. doi: 10.1111/1468-5949.00206.

Wendy Chapman, John Dowling, and David Chu. ConText: An algorithm for identifying contextual features from clinical text. In K. Bretonnel Cohen, Dina Demner-Fushman, Carol Friedman, Lynette Hirschman, and John Pestian, editors, *Biological, translational, and clinical language processing*, pages 81-88, Prague, Czech Republic, June 2007. Association for Computational Linguistics. URL https://aclanthology.org/W07-1011.

Wendy W. Chapman, Will Bridewell, Paul Hanbury, Gregory F. Cooper, and Bruce G. Buchanan. A simple algorithm for identifying negated findings and diseases in discharge summaries. *Journal of Biomedical Informatics*, 34(5):301-310, 2001. ISSN 15320464. doi: 10.1006/jbin.2001.1029. Publisher: Academic Press Inc.

Chaomei Chen, Min Song, and Go Eun Heo. A Scalable and Adaptive Method for Finding Semantically Equivalent Cue Words of Uncertainty. Technical report, 2017. URL http://www.elsevier.com/open-access/userlicense/1.0/.
Chaomei Chen, Min Song, and Go Eun Heo. A scalable and adaptive method for finding semantically equivalent cue words of uncertainty. *Journal of Informetrics*, 12 (1):158–180, February 2018. ISSN 17511577. doi: 10.1016/j.joi.2017.12.004. URL https://linkinghub.elsevier.com/retrieve/pii/S1751157717301712.

David Clausen. HedgeHunter: A system for hedge detection and uncertainty classification. In Richárd Farkas, Veronika Vincze, György Szarvas, György Móra, and János Csirik, editors, *Proceedings of the Fourteenth Conference on Computational Natural Language Learning – Shared Task*, pages 120–125, Uppsala, Sweden, July 2010. Association for Computational Linguistics. URL https://aclanthology.org/W10-3017.

Janez Demšar. Statistical comparisons of classifiers over multiple data sets. *J. Mach. Learn. Res.*, 7:1–30, December 2006. ISSN 1532-4435.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. 2018. doi: 10.48550/ARXIV.1810.04805. URL https://arxiv.org/abs/1810.04805. Publisher: [object Object] Version Number: 2.

Richárd Farkas, Veronika Vincze, György Móra, János Csirik, and György Szarvas. The CoNLL-2010 shared task: Learning to detect hedges and their scope in natural language text. In Richárd Farkas, Veronika Vincze, György Szarvas, György Móra, and János Csirik, editors, *Proceedings of the Fourteenth Conference on Computational Natural Language Learning – Shared Task*, pages 1–12, Uppsala, Sweden, July 2010. Association for Computational Linguistics. URL https://aclanthology.org/W10-3001.

Eraldo R. Fernandes, Carlos E. M. Crestana, and Ruy L. Milidiú. Hedge detection using the relhunter approach. In *Proceedings of the Fourteenth Conference on Computational Natural Language Learning – Shared Task*, CoNLL ’10: Shared Task, page 64–69, USA, 2010. Association for Computational Linguistics. ISBN 9781932432848.

Viola Ganter and Michael Strube. Finding hedges by chasing weasels: Hedge detection using Wikipedia tags and shallow linguistic features. In Keh-Yih Su, Jian Su, Janyce Wiebe, and Haizhou Li, editors, *Proceedings of the ACL-IJCNLP 2009 Conference Short Papers*, pages 173–176, Suntec, Singapore, August 2009. Association for Computational Linguistics. URL https://aclanthology.org/P09-2044.

Louie Giray. Prompt engineering with ChatGPT: A guide for academic writers. 51 (12):2629–2633, 2023. ISSN 0090-6964, 1573-9686. doi: 10.1007/s10439-023-03272-4. URL https://link.springer.com/10.1007/s10439-023-03272-4.

Kishaloy Halder, Alan Akbik, Josip Krapac, and Roland Vollgraf. Task-aware representation of sentences for generic text classification. In *Proceedings of the 28th International Conference on Computational Linguistics*, pages 3202–3213. International Committee on Computational Linguistics, 2020. doi: 10.18653/v1/2020.coling-main.285. URL https://www.aclweb.org/anthology/2020.coling-main.285.

Henk Harkema, John N. Dowling, Tyler Thornblade, and Wendy W. Chapman. Context: An algorithm for determining negation, experiencer, and temporal status from
clinical reports. *Journal of Biomedical Informatics*, 42(5):839-851, 2009. ISSN 1532-0464. doi: https://doi.org/10.1016/j.jbi.2009.05.002. URL https://www.sciencedirect.com/science/article/pii/S1532046409000744. Biomedical Natural Language Processing.

Janusz A. Hołyst, Philipp Mayr, Michael Thelwall, Ingo Frommholz, Shlomo Havlin, Alon Sela, Yoed N. Kenett, Denis Helic, Aljoša Rehar, Sebastijan R. Maček, Przemysław Kazienko, Tomasz Kajdanowicz, Przemysław Biecek, Bolesław K. Szymanski, and Julian Sienkiewicz. Protect our environment from information overload. *Nature Human Behaviour*, 2024. ISSN 2397-3374. doi: 10.1038/s41562-024-01833-8. URL https://www.nature.com/articles/s41562-024-01833-8.

Yan Hu, Qingyu Chen, Jingcheng Du, Xueqing Peng, Vipina Kuttichi Keloth, Xu Zuo, Yujia Zhou, Zehan Li, Xiaoqian Jiang, Zhiyong Lu, Kirk Roberts, and Hua Xu. Improving large language models for clinical named entity recognition via prompt engineering. page ocad259, 2024. ISSN 1067-5027, 1527-974X. doi: 10.1093/jamia/ocad259. URL https://academic.oup.com/jamia/advance-article/doi/10.1093/jamia/ocad259/7590607.

Ken Hyland. Talking to the Academy: Forms of Hedging in Science Research Articles. *Written Communication*, 13(2):251-281, April 1996. ISSN 0741-0883. doi: 10.1177/0741088396013002004. URL https://doi.org/10.1177/0741088396013002004.
Publisher: SAGE Publications Inc.

Ken Hyland. Developments in english for specific purposes: a multi-disciplinary approach; tony dudley-evans and maggie-jo st john. cambridge, cambridge university press, 1998, 301 pp. *English for Specific Purposes*, 19:297-300, 2000. URL https://api.semanticscholar.org/CorpusID:143085816.

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b. 2023. doi: 10.48550/ARXIV.2310.06825. URL https://arxiv.org/abs/2310.06825. Publisher: [object Object] Version Number: 1.

Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. SpanBERT: Improving pre-training by representing and predicting spans. *Transactions of the Association for Computational Linguistics*, 8:64-77, 2020. doi: 10.1162/tacl_a_00300. URL https://aclanthology.org/2020.tacl-1.5.

Aditya Khandelwal and Benita Kathleen Britto. Multitask learning of negation and speculation using transformers. In Eben Holderness, Antonio Jimeno Yepes, Alberto Lavelli, Anne-Lyse Minard, James Pustejovsky, and Fabio Rinaldi, editors, *Proceedings of the 11th International Workshop on Health Text Mining and Information Analysis*, pages 79-87, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.louhi-1.9. URL https://aclanthology.org/2020.louhi-1.9.
Mohsen Khedri and Konstantinos Kritsis. How do we make ourselves heard in the writing of a research article? A study of authorial references in four disciplines. *Australian Journal of Linguistics*, 40(2):194–217, April 2020. ISSN 0726-8602, 1469-2996. doi: 10.1080/07268602.2020.1753011. URL https://www.tandfonline.com/doi/full/10.1080/07268602.2020.1753011.

Jin-Dong Kim, Tomoko Ohta, and Jun'ichi Tsujii. Corpus annotation for mining biomedical events from literature. *BMC Bioinformatics*, 9(1):10, January 2008. ISSN 1471-2105. doi: 10.1186/1471-2105-9-10. URL https://doi.org/10.1186/1471-2105-9-10.

G. Klir and T. A. Folger. *Fuzzy sets, uncertainty and information*. Englewood Cliffs, N.J.: Prentice Hall, United States of America, 1988. ISBN 0-13-345984-5 025.

Varada Kolhatkar, Hanhan Wu, Luca Cavasso, Emilie Francis, Kavan Shukla, and Maite Taboada. The SFU opinion and comments corpus: A corpus for the analysis of online news comments. *Corpus Pragmat.*, 4(2):155–190, 2020.

Johannes Lehmann and Matthias Rillig. Distinguishing variability from uncertainty. *Nat. Clim. Chang*, 4, 02 2014. doi: 10.1038/nclimate2133.

Xinxin Li, Jianping Shen, Xiang Gao, and Xuan Wang. Exploiting rich features for detecting hedges and their scope. In Richárd Farkas, Veronika Vincze, György Szarvas, György Móra, and János Csirik, editors, *Proceedings of the Fourteenth Conference on Computational Natural Language Learning – Shared Task*, pages 78–83, Uppsala, Sweden, July 2010. Association for Computational Linguistics. URL https://aclanthology.org/W10-3011.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach. 2019. doi: 10.48550/ARXIV.1907.11692. URL https://arxiv.org/abs/1907.11692. Publisher: [object Object] Version Number: 1.

Liliana Mamani Sánchez, Baoli Li, and Carl Vogel. Exploiting CCG structures with tree kernels for speculation detection. In Richárd Farkas, Veronika Vincze, György Szarvas, György Móra, and János Csirik, editors, *Proceedings of the Fourteenth Conference on Computational Natural Language Learning – Shared Task*, pages 126–131, Uppsala, Sweden, July 2010. Association for Computational Linguistics. URL https://aclanthology.org/W10-3018.

Ben Medlock and Ted Briscoe. Weakly Supervised Learning for Hedge Classification in Scientific Literature. In *Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics*, Prague, Czech Republic, June 2007. Association for Computational Linguistics. URL https://aclanthology.org/P07-1125.

Alessandro Miani, Thomas Hills, and Adrian Bangerter. LOCO: The 88-million-word language of conspiracy corpus. *Behavior Research Methods*, 54(4):1794–1817, August 2022. ISSN 1554-3528. doi: 10.3758/s13428-021-01698-z.
Roser Morante and Walter Daelemans. Learning the scope of hedge cues in biomedical texts. In K. Bretonnel Cohen, Dina Demner-Fushman, Sophia Ananiadou, John Pestian, Jun'ichi Tsujii, and Bonnie Webber, editors, *Proceedings of the BioNLP 2009 Workshop*, pages 28–36, Boulder, Colorado, June 2009. Association for Computational Linguistics. URL https://aclanthology.org/W09-1304.

Pilar Mur-Dueñas. There may be differences: Analysing the use of hedges in english and spanish research articles. *Lingua*, 260:103131, 2021. ISSN 0024-3841. doi: https://doi.org/10.1016/j.lingua.2021.103131. URL https://www.sciencedirect.com/science/article/pii/S0024384121001030.

Marcus Müller. Corpus Approaches to Analysing Uncertainty and Ignorance in Academic Discourse. *Fachsprache*, 45(1-2):28–47, May 2023. ISSN 1017-3285. doi: 10.24989/fs.v45i1-2.2232. URL https://ejournals.facultas.at/index.php/fachsprache/article/view/2232.

Marcus Müller, Jens Steffek, and Ana Schenk. The Darmstadt International Relations Corpus (DIReC). 2020. doi: 10.25534/TUPRINTS-00013063. URL https://tuprints.ulb.tu-darmstadt.de/id/eprint/13063. Publisher: UNSPECIFIED.

Marcus Müller, Sabine Bartsch, and Jens O. Zinn. Communicating the unknown: An interdisciplinary annotation study of uncertainty in the coronavirus pandemic. *International Journal of Corpus Linguistics*, 26(4):498–531, November 2021. ISSN 1384-6655, 1569-9811. doi: 10.1075/ijcl.21096.mul. URL https://www.jbe-platform.com/content/journals/10.1075/ijcl.21096.mul. Publisher: John Benjamins.

Panggih Kusuma Ningrum and Iana Atanassova. Scientific Uncertainty: an Annotation Framework and Corpus Study in Different Disciplines. In *19th International Conference of the International Society for Scientometrics and Informetrics (ISSI 2023)*, Bloomington, Indiana, US, 7 2023.

Panggih Kusuma Ningrum and Iana Atanassova. Annotation of scientific uncertainty using linguistic patterns. *Scientometrics*, May 2024.

Panggih Kusuma Ningrum and Iana Atanassova. AURORA-MESS: Annotated uncertainty and reference in open research articles for multidisciplinary and empirical social science, 2025.

Panggih Kusuma Ningrum, Philipp Mayr, and Iana Atanassova. UnScientify: Detecting Scientific Uncertainty in Scholarly Full Text. In *Joint Workshop of the 4th Extraction and Evaluation of Knowledge Entities from Scientific Documents (EEKE2023) and the 3rd AI + Informetrics (AIi2023)*, number Vol-3451, pages 52–58, Aachen, Germany, 2023. CEUR Workshop Proceedings. URL https://ceur-ws.org/Vol-3451/paper9.pdf.

Paolo Omero, Massimiliano Valotto, Riccardo Bellana, Ramona Bongelli, Ilaria Riccioni, Andrzej Zuczkowski, and Carlo Tasso. Writer's uncertainty identification in scientific biomedical articles: a tool for automatic if-clause tagging. *Language Resources and Evaluation*, 54:1–21, June 2020. doi: 10.1007/s10579-020-09491-8.
OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Łukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Łukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O'Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss,
Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. GPT-4 technical report. 2023. doi: 10.48550/ARXIV.2303.08774. URL https://arxiv.org/abs/2303.08774. Publisher: [object Object] Version Number: 6.

Eyal Orbach, Lev Haikin, Nelly David, and Avi Faizakof. Span-aggregatable, contextualized word embeddings for effective phrase mining, 2024. URL https://arxiv.org/abs/2405.07263.

Yifan Peng, Xiaosong Wang, Le Lu, Mohammadhadi Bagheri, Ronald Summers, and Zhiyong Lu. Negbio: a high-performance tool for negation and uncertainty detection in radiology reports, 2017. URL https://arxiv.org/abs/1712.05898.

Ethan Perez, Douwe Kiela, and Kyunghyun Cho. True few-shot learning with language models. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, *Advances in Neural Information Processing Systems*, volume 34, pages 11054–11070. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper_files/paper/2021/file/5c04925674920eb58467fb52ce4ef728-Paper.pdf.

Karl Popper. *Karl Popper: The Logic of Scientific Discovery*. Routledge, London and New York, 1 edition, 2002. ISBN 0-203-99462-0.

Brett Powley and Robert Dale. *Evidence-Based Information Extraction for High Accuracy Citation and Author Name Identification*. January 2007.

Jens Christian Refsgaard, Jeroen P. van der Sluijs, Anker Lajer Højberg, and Peter A. Vanrolleghem. Uncertainty in the environmental modelling process - a framework and guidance. *Environ. Model. Softw.*, 22(11):1543–1556, nov 2007. ISSN 1364-8152. doi: 10.1016/j.envsoft.2007.02.004. URL https://doi.org/10.1016/j.envsoft.2007.02.004.

Marek Rei and Ted Briscoe. Combining manual rules and supervised learning for hedge cue and scope detection. In Richárd Farkas, Veronika Vincze, György Szarvas, György Móra, and János Csirik, editors, *Proceedings of the Fourteenth Conference on Computational Natural Language Learning – Shared Task*, pages 56–63, Uppsala, Sweden, July 2010. Association for Computational Linguistics. URL https://aclanthology.org/W10-3008.

Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence embeddings using siamese BERT-networks. 2019. doi: 10.48550/ARXIV.1908.10084. URL https://arxiv.org/abs/1908.10084. Publisher: [object Object] Version Number: 1.

François-C. Rey. *Ontologie de l'incertitude et annotation sémantique d'un corpus autour du changement climatique*. Theses, Université Bourgogne Franche-Comté, May 2022. URL https://hal.science/tel-03780719.
Ilaria Riccioni, Ramona Bongelli, and Andrzej Zuczkowski. Self-mention and uncertain communication in the British Medical Journal (1840-2007): The decrease of subjectivity uncertainty markers. *Open Linguistics*, 7(1):739-759, January 2021. ISSN 23009969. doi: 10.1515/OPLI-2020-0179/MACHINEREADABLECITATION/RIS. URL https://www.degruyter.com/document/doi/10.1515/opli-2020-0179/html?lang=en. Publisher: Walter de Gruyter GmbH.

Georg Ruhrmann, Lars Guenther, Sabrina Heike Kessler, and Jutta Milde. Frames of scientific evidence: How journalists represent the (un)certainty of molecular medicine in science television programs. *Public Understanding of Science*, 24(6):681-696, 2013. ISSN 13616609. doi: 10.1177/0963662513510643. Publisher: SAGE Publications Ltd.

Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. 2019. doi: 10.48550/ARXIV.1910.01108. URL https://arxiv.org/abs/1910.01108. Publisher: [object Object] Version Number: 4.

Souvika Sarkar, Mohammad Fakhruddin Babar, Monowar Hasan, and Shubhra Kanti Karmaker. Lms as on-demand customizable service, 2024.

Roser Saurí and James Pustejovsky. Factbank: A corpus annotated with event fac-tuality. *Language Resources and Evaluation*, 43(3):227-268, September 2009. ISSN 1574020X. doi: 10.1007/s10579-009-9089-9.

Thomas A. Schwandt. Credible Evidence of Effectiveness: Necessary but Not Suffi-cient. In *Credible and Actionable Evidence*. SAGE Publications, Inc., United States of America, 2 edition, 2015. ISBN 978-1-4833-0625-4.

Si Shen, Jiangfeng Liu, Litao Lin, Ying Huang, Lin Zhang, Chang Liu, Yu-tong Feng, and Dongbo Wang. SSciBERT: a pre-trained language model for social science texts. 128(2):1241-1263, 2023. ISSN 0138-9130, 1588-2861. doi: 10.1007/s11192-022-04602-4. URL https://link.springer.com/10.1007/s11192-022-04602-4.

Katja Sigel, Bernd Klauer, and Claudia Pahl-Wostl. Conceptualising uncertainty in environmental decision-making: The example of the eu water framework directive. *Ecological Economics*, 69(3):502-510, 2010. ISSN 0921-8009. doi: https://doi.org/10.1016/j.ecolecon.2009.11.012. URL https://www.sciencedirect.com/science/article/pii/S0921800909004613.

Chandan Singh, Jeevana Priya Inala, Michel Galley, Rich Caruana, and Jianfeng Gao. Rethinking interpretability in the era of large language models, 2024.

S. Holly Stocking and Lisa W. Holstein. Constructing and Reconstructing Scientific Ignorance: Ignorance Claims in Science and Journalism. *Knowledge*, 15(2):186-210, December 1993. ISSN 0164-0259. doi: 10.1177/107554709301500205. URL https://doi.org/10.1177/107554709301500205. Publisher: SAGE Publications.
William F. Styler IV, Steven Bethard, Sean Finan, Martha Palmer, Sameer Pradhan, Piet C. de Groen, Brad Erickson, Timothy Miller, Chen Lin, Guergana Savova, and James Pustejovsky. Temporal Annotation in the Clinical Domain. *Transactions of the Association for Computational Linguistics*, 2:143–154, April 2014. ISSN 2307-387X. URL https://transacl.org/ojs/index.php/tacl/article/view/305.

Jumino Suhadi. Epistemic modality and deontic modality: Two sides of a coin. 2017.

György Szarvas, Veronika Vincze, Richárd Farkas, and János Csirik. The BioScope corpus: annotation for negation, uncertainty and their scope in biomedical texts. *BioNLP*, pages 38–45, 2008. URL www.inf.u-szeged.hu/rgai/bioscope.

György Szarvas, Veronika Vincze, Richárd Farkas, and Iryna Gurevych. Cross-Genre and Cross-Domain Detection of Semantic Uncertainty. Technical report, 2012.

Buzhou Tang, Xiaolong Wang, Xuan Wang, Bo Yuan, and Shixi Fan. A cascade method for detecting hedges and their scope in natural language text. In Richárd Farkas, Veronika Vincze, György Szarvas, György Móra, and János Csirik, editors, *Proceedings of the Fourteenth Conference on Computational Natural Language Learning – Shared Task*, pages 13–17, Uppsala, Sweden, July 2010. Association for Computational Linguistics. URL https://aclanthology.org/W10-3002.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. 2023. doi: 10.48550/ARXIV.2307.09288. URL https://arxiv.org/abs/2307.09288. Publisher: [object Object] Version Number: 2.

Lewis Tunstall, Nils Reimers, Unso Eun Seo Jo, Luke Bates, Daniel Korat, Moshe Wasserblat, and Oren Pereg. Efficient few-shot learning without prompts. 2022. doi: 10.48550/ARXIV.2209.11055. URL https://arxiv.org/abs/2209.11055. Publisher: [object Object] Version Number: 1.

Veronika Vincze, György Szarvas, Richárd Farkas, György Móra, and János Csirik. The BioScope corpus: Biomedical texts annotated for uncertainty, negation and their scopes. In *BMC Bioinformatics*, volume 9, November 2008. doi: 10.1186/1471-2105-9-S11-S9. Issue: SUPPL. 11 ISSN: 14712105.
W.E. Walker, P. Harremoës, J. Rotmans, J.P. van der Sluijs, M.B.A. van Asselt, P. Janssen, and M.P. Krayer von Krauss. Defining Uncertainty: A Conceptual Basis for Uncertainty Management in Model-Based Decision Support. *Integrated Assessment*, 4(1):5–17, March 2003. ISSN 1389-5176. doi: 10.1076/iaij.4.1.5.16466. URL http://www.tandfonline.com/doi/abs/10.1076/iaij.4.1.5.16466.

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davidson, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. HuggingFace's transformers: State-of-the-art natural language processing. 2019. doi: 10.48550/ARXIV.1910.03771. URL https://arxiv.org/abs/1910.03771. Publisher: [object Object] Version Number: 5.

Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mT5: A massively multilingual pretrained text-to-text transformer. In *Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*, pages 483–498. Association for Computational Linguistics, 2021. doi: 10.18653/v1/2021.naacl-main.41. URL https://aclanthology.org/2021.naacl-main.41.

Mingxin Yao, Ying Wei, and Huiyu Wang. Promoting research by reducing uncertainty in academic writing: a large-scale diachronic case study on hedging in science research articles across 25 years. *Scientometrics*, 128(8):4541–4558, May 2023. ISSN 0138-9130. doi: 10.1007/s11192-023-04759-6. URL https://doi.org/10.1007/s11192-023-04759-6.

Shaodian Zhang, Hai Zhao, Guodong Zhou, and Bao-Liang Lu. Hedge detection and scope finding by sequence labeling with procedural feature selection. In Richárd Farkas, Veronika Vincze, György Szarvas, György Móra, and János Csirik, editors, *Proceedings of the Fourteenth Conference on Computational Natural Language Learning – Shared Task*, pages 92–99, Uppsala, Sweden, July 2010. Association for Computational Linguistics. URL https://aclanthology.org/W10-3013.

H. J. Zimmermann. An application-oriented view of modeling uncertainty. *European Journal of Operational Research*, 122(2):190–198, April 2000. ISSN 0377-2217. doi: 10.1016/S0377-2217(99)00228-3. URL https://www.sciencedirect.com/science/article/pii/S0377221799002283.
# Appendices

## Appendix A

<table><caption>Table Appendix A: SU Pattern Groups and examples of annotated sentences with SU spans written in bold</caption><thead><tr><td>Pattern Group</td><td>Description</td><td>Examples</td></tr></thead><tbody><tr><td>Explicit SU</td><td>Explicit SU group displays expressions with obvious scientific uncertainty keywords, indicating direct and explicit uncertainty expression</td><td>1) In addition, the role of the public is often unclear.<br>2) ... the functional relevance of G4 in vivo in mammalian cells remains controversial.</td></tr><tr><td>Modality</td><td>The modality group comprises expressions that indicate uncertainty through the use of modal language</td><td>1) Different voters might have different interpretations about ...<br>2) There may also be behavioral effects.</td></tr><tr><td>Conditional Expression</td><td>The conditional expression group includes expressions that indicate uncertainty by presenting a condition or circumstance that must be met for a certain outcome to occur</td><td>1) If persons perceive the media as hostile, it is probable that the mere-exposure effect is weakened thus we hypothesize...<br>2) If there are any violations, subsequent inferential procedures may be invalid, and if so, the conclusions would be faulty.</td></tr><tr><td>Hypothesis</td><td>The hypothesis group encompasses expressions that indicate uncertainty by proposing a tentative explanation or assumption that requires further testing and verification to be confirmed</td><td>1) Hypotheses predict that aggregate support for markets should be stronger...<br>2) We assume that post-materialistic individuals may have differing attitudes towards doctors than those...</td></tr><tr><td>Prediction</td><td>The prediction group comprises expressions that indicate uncertainty by proposing a forecast or projection that may or may not come to fruition, thereby introducing an element of uncertainty</td><td>1) In July 2017, the National Grid's Future Energy Scenarios projected that the UK government...<br>2) Since aging leads to decreased Sir2, we predicted that, in young cells...</td></tr><tr><td>Interrogative Expression</td><td>The interrogative expression group includes expressions that indicate uncertainty by posing a question or series of questions, which may suggest doubt or uncertainty about a particular concept or phenomenon</td><td>1) The study aims to determine whether the observed results can be replicated across different populations.<br>2) ...this research literature has also contested whether or not citizens' knowledge about these issues is accurate...</td></tr><tr><td>Non-generalizable statement</td><td>The non-generalizable statement group expresses uncertainty with limited scope or applicability, which may not represent a broader context or population</td><td>1) Our study ... thus cannot be directly generalized to low-income nations nor extrapolated into the long-term future.<br>2) ...estimates may not be generalisable to women in other ancestry groups...</td></tr><tr><td>Adverbial SU</td><td>The scientific uncertainty group includes adverbs that modify or shift the sentence's meaning, introducing uncertainty</td><td>1) ...direct and indirect readout during the transition from search to recognition mode is poorly understood.<br>2) It will be quite certain that they belong to the subpopulation of gender heterogenous...</td></tr></tbody></table>

Continued on next page
<table><thead><tr><td>Pattern Group</td><td>Description</td><td>Examples</td></tr></thead><tbody><tr><td>Negation</td><td>The negation group comprises expressions that indicate uncertainty through the use of negation which may alter the meaning of the sentence and introduce an element of uncertainty</td><td>1) The identity of C34 modification in... is <strong>not</strong> clear.<br>2) There was <strong>no consistent</strong> evidence for a causal relationship between age at menarche and lifetime number of sexual partners...</td></tr><tr><td>Subjectivity</td><td>The subjectivity group includes expressions indicating uncertainty through subjective language like opinions, beliefs, or personal experiences</td><td>1) <strong>We believe that</strong> there are good reasons for voters to care about...<br>2) <strong>To our knowledge</strong>, this is the first study to provide global...</td></tr><tr><td>Conjectural</td><td>The conjectural group expresses uncertainty through conjecture or speculation, using guessing or suppositions without concrete evidence</td><td>1) This belief <strong>seems to be</strong> typical for moderate religiosity.<br>2) Better performance <strong>seems to be linked</strong> to life satisfaction, suggesting that individuals...</td></tr><tr><td>Disagreement</td><td>The disagreement group includes expressions that express uncertainty through disagreement or contradiction, often indicating opposing viewpoints or conflicting evidence</td><td>1) <strong>In contrast to previous studies</strong>, our results did not show a significant effect...<br>2) <strong>On the one hand</strong>, some researchers argue that the use of technology in the classroom can enhance...</td></tr></tbody></table>
Appendix B

<figure><img src="image_7.png" /><figcaption>Figure Appendix B: UnScientify Interface and Output</figcaption></figure>
# Appendix C

<table><caption>Table Appendix C: The list of cues that compose the data for data sampling with the cue mapping method</caption><thead><tr><th>Source</th><th>Category</th><th>Cues / Markers</th></tr></thead><tbody><tr><td>Hyland (1996)</td><td>–</td><td>Would (not); may (not); could; might (not); should; cannot; will (not); must; shall; ought to</td></tr><tr><td>Chen et al. (2017)</td><td>–</td><td>Unclear; suspect; controversial; ambiguity; inconclusive; unexpected; consensus; contrary; inconsistent; paradoxical; confusing; unusual; uncertain; flaw; uncertainty; dispute; unknown; impossible; ambiguous; misleading; incomplete; unexplained; contradictory; contentious; paradox; incompatible; surprising</td></tr><tr><td rowspan="4">Bongelli et al. (2019)</td><td>Epistemic verbs</td><td>I suppose; suggest/s; seem/s; suggesting; assuming; we think; I believe; it seems; expect; appear; look/s; suspect/suspected; do not seem; no one has proven; not sure</td></tr><tr><td>Epistemic non-verbs (Adjectives, Adverbs, Nouns, Personal Attributions)</td><td>Unlikely; likely/morelikely; probably; perhaps; maybe; possible; possibility; seemingly; likelihood; not likely; plausibility; possibly; potentially; potential; to our knowledge; unclear; according to my view; in my opinion; perhaps; doubt; impression; probably; unclear; apparently; uncertainty; uncertain; apparent; assumption; confident; hypothesis; plausibly</td></tr><tr><td>Modal verbs in the simple present</td><td>Can; may; may not; must</td></tr><tr><td>Modal verbs in the conditional mood</td><td>Could; would; might; should</td></tr></tbody></table>
# Appendix D

## UNCERTAINTY

Definition: **Not knowing** for sure if a statement is **true** or **false**

<table><thead><tr><th>Entities</th><th>Description</th><th>Labels</th></tr></thead><tbody><tr><td>Scientific uncertainty expression</td><td>A sentence expresses uncertainty if it (explicitly) expresses a lack of knowledge or a lack of precision in the information on a subject or concept that is identified.</td><td>Yes/No</td></tr><tr><td>Reference</td><td>According to Stocking and Holstein (1993), a typical scientific text may contain a variety of statements and information that discuss not only the current study but also previous studies. This theory serves as the foundation for the first dimension in the current framework, addressing the "Who" or reference of the scientific uncertainty expression, whether it refers to the author(s) of the observed journal article or the third party or author(s) from former research.</td><td>1:author(s)<br>2:former study(ies)<br>3:both author & former study(ies)</td></tr></tbody></table>

## Identifying Spans

Scientific uncertainty (SU) spans is the element within a sentence that conveys or contributes to the expression of scientific uncertainty. This span includes words, phrases, or clauses that indicate doubt, probability, variability, or lack of definite knowledge, thereby reflecting the tentative nature of scientific findings or hypotheses. These elements can manifest through linguistic markers such as modal verbs, qualifiers, hedges, or references to conflicting evidence or incomplete data, highlighting the inherent uncertainty in scientific discourse.

### Example 1

"It is often unclear how much people know about the policies in their country <CITATION>"

"It is often unclear whether smoking can give more benefits or disadvantages."

"It is often unclear how significant X will affect Y"

"It is often unclear what factors that contribute to the increase of Z"

### Other examples

"The result is highly ambiguous in the sense that..."

"This assumption is problematic."

"Hypotheses predict that aggregate support for markets should be stronger..."

"We assume that post-materialistic individuals may have differing attitudes towards doctors than those..."

"There may also be behavioral effects."

"If there are any violations, subsequent inferential procedures may be invalid, and if so, the conclusions would be faulty."

"... we predicted that, in young cells..."

"... this research literature has also contested whether or not citizens' knowledge about these issues is accurate..."

"... during the transition from search to recognition mode is poorly understood ..."

### Explanation

The most important element in the sentence that expresses SU is "is often unclear" while the other elements of the sentence can be replaced with any other information or contents. Therefore, we consider "is often unclear" as the SU span.

<caption>Figure Appendix D: Annotation codebook</caption>
Appendix E

<figure><img src="image_8.png" /><figcaption>Figure Appendix E: SU patterns formulation</figcaption></figure>
Appendix F

<table><caption>Table Appendix F: spaCy model evaluation for extracting linguistic features from text</caption><thead><tr><td>Model</td><td>Accuracy</td><td>Precision</td><td>Recall</td><td>F1 Score</td></tr></thead><tbody><tr><td>en_core_web_sm</td><td>0.806</td><td>0.830</td><td>0.821</td><td>0.806</td></tr><tr><td>en_core_web_md</td><td>0.807</td><td>0.831</td><td>0.822</td><td>0.807</td></tr><tr><td>en_core_web_lg</td><td>0.805</td><td>0.829</td><td>0.820</td><td>0.805</td></tr><tr><td>en_core_web_trf</td><td>0.808</td><td>0.832</td><td>0.823</td><td>0.808</td></tr></tbody></table>