# Extracting Information in a Low-resource Setting: Case Study on Bioinformatics Workflows

Clémence Sebe, Sarah Cohen-Boulakia, Olivier Ferret, Aurélie Névéol

► To cite this version:

Clémence Sebe, Sarah Cohen-Boulakia, Olivier Ferret, Aurélie Névéol. Extracting Information in a Low-resource Setting: Case Study on Bioinformatics Workflows. Symposium on Intelligent Data Analysis (IDA 2025), May 2025, Konstanz, Germany. pp.274-287, 10.1007/978-3-031-91398-3_21 . hal-05244222

HAL Id: hal-05244222

https://hal.science/hal-05244222v1

Submitted on 8 Sep 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# Extracting Information in a Low-resource Setting:
## Case Study on Bioinformatics Workflows

Clémence Sebe<fnref n="1" />, Sarah Cohen-Boulakia<fnref n="1" />, Olivier Ferret<fnref n="2" />, and Aurélie
Névéol<fnref n="1" />

**Abstract.** Bioinformatics workflows are essential for complex biological data analyses and are often described in scientific articles with source code in public repositories. Extracting detailed workflow information from articles can improve accessibility and reusability but is hindered by limited annotated corpora. To address this, we framed the problem as a low-resource extraction task and tested four strategies: 1) creating a tailored annotated corpus, 2) few-shot named-entity recognition (NER) with an autoregressive language model, 3) NER using masked language models with existing and new corpora, and 4) integrating workflow knowledge into NER models. Using BioToFlow, a new corpus of 52 articles annotated with 16 entities, a SciBERT-based NER model achieved a 70.4 F-measure, comparable to inter-annotator agreement. While knowledge integration improved performance for specific entities, it was less effective across the entire information schema. Our results demonstrate that high-performance information extraction for bioinformatics workflows is achievable.

**Keywords:** Natural Language Processing · Information Extraction · Bioinformatics Workflows

## 1 Introduction

In data-intensive sciences, such as bioinformatics, scientific results are produced using complex data analysis pipelines taking in vast amounts of experimental data. These pipelines consist of numerous steps making calls to bioinformatics tools and require considerable computational time. They can be implemented using scripts (Bash, Python, etc.) that orchestrate tool execution and serve as the glue between data, processes, tools, and the execution environment. However, the development and use of such scripts lead to challenges in the design and execution of the processing chain, as well as maintenance and reproducibility issues. In response to these challenges, efforts have been made over the past two decades to provide bioinformaticians with approaches that guide them toward better automation of data analysis pipelines: the scientific workflow systems [4]. Two systems stand out today in bioinformatics: Nextflow [5] and Snakemake [16].

<footnote n="1">¹ Université Paris-Saclay, LISN, CNRS, Orsay, 91400, France
`firstname.lastname@lisn.fr`</footnote>

<footnote n="2">² Université Paris-Saclay, CEA, List, F-91120, Palaiseau, France
`olivier.ferret@cea.fr`</footnote>
Currently, we observe: (1) an increase in scientific articles describing bioinformatics workflows in a descriptive form (steps described without being executable) and (2) a rise in bioinformatics workflows available on repositories like GitHub that are in a programmatic form (implementation available but the lack of documentation makes them difficult to reuse).

In Natural Language Processing (NLP), bioinformatics workflows remain underrepresented. To our knowledge, there is no comprehensive corpus encompassing a rich and diverse set of entities describing workflows (bioinformatics tools, data, management systems, etc.). However, some corpora focus on specific types of entities, such as those centered on bioinformatics tools [6,14]. This lack of resources makes bioinformatics a low-resource context, making it difficult to structure information or analyze texts with the limited annotated data available.

To overcome the low-resource challenges in the field, we consider in this paper several strategies. First, decoder-based approaches, such as those using generative models, offer a potential solution, although they may not always outperform other methods in terms of extraction accuracy [3,18]. Next, encoder-based approaches can be explored in two ways: using larger, related corpora to enrich model training, or by developing a small, domain-specific training corpus to focus on the extraction of entities relevant to the field. Finally, another promising approach is the injection of domain-specific knowledge directly into language models.

From a methodological perspective, we address in this paper several research questions related to low-resource settings. In a specific domain, what resources should be annotated, in what quantity and how should domain experts be involved? Which existing resources can be reused or adapted for the new task? What methods are the most effective for the target task and how can they be combined? From a more application-oriented perspective, the challenge is to develop a simple and automated approach capable of extracting information about workflows described in the literature, aiming to not only systematically document workflows presented in the literature but also facilitate their registration in workflow repositories.

In preliminary work [22], we presented a corpus of scientific articles annotated with entities representing bioinformatics workflow information (BioToFlow). We also reported preliminary experiments with Named-Entity Recognition (NER) from the corpus with a specific focus on the memorization vs. generalization abilities of statistical and rule-based methods. Herein, we provide a brief description of BioToFlow and we present new contributions as follows: (a) experiments with few-shot NER using an autoregressive language model; (b) NER experiments using a pre-existing corpus and BioToFlow and (c) integration of workflow knowledge into NER models. Our code is publicly available<fnref n="3" />.

<footnote n="3">³ https://gitlab.liris.cnrs.fr/sharefair/ner_workflows</footnote>
# 2 Workflow Information Representation

## 2.1 Workflow Representation Schema

To systematically describe the bioinformatics workflows and their components, we relied on discussions with workflow experts (notably from the ELIXIR and BOSC International Bioinformatics communities) and the review of several articles describing workflows (e.g., [4,5,15,16,27]). Generally, bioinformatics workflows consist of data analysis steps, each containing a script that may or may not invoke a bioinformatics tool. For a workflow to be executed, it is crucial to maintain a record of its execution environment. Figure 1 presents the proposed schema, which distinguishes between three main categories of entities. The first category is *core* entities, representing major components of a workflow, such as the name of the workflow described, bioinformatics tools, and data. *Tools* can also be further classified based on the context: whether they are explicitly identified as bioinformatics tools (BioInfo), laboratory-related tools (Lab), tools mentioned but not used in the final workflow (Context), or tools referenced in a general sense without specific details (General). One entity is dedicated to methods, i.e. the descriptions of workflow steps, but without mentioning specific tool names (e.g., "mapping DNA sequences against a genome" vs. "BWA"). The data refers to the information processed by the workflow (e.g., "DNA sequence"). Articles exhibit different levels of specificity and can contain mentions of a database name or the exact filenames that were used. However, in other cases, they remain more general and only refer to broad categories of data. The second category includes *environment* entities, which capture the resources and configurations required for workflow execution, such as the execution environment (e.g., the management system used, the hardware it runs on) or programming languages used in scripts. Finally, the third category addresses more specific details, including versioning information, bibliographic references describing tools or libraries, as well as descriptions and parameters associated. This additional information can be associated with the entities present in the workflow body or be more closely related to the workflow environment, providing further context about it.

## 2.2 Annotation of Workflow Information in Articles: *BioToFlow*

**Article Selection Criteria** Articles describing bioinformatics workflows with a link to workflow code in Github have been targeted using a query on PubMed and PubMedCentral [20]^4. During the annotation phase (February 2024), this query returned 89 and 91 articles describing workflows from Nextflow and Snakemake systems respectively, from which we randomly selected articles for annotation. As of 11/18/2024, the same query now returns 124 and 123 articles, demonstrating a continuous increase in the number of published articles on these workflows. Articles were pre-processed to select the contents of the *Material and Methods* and *Implementation* sections, which commonly contain workflow descriptions that we aim to annotate.

<footnote>^4 PubMed query: (nextflow[Abstract] OR snakemake[Abstract] OR nextflow[Title] OR snakemake[Title]) AND github[All Fields]</footnote>
<figure><img src="image_2.png" /><figcaption>Fig. 1. Bioinformatics workflow representation schema.</figcaption></figure>

**Creation of an Annotated Corpus** To validate the information representation schema described in section 2.1 and create a resource for evaluating the extraction of such information, we developed an annotated corpus using a collaborative method [7] with the help of an annotation guide that we designed. Annotations were performed using BRAT (Brat Rapid Annotation Tool [23], v1.3 p1), recognized for its qualities including the ability to process nested entities, an existing annotation schema, and pre-annotations [19].

Seven annotators participated in the annotation campaigns, which were conducted in multiple stages: the annotators first worked on common texts, then on different texts. Inter-annotator agreement (IAA) was calculated using the F-measure<fnref n="5" /> with BRAT-Eval [25](v0.0.2), a tool that compares two sets of annotations and analyzes divergences. The evaluation of agreement can be strict (agreement between two annotations only if identical text portions are annotated with the same label) or relaxed (agreement if two identical or overlapping text portions are annotated with the same label). The score ranges from 0 (no agreement) to 1 (perfect agreement). IAA obtained were all higher than 70% in relaxed mode. When we look in more detail at the extraction scores for each entity, we found a heterogeneity in the scores showing that some are easier to annotate than others. There are several causes for these differences, including ambiguity in the tagset, criteria for discrimination [8], and, finally, the expertise domain of the annotators. Figure 2 shows entity annotations using our schema on an excerpt from an article describing a Nextflow workflow (PMID 35171290).

Overall, our corpus, named *BioToFlow*<fnref n="6" />, is a small corpus composed of 52 articles (26 articles related to Nextflow workflows and 26 on Snakemake workflows, randomly selected from PubMed while ensuring a balanced representation of both

<footnote n="5">⁵ For a detailed review of inter-annotator agreement measures and their usage context, we refer the reader to [1]. In the case of named entity annotation, [10] demonstrates that the F-measure and κ can be considered equivalent.</footnote>

<footnote n="6">⁶ https://doi.org/10.5281/zenodo.14900544</footnote>
<figure><img src="image_3.png" /><figcaption>**Fig. 2.** Excerpt from the annotated corpus using the BRAT software.</figcaption></figure>

workflow management systems) with a total of 78,419 tokens (27,786 annotated tokens). However, the entities are diversified and distributed according to Table 1. In our corpus, we allow the presence of nested entities (~8% of entities in BioToFlow; an example can be seen in Figure 2, where a WorkflowName entity is nested within a Biblio entity).

<table><caption>**Table 1.** BioToFlow: number of occurrences per entity.</caption><thead><tr><td>Entities</td><td>Occurrences</td><td>Entities</td><td>Occurrences</td></tr></thead><tbody><tr><td>Data</td><td>2,434</td><td>Version</td><td>454</td></tr><tr><td>Tool</td><td>1,482</td><td>Hardware</td><td>429</td></tr><tr><td>Description</td><td>1,300</td><td>Database</td><td>288</td></tr><tr><td>Biblio</td><td>1,251</td><td>ManagementSystem</td><td>243</td></tr><tr><td>Method</td><td>936</td><td>Container</td><td>108</td></tr><tr><td>WorkflowName</td><td>851</td><td>ProgrammingLanguage</td><td>104</td></tr><tr><td>File</td><td>780</td><td>LibraryPackage</td><td>101</td></tr><tr><td>Parameter</td><td>464</td><td>Environment</td><td>83</td></tr></tbody></table>

Using this corpus and its annotations as a starting point, we explored different extraction approaches, starting with decoder-based approaches and then moving on to encoder-based approaches.

## 3 Decoder-based Approach for Named Entity Recognition

Because of the small size of the BioToFlow corpus, we sought to leverage an auto-regressive language model to perform few-shot named entity recognition. Specifically, we used Llama-3-8B-Instruct, which demonstrated the best performance among generative models in a comparative study of masked language models and generative models utilizing prompting techniques [18]. Following the same methodology as in this study<fnref n="7" />, we conducted a series of few-shot experiments, varying the number of examples provided (either 5 or 10) and testing different prompt formulations, such as providing or omitting entity definitions and task framing.

We use random sampling to split the BioToFlow corpus: we use 75% of the corpus (39 articles) for training and the remaining 25% (13 articles) for testing.

<footnote n="7">⁷ The code is available at https://github.com/marconaguib/autoregressive_ner</footnote>
The results obtained on the BioToFlow test set using the Llama-3-8B-Instruct model with the best hyperparameters are presented in Table 2.

<table><caption>**Table 2.** Performance of Llama-3-8B-Instruct model with the best hyperparameters (relaxed mode).</caption><thead><tr><th>Entities</th><th>P</th><th>R</th><th>F1</th><th>Entities</th><th>P</th><th>R</th><th>F1</th></tr></thead><tbody><tr><td>Biblio</td><td>93.1</td><td>82.2</td><td>87.3</td><td>LibraryPackage</td><td>3.8</td><td>5.6</td><td>4.5</td></tr><tr><td>Container</td><td>29.2</td><td>50.0</td><td>36.8</td><td>ManagementSystem</td><td>41.1</td><td>38.3</td><td>39.7</td></tr><tr><td>Data</td><td>31.2</td><td>27.6</td><td>29.3</td><td>Method</td><td>15.4</td><td>12.1</td><td>13.6</td></tr><tr><td>Database</td><td>28.3</td><td>24.6</td><td>26.3</td><td>Parameter</td><td>11.4</td><td>8.8</td><td>9.9</td></tr><tr><td>Description</td><td>26.2</td><td>19.4</td><td>22.3</td><td>ProgrammingLanguage</td><td>23.5</td><td>14.3</td><td>17.8</td></tr><tr><td>Environment</td><td>60.0</td><td>13.0</td><td>21.4</td><td>Tool</td><td>54.9</td><td>59.1</td><td>56.9</td></tr><tr><td>File</td><td>30.7</td><td>34.8</td><td>32.6</td><td>Version</td><td>67.1</td><td>39.5</td><td>49.7</td></tr><tr><td>Hardware</td><td>20.5</td><td>26.3</td><td>23.1</td><td>WorkflowName</td><td>29.1</td><td>37.7</td><td>32.8</td></tr><tr><td>Overall</td><td>40.1</td><td>36.7</td><td>38.3</td><td></td><td></td><td></td><td></td></tr></tbody></table>

We obtained an overall performance below 40% in relaxed mode and 31.2% in strict mode (not shown). These results are consistent with findings on other biomedical corpora [18]. Bioinformatics workflows are complex, and although they are closely related to the medical domain, the entities are often very specific. This leads us to explore alternative methods leveraging encoder-based models.

# 4 Encoder-based Approach for Named Entity Recognition

Encoder-based approaches for NER typically require a substantial amount of annotated corpus. In our experiments, we leverage a large corpus with annotations covering a subset of workflow information and a small corpus with comprehensive coverage of workflow information.

## 4.1 Experiments on a Large Corpus: SoftCite

To our knowledge, no corpus fully covers workflows. However, we identified corpora with annotations partially aligned with our schema, such as those focusing on datasets [21] or bioinformatics tools. We focused on bioinformatics tools, as identifying them offers crucial insights into a workflow's functionality. We can cite the CZ Software Mentions dataset [14], which includes software mentions extracted using a trained SciBERT model from diverse sources like PubMed Central and the Chan Zuckerberg Initiative. Another relevant corpus is SoftCite [6], a gold-standard dataset where all articles were manually annotated. SoftCite comprises 4,971 articles from the fields of biomedicine and economics. We chose SoftCite due to its manual annotation process and, to assess its relevance, we selected a subset of 1,159 articles specifically related to biology.
Since SoftCite does not contain the same entity types as BioToFlow, our first step was to identify possible overlaps between the two entity schemas and perform a mapping to convert SoftCite entity names to those used in BioToFlow. Table 3 represents the correspondence between entities in SoftCite (left column) and those in BioToFlow (right column). Some entities in the SoftCite corpus, *publisher\_person*, *figure*, *table*, and *formula*, have no equivalent in BioToFlow. Similarly, certain entities in BioToFlow, including *data*, *container*, *hardware*, and *description*, are not present in SoftCite. After the conversion, the number of entities and occurrences are presented in Table 4.

<table><caption>**Table 3.** Entity correspondence between SoftCite and BioToFlow.</caption><thead><tr><th></th><th>SoftCite</th><th>BioToFlow</th></tr></thead><tbody><tr><td>software</td><td></td><td>Tool</td></tr><tr><td></td><td>environment</td><td>Tool</td></tr><tr><td></td><td>url</td><td>Biblio</td></tr><tr><td></td><td>component</td><td>LibraryPackage</td></tr><tr><td></td><td>implicit</td><td>Tool_general</td></tr><tr><td>publisher</td><td></td><td>Biblio</td></tr><tr><td></td><td>environment</td><td>Environment</td></tr><tr><td>bibr</td><td></td><td>Biblio</td></tr><tr><td>version</td><td></td><td>Version</td></tr><tr><td>url</td><td></td><td>Biblio</td></tr><tr><td>language</td><td></td><td>ProgrammingLanguage</td></tr></tbody></table>

<table><caption>**Table 4.** SoftCite: Number of occurrences per entity.</caption><thead><tr><th>Entities</th><th>Occurrences</th><th>Entities</th><th>Occurrences</th></tr></thead><tbody><tr><td>Tool</td><td>4,142</td><td>Biblio</td><td>3,946</td></tr><tr><td>Version</td><td>1,349</td><td>LibraryPackage</td><td>65</td></tr><tr><td>ProgrammingLanguage</td><td>61</td><td></td><td></td></tr></tbody></table>

For the extraction of our target entities, we chose a biLSTM-CRF neural model (combination of a bidirectional LSTM (for sequence processing) with CRF (for prediction)) [26], implemented by the NLStruct Python library package<fnref n="8" /> (v0.2.0), which performs multiclass classification. All the experiments with NLStruct were run with the hyperparameters provided in Appendix A. This library offers the dual advantage of supporting the detection of nested entities, which is essential for certain entity types in our dataset, and accepting input files in the BRAT format. Additionally, it has demonstrated strong performance in the biomedical domain,

<footnote n="8">⁸ https://github.com/percevalw/NLStruct</footnote>
which closely aligns with our field. More specifically, we used this model with the SciBERT language model [2], more specifically trained on scientific articles. We conventionally chose to randomly divide the articles into two main sets: the first set, 75% of the corpus (927 articles), was used to create the training set, while the remaining 25% (232 articles) constituted the test set. Within the training set, two-thirds of the articles (649 articles) were used for the actual training of the models, and one-third (278 articles) for their validation. A distinctive aspect of our approach is that we constructed five splits through random sampling to minimize dependency on a specific division. For each split, five versions of the entity recognition model were produced using different random seeds, ensuring more robust and reproducible results.

<table><caption>**Table 5.** Average performance metrics on five splits and five random seeds (with standard deviation) of percentage scores obtained using SciBERT and NLStruct in relaxed mode on SoftCite corpora.</caption><thead><tr><th rowspan="2"></th><th colspan="3">Softcite</th></tr><tr><th>P</th><th>R</th><th>F1</th></tr></thead><tbody><tr><td>Biblio</td><td>90.6±1.4</td><td>99.0±0.4</td><td>94.6±0.9</td></tr><tr><td>Environment</td><td>0.0±0.0</td><td>0.0±0.0</td><td>0.0±0.0</td></tr><tr><td>LibraryPackage</td><td>2.5±2.4</td><td>14.3±10.5</td><td>4.1±3.9</td></tr><tr><td>ProgLanguage</td><td>48.9±5.3</td><td>88.8±4.8</td><td>62.8±4.8</td></tr><tr><td>Tool</td><td>73.4±1.9</td><td>51.0±0.6</td><td>60.2±1.0</td></tr><tr><td>Version</td><td>49.8±2.2</td><td>93.5±0.6</td><td>65.0±1.9</td></tr><tr><td><strong>Overall-focused</strong></td><td>73.1±1.1</td><td>70.7±0.6</td><td>71.9±0.8</td></tr></tbody></table>

We test the results on 13 test articles from BioToFlow to see if the entities of the SoftCite corpus could be used to train and extract a part of the BioToFlow entities. The performance on a subset of entities (Table 5) using this corpus is significantly better than those achieved with a decoder-based approach.

## 4.2 Experiments on a Small Corpus: BioToFlow

Similar to the SoftCite corpus experiments, we conducted experiments on BioToFlow using NLStruct and divided the data as follows: 75% of the corpus (39 articles) in the training set, with two-thirds of these (26 articles) used for model training and one-third (13 articles) for validation and 25% (13 articles) formed the test set. The results obtained are presented in Table 6.

The first results show that training a model on our little corpus obtains higher results for all the entities (see Tables 2, 5, and 6).

## 4.3 Fusion of SoftCite and BioToFlow Corpora

The previous results show that SoftCite and BioToFlow, when used independently, achieve good scores (above 70%). We also merged the two corpora and trained
<table><caption>**Table 6.** Average performance on five splits and five random seeds (with standard deviation) of percentage scores obtained with SciBERT and NLStruct in relaxed mode on the BioToFlow corpus. **Overall-focused** results are directly comparable to Table 5.</caption><thead><tr><th>Entities</th><th>P</th><th>R</th><th>F1</th><th>Entities</th><th>P</th><th>R</th><th>F1</th></tr></thead><tbody><tr><td>Biblio</td><td>97.9±0.4</td><td>97.4±1.0</td><td>97.7±0.4</td><td>LibraryPackage</td><td>39.8±12.3</td><td>30.9±6.7</td><td>33.9±7.4</td></tr><tr><td>Container</td><td>78.3±11.2</td><td>80.2±12.5</td><td>78.9±11.8</td><td>MgmtSystem</td><td>72.9±5.6</td><td>90.9±4.0</td><td>80.6±3.4</td></tr><tr><td>Data</td><td>70.9±2.1</td><td>57.1±2.3</td><td>63.2±1.9</td><td>Method</td><td>38.6±2.6</td><td>68.3±3.4</td><td>49.2±2.9</td></tr><tr><td>Database</td><td>63.8±3.6</td><td>54.8±4.3</td><td>58.7±1.8</td><td>Parameter</td><td>53.2±3.3</td><td>53.1±4.8</td><td>52.9±2.9</td></tr><tr><td>Description</td><td>65.4±3.9</td><td>59.3±1.8</td><td>62.1±1.5</td><td>ProgLanguage</td><td>86.9±2.1</td><td>87.2±2.9</td><td>86.9±2.4</td></tr><tr><td>Environment</td><td>50.3±17.9</td><td>79.4±10.1</td><td>60.3±17.3</td><td>Tool</td><td>77.7±3.2</td><td>72.2±2.8</td><td>74.8±1.0</td></tr><tr><td>File</td><td>84.3±0.5</td><td>82.5±1.4</td><td>83.4±0.6</td><td>Version</td><td>86.8±2.6</td><td>90.8±2.0</td><td>88.7±0.9</td></tr><tr><td>Hardware</td><td>65.8±1.2</td><td>79.2±4.4</td><td>71.7±1.7</td><td>WorkflowName</td><td>67.1±2.2</td><td>77.8±4.4</td><td>72.0±2.1</td></tr><tr><td><strong>Overall</strong></td><td>71.2±1.3</td><td>69.6±0.9</td><td>70.4±0.8</td><td><strong>Overall-focused</strong></td><td>85.5±1.2</td><td>83.4±1.2</td><td>84.4±0.5</td></tr></tbody></table>

models on the combined dataset to see the contribution of the SoftCite corpus. Two configurations of the SoftCite corpus were considered: (a) entities were only converted to match BioToFlow entity names and (b) each SoftCite article was annotated using predictions from a model trained on BioToFlow.

The scores for specific entities are provided in Table 7. The results suggest that the SoftCite corpus, in its silver form, achieves a slightly higher F1 score compared to a model trained only on BioToFlow (Table 6). These corpus fusions resulted in some entity scores increasing while others decreased.

<table><caption>**Table 7.** Average performance metrics on five splits and five random seeds (with standard deviation) of percentage scores obtained using SciBERT and NLStruct in relaxed mode on the fusion of BioToFlow and SoftCite corpora.</caption><thead><tr><th rowspan="2"></th><th colspan="3">With SoftCite conversion</th><th colspan="3">With SoftCite silver</th></tr><tr><th>P</th><th>R</th><th>F1</th><th>P</th><th>R</th><th>F1</th></tr></thead><tbody><tr><td>Data</td><td>56.9±2.3</td><td>64.1±1.5</td><td>60.1±0.7</td><td>75.8±2.6</td><td>56.8±2.1</td><td>64.9±0.8</td></tr><tr><td>Description</td><td>62.9±3.7</td><td>62.5±1.3</td><td>62.6±1.2</td><td>71.0±0.8</td><td>57.4±0.9</td><td>63.5±0.7</td></tr><tr><td>LibraryPackage</td><td>36.2±7.9</td><td>38.1±6.3</td><td>36.1±7.0</td><td>25.3±6.4</td><td>37.2±8.9</td><td>29.0±5.7</td></tr><tr><td>ProgLanguage</td><td>88.6±5.4</td><td>87.6±4.0</td><td>87.9±2.4</td><td>90.3±1.3</td><td>88.3±3.0</td><td>89.2±1.4</td></tr><tr><td>Tool</td><td>76.9±1.3</td><td>72.2±1.2</td><td>74.5±0.9</td><td>78.8±1.9</td><td>70.0±0.9</td><td>74.1±1.1</td></tr><tr><td><strong>Overall</strong></td><td>64.4±1.0</td><td>73.6±0.9</td><td>68.7±0.3</td><td>73.0±0.5</td><td>68.5±1.4</td><td>70.7±0.6</td></tr></tbody></table>

# 5 Integration of Knowledge into Language Models

Since the various tested approaches did not show any significant improvement in NER scores compared to a masked language model trained on BioToFlow, we explored one final approach: the integration of knowledge into language models.
**Used of a Pre-initialized Encoder** In the spirit of work exploiting pretraining for NER [17], we trained models on BioToFlow following the same methodology as in Section 4.2 but with an encoder initialized with one of the models trained during the NER experiments on the SoftCite corpus (see Section 4.1). The scores of Table 8 for a subset of entities show that the results obtained are nearly identical to those achieved with a model whose encoder was not pre-initialized.

<table><caption>**Table 8.** Average performance metrics on five splits and five random seeds (with standard deviation) of percentage scores for a subset of entities obtained using SciBERT and NLStruct in relaxed mode on BioToFlow corpus with the encoder initialization.</caption><thead><tr><th></th><th>P</th><th>R</th><th>F1</th><th></th><th>P</th><th>R</th><th>F1</th></tr></thead><tbody><tr><td>Database</td><td>63.9±3.2</td><td>56.2±3.3</td><td>59.7±1.3</td><td>Method</td><td>38.2±1.2</td><td>68.1±3.3</td><td>48.9±1.6</td></tr><tr><td>Environment</td><td>47.7±18.9</td><td>80.0±14.3</td><td>58.5±20.1</td><td>ProgLanguage</td><td>87.1±1.0</td><td>85.9±5.0</td><td>86.2±2.5</td></tr><tr><td>LibraryPackage</td><td>40.2±11.5</td><td>32.4±5.6</td><td>35.1±5.9</td><td>Tool</td><td>77.3±3.1</td><td>72.1±2.2</td><td>74.5±0.6</td></tr><tr><td>MgmtSystem</td><td>72.7±5.9</td><td>94.4±5.7</td><td>82.0±4.2</td><td><strong>Overall</strong></td><td>71.3±0.8</td><td>69.6±1.3</td><td>70.4±0.8</td></tr></tbody></table>

**Adding Vocabulary Tools** are considered a key entity in our work because each tool has a specific function in the workflow, and it is essential for the overall understanding of the workflow task that all tools are extracted properly. To improve the extraction of bioinformatics tool names, another strategy involves incorporating external knowledge into the encoder. Since the knowledge to inject in our case is made of lists of names, we chose to perform this injection at the level of its vocabulary, similarly to work such as [9,12]. Various knowledge bases of bioinformatics tools exist, from which tool and binary names (the executable files or commands used to run the tools in a program) can be extracted. We utilized four knowledge bases: *Biotools* [13], *Bioconda* [11], *Biocontainers* [24] and *Bioweb*<fnref n="9" />. From these knowledge bases, we aggregated the names of tools and binaries to create an enriched vocabulary. This new vocabulary was then injected into the SciBERT language model to improve its ability to recognize bioinformatics tool names. We evaluated this strategy by conducting two experiments on BioToFlow, testing two approaches: (a) fine-tuning the SciBERT model with the new vocabulary and (b) using the enriched vocabulary without additional fine-tuning. In both cases, the embeddings of the added tokens were initialized by averaging the embeddings of their subtokens.

Adding bioinformatics tool vocabulary appears to improve the extraction of tool names, particularly when fine-tuning the SciBERT model after incorporating the new vocabulary, with an F1 score increasing from 74.8±1.0% to 77.0±2.8%. As observed in previous experiments, some entities benefit from the added knowledge, while others experience slight decreases in performance (see Table 9). Specifically,

<footnote n="9">⁹ https://bioweb.pasteur.fr/welcome</footnote>
the *Data* entity shows improvements with vocabulary addition in both fine-tuning and non-fine-tuning scenarios. Entities like *LibraryPackage* and *Tool* also benefit from fine-tuning with the added vocabulary. Conversely, scores for *ManagementSystem* and *ProgrammingLanguage* entities show slight declines.

<table><caption>**Table 9.** Average performance metrics on five splits and five random seeds (with standard deviation) of percentage scores obtained using SciBERT and NLStruct in relaxed mode with the addition of bioinformatics tool vocabulary, with or without fine-tuning.</caption><thead><tr><th rowspan="2"></th><th colspan="3">Add voc - finetune</th><th colspan="3">Add voc - no finetune</th></tr><tr><th>P</th><th>R</th><th>F1</th><th>P</th><th>R</th><th>F1</th></tr></thead><tbody><tr><td>Data</td><td>70.0±7.2</td><td>63.4±10.9</td><td>66.4±8.9</td><td>71.7±2.4</td><td>56.9±2.8</td><td>63.4±2.3</td></tr><tr><td>LibraryPackage</td><td>52.5±7.1</td><td>57.4±15.6</td><td>52.3±10.0</td><td>37.1±8.0</td><td>30.4±7.2</td><td>32.7±6.0</td></tr><tr><td>ManagementSystem</td><td>75.6±7.9</td><td>76.8±6.7</td><td>75.5±7.0</td><td>69.7±3.9</td><td>93.4±3.3</td><td>79.7±1.7</td></tr><tr><td>ProgrammingLanguage</td><td>81.9±12.6</td><td>82.1±11.5</td><td>81.9±12.0</td><td>86.0±2.1</td><td>85.8±2.3</td><td>85.8±2.1</td></tr><tr><td>Tool</td><td>77.2±3.7</td><td>76.9±2.7</td><td>77.0±2.8</td><td>74.5±2.5</td><td>73.9±2.6</td><td>74.1±0.3</td></tr><tr><td>Overall</td><td>70.4±1.7</td><td>70.2±1.5</td><td>70.3±0.4</td><td>70.6±1.2</td><td>69.5±1.2</td><td>70.1±0.9</td></tr></tbody></table>

# 6 Conclusion

Based on a workflow representation schema, we developed a small entity-rich corpus, BioToFlow. Using this corpus, we experimented with various entity extraction techniques, including decoder-based models, which achieved scores below 50%, and encoder-based methods. These included leveraging a large corpus with annotations covering a subset of the schema and our BioToFlow corpus. A final approach sought to incorporate additional knowledge into the models. The most effective strategy involved training a model on our small corpus while integrating a vocabulary list specific to bioinformatics tools, which improved the extraction of this entity type. All these experiments and combinations of approaches demonstrate that bioinformatics workflows represent a highly specialized branch of biology. This implies that named entity extraction requires a large amount of domain-specific annotated data, as well as the use of language models tailored to bioinformatics. To improve entity extraction, additional few-shot learning approaches should be explored, such as using NER models trained to identify any type of entity, like GLiNER [28]. To further extend the approach of incorporating domain knowledge, we will also investigate various ways to enhance the embeddings of the added vocabulary, starting with their adaptation during the training of the NER models, similarly to [12]. More globally, the entity linking between the components extracted from workflows described in scientific literature and those found in code from public repositories is an essential task that we will consider to merge information from both sources.
**Acknowledgments.** We sincerely thank N. Bossut, A. Gaignard, G. Marchment, M. Schmit, H. Menager, F. Lemoine for their help with the annotation process. We also thank M. Naguib for his help with the decoder-based experiments. This work has received support from the French government (Agence Nationale pour la Recherche) under the France 2030 program grant agreement ANR-22-PESN-0007 (ShareFAIR).

**Disclosure of Interests.** The authors have no competing interests.

## A Hyperparameters used with NLStruct

<table><caption>Table 10. Hyperparameters used with NLStruct.</caption><thead><tr><th>Parameter</th><th>Value</th><th>Parameter</th><th>Value</th></tr></thead><tbody><tr><td>Base encoder</td><td>BERT-base-uncased<br>SciBERT_scivocab_uncased</td><td>Learning rate</td><td>1e-3</td></tr><tr><td>Sequence length</td><td>256</td><td>Dropout</td><td>0</td></tr><tr><td>Train iteration</td><td>4,000</td><td>Warmup ratio</td><td>0</td></tr><tr><td>Optimizer</td><td>AdamW</td><td>Random seed</td><td>1 - 8 - 22 - 42 -100</td></tr></tbody></table>

## References

1. Artstein, R., Poesio, M.: Inter-coder agreement for computational linguistics. Computational linguistics 34(4), 555–596 (2008)

2. Beltagy, I., Lo, K., Cohan, A.: SciBERT: A Pretrained Language Model for Scientific Text. In: Inui, K., Jiang, J., Ng, V., Wan, X. (eds.) Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). pp. 3615–3620. Association for Computational Linguistics, Hong Kong, China (Nov 2019). https://doi.org/10.18653/v1/D19-1371, https://aclanthology.org/D19-1371

3. Chen, Q., Hu, Y., Peng, X., Xie, Q., Jin, Q., Gilson, A., Singer, M.B., Ai, X., et al.: A systematic evaluation of large language models for biomedical natural language processing: benchmarks, baselines, and recommendations (Sep 2024). https://doi.org/10.48550/arXiv.2305.16326, http://arxiv.org/abs/2305.16326

4. Cohen-Boulakia, S., Belhajjame, K., Collin, O., Chopard, J., Froidevaux, C., et al.: Scientific workflows for computational reproducibility in the life sciences: Status, challenges and opportunities. Future Generation Computer Systems 75, 284–298 (2017)

5. Di Tommaso, P., Chatzou, M., Floden, E.W., Barja, P., Palumbo, E., Notredame, C.: Nextflow enables reproducible computational workflows. Nature Biotech 35, 316–319 (Apr 2017). https://doi.org/10.1038/nbt.3820

6. Du, C., Cohoon, J., Lopez, P., Howison, J.: Softcite dataset: A dataset of software mentions in biomedical and economic research publications. Journal of the Association for Information Science and Technology 72(7), 870–884 (2021), https://onlinelibrary.wiley.com/doi/abs/10.1002/asi.24454
7. Fort, K.: Collaborative Annotation for Reliable Natural Language Processing: Technical and Sociological Aspects. Wiley-ISTE (Jul 2016), https://hal.archives-ouvertes.fr/hal-01324322

8. Fort, K., Nazarenko, A., Rosset, S.: Modeling the complexity of manual annotation tasks: a grid of analysis. In: Kay, M., Boitet, C. (eds.) Proceedings of COLING 2012. pp. 895–910. The COLING 2012 Organizing Committee, Mumbai, India (Dec 2012), https://aclanthology.org/C12-1055

9. Gee, L., Zugarini, A., Rigutini, L., Torroni, P.: Fast vocabulary transfer for language model compression. In: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: Industry Track. pp. 409–416. Association for Computational Linguistics, Abu Dhabi, UAE (Dec 2022), https://aclanthology.org/2022.emnlp-industry.41

10. Grouin, C., Rosset, S., Zweigenbaum, P., Fort, K., Galibert, O., Quintard, L.: Proposal for an extension of traditional named entities: From guidelines to evaluation, an overview. In: Proc. of Linguistic Annotation Workshop (LAW-V). pp. 92–100. Portland, OR (23-24 june 2011), http://www.aclweb.org/anthology/W11-0411

11. Grüning, B., Dale, R., Sjödin, A., Chapman, B.A., Rowe, J., Tomkins-Tinch, C.H., Valieris, R., Köster, J.: Bioconda: sustainable and comprehensive software distribution for the life sciences. Nature Methods 15(7), 475–476 (Jul 2018). https://doi.org/10.1038/s41592-018-0046-7

12. Hong, J., Kim, T., Lim, H., Choo, J.: AVocaDo: Strategy for Adapting Vocabulary to Downstream Domain. In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. pp. 4692–4700. Association for Computational Linguistics, Online and Punta Cana, Dominican Republic (Nov 2021). https://doi.org/10.18653/v1/2021.emnlp-main.385

13. Ison, J., Rapacki, K., Ménager, H., Kalaš, M., Rydza, E., Chmura, P., Anthon, C., Beard, N., Berka, K., Bolser, D., Booth, T., Bretaudieu, A., Brezovsky, J., Casadio, R., Cesareni, G., Coppens, F., Cornell, M., Cuccuru, G., et al.: Tools and data services registry: a community effort to document bioinformatics resources. Nucleic Acids Research 44(D1), D38–D47 (Jan 2016). https://doi.org/10.1093/nar/gkv1116, https://doi.org/10.1093/nar/gkv1116

14. Istrate, A.M., Li, D., Taraborelli, D., Torkar, M., Veytsman, B., Williams, I.: A large dataset of software mentions in the biomedical literature (Sep 2022), http://arxiv.org/abs/2209.00693

15. Kieser, S., Brown, J., Zdobnov, E.M., Trajkovski, M., McCue, L.A.: ATLAS: a Snakemake workflow for assembly, annotation, and genomic binning of metagenome sequence data. BMC Bioinformatics 21(1), 257 (Jun 2020). https://doi.org/10.1186/s12859-020-03585-4, https://doi.org/10.1186/s12859-020-03585-4

16. Köster, J., Rahmann, S.: Snakemake - a scalable bioinformatics workflow engine. Bioinformatics (Oxford, England) 28, 2520–2 (Aug 2012). https://doi.org/10.1093/bioinformatics/bts480

17. Liu, Z., Jiang, F., Hu, Y., Shi, C., Fung, P.: NER-BERT: a pre-trained model for low-resource entity tagging. arXiv preprint arXiv:2112.00405 (2021)

18. Naguib, M., Tannier, X., Névéol, A.: Few-shot clinical entity recognition in English, French and Spanish: masked language models outperform generative model prompting. In: Al-Onaizan, Y., Bansal, M., Chen, Y.N. (eds.) Findings of the Association for Computational Linguistics: EMNLP 2024. pp. 6829–6852. Association for Computational Linguistics, Miami, Florida, USA (Nov 2024), https://aclanthology.org/2024.findings-emnlp.400
19. Neves, M., Ševa, J.: An extensive review of tools for manual annotation of documents. Briefings in bioinformatics **22**, 146–163 (Dec 2019). https://doi.org/10.1093/bib/bbz130

20. NLM: MEDLINE, PubMed, and PMC (PubMed Central): How are they different? (2021), https://www.nlm.nih.gov/bsd/difference.html

21. Pan, H., Zhang, Q., Dragut, E., Caragea, C., Latecki, L.J.: DMDD: A Large-Scale Dataset for Dataset Mentions Detection. Transactions of the Association for Computational Linguistics **11**, 1132–1146 (Sep 2023). https://doi.org/10.1162/tacl_a_00592, https://doi.org/10.1162/tacl_a_00592

22. Sebe, C., Cohen-Boulakia, S., Ferret, O., Névéol, A.: Extraction d'entités nommées décrivant des chaînes de traitement bioinformatiques dans des articles scientifiques en anglais. In: Balaguer, M., Bendahman, N., Ho-dac, L.M., Mauclair, J., G Moreno, J., Pinquier, J. (eds.) Actes de la 31ème Conférence sur le Traitement Automatique des Langues Naturelles, volume 1 : articles longs et prises de position. pp. 422–434. ATALA and AFPC, Toulouse, France (Jul 2024), https://aclanthology.org/2024.jeptalnrecital-taln.30/

23. Stenetorp, P., Pyysalo, S., Topic, G., Ohta, T., Ananiadou, S., Tsujii, J.: brat: a Web-based Tool for NLP-Assisted Text Annotation. In: Proceedings of the Demonstrations at the 13th Conference of the European Chapter of the Association for Computational Linguistics. pp. 102–107 (Apr 2012)

24. da Veiga Leprevost, F., Grüning, B.A., Alves Aflitos, S., Röst, H.L., Uszkoreit, J., Barsnes, H., Vaudel, M., et al.: BioContainers: an open-source and community-driven framework for software standardization. Bioinformatics **33**(16), 2580–2582 (Aug 2017). https://doi.org/10.1093/bioinformatics/btx192, https://doi.org/10.1093/bioinformatics/btx192

25. Verspoor, K., Jimeno-Yepes, A., Cavedon, L., McIntosh, T., Herten-Crabb, A., Thomas, Z., Plazzer, J.P.: Annotating the biomedical literature for the Human Variome. Database : the journal of biological databases and curation **2013**, bat019 (Jan 2013). https://doi.org/10.1093/database/bat019

26. Wajsbürt, P.: Extraction et normalisation d'entités simples et structurées dans les documents médicaux. Thèse de doctorat, Sorbonne université (Dec 2021), https://www.theses.fr/2021SORUS541

27. Yates, J.A.F., Lamnidis, T.C., Borry, M., Valtueña, A.A., Fagernäs, Z., Clayton, S., Garcia, M.U., Neukamm, J., Peltzer, A.: Reproducible, portable, and efficient ancient genome reconstruction with nf-core/eager. PeerJ **9**, e10947 (Mar 2021), https://peerj.com/articles/10947

28. Zaratiana, U., Tomeh, N., Holat, P., Charnois, T.: GLiNER: Generalist model for named entity recognition using bidirectional transformer. In: Duh, K., Gomez, H., Bethard, S. (eds.) Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). pp. 5364–5376. Association for Computational Linguistics, Mexico City, Mexico (Jun 2024). https://doi.org/10.18653/v1/2024.naacl-long.300, https://aclanthology.org/2024.naacl-long.300/