dataesr/scientific-paragraphs-categorization huggingface dataset:
Impact of OCR errors on NER All experiments we will perform in this article are exposed in  Hamdi et al. (2020). The first one was conducted over clean corpora. Then, the same experiment was run on the OCRed datasets. We evaluated four NER systems, one machine learning-based system and three deep learning-based systems. As results showed that deep learning-based systems have very similar performances and outperform the machine learning-based system, we reuse in this paper the results obtained by the BiLSTM-CNN-CRF model k  (Ma and Hovy 2016). The model uses a forward LSTM and a backward LSTM that encode the left and right contexts, respectively. The forward and backward LSTM pair is referred to as a bidirectional LSTM (BiLSTM). Then, a CRF layer generates the most probable sequence of predicted labels from surrounding words.
7). In order to analyze the impact of OCR degradation on the NEL task, experiments were conducted using the system of  Ganea and Hofmann (2017)  on OCRed datasets. Ganea and Hofmann proposed a deep learning model l that represents entities and words in a common vector space. l The model is publicly available: https://github. com/dalab/deep-ed Their model uses a neural attention mechanism over local context windows and a conditional random field that collectively disambiguates the mentions in a document. Their model was pre-trained using the word embedding model Word2Vec with vectors of 300 dimensions from the Wikipedia corpus published in February 2014. Then, it was trained on the AIDA-CoNLL dataset  (Hoffart et al. 2011). Table  3  shows the performance of the Ganea and Hofmann's system (Ganea and Hofmann 2017) on datasets with multiple levels of OCR quality, in terms of micro-averaged F1 scores.
It can be seen that the CER and WER generated by OCR degradation are globally correlated to the performance of NEL. While the LEV-0 and bleed degradation had the lowest CER and WER levels and generated the least impact on the F1 scores, character degradation produced larger CER and WER values and caused a bigger drop in the NEL performance. In contrast to the effect on the NER datasets, the blur degradation had a similar impact on the NEL performance as the bleed degradation. Among all datasets, ACE2004 was the most affected by OCR degradation, with a drop of almost 20 percentage points for the LEV-MIX degradation. The combination of the OCR degradations (LEV-MIX) generated the highest CER and WER. Despite high error rates, this does not appear to have a significant impact on the NEL performance (decrease rate on the F1-scores up to 11%).
In-depth OCR error analysis This section contains a key contribution of this paper. We propose an in-depth analysis of OCR errors and their impact on NER and NEL. Based on the conclusions from our analysis, one can observe that some OCR errors have a stronger impact on the tasks of named entity recognition and linking. Noting this is notably useful for the implementation of effective OCR post-processing approaches related to NER and NEL. As discussed in Section 3. 2 and detailed by  Hamdi et al. (2020)  and Linhares Pontes et al. (  2019 ), the impact of OCR quality on NER and NEL is very similar regardless of the system or the dataset used on each task. For NER, all the systems achieve satisfying results when the OCR error rates are between 1% and 5% at the character level and 8% and 15% at the word level.
Similarly all NEL systems reach good results when the CER is less than 7% and the WER does not exceed 17%. From these rates, NER and NEL are harmed by OCR errors. For these reasons, we use for each task one system and one dataset including all the noisy versions (Section 3). However, the BiLSTM-CNN-CRF model and the system of  Ganea and Hofmann (2017)  showed slightly better results for NER and NEL respectively. We used the outputs of these systems to conduct our analysis. In terms of corpora, we used the English CoNLL-03 corpus for NER and its extension the AIDA corpus for NEL. Indeed, they are mostly used to evaluate state-of-the-art systems for the two tasks separately rather than through an end-to-end approach. Our in-depth analysis aims at covering the most relevant phenomena that define good strategies for post-OCR correction (edit operations, Levenshtein distance, length effects, first character errors and segmentation errors).
Recognition and linking of contaminated named entities Contaminated named entities are named entities that are wrongly recognized from the image by the OCR process. As we mentioned earlier, NER and NEL systems are able in some cases to face OCR errors and to correctly recognize and link NEs. However, in many cases NER and NEL systems fail to overcome OCR errors. To the best of our knowledge, all post-OCR research focused on general words  Nguyen et al. (2021). No previous research study has been conducted on the analysis OCR errors over named entities. In order to study post-OCR on named entities, we have first extracted the contaminated entities in all the noisy versions of our datasets. Then, we have identified among them the NEs correctly recognized/linked and those wrongly recognized/linked in order to analyse them and identify OCR aspects that impacted the effectiveness of NER and NEL systems.
Among the degraded entities, the main OCR errors in the AIDA dataset are: deletion of characters ("outh Korea" and "South Korea"), accent ("Jerusalém" and "Jerusalem"), uppercase character ("indonesia" and "Indonesia", and "United states" and "United States"), and punctuation ("Australia. " and "Australia"). Unsurprisingly, the more NEs are contaminated, the more NER results are degraded. Things are less clear with NEL. For example, the phantom degradation affected more NEs than the bleedthrough. However, more contaminated NEs are correctly linked in data with phantom degradation. In addition, all the contaminated NEs in the blurred data are successfully linked despite the number of contaminated NEs compared to other degraded data. These observations show that an additional analysis is required in order to understand different OCR aspects and their impact on NER and NEL.
Length effects It was already observed that shorter words are more affected by OCR engines  (Kukich 1992). This section aims to examine this aspect on NE tokens. In practice, the length of OCRed tokens may differ from the actual length of the tokens in the ground truth. For example, the OCRed token "Japgfl" of length 6 comes from the ground truth (GT) word "Japan" of length 5. We therefore analysed the effect of length over the OCRed tokens, since post-OCR algorithms focus on those rather than on the GT tokens. To do so, we first categorized contaminated NEs according to the length of their tokens and then identified among them those that were correctly tagged by the NER and the NEL systems. Counts of correct/incorrect NE recognized according to their lengths on our datasets are shown in Figure  2.
The analysis of OCRed token lengths (see Figure  2 ) shows two main findings. First, the vast majority of contaminated tokens are of length between 4 and 10 (about 78%). Second, most tokens outside this interval are correctly recognised despite OCR errors. We therefore suggest a post-OCR correction of tokens of lengths between 4 and 10. Contrary to NER, the length effect does not show a significant impact on NEL. The model is mostly able to overcome OCR errors regardless of the entity length. Table  5  shows the number of noisy entities in the AIDA dataset that are correctly (or incorrectly) linked according to their lengths.
Levenshtein distance The Levenshtein distance is a measure of the difference between two strings. It considers the edit distance required to convert a string into another one based on classical edition operations (insertion, deletion and substitution). With it, we can estimate the degree of modification that the OCR degradation generated compared to the clean version and how this degradation affected the performance of NER/NEL systems. Percentages of errors based on Levenshtein distances (edit distances) of our datasets are shown in Figure  3. Depending on edit distance, there are singleerror tokens (e. g. "Spain" vs. "Span") and multi-error tokens with edit distances higher than 1 (e. g. "Spain" vs. "Syin"). Mitton (1987)  reported that single-error words largely exceed multierror words in OCR outputs.
First character errors In named entities the first character generally has more importance than the other characters, for instance when it is a capital letter. This is illustrated by the fact that early systems for named entity extraction only focused on capital letters  (McDonald 1993; Mikheev 1999). These systems based on capital letters were using them to identify and delimit named entities (in English). Each word (or sequence of words) which did not occur in an ambiguous position (such as the beginning of a sentence, or in a capitalized title) with a capitalized first letter was considered as a named entity. However, with OCR degradation, upper and lower case characters may be mixed up. NER results are, therefore, very impacted by case modifications (e. g. Apple vs apple). Mitton (1987)  described that 7% of the misspellings of his dataset appeared at the first character.
Segmentation errors Also known as spacing errors, they occur in two cases: • over-segmentation: when a word is split into several words (generally due to different text alignments and spacing). • under-segmentation: when multiple words are wrongly joined. It is worth noting that over-and under-segmentation can occur simultaneously. In the case of NEs, segmentation errors occur when a white space character is omitted between words in multiword NEs or when white space characters are erroneously inserted between two characters in at least one token of a named entity. Table  9  shows the percentages of NEs impacted by segmentation errors on the CoNLL-03 corpus. The impact is dramatic as only 19. 25% of the contaminated NEs are correctly recognized. All types of degradation in the AIDA dataset generated a few cases of segmentation errors that were correctly disambiguated by the NEL system.
2 of the paper (blurring, bleeding effect, phantom and character degradation) as well as the mix of all degradations. In order to compare the performance of the NEL system in the disambiguation-only (Section 3. 2. 2) and the end-to-end scenarios, we used our NER approach (BiLSTM-CNN-CRF) to recognise the entities on the AIDA dataset and then Ganea and Hofmann's approach to disambiguate them to a KB. Table  10  shows the performance evolution between the NEL system only (from Section 3. 2. 2) and the end-to-end combination for each OCR degradation. Table  3  shows the impact of the propagation of OCR errors from NER to NEL. For comparative reasons we report the results of the NEL-only experiments. Despite the good NEL performance in the NEL-only scenario for all versions on the AIDA dataset, we can observe that the combination of NER and NEL caused a performance drop of 12%.
Among all OCR versions, the LEV-MIX achieved the worst results. 10. F-scores values of the OCR impact on the NEL-only and the end-to-end NEL performances in the AIDA dataset, and its associated loss. As Table  10  shows, the NEL-only model unsurprisingly outperforms the end-to-end model. Linking the output of the NER system (end-to-end) is more complicated than disambiguating GT named entities annotated in the clean version. In the noisy versions, it is clear that NER errors on OCRed data are impacting the NEL process, leading to a performance drop of the end-to-end model compared to the NEL-only model. Nevertheless, despite this drop, we can consider that the end-to-end NEL pipeline achieved good results (almost 0. 8 m in F-scores), which shows that the combination of NER and NEL systems can provide satisfying results for OCRed documents even without post-OCR correction.
Discussion To perform our analysis, the selection of the dataset was thus intended to target and exhaust a dataset with variable OCR types and a realistic number of errors. The spectrum of the analysis is meant to be broad in the types of degradation rather than in the amount of data processed. As shown in Section 3, the CoNLL-03 and AIDA corpora, which are frequently used in NER and NEL tasks, contain diverse types of OCR errors, and the five aspects analysed in this section are sufficiently distributed in the noisy versions of each of them. As a matter of fact, OCR error rates in the CoNLL-03 corpus vary from 2% to 7% at the character level and from 8% to 23% at the word level whereas in the AIDA corpus the character error rate is between 1% and 5% and the word error rate is between 4% and 18% (cf.
We selected a dataset that, on the one hand, has annotations for both tasks, and on the other hand, contains variable OCR error types. The study covers 5 types of OCR errors and the analysis of the impact on NER and NEL output led to many interesting findings. The length effects demonstrated that most contaminated NEs, which the NER/NEL systems fail to recognise/link, are of length between 5 and 15 characters. The edit distance analysis showed that most of the contaminated NEs contain either singleor double-character errors. Post-OCR techniques should therefore be able to fix about 81. 49% of the impacted named entities with an edit distance threshold of 2. Moreover, our observations showed that character deletion errors are the most easily overcome by NER/NEL systems contrary to character insertion and substitution errors.
Post-OCR algorithms are recommended to primarily m Performances are comparable to efficient state-of-the-art end-to-end systems on the AIDA dataset  (Kolitsas et al. 2018; van Hulst et al. 2020; Ravi et al. 2021)  favor the correction of character substitutions, then insertions. For NEL, post-OCR can be limited to correct substitutions, as the edit operation analysis showed that the NEL system is able to deal with errors generated from deletion and insertion. When it comes to the position of erroneous characters, our observations showed that NER systems are particularly vulnerable to errors made with the first character notably when the case is changed. For NEL, however, case sensitivity has little impact, while the substitution of the first character did. Finally, the analysis showed that segmentation errors had a very strong impact on NER performance.
(Guo and Barbosa 2014)ata that was used for the NER task. This dataset is divided into AIDA-train for training, AIDA-A for validation, and AIDA-B for testing. This dataset contains 1,393 Reuters news articles and 27,817 linkable mentions. •AQUAINTdataset (Guo and Barbosa 2014) is composed of 50 short news documents (250-300 words) from the Xinhua News Service, the New York Times, and the Associated Press. This dataset contains 727 mentions. •ACE2004 dataset (Guo and Barbosa 2014) is a subset of the ACE2004 coreference documents with 57 articles and 306 mentions, annotated through crowdsourcing. •MSNBC dataset(Guo and Barbosa 2014)is composed of 20 news articles from 10 different.
(Guo and Barbosa 2014)ata that was used for the NER task. This dataset is divided into AIDA-train for training, AIDA-A for validation, and AIDA-B for testing. This dataset contains 1,393 Reuters news articles and 27,817 linkable mentions. •AQUAINTdataset (Guo and Barbosa 2014) is composed of 50 short news documents (250-300 words) from the Xinhua News Service, the New York Times, and the Associated Press. This dataset contains 727 mentions. •ACE2004 dataset (Guo and Barbosa 2014) is a subset of the ACE2004 coreference documents with 57 articles and 306 mentions, annotated through crowdsourcing. •MSNBC dataset(Guo and Barbosa 2014)is composed of 20 news articles from 10 different topics (two articles per topic: Business, U. S. Politics, Entertainment, Health, Sports, Tech & Science, Travel, News).
The workshop grew from a previous edition organised during  [BC] 2 2019 focused on logical modelling  [1]. The second edition brought together scientists with various research backgrounds and from different working groups such as BioModels  [2] , a central repository of mathematical models of biological/biomedical processes; the Computational Modelling in Biology Network initiative (COMBINE)  [3] ; CoLoMoTo,  [4] ; SysMod,  [5] ; the Systems Biology Graphical Notation (SBGN) project  [6] ; the systems biology markup language (SBML)  [7]  and simulation experiment description markup language (SED-ML)  [8] , to exchange and expand on several key topics of common interest (Figure  1 ). While the modelling approaches across these communities differ, several critical points are shared, such as (i) the importance of annotations for reproducibility, (ii) the use of community standards for exchange and annotation encoding, (iii) the need to implement standards in tools and platforms to boost reusability and interoperability, (iv) the importance of transparency of modelling frameworks in publications and (v) the use of shared repositories to enhance model accessibility (Figure  2 ).
Efforts to formalise process description (PD) diagrams using SBGN schemes  [6]  have been made within the Disease Maps project (https://disease-maps. org/)  [13]  and, more recently, within the COVID-19 Disease Map community  [14]. Tools to automate the translation of Disease Maps into executable Boolean models, such as the CellDesigner as SBML-Qual (CaSQ) tool, have eased the creation of models in the SBMLqual format  [15, 16]. This initiative has demonstrated the absolute need for coordinated efforts among different systems biology communities to establish best practices for model curation and annotation  [1]. The use of standards, especially SBGN  [6] , reduces uncertainty and disambiguates knowledge representation. SBGN includes three complementary and orthogonal visual languages which represent alternative projections of the underlying biological system: (a) the SBGN PD language  [17] , which shows the sequence of interactions between biochemical entities on a mechanistic level; (b) the SBGN Entity Relationship language  [18]  that represents the relationships in which an entity can participate regardless of the sequential order of these events and also shows information on a mechanistic level and (c) the SBGN Activity Flow language  [19]  that depicts influences between entities in a network on a more abstract level.
Tools are needed to implement the three SBGN languages in a straightforward, user-friendly way. An example is SBGN-ED  [20] , an editor capable of handling all three SBGN languages. SBGN-ED allows the user to check for correctness using rule-based methods, combine SBGN bricks to build networks, translate diagrams between formats  [21] , apply automated layouts to improve the readability of maps  [22] [23] [24] , and perform data integration tasks such as converting high-throughput (omic) data into pathway maps  [25]. Such pathway maps can also be imported in Cytoscape  [26]  and analysed as graphs for their topology, revealing important structural properties such as various centrality measurements, in and out degrees, and network connectivity. An important aspect for model reusability is the capability to merge and combine existing models.
MSMB supports multistate models created using different modelling styles and is based on Java and COmplex PAthway SImulator (COPASI)  [32]  APIs. Simulink, a MATrix LABoratory (MATLAB) toolbox, is a block-diagram environment for multidomain simulation and model-based design. It includes a merge functionality allowing the merger of two versions of a model  [33]. Recent developments of SBML have enabled the support of multiple modelling methodologies  [7]. SBML is a powerful language whose syntax supports annotations and covers various modelling frameworks, including rule-based, logical, spatial, kinetic methodologies and multiple types of analysis, including time-course sim-ulations, parameter estimation, sensitivity analysis, flux balance analysis and visualization. However, one key field challenge that remains to be addressed is to capture both the mathematics and the semantics that define models and simulations in an implementationindependent manner.
They also used the model checker New Symbolic Model Verifier -Action Restricted Computation Tree Logic (NuSMV-ARCT)  [54]  to perform model checking analysis and finally MaBoSS for stochastic simulations  [55]. While not all tools use standard formats, they can produce compatible, intermediate files that can be imported and analysed further. However, adopting standardised practices through shared libraries and open-source code on how tools handle model building and analysis could significantly accelerate community-driven software updates and the development of multifunctional and seamless analytical pipelines. To support this adoption, the CoLoMoTo interactive notebook  [56]  is a communitydriven effort to improve reproducibility and reusability within the subdomain of logical models and software tools.
It includes a knowledge base for information about individual biological processes, such as identifying direct protein-protein interactions. The tool also includes a reference section where users can enter references using PubMed IDs. Finally, the tool provides a large model repository of Boolean and, recently, constraintbased models, and includes analytical methods for both modelling approaches. With similar goals, another interface, WebMaBoSS  [62] , was developed for simulating models with the Markovian Boolean stochastic simulator (MaBoSS) tool. The MaBoSS software is based on stochastic simulations of Boolean networks  [55] : transition rates are associated with each node of the model, and probabilities for the states of the model are computed over time. In addition, the framework was expanded with an easy-to-use web interface.
New models can be loaded from local files or imported directly from existing databases, including BioModels (https://www. ebi. ac. uk/ biomodels/), Cell Collective (https://cellcollective. org/) and GINsim (http://ginsim. org/models_repository) repositories. Models are then stored in a personal database and modified, keeping track of all versions. This new interface comes with an update of MaBoSS, which improves its compatibility with community formats by adding native support of SBML-qual  (16)  and BoolNet  [63]  model representations.
Transparency through proper documentation across modelling frameworks The need for curation guidelines and standard formats for archiving information also applies to metabolic models. Genome-scale metabolic models describe an organism's metabolism and need to be integrated when studying microbial communities at the metabolic level. Building comprehensive models requires integrating all relevant information from multiple biological databases with different data types. Data integration is not straightforward, though, and choices of data and analytical tools need to be made. Furthermore, for models to be fully exploited by the community, reconstruction steps must be well documented and described as metadata in a standardised workflow language. These metadata allow other researchers to reproduce and improve models or reuse models as blueprints to create new ones.
Recent efforts include the dedicated padmet format to generate customised metabolic models through transparent reproducible pipelines  [64]. Moreover, the automatic generation of local wikis and query-based interfaces facilitates the exploration of models and metadata  [65]. The same issues apply to the construction of multiscale models. Currently, there is limited interoperability of virtual-tissue simulation environments as they often handle one methodology per scale. In addition, standards for describing cellular and multicellular experiments and models are lacking. There is a clear need to develop crossscale integration languages and ontology-based model specification languages. CompuCell3D is an open-source simulation platform that allows tissue-scale models to integrate easily with existing physiologically based pharmacokinetic models, with subcellular models of biological networks and with artificial intelligence (AI)  [66].