dataesr/scientific-paragraphs-categorization huggingface dataset:
Conclusion In this study, I explored sparse annotation-based training of deep learning models for cell image segmentation. Widely used cell segmentation algorithms, StarDist and Cellpose, were made compatible with sparse annotation. I trained the models of the modified versions of StarDist and Cellpose, and ELEPHANT's segmentation module with sparse annotations generated under multiple conditions, including three strategies: (i) dim nuclei preferentially, (ii) bright nuclei preferentially, and (iii) both mixed, and five annotation percentages ranging from 4. 6% to 100%. I showed that for training StarDist and Cellpose models, background information is essential. I found that recall and precision scores changed depending on the annotation strategy; prioritizing dim nuclei produced higher recall scores, and prioritizing bright nuclei produced higher precision scores.
Methods Adaptation of StarDist and Cellpose algorithms to sparse annotation. StarDist and Cellpose are widely used algorithms, but require densely annotated data for training with custom data. I adapted these algorithms to sparse annotation with the same idea as implemented in ELEPHANT. The modification is very simple: the loss function is calculated by ignoring unannotated pixels in the training phase. These changes can be found in the following GitHub repositories (https://github. com/ ksugar/stardist-sparse and https://github. com/ksugar/cellpose-sparse). Dataset preparation. I used a dataset from the StarDist paper  (1). The dataset contains 447 training data and 50 validation data, which are a subset of the Data Science Bowl 2018, an annotated collection of 2D nuclear images acquired by fluorescence microscopy.
Acknowledgements I am grateful to  Michalis Averof  ( IGFL, CNRS ) in whose lab this work was initiated and carried out, and to  Shuichi Onami  ( RIKEN BDR ) in whose lab part of this work was carried out. I thank  Constantin Pape  ( Georg-August University Göttingen ) for the helpful discussion;  Michalis Averof  for feedback on the manuscript. This research was supported by the  European Research Council , under the  European Union   Horizon 2020 programme , grant  ERC-2015-AdG  # 694918 , and by the  Agence Nationale de la Recherche of France , grant  ANR-21-CE13-0044-01.
Code availability The implementation of StarDist with support for sparse annotation is available at https://github. com/ksugar/ stardist-sparse. The implementation of Cellpose with support for sparse annotation is available at https: //github. com/ksugar/cellpose-sparse. Programs for training and evaluation of StarDist, Cellpose, and ELEPHANT with sparse annotations are available at https://github. com/ksugar/ cellsparse-core, https://github. com/ ksugar/cellsparse-api and https://github. com/ksugar/qupath-extension-cellsparse. The implementation of SAM for QuPath is available at https://github. com/ksugar/ samapi and https://github. com/ksugar/ qupath-extension-sam.
Acknowledged some twenty years ago, NE processing has undergone major evolution since then, from entity recognition and classification to entity disambiguation and linking, and is representative of the evolution of information extraction from a document-to a semantic-centric view point  [156]. As for most NLP research areas, recent developments around NE processing are dominated by deep neural networks and the usage of embedded language representations  [37, 110]. Since their inception up to now, NE-related tasks are of ever-increasing importance and at the core of virtually any text mining application. From the NLP perspective, NE processing is useful first and foremost in information retrieval, or the activity of retrieving a specific set of documents within a collection given an input query.
NER in a nutshell 3. 2. 1 A sequence labelling task. Named entity recognition and classification is defined as a sequence labelling task where, given a sequence of tokens, a system seeks to assign labels (NE classes) to this sequence. The objective for a system is to observe, in a set of labelled examples, the word-labels correspondences and their most distinctive features in order to learn identification and classification patterns which can then be used to infer labels for new, unseen sequences of tokens. This excerpt from the CoNLL-03 English test dataset  [190]  illustrates a training example (or the predictions a system should output): ( Such input is often represented with the IOB tagging scheme, where each token is marked as being at the beginning (B), inside (I) or outside (O) of an entity of a certain class  [155].
1% F-score on the NER CoNLL-03 English dataset. Soon, BiLSTM networks became the de facto standard for context-dependent sequence labelling, giving rise to a body of work including Lample et al. [110] , Chiu et al. [110] , and Ma et al. [121]  (to name but a few). Besides making use of bidirectional variants of RNN, these work also experiment with various input representations, in most cases combining learned character-based representations with pre-trained word embeddings. Character information has proven useful for inferring information for unseen words and for learning morphological patterns, as demonstrated by the 91. 2% F-score of Ma et al. [121]  on CoNLL-03, and the systematically better results of Lample et al. [110]  on the same dataset when using character information. A more recent study by Taillé et al.
8% NER F-score on CoNLL-03. More recently,  Yamada et al. [201]  proposed an entity-aware self-attention architecture which achieved 94. 3% F-score on the same dataset. Transformer-based architectures are the focus of extensive research and many model variants were proposed, of which Tay et al. [187]  propose an overview. Overall, two points should be noted. First, that beyond the race for the leader board (based on the fairly clean English CoNLL-03 dataset), pre-trained embeddings and language models play a crucial role and are becoming a new paradigm in neural NLP and NER (the 'NLP's ImageNet moment'  [167] ). Second, that powerful language models are also paving the way for transfer learning, a method particularly useful with low-resource languages and out-of-domain contexts, as is the case with challenging, historical texts.
CHALLENGES Named entity recognition on historical documents faces four main challenges for which systems developed on contemporary datasets are often ill-equipped. Those challenges are intrinsic to the historical setting, like time evolution and types of documents, and endemic to the text acquisition process, like OCR noise. This translates into a variable and sparse feature space, a situation compounded by the lack of resources. This section successively considers the challenges of document type and domain variety, noisy input, dynamics of language, and lack of resources.
Finally, in terms of time periods covered, we observe a high concentration of corpora in the 19C, directly followed by 20C and 21C, while corpora for previous centuries are either scarce or absent. 9 Inventory as of June 2021. The Voices of the Great War corpus  [27]  is not included for not released under an open license. 10 For comparison, the CoNLL-03 dataset contains ca. 70k mentions for English and 20k for German  [190] , while OntoNotes v5. 0 contains 194k mentions for English, 130k for Chinese and 34k for Arabic  [151]. guidelines  [54] , which are based on and are retro-compatible with the Quaero guidelines  [166]. The annotation tag set comprises 5 coarse-grained and 23 fine-grained entity types, and includes entity components as well as nested entities. Wrongly OCRed entity surface forms are manually corrected and entities are linked towards Wikidata.
NERC and EL annotations reached an average IAA across languages of 0. 8 (Krippendorf's alpha). The NewsEye dataset 14   [81]  is a large-sized corpus composed of articles extracted from newspapers published between mid 19C and mid 20C in French, German, Finnish, and Swedish. Four entity types were considered (person, location, organisation and human product) and annotated according to guidelines  15  similar to the impresso ones; entities are linked towards Wikidata and articles are further annotated with authors' stances. The annotation reaches high IAAs exceeding 0. 8 for Swedish and 0. 9 for German, French and Swedish (Cohen's kappa). The Quaero Old Press Extended NE corpus  16  [165] is a very large annotated corpus composed of 295 pages sampled from French newspapers of December 1890.
Due to the large time span covered, texts adhere to different orthographic conventions. Named entity annotations correspond to locations, organisations, works, events and miscellaneous entities, and were automatically produced (silver annotations). They constitute only one of the many layers of linguistic annotations of this corpus, alongside POS tags, syntactic analysis and semantic roles. Next, the LitBank 20   [12]  dataset is a medium-sized corpus composed of 100 English literary texts published between mid 19C and beginning 20C. Entities were annotated following the ACE guidelines-with the only exception of weapons as rarely attested-and include noun phrases as well as nested entities. Finally, the Deutsches ROman Corpus (DROC)  [106]  is a set of 90 richly-annotated fragments of German novels published between 1650 and 1950.
Each annotated text is available in its original version, as well as with automatic and manual historical spelling normalization. Despite its limited size, this corpus can be a valuable resource for researchers investigating the effects of historical normalisation on NER. Finally, moving from literature to scholarly literature, three corpora should be mentioned. First, BIOfid  23  [2] is a large NE-annotated corpus composed of ca. 1000 articles sampled from German books and scholarly journals in the domain of biodiversity and published between 18C and 20C. The annotation guidelines used for this corpus build upon those used for the GermEval dataset  [18] , with the addition of time expressions and taxonomies (Taxon), i. e. systematic classifications of organisms by their characteristics (e.
[87]  published a collection of static (word2vec, fastText) and contextualised embeddings (flair) trained on the Microsoft British Library (MBL) corpus. MBL is a large-scale corpus composed of 47,685 OCRed books in English (1760-1900) which cover a wide range of subject areas including philosophy, history, poetry and literature, for a total of approximately 5. 1 billion tokens. For each architecture, authors released models trained either on the whole corpus or on books published before 1850. Word-level LM embeddings trained on historical data are available for Latin, French, German and English. Latin BERT is a LM for Latin trained on 640 million tokens spanning 22 centuries. 33  In order to reach a sufficiently large volume of training material, a wide variety of datasets was employed including the Perseus Digital Library, the Latin Wikipedia (Vicipaedia), and Latin texts of the Internet Archive.
Extrinsic evaluation of the model was performed on POS tagging and word sense disambiguation, for which Latin BERT demonstrated state-of-the-art results. For historical German and French, Schweter  [171]  published BERT and ELECTRA models trained on two subsets of the Europeana newspapers corpus, consisting of 8 and 11 billion tokens for German and French respectively. The German models were evaluated on two historical NE datasets, on which the ELECTRA models over-performed the BERT ones, leading to an overall improvement on the current state-of-the-art results reported by Schweter and Baiter  [172]. Finally, for 19C English, BERT-based language models trained on the MBL corpus are available in the histLM model collection  [87]. One model was trained on the entire corpus, and additional models were created for different time slices to enable the study of linguistic and cultural changes over the 19C, by fine-tuning an existing contemporary model (BERT base uncased).
Traditional Machine Learning Approaches Machine learning algorithms inductively learn statistical models from annotated data on the basis of manually selected features (see Section 3. 4. 2). Heavily researched and applied in the 2000s, machine learning-based approaches contributed strong baselines for mainstream NER, and were rapidly adopted for NER on historical documents. In this section we review the usage of such traditional, pre-neural machine learning approaches on historical material, first considering works which apply already existing models, second which train new ones.
Applying existing models. Early achievements adopted the 'off-the-shelf' strategy with the application of pre-trained NER systems or web services to various historical documents, mainly with the objectives of assessing baselines and/or comparing system performances. This is the case of Rodriquez et al. [163] , who compared the performances of four NER systems (Stanford CRF classifier, OpenNLP, AlchemyAPI, and OpenCalais) on two English datasets related to WWII: individual Holocaust survivor testimonies from the Wiener Library of London and letters of soldiers from King's College archive. Evaluated on a small dataset, the recognition of Person, Location and Organization reached an F-score between 47% and 54% for the testimonies (Stanford CRF being the most accurate), and between 32% and 36% for the letters (OpenCalais performing best).
Training models. Other work trained NER systems anew on custom material. Early attempts include the experiments of Nissim et al. [135]  on Location entity type in manually transcribed Scottish parish registers of the late 18C and early 19C. They trained a maximum entropy tagger with its in-built standard features on a dataset of ca. 6000 location mentions and obtained very satisfying performance (94. 2% F-score), which they explained by the custom training data and the binary classification task (location vs non-location). Subsequently, the most frequently used system is the Stanford CRF classifier 35   [63] , particularly on historical newspapers. Working with the press collection of the National Library of Australia, Kim et al. [103]   (slightly) fine-tuned the CRF baseline provided by the organisers and reached 66% on all types (exact match), two points more than the baseline.
Going back in time, Aguilar et al. [1]  experimented NER on manually transcribed Latin medieval charters from the 10C to 13C. Focusing on person and place names, they used dedicated preprocessing and trained a CRF classifier using the Wapiti toolkit. 36 Results are remarkable, on average in the 90% for both types, certainly due to the regularity of the documents in terms of names, naming conventions, context and overall structure. Finally, Passaro et al. [141]  attempted to extract entities from WWI and WWII Italian official war bulletins. They focused on the traditional entity types, plus Military organisations, Ships and Airplanes. The Stanford system was trained (without gazetteers) on semi-automatically annotated data from the two periods as well as on contemporary Italian news, and various experiments mixing in-vs.
out-of-time data were carried out. Results showed that performances are highest when the model is trained on data close in time, that entities of type Location are systematically better recognised, and that custom types (ships, military organisations, etc. ) are poorly recognised. Conclusion on traditional machine learning approaches. Overall, the availability of machine learning-based NER systems that could either be applied as such or trained on new material greatly fostered a second wave of experiments on historical documents. Settings are quite diverse, and so are the performances, but F-scores are usually in the order of 60 -70%, which is significantly lower than those usually obtained on contemporary material (frequently in the 90%). The Stanford CRF classifier is by far the most commonly used, as well as CRF in general.
Approaches based on static embeddings. First attempts are based on state-of-the-art BiLSTM-CRF and investigate the transferability of various types of pre-trained static embeddings to historical material. They all use traditional CRFs as baseline. Focusing on location names in 19-20C English travelogues,  37  Sprugnoli  [178]  compares two classifiers, Stanford CRF and BiLSTM-CRF, and experiment with different word embeddings: GloVe embeddings, based on linear bag-of-words contexts and trained on Common Crawl data  [143] , Levy and Goldberg embeddings, produced from the English Wikipedia with a dependency-based approach [115], and fastText embeddings, also trained on the English Wikipedia but using sub-word information  [21]. Additionally to these pre-trained vectors, Sprugnoli trains each embedding type afresh on historical data (a subset of the Corpus of Historical American English), ending up with 3×2 input options for the neural model.
[108]  hereafter). Experimental settings consist of: two sets of German labelled corpora, with large contemporary datasets (CoNNL-03 and GermEval) and small historical ones (from the Friedrich Temann and Austrian National library); two types of classifiers, CRFs (Stanford and GermaNER) and BiLSTM-CRF; finally, for the neural system, usage of fastText embeddings derived from generic (Wikipedia) and in-domain (Europeana corpus) data. On this base, authors perform three experiments. The first investigates the performances of the two types of systems on the contemporary datasets. On both GermEval and CoNNL, the BiLSTM-CRF models outperform the traditional CRF ones, with Wikipedia-based embeddings yielding better results than the Europeanabased ones. It is noteworthy that the GermaNER CRF model performs better than the LSTM of Lample et al.
[110]  on CoNLL-03, but suffers from low recall compared to BiLSTM. The second experiment focuses on all-corpora crossing, with each system being trained and evaluated on all possible combinations of contemporary and historical corpora pairs. With no surprise, best results are obtained when models are trained and evaluated on the same material. Interestingly, CRFs perform better than BiLSTM in the historical setting (i. e. train and test sets from historical corpora) by quite a margin, suggesting that although not optimised for historical texts, CRFs are more robust than BiLSTM when faced with small training datasets. The type of embeddings (Wikipedia vs. Europeana) plays a minor role in the BiLSTM performance in the historical setting. Ultimately, the third experiment explores how to overcome this neural net dependence on large data with domain adaptation transfer learning: the model is trained on a contemporary corpus until convergence and then further trained on a historical one for a few more epochs.
Results show consistent benefits for BiLSTM on historical datasets (ca. +4 F-score percentage points). In general, main difficulties relate to OCR mistakes and wrongly hyphenated words due to line breaks, and to the Organisation type. Overall, this work shows that BiLSTM and CRF achieve similar performances in a small-data historical setting, but that BiLSTM-CRF outperforms CRF when supplied with enough data or in a transfer learning setting. This first set of work confirms the suitability of the state-of-the-art BiLSTM-CRF approach for historical documents, with the major advantage of not requiring feature engineering. Provided that there is enough in-domain training data, this architecture obtains better performances than traditional CRFs (the latter performing on par or better otherwise).
Suitable for reliable locationbased search but suffering from low recall, this approach is then compared with a BiLSTM-CRF architecture. The neural system is fed with stacked embeddings composed of in-domain contextual string embeddings pre-trained on the alpine corpus concatenated with general-purpose fastText word embeddings pre-trained on web data, and trained on a silver training set containing 28k annotations obtained via the application of the gazetteer-based approach. The model leads to an increase of recall for the most frequent categories, without degrading precision scores (76% F-score). This shows the generalisation capacity of the neural approach in combination with contextsensitive string embeddings and given sufficient training data. Finally, authors experiment with crowd-corrected annotations and observe that already a small number of corrections on the silver data has a positive impact (+3 F-score percentage point).
Swaileh et al. [184]  target even more specific entity types in French and German financial yearbooks from the first half of 20C. They apply a BiLSTM-CRF network trained on custom data and fed with modern flair embeddings. Results are very good (between 85% to 95% F-score depending on the book sections), with the CRF baseline and the BiLSTM model performing on par for French books, and BiLSTM being better than CRF for the German one, which has a lower OCR quality. Overall, these performances can be explained by the regularity of the structure and language as well as the quality of the considered material, resulting in stable contexts and non-noisy entities.
[203]  face the problems of the complexity of classical Chinese and of the absence of appropriate training data in their attempt to recognise Person and Location. Their BiLSTM-CRF model is trained on a NE-annotated modern Chinese corpus and makes use of modern Chinese BERT embeddings in a feature extraction setting (frozen). Evaluated on a (small) dataset representative of the time span of the target corpus, the model achieves relatively good performances (from 72% to 82% F-score depending on the book), with a pretty good P/R balance, better results for Location than for Person, and on recent books. Given the complete 'modern' setting of embeddings and training labelled data, those results shows the benefit of large LM-based embeddings-keeping in mind the small size of the test set and perhaps the regularity of entity occurrences in the material, not detailed in the paper.
Higher gains come with larger labelled data, however the absence of ablation tests hinders the complete understanding of the contribution of the historical part of this labelled data, and the use of two implementation frameworks does not warrant full results comparability. Both Schweter et al. [172]  and Labusch et al. [108]  build on the work of Riedl et al. [158] and try to improve NER performances on the same historical German evaluation datasets, thereby constituting (with HIPE-2020) one of the few sets of comparable experiments. Schweter et al. seek to offset the lack of training data by using only unlabelled data via pre-trained embeddings and language models. They use the Flair framework to train and combine ("stack") their language models, and to train a BiLSTM-CRF model. Their first experiment consists in testing various static word representations, with: character embeddings learned during training, fastText embeddings pre-trained on Wikipedia or Common Crawl (with no sub-word information), and the combination of all of these.
With the objective of developing a versatile approach that performs decently on texts of different epochs without intense adaptation, Labusch et al. [108]  experiment with BERT under different pre-training and fine-tuning settings. In a nutshell, they apply a model based on multilingual BERT embeddings, which is further pre-trained on large OCRed historical German unlabelled data (the Digital Collection of the Berlin State Library) and subsequently fine-tuned on several NE-labelled datasets (CoNLL-03, GermEval, and the German part of Europeana NER corpora). Tested across different contemporary/historical dataset pairs (similar to the all-corpora crossing of  Riedl et al. [158] ), it appears that additional in-domain pre-training is most of the time beneficial for historical pairs, while performances worsen on contemporary ones.