dataesr/scientific-paragraphs-categorization huggingface dataset:
The combination of several task-specific training datasets has positive yet less important impact than BERT pre-training, as already observed by Schweter et al. [172]. Overall, this work shows that an appropriately pre-trained BERT model delivers decent recognition performances in a variety of settings. In order to further improve them, authors purpose to use the BERT large instead of the BERT base model, to build more historical labelled training data, and to improve the OCR quality of the collections. The same spirit of combinatorial optimization drove the work of  Todorov et al. [191]  and  Schweter et al. [173]  in the context of HIPE-2020. Todorov et al. build on the bidirectional LSTM-CRF architecture of Lample et al. and introduce a multi-task approach by splitting the top layers for each entity type.
The best final combination includes fastText and BERT, leading to 65% F-score on coarse literal NER (strict setting). Finally, Boros et al. [25]  also tackled NER tagging for HIPE-2020 in all languages and achieved best results. They used a hierarchical transformer-based model  [196]  built upon BERT in a multitask learning setting. On top of the pre-trained BERT blocks (multilingual BERT for all languages, additionally Europeana BERT for German  43  and CamemBERT for French  [124] ), two task-specific transformer layers were optionally added to alleviate data sparsity issues, for instance out-ofvocabulary words, spelling variations, or OCR errors in the HIPE dataset. A state-of-the-art CRF layer was added on top in order to model the context dependencies between entity tags. For base BERT with a limited context of 512 sub-tokens, documents are too long and newspaper lines are too short for proper contextualization.
What conclusions can be drawn from all this? First, the twenty or so papers reviewed above illustrate the growing interest of researchers and practitioners from different fields in the application of deep learning approaches to NER on historical collections. Second, it is obvious that these many publications also equate with a great diversity in terms of document, system and task settings. Apart from the historical German  [108, 158, 172]  and HIPE papers, most publications use different datasets and evaluation settings, which prevents result comparison; what is more, the sensitivity of DL approaches to experimental settings (pre-processing, embeddings, hyper-parameters, hardware) usually undermines any attempt to compare or reproduce experiments, and often leads to misconceptions about what works and what does not  [202].
ACKNOWLEDGMENTS The work of  Maud Ehrmann  and  Matteo Romanello  was supported by the  Swiss National Science Foundation  under the grants number  CR-SII5_173719  ( Impresso -Media Monitoring  of the Past) and number  PZ00P1_186033  (only for MR). The work of  Ahmed Hamdi ,  Elvys Linhares Pontes  (now employed at the  Trading Central Labs company ) and  Antoine Doucet  was supported by the  European Union 's  Horizon 2020 research and innovation program  under grant  770299  (NewsEye).
It consists of newspaper articles sampled from Swiss, Luxembourgish and American newspaper collections covering a time span of ca. 200 years (1798-2018). OCR quality of the corpus corresponds to real-life setting and varies depending on the digitisation time and preservation state of original documents. The corpus was annotated following the impresso 11 https://github. com/EuropeanaNewspapers/ner-corpora 12 https://digi. kansalliskirjasto. fi/opendata/submit (Digitalia (2017-2019) package). 13  Version 1. 3, https://github. com/impresso/CLEF-HIPE-2020/tree/master/data.
An unresolved issue in contemporary biomedicine is the overwhelming number and diversity of complex images that require annotation, analysis and interpretation. Recent advances in Deep Learning have revolutionized the field of computer vision, creating algorithms that compete with human experts in image segmentation tasks. However, these frameworks require large humanannotated datasets for training and the resulting "black box" models are difficult to interpret. In this study, we introduce Kartezio, a modular Cartesian Genetic Programming-based computational strategy that generates fully transparent and easily interpretable image processing pipelines by iteratively assembling and parameterizing computer vision functions. The pipelines thus generated exhibit comparable precision to state-of-the-art Deep Learning approaches on instance segmentation tasks, while requiring drastically smaller training datasets.
By merging CGP-based IS with CV measures and classical unsupervised machine learning (ML), we designed a highly effective AI solution to extract quantitative information from immunofluorescent (IF) and immunohistochemical (IHC) images. Our CGP-based strategy is: (i) trainable on small datasets, (ii) transparent and interpretable by humans, and (iii) capable of integrating human knowledge. The strategy presented in this manuscript (summarized in Supplementary Fig. 1 ) represents a substantially new conceptual framework for the design of image processing pipelines, and offers a novel approach to tackle the challenge of image analysis in contemporary medical and biological sciences.
Evaluation of Kartezio performance versus state-of-the art Deep Learning models To evaluate the performance of this approach against state-of-the-art DL models, we compared Kartezio to Cellpose  23  , a recently released DL approach for IS in biology with both specialist and generalist models, as well as to two previously released approaches, Mask R-CNN  25  and StarDist  24. As described below, we first evaluated Kartezio on the Cellpose dataset (originally derived from Cell Image Library)  28  and compared it to the reported performance of the Cellpose specialist model (Fig. 2a, b )  23  ; we then assessed both Kartezio and the Cellpose generalist model side-by-side on our own recently published dataset (Fig. 2c, d )  29. In their initial study, Stringer et al. introduced the Cellpose model to perform IS on diverse types of cells or nuclei; this generalist model was trained on over 70,000 segmented objects  23.
The authors also proposed a distinct specialist model, trained on in vitro cultured neurons (individual channels from a representative image are displayed in Supplementary Fig. 2a ; reproduced with permission); this dataset was composed of 89 training images and 11 test images  23, 28. Since Kartezio is a modular framework for generating specialist models, we chose to initially compare our approach to the Cellpose specialist model, and accordingly utilized the 89-image training dataset and 11-image test dataset previously used in the generation of the Cellpose specialist model  23, 28. We utilized randomly selected subsets of these 89 images (ranging from 1 to 89 images) to train Kartezio from scratch and subsequently evaluated its performance on the previously unseen 11-image Cellpose test dataset.
Our results also show that the best score obtained by Kartezio was 0. 89, a value comparable to the reported best performance of Cellpose (0. 91). Of note, increasing the size of the training set in 10-image increments from 30 to 80 images did not perceptibly change Kartezio's performance. The capacity of Kartezio to extrapolate from only a few training events was also evident when we compared the scores obtained on training subsets with those obtained on the 11-image test dataset. In this analysis, a score of 0 is optimal as it shows no decrease in accuracy when transitioning from the training to the test set. As shown in Fig. 2b , the difference of mean average precision (AP) between the training and test sets was close to 0 when only 9-10 images were used for training. These data indicate that Kartezio generates pipelines of comparable precision to state-of-the-art DL approaches but with a drastic reduction in the size of the training dataset.
Interestingly, Kartezio generalized well on this dataset as the training versus testing scores were comparable (Supplementary Table  2 ). In a second exercise, we compared the performance of Kartezio to the Cellpose generalist model (CPx) in the context of identifying individual melanoma nuclei within complex real world histopathology images (Fig. 2c, d ). The original dataset, consisting of tissue samples from melanoma patients and the associated clinical data, has been previously described  29. In the present study, we employed 12 histopathology images for Kartezio training, and a further 10 images for Kartezio and Cellpose side-by-side comparison (cropped as shown in Fig. 2d ). As depicted in Fig. 2c , we also used this opportunity to evaluate different non-evolvable nodes in the Kartezio pipeline: RGB versus HSV versus HED in the pre-processing node, and Marker Controlled Watershed (MCW) versus No Endpoint in the endpoint node (n = 10 experiments per condition).
Individual RGB, HSV, and HED channels are shown in Supplementary Fig. 2b. All six CGP-based approaches significantly outperformed CPx (AP50 = 0. 605) on this dataset (RGB MCW p = 0. 000030, t = 7. 697; RGB No Endpoint p = 0. 000049, that creates an intentional bottleneck to drive the heuristics optimization. While the Endpoint is part of the evaluation of the genotype, it is not subjected to evolution and is chosen by the user depending on the task to solve. t = 7. 236; HSV MCW p = 0. 000018, t = 8. 196; HSV No Endpoint p = 0. 000015, t = 8. 369; HED MCW p = 0. 000003, t = 13. 30; HED No Endpoint p = 0. 00695, t = 3. 479, by one-sample t-test). Furthermore, the addition of the MCW endpoint significantly improved Kartezio's performance (F = 5. 649; p = 0. 0003, 1-way ANOVA) compared to the No Endpoint control, when used in combination with either HSV (p = 0.
Both explicability and accessibility are key features in clinical settings. We then illustrate three additional Use Cases in which Kartezio successfully performs IS of biological entities (from nanoscale extracellular particles to microscale immune cells), again based on very few annotations. All the images that we included in our study are conceptually focused on the theme of cytotoxic T lymphocyte (CTL) and tumor cell interactions in the context of tumor immunology. Supplementary Table  3  outlines the key features of dataset preparation and analysis for the four different Use Cases in the portfolio. Supplementary Table  4  summarizes the model parameters for each Use Case and Supplementary Table  5  includes the fitness scores (min, max, and mean) for testing and training datasets. All four Use Cases utilize the same default library of 42 functions (Supplementary Table  1 ) and the same hyperparameters (e.
Use Case 1: Explainable and accessible models for semantic segmentation of tumor nodules in patient tissue samples We propose an end-to-end translational application of Kartezio for tumor nodule SS in patient tissue samples. The goal is to provide clinicians with an easy-to-apply and fully interpretable tool that is automatically generated starting from a small annotated dataset. For an initial application, we profited from an off-the-shelf tissue sample cohort of metastatic melanoma patients we have previously described  29. In this application, as shown in Fig. 3a , Kartezio was employed to create a SS mask of tumor nodules within IHC-stained tissue samples. Melanoma cells and CTLs were identified by Sox10 (a nuclear marker of melanocytic lineage, orange) and CD8 (a CTL marker, purple) expression respectively and cell lysosomal content was identified by CD107a staining (black).
All cells were counterstained with hematoxylin (blue). In the present study, images were preprocessed by transforming RGB channels into HSV color space (Supplementary Fig. 2c ). We limited the analysis to a dataset containing 24 images. Supplementary Fig. 4  depicts the 12 images used for training (Supplementary Fig. 4a ) and the 12 used for testing (Supplementary Fig. 4b ). Both image series have been annotated by an expert pathologist (green lines); illustrative examples are shown in Fig. 3a. Based on 100 runs, Kartezio obtained a mean IoU score of 0. 87 ( ± 0. 018) on the training dataset, with a maximum score of 0. 91, as shown in Supplementary Table  5. Kartezio obtained a mean score of 0. 85 ( ± 0. 047) on the test dataset, with a maximum score of 0. 89. For each image in the training and test dataset, we merged the predictions of n = 100 trained models to generate composite heatmaps providing integrated representations of the obtained masks, as shown in Fig.
3a  and Supplementary Fig. 4. These composite masks can also be evaluated against the ground truth to provide IoU scores. As shown in Supplementary Fig. 4c-d , we evaluated the fitness of the ensemble of 100 pipelines at multiple different threshold values (between t = 0. 0 and t = 1. 0 in increments of 0. 01; illustrative images shown at t = 0, 0. 35, 0. 5, 0. 6 and 0. 75). This analysis demonstrated that fitness is highly dependent upon threshold value; using the training dataset, we calculated an optimal threshold of t = 0. 35. Across the 12 tumor nodules of the test dataset, the pipeline ensemble yielded a mean fitness score of 0. 90. At this optimal threshold, the ensemble outperformed individual pipelines on average and even outperformed the best individual pipeline, which had a score of 0.
89 on the test set, illustrating the robustness of the ensemble approach for semantic segmentation. It is worth noting that fitness scores ranged from 0. 81 to 0. 96 across the 12 images for the test dataset, indicating that the pipeline ensemble performed consistently (with an SD between images of 0. 054) and with high degree of accuracy in every tumor nodule tested, an important feature for clinical applications. Notably, each of the 100 Kartezio-generated algorithms can be interpreted and explained. As an example, we selected the pipeline that obtained the best IoU score and dissected its behavior (Fig. 3b ). The first step (node 3) of this pipeline consisted of subtracting the Hue channel (node 0) from the Saturation channel (node 1), allowing for the exclusion of pixels corresponding to CD8 + T cells and to other cells belonging to the tumor microenvironment.
Furthermore, neither of these two strategies involved re-training Kartezio on high resolution images, but rather repurposing the masks (first strategy) or the pipeline ensemble (second strategy) from the low resolution dataset. The ability to train Kartezio on low resolution images to generate pipelines that perform well on high resolution images provides a substantial computational advantage, since the extremely cumbersome amount of data associated with high resolution IHC images makes them challenging to use in a training context. This easily interpretable tool to rapidly segment tumor nodules has the potential to allow automatic assessment of various parameters that are critical for cancer patient management, such as tumor nodule size, morphological features, tumor localization, its borders with   surrounding tissue, and distance from the resection margins, as described in more detail in the discussion.
Although conceptually simple, this Use Case enabled us to demonstrate the performance of Kartezio in the context of single-particle high resolution imaging. Using Kartezio, we created an automated procedure to perform IS, which permitted the subsequent identification of single-and double-positive particles and characterization of their features. For each channel, we first trained Kartezio on a dataset consisting of one manually-labelled image, split into quarters (the number of ROIs is indicated in Supplementary Table  3 ). Based on 35 training runs, Kartezio achieved an average AP50 score of 0. 81 ( ± 0. 037) and 0. 79 ( ± 0. 030) for the WGA and DiO channels respectively, with a maximum score of 0. 87 and 0. 84 for WGA and DiO respectively. To assess generalizability, all IS pipelines were then tested on a dataset of one manually-labelled previously unseen image, split into quarters.
In order to identify doublestained particles post-segmentation, the WGA and DiO instances previously detected using Kartezio were tested for matching using a pairing mechanism based on the Intersection over Union (IoU) metric (detailed in Methods). Instances detected in the WGA and DiO channels with the IoU score above a threshold of 0. 05 were identified as double-positive instances. The entire process enabled the identification of three sets of particles: WGA + DiO -(cyan), WGA -DiO + (magenta), and WGA + DiO + (orange) (Fig. 4c ). From these datasets we could readily calculate the mean fluorescence intensity (MFI) and area of each instance as shown in Fig. 4d. The panel displays a scatterplot of WGA + DiO -(cyan), WGA -DiO + (magenta), and WGA + DiO + particles (orange) released from CTLs.
This analysis proved capable of distinguishing extracellular vesicles (WGA -DiO + and WGA + DiO + particles) from putative SMAPs, which were previously shown to be WGA + but negative for lipophilic plasma membrane dyes  33. Taken together, the above results show that Kartezio can successfully perform rapid and accurate IS of extracellular particles at a multi-molecular scale, providing access to further analysis of a   5. statistically relevant number of events. It is important to note that the pipelines generated by a Kartezio can be exported to standard Python scripts, which can be readily re-used on additional images acquired in comparable experimental conditions.
For the initial analysis, we applied Kartezio to series of z-stacks containing CD45 staining only (Fig. 5a ). To achieve this aim, we upgraded Kartezio to segment cells in 3D stacks to provide accurate 2D masks of each identified cell. Kartezio-generated pipelines were sequentially applied to each layer of the z-stack to produce initial Watershed masks of each z-stack. All the masks were merged with an averaging operator which allowed us to obtain the maximal area of the cells in the z-dimension (Fig. 5b ). Local Max Watershed was applied to produce the IS masks of the cells (individual segments, Fig. 5c ). For this analysis, the training fitness was more stringent than in the previous analysis (AP with a threshold of 0. 7 instead of 0. 5) to avoid overfitting. The training dataset was limited to 5 images containing 45 cells, highlighting the aforementioned capacity of Kartezio to be trained for cell segmentation on a very small set of images.
On 35 independent runs, Kartezio obtained an average score of 0. 98 ( ± 0. 007) on the training dataset (n = 5 images with 45 cells) and 0. 83 ( ± 0. 028) on the test dataset (n = 4 images containing 34 cells), with a maximum score of 1. 000 during training and 0. 89 during testing (results shown in Supplementary Table  5 ). The best overall pipeline had a score of 1. 00 during training and 0. 85 during testing. After the individual cells were segmented using Kartezio's best pipeline, we measured staining of the three lytic granule components (perforin, granzyme B, and CD107a) in each z-section to derive a single feature vector for each cell. This vector was a composite of different quantitative readouts and integrated both individual and combined counts of positive pixels, sums of pixel fluorescence intensities, and average intensities measured in the different channels (see Material and Methods).
6μm, based on manual user annotations)   5. to micrometer-scale analysis of CTL phenotype (average mask diameter of ~12μm, based on manual user annotations), and ultimately to centimeter-scale detection of cancer nodules in patient tissue samples (average estimated mask diameter of ~1. 1 cm, based on pathologist annotations). This stringent testing strategy provided a crucial validation of Kartezio's capabilities across a wide variety of image types and across scales differing by five orders of magnitude. A conceptual step-by-step summary of the Kartezio workflow is illustrated in Supplementary Fig. 1. The experimentalist first prepares the training and testing datasets by manually annotating a limited number of images. Then, the training images are provided to Kartezio, which generates initial pipelines composed of functions randomly drawn from the default function library, randomly parameterized and randomly arranged into an image processing pipeline.
Each pipeline is evaluated against user defined annotations and the best pipeline is selected to proceed through the evolutionary selection process (illustrated in Supplementary Movie 1 using images from the Cellpose dataset, originally derived from Cell Image Library  23, 28  ). The optimized pipeline proceeds to the testing phase and, if satisfactory, it can be then deployed on a previously unseen dataset. As demonstrated in the four Use Cases, once image segmentation (IS or SS) has been achieved using Kartezio, post-segmentation analysis of the segmented images can be performed, ranging from simple feature extraction from instances (e. g. MFI, size of particle, etc. ) to MLbased analyses, depending upon the experimental design and needs of the experimentalist (illustrated for the four Use Cases in Supplementary Table  3 ).
As shown in Figs. 1  and 3 , as well as Supplementary Fig. 3 , Kartezio is natively interpretable, meaning that the generated pipelines can be humanly read, evaluated and tested for provability. Given the potential applications for Kartezio in clinical pathology and related fields, its fully interpretable "white-box" nature presents a major conceptual advantage  14. Recent advances in Large Language Models (LLM) are also in line with the quest for more interpretable data analysis pipelines. Indeed, LLM such as ChatGPT  41, 42  , SantaCoder  43  and GitHub Copilot  44  can directly generate the code necessary to analyze a given image or dataset, provided that appropriate prompts are specified by the expert (e. g. type of input image, expected output, functions to use in the analysis pipeline, etc.
It is therefore crucial to minimize experts' effort by designing ML algorithms that can handle the smallest datasets possible, while keeping adequate generalization capacity. To address this point, our study utilizes CGP to generate solutions that leverage the cumulative expertise of human CV experts. In other words, CGP produces pipelines by assembling algorithms that integrate decades of existing research and development in CV. While the number of images used in our study is already remarkably low, it is conceivable to upgrade Kartezio to an active "human-in-the-loop" training mode to further reduce the size requirement of the training dataset and improve performance. In such a configuration, intermediate results from Kartezio-generated pipelines could be displayed while the expert annotates.
At a given point, the expert would not annotate anymore, but simply amend the automatically generated annotation. In turn, expert-operated adjustments would be informative to pursue the training of Kartezio. In addition to these major conceptual advantages, Kartezio offers several concrete technical improvements over existing IS approaches. For example, Kartezio can handle images of diverse dimensions within the same dataset. In addition, it can be trained on small patches and later deployed on much larger images. As shown in Fig. 3 , Kartezio can also be trained on low resolution images and the resultant pipelines applied to high resolution images, which removes the need to train the models on cumbersome high resolution images. Finally, Kartezio is not restricted to gray scale input images and can easily accommodate RGB, HSV, and HED formats.