dataesr/scientific-paragraphs-categorization huggingface dataset:
Inter-folds indicators As observed in  [6]  and in  [55] , some datasets have a high overlap of mentions between training and test sets. This could lead to relatively optimistic results. Thus, we compute a redundancy indicator, which is the percentage of examples (i. e. not the mentions, as in other studies) seen in the training set and seen again in the test set. Conversely, we compute a ZSL indicator (i. e. for Zero-Shot-Learning) which is the percentage of concepts met in the test set but never in the training set. For instance, the "zero-shot split" of the COMETA corpus has a ZSL of 100%: no concept of the test set is seen in the training material. These ZSL cases are considered challenging, and we think that the proportion of these in a dataset can give an estimation of the difficulty of a task.
1% of multi-normalization cases in its training set. If the test set has a similar ratio, this would result in at least 4 points less in accuracy for methods which do not address this problem. In contrast, as indicated in  [32] , all multinormalization cases (called "ambiguous mentions" in the study) have been removed from the Random CADEC corpus to build the Custom CADEC corpus. The most notable difference between the three datasets is the rate of ambiguous mentions. Indeed, while NCBI-DC and Custom CADEC contain a low ambiguity rate (less than 5%), almost one in five BB4 mentions is ambiguous. To resolve those ambiguities, it would be necessary to use the context of the mentions, but to our knowledge, this information is never used by existing methods in entity normalization (at the difference of entity linking methods).
It is noteworthy that in the three datasets, there are no NIL mentions. The hierarchy of the OntoBiotope and MEDIC ontologies means that all mentions can at least be normalized by their root. Nevertheless, in practice, there are no mentions normalized by the root in these datasets: a NIL mention could be resolved by completing the ontology with a more specific concept during the corpus building (e. g. it was the case for the BB4 dataset). Moreover, a direct mention in text of a root concept (e. g. "bacterial habitat") can usually be considered as too uninformative to be extracted. Finally, the NIL mentions can simply be deleted from the corpus. For instance, because of the non-hierarchical structure of the AMT part of the Custom CADEC dataset, there are NIL mentions in the initial CADEC corpus.
Inter-folds As shown in Table  6 , redundancy and zero-shot-learning indicators can also clearly differentiate the three corpora. As redundancy was a major bias in the Random CADEC corpus, we observe indeed that there is nothing left in the Custom CADEC. The authors were able to obtain 5 folds (rather than 10 initially) by redistributing examples (seeming to duplicate examples from the initial corpus between batches) and obtain only 181 used concepts (rather than 1036 initially). This modification could also explain the higher number of distinct examples by concepts observed in Table  4. Compared to this, the NCBI Disease Corpus displays a high rate of redundancy (63. 3%), close to the initial rate of the Random CADEC dataset. Because of their relatively low rate of ambiguities (see Table  5 ), redundancy appears also as a major biais in the NCBI Disease Corpus.
The redundancy is lower in the BB4 dataset (24. 6%). ZSL is also an interesting indicator to differentiate the datasets. Notably, although Custom CADEC addressed the redundancy problem, it keeps a distribution of examples between sets, which results in a perfect overlap between concepts used in the training sets and their associated test sets. Compared to this one, around a quarter of concepts to predict in the test set have never been used in the training set of the NCBI Disease Corpus, and around a half for the BB4 dataset. Without judging the relevance of having a strong ZSL value in an evaluation corpus, a null value does not allow to evaluate the capacity of a system to predict a concept it has never seen.
Significativity/variability of methods A screening of all methods' parameters has a significant cost in time. This time needs to be multiplied to estimate the significativity and the variability (due to stochastic parts in numerous methods) of the results. These efforts are particularly important when the aim is to compare different methods that obtain very similar scores on the same dataset. This may be the reason why some authors do not test for significance and present only one result in their publication (e. g. BioSyn). This represents a bias when we try to unknowingly compare a method that would give a significant score with a method that would give its best score, enough to change a leaderboard. Nevertheless, in this study, we seek to observe important differences. Based on the articles of the BioSyn and the C-Norm methods, and on our tests, we assumed in our experiments that a difference of 3 points is significant enough to make observations on single runs, regardless of the modifications made.
Influence of preprocessing and used resources We evaluate the impact of preprocessing on the performance of the methods. If a preprocessing gives a positive impact on a dataset, not on another one, then it reveals that these preprocessing are somehow domain-specific, thus lacking robustness. We think that a realistic and non-domain-specific baseline method can also help to characterize  a corpus. This baseline serves as a control through different datasets. Rather than only measuring the score of a method alone on a dataset, observing the difference between this score and that of the baseline should give a better appreciation of the quality of the results.
BioSyn For BioSyn, it should be noted that score variation doesn't come from training (with same BioBERT version at least), but only from preprocessing. As shown in Table  7 , the whole preprocessing seems to have a marginal influence on the accuracy of the method on the BB4 dataset (+ 2. 5 points), and has even a slightly negative influence on our baseline method (-1. 3 points). On the other hand, preprocessing has a global positive influence on the accuracy on the NCBI-DC dataset (+ 8. 3 points). It seems to indicate that preprocessing is globally over-specialized to the NCBI-DC dataset. However, on a single run, it is difficult to estimate the real individual contribution of each preprocessing option (maximum observed contribution of 2. 8 points). Surprisingly, it can be noted that removing the typo resolution, though specialized to the typos contained in the NCBI-DC corpus, gave a slightly higher score (+ 0.
6 points) for the method with all the preprocessing. Thus, it is not a significant observation, but it is enough to question the relevance of this preprocessing which would require important manual efforts while being strongly domain-specific. We also observe a huge gap between the results of BioSyn on the two datasets: + 30. 5 points on the NCBI-DC dataset compared to the BB4. We have a similar observation with our baseline method (+ 23. 3 points) or with our baseline with BioSyn preprocessing (+ 28. 8 points). The difference of measures presented earlier between the two corpora may explain this difference. Moreover, from our observations, BB4 ambiguous cases in particular lead BioSyn to predict too many concepts. And finally, we show that BioSyn outperforms our baseline, with or without BioSyn preprocessing (+ 3.
C-Norm In Table  8 , we compare the impact of BioSyn preprocessing on the results of C-Norm. Surprisingly, C-Norm does not seem to gain any particular benefit from the NCBI-DC over-specialized BioSyn preprocessing. Thus, the method seems relatively robust and can mitigate the need of these kinds of preprocessing. Despite the suboptimal use of only 3000 convolutional filters on the NCBI-DC dataset, we can still observe a gap between the results of C-Norm on the two datasets: + 9. 0 points on the NCBI-DC dataset compared to the BB4. C-Norm applies as preprocessing only lowercasing, lemmatization and stopword removal and no strongly domain-specific preprocessing is used. But on the contrary, C-Norm is based on a specialized Word2Vec embedding set, optimized directly on the BB4 development set with another normalization method  [17].
As shown in Table  9 , the use of a standard embedding set (even specialized for the biomedical domain and trained on the same type of literature) has a negative impact on the C-Norm results: -5. 9 points on the BB4 and -6. 8 points on the NCBI-DC. A similar observation can be done with our baseline method on the BB4 (-3. 4 points) but not on the NCBI-DC (+ 6. 4 points). For C-Norm, which is strongly based on word embeddings, the main explanation could be the number of out-of-vocabulary tokens. Indeed, 31. 7% of tokens from NCBI-DC dataset and MEDIC terminology are out-of-vocabulary in the BB4 embedding set. The DictB-SymR submethod of our baseline, which is not based on embeddings, could explain why this one performs better. As noted, a minimized value of the number of filters seems to have a real negative impact on C-Norm scores on NCBI-DC.
Influence of reference size As noticed in  [54] , some datasets use a subpart of the initial reference ontology. We analyze the impact of choosing different parts of the ontology. Mainly, we built smaller ontologies in keeping only the concepts used in train and/or test sets. For OntoBiotope, we also use the whole ontology with the phenotype part. As shown in Table  10 , the BB4 dataset used in our experiment contains 3172 concepts of bacterial habitats (88%) out of the 3602 of the whole OntoBiotope ontology. The habitat part is the main part of OntoBiope. Compared to that, the train + dev dataset represents only 294 concepts (8% of the whole, 9% of the habitat part). As shown in Table  11 , these differences of the possible target concepts have a strong influence on the results on the dev set for C-Norm (+ 9.
2 points). The gain is less visible for BioSyn (+ 2. 7 points) and our baseline (+ 3. 8 points). Certainly because of the relatively low number of overlaps between used concepts in the train and the dev (51. 8%, see ZSL indicator in Table  6 ), all the methods strongly decrease their results when using only the concepts from the train (-14. 1 points for C-Norm, -16. 5 for BioSyn and -12. 6 for our baseline). We also observe that C-Norm has a slight loss when using the whole ontology (-1 point), notably compared to BioSyn (-5. 6 points). Thus, C-Norm seems more robust to concepts addition than BioSyn on the BB4 dataset. With the availability of the annotated test set of the NCBI-DC dataset, the methods can use more different subsampled ontologies. We do not use the MEDIC ontology with only the concepts used in the test set, which would be borderline, as the dev Habitat Ontobiotope for BB4.
As shown in Table  12 , the NCBI-DC dataset used in our  experiment contains 9664 concepts, with the train + dev + test dataset representing only 751 concepts (8%) and the train + dev dataset only 695 (7%). There is thus a certain similarity of distribution between the concepts used in the annotated data and all the predictable concepts. As shown in Table  13 , the use of the train + dev + test MEDIC has no visible influence on the results of BioSyn (+ 1. 2 points), and a weak gain for our baseline (+ 3. 2 points). In contrast, it has a strong influence on C-Norm (+ 12. 5 points). Thus, we can observe here a similar behavior of the three methods whether on the BB4 or on the NCBI-DC datasets. Because of a higher number of overlaps between used concepts in the train + dev and the test (72. 7%, see ZSL indicator in Table  6 ), the effect of using only the train + dev concepts seems to be less drastic (-10.
Influence of scoring metric The BioSyn script proposes a scoring evaluation with different features than ours (see Eq. 3). Notably, it evaluates multi-labeled mentions differently: if the predicted concept is in the list of correct concepts, it is considered as a good prediction (i. e. maximum points for the current mention). With our metric, a good prediction means all expected concepts are predicted. From Table  14 , we see that this difference has not a significant impact on the NCBI-DC dataset (0. 1 point variation for our baseline, 0. 4 for C-Norm and 0. 6 point for BioSyn). The reason is that there are only 2. 3% of multi-normalization cases in the NCBI-DC (see Table  5 ), and therefore even fewer multi-labeled mentions. On the contrary, for the BB4 dataset, there are exactly 8. 1% multi-labeled mentions in the train + dev dataset (see Table  5 ).
Discussion To better evaluate the robustness and the adaptability of methods, evaluation on distinct datasets are needed. Nevertheless, on the existing datasets, some have no publicly available resources (e. g. the specific version of the SNOMED CT-AU annotating part of the CADEC datasets is not open source, and not free for everyone). This is an obstacle for the reproducibility of the work. In this way, the work described in this article could be extended to more datasets and more methods. The community should also make efforts to produce datasets from outside the biomedical (or biological) domain. An open source warehouse with many datasets could support this purpose. A personal evaluation system seems to us to be a significant source of possible errors. Notably, we showed that it can induce a bias if there are differences in the scoring measure used.
The full access to the annotated test set can also induce a bias by enabling some concepts filtering in the ontological reference making the task easier and/or less real world oriented. For these reasons, an independent online evaluation platform, similar to the one used by the Bacteria Biotope tasks, seems to be essential. Nevertheless, the unavailability of the annotated test set limits the possibility of discussions (e. g. on possible errors of annotation, on comparison of mentions/ concepts distribution between train/dev dataset and test set or on designing k-fold cross-validation). By the way, for significativity purposes, k-fold cross-validation should become a standard, as well as a ZSL evaluation, notably to better apprehend the capacity of a method to address real world tasks. We were surprised by the number of recently published methods that do not even share their code.
But without taking into account this bad practice, there can also be a usability problem with the methods sharing their code. For instance, C-Norm needs a lot of computing resources, which means that even if it can be easily installed, it is not very usable without having access to a suitable computer. Compared to C-Norm, Bio-Syn seems much more usable (as we could run it on a free Google Colab GPU server). A standardization of the normalization task could be a step forward for reproducible work and fairer comparison. For instance, some datasets already resolve the composite mentions in the provided dataset (e. g. BB4) while some others do not (e. g. NCBI-DC). Some datasets use a conceptless concept (e. g. CADEC), some others remove them (e. g. Custom CADEC) and some others give a hierarchical ontology which means that any mention can at least be normalized by the root concept (e.
g. BB4). Some datasets are designed with some simplifications (e. g. ambiguous mentions have been removed for Custom CADEC). Some datasets had an intervention on the distribution of the examples (e. g. removing duplicate mentions in Custom CADEC). And as shown previously, there are some distinct accuracy functions. On this subject, our recommendations are: • Separate into two tasks the resolution of composite mentions and the entity normalization. • Promote hierarchical ontology rather NIL concept. Minimally, a list of concepts can be attached to a root that represents the entity type (i. e. the type of entity targeted by the prior recognition). • Avoid simplifying datasets, and instead intervene to make them more realistic. In particular, redundancy should be minimized and ZSL cases should be amplified, because of the performance biases.
Separate evaluations (e. g. cases of multinorm, redundancy, ZSL and ambiguity) could also help to estimate the strengths and weaknesses of methods. What is often missing is a comparison with a good baseline that is relatively robust. This would allow to estimate the difficulty of a dataset, and also to estimate a bit better the real performance of a method by observing the difference from it. Ideally, such a baseline should be openly shared and easily manipulable. The baseline we have proposed here, if efforts to make it more easily usable were (e. g. by integrating it into a standard Python library), seems to us to meet this need. It also has the advantage of being able to perform a training phase and a prediction phase in only a few minutes. We expect to improve our analysis by expanding our experiment to some other entity normalization datasets and methods in future work, and to study in more depth the performance of the methods according to the different cases mentioned.
Conclusions Even if some methods reach scores higher than 90 points of accuracy on some entity normalization datasets, we show that there exists some evaluation bias that renders comparisons between methods hazardous, and tends to distort the real performance of the state of the art methods. We hope that these biases will be taken into consideration by the entity normalization community, notably by the dataset/method providers. We were able to make some suggestions that could avoid these biases and therefore considerably support a methodological research on entity normalization.
Modeling: ECS Components The objective of this part is to define the models used for the main components of ECS. With this in mind, different methodologies can be used. First, many models can be obtained from analytical equations based on physical relationships. Then, scaling laws are useful to define characteristics from a reference  [34, 35]. Finally, some models can be obtained from surrogate models, via an existing dataset or via numerical simulations to generate this data  [36]. For all of the components under study, their mass, their electrical consumption, and their additional drag will be considered.
Sample Size Calculation and Rater Selection This research was conducted according to the guidelines for reporting reliability and agreement study  [15]. No data were available in the literature regarding the inter-rater agreement and reliability of abomasal scoring systems in veal calves. The sample size was determined based on an expected lesion prevalence of 70% and the ability to detect true kappa reliability coefficients greater than 0. 4, 0. 5, and 0. 6. Different scenarios were developed, taking into account the presence versus absence of a specific lesion (binary classification) using a freely available software (package irr  [16] , argument N. cohen. kappa; R  [17] ). A sample size of 70 abomasa was determined as adequate for all scenarios, with a maximal Type I error (α) set at 5% and Type-II error (β) set at 20%.
Data Collection The abomasa were collected from 27 June to 11 July 2022 in a large veal slaughterhouse in Saint-Germain de Grantham, central Quebec, Canada, with an average slaughter capacity of 1200 white veal calves per week. For 7 days, between 10 to 15 abomasa per day were retrieved from white veal calves raised for meat production. During the slaughtering process, the abomasum was separated from the intestinal tract by experienced technicians working in the packing plant as a standard operation process. Immediately after separation, the organs were rinsed with tap water, collected in a box, and transported for a maximum of 40 min at room temperature to the Faculty of Veterinary Medicine in St-Hyacinthe (Quebec, QC, Canada). Upon arrival at the necropsy laboratory, the abomasa were opened along the greater curvature from the omasoabomasal orifice to the duodenum and rinsed a second time with tap water before examination.
Statistical Analysis All data from the scoring systems were stored in an Excel file (Excel 2016, Microsoft Corp. , Redmond, WA, USA). Analyses were performed using the open-source R software v 4. 3. 3 (https://www. r-project. org/ (accessed on 8 September 2022)). The detailed coding script can be found in the Supplementary Data. Scores were calculated per rater and per abomasum in accordance with the Welfare Quality Consortium protocol for veal calves, 2009  [2]. Only lesions in the pyloric area and torus pyloricus were taken into account. The surface of each lesion was estimated based on the size measurements of the rater, and was classified as follows: small (<0. 5 cm 2 ), medium (0. 5-1. 0 cm 2 ), and large (>1 cm 2 ). The number of lesions was set to 0 when no lesion was present and was truncated to 4 when 4 or more lesions were present.
INTRODUCTION M ODERN software solutions, for obvious cost reasons, are shared by several users. This may be achieved by supply pooling (a publisher creates a solution and sells it to "clients") or by demand pooling (a group of actors coordinates to develop a resource that addresses shared or complementary needs), with several intermediate solutions (e. g. a group of actors mandates a provider to develop a solution for them). The possible technical, organizational, and economic arrangements vary according to who allocates and who pays for the human and technical resources needed to develop two main capacities  [1] : the capacity to specify users' needs into technological requirements and code, and the mastery of the development process (prioritizing and integrating the code into a coherent solution).
This article studies one such arrangement, the FLOSS project CommunesPlone, which has developed several dedicated public software solutions based on one of the oldest FLOSS Content Management Systems Plone. Initiated by municipalities in the Walloon Region of Belgium, it is also used, however marginally, in France and Italy 1. It includes online services (eGuichet), deliberation management (Plone Meeting) and web site template (CPSkin). 1. The project's international initiative, PloneGov, has never really taken off, illustrating the difficulty in sharing business developments between different countries, with different regulatory frameworks, even in Europe. The website is down. Some challenges addressed by this case are common to any project: how to respond to users' needs, especially non IT specialists, how to share the costs fairly.
Finally they benefit from others' efforts to meet the shared needs and to propose functionalities they may need but have not yet identified. Public administrations have long been sensitive to these arguments  [3] , and have deployed generic open source software (operating systems, email servers, office automation, etc. )  [4]. They have also developed dedicated FLOSS software (e. g. the Lutèce CMS developed by the City of Paris). The French non profit Adullact has been coordinating public authorities around FLOSS project development since 2002, with the goal to help them to regain control over how their needs are addressed, but also to better manage public IT spending (not paying twice for the same solution). Despite these examples, mostly from large public administrations, benefiting from major pools of IT human resources, public authorities report difficulties in embracing FLOSS  [5].