dataesr/scientific-paragraphs-categorization huggingface dataset:
Effects of the local iterations In this section, we studied the impact of increasing the number of iterations or epochs locally for each client participating in the FL process for model training. Indeed, since the training process in FL is distributed across multiple devices, the number of epochs corresponds to the number of complete iterations on the training dataset on each client. Thus, for this experiment, we first consider a scenario where the epochs parameter is set to 1, with 3 clients and 50 rounds. Then, keeping the same configuration (3 clients and 50 rounds), we increased the value of epochs to 5 in order to compare the corresponding performance for these two scenarios. Table  5  shows the experimental results based on the specifications. Looking at the results in Table  5 , we generally observe an improvement in performance for all models studied when the number of local epochs is increased from 1 to 5.
Performance analysis across datasets In this section, we conduct a comparative performance study of the different models studied for each of the four datasets from the PlantVillage platform, namely: Apple, Corn, Grape and Tomato. Table  6  summarizes the performance results of the models tested on our four datasets for a 3-client, 30-round configuration. Note that for these experiments, we used the same configuration with the parameters defined in Table  2  for each of the datasets. We can observe from the results presented in Table  6  that the performance of the models varies depending on each dataset. In general, we can notice that all models obtain better performances on the Grape dataset, while the performances are worse on the Tomato dataset. The ResNet50 model presents the best performances among all the models, whatever the dataset considered.
Indeed, it manages to reach a F1-Score and an Accuracy of 100% on the Grape dataset. The ViT B16 model also performs well, slightly less than ResNet50, but slightly better than the other models on the Grape, Apple and Tomato datasets. More precisely, it obtains respectively a F1-Score and Accuracy of 98. 57% and 98. 56% on the Grape dataset, a F1-Score and Accuracy of 98. 57% and 98. 56% on the Apple dataset, and finally a F1-Score and Accuracy of 92. 57% and 93. 57% on the Tomato dataset. The Vgg16 and ViT B32 models show almost similar performance on the Grape, Apple and Corn datasets. However, these performances are slightly lower than those of ResNet50 and ViT B16. The F1-scores and Accuracies obtained by the two models are close to 98. 2% for the Grape dataset, 98% for the Apple dataset, and 95% for the Corn dataset respectively.
MobileNetV2 and DenseNet121 also perform well, varying by dataset, and slightly worse in most cases than ResNet50, ViT B16, Vgg16 and ViT B32. The InceptionV3 model performs poorly overall, compared to the other models tested, regardless of the dataset. InceptionV3 performs worse than all other models, but peaks on the Grape dataset with a F1-Score and Accuracy of 96. 11% and 96. 20% respectively. We found that the majority of models exhibit superior performance, with very high F1-Score and Accuracy on the Grape dataset. This could mean that the images in this dataset are easier to classify, by the simple fact that they are overall sharper and have a better representation of the different classes. The performance of the models on the Tomato dataset is generally the poorest, compared to the other datasets.
Although these images are of lower quality than the Grape dataset, it is clear that they are more difficult for the models to classify. Since the size of the Tomato dataset and the number of classes (10 classes) are significantly larger than those of the other datasets, the poor performance of the deep architectures on this dataset could be explained by the fact that the distribution of examples between classes is less balanced, thus making the classification task more difficult. It could also be due to the fact that the models need more examples to learn the discriminative features of the different classes of tomato leaves. Since the size and number of classes for this dataset is larger, we believe that tuning the hyperparameters taking into account the size of this dataset could improve the performance of the models.
The Apple and Corn datasets have the same number of classes (4 classes) and almost similar sizes as the Grape dataset. Due to the simpler distribution of classes and the size of these datasets, the deep architectures benefit from fast learning, which justifies the good performance of the models on these datasets, unlike the Tomato dataset. However, the variation in performance on the Grape, Apple, and Corn datasets could be due to the difference in data quality between these datasets. Thus, we have models that might be more robust to variations in data quality, while others are not. As an example, apart from the ResNet50 model, we can see through the results in Table  6  that the vision model ViT B16 (see also ViT B32) is only slightly impacted by data variations by offering slightly lower performances than ResNet50 compared to the other models for most datasets.
The robustness of these models in this case is certainly explained by the use of attention mechanism. This mechanisms allow this type of model to capture information at different scales and levels of abstraction, thus improving their ability to learn more general and abstract visual features from the input images. Compared with the results obtained by the CNNs in  [3] ,  [18] ,  [8] , as well as the models based on attention mechanisms in  [19] ,  [20]  which used the centralized approach and the "PlantVillage" platform datasets, we note that these results are close to those obtained in our work with the FL decentralized approach. In other words, in addition to its many advantages, FL is capable of performing the same tasks and obtaining results almost identical to those obtained with the centralized approach.
Second, the number of communication rounds has an impact on the performance of the deep architectures, where ResNet50 proved to be an optimal choice for a balance between performance, computational cost, and model complexity. In addition, increasing the number of epochs locally for each client generally improves the performance of all models studied, with ResNet50 providing the best performance. The ViT B16 and ViT B32 Vision Transformers, while offering better performance than some CNNs, require more computational time, making them less suitable in a FL scenario. Finally, the performance of the deep models varies with each dataset, with significant differences depending on the quality of the data and the number of classes in each dataset. It is clear that federated learning has a crucial role to play in crop disease classification.
C i=1 k i ; the number of communication rounds R = R i=1 r i ; the number of epochs E c = E i=1 e i ; the local learning rate L R ; and the size of local batches B c. We therefore vary these different parameters during our experiments in order to observe the impact of each of them in the different model's performances. To do so, we perform a simulation of a collaborative set of clients who share a dataset with the same characteristics (consisting of healthy and infected leaves) for training and testing the model. The data partitioning of the dataset among the different clients is performed equally and randomly distributed. In other words, D = n k=1 P k , where D denotes the size of the dataset, with P k (k = 1, 2,. , n) the partition of each client, and P 1 = P 2 =. = P n.
To the best of the authors' knowledge, this is the first cacheaware allocation method for multi-DAGs systems that is decoupled from (i) the knowledge of instruction and data usage of the software and (ii) detailed settings as well as additional facilities of the underlying hardware. Extensive experimental results, including a real-world case study, validate the concept of the proposed cache-aware allocation approach and demonstrate the AJLR can outperform the existing methods even with errors existing in the CRP models. Paper Organisation: The related work is described in Section 2. Then, based on the system model presented in Section 3, we construct the AJLR in Section 4, with an implementation of the CRP described in Section 5. Finally, we present the evaluation in Section 6 and conclude the paper in Section 7.
Real-world Example We now use a real-world example to (i) demonstrate the construction of a CRP for a given task, and (ii) validate the fundamental hypothesis of the CRP model, i. e. tasks do benefit from the content remained in cache following the prior instance, resulting in execution speed-ups. More importantly, through this case study, we show that the CRP exists for tasks in real systems. Later in Section 6. 2 a proof of concept is provided to show the CRP can lead to better scheduling results using the same case study. The targeted tasks are taken from the TacleBench real-time benchmarks suite  [14] , which covers a wide variety of computational and  cache-related behaviours of tasks in the real world. The considered platform is a general-purpose quad-core Intel Core i5-6500 with a standard Linux Operating System.
For a job 𝐽 𝑖,𝑘 released by 𝜏 𝑖 , its recency distance is the sum of 𝑁𝑜𝑈𝐶 of all jobs from other tasks executed between 𝐽 𝑖,𝑘-1 and 𝐽 𝑖,𝑘. Note, the constructed allocation does not rely on the 𝑁𝑜𝑈𝐶 of each node being precisely accurate. An example of the profiled data (of ndes in TacleBench), and the learned model based on it is given in Figure  5. The profiled data is collected as a percentage of the WCET. For example, 50% means the execution time on measurement is 50% of the WCET. The recency distance (x-axis) is calculated based on Equation  2. Finally, a segmented linear function (the yellow lines) is fitted to the data with error minimised using piecewise linear regression. In section 6. 5 we evaluate the effectiveness of the proposed method when considering different amounts of error in the CRP model, and show that the method remains effective when errors exist.
The main contributions of this work are as follows: • We propose a data-centric method for formulating the KG-based learning objectives of triple classification and entity/link prediction in the language modelling paradigm, and implement a framework for training transformers using the UMLS knowledge base in parallel with masked-language pre-training. • Pre-training on the UMLS alongside the European Clinical Case Corpus  (Minard et al. , 2021; Magnini et al. , 2020) , we show that this method brings improvements to pre-trained models across a range of biomedical entity recognition tasks in three different languages, as well as functioning as a competitive pre-training strategy that requires much less training data in comparison to state-ofthe-art transformer models. We release the monolingual and multilingual model weights trained in this way, UMLS-KGI-BERT, as open-source resources for the clinical NLP research community.
Pre-trained LMs for Medical Applications In general, the standard methodology for adapting neural text encoders to the biomedical domain has been to take a model that has been pre-trained on general-domain text corpora and continue this unsupervised pre-training on a medical corpus  (Alrowili and Shanker, 2021; Lee et al. , 2020; Alsentzer et al. , 2019). However, recent work has suggested that, given enough training data, it is preferable to pre-train these models on large domain-specific corpora only, without starting from a general-domain checkpoint  (Gu et al. , 2021; Rasmy et al. , 2021). In this work we explore both approaches, extending existing biomedical and general-domain models as well as training BERT models from scratch on our own generated datasets.
Knowledge-enhanced LMs Techniques for the incorporation of knowledge graph structure into BERT models can, broadly speaking, be divided into three categories, each focusing on one of the three fundamental components of a machine learning system, i. e. 1) the training data, 2) the model architecture and 3) the objective function to be optimised. The first type of approach prioritises the augmentation of BERT's input data with information extracted from a knowledge graph. This extra information can be numerical, e. g. precomputed graph embeddings  (Jeong et al. , 2019)  or textual, e. g. KG triples linked to input sentences  (Liu et al. , 2019). The second type of approach focuses on adapting the architecture of BERT so that its language representations become fused with knowledge graph embeddings (KGEs)  (Wang et al.
, 2020; Mondal et al. , 2019). In particular, there have been multiple successful efforts to integrate the UMLS knowledge graph into BERT models, notably UmlsBERT  (Michalopoulos et al. , 2021) , which proposes a data-augmentation technique allowing for concept and semantic type information to be linked to input text, and SapBERT  (Liu et al. , 2021b,a) , which introduced a self-alignment strategy for learning from UMLS synonym pairs via a multi-similarity (MS) loss function to force related concepts closer to one another in BERT's repre- sentation space. Yuan et al. (2022)  build on this strategy by applying MS loss to relation triples. In contrast, in this work we show that information from the UMLS can be incorporated into BERT models in a simpler way, using only cross-entropy classification loss, while also balancing this training process with standard masked-language BERT pre-training.
Recent general overviews of the landscape of AI research have highlighted the importance of data-centric approaches to building models  (Zha et al. , 2023; Hamid, 2022; Jakubik et al. , 2022)  and in light of these trends this work focuses on types 1) and 3) of knowledge base integration described above, i. e. on improving the performance of standard model architectures by constructing high-quality datasets that can be integrated into the self-supervised language modelling paradigm by modifying the BERT objective function. The motivation for this kind of approach is also to provide a pre-training framework that is more widely transferable and does not rely on any particular transformer-encoder architecture.
Dataset Construction Formally, we consider the UMLS KG in the standard fashion, as a directed graph G = (C, E, R) where C is the set of all medical concepts in the KG, E the set of all edges or relations that link these concepts to one another, and R the set of possible relation types, i. e. the labels r for each e ∈ E. The training sequences are thus generated from the KG dataset of ordered triples (h, r, t) where (h, r) ∈ C × C and r ∈ R. As a compendium of multiple different sources of taxonomic biomedical information, the UMLS metathesaurus contains multiple levels of granularity at which meaning representation can be analysed. We consider three such levels of granularity in our work: • Terms -string descriptors for conceptual enti- • Semantic Groups -these are groupings of concepts that can be considered to define the type of entity a concept represents; e.
(2019) , and [REL] is one of the relation tokens. For link prediction, we construct a dataset of variable-length paths through the KG by iteratively selecting a list of triples (h 1 , r 1 , t 1 ),. , (h n , r n , t n ) where h i+1 = t i to form a path p = (h 1 , r 1 , h 2 ,. , r n , t n ).
Entity Prediction The entity classification task can be trivially integrated into the masked-language objective of BERT, by masking the tokens associated with the concept t. Link Prediction We formulate link prediction as a narrow masked-language task by masking the relation tokens in the path dataset with another hidden relation token, for which the model is trained to fill in one of six relation types -as the triple classification and entity prediction tasks already have the partial goal of improving the model's capability to associate synonymous terms with each other, we exclude synonym relations from the path dataset. Triple Classification Following the work of  Hao et al. (2020) , the triple classification objective is formulated as a binary classification problem where the model is tasked with classifying triples as true or false.
In order to generate training examples of false triples, we use two different negative sampling strategies. Firstly, to provide directly contrastive examples for existing relations, we sample triples (h, r, t) where h and t belong to different semantic groups and construct corresponding false triples with the same relation type and semantic group categories, i. e. ( ĥ, r, t) / ∈ G where ĥ and t are of the same semantic group as h and t respectively. Secondly, to provide contrastive examples for relation types, we sample triples for which h and t are of the same semantic group, and form the negative training example by changing the relation type r. To ensure balance, the triple classification datasets used in this work are made up of 50% positive examples (real triples from the KG), 25% examples generated by the first negative sampling method and the rest by the second.
We perform stratified sampling on the base knowledge graph according to semantic groups, i. e. we ensure that the proportional representation of each semantic group in the knowledge-base triples for each language is maintained in the training datasets. Mixed Objective Function In order to train BERT models using the UMLS-based reason-ing tasks described above alongside the maskedlanguage objective, each training example is augmented with an indicator label that tells the model which loss function to apply to the sequence in question. The overall loss function is then calculated as L = L MLM + α 1 L EP + α 2 L LP + α 3 L TC (2) where the α i are scalar task-weighting coefficients and L MLM , L EP , L LP , and L TC correspond to the loss values for masked language modelling, entity prediction, link prediction and triple classification respectively.
KG-integrated pre-training Pre-training corpora As a resource for maskedlanguage pre-training, we utilise the European Clinical Case Corpus (E3C) version 2. 0. 0 2 , a freelyavailable multilingual corpus of clinical narratives. We evaluate our method in three different languages; English, French and Spanish. These languages were chosen as they are the three most well-represented languages in the metathesaurus for which we have access to pre-trained clinical BERT models for comparison. The sizes of the combined UMLS-E3C datasets used are shown in Table  1. For each language, we compare the performance of 1) a transformer model trained from scratch on each monolingual dataset (KGI-BERT EN,F R,ES ) against 2) a multilingual version of the same model trained on all three datasets (KGI-BERT m ), 3) a pre-trained monolingual biomedical model and 4) the same pre-trained model with supplementary training on the corresponding monolingual UMLS-E3C dataset.
The UMLS-KGI models were trained for 64 epochs on each dataset, using the PyTorch implementation of the weighted ADAM optimizer 1 https://huggingface. co/datasets 2 https://live. european-language-grid. eu/ catalogue/corpus/7618  (Loshchilov and Hutter, 2019)  with default parameters. We use a maximal sequence length of 256 for the masked-language modelling sequences, an effective batch size of 1500 and a triangular learning rate schedule peaking at 7. 5 × 10 -4. To take into account the varying sizes of the components of the pre-training dataset we set the values of the coefficients of the loss function such that they are inversely proportional to the number of documents available: α i = 3 j=0,j̸ =i n j 2 3 k=0 n k where the n k correspond to the number of documents in the training set for each UMLS-based task.
In this way, the E3C masked-language loss has the same weighting as the UMLS-based task losses. Pre-trained models For supplementary training, we make use of what are, to the best of our knowledge, the overall best-performing biomedical BERT models of their size (pre-trained using masked-language tasks only) for each language, according to baseline experiments on the NER tasks. For French, we use DrBERT  (Labrak et al. , 2023) , for Spanish the RoBERTa-based biomedical model released by Carrino et al. (  2021 ), which we refer to as BioRoBERTa-ES, and for English PubMedBERT  (Gu et al. , 2021). For training from scratch, we use the DistilBERT model configuration  (Sanh et al. , 2019)  with 12 encoder layers and 12 attention heads.
NCBI-Disease BioRED  (Luo et al. , 2022)  This corpus is designed for biomedical relation extraction and en-tity recognition; we focus on the latter in this work. This task can be considered a more semantically general version of the NCBI disease recognition task, in that the BioRED corpus consists of PubMed abstracts annotated with a diverse range of entity types including genes, proteins and chemicals. JNLPBA04 NER Dataset  (Collier and Kim, 2004)  Developed in the context of a biomedical entity recognition shared task, this corpus consists of Medline documents annotated with mentions of DNA, RNA, proteins, cell types and cell lines. We report the macro-averaged precision, recall and F1-score for each task. Results for the French, English and Spanish tasks can be seen in Tables  2, 3 , and 4 respectively.
Conclusions and Future Work This paper introduces UMLS-KGI, a framework for training BERT models using knowledge graphs requiring highly minimal adjustments to the standard language modelling paradigm. We show the potential of this method to increase the performance of BERT models on various NER tasks. The results presented in this paper suggest that for clinical NER tasks, high-quality small-scale datasets derived from structured information, alongside alongside relatively small clinical text corpora, can be as effective as large-scale corpora for pre-training BERT models. We make our models and dataprocessing pipelines freely available online. Future work in this direction will involve the incorporation of more diverse graph-based reasoning tasks in the pre-training strategy with more finegrained representation of relation types, as well as intrinsic evaluation of the UMLS-KGI-BERT language representations via embedding visualisation and interpretability studies.
The AGATA and GRETA spectrometers are large arrays of highly segmented HPGe detectors that use the technique of gamma ray tracking to reconstruct the scattering path of gamma rays interacting within the active material. A basic requirement is a precise reconstruction of the individual interaction locations within the detectors. This is possible through the use of pulse shape analysis which has to be conducted in real time due to the high data rates generated by the spectrometer. The methodologies that have been evaluated to perform this for AGATA are discussed along with the approaches used to calculate the pulse shape databases required by these algorithms. Finally, the performance and limitations of the existing approaches are reviewed.
The AGATA detectors provide a course degree of position information through the 36 segments on each detector. The resulting position resolution is however insufficient to precisely locate the interaction positions. PSA allows for the signal collected from each of the charge sensitive preamplifiers connected to these segments, and the core electrode, to be analysed. The unique shape profile of the charge signal can be used to identify the positions(s) of interaction. In practice PSA uses a database of pulse shapes (basis) to describe how these signals look at each interaction position inside the detector. By comparing the signals measured with the basis the unique interaction positions can be found. The basis is created from a calculation, and it is validated by experimental measurements which are obtained from detector characterisation systems.