dataesr/scientific-paragraphs-categorization huggingface dataset:
A simple metric to consider, and one that is widely used, is the accuracy metric, which returns the percentage of correctly identified events. However, class imbalance can exert a major impact on the accuracy metric, modifying its meaning and reliability. In particular, excellent accuracy may be obtained by classifiers simply guessing the majority (background) class  [41]. The simplicity of the accuracy metric leads to a bias towards the majority class, leading to the metric being less discriminatory and less informative  [42]. The imbalanced dataset problem has been studied in the machine learning community, with many looking to improve on the performance of the accuracy metric  [41] [42] [43] [44] [45]. A popular choice of metric amongst the high energy physics community is the area under the receiver operating characteristics curve (auc), which reflects the overall ranking performance of a classifier  [42].
This metric considers the true and false positive identification rates. This metric is a standard metric for use in machine learning for particle physics, and has shown reasonable performance for imbalanced datasets  [46]. As an alternative metric, the F-score  [47]  may be considered to evaluate the classification power of a model. The F-score metric is a measure of accuracy, but has been reported to perform better as a discriminator than the accuracy metric, which simply measures the number of predictions the model made correctly  [42]. It is a popular metric for imbalanced datasets  [48]  but is not usually considered in high energy physics analyses, and may improve on the auc performance  [49]. Finally, the ams metric is equivalent to the significance used in traditional cut-and-count analyses, measuring the discovery potential of an analysis.
It is calculated using a formula for the discovery significance for Poisson data. In the machine learning context, and particularly with very imbalanced datasets, it is too volatile to be used in training or hyperparameter selection, but may be used to evaluate the model at the final stage given the number of true background and signal events selected as signal. The difficulties introduced by large class imbalances may be handled by a careful choice of metric, or by altering the size of the training set by rescaling of the weights (discussed in section 2. 3) to balance the classes. However, testing must take place on the true size of classes to represent real world performance, and test results often suffer from the sheer difference in numbers between signal and background. The advantage of rescaling weights in training is metric-dependent, but the auc metric performs well on balanced training classes  [44].
In particular, for β = 1, we define the F-1 score metric, where precision and recall are equally important. One may also consider an unweighted F-score by modifying the value of β to emphasise either the precision or recall, where higher precision imposes stricter selections for a positive result, and higher recall allows more events to be identified as positive. While the former will lead to a very pure sample of positively identified events, the latter will reduce the number of false negatives, which may be crucial in a highly imbalanced particle physics dataset with very few true signal events. A large area under the precision-recall curve (the plot of the precision versus the recall for varying probability thresholds) indicates a high recall and high precision rate, or low false-negative and false-positive rates.
Weights In a high energy physics dataset, simulated events almost always have associated to them a weight, which indicates that a given event in the sample usually corresponds to more events in reality, although in some weighting cases it corresponds to fewer. Weights may in principle be arbitrarily small, leading to large demands on CPU. That is, the sum of weights of a dataset will generally sum to the cross section of the process at hand. For use in our case, where the weights are utilised by the ams metric, we must have that the sum of weights corresponds to the number of expected events. As such, the formula N events = L × σ (2. 10) should be used as appropriate. Here L is the integrated luminosity, which measures the rate of collisions produced in a detector. Given this sum of weights, the purity of the dataset can be defined as p = s s + (b + ∆b) = i w s i i w s i + j (w b j + ∆w b j ) , (2.
11) for i signal events with weights w i , and j background events with weights w j , where the background events are associated to an uncertainty ∆b due to the shape of their distribution. The ams metric is normalised by the sum of weights of the data on which it is evaluated, where this sum should be the number of expected events, as in eq. (2. 10). This implies that weights must be handled carefully. If weights are arbitrarily rescaled, the ams value computed on them will be unrealistic. For any training, testing, and validation subset, the sum of the weights must be the same in order to obtain meaningful ams values. When analysing x% of the dataset, weights should be rescaled by 1/x%. Failure to do this may lead to lower values of the ams score. It should be clear that these weights are associated to the simulation of physics processes, and should be handled with care.
Event generation and tools Training and testing of the machine learning toolkit will be done on simulated samples, including both background and signal processes. The MadGraph5 aMC@NLO (MG5 aMC) framework  [58]  has been used for the simulation at colliders for both the signal and background processes, where the decays of the smuons are handled by Mad-Spin  [59]  and MadWidth  [60]. The UFO model files  [61]  for use with MG5 aMC were created using the FeynRules  [62, 63]  package and are publicly available  [64]. In each case, the leading order set of NNPDF 2. 3 parton densities NNPDF23 lo as 0130 qed  [65]  is convoluted with the partonic cross section. In the case of the signal simulations using MG5 aMC, additional parton-level jets are allowed at the matrix element level, and an MLM merging procedure is employed to avoid double counting after matching with parton showering  [66, 67].
The simulation of collision events and reconstruction has been done in conjunction with Pythia 8  [68]  to describe parton showering and hadronisation, and with the Delphes 3  [69]  software package that includes the anti-k T algorithm  [70]  as implemented in FastJet 3  [71]  for event reconstruction. The default Delphes card for the CMS detector was utilised, which is included in the standard MG5 aMC installation, and proton collisions simulated at a centre of mass energy of √ s = 13 TeV. A number of background processes are relevant to the considered final state, where SM processes featuring a final state pair of opposite sign leptons and neutrinos constitute the dominant background. Of particular relevance are the t t processes where both top quarks Sample νν W + W - t t Signal Q match 37.
With the goal of understanding the feature space in mind, a two-step feature investigation was performed; an investigation into feature dimensionality and relevance was performed using a principal component analysis (PCA)  [75] , followed by feature selection utilising Shapley values  [76]  and feature permutation importance investigation. The PCA is an unsupervised technique which is widely used for dimensionality reduction, but in this case is utilised to investigate the variance of the feature space. The XGBoost package features a supervised machine learning technique, but the use of the unsupervised PCA (where signal and background labels are not considered) allows for general consideration of the behaviour of the features. In contrast, feature permutation and Shapley investigations allow for the selection of the features which contribute most heavily to the supervised training.
It should be noted that decision trees do not require the same level of feature selection as other machine learning methods, as they are not significantly affected by a large number of variables, nor are they vulnerable to duplicate variables. They are able to handle discrete and continuous variables simultaneously, and are insensitive to the order in which the variables are introduced  [23]. Instead, the dominant gain in feature selection arises in CPU improvements. It is nonetheless crucial to understand well the features included in the dataset to avoid a black-box implementation.
Features with higher variation are not necessarily more useful in a physics analyses than those with lower variation, but variance analysis of features in an uncorrelated format does offer insight into how each of the features within the dataset will contribute to the target concept being learnt. Following this investigation into the feature behaviour, we are satisfied that momentumbased features are expected to play a significant role, as would be expected from the physics case. It is also of interest that only two of the ratios of eq. (3. 6) appear to be selected by the PCA, suggesting that the machine learning algorithm may be able to perform without needing all of the constructed variables which were used in the cut-and-count. We are therefore motivated to continue with a feature importance investigation which will use supervised learning to select the most valuable features.
In employing any feature selection, it is important to take the results with a grain of salt, and we should be careful how we interpret ranking. We consider the scenario where feature A and feature B are very similar, but B is always slightly less important than feature A, here feature B will always be ranked very low (as it does not add to the selection already made by feature A). However, if we then remove feature A, feature B will suddenly dramatically increase in importance. For this reason it is important to understand the physics variables employed in an analysis and their possible correlations, as well as carefully permuting through a feature importance investigation, which is employed in figure  3  (right). In this final feature investigation technique, the function uses a permutation to randomly shuffle features in a dataset, each time calculating the model score.
Hyperparameter selection Finally, we turn to the hyperparameter selection for the XGBoost model. The values of these parameters, which control the structure and functioning of the machine learning optimisation, directly affect the performance and predictive power of a machine learning toolkit  [36] , and so should be carefully considered. It has long since been established that different hyperparameters work for different datasets  [79] , and also that the tuning of such hyperparameters for a given dataset improves the performance of a given machine learning algorithm with respect to its default parameters  [36, 80, 81]. Manual optimisation of hyperparameters is possible, but automatic methods are more easily repeated and generalisable  [82] , and also can produce better results by covering larger parameter spaces.
Hyperparameters also often depend on each other, making tuning them independently unreasonable  [36]. Tuning may be carried out via a grid search  [83] , where the dimensions of the space of hyperparameters is reduced to a finite set of values and evaluated, or the aptly named random search method  [84] , where arbitrary configurations in hyperparameter space are tested, usually for a set number of iterations. Additional optimisation algorithms have also been studied  [36]. It is not possible to predict which hyperparameters will best suit a given model before they are optimised, and one set of parameters can not be widely applied across datasets, although a suspected range of validity can be provided, and is discussed below. The hyperparameters should be optimised using an automated method, where assumptions about the data can be introduced via the task (logistic or linear regression, or classification) and via bounds on the appropriate hyperparameters.
Feature utility The cut-and-count analysis of Ref. [29]  utilised sophisticated mass ratios which were chosen to provide a better discrimination between the signal and backgrounds. The utility of these ratios in a BDT algorithm were investigated first, by training and testing the BDTs at each benchmark with and without the ratios. We define three test cases; all ratios included, H T and S T values removed but ratios included, and ratios removed with H T and S T values included. We compare the use of the auc and F-score metrics on the training set, which is a subset of the full dataset. We compare the use of the F-score and auc metrics in each case (including all ratios, including only H T and S T and including no ratios or H T and S T ) by computing the ams score of the trained models on the test data.
Model generalisability Finally, we have investigated the applicability of models trained on a given benchmark to the other benchmark datasets. The ability to train a model on a certain benchmark and apply it across the mass spectrum would be advantageous as the BSM model at hand features a largely unconstrained two-dimensional parameter space of possible masses for the smuon and neutralino. It would be preferable to be able to train a BDT model on one benchmark, and apply it across a large mass range, or indeed across all mass combinations. This would be subject to the similarities in features across the benchmarks. One may expect significantly different kinematic behaviour across mass separations, as the available phase space will modify final momenta and angular distributions. It might be expected then that a model trained within a certain mass regime would only produce similar results on benchmarks with a similar mass ratio m μ/m χ.
Summary and outlook In this work we have outlined the application of a BDT algorithm to a generic particle physics problem. We have outlined the key concepts in such a study, including a comparison of the widely used auc metric to the F-score metric, where the latter may better handle the imbalanced datasets characteristic of particle physics problems. Indeed, we find that the F-score is a more stable choice of metric and achieves better results. An in-depth feature exploration and analysis was performed, where a PCA was used in conjunction with a Shapley analysis and a feature permutation tool in order to assess the utility of the features at hand. In such an analysis this step is critical in understanding the data at hand in order to use its features correctly in an analysis. We have compared the use of sophisticated features in previous cut-and-count analyses with respect to machine learning applications, and found that they may not be necessary in the given scenario.
Acknowledgements The authors would like to thank  Jack Araz  for useful discussions. ASC is supported in part by the  National Research Foundation of South Africa (NRF). LM is supported by the  UJ GES 4IR  initiative and thanks  Nikita Kazeev  for useful discussions. GEH is supported by  UJ  and would like to thank  Adrian Snyman  for assistance with running the codes on their servers.
EPHOR mega cohort approach For the EPHOR Mega Cohort approach, we build on an inventory of European cohorts, with extensive information on employment history conducted in collaboration with the OMEGA-NET project. 17, 18 This inventory (which is available at: https://occupationalcohorts. net/) currently includes over 140 cohorts adding up to millions of subjects and continues to grow. Actual pooling of cohort data across country and institute boundaries is challenging due to privacy and ethics legislations. Therefore, a framework is being developed using DataSHIELD  19, 20  for joint decentralized analyses across cohorts, through virtual pooling of the individual cohorts. Meta-analysis will be used when virtual pooling is not possible. Through this framework, combinations of cohorts that are selected for each research question can be flexibly combined, resulting in a dynamic EPHOR Mega Cohort.
A selected number of cohorts directly linked to the EPHOR consortium partners (≈10) has been invited for a pilot phase in which methods for data harmonization, meta data cataloging, and decentralized data analyses are developed. Additional cohorts will be invited for Mega Cohort analyses after research questions, and end points have been defined. Data on health end points, relevant nonoccupational exposures that may be directly or indirectly linked to the working life, and potential confounders will be harmonized across the cohorts. A defined set of core variables that should be derivable by the majority of relevant cohorts and will be frequently used as covariates in working life research (e. g. , employment history, smoking status, socioeconomic status, region/address, age) has been defined.
A data harmonization protocol is being developed based on the available variable labels, categories, and values for these variables in the pilot phase cohorts. This harmonization protocol will guide local harmonization of data by cohort owners. Based on these harmonized core variables, an online data catalogue will be developed. Employment histories will be used to estimate occupational exposures by EuroJEM, explained in more detail below. The following types of targeted research questions will be addressed in the Mega Cohort analyses: What are the most important working life risk factors for the development of a specific NCD over the life course? To what extent do working life exposures (including occupational, lifestyle, and SES) interact in their relation to an NCD? Are certain subpopulations more vulnerable? Can critical exposure time windows across the life course be identified? These targeted analyses will be performed in order to address predefined research questions focusing on specific (sets of) exposures and health outcomes.
22 In total, 4000 subjects who participated in ECRHS between 2010 2012 or in Constances between 2012 and 2016 will be invited for the EPHOR-specific follow-up. A participation rate of 75% is expected, resulting in approximately 3000 individuals with EPHOR-specific follow-up data. Follow-up among these individuals will include blood sampling, urine sampling, measurement of lung function, and questionnaires. Among these, a subgroup of around 400 mild asthmatics, with current occupational exposure to airborne irritants, will be selected. Jobs with airborne exposure to irritants will be identified based on a job exposure matrix (JEM). 23 hese 400 individuals will be subjected to a detailed one-week study, in which new data on external exposure, internal exposure, biomarkers, and omics will be collected, described below.
In total, approximately 800 participants currently working in a job involving night shifts or day shifts will be recruited. Among these individuals, new external exposure, internal exposure, biomarkers, and omics data will be collected, as described below for analyses of pathways related to shift work. The exposome protocol will also incorporate analyses to evaluate key body functions closely related to aging (e. g. , metabolic syndrome, cognitive function, and hallmarks of aging  26  ) and the development of cardiovascular disease through a combination of biomarkers, biochemical analyses, harmonized tests, clinical evaluations, and questionnaires.
Digital questionnaires To supplement sensor and passive sampler data collection, an app-based questionnaire  47  will be used to obtain information on daily habits and symptoms. This system involves questions on daily sleep and wakening times, commute times, working times, food/drink consumption times, use of personal protective equipment, stress, other exposure factors, symptoms, and contextual information regarding the occupational setting, i. e. , use of personal protective equipment. Collection of new high-resolution internal exposure and effect data in the case studies The internal exposome characterizes exposure biomarkers and biological pathways to link external exposure and health effects. In addition to blood collection using phlebotomy, minimally invasive biomatrices such as saliva, exhaled breath (EB) condensate, and urine will also be collected for analyses of several biomarkers.
We will compile these methods, apply, and compare them in the EPHOR Mega Cohort. In parallel, we will investigate to what extent existing ETR models can be modified to allow for multiple exposures and nonlinear exposure-response relations and to incorporate biological data to strengthen these models (effect markers, omics). 62 Hierarchical regression will be used as a general approach, including Semi-Bayes adjustment, that aims at improving the validity of standard maximum-likelihood estimates in the presence of multiple comparisons by incorporating similarities between the exposures of interest in a second-stage model. 63.
Biological pathways Internal exposure and effect data (omics, biomarkers) will be subjected to pathway analyses to understand relations between external exposure and health outcome. The first level of analysis involves qualitative pathway analysis to retrieve sets of genes, the expression of which has changed due to exposure. This analysis will include combinations of different omics data sets and also compare exhaled breath and blood omics to develop noninvasive biomarkers. 5] [66] [67]  The second level of bioinformatics involves quantitative pathway-based exposure to health effect modeling. Existing scientific knowledge, exposure, and pathway content will be retrieved from text mining of scientific literature and toxicological databases, to construct qualitative exposure to adverse outcome pathway networks.
Ethical considerations The approach adopted in EPHOR for the advancement of occupational health gives rise to ethical and legal challenges regarding in particular issues of consent, privacy, and transborder flow of health-related data. For the large-scale pooling of existing cohorts from different countries, data transfer, especially across borders, has proven difficult due to legal and ethical challenges. This is addressed through the use of DataSHIELD software, which secures the privacy-preserving analysis of data that remain at the institute that owns the data. Data access agreements are being realized in compliance with the general data protection regulation and the national regulatory frameworks of participating countries. Collection of new data for the case studies will be undertaken in multiple study centers spread over different countries.
Working life exposome toolbox and stakeholder involvement The knowledge and tools developed in the EPHOR project will be tailored to three groups of stakeholders. For scientists, methods and tools will be made available to expand the current knowledge base on the working life exposome in relation to health. These methods and tools include inventories, protocols, proof of concepts, decision support, and visualization tools and tutorials. For policy makers, data and tools for analyzing and assessing the impact of the working life exposome will be produced to support development of evidence-based and cost-effective preventive policies. For occupational health practitioners, data and tools for developing evidence-based and cost-effective preventive actions will be made available. All groups of stakeholders will be consulted during the development of all aspects of the toolbox, to take into account the values, needs, and expectations of these stakeholders during development.