dataesr/scientific-paragraphs-categorization huggingface dataset:
Background Work circumstances can substantially negatively impact health. To explore this, large occupational cohorts of free-text job descriptions are manually coded and linked to exposure. Although several automatic coding tools have been developed, accurate exposure assessment is only feasible with human intervention. Methods We developed OPERAS, a customizable decision support system for epidemiological job coding. Using 812,522 entries, we developed and tested classification models for the Professions et Catégories Socioprofessionnelles (PCS)2003, Nomenclature d'Activités Française (NAF)2008, International Standard Classifications of Occupation (ISCO)-88, and ISCO-68. Each code comes with an estimated correctness measure to identify instances potentially requiring expert review. Here, OPERAS' decision support enables an increase in efficiency and accuracy of the coding process through code suggestions.
This implies that only JEMs using these national classifications can be applied, limiting the use of these tools in global exposure assessment studies requiring other classifications. Consequently, we developed, tested, and validated OPERAS, a decision support system for epidemiological job coding that utilizes ML-based classification models for four (inter)national classification systems. Its goal is to support expert coders during occupational coding through both automatic coding as well as decision support through code suggestions. Our evaluation shows that OPERAS outperforms existing automatic coding tools and expert coders, leading to a substantial workload reduction and highly accurate exposure assessment.
Methods OPERAS has been developed via six phases. First, we acquired datasets and identified classification performance barriers. Additionally, we specify the origin, available input classes, and used occupational classification systems. Second, to enable complete code suggestions, we removed missing and incomplete codes during data preparation. Furthermore, to ensure optimal classification performance, we reduced the dimensionality and retained crucial information of the free-text job descriptions through multiple Natural Languange Processing (NLP) techniques. Third, using state-of-the-art gradient tree boosting, we trained the classification models. We optimized for generalizability by using pre-defined parameter settings and the inclusion of generally described job descriptions from the coding index in the training set.
Fourth, using prediction accuracy and Cohen's kappa, we evaluated the classification models on their classification performance and present its impact on the occupational coding process. Fifth, based on two occupational inter-coder reliability studies, we compared OPERAS' inter-coder reliability to expert coders. Sixth, using four JEMs, we evaluated the NAF, PCS, ISCO-88, and ISCO-68 classification models on their exposure assessment for two groups: 1) all individuals, and 2) exposed individuals. In the following subsections, each phase will be explained in detail. Datasets. Using French 23 , Asian  24  , and Dutch 25 datasets, we developed OPERAS' classification models. These datasets respectively contain French, English, and Dutch manually expertcoded free-text job descriptions in four hierarchically structured occupational/activity sector coding classifications.
Each classification consists of multiple numbers and/or letters (e. g. , "211A") where each added (combination of) character(s) provides additional detail about the job description. For example, the major, sub-major, minor, and unit groups of the ISCO-88 code "2221" are "Professionals" (2), "Life Sciences and Health Professionals" (22), "Health Professionals" (222), and "Medical Doctors" (2221), respectively. For insufficiently detailed job descriptions the expert coder will replace not codable levels with a pre-defined character (e. g. , "#"). In the following paragraphs, each dataset and coding classification will be further described. The descriptive statistics of each dataset can be found in Tables  2  and 3. Constances is a general-purpose cohort of French adults aged 18-69 with a focus on occupational and environmental factors  23.
The dataset contains free-text answers to open-ended questions regarding participants' occupations held and sector of activity. This information was manually coded into the French Nomenclature d'Activités Française (NAF)2008 and Professions et Catégories Socioprofessionnelles (PCS)2003 classifications by expert coders using a dedicated application showing the entire summary of a participant's occupational history  26. Here, a NAF code describes one's activity sector and consists of four digits and one letter (e. g. , "4231A"), whereas a PCS code describes occupation and consists of three digits and a letter (e. g. , "211A"). The Constances cohort has been reviewed and approved by the Institutional Review Boards of the French Institute of Medical Research and Health, the "Commission Nationale de l'Informatique et des Libertés", and the "Conseil National de l'Information Statistique".
Written informed consent was obtained from all participants. Access and permission to reuse the restricted data for the current study were provided by the Principal Investigators (PIs) of the original study. The AsiaLymph dataset is a hospital-based case-control study of lymphoma among Chinese in Eastern Asia  24. This dataset contains free-text traditional and simplified Chinese occupations, tasks, employers, and products that were translated into English. They were manually coded using the International Standard   The Lifework cohort is a large federated prospective cohort from the Netherlands that quantifies the health effects of occupational and environmental exposure  25. The dataset includes job names, descriptions, and company types that were manually coded into ISCO-68 occupational codes, which is an earlier version of ISCO-88.
Data preparation. The Constances, Asialymph, and Lifework datasets contain 637,148, 36,179, and 12,120 entries, respectively. To ensure complete code suggestions, entries with incomplete codes were considered nonviable for classification model training. After data cleaning, 281,418, 483,090, 36,007, and 12,007 viable entries remained to train the NAF, PCS, ISCO-88, and ISCO-68 models, respectively. A large number of incomplete NAF and PCS entries were removed using this strategy due to the insufficiently detailed descriptions before occupational coding. However, this was warranted as it ensures optimal decision support during the occupational coding process through complete code suggestions. As all data were collected in an open-ended format, tokenizing, removing punctuation marks, converting all input to lowercase, and stemming all words to their root form allowed for the reduction of the random noise and dimensionality of the data  27.
In the current context, this would result in similar job descriptions being represented as such. When multiple input classes per occupational code were required for classification model training (e. g. , job description and sector), we summed the resulting feature vectors to combine all information  31. To increase the generalizability of the models, we randomly split up the prepared data from each dataset into a training (60%), test (30%), and validation (10%) set, and supplemented the training set of each occupational classification with job descriptions from their respective coding indexes. As classification models tend to overly represent majority classes, the class sizes of the outcome categories should be balanced  32. However, algorithms that synthetically provide data balance could not be applied to the current datasets due to the difference in class size being too large  33, 34.
Classification model training. Using the state-of-the-art gradient tree boosting algorithm XGBoost, we developed the NAF, PCS, ISCO-88, and ISCO-68 classification models  35  (see Fig. 1 ). We chose this algorithm due to a study performed by Schierholz and Schonlau  22  , where multiple Machine Learning (ML) algorithms were evaluated on their classification performance based on five occupational datasets. They found that, when the training data was supplemented with the job descriptions from the coding index, the XGBoost algorithm performed best. XGBoost generates an ensemble of weak Classification and Regression Trees (CARTs) and combines their predictions to produce a strong, accurate model  35. The model is trained by minimizing a regularized objective function, which includes a loss function to measure the error between the target class (i.
Schierholz and Schonlau  22  empirically optimized the hyperparameters for the largest occupational dataset and proposed a default setting. As the datasets used in ref. 22  are similar to the current datasets in terms of data collection (i. e. , free-text job descriptions), (number of) outcome categories (i. e. , occupational codes), and average word count, we expected similar relative performance when the proposed default set of hyperparameters is used in the current study  36. Following the instructions from Schierholz and Schonlau  22  we have set lower values for η and max_depth for optimal classification performance. Hence, we adopted and used the following hyperparameter settings for the development of each classification model: Early stopping if performance does not improve for 1 round, η = 0.
6, max_delta_step = 1, max_depth = 20, γ = 1. 5, λ = 1e -4 , min_child_weight = 0, subsample = 0. 75, colsample_by_tree = 1, and colsample_by_level = 1. We trained the NAF, PCS, and ISCO-68 classification models in the original language of the corresponding dataset, whereas we trained the ISCO-88 model using its English translation. Here, the classification models were solely trained using the training data. The validation set was used during training to allow for early stopping of training to reduce the chance of overfitting  37. We performed feature selection to find the combination of input classes that yield the best predictive performance for each dataset. Here, we used an exhaustive wrapper approach, where a classification model is trained for each combination of input classes in a dataset and evaluated to find the combination which provides the best predictive performance  38.
Using exclusively the test set, we measured the performance of the classification models. Evaluation metrics. Using the entries in the test set of the corresponding datasets, we assessed the accuracy and inter-coder reliability of OPERAS' classification models using accuracy and Cohen's Kappa (κ), respectively. Here, we consider the manually expert-coded job descriptions from the aforementioned datasets as the gold standard. We deem accuracy to be an important metric in the current study, as it indicates the proportion of occupational codes which will potentially require less coding time during the occupational coding process. We define accuracy as the number of correctly predicted codes relative to the total number of predicted codes  39, 40.
Accuracy ð%Þ ¼ Number of correct predictions Total number of predictions 100% For each classification model, we computed the accuracy per code level and major occupational group. Since OPERAS autonomously codes the entries from the test set, it can be considered an (automatic) coder. Hence, OPERAS' inter-coder reliability can be calculated using κ, which takes into account the correct prediction occurring by chance. This is defined as: ðP o À P e Þ=ð1 À P e Þ; ð2Þ where P o represents the probability of overall agreement over the label assignments between the classifier and true process. P e represents the chance agreement over the labels. This is defined as the sum of the proportion of examples assigned to a class times the proportion of true labels of that class in the dataset  41. OPERAS contains a feature to automatically process predicted codes with a confidence score above a pre-defined threshold.
The accuracy and κ in both groups are calculated using Eq. (1) and Eq. (  2 ) respectively. Additionally, we grouped the predicted exposure assessments on the confidence score of the original code in intervals of 5% ranging from 0% to 100% and evaluated the accuracy of each group. To resemble OPERAS' real-world usage of the automatic coding function, we used the minimum confidence score of the NAF or PCS code for the Formaldehyde-JEM and Silica-JEM. In the following sections, we provide additional information on the content and application of the JEMs. Formaldehyde-JEM. The Formaldehyde-JEM has been developed by "Santé Publique France" (the French National Health Surveillance Agency), as part of its "Matgéné" program to assess formaldehyde exposure in the French population  43. It has been developed by a team of occupational experts through a metaanalysis of 469 scientific, medical, and technical sources.
Silica-JEM. The Silica-JEM assesses the crystalline silica exposure in the French population  44. Similar to the Formaldehyde-JEM, it has been developed by a team of occupational experts from "Santé Publique France", as part of its "Matgéné" program through a meta-analysis of 469 scientific, medical, and technical sources. This resulted in a matrix evaluating exposure to crystalline silica based on occupational histories describing occupation, sector, and corresponding dates between 1947 and 2007. Similar exposure indices, namely the probability, intensity, and frequency of exposure are provided for each combination of NAF2008 and PCS2003 codes for different calendar periods. As this study concerned a meta-review, no approval by an Institutional Review Board was needed. For each job episode, we applied the Silica-JEM to the gold standard manually coded and OPERAS-predicted NAF and PCS codes.
Additionally, OPERAS provides a confidence score indicating the probability of correctness of a suggested code for each prediction. This is used to interpret the quality and usefulness of a classification model (see Fig. 2 ). Model evaluation. NAF and PCS use Constances' input-classes occupation and sector with an accuracy per major occupational group ranging between 52. 17-95. 91% and 32. 45-90. 56%, respectively. 57. 41% of the suggested NAF codes have a confidence score in the 95-100% range, with an accuracy of 97. 02%. 43. 11% of the suggested PCS codes are within this range, with 94. 47% correctly predicted. Using the Formaldehyde-JEM and Silica-JEM, exposure assessment for all individuals gave 98. 09% (κ = 0. 84) and 98. 41% (κ = 0. 67) accuracy, respectively. Here, 31. 28% of the suggested codes had a 95-100% minimum confidence score for both the Formaldehyde-JEM and Silica-JEM, with 99.
Discussion To build robust generic classification models, we refrained from tweaking hyperparameter settings. Instead, we used default values and included generalized job descriptions from the coding indexes in the training data  22. This provided us with a good baseline performance that can be expected to generalize to other datasets  36. Further, OPERAS proposes a limited set of probable codes for similar job descriptions. This holds the potential for improved inter-coder reliability among expert coders utilizing OPERAS  13  , which can contribute to the occupational coding process' robustness. Moreover, OPERAS' adaptable automatic coding can enable a substantial workload reduction for a wide range of applications. For example, the default 95% confidence score threshold could be lowered to increase workload reduction for applications that are tolerant to less accurate codes or solely require high accuracy on lower coding levels.
Differences in classification performance can be attributed to the difference in available job descriptions per classification model. The NAF and PCS were respectively trained with 281,418 and 483,090 entries, whereas the ISCO-88 and ISCO-68 models were trained with respectively 36,007 and 12,007 entries. Additionally, the difference in combined job description length between datasets could also have contributed to the difference in classification performance. Longer job descriptions decrease coding reliability if the additional information does not directly correspond to the definition of the occupational code  12. This is due to an increase in potential occupational codes that can be assigned to a job description. In classification model training, the information increases variance within outcome categories, resulting in decreased classification performance  49.
2b, c ). However, a moderate increase in reliability can be observed during the dichotomous evaluation for exposed individuals of the Formaldehyde-JEM and Silica-JEM. The correlation coefficients for all exposure levels also show strong and moderate correlation for respectively the same JEMs. However, they are not near perfect as evidenced by their inter-rater reliability for all individuals. Hence, to ensure reliable health outcomes it is crucial to take these factors into account during (semi-automatic) occupational coding  47, 50. The class imbalance in current datasets presents a substantial challenge for the development and deployment of the classification models. This is due to the fact that ML models tend to overclassify the majority class, thereby compromising the classification performance of minority classes  32.
This is a large issue, particularly in the current domain where the misclassification of minority classes can have critical implications on health outcomes. Traditional methods designed to tackle this issue, such as SMOTE, are in this case inadequate as the representation of the minority class in the datasets is excessively low  33, 51. As such, it is important that alternative data augmentation techniques are developed that can effectively enhance the representation of minority classes in datasets where the representation of the minority class is (close to) one. This will contribute to a more balanced and accurate performance of ML models, thereby increasing their generalizability and applicability in critical domains. OPERAS' classification models are trained using one occupational dataset each.
However, job description characteristics often differ between occupational datasets, possibly resulting in a lower out-of-distribution performance  11, 20. Additionally, OPERAS has been trained and tested on expert-coded occupational data coded on the least aggregated level. Although OPERAS' data preprocessing pipeline can manage and produce predictions for ambiguous or incomplete data, its performance in such scenarios remains untested. Hence, to assess the generalizability of OPERAS' classification models, its evaluation could be extended using external validation sets. To further improve the classification model performances, other classification techniques, hyperparameter settings, and the use of additional datasets could be considered  52. Additionally, OPERAS' human-model cross-validation is currently based on external occupational inter-coder reliability studies containing different occupational characteristics compared to the current ones  9, 11, [23] [24] [25].
Hence, to gain more insight into OPERAS' performance against expert-coders, additional cross-validations using the same underlying datasets could be performed. With OPERAS, a decision support system for epidemiological job coding of free-text job descriptions is introduced, which outperforms both expert coders  9, 11  and state-of-the-art coding tools  18, 53. This is achieved through the ML-based classification models for four (inter)national classification systems. OPERAS' codes are accompanied by confidence scores that can be used by the expert coder to partially automate the coding process with accurate exposure assessment, resulting in a substantial workload reduction. Additional insight into OPERAS' classification performance can be gained through the use of external data sets and comparisons with expert coders using the same underlying data.
Acknowledgements The authors thank the  Constances team  for providing the job histories of the cohort participants and  Fabien Gilbert  ( IRSET, University of Angers ) for supervising their coding. The authors would also like to thank the  National Cancer Institute  for providing the AsiaLymph job descriptions. The OPERAS project discloses support for the research of this work from  ANSES  [ PNR EST-2018/1/106 ]. The  Constances cohort  is supported and funded by the  Caisse Nationale d'Assurance Maladie (CNAM). It benefits from a grant from the  Agence nationale de la recherche  [ ANR-11-INBS-0002 ] and from the  French Ministry of Research. Lifework was funded by the  Netherlands Organisation for Health Research and Development (ZonMw)  within the  Electromagnetic Fields and Health Research programme  [ 85200001 ] and [ 85500003 ].
S. P. is supported by the  Exposome Project for Health and Occupational Research (EPHOR)  which is funded by the  European Union 's  Horizon 2020 research and innovation programme  [ 874703 ]. R. C. H. V. is supported by the  Gravitation program  of the  Dutch Ministry of Education, Culture, and Science and the Netherlands Organization for Scientific Research  through the  EXPOSOME-NL [024. 004. 017 ]. N. R. , Q. L. , and M. F. were supported by the  Intramural Research Program of the Division of Cancer Epidemiology and Genetics, NCI ,  NIH.
Data availability The Constances  23  , Asialymph  24  , and Lifework  25  datasets are not publicly available. Access to these datasets should be requested from the authors of the original studies. The Formaldehyde  43  and Silica  44  JEMs can be consulted on the Exp-pro portal of "Santé publique France" at https://exppro. santepubliquefrance. fr/matgene. Access to the computer files of the matrices for their use in the context of epidemiological studies should be requested from the authors of the original study. The ALOHA  45  and DOM  46  JEMs are not publicly available and should be requested from the authors of the original study. Source data underlying the graphs in Fig. 2  are available as Supplementary Data 1.
Code availability The development and evaluation code of the current classification models is available at https://zenodo. org/records/8390811  54. The used test sets from the Constances, Asialymph, and Lifework datasets can only be obtained with permission from the authors of the original studies. The code has been developed in Python 3. 8 using the following Python packages: flair 0. 8. 0. post1, nltk 3. 6. 2, numpy 1. 19. 5, pandas 1. 2. 4, scikit-learn 0. 24. 1, tokenizers 0. 10. 2, and XGBoost 1. 5. 2. The OPERAS software is still in development and future versions might use classification models utilizing different packages and techniques than the ones currently described.
Lake fish telemetry group (LFTG) Fine-scale telemetry systems tend to be located in freshwater systems, mainly lakes, due to the need for enclosed systems to perform long-term tracking without losing tracked fish through emigration  [1]. Lakes also allow dense receiver networks to conduct positioning, which is less tractable in saltwater environments where acoustic signals are attenuated more quickly. The LFTG originated from the desire to build collaborative projects from groups working with fine-scale acoustic telemetry systems in freshwater lentic environments. The concept emerged during a lunch time meeting in Berlin among Ivan Jarić, Milan Říha, Robert Arlinghaus and Christopher Monk, during which they intended to organize a joint project investigating the winter ecology of freshwater fish species in their respective acoustic telemetry datasets.