DOI: 10.5061/dryad.ncjsxkszr
title: Not so weak-PICO: Leveraging weak supervision for Participants, Interventions, and Outcomes recognition for systematic review automation

description: Objective: PICO (Participants, Interventions, Comparators, Outcomes)
 analysis is vital but time-consuming for conducting systematic reviews
 (SRs). Supervised machine learning can help fully automate it, but a lack
 of large annotated corpora limits the quality of automated PICO
 recognition systems. The largest currently available PICO corpus is
 manually annotated, which is an approach that is often too expensive for
 the scientific community to apply. Depending on the specific SR question,
 PICO criteria are extended to PICOC (C-Context), PICOT (T-timeframe), and
 PIBOSO (B-Background, S-Study design, O-Other) meaning the static
 hand-labelled corpora need to undergo costly re-annotation as per the
 downstream requirements. We aim to test the feasibility of designing a
 weak supervision system to extract these entities without hand-labelled
 data. Methodology: We decompose PICO spans into its constituent entities
 and re-purpose multiple medical and non-medical ontologies and
 expert-generated rules to obtain multiple noisy labels for these entities.
 These labels obtained using several sources are then aggregated using
 simple majority voting and generative modelling approaches. The resulting
 programmatic labels are used as weak signals to train a weakly-supervised
 discriminative model and observe performance changes. We explore mistakes
 in the currently available PICO corpus that could have led to inaccurate
 evaluation of several automation methods. Results: We present Weak-PICO, a
 weakly-supervised PICO entity recognition approach using medical and
 non-medical ontologies, dictionaries and expert-generated rules. Our
 approach does not use hand-labelled data. Conclusion: Weak supervision
 using weak-PICO for PICO entity recognition has encouraging results, and
 the approach can potentially extend to more clinical entities readily.
description: This upload contains four main zip files. 
 <strong>ds_cto_dict.zip:</strong> This zip file
 contains the four distant supervision dictionaries (P: participant.txt, I
 = intervention.txt, intervetion_syn.txt, O: outcome.txt) generated from
 clinicaltrials.gov using the Methodology described in Distant-CTO (https://aclanthology.org/2022.bionlp-1.34/). These dictionaries were used to create distant supervision labelling functions as described in the Labelling sources subsection of the Methodology. The data was derived from https://clinicaltrials.gov/<br><br> <strong>handcrafted_dictionaries.zip:</strong> This zip folder contains three files 1) gender_sexuality.txt: a list of possible genders and sexual orientations found across the web. The list needs to be more comprehensive. 2) endpoints_dict.txt: contains outcome names and the names of questionnaires used to measure outcomes assembled from PROM questionnaires and PROMs. and 3) comparator_dict: contains a list of idiosyncratic comparator terms like a sham, saline, placebo, etc., compiled from the literature search. The list needs to be more comprehensive.<br><br> <strong>test_ebm_correctedlabels.tsv:</strong> EBM-PICO is a widely used dataset with PICO annotations at two levels: span-level or coarse-grained and entity-level or fine-grained. Span-level annotations encompass the full information about each class. Entity-level annotations cover the more fine-grained information at the entity level, with PICO classes further divided into fine-grained subclasses. For example, the coarse-grained Participant span is further divided into participant age, gender, condition and sample size in the randomised controlled trial. This dataset comes pre-divided into a training set (n=4,933) annotated through crowd-sourcing and an expert annotated gold test set (n=191) for evaluation. The EBM-PICO annotation guidelines caution about variable annotation quality. Abaho et al. developed a framework to post-hoc correct EBM-PICO outcomes annotation inconsistencies. Lee et al. studied annotation span disagreements suggesting variability across the annotators. Low annotation quality in the training dataset is excusable, but the errors in the test set can lead to faulty evaluation of the downstream ML methods. We evaluate 1% of the EBM-PICO training set tokens to gauge the possible reasons for the fine-grained labelling errors and use this exercise to conduct an error-focused PICO re-annotation for the EBM-PICO gold test set. The file 'test_ebm_correctedlabels.tsv' has error corrected EBM-PICO gold test set. This dataset could be used as a complementary evalution set along with EBM-PICO test set.<br><br> <strong>error_analysis.zip: </strong>This .zip file contains three .tsv files for each PICO class to identify possible errors in about 1% (about 12,962 tokens) of the EBM-PICO training set. 
description: All the datasets could be opened using text editors or Google
 sheets. The .zip files in the dataset can be opened using the
 archive utility on Mac OS and unzip functionality in Linux. (All Windows
 and Apple operating systems support the use of ZIP files without
 additional third-party software)