Cost-effectiveness of MAG and automation for maintenance of a living Covid-19 map
Creators: name: Arno, Anneliese
Publisher: name: Open Science Framework
Year: 2022
Subjects: subject: Clinical Epidemiology
subject: Public Health
subject: Medicine and Health Sciences

description: Abstract Background: Globally, the research sector spends billions of dollars each year on finding newly published research articles that meet eligibility criteria for indexing in living maps, specialised registers, and tertiary databases on various topics. Machine learning (ML) tools, developed to semi-automate the process of selecting new, eligible articles, have been in use for several years. However, until now, searching for eligible articles has relied primarily on information specialists conducting Boolean searches of secondary databases and manually processing the results, including deduplication between these multiple sources. Microsoft Academic Graph (MAG) is a continuously updated, open access dataset comprising &gt;250 million bibliographic records of research articles from across science, connected in a large network graph of conceptual and citation relationships. Searching a single, comprehensive source, rather than multiple databases, would save time and resources; and MAG’s network graph structure provides additional metrics that can be exploited as features in ML. However, research is needed to establish the relative advantage of using MAG as a single source, compared with conventional searches of multiple databases. In the current use scenario, we needed to establish: (a) How comprehensive is the MAG dataset as a single source for identifying records of study reports eligible to be indexed in our living map of COVID-19 research (i.e., recall)?, and (b) Can we can identify eligible study reports from the MAG dataset with an acceptably high level of specificity (i.e., precision)? We therefore conducted an economic evaluation to assess the costs and effects of our novel MAG-enabled workflow for semi-automated study identification in our living map of COVID-19 research. Methods: We conducted a pragmatic, eight-arm cost-effectiveness analysis (simulation study) to assess the costs, recall and precision of our semi-automated MAG-enabled workflow versus conventional searches of MEDLINE and Embase (with and without ML classifiers, active learning and/or fixed screening targets) for maintaining a living map of COVID-19 research. The analysis adopted a single employer (university) perspective. Workflow performance data were assembled in EPPI-Reviewer, an online systematic review software. Resource use data (time use) were collected from information specialists and other researchers involved in map production. Illustrative unit costs were obtained from university salary scales (GBP and AUD, 2021), and the time horizon was one month. Results: MAG-enabled workflows dominated MEDLINE-Embase workflows in both the base case and sensitivity analyses. At one month (base case analysis) our MAG-enabled workflow with ML classifiers, active learning and fixed screening targets identified n=469 more new, eligible articles for inclusion in our living map – and cost £3,179 GBP ($5,691 AUD) less – than conventional MEDLINE-Embase searches without any automation or fixed screening targets. Conclusions: This study indicated that our novel continuous surveillance workflow, comprising automated searches of MAG as a single source, exploiting network graph and text features of records, combined with automated selection and prioritisation of those records for manual screening, would enable us to maintain our living map of COVID-19 research more effectively and efficiently than we could if we continued to use conventional search and study selection methods. We therefore adopted this MAG-enabled workflow to maintain our map. When implemented at scale, MAG-enabled continuous surveillance workflows have the potential to revolutionise study identification methods for living maps, specialised registers, databases of research studies and/or collections of systematic reviews, by increasing their recall and coverage, whilst also reducing production costs.