# Reproducibility and Data Storage for Active Learning-Aided Systematic Reviews
###### Abstract  
In the screening phase of a systematic review, screening prioritization via active learning effectively reduces the workload. However, the PRISMA guidelines are not sufficient for reporting the screening phase in a reproducible manner. Text screening with active learning is an iterative process, but the labeling decisions and the training of the active learning model can happen independently of each other in time. Therefore, it is not trivial to store the data from both events so that one can still know which iteration of the model was used for each labeling decision. Moreover, many iterations of the active learning model will be trained throughout the screening process, producing an enormous amount of data (think of many gigabytes or even terabytes of data), and machine learning models are continually becoming larger. This article clarifies the steps in an active learning-aided screening process and what data is produced at every step. We consider what reproducibility means in this context and we show that there is tension between the desire to be reproducible and the amount of data that is stored. Finally, we present the RDAL Checklist (Reproducibility and Data storage for Active Learning-Aided Systematic Reviews Checklist), which helps users and creators of active learning software make their screening process reproducible. systematic review; meta-analysis; active learning; transparency; open science; data storage; reproducibility +
Footnote †: journal: Medical  
thousands of publications that must be read and screened for relevance, literature screening is extremely time-consuming [7]. Artificial intelligence can help to speed up the process of searching through large amounts of text data. Machine-supported pipelines have been developed that assist in finding relevant texts for search tasks; for overviews, see [8; 9; 10; 11; 12]. A well-established approach to increase the efficiency of title and abstract screening is screening prioritization [13; 14] via active learning (AL) [15]: a constant interaction between a human and a machine; for an explanation, see Box 1 and Figure 1. Active learning is extremely effective in screening large amounts of textual data [9; 14; 16; 17; 18; 19] and is successfully implemented in screening software like Abstrackr [17], ASReview [9], Colandr [18], FASTREAD [16], Rayyan [20], RobotAnalyst [21], Research Screener [22], DistilerSR [23], and robotreviewer [24]. Note that this is different from the classical notion of active learning in machine learning; see Box 2 for an explanation of the differences. **Box 1. ** Active learning explained. Researcher-in-the-loop active learning (RITL) is a form of active learning (AL) where the following three criteria need to be satisfied [9; 25]:  
* The primary output is a list of labeled records, and the machine learning model is only secondary. * The annotator needs to have seen the relevant records at the end of the active learning process. * The active learning process should be transparent and reproducible. The process starts with a database search to obtain a dataset of unlabeled records containing metadata such as titles, abstracts, and keywords of scientific papers. This is followed by constructing an initial labeled _training set_, where the labels are provided by the _annotator_. This consists of at least one _label relevant_ and _label irrelevant_ record. Next, an active learning model needs to be selected, including a _feature extraction_ technique, a _classification algorithm_ (i.e., a machine learning model), a query strategy, and a balancing strategy to deal with the extreme imbalance between the number of relevant and irrelevant records in the data. Then, the active learning cycle starts (see also Figure 1):  
* The chosen model is trained on the labeled records, and the model is used to produce _relevance scores_ for all unlabeled records. * The annotator samples a record from the unlabeled records, for example, the record with the highest relevance score (certainty-based sampling [26]).