# Reproducibility and Data Storage for Active Learning-Aided Systematic Reviews
###### Abstract
* The annotator screens this record and provides a label, i.e., relevant or irrelevant, and goes back to step 2. If there are enough new labeled records, step 1 is also triggered. This cycle is repeated until the annotator thinks they have seen all relevant records. The list of relevant records is the output of the system and will be used in the subsequent data extraction phase. The goal is to save time by screening fewer records than exist in the entire pool because the active learning model puts all of the most likely relevant records at the front of the list. Figure 1: The human-in-the-loop (HITL) active learning cycle with two types of output: on the top right, the trained model, which is the main output in the classical active learning; and at the bottom left, all relevant papers, which is the main output of researcher-in-the-loop (RITL) active learning. **Box 2. ** Differences between classical active learning and RITL. Classical active learning refers to a set of techniques where a model is trained on a subset of the available data, and then actively selects additional data points for which it requests labels. The goal is to improve the performance of the model while minimizing the amount of data that needs to be annotated. The main output is a trained model that can be used on unseen records, and the annotated data is a byproduct. It can be particularly useful in cases where obtaining new labels is expensive, for example, because a (human) expert needs to look at the unlabeled records. Similarly, in the RITL set-up, obtaining new labels is an expensive process. However, unlike in the classical set-up, the main output is the set of labeled records, not the trained model. An example of classical active learning would be training a model to classify images. The model selects the images for which a human annotator should provide labels. The model selects the images from which it can learn the most, so that the annotator needs to label as few of them as possible. In the end, the trained model can be used to classify new unseen images. The reproducibility of the process enhances our trust in the process; it may be more important, however, that others can reuse the results and data to improve or analyze the model. A typical application of RITL would be to find all relevant articles for determining a dosage guideline for a medical drug. First, databases are searched for possible articles, and these articles are then screened by a human annotator. Instead of annotating all articles, a model is iteratively trained to suggest the most relevant articles the annotator should look at first. The list of relevant articles is the main output of the system. It is important that the human annotator sees all relevant articles because we do not want the model to make the final decision about whether an article is relevant. The main reason we want the process to be transparent and reproducible is to increase the trust in the dosage guideline. Obviously these differences between classical and RITL active learning have implications about what it means to be reproducible, and what data is important to store. Clearly in the classical setting it is more important to store information regarding the training of the model, while in the RITL setting, the labeling decisions have higher priority. As we explain in Section 3, the model data are typically larger than the labeling data. In the classical setting, the larger model data are the primary output. In contrast, in the RITL setting, the labeling data are the primary output and storing the secondary model data means a large increase in storage size. Therefore, data storage size is more of a consideration for RITL. While the active learning pipeline can significantly reduce the time spent on systematically screening, it also presents challenges regarding transparency and repro-ducibility. At the end of the AL-aided screening process, there is a set of seen and la-beled papers and a set of unseen papers without labels.