# Reproducibility and Data Storage for Active Learning-Aided Systematic Reviews
###### Abstract
To understand why the anno-tator did not see these precise papers, we need to have insight into the model's deci-sions at any time in the process. This presents theoretical and practical challenges. Although the field of explainable Artificial Intelligence is gaining lots of attention [27], it can be difficult or even impossible to explain how a system comes to its exact deci-sion, especially when neural nets are used [28]. But before explainable Artificial Intelligence can even be applied to an active learning-aided screening pipeline, the model and the data the model produces should be stored. This presents difficulties for two reasons. Firstly, text screening with active learning is an iterative process, but the labeling decisions and the training of the active learning model can happen independently of each other in time. So it is not trivial to store the data from both events in such a way that you can still know which iteration of the model was used for each labeling decision. It is not necessarily the case that a new model will be trained after each labeling decision. Secondly, many iterations of the active learning model will be trained throughout the review, producing an enormous amount of data (think of many gigabytes or even terabytes of data), and machine learning models are continually becoming larger using even more model parameters [29]. Together this can add up to an undesirable amount of data when naively storing all the data produced at every iteration of the active learning pipeline. Also, many studies fail to report enough detailed information about the datasets and machine learning algorithms used, limiting the reproducibility of the studies assessed [30; 31]. Thus, in AL-aided systematic screening, there is a tension between the desire for reproducibility and transparency on the one hand, and the practical difficulties of storing and interpreting machine learning models on the other hand. More data must be saved if a more complete and straightforward reproducible result is wanted. In classical systematic reviews, there are clear guidelines on what to report regarding the output and the process of creating the review [4]. However, there is no such consensus for AL-aided systematic reviews, and PRISMA only offers general recommendations [32]:  
_Specify the methods used to decide whether a study met the inclusion criteria of the review, including how many reviewers screened each record and each report retrieved, whether they worked independently, and if applicable, details of automation tools used in the process._  
In the expanded checklist (source: [[https://prisma-statement.org//documents/PRIS](https://prisma-statement.org//documents/PRIS)]([https://prisma-statement.org//documents/PRIS](https://prisma-statement.org//documents/PRIS)) MA_2020_expanded_checklist.pdf (accessed on 25 April 2024)), this is elaborated further:  
_Recommendations for reporting in systematic reviews using automation tools in the selection process:_  
* _Report how automation tools were integrated within the overall study selection process._ [...]
* _If machine learning algorithms were used to prioritize screening (whereby unscreened records are continually re-ordered based on screening decisions), state the software used and provide details of any screening rules applied._  
However, as we demonstrate in the current paper, the updated PRISMA guidelines are not enough when using AL-aided pipelines. In AL-aided screening, this is more difficult because the annotator sees only part of the data. Suppose two people start reading the same dataset. In that case, the active learning model might suggest different records to read because they started with a different training set or selected a different model, there was difference of opinion between the annotators, or because of one of many more possible reasons. As a result, they might end up with different results (i.e., a different set of seen and labeled records), even though the dataset and inclusion/exclusion criteria were identical.