# Reproducibility and Data Storage for Active Learning-Aided Systematic Reviews
## 4 A Trade-Off between Reproducibility and Data Storage  
In practice, there is a trade-off between data size on the one hand and ease of reproducibility on the other hand. By using extra data storage, we can make it easier to reproduce the state of the screening at a certain iteration or moment in time. However, when these data (model parameters, relevance scores, etc.) become much larger than the primary output (a list of labeled records), this becomes a problem. Therefore, we propose only to store some essential components to make the output of the AL-aided pipeline transparent and reproducible; for an overview, see Table 1.  
At one extreme of the spectrum between data size and time, the least amount of storage capacity involves only storing the starting point (input dataset, initial training data, model settings, seed values, and inclusion/exclusion criteria). Someone reproducing the results can import the data and select the same model, and by screening the records in the order they are suggested and applying the inclusion criteria, they should obtain the same set of labeled records and the same trained active learning models. This will cost a large amount of time, though; the annotator needs to read all the records, and the active learning system needs to train all the iterations of the model again.  
Moreover, some accuracy will be lost. Humans make mistakes when reading and labeling records [7], and making one different labeling decision will result in a different set of records shown to the annotator. Also, many machine learning algorithms use (pseudo)random number generators to simulate probabilistic behavior. To make these algorithms deterministic, the random seed needs to be stored at the start. However, many algorithms designed for a graphics processing unit (GPU) are not deterministic due to the many parallelized computations that occur simultaneously. In recent years, support for deterministic training of machine learning models on GPUs has increased [40; 41]. In the end, the results (i.e., the set of seen and included records) might be replicated, but the entire process is not reproduced precisely because of both human errors and probabilistic models.  
At the other extreme of the reproducibility spectrum, one can try to store absolutely everything. This means storing every iteration of the model, with all its model parameters and the relevance scores that it produced for every record. In most cases, this becomes infeasible and, in fact, undesirable; if the data size becomes too big, it will be more difficult for other people to use the data and investigate its contents. For example, storing the relevance scores of a thousand trained models on a dataset of a thousand records takes approximately 8 MB, and the scores of ten thousand models on ten thousand records take approximately 800 MB. With every increase in the size of the dataset by a factor of 10, the data storage size increases by a factor of 100. When the dataset has in the order of a million records, the data size becomes in the order of terabytes, and this is just for storing the relevance scores; imagine how much storage capacity is needed to store the entire output of the model for each iteration.