# Reproducibility and Data Storage for Active Learning-Aided Systematic Reviews
## 2 Reproducibility in the Context of Systematic Screening  
Reproducible research is not an easy concept to define. In very broad terms, the meaning is to describe the data, methods, and results of the research in such a way that others can start with the same data and use the same methods to arrive at the same results. What'same', 'data','methods', and'results' mean depends very much on the context of the research and the goal of reproducibility. The confusion around the word'reproducibility' is confounded by the fact that there is also the word'replicability', and the meaning of these two words can be completely opposite depending on the context. See [PERSON] and [PERSON] [34] for a review and [PERSON] [35] for an overview of the usage of the term reproducibility.  
A systematic review is a process with many steps, and in all of those steps one can debate about what is necessary to store for it to be reproducible. In every step, the answer will depend on the goal of reproducibility. In the case of a systematic screening, the main goal is to find all texts relevant to a research question. Therefore, the main goal of reproduction should be to understand and verify each step that led to this list of texts. This builds trust in the process as a whole. In the case of AI-aided screening, there is the extra goal of understanding and verifying the AI-method used and the specific implementation used. Then there are secondary goals: allowing easy reuse of the data for further analyses of the results, producing a trained AI model, and improving the AI method. In the case of classical active learning, we would have a different primary goal, namely, to produce a model that can accurately label texts. Of course, this means that the model training steps of the process become more important for reproducibility. Thus, we see that the goal influences what data we want to store.  
Another important consideration is the ease of reproducibility. Some things are very easy to store and reproduce, such as a list of labels 'included' or 'excluded' for each record in the screening phase. Other information is more difficult to store, such as the state of an AI model after training it on certain records. We could store all the information on the model, but this might become very large, especially with large, modern machine learning models. On the other hand, we can store only the information needed to train the model again, but then it will take more time to reproduce the model; or, it will even be impossible to reproduce the model for someone that does not have access to sufficiently powerful hardware to run the model. In such a case, it would be reproducible for certain people, but irreproducible for others.  
We take a detailed look at all the steps in the screening phase of a systematic review. For each of these steps, we look at the data that goes in and comes out. With this information at hand, and with our main goal of reproducibility in mind, we give recommendations on what data should be stored, and how it should be stored, in order to strike a good balance between ease of reproduction and data storage.  
Note that we focus on those ingredients relevant to the active learning aspect of the pipeline, thereby ignoring other important issues like search strategies, deduplication, abstract versus full-text screening, and inter-rater reliability. Such topics are already clearly described in textbooks (e.g., [32]) and the PRISMA guidelines. We also do not consider using artificial intelligence tools in other steps than in the screening phase.