# Reproducibility and Data Storage for Active Learning-Aided Systematic Reviews
## 2 Reproducibility in the Context of Systematic Screening
### Defining Three Phases  
We can identify three main phases in a systematic review; see Figure 2:  
1. the pre-processing phase;
2. the screening phase;
3. the post-processing phase.  
The pre-processing phase results in an input dataset containing the information for the screening phase. To create such a dataset, the researcher formulates a research question and determines the inclusion and exclusion criteria for selecting relevant records. Then, they create a dataset that contains all records that might be relevant to the question. Typically this is conducted using database searches in multiple databases, after which the results are combined and deduplicated. The input dataset is a set of records containing meta-data of, for example, scientific papers (i.e., titles, abstracts, and persistent identifiers like the DOI).  
In the screening phase, it is determined which of the records in the input dataset are relevant based on a predefined set of inclusion and exclusion criteria. The screening phase is an iterative and synchronous process, except for the training of the active learning model, which can happen asynchronously; see Figure 3. In each iteration, a model is trained on the labeled records, starting with the initially selected prior knowledge. The trained model then predicts relevance scores for the unseen records. It ranks these records from high to low. At the same time, the annotator reads the records in the current ranking order and assigns each record a label. The new label(s) are then used to train a new model. After each labeling decision, the size of the training data increases.  
Figure 3: The two cycles of labeling and training.  
Figure 2: Main steps in a systematic review using active learning.  
Note that both these processes (training a model and labeling records) are continuous and can be parallel to each other; at any moment, a new model can be trained using the currently labeled records, and at any moment, the annotator can take a new record from the pool of ranked, unlabeled records and assign it a label. The annotator and the model do not need to wait for each other. Of course, it is an option to implement the labeling and model training in a serial fashion, one after the other, but this is a priori not necessary and is inefficient. The user always wants to be able to request a new record instead of having to wait for the model to finish with training. Which record is selected depends on the chosen query strategy. It can be the record with the highest relevance score, called certainty-based sampling [26], which is most often used for HITL active learning.  
This asynchronicity is a departure from the classical active learning setting. In that setting, the model can request which records the annotator should label. Labeling extra records from an old iteration of the model could mean wasted time by the annotator since the new iteration of the model might request different records from which it can learn more. Therefore, making the model training and the labeling phases asynchronous can mean that resources are wasted. In the RITL setting, this is not a problem since the set of labeled records is the main output anyway, and thus an asynchronous set-up means there is less downtime for the annotator.  
Once the screener decides to stop labeling, the relevant records can be exported for post-processing (e.g., data extraction). We define the output of this phase as the set of relevant records. The included records from the output dataset are used in the post-processing phase. In the post-processing phase, the data of the selected relevant records are extracted and analyzed in order to answer the research question. Thus, the output of this phase is the conclusion of the study.