# Reproducibility and Data Storage for Active Learning-Aided Systematic Reviews
## 6 Reproducibility and Data Storage for Active Learning-Aided Systematic Screening--The Checklist
### Example of the Technical Data  
To provide an example of how the data produced by the human and by the model can be stored (item 9 in combination with items 12-14 of the checklist), we provide an illustration based on the output of the open-source software ASReview v1.0. For each iteration, ASReview stores the number of records in the training set and the type of model used. We assume one screener is active, only one model is trained at a time starting with the default settings of ASReview (TF-IDF + Naive Bayes), and the user wants to switch to a different model (sBert + Neural Network) after screening ten records. The screener starts with two records as prior knowledge; see the value '\(-1\)' for the training set size in the first two rows in Table 3. Note that any size of prior knowledge can be used. If you used a model that does not use prior knowledge, this would be indicated by a '0' in the table. After importing the data and selecting the prior knowledge, you have to wait for the text to be transformed into a format the model can understand using a feature extraction method. Using the feature matrix, a first classifier is trained (denoted by M1), which produces relevance scores. The most likely relevant record is shown to the annotator (first in the queue of M1), who assigns the label '1'. This label is based on a model with a training set size of 2. In the implementation of ASReview, at most one model is being trained at any time. After the annotator makes a labeling decision, the program checks if there is no model training in which case a new model is triggered. In our example, this is done after the label is provided by the screener in row number 3 and the training of the model M2 is started using the labels of the first 3 rows. Instead of having to wait for M2 to finish training, the annotator already sees the fourth record, which was ranked second highest in M1. Assuming the annotator needs some time to read the fourth record, M3 has finished training. Thus, the fifth record presented is based on M2, etcetera. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline  & \multicolumn{4}{c}{**Labeling Data**} & \multicolumn{4}{c}{**Training Data**} \\ \hline \hline \multicolumn{2}{c}{**Row**} & \multicolumn{1}{c}{**Record**} & \multirow{2}{*}{**Label**} & \multirow{2}{*}{**Labeling Time**} & \multicolumn{1}{c}{**Training**} & \multicolumn{1}{c}{**Record from**} & \multicolumn{1}{c}{**Model**} \\ \cline{3-8} \multicolumn{2}{c}{**Number**} & & & **Identifier** & & **Set Size** & **Queue** & **Model** \\ \hline \hline \multirow{8}{*}{2} & 1 & 145 & 0 & 2022-12-23 09:32:08.46 & \(-1\) & - & - \\ \cline{2-8}  & 2 & 56 & 1 & 2022-12-23 09:32:40.13 & \(-1\) & - & - \\ \cline{2-8}  & \multicolumn{4}{c}{Composing feature matrix (using TF-IDF) + training of 1 st iteration of 1 st model (NB) with two priors} \\ \cline{2-8}  & 3 & 120 & 1 & 2022-12-23 09:13:26.48 & 2 & 1 st of M1 & TF-IDF + NB \\ \cline{2-8}  & 4 & 442 & 0 & 2022-12-23 09:14:26.55 & 2 & 2 nd of M1 & TF-IDF + NB \\ \cline{2-8}  & 5 & 247 & 1 & 2022-12-23 09:15:27.11 & 3 & 1 st of M2 & TF-IDF + NB \\ \cline{2-8}  & 6 & 491 & 1 & 2022-12-23 09:16:28.43 & 4 & 1 st of M3 & TF-IDF + NB \\ \cline{2-8}  & 7 & 102 & 0 & 2022-12-23 09:16:28.59 & 5 & 1 st of M4 & TF-IDF + NB \\ \cline{2-8}  & 8 & 243 & 0 & 2022-12-23 09:17:30.12 & 5 & 2 nd of M4 & TF-IDF + NB \\ \cline{2-8}  & 9 & 401 & 0 & 2022-12-23 09:18:40.22 & 6 & 1 st of M5 & TF-IDF + NB \\ \cline{2-8}  & 10 & 279 & 0 & 2022-12-23 09:19:59.36 & 8 & 1 st of M6 & TF-IDF + NB \\ \cline{2-8}  & \multicolumn{4}{c}{Model Switch} \\ \cline{2-8}  & \multicolumn{4}{c}{Training of 1 st iteration 1 st model with sBert + NN (M7) with ten priors} \\ \cline{2-8}  & 11 & 4273 & 1 & 2022-12-27 09:00:47.83 & 10 & 1 st of M7 & sBert + NN \\ \cline{2-8}  & 12 & 366 & 0 & 2022-12-27 09:01:13.26 & 10 & 2 nd of M7 & sBert + NN \\ \cline{2-8}  &... &... &... &... &... & \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Slice of the model data.