# Reproducibility and Data Storage for Active Learning-Aided Systematic Reviews
## 8 Conclusions  
Text reading, processing, and screening require enormous amounts of time in almost any professional organization. With the emergence of online publishing, the number of scientific papers on any topic is skyrocketing. The COVID-19 crisis has illustrated how crucial and critical it can be to develop fast but rigorous systematic overviews of the literature. At the same time, accurate information retrieval is essential to provide access to facts. Orchestrated campaigns are spreading untruths, disinformation, mal-information, and misinformation, which are often unwittingly shared on social media. This raises questions about the quality, impact, and credibility of researchers, the judiciary, physicians, journalists, media networks, governments, and all other professionals, institutions, and networks acting as agents of knowledge. Therefore, reproducible and verifiable methods are needed to identify, select, and critically appraise all relevant decisions in the process of selecting textual data. Transparency will help increase quality and accountability to gain public trust. Addressing the issue of reproducibility is a complex task, especially in the context of AI-aided systematic screening. The primary aim of our checklist is to address the gaps present in the PRISMA checklist when using AI-aided screening tools. It specifies the essential information that should be provided to ensure reproducible results. The checklist guarantees that all data inputs and outputs within the AI-aided screening process are systematically recorded and made accessible. This allows the same data to be used by others to replicate the study outcomes. We provide clear documentation of the algorithms, model parameters, and computational processes used, ensuring that the methods can be precisely replicated by other researchers. The checklist advocates for the transparency of both interim and final screening results, including the classification of data by AI tools. This ensures that results can be consistently replicated across studies using the same methodologies and data. By ensuring the reproducibility of data, methods, and results, our checklist indirectly supports the reproducibility of conclusions, provided that the analysis is consistently applied. However, we need to strike a balance between data storage and reproducibility. On the one hand, storing the starting point is not enough to call the review reproducible. On the other hand, storing everything is overkill because not all of the enormous amount of information produced by the model is relevant. Where on this spectrum an AL-aided review pipeline should be, depends on the specific application. There is a scale on which we can measure the software used for AL-aided screening and the needs of the users of this software. On the one side of this scale is the software that only provides the main output. The steps in between remain a black box because nothing gets reported on them. Users of this software should only be interested in the final results and not in reporting on the process of how these results were obtained. Software on the other side of the scale stores absolutely everything about the whole screening process. As we discussed, this can lead to considerable data storage sizes. For users on this side of the scale, it should be imperative to account for all the steps in the screening process, or the storage size is just not an essential factor for these users. For classical reviews, the PRISMA guidelines go far toward storing everything. However, during pre-processing, the guidelines require saving the search string but not the actual screening input dataset. For AL-aided reviews, they shift away from storing everything since they only ask to report on which software was used and how it was used. They do not ask to store any information on the models that were used. As we noted, this means that we cannot reconstruct how the decisions of the model were made. Users should think about where on the scale they want to be.