# Computer-Assisted Relevance Assessment: A Case Study of Updating Systematic Medical Reviews
## 3 Materials and Methods
### Experiment Design
#### 3.3.1 Data Collection  
As mentioned, the main focus of this study was to measure how human performance changes according to the length of evidence shown. To carry such an experiment, we had to collect a set of documents with actual relevance pre-judged by domain experts. In both tasks of the experiment, users viewed a list of clinical studies associated with a systematic review extracted from the Cochrane Database of Systematic Reviews (CSDR) ([[https://www.cochranelibrary.com/cdsr/about-cdsr](https://www.cochranelibrary.com/cdsr/about-cdsr)]([https://www.cochranelibrary.com/cdsr/about-cdsr](https://www.cochranelibrary.com/cdsr/about-cdsr))). The CSDR is a leading database for systematic reviews helping medical experts in the decision-making through evidence-based information. A Cochrane review identifies, appraises, and synthesizes evidence related to a specific research question given pre-specified eligibility criteria. Generally, to collect studies for systematic reviews, experts construct complex boolean queries that include logical operators such as AND, OR, and NOT to maximize the recall. The results from the query search pass through two elimination phases, abstract-screening and full-text screening. The abstract-screening involves a review of article titles and abstracts to give initial feedback on the relevancy of the study. This is followed by a full-text screening that consists of a thorough analysis to decide whether the study meets the inclusion criteria or not. Each systematic review included in the Cochrane database has its bibliography divided into three: _Included_, _Excluded_, _and Additional_ sections, with references to other articles. Studies in the _Included_ section are the ones found relevant to the systematic review based on the full content of the study, in other words, after passing through both elimination stages, whereas the _Excluded_ section includes studies that were initially considered relevant in the abstract screening stage but later dismissed in the full-text screening stage. On the other hand, the _Additional_ section includes references that are neutral and considered irrelevant to the study. To fully assess the human judgment, the list of documents to be reviewed by the participants needs to be uniform, i.e., to maintain a good balance between relevant and irrelevant documents included in the list. Another constraint to account for while choosing the candidate documents is the difficulty level in assessing the documents; it should be as tricky as it is for experts when updating a systematic review in reality. With these goals in mind, we selected six published intervention systematic reviews, from the heart and circulation category, that are available in the CSDR. The chosen reviews are assigned the _New Search_ tag, which indicates that the authors of the review published an updated version. We limited the scope of the experiment to the abstract screening stage, and hence references in the _Included_ and _Excluded_ sections are both considered relevant studies at the abstract level. Similarly, all references to studies in the _Additional_ section are deemed irrelevant. We note that we excluded any reference to studies not related to the biomedical field, such as studies with general guidelines on how to conduct systematic reviews or interpret bias. In some cases, we re-submitted the boolean query, if provided, to the corresponding medical database and the retrieved studies were also considered irrelevant even if not included in the _Additional_ section. Our list of documents remains, however, random and un-ordered. Assessing and understanding human judgments and responses to ranked lists of documents is more complicated, and we leave it for future work.