## 4 Case Study
### Results
#### 4.5.2 Observations from Data Cleaning  
Cleaning the data after data extraction can be a time-consuming process. We applied a few strategies to refine our IVD prompts to reduce these time constraints. Our data-cleaning process incorporated the following steps:  
* Standardizing formats: This involved applying uniform names to terms that may have several variations. For example, we standardized \"activPAL\" from \"activPAL3\", \"ActivPAL3\", \"activPAL3c\", \"activPAL3\", and \"activPAL 3 micro\" to reduce the sensitivity of the search. * Handling missing values: Certain headings, such as \"_devicemodel_\" had a high prevalence of missing data. The LLM could not differentiate between a device's name and its model. Extracting such missing values involved identifying articles that met the selection criteria, particularly those with missing values in the identifiers, and searching for the specific missing information. * Error removal: This involved identifying and rectifying errors in the output. Specific fields that required attention included the subjective measurements and the device model. These fields were reviewed for any inaccuracies or inconsistencies and corrected with information from the original article. Although LLM output accuracy to assist with SR heavily relies on specific prompt engineering and may vary across discipline areas, we obtained a high efficiency for identifying the inclusion (\(n=170\)) from full-text review articles (\(n=194\)) simply by filtering and screening keywords and concise sentences from the LLM output. Figure 8 illustrates the efficiency of prompt output for the inclusion sector (a) and exclusion sector (b) once the prompt development and LLM executions finished. We calculated completeness as the number of rows, i.e., the article for which a value was found by the LLM in the full-text article for the specific prompt/CSV headers, compared to the total number of rows. So, for a prompt \(p\):  
\[completeness\ (p)=\frac{Number\ of\ articles\ with\ a\ valid\ extraction\ for\ p}{Total\ number\ of\ articles}\times 100 \tag{1}\]  
Figure 8: _Cont._Identifiers have a high rate of completeness, meaning the process of identifier filtering works with high accuracy, thus reducing the human workload during data cleaning. It is not surprising that all verifiers used in this case study (see Figure 3) had perfect completeness (100%) when asking for article summaries. The data fields (some of which also acted as identifiers) are relatively less filled in the CSV but still have over 50% completeness for most fields of the _included_ articles. On the other hand, the _identifier_ and _data field_ completeness are considerably lower for the same prompts for the articles that were ultimately excluded. This shows that the LLM largely succeeds in extracting the desired data from the full-text articles, largely reducing the human's responsibility to screen the full-text articles. Prompt development and testing are an iterative process during which humans do not have a clear idea of which article is included or not. However, as we started with a set of 50 articles that strongly met the selection criteria, we expected that at the end of the prompt development and testing, the prompts would have a very high level of completeness. To provide an additional estimate of the time-saving capacity of our hybrid workflow, we measured the character length of the identifiers and verifiers (Table 4), comparing the speed at which human readers may be expected to perform the filtering steps using the LLM output compared to performing a full-text review to determine article inclusion or exclusion for an SR. Identifiers had a range of 10-25 characters, which were much smaller than the verifiers (150-300 characters). Both of these could be considered negligible compared with reading a full-text article with thousands of words.