# Large Language Models' Trustworthiness in the Light of the EU AI Act--A Systematic Mapping Study
## 6 Threats to the Validity of the Results  
When performing a systematic mapping study, various issues can arise that may threaten the validity of the research. This section outlines the approach for addressing these validity threats, based on the guidelines provided in [73].  
The construct validity in systematic mapping studies ensures that the extracted data accurately reflects the research questions. For this particular study, we followed the guidelines outlined in [20,33], confidently selecting primary studies that represent the research questions. The search string was carefully crafted using the PICO method. The most widely used digital libraries in the fields of engineering and technology were selected for automated search, enhancing the depth of the study on various dimensions of trustworthiness in LLMs, as discussed in Section 3.1.3. The publications identified through the search were reviewed based on their titles and abstracts. If a publication's relevance could not be determined from these, full-text screening was conducted to assess its suitability for the research area. A random selection of 260 studies from a pool of 7615 was reviewed by the selection team to assess overall agreement, achieving a high inter-rater agreement of 0.93, which indicates near-perfect consistency. Also, the classification schema for mapping was refined through expert discussions and iterative meetings, with systematic keywording ensuring coherence.  
Nevertheless, certain limitations exist:  
* **Potential Missing Studies**: This study underwent a stringent screening process with well-defined inclusion and exclusion criteria. Although the decision to not use \"snowball sampling\" may limit this study's credibility, the large number of primary studies (7615) ensures comprehensive coverage of the intended scope. Moreover, following T-A-K skimming Section 3.2.2, some relevant studies could have been excluded due to incomplete or misleading abstracts. Despite these factors, it is believed that, even if a few studies were missed, they are unlikely to significantly affect the overall results of our analysis.
* **Limited Critical Depth**: Although the use of the T-A-K method in this study effectively facilitated the identification of research trends, developments, underexplored areas, and methodological patterns in the research on LLM trustworthiness, its limitations  
Figure 21: Distribution of publications based on academic and industrial affiliation in LLM trustworthiness studies.  
hinder a more critical exploration of contributing factors that lead to interdisciplinary fragmentation. In particular, how certain trustworthiness dimensions intersect with domain-specific challenges or how they vary across established and emerging models.
* **Single Regulatory Framework**: The classification scheme for the trustworthiness aspects is based solely on the AI Act. We believe that this approach ensures proper alignment and contextual understanding, particularly when differing definitions or interpretations may arise from other AI-related legislative frameworks. Also, the classification was conducted independently by different authors, and using the AI Act as the primary reference point allowed us to achieve consistency across our analysis.
* **Language and Publication Type Bias**: To enhance validity, only peer-reviewed, English-language publications were considered, potentially excluding relevant non-English or non-peer-reviewed literature.
* **Exclusion of Industry Patents**: We consider that certain application domain works, particularly those originating from the industry, might be more appropriately presented in patents rather than academic publications. Since patents were not included within the scope of our study, this exclusion could potentially influence our results.