# Cyber Resilience Limitations in Space Systems Design Process: Insights from Space Designers
## 3 Methods
### Data Analytics  
We followed a step-by-step process for creating and refining the codebook, as [PERSON] et al. [82] outlined. Our approach involved the following stages:  
1. Data immersion and familiarisation;
2. Initial code generation;
3. Theme identification; \begin{table}
\begin{tabular}{c l l} \hline \hline
**No** & **Theme** & **Question** \\ \hline Q6 & Cybersecurity and Resilience Awareness & Current level of cybersecurity and resilience in ground stations used for operations \\ Q7 & Challenges and Barriers & Main challenges to integrating cyber \\ Q8 & Challenges and Barriers & resilience into system design \\ Q9 & Challenges and Barriers & Significant challenges faced when ensuring \\  & & the cyber resilience of complex systems \\  & & Potential challenges or barriers that might \\ Q9 & Challenges and Barriers & arise during the implementation of the \\  & & cyber resilience engineering framework \\ Q10 & Standards, Regulations, and & Impact of international and domestic \\  & Architectural Aspects & standards and regulations on cybersecurity \\ Q11 & Standards, Regulations, and & Crucial system-level architectural aspects \\  & & for achieving cyber resilience in systems \\  & & engineering \\ Q12 & Framework Development and & Specific requirements or features expected \\  & & in a new engineering framework for \\  & & incorporating cyber resilience \\ Q13 & Framework Development and & Integration of the cyber resilience \\  & & framework into engineering processes and \\  & & its flexibility and adaptability \\ Q14 & Framework Development and & Metrics or indicators for assessing the \\  & & effectiveness of the framework in \\  & & enhancing cyber resilience \\ Q15 & Framework Development and & Effective training formats and resources for \\  & & promoting the incorporation of the \\  & & framework into projects \\ \hline \hline \end{tabular}
\end{table} 
Table 2: _Cont._  
Figure 2: Flow chart of the research procedure. * _Theme review and refinement;_
* _Theme definition and nomenclature;_
* _Report compilation and presentation._  
This iterative process was a collaborative effort. We compared our codes with those generated by AtlasTI_ _[_83; 84_]_ _to validate the emerging themes. Additionally, we used inter-rater reliability metrics to ensure consistency and accuracy_ _[_85_]__. Finally, all codes were human verified (Table_ 3_). Throughout the analysis, we maintained a reflexive approach, continuously acknowledging and addressing any biases and preconceptions that could influence the results. Furthermore, we used Python's Natural Language Processing (NLP) library spaCy, employing its pre-trained model \"en core web\" to validate our qualitative coding through systematic and reproducible methods_ _[_86_]__. spaCy is a powerful and efficient natural language processing library in Python that researchers have used for qualitative coding and validation tasks due to its wide range of features such as tokenisation, part-of-speech tagging, dependency parsing, and named entity recognition_ _[_87_]__. Data analysed by a recent study compared human coding to NLP algorithms and found that using a merit-based fusion of BERT, XLM-RoBERTa, and custom LSTM models achieved an F1-score of 0.852, outperforming individual models and existing methods_ _[_88_]__. Another research study argued that partnering an NLP model with a qualitative researcher (NLP+) can significantly reduce time and costs while yielding results aligned with traditional qualitative methods_ _[_89_]__._  
Our data collection, coding, and analysis approach was thorough and recursive. We used humans and machines to streamline and validate our approach. We identified the point of saturation, which is defined as the stage in the research process where no novel properties or dimensions emerged from the coding procedure_ _[_82_]__. This cyclical methodology allowed for a thorough data exploration, ensuring that all relevant themes and concepts were captured and examined until no further insights could be generated.