## 4 Evaluation
### Success Criterion SC-14 Sufficient Accuracy  
On the one hand, performance parameters of dependency analysis in an industrial context are used as the basis for evaluation. On the other hand, a survey with pilot users (questionnaire and interviews) is conducted. The performance characteristics of the dependency analysis allow a conclusion on the accuracy of automatically generated information. Whether the follow-up steps, based on this information, allow a sufficiently accurate assessment of the risk of change in the primarily supported decision situations is determined by questions 23 to 27 (cf. Appendix C). The queried decision situations are derived from the use cases (cf. Section 2). The evaluation is based on expert estimates, since a data-based accuracy evaluation is not possible. It would require documentation of the history of uncertainties and requirement changes (including initiator and impact) as well as selected actions over a longer period of time. This is judged to be inappropriate and expert judgment is used instead as an indicator for accuracy. SC-14 is considered fulfilled if the average of users' ratings for each question is \(\geq\) 0 (neutral). In this case, it is assumed that the method is sufficiently accurate under usual conditions. Evaluation result: Performance parameters of dependency analysis are high for the articulated-arm robot case study (cf. Table 4). None as a major class has very high accuracy, but also minor class Dependent reaches high accuracy. Due to the limited extent of training data, dependency types are not distinguished by type in order to reduce the number of classes to be differentiated. Since the dataset contains a major class (None) with many entries and a minor class with few entries (Dependent), the criteria needed to be viewed macro averaged (equal weight on all classes) to be significant [75]. The application for industrial case study on engine control unit revealed the importance of a context-specific dependency model. Using a context-independent dependency model, accuracy was poor for minor classes, but increases significantly with the ratio of context-specific data for training the dependency model. Due to the high effort to create such context-specific data (cf. SC-12), pilot users suggested the gathering of required dependency data in the course of future development projects. The suggested approach was to adapted in documentation guidelines and commits developers to capture dependency data. This not just helps to create a context-specific database, but also ensures the fulfillment of common development standards (cf. SC-10). To further investigate the influence of context-specific training data on accuracy, data from four student development projects on power tools (cf. Section 2) were used. The accuracy increased from F1 = 92% (None) and F1 = 47% (Dependent macro avg.) without \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Dependencies** & **Number** & **Precision** & **Recall** & **F1** \\ \hline None & \(-\)081 & 99.12\% & 99.06\% & **99.09\%** \\ \hline Dependent & 92 & 69.15\% & 70.65\% & 69.89\% \\ \hline Macro avg. & - & 84.14\% & 84.86\% & **84.49\%** \\ \hline Weighted avg. & - & 98.25\% & 98.24\% & 98.24\% \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Performance of dependency analysis for the articulated-arm robot case study [75]. context-specific data to F1 = 98% (None) and F1 = 96% (Dependent macro avg.) with context-specific data as part of the training set. In consequence, the importance of context-specific data to train the dependency model is highlighted. However, the ability to reach high accuracy of dependency analysis for different contexts is also shown [75]. As a second step, pilot users' feedback was analyzed. Feedback indicates sufficient accuracy for all decision situations (cf. Appendix C). Most questions have an average rating close to 1 (agree). Improvement potential is given for the selection of proactive measures (average rating = 0.3).