## 4 Evaluation
### Support Evaluation
#### 4.1.1 Success Criteria SC-4: Analysis of Change Propagation  
Internal Functional Testing  
For the internal functional test of dependency analysis, a dataset was built (case study articulated robot arm). The dataset consisted of 145 requirements and a manually created dependency matrix with 21.025 entries. The dependency type was differentiated in the dependency matrix as an indicator for propagation behavior. For testing, the dataset was divided into a test set and training set: 80% training data and 20% test data. Since no fully objective data on dependencies exist, the systematic assessment of experts involved in system development is assumed to be ground truth data quality. Therefore, the dataset was created by the developer of the articulated robot arm. Next, the four solution classes identified in the literature review were implemented as an algorithm. These are refines, is refined by, requires and is required by. Since not all approaches are able to differentiate the dependency types, individual statistical classifiers were used to evaluate the quality of the results. Comparing the solution classes, the dataset was analyzed with all algorithms and the performance was compared. For this purpose, the metrics precision, recall, F1 and ROC_AUC were determined (cf. Table 2). The evaluation shows that the BERT approach produces the best results. Therefore, this approach was favored. Due to the high computational effort (for instance, compared to LSTM), the decision was made in consultation with the industry representatives [W10]. Figure 15: Articulated robot arm with case study data. Validation Workshops with Industry Users [W8, 10 and 11]  
Due to the complexity of the dependency analysis method, the validation was performed step by step in three workshops. First, the basic industrial suitability of AI-supported language processing of requirements descriptions with respect to application effort and availability of information was verified [W8]. Irrespective of the specific solution approach, the requirement of partly extensive training data for training the machine learning models as well as the lack of traceability were discussed. Both were considered acceptable. After comparing the solution classes with respect to quality of results (cf. internal function test), BERT was selected as the most promising approach in a second workshop [W10]. As an alternative, the LSTM approach was explained, which achieves a lower performance, but causes significantly less computational effort. Here, the performance was rated as more important than the speed of the calculation by the industrial users. The quality of the results was rated as fulfilled, since both the dependencies are recognized and their type is differentiated. A result of the workshop was also that a differentiation between types with coercive propagation is not necessary if a faster calculation of the dependencies is made possible by the simplification. In a final workshop, the learning mechanism of the dependency analysis was validated [W11]. The industrial users rated the application effort as well as the availability of the information as fulfilled and the continuous improvement of the quality of the results as positive. Validation result: SC-4 is fulfilled. 1.2 Success Criteria SC-5: Analysis of Exogenous and Endogenous Change Likelihood Internal Function Test  
The method for assessing the likelihood of occurrence of a change was validated in three parts. First, the assessment of exogenous change likelihood was examined. Since the categories of change initiators were derived from existing literature approaches, only an internal functional test of the software functionalities (assignment of initiators, selection of reference projects) was performed by comparison with manual calculation. The evaluation of endogenous likelihood of change is based on the PageRank algorithm, so that only the context-specific adjustments were checked for contextual correctness.