## 4 Evaluation
### Application Evaluation
#### 4.4.2 Application Success Criteria  
Success Criterion SC-8: Easy to Use  
The results of the workshops for support evaluation (cf. Section 4.1) as well as the feedback from pilot users (questionnaire and interviews) are used as the basis for the evaluation. The evaluation of the method steps regarding ease of use was performed in two steps to reduce biases caused by the user interface of the software prototype [46]. For example, unavoidable weaknesses of a prototype such as missing secondary functionalities (for instance, role and rights management) can lead to user frustration and bias [46]. First, the individual method steps are evaluated and based on the experimental method Participative design [76]. For this, the respective step was demonstrated in the workshops  
Figure 16: Runtime of dependency analysis for different size requirement sets. of support evaluation (cf. Section 4.1) and asked whether it is easy to apply or whether usability adaptations are necessary. By demonstrating the method, the feedback could be actively focused on the method and the influence of the user interface and missing secondary functionalities were reduced. Verification of the ease of use of the user interface (in terms of usability according to [77]) was carried out in the second step. For this purpose, pilot users independently apply the ProMaRC method and evaluate it based on predefined criteria (cf. Appendix C). This approach was chosen because the pilot users were not involved in the development phase and therefore were unbiased. An established questionnaire was used for the evaluation: system usability scale [78]. Compared to other standardized questionnaires (for instance, user experience questionnaire [79]), this questionnaire has the advantage that an overall evaluation can be determined and interpreted on the basis of predefined threshold values. Formulating individual questions was declined, since standardized and validated questionnaires on usability of user interfaces are available and allow low-bias evaluation as well as sound interpretation of results. SC-8 is considered fulfilled for the method if ease of use is confirmed in the workshops and no adjustments are pending. The success criterion is considered fulfilled for the user interface if the SUS score of the user assessments is above 51 [0; 100]. For SUS score \(\geq\) 51 the user-friendliness is considered OK [80], which is seen as threshold for a software prototype to be adequate. If both the method and user interface are completely fulfilled, SC-8 is considered fulfilled. Evaluation result: If adaptations to the method application or software functionality were suggested from industry experts during the workshops (for instance, integration of a status display for dependency analysis), those were implemented without exceptions. The adaptations were presented and discussed again during the subsequent workshop. Since all adaptations were implemented and evaluated as satisfactory, the overall ease of use is considered to be fulfilled. Usability of the software prototype was rated with a SUS score of 69.2 (average value), which is above the predefined threshold and close to the next usability level (\"Good\") which requires a rating SUS \(\geq\) 71. As a result, ease of use is approved for the method and user interface. Validation result: SC-8 is fulfilled. Success Criterion SC-9: Availability of Required Software  
Availability is rated by means of the software costs. For this, licensing conditions of open-source software and product costs as well as market penetration are used in the case of commercial software. Market penetration is seen as an indicator for costs, since no procurement costs are expected for standard products already available in the company (e.g., Microsoft Office). SC-9 is considered fulfilled if there are no costs for using the method in an industrial context. Evaluation result: The following software is required for method application: Python and Microsoft Excel.