# Data-Based Stakeholder Identification in Technical Change Management
## 5 Methodology for Data-Based SI
### Integration of Expert Knowledge  
After the most suitable classification model has been identified, the model quality is to be further improved by integrating expert knowledge. For this purpose, workshops are held with relevant employees on the basis of the classification results obtained. In these workshops, the FN classification cases are analyzed by the employees who were actually affected. For each workshop, lists of FN results are prepared (e.g., in an Excel file) and then reviewed with an employee who would be affected by the changes. Workshops are conducted with individual employees and the duration depends on how many FN cases are to be analyzed, although the objective is to have as many as possible. Based on the use case described in Section 5 in this contribution, it can be assumed that a duration of one minute per FN case should be calculated. The information on which employees are relevant is obtained from past change data, as described in Section 5.1. This traceability is essential as otherwise employees give their assessment without being aware of the background and details of the change. Changes that cannot be clearly assigned, e.g., due to reorganizations or changes in the workforce, are not considered in this step. In these workshops, the employees use the textual information of the change under consideration and indicate, based on their expert knowledge, which words or sentence components would have implied the relevance of their business unit. In this way, a vocabulary is derived, which includes words and word combinations that indicate the relevance of the business unit under consideration and therefore offers the potential to support the classification model. The identified word combinations from all experts are then sorted into categories summarizing terms with similar or synonymous meanings. For example, within a workshop, the word combinations \"adjustment steering wheel\", \"adding screw\", and \"modification side member\" are identified as relevant word combinations. The category \"Modification\" can then be formed from the terms \"addition\", \"modification\", and \"adaptation\". In contrast, the terms \"steering wheel\", \"screw\" and \"side member\" are clustered as \"Product Parts\". The terms inside these categories are then automatically combined to obtain all possible word combinations. With the two exemplary lists \"Modification\" and \"Components\" a new list of word combinations is created, in which \"adjustment steering wheel\", \"adjustment screw\" or \"adjustment side member\", and \"add steering wheel\" are contained. However, when creating the combinations, the words within a category are not combined because it is assumed that these combinations (e.g.,\"adjustment add\") do not offer any added value for the classification due to the similarity of their meanings. The formation of this multitude of word combinations and the subsequent evaluation of their added value enables the identification and inclusion of actually relevant word combinations in the classification model. These workshop-specific lists are then merged per derived category of terms of similar meaning, and duplicates are deleted. For each individual CR, it is then checked which word combinations it contains. The word combinations do not have to occur directly next to each other in the text but may also be distributed throughout the text. The word combinations that do not appear in any CRs are then removed. The remaining word combinations are then combined with the results of the previously trained text-classification model. For each word combination, the effect on the overall model quality of relabeling the text classification model from \"not relevant\" to \"relevant\" based on their occurrence is examined. By focusing on the cases recognized as \"not relevant\" (FN or TN), either a change from FN to TP or from TN to FP can be achieved. It is checked for each word combination whether a reduction in FN is achieved without at the same time disproportionately increasing the number of FP.