# Data-Based Stakeholder Identification in Technical Change Management
## 5 Methodology for Data-Based SI
### Training of Machine Learning Models  
After categorizing the CRs, generating training word embeddings, and analyzing the additional data, text-classification models are trained and then evaluated in order to build a target-oriented model. Within the training, an iterative procedure is used. At first, a model is trained, which only considers textual information as an input. The textual information provides the basis for the classification and is therefore integrated into all trained models because textual descriptions are the most precise way to understand the details of the actual change. Based on this basic model, several models are then trained with different combinations of additional features to identify the most successful classification models. An analysis of the additional data individually, without the text data is not performed because the additional data consists only of single sentences without context. For each model training, the data used must be preprocessed and converted into numerical data. For this purpose, decimal numbers, punctuation, and special characters are removed, abbreviations are replaced with spelled-out words, umlauts are converted, all words arelowercased, numbers between 0 and 9 are converted into words, and finally tokenization is performed. These steps, as shown in [PERSON] and [PERSON] (1998), [PERSON] (1998), or [PERSON] (1998), are common steps in preparing text for data analysis. Afterward, each tokenized word is converted to its word vector, which serves as input variables into the HAN. After training, the models can be evaluated and compared by using statistical metrics. By performing this training cycle for different configurations of the data attributes, different models with varying classification quality are generated. The objective is to select the most suitable model for each use case by comparing the metrics of each model. For the training and evaluation of the models, the database data is split into a training dataset, a validation dataset, and a test dataset. The same data split is used for each model training so that the trained models can be compared. The training dataset consists of 70% of the data. The validation dataset and the test data set each consist of 15%. The splitting of the data is done in order to train the text-classification model on the largest possible database and then to objectively evaluate the quality of the model by using unseen data. A trained text-classification model is to decide for new CRs whether they are \"relevant to business unit\" or \"not relevant to business unit\". Within the training cycle, the test dataset is used to test whether the trained model delivers target-oriented results. Subsequently, the results are compared with the categories derived from Section 5.1. From this, the number of true positive (TP), true negative (TN), false positive (FP), and false negative (FN) results can be derived and used to evaluate the model quality based on statistical metrics. In general, the choice of the appropriate evaluation metrics for machine learning models strongly depends on the considered use case. The relevance of incorrect and correct classification varies significantly. In the case of change management, the FN classifications, in particular, are crucial, as they address cases wherein relevant stakeholders are missed. This means that the expertise and assessment of the employees cannot be utilized in the planning and implementation of the change, which increases the risk of unplanned change impacts. FP cases, in contrast, are considered significantly less critical, despite the waste of resources due to unnecessary efforts, as in these cases, there is often only a brief review by the contacted stakeholder, and the non-relevance is quickly recognizable. In addition, it became clear in discussions with industry experts that overall a tendency of the model to classify as \"relevant\" is to be preferred. In this way, the experts are contacted in uncertain cases and can thus perform a review of the change.