## 5 Analytical Comparison of MLP, SVM, and RF in Classifying Error in Simulated CPSs
### Experiment Method  
The provided code demonstrates an experiment comparing the performance of three machine-learning models (multi-layer perceptron (MLP), support vector machine (SVM), and random forest) on a classification task using a given dataset. The experiment compares models using the provided dataset, assesses their accuracy, and visualises the results. It also employs the PyCaret library for further model evaluation and selection. The experiment includes the following steps:  
1. Sample Data: 1.1 A random seed for reproducibility was set. 1.2 The number of samples as \"num_samples\" is defined. 1.3 Random data were generated using \"np.random.randn\" with dimensions \"num_samples\" by 2. 1.4 Random labels ranging from 0 to 2 using \"np.random.randint\" for \"num_samples\" times were generated. 2. Saving the Data to a CSV File: 2.1 A pandas DataFrame called \"df\" with columns named \"Error\", \"Warning\", and \"Label\" was created. 2.2 The DataFrame was saved to a CSV file and specified by \"csv_file\" using the \"to_csv\" function. 3. Loading the Data from the CSV File: 3.1 The CSV file was read into a pandas DataFrame called \"df_loaded\". Figure 12: Random forest accuracy chart. * 3.2 The \"Error\" and \"Warning\" columns were extracted from \"df_loaded\" and assigned to \"data_loaded\". * 3.3 The \"Label\" column was extracted from \"df_loaded\" and set to \"labels_loaded\". * 4 Multi-Layer Perceptron (MLP) Example: 4.1 The function \"mlp_self_healing\" was defined to train an MLPClassifier model on the \"data\" and \"labels\". 4.2 Prediction using the trained MLP model and calculating the accuracy was performed. 4.3 The \"real\" data were randomly generated, and predicted labels were plotted into a scatter chart. * 5 Support Vector Machine (SVM) Example: 5.1 The function \"svm_self_healing\" was defined to train an SVC model with a linear kernel on the \"data\" and \"labels\". 5.2 Prediction using the trained SVM model and calculating the accuracy was performed. 5.3 The \"real\" data were randomly generated, and predicted labels were plotted into a scatter chart. * 6 Random Forest Example: 6.1 The function \"random_forest_self_healing\" was defined to train a RandomForestClassifier model with 100 estimators on the \"data\" and \"labels\". 6.2 Perform prediction using the trained random forest model and calculate the accuracy was performed. 6.3 The \"real\" data were randomly generated, and predicted labels were plotted into a scatter chart. * 7 Calling the Self-Healing Approaches Using the Loaded Data: 7.1 The \"mlp_self_healing\" function with \"data_loaded\" and \"labels_loaded\" was invoked. 7.2 The \"svm_self_healing\" function with \"data_loaded\" and \"labels_loaded\" was invoked. 7.3 The \"random_forest_self_healing\" function with \"data_loaded\" and \"labels_loaded\" was invoked. * 8 Preparing the Data for PyCaret: 8.1 Pandas DataFrame called \"df_pycaret\" was created by concatenating \"data_loaded\" and \"labels_loaded\" along the columns. 8.2 The column names (\"Error\", \"Warning\", \"Label\") were set. * 9 Initialising PyCaret Classification Setup: 9.1 PyCaret \"setup\" function to initialise the classification task with \"df_pycaret\" as the dataset and \"Label\" as the target variable was used. * 10 Comparing Models and Select the Best One: 10.1 PyCaret's \"compare_models\" function was used to compare the performance of the available models (MLP, SVM, RF). 10.2 The best-performing model based on the comparison was selected. * 11 Evaluation of the Performance of the Best Model: 11.1 PyCaret's \"evaluate_model\" function was used to evaluate the performance of the best model selected in the previous step. The experimental results of the PyCarat analysis for different models are shown in Table 4. PyCarat is a performance analysis tool that provides insights into model performance using various evaluation metrics.