## 2 Materials and Methods
### Design Phase
#### 2.2.7 Algorithm Selection with Machine Learning  
Due to the introduced scoring system, machine learning algorithms can learn which solver tends to provide satisfying results for a given instance. Machine learning can be divided into three major areas: supervised learning, unsupervised learning, and reinforcement learning [60]. Supervised learning is the form of machine learning that is most widely used in practice [61].  
The abovementioned problem was modeled as a multi-class classification problem, which implies classes and a ground truth. The classes were the winning CP solvers, either CP Optimizer (CPO) or OR-Tools, and the previously determined features. The goal was to learn a mapping from inputs x that corresponded to the features of the dataset and to outputs y, where y \(\in\) {1,..., K}, with K being the number of classes, in this case the set of solution algorithms.  
There are several classification algorithms available for predicting the output of a set of labeled input features. Among the most popular classification algorithms are logistic regression, decision trees, random forest, K-nearest neighbors (KNN), support vector machine, and naive Bayes. Furthermore, neural networks are also applicable to classification problems.  
To make the final selection of one machine learning algorithm to determine the optimal solver, they have to be tested on the dataset. This is because their performance significantly depends on the given input data. Therefore, only a pre-selection based on preliminary criteria is reasonable, and the final decision takes place after applying them to the problems. Due to their characteristics and suitability for the algorithm selection approach, logistic regression, decision trees, random forest, KNN, and neural networks were pre-selected. Logistic regression uses the sigmoid function to return the probability of a label. It is often used for binary classification problems, for instance, true or false and positive or negative. The decision tree algorithm is one of the most popular algorithms in use today. Decision trees build tree branches using a hierarchy approach, where each branch can be considered to be an if-else statement [62; 63]. The branches develop by partitioning the dataset into subsets based on the most important features. Their scale of explainability was ranked comparably very high in a study assessing several machine learning techniques in terms of their explainability. Especially when considering the fear of a majority regarding the black-box character of machine learning, this technique might provide broader consent [64]. Random forest represents a collection of decision trees. Because of the additional layers added to the model, random forest has better generalization but is less interpretable. KNN represents each data point in an n-dimensional space and calculates the distance between one point and another. Afterwards, it assigns labels to unobserved data based on the labels of the nearest observed data points [63]. Neural networks represent a set of algorithms that are modeled by the inspiration of the human brain. In this way, they are supposed to recognize patterns in data.  
The algorithms were implemented with the Scikit-learn library, an open-source machine learning library in Python for predictive data analysis. It is publicly available and was built using NumPy, SciPy, and matplotlib. Furthermore, the neural network was implemented with two approaches, one approach with Scikit-learn and the other with Keras. It is the most widely used deep learning framework among the five top teams on Kaggle. The parameters were set to defaults if not otherwise mentioned.