## 3 Methodology
### Model Selection
#### 3.6.2 Logistic Regression  
Logistic Regression (LR) is a supervised learning binary classification algorithm that predicts the probability of a sample belonging to a certain class. Equation (2) formulates this process, where the equation takes all input features as weighted variables, and the output would be either 0 or 1 with each extremity representing the two classes [56].  
\[y=w_{0}+w_{1}x_{1}+w_{2}x_{2}+w_{3}x_{3}+\dots+w_{n}x_{n} \tag{2}\]  
In Equation (2), \(y\) is either 0 or 1, the input features are \(x_{1},x_{2},x_{3}\),..., \(x_{n}\), and the weights of the features are given by \(w_{1},w_{2},w_{3},\)..., \(w_{n}\), where \(n\) is the number of features.  
The output can be plotted as an S-shaped curve, which is also called a _Sigmoid_ function. The curve helps to visualize the predicted probabilities and how outputs are assigned. This model is called _regression_ since the prediction is a range of values between 0 and 1, but it performs binary classification since the output can only have two possible values, 0 or 1.  
During the model training process, the weights are calculated based on the output values. Then, during model evaluation, the testing samples are assigned output values by substituting the values of their input features into the equation, multiplying each of them with their respective calculated weights, adding them up, and the resulting output would be assigned 0 or 1 based on the value to which it is closest.  
An optimization algorithm is used to find the optimal weights and bias during training. Conventionally, gradient descent is used for this purpose. An LR Model trained on the same dataset with different optimization algorithms may result in different outcomes.  
To prevent overfitting the S-shaped curve to the training data, there are regularization methods called \(L1\) Loss and \(L2\) Loss functions. An LR model can be trained using either one of these loss functions or neither of them. Implementing a loss function improves model performance by penalizing incorrect predictions.  
Using Python's scikit-learn library, we experimented with several parameters for Logistic Regression. The _penalty_ parameter trains the LR model with either \(L1\) or \(L2\) loss functions (or none) to regularize the data and avoid overfitting to the training set. The  
\begin{table}
\begin{tabular}{l c c c c c} \hline \hline
**Weights** & \(k\) **= 1** & \(k\) **= 3** & \(k\) **= 5** & \(k\) **= 7** & \(k\) **= 9** \\ \hline Uniform & \(k\)NN\_M1 & \(k\)NN\_M3 & \(k\)NN\_M5 & \(k\)NN\_M7 & \(k\)NN\_M9 \\ \hline Distance & \(k\)NN\_M2 & \(k\)NN\_M4 & \(k\)NN\_M6 & \(k\)NN\_M8 & \(k\)NN\_M10 \\ \hline \hline \end{tabular}
\end{table}
Table 4: \(k\)NN models and their parameters.  
class_weight_ parameter is _None_ by default but can be set to _balanced_ for an imbalanced training set. This assigns samples of the minority class with a higher weight and majority class samples with a lower weight. This may improve model performance. The _solver_ parameter implements different optimization algorithms. The \(C\) parameter specifies the strength of regularization, whereas a smaller \(C\) would specify stronger regularization [54].  
We experimented with the _penalty_, _solver_, and \(C\) parameters and found that _class_weight_\(=\)_'balanced_' and _penalty_\(=\)'L2' worked well for all datasets. We then varied \(C\) and _solver_, and evaluated the performance of the logistic regression models. Table 5 summarizes the different LR models resulting from parameter variations.