# Machine Learning Ship Classifiers for Signals from Passive Sonars
## 3 Classifier Models
### Logistic Regression  
Logistic regression is a statistical model used for binary classification in which the output is a categorical variable with two possible outcomes. A logistic regression estimates the probability of an event occurring [30]. It is based on the logistic function, also known as the sigmoid function, which maps any real-valued number into the range \((0,1)\) and allows logistic regression to output probabilities. The logistic function is given by  
\[\sigma(z)=\frac{1}{1+e^{-z}},\]  
where \(z=X\beta\), \(X\) is a linear combination of input features, \(\beta\) are the model coefficients, and \(e\) is the base of the natural logarithm. This function takes any value of \(z\), which can range from \(-\infty\) to \(\infty\), and compresses it in the range \((0,1)\). For logistic regression, \(z\) represents the log odds of the target variable being one. In logistic regression, we model the log odds of the event, that is, the logarithm of the ratio between the probability of success and the probability of failure, as a linear function of the input features:  
\[\log\biggl{(}\frac{P(y=1\mid X)}{1-P(y=1\mid X)}\biggr{)}=X\beta, \tag{4}\]  
where \(P(y=1\mid X)\) is the probability of the event occurring (the output being 1), \(X\) is the matrix of input features, and \(\beta\) is the vector of coefficients. The left-hand side of Equation (4) shows the log odds, and its right-hand side shows a linear combination of the input features. Taking the exponential on both sides yields  
\[\frac{P(y=1\mid X)}{1-P(y=1\mid X)}=e^{X\beta}.\]  
Solving this equation for \(P(y=1\mid X)\), we obtain the logistic regression model  
\[P(y=1\mid X)=\frac{1}{1+e^{-X\beta}},\]  
which represents the probability that the outcome is 1 given the input features. Once the probability \(P(y=1\mid X)\) is calculated, we can make the following prediction:  
* If \(P(y=1\mid X)\geq 0.5\), \(y=1\) is predicted;
* If \(P(y=1\mid X)<0.5\), \(y=0\) is predicted. To fit the logistic regression model, we estimate the coefficients \(\beta\). This is achieved by minimizing a cost function that measures the difference between the predicted probabilities and actual labels. For logistic regression, the cost function is the log loss or binary cross-entropy:  
\[J(\beta)=-\frac{1}{n}\sum_{i=1}^{n}[y_{i}\log(P(y_{i}\mid X_{i}))+(1-y_{i}) \log(1-P(y_{i}\mid X_{i}))],\]  
where \(y_{i}\) is the actual label for sample \(i\), \(P(y_{i}\mid X_{i})\) is the predicted probability for sample \(i\), and \(n\) is the number of samples. The log-loss function penalizes incorrect predictions more heavily, as the predicted probability diverges from the true label. Gradient descent is typically used to find the optimal coefficients \(\beta\) because the cost function in logistic regression is nonlinear. Gradient descent iteratively updates (represented by the symbol :=) the coefficients by moving in the direction of the negative gradient of the cost function:  
\[\beta:=\beta-\alpha\frac{\partial J(\beta)}{\partial\beta},\]  
where \(\alpha\) is the learning rate, which controls the step size, and \(\frac{\partial J(\beta)}{\partial\beta}\) is the gradient of the cost function with respect to \(\beta\). The gradient descent continues until the convergence criterion is satisfied. Similar to linear regression, regularization can be added to logistic regression to prevent overfitting. The two common types of regularization are as follows:  
* \(\ell_{2}\) regularization (ridge) adds a penalty term that is proportional to the square of the coefficients: \[J(\beta)=-\frac{1}{n}\sum_{i=1}^{n}[y_{i}\log(P(y_{i}\mid X_{i}))+(1-y_{i}) \log(1-P(y_{i}\mid X_{i}))]+\frac{\lambda}{2}\sum_{j=1}^{p}\beta_{j}^{2};\]
* \(\ell_{1}\) regularization (Lasso) adds a penalty proportional to the absolute value of the coefficients: \[J(\beta)=-\frac{1}{n}\sum_{i=1}^{n}[y_{i}\log(P(y_{i}\mid X_{i}))+(1-y_{i}) \log(1-P(y_{i}\mid X_{i}))]+\lambda\sum_{j=1}^{p}\beta_{j}.\]  
Here, \(\lambda\) controls the regularization strength.