A note about documentation on the logistic regression.

### Describe the issue linked to the documentation

Dear Authors.
First of all, let me express my respect for your hard work and success, acclaimed and used by thousands of ML specialists. 
I would like to report a small issue in the documentation, providing a justification for my concern and proposal.

The documentation [says](https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression):

> Logistic regression, despite its name, is a linear model for classification rather than regression.

which is not true, alas. Logistic regression, invented and further developed in terms of the **Generalized Linear Model** and **Generalized Additive Model** by Cox, Nelder, Hastie, Tibshirani and others, served exactly to address regression problems with categorical outcomes (via Maximum Likelihood Estimation).

Every single regression model predicts some **statistic** (typically expected value, but can be also quantile, trimmed mean) **of the conditional distribution of the response**, which is always **a numerical outcome**. This applies to  the linear, logistic, probit, ordinal, multinomial, Poisson (BTW, this explains why it returns fractions rather than integers), negative binomial, fractional logistic regression, beta, gamma, Cox, and many, many more regression models.

The logistic regression is no different - the conditional distribution here is Bernoulli (or binomial with k=1), and the interpretation of the E(Y|X=x) is by def. the probability of the event (if success/occurrence/presence is coded as 1 and the failure/absence as 0).

In terms of the Generalized Linear Model, it's simply a GLM with binomial/Bernoulli conditional distribution and logit link. Or simpler - binomial regression with logit link. There are lots of monographs devoted exclusively to the regression aspects of the LR. There are also materials written by typically ML-oriented specialists like Prof. Andrew Ng (Stanford), placing it in frames of the GLM in his papers ([example](https://see.stanford.edu/materials/aimlcs229/cs229-notes1.pdf) page 22) and admitting this can be used for both applications: regression and classification. I will put also a few book covers below for reference.

For the last 60+ years the logistic regression has been one of the **core regression tools in experimental science** to analyse trials/studies with binary endpoints in: clinical trials (drug evaluation and approval part of the pharmaceutical industry), clinical laboratory diagnostics, epidemiology, ecology, econometric, medical biostatistics, actuary, experimental physics, psychology, toxicology and so on, including the causal analysis in these areas. This is also my daily regression tool to make inference in drug approval studies. Please check this query: [https://www.google.com/search?q=site%3Aclinicaltrials.gov+SAP+%2B+"logistic+regression"](https://www.google.com/search?q=site%3Aclinicaltrials.gov+SAP+%2B+"logistic+regression")

OK, but how is it used for regression purposes? It assesses how a certain set of independent variables affects the odds-ratio or % of successes (depending on formulation; doesn't matter) through the assessment of the:
- marginal and interaction effects: direction, magnitude, inference (p-value, confidence interval)
- simple effects (for categorical covariates): magnitude of difference/ratio + inference (p-value, CI)

... and, when the model is followed by the AN(C)OVA-like type analysis, here called the "_analysis of deviance"_, it gives us also the main and their interaction effects, e.g. "does gender and treatment affect the % of successes". It's done either via Likelihood Ratio tests of Wald's joint analysis of the model coefficients. The analysis [employs the EM-means ](https://www.r-bloggers.com/2020/04/estimating-and-testing-glms-with-emmeans/) (estimated marginal means), which - again - is a typical regression outcome.

It is used also in the [IPW - inverse probability weighting.](https://www.r-bloggers.com/2017/12/when-you-use-inverse-probability-weighting-for-estimation-what-are-the-weights-actually-doing/).

The logistic regression was greatly extended to handle repeated or clustered observations via **GEE** (Generalized Estimated Equations) estimation and **Generalized Linear Mixed Models**. There is also the [Alternating Logistic Regression](https://www.google.com/search?q=alternating+logistic+regression) also used to handle repetitions. Further generalizations occur as well, for example the Ordinal Logistic Regression (aka proportional odds regression model) is the [generalization ](https://hbiostat.org/blog/post/wpo/) of the Mann-Whitney (Wilcoxon) test with covariates. Further extension includes GAM (to relax linearity in parameters) and regularization (LASSO, ridge) to address multi-collinearity. For multiple classes, that are somehow dependent/ordered/nested we have the [Nested LR](https://www.bauer.uh.edu/rsusmel/phd/ec1-20.pdf). It shows well, how rich is the set of regression-oriented tools built around the Logistic Regression.

As every regression, it is also used for predictions. And now, having the predicted probability, by applying a threshold to it (usually, but not necessarily 50%) one can use it for classification. It's then the **logistic classifier based on the result of the logistic regression**.

**There is no conflict of meanings**. Regression is for numerical outcome - and every regression gives numerical outcome by its very definition, including the Logistic Regression (probability is numeric). Classification - for binary outcome - and this is given by the Logistic Classifier. Good!

**There is a conflict**, however, if some says that "_logistic regression is not a regression_", because it:
1. denies the basic knowledge about how regression works (conditional expectation),
2. ignores the fact, the the LR is used by big thousands of researchers and data scientists for regression purposes,
3. limits those, who won't be able to complete a basic regression analysis of percentages believing that they have no tool for it.
4. Not to mention, that some learners, who read that, may repeat it during an exam or interview and fail it.

The books I mentioned above:
![obraz](https://user-images.githubusercontent.com/60151643/194738074-52d2a5bf-9b50-42fe-8283-c8fefb8b5944.png)

Especially this one explains the idea of regression as the conditional expectation (including the GLM):
![obraz](https://user-images.githubusercontent.com/60151643/194738123-bf57cdba-1240-49ab-ab2b-edd3b3699b6b.png)


### Suggest a potential alternative/fix

To conclude, I would like to propose changing the sentence to:

> Logistic regression is a special case of the generalized linear regression model (GLM). In Machine Learning it is used mainly for classification by taking the predicted probability and applying a decision rule to it. This makes the Logistic Classifier.

(prof. Ng can be referred)

Like on this quick and dirty illustration:
![obraz](https://user-images.githubusercontent.com/60151643/194738359-7e09938e-d414-44f8-9d16-6877685008ae.png)

This is a **win-win** definition:
1. agrees with statistical definition of regression
2. agrees with ML terminology
5. finally one doesn't have to deny the logic, common sense and struggle with a question: "why is it named a regression if it classifies stuff?"
6. it teaches people, that there exists the GLM (with extensions), which opens big possibilities in data analysis, if - one day - a ML specialist will have to analyse such data.

COMMENT:
I could work on this and update `doc/modules/linear_model.rst`

Just to clarify:
The first sentence would be modified to:

> LogisticRegression, despite its name, is implemented as a linear model for classification rather than regression in terms of scikit-learn/ML nomenclature.

In a note below that we would elaborate:

> Logistic regression is a special case of the Generalized Linear Regression Model (GLM). The numerical output of the logistic regression, which is the predicted probability, can be used to create a classifier by applying a threshold (by default 0.5) to it. This is how we use it in scikit-learn, so here it expects categorical target. This makes it the Logistic (Regression) Classifier.

Anything else that I may have missed? @Generalized @lorentzenchr @glemaitre

Labels: Documentation