SGDOneClassSVM model does not converge with default stopping criteria(stops prematurely)

### Describe the bug

SGDOneClassSVM does not converge with default early stopping criteria, because the used loss is not actual loss, but only error, which can be easily 0.0 and then increase as the model converges to adequate solution. That is, the used for stopping and reported with verbose "loss" value doesn't accout for the full model formula/regularization. Also, pay attention to bias term to gauge convergence.
https://github.com/scikit-learn/scikit-learn/blob/c7839c48363d1531af9a00abfcb9d911ecfcb2b2/sklearn/linear_model/_sgd_fast.pyx.tp#L482
The optimization almost always stops after 6 epochs, the initial epoch, plus the 5 for stopping tolerance (can't change the number of epochs for the stopping tolerance btw).
The problem does not manifest with toy data(small dimensiaonal), becasue 6 epochs is likely enough for convergence to satisfactory solution.
In the reproduction code, mind the console output and comments. Possible workaround at the end of reproduction code, is to use tol=None with manual epoch limit(max_iter), but that slows the optimization down by a lot, since forbids the use of learning_rate="adaptive".

### Steps/Code to Reproduce
```python
import numpy as np
from matplotlib import pyplot as plt
import pandas as pd
#from sklearn.linear_model import Ridge
from sklearn.datasets import make_regression
from timeit import timeit
from sklearn.linear_model import SGDOneClassSVM
from sklearn.svm import OneClassSVM
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler, MinMaxScaler

np.random.seed(123)

#no matter the feature count, optimization stops in 6 epochs
print("fitting different feature counts:")
featCnts = [10, 1000, 25000]
for featCnt in featCnts:
    print("\n10k samples, {} features".format(featCnt))
    x, y = make_regression(10000, featCnt, n_informative=featCnt // 10)
    x = MinMaxScaler().fit_transform(x) + 1.0 #make positive
    model = SGDOneClassSVM(nu=0.01, verbose=10)#, tol=1e-10)
    model.fit(x) #see console output

#model reports 0 "loss" even after just 1 sample
print("\n\npartial fit test:")
x, y = make_regression(10000, 2500//4, n_informative=250)
xPos = StandardScaler().fit_transform(x) + 100 #highly positive data
#Note: linear one class svm has to run with e.g. positive data for correct function(not centered about 0), otherwise data is not separable from origin
print("xPos data mean", xPos.mean())
model = SGDOneClassSVM(nu=0.01, verbose=10)#, tol=1e-10)
print("fit 1 sample")
model.partial_fit(xPos[:1], y) #gives avg loss 0.0
print("fit 100 samples")
model.partial_fit(xPos[:100], y)

#demonstrate that model output(decision function) is far from expected
# by comparing with a slow reference model
print("\n\nreference comparison:")
testNu = 0.5
modelSgd = make_pipeline( MinMaxScaler(), SGDOneClassSVM(nu=testNu, verbose=10) )
modelSgd.fit(x)
modelSgdDecFn = modelSgd.decision_function(x)
modelSgdClass = modelSgd.predict(x)

refMinMaxModel = make_pipeline( MinMaxScaler(), OneClassSVM(nu=testNu, verbose=10, tol=1e-10, kernel="linear", shrinking=False) ).fit(x)
refMinMax = refMinMaxModel.decision_function(x)
refMinMaxClass = refMinMaxModel.predict(x)

#slow sgd, manually tuned learning rate model
invScalingPower = np.emath.logn( len(x) * 9000, 100 ) #target lr reduction coefficient after N samples (len(x) * epoch count to get sample count)
#invScalingPower =0.2
print("invScalingPower", invScalingPower )
sgdMinMaxModel = make_pipeline( MinMaxScaler(), SGDOneClassSVM(nu=testNu, verbose=10, max_iter=10000//10, tol=None,
    learning_rate="constant", eta0=0.001, power_t=invScalingPower, average=True) ).fit(x)#, sgdoneclasssvm__coef_init=np.random.normal(-100.0, 1.0, x.shape[1])) # sgdoneclasssvm__offset_init=1000)
sgdMinMax = sgdMinMaxModel.decision_function(x)
sgdMinMaxClass = sgdMinMaxModel.predict(x)


df = pd.DataFrame({"sklearn default modelSgd": modelSgdDecFn,  "reference ocsvm refMinMax": refMinMax, "manual stopping sgdMinMax": sgdMinMax, })
print("decision function:\n", df)
print("correlation between decision_function:\n", df.corr())

df = pd.DataFrame({"sklearn default modelSgd": modelSgdClass,  "reference ocsvm refMinMax": refMinMaxClass, "manual stopping sgdMinMax": sgdMinMaxClass, })
print("correlation between predict:\n", df.corr())
```

### Expected Results

Model convergence criteria works adequately (especially important for the most efficient learning_rate="adaptive"), and model reaches convergence.

### Actual Results

Optimization stops prematurely, usually after 6 epochs:
```
fitting different feature counts:

10k samples, 10 features
-- Epoch 1
Norm: 1.41, NNZs: 10, Bias: -5.067461, T: 10000, Avg. loss: 0.000088
Total training time: 0.00 seconds.
-- Epoch 2
Norm: 1.66, NNZs: 10, Bias: -6.148603, T: 20000, Avg. loss: 0.000187
Total training time: 0.00 seconds.
-- Epoch 3
Norm: 1.80, NNZs: 10, Bias: -6.766193, T: 30000, Avg. loss: 0.000267
Total training time: 0.00 seconds.
-- Epoch 4
Norm: 1.87, NNZs: 10, Bias: -7.203003, T: 40000, Avg. loss: 0.000191
Total training time: 0.00 seconds.
-- Epoch 5
Norm: 1.97, NNZs: 10, Bias: -7.528345, T: 50000, Avg. loss: 0.000193
Total training time: 0.00 seconds.
-- Epoch 6
Norm: 2.01, NNZs: 10, Bias: -7.798230, T: 60000, Avg. loss: 0.000285
Total training time: 0.00 seconds.
Convergence after 6 epochs took 0.01 seconds

10k samples, 1000 features
-- Epoch 1
Norm: 0.95, NNZs: 1000, Bias: -5.741972, T: 10000, Avg. loss: 0.000000
Total training time: 0.01 seconds.
-- Epoch 2
Norm: 0.47, NNZs: 1000, Bias: -7.123019, T: 20000, Avg. loss: 0.000000
Total training time: 0.02 seconds.
-- Epoch 3
Norm: 0.32, NNZs: 1000, Bias: -7.932197, T: 30000, Avg. loss: 0.000000
Total training time: 0.03 seconds.
-- Epoch 4
Norm: 0.24, NNZs: 1000, Bias: -8.506685, T: 40000, Avg. loss: 0.000000
Total training time: 0.05 seconds.
-- Epoch 5
Norm: 0.38, NNZs: 1000, Bias: -8.948081, T: 50000, Avg. loss: 0.000001
Total training time: 0.06 seconds.
-- Epoch 6
Norm: 0.32, NNZs: 1000, Bias: -9.312374, T: 60000, Avg. loss: 0.000000
Total training time: 0.07 seconds.
Convergence after 6 epochs took 0.07 seconds

10k samples, 25000 features
-- Epoch 1
Norm: 4.73, NNZs: 25000, Bias: -5.741972, T: 10000, Avg. loss: 0.000000
Total training time: 0.24 seconds.
-- Epoch 2
Norm: 2.37, NNZs: 25000, Bias: -7.123019, T: 20000, Avg. loss: 0.000000
Total training time: 0.48 seconds.
-- Epoch 3
Norm: 1.58, NNZs: 25000, Bias: -7.932197, T: 30000, Avg. loss: 0.000000
Total training time: 0.71 seconds.
-- Epoch 4
Norm: 1.19, NNZs: 25000, Bias: -8.506685, T: 40000, Avg. loss: 0.000000
Total training time: 0.95 seconds.
-- Epoch 5
Norm: 0.95, NNZs: 25000, Bias: -8.952446, T: 50000, Avg. loss: 0.000000
Total training time: 1.19 seconds.
-- Epoch 6
Norm: 0.79, NNZs: 25000, Bias: -9.316738, T: 60000, Avg. loss: 0.000000
Total training time: 1.43 seconds.
Convergence after 6 epochs took 1.43 seconds


partial fit test:
xPos data mean 100.00000000000011
fit 1 sample
-- Epoch 1
Norm: 9397.76, NNZs: 625, Bias: 4.722997, T: 1, Avg. loss: 0.000000
Total training time: 0.00 seconds.
fit 100 samples
-- Epoch 1
Norm: 3262.77, NNZs: 625, Bias: 2.619430, T: 100, Avg. loss: 0.000000
Total training time: 0.00 seconds.


reference comparison:
-- Epoch 1
Norm: 1.21, NNZs: 625, Bias: -13.904809, T: 10000, Avg. loss: 0.001472
Total training time: 0.01 seconds.
-- Epoch 2
Norm: 1.32, NNZs: 625, Bias: -15.210391, T: 20000, Avg. loss: 0.001823
Total training time: 0.02 seconds.
-- Epoch 3
Norm: 1.38, NNZs: 625, Bias: -15.971154, T: 30000, Avg. loss: 0.002099
Total training time: 0.03 seconds.
-- Epoch 4
Norm: 1.43, NNZs: 625, Bias: -16.509796, T: 40000, Avg. loss: 0.002299
Total training time: 0.04 seconds.
-- Epoch 5
Norm: 1.46, NNZs: 625, Bias: -16.926977, T: 50000, Avg. loss: 0.002480
Total training time: 0.04 seconds.
-- Epoch 6
Norm: 1.49, NNZs: 625, Bias: -17.267392, T: 60000, Avg. loss: 0.002436
Total training time: 0.05 seconds.
Convergence after 6 epochs took 0.05 seconds
..*
optimization finished, #iter = 2497
obj = 1924865835.187691, rho = 776455.657200
nSV = 5001, nBSV = 4999
[LibSVM]invScalingPower 0.25143814733164016
-- Epoch 1
Norm: 0.40, NNZs: 625, Bias: -3.925000, T: 10000, Avg. loss: 0.000117
Total training time: 0.01 seconds.
-- Epoch 2
Norm: 0.79, NNZs: 625, Bias: -8.770000, T: 20000, Avg. loss: 0.000492
Total training time: 0.02 seconds.
-- Epoch 3

...

-- Epoch 998
Norm: 24.84, NNZs: 625, Bias: -309.593000, T: 9980000, Avg. loss: 1.322528
Total training time: 14.50 seconds.
-- Epoch 999
Norm: 24.80, NNZs: 625, Bias: -309.597000, T: 9990000, Avg. loss: 1.315051
Total training time: 14.52 seconds.
-- Epoch 1000
Norm: 24.80, NNZs: 625, Bias: -309.597000, T: 10000000, Avg. loss: 1.310013
Total training time: 14.53 seconds.
decision function:
       sklearn default modelSgd  reference ocsvm refMinMax  manual stopping sgdMinMax
0                     0.152997               -8169.563012                  -2.922611
1                     0.208675               -5888.609526                  -2.068050
2                     0.460575                4671.620928                   1.890327
3                     0.324515                -993.308774                  -0.232038
4                     0.376079                1109.737641                   0.558291
...                        ...                        ...                        ...
9995                  0.203837               -5984.511457                  -2.103481
9996                  0.263156               -3678.767938                  -1.238047
9997                  0.408527                2547.241215                   1.097869
9998                  0.408359                2477.709831                   1.068493
9999                  0.534379                7887.142835                   3.098472

[10000 rows x 3 columns]
correlation between decision_function:
                            sklearn default modelSgd  reference ocsvm refMinMax  manual stopping sgdMinMax
sklearn default modelSgd                   1.000000                   0.999918                   0.999919
reference ocsvm refMinMax                  0.999918                   1.000000                   1.000000
manual stopping sgdMinMax                  0.999919                   1.000000                   1.000000
correlation between predict:
                            sklearn default modelSgd  reference ocsvm refMinMax  manual stopping sgdMinMax
sklearn default modelSgd                   1.000000                   0.195700                   0.203691
reference ocsvm refMinMax                  0.195700                   1.000000                   0.960769
manual stopping sgdMinMax                  0.203691                   0.960769                   1.000000
```

### Versions

```shell
System:
    python: 3.7.3 (v3.7.3:ef4ec6ed12, Mar 25 2019, 22:22:05) [MSC v.1916 64 bit (AMD64)]
executable: E:\Program Files\Python\Python37\python.exe
   machine: Windows-10-10.0.17763-SP0

Python dependencies:
          pip: 24.0
   setuptools: 68.0.0
      sklearn: 1.0.2
        numpy: 1.21.6
        scipy: 1.7.3
       Cython: 3.0.11
       pandas: 1.3.5
   matplotlib: 3.5.3
       joblib: 1.0.1
threadpoolctl: 2.1.0

Built with OpenMP: True
```

COMMENT:
@glemaitre 

> Actually, I'm under the impression that this bug is for all models that uses the SGD solver.

Yes, indeed that's the case. However, in my tests with SGDRegressor/Classifier, the issue is not nearly as pronounced since usually full loss with regularization and just the error closely match, so the stopping criteria works fine. Regarding the documentation and verbose output, I really got confused by the error being called "loss" in the verbose mode, as well as the term "objective function" in the docs, because that made me think of the full objective, not just the error.

Labels: Bug