Talip7/scikit-learn-issues-multilabel huggingface dataset:
Add `strata` to `Stratified*Split` CV splitters

Right now `Stratified*Split` classes take `y` as the strata, while that's not always the case.

In `train_test_split` we allow a `stratify` arg (which I'm wondering if it should be called `strata`), which defines the groups samples belong to. And inside, we basically do this:

```py
    cv = StratifiedShuffleSplit(test_size=n_test, train_size=n_train, random_state=random_state)

    train, test = next(cv.split(X=arrays[0], y=stratify))

    return list(
        chain.from_iterable(
            (_safe_indexing(a, train), _safe_indexing(a, test)) for a in arrays
        )
    )
```

As you can see, we're passing `stratify` as `y` to the splitter. I think it would make sense to add a `strata` arg to the splitters, and if `None`, we'd take values in `y` instead, as it is now.

Note that now that we have SLEP6, we won't need a separate class for them, and we'd simply request `strata` for the splitter:

```py
cv = StratifiedShuffleSplit().set_split_request(strata=True)
...
GridSearchCV(model, param_grid, cv=cv).fit(X, y, strata=strata_values)
cross_validate(model, X, y, cv=cv, props={"strata": strata_values})
```

cc @marenwestermann

COMMENT:
We're at a deadlock here and that's not a state I'd like to leave this at.

I personally don't have the datasets (they're all private) nor the energy to write up what @ogrisel would need to be happy with this moving forward.

At the same time, according to @ogrisel 's arguments, we should remove stratification alltogether, are we gonna do that?

I see enough support here to move forward. Should we call for a vote? What's the realistically actionable path forward? Anything other than this being blocked and stalled.

Labels: New Feature, RFC
Add `strata` to `Stratified*Split` CV splitters

Right now `Stratified*Split` classes take `y` as the strata, while that's not always the case.

In `train_test_split` we allow a `stratify` arg (which I'm wondering if it should be called `strata`), which defines the groups samples belong to. And inside, we basically do this:

```py
    cv = StratifiedShuffleSplit(test_size=n_test, train_size=n_train, random_state=random_state)

    train, test = next(cv.split(X=arrays[0], y=stratify))

    return list(
        chain.from_iterable(
            (_safe_indexing(a, train), _safe_indexing(a, test)) for a in arrays
        )
    )
```

As you can see, we're passing `stratify` as `y` to the splitter. I think it would make sense to add a `strata` arg to the splitters, and if `None`, we'd take values in `y` instead, as it is now.

Note that now that we have SLEP6, we won't need a separate class for them, and we'd simply request `strata` for the splitter:

```py
cv = StratifiedShuffleSplit().set_split_request(strata=True)
...
GridSearchCV(model, param_grid, cv=cv).fit(X, y, strata=strata_values)
cross_validate(model, X, y, cv=cv, props={"strata": strata_values})
```

cc @marenwestermann

COMMENT:
@adrinjalali - I'll set aside some time this week to try and put together an argument & synthetic example illustrating the issue that this would solve, for those that haven't faced it before. Can any vote wait until the beginning of next week?

(Apologies for my delayed response.)

Labels: New Feature, RFC
Add `strata` to `Stratified*Split` CV splitters

Right now `Stratified*Split` classes take `y` as the strata, while that's not always the case.

In `train_test_split` we allow a `stratify` arg (which I'm wondering if it should be called `strata`), which defines the groups samples belong to. And inside, we basically do this:

```py
    cv = StratifiedShuffleSplit(test_size=n_test, train_size=n_train, random_state=random_state)

    train, test = next(cv.split(X=arrays[0], y=stratify))

    return list(
        chain.from_iterable(
            (_safe_indexing(a, train), _safe_indexing(a, test)) for a in arrays
        )
    )
```

As you can see, we're passing `stratify` as `y` to the splitter. I think it would make sense to add a `strata` arg to the splitters, and if `None`, we'd take values in `y` instead, as it is now.

Note that now that we have SLEP6, we won't need a separate class for them, and we'd simply request `strata` for the splitter:

```py
cv = StratifiedShuffleSplit().set_split_request(strata=True)
...
GridSearchCV(model, param_grid, cv=cv).fit(X, y, strata=strata_values)
cross_validate(model, X, y, cv=cv, props={"strata": strata_values})
```

cc @marenwestermann

COMMENT:
> I personally don't have the datasets (they're all private) nor the energy to write up what @ogrisel would need to be happy with this moving forward.

+1 for moving forward, and I will try to find the time to educate myself proactively on valid potential uses of this option so that we can document how to properly use it in the docstring / user guide or examples.

> @adrinjalali - I'll set aside some time this week to try and put together an argument & synthetic example illustrating the issue that this would solve, for those that haven't faced it before.

That would be great if you could contribute to this. Thanks!

> At the same time, according to @ogrisel 's arguments, we should remove stratification alltogether, are we gonna do that?

I don't think we are ever going to do that because:

- we would first need to update our CV tools to handle single class CV folds gracefully first (and I am not sure what's the proper way to address all edge cases correctly and without too much complexity);
- but even if we did, I think this would be a disruptive breaking change / noisy deprecation cycle to handle, and I am really not sure if it's worth it.

Labels: New Feature, RFC
Add `strata` to `Stratified*Split` CV splitters

Right now `Stratified*Split` classes take `y` as the strata, while that's not always the case.

In `train_test_split` we allow a `stratify` arg (which I'm wondering if it should be called `strata`), which defines the groups samples belong to. And inside, we basically do this:

```py
    cv = StratifiedShuffleSplit(test_size=n_test, train_size=n_train, random_state=random_state)

    train, test = next(cv.split(X=arrays[0], y=stratify))

    return list(
        chain.from_iterable(
            (_safe_indexing(a, train), _safe_indexing(a, test)) for a in arrays
        )
    )
```

As you can see, we're passing `stratify` as `y` to the splitter. I think it would make sense to add a `strata` arg to the splitters, and if `None`, we'd take values in `y` instead, as it is now.

Note that now that we have SLEP6, we won't need a separate class for them, and we'd simply request `strata` for the splitter:

```py
cv = StratifiedShuffleSplit().set_split_request(strata=True)
...
GridSearchCV(model, param_grid, cv=cv).fit(X, y, strata=strata_values)
cross_validate(model, X, y, cv=cv, props={"strata": strata_values})
```

cc @marenwestermann

COMMENT:
> While I am not fundamentally opposed to this feature, I wonder what concrete need it fulfills and what precise statistical question it answers. If we do decide to implement it, we should start with a convincing practical usage example that highlights how the lack of stratification in the CV evaluation loop would have otherwise led to mistaken conclusions.

I just found [this paper](https://eprints.whiterose.ac.uk/3570/1/willettp5.pdf) (full version [here](https://onlinelibrary.wiley.com/doi/abs/10.1002/qsar.200290002) and related code [here](https://github.com/rdkit/rdkit/blob/master/Code/SimDivPickers/MaxMinPicker.h)) with a usecase in chemistry. It claims:
> The systematic subsets contain more active molecules than would be expected by simple random
selection

In my understanding, stratifying on X ("structural diversity" in the terminology of said paper) mitigates data sampling biases. In the case of chemistry, a given lab may be more prompt to oversample a small region of the feature space (thus leading to over-representation of some combinations of features) whereas we want to cross-validate on folds with data coming from a feature-space as broad as possible.

![Image](https://github.com/user-attachments/assets/07c245a9-5418-4df0-a8bf-e1f73f3a1801)

![Image](https://github.com/user-attachments/assets/3312aa67-5075-4410-982d-61f568acb220)

Labels: New Feature, RFC
Add `strata` to `Stratified*Split` CV splitters

Right now `Stratified*Split` classes take `y` as the strata, while that's not always the case.

In `train_test_split` we allow a `stratify` arg (which I'm wondering if it should be called `strata`), which defines the groups samples belong to. And inside, we basically do this:

```py
    cv = StratifiedShuffleSplit(test_size=n_test, train_size=n_train, random_state=random_state)

    train, test = next(cv.split(X=arrays[0], y=stratify))

    return list(
        chain.from_iterable(
            (_safe_indexing(a, train), _safe_indexing(a, test)) for a in arrays
        )
    )
```

As you can see, we're passing `stratify` as `y` to the splitter. I think it would make sense to add a `strata` arg to the splitters, and if `None`, we'd take values in `y` instead, as it is now.

Note that now that we have SLEP6, we won't need a separate class for them, and we'd simply request `strata` for the splitter:

```py
cv = StratifiedShuffleSplit().set_split_request(strata=True)
...
GridSearchCV(model, param_grid, cv=cv).fit(X, y, strata=strata_values)
cross_validate(model, X, y, cv=cv, props={"strata": strata_values})
```

cc @marenwestermann

COMMENT:
@adrinjalali opened this issue because at the time I was woking on a machine learning project for my company that was focussing on predicting overcapacities on Germany's national railway network and this feature would have been useful for us. (Unfortunately I can't give more details without seeking approval first.)

I was thinking that maybe we could do a user survey on this (maybe just a LinkedIn post). We could make a simple poll on whether people think this feature is useful and also ask people if they could provide us with use cases. However, I'm aware that this involves a bit of work and I don't know if anyone here currently has the capacity to look after such a survey. 

I'm pining @OpenRailAssociation here because this might potentially be relevant for other European railway organisations as well.

Labels: New Feature, RFC
Clarify the differences between `LinearSVC` and `SVC`

### Describe the issue linked to the documentation

Currently the documentation for `LinearSVC` says that is "Similar to SVC with parameter `kernel="linear"`, but implemented in terms of liblinear rather than libsvm, so it has more flexibility in the choice of penalties and loss functions and should scale better to large numbers of samples".

This is confusing, because by default they are not equivalent. `LinearSVC` has a different default loss function and regularizes the intercept, as mentioned in these StackOverflow posts:
- https://stackoverflow.com/questions/45384185/what-is-the-difference-between-linearsvc-and-svckernel-linear
- https://stackoverflow.com/questions/35076586/when-should-one-use-linearsvc-or-svc
- https://stackoverflow.com/questions/33843981/under-what-parameters-are-svc-and-linearsvc-in-scikit-learn-equivalent

As a result, the current documentation can mislead and give result to research that does the wrong thing, as it almost happened to me.

### Suggest a potential alternative/fix

A big red box with a warning should be at the beginning of the documentation for `LinearSVC`, describing the ways in which differs from `SVC` and that it is not a "true" linear SVM (because of the intercept regularization).

In the future, it would be desirable in my opinion to change the default values/implementation of `LinearSVC` so that both `LinearSVC` and `SVC` with parameter `kernel="linear"` solve the same optimization problem by default. Otherwise the risk of it leading to wrong research results is very big.

COMMENT:
I think a nice addition would be to have a small line writing with what parameter combination they'd be equivalent. That should be enough to show their differences. I'd be happy to review a PR for that.

Labels: Documentation
Clarify the differences between `LinearSVC` and `SVC`

### Describe the issue linked to the documentation

Currently the documentation for `LinearSVC` says that is "Similar to SVC with parameter `kernel="linear"`, but implemented in terms of liblinear rather than libsvm, so it has more flexibility in the choice of penalties and loss functions and should scale better to large numbers of samples".

This is confusing, because by default they are not equivalent. `LinearSVC` has a different default loss function and regularizes the intercept, as mentioned in these StackOverflow posts:
- https://stackoverflow.com/questions/45384185/what-is-the-difference-between-linearsvc-and-svckernel-linear
- https://stackoverflow.com/questions/35076586/when-should-one-use-linearsvc-or-svc
- https://stackoverflow.com/questions/33843981/under-what-parameters-are-svc-and-linearsvc-in-scikit-learn-equivalent

As a result, the current documentation can mislead and give result to research that does the wrong thing, as it almost happened to me.

### Suggest a potential alternative/fix

A big red box with a warning should be at the beginning of the documentation for `LinearSVC`, describing the ways in which differs from `SVC` and that it is not a "true" linear SVM (because of the intercept regularization).

In the future, it would be desirable in my opinion to change the default values/implementation of `LinearSVC` so that both `LinearSVC` and `SVC` with parameter `kernel="linear"` solve the same optimization problem by default. Otherwise the risk of it leading to wrong research results is very big.

COMMENT:
`LinearSVC` and `SVC(kernel='linear')` are similar in that they both use a linear kernel, but they differ in their implementation, loss functions, and penalties. `LinearSVC` is implemented using the library `liblinear`, while `SVC` is implemented using `libsvm`. By default, `LinearSVC` minimizes the **squared hinge loss**, while `SVC` minimizes the regular **hinge loss**. Additionally, `LinearSVC` uses the **One-vs-Rest** multiclass reduction while `SVC` uses the **One-vs-One** multiclass reduction.

Labels: Documentation
Clarify the differences between `LinearSVC` and `SVC`

### Describe the issue linked to the documentation

Currently the documentation for `LinearSVC` says that is "Similar to SVC with parameter `kernel="linear"`, but implemented in terms of liblinear rather than libsvm, so it has more flexibility in the choice of penalties and loss functions and should scale better to large numbers of samples".

This is confusing, because by default they are not equivalent. `LinearSVC` has a different default loss function and regularizes the intercept, as mentioned in these StackOverflow posts:
- https://stackoverflow.com/questions/45384185/what-is-the-difference-between-linearsvc-and-svckernel-linear
- https://stackoverflow.com/questions/35076586/when-should-one-use-linearsvc-or-svc
- https://stackoverflow.com/questions/33843981/under-what-parameters-are-svc-and-linearsvc-in-scikit-learn-equivalent

As a result, the current documentation can mislead and give result to research that does the wrong thing, as it almost happened to me.

### Suggest a potential alternative/fix

A big red box with a warning should be at the beginning of the documentation for `LinearSVC`, describing the ways in which differs from `SVC` and that it is not a "true" linear SVM (because of the intercept regularization).

In the future, it would be desirable in my opinion to change the default values/implementation of `LinearSVC` so that both `LinearSVC` and `SVC` with parameter `kernel="linear"` solve the same optimization problem by default. Otherwise the risk of it leading to wrong research results is very big.

COMMENT:
@greyisbetter I think that was clear from the OPs post.

Labels: Documentation
Clarify the differences between `LinearSVC` and `SVC`

### Describe the issue linked to the documentation

Currently the documentation for `LinearSVC` says that is "Similar to SVC with parameter `kernel="linear"`, but implemented in terms of liblinear rather than libsvm, so it has more flexibility in the choice of penalties and loss functions and should scale better to large numbers of samples".

This is confusing, because by default they are not equivalent. `LinearSVC` has a different default loss function and regularizes the intercept, as mentioned in these StackOverflow posts:
- https://stackoverflow.com/questions/45384185/what-is-the-difference-between-linearsvc-and-svckernel-linear
- https://stackoverflow.com/questions/35076586/when-should-one-use-linearsvc-or-svc
- https://stackoverflow.com/questions/33843981/under-what-parameters-are-svc-and-linearsvc-in-scikit-learn-equivalent

As a result, the current documentation can mislead and give result to research that does the wrong thing, as it almost happened to me.

### Suggest a potential alternative/fix

A big red box with a warning should be at the beginning of the documentation for `LinearSVC`, describing the ways in which differs from `SVC` and that it is not a "true" linear SVM (because of the intercept regularization).

In the future, it would be desirable in my opinion to change the default values/implementation of `LinearSVC` so that both `LinearSVC` and `SVC` with parameter `kernel="linear"` solve the same optimization problem by default. Otherwise the risk of it leading to wrong research results is very big.

COMMENT:
`LinearSVC(loss='hinge')` and `SVC(kernel="linear", tol=0.0001, max_iter=1000, decision_function_shape='ovr')` are likely to produce similar results, despite  the fact that `SVC(kernel="linear")` may take longer to train compared to `LinearSVC`

Labels: Documentation
Clarify the differences between `LinearSVC` and `SVC`

### Describe the issue linked to the documentation

Currently the documentation for `LinearSVC` says that is "Similar to SVC with parameter `kernel="linear"`, but implemented in terms of liblinear rather than libsvm, so it has more flexibility in the choice of penalties and loss functions and should scale better to large numbers of samples".

This is confusing, because by default they are not equivalent. `LinearSVC` has a different default loss function and regularizes the intercept, as mentioned in these StackOverflow posts:
- https://stackoverflow.com/questions/45384185/what-is-the-difference-between-linearsvc-and-svckernel-linear
- https://stackoverflow.com/questions/35076586/when-should-one-use-linearsvc-or-svc
- https://stackoverflow.com/questions/33843981/under-what-parameters-are-svc-and-linearsvc-in-scikit-learn-equivalent

As a result, the current documentation can mislead and give result to research that does the wrong thing, as it almost happened to me.

### Suggest a potential alternative/fix

A big red box with a warning should be at the beginning of the documentation for `LinearSVC`, describing the ways in which differs from `SVC` and that it is not a "true" linear SVM (because of the intercept regularization).

In the future, it would be desirable in my opinion to change the default values/implementation of `LinearSVC` so that both `LinearSVC` and `SVC` with parameter `kernel="linear"` solve the same optimization problem by default. Otherwise the risk of it leading to wrong research results is very big.

COMMENT:
@greyisbetter again, the question is not how to get similar results, the question is how to document parameters which should in theory give the same results.

Labels: Documentation
Clarify the differences between `LinearSVC` and `SVC`

### Describe the issue linked to the documentation

Currently the documentation for `LinearSVC` says that is "Similar to SVC with parameter `kernel="linear"`, but implemented in terms of liblinear rather than libsvm, so it has more flexibility in the choice of penalties and loss functions and should scale better to large numbers of samples".

This is confusing, because by default they are not equivalent. `LinearSVC` has a different default loss function and regularizes the intercept, as mentioned in these StackOverflow posts:
- https://stackoverflow.com/questions/45384185/what-is-the-difference-between-linearsvc-and-svckernel-linear
- https://stackoverflow.com/questions/35076586/when-should-one-use-linearsvc-or-svc
- https://stackoverflow.com/questions/33843981/under-what-parameters-are-svc-and-linearsvc-in-scikit-learn-equivalent

As a result, the current documentation can mislead and give result to research that does the wrong thing, as it almost happened to me.

### Suggest a potential alternative/fix

A big red box with a warning should be at the beginning of the documentation for `LinearSVC`, describing the ways in which differs from `SVC` and that it is not a "true" linear SVM (because of the intercept regularization).

In the future, it would be desirable in my opinion to change the default values/implementation of `LinearSVC` so that both `LinearSVC` and `SVC` with parameter `kernel="linear"` solve the same optimization problem by default. Otherwise the risk of it leading to wrong research results is very big.

COMMENT:
Also, I am not sure if it is really possible to remove the intercept regularization in `LinearSVC`.

Labels: Documentation
Clarify the differences between `LinearSVC` and `SVC`

### Describe the issue linked to the documentation

Currently the documentation for `LinearSVC` says that is "Similar to SVC with parameter `kernel="linear"`, but implemented in terms of liblinear rather than libsvm, so it has more flexibility in the choice of penalties and loss functions and should scale better to large numbers of samples".

This is confusing, because by default they are not equivalent. `LinearSVC` has a different default loss function and regularizes the intercept, as mentioned in these StackOverflow posts:
- https://stackoverflow.com/questions/45384185/what-is-the-difference-between-linearsvc-and-svckernel-linear
- https://stackoverflow.com/questions/35076586/when-should-one-use-linearsvc-or-svc
- https://stackoverflow.com/questions/33843981/under-what-parameters-are-svc-and-linearsvc-in-scikit-learn-equivalent

As a result, the current documentation can mislead and give result to research that does the wrong thing, as it almost happened to me.

### Suggest a potential alternative/fix

A big red box with a warning should be at the beginning of the documentation for `LinearSVC`, describing the ways in which differs from `SVC` and that it is not a "true" linear SVM (because of the intercept regularization).

In the future, it would be desirable in my opinion to change the default values/implementation of `LinearSVC` so that both `LinearSVC` and `SVC` with parameter `kernel="linear"` solve the same optimization problem by default. Otherwise the risk of it leading to wrong research results is very big.

COMMENT:
I'll open a PR to add the comparisons to the docs.

Labels: Documentation