Talip7/scikit-learn-issues-multilabel huggingface dataset:
[RANSACRegressor] Strange behaviour with rectangular regions

### Describe the bug

I'm using sklearn in order to find a line that best fits a rectangular region using the `RANSACRegressor`. In general RANSAC works pretty good but it seems to have some problem with vertical rectangular regions.

### Steps/Code to Reproduce

```python
from sklearn import linear_model
import matplotlib.pyplot as plt
import numpy as np

# Create the dataset
WIDTH = 200
HEIGHT = 400
x = np.empty((WIDTH*HEIGHT))
y = np.empty_like(x)
count = 0
for i in range(WIDTH):
    for j in range(HEIGHT):
        x[count] = i
        y[count] = j
        count+=1

# Fit the line
ransac = linear_model.RANSACRegressor(residual_threshold = 100, max_trials= 1000)
ransac.fit(x.reshape(-1,1), y)

# Show the results
R = np.array([ransac.estimator_.coef_[0], ransac.predict([[0]])[0]])
inlier_mask = ransac.inlier_mask_
xp = np.linspace(0, 200)
p = np.poly1d(R)
plt.axes().set_aspect('equal')
plt.plot(x, y, '.', x[inlier_mask], y[inlier_mask], "-", xp, p(xp), '-',)
```

### Expected Results

I expected that in 1000 trials the regressor finds the vertical line (or at leasts one of the diagonal of the rectangle) as the best line

### Actual Results

The actual results is something similar to this:

![nvMfH](https://user-images.githubusercontent.com/34270884/154860865-b39b069c-5f61-45dc-bb08-bfaccc6fccf4.png)

The blue points are the outliers, the orange points are the inliers, the green line is the fitted regressor.

### Versions

```shell
System:
    python: 3.9.7 (default, Sep 10 2021, 14:59:43)  [GCC 11.2.0]
executable: /bin/python3
   machine: Linux-5.13.0-28-generic-x86_64-with-glibc2.34

Python dependencies:
          pip: 20.3.4
   setuptools: 52.0.0
      sklearn: 1.0.2
        numpy: 1.17.4
        scipy: 1.8.0
       Cython: None
       pandas: 1.3.4
   matplotlib: 3.5.1
       joblib: 1.1.0
threadpoolctl: 3.1.0

Built with OpenMP: True
```

COMMENT:
In this case I think it is expected behavior. With your code snippet, the line found is different with every run. This happens because the RANSAC algorithm samples the data in each trial controlled by the `min_samples` parameter. By default, `min_samples` is `X.shape[1] + 1`, which is equal to 2 in your example. This means each trials uses two random points to fit a line and then uses that line to find inliers. This leads to a different line with every run.

If you want the vertical line, you can sample all the points by setting `min_samples=1.0`:

```python
RANSACRegressor(residual_threshold = 100, max_trials= 100, min_samples=1.0)
```

which results in:

![Screen Shot 2022-05-05 at 2 41 28 PM](https://user-images.githubusercontent.com/5402633/167000260-48936441-afb3-486e-ae94-239d0b16cf79.jpg)

Labels: Bug, module:linear_model, Needs Triage
ColumnTransformer: integer column index in dataframes unexpected behaviour and error (column selectors vs _get_column_indices)

### Describe the bug

Unexpected behaviour when using integer column index in a DataFrame, other than natural ordering [0, 1, ...].

```
select_int = make_column_selector(dtype_include=np.int_)
ct = ColumnTransformer([('t2', Normalizer(norm="l1"), select_int)])
df1 = pd.DataFrame({'1': [1, 2, 3], '2': [9, 8, 7]})
df2 = pd.DataFrame({1: [1, 2, 3], 2: [9, 8, 7]})
ct.fit_transform(df1) # OK
ct.fit_transform(df2) # IndexError
```

The only difference between `df1` and `df2` is the type of column index. In my opinion, the results for these dataframes must be similar, but an error is raised for the latter.

As far as I could see, the problem stems from semantic ambiguity as to when to use `iloc`-based indexing vs `loc`-based indexing. In `_get_column_indices` [L382](https://github.com/scikit-learn/scikit-learn/blob/998e8f2068c7d9f43a30d997b63e4875688d0b1a/sklearn/utils/__init__.py#L382) this decision is based on the type of index and not on the type of the array. Whichever criterion is chosen, if it followed consistently in column selectors, the error shall be avoided. Probably.

### Steps/Code to Reproduce

(See above)

### Expected Results

(See above)

### Actual Results

(See above)

### Versions

```shell
Python dependencies:
          pip: 22.0.3
   setuptools: 60.8.1
      sklearn: 1.1.dev0
        numpy: 1.22.2
        scipy: 1.8.0
       Cython: 0.29.27
       pandas: 1.3.5
   matplotlib: 3.5.0
       joblib: 1.1.0
threadpoolctl: 3.1.0

commit b28c5bba66529217ceedd497201a684e5d35b73c (upstream/main, origin/main, origin/HEAD, main)
Author: Thomas J. Fan 
Date:   Tue Feb 15 11:46:54 2022 -0500

    FIX DummyRegressor overriding constant (#22486)
```

COMMENT:
I agree that using `make_column_selector` when selecting a DataFrame with integer column names is not great.

As you noted `ColumnTransformer` interprets integer selection as positional indices of the DataFrame. This behavior is detailed in the [ColumnTransformer](https://scikit-learn.org/stable/modules/generated/sklearn.compose.ColumnTransformer.html#sklearn.compose.ColumnTransformer) docstring:

> Integers are interpreted as positional columns, while strings can reference DataFrame columns by name.

The semantics for `ColumnTransformer` was decided a while ago and changing it would be backward incompatible. For now, the workaround would be to avoid using integer column names.

Assuming we do not change `ColumnTransformer`, here is a API proposal: We can add a `returns=` kwarg to `make_column_selector` such that:

1. `make_column_selector(..., returns="names")`: The default which is what we do now.
2. `make_column_selector(..., return="positional")`: Returns the columns as positions. For your use case, you can use this to select on any datafarme regardless of dtype of the columns.

Labels: New Feature, API, Needs Decision, module:compose
ColumnTransformer: integer column index in dataframes unexpected behaviour and error (column selectors vs _get_column_indices)

### Describe the bug

Unexpected behaviour when using integer column index in a DataFrame, other than natural ordering [0, 1, ...].

```
select_int = make_column_selector(dtype_include=np.int_)
ct = ColumnTransformer([('t2', Normalizer(norm="l1"), select_int)])
df1 = pd.DataFrame({'1': [1, 2, 3], '2': [9, 8, 7]})
df2 = pd.DataFrame({1: [1, 2, 3], 2: [9, 8, 7]})
ct.fit_transform(df1) # OK
ct.fit_transform(df2) # IndexError
```

The only difference between `df1` and `df2` is the type of column index. In my opinion, the results for these dataframes must be similar, but an error is raised for the latter.

As far as I could see, the problem stems from semantic ambiguity as to when to use `iloc`-based indexing vs `loc`-based indexing. In `_get_column_indices` [L382](https://github.com/scikit-learn/scikit-learn/blob/998e8f2068c7d9f43a30d997b63e4875688d0b1a/sklearn/utils/__init__.py#L382) this decision is based on the type of index and not on the type of the array. Whichever criterion is chosen, if it followed consistently in column selectors, the error shall be avoided. Probably.

### Steps/Code to Reproduce

(See above)

### Expected Results

(See above)

### Actual Results

(See above)

### Versions

```shell
Python dependencies:
          pip: 22.0.3
   setuptools: 60.8.1
      sklearn: 1.1.dev0
        numpy: 1.22.2
        scipy: 1.8.0
       Cython: 0.29.27
       pandas: 1.3.5
   matplotlib: 3.5.0
       joblib: 1.1.0
threadpoolctl: 3.1.0

commit b28c5bba66529217ceedd497201a684e5d35b73c (upstream/main, origin/main, origin/HEAD, main)
Author: Thomas J. Fan 
Date:   Tue Feb 15 11:46:54 2022 -0500

    FIX DummyRegressor overriding constant (#22486)
```

COMMENT:
~~What could possibly break if you apply the same rule:~~

> Integers are interpreted as positional columns, while strings can reference DataFrame columns by name.

~~to `_get_column_indices`  ?~~

~~I skimmed through an older version of code, and I think _it used to be exactly this way_, see e.g. [this commit](https://github.com/scikit-learn/scikit-learn/commit/5f37ec1a997ec84a63739c2dc036d748bb47ae67) of #14035 . Probably some time after that, someone changed the logic: instead of checking the type of passed indices they decided to check the type of the dataframe's column Index.~~

P.S. I got a bit confused with the details, but I have a strong feeling that the default behaviour of `make_column_selector` can be changed without any harm done. I see a little problem with your proposition: if I want my pipeline to work well when the data is like `df['1']` and also when it is `df[1]`, I have to use `make_column_selector(..., return="positional")` and completely forgo the convenience offered by strings.

Labels: New Feature, API, Needs Decision, module:compose
ColumnTransformer: integer column index in dataframes unexpected behaviour and error (column selectors vs _get_column_indices)

### Describe the bug

Unexpected behaviour when using integer column index in a DataFrame, other than natural ordering [0, 1, ...].

```
select_int = make_column_selector(dtype_include=np.int_)
ct = ColumnTransformer([('t2', Normalizer(norm="l1"), select_int)])
df1 = pd.DataFrame({'1': [1, 2, 3], '2': [9, 8, 7]})
df2 = pd.DataFrame({1: [1, 2, 3], 2: [9, 8, 7]})
ct.fit_transform(df1) # OK
ct.fit_transform(df2) # IndexError
```

The only difference between `df1` and `df2` is the type of column index. In my opinion, the results for these dataframes must be similar, but an error is raised for the latter.

As far as I could see, the problem stems from semantic ambiguity as to when to use `iloc`-based indexing vs `loc`-based indexing. In `_get_column_indices` [L382](https://github.com/scikit-learn/scikit-learn/blob/998e8f2068c7d9f43a30d997b63e4875688d0b1a/sklearn/utils/__init__.py#L382) this decision is based on the type of index and not on the type of the array. Whichever criterion is chosen, if it followed consistently in column selectors, the error shall be avoided. Probably.

### Steps/Code to Reproduce

(See above)

### Expected Results

(See above)

### Actual Results

(See above)

### Versions

```shell
Python dependencies:
          pip: 22.0.3
   setuptools: 60.8.1
      sklearn: 1.1.dev0
        numpy: 1.22.2
        scipy: 1.8.0
       Cython: 0.29.27
       pandas: 1.3.5
   matplotlib: 3.5.0
       joblib: 1.1.0
threadpoolctl: 3.1.0

commit b28c5bba66529217ceedd497201a684e5d35b73c (upstream/main, origin/main, origin/HEAD, main)
Author: Thomas J. Fan 
Date:   Tue Feb 15 11:46:54 2022 -0500

    FIX DummyRegressor overriding constant (#22486)
```

COMMENT:
> if I want my pipeline to work well when the data is like df['1'] and also when it is df[1], I have to use make_column_selector(..., return="positional") and completely forgo the convenience offered by strings.

Selecting columns based on two different types is not well supported and would require a design discussion. It also breaks backward compatibility.

If this is a major issue, I think we could add a `use_index_directly` parameter to `ColumnTransformer` that turns off these semantics and selects from the column directly. For example:

```python
ct = ColumnTransformer([('t2', Normalizer(norm="l1"), [1, 2])], use_index_directly=True)
df2 = pd.DataFrame({1: [1, 2, 3], 2: [9, 8, 7]})

ct.fit_transform(df2)
# Internally we will do `df2[[1, 2]]` for selection.
```

What do you think?

Labels: New Feature, API, Needs Decision, module:compose
ColumnTransformer: integer column index in dataframes unexpected behaviour and error (column selectors vs _get_column_indices)

### Describe the bug

Unexpected behaviour when using integer column index in a DataFrame, other than natural ordering [0, 1, ...].

```
select_int = make_column_selector(dtype_include=np.int_)
ct = ColumnTransformer([('t2', Normalizer(norm="l1"), select_int)])
df1 = pd.DataFrame({'1': [1, 2, 3], '2': [9, 8, 7]})
df2 = pd.DataFrame({1: [1, 2, 3], 2: [9, 8, 7]})
ct.fit_transform(df1) # OK
ct.fit_transform(df2) # IndexError
```

The only difference between `df1` and `df2` is the type of column index. In my opinion, the results for these dataframes must be similar, but an error is raised for the latter.

As far as I could see, the problem stems from semantic ambiguity as to when to use `iloc`-based indexing vs `loc`-based indexing. In `_get_column_indices` [L382](https://github.com/scikit-learn/scikit-learn/blob/998e8f2068c7d9f43a30d997b63e4875688d0b1a/sklearn/utils/__init__.py#L382) this decision is based on the type of index and not on the type of the array. Whichever criterion is chosen, if it followed consistently in column selectors, the error shall be avoided. Probably.

### Steps/Code to Reproduce

(See above)

### Expected Results

(See above)

### Actual Results

(See above)

### Versions

```shell
Python dependencies:
          pip: 22.0.3
   setuptools: 60.8.1
      sklearn: 1.1.dev0
        numpy: 1.22.2
        scipy: 1.8.0
       Cython: 0.29.27
       pandas: 1.3.5
   matplotlib: 3.5.0
       joblib: 1.1.0
threadpoolctl: 3.1.0

commit b28c5bba66529217ceedd497201a684e5d35b73c (upstream/main, origin/main, origin/HEAD, main)
Author: Thomas J. Fan 
Date:   Tue Feb 15 11:46:54 2022 -0500

    FIX DummyRegressor overriding constant (#22486)
```

COMMENT:
Your solution seems reasonable to me. I think that some users might want `ColumnTransformer` to have this parameter even if they don't use `make_column_selector` or other callables. I would also be fine even with a cautionary note in the documentation as an interim solution.

What bothers me is why `ColumnsTransformer` and `make_column_selector` use different semantics in the first place. Namely, the output of `make_column_selector` is always _names_ (str or int), while the input of `ColumnsTransformer` is either _names_ (str) or _positional_ (int). Afaiac, they were introduced together to be used together.

Notwithstanding your proposed solution, would it make sense to change the behaviour of `make_column_selector` and break the backward compatibility in the future? (if the policy allows, through a deprecation cycle or somehow). I wonder, whether `ColumnsTransformer` is the only "user" of `make_column_selector` or if there are conceivable alternative use cases for `make_column_selector`, in which the lack backward compatibility can break actual code.

Labels: New Feature, API, Needs Decision, module:compose
ColumnTransformer: integer column index in dataframes unexpected behaviour and error (column selectors vs _get_column_indices)

### Describe the bug

Unexpected behaviour when using integer column index in a DataFrame, other than natural ordering [0, 1, ...].

```
select_int = make_column_selector(dtype_include=np.int_)
ct = ColumnTransformer([('t2', Normalizer(norm="l1"), select_int)])
df1 = pd.DataFrame({'1': [1, 2, 3], '2': [9, 8, 7]})
df2 = pd.DataFrame({1: [1, 2, 3], 2: [9, 8, 7]})
ct.fit_transform(df1) # OK
ct.fit_transform(df2) # IndexError
```

The only difference between `df1` and `df2` is the type of column index. In my opinion, the results for these dataframes must be similar, but an error is raised for the latter.

As far as I could see, the problem stems from semantic ambiguity as to when to use `iloc`-based indexing vs `loc`-based indexing. In `_get_column_indices` [L382](https://github.com/scikit-learn/scikit-learn/blob/998e8f2068c7d9f43a30d997b63e4875688d0b1a/sklearn/utils/__init__.py#L382) this decision is based on the type of index and not on the type of the array. Whichever criterion is chosen, if it followed consistently in column selectors, the error shall be avoided. Probably.

### Steps/Code to Reproduce

(See above)

### Expected Results

(See above)

### Actual Results

(See above)

### Versions

```shell
Python dependencies:
          pip: 22.0.3
   setuptools: 60.8.1
      sklearn: 1.1.dev0
        numpy: 1.22.2
        scipy: 1.8.0
       Cython: 0.29.27
       pandas: 1.3.5
   matplotlib: 3.5.0
       joblib: 1.1.0
threadpoolctl: 3.1.0

commit b28c5bba66529217ceedd497201a684e5d35b73c (upstream/main, origin/main, origin/HEAD, main)
Author: Thomas J. Fan 
Date:   Tue Feb 15 11:46:54 2022 -0500

    FIX DummyRegressor overriding constant (#22486)
```

COMMENT:
Okay, I see why changing the default behaviour of `make_column_selector` in the future is even worse than changing it now.

After your proposed solution is implemented, I will be using the parameter `ColumnTransformer(..., use_index_directly=True)` to make the pipeline work with both str and int column _names_. If `make_column_selector` is changed after that, then I will have to remove `use_index_directly=True` from all my code. So it is now or never.

Labels: New Feature, API, Needs Decision, module:compose
ColumnTransformer: integer column index in dataframes unexpected behaviour and error (column selectors vs _get_column_indices)

### Describe the bug

Unexpected behaviour when using integer column index in a DataFrame, other than natural ordering [0, 1, ...].

```
select_int = make_column_selector(dtype_include=np.int_)
ct = ColumnTransformer([('t2', Normalizer(norm="l1"), select_int)])
df1 = pd.DataFrame({'1': [1, 2, 3], '2': [9, 8, 7]})
df2 = pd.DataFrame({1: [1, 2, 3], 2: [9, 8, 7]})
ct.fit_transform(df1) # OK
ct.fit_transform(df2) # IndexError
```

The only difference between `df1` and `df2` is the type of column index. In my opinion, the results for these dataframes must be similar, but an error is raised for the latter.

As far as I could see, the problem stems from semantic ambiguity as to when to use `iloc`-based indexing vs `loc`-based indexing. In `_get_column_indices` [L382](https://github.com/scikit-learn/scikit-learn/blob/998e8f2068c7d9f43a30d997b63e4875688d0b1a/sklearn/utils/__init__.py#L382) this decision is based on the type of index and not on the type of the array. Whichever criterion is chosen, if it followed consistently in column selectors, the error shall be avoided. Probably.

### Steps/Code to Reproduce

(See above)

### Expected Results

(See above)

### Actual Results

(See above)

### Versions

```shell
Python dependencies:
          pip: 22.0.3
   setuptools: 60.8.1
      sklearn: 1.1.dev0
        numpy: 1.22.2
        scipy: 1.8.0
       Cython: 0.29.27
       pandas: 1.3.5
   matplotlib: 3.5.0
       joblib: 1.1.0
threadpoolctl: 3.1.0

commit b28c5bba66529217ceedd497201a684e5d35b73c (upstream/main, origin/main, origin/HEAD, main)
Author: Thomas J. Fan 
Date:   Tue Feb 15 11:46:54 2022 -0500

    FIX DummyRegressor overriding constant (#22486)
```

COMMENT:
> After your proposed solution is implemented, I will be using the parameter ColumnTransformer(..., use_index_directly=True) to make the pipeline work with both str and int column names.

Yup! That is the goal. The other benefit of `use_index_directly` is that it allows for using panda's multiindex: https://github.com/scikit-learn/scikit-learn/issues/13781

Labels: New Feature, API, Needs Decision, module:compose
LinearRegression in User Guide seems to have $w_0=0$

From the documentation of https://scikit-learn.org/stable/modules/linear_model.html#ordinary-least-squares, one could understand that LinearRegression fits a model where the intercept $w_0$ is absent, that is, where $w_0=0$.

To prevent this misinterpretation, I suggest rephrasing (i) "LinearRegression fits a linear model with coefficients $w = (w_1, ..., w_p)" as "LinearRegression fits a linear model with coefficients $w = (w_0, w_1, ..., w_p)" AND (ii) "and will store the coefficients of the linear model in its coef_ member" as "and will store the coefficients of the linear model in its coef_ and intercept_ members"

COMMENT:
Not a core maintainer, but I agree with this assessment and the general direction of the proposed solution

Labels: Documentation, module:linear_model
LinearRegression in User Guide seems to have $w_0=0$

From the documentation of https://scikit-learn.org/stable/modules/linear_model.html#ordinary-least-squares, one could understand that LinearRegression fits a model where the intercept $w_0$ is absent, that is, where $w_0=0$.

To prevent this misinterpretation, I suggest rephrasing (i) "LinearRegression fits a linear model with coefficients $w = (w_1, ..., w_p)" as "LinearRegression fits a linear model with coefficients $w = (w_0, w_1, ..., w_p)" AND (ii) "and will store the coefficients of the linear model in its coef_ member" as "and will store the coefficients of the linear model in its coef_ and intercept_ members"

COMMENT:
Thank you for opening this issue! I agree the documentation can be improved here. I would also remove the `w =` part slightly above in the "1.1. Linear Models" section to avoid defining `w` two different ways so close together:

https://github.com/scikit-learn/scikit-learn/blob/998e8f2068c7d9f43a30d997b63e4875688d0b1a/doc/modules/linear_model.rst?plain=1#L14-L17

@thomasoliveira Are you interested in opening a PR to adjust the documentation?

Labels: Documentation, module:linear_model
LinearRegression in User Guide seems to have $w_0=0$

From the documentation of https://scikit-learn.org/stable/modules/linear_model.html#ordinary-least-squares, one could understand that LinearRegression fits a model where the intercept $w_0$ is absent, that is, where $w_0=0$.

To prevent this misinterpretation, I suggest rephrasing (i) "LinearRegression fits a linear model with coefficients $w = (w_1, ..., w_p)" as "LinearRegression fits a linear model with coefficients $w = (w_0, w_1, ..., w_p)" AND (ii) "and will store the coefficients of the linear model in its coef_ member" as "and will store the coefficients of the linear model in its coef_ and intercept_ members"

COMMENT:
@thomasjpfan , thanks. I have opened the PR #22561 to adjust the documentation.

Labels: Documentation, module:linear_model
LinearRegression in User Guide seems to have $w_0=0$

From the documentation of https://scikit-learn.org/stable/modules/linear_model.html#ordinary-least-squares, one could understand that LinearRegression fits a model where the intercept $w_0$ is absent, that is, where $w_0=0$.

To prevent this misinterpretation, I suggest rephrasing (i) "LinearRegression fits a linear model with coefficients $w = (w_1, ..., w_p)" as "LinearRegression fits a linear model with coefficients $w = (w_0, w_1, ..., w_p)" AND (ii) "and will store the coefficients of the linear model in its coef_ member" as "and will store the coefficients of the linear model in its coef_ and intercept_ members"

COMMENT:
@thomasjpfan @lorentzenchr 
Looks like this issue is stalled and still open. Can I take it and correct the documentation as mentioned above?

Labels: Documentation, module:linear_model
LinearRegression in User Guide seems to have $w_0=0$

From the documentation of https://scikit-learn.org/stable/modules/linear_model.html#ordinary-least-squares, one could understand that LinearRegression fits a model where the intercept $w_0$ is absent, that is, where $w_0=0$.

To prevent this misinterpretation, I suggest rephrasing (i) "LinearRegression fits a linear model with coefficients $w = (w_1, ..., w_p)" as "LinearRegression fits a linear model with coefficients $w = (w_0, w_1, ..., w_p)" AND (ii) "and will store the coefficients of the linear model in its coef_ member" as "and will store the coefficients of the linear model in its coef_ and intercept_ members"

COMMENT:
/take

Labels: Documentation, module:linear_model