Talip7/scikit-learn-issues-multilabel huggingface dataset:
Add sample_weight support to binning in HGBT

Use `sample_weight` in the binning of `HistGradientBoostingClassifier` and `HistGradientBoostingRegressor`, or allow it via an option.

Currently, sample weights are ignored in the `_BinMapper`.

Some more context and history summarized by @NicolasHug [here](https://github.com/scikit-learn/scikit-learn/issues/24728#issuecomment-1288084267):
> I agree that it would make sense to support SW in the binner (although this is nuanced, see note below). Reading back the original PR, this was discussed extensively:
> 
>   - LightGBM implem doesn't take weights into account when binning [ENH Support sample weights in HGBT #14696 (comment)](https://github.com/scikit-learn/scikit-learn/pull/14696#issuecomment-545138460)
> 
>   - I had a proposal to support SW in the binning: [ENH Support sample weights in HGBT #14696 (comment)](https://github.com/scikit-learn/scikit-learn/pull/14696#issuecomment-556074060). Olivier and Andy seemed to be happy with it [ENH Support sample weights in HGBT #14696 (comment)](https://github.com/scikit-learn/scikit-learn/pull/14696#issuecomment-581608597); there were some concerns from Adrin. To unblock the rest of the work, we were all happy to just not implement SW support in the Binner and leave that as potential future work.
> 
> 
> The estimators were still experimental at the time so we had more flexibility. Now that they're stable, bringing SW support in the Binner might require BC mitigations.
> 
> Also as a side note, [#15657 (comment)](https://github.com/scikit-learn/scikit-learn/issues/15657#issuecomment-556065553) may be relevant here: it's not super clear to me how we should handle SW in an estimator that performs some sort of subsambling during the training process (as is the case here during Binning).

COMMENT:
More of a "for my education" question but also maybe useful to think about: is there a way to determine a "correct" or "incorrect" set of bin edges for a dataset?

I think there is no way to define what a "correct" set of bin edges is. There are smarter and less smart ways to choose the edges, and some sets of edges will lead to improved performance than others. But I find it hard to come up with a way to define that a set of edges is "incorrect".

When you compute how many samples are between two edges you do need to take into account the weights, at least I think it is "incorrect" not to.

The point being: maybe it is simpler to ignore the weights when determining the edges but take them into account when calculating how many samples are in each bin.

Labels: New Feature, Needs Decision, module:ensemble
Add sample_weight support to binning in HGBT

Use `sample_weight` in the binning of `HistGradientBoostingClassifier` and `HistGradientBoostingRegressor`, or allow it via an option.

Currently, sample weights are ignored in the `_BinMapper`.

Some more context and history summarized by @NicolasHug [here](https://github.com/scikit-learn/scikit-learn/issues/24728#issuecomment-1288084267):
> I agree that it would make sense to support SW in the binner (although this is nuanced, see note below). Reading back the original PR, this was discussed extensively:
> 
>   - LightGBM implem doesn't take weights into account when binning [ENH Support sample weights in HGBT #14696 (comment)](https://github.com/scikit-learn/scikit-learn/pull/14696#issuecomment-545138460)
> 
>   - I had a proposal to support SW in the binning: [ENH Support sample weights in HGBT #14696 (comment)](https://github.com/scikit-learn/scikit-learn/pull/14696#issuecomment-556074060). Olivier and Andy seemed to be happy with it [ENH Support sample weights in HGBT #14696 (comment)](https://github.com/scikit-learn/scikit-learn/pull/14696#issuecomment-581608597); there were some concerns from Adrin. To unblock the rest of the work, we were all happy to just not implement SW support in the Binner and leave that as potential future work.
> 
> 
> The estimators were still experimental at the time so we had more flexibility. Now that they're stable, bringing SW support in the Binner might require BC mitigations.
> 
> Also as a side note, [#15657 (comment)](https://github.com/scikit-learn/scikit-learn/issues/15657#issuecomment-556065553) may be relevant here: it's not super clear to me how we should handle SW in an estimator that performs some sort of subsambling during the training process (as is the case here during Binning).

COMMENT:
> The point being: maybe it is simpler to ignore the weights when determining the edges but take them into account when calculating how many samples are in each bin.

I don't follow.

The general point is more that if you interpret weights as frequency weights (which is allowed) then you might very well want to take them into account for calculating quantiles as bin thresholds such that about equal sum of weights ends up in each bin.

Labels: New Feature, Needs Decision, module:ensemble
Add sample_weight support to binning in HGBT

Use `sample_weight` in the binning of `HistGradientBoostingClassifier` and `HistGradientBoostingRegressor`, or allow it via an option.

Currently, sample weights are ignored in the `_BinMapper`.

Some more context and history summarized by @NicolasHug [here](https://github.com/scikit-learn/scikit-learn/issues/24728#issuecomment-1288084267):
> I agree that it would make sense to support SW in the binner (although this is nuanced, see note below). Reading back the original PR, this was discussed extensively:
> 
>   - LightGBM implem doesn't take weights into account when binning [ENH Support sample weights in HGBT #14696 (comment)](https://github.com/scikit-learn/scikit-learn/pull/14696#issuecomment-545138460)
> 
>   - I had a proposal to support SW in the binning: [ENH Support sample weights in HGBT #14696 (comment)](https://github.com/scikit-learn/scikit-learn/pull/14696#issuecomment-556074060). Olivier and Andy seemed to be happy with it [ENH Support sample weights in HGBT #14696 (comment)](https://github.com/scikit-learn/scikit-learn/pull/14696#issuecomment-581608597); there were some concerns from Adrin. To unblock the rest of the work, we were all happy to just not implement SW support in the Binner and leave that as potential future work.
> 
> 
> The estimators were still experimental at the time so we had more flexibility. Now that they're stable, bringing SW support in the Binner might require BC mitigations.
> 
> Also as a side note, [#15657 (comment)](https://github.com/scikit-learn/scikit-learn/issues/15657#issuecomment-556065553) may be relevant here: it's not super clear to me how we should handle SW in an estimator that performs some sort of subsambling during the training process (as is the case here during Binning).

COMMENT:
> The general point is more that if you interpret weights as frequency weights (which is allowed) then you might very well want to take them into account for calculating quantiles as bin thresholds such that about equal sum of weights ends up in each bin.

I agree that it is smart to do this. What I was wondering is if it is too hard/it would be easier to ignore the weights when computing the edges and only taking them into account when computing the contents of each bin. Hence the question about whether it is incorrect/wrong to ignore the weights when computing the edges or (potentially) just leads to lower performance?

For example using equally spaced bins between feature minimum and feature maximum isn't wrong, but likely to lead to lower performance than the quantiles based binning.

Labels: New Feature, Needs Decision, module:ensemble
Add sample_weight support to binning in HGBT

Use `sample_weight` in the binning of `HistGradientBoostingClassifier` and `HistGradientBoostingRegressor`, or allow it via an option.

Currently, sample weights are ignored in the `_BinMapper`.

Some more context and history summarized by @NicolasHug [here](https://github.com/scikit-learn/scikit-learn/issues/24728#issuecomment-1288084267):
> I agree that it would make sense to support SW in the binner (although this is nuanced, see note below). Reading back the original PR, this was discussed extensively:
> 
>   - LightGBM implem doesn't take weights into account when binning [ENH Support sample weights in HGBT #14696 (comment)](https://github.com/scikit-learn/scikit-learn/pull/14696#issuecomment-545138460)
> 
>   - I had a proposal to support SW in the binning: [ENH Support sample weights in HGBT #14696 (comment)](https://github.com/scikit-learn/scikit-learn/pull/14696#issuecomment-556074060). Olivier and Andy seemed to be happy with it [ENH Support sample weights in HGBT #14696 (comment)](https://github.com/scikit-learn/scikit-learn/pull/14696#issuecomment-581608597); there were some concerns from Adrin. To unblock the rest of the work, we were all happy to just not implement SW support in the Binner and leave that as potential future work.
> 
> 
> The estimators were still experimental at the time so we had more flexibility. Now that they're stable, bringing SW support in the Binner might require BC mitigations.
> 
> Also as a side note, [#15657 (comment)](https://github.com/scikit-learn/scikit-learn/issues/15657#issuecomment-556065553) may be relevant here: it's not super clear to me how we should handle SW in an estimator that performs some sort of subsambling during the training process (as is the case here during Binning).

COMMENT:
The `_BinMapper` already uses quantiles. So it would *simply* be a matter of using weighted quantiles instead.

I think, we also should expose the `subsample` parameter of the bin mapper to allow users more control.

Labels: New Feature, Needs Decision, module:ensemble
Add sample_weight support to binning in HGBT

Use `sample_weight` in the binning of `HistGradientBoostingClassifier` and `HistGradientBoostingRegressor`, or allow it via an option.

Currently, sample weights are ignored in the `_BinMapper`.

Some more context and history summarized by @NicolasHug [here](https://github.com/scikit-learn/scikit-learn/issues/24728#issuecomment-1288084267):
> I agree that it would make sense to support SW in the binner (although this is nuanced, see note below). Reading back the original PR, this was discussed extensively:
> 
>   - LightGBM implem doesn't take weights into account when binning [ENH Support sample weights in HGBT #14696 (comment)](https://github.com/scikit-learn/scikit-learn/pull/14696#issuecomment-545138460)
> 
>   - I had a proposal to support SW in the binning: [ENH Support sample weights in HGBT #14696 (comment)](https://github.com/scikit-learn/scikit-learn/pull/14696#issuecomment-556074060). Olivier and Andy seemed to be happy with it [ENH Support sample weights in HGBT #14696 (comment)](https://github.com/scikit-learn/scikit-learn/pull/14696#issuecomment-581608597); there were some concerns from Adrin. To unblock the rest of the work, we were all happy to just not implement SW support in the Binner and leave that as potential future work.
> 
> 
> The estimators were still experimental at the time so we had more flexibility. Now that they're stable, bringing SW support in the Binner might require BC mitigations.
> 
> Also as a side note, [#15657 (comment)](https://github.com/scikit-learn/scikit-learn/issues/15657#issuecomment-556065553) may be relevant here: it's not super clear to me how we should handle SW in an estimator that performs some sort of subsambling during the training process (as is the case here during Binning).

COMMENT:
> The `_BinMapper` already uses quantiles. So it would _simply_ be a matter of using weighted quantiles instead.

IIRC we use the `percentile` function from numpy which doesn't support weights. I assume the reason they don't support it is because it is difficult to implement an efficient algorithm, but that is speculation.

Labels: New Feature, Needs Decision, module:ensemble
Add sample_weight support to binning in HGBT

Use `sample_weight` in the binning of `HistGradientBoostingClassifier` and `HistGradientBoostingRegressor`, or allow it via an option.

Currently, sample weights are ignored in the `_BinMapper`.

Some more context and history summarized by @NicolasHug [here](https://github.com/scikit-learn/scikit-learn/issues/24728#issuecomment-1288084267):
> I agree that it would make sense to support SW in the binner (although this is nuanced, see note below). Reading back the original PR, this was discussed extensively:
> 
>   - LightGBM implem doesn't take weights into account when binning [ENH Support sample weights in HGBT #14696 (comment)](https://github.com/scikit-learn/scikit-learn/pull/14696#issuecomment-545138460)
> 
>   - I had a proposal to support SW in the binning: [ENH Support sample weights in HGBT #14696 (comment)](https://github.com/scikit-learn/scikit-learn/pull/14696#issuecomment-556074060). Olivier and Andy seemed to be happy with it [ENH Support sample weights in HGBT #14696 (comment)](https://github.com/scikit-learn/scikit-learn/pull/14696#issuecomment-581608597); there were some concerns from Adrin. To unblock the rest of the work, we were all happy to just not implement SW support in the Binner and leave that as potential future work.
> 
> 
> The estimators were still experimental at the time so we had more flexibility. Now that they're stable, bringing SW support in the Binner might require BC mitigations.
> 
> Also as a side note, [#15657 (comment)](https://github.com/scikit-learn/scikit-learn/issues/15657#issuecomment-556065553) may be relevant here: it's not super clear to me how we should handle SW in an estimator that performs some sort of subsambling during the training process (as is the case here during Binning).

COMMENT:
We have our own (quite simple) weighted percentile function in utils.

Labels: New Feature, Needs Decision, module:ensemble
Add sample_weight support to binning in HGBT

Use `sample_weight` in the binning of `HistGradientBoostingClassifier` and `HistGradientBoostingRegressor`, or allow it via an option.

Currently, sample weights are ignored in the `_BinMapper`.

Some more context and history summarized by @NicolasHug [here](https://github.com/scikit-learn/scikit-learn/issues/24728#issuecomment-1288084267):
> I agree that it would make sense to support SW in the binner (although this is nuanced, see note below). Reading back the original PR, this was discussed extensively:
> 
>   - LightGBM implem doesn't take weights into account when binning [ENH Support sample weights in HGBT #14696 (comment)](https://github.com/scikit-learn/scikit-learn/pull/14696#issuecomment-545138460)
> 
>   - I had a proposal to support SW in the binning: [ENH Support sample weights in HGBT #14696 (comment)](https://github.com/scikit-learn/scikit-learn/pull/14696#issuecomment-556074060). Olivier and Andy seemed to be happy with it [ENH Support sample weights in HGBT #14696 (comment)](https://github.com/scikit-learn/scikit-learn/pull/14696#issuecomment-581608597); there were some concerns from Adrin. To unblock the rest of the work, we were all happy to just not implement SW support in the Binner and leave that as potential future work.
> 
> 
> The estimators were still experimental at the time so we had more flexibility. Now that they're stable, bringing SW support in the Binner might require BC mitigations.
> 
> Also as a side note, [#15657 (comment)](https://github.com/scikit-learn/scikit-learn/issues/15657#issuecomment-556065553) may be relevant here: it's not super clear to me how we should handle SW in an estimator that performs some sort of subsambling during the training process (as is the case here during Binning).

COMMENT:
Note that we the training set is large, we subsample it and find the bin thresholds only on those on the subsample. In that case we should probably use the weights when subsampling only as proposed in https://github.com/scikit-learn/scikit-learn/pull/14696#issuecomment-556074060.

However, subsampling does not happen if the number of data points is smaller than `int(2e5)`. In that case we could make the `_find_binning_thresholds` respect `sample_weight` by calling our own `_weighted_percentile` instead of `np.percentile` internally. The case where `len(distinct_values) <= max_bins` might be tricky to get right though. Maybe we should no longer special-case it and instead call `np.unique` on the thresholds and expect that it can be lower than 256.

Labels: New Feature, Needs Decision, module:ensemble
Add baseline estimator to HGBT

### Describe the workflow you want to enable

I would like to specify a baseline estimator like in `GradientBoostingRegressor(init=MyCoolBaselineEstimator_Maybe_a_linear_model)`.

### Describe your proposed solution

Add a parameter `baseline` (or if has to be the same, then `init`) to `HistGradientBoostingRegressor` and `HistGradientBoostingClassifier`.

### Describe alternatives you've considered, if relevant

The `StackingClassifier` does not do the same thing.

### Additional context

In particular a linear model as baseline could
1. Improve fit time
2. Allow for non-constant extrapolation on important features, quite in contrast to trees.

COMMENT:
Hello @lorentzenchr , I have tried to add the baseline parameter in the `HistGradientBoostingRegressor` and I am almost there. But I want to test so how do I test it? Please guide me.(I am new in open source so I have no idea)

Labels: New Feature, module:ensemble, Needs Decision - Include Feature
Add baseline estimator to HGBT

### Describe the workflow you want to enable

I would like to specify a baseline estimator like in `GradientBoostingRegressor(init=MyCoolBaselineEstimator_Maybe_a_linear_model)`.

### Describe your proposed solution

Add a parameter `baseline` (or if has to be the same, then `init`) to `HistGradientBoostingRegressor` and `HistGradientBoostingClassifier`.

### Describe alternatives you've considered, if relevant

The `StackingClassifier` does not do the same thing.

### Additional context

In particular a linear model as baseline could
1. Improve fit time
2. Allow for non-constant extrapolation on important features, quite in contrast to trees.

COMMENT:
Hello @lorentzenchr ,Thanks for reporting the issue. I am willing to add the `baseline` estimator in `HistGradientBoostingClassifier` and `HistGradientBoostingRegressor`. 
I have started working in this.

Labels: New Feature, module:ensemble, Needs Decision - Include Feature
Add baseline estimator to HGBT

### Describe the workflow you want to enable

I would like to specify a baseline estimator like in `GradientBoostingRegressor(init=MyCoolBaselineEstimator_Maybe_a_linear_model)`.

### Describe your proposed solution

Add a parameter `baseline` (or if has to be the same, then `init`) to `HistGradientBoostingRegressor` and `HistGradientBoostingClassifier`.

### Describe alternatives you've considered, if relevant

The `StackingClassifier` does not do the same thing.

### Additional context

In particular a linear model as baseline could
1. Improve fit time
2. Allow for non-constant extrapolation on important features, quite in contrast to trees.

COMMENT:
Could you explain a bit more what you mean with a baseline model? Is it the model that gets boosted (by default a decision tree)?

Labels: New Feature, module:ensemble, Needs Decision - Include Feature
Add baseline estimator to HGBT

### Describe the workflow you want to enable

I would like to specify a baseline estimator like in `GradientBoostingRegressor(init=MyCoolBaselineEstimator_Maybe_a_linear_model)`.

### Describe your proposed solution

Add a parameter `baseline` (or if has to be the same, then `init`) to `HistGradientBoostingRegressor` and `HistGradientBoostingClassifier`.

### Describe alternatives you've considered, if relevant

The `StackingClassifier` does not do the same thing.

### Additional context

In particular a linear model as baseline could
1. Improve fit time
2. Allow for non-constant extrapolation on important features, quite in contrast to trees.

COMMENT:
@betatim Same as [`GradientBoostingRegressor(init=...)`](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingRegressor.html#sklearn.ensemble.GradientBoostingRegressor), i.e. the starting point from where to begin boosting, i.e. it is still tree boosting.

Labels: New Feature, module:ensemble, Needs Decision - Include Feature
Add baseline estimator to HGBT

### Describe the workflow you want to enable

I would like to specify a baseline estimator like in `GradientBoostingRegressor(init=MyCoolBaselineEstimator_Maybe_a_linear_model)`.

### Describe your proposed solution

Add a parameter `baseline` (or if has to be the same, then `init`) to `HistGradientBoostingRegressor` and `HistGradientBoostingClassifier`.

### Describe alternatives you've considered, if relevant

The `StackingClassifier` does not do the same thing.

### Additional context

In particular a linear model as baseline could
1. Improve fit time
2. Allow for non-constant extrapolation on important features, quite in contrast to trees.

COMMENT:
@GyanPrakashkushwaha This is marked as "Needs Decision" in the labels.

Labels: New Feature, module:ensemble, Needs Decision - Include Feature
Add baseline estimator to HGBT

### Describe the workflow you want to enable

I would like to specify a baseline estimator like in `GradientBoostingRegressor(init=MyCoolBaselineEstimator_Maybe_a_linear_model)`.

### Describe your proposed solution

Add a parameter `baseline` (or if has to be the same, then `init`) to `HistGradientBoostingRegressor` and `HistGradientBoostingClassifier`.

### Describe alternatives you've considered, if relevant

The `StackingClassifier` does not do the same thing.

### Additional context

In particular a linear model as baseline could
1. Improve fit time
2. Allow for non-constant extrapolation on important features, quite in contrast to trees.

COMMENT:
@lorentzenchr I am trying to add the `init` parameter.

Labels: New Feature, module:ensemble, Needs Decision - Include Feature
AttributeError: 'LogisticRegression' object has no attribute 'feature_names_in_'

### Describe the bug

I created two model successfully (one being Decision Tree, and the other Logistic Regression). Those were exported as pickle `.pkl` files. Because I used one-hot encoding in both models and I have a lot of categories I needed to get name of features. But weird thing is that I use `feature_names_in_` for both of them and for some reason I get this error  `AttributeError: 'LogisticRegression' object has no attribute 'feature_names_in_'` . On the other hand this works fine for Decision Tree. I also checked the documentation for [Logistic Regression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) to double check if this even exists and it does. Also, I have checked version of `sklearn` by running `sklearn.__version__` and indeed I have confirmed latest `1.3.0` version. 

### Steps/Code to Reproduce

```
model = pickle.load(open(model_path, "rb"))
print(f"Version: {sklearn.__version__}")
ordered_columns = model.feature_names_in_
```

### Expected Results

No error is thrown

### Actual Results

```
    prediction = create_prediction(list(model_input_items), "./path_to_model.pkl")
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 create_prediction
    ordered_columns = model.feature_names_in_
                      ^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'LogisticRegression' object has no attribute 'feature_names_in_'
```

### Versions

```shell
Version: 1.3.0
```

COMMENT:
Thanks for reporting the issue. Could you please also include a [minimal and reproducible example](http://sscce.org/) for us to better diagnose the problem?

Ideally we should be able to copy/paste your code to reproduce the issue.

In your case, please include a minimal example including a toy training data that results in this issue.

Labels: Bug, Needs Reproducible Code