Talip7/scikit-learn-issues-multilabel huggingface dataset:
Allow X in func or transformer in TransformedTargetRegressor

### Describe the workflow you want to enable

I would like to enable target transformation based on X in data. 

This will come in handy when predicting a quantity that has a natural dependency on a feature. E.g. target value is summed salary of a household and the function is to divide by number of adults in household. This creates a natural scaling of the target but requires features in training data to accomplish. 

In this case the proposed solution would look something like this

```
def func(X, y):
   return y / x[1]

def inverse(X, y):
   return y * X[1]
```

### Describe your proposed solution

Check signature on func and transformer calls to see if it requires X as parameter and if so add it to function calls similar to https://github.com/scikit-learn/scikit-learn/pull/21935

Here is a hacky solution I made myself to get it to work

```
class TargetValueTransformer(TransformedTargetRegressor):
    def __init__(
        self,
        regressor=None,
        *,
        transformer=None,
        func=None,
        inverse_func=None,
        check_inverse=False,
    ):        
        super(TargetValueTransformer, self).__init__(regressor,        
        transformer=transformer,
        func=func,
        inverse_func=inverse_func,
        check_inverse=check_inverse,
)
      
    def _fit_transformer(self, y):
        pass
        
    def fit(self, X, y):
        self.regressor.fit(X, self.func(X, y))
        return self
    
    def predict(self, X):
        return self.inverse_func(X, self.regressor.predict(X))
```
### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
@GLevV @glemaitre thanks for checking out this issue. If I understand your suggestion correctly I don't think that will work. Perhaps the dosctring for the class describes the problem the best:

```
    The computation during :meth:`fit` is::
        regressor.fit(X, func(y))
    or::
        regressor.fit(X, transformer.transform(y))
    The computation during :meth:`predict` is::
        inverse_func(regressor.predict(X))
```

What I want to be possible is to also allow `X` in the func/transformer e.g:

```
    The computation during :meth:`fit` is::
        regressor.fit(X, func(X=X, y=y))
    or::
        regressor.fit(X, transformer.transform(y))
    The computation during :meth:`predict` is::
        inverse_func(X=X, y=regressor.predict(X))
```

Labels: New Feature, module:preprocessing, Needs Decision - Include Feature
Allow X in func or transformer in TransformedTargetRegressor

### Describe the workflow you want to enable

I would like to enable target transformation based on X in data. 

This will come in handy when predicting a quantity that has a natural dependency on a feature. E.g. target value is summed salary of a household and the function is to divide by number of adults in household. This creates a natural scaling of the target but requires features in training data to accomplish. 

In this case the proposed solution would look something like this

```
def func(X, y):
   return y / x[1]

def inverse(X, y):
   return y * X[1]
```

### Describe your proposed solution

Check signature on func and transformer calls to see if it requires X as parameter and if so add it to function calls similar to https://github.com/scikit-learn/scikit-learn/pull/21935

Here is a hacky solution I made myself to get it to work

```
class TargetValueTransformer(TransformedTargetRegressor):
    def __init__(
        self,
        regressor=None,
        *,
        transformer=None,
        func=None,
        inverse_func=None,
        check_inverse=False,
    ):        
        super(TargetValueTransformer, self).__init__(regressor,        
        transformer=transformer,
        func=func,
        inverse_func=inverse_func,
        check_inverse=check_inverse,
)
      
    def _fit_transformer(self, y):
        pass
        
    def fit(self, X, y):
        self.regressor.fit(X, self.func(X, y))
        return self
    
    def predict(self, X):
        return self.inverse_func(X, self.regressor.predict(X))
```
### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
True, we don't forward `X` internally. This makes me think about the Impact Encoder: https://github.com/scikit-learn/scikit-learn/pull/17323

I recall some API issues but I don't know if they would apply here.

Labels: New Feature, module:preprocessing, Needs Decision - Include Feature
Allow X in func or transformer in TransformedTargetRegressor

### Describe the workflow you want to enable

I would like to enable target transformation based on X in data. 

This will come in handy when predicting a quantity that has a natural dependency on a feature. E.g. target value is summed salary of a household and the function is to divide by number of adults in household. This creates a natural scaling of the target but requires features in training data to accomplish. 

In this case the proposed solution would look something like this

```
def func(X, y):
   return y / x[1]

def inverse(X, y):
   return y * X[1]
```

### Describe your proposed solution

Check signature on func and transformer calls to see if it requires X as parameter and if so add it to function calls similar to https://github.com/scikit-learn/scikit-learn/pull/21935

Here is a hacky solution I made myself to get it to work

```
class TargetValueTransformer(TransformedTargetRegressor):
    def __init__(
        self,
        regressor=None,
        *,
        transformer=None,
        func=None,
        inverse_func=None,
        check_inverse=False,
    ):        
        super(TargetValueTransformer, self).__init__(regressor,        
        transformer=transformer,
        func=func,
        inverse_func=inverse_func,
        check_inverse=check_inverse,
)
      
    def _fit_transformer(self, y):
        pass
        
    def fit(self, X, y):
        self.regressor.fit(X, self.func(X, y))
        return self
    
    def predict(self, X):
        return self.inverse_func(X, self.regressor.predict(X))
```
### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Is there some reason we should not allow this suggestion? I can give a stab at creating a PR, from a first glance it seems straightforward. My idea would basically be to just inspect the input function/transformer and if it requires X it will be used otherwise we have default to current functionality.

Labels: New Feature, module:preprocessing, Needs Decision - Include Feature
Allow X in func or transformer in TransformedTargetRegressor

### Describe the workflow you want to enable

I would like to enable target transformation based on X in data. 

This will come in handy when predicting a quantity that has a natural dependency on a feature. E.g. target value is summed salary of a household and the function is to divide by number of adults in household. This creates a natural scaling of the target but requires features in training data to accomplish. 

In this case the proposed solution would look something like this

```
def func(X, y):
   return y / x[1]

def inverse(X, y):
   return y * X[1]
```

### Describe your proposed solution

Check signature on func and transformer calls to see if it requires X as parameter and if so add it to function calls similar to https://github.com/scikit-learn/scikit-learn/pull/21935

Here is a hacky solution I made myself to get it to work

```
class TargetValueTransformer(TransformedTargetRegressor):
    def __init__(
        self,
        regressor=None,
        *,
        transformer=None,
        func=None,
        inverse_func=None,
        check_inverse=False,
    ):        
        super(TargetValueTransformer, self).__init__(regressor,        
        transformer=transformer,
        func=func,
        inverse_func=inverse_func,
        check_inverse=check_inverse,
)
      
    def _fit_transformer(self, y):
        pass
        
    def fit(self, X, y):
        self.regressor.fit(X, self.func(X, y))
        return self
    
    def predict(self, X):
        return self.inverse_func(X, self.regressor.predict(X))
```
### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
During the triaging meeting, we decided that this still needs a compelling use case. If one needs to transform the target, they can divide `y` before passing it into the estimator.

For me, I prefer not to add more complexity to `TransformedTargetRegressor` unless there is compelling use case. As shown in https://github.com/scikit-learn/scikit-learn/issues/24199#issue-1343035828, one can subclass and pass in `X` without too much difficulty. We do have [an example](https://scikit-learn.org/stable/auto_examples/linear_model/plot_poisson_regression_non_normal_loss.html#the-french-motor-third-party-liability-claims-dataset) that computes a new target based on `claims` (y) and `exposure` (in `X`). If `TransformedTargetRegressor` also passed in `X`, I would not use the feature in the example, because of the additional complexity.

Labels: New Feature, module:preprocessing, Needs Decision - Include Feature
Allow X in func or transformer in TransformedTargetRegressor

### Describe the workflow you want to enable

I would like to enable target transformation based on X in data. 

This will come in handy when predicting a quantity that has a natural dependency on a feature. E.g. target value is summed salary of a household and the function is to divide by number of adults in household. This creates a natural scaling of the target but requires features in training data to accomplish. 

In this case the proposed solution would look something like this

```
def func(X, y):
   return y / x[1]

def inverse(X, y):
   return y * X[1]
```

### Describe your proposed solution

Check signature on func and transformer calls to see if it requires X as parameter and if so add it to function calls similar to https://github.com/scikit-learn/scikit-learn/pull/21935

Here is a hacky solution I made myself to get it to work

```
class TargetValueTransformer(TransformedTargetRegressor):
    def __init__(
        self,
        regressor=None,
        *,
        transformer=None,
        func=None,
        inverse_func=None,
        check_inverse=False,
    ):        
        super(TargetValueTransformer, self).__init__(regressor,        
        transformer=transformer,
        func=func,
        inverse_func=inverse_func,
        check_inverse=check_inverse,
)
      
    def _fit_transformer(self, y):
        pass
        
    def fit(self, X, y):
        self.regressor.fit(X, self.func(X, y))
        return self
    
    def predict(self, X):
        return self.inverse_func(X, self.regressor.predict(X))
```
### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
I would like to see this feature implemented to handle X-dependent y transformations in a uniform manner in the sklearn pipeline.

Labels: New Feature, module:preprocessing, Needs Decision - Include Feature
Allow X in func or transformer in TransformedTargetRegressor

### Describe the workflow you want to enable

I would like to enable target transformation based on X in data. 

This will come in handy when predicting a quantity that has a natural dependency on a feature. E.g. target value is summed salary of a household and the function is to divide by number of adults in household. This creates a natural scaling of the target but requires features in training data to accomplish. 

In this case the proposed solution would look something like this

```
def func(X, y):
   return y / x[1]

def inverse(X, y):
   return y * X[1]
```

### Describe your proposed solution

Check signature on func and transformer calls to see if it requires X as parameter and if so add it to function calls similar to https://github.com/scikit-learn/scikit-learn/pull/21935

Here is a hacky solution I made myself to get it to work

```
class TargetValueTransformer(TransformedTargetRegressor):
    def __init__(
        self,
        regressor=None,
        *,
        transformer=None,
        func=None,
        inverse_func=None,
        check_inverse=False,
    ):        
        super(TargetValueTransformer, self).__init__(regressor,        
        transformer=transformer,
        func=func,
        inverse_func=inverse_func,
        check_inverse=check_inverse,
)
      
    def _fit_transformer(self, y):
        pass
        
    def fit(self, X, y):
        self.regressor.fit(X, self.func(X, y))
        return self
    
    def predict(self, X):
        return self.inverse_func(X, self.regressor.predict(X))
```
### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Agreed, this would be useful

Labels: New Feature, module:preprocessing, Needs Decision - Include Feature
Allow X in func or transformer in TransformedTargetRegressor

### Describe the workflow you want to enable

I would like to enable target transformation based on X in data. 

This will come in handy when predicting a quantity that has a natural dependency on a feature. E.g. target value is summed salary of a household and the function is to divide by number of adults in household. This creates a natural scaling of the target but requires features in training data to accomplish. 

In this case the proposed solution would look something like this

```
def func(X, y):
   return y / x[1]

def inverse(X, y):
   return y * X[1]
```

### Describe your proposed solution

Check signature on func and transformer calls to see if it requires X as parameter and if so add it to function calls similar to https://github.com/scikit-learn/scikit-learn/pull/21935

Here is a hacky solution I made myself to get it to work

```
class TargetValueTransformer(TransformedTargetRegressor):
    def __init__(
        self,
        regressor=None,
        *,
        transformer=None,
        func=None,
        inverse_func=None,
        check_inverse=False,
    ):        
        super(TargetValueTransformer, self).__init__(regressor,        
        transformer=transformer,
        func=func,
        inverse_func=inverse_func,
        check_inverse=check_inverse,
)
      
    def _fit_transformer(self, y):
        pass
        
    def fit(self, X, y):
        self.regressor.fit(X, self.func(X, y))
        return self
    
    def predict(self, X):
        return self.inverse_func(X, self.regressor.predict(X))
```
### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
To add a potential compelling use case:

I want to predict the density of a mixture. Knowing the individual densities of the components X, I can already calculate the ideal densities y by creating a simple linear function y = f(X). This means, my model only needs to predict the difference between the ideal and real densities, corresponding to volume contraction/expansion effects. So far, I was fine with calculating this difference before passing it to the sklearn pipeline. But now I wanted to combine both steps into one TransformedTargetRegressor to directly calculate the real density. Furthermore, I can then export it with pickle and load it in a different program later, in which I already utilized other TransformedTargetRegressors with a more simple transformation (like subtracting a fixed value from all targets y).

The 'hacky' workaround of @mattiasAngqvist works, but in this case, the X-dependent target transformation would have come in handy.

Labels: New Feature, module:preprocessing, Needs Decision - Include Feature
sklearn implementation of the square-root LASSO

### Describe the workflow you want to enable

The unconstrained LASSO is currently implemented in `lasso_path`. However, the parameter for unconstrained LASSO is sensitive to the measurement noise. Instead, the square-root LASSO has objective norm(Ax-b) + alpha * norm(x, 1) and its parameter, alpha, is insensitive to the noise level. I would like an `sklearn` interface (with `fit`, `predict`, etc.) coupled with a `lasso_path`-like function, `srlasso_path`, which solves the square-root LASSO optimization program. 

### Describe your proposed solution

I propose an `sklearn`-friendly wrapper around the `cvxpy` or `cvxopt` interface. More ambitiously, I propose an implementation based on [1] that implements a fast (accelerated?) proximal gradient method for solving square-root LASSO.

[1] "On fast convergence of proximal algorithms for SQRT-LASSO optimization: Don’t worry about its nonsmooth loss function" by Li, Xinguo,  Jiang, Haoming, et al. (2020) PMLR. 

### Describe alternatives you've considered, if relevant

Currently, `statsmodels` offers a square-root LASSO solver implemented via the Python package `cvxopt` - this approach is slow. Using `cvxpy` with another interior point solver like MOSEK offers only 2x speed-up (it is snails compared to `lasso_path`). 

Separately, if the problem is appropriately regularized to avoid over-fitting, some work has shown that it's possible to use accelerated proximal gradient descent to achieve convergence. However, the paper offers an algorithm with several critical implementation details unprovided. See [1] above.

### Additional context

_No response_

COMMENT:
Note that the characterization of srLASSO as a second-order conic programming (SOCP) problem is given in [2]

[2] Belloni, A., Chernozhukov, V. and Wang, L. Square-Root LASSO: Pivotal recovery of sparse signals via conic programming https://arxiv.org/pdf/1009.5689.pdf

Labels: New Feature, module:linear_model, Needs Triage
sklearn implementation of the square-root LASSO

### Describe the workflow you want to enable

The unconstrained LASSO is currently implemented in `lasso_path`. However, the parameter for unconstrained LASSO is sensitive to the measurement noise. Instead, the square-root LASSO has objective norm(Ax-b) + alpha * norm(x, 1) and its parameter, alpha, is insensitive to the noise level. I would like an `sklearn` interface (with `fit`, `predict`, etc.) coupled with a `lasso_path`-like function, `srlasso_path`, which solves the square-root LASSO optimization program. 

### Describe your proposed solution

I propose an `sklearn`-friendly wrapper around the `cvxpy` or `cvxopt` interface. More ambitiously, I propose an implementation based on [1] that implements a fast (accelerated?) proximal gradient method for solving square-root LASSO.

[1] "On fast convergence of proximal algorithms for SQRT-LASSO optimization: Don’t worry about its nonsmooth loss function" by Li, Xinguo,  Jiang, Haoming, et al. (2020) PMLR. 

### Describe alternatives you've considered, if relevant

Currently, `statsmodels` offers a square-root LASSO solver implemented via the Python package `cvxopt` - this approach is slow. Using `cvxpy` with another interior point solver like MOSEK offers only 2x speed-up (it is snails compared to `lasso_path`). 

Separately, if the problem is appropriately regularized to avoid over-fitting, some work has shown that it's possible to use accelerated proximal gradient descent to achieve convergence. However, the paper offers an algorithm with several critical implementation details unprovided. See [1] above.

### Additional context

_No response_

COMMENT:
@asberk Thanks for proposing this new estimator.
A few remarks: Without penalty, taking the square root of the sum of squares makes no difference (in minimization).
Then it is clear, that only the (effect of) the penalty changes. While the theoretical arguments of the linked paper seem valid, in practice, one has to choose a penalty strength (alpha). But the formulae for the penalty strength in the linked paper always contains "some constant c>0".

Conclusion: I'm not convinced that the added complexity justifies the benefits in applications.

@agramfort @TomDLT @mathurinm you might be interested.

Labels: New Feature, module:linear_model, Needs Triage
sklearn implementation of the square-root LASSO

### Describe the workflow you want to enable

The unconstrained LASSO is currently implemented in `lasso_path`. However, the parameter for unconstrained LASSO is sensitive to the measurement noise. Instead, the square-root LASSO has objective norm(Ax-b) + alpha * norm(x, 1) and its parameter, alpha, is insensitive to the noise level. I would like an `sklearn` interface (with `fit`, `predict`, etc.) coupled with a `lasso_path`-like function, `srlasso_path`, which solves the square-root LASSO optimization program. 

### Describe your proposed solution

I propose an `sklearn`-friendly wrapper around the `cvxpy` or `cvxopt` interface. More ambitiously, I propose an implementation based on [1] that implements a fast (accelerated?) proximal gradient method for solving square-root LASSO.

[1] "On fast convergence of proximal algorithms for SQRT-LASSO optimization: Don’t worry about its nonsmooth loss function" by Li, Xinguo,  Jiang, Haoming, et al. (2020) PMLR. 

### Describe alternatives you've considered, if relevant

Currently, `statsmodels` offers a square-root LASSO solver implemented via the Python package `cvxopt` - this approach is slow. Using `cvxpy` with another interior point solver like MOSEK offers only 2x speed-up (it is snails compared to `lasso_path`). 

Separately, if the problem is appropriately regularized to avoid over-fitting, some work has shown that it's possible to use accelerated proximal gradient descent to achieve convergence. However, the paper offers an algorithm with several critical implementation details unprovided. See [1] above.

### Additional context

_No response_

COMMENT:
I agree with @lorentzenchr. it can simplify model selection as the regularization does not depend of the noise level but you still need a search. Now it can be of interest in [skglm](https://github.com/scikit-learn-contrib/skglm). 

cc @mathurinm

Labels: New Feature, module:linear_model, Needs Triage