Allows finer costs to be taken into account in learning

### Describe the workflow you want to enable

I am generally trying to take into account costs in learning. The set-up is as follows: a statistical learning problem with usuall X and y, where y is imbalanced (roughly 1% of ones). I also have costs matrices C (see below).

Scikit learn usually offers wights parameters where you can set up weights matching imbalance. So the weights are depending on the target. Assigning weights will transform the log loss into weighted log loss as seen below.

$\text{Log Loss} = -\frac{1}{N} \sum_{i=1}^{N} \left[ y_i \log(\hat{y}_i) + (1 - y_i) \log(1 - \hat{y}_i) \right]$

$\text{Weighted Log Loss} = -\frac{1}{N} \sum_{i=1}^{N} w_{y_i} \left[ y_i \log(\hat{y}_i) + (1 - y_i) \log(1 - \hat{y}_i) \right]$

As you see the weights $w$ is constant on each classes and only depends on $y_i$. I am generally looking for specifying the weights in terms of errors costs. More specifically, I have costs matrixes C associated with:

- $c_{1,1}$, cost associated with True Positives (correctly identified positives)
- $c_{0,1}$, cost associated with False Positives (Type 1 error)
- $c_{1,0}$, cost associated with False Negatives (Type 2 error)
- $c_{0,0}$, cost associated with True Negatives (correctly identified negatives)

With three sub-cases:

1) $c_{y_i,1,1}, c_{y_i,0,1}, c_{y_i,1,0}, c_{y_i,0,0}$ depends only on classes, typically I have classifications costs for each classes (8 parameters in total)
2) $c_{i,1,1}, c_{i,0,1}, c_{i,1,0}, c_{i,0,0}$ depends on instances, so I have four values for each instances.
3) $c_{i,1,1}, c_{i,0,1}, c_{i,1,0}, c_{i,0,0}$ depends both on instances and models outputs $\hat{y}_i$. I think the most generic approach would be to take: 

```math
C = \begin{bmatrix}
\hat{y}_i* w_i & 0 \\
(\hat{y}_i-1)*w_i & 0
\end{bmatrix}
```

$c_{0,1}=c_{1,1}=0$ as we predict the rare event and refuse the relation
$c_{0,0}=\hat{y}_i*w_i $ as we accept the relation and charge proportionally to the estimated risk times some instance notional.
$c_{1,0}=\hat{y}_i*w_i-w_i $ as we charge proportionally to risks but loose the notional.

It appears the cost function can be rewritten as:

```math
\text{Cost-Sensitive Log Loss} = -\frac{1}{N} \sum_{i=1}^{N} \left[ 
y_i \cdot \left( c_{1,0} \cdot \log(\hat{y}_i) + c_{1,1} \cdot \log(1 - \hat{y}_i) \right) + 
(1 - y_i) \cdot \left( c_{0,1} \cdot \log(\hat{y}_i) + c_{0,0} \cdot \log(1 - \hat{y}_i) \right) 
\right]
```

However, it doesn't really seems standard to passes those costs matrices to scikit-learn. 

### Describe your proposed solution

For the different subcases:

1) Ideally, the weights in estimators (class weights) should be able to take into account cost matrices. 
2) Ideally, the weights in estimators (sample weights) should be able to take into account cost matrices.
3) Seems trickier as it would requires to accept customs losses with the costs implemented as a function of $\hat{y}$. Maybe this is not possible to make a generic implementation. But I think the case mentioned is generic enough to be available somewhere.

### Describe alternatives you've considered, if relevant

Mentioning custom losses, all cases could be treatable with it. However, it is not entirely clear how it would works, notably accessing classes / instances weights. 

### Additional context

_No response_

COMMENT:
Metadata routing is only not for ML eng. problems. This example shows a cost sensitive metric: https://scikit-learn.org/stable/auto_examples/model_selection/plot_cost_sensitive_learning.html#sphx-glr-auto-examples-model-selection-plot-cost-sensitive-learning-py

I think that part of the issue is already solved by metadata routing. What can be useful is adding custom loss support to our estimators where applicable. But I'm not sure if that would work. Maybe @lorentzenchr would have an idea?

Labels: Documentation