Issue 1:
Description

Output of SVC.decision_function for one observation depends on the other observations --> it is not independant.

Steps/Code to Reproduce

import numpy
from sklearn.svm import SVC
from sklearn.datasets import load_iris

iris = load_iris()
X = iris.data[:, :3]
y = iris.target
model = SVC()
model.fit(X, y)

X1 = X[:1] * 0.10
X2 = X[1:2]
X3 = X[2:3]
X1X2 = numpy.vstack([X1, X2])
X2X3 = numpy.vstack([X2, X3])
X1X3 = numpy.vstack([X1, X3])
X1X2X3 = numpy.vstack([X1, X2, X3])

print('X1')
print(model.decision_function(X1))
print('X2')
print(model.decision_function(X2))
print('X3')
print('')
print('X1 X2')
print(model.decision_function(X1X2))
print('X2 X3')
print(model.decision_function(X2X3))
print('')
print('X1 X2 X3')
print(model.decision_function(X1X2X3))

Expected Results

decision_function should verify:

decision_function(X1) == decision_function(X1X2)[:1]

Actual Results

X1
[[-0.49153396 0.99153396 2.5 ]]
X2
[[ 2.5 0.77197248 -0.27197248]]
X3

X1 X2
[[-0.04698652 0.99919072 2.0477958 ]
[ 2.5 0.77197248 -0.27197248]]
X2 X3
[[ 2.48614274 0.77829215 -0.26443489]
[ 2.5 0.75104163 -0.25104163]]

X1 X2 X3
[[-0.04568431 0.99921315 2.04647116]
[ 2.48614274 0.77829215 -0.26443489]
[ 2.5 0.75104163 -0.25104163]]

Reasons

decision_function calls _ovr_decision_function which normalizes outputs based on the whole set of observations.

https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/utils/multiclass.py#L432

Versions

Windows-10-10.0.17134-SP0

Python 3.7.0 (v3.7.0:1bf9cc5093, Jun 27 2018, 04:59:51) [MSC v.1914 64 bit (AMD64)]

NumPy 1.15.3

SciPy 1.1.0

Scikit-Learn 0.20.0

Issue 2:
Describe the workflow you want to enable

In my dreams, a new InferenceData class would simplify training and prediction to look more like

import ... as learner

data = InferenceData(
df=..., # a data frame
meta=Meta(
y_cols=..., # name(s) of output variable
... # additional metadata fields
)
)
train_data, test_data = data.split(train_fraction=0.7, ...)
learner.fit(train_data)
predictions = learner.predict(test_data.x)

Note that this introduces just two data variables train_data and test_data instead of the current standard four (X_train, X_test, y_train, y_test).

In addition, InferenceData could easily be extended to allow the above pipeline to handle related metadata such as feature weights, replacing a step like learner.fit(train_data) by learner.fit(train_data, weights=train_data.row_weights), for example.

Describe your proposed solution

A solution could look something like this:

@dataclass
class Meta:
"""Metadata for a Data class."""
y_cols: Optional[list[str]] = None
row_weights_col: Optional[str] = None

@property
def y(self) -> list[str]:
"""Output variable column names."""
if not self.y_cols:
return []
return self.y_cols

@property
def columns(self) -> set[str]:
"""All metadata columns."""
cols = set(self.y)
if self.row_weights_col:
cols.add(row_weights_col)
return cols

@dataclass
class InferenceData:
"""A data frame container that includes metadata relevant for machine learning and inference."""

df: DataFrame
meta: Meta = Meta()

def __post_init__(self) -> None:
"""Parameter validation."""
if (self.y is not None) and (self.n_rows != len(self.y)):
raise ValueError("Expected y to have the same number of data points as x has.")
# TODO: also validate that all columns referenced in self.meta exist in self.df etc

@property
def x(self) -> DataFrame:
"""The data frame of predictor variables, excluding output variables and weights, etc."""
non_metadata_cols = [col for col in self.df if col not in self.meta.columns]
return self.df[non_metadata_cols]

@property
def y(self) -> None | np.ndarray:
"""Output features."""
if not self.meta.y_cols:
return None
if len(self.meta.y_cols) == 1:
y_col = self.meta.y_cols[0]
return self.df[y_col].to_numpy()
return self.df[self.meta.y_cols].to_numpy()

@property
def w(self) -> None | np.ndarray:
"""The data frame of predictor variables, excluding output variables and weights, etc."""
return self.df[row_weights_col]

def iloc(self, positional_indexes: Iterable) -> "InferenceData":
"""Return the subset of the data in self corresponding to the specificied positional indices."""
return InferenceData(df=df.iloc[positional_indexes], meta=self.meta)

@property
def n_rows(self) -> int:
"""Number of rows of data."""
return self.df.shape[0]

Of course, the usefulness of this solution will depend on wrapping existing machine learning algorithms to accept an InferenceData class as input.

Describe alternatives you've considered, if relevant

No response

Additional context

No response