RFC should the scikit-learn metrics return a Python scalar or a NumPy scalar?

While working on the representation imposed by NEP51, I found out that we recently made the `accuracy_score` to return a Python scalar while, up-to-now, other metric are returning NumPy scalar.

This change was made due to the array API work:

https://github.com/scikit-learn/scikit-learn/blob/b0da1b7706054f0b78f0a0582a9362a188e1fa38/sklearn/utils/_array_api.py#L448-L454

I assume that we are getting to an intersection where we should make the output of our metrics consistent but also foresee potential requirements: as the comment indicate, calling `float()` will be a sync point but it might not be the best strategy for lazy computation.

This RFC is a placeholder to discuss what strategy we should be implementing.

COMMENT:
As hinted by @glemaitre, the accepted NEP51 proposes to change the representation `__repr__` of numpy scalar arrays:

https://numpy.org/neps/nep-0051-scalar-representation.html

It is being implemented for numpy 2.0 scheduled for next year and is causing many of our doctests to fail.

To address NEP51, we could go either way.

## [solution-a] Do nothing to our scoring metrics (continue returning scalar numpy arrays)

- accept that this is a more informative `__repr__` and still decide to use numpy scalar arrays in the output of our metric functions in scikit-learn. The users would get more verbose output when using scikit-learn in interactive model (e.g. in a jupyter notebook sesssion) but at least it would be more explicit about the precision of floating point computation for instance,
- we could either update our docstest so that they would pass on numpy 2.0.0.dev0+ and stop running the doctests on or numpy < 2.0 CIs or alternatively keep the doctests unchanged for now but stop running the doctests on the `[scipy-dev]` build and reexplore this decision once numpy 2.0 is out (or at least an official beta).
- to add array API support, we might need to change our test to call `float()` manually on the result of `accuracy_score` and the like explicitly when needed (e.g. to trigger a lazy computation, move the results to CPU and be able to compare to another Python scalar value).


## [solution-b] Change our scoring metrics to always return a Python scalar

So we could decide to make our scalar metric function always call `float()` (and maybe `int()` in some cases) internally.

This means that the precision level of the computation would be lost.

This also means that eager/blocking execution semantics are forced when calling a metric function with lazy Array API compatible inputs. It would also always move back the resulting scalar value to the CPU without the user having to do a library specific operation.

But this will make our doctests suite pass both for numpy 1.x and numpy 2.x+ unchanged.

This is always less verbose for scikit-learn users calling `accuracy_score` and similar in their notebooks.


### [solution-c] Add a new flag `return_as_python_scalar=True` to all metrics

This way we do the conversion to Python scalar by default but we give control to the user, if they need to access the dtype of the result or keep the computation as lazy as possible when using the Array API.

### More context about the possibility of lazy computation via Array API in scikit-learn

Note that the `fit` method of estimators that use an iterative solver with a `tol`-based stopping criterion will by construction need to trigger eager computation internally by design anyway, so we cannot really hope to express 100% lazy machine learning pipelines. More precisely:

```python
est = IterativeEstimator(tol=1e-4).fit(X_lazy_array_train, y_lazy_array_train)
test_score = est.score(X_lazy_array_test, y_lazy_array_test)
```

even if `test_score` is kept as a lazy scalar array until calling `float(test_score)`, a significant computation would have already been triggered (implicit eager evaluation) when calling `fit` due to the `tol`-based checks at the end of each iteration of the internal solver.

Also note:

- I opened an issue to discuss lazy Array API in scikit-learn in #26724
- Right now, dask does not really implement the Array API but it could quite quickly via array-api-compat: https://github.com/data-apis/array-api-compat/issues/17
- JAX's own Array API support might be enough for some scikit-learn estimators (e.g. PCA) but I haven't tried and we have not included JAX in our array-api compliance test framework yet.
- PyTorch uses eager computation semantics by default, unless it is wrapped in a function decorated by `torch.compile`. By default the compiler attempts to discover blocks of code that can be executed natively (with lazy evaluation semantics) and automatically issue synchronized blocking calls to the Python interpreter whenever its needed.

Labels: help wanted, RFC