Inconsistency in `DecisionTreeClassifier` Threshold Behavior

### Describe the bug

I've encountered an unexpected behavior in `DecisionTreeClassifier` when using a decision stump (a tree with one root node and two leaf children). My assumption is based on the standard decision tree logic where a feature value `x` is classified to the left child if `x <= threshold`. Therefore, I expect the following assertion to always be true:
```python
assert clf.apply([[clf.tree_.threshold[0]]]) == 1
```

This assertion is meant to test that a feature value equal to the root node's threshold is classified to the left child node, in accordance with the `x <= threshold` rule. However, I have observed that in approximately 25% of the cases, the data point is unexpectedly classified to the right child node instead of the left.

The issue persists regardless of whether the `threshold[0]` is explicitly converted to `float32` or not. Given scikit-learn's documentation stating that
> All decision trees use `np.float32` arrays internally. If training data is not in this format, a copy of the dataset will be made

this behavior is puzzling. Precise thresholding is crucial for my application. Thank you for your time and effort in maintaining this important library and for any insights you can provide regarding this issue.

### Steps/Code to Reproduce

```python
import numpy as np
from sklearn.tree import DecisionTreeClassifier

correct, wrong = [], []
n_trials = 500
for seed in range(n_trials):
    rand = np.random.RandomState(seed)
    X = rand.normal(size=(100, 1))
    y = rand.randint(0, 2, size=X.shape[0])
    clf = DecisionTreeClassifier(max_depth=1, random_state=0).fit(X, y)
    
    thres = clf.tree_.threshold[0]
    # thres = np.float32(thres) # gives the same correct-wrong distribution
    if clf.apply([[thres]]) == 1:
        correct.append(thres)
    else:
        wrong.append(thres)

    assert clf.apply([[np.nextafter(np.float32(thres), np.float32(-np.inf))]]) == 1 # this is true though
print(f'{len(correct) / n_trials:%}') # 75%
print(f'{len(wrong) / n_trials:%}') # 25%
```

### Expected Results

```
100.0000%
0.0000%
```

### Actual Results

```
75.0000%
25.0000%
```

### Versions

```shell
System:
    python: 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0]
executable: /usr/bin/python3
   machine: Linux-6.1.58+-x86_64-with-glibc2.35

Python dependencies:
      sklearn: 1.2.2
          pip: 23.1.2
   setuptools: 67.7.2
        numpy: 1.23.5
        scipy: 1.11.4
       Cython: 3.0.8
       pandas: 1.5.3
   matplotlib: 3.7.1
       joblib: 1.3.2
threadpoolctl: 3.2.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
    num_threads: 2
         prefix: libopenblas
       filepath: /usr/local/lib/python3.10/dist-packages/numpy.libs/libopenblas64_p-r0-742d56dc.3.20.so
        version: 0.3.20
threading_layer: pthreads
   architecture: Haswell

       user_api: openmp
   internal_api: openmp
    num_threads: 2
         prefix: libgomp
       filepath: /usr/local/lib/python3.10/dist-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
        version: None

       user_api: blas
   internal_api: openblas
    num_threads: 2
         prefix: libopenblas
       filepath: /usr/local/lib/python3.10/dist-packages/scipy.libs/libopenblasp-r0-23e5df77.3.21.dev.so
        version: 0.3.21.dev
threading_layer: pthreads
   architecture: Haswell
```

COMMENT:
I don't know yet if we should solve this floating-point error or not.

At inference time (when calling `apply`), I would be under the impression that we would save a copy of the dataset if we were using fused type. We would only need to upcast the value of the threshold if `X` is `np.float32`. But we are currently always downcasting from `np.float32` to `np.float64` to make all comparison so we should not have a regression there.

Also making a `np.float64` comparison should not be less efficient than `np.float32` since that there is not vectorization at this stage or whatever low-level optimization.

However, I might miss some consideration. @ogrisel @jeremiedbb @adrinjalali Do you see something for which my reasoning would fall short?

Labels: Bug