Repository: Lightning-AI/torchmetrics

Issue:
for BinaryCalibrationError the n_bins needs to be 1 less than actual to match manual calculation
## 🐛 Bug

<!-- A clear and concise description of what the bug is. -->

given this example 

```python
from torch import tensor
from torchmetrics.classification import BinaryCalibrationError
preds = tensor([0.25, 0.25, 0.55, 0.75, 0.75])
target = tensor([0, 0, 1, 1, 1])
metric = BinaryCalibrationError(n_bins=2, norm='l1')
metric(preds, target)
```

the number of bins are 2 but they should be 3 since there are three probability values -- not sure if this is a documentation error or a bug
<img width="988" alt="Screenshot 2023-07-11 at 12 33 24 PM" src="https://github.com/Lightning-AI/torchmetrics/assets/93558515/042418b1-606b-4df6-9d9e-a9620c224a94">

### To Reproduce

Steps to reproduce the behavior...

<!-- If you have a code sample, error messages, stack traces, please provide it here as well -->

<details>
  <summary>Code sample</summary>

<!-- Ideally attach a minimal code sample to reproduce the decried issue.
Minimal means having the shortest code but still preserving the bug. -->

</details>

### Expected behavior

<!-- A clear and concise description of what you expected to happen. -->

### Environment

- TorchMetrics version (and how you installed TM, e.g. `conda`, `pip`, build from source):
- Python & PyTorch Version (e.g., 1.0):
- Any other relevant information such as OS (e.g., Linux):

### Additional context

<!-- Add any other context about the problem here. -->



Discussion:
Hi! thanks for your contribution!, great first issue!
Hi @mshergad, thanks for raising this issue.
Not completely sure what is the problem here. The `n_bins` is just a freely chosen parameter that determines how precise your approximation of the calibration error is. So for the sake of the example it does not really matter what value I set it to.
Regarding the line you are referring to in the code, that is the normal way to define the bin boundaries. If I have `n_bin=2` and I then evaluate
```python
torch.linspace(0,1,n_bin)
```
I get `tensor([0., 1.])` which is not correct, this only gives me one bin from 0 to 1. Adding 1 to the number of steps:
```python
torch.linspace(0,1,n_bin+1)
```
I get `tensor([0.0000, 0.5000, 1.0000])` which correctly are the two bins I would expect, one from 0 to 0.5 and one from 0.5 to 1.


Thanks for the quick response. Here is the example I tried:

preds is this tensor([0.9000, 0.9000, 0.9000, 0.9000, 0.9000, 0.8000, 0.8000, 0.0100, 0.3300,
        0.3400, 0.9900, 0.6100], dtype=torch.float64)

say target is tensor([1, 1, 1, 0, 0, 1, 0, 1, 0, 1, 0, 0])

for number of bins as 100 and 99 the value is different --

<img width="632" alt="Screenshot 2023-07-12 at 9 31 09 AM" src="https://github.com/Lightning-AI/torchmetrics/assets/93558515/50962563-0b9f-42d3-a888-0fdad20e0d12">

if I consider 100 bins split for 0-1 I would have the right boundary of each bin in increments of 0.01 
for the 100 bin case and using scikit learn and manual calculation of accuracy and weighted average I get the same calculation as that of 99 bins not 100 bins 
<img width="852" alt="Screenshot 2023-07-12 at 9 32 49 AM" src="https://github.com/Lightning-AI/torchmetrics/assets/93558515/41c7c1c9-91e3-4522-9071-6f8ec23339ce">



Thanks for the example. I can definitely see now this is some corner case that we are not covering right now.
I tried to remove the `+1` for the code and run our unittest and there everything is starting to fail, so I do not think that is the solution.
Internally we compare against 
https://github.com/EFS-OpenSource/calibration-framework
which does work for the example you have provided:
```python
from torchmetrics.functional.classification import binary_calibration_error
from netcal.metrics import ECE
import torch
from torch import tensor
preds =  tensor([0.9000, 0.9000, 0.9000, 0.9000, 0.9000, 0.8000, 0.8000, 0.0100, 0.3300, 0.3400, 0.9900, 0.6100], dtype=torch.float64)
target = tensor([1, 1, 1, 0, 0, 1, 0, 1, 0, 1, 0, 0])

print(binary_calibration_error(preds, target, n_bins=99, norm="l1"))  # tensor(0.4733, dtype=torch.float64)
print(binary_calibration_error(preds, target, n_bins=100, norm="l1"))  # tensor(0.4183, dtype=torch.float64)

print(ECE(99).measure(preds.numpy(), target.numpy()))  # 0.4733333333333334
print(ECE(100).measure(preds.numpy(), target.numpy()))  # 0.4733333333333333
```
they also have a `+1` in their code regarding the bin boundaries:
https://github.com/EFS-OpenSource/calibration-framework/blob/45bebd15c873ae399348b8148eb2ea5c89254d27/netcal/metrics/Miscalibration.py#L379
to me this means that it is later in the code that we are doing something wrong that is causing this example to fail.
Just saw that calibration curve in sklearn also has a `+1`:
https://github.com/scikit-learn/scikit-learn/blob/7f9bad99d6e0a3e8ddf92a7e5561245224dab102/sklearn/calibration.py#L1052

Resolving patch:
diff --git a/src/torchmetrics/functional/classification/calibration_error.py b/src/torchmetrics/functional/classification/calibration_error.py
--- a/src/torchmetrics/functional/classification/calibration_error.py
+++ b/src/torchmetrics/functional/classification/calibration_error.py
@@ -84,7 +84,7 @@ def _ce_compute(
 
     """
     if isinstance(bin_boundaries, int):
-        bin_boundaries = torch.linspace(0, 1, bin_boundaries + 1, dtype=torch.float, device=confidences.device)
+        bin_boundaries = torch.linspace(0, 1, bin_boundaries + 1, dtype=confidences.dtype, device=confidences.device)
 
     if norm not in {"l1", "l2", "max"}:
         raise ValueError(f"Argument `norm` is expected to be one of 'l1', 'l2', 'max' but got {norm}")


Test patch:
diff --git a/tests/unittests/classification/test_calibration_error.py b/tests/unittests/classification/test_calibration_error.py
--- a/tests/unittests/classification/test_calibration_error.py
+++ b/tests/unittests/classification/test_calibration_error.py
@@ -24,7 +24,7 @@
     binary_calibration_error,
     multiclass_calibration_error,
 )
-from torchmetrics.utilities.imports import _TORCH_GREATER_EQUAL_1_9
+from torchmetrics.utilities.imports import _TORCH_GREATER_EQUAL_1_9, _TORCH_GREATER_EQUAL_1_13
 
 from unittests import NUM_CLASSES
 from unittests.classification.inputs import _binary_cases, _multiclass_cases
@@ -108,7 +108,8 @@ def test_binary_calibration_error_differentiability(self, inputs):
     def test_binary_calibration_error_dtype_cpu(self, inputs, dtype):
         """Test dtype support of the metric on CPU."""
         preds, target = inputs
-
+        if dtype == torch.half and not _TORCH_GREATER_EQUAL_1_13:
+            pytest.xfail(reason="torch.linspace in metric not supported before pytorch v1.13 for cpu + half")
         if (preds < 0).any() and dtype == torch.half:
             pytest.xfail(reason="torch.sigmoid in metric does not support cpu + half precision")
         self.run_precision_test_cpu(
@@ -123,6 +124,8 @@ def test_binary_calibration_error_dtype_cpu(self, inputs, dtype):
     @pytest.mark.parametrize("dtype", [torch.half, torch.double])
     def test_binary_calibration_error_dtype_gpu(self, inputs, dtype):
         """Test dtype support of the metric on GPU."""
+        if dtype == torch.half and not _TORCH_GREATER_EQUAL_1_13:
+            pytest.xfail(reason="torch.searchsorted in metric not supported before pytorch v1.13 for gpu + half")
         preds, target = inputs
         self.run_precision_test_gpu(
             preds=preds,
@@ -246,3 +249,23 @@ def test_multiclass_calibration_error_dtype_gpu(self, inputs, dtype):
             metric_args={"num_classes": NUM_CLASSES},
             dtype=dtype,
         )
+
+
+def test_corner_case_due_to_dtype():
+    """Test that metric works with edge case where the precision is really important for the right result.
+
+    See issue: https://github.com/Lightning-AI/torchmetrics/issues/1907
+
+    """
+    preds = torch.tensor(
+        [0.9000, 0.9000, 0.9000, 0.9000, 0.9000, 0.8000, 0.8000, 0.0100, 0.3300, 0.3400, 0.9900, 0.6100],
+        dtype=torch.float64,
+    )
+    target = torch.tensor([1, 1, 1, 0, 0, 1, 0, 1, 0, 1, 0, 0])
+
+    assert np.allclose(
+        ECE(99).measure(preds.numpy(), target.numpy()), binary_calibration_error(preds, target, n_bins=99)
+    ), "The metric should be close to the netcal implementation"
+    assert np.allclose(
+        ECE(100).measure(preds.numpy(), target.numpy()), binary_calibration_error(preds, target, n_bins=100)
+    ), "The metric should be close to the netcal implementation"