internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: Lightning-AI/torchmetrics

Issue:
Iteration over a metric never ends
## 🐛 Bug
If you try to iterate over a metric (by mistake in my case), it just spins forever, making it appear that the program is hung. I'm not sure internally to torchmetrics exactly why this behavior occurs and if it is desired, but it would be nice if it errored out or something, unless there is a use for iterating over a metric that I am not aware of?

```py
from torchmetrics.classification import Accuracy
acc = Accuracy()
for i, item in enumerate(acc):
    print(i, item)
```

The above code will spin forever, printing something like this
```
8144 CompositionalMetric(
  <lambda>(
    Accuracy(),
    None
  )
)
```

### Expected behavior
I would expect an error to be raised if I try to iterate over a metric.

### Environment
- TorchMetrics version (and how you installed TM, e.g. `conda`, `pip`, build from source):  `0.10.2`




Discussion:

Resolving patch:
diff --git a/src/torchmetrics/metric.py b/src/torchmetrics/metric.py
--- a/src/torchmetrics/metric.py
+++ b/src/torchmetrics/metric.py
@@ -858,6 +858,9 @@ def __getitem__(self, idx: int) -> "Metric":
     def __getnewargs__(self) -> Tuple:
         return (Metric.__str__(self),)
 
+    def __iter__(self):
+        raise NotImplementedError("Metrics does not support iteration.")
+
 
 def _neg(x: Tensor) -> Tensor:
     return -torch.abs(x)


Test patch:
diff --git a/tests/unittests/bases/test_metric.py b/tests/unittests/bases/test_metric.py
--- a/tests/unittests/bases/test_metric.py
+++ b/tests/unittests/bases/test_metric.py
@@ -464,3 +464,10 @@ def test_custom_availability_check_and_sync_fn():
     acc.compute()
     dummy_availability_check.assert_called_once()
     assert dummy_dist_sync_fn.call_count == 4  # tp, fp, tn, fn
+
+
+def test_no_iteration_allowed():
+    metric = DummyMetric()
+    with pytest.raises(NotImplementedError, match="Metrics does not support iteration."):
+        for m in metric:
+            continue
Repository: Lightning-AI/torchmetrics

Issue:
bug in pairwise_euclidean_distance
https://github.com/Lightning-AI/metrics/blob/e1c3fda24f90367803c2b04315ad7c8bced719db/torchmetrics/functional/pairwise/euclidean.py#L34
this line can become negative, resulting in a failure with the sqrt function and thus return "nan"

you can test this easily by checking this code:

`pairwise_euclidean_distance(torch.tensor([[772., 112.], [772.20001, 112.], [772.20001, 112.], [772., 112.00000], [772.2, 112.00000], [772.0, 112.00000], [772.01, 112.00000], [772.00000000000001, 112.00000], [772.000001, 112.00000], [772.00001, 112.00000], [772.0001, 112.00000], [772.001, 112.00000], [772.01, 112.00000], [772.99, 112.00000]], dtype=torch.float32))`


Discussion:
Hi! thanks for your contribution!, great first issue!

Resolving patch:
diff --git a/src/torchmetrics/functional/pairwise/euclidean.py b/src/torchmetrics/functional/pairwise/euclidean.py
--- a/src/torchmetrics/functional/pairwise/euclidean.py
+++ b/src/torchmetrics/functional/pairwise/euclidean.py
@@ -13,6 +13,7 @@
 # limitations under the License.
 from typing import Optional
 
+import torch
 from torch import Tensor
 from typing_extensions import Literal
 
@@ -30,9 +31,13 @@ def _pairwise_euclidean_distance_update(
         zero_diagonal: determines if the diagonal of the distance matrix should be set to zero
     """
     x, y, zero_diagonal = _check_input(x, y, zero_diagonal)
-    x_norm = x.norm(dim=1, keepdim=True)
-    y_norm = y.norm(dim=1).T
-    distance = x_norm * x_norm + y_norm * y_norm - 2 * x.mm(y.T)
+    # upcast to float64 to prevent precision issues
+    _orig_dtype = x.dtype
+    x = x.to(torch.float64)
+    y = y.to(torch.float64)
+    x_norm = (x * x).sum(dim=1, keepdim=True)
+    y_norm = (y * y).sum(dim=1)
+    distance = (x_norm + y_norm - 2 * x.mm(y.T)).to(_orig_dtype)
     if zero_diagonal:
         distance.fill_diagonal_(0)
     return distance.sqrt()


Test patch:
diff --git a/tests/unittests/pairwise/test_pairwise_distance.py b/tests/unittests/pairwise/test_pairwise_distance.py
--- a/tests/unittests/pairwise/test_pairwise_distance.py
+++ b/tests/unittests/pairwise/test_pairwise_distance.py
@@ -115,3 +115,20 @@ def test_error_on_wrong_shapes(metric):
 
     with pytest.raises(ValueError, match="Expected reduction to be one of .*"):
         metric(torch.randn(10, 5), torch.randn(10, 5), reduction=1)
+
+
+@pytest.mark.parametrize(
+    "metric_functional, sk_fn",
+    [
+        (pairwise_cosine_similarity, cosine_similarity),
+        (pairwise_euclidean_distance, euclidean_distances),
+        (pairwise_manhattan_distance, manhattan_distances),
+        (pairwise_linear_similarity, linear_kernel),
+    ],
+)
+def test_precison_case(metric_functional, sk_fn):
+    """test that metrics are robust towars cases where high precision is needed."""
+    x = torch.tensor([[772.0, 112.0], [772.20001, 112.0]])
+    res1 = metric_functional(x, zero_diagonal=False)
+    res2 = sk_fn(x)
+    assert torch.allclose(res1, torch.tensor(res2, dtype=torch.float32))
Repository: Lightning-AI/torchmetrics

Issue:
module 'torchmetrics.classification' has no attribute 'AUC'
## 🐛 Bug

Importing all the classification metrics causes the `AttributeError`:
`from torchmetrics.classification import *`

`AttributeError: module 'torchmetrics.classification' has no attribute 'AUC'`

Environment
torchmetrics 0.11.0
pytorch 1.13.0

In order to fix it someone should remove AUC from the list __all__ (src/torchmetrics/classification/__init__.py)


Discussion:
Hi! thanks for your contribution!, great first issue!

Resolving patch:
diff --git a/src/torchmetrics/classification/__init__.py b/src/torchmetrics/classification/__init__.py
--- a/src/torchmetrics/classification/__init__.py
+++ b/src/torchmetrics/classification/__init__.py
@@ -119,7 +119,6 @@
     "BinaryAccuracy",
     "MulticlassAccuracy",
     "MultilabelAccuracy",
-    "AUC",
     "AUROC",
     "BinaryAUROC",
     "MulticlassAUROC",


Test patch:
Repository: Lightning-AI/torchmetrics

Issue:
Small typo in PESQ docs
## 📚 Documentation

[Perceptual Evaluation Of Speech Quality (PESQ) documentation](https://torchmetrics.readthedocs.io/en/stable/audio/perceptual_evaluation_speech_quality.html) states following for both Module Interface and Functional Interface:
Raises ModuleNotFoundError – If **peqs** package is not installed.



Discussion:
Hi! thanks for your contribution!, great first issue!

Resolving patch:
diff --git a/src/torchmetrics/audio/pesq.py b/src/torchmetrics/audio/pesq.py
--- a/src/torchmetrics/audio/pesq.py
+++ b/src/torchmetrics/audio/pesq.py
@@ -55,7 +55,7 @@ class PerceptualEvaluationSpeechQuality(Metric):
 
     Raises:
         ModuleNotFoundError:
-            If ``peqs`` package is not installed
+            If ``pesq`` package is not installed
         ValueError:
             If ``fs`` is not either  ``8000`` or ``16000``
         ValueError:
diff --git a/src/torchmetrics/functional/audio/pesq.py b/src/torchmetrics/functional/audio/pesq.py
--- a/src/torchmetrics/functional/audio/pesq.py
+++ b/src/torchmetrics/functional/audio/pesq.py
@@ -62,13 +62,13 @@ def perceptual_evaluation_speech_quality(
 
     Raises:
         ModuleNotFoundError:
-            If ``peqs`` package is not installed
+            If ``pesq`` package is not installed
         ValueError:
             If ``fs`` is not either  ``8000`` or ``16000``
         ValueError:
             If ``mode`` is not either ``"wb"`` or ``"nb"``
         RuntimeError:
-            If ``preds`` and ``target`` does not have the same shape
+            If ``preds`` and ``target`` do not have the same shape
 
     Example:
         >>> from torchmetrics.functional.audio.pesq import perceptual_evaluation_speech_quality


Test patch:
Repository: Lightning-AI/torchmetrics

Issue:
SMAPE formula typo
## 📚 Documentation


There's a typo in the [SMAPE formula](https://torchmetrics.readthedocs.io/en/stable/regression/symmetric_mean_absolute_percentage_error.html). It should be `{SMAPE} = \frac{2}{n}\sum_1^n\frac{|   y_i - \hat{y_i} |}{\max(| y_i | + | \hat{y_i} |, \epsilon)}` instead of `{SMAPE} = \frac{2}{n}\sum_1^n max(\frac{|   y_i - \hat{y_i} |}{| y_i | + | \hat{y_i} |, \epsilon})`. The attached screenshot shows the typo and its correction.
![smape](https://user-images.githubusercontent.com/44662992/213825207-21c308b1-a407-4830-813f-a3dbe87ddb41.png)



Discussion:
Hi! thanks for your contribution!, great first issue!

Resolving patch:
diff --git a/src/torchmetrics/regression/symmetric_mape.py b/src/torchmetrics/regression/symmetric_mape.py
--- a/src/torchmetrics/regression/symmetric_mape.py
+++ b/src/torchmetrics/regression/symmetric_mape.py
@@ -25,7 +25,7 @@
 class SymmetricMeanAbsolutePercentageError(Metric):
     r"""Computes symmetric mean absolute percentage error (`SMAPE`_).
 
-    .. math:: \text{SMAPE} = \frac{2}{n}\sum_1^n max(\frac{|   y_i - \hat{y_i} |}{| y_i | + | \hat{y_i} |, \epsilon})
+    .. math:: \text{SMAPE} = \frac{2}{n}\sum_1^n\frac{|   y_i - \hat{y_i} |}{\max(| y_i | + | \hat{y_i} |, \epsilon)}
 
     Where :math:`y` is a tensor of target values, and :math:`\hat{y}` is a tensor of predictions.
 


Test patch:
Repository: Lightning-AI/torchmetrics

Issue:
change `targets.dtype` to the type of `outputs` in the intern of `StructuralSimilarityIndexMeasure().update()`?
## 🚀 Feature

<!-- A clear and concise description of the feature proposal -->

### Motivation

<!-- Please outline the motivation for the proposal. Is your feature request related to a problem? e.g., I'm always frustrated when [...]. If this is related to another GitHub issue, please link here too -->
Now, `StructuralSimilarityIndexMeasure().update(outputs, targets)` will call `_ssim_check_inputs(outputs, targets)` to `assert outputs.dtype == targets.dtype`. However, in auto mixed precision training:

```python
from torch.cuda.amp.autocast_mode import autocast
with autocast():
    outputs = model(samples)  # outputs.dtype == torch.float16, while targets.dtype == torch.float32
    loss = loss_function(outputs, targets)
metric(outputs, targets)  # assert failure
```

How about change `targets.dtype` to the type of `outputs` in the intern of `StructuralSimilarityIndexMeasure().update()`?

Just `targets.as_type(preds)`?

### Pitch

<!-- A clear and concise description of what you want to happen. -->

### Alternatives

<!-- A clear and concise description of any alternative solutions or features you've considered, if any. -->
Or is there any method to skip this check like `validate_args = False`?
### Additional context

<!-- Add any other context or screenshots about the feature request here. -->



Discussion:

Resolving patch:
diff --git a/src/torchmetrics/functional/image/ssim.py b/src/torchmetrics/functional/image/ssim.py
--- a/src/torchmetrics/functional/image/ssim.py
+++ b/src/torchmetrics/functional/image/ssim.py
@@ -33,10 +33,7 @@ def _ssim_check_inputs(preds: Tensor, target: Tensor) -> Tuple[Tensor, Tensor]:
     """
 
     if preds.dtype != target.dtype:
-        raise TypeError(
-            "Expected `preds` and `target` to have the same data type."
-            f" Got preds: {preds.dtype} and target: {target.dtype}."
-        )
+        target = target.to(preds.dtype)
     _check_same_shape(preds, target)
     if len(preds.shape) not in (4, 5):
         raise ValueError(


Test patch:
diff --git a/tests/unittests/image/test_ssim.py b/tests/unittests/image/test_ssim.py
--- a/tests/unittests/image/test_ssim.py
+++ b/tests/unittests/image/test_ssim.py
@@ -231,11 +231,8 @@ def test_ssim_half_gpu(self, preds, target, sigma):
     ],
 )
 def test_ssim_invalid_inputs(pred, target, kernel, sigma):
-    pred_t = torch.rand(pred, dtype=torch.float32)
-    target_t = torch.rand(target, dtype=torch.float64)
-    with pytest.raises(TypeError):
-        structural_similarity_index_measure(pred_t, target_t)
-
+    """Test that an value errors are raised if input sizes are different, kernel length and sigma does not match
+    size or invalid values are provided."""
     pred = torch.rand(pred)
     target = torch.rand(target)
     with pytest.raises(ValueError):
Repository: Lightning-AI/torchmetrics

Issue:
0.11.2 and 0.11.3 not released on PyPI
The 0.11.2 and 0.11.3 releases have both been released on GitHub, but haven't yet been released on PyPI. Is this planned for the future, or is the release process broken?


Discussion:
Hi! thanks for your contribution!, great first issue!

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -78,6 +78,9 @@ def _parse_requirements(strs: Union[str, Iterable[str]]) -> Iterator[_Requiremen
                 line += next(lines)
             except StopIteration:
                 return
+        if "@" in line or re.search("https?://", line):
+            # skip lines with links like `pesq @ git+https://github.com/ludlows/python-pesq`
+            continue
         # If there's a pip argument, save it
         if line.startswith("--"):
             pip_argument = line


Test patch:
Repository: Lightning-AI/torchmetrics

Issue:
MS-SSIM for 3D input
Similar to the ssim index in 

https://github.com/Lightning-AI/metrics/blob/a6320cfae434b96d41fdf94d6387f4aed02b9062/src/torchmetrics/functional/image/ssim.py#L161

the contrast_sensitivity should remove the pad_d in the 3D case in  

https://github.com/Lightning-AI/metrics/blob/a6320cfae434b96d41fdf94d6387f4aed02b9062/src/torchmetrics/functional/image/ssim.py#L167


Discussion:
Hi! thanks for your contribution!, great first issue!
Hey @Warvito , would you be interested in contributing a 3D port?

Resolving patch:
diff --git a/src/torchmetrics/functional/image/ssim.py b/src/torchmetrics/functional/image/ssim.py
--- a/src/torchmetrics/functional/image/ssim.py
+++ b/src/torchmetrics/functional/image/ssim.py
@@ -168,7 +168,10 @@ def _ssim_update(
 
     if return_contrast_sensitivity:
         contrast_sensitivity = upper / lower
-        contrast_sensitivity = contrast_sensitivity[..., pad_h:-pad_h, pad_w:-pad_w]
+        if is_3d:
+            contrast_sensitivity = contrast_sensitivity[..., pad_h:-pad_h, pad_w:-pad_w, pad_d:-pad_d]
+        else:
+            contrast_sensitivity = contrast_sensitivity[..., pad_h:-pad_h, pad_w:-pad_w]
         return ssim_idx.reshape(ssim_idx.shape[0], -1).mean(-1), contrast_sensitivity.reshape(
             contrast_sensitivity.shape[0], -1
         ).mean(-1)


Test patch:
diff --git a/tests/unittests/image/test_ms_ssim.py b/tests/unittests/image/test_ms_ssim.py
--- a/tests/unittests/image/test_ms_ssim.py
+++ b/tests/unittests/image/test_ms_ssim.py
@@ -96,3 +96,13 @@ def test_ms_ssim_differentiability(self, preds, target):
                 "betas": single_beta,
             },
         )
+
+
+def test_ms_ssim_contrast_sensitivity():
+    """Test that the contrast sensitivity is correctly computed with 3d input."""
+    preds = torch.rand(1, 1, 182, 182, 182)
+    target = torch.rand(1, 1, 182, 182, 182)
+    out = multiscale_structural_similarity_index_measure(
+        preds, target, data_range=1.0, kernel_size=11, betas=(1.0, 0.5, 0.25)
+    )
+    assert isinstance(out, torch.Tensor)
Repository: Lightning-AI/torchmetrics

Issue:
`MultioutputWrapper` does not squeeze `kwargs`
Hi there,

`MultioutputWrapper` class provides the option `squeeze_outputs`. If this is set the output should be squeezed on the index select dimension.
Altough this happens for the `args` it does not happen for `kwargs`

Here is are the where it happens, but it is only done for the `args` not for the `kwargs`.
https://github.com/Lightning-AI/torchmetrics/blob/af52ba6a422e9f8c99853396d78882015a3c6fe3/src/torchmetrics/wrappers/multioutput.py#L105-L108

Should be a one-line change. if needed i can provide an PR

Greetings Andi


Discussion:
Hi! thanks for your contribution!, great first issue!
Hi @apointa, feel free to send a PR with the change :)

Resolving patch:
diff --git a/src/torchmetrics/wrappers/multioutput.py b/src/torchmetrics/wrappers/multioutput.py
--- a/src/torchmetrics/wrappers/multioutput.py
+++ b/src/torchmetrics/wrappers/multioutput.py
@@ -106,6 +106,7 @@ def _get_args_kwargs_by_output(self, *args: Tensor, **kwargs: Tensor) -> List[Tu
 
             if self.squeeze_outputs:
                 selected_args = [arg.squeeze(self.output_dim) for arg in selected_args]
+                selected_kwargs = {k: v.squeeze(self.output_dim) for k, v in selected_kwargs.items()}
             args_kwargs_by_output.append((selected_args, selected_kwargs))
         return args_kwargs_by_output
 


Test patch:
diff --git a/tests/unittests/wrappers/test_multioutput.py b/tests/unittests/wrappers/test_multioutput.py
--- a/tests/unittests/wrappers/test_multioutput.py
+++ b/tests/unittests/wrappers/test_multioutput.py
@@ -134,3 +134,12 @@ def test_reset_called_correctly():
     res = cf(tensor([[1, 1]]), tensor([[0, 0]]))
     assert torch.allclose(res[0], tensor([[0, 1], [0, 0]]))
     assert torch.allclose(res[1], tensor([[0, 1], [0, 0]]))
+
+
+def test_squeeze_argument():
+    """Test that the squeeze_outputs argument works as expected."""
+    m = MultioutputWrapper(ConfusionMatrix(task="binary"), num_outputs=3)
+    m.update(torch.randint(2, (10, 3)), torch.randint(2, (10, 3)))  # as args
+    m.update(preds=torch.randint(2, (10, 3)), target=torch.randint(2, (10, 3)))  # as kwargs
+    val = m.compute()
+    assert val.shape == (3, 2, 2)
Repository: Lightning-AI/torchmetrics

Issue:
MeanAveragePrecision box format
https://torchmetrics.readthedocs.io/en/stable/detection/mean_average_precision.html

Docs say

> Predicted boxes and targets have to be in Pascal VOC format (xmin-top left, ymin-top left, xmax-bottom right, ymax-bottom right)

However, it accepts `box_format` which looks like it overrides this requirement. Therefore this sentence needs to be deleted. 


Discussion:
Hi! thanks for your contribution!, great first issue!

Resolving patch:
diff --git a/src/torchmetrics/detection/mean_ap.py b/src/torchmetrics/detection/mean_ap.py
--- a/src/torchmetrics/detection/mean_ap.py
+++ b/src/torchmetrics/detection/mean_ap.py
@@ -150,9 +150,12 @@ def _segm_iou(det: List[Tuple[np.ndarray, np.ndarray]], gt: List[Tuple[np.ndarra
 class MeanAveragePrecision(Metric):
     r"""Compute the `Mean-Average-Precision (mAP) and Mean-Average-Recall (mAR)`_ for object detection predictions.
 
-    Predicted boxes and targets have to be in Pascal VOC format (xmin-top left, ymin-top left, xmax-bottom right,
-    ymax-bottom right). The metric can both compute the mAP and mAR values per class or as an global average over all
-    classes.
+    .. math::
+        \text{mAP} = \frac{1}{n} \sum_{i=1}^{n} AP_i
+
+    where :math:`AP_i` is the average precision for class :math:`i` and :math:`n` is the number of classes. The average
+    precision is defined as the area under the precision-recall curve. If argument `class_metrics` is set to ``True``,
+    the metric will also return the mAP/mAR per class.
 
     As input to ``forward`` and ``update`` the metric accepts the following input:
 


Test patch:
Repository: Lightning-AI/torchmetrics

Issue:
NLTK punkt bug
## 🐛 Bug

[This line](https://github.com/Lightning-AI/torchmetrics/blob/17c0e9fdf636cff7299fc3d86fa5e3709b603737/src/torchmetrics/functional/text/rouge.py#L52) `nltk.data.find("tokenizers/punkt.zip")` always raises LookupError:
```
LookupError: 
**********************************************************************
  Resource punkt not found.
  Please use the NLTK Downloader to obtain the resource:

  >>> import nltk
  >>> nltk.download('punkt')
  
  For more information see: https://www.nltk.org/data.html

  Attempted to load tokenizers/punkt.zip

  Searched in:
    - '/home/**/nltk_data'
    - '/data/**/anaconda3/nltk_data'
    - '/data/**/anaconda3/share/nltk_data'
    - '/data/**/anaconda3/lib/nltk_data'
    - '/usr/share/nltk_data'
    - '/usr/local/share/nltk_data'
    - '/usr/lib/nltk_data'
    - '/usr/local/lib/nltk_data'
**********************************************************************
```

Maybe it should be `nltk.data.find("tokenizers/punkt")`?
After the downloading that follows, this calling to `find` without .zip extension works successfully.

### To Reproduce

```
import nltk

try:
    nltk.data.find("tokenizers/punkt")
except LookupError:
    try:
        nltk.download("punkt", quiet=True, force=False, halt_on_error=False, raise_on_error=True)
    except ValueError:
        raise OSError(
            "`nltk` resource `punkt` is not available on a disk and cannot be downloaded as a machine is not "
            "connected to the internet."
        )

nltk.data.find("tokenizers/punkt.zip")
```

### Expected behavior

No error raised.

### Environment

- TorchMetrics version (and how you installed TM, e.g. `conda`, `pip`, build from source): 0.11.4, pip installed
- Python & PyTorch Version (e.g., 1.0): Python 3.7.10, PyTorch 1.13.0
- Any other relevant information such as OS (e.g., Linux): Ubuntu 18.04, nltk 3.8.1

### Additional context

The above information should be enough.



Discussion:
Hi! thanks for your contribution!, great first issue!
Same problem. And the function always raises the error in epoch 2 or later, it has terminated my training serval times......

Resolving patch:
diff --git a/src/torchmetrics/functional/text/rouge.py b/src/torchmetrics/functional/text/rouge.py
--- a/src/torchmetrics/functional/text/rouge.py
+++ b/src/torchmetrics/functional/text/rouge.py
@@ -49,7 +49,7 @@ def _ensure_nltk_punkt_is_downloaded() -> None:
     import nltk
 
     try:
-        nltk.data.find("tokenizers/punkt.zip")
+        nltk.data.find("tokenizers/punkt")
     except LookupError:
         try:
             nltk.download("punkt", quiet=True, force=False, halt_on_error=False, raise_on_error=True)


Test patch:
Repository: Lightning-AI/torchmetrics

Issue:
Wrong argument check in `MetricCollection.plot`
https://github.com/Lightning-AI/torchmetrics/blob/2d356501cd686b966071c2c79d45d7cbde8cf8f2/src/torchmetrics/collections.py#L596C17-L601

It should be:
```python
if (
    not together and (
    not isinstance(ax, Sequence) 
    or not all(isinstance(a, _AX_TYPE) for a in ax) 
    or len(ax) != len(self))
):
```


Discussion:

Resolving patch:
diff --git a/src/torchmetrics/collections.py b/src/torchmetrics/collections.py
--- a/src/torchmetrics/collections.py
+++ b/src/torchmetrics/collections.py
@@ -593,11 +593,8 @@ def plot(
                 raise ValueError(
                     f"Expected argument `ax` to be a matplotlib axis object, but got {type(ax)} when `together=True`"
                 )
-            if (
-                not together
-                and not isinstance(ax, Sequence)
-                and not all(isinstance(a, _AX_TYPE) for a in ax)
-                and len(ax) != len(self)
+            if not together and not (
+                isinstance(ax, Sequence) and all(isinstance(a, _AX_TYPE) for a in ax) and len(ax) == len(self)
             ):
                 raise ValueError(
                     f"Expected argument `ax` to be a sequence of matplotlib axis objects with the same length as the "


Test patch:
diff --git a/tests/unittests/utilities/test_plot.py b/tests/unittests/utilities/test_plot.py
--- a/tests/unittests/utilities/test_plot.py
+++ b/tests/unittests/utilities/test_plot.py
@@ -824,6 +824,14 @@ def test_plot_method_collection(together, num_vals):
         assert all(isinstance(f[0], plt.Figure) for f in fig_ax)
         assert all(isinstance(f[1], matplotlib.axes.Axes) for f in fig_ax)
 
+    # test ax arg
+    fig, ax = plt.subplots(nrows=len(m_collection), ncols=1)
+    m_collection.plot(ax=ax.tolist())
+
+    fig, ax = plt.subplots(nrows=len(m_collection) + 1, ncols=1)
+    with pytest.raises(ValueError, match="Expected argument `ax` to be a sequence of matplotlib axis objects with.*"):
+        m_collection.plot(ax=ax.tolist())
+
 
 @pytest.mark.parametrize(
     ("metric_class", "preds", "target"),