[Bug]: Setting norm with existing colorbar fails with 3.6.3
### Bug summary

Setting the norm to a `LogNorm` after the colorbar has been created (e.g. in interactive code) fails with an `Invalid vmin` value in matplotlib 3.6.3.

The same code worked in previous matplotlib versions.

Not that vmin and vmax are explicitly set to values valid for `LogNorm` and no negative values (or values == 0) exist in the input data.

### Code for reproduction

```python
import matplotlib.pyplot as plt
from matplotlib.colors import LogNorm
import numpy as np

# create some random data to fill a 2d plot
rng = np.random.default_rng(0)
img = rng.uniform(1, 5, (25, 25))

# plot it
fig, ax = plt.subplots(layout="constrained")
plot = ax.pcolormesh(img)
cbar = fig.colorbar(plot, ax=ax)

vmin = 1
vmax = 5

plt.ion()
fig.show()
plt.pause(0.5)

plot.norm = LogNorm(vmin, vmax)
plot.autoscale()
plt.pause(0.5)
```


### Actual outcome

```
Traceback (most recent call last):
  File "/home/mnoethe/.local/conda/envs/cta-dev/lib/python3.9/site-packages/matplotlib/backends/backend_qt.py", line 454, in _draw_idle
    self.draw()
  File "/home/mnoethe/.local/conda/envs/cta-dev/lib/python3.9/site-packages/matplotlib/backends/backend_agg.py", line 405, in draw
    self.figure.draw(self.renderer)
  File "/home/mnoethe/.local/conda/envs/cta-dev/lib/python3.9/site-packages/matplotlib/artist.py", line 74, in draw_wrapper
    result = draw(artist, renderer, *args, **kwargs)
  File "/home/mnoethe/.local/conda/envs/cta-dev/lib/python3.9/site-packages/matplotlib/artist.py", line 51, in draw_wrapper
    return draw(artist, renderer)
  File "/home/mnoethe/.local/conda/envs/cta-dev/lib/python3.9/site-packages/matplotlib/figure.py", line 3082, in draw
    mimage._draw_list_compositing_images(
  File "/home/mnoethe/.local/conda/envs/cta-dev/lib/python3.9/site-packages/matplotlib/image.py", line 131, in _draw_list_compositing_images
    a.draw(renderer)
  File "/home/mnoethe/.local/conda/envs/cta-dev/lib/python3.9/site-packages/matplotlib/artist.py", line 51, in draw_wrapper
    return draw(artist, renderer)
  File "/home/mnoethe/.local/conda/envs/cta-dev/lib/python3.9/site-packages/matplotlib/axes/_base.py", line 3100, in draw
    mimage._draw_list_compositing_images(
  File "/home/mnoethe/.local/conda/envs/cta-dev/lib/python3.9/site-packages/matplotlib/image.py", line 131, in _draw_list_compositing_images
    a.draw(renderer)
  File "/home/mnoethe/.local/conda/envs/cta-dev/lib/python3.9/site-packages/matplotlib/artist.py", line 51, in draw_wrapper
    return draw(artist, renderer)
  File "/home/mnoethe/.local/conda/envs/cta-dev/lib/python3.9/site-packages/matplotlib/collections.py", line 2148, in draw
    self.update_scalarmappable()
  File "/home/mnoethe/.local/conda/envs/cta-dev/lib/python3.9/site-packages/matplotlib/collections.py", line 891, in update_scalarmappable
    self._mapped_colors = self.to_rgba(self._A, self._alpha)
  File "/home/mnoethe/.local/conda/envs/cta-dev/lib/python3.9/site-packages/matplotlib/cm.py", line 511, in to_rgba
    x = self.norm(x)
  File "/home/mnoethe/.local/conda/envs/cta-dev/lib/python3.9/site-packages/matplotlib/colors.py", line 1694, in __call__
    raise ValueError("Invalid vmin or vmax")
ValueError: Invalid vmin or vmax
```

### Expected outcome

Works, colorbar and mappable are updated with new norm.

### Additional information

_No response_

### Operating system

Linux

### Matplotlib Version

3.6.3 (works with 3.6.2)

### Matplotlib Backend

Multpiple backends tested, same error in all (Qt5Agg, TkAgg, agg, ...)

### Python version

3.9.15

### Jupyter version

not in jupyter

### Installation

conda