Repository: napari/napari

Issue:
pydantic 1.8 release
## 🐛 Bug

Looks like the pydantic 1.8 release has broken something. Not sure what yet. Could make a quick pin < 1.8 or quick fix cc @tlambert03 


Discussion:
```python-traceback
Traceback (most recent call last):
  File "examples/add_image.py", line 6, in <module>
    import napari
  File "/Users/nsofroniew/GitHub/napari/napari/__init__.py", line 7, in <module>
    from .viewer import Viewer  # isort:skip
  File "/Users/nsofroniew/GitHub/napari/napari/viewer.py", line 3, in <module>
    from .components import ViewerModel
  File "/Users/nsofroniew/GitHub/napari/napari/components/__init__.py", line 18, in <module>
    from .dims import Dims
  File "/Users/nsofroniew/GitHub/napari/napari/components/dims.py", line 10, in <module>
    class Dims(EventedModel):
  File "/Users/nsofroniew/GitHub/napari/napari/utils/events/evented_model.py", line 78, in __new__
    cls = super().__new__(mcs, name, bases, namespace, **kwargs)
  File "pydantic/main.py", line 287, in pydantic.main.ModelMetaclass.__new__
  File "pydantic/fields.py", line 392, in pydantic.fields.ModelField.infer
  File "pydantic/fields.py", line 327, in pydantic.fields.ModelField.__init__
  File "pydantic/fields.py", line 438, in pydantic.fields.ModelField.prepare
  File "pydantic/fields.py", line 651, in pydantic.fields.ModelField.populate_validators
  File "pydantic/validators.py", line 710, in find_validators
RuntimeError: error checking inheritance of typing.Tuple[float, float, float] (type: Tuple[float, float, float])
```
There is plan for some fast release (0.4.5.post1 differed from 0.4.5 with one commit) with such pin? It touch also napari in 0.4.5.
I think we could start release process for 0.4.6, it's been two weeks. Earliest that would be out is monday/ tuesday though. Thoughts @jni? What do you think we should do?
But it is still few days when the new installation of napari will be broken if the user does not have `pydantic` already installed. 
I'd vote for releasing a new version of napari to patch this. It took me a while to work out why napari wasn't working in a fresh environment. 
> It took me a while to work out why napari wasn't working in a fresh environment.

Same here, just now. As I was building napari inside a Docker container (something I am not that familiar with) I was debugging my Dockerfile for a while before looking for the issue here.
So +1

this has been fixed upstream, and will presumably be in the next pydantic release: https://github.com/samuelcolvin/pydantic/pull/2421

(not that we shouldn't release with `pydantic<1.8.0`)
When new version of pydantic will be released? 
> When new version of pydantic will be released?

at this point, we don't know
We've started the `0.4.6` release process now with a pin `pydantic<1.8.0`, will have an `rc` to test soon.
Ok `0.4.6rc0` has been released and can be tested with `pip install --pre napari`. Please let us know how that's working on the Zulip release discussion - https://napari.zulipchat.com/#narrow/stream/215289-release/topic/0.2E4.2E6/near/228264146
> We've started the `0.4.6` release process now with a pin `pydantic<1.8.0`, will have an `rc` to test soon.

That won't totally work, pip has a dependency resolver, and might decide to downgrade napari for end users instead of downgrading pydantic.
> That won't totally work, pip has a dependency resolver, and might decide to downgrade napari for end users instead of downgrading pydantic.

Not saying that you should not make a release pinning pydantic to <1.8.0; but that users may end up upgrading pydantic and downgrading napari when manipulating other dependencies, updating plugins.... (had the same issue with IPython/Jedi recently. 


Interesting... is there any better alternative?  Or just something were kinda stuck with?
I would suggest we release with `pydantic!=1.8.0` so that we automatically work with subsequent versions, since the fix is in pydantic master?
> Interesting... is there any better alternative? Or just something we're kinda stuck with?

Yeah, it's something we are stuck with. It's an inherent problem of package manager that don't allow to update package meta informations after the release, it also affects conda, but in a lesser form, as you can hotpatch repo index in conda/conda-forge, and the fact that we can't know the future (yet). 

It's worse in pip/pypi as the info is _in_ the wheel, **and** pip recently changed to a resolver. 

Project that are 1/2 apps 1/2 libraries (like IPython and napari) also have a tough time as the application side push us toward pinning, and the library side toward letting dependencies version float.

See [recent twitter thread](https://twitter.com/Mbussonn/status/1362203038136508416), among others, which is about wether one should pin to python<4 or not.

 @Carreau you're exactly right (here and on twitter), metadata fixed at release time causes so many problems. :facepalm: 
I've updated our `rc` to just avoid `1.8.0`
> @Carreau you're exactly right (here and on twitter), metadata fixed at release time causes so many problems

If ever you think you think a pep is necessary, I'm happy to contribute. If it becomes a long time problem we can try to pull resources and get funding to  try to fix this upstream. 
> this has been fixed upstream, and will presumably be in the next pydantic release: samuelcolvin/pydantic#2421

Trying pydantic-1.8.0 with the fix, napari-0.4.6 still fails to start (on Windows):

```Python-traceback
> py -m napari
Traceback (most recent call last):
  File "X:\Python38\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "X:\Python38\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "X:\Python38\lib\site-packages\napari\__main__.py", line 327, in <module>
    sys.exit(main())
  File "X:\Python38\lib\site-packages\napari\__main__.py", line 323, in main
    _run()
  File "X:\Python38\lib\site-packages\napari\__main__.py", line 249, in _run
    _viewer = view_path(  # noqa: F841
  File "X:\Python38\lib\site-packages\napari\view_layers.py", line 6, in view_path
    function signature is a merged version of one of the ``Viewer.__init__`` method
  File "X:\Python38\lib\site-packages\napari\viewer.py", line 52, in __init__
    self._window = Window(self, show=show)
  File "X:\Python38\lib\site-packages\napari\_qt\qt_main_window.py", line 251, in __init__
    self._add_view_menu()
  File "X:\Python38\lib\site-packages\napari\_qt\qt_main_window.py", line 414, in _add_view_menu
    toggle_theme.triggered.connect(self.qt_viewer.viewer._toggle_theme)
TypeError: unhashable type: 'Viewer'
```

 Reverting to pydantic-1.7.3 works.
thanks @cgohlke.  I just tried with the current pydantic master and napari 0.4.6, but it worked for me on my mac (strange though... doesn't seem this would be an OS specific error).  I can get on this for windows though and we can try to make sure to have a napari release that works for it when it comes out.
Thanks for flagging @cgohlke - is that fixed by https://github.com/samuelcolvin/pydantic/pull/2423?
Yes, using pydantic master works. Sorry for the noise.
> Yes, using pydantic master works. Sorry for the noise.

No worries, great to have the additional confirmation! Always keep your bug reports coming :-)
1.8.1 just came out, not sure if it's breaking `4.6.0`, but i suspect it is breaking master due to

```
  E   TypeError: Object of type 'cycle' is not JSON serializable` error, 
```
failing in the colormanger now see https://github.com/napari/napari/pull/2344/checks?check_run_id=2026313230

cc @tlambert03 @kevinyamauchi 

not sure what could have happened from the changelog https://pydantic-docs.helpmanual.io/changelog/#v181-2021-03-03
Ok, yup I've checked this now, it is the `1.8.1` update, i havn't tried to find the PR yet, but this is the full traceback on master right now 

```python-traceback
_____________________________________________________________________________________ test_init_color_manager_colormap _____________________________________________________________________________________

    def test_init_color_manager_colormap():
        n_colors = 10
        color_cycle = [[0, 0, 0, 1], [1, 1, 1, 1]]
        properties = {'point_type': _make_cycled_properties([0, 1.5], n_colors)}
        color_manager = ColorManager._from_layer_kwargs(
            colors='point_type',
            properties=properties,
            n_colors=n_colors,
            continuous_colormap='gray',
            contrast_limits=None,
            categorical_colormap=color_cycle,
        )

        assert len(color_manager.colors) == n_colors
        assert color_manager.color_mode == 'colormap'
        color_array = transform_color(['black', 'white'] * int(n_colors / 2))
        colors = color_manager.colors.copy()
        np.testing.assert_allclose(colors, color_array)
        np.testing.assert_allclose(color_manager.current_color, [1, 1, 1, 1])
        assert color_manager.color_properties.current_value == 1.5

        # test that colormanager state can be saved and loaded
        cm_dict = color_manager.dict()
        color_manager_2 = ColorManager._from_layer_kwargs(
            colors=cm_dict, properties=properties
        )
        assert color_manager == color_manager_2

        # test json serialization
>       json_str = color_manager.json()

napari/layers/utils/_tests/test_color_manager.py:538:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
pydantic/main.py:551: in pydantic.main.BaseModel.json
    ???
../../opt/anaconda3/lib/python3.7/json/__init__.py:238: in dumps
    **kw).encode(obj)
../../opt/anaconda3/lib/python3.7/json/encoder.py:199: in encode
    chunks = self.iterencode(o, _one_shot=True)
../../opt/anaconda3/lib/python3.7/json/encoder.py:257: in iterencode
    return _iterencode(o, 0)
pydantic/json.py:107: in pydantic.json.custom_pydantic_encoder
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

>   ???
E   TypeError: Object of type 'cycle' is not JSON serializable

pydantic/json.py:95: TypeError
```
Oh now that i think about this, it could have broken in `1.8.0`, but we just hadn't seen it yet!

Resolving patch:
diff --git a/napari/utils/events/evented_model.py b/napari/utils/events/evented_model.py
--- a/napari/utils/events/evented_model.py
+++ b/napari/utils/events/evented_model.py
@@ -84,7 +84,12 @@ def __new__(mcs, name, bases, namespace, **kwargs):
             # NOTE: a _json_encode field must return an object that can be
             # passed to json.dumps ... but it needn't return a string.
             if hasattr(f.type_, '_json_encode'):
-                cls.__config__.json_encoders[f.type_] = f.type_._json_encode
+                encoder = f.type_._json_encode
+                cls.__config__.json_encoders[f.type_] = encoder
+                # also add it to the base config
+                # required for pydantic>=1.8.0 due to:
+                # https://github.com/samuelcolvin/pydantic/pull/2064
+                EventedModel.__config__.json_encoders[f.type_] = encoder
         return cls
 
 


Test patch:
diff --git a/napari/utils/events/_tests/test_evented_model.py b/napari/utils/events/_tests/test_evented_model.py
--- a/napari/utils/events/_tests/test_evented_model.py
+++ b/napari/utils/events/_tests/test_evented_model.py
@@ -284,3 +284,19 @@ class Model(EventedModel):
     assert raw == '{"obj": {"a": 1, "b": "hi"}, "shaped": [1.0, 2.0, 3.0]}'
     deserialized = Model.parse_raw(raw)
     assert deserialized == m
+
+
+def test_nested_evented_model_serialization():
+    """Test that encoders on nested sub-models can be used by top model."""
+
+    class NestedModel(EventedModel):
+        obj: MyObj
+
+    class Model(EventedModel):
+        nest: NestedModel
+
+    m = Model(nest={'obj': {"a": 1, "b": "hi"}})
+    raw = m.json()
+    assert raw == r'{"nest": {"obj": {"a": 1, "b": "hi"}}}'
+    deserialized = Model.parse_raw(raw)
+    assert deserialized == m