Repository: napari/napari

Issue:
Error when returning named shapes/surface layer multiple times (Dock widget)
Running a dock widget multiple times (i.e. pressing the `call_button`) will cause an error with named `shapes` (and also `surface`, not shown) output layers. 

## Code

Minimal dock widget to reproduce the issue.

```python
@magic_factory(call_button=True)
def widget() -> List[napari.types.LayerDataTuple]:
    polygons = [np.array([[11, 13], [111, 113], [22, 246]])]
    # return [(polygons, dict(shape_type='polygon'), 'shapes')] # this works
    return [(polygons, dict(name='Test', shape_type='polygon'), 'shapes')]

@napari_hook_implementation
def napari_experimental_provide_dock_widget():
    return widget
```

![image](https://user-images.githubusercontent.com/231996/110045579-2e6f5700-7d4b-11eb-94ca-6cff8f4b9779.png)

## Expected behavior

Not to throw an error. Would have expected that the layer is either replaced (as happens for `image` and `labels` layers) or  a new layer is added with a name suffix (e.g. `Test [1]`) as happens if the layer is not named.

## Environment

![image](https://user-images.githubusercontent.com/231996/110044142-eea77000-7d48-11eb-930a-ee5d195b1103.png)


Discussion:
Looks like something funky is definitely happening, the error message "three vertices to a rectangle" is relatively deep in the code, not quite sure what would have caused that. We'll look into it.
The reason this happens is because when we receive a `layer_data_tuple` from a widget whose `name` is that of an existing layer in the viewer, we try to overwrite the layer's `data`, instead of adding a new layer. Since `shapes` layers store the type of each shape in the metadata dict, the `shape_type` gets lost, and the data setter defaults to rectangle. See [here](https://github.com/napari/napari/blob/102a7e8f845893c874d2b86f9371d41130100b89/napari/utils/_magicgui.py#L180) for the relevant code.

In the case of `surface`, we don't provide a data setter at all, so this also fails with an `AttributeError: can't set attribute`

**Options**:
1. Special case `shapes` and `surface` - I think this is a bad idea as we'd have to special case anything whose `data` property is not easily settable/requires associated metadata
2. Never replace data, always add new layer
3. Replace the layer itself i.e. remove old layer, add newly passed layer
4. Keep the data setting for all layer types for which it works, pick one of 2. or 3. for other layer types

@jni  @sofroniewn @tlambert03 thoughts on preferred option? Any other options I haven't thought of?
Hmm, this is a tricky one, I'm not sure. It's not ideal that the data property here isn't a complete descriptor. I will have to think on it, but thanks for the clear description!
@sofroniewn I agree it's a deeper problem, but I think Option 4 is a nice, non-intrusive way to fix this specific bug while we think about improving our data models:

```python
try:
    set_data()
except:
    replace_whole_layer()
```

As I understand the only downside to replacing the whole layer always is performance, right?
I think this could actually be fixed by changing it here https://github.com/napari/napari/blob/767f4bb2282f1ee965a041992d4ab801c3a6cc44/napari/layers/shapes/shapes.py#L537-L545

to be 

```python
    @data.setter
    def data(self, data, shape_type=None):
        self._finish_drawing()
        self._data_view = ShapeList()
        if shape_type is None:
              shape_type = self.shape_type # or getattr(self, shape_type, 'rectangle')
        self.add(data, shape_type=shape_type)
```

Would that work @DragaDoncila? I think it is better behavior independent of the dockwidget stuff
Ah it is maybe a little more complex if the length of data and shape_type is changing, then we need to pad/ trim - but I think we do that elsewhere https://github.com/napari/napari/blob/master/napari/layers/shapes/shapes.py#L537-L545

I can look into this more later, but i think we can fix this inside `shapes.py`
Yeah I was going to say that knowing the current `shape_type` is insufficient. Also we have to keep in mind that if we fix it inside shapes we'll have to also add a data setter for `surfaces` (which maybe we want to do anyway) in order to keep this consistent. We'd also have to make sure that any new layer types we add support setting data exclusively through the data property without any associated metadata.
> Also we have to keep in mind that if we fix it inside shapes we'll have to also add a data setter for surfaces (which maybe we want to do anyway) in order to keep this consistent. We'd also have to make sure that any new layer types we add support setting data exclusively through the data property without any associated metadata.

I still think this is a good idea, we do this for the Points layer, though the setter is quite a bit more complex https://github.com/napari/napari/blob/767f4bb2282f1ee965a041992d4ab801c3a6cc44/napari/layers/points/points.py#L410-L474

I think we should change the shapes setter to be more like this. Drop the optional `shape_type`, and then if the number of shapes isn't changing just replace the actual data (keep shape type/ colors etc the same), if the number of shapes is decreasing trim those other lists from the end, if the number of shapes is growing use the `current_*` property if we have one or the next best sensible default.

How does that sound as reasonable behaviour? If we fix that then this will be fixed for shapes. We can then look into fixing for surface.
> Drop the optional shape_type, and then if the number of shapes isn't changing just replace the actual data (keep shape type/ colors etc the same), if the number of shapes is decreasing trim those other lists from the end, if the number of shapes is growing use the current_* property if we have one or the next best sensible default.

@sofroniewn I think those are all worse band-aids than option 4. The problem, as you pointed out, is that data is not a complete descriptor of the data. So if someone wants to change the number of shapes *and* change the shape types at the same time, we'll still get an error, no?
> I think those are all worse band-aids than option 4. 

the problem with option 4 is i can imagine times when it won't error, but we'll do wrong thing. It's also making a big change to our dock widget return type handling when I think the problem is with the shapes layer data model.

> if someone wants to change the number of shapes and change the shape types at the same time, we'll still get an error, no

We could also optionally let data be a tuple (array_data, shape_type), which get separated into their respective components in the setter. I think this would solve that use case for people that need it. Thoughts?
> Drop the optional shape_type, and then if the number of shapes isn't changing just replace the actual data (keep shape type/ colors etc the same), if the number of shapes is decreasing trim those other lists from the end, if the number of shapes is growing use the current_* property if we have one or the next best sensible default.

I'm not against this, but I think I am a little confused. If I have three shapes in my layer - a polygon, a rectangle and an ellipse, then 

`layer.shape_type = ['polygon', 'rectangle', 'ellipse']`

> if the number of shapes isn't changing just replace the actual data (keep shape type/ colors etc the same)

If the data I passed in didn't actually line up with the shape type e.g. I pass in three triangle polygons, then wouldn't we run into the same issue with this approach i.e. `rectangle expects 4 corner vertices, 3 provided`?

>  if the number of shapes is growing use the current_* property if we have one or the next best sensible default

Is the suggestion that we try to introspect the shape type from the data? Because otherwise I'm not sure what "the next best sensible default" is that wouldn't lead us into the same spot we are now.

> We could also optionally let data be a tuple (array_data, shape_type)

This seems like the closest approach to what we actually want, since I don't think there's a sane way to dissociate the vertices of the shape from the type of the shape we're adding e.g. 4 vertices could be a rectangle or an ellipse. 

Maybe I'm misunderstanding something though, I don't have a lot of experience with Shapes layers.
> > We could also optionally let data be a tuple (array_data, shape_type)
>
> This seems like the closest approach to what we actually want, since I don't think there's a sane way to dissociate the vertices of the shape from the type of the shape we're adding e.g. 4 vertices could be a rectangle or an ellipse.
>
> Maybe I'm misunderstanding something though, I don't have a lot of experience with Shapes layers.

I think your summary was good @DragaDoncila, I think we might need this approach. We could also change the default type to `polygon` which is a little more forgiving then `rectangle`, but that might cause more problems for people. What does @jni think about the latest discussion?
I'm just starting to think that we should change the shape type to be an object for each shape. As we've said above, the fundamental issue is that 'data' doesn't contain the full information needed to define a shape. I think I was probably one of the early detractors of the "let's make a special object" model, but in this case it was probably the wrong call. 'data' should contain the minimal information to draw *something*, and that's currently broken in Shapes. The tuple is "ok" but it still feels a bit off in that the shapes and the "shapedata" (points) are separated.

ie, maybe if we decide *not* to go for a full object, we could go for data being a list of (points, datatype) tuples? I think that is the least intrusive option while still maintaining data/shapetype parity.
We do have an internal shape object already (right now is also contains the dims to be displayed, which we'll want to change) https://github.com/napari/napari/blob/master/napari/layers/shapes/_shapes_models/shape.py

> ie, maybe if we decide not to go for a full object, we could go for data being a list of (points, datatype) tuples? I think that is the least intrusive option while still maintaining data/shapetype parity.

I was imagining `Array | Tuple(Array, str) | List[Array | Tuple(Array, str)] | Tuple(List[Array], str)` as the full set of allowed inputs. I think this would be the most minimal and backwards compatible change we could make. Currently we support `Array | List[Array]` and I don't think we'd want to loose that.

I also think most users will have a single shape type per layer most of the time, and we don't want to make things unnecessarily hard for them
> I was imagining Array | Tuple(Array, str) | List[Array | Tuple(Array, str)] | Tuple(List[Array], str) as the full set of allowed inputs.

One thing to consider is does that mean we're removing the `shape_type` from the `metadata`? Because otherwise if we're going to support it in both places I can definitely see sources of confusion and/or inconsistent behaviour there.
> One thing to consider is does that mean we're removing the shape_type from the metadata? Because otherwise if we're going to support it in both places I can definitely see sources of confusion and/or inconsistent behaviour there.

I think i'd probably keep it, and inside the data setter split out the shape type so that `layer.shape_type` is always returns `List[str]` and `layer.data` always returns `List[Array]`. If something is passed to `layer.data` then it overwrites anything in `layer.shape_type`.

Note in the motivating example above you'd need to now do

```python
def widget() -> List[napari.types.LayerDataTuple]:
    polygons = [np.array([[11, 13], [111, 113], [22, 246]])]
    # return [(polygons, dict(shape_type='polygon'), 'shapes')] # this works
    # return [(polygons, dict(name='Test', shape_type='polygon'), 'shapes')] # this still wouldn't work
    return [((polygons, 'polygon'), dict(name='Test'), 'shapes')] # this would now work
```

which doesn't look too bad to me ...

Another thought is that we should swap the order of line 180 and the for loop lines (181, 182) here https://github.com/napari/napari/blob/102a7e8f845893c874d2b86f9371d41130100b89/napari/utils/_magicgui.py#L180

and then if we additionally go with the approach I propose above

>  if the number of shapes isn't changing just replace the actual data (keep shape type/ colors etc the same), if the number of shapes is decreasing trim those other lists from the end, if the number of shapes is growing use the current_* property if we have one or the next best sensible default.

and the optional tuple, then  `return [(polygons, dict(name='Test', shape_type='polygon'), 'shapes')]` will just work without extra thought, and all the special cases can be handled.

I know this discussion has gotten quite involved now, but I do see a way forward that will fix this use case, allow for additional flexibility, and be mostly backwards compatible
> If something is passed to layer.data then it overwrites anything in layer.shape_type

I like this, I think it makes the most sense. 

I think this is a reasonable approach to fix the use case and provide more flexibility overall, and I think the 
`List[Tuple(Array, str)]` is quite readable as well. @jni ?
all clear from me! 😊 
did #2544 close this?
> did #2544 close this?

Yes I think so, combined with #2507 
Sorry, I'm still having an issue when I additionally set `edge_width` (napari 0.4.8rc2, magicgui 0.2.8.dev11+g260f63e):

```python
@magic_factory(call_button=True)
def widget() -> List[napari.types.LayerDataTuple]:
    polygons = [np.array([[11, 13], [111, 113], [22, 246]])]
    return [(polygons, dict(name='Test', shape_type='polygon', edge_width=0.5), 'shapes')]

@napari_hook_implementation
def napari_experimental_provide_dock_widget():
    return widget
```

```python
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
~/research/stardist/napari/magicgui/magicgui/widgets/_bases/value_widget.py in _on_value_change(self=PushButton(value=False, annotation=None, name='call_button'), *args=(False,))
     45     def _on_value_change(self, *args):
     46         """Called when the widget value changes.  args come from the widget itself."""
---> 47         self.changed(value=args[0] if args else None)
        self.changed = <magicgui.events.EventEmitter object at 0x7fceb6f2e9a0>
        global value = undefined
        args = (False,)
     48 
     49     def get_value(self):

~/research/stardist/napari/magicgui/magicgui/events.py in __call__(self=<magicgui.events.EventEmitter object>, *args=(), **kwargs={'value': False})
    654                     continue
    655 
--> 656                 self._invoke_callback(cb, event)
        self._invoke_callback = <bound method EventEmitter._invoke_callback of <magicgui.events.EventEmitter object at 0x7fceb6f2e9a0>>
        cb = <function FunctionGui.__init__.<locals>._disable_button_and_call at 0x7fceb6ed74c0>
        event = <Event blocked=False handled=False native=None source=None sources=[] type=changed>
    657                 if event.blocked:
    658                     break

~/research/stardist/napari/magicgui/magicgui/events.py in _invoke_callback(self=<magicgui.events.EventEmitter object>, cb=<function FunctionGui.__init__.<locals>._disable_button_and_call>, event=<Event blocked=False handled=False native=None source=None sources=[] type=changed>)
    675             cb(event)
    676         except Exception:
--> 677             _handle_exception(
        global _handle_exception = <function _handle_exception at 0x7fcf775b1ca0>
        self.ignore_callback_errors = False
        self.print_callback_errors = 'reminders'
        self = <magicgui.events.EventEmitter object at 0x7fceb6f2e9a0>
        global cb_event = undefined
        cb = <function FunctionGui.__init__.<locals>._disable_button_and_call at 0x7fceb6ed74c0>
        event = <Event blocked=False handled=False native=None source=None sources=[] type=changed>
    678                 self.ignore_callback_errors,
    679                 self.print_callback_errors,

~/research/stardist/napari/magicgui/magicgui/events.py in _handle_exception(ignore_callback_errors=False, print_callback_errors='reminders', obj=<magicgui.events.EventEmitter object>, cb_event=(<function FunctionGui.__init__.<locals>._disable_button_and_call>, <Event blocked=False handled=False native=None source=None sources=[] type=changed>), node=None)
    139     # Handle
    140     if value and not ignore_callback_errors:
--> 141         raise value
        value = AttributeError("can't set attribute")
    142     if print_callback_errors != "never":
    143         this_print: Optional[str] = "full"

~/research/stardist/napari/magicgui/magicgui/events.py in _invoke_callback(self=<magicgui.events.EventEmitter object>, cb=<function FunctionGui.__init__.<locals>._disable_button_and_call>, event=<Event blocked=False handled=False native=None source=None sources=[] type=changed>)
    673     def _invoke_callback(self, cb: Callback, event: Event):
    674         try:
--> 675             cb(event)
        cb = <function FunctionGui.__init__.<locals>._disable_button_and_call at 0x7fceb6ed74c0>
        event = <Event blocked=False handled=False native=None source=None sources=[] type=changed>
    676         except Exception:
    677             _handle_exception(

~/research/stardist/napari/magicgui/magicgui/widgets/_function_gui.py in _disable_button_and_call(val=<Event blocked=False handled=False native=None source=None sources=[] type=changed>)
    185                     self._call_button.enabled = False
    186                     try:
--> 187                         self.__call__()
        global self.__call__ = undefined
    188                     finally:
    189                         self._call_button.enabled = True

~/research/stardist/napari/magicgui/magicgui/widgets/_function_gui.py in __call__(self=<FunctionGui widget() -> List[LayerDataTuple]>, *args=(), **kwargs={})
    295 
    296             for callback in _type2callback(return_type):
--> 297                 callback(self, value, return_type)
        callback = <function add_layer_data_tuples_to_viewer at 0x7fcf774fc5e0>
        self = <FunctionGui widget() -> List[LayerDataTuple]>
        value = [([array([[ 11,  13],
       [111, 113],
       [ 22, 246]])], {'name': 'Test', 'shape_type': 'polygon', 'edge_width': 0.5}, 'shapes')]
        return_type = typing.List[LayerDataTuple]
    298         self.called(value=value)
    299         return value

~/sw/miniconda3/envs/napari-dev/lib/python3.8/site-packages/napari/utils/_magicgui.py in add_layer_data_tuples_to_viewer(gui=<FunctionGui widget() -> List[LayerDataTuple]>, result=[([array([[ 11,  13],
       [111, 113],
       [ 22, 246]])], {'edge_width': 0.5, 'name': 'Test', 'shape_type': 'polygon'}, 'shapes')], return_type=typing.List[LayerDataTuple])
    195                     layer.data = layer_datum[0]
    196                     for k, v in layer_datum[1].items():
--> 197                         setattr(layer, k, v)
        global setattr = undefined
        layer = <Shapes layer 'Test' at 0x7fceb6eedd30>
        k = 'edge_width'
        v = 0.5
    198                     continue
    199                 except KeyError:  # layer not in the viewer

AttributeError: can't set attribute
```
@DragaDoncila can you check this out again?
@sofroniewn on it now

Resolving patch:
diff --git a/napari/layers/shapes/shapes.py b/napari/layers/shapes/shapes.py
--- a/napari/layers/shapes/shapes.py
+++ b/napari/layers/shapes/shapes.py
@@ -987,11 +987,59 @@ def edge_width(self):
         """list of float: edge width for each shape."""
         return self._data_view.edge_widths
 
+    @edge_width.setter
+    def edge_width(self, width):
+        """Set edge width of shapes using float or list of float.
+
+        If list of float, must be of equal length to n shapes
+
+        Parameters
+        ----------
+        width : float or list of float
+            width of all shapes, or each shape if list
+        """
+        if isinstance(width, list):
+            if not len(width) == self.nshapes:
+                raise ValueError(
+                    trans._('Length of list does not match number of shapes')
+                )
+            else:
+                widths = width
+        else:
+            widths = [width for _ in range(self.nshapes)]
+
+        for i, width in enumerate(widths):
+            self._data_view.update_edge_width(i, width)
+
     @property
     def z_index(self):
         """list of int: z_index for each shape."""
         return self._data_view.z_indices
 
+    @z_index.setter
+    def z_index(self, z_index):
+        """Set z_index of shape using either int or list of int.
+
+        When list of int is provided, must be of equal length to n shapes.
+
+        Parameters
+        ----------
+        z_index : int or list of int
+            z-index of shapes
+        """
+        if isinstance(z_index, list):
+            if not len(z_index) == self.nshapes:
+                raise ValueError(
+                    trans._('Length of list does not match number of shapes')
+                )
+            else:
+                z_indices = z_index
+        else:
+            z_indices = [z_index for _ in range(self.nshapes)]
+
+        for i, z_idx in enumerate(z_indices):
+            self._data_view.update_z_index(i, z_idx)
+
     @property
     def selected_data(self):
         """set: set of currently selected shapes."""


Test patch:
diff --git a/napari/layers/shapes/_tests/test_shapes.py b/napari/layers/shapes/_tests/test_shapes.py
--- a/napari/layers/shapes/_tests/test_shapes.py
+++ b/napari/layers/shapes/_tests/test_shapes.py
@@ -1618,6 +1618,20 @@ def test_edge_width():
     assert len(layer.edge_width) == shape[0] - 1
     assert layer.edge_width == [width_list[1]] + width_list[3:] + [4]
 
+    # Test setting edge width with number
+    layer.edge_width = 4
+    assert all([width == 4 for width in layer.edge_width])
+
+    # Test setting edge width with list
+    new_widths = [2] * 5 + [3] * 4
+    layer.edge_width = new_widths
+    assert layer.edge_width == new_widths
+
+    # Test setting with incorrect size list throws error
+    new_widths = [2, 3]
+    with pytest.raises(ValueError):
+        layer.edge_width = new_widths
+
 
 def test_z_index():
     """Test setting z-index during instantiation."""
@@ -1649,6 +1663,20 @@ def test_z_index():
     assert len(layer.z_index) == shape[0] - 1
     assert layer.z_index == [z_index_list[1]] + z_index_list[3:] + [4]
 
+    # Test setting index with number
+    layer.z_index = 4
+    assert all([idx == 4 for idx in layer.z_index])
+
+    # Test setting index with list
+    new_z_indices = [2] * 5 + [3] * 4
+    layer.z_index = new_z_indices
+    assert layer.z_index == new_z_indices
+
+    # Test setting with incorrect size list throws error
+    new_z_indices = [2, 3]
+    with pytest.raises(ValueError):
+        layer.z_index = new_z_indices
+
 
 def test_move_to_front():
     """Test moving shapes to front."""