Repository: napari/napari

Issue:
Refactored properties raise key error if index is set in dataframe
## 🐛 Bug

<!-- A clear and concise description of what the bug is. -->

## To Reproduce

Steps to reproduce the behavior:

This test passes on 0.4.1.2 but fails on the current main (this is short reproduction of code from PartSeg):

```python
def test_get_status():
    data = np.zeros((10, 10), dtype=np.uint8)
    data[2:5, 2:-2] = 1
    data[5:-2, 2:-2] = 2
    layer = Labels(data)
    layer.properties = pd.DataFrame({"text1": [1, 3], "text2": [7, -2], "index" : [1, 2]}, index=[1, 2])
    assert layer.get_status((0, 0)) == "Labels [0 0]: 0; [No Properties]"
    assert layer.get_status((3, 3)) == "Labels [3 3]: 1; text1: 1, text2: 7"
    assert layer.get_status((6, 6)) == "Labels [6 6]: 2; text1: 3, text2: -2"
```

<!-- If you have a code sample, error messages, stack traces, please provide it here as well -->

```
napari/layers/labels/_tests/test_labels.py::test_get_status FAILED       [100%]
napari/layers/labels/_tests/test_labels.py:1347 (test_get_status)
def test_get_status():
        data = np.zeros((10, 10), dtype=np.uint8)
        data[2:5, 2:-2] = 1
        data[5:-2, 2:-2] = 2
        layer = Labels(data)
        layer.properties = pd.DataFrame({"text1": [1, 3], "text2": [7, -2], "index" : [1, 2]}, index=[1, 2])
        assert layer.get_status((0, 0)) == "Labels [0 0]: 0; [No Properties]"
>       assert layer.get_status((3, 3)) == "Labels [3 3]: 1; text1: 1, text2: 7"

napari/layers/labels/_tests/test_labels.py:1355: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
napari/layers/labels/labels.py:1345: in get_status
    properties = self._get_properties(
napari/layers/labels/labels.py:1415: in _get_properties
    return [
napari/layers/labels/labels.py:1420: in <listcomp>
    and v[idx] is not None
../../.pyenv/versions/napari/lib/python3.8/site-packages/pandas/core/series.py:871: in __getitem__
    result = self.index.get_value(self, key)
../../.pyenv/versions/napari/lib/python3.8/site-packages/pandas/core/indexes/base.py:4405: in get_value
    return self._engine.get_value(s, k, tz=getattr(series.dtype, "tz", None))
pandas/_libs/index.pyx:80: in pandas._libs.index.IndexEngine.get_value
    ???
pandas/_libs/index.pyx:90: in pandas._libs.index.IndexEngine.get_value
    ???
pandas/_libs/index.pyx:138: in pandas._libs.index.IndexEngine.get_loc
    ???
pandas/_libs/hashtable_class_helper.pxi:998: in pandas._libs.hashtable.Int64HashTable.get_item
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   ???
E   KeyError: 0

pandas/_libs/hashtable_class_helper.pxi:1005: KeyError
```

## Expected behavior
The text, may change, but the code should work, or at least not end with an error. 

<!-- A clear and concise description of what you expected to happen. -->

## Environment

last main

## Additional context

<!-- Add any other context about the problem here. -->



Discussion:
Previously `dataframe_to_properties` would ignore the index of the passed `DataFrame` when generating the `dict[str, np.ndarray]`. 

In all layer types except `Labels`, [we explicitly reset the index to be the default range-index](https://github.com/napari/napari/blob/12b8656d5f387ba6d7d5e5c44156171231dfa4fd/napari/layers/utils/layer_utils.py#L718) based on the number of elements in the layer (e.g. the number of points). But for `Labels`, we don't know how many elements (i.e. unique label values) there are, so we don't do that and in this test case we retain the given index.

I see a couple of potential fixes for this.
1. Replace `[]` with `iloc[]` in `Labels._get_properties` and everywhere else in `Labels` (and maybe other layers too).
2. Always reset the index to the default one when validating/coercing the given `DataFrame`.

I lean towards option (2) for a few reasons.
- Row names (e.g. a non-default pandas index) are not supported by [the current DataFrame protocol](https://data-apis.org/dataframe-protocol/latest/API.html#design-decisions).
- We support appends to the features table (e.g. adding a new point adds a new row to it), so if the existing index is not the default one, then we'll end up with a messy index with possible repeats. This is one reason we reset the index when removing rows too.
- It's most similar to the previous behavior.
- Forgetting to use `iloc[]` instead of the plain `[]` may cause future bugs.

That being said, we don't need to rely on the index anywhere, so we could just be careful and use `iloc[]`. But my suggestion would be try the more restrictive option (2), and only add the ability to retain the index later if it's a strong need/desire for users.

@Czaki : curious to hear your thoughts. Also adding @jni 
If the reset index will not affect the source data frame then I'm ok with this solution. 
Yep, the way I'm thinking about resetting the index is *not* using `inplace=True`, so it does not affect the source data frame. For example:

```python
In [20]: df = pd.DataFrame({"text1": [1, 3], "text2": [7, -2], "index": [1, 2]}, index=[1, 2])

In [21]: df
Out[21]: 
   text1  text2  index
1      1      7      1
2      3     -2      2

In [22]: df2 = df.reset_index(drop=True)

In [23]: df2
Out[23]: 
   text1  text2  index
0      1      7      1
1      3     -2      2
```

That means there are some potentially non-performant copies occurring on validation/coercion, but I think that's OK for now.
And what do you think about using dataframe index if the index column is missed? 
> And what do you think about using dataframe index if the index column is missed?

Initially, I'd prefer not to rely on `DataFrame.index` publicly at all, mostly because the proposed dataframe protocol does not support row names/labels.

That being said, for `Labels`, it would be pretty nice to have `features.index` store the label values and also act as indices, thereby removing the need for `Labels._label_index`. So I'm happy to think a bit more about this, and expand what we support later.

Resolving patch:
diff --git a/napari/layers/utils/layer_utils.py b/napari/layers/utils/layer_utils.py
--- a/napari/layers/utils/layer_utils.py
+++ b/napari/layers/utils/layer_utils.py
@@ -695,7 +695,9 @@ def _validate_features(
     ----------
     features : Union[Dict[str, np.ndarray], pd.DataFrame]
         The features table input, which will be passed to the pandas
-        DataFrame initializer.
+        DataFrame initializer. If this is a pandas DataFrame with a
+        non-default index, that index (except its length) will be
+        ignored.
     num_data : Optional[int]
         The number of the elements in the layer calling this, such as
         the number of points, which is used to check that the features
@@ -705,9 +707,8 @@ def _validate_features(
     Returns
     -------
     pd.DataFrame
-        The pandas DataFrame created from the input features table.
-        If the input features are already a DataFrame, the data will not
-        be copied, otherwise they will.
+        The pandas DataFrame created from the input features table. This
+        may or may not store a copy of the input data.
 
     Raises
     ------
@@ -715,6 +716,8 @@ def _validate_features(
         If the input feature columns are not all the same length, or if
         that length is not equal to the given num_data.
     """
+    if isinstance(features, pd.DataFrame):
+        features = features.reset_index(drop=True)
     index = None if num_data is None else range(num_data)
     return pd.DataFrame(data=features, index=index)
 


Test patch:
diff --git a/napari/layers/labels/_tests/test_labels.py b/napari/layers/labels/_tests/test_labels.py
--- a/napari/layers/labels/_tests/test_labels.py
+++ b/napari/layers/labels/_tests/test_labels.py
@@ -1344,3 +1344,18 @@ def test_negative_label_doesnt_flicker():
     # the label colors. Now -1 is seen so it is taken into account in the
     # indexing calculation, and changes color
     assert tuple(layer.get_color(-1)) == minus_one_color_original
+
+
+def test_get_status_with_custom_index():
+    """See https://github.com/napari/napari/issues/3811"""
+    data = np.zeros((10, 10), dtype=np.uint8)
+    data[2:5, 2:-2] = 1
+    data[5:-2, 2:-2] = 2
+    layer = Labels(data)
+    df = pd.DataFrame(
+        {'text1': [1, 3], 'text2': [7, -2], 'index': [1, 2]}, index=[1, 2]
+    )
+    layer.properties = df
+    assert layer.get_status((0, 0)) == 'Labels [0 0]: 0; [No Properties]'
+    assert layer.get_status((3, 3)) == 'Labels [3 3]: 1; text1: 1, text2: 7'
+    assert layer.get_status((6, 6)) == 'Labels [6 6]: 2; text1: 3, text2: -2'
diff --git a/napari/layers/utils/_tests/test_layer_utils.py b/napari/layers/utils/_tests/test_layer_utils.py
--- a/napari/layers/utils/_tests/test_layer_utils.py
+++ b/napari/layers/utils/_tests/test_layer_utils.py
@@ -432,3 +432,17 @@ def test_remove_features():
         new_features['confidence'],
         [0.2, 1],
     )
+
+
+def test_validate_features_with_custom_index_without_num_data():
+    input = pd.DataFrame({'a': [1, 3], 'b': [7.5, -2.1]}, index=[1, 2])
+    actual = _validate_features(input)
+    expected = input.reset_index(drop=True)
+    pd.testing.assert_frame_equal(actual, expected)
+
+
+def test_validate_features_with_custom_index_and_num_data():
+    input = pd.DataFrame({'a': [1, 3], 'b': [7.5, -2.1]}, index=[1, 2])
+    actual = _validate_features(input, num_data=2)
+    expected = input.reset_index(drop=True)
+    pd.testing.assert_frame_equal(actual, expected)