Repository: open-mmlab/mmsegmentation

Issue:
Precarious label map creation when using custom classes
When using custom classes, [the label map is created in this way](https://github.com/open-mmlab/mmsegmentation/blob/b997a13e28da5e5d4eb7639edd749c20b23b3807/mmseg/datasets/custom.py#L325-L333) (copied below). This code fragment is fragile because missing classes are mapped to "-1", but the "-1" is never handled properly.

```python
# dictionary, its keys are the old label ids and its values
# are the new label ids.
# used for changing pixel labels in load_annotations.
self.label_map = {}
for i, c in enumerate(self.CLASSES):
    if c not in class_names:
        self.label_map[i] = -1
    else:
        self.label_map[i] = class_names.index(c)
```

I've tried to understand how this "-1" label is handled when training. The expected behavior is that this "-1" label should be ignored. However, this behavior is achieved very indirectly and precariously (prone to bugs):

In [loading.py](https://github.com/open-mmlab/mmsegmentation/blob/b997a13e28da5e5d4eb7639edd749c20b23b3807/mmseg/datasets/pipelines/loading.py):

```python
gt_semantic_seg = mmcv.imfrombytes(
    img_bytes, flag='unchanged',
    backend=self.imdecode_backend).squeeze().astype(np.uint8)
```

`gt_semantic_seg` is created with type `uint8`. Then,

```python
if results.get('label_map', None) is not None:
    for old_id, new_id in results['label_map'].items():
        gt_semantic_seg[gt_semantic_seg == old_id] = new_id
```
When `new_id` is -1, uint8 **underflows** to 255. Then, since the default `ignore_index=255` in [custom.py](https://github.com/open-mmlab/mmsegmentation/blob/b997a13e28da5e5d4eb7639edd749c20b23b3807/mmseg/datasets/custom.py), the "-1" label is successfully ignored.

My issue is that this process is not explicit and can be prone to bugs. For example, the uint8 masks are eventually converted to `torch.Tensor` of type `int64`, which can contain negative values. The code only works just because an intermediate step happened to use uint8 and the underflow occurred.

I think the code should be improved and the "-1" label should be handled more explicitly, by, for example, always ignoring negative labels in addition to ignoring the (optional) `ignore_index` label.



Discussion:
Thanks for your bug report, and for other high-quality PR from you. 👍 

We also found it is not robust, and modify it in [MMSegmentation 1.x
](https://github.com/open-mmlab/mmsegmentation/releases/tag/v1.0.0rc0)
https://github.com/open-mmlab/mmsegmentation/blob/a115b103237c813197fe4b6f8e3c510afd38b92b/mmseg/datasets/basesegdataset.py#L181-L185

In new mmseg, we give up the unstable **underflows**, just set the default ignore value 255 in decode head of mmseg model.

Do you have any comments about this modification? Are there any potential bugs we haven't thought of?
Thanks for your prompt replies!

>Do you have any comments about this modification? Are there any potential bugs we haven't thought of?

Thanks for pointing out that this has been fixed in 1.x. I think for now, this is an okay fix, but it can be improved. 

Currently, at various places in both the master and 1.x branches, `ignore_index` defaults sometime to 255 ([see here](https://github.com/open-mmlab/mmsegmentation/blob/1.x/mmseg/models/losses/dice_loss.py#L19)), -100 ([see here](https://github.com/open-mmlab/mmsegmentation/blob/1.x/mmseg/models/losses/cross_entropy_loss.py#L18)), `None` ([see here](https://github.com/open-mmlab/mmsegmentation/blob/1.x/mmseg/models/losses/accuracy.py#L6)), etc. This could lead to a potential bug. For example, if someone wants to ignore the index 1 for some reason and sets `ignore_index=1`, then 255 will not be ignored. However, logically, the label `255` should **always** be ignored regardless of which other labels are being ignored, because `reduce_zero_label` and `label_map` assume that 255 will be ignored. I think there can be two solutions to deal with this:
1. Just remove the `ignore_index` option; an index should only be ignored if (a) `reduce_zero_label` is set to `True` or (2) labels are remapped via `label_map` or (3) a subset of the dataset's `classes` are specified; all these options use 255 to ignore indices anyway. I don't see any other use cases of explicitly setting the `ignore_index` so it might be better to just remove it and reduce the chance of bugs.
2. Keep `ignore_index`, but make it a list (or better yet, a set). Always insert 255 into the set along with potentially any other labels.

I think this is not urgent; the current fix you have works for now.

I had no idea that a 1.x branch existed! Thanks for pointing it out. I have a few questions:

1. Do you recommend switching from `master` to `1.x` at this point?
2. When will `1.x` be released?
3. When it does release, will all the bug fixes in the current `master` (e.g. #2500, #2504) be inherited?

In the meantime, I'll create a PR for `master` to set `label_map[i] = 255`.

Resolving patch:
diff --git a/mmseg/datasets/custom.py b/mmseg/datasets/custom.py
--- a/mmseg/datasets/custom.py
+++ b/mmseg/datasets/custom.py
@@ -349,7 +349,7 @@ def get_classes_and_palette(self, classes=None, palette=None):
             self.label_map = {}
             for i, c in enumerate(self.CLASSES):
                 if c not in class_names:
-                    self.label_map[i] = -1
+                    self.label_map[i] = 255
                 else:
                     self.label_map[i] = class_names.index(c)
 
@@ -364,7 +364,7 @@ def get_palette_for_custom_classes(self, class_names, palette=None):
             palette = []
             for old_id, new_id in sorted(
                     self.label_map.items(), key=lambda x: x[1]):
-                if new_id != -1:
+                if new_id != 255:
                     palette.append(self.PALETTE[old_id])
             palette = type(self.PALETTE)(palette)
 


Test patch:
diff --git a/tests/test_data/test_loading.py b/tests/test_data/test_loading.py
--- a/tests/test_data/test_loading.py
+++ b/tests/test_data/test_loading.py
@@ -187,7 +187,7 @@ def test_load_seg_custom_classes(self):
             # classes=["A", "C", "D"] which removes class "B".
             label_map={
                 0: 0,
-                1: -1,  # simulate removing class 1
+                1: 255,  # simulate removing class 1
                 2: 1,
                 3: 2
             },
@@ -204,7 +204,7 @@ def test_load_seg_custom_classes(self):
 
         true_mask = np.ones_like(gt_array) * 255  # all zeros get mapped to 255
         true_mask[2:4, 2:4] = 0  # 1s are reduced to class 0 mapped to class 0
-        true_mask[2:4, 6:8] = -1  # 2s are reduced to class 1 which is removed
+        true_mask[2:4, 6:8] = 255  # 2s are reduced to class 1 which is removed
         true_mask[6:8, 2:4] = 1  # 3s are reduced to class 2 mapped to class 1
         true_mask[6:8, 6:8] = 2  # 4s are reduced to class 3 mapped to class 2