One of the class merged with Ignore class while training semantic segmentation

Shashank_Shekhar: <p>While training semantic segmentation  i had 23 classes including Ignore label, during my prediction i get only 22 classes. I have used <strong>cross entropy loss</strong> and provided <strong>ignore index as 255</strong>.</p>
<p>What looks from the predicted image is one of the class <strong>(class_name:vegetation, id:21)</strong>  merge with ignore index as seen in the below image.</p>
<p><strong>Below image shows the Unique values for Ground truth and Prediction along with Original, GT  &amp; Predicted image.</strong></p>
<p>Please help with what may have caused this!!!</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/a/7a8b44a08f9f1b78e4e78213f4665cce28737118.png" data-download-href="https://discuss.pytorch.org/uploads/default/7a8b44a08f9f1b78e4e78213f4665cce28737118" title="Screenshot from 2020-01-10 18-08-21"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/a/7a8b44a08f9f1b78e4e78213f4665cce28737118_2_690x200.png" alt="Screenshot from 2020-01-10 18-08-21" data-base62-sha1="hu4DaU8Ri7gKDTbdHteLCgNkBte" width="690" height="200" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/a/7a8b44a08f9f1b78e4e78213f4665cce28737118_2_690x200.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/a/7a8b44a08f9f1b78e4e78213f4665cce28737118_2_1035x300.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/7/a/7a8b44a08f9f1b78e4e78213f4665cce28737118.png 2x" data-dominant-color="DAD1D3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2020-01-10 18-08-21</span><span class="informations">1154×336 115 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>What is the shape of your output activation?<br>
In a segmentation use case, your output activation will most likely have the shape <code>[batch_size, nb_classes, height, width]</code>.<br>
If <code>nb_classes</code> was defined as 22, you will only get these labels.</p>

Shashank_Shekhar: <p>Thanks for replying <a class="mention" href="/u/ptrblck">@ptrblck</a> , I am using deeplab v3+ architecture and my segmentation output shape is in <strong>[batch_size, nb_classes, height, width]</strong> format. I just printed out the unique values to check whether ignore index is their or not.</p>
<p>During training output tensor is of shape [batch, 23, 512, 512] after that i calculated the labels using <code>_, pred = torch.max(scores, dim=1)</code> .List of unique values of pred contain <strong>22 (which is ignore index)</strong>,</p>
<p>But at inference time when i use the trained weights, output is of shape [batch, 23, 512, 512] and the the unique values contains <strong>max label upto 21</strong>.</p>

ptrblck: <p>It seems that the model output contains the background class + all other 22 classes.<br>
If I understand your work flow correctly, you have passed the background class index as <code>ignore_index</code>, so that your model never learned how to predict the background?</p>
<p>If that’s the case, I would expect the model never to output the background class during inference either.</p>

Shashank_Shekhar: <p>FYI i am using appoloscapes dataset, below are the labels available and in the trainID column <strong>we don’t have any classes for background</strong>. So i am using <strong>ignore index + 22 other classes</strong>.</p>
<p>For the architecture, loss i am passing below argument:</p>
<pre><code class="lang-auto">model = DeepLab (num_classes=23,backbone=resnet101,output_stride=16)
criterion = nn.CrossEntropyLoss(ignore_index=255)

</code></pre>
<pre><code class="lang-auto">labels = [
    #     name                    clsId    id   trainId   category  catId  hasInstanceignoreInEval   color
    Label('others'              ,    0 ,    0,   255   , '其他'    ,   0  ,False , True  , 0x000000 ),
    Label('rover'               , 0x01 ,    1,   255   , '其他'    ,   0  ,False , True  , 0X000000 ),
    Label('sky'                 , 0x11 ,   17,    0    , '天空'    ,   1  ,False , False , 0x4682B4 ),
    Label('car'                 , 0x21 ,   33,    1    , '移动物体',   2  ,True  , False , 0x00008E ),
    Label('car_groups'          , 0xA1 ,  161,    1    , '移动物体',   2  ,True  , False , 0x00008E ),  
    Label('motorbicycle'        , 0x22 ,   34,    2    , '移动物体',   2  ,True  , False , 0x0000E6 ),
    Label('motorbicycle_group'  , 0xA2 ,  162,    2    , '移动物体',   2  ,True  , False , 0x0000E6 ),
    Label('bicycle'             , 0x23 ,   35,    3    , '移动物体',   2  ,True  , False , 0x770B20 ),
    Label('bicycle_group'       , 0xA3 ,  163,    3    , '移动物体',   2  ,True  , False , 0x770B20 ),
    Label('person'              , 0x24 ,   36,    4    , '移动物体',   2  ,True  , False , 0x0080c0 ),
    Label('person_group'        , 0xA4 ,  164,    4    , '移动物体',   2  ,True  , False , 0x0080c0 ),
    Label('rider'               , 0x25 ,   37,    5    , '移动物体',   2  ,True  , False , 0x804080 ),
    Label('rider_group'         , 0xA5 ,  165,    5    , '移动物体',   2  ,True  , False , 0x804080 ),
    Label('truck'               , 0x26 ,   38,    6    , '移动物体',   2  ,True  , False , 0x8000c0 ),
    Label('truck_group'         , 0xA6 ,  166,    6    , '移动物体',   2  ,True  , False , 0x8000c0 ), 
    Label('bus'                 , 0x27 ,   39,    7    , '移动物体',   2  ,True  , False , 0xc00040 ),
    Label('bus_group'           , 0xA7 ,  167,    7    , '移动物体',   2  ,True  , False , 0xc00040 ),
    Label('tricycle'            , 0x28 ,   40,    8    , '移动物体',   2  ,True  , False , 0x8080c0 ),
    Label('tricycle_group'      , 0xA8 ,  168,    8    , '移动物体',   2  ,True  , False , 0x8080c0 ),
    Label('road'                , 0x31 ,   49,    9    , '平面'    ,   3  ,False , False , 0xc080c0 ),
    Label('siderwalk'           , 0x32 ,   50,    10   , '平面'    ,   3  ,False , False , 0xc08040 ),
    Label('traffic_cone'        , 0x41 ,   65,    11   , '路间障碍',   4  ,False , False , 0x000040 ),
    Label('road_pile'           , 0x42 ,   66,    12   , '路间障碍',   4  ,False , False , 0x0000c0 ),
    Label('fence'               , 0x43 ,   67,    13   , '路间障碍',   4  ,False , False , 0x404080 ),
    Label('traffic_light'       , 0x51 ,   81,    14   , '路边物体',   5  ,False , False , 0xc04080 ),
    Label('pole'                , 0x52 ,   82,    15   , '路边物体',   5  ,False , False , 0xc08080 ),
    Label('traffic_sign'        , 0x53 ,   83,    16   , '路边物体',   5  ,False , False , 0x004040 ),
    Label('wall'                , 0x54 ,   84,    17   , '路边物体',   5  ,False , False , 0xc0c080 ),
    Label('dustbin'             , 0x55 ,   85,    18   , '路边物体',   5  ,False , False , 0x4000c0 ),
    Label('billboard'           , 0x56 ,   86,    19   , '路边物体',   5  ,False , False , 0xc000c0 ),
    Label('building'            , 0x61 ,   97,    20   , '建筑'    ,   6  ,False , False , 0xc00080 ),
    Label('bridge'              , 0x62 ,   98,    255  , '建筑'    ,   6  ,False , True  , 0x808000 ),
    Label('tunnel'              , 0x63 ,   99,    255  , '建筑'    ,   6  ,False , True  , 0x800000 ),
    Label('overpass'            , 0x64 ,  100,    255  , '建筑'    ,   6  ,False , True  , 0x408040 ),
    Label('vegatation'          , 0x71 ,  113,    21   , '自然'    ,   7  ,False , False , 0x808040 ),
    Label('unlabeled'           , 0xFF ,  255,    255  , '未标注'  ,   8  ,False , True  , 0xFFFFFF ),
]
</code></pre>

ptrblck: <p>Thanks for the information.</p>
<p>Could you check, what the white color represents in your ground truth image?<br>
I would assume it’s class0 (sky), but that wouldn’t make sense, as part of the buildings and car is also marked as this color.</p>

Shashank_Shekhar: <p>what you told is correct ,actually all the white color in the image is ignore index. Sky, building and host vehicle all should have corresponding classes but the dataset provided by them labels some parts as 255.<br>
Is their any workaround?</p>

ptrblck: <p>This would explain, why your model predicts random classes for these pixel locations.<br>
Basically, your model will not get any information for these locations, as they are ignored, and can just output any random class. In your case it seems the vegetation class was picked.</p>
<p>You could transform all 255 class values to an “unknown” or background class and let the model learn this additional class instead.</p>

Shashank_Shekhar: <p>It worked, i trained my model for 10 epoch by adding all the ignore labels to extra class(class id:22). I have attached the results i got after 10 epochs.</p>
<p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> for your support, i have closed the issue with your last answer.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/3/23f7077cf7e173bd698a8e825d8c95da90c996ac.png" data-download-href="https://discuss.pytorch.org/uploads/default/23f7077cf7e173bd698a8e825d8c95da90c996ac" title="Screenshot from 2020-01-13 18-32-21"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/3/23f7077cf7e173bd698a8e825d8c95da90c996ac_2_690x239.png" alt="Screenshot from 2020-01-13 18-32-21" data-base62-sha1="589Y5A7RvvJx2Ua3aETsOB5ZQvi" width="690" height="239" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/3/23f7077cf7e173bd698a8e825d8c95da90c996ac_2_690x239.png, https://discuss.pytorch.org/uploads/default/original/3X/2/3/23f7077cf7e173bd698a8e825d8c95da90c996ac.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/2/3/23f7077cf7e173bd698a8e825d8c95da90c996ac.png 2x" data-dominant-color="DDD4D9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2020-01-13 18-32-21</span><span class="informations">951×330 112 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>