Instance Segmentation Using Mutex Watershed

Assefa_Seyoum: <h1><a name="occluded-segments-of-the-same-instance-assigned-different-ids-after-mutex-watershed-1" class="anchor" href="#occluded-segments-of-the-same-instance-assigned-different-ids-after-mutex-watershed-1"></a>Occluded segments of the same instance assigned different IDs after Mutex Watershed</h1>
<p>I am doing the instance segmentation of cables by predicting affinities then post-processing the result using mutex watershed.</p>
<p>However, I couldn’t figure out how to use the mutex watershed algorithm to model model occlusion.</p>
<p>A short explanatory example can be found at <a href="https://github.com/aswahd/Mutex-Watershed-Instance-segmentation" rel="noopener nofollow ugc">Mutex-Watershed-Instance-segmentation</a>.</p>
<p>Use two types of neighborhoods / offsets:</p>
<ol>
<li>Offset 1</li>
</ol>
<pre><code class="lang-auto">offsets = [
    [0, -1, 0], [0, 0, -1],
    [0, -1, -1], [0, 1, 1], [0, -1, 1], [0, 1, -1],
    [0, -9, 0], [0, 0, -9],
    [0, -9, -9], [0, 9, -9], [0, -9, -4],
    [0, -4, -9], [0, 4, -9], [0, 9, -4],
    [0, -27, 0], [0, 0, -27], [0, -27, 27]
]
</code></pre>
<ol start="2">
<li>4 neighborhoods at distance of {1, 2, 4, 8, 32, 64} (Similar to <a href="https://arxiv.org/abs/1811.10870" rel="noopener nofollow ugc">Affinity Derivation and Graph Merge for Instance Segmentation</a>).<br>
i.e.,</li>
</ol>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/f/6f1a7e76fe62660396fa47ff5d511ca03b1fec8a.png" data-download-href="https://discuss.pytorch.org/uploads/default/6f1a7e76fe62660396fa47ff5d511ca03b1fec8a" title="gmis-kernel"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/f/6f1a7e76fe62660396fa47ff5d511ca03b1fec8a_2_690x239.png" alt="gmis-kernel" data-base62-sha1="fQRLgvIcsTuDR4S9xWPDETIT6nU" width="690" height="239" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/f/6f1a7e76fe62660396fa47ff5d511ca03b1fec8a_2_690x239.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/f/6f1a7e76fe62660396fa47ff5d511ca03b1fec8a_2_1035x358.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/f/6f1a7e76fe62660396fa47ff5d511ca03b1fec8a_2_1380x478.png 2x" data-dominant-color="C8CBC8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">gmis-kernel</span><span class="informations">1406×489 188 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<h1><a name="h-1-on-cvppp-plant-dataset-2" class="anchor" href="#h-1-on-cvppp-plant-dataset-2"></a>1. On CVPPP Plant dataset</h1>
<p>Input Image:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/f/2fefa04d8964efc1b486880d132fbbe252725088.png" alt="plant017_input" data-base62-sha1="6Q3PWETh43MOo4nbYFb4aFUF6Aw" width="405" height="418"></p>
<p>Output After Affinity and Mutex Watershed:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/1/5/153922052f01b113afe48dae708e7f9d815f0716.png" alt="plant017_output" data-base62-sha1="31KqSvyU72RvfQnZxmFdgTVAjvU" width="405" height="418"></p>
<p>Note that the mutex watershed perfectly reconstructs the ground truth.</p>
<h1><a name="h-2-on-cable-dataset-3" class="anchor" href="#h-2-on-cable-dataset-3"></a>2. On Cable dataset</h1>
<p>Input Image:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/8/7/871349ecb0c8adcface6dba5da1a55279b62327e.png" alt="cable_input" data-base62-sha1="jgVMDDy0D7rfp6krIRox8PJvXzw" width="430" height="418"></p>
<p>Output After Affinity and Mutex Watershed:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/f/0fa90da85ba667da5bcc07dea3671def74c974b9.png" alt="cable_output" data-base62-sha1="2exm92a0Xbz8RZJrcpS4w327093" width="430" height="418"></p>
<p>Disjoint parts of the same cable are not connected after mutex watershed even though the area of the kernel / neighborhood (64) is larger than the width of a cable (10-15px)</p>
<p>Zoomed-in view of the disconnected segments</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/7/37dad58c68c3c900cad8b77ffc63f71a3fb22f99.png" data-download-href="https://discuss.pytorch.org/uploads/default/37dad58c68c3c900cad8b77ffc63f71a3fb22f99" title="zoomed-in"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/7/37dad58c68c3c900cad8b77ffc63f71a3fb22f99_2_312x250.png" alt="zoomed-in" data-base62-sha1="7Y76SpCTKkO84iu77ly6wndBVsB" width="312" height="250" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/7/37dad58c68c3c900cad8b77ffc63f71a3fb22f99_2_312x250.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/7/37dad58c68c3c900cad8b77ffc63f71a3fb22f99_2_468x375.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/7/37dad58c68c3c900cad8b77ffc63f71a3fb22f99_2_624x500.png 2x" data-dominant-color="533F6C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">zoomed-in</span><span class="informations">673×539 12.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Why does this happen, and how to correct it?</p>
<p>Thanks.</p>

ptrblck: <p>The issue doesn’t seem to be PyTorch-related, so I would recommend to ask the question in a CV-related discussion board or the authors in the repository.</p>