Differentiable binary mask centroid computation

towardsautonomy: <p>Hello, I am wondering if there are ways to compute the centroid of a binary blob in an image such that they are differentiable? Essentially, I am looking for a differentiable twin of the loss function shown here:</p>
<pre><code class="lang-auto">class CentroidLoss(nn.Module):
    def __init__(self):
        super(CentroidLoss, self).__init__()
        
    def forward(self, rendered_silhouette, original_silhouette):
        rendered_sil_indices = torch.nonzero(rendered_silhouette)
        rendered_sil_centroid = torch.Tensor.float(rendered_sil_indices).mean(dim=0)

        original_sil_indices = torch.nonzero(original_silhouette)
        original_sil_centroid = torch.Tensor.float(original_sil_indices).mean(dim=0)

        centroid_dist = ((rendered_sil_centroid[0] - original_sil_centroid[0]) ** 2 \
                        + (rendered_sil_centroid[0] - original_sil_centroid[0]) ** 2 ) \
                        / (rendered_silhouette.shape[0]*rendered_silhouette.shape[1])

        return centroid_dist
</code></pre>
<p>“rendered_silhouette” and “original_silhouette” are binary images, each containing a blob.</p>

KFrank: <p>Hi Shubham!</p>
<aside class="quote no-group" data-username="towardsautonomy" data-post="1" data-topic="115888" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/towardsautonomy/48/36247_2.png" class="avatar"> towardsautonomy:</div>
<blockquote>
<p>“rendered_silhouette” and “original_silhouette” are binary images, each containing a blob.</p>
</blockquote>
</aside>
<p>The short answer is to compute the center of mass (the “weighted<br>
centroid”) of  <code>rendered_silhouette</code> before thresholding.</p>
<p>Let me speculate as to what your use case might be:</p>
<p>Your <code>original_silhouette</code> plays the role of some kind of “ground-truth”<br>
target.  Perhaps your network is trying to reconstruct it somehow.</p>
<p><code>rendered_silhouette</code> is generated by a network and you want to<br>
train the weights of the network so that <code>rendered_silhouette</code><br>
approximates <code>original_silhouette</code>.  Therefore you want a differentiable<br>
loss function so that you can use backpropagation for training.</p>
<p>In all probability your network generates an image that is not binary.<br>
That is, the value of each pixel in the generated image is the probability<br>
(or perhaps the logit) for that pixel being <code>1</code> (rather than <code>0</code>) in the final<br>
binary image.  You then threshold this “probability” image to get your<br>
final <code>rendered_silhouette</code> binary image.</p>
<p>This thresholding is not (usefully) differentiable, so you can’t<br>
backpropagate through it.</p>
<p>What you can do instead is calculate the center of mass of the (blob<br>
in the) probability image.  This will be a properly-differentiable version<br>
of the (non-differentiable) centroid of the thresholded image.  Use it<br>
to compute <code>centroid_dist</code> returned by your <code>CentroidLoss</code> function.</p>
<p>You can now use this differentiable loss to backpropagate and train<br>
the weights in your network.</p>
<p>Best.</p>
<p>K. Frank</p>

towardsautonomy: <p>Hi Frank,</p>
<p>Thanks for responding, yeah your speculation is correct. How do one go about computing the center of the mass though?</p>
<p>In essence, one need to multiply the indices with the pixel value and then take mean of it all. But I can’t think of a differentiable way of obtaining indices.</p>
<p>Thanks,<br>
Shubham</p>

KFrank: <p>Hi Shubham!</p>
<aside class="quote no-group" data-username="towardsautonomy" data-post="3" data-topic="115888" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/towardsautonomy/48/36247_2.png" class="avatar"> towardsautonomy:</div>
<blockquote>
<p>In essence, one need to multiply the indices with the pixel value and then take mean of it all.</p>
</blockquote>
</aside>
<p>Yes, this is correct.  More precisely, you need to take the<br>
pixel-value-weighted mean of the pixel locations (indices)<br>
to get the “center-of-mass” location.</p>
<aside class="quote no-group">
<blockquote>
<p>But I can’t think of a differentiable way of obtaining indices.</p>
</blockquote>
</aside>
<p>You don’t need to get “differentiable” indices.  What matters is that<br>
your (pre-thresholding) pixel values are differentiable (with respect<br>
to your network weights), and presumably they are.  Your indices<br>
(pixel locations) are fixed constants.  They do not need to be – and<br>
are not – differentiable.</p>
<p>Best.</p>
<p>K. Frank</p>