How to select pixels of ROI from feature map

sal: <p>In Faster RCNN or Mask RCNN the ROI Align layers takes different sized roi’s as input and projects them onto a uniform layer size.<br>
I’m currently implementing <a href="https://arxiv.org/pdf/1803.03474.pdf" rel="nofollow noopener">this paper</a> where the authors take rotated boxes as inputs to inception layer for further detection. It is an end-to-end text recognition pipeline which first does text box detection( same task as object detection), then the text boxes are fed to an encoder-decoder to recognize the characters.</p>
<p>The output of the box detection as rotated bounding box, score map and last feature map   are of the format :</p>
<pre><code class="lang-auto">Feature Map from text detector Shape torch.Size([1, 256, 128, 128])
ROI from text detector Shape torch.Size([1, 5, 128, 128])
Score from text detector Shape torch.Size([1, 2, 128, 128])
</code></pre>
<p>The five parameters represent the distances of the current point to the top, bottom, left and right<br>
sides of an associated bounding box, together with its inclined orientation. With these configurations, the detection branch is able to predict a quadrilateral of arbitrary orientation for each text instance.</p>
<p>The original text detector architecture is<a href="https://arxiv.org/pdf/1704.03155v2.pdf" rel="nofollow noopener">based on East.</a>.<br>
I have been able to match it’s base line scores.</p>
<p>I’m stuck on the grid sampling part currently.<br>
I’m first converting the 5 values - {4 distances , 1 angle} to corresponding 8 coordinates for a rotated bbox as a quadrilateral (scaled to feature map size, filtered by scores) to produce coordinates tensor of the format :</p>
<pre><code class="lang-auto">raw coord coords shape {numberOfPredictionsX8}
torch.Size([2960, 8])
</code></pre>
<p>Now I have to select only these pixels between these coordinates from the top feature map and then passing them to a <code>nn.Functional.Interpolate()</code> layer.</p>
<p>How to do this while keeping gradient intact.<br>
I naively did the fillpoly directly on feature map directly first, but currently code looks like :</p>
<pre><code class="lang-auto">    fm_ = fm.clone().detach().numpy()
    coords_ = coords.clone().detach().numpy()

    masks = np.zeros((h_, w_), dtype=np.uint8)

    cv2.fillPoly(masks, (coords_.reshape(-1, 4, 2)).astype(np.int32), (1,1,1), 8)
    masks = torch.tensor((masks), dtype=torch.float32)

    fm *= masks
</code></pre>
<p>to preserve only pixel values from <code>fm_</code> present inside a quadrilateral described by points in <code>coords_</code><br>
but cv function only works on np values.<br>
But converting to np would lead to loss of gradient flow.</p>
<p>If I pass this fm to interpolate layer would that work correctly ?</p>
<p>updated code with suggestion, but still this approach is selecting extra pixels.</p>

ptrblck: <p>It looks like you are just creating the mask using <code>numpy</code> and apply it on your tensor <code>fm</code>.<br>
If you don’t need gradients for the mask, this approach should generally work.<br>
You might have to transform <code>masks</code> to a valid (non-differentiable) tensor before multiplying it with <code>fm</code>.<br>
Have you tried your approach and noticed that it’s not working?</p>

sal: <p>Yep it’s a makeshift approach for now to just mask out all pixels that are not in my quadrilateral.<br>
But the aim is to select only the pixels belonging to the quadrilateral roi’s and then pass those to the f.interpolate().<br>
Anyway to do that ? kind of what tf.crop_and_resize does. but for floating point coordinates for a region defined by 8 coordinates and not 4 corners of a horizontal box.<br>
Basically a perspective transform on the feature map, but with an operation that is differentiable.</p>

ptrblck: <p>I think in that case <a href="https://pytorch.org/docs/stable/nn.html#grid-sample" rel="nofollow noopener">grid_sample</a> might be a useful function, which uses pixel locations from the specified grid.</p>

eslam_bakr: <p>Hello <a class="mention" href="/u/ptrblck">@ptrblck</a>,<br>
Please I am facing the same problem, I have 5 points that are produced from simple NN and I want to get the area inside them to generate a mask to be able to apply a segmentation loss on the mask.<br>
I implemented it using cv2.fillploy after converting the 5 points to numpy but unfortunately, I lost the gradients.</p>
<p>I thought about your comment to use grid_sample but unfortunately, I don’t know how could I generate the mask with respect to the 5 points without losing the gradient tracing</p>
<p>My main problem is that, I don’t have the all desired position to create my grid, and If I use fillpoly to get the positions then I will loss the gradient trace.</p>
<p>And suggestions?</p>

ptrblck: <p>Usually you would create the target mask before the training so that you wouldn’t need gradients for this operation.<br>
Could you explain your use case a bit and how you would like to use these gradients for the mask?</p>

eslam_bakr: <p>First of all thank you for your fast reply.</p>
<p>Right, For the GT There is no need but my network generate for example 4 point which represent rotated box each point represented by (x,y) then I need to convert the network output to be mask as well to be ale to apply segmentation loss on both the predicted mask (after converting predicted point to mask and the GT mask)</p>
<p>I tried to convert the predicted points to mask using cv2.fillpoly but the loss wasn’t change during training, after investigation I realized that this behaviour because I loss the gredient trace when I convert the predections to numpy to use fillpoly, so I need another way to achieve that to be able to backprop on it</p>
<p>I hope this time I was clear enough</p>

ptrblck: <p>I’m unsure, but take a look at <a href="https://discuss.pytorch.org/t/how-to-collect-coordinates-within-some-bounding-boxes-efficiently-by-pytorch/6347/7">this approach</a> and see, if you could reuse it.<br>
If I understand it correctly, you are dealing with mask targets, but your model outputs just coordinates so you would want to create a mask using these coordinates?</p>

eslam_bakr: <p>You are completely right, but unfortunate the link will not help me as it is talking about boxes, I think it is easy to generate mask from the coordinates if they will shape a box</p>
<p>But my case the coordinates will create an arbitrary shape for example a star shape so this approach which is prposed in this link will not help me, I need to use something like cv2.fillpoly but in pytorch to keep tracing gradient.</p>

ptrblck: <p>I see. You could try to check the source code of <code>fillpoly</code> and see, if you could either port it directly to PyTorch, so that Autograd would create the backward pass automatically for you, implement the backward function manually (if possible at all), or alternatively let your model output the mask directly (might be the easiest approach).</p>

JasonChenhx: <p>Hello, I’m sorry to bother you.<br>
May I ask you a question?<br>
My question is that I have a feature vectors that operate a original image through a CNN and I also have a segmentation result. But I want to get the feature vectors of the yellow and green regions.<br>
The segmentation result is shown.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/2/6/267f7527e4486fc6ad21296a73a53039264b09db.png" alt="2007_000063" data-base62-sha1="5uzdG1PNxRfAIuJNxvmSytuylo7" width="500" height="375"></p>

ptrblck: <p>How would these feature vectors be defined for these regions?<br>
I.e. would you like to get a specific activation in a previous layer or would you like to process the output in a special way?</p>

JasonChenhx: <p>The feature vectors means that feature maps.<br>
I want to process the output in a special way.<br>
In other words, I want to the feature maps of the yellow and green regions.</p>

ptrblck: <p>To get the activation maps you could use forward hooks as described <a href="https://discuss.pytorch.org/t/how-can-l-load-my-best-model-as-a-feature-extractor-evaluator/17254/6">here</a>.<br>
However, the pixel locations of these maps might not correspond to the output locations and it depends on the architecture of your model.<br>
E.g. convolution layers will use filters with a specific window, stride, dilation, etc., so that you would have to calculate the receptive field of the output locations for each activation map.</p>

JasonChenhx: <p>If I interpolate the feature maps with bilinear into the same size as the original image, and then use the pixel position to correspond one to one, is it ok?</p>

ptrblck: <p>No, this still wouldn’t work, as each output pixel position might be calculated by a larger field of the input activation(s).<br>
If all your convolutions use a <code>1x1</code> kernel, the output pixel locations would correspond to the input locations.<br>
<a href="https://captum.ai/">Captum</a> uses specific methods for model interpretability, which e.g. use the gradient flow to visualize which activations were “important” for which output part, but that doesn’t seem to be your use case.</p>

JasonChenhx: <p>Thanks your reply.<br>
I understand what you’re saying.<br>
But I still can’t figure out what method I should use if I want to extract feature maps of ROI for irregular shapes.</p>

JasonChenhx: <p>But I remember using ROI Align in Mask R-CNN， which can extract the feature maps of the candidate box.</p>

ptrblck: <p><code>ROIAlign</code> interpolates the proposals, which are overlaid on the feature maps, no?</p>
<p>As I said, you could calculate the receptive field, if you know all conv setups.<br>
Interpolating an activation map to the same shape as the output will not create a 1 to 1 mapping.</p>

JasonChenhx: <p>Thanks your reply.<br>
I understand your means. And I know what I should do next.<br>
Thanks again.</p>