Rotate and rescale images in batch

simenthys: <p>I have an image named <code>adv_patch</code> which is a tensor.<br>
I also have a batch of images with known bounding box locations, and a pretrained image detection network.<br>
I want to apply this <code>adv_patch</code> to the batch of images, meaning i have to rescale <code>adv_patch</code>, rotate it, and put it on the image at each of the locations indicated by the bounding boxes.</p>
<p>The goal is to optimize <code>adv_patch</code> by minimizing the loss of the network w.r.t. <code>adv_patch</code>.<br>
This loss would be defined by how well the network still detects the objects in the images from my batch after <code>adv_patch</code> being applied.</p>
<p>In the end i would like <code>adv_patch</code> to be able to reliably ‘fool’ my detector.</p>
<p>As I understand, this would require me to use operations over which autograd can compute gradiënts in order to be able to backpropagate all the way back to <code>adv_patch</code>. So the functions for rotating and rescaling in <a href="https://pytorch.org/docs/stable/torchvision/transforms.html" rel="nofollow noopener">torchvision.transforms</a> are not an option, as they perform only on PIL images and transforming would mean detaching from graph and no gradients</p>
<p>I had some success in using <code>affine_grid</code> and <code>grid_sample</code> to rotate and rescale a single <code>adv_patch</code>, but applying an <code>adv_patch</code> on multiple detections on a batch of images is a whole different case.</p>
<p>Is there any way I could make this work?</p>

ptrblck: <p>Where are you stuck currently?<br>
I’ve written a simple rotation using a rotation matrix and <code>meshgrid</code> for batched inputs <a href="https://github.com/ptrblck/pytorch_misc/blob/master/image_rotation_with_matrix.py" rel="nofollow noopener">here</a>.<br>
Would that help somehow?</p>

simenthys: <p>Thank you for trying to help me. I used <code>.expand()</code> to get a batch of <code>adv_patch</code> images, of the size [N, M, C, H, W] where N is the number of images in my batch I should apply the patch on, and M is the number of detections per image in my batch. The problem I’m facing right now is that i would like to rotate the images in my <code>adv_patch</code> batch, but each give them a different (random) rotation.<br>
I also have to resize each of these images to a specific size, defined in another tensor.</p>
<p>Thanks for sharing your code for a batched rotation. I used the code of your <a href="https://discuss.pytorch.org/t/affine-transformation-matrix-paramters-conversion/19522/2?u=simenthys">answer in another thread</a> to rotate a single tensor. The problem I’m facing is that I have to give a <em>different</em> rotation to each of the  <code>adv_patch</code> images in my batch. I made a tensor the same size of my batch, containing random angles in [0, 2*pi]. How can i do the rotation taking as input the batch of <code>adv_patch</code>  images and the batch of rotations and giving me a batch of <code>adv_patch</code> images, each one rotated along the angles in the rotation batch? Is it possible on a batch level or should i use a loop for that?</p>

simenthys: <p>Thanks for your time, I figured it out! It came down to calculating a batch of rotation matrices <code>theta</code> and constructing a batch of <code>affine_grid</code> based on that.</p>
<p>The only problem is that the first call to <code>affine_grid</code> takes a long time. I suspect it’s doing some kind of initialization?</p>

edgarriba: <p>you can also use torchgeometry,<br>
<a href="https://torchgeometry.readthedocs.io/en/latest/warp_affine.html" class="onebox" target="_blank" rel="nofollow noopener">https://torchgeometry.readthedocs.io/en/latest/warp_affine.html</a></p>

simenthys: <p>Thanks! That would have been very helpful! It will come in handy for future projects.</p>

edgarriba: <p>quick update: this can be found in kornia:<br>
<a href="https://kornia.readthedocs.io/en/latest/geometry.transform.html#kornia.geometry.transform.rotate" class="onebox" target="_blank" rel="nofollow noopener">https://kornia.readthedocs.io/en/latest/geometry.transform.html#kornia.geometry.transform.rotate</a><br>
AND<br>
<a href="https://kornia.readthedocs.io/en/latest/geometry.transform.html#kornia.geometry.transform.scale" class="onebox" target="_blank" rel="nofollow noopener">https://kornia.readthedocs.io/en/latest/geometry.transform.html#kornia.geometry.transform.scale</a></p>

Meysam: <p>Hey Simenthys<br>
I have a similar problem as you posted here. I want to apply a rotated patch to an image and pass it through a neural net and calculate the loss. Then do backprop and revise the patch.</p>
<p>So, do you have any update on the best approach that one can use? Which of the answers do you suggest?</p>

yannf: <p>Thank you a lot  <a class="mention" href="/u/ptrblck">@ptrblck</a> for your code. I created the rotation function I wanted to have with your lines 21-33.<br>
I have just changed the centor location to x_mid=im.size(2) / 2 and y_mid=im.size(2)/ 2. as the +1 is already included into the im.size. Hope I am right. It improved quite a lot my rotation.</p>