Why is .clone() necesary after expand

Anakin: <p>Hello all,</p>
<p>as doc of <code>clone</code> function shows</p>
<blockquote>
<p>This function is differentiable, so gradients will flow back from the result of this operation to <code>input</code> . To create a tensor without an autograd relationship to <code>input</code> see <a href="https://pytorch.org/docs/stable/autograd.html#torch.Tensor.detach" rel="noopener nofollow ugc"><code>detach()</code></a>.</p>
</blockquote>
<p>for instance,</p>
<pre><code class="lang-auto">src = torch.rand(3, 5, requires_grad =True)
dst = src.clone()
</code></pre>
<p>The gradients that related to <code>dst</code> tensor will also flow back to <code>src</code> tensor. As so far I guess I’m right.</p>
<p>Here is the case where I donot understand why clone is needed.</p>
<pre><code class="lang-auto">    def forward(self, image_size, feats, boxes):
        """
        image_size: image size, height, width
        feats:
        boxes:
        """
        B = boxes.size(0)
        print("boxes.size", boxes.size())
        
        # convert box to affine matrix
        self.affine_regressor.set_size(image_size)
        theta = self.affine_regressor.forward(boxes, image_size)
         
        # generate grids
        grid = self.grid_generator.forward(theta).to(feats.device)
        print("grid.size", grid.size())
        
        # expand inputs for bilinear sampling
        print("feats.size", feats.size())
        feats = feats.expand(B, feats.size(0), feats.size(1), feats.size(2))
        # ^^^^^^^^^^ why is clone needed here??^^^^^^^^^^^^^^^^^^^^ 

        # bilinear sampling
        box_features = self.roi_sampler.forward(feats, grid)
        return box_features
</code></pre>
<p>The function convert detected boxes <code>Bx4</code> into <code>Bx2x3</code> normalized matrixes, then using these matrixes to generate grids for bilinear sampling, as shown at the last line. The definition of roi_sampler can be located <a href="https://github.com/qassemoquab/stnbhwd/blob/master/BilinearSamplerBHWD.cu" rel="noopener nofollow ugc">BilinearSamplerBHWD.cu</a>.</p>
<p>After bilinear sampling, I visualized the result, as the following image shows,  <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/a/2af4fc0e5cce10c755914d70c28e710b60709dbc.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/2af4fc0e5cce10c755914d70c28e710b60709dbc" title="kitten_rgb"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/a/2af4fc0e5cce10c755914d70c28e710b60709dbc_2_690x377.jpeg" alt="kitten_rgb" data-base62-sha1="680VI77wLDroKfT51fnd6wInJsU" width="690" height="377" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/a/2af4fc0e5cce10c755914d70c28e710b60709dbc_2_690x377.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/2/a/2af4fc0e5cce10c755914d70c28e710b60709dbc_2_1035x565.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/a/2af4fc0e5cce10c755914d70c28e710b60709dbc_2_1380x754.jpeg 2x" data-dominant-color="DEDFDD"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">kitten_rgb</span><span class="informations">2523×1380 204 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div>   the cats in each box are covered by unexpected colors.  If I add a <code>clone()</code> after expanding the input <code>feats (3, H, W)</code> to <code>(B, 3, H, W)</code>. I get the correcte results, say in each box, the cat is interpolated with given expanded feats w.r.t. the generated grids (feats are  outputs of backbone model, here VGG16), as depicted in the following figure. <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/5/7586168722c304376379096ae9075246824f2998.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/7586168722c304376379096ae9075246824f2998" title="kitten_ok"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/5/7586168722c304376379096ae9075246824f2998_2_690x377.jpeg" alt="kitten_ok" data-base62-sha1="gLF94d1ZIqmAp6QqwlQ0aUCj0Cs" width="690" height="377" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/5/7586168722c304376379096ae9075246824f2998_2_690x377.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/7/5/7586168722c304376379096ae9075246824f2998_2_1035x565.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/7/5/7586168722c304376379096ae9075246824f2998_2_1380x754.jpeg 2x" data-dominant-color="E2E3DA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">kitten_ok</span><span class="informations">2523×1380 236 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>My question:</p>
<ul>
<li>
<p>After expanding, the feas sizes with or without clone are the same <code>6x3xHxW</code>.  What kind of role is played by the clone function. The feats are already expanded in the correct dims.</p>
</li>
<li>
<p>With clone(), the gradients will flow back to the expanded tensor <code>(B, 3, H, W)</code>, which are originally based on <code>(3, H, W)</code>. Do the gradients flow back further to this base tensor. In my case, I need the gradients of the base tensor.</p>
</li>
</ul>
<p>Thanks.</p>

albanD: <p>Hi,</p>
<p>What <code>.clone()</code> does is allocate brand new memory and copy the content of the Tensor into it. It always creates a new contiguous Tensor as output.<br>
The thing with <code>expand</code> is that it does not actually allocate new memory but play with stride. Unfortunately, not all functions support non-standard strides (non-contiguous Tensors).<br>
I guess that your function is one of these and so the clone fixes it by making sure the input is contiguous.</p>
<p>A more explicit way to do this, is to do <code>feats = feats.contiguous()</code> so that, if <code>feats</code> is already contiguous, nothing will happen. Compared to clone which will always make a full copy.</p>

Anakin: <p>Hi albanD,</p>
<p>yes, you are absolutely right. The tensor <code>feats</code> I tested here is dummy tensor, not the direct output of a backbone model. If I put my bilinear sampling module into my complete model. The problem is automatically resolved. Actually, if there is tensor that is non-contiguous, it might trigger runtime error if some ops need the tensor to be contiguous. Thanks for your input.</p>