Torch.repeat and torch.expand which to use?

Hanxiong_Chen: <p>I have one vector and one 2-D vector. Say embed1= [1, 2, 3] shape is (3) and embed2=[[1, 2, 3], [4,5,6], [7,8,9]] shape is (3, 3). Now I want to expand embed1 to [[1, 2, 3], [1, 2, 3], [1, 2, 3]] so that I can do torch.cat() to concatenate embed1 with embed2 and send them to a MLP. Actually this is to avoid of doing for loop by concatenating embed1 with each vector in embed2 and then send each concatenated vector to MLP.  When I read the document, it says that torch.repeat will copy data so I don’t know if this will affect the backpropagation since data are copied instead of the original one. I hope all the autograd process will only update my original embed1.</p>

albanD: <p>Hi,</p>
<p>Backprop-wise, they will give the exact same result.<br>
The difference is that if the original dimension you want to expand is of size 1, you can use <code>torch.expand()</code> to do it without using extra memory.<br>
If the dimension you want to expand is of size more than 1, then you actually want to repeat what is at that dimension and you should use <code>torch.repeat()</code>. It will use extra memory (there is no way around that).</p>

tomahawk810: <p>Therefore, when expanding a dimension of size 1, should <code>torch.expand()</code> always be used instead of <code>torch.repeat()</code>?</p>

albanD: <p>Yes.<br>
Keep in mind though that if you plan on changing this expanded tensor inplace, you will need to use <code>.clone()</code> on it before so that it actually is a full tensor (with memory for each element). But even <code>.expand().clone()</code> should be faster than <code>.repeat()</code> I think.</p>

tomahawk810: <p>What about <code>.contiguous</code> instead of <code>.clone</code> ? Wouldn’t those 2 operations achieve the same goal before inplace modifications?</p>
<p>I might be going a bit off-topic, I am new to PyTorch and am trying to get what are best practices.</p>

albanD: <p>Hi,</p>
<p>In this case yes they will do the same thing. Because the input is allways non contiguous <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=6" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Hanxiong_Chen: <p>Thank you for your answer.</p>

solsol: <p>Hi,</p>
<p>A and B are feature maps, in shape of [12, 9, 64] each.<br>
I have this code to concatenate them 9 times in different cells:<br>
A = A.repeat(1, 1, 9).view(12, 81, 64)<br>
B = B.repeat(1, 9, 1)</p>
<p>C = torch.cat((A, B), dim=2)</p>
<p>However, I faced with lack of memory to run this code. Is it possible to use expand instead of view, here also?<br>
Thanks!</p>

tom: <p>So to make this a self-contained example:</p>
<pre><code class="lang-python">A = torch.randn([12, 9, 64])
B = torch.randn([12, 9, 64])
Ar = A.repeat(1, 1, 9).view(12, 81, 64)
Br = B.repeat(1, 9, 1)
C = torch.cat((Ar, Br), dim=2)
D = torch.cat([A.unsqueeze(2).expand(-1, -1, 9, -1), 
               B.unsqueeze(1).expand(-1, 9, -1, -1)], dim=-1).view(12, 81, 128)
print ((C-D).abs().max().item())
</code></pre>
<p>gives 0 and from A, B to D only the memory for D is newly allocated.</p>
<p>Best regards</p>
<p>Thomas</p>
<p>P.S.: Enclose multiple lines of source code with triple backticks ``` to have them rendered nicely. Single backticks for “in-text” commands.</p>

solsol: <p>Thomas thanks for your reply. It was helpful, however, still there is memory problem!!<br>
“cuda out of memory”</p>

pkmandke: <p>Hi.</p>
<p>I have a somewhat related query so thought I’d ask here instead of starting a new thread.</p>
<p>So, I am trying to use VGG as a feature extractor. My model returns a (grayscale) image tensor of size (1, 512, 512) that I want to feed into VGG which takes in 3 channel RGB images. The idea is to compute a loss based on the output of VGG and backprop that into my model.</p>
<p>So, if I repeat the grayscale across the 3 dimensions before feeding into VGG, will .backward() successfully backprop into my model? Some code snippets would be quite helpful.</p>
<p>Thanks.</p>

albanD: <p>Hi,</p>
<p>Ye, both repeat and expand are differentiable. So you can use them in your net as any other op.</p>

WolfLo: <p>Hi, in the following example, is <code>Ar2</code> a better alternative to <code>Ar</code>, or does the <code>reshape()</code> function also allocates new memory, offsetting the advantage of <code>expand()</code> over <code>repeat()</code> ?</p>
<pre><code class="lang-auto">A = torch.randn([2, 3])
Ar = A.repeat(3,1)
Ar2 = A.unsqueeze(0).expand(3,-1,-1).reshape(3*A.shape[0],-1)
</code></pre>
<p>Thanks in advance</p>