Should I use clone when I am using a feature in multiple branch?

username: seyeeet
cooked: <p>So I have this confusion for months now, should I use clone in the folliwng code for <code>x</code>?<br>
I am not sure if it is important or not, but I appreciate if someone can help me understand it.<br>
Is it gonna effect the learning if I go with any of the following samples<br>
Lets say I have a model that somewhere in it I have to use branches of an input and compute different output. I am wondering if I need to use clone or not.</p>
<p>Which one of the following models is correct?</p>
<ol>
<li>
</li>
</ol>
<pre><code class="lang-auto">class Model(nn.Module):
  def __init__(self,):
    super(Model, self).__init__()
    self.conv = nn.Conv2d(3,6,3)
    self.conv_2 = nn.Conv2d(6,2,3)
    self.conv_3= nn.Conv2d(6,20,3) 
  def forward(self,input):
    x = self.conv(input)
    x_2 = self.conv_2(x)
    x_3 = self.conv_3(x)
    return x_2,x_3
</code></pre>
<ol start="2">
<li>
</li>
</ol>
<pre><code class="lang-auto">class Model(nn.Module):
  def __init__(self,):
    super(Model, self).__init__()
    self.conv = nn.Conv2d(3,6,3)
    self.conv_2 = nn.Conv2d(6,2,3)
    self.conv_3= nn.Conv2d(6,20,3) 
  def forward(self,input):
    x = self.conv(input)
    x_2 = self.conv_2(x.clone())
    x_3 = self.conv_3(x)
    return x_2,x_3
</code></pre>
<ol start="3">
<li>
</li>
</ol>
<pre><code class="lang-auto">class Model(nn.Module):
  def __init__(self,):
    super(Model, self).__init__()
    self.conv = nn.Conv2d(3,6,3)
    self.conv_2 = nn.Conv2d(6,2,3)
    self.conv_3= nn.Conv2d(6,20,3) 
  def forward(self,input):
    x = self.conv(input)
    x_2 = self.conv_2(x.clone())
    x_3 = self.conv_3(x.clone())
    return x_2,x_3
</code></pre>

username: ptrblck
cooked: <p>You would have to <code>clone()</code> the input to the separate branches, in case they are using inplace operations in the first branch, as it would yield unexpected results in the second (and further) branch as seen here:</p>
<pre><code class="lang-python">lin = nn.Linear(10, 10)
branch1 = nn.ReLU(inplace=True)
branch2 = nn.Sigmoid()

# without clone
x = torch.randn(1, 10)
x = lin(x)
print(x)
&gt; tensor([[-0.1115, -0.4749, -0.0076, -0.1911, -0.0425, -0.7667, -1.2244, -0.2573,
            1.6056, -0.1653]], grad_fn=&lt;AddmmBackward&gt;)

x1 = branch1(x)
print(x) # changed inplace!
&gt; tensor([[0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 1.6056,
           0.0000]], grad_fn=&lt;ReluBackward1&gt;)

print(x1)
&gt; tensor([[0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 1.6056,
           0.0000]], grad_fn=&lt;ReluBackward1&gt;)

x2 = branch2(x) # unexpected results
print(x2)
&gt; tensor([[0.5000, 0.5000, 0.5000, 0.5000, 0.5000, 0.5000, 0.5000, 0.5000, 0.8328,
           0.5000]], grad_fn=&lt;SigmoidBackward&gt;)

# with clone
x = torch.randn(1, 10)
x = lin(x)
print(x)
&gt; tensor([[-0.1678, -0.0383,  0.3958,  0.0245, -0.3356,  0.2320,  0.0355, -0.1278,
            0.1390,  0.5725]], grad_fn=&lt;AddmmBackward&gt;)

x1 = branch1(x.clone())
print(x) # not changed
&gt; tensor([[-0.1678, -0.0383,  0.3958,  0.0245, -0.3356,  0.2320,  0.0355, -0.1278,
            0.1390,  0.5725]], grad_fn=&lt;AddmmBackward&gt;)

print(x1)
&gt; tensor([[0.0000, 0.0000, 0.3958, 0.0245, 0.0000, 0.2320, 0.0355, 0.0000, 0.1390,
           0.5725]], grad_fn=&lt;ReluBackward1&gt;)

x2 = branch2(x)
print(x2)
&gt; tensor([[0.4581, 0.4904, 0.5977, 0.5061, 0.4169, 0.5578, 0.5089, 0.4681, 0.5347,
           0.6393]], grad_fn=&lt;SigmoidBackward&gt;)
</code></pre>

username: seyeeet
cooked: <p>Thank you for clarification.<br>
Would it cause any issue if I clone to both first and second branch instead of just second branch? I mean technically it should be equal to the case that we clone only in the second branch… (or maybe not?)<br>
Can you please enlighten me about the difference</p>

username: ptrblck
cooked: <p>If you are unsure which branch might manipulate the input inplace, you could <code>clone</code> both, but would thus use a bit more memory.</p>

username: char-t
cooked: <p>Hi! The model in my current project uses lots of branches and this thread caught my interest. Is it ok to not clone() the input to any branches if we have taken care to avoid inplace operations? Motivation for this would be reducing memory usage.</p>

username: ptrblck
cooked: <p>Yes, it should be fine. Just make sure not to reassign the same variable name:</p>
<pre><code class="lang-python">x = self.module1(x)
x = self.module2(x)
</code></pre>
<p>in case you want to use the same “input” <code>x</code> in <code>self.module2</code>.</p>

username: seyeeet
cooked: <p><a class="mention" href="/u/char-t">@char-t</a> can you please clarify how to <code>taken care to avoid inplace operations? </code> without using the clone?</p>
<p><a class="mention" href="/u/char-t">@char-t</a>  <a class="mention" href="/u/ptrblck">@ptrblck</a><br>
do you mean something like this is fine to do and it is similar to using the clone?:</p>
<pre><code class="lang-auto">x1 = input
x2 = input
x1 = self.module1(x1)
x2 = self.module2(x2)
</code></pre>
<p>and it is okay to do it like this?</p>

username: ptrblck
cooked: <p>No, this won’t work, as <code>x1</code> and <code>x2</code> would be a view to <code>input</code>.<br>
You could use my posted code snippet for a quick check:</p>
<pre><code class="lang-python">lin = nn.Linear(10, 10)
branch1 = nn.ReLU(inplace=True)
branch2 = nn.Sigmoid()

# without clone
x = torch.randn(1, 10)
x = lin(x)
print(x)
&gt; tensor([[-0.9229, -0.3109, -0.3517,  0.1832,  0.2353, -0.5287, -0.9106, -0.1691,
           -0.6657,  0.1129]], grad_fn=&lt;AddmmBackward&gt;)

x1 = x
x2 = x
x1_out = branch1(x1)
print(x) # changed inplace!
&gt; tensor([[0.0000, 0.0000, 0.0000, 0.1832, 0.2353, 0.0000, 0.0000, 0.0000, 0.0000,
           0.1129]], grad_fn=&lt;ReluBackward1&gt;)
print(x1) # changed inplace!
&gt; tensor([[0.0000, 0.0000, 0.0000, 0.1832, 0.2353, 0.0000, 0.0000, 0.0000, 0.0000,
           0.1129]], grad_fn=&lt;ReluBackward1&gt;)
print(x1_out)
&gt; tensor([[0.0000, 0.0000, 0.0000, 0.1832, 0.2353, 0.0000, 0.0000, 0.0000, 0.0000,
           0.1129]], grad_fn=&lt;ReluBackward1&gt;)
print(x2)
&gt; tensor([[0.0000, 0.0000, 0.0000, 0.1832, 0.2353, 0.0000, 0.0000, 0.0000, 0.0000,
           0.1129]], grad_fn=&lt;ReluBackward1&gt;)
</code></pre>