Question about the input x of the forward pass (nn.module)

exponential: <p>Hi guys,<br>
Total noob here.</p>
<p>So, I have the following code</p>
<pre><code class="lang-auto">class MyModel(nn.Module):
    def __init__(self, depth, some var, some var1):
         super(MyModel self).__init__()

  def L1_block(self, x, before_trans = True):
    some code

  def L2_block(self, x, before_trans = True):
    some code

 def U1_block(self, x, before_trans = True):
    some code

 def U2_block(self, x, before_trans = True):
    some code

  def forward(self, x, concat_only=True):
        output_var = ['l1_out', 'l2_out', 'u1_out', 'u2_out']
        output_var_t = ['l1_out_t', 'l2_out_t', 'u1_out_t', 'u2_out_t']
        l1_out = self.L1_block(x, before_trans=True)
        l1_out_t = self.L1_block(x, before_trans=False)
        l2_out = self.L2_block(l1_out_t, before_trans=True)
        l2_out_t = self.L2_block(l1_out_t, before_trans=False)
        u1_out = self.U1_block(l2_out_t, l1_out, l2_out, x, before_trans=True)
        u1_out_t = self.U1_block(l2_out_t, l1_out, l2_out, x, before_trans=False)
        u2_out = self.U2_block(u1_out_t, l1_out, l2_out, u1_out, x, before_trans=True)
        u2_out_t = self.U2_block(u1_out_t, l1_out, l2_out, u1_out, x, before_trans=False)
        if concat_only:
            return dict(zip(output_var, (l1_out, l2_out, u1_out, u2_out)))
        else:
            return dict(zip(output_var_t, (l1_out_t, l2_out_t, u1_out_t, u2_out_t)))
</code></pre>
<p>My question is, in the forward pass, will the x be the same all through, or will it change at each line of code? x is the input data (I’d assume). I want the x, in this case, to be the same all through the forward, hence the reason for calling it at each block of code.<br>
Thanks so much for the help.</p>

ptrblck: <p>Yes, <code>x</code> is the passed input data and will be mapped to <code>input</code> in this example:</p>
<pre><code class="lang-python">output = model(input)
</code></pre>
<p>It depends on the used modules if <code>x</code> stays the same or will be modified.<br>
I.e. if any module uses the <code>x</code> input in an inplace operation as the first operation, <code>x</code> would be manipulated.<br>
Here is a small example:</p>
<pre><code class="lang-python">class MyModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.module1 = nn.Linear(1, 1)
        self.module2 = nn.ReLU()
        self.module3 = nn.ReLU(inplace=True)
        
    def forward(self, x):
        print(x) # original input
        out1 = self.module1(x)
        print(x) # unmodified
        out2 = self.module2(x)
        print(x) # unmodified
        out3 = self.module3(x)
        print(x) # modified!
        return out3
    
model = MyModel()
x = torch.tensor([[-0.5], [1.0]])
print(x)
# tensor([[-0.5000],
#        [ 1.0000]])

out = model(x)
# tensor([[-0.5000],
#         [ 1.0000]])
# tensor([[-0.5000],
#         [ 1.0000]])
# tensor([[-0.5000],
#         [ 1.0000]])
# tensor([[0.],
#         [1.]])
</code></pre>

exponential: <p>hi ptrblck,</p>
<p>Thanks for your swift response.</p>
<p>what if an inplace operation was part of the function I used in the forward pass?</p>
<p>say for example an in-place operation as part of the functions of the class I used in my Model class. Will this affect the input. if yes how can I overturn this in my forward pass?</p>
<p>I am sorry if my question sounds amateur.<br>
Thank you.</p>

ptrblck: <aside class="quote no-group" data-username="exponential" data-post="3" data-topic="155123">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/exponential/48/50889_2.png" class="avatar"> exponential:</div>
<blockquote>
<p>say for example an in-place operation as part of the functions of the class I used in my Model class. Will this affect the input.</p>
</blockquote>
</aside>
<p>I think this would match my example in <code>self.module3</code> and will thus change the input <code>x</code> or what would the difference be?</p>
<aside class="quote no-group" data-username="exponential" data-post="3" data-topic="155123">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/exponential/48/50889_2.png" class="avatar"> exponential:</div>
<blockquote>
<p>if yes how can I overturn this in my forward pass?</p>
</blockquote>
</aside>
<p>You could <code>clone()</code> the tensor before passing it to the module, which would create a copy (thus use more memory) but would avoid the inplace manipulation on the original tensor.</p>

exponential: <p>Thanks once again ptrblck!</p>
<p>so, about cloning, do you advise I clone the input in the function of the class or in  Mymodel class? Which is faster?</p>
<pre><code class="lang-auto">class MyFunctionclass(nn.Module):
   def __init__(self, somevar, somevar2)
       super(MyFunctionclass, self).__init__()
   
   def forward(self, x):
     x_ = x.clone()
     some code
 
</code></pre>
<p>Or</p>
<pre><code class="lang-auto">class MyModel(nn.Module):
    def __init__(self, depth, some var, some var1):
         super(MyModel self).__init__()

  def L1_block(self, x, before_trans = True):
    some code

  def L2_block(self, x, before_trans = True):
    some code

 def U1_block(self, x, before_trans = True):
    some code

 def U2_block(self, x, before_trans = True):
    some code

  def forward(self, x, concat_only=True):
        x_ = x.clone()
        output_var = ['l1_out', 'l2_out', 'u1_out', 'u2_out']
        output_var_t = ['l1_out_t', 'l2_out_t', 'u1_out_t', 'u2_out_t']
        l1_out = self.L1_block(x_, before_trans=True)
        l1_out_t = self.L1_block(x_, before_trans=False)
        l2_out = self.L2_block(l1_out_t, before_trans=True)
        l2_out_t = self.L2_block(l1_out_t, before_trans=False)
        u1_out = self.U1_block(l2_out_t, l1_out, l2_out, x_, before_trans=True)
        u1_out_t = self.U1_block(l2_out_t, l1_out, l2_out, x_, before_trans=False)
        u2_out = self.U2_block(u1_out_t, l1_out, l2_out, u1_out, x_, before_trans=True)
        u2_out_t = self.U2_block(u1_out_t, l1_out, l2_out, u1_out, x_, before_trans=False)
        if concat_only:
            return dict(zip(output_var, (l1_out, l2_out, u1_out, u2_out)))
        else:
            return dict(zip(output_var_t, (l1_out_t, l2_out_t, u1_out_t, u2_out_t)))
</code></pre>

ptrblck: <p>The second approach could still hit the issue as you are reusing the same <code>x_</code>. If <code>self.L1_block</code> applies an inplace operation to <code>x_</code> the next usage could be wrong.<br>
This would make sure to pass the original inputs to each module:</p>
<pre><code class="lang-python">def forward(self, x):
    out1 = self.layer1(x.clone())
    out2 = self.layer2(x.clone())
</code></pre>
<p>Note that this will add an overhead since copies are now created, but would be safe from the functional perspective in case you are unsure if any module uses an inplace op as the first operation.</p>

exponential: <p>thank you for the help!</p>