Control the behavior of neural network by condition

username: MrRobot
cooked: <h2>Problem</h2>
<p>I am trying to develop an architecture that haves some computation that is only triggered by some external signal (for example, when the loss between two epochs does not change much).</p>
<p>However, I am not sure how I would implement this because of two reasons</p>
<ul>
<li>If I write a conditional in the model definition that looks like following. I am not sure this conditional will be executed</li>
</ul>
<pre><code class="lang-python">class NeuralNet(nn.Module):
  # ...
  def forward(self, x):
    if &lt;conditional&gt;:
      # do something
    x = ...
    return x
</code></pre>
<ul>
<li>I could not use hook function to get this done since my operation will affect both forward and backward pass, but there is a known bug of <code>register_backward_hook</code>.</li>
</ul>
<p>Any input is appreciated!</p>

username: beaupreda
cooked: <p>Hello,</p>
<aside class="quote no-group" data-username="MrRobot" data-post="1" data-topic="62294">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/b38774/48.png" class="avatar"> MrRobot:</div>
<blockquote>
<p>If I write a conditional in the model definition that looks like following. I am not sure this conditional will be executed</p>
</blockquote>
</aside>
<p>Yes conditionals in the the <code>forward</code> function are executed since PyTorch builds the computation graph dynamically (at runtime) from the operations in the <code>forward</code> function. For example, here is the <code>forward</code> function of a <code>BasicBlock</code> (<a href="https://github.com/pytorch/vision/blob/master/torchvision/models/resnet.py#L57" rel="noopener nofollow ugc">code</a>) in PyTorch’s implementation of ResNet:</p>
<pre><code class="lang-auto">    def forward(self, x):
        identity = x

        out = self.conv1(x)
        out = self.bn1(out)
        out = self.relu(out)

        out = self.conv2(out)
        out = self.bn2(out)

        if self.downsample is not None:
            identity = self.downsample(x)

        out += identity
        out = self.relu(out)

    return out
</code></pre>

username: MrRobot
cooked: <p>Thank you for your reply. I think this is a major advantage of PyTorch over other TF.</p>
<p>Then here comes the second question.</p>
<p>In the implementation of ResNet, the <code>downsample</code> variable is passed to <code>BasicBlock</code> while initialization (what I call “internal” signal). However, in my use case, the conditional is activated by some “external” signal like number of epochs, so how do I pass this external signal to forward pass.</p>
<p>There are two thoughts</p>
<ul>
<li>Declare some global variables using <code>global VAR</code> and let forward function to access it.</li>
<li>Directly pass <code>VAR</code> through <code>forward</code>, which makes signature look like</li>
</ul>
<pre><code class="lang-python">def forward(self, x, VAR):
   if VAR:
      # do something
</code></pre>
<p>So which one is correct and preferred?</p>

username: beaupreda
cooked: <p>I think both can be correct since they will achieve the same results, but I would prefer the second approach.  I do not like to use global variables and they are generally seen as bad practice. Maybe someone more knowledgeable will chime in and give you a different answer, that’s just my two cents <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>