Was this possible in PyTorch previous releases?

username: blackbirdbarber
cooked: <p>Found some <a href="https://github.com/XavierLinNow/deepdream_pytorch/blob/master/deepdream.py#L32" rel="noopener nofollow ugc">deep dream code</a></p>
<pre><code class="lang-auto">        act_value = model.forward(X_Variable, end_layer)
        diff_out = object(act_value, guide_features)
        act_value.backward(diff_out)
</code></pre>
<p>As I checked the <code>model.forward</code> should not have the second parameter. Just the one input tensor.</p>
<p>Can someone explain me the third line. Since <code>act_value</code> should be the activation of some <code>end_layer</code> what is <code>diff_out</code>. Should this be a gradient of some kind, since <a href="https://pytorch.org/docs/stable/generated/torch.Tensor.backward.html" rel="noopener nofollow ugc"><code>backward</code></a>?</p>
<p>The only thing I liked with this “solution” was the deep dreaming like:</p>
<pre><code class="lang-auto">out=model(input)
out.backward()
</code></pre>
<p>Is this possible or I need a loss function?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="blackbirdbarber" data-post="1" data-topic="159597">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/6de8d8/48.png" class="avatar"> blackbirdbarber:</div>
<blockquote>
<p>As I checked the <code>model.forward</code> should not have the second parameter. Just the one input tensor.</p>
</blockquote>
</aside>
<p>The user defines a custom <code>MyResNet</code> which accepts two input arguments in its <code>forward</code> as seen <a href="https://github.com/XavierLinNow/deepdream_pytorch/blob/8b70139bab51b339f45c026a000eb4631f1f9fd8/resnet.py#L89-L103">here</a>:</p>
<pre data-code-wrap="python"><code class="lang-python">class MyResnet(models.resnet.ResNet):
    def forward(self, x, n_layer=3):
        x = self.conv1(x)
        x = self.bn1(x)
        x = self.relu(x)
        x = self.maxpool(x)

        layers = [self.layer1, self.layer2, self.layer3, self.layer4]

        for i_layer in range(n_layer):
            x = layers[i_layer](x)
        #         x = self.layer2(x)
        #         x = self.layer3(x)
        #         x = self.layer4(x)
        return x
</code></pre>
<aside class="quote no-group" data-username="blackbirdbarber" data-post="1" data-topic="159597">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/6de8d8/48.png" class="avatar"> blackbirdbarber:</div>
<blockquote>
<p>Can someone explain me the third line. Since <code>act_value</code> should be the activation of some <code>end_layer</code> what is <code>diff_out</code>. Should this be a gradient of some kind, since <a href="https://pytorch.org/docs/stable/generated/torch.Tensor.backward.html"><code>backward</code></a>?</p>
</blockquote>
</aside>
<p>Yes, <code>diff_out</code> is the gradient.</p>

username: blackbirdbarber
cooked: <p>Thanks for reaching out. It is still confusing for me.<br>
How do you justify this part:</p>
<pre><code class="lang-auto">layers = [self.layer1, self.layer2, self.layer3, self.layer4]

        for i_layer in range(n_layer):
            x = layers[i_layer](x)
        #         x = self.layer2(x)
        #         x = self.layer3(x)
        #         x = self.layer4(x)
        return x
</code></pre>
<p>I don’t see <code>self.layer2</code> is defined anywhere. If it is not defined it should be <code>None</code> right?</p>
<p>And a second question; it is not so obvious for me.</p>
<pre><code class="lang-auto">out=model(input)
out.backward(gradient)
</code></pre>
<p>I know input has requires_grad. When I <code>backward()</code> gradients should be calculated for every requires grad variable. Input is a leaf since it is user defined. What is the use of the gradient?</p>
<p>In the <a href="https://pytorch.org/docs/stable/generated/torch.Tensor.backward.html" rel="noopener nofollow ugc">documentation</a> students read:</p>
<blockquote>
<p>It should be a tensor of matching type and location, that contains the gradient of the differentiated function w.r.t. <code>self</code>.</p>
</blockquote>
<p>What is the meaning of w.r.t <code>self</code>?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="blackbirdbarber" data-post="3" data-topic="159597">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/6de8d8/48.png" class="avatar"> blackbirdbarber:</div>
<blockquote>
<p>I don’t see <code>self.layer2</code> is defined anywhere. If it is not defined it should be <code>None</code> right?</p>
</blockquote>
</aside>
<p><code>MyResnet</code> is derived from <code>torchvision.models.resnet.Resnet</code> as seen in my link, which defines <code>layer2</code> as:</p>
<pre data-code-wrap="python"><code class="lang-python">print(model.layer2)
Sequential(
  (0): Bottleneck(
    (conv1): Conv2d(256, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
    (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
    (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (conv3): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
    (bn3): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (relu): ReLU(inplace=True)
    (downsample): Sequential(
      (0): Conv2d(256, 512, kernel_size=(1, 1), stride=(2, 2), bias=False)
      (1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    )
  )
  (1): Bottleneck(
    (conv1): Conv2d(512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
    (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
    (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (conv3): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
    (bn3): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (relu): ReLU(inplace=True)
  )
  (2): Bottleneck(
    (conv1): Conv2d(512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
    (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
    (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (conv3): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
    (bn3): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (relu): ReLU(inplace=True)
  )
  (3): Bottleneck(
    (conv1): Conv2d(512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
    (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
    (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (conv3): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
    (bn3): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (relu): ReLU(inplace=True)
  )
)
</code></pre>
<aside class="quote no-group" data-username="blackbirdbarber" data-post="3" data-topic="159597">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/6de8d8/48.png" class="avatar"> blackbirdbarber:</div>
<blockquote>
<p>What is the use of the gradient?</p>
</blockquote>
</aside>
<p>The gradient is used to update trainable parameters of your model.</p>
<aside class="quote no-group" data-username="blackbirdbarber" data-post="3" data-topic="159597">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/6de8d8/48.png" class="avatar"> blackbirdbarber:</div>
<blockquote>
<p>What is the meaning of w.r.t <code>self</code>?</p>
</blockquote>
</aside>
<p><code>self</code> refers to the module itself, which <code>.backward()</code> is called on.<br>
In your example you are passing the gradient directly to <code>out</code>’s <code>backward</code>. If you are not passing the gradient explicitly to <code>backward</code> a gradient of <code>dL/dL = 1.</code> will be used by default.</p>

username: blackbirdbarber
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="159597">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>In your example you are passing the gradient directly to <code>out</code>’s <code>backward</code>.</p>
</blockquote>
</aside>
<p>
Practically you can define the root initial gradient this way, which would be 1 otherwise, since <code>out</code> is the “root” node as it is called. But thinking of this more, could we control this with the lambda?
</p>
<p>After some checking …<br>
Gradients are there to suppress <em>RuntimeError: grad can be implicitly created only for scalar outputs</em> which happens in case we don’t use the reducer function for the output such as <code>torch.sum</code>.</p>
<p>Now this brings me to the second row, I was not able to create example so that the line works:</p>
<pre><code class="lang-auto">diff_out = object(act_value, guide_features)
</code></pre>
<p>What I did so far:</p>
<pre><code class="lang-auto">act_value = torch.tensor([1.0, 2.0, 3.0])
guide_features = torch.tensor([3.0, 4.0, 5.0])

diff_out = object(act_value, guide_features) #TypeError: object() takes no arguments

print(act_value)
print(guide_features)
print(diff_out)
</code></pre>
<p>Re <code>layer2</code> got that. This is present in <code>torchvision</code>. I didn’t expect <code>torchvision</code> was 5 years old. Some Resnet hub models I used just have <code>layers</code>.</p>

username: blackbirdbarber
cooked: <p>Can someone provide some clue on this second line:</p>
<pre><code class="lang-auto">diff_out = object(act_value, guide_features)
</code></pre>
<p>Was this <code>diff_out</code> gradient creation possible just in Python 2? Any clue how this would work not getting the TypeError as above.</p>

username: blackbirdbarber
cooked: <p>I see it now. <code>object</code> is the objective function taken from the program arguments.</p>