Forward and backward about pytorch

AndrewSoul: <p>Hi, I want to ask about the difference between the following two pieces of code:</p>
<pre><code class="lang-auto">class ModelOutputs():
    """ Class for making a forward pass, and getting:
    1. The network output.
    2. Activations from intermeddiate targetted layers.
    3. Gradients from intermeddiate targetted layers. """

    def __init__(self, model, target_layers):
        self.model = model
        self.target_layers = target_layers
        self.gradients = None

    def save_gradient(self, grad):
        self.gradients = grad

    def __call__(self, x):
        conv_outputs = None
        for name, module in self.model.features._modules.items():
            x = module(x)
            if name in self.target_layers:
                x.register_hook(self.save_gradient)
                conv_outputs = x
        output = x
        output = output.view(output.size(0), -1)
        output = self.model.classifier(output)
        return output, conv_outputs
</code></pre>
<p>I passed vgg to model in ModelOutputs()<br>
and</p>
<pre><code class="lang-auto">class VGG(nn.Module):
    def __init__(self, vgg_name):
        super(VGG, self).__init__()
        self.features = self._make_layers(cfg[vgg_name])
        self.classifier = nn.Linear(512, 10)
        self.gradients = None

    def save_gradient(self, grad):
        self.gradients = grad

    def forward(self, x):
        x = self.features(x)
        x.register_hook(self.save_gradient)
        inter = x
        x = x.view(x.size(0), -1)
        x = self.classifier(x)
        return x, inter
</code></pre>
<p>I’m just curious about what the difference between the two pieces of code above for backward and forward is? When the loss backward, will the gradients be different?<br>
Could someone help me?<br>
Appreciated for your help!</p>

ptrblck: <p>You could use some dummy data and target, calculate the loss and gradients, and compare the gradients of (some) layers using something like this:</p>
<pre><code class="lang-python">print(torch.allclose(modelA.model.features[0].weight.grad),
      modelB.features[0].weight.grad))
</code></pre>
<p>I would recommend using the second approach, since the first one is not defined as an <code>nn.Module</code>, which will yield errors, e.g. if you try to call <code>model.parameters()</code> etc.</p>

AndrewSoul: <p>Appreciated for your reply!</p>
<pre><code class="lang-auto">m_o = ModelOutputs(net, layer=40)
output, conv_output = m_o(input)
loss = cross_entropy(output, label)
loss.backward()
grad_m_o = m_o.gradients
grad_net = net.features[40].weight.grad
</code></pre>
<p>I find the grad_m_o  is different from grad_net, why?</p>
<p>And I got  another proplem: I want to load model in DataParallel and use multi-GPU in one server. like this:</p>
<pre><code class="lang-auto">os.environ['CUDA_VISIBLE_DEVICES'] = '0,1,2'
device = 'cuda' if torch.cuda.is_available() else 'cpu'
</code></pre>
<pre><code class="lang-auto">... ...
net.to(device)
net= torch.nn.DataParallel(model)
input.to(device) 
...
</code></pre>
<p>And I use the first approach in the code above.</p>
<blockquote>
<p>class ModelOutputs():</p>
</blockquote>
<pre><code class="lang-auto">m_o = ModelOutputs(net, layer)
output, conv_output = m_o(input)
</code></pre>
<p>I passed the net in ModelOutpus. When I ran the code, it did’t use multi-GPU but gpu’0.<br>
Is the problem caused by not defined as nn.Module?</p>

ptrblck: <aside class="quote no-group" data-username="AndrewSoul" data-post="4" data-topic="50089">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/andrewsoul/48/13963_2.png" class="avatar"> AndrewSoul:</div>
<blockquote>
<p>I find the grad_m_o is different from grad_net, why?</p>
</blockquote>
</aside>
<p>How large is the difference?<br>
I’m not sure, if the code you’ve posted is still the currently used one, but it seems to be that you should pass <code>targets_layers</code> as a list of strings.</p>
<aside class="quote no-group" data-username="AndrewSoul" data-post="4" data-topic="50089">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/andrewsoul/48/13963_2.png" class="avatar"> AndrewSoul:</div>
<blockquote>
<p>Is the problem caused by not defined as nn.Module?</p>
</blockquote>
</aside>
<p>I think you are not using multiple GPUs, since you are never calling the model itself (and thus its <code>forward</code> method), but split the call into submodules and call them in a loop.<br>
Could you try to call <code>net(input)</code> and check the GPU utilizations?</p>

AndrewSoul: <aside class="quote no-group" data-username="ptrblck" data-post="6" data-topic="50089">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>should pass <code>targets_layers</code> as a list of strings</p>
</blockquote>
</aside>
<p>Sorry to reply late. Yeah, I passed string actually, like: [‘40’].</p>
<aside class="quote no-group" data-username="ptrblck" data-post="6" data-topic="50089">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>How large is the difference?</p>
</blockquote>
</aside>
<p>The shape between the m_o.gradients and net.features[40] is different.</p>
<aside class="quote no-group" data-username="ptrblck" data-post="6" data-topic="50089">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Could you try to call <code>net(input)</code> and check the GPU utilizations?</p>
</blockquote>
</aside>
<p>Yes, you’re right. I didn’t call net(input) directly. How can I use multiple GPUs  using ModelOutput? Inheriting class nn.Module and define a forward function?</p>

ptrblck: <p>I’m not sure, why the shapes differ, but apparently the wrong gradients are stored.<br>
Here is a small dummy example using <code>vgg16</code>:</p>
<pre data-code-wrap="python"><code class="lang-python">grads = []
def save_grad(grad):
    grads.append(grad)

# Create model
model = models.vgg16()
model.eval()

# First approach
x = torch.randn(1, 3, 224, 224)
output = model.features(x)
output.register_hook(lambda x: save_grad(x))
output = model.avgpool(output)
output = output.view(output.size(0), -1)
output = model.classifier(output)
output.mean().backward()

# Reset
model.zero_grad()

# Second approach
output = x.clone()
for name, module in model.features._modules.items():
    output = module(output)
    if '30' in name:
        output.register_hook(lambda x: save_grad(x))

output = model.avgpool(output)
output = output.view(output.size(0), -1)
output = model.classifier(output)
output.mean().backward()

# Compare gradients
print((grads[0] == grads[1]).all())
&gt; tensor(1, dtype=torch.uint8)
</code></pre>
<p>I tried to stick to your approach and as you can see, both methods yield the same gradients.</p>
<aside class="quote no-group" data-username="AndrewSoul" data-post="7" data-topic="50089">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/andrewsoul/48/13963_2.png" class="avatar"> AndrewSoul:</div>
<blockquote>
<p>Yes, you’re right. I didn’t call net(input) directly. How can I use multiple GPUs using ModelOutput? Inheriting class nn.Module and define a forward function?</p>
</blockquote>
</aside>
<p>I guess data parallel isn’t being used, since you are not calling the model directly, but each submodule.<br>
This might be related to <a href="https://discuss.pytorch.org/t/multiple-gpu-error-data-parallel/50288/3">this issue</a>.</p>

AndrewSoul: <p>Thanks for your quick reply!</p>
<aside class="quote no-group" data-username="ptrblck" data-post="8" data-topic="50089">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>I tried to stick to your approach and as you can see, both methods yield the same gradients.</p>
</blockquote>
</aside>
<p>Yes, with register_hood. Both method yield the same gradients. Before this, I have tried:</p>
<aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="50089">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>print(torch.allclose(modelA.model.features[0].weight.grad), modelB.features[0].weight.grad))</p>
</blockquote>
</aside>
<p>I think the gradients of ModelOutputs(net, layer=40) is different from net.features[40].weight.grad.</p>

ptrblck: <aside class="quote no-group" data-username="AndrewSoul" data-post="9" data-topic="50089">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/andrewsoul/48/13963_2.png" class="avatar"> AndrewSoul:</div>
<blockquote>
<p>I think the gradients of ModelOutputs(net, layer=40) is different from net.features[40].weight.grad.</p>
</blockquote>
</aside>
<p>That might be the case and I suspect you might be registering the weight parameter and overriding it with the bias parameter later, since both will match the condition.<br>
Could you check the shape and see, if you are in fact storing the bias gradient?</p>

AndrewSoul: <aside class="quote no-group" data-username="ptrblck" data-post="10" data-topic="50089">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Could you check the shape</p>
</blockquote>
</aside>
<p>Hi, <a class="mention" href="/u/ptrblck">@ptrblck</a>. I have checked the shape.  The ConvNd weight.shape is defined as：</p>
<blockquote>
<p>self.weight = Parameter(torch.Tensor(<br>
out_channels, in_channels // groups, *kernel_size))</p>
</blockquote>
<p>So the shape of features[40].weight.grad is the same as weight’s shape.</p>
<blockquote>
<p>(C<sub>out</sub>,C<sub>in</sub>//groups, *kernel_size)</p>
</blockquote>
<p>ModelOutputs(net, layer=40): shape: (N, C<sub>out</sub>, H<sub>out</sub>, W<sub>out</sub>)<br>
I think it explains why the shape is different. Am I right?</p>

ptrblck: <p>Maybe I misunderstood the issue, but I thought you had a shape mismatch comparing the two gradient tensors?<br>
Are the shapes equal?</p>

AndrewSoul: <p>Sorry, maybe I wasn’t clear enough. I didn’t mean to mislead you. <img src="https://discuss.pytorch.org/images/emoji/apple/grimacing.png?v=12" title=":grimacing:" class="emoji" alt=":grimacing:" loading="lazy" width="20" height="20"> And thanks for your patience!</p>
<aside class="quote no-group" data-username="ptrblck" data-post="8" data-topic="50089">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Here is a small dummy example using <code>vgg16</code> :</p>
</blockquote>
</aside>
<pre><code class="lang-auto">grads = []
def save_grad(grad):
    grads.append(grad)

# Create model
model = models.vgg16()
model.eval()

# First approach
x = torch.randn(1, 3, 224, 224)
output = model.features(x)
output.register_hook(lambda x: save_grad(x))
output = model.avgpool(output)
output = output.view(output.size(0), -1)
output = model.classifier(output)
output.mean().backward()

# Reset
model.zero_grad()

# Second approach
output = x.clone()
for name, module in model.features._modules.items():
    output = module(output)
    if '30' in name:
        output.register_hook(lambda x: save_grad(x))

output = model.avgpool(output)
output = output.view(output.size(0), -1)
output = model.classifier(output)
output.mean().backward()

# Compare gradients
print((grads[0] == grads[1]).all())
&gt; tensor(1, dtype=torch.uint8)
</code></pre>
<p>The code you provide above yield the same gradients. That is, grads[0] and grads[1] are the same. But  both grads[0] and grads[1] are different from model.features[30].weight.grad in shape. Because the definition of ConvNd’s weight is:</p>
<aside class="quote no-group" data-username="AndrewSoul" data-post="11" data-topic="50089">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/andrewsoul/48/13963_2.png" class="avatar"> AndrewSoul:</div>
<blockquote>
<p>self.weight = Parameter(torch.Tensor(<br>
out_channels, in_channels // groups, *kernel_size))</p>
</blockquote>
</aside>
<p>So the shape of model.features[30].weight.grad and shape of  grads[0] /grads[1] are different, I think. <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=12" title=":grinning:" class="emoji" alt=":grinning:" loading="lazy" width="20" height="20"></p>

ptrblck: <p>I might still misunderstand the use case, but in your code example you are storing the gradients of the output of <code>self.features</code>:</p>
<pre><code class="lang-python">    def forward(self, x):
        x = self.features(x)
        x.register_hook(self.save_gradient)
</code></pre>
<p>, while you are now comparing these gradients to the <code>weight.grad</code>.</p>
<p>Also, which model are you using?<br>
I assumed it’s <code>vgg16</code> and if that’s the case, <code>features[30]</code> is an <code>nn.MaxPool2d</code> layer, without any <code>weight</code> and <code>bias</code> parameters.</p>

AndrewSoul: <p>Sorry,  I didn’t mean to confuse you. I made a mistake in my last post.</p>
<aside class="quote no-group" data-username="ptrblck" data-post="14" data-topic="50089">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Also, which model are you using?</p>
</blockquote>
</aside>
<p>I’m using vgg16 for cifar10 and I choose the same layer to compare the gradients.</p>
<aside class="quote no-group" data-username="ptrblck" data-post="14" data-topic="50089">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>I assumed it’s <code>vgg16</code> and if that’s the case, <code>features[30]</code> is an <code>nn.MaxPool2d</code> layer, without any <code>weight</code> and <code>bias</code> parameters.</p>
</blockquote>
</aside>
<p>Yeah, as you said, there isn’t any <code>weight</code> and <code>bias</code> parameters in features[30]. So I choose features[37], which is Conv2d layer in my model.</p>
<aside class="quote no-group" data-username="AndrewSoul" data-post="1" data-topic="50089">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/andrewsoul/48/13963_2.png" class="avatar"> AndrewSoul:</div>
<blockquote>
<pre><code class="lang-auto">class ModelOutputs():
</code></pre>
</blockquote>
</aside>
<p>I defined m_o like:<br>
<code>m_o = ModelOutputs (model, target_layers=['37'])</code><br>
So I compared the gradient of  ModelOutputs in layer[‘37’] and model.features[37].weight.grad.<br>
The shapes are different, like I said in my last post:</p>
<aside class="quote no-group" data-username="AndrewSoul" data-post="11" data-topic="50089">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/andrewsoul/48/13963_2.png" class="avatar"> AndrewSoul:</div>
<blockquote>
<p>self.weight = Parameter(torch.Tensor(<br>
out_channels, in_channels // groups, *kernel_size))</p>
</blockquote>
</aside>
<p>And  m_o.gradients: shape: (N, C<sub>out</sub>, H<sub>out</sub>, W<sub>out</sub>)<br>
Maybe, that can explain the difference in shape?  Did I make myself understood?<img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=12" title=":grinning:" class="emoji" alt=":grinning:" loading="lazy" width="20" height="20"></p>

ptrblck: <aside class="quote no-group" data-username="AndrewSoul" data-post="15" data-topic="50089">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/andrewsoul/48/13963_2.png" class="avatar"> AndrewSoul:</div>
<blockquote>
<p>Yeah, as you said, there isn’t any <code>weight</code> and <code>bias</code> parameters in features[30]. So I choose features[37], which is Conv2d layer in my model.</p>
</blockquote>
</aside>
<p><code>vgg16</code> does only have 30 layers on <code>.features</code>. Are you using <code>vgg16_bn</code>?</p>
<aside class="quote no-group" data-username="AndrewSoul" data-post="15" data-topic="50089">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/andrewsoul/48/13963_2.png" class="avatar"> AndrewSoul:</div>
<blockquote>
<p>So I compared the gradient of ModelOutputs in layer[‘37’] and model.features[37].weight.grad.<br>
The shapes are different, like I said in my last post:</p>
</blockquote>
</aside>
<p>Maybe, because there is still some mixup between model architectures?<br>
As you can see in my dummy example, you’ll get the same gradients with the same shapes, if you use the same model architecture.</p>
<p>Could you make sure to compare exactly the same models?<br>
Try to print the modules using <code>print(model)</code> and also compare the number of parameters.<br>
If you have a shape mismatch in a certain layer, it means that the model architecture is different.</p>

AndrewSoul: <aside class="quote no-group" data-username="ptrblck" data-post="16" data-topic="50089">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p><code>vgg16</code> does only have 30 layers on <code>.features</code> . Are you using <code>vgg16_bn</code> ?</p>
</blockquote>
</aside>
<p>Yes, I’m using vgg16_bn for cifar10:</p>
<pre><code class="lang-auto">odict_items([('features', Sequential(
  (0): Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (2): ReLU(inplace)
  (3): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (4): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (5): ReLU(inplace)
  (6): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  (7): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (8): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (9): ReLU(inplace)
  (10): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (11): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (12): ReLU(inplace)
  (13): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  (14): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (15): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (16): ReLU(inplace)
  (17): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (18): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (19): ReLU(inplace)
  (20): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (21): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (22): ReLU(inplace)
  (23): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  (24): Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (25): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (26): ReLU(inplace)
  (27): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (28): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (29): ReLU(inplace)
  (30): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (31): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (32): ReLU(inplace)
  (33): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  (34): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (35): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (36): ReLU(inplace)
  (37): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (38): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (39): ReLU(inplace)
  (40): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (41): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (42): ReLU(inplace)
  (43): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  (44): AvgPool2d(kernel_size=1, stride=1, padding=0)
)), ('classifier', Linear(in_features=512, out_features=10, bias=True))])
</code></pre>
<aside class="quote no-group" data-username="ptrblck" data-post="16" data-topic="50089">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Could you make sure to compare exactly the same models?</p>
</blockquote>
</aside>
<p>I just use the same model as mentioned above. And I compare the features[37] with same model using m_o.gradients and model.features[37].weight.grad.</p>