nn.Sequential() of the children of a model has different output than that model

username: David_Dong
cooked: <p>I am loading a Resnet18 architecture from <code>torchvision.models</code>. To do some testing, just with the original <code>torchvision.model.resnet18()</code>, we get this:</p>
<pre><code class="lang-auto">resnet18 = torchvision.models.resnet18()
resnet18(
    torch.tensor(np.random.uniform(-42, 42, (16, 3, 128, 128))).float()
)
</code></pre>
<p>With the output<br>
<code>torch.Size([16, 1000])</code>.</p>
<p>However, when I do the following:</p>
<pre><code class="lang-auto">resnet18_ls = nn.Sequential(
    *list(resnet18.children())
)
resnet18_ls(
    torch.tensor(np.random.uniform(-42, 42, (16, 3, 128, 128))).float()   
)
</code></pre>
<p>I get the runtime error below:</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-73-2262642ca1ec&gt; in &lt;module&gt;()
      3 )
      4 resnet18_ls(
----&gt; 5     torch.tensor(np.random.uniform(-42, 42, (16, 3, 128, 128))).float()
      6 )

4 frames
/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py in linear(input, weight, bias)
   1846     if has_torch_function_variadic(input, weight, bias):
   1847         return handle_torch_function(linear, (input, weight, bias), input, weight, bias=bias)
-&gt; 1848     return torch._C._nn.linear(input, weight, bias)
   1849 
   1850 

RuntimeError: mat1 and mat2 shapes cannot be multiplied (8192x1 and 512x1000)
</code></pre>
<p>But aren’t these the same network? I understand that the <code>8192</code> comes from the last layer output of 512 multiplied by the batch size of 16, but I don’t see how that makes the matrix shape <code>8192 x 1</code>.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="David_Dong" data-post="1" data-topic="149283">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/david_dong/48/48702_2.png" class="avatar"> David_Dong:</div>
<blockquote>
<p>But aren’t these the same network?</p>
</blockquote>
</aside>
<p>No, since the original model uses a custom <code>forward</code> method seen <a href="https://github.com/pytorch/vision/blob/10bb5b18831ed1cc7396b599829c41954551f018/torchvision/models/resnet.py#L264-L283">here</a> while your <code>nn.Sequential</code> container will call all initialized modules in a sequential way and will thus miss the functional API calls, e.g. the <code>x = torch.flatten(x, 1)</code> line of code.</p>

username: David_Dong
cooked: <p>Thanks! I actually just saw it looking it up. Is there a reason why it skips over those calls?</p>

username: ptrblck
cooked: <p>Yes, because these functional calls are not modules and are thus not returned by <code>model.children()</code>.<br>
PyTorch allows you to define the <code>forward</code> pass as you wish including loops, conditions, functional API calls etc.<br>
<code>nn.Sequential</code> containers are used for simple models where the execution is strictly sequential and between <code>nn.Module</code>s.</p>