Run inference manually on a pytorch model layer by layer

gohn: <p>Hi all,</p>
<p>My goal is to replicate the forward pass of the model manually, layer by layer, and verify the results.</p>
<h3><a name="p-454278-setup-1" class="anchor" href="#p-454278-setup-1"></a>Setup:</h3>
<p>1.I first run the model end-to-end using PyTorch forward method and save the intermediate layer inputs, weights, and outputs (for both convolution and fully connected layers) in CSV format in a directory (original)</p>
<p><span class="hashtag-raw">#hooks</span> to save intermediate layer output</p>
<p>def attach_hooks(model):<br>
for name, layer in model.named_modules():<br>
if isinstance(layer, (torch.nn.Conv2d, torch.nn.Linear, torch.nn.ReLU, torch.nn.MaxPool2d, torch.nn.AdaptiveAvgPool2d, torch.nn.Dropout)):<br>
layer.register_forward_hook(lambda layer, inp, out, name=name: save_activation(name, layer, inp, out))</p>
<h1><a name="p-454278-run-inference-on-model-2" class="anchor" href="#p-454278-run-inference-on-model-2"></a>run inference on model</h1>
<pre><code>model = load_pretrained_vgg11_model()
attach_hooks(model)
output = model(input_tensor)
</code></pre>
<p>…</p>
<ol start="2">
<li>
<p>I created a second script that reads these CSV files and performs the same computations manually for each layer (e.g., convolution, ReLU, pooling, linear layers) and saves the outputs of each layer in a another directory (layer_by_layer)</p>
</li>
<li>
<p>I compare the outputs from my manual layer-by-layer calculations with the original outputs that were saved during the initial end-to-end run.</p>
</li>
</ol>
<h3><a name="p-454278-the-problem-3" class="anchor" href="#p-454278-the-problem-3"></a>The Problem:</h3>
<ol>
<li>When I compare the output tensors from my manual calculations with the stored output tensors for each layer, there are significant differences, especially in deeper layers.</li>
<li>The final prediction from my layer-by-layer inference is <strong>incorrect</strong>. Instead of predicting “beagle” (which was the correct prediction from the original inference), the model predicts “chain” after softmax is applied to the output of the last fully connected layer.</li>
</ol>
<h3><a name="p-454278-what-ive-noticed-4" class="anchor" href="#p-454278-what-ive-noticed-4"></a>What I’ve Noticed:</h3>
<ol>
<li>The maximum difference and relative errors between the computed and stored outputs grow progressively larger in deeper layers. For instance:<br>
For features.0, the maximum difference is ~13.91, with a relative error of ~4569017.<br>
For features.3, the maximum difference is ~45.15, with a relative error of ~475987.<br>
For classifier.6 (the final layer), the maximum difference is ~18.54, with a relative error of ~219.</li>
</ol>
<h3><a name="p-454278-questions-5" class="anchor" href="#p-454278-questions-5"></a>Questions:</h3>
<ol>
<li>Why are there significant differences in the intermediate outputs of the layers, and how might this lead to an incorrect final prediction?</li>
<li>Are there any differences between how PyTorch performs computations in an end-to-end inference compared to my manual layer-by-layer implementation that could explain these discrepancies and the wrong final prediction?</li>
<li>Has anyone encountered similar issues, and how can I reduce these differences to improve accuracy in the layer-by-layer manual calculations?</li>
</ol>
<p>Any guidance on resolving these discrepancies or improving the accuracy of my manual calculations would be greatly appreciated!</p>
<p>Thanks in advance!</p>

ptrblck: <ol>
<li>You didn’t describe the model architecture, but I see you are registering the forward hook on <code>nn.Dropout</code> layers, so I would start by disabling these as it’s expected to see different results during the training phase unless the code is carefully seeded.</li>
<li>Yes, there could be differences depending on the actual model implementation in in particular its <code>forward</code> method. You should make sure that your layer-by-layer execution matches the original <code>forward</code> pass.</li>
<li>Yes, but the mismatches were expected due to e.g. the usage of dropout layers.</li>
</ol>