Bottleneck attribution error

Felix_Labelle: <p>I’m trying to profile code using the bottleneck util. Running <code>CUDA_VISIBLE_DEVICES=2 python3 -m torch.utils.bottleneck /root/mount/Matterport3DSimulator/scripts/multimodal_filter.py --mode eval --batch_size 3000 --checkpoint best_model.pt</code>, gives the following error:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/a/3a051f46024d8f96fa7b33469e7486f6688d971d.png" data-download-href="https://discuss.pytorch.org/uploads/default/3a051f46024d8f96fa7b33469e7486f6688d971d" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/a/3a051f46024d8f96fa7b33469e7486f6688d971d_2_690x161.png" alt="image" data-base62-sha1="8hgFnKco9CZNdhjGafRSy716QAl" width="690" height="161" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/a/3a051f46024d8f96fa7b33469e7486f6688d971d_2_690x161.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/a/3a051f46024d8f96fa7b33469e7486f6688d971d_2_1035x241.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/a/3a051f46024d8f96fa7b33469e7486f6688d971d_2_1380x322.png 2x" data-dominant-color="060606"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1898×445 28.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The class its referring to is a class declared in the multimodal_filter script. That script works with no issues.</p>
<p>I’m using docker (nvidia-docker), PyTorch 1.4.0, Cuda compilation tools, release 10.0 (I’m assuming this is the cuda version).</p>

ptrblck: <p>Did you store the complete model or its <code>state_dict</code>?<br>
The latter case is recommended, but I haven’t seen this particular issue with the former approach yet, so not sure, if this might cause the error.</p>

Felix_Labelle: <p>The model is copied using deepcopy, stored in a dictionary with other information required to run the model, and pickled (Code below). I’ll try using the <code>state_dict</code> instead of storing the entire, see if that fixes the issue.</p>
<p>Out of curiosity, why is saving only the <code>state_dict</code> recommend?</p>
<p>Here is storage part of the code:</p>
<pre><code class="lang-auto">        if best_test_acc &lt;= (test_correct/test_total):
            print("Best model saved")
            bestModel = deepcopy(model)
            best_test_acc = (test_correct/test_total)
    checkpoint_data = {
            "vocab": vocab,
            "model" : bestModel,
            "processor" : processor,
            "test_acc" : best_test_acc
            }
    return checkpoint_data

</code></pre>
<p>Here is the pickling, the code above returns the variable <code>checkpoint</code>.</p>
<pre><code class="lang-auto">        checkpoint = train(args)
        with open(args.checkpoint, 'wb+') as pk_fh:
            pickle.dump(checkpoint, pk_fh)
</code></pre>

ptrblck: <p>If you store the complete model and would like to reload it, you would have to make sure to recreate the directory structure as described in the <a href="https://pytorch.org/docs/stable/notes/serialization.html">serialization docs</a> and might thus easily break.</p>

Felix_Labelle: <p>Serialization of the entire model was the issue. Thanks for your help!</p>