State dict and loading of buffers

isentropic: <p>I’m training a simple vision model and for some reason loading saved buffers does not work. Other weights seem to load just fine. Not sure what could be the reason. Any suggestions are welcome</p>
<pre><code class="lang-auto">In [21]: pretrained_dict['layer1.0.relu1.running_max']
Out[21]: tensor(2.8699, device='cuda:1')

In [22]: net.layer1[0].relu1.running_max
Out[22]: tensor(2)

In [23]: net.load_state_dict(pretrained_dict)
Out[23]: &lt;All keys matched successfully&gt;

In [24]: net.layer1[0].relu1.running_max
Out[24]: tensor(2)
</code></pre>

ptrblck: <p>Could you post a minimal and executable code snippet reproducing the issue?</p>

isentropic: <p>I have resolved my issue, for those who might face a similar issue I’m providing the solution and the reason for this problem.<br>
Here’s roughly my code (doing similar stuff to batch norm, except that it is batch max)</p>
<pre data-code-wrap="python"><code class="lang-python">def Module(nn.Module):
    def __init__(self, op_mode):
        super().__init__()
        self.register_buffer('running_max', torch.tensor(1)) # the problem is here it should have been 1.0
    def forward(x):
         xmax = torch.max(torch.abs(x))
         self.running_max = (1 - self.momentum) * self.running_max + self.momentum * xmax
</code></pre>
<p>The reason why the loading is not working is that the buffer is initially assigned to be an <code>int</code> as opposed to <code>float32</code> that’s why it is silently ignored / unpromoted to float32</p>
<p>The</p>

isentropic: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> do you think this deserves an issue in torch github issue tracker?</p>

ptrblck: <p>I would assume this is expected behavior since you are explicitly defining <code>running_max</code> as an integer type via:</p>
<pre data-code-wrap="python"><code class="lang-python">torch.tensor(1).type()
# 'torch.LongTensor'
</code></pre>
<p>Copying data into the <code>LongTensor</code> will transform it instead of changing the <code>dtype</code> of the target tensor:</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.tensor(1)
x.copy_(2.3)
print(x)
# tensor(2)
</code></pre>
<p>which I would also expect.</p>

isentropic: <p>Shouldn’t there be at least some sort of warnings when loading the weights with different types instead of silently promoting them? This really threw me off:</p>
<pre><code class="lang-auto">In [23]: net.load_state_dict(pretrained_dict)
Out[23]: &lt;All keys matched successfully&gt;
</code></pre>
<p>Shouldn’t it output like <code>All keys matched, some keys had types promoted</code> or something among those lines? Spotting the difference between <code>1</code> and <code>1.0</code> I guess is something you get used to do, but man this took a while to diagnose.</p>

ptrblck: <p>It might be a good idea to add a warning and you could propose this as a feature on GitHub.<br>
If you want to use the metadata of the <code>state_dict</code>, you could use <code>m.load_state_dict(sd, assign=True)</code>.</p>

isentropic: <p>Thanks a lot I raised an issue <a href="https://github.com/pytorch/pytorch/issues/126000" class="inline-onebox" rel="noopener nofollow ugc">Inform when weights loading promotes types · Issue #126000 · pytorch/pytorch · GitHub</a><br>
Could you suggest of a way to implement that? In case this featured deemed desired I’d like to try sending PR if it is not that difficult</p>