Loading tensors from a file

sinan_mutlu: <p>Hi,</p>
<p>I have a model structured in small networks in a network. After training it, I save the model with torch.save(model.state_dict() … and it can be loaded it with model.load_state_dict without any errors. To transfer the weights &amp; biases for the C++ implementation from python implementation I used the torch::jit to save them into a file. I traverse the network to get weight and bias values at each node.<br>
To test if it works I load the weights &amp; biases from the file in python-pytorch after instantiating the model, and continue to train the model but the loss value becomes very high and model parameters are not updating.</p>
<p>If I first load the trained model with model.load_state_dict and update the weights &amp; biases from the file which was saved with torch::jit the model continues to train from where it was. To hold all the layers in the networks I used torch.ModuleLists.</p>
<p>To load the parameters directly from a file after instantiating the model is there any specific thing that I have to do?</p>
<p>Thanks.</p>

ptrblck: <aside class="quote no-group" data-username="sinan_mutlu" data-post="1" data-topic="88123">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sinan_mutlu/48/26350_2.png" class="avatar"> sinan_mutlu:</div>
<blockquote>
<p>To load the parameters directly from a file after instantiating the model is there any specific thing that I have to do?</p>
</blockquote>
</aside>
<p>Not that I’m aware of and the <code>model.load_state_dict()</code> call should raise a warning, if missing or unexpected keys were found.</p>
<p>Could you post a small code snippet, which shows how you are restoring the model (in the non-working case) and how you are fixing this issue?</p>

sinan_mutlu: <p>Hi, thanks for the comment.<br>
If I load the model like this:<br>
model.load_state_dict( torch.load(path_to_saved_model), there is no problem.</p>
<p>But If I load the tensors from a file after instantiating the model I don’t get same inference performance, loss is higher than what it would be in the first epoch of the training if I had started to train it with random weights. Also it seems like the weights are not updating. The model is structured similar to this:</p>
<p>class SmallNet(nn.Module):<br>
def <strong>init</strong>(self, …)<br>
self.layers = nn.ModuleList()<br>
self.normLayers = nn.ModuleList()<br>
…<br>
…</p>
<pre><code>def setParams(self, start_ind, v_ParamList):

    for i in range(len(self.layers)):
        self.layers[i].weight.data = v_ParamList[start_ind].cuda()
        self.layers[i].weight.requires_grad = True
        self.layers[i].bias.data   = v_ParamList[start_ind+1].cuda()
        self.layers[i].bias.requires_grad = True
        start_ind += 2

    for i in range(len(self.normLayers)):
        if self.normLayers[i]!=None:
            self.normLayers [i].weight.data = v_ParamList[start_ind].cuda()
            self.normLayers[i].bias.data  = v_ParamList[start_ind+1].cuda()
            self.normLayers[i].running_mean.data = v_ParamList[start_ind+2].cuda()
            self.normLayers[i].running_var.data = v_ParamList[start_ind+3].cuda()
            self.normLayers[i].num_batches_tracked.data = v_ParamList[start_ind+4].cuda()
            start_ind += 5

    return start_ind
</code></pre>
<p>class BigNet(nn.Module):<br>
def <strong>init</strong>(self, …):<br>
self.Layer_1 = SmallNet(…)<br>
self.Layer_2 = SmallNet(…)<br>
self.Layer_3 = nn.ModuleList()</p>
<pre><code>    for i in range(10):
          self.Layer_3.append(SmallNet(....))

def setParams(self, start_ind, v_ParamList):

    start_ind = self.Layer_1.setParams(start_ind, v_ParamList)
    start_ind = self.Layer_2.setParams(start_ind, v_ParamList)
    for i in range(len(self.Layer_3)):
        start_ind = self.Layer_3[i].setParams(start_ind, v_ParamList)

    return start_ind

  def loadModelParams(self, path):
      container = torch.jit.load(path)
      v_tensors = getattr(container, "weights")
      start_ind = self.setParams(0, v_tensors)
      ....
</code></pre>
<hr>
<p>model = BigNet()<br>
model = model.float()<br>
device = utils.get_device(“cuda”)</p>
<p>if cuda == 'cuda<br>
model.to(device)</p>
<p>model.loadModelParams("…pt")</p>
<hr>
<p>If I use one SmallNet to train on MNIST dataset and save the weights&amp;biases to a file and reload them again, I don’t have any problems. But loading the weights&amp;biases of BigNet from a file doesn’t work. I checked the difference between the tensors if the model is loaded with “load_state_dict” or from a tensors file, they are same. I don’t undestand what BigNet changes when I want to load the tensors directly from a file.</p>
<p>Thanks a lot!</p>

sinan_mutlu: <p>Hi,<br>
I found the problem, I forgot to save the weights in the activation functions. PReLU for instance has a weight. After changing the code the problem is fixed.<br>
Thanks</p>