How to copy custom weights tensor into model layer weights

username: Andoni_Cortes
cooked: <p>Hello everyone,</p>
<p>I am trying to copy a tensor with custom weights into an AlexNet convolutional layer weights. I am doing as indicated below and I dont get any error. However when I load the model using torch::load and check the weights, they are different than the values that I saved with torch::save function. Am I missing something?</p>
<pre><code class="lang-auto">AlexNet model;

 model-&gt;named_parameters()["conv1.weights"] = tensor_with_custom_weights.clone().detach();

// if I check it here weights are correct, the ones that I assigned
torch::save("alexnet.pt",model);

AlexNet model1;
torch::load("alexnet.pt",model1);
//Here model1.conv1.weights are random

</code></pre>
<p>Thanks</p>

username: ptrblck
cooked: <p>After you’ve reassigned the tensor to <code>conv1.weights</code>, could you check, if these parameters are shown via:</p>
<pre><code class="lang-auto">for (const auto&amp; pair : net.named_parameters()) {
  std::cout &lt;&lt; pair.key() &lt;&lt; ": " &lt;&lt; pair.value() &lt;&lt; std::endl;
}
</code></pre>
<p>Usually, the attribute is called <code>weight</code> (without the s at the end), so maybe you are just creating a new key in the map?</p>

username: Andoni_Cortes
cooked: <p>sorry, just a typo. I meant</p>
<pre><code class="lang-auto">model-&gt;named_parameters()["conv1.weight"] = tensor_with_custom_weights.clone().detach();
</code></pre>
<p>my model structure is this:</p>
<pre><code class="lang-auto">AlexNetImpl(
  (linear1): torch::nn::Linear(in_features=9216, out_features=4096, bias=true)
  (linear2): torch::nn::Linear(in_features=4096, out_features=4096, bias=true)
  (linear3): torch::nn::Linear(in_features=4096, out_features=21, bias=true)
  (dropout): torch::nn::Dropout(p=0.5, inplace=false)
  (conv1): torch::nn::Conv2d(3, 96, kernel_size=[11, 11], stride=[4, 4])
  (conv2): torch::nn::Conv2d(96, 256, kernel_size=[5, 5], stride=[1, 1], padding=[2, 2])
  (conv3): torch::nn::Conv2d(256, 384, kernel_size=[3, 3], stride=[1, 1], padding=[1, 1])
  (conv4): torch::nn::Conv2d(384, 384, kernel_size=[3, 3], stride=[1, 1], padding=[1, 1])
  (conv5): torch::nn::Conv2d(384, 256, kernel_size=[3, 3], stride=[1, 1], padding=[1, 1])
  (lrnorm1): torch::nn::LocalResponseNorm(5, alpha=0.0001, beta=0.75, k=2)
)
</code></pre>
<p>and this is the output of the for-loop (instead of tensor values, just the sizes) :</p>
<pre><code class="lang-auto">Parameters of the model: AlexNetImpl
-----------------------------------------
   &gt; linear1.weight [4096, 9216]
   &gt; linear1.bias [4096]
   &gt; linear2.weight [4096, 4096]
   &gt; linear2.bias [4096]
   &gt; linear3.weight [21, 4096]
   &gt; linear3.bias [21]
   &gt; conv1.weight [96, 3, 11, 11]
   &gt; conv1.bias [96]
   &gt; conv2.weight [256, 96, 5, 5]
   &gt; conv2.bias [256]
   &gt; conv3.weight [384, 256, 3, 3]
   &gt; conv3.bias [384]
   &gt; conv4.weight [384, 384, 3, 3]
   &gt; conv4.bias [384]
   &gt; conv5.weight [256, 384, 3, 3]
   &gt; conv5.bias [256]
</code></pre>
<p>but still I don’t get the correct values after loading the model.<br>
by the way, thanks for the help.</p>

username: Andoni_Cortes
cooked: <p>Hello again, I tried another model modification. I am using <a href="https://github.com/pytorch/vision/blob/master/torchvision/csrc/models/alexnet.cpp" rel="nofollow noopener">torchvision alexnet model</a></p>
<pre><code class="lang-auto">AlexNetImpl(
  (features): torch::nn::Sequential(
    (0): torch::nn::Conv2d(3, 64, kernel_size=[11, 11], stride=[4, 4], padding=[2, 2])
    (1): torch::nn::Functional()
    (2): torch::nn::Functional()
    (3): torch::nn::Conv2d(64, 192, kernel_size=[5, 5], stride=[1, 1], padding=[2, 2])
    (4): torch::nn::Functional()
    (5): torch::nn::Functional()
    (6): torch::nn::Conv2d(192, 384, kernel_size=[3, 3], stride=[1, 1], padding=[1, 1])
    (7): torch::nn::Functional()
    (8): torch::nn::Conv2d(384, 256, kernel_size=[3, 3], stride=[1, 1], padding=[1, 1])
    (9): torch::nn::Functional()
    (10): torch::nn::Conv2d(256, 256, kernel_size=[3, 3], stride=[1, 1], padding=[1, 1])
    (11): torch::nn::Functional()
    (12): torch::nn::Functional()
  )
  (classifier): torch::nn::Sequential(
    (0): torch::nn::Dropout(p=0.5, inplace=false)
    (1): torch::nn::Linear(in_features=9216, out_features=4096, bias=true)
    (2): torch::nn::Functional()
    (3): torch::nn::Dropout(p=0.5, inplace=false)
    (4): torch::nn::Linear(in_features=4096, out_features=4096, bias=true)
    (5): torch::nn::Functional()
    (6): torch::nn::Linear(in_features=4096, out_features=1000, bias=true)
  )
)
</code></pre>
<p>I load it with torch::load and modify last output layer  nodes number in this way:</p>
<pre><code class="lang-auto">AlexNet model_from_torchvision;
torch::load(model_from_torchvision, "alexnet.pt");

model_from_torchvision.get()-&gt;named_modules()["classifier"]-&gt;unregister_module("6");
model_from_torchvision.get()-&gt;named_modules()["classifier"]-&gt;register_module("6", torch::nn::Linear(4096, 12));

</code></pre>
<p>The <strong>model</strong> seems to have been modified (12 nodes in the output layer):</p>
<pre><code class="lang-auto">AlexNetImpl(
  (features): torch::nn::Sequential(
    (0): torch::nn::Conv2d(3, 64, kernel_size=[11, 11], stride=[4, 4], padding=[2, 2])
    (1): torch::nn::Functional()
    (2): torch::nn::Functional()
    (3): torch::nn::Conv2d(64, 192, kernel_size=[5, 5], stride=[1, 1], padding=[2, 2])
    (4): torch::nn::Functional()
    (5): torch::nn::Functional()
    (6): torch::nn::Conv2d(192, 384, kernel_size=[3, 3], stride=[1, 1], padding=[1, 1])
    (7): torch::nn::Functional()
    (8): torch::nn::Conv2d(384, 256, kernel_size=[3, 3], stride=[1, 1], padding=[1, 1])
    (9): torch::nn::Functional()
    (10): torch::nn::Conv2d(256, 256, kernel_size=[3, 3], stride=[1, 1], padding=[1, 1])
    (11): torch::nn::Functional()
    (12): torch::nn::Functional()
  )
  (classifier): torch::nn::Sequential(
    (0): torch::nn::Dropout(p=0.5, inplace=false)
    (1): torch::nn::Linear(in_features=9216, out_features=4096, bias=true)
    (2): torch::nn::Functional()
    (3): torch::nn::Dropout(p=0.5, inplace=false)
    (4): torch::nn::Linear(in_features=4096, out_features=4096, bias=true)
    (5): torch::nn::Functional()
    (6): torch::nn::Linear(in_features=4096, out_features=12, bias=true)
  )
)
</code></pre>
<p>Now I simply save the model and then I load it into a new object.</p>
<pre><code class="lang-auto">torch::save(model_from_torchvision, "savedmodel.pt");

AlexNet model_tmp;
torch::load(model_tmp, "savedmodel.pt");
</code></pre>
<p>But <strong>model_tmp structure</strong> is not the modified one but the original one (with 1000 nodes in the output layer).</p>
<pre><code class="lang-auto">AlexNetImpl(
  (features): torch::nn::Sequential(
    (0): torch::nn::Conv2d(3, 64, kernel_size=[11, 11], stride=[4, 4], padding=[2, 2])
    (1): torch::nn::Functional()
    (2): torch::nn::Functional()
    (3): torch::nn::Conv2d(64, 192, kernel_size=[5, 5], stride=[1, 1], padding=[2, 2])
    (4): torch::nn::Functional()
    (5): torch::nn::Functional()
    (6): torch::nn::Conv2d(192, 384, kernel_size=[3, 3], stride=[1, 1], padding=[1, 1])
    (7): torch::nn::Functional()
    (8): torch::nn::Conv2d(384, 256, kernel_size=[3, 3], stride=[1, 1], padding=[1, 1])
    (9): torch::nn::Functional()
    (10): torch::nn::Conv2d(256, 256, kernel_size=[3, 3], stride=[1, 1], padding=[1, 1])
    (11): torch::nn::Functional()
    (12): torch::nn::Functional()
  )
  (classifier): torch::nn::Sequential(
    (0): torch::nn::Dropout(p=0.5, inplace=false)
    (1): torch::nn::Linear(in_features=9216, out_features=4096, bias=true)
    (2): torch::nn::Functional()
    (3): torch::nn::Dropout(p=0.5, inplace=false)
    (4): torch::nn::Linear(in_features=4096, out_features=4096, bias=true)
    (5): torch::nn::Functional()
    (6): torch::nn::Linear(in_features=4096, out_features=1000, bias=true)
  )
)
</code></pre>
<p>Why could this be?<br>
It seems that any change I make from outside the model is not persistent and it is lost after saving.<br>
Thanks.</p>

username: ptrblck
cooked: <p>That’s interesting and it seems that maybe only the parameters were loaded? Although I would assume that this would raise an error, as you would have a shape mismatch in the last linear layer.</p>
<p>Note that I don’t use the C++ frontend for training, so I’m not sure how the serialization is working correctly and I usually just stick to the <a href="https://github.com/pytorch/pytorch/blob/master/test/cpp/api/serialize.cpp">tests</a> to see a proper usage.<br>
However, I cannot find a comparable test for the fine-tuning use case.</p>
<p>I think <a class="mention" href="/u/yf225">@yf225</a> would know, what’s going on. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>