Weights loaded incorrectly

sigma_x: <p>I was f’tuning VGG network for style transfer (Gatys et al, 2015) and kept getting results that made no sense, so after some debugging it turned out I uploaded the weights incorrectly. Can someone point out my error, because I tried it on other models, and it seemed to work?</p>
<pre><code class="lang-auto">import vgg16
pretrained_weights = torch.load(os.path.join('pretrained_weights', 'vgg16-00b39a1b.pth'))
feature_extractor = vgg16.vgg16(pretrained=False)

for _n,par in feature_extractor.named_parameters():  
    par.requires_grad = False
    par = copy.deepcopy(pretrained_weights[_n]) 
    par = par.to(device)
</code></pre>
<p>vgg16 is the same vgg16.py from torchvision.models.</p>
<p>I believe <code>copy.deepcopy</code> didn’t change the parameters for some reason</p>

alex.veuthey: <p>Did you try the recommended way of loading weights <a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html" rel="nofollow noopener">in the tutorials</a> ? Does it not work in your case ?</p>

sigma_x: <p>I did later, and it fixed the issue. My question is, why this doesn’t work.</p>

albanD: <p>Hi,</p>
<p>This does not work because <code>par = foo</code> is assigning to the object foo the new name of par. What was in <code>par</code> before is deleted.<br>
If you want to write in the par Tensor, you need to use an inplace operation like <code>par.copy_(foo)</code> to copy into the Tensor that is in <code>par</code>.</p>
<p>Also the <code>.to()</code> operation is always out of place, so you cannot do it like that. You will have to use <code>feature_extractor.to(device)</code> to move your weifghts to the right device.</p>

sigma_x: <p>Thanks, I’m sorry why is that operation ‘out of place’? This won’t put the tensor on CUDA?</p>

albanD: <p>The code below should be clear.</p>
<pre><code class="lang-auto">t = some_cpu_tensor
t.cuda()
# here t is not on the the gpu

t = some_cpu_tensor
tc = t.cuda()
# here tc is on the the gpu

t = some_cpu_tensor
t = t.cuda()
# here t is on the the gpu
</code></pre>

mailcorahul: <p><a class="mention" href="/u/sigma_x">@sigma_x</a> Just curious, why aren’t you using<br>
<code> feature_extractor = vgg16.vgg16(pretrained=True)</code>?</p>

sigma_x: <p>but in my code</p>
<p><code>par=par.to(device)</code></p>
<p>are you saying that <code>par</code> tensor will not be placed on device?</p>

albanD: <p>The thing is that the Tensor itself and the python name associated to it are two different things.<br>
if you have <code>par</code> being some tensor. When you do <code>par.copy_()</code>, you write inplace into the tensor named <code>par</code>. If you do <code>par = par + 1</code> then you create a new tensor that will contain the value of <code>par</code> plus <code>1</code>, then associate that new tensor the name <code>par</code>. See that the Tensor that corresponds to <code>par</code> at the end is not the same as the original one.</p>
<p>In your case, if you do <code>par = par.cuda()</code>, after that line, <code>par</code> will point to a cuda tensor. But the original Tensor that is contained in the network won’t be changed.</p>

sigma_x: <p>In your code snippet</p>
<pre><code class="lang-auto">t = some_cpu_tensor
t = t.cuda()
</code></pre>
<p>isn’t this what I did to put <code>par</code> on CUDA with <code>par=par.to(device)</code>. I don’t see the difference between putting the whole model <code>model=model.to(device)</code> and <code>par</code> tensor. Are you saying that a single tensor can only be put on cuda with <code>t=t.cuda()</code> command?</p>

albanD: <p>Hi,</p>
<p>The difference is that <code>model.to(device)</code> is inplace while <code>tensor.to(device)</code> is not:</p>
<p>Does the sample below helps?</p>
<pre><code class="lang-python">import torch
import copy

# I will use `.double()` instead of `.cuda()` because my local
# machine does not have cuda, but they behave the same way

tensor = torch.zeros(10)
model = torch.nn.Linear(10, 10)

new_val = torch.ones(10)
def your_copy(par):
  par = copy.deepcopy(new_val)
  par = par.double()

print("Your changes are out of place and so not reflected outside")
print("Before your copy on tensor")
print(tensor)
print(tensor.type())

your_copy(tensor)

print("After your copy on tensor")
print(tensor)
print(tensor.type())

print("Models are changed inplace:")
print("Before on module")
print(model.weight.type())

model.double()

print("After on module")
print(model.weight.type())


def my_suggestion(par):
  par.copy_(new_val)
  # No way to change type here inplace

print("With inplace change, you can change value and have it reflected outside")
print("Before my copy")
print(tensor)

my_suggestion(tensor)

print("After my copy")
print(tensor)
</code></pre>
<pre><code class="lang-auto"></code></pre>

sigma_x: <p><span class="mention">@malicorahul</span>: some layers can be modified so the number of weights does not match. It’s better then to load weights manually.</p>