Finetuning Image Colorization model

username: skhu17
cooked: <p>Hi, I was wondering if anyone can off their aid.<br>
I am using someone else’s repo that I want to finetune, it is a image colorization nn. I want to utilise the models and introduce my own dataset to colorize various natural scenery.<br>
I am having difficulty finetuning as most of the tutorials are based on the pretrained models. I am currently struggling on loading their models and training my dataset.<br>
The git hub is called “InstColorization” by ericsujw. <a href="https://github.com/ericsujw/InstColorization" rel="noopener nofollow ugc">https://github.com/ericsujw/InstColorization</a></p>
<p>Thanks in advance! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: ptrblck
cooked: <p>Where are you stuck and what is not working? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: skhu17
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, where do I begin <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=9" title=":sweat_smile:" class="emoji" alt=":sweat_smile:">. I solved some of my preexisting issues but the first issue I had was listed here, <a href="https://discuss.pytorch.org/t/solved-keyerror-unexpected-key-module-encoder-embedding-weight-in-state-dict/1686/14" class="inline-onebox">[solved] KeyError: 'unexpected key "module.encoder.embedding.weight" in state_dict'</a>.<br>
I originally tried to change my model to nn.dataparallel but I was running into all sorts of attribute errors, so I eventually used ‘load_state_dict(state_dict,strict=False)’. My question now is that will having strict=false effect my model, or should I go for the solution in the above link to remove the ‘.module’ prefix.</p>
<p>Cheers <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=9" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:"></p>

username: ptrblck
cooked: <p>Go for the <code>.module</code> solution. Otherwise <code>strict=False</code> might just drop mismatched keys and your could end up with a randomly initialized model.</p>

username: skhu17
cooked: <p>Thanks, I will definitely give that a shot!<br>
I have another question regarding fine tuning, I am following this tutorial: <a href="https://pytorch.org/tutorials/beginner/finetuning_torchvision_models_tutorial.html" rel="noopener nofollow ugc">https://pytorch.org/tutorials/beginner/finetuning_torchvision_models_tutorial.html</a></p>
<p>I have a param_to_update tensor which I want to feed into my optim function. Currently my optimizer func is called as self.optimizer_G = torch.optim.Adam(self.netG.parameters(),<br>
lr=opt.lr, betas=(opt.beta1, 0.999))</p>
<p>If I want to call the function, how do I set the netG.parameters() = param_to_update?<br>
Sorry if my explanation is a bit confusing.</p>

username: ptrblck
cooked: <p>If <code>param_to_update</code> is not part of the <code>netG.parameters()</code>, you could pass it additionally to the optimizer as:</p>
<pre><code class="lang-python">self.optimizer_G = torch.optim.Adam(list(self.netG.parameters()) + [param_to_update], lr=opt.lr, betas=(opt.beta1, 0.999))
</code></pre>

username: skhu17
cooked: <p>So in the tutorial they passed param_to_update through the optim function as such <code>optimizer_ft = optim.SGD(params_to_update, lr=0.001, momentum=0.9)</code>.</p>
<p>So in my case I was hoping to pass my param_to_update into my optimizer function by only calling the function. The function is in a different script file so I thought the only way to call the funcion with my param_to_update tensor was to update my self.netG.parameters(). So is there a way to set self.netG.parameters() = param_to_update</p>
<p>Sorry for the silly questions and thanks for the help!</p>

username: ptrblck
cooked: <p>I don’t understand, why you would need to add this parameter to the model parameters.<br>
Wouldn’t it work, if you pass the parameter with <code>netG.parameters()</code> directly to the optimizer?<br>
If you want to add the parameter later, you could still use <code>optimizer.add_param_group</code>.</p>

username: skhu17
cooked: <p>I was hoping to pass <code>param_to_update</code> into the optimizer instead of <code>netG.parameters()</code>, but im not entirely too sure how to achieve this.</p>
<p>I will try follow your method though, thanks for the help <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: skhu17
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, I was hoping you could aid me on another problem. I am currently getting this error <code>RuntimeError: element 0 of variables does not require grad and does not have a grad_fn</code> and I have seen your previous solutions to this. The error occurs when the .backward() func is called, and I believe it is due to the loss functions and them possibly having the required_grad = False?</p>
<p>Here is my optimize_parameter and forward() func:</p>
<blockquote>
<p>def <strong>forward</strong>(self):</p>
<pre><code>    if self.opt.stage == 'full' or self.opt.stage == 'instance':

        (_, self.fake_B_reg) = self.netG(self.real_A, self.hint_B, self.mask_B)
    else:
        print('Error! Wrong stage selection!')
        exit()
</code></pre>
</blockquote>
<blockquote>
<p>def <strong>optimize_parameters</strong>(self, optimize):</p>
<pre><code>    self.forward()

    optimize.zero_grad()

    if self.opt.stage == 'full' or self.opt.stage == 'instance':

        self.loss_L1 = torch.mean(self.criterionL1(self.fake_B_reg.type(torch.cuda.FloatTensor),

                                                    self.real_B.type(torch.cuda.FloatTensor)))

        self.loss_G = 10 * torch.mean(self.criterionL1(self.fake_B_reg.type(torch.cuda.FloatTensor),

                                                    self.real_B.type(torch.cuda.FloatTensor)))
</code></pre>
<p>else:</p>
<pre><code>        print('Error! Wrong stage selection!')

        exit()

    self.loss_G.backward()

    optimize.step()
</code></pre>
</blockquote>
<p>Any help is much appreciated <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

username: ptrblck
cooked: <p>The computation graph seems to be detached at one point.<br>
Could you check, if the model output and the loss tensors have a valid <code>.grad_fn</code>?</p>

username: skhu17
cooked: <p>I have tried to print the .grad_fn and the requires_grad of the model and the loss tensors. On both occasions the loss tensors does not print anything. Not entirely sure If I am getting this right.</p>
<p>I tried print the first iteration of the loss tensor and it displayed <code>tensor(2.155, device:'cuda:0')</code></p>

username: ptrblck
cooked: <p>Are you wrapping the forward pass in a <code>with torch.no_grad()</code> block or disabling the gradient calculation globally?<br>
If not, could you post the model definition, please?</p>

username: skhu17
cooked: <p>I have not used the torch.no_grad(), Here’s the model that I am working on</p>
<p>class SIGGRAPHGenerator(nn.Module):</p>
<pre><code>def __init__(self, input_nc, output_nc, norm_layer=nn.BatchNorm2d, use_tanh=True, classification=True):

    super(SIGGRAPHGenerator, self).__init__()

    self.input_nc = input_nc

    self.output_nc = output_nc

    self.classification = classification

    use_bias = True

    # Conv1

    model1=[nn.Conv2d(input_nc, 64, kernel_size=3, stride=1, padding=1, bias=use_bias),]

    model1+=[nn.ReLU(True),]

    model1+=[nn.Conv2d(64, 64, kernel_size=3, stride=1, padding=1, bias=use_bias),]

    model1+=[nn.ReLU(True),]

    model1+=[norm_layer(64),]

    # add a subsampling operation

    # Conv2

    model2=[nn.Conv2d(64, 128, kernel_size=3, stride=1, padding=1, bias=use_bias),]

    model2+=[nn.ReLU(True),]

    model2+=[nn.Conv2d(128, 128, kernel_size=3, stride=1, padding=1, bias=use_bias),]

    model2+=[nn.ReLU(True),]

    model2+=[norm_layer(128),]

    # add a subsampling layer operation

    # Conv3

    model3=[nn.Conv2d(128, 256, kernel_size=3, stride=1, padding=1, bias=use_bias),]

    model3+=[nn.ReLU(True),]

    model3+=[nn.Conv2d(256, 256, kernel_size=3, stride=1, padding=1, bias=use_bias),]
    model3+=[nn.ReLU(True),]
    model3+=[nn.Conv2d(256, 256, kernel_size=3, stride=1, padding=1, bias=use_bias),]

    model3+=[nn.ReLU(True),]

    model3+=[norm_layer(256),]

    # add a subsampling layer operation

    # Conv4
    model4=[nn.Conv2d(256, 512, kernel_size=3, stride=1, padding=1, bias=use_bias),]
    model4+=[nn.ReLU(True),]

    model4+=[nn.Conv2d(512, 512, kernel_size=3, stride=1, padding=1, bias=use_bias),]

    model4+=[nn.ReLU(True),]

    model4+=[nn.Conv2d(512, 512, kernel_size=3, stride=1, padding=1, bias=use_bias),]

    model4+=[nn.ReLU(True),]

    model4+=[norm_layer(512),]

    # Conv5

    model5=[nn.Conv2d(512, 512, kernel_size=3, dilation=2, stride=1, padding=2, bias=use_bias),]
    model5+=[nn.ReLU(True),]


    model5+=[nn.Conv2d(512, 512, kernel_size=3, dilation=2, stride=1, padding=2, bias=use_bias),]

    model5+=[nn.ReLU(True),]

    model5+=[nn.Conv2d(512, 512, kernel_size=3, dilation=2, stride=1, padding=2, bias=use_bias),]

    model5+=[nn.ReLU(True),]

    model5+=[norm_layer(512),]

    # Conv6

    model6=[nn.Conv2d(512, 512, kernel_size=3, dilation=2, stride=1, padding=2, bias=use_bias),]

    model6+=[nn.ReLU(True),]

    model6+=[nn.Conv2d(512, 512, kernel_size=3, dilation=2, stride=1, padding=2, bias=use_bias),]

    model6+=[nn.ReLU(True),]

    model6+=[nn.Conv2d(512, 512, kernel_size=3, dilation=2, stride=1, padding=2, bias=use_bias),]

    model6+=[nn.ReLU(True),]

    model6+=[norm_layer(512),]

    # Conv7

    model7=[nn.Conv2d(512, 512, kernel_size=3, stride=1, padding=1, bias=use_bias),]

    model7+=[nn.ReLU(True),]

    model7+=[nn.Conv2d(512, 512, kernel_size=3, stride=1, padding=1, bias=use_bias),]

    model7+=[nn.ReLU(True),]

    model7+=[nn.Conv2d(512, 512, kernel_size=3, stride=1, padding=1, bias=use_bias),]

    model7+=[nn.ReLU(True),]

    model7+=[norm_layer(512),]

    # Conv7

    model8up=[nn.ConvTranspose2d(512, 256, kernel_size=4, stride=2, padding=1, bias=use_bias)]

    model3short8=[nn.Conv2d(256, 256, kernel_size=3, stride=1, padding=1, bias=use_bias),]

    model8=[nn.ReLU(True),]

    model8+=[nn.Conv2d(256, 256, kernel_size=3, stride=1, padding=1, bias=use_bias),]
    model8+=[nn.ReLU(True),]


    model8+=[nn.Conv2d(256, 256, kernel_size=3, stride=1, padding=1, bias=use_bias),]

    model8+=[nn.ReLU(True),]

    model8+=[norm_layer(256),]

    # Conv9

    model9up=[nn.ConvTranspose2d(256, 128, kernel_size=4, stride=2, padding=1, bias=use_bias),]
    model2short9=[nn.Conv2d(128, 128, kernel_size=3, stride=1, padding=1, bias=use_bias),]

    # add the two feature maps above        

    model9=[nn.ReLU(True),]

    model9+=[nn.Conv2d(128, 128, kernel_size=3, stride=1, padding=1, bias=use_bias),]

    model9+=[nn.ReLU(True),]

    model9+=[norm_layer(128),]

    # Conv10

    model10up=[nn.ConvTranspose2d(128, 128, kernel_size=4, stride=2, padding=1, bias=use_bias),]

    model1short10=[nn.Conv2d(64, 128, kernel_size=3, stride=1, padding=1, bias=use_bias),]

    # add the two feature maps above

    model10=[nn.ReLU(True),]

    model10+=[nn.Conv2d(128, 128, kernel_size=3, dilation=1, stride=1, padding=1, bias=use_bias),]

    model10+=[nn.LeakyReLU(negative_slope=.2),]

    # classification output - possibly change this output

    model_class=[nn.Conv2d(256, 529, kernel_size=1, padding=0, dilation=1, stride=1, bias=use_bias),]

    # regression output

    model_out=[nn.Conv2d(128, 2, kernel_size=1, padding=0, dilation=1, stride=1, bias=use_bias),]

    if(use_tanh):

        model_out+=[nn.Tanh()]

    self.model1 = nn.Sequential(*model1)

    self.model2 = nn.Sequential(*model2)

    self.model3 = nn.Sequential(*model3)

    self.model4 = nn.Sequential(*model4)

    self.model5 = nn.Sequential(*model5)

    self.model6 = nn.Sequential(*model6)

    self.model7 = nn.Sequential(*model7)

    self.model8up = nn.Sequential(*model8up)

    self.model8 = nn.Sequential(*model8)

    self.model9up = nn.Sequential(*model9up)

    self.model9 = nn.Sequential(*model9)

    self.model10up = nn.Sequential(*model10up)

    self.model10 = nn.Sequential(*model10)

    self.model3short8 = nn.Sequential(*model3short8)

    self.model2short9 = nn.Sequential(*model2short9)

    self.model1short10 = nn.Sequential(*model1short10)

    self.model_class = nn.Sequential(*model_class)

    self.model_out = nn.Sequential(*model_out)

    self.upsample4 = nn.Sequential(*[nn.Upsample(scale_factor=4, mode='nearest'),])

    self.softmax = nn.Sequential(*[nn.Softmax(dim=1),])

def forward(self, input_A, input_B, mask_B):

    conv1_2 = self.model1(torch.cat((input_A,input_B,mask_B),dim=1))

    conv2_2 = self.model2(conv1_2[:,:,::2,::2])

    conv3_3 = self.model3(conv2_2[:,:,::2,::2])

    conv4_3 = self.model4(conv3_3[:,:,::2,::2])

    conv5_3 = self.model5(conv4_3)

    conv6_3 = self.model6(conv5_3)

    conv7_3 = self.model7(conv6_3)

    conv8_up = self.model8up(conv7_3) + self.model3short8(conv3_3)

    conv8_3 = self.model8(conv8_up)

    if(self.classification):

        out_class = self.model_class(conv8_3)

        conv9_up = self.model9up(conv8_3.detach()) + self.model2short9(conv2_2.detach())

        conv9_3 = self.model9(conv9_up)

        conv10_up = self.model10up(conv9_3) + self.model1short10(conv1_2.detach())

        conv10_2 = self.model10(conv10_up)

        out_reg = self.model_out(conv10_2)

    else:

        out_class = self.model_class(conv8_3.detach())

        conv9_up = self.model9up(conv8_3) + self.model2short9(conv2_2)

        conv9_3 = self.model9(conv9_up)

        conv10_up = self.model10up(conv9_3) + self.model1short10(conv1_2)

        conv10_2 = self.model10(conv10_up)

        out_reg = self.model_out(conv10_2)

    return (out_class, out_reg)</code></pre>

username: ptrblck
cooked: <p>Your model definition works and the output tensors have valid <code>grad_fn</code>s, so I’m unsure why they are <code>None</code> in your script:</p>
<pre><code class="lang-python">model = SIGGRAPHGenerator(3, 1)
x = torch.randn(1, 1, 24, 24)
out = model(x, x, x)
print(out[0].grad_fn)
&gt; &lt;ThnnConv2DBackward object at ...&gt;

print(out[1].grad_fn)
&gt; &lt;TanhBackward object at ...&gt;
</code></pre>

username: skhu17
cooked: <p>Does this mean that the loss functions does not have the attribute requires_grad=True ? And if not, how do I set these tensors to have this attribute.<br>
I saw in another post that you said loss.requries_grad =True will result in undesirable things. Thanks!</p>

username: ptrblck
cooked: <p>Could you verify, that your model outputs also have valid <code>.grad_fn</code>s?<br>
From your last post I understood that’s not the case.</p>

username: skhu17
cooked: <p>Apologies for the late reply. I followed your steps and I obtained similar results and were able to print out<br>
&lt;ThnnConv2DBackward object at …&gt;</p>

username: ptrblck
cooked: <p>Ah OK, could you post the loss function then?<br>
If the model output contains a valid <code>grad_fn</code>, while the loss doesn’t, the loss function might detach the graph.</p>

username: skhu17
cooked: <p>Hi ptrblck, I have found that when i use <code>self.loss_G = Variable(self.loss_G, requires_grad=True)</code>, the error doesn’t occur, so I assume it is due to the loss tensors, but if there are some other reason for the error please let me know! Again, thanks for all the help!</p>
<p>However, here are the loss functions if you are still interested:</p>
<pre><code class="lang-auto">self.criterionL1 = networks.HuberLoss(delta=1. / opt.ab_norm)
</code></pre>
<pre><code class="lang-auto">class HuberLoss(nn.Module):
    def __init__(self, delta=.01):
        super(HuberLoss, self).__init__()
        self.delta=delta

    def __call__(self, in0, in1):
        mask = torch.zeros_like(in0)
        mann = torch.abs(in0-in1)
        eucl = .5 * (mann**2)
        mask[...] = mann &lt; self.delta

        # loss = eucl*mask + self.delta*(mann-.5*self.delta)*(1-mask)
        loss = eucl*mask/self.delta + (mann-.5*self.delta)*(1-mask)
        return torch.sum(loss,dim=1,keepdim=True)
</code></pre>