How to change pretrained model from 3 RGB Channels to 4 Channels without causing non- leaf tensor value error?

Jason_Sheen: <p>Hello guys, I have been trying to change the pretrained PyTorch Densenet’s first conv layer from 3 channels to 4 channels while maintaining its original RGB channel’s pretrained weights. I have done the following codes, but the optimizer part throws me this error:  <code>"ValueError: can't optimize a non-leaf Tensor"</code> .</p>
<pre><code class="lang-auto"> import torchvision.models as models
    import torch.nn as nn
    backbone = models.__dict__['densenet169'](pretrained=True)
    
    
    weight1 = backbone.features.conv0.weight.data.clone()
    new_first_layer  = nn.Conv2d(4, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
    with torch.no_grad():
        new_first_layer.weight[:,:3] = weight1
    
    backbone.features.conv0 = new_first_layer
    optimizer = torch.optim.SGD(backbone.parameters(), 0.001,
                                     weight_decay=0.1)  # Changing this optimizer from SGD to ADAM
</code></pre>
<p>I have also tried to remove the argument <code>with torch.no_grad():</code> but this issue still remains:</p>
<pre><code class="lang-auto">      ValueError                                Traceback (most recent call last)
    &lt;ipython-input-343-5fc87352da04&gt; in &lt;module&gt;()
         11 backbone.features.conv0 = new_first_layer
         12 optimizer = torch.optim.SGD(res.parameters(), 0.001,
    ---&gt; 13                                  weight_decay=0.1)  # Changing this optimizer from SGD to ADAM
    
    ~/anaconda3/envs/detectron2/lib/python3.6/site-packages/torch/optim/sgd.py in __init__(self, params, lr, momentum, dampening, weight_decay, nesterov)
         66         if nesterov and (momentum &lt;= 0 or dampening != 0):
         67             raise ValueError("Nesterov momentum requires a momentum and zero dampening")
    ---&gt; 68         super(SGD, self).__init__(params, defaults)
         69 
         70     def __setstate__(self, state):
    
    ~/anaconda3/envs/detectron2/lib/python3.6/site-packages/torch/optim/optimizer.py in __init__(self, params, defaults)
         50 
         51         for param_group in param_groups:
    ---&gt; 52             self.add_param_group(param_group)
         53 
         54     def __getstate__(self):
    
    ~/anaconda3/envs/detectron2/lib/python3.6/site-packages/torch/optim/optimizer.py in add_param_group(self, param_group)
        231                                 "but one of the params is " + torch.typename(param))
        232             if not param.is_leaf:
    --&gt; 233                 raise ValueError("can't optimize a non-leaf Tensor")
        234 
        235         for name, default in self.defaults.items():
    
    ValueError: can't optimize a non-leaf Tensor
</code></pre>
<p>My PyTorch version is: 1.7.0.</p>
<p>Could you guys please help? Thanks alot!</p>
<p>Regards.</p>

Eta_C: <p>Hey, I copied your code and it ran fine.</p>

Jason_Sheen: <p>What?! Really, could you please let me know what your pytorch version is? Thanks!</p>

VirginieBfd: <p>Here is a working solution for ResNet:</p><aside class="quote quote-modified" data-post="10" data-topic="2623">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/v/858c86/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-to-modify-the-input-channels-of-a-resnet-model/2623/10">How to modify the input channels of a Resnet model</a> 
  </div>
  <blockquote>
    Here is a generic function to increase the channels to 4 or more channels. 
One key point is that the additional channel weights can be initialized with one original channel rather than being randomized. 
new_in_channels = 4
model = models.resnet18(pretrained=True)

layer = model.conv1
        
# Creating new Conv2d layer
new_layer = nn.Conv2d(in_channels=new_in_channels, 
                  out_channels=layer.out_channels, 
                  kernel_size=layer.kernel_size, 
                  stri…
  </blockquote>
</aside>


Jason_Sheen: <p>Guys, I have resolved this. Here is the code I did:</p>
<pre><code class="lang-auto">import torchvision.models as models
import torch.nn as nn
from torch.autograd import Variable
backbone = models.__dict__['densenet169'](pretrained=True)
weight1 = backbone.features.conv0.weight.clone()
new_first_layer  = nn.Conv2d(4, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False).requires_grad_()
new_first_layer.weight[:,:3,:,:].data[...] =  Variable(weight1, requires_grad=True)
backbone.features.conv0 = new_first_layer
optimizer = torch.optim.SGD(res.parameters(), 0.001,
                                 weight_decay=0.1)
</code></pre>