Transfer Learning: How to modify the first conv2d layer of Alexnet to accomodate for 9 channel input?

jupiter: <p>I am using Alexnet as a feature extractor on a 9 channel image (3 images concatenated). So I need to modify the first conv2d layer to receive 9 channel inputs. What I tried:</p>
<p><code>model_conv = torchvision.models.alexnet(pretrained=True) new_features = nn.Sequential(*list(model_conv.features.children())) model_conv.features = new_features new_9_tensor = torch.cat((new_features.state_dict().get('0.weight'), new_features.state_dict().get('0.weight'), new_features.state_dict().get('0.weight')), 1) new_features.state_dict().__setitem__('0.weight', new_9_tensor) print("named modules", list(new_features.named_modules()))</code></p>
<p>Although the <code>state_dict</code> is modified to show the required 64x9x11x11 tensor, the output of last line (list of named modules) shows no change in conv2d layer. Can someone please help me on what changes do I need to make to make Alexnet work for a 9 channel input? Is there a easier way to do this without accessing private attributes?</p>
<p>Is there a way to remove first layer and initialize it with <code>Conv2d(9, 64, kernel_size=(11, 11), stride=(4, 4), padding=(2, 2))</code></p>
<p>Any sort of help is really appreciated. Thanks!</p>

dgriff: <p>I believe you want Conv2d(9, 64, kernel_size=11, stride=4, padding=2)</p>

jupiter: <p>My bad! typo in the post but not in the original code! Reinitialization of conv2d will randomly initialize the weights. Besides, the <code>self.features = nn.Sequential( ...</code> container doesn’t allow to change only the first conv2d layer (please correct me if I’m wrong) while keeping all the weights as they are that it received after the first statement above. Still looking for a clue on how to modify the first layer <img src="//discuss.pytorch.org/images/emoji/apple/neutral_face.png?v=5" title=":neutral_face:" class="emoji" alt=":neutral_face:"></p>

aitutakiv: <p>I think the easiest way to do this is to create your model by modifying the Alexnet model from Vision so that the first conv layer has 9 channels<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/vision/blob/master/torchvision/models/alexnet.py#L18" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/vision/blob/master/torchvision/models/alexnet.py#L18" target="_blank" rel="nofollow noopener">pytorch/vision/blob/master/torchvision/models/alexnet.py#L18</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="8" style="counter-reset: li-counter 7 ;">
<li>model_urls = {</li>
<li>'alexnet': 'https://download.pytorch.org/models/alexnet-owt-4df8aa71.pth',</li>
<li>}</li>
<li>
</li>
<li>
</li>
<li>class AlexNet(nn.Module):</li>
<li>
</li>
<li>def __init__(self, num_classes=1000):</li>
<li>    super(AlexNet, self).__init__()</li>
<li>    self.features = nn.Sequential(</li>
<li class="selected">        nn.Conv2d(3, 64, kernel_size=11, stride=4, padding=2),</li>
<li>        nn.ReLU(inplace=True),</li>
<li>        nn.MaxPool2d(kernel_size=3, stride=2),</li>
<li>        nn.Conv2d(64, 192, kernel_size=5, padding=2),</li>
<li>        nn.ReLU(inplace=True),</li>
<li>        nn.MaxPool2d(kernel_size=3, stride=2),</li>
<li>        nn.Conv2d(192, 384, kernel_size=3, padding=1),</li>
<li>        nn.ReLU(inplace=True),</li>
<li>        nn.Conv2d(384, 256, kernel_size=3, padding=1),</li>
<li>        nn.ReLU(inplace=True),</li>
<li>        nn.Conv2d(256, 256, kernel_size=3, padding=1),</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>Then to copy the weights over you can copy them from the original Alexnet <code>state_dict</code> to the <code>state_dict</code> of your modified model. It should be direct assignment except for the first conv layer.</p>

Yongkai_Liu: <p>Does this work very well? Is there any way better?</p>