How to implement many filters/weights in single Conv layer?

Fahd_Jerbi: <p>Hi, I’m trying to make a CNN model that use custom filters/weights. I started by using a pretrained model and changed it according to my need (figure below to better explanation of the idea). The goal is to have a 3 channels image then filter the input with all filters in each layer. I want to ask if the weights implementation is done right. Here is the code I have done, I know it might not be clean sorry for that.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/b/8b75fc9b103b1f3bad0bcb2248501953f3094226.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/8b75fc9b103b1f3bad0bcb2248501953f3094226" title="my_model"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/b/8b75fc9b103b1f3bad0bcb2248501953f3094226_2_517x291.jpeg" alt="my_model" data-base62-sha1="jTJ9xhsmJGS2MbPFzGSBKd13N0a" width="517" height="291" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/b/8b75fc9b103b1f3bad0bcb2248501953f3094226_2_517x291.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/8/b/8b75fc9b103b1f3bad0bcb2248501953f3094226_2_775x436.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/b/8b75fc9b103b1f3bad0bcb2248501953f3094226_2_1034x582.jpeg 2x" data-dominant-color="FBFBFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">my_model</span><span class="informations">1280×720 51.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<pre><code class="lang-auto">model = models.alexnet(pretrained= True)
# replace avgpool: 
 class Identity(nn.Module): 
   def __init__(self): 
     super(Identity, self).__init__() 

   def forward(self, x): 
     return x

model.avgpool= Identity()

# remove some layer:
model.features[8] = nn.MaxPool2d(3, stride=2, padding=0, dilation=1, ceil_mode=False)
model.features = nn.Sequential(*[model.features[i] for i in range(9)])
model.classifier = nn.Sequential(nn.Dropout(p=0.5),
                                nn.Linear(3,3),
                                nn.ReLU(inplace=True),
                                nn.Linear(3,3))
############ 1st hidden layer:
#replace Conv2d:
model.features[0]= nn.Conv2d(3, 1, 3, stride=1, groups=1)
#change weights:
model.features[0].weight = nn.Parameter(data= torch.tensor([
                       [[1, 2, 1],
                        [0, 0, 0],
                        [-1, -2, -1]],
                        
                       [[-1, 0, 1],
                        [-2, 0, 2],
                        [-1, 0, 1]],

                       [[2, 1, 0],
                        [1, 0, -1],
                        [0, -1, -2]],
                          
                       [[0, -1, -2],
                        [1, 0, -1],
                        [2, 1, 0]]], dtype= torch.float32), requires_grad= True)
###########2nd hidden layer:
#replace 2nd Conv2d:
model.features[3]= nn.Conv2d(3, 1, 3, stride=1, groups=1)
#add weights: 
model.features[3].weight = nn.Parameter(data= torch.tensor([
                     [[0.081, 0.17789, 0.081],
                      [0, 0, 0],
                      [-0.081, -0.17789, -0.081]],
                                         
                     [[0.081, 0, -0.081],
                      [0.17789, 0, -0.17789],
                      [0.081, 0, -0.081]],

                     [[0.17789, 0.081 , 0],
                      [0.081, 0, -0.081],
                      [0, -0.081, -0.17789]],

                      [[0, -0.081, -0.17789],
                       [0.081, 0, -0.081],
                       [0.17789, 0.081, 0]]], dtype= torch.float32), requires_grad= False)
#######3rd Hidden layer: 
#replace 3rd Conv2d:
model.features[6]= nn.Conv2d(3, 1, 3, stride=1, groups=1)
#add weights:
model.features[6].weight= nn.Parameter(data= torch.tensor([
          [[0.0455, -0.1789, 0.0455],
           [0.1, -0.388, 0.1],
           [0.0455, -0.1789, 0.0455]],

          [[0.0455, 0.1, 0.0455],
           [-0.1789, -0.388, -0.1789],
           [0.0455, 0.1, 0.0455]],

          [[0.1272, 0, -0.1272],
           [0, 0, 0],
           [-0.1272, 0, 0.1272]],
           
          [[-0.1272, 0, 0.1272],
           [0,0,0],
           [0.1272, 0, -0.1272]]], dtype=torch.float32), requires_grad= False)
</code></pre>

ptrblck: <p>The <code>model.features[0].weight</code> parameter has a shape of <code>[4, 3, 3]</code> so one dimension is missing.<br>
<code>nn.Conv2d</code> defines the <code>weight</code> parameter as <code>[out_channels, in_channels, height, width]</code>.<br>
Based on the current code, I guess that the number of <code>in_channels</code> is missing?<br>
If that’s the case, you could <code>unsqueeze</code> the tensor in <code>dim1</code> and <code>repeat it </code>3` times.</p>

Fahd_Jerbi: <p>Yes Mr. <a class="mention" href="/u/ptrblck">@ptrblck</a>, <code>in_channels</code> was missing and the weights implementation was wrong but now it worked as this:</p>
<pre><code class="lang-auto">model.features[0].weight = nn.Parameter(data= torch.tensor([
                        [[[1, 2, 1], [0, 0, 0], [-1, -2, -1]],
                         [[1, 2, 1], [0, 0, 0], [-1, -2, -1]],   
                         [[1, 2, 1], [0, 0, 0], [-1, -2, -1]]], 
                                          ###############  end of filter n°1
                        [[[-1, 0, 1], [-2, 0, 2], [-1, 0, 1]],
                         [[-1, 0, 1], [-2, 0, 2], [-1, 0, 1]],
                         [[-1, 0, 1], [-2, 0, 2], [-1, 0, 1]]],
                                            ############## end of filter n°2
                         [[[2, 1, 0], [1, 0, -1], [0, -1, -2]],
                          [[2, 1, 0], [1, 0, -1], [0, -1, -2]],
                          [[2, 1, 0], [1, 0, -1], [0, -1, -2]]],
                                            ############## end of filter n°3
                         [[[0, -1, -2], [1, 0, -1], [2, 1, 0]],
                          [[0, -1, -2], [1, 0, -1], [2, 1, 0]],
                          [[0, -1, -2], [1, 0, -1], [2, 1, 0]]]],  dtype= torch.float32), requires_grad= True)
</code></pre>
<p>I just encountered a new error since I changed Linear layer :</p>
<blockquote>
<p>mat1 dim 1 must match mat2 dim 0</p>
</blockquote>
<p>If you would please to suggest any solution that may work it would be appreciated</p>

ptrblck: <p>This error might be raised, if the number of input features of an activation doesn’t match the expected <code>in_features</code> of a linear layer.<br>
Try to print the shape of the flattened activation before feeding it into a linear layer and make sure the features match.</p>