Binary weight backward function

Mate_Nagy: <p>I want to make an Autoencoder with binary weights at the first layer. You can see attached the network layout:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/5/45fcd84812dfe80713358e505eb4d7a624f22bc3.png" data-download-href="https://discuss.pytorch.org/uploads/default/45fcd84812dfe80713358e505eb4d7a624f22bc3" title="dcan"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/5/45fcd84812dfe80713358e505eb4d7a624f22bc3_2_690x278.png" alt="dcan" data-base62-sha1="9Z8F6M96BzGD5cFP2w1z6ptyvDR" width="690" height="278" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/5/45fcd84812dfe80713358e505eb4d7a624f22bc3_2_690x278.png, https://discuss.pytorch.org/uploads/default/original/3X/4/5/45fcd84812dfe80713358e505eb4d7a624f22bc3.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/4/5/45fcd84812dfe80713358e505eb4d7a624f22bc3.png 2x" data-dominant-color="DDDDDD"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">dcan</span><span class="informations">850×343 63.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
(<a href="https://www.researchgate.net/figure/The-deep-convolutional-auto-encoder-network-DCAN-architecture-The-DCAN-is-a_fig1_322937707" rel="noopener nofollow ugc">Source</a>)<br>
I think I have succesfully implemented almost everything except the backward for binary layer. I am still a beginner in NN’s so I really need help to make this network better.<br>
Here are my results right now:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/c/0ce553c310a804dd1b3da79b0e55bfdcc61f496b.png" data-download-href="https://discuss.pytorch.org/uploads/default/0ce553c310a804dd1b3da79b0e55bfdcc61f496b" title="dcan_own_result"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/c/0ce553c310a804dd1b3da79b0e55bfdcc61f496b_2_690x143.png" alt="dcan_own_result" data-base62-sha1="1Q53M50NWLDoUuFkNTPS5gSZv99" width="690" height="143" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/c/0ce553c310a804dd1b3da79b0e55bfdcc61f496b_2_690x143.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/c/0ce553c310a804dd1b3da79b0e55bfdcc61f496b_2_1035x214.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/0/c/0ce553c310a804dd1b3da79b0e55bfdcc61f496b_2_1380x286.png 2x" data-dominant-color="B5B5B5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">dcan_own_result</span><span class="informations">1923×400 153 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
But as you can see this is far from perfect. The problem is that I should also optimize the binary filters to get the best result. Here is a part of my code:</p>
<pre><code class="lang-auto">class BinaryFilterLayer(nn.Module):
  def __init__(self,number_of_features):
    super(BinaryFilterLayer, self).__init__()
    self.number_of_features = number_of_features
    # Generating the initial binary filters (only 0 and 1)
    self.weight = nn.Parameter(torch.randint(high = 2,size=(1,self.number_of_features,128,128)).float())

  def forward(self, input):
      output = input * self.weight
      return output
  # here i need the backward but i dont really know how


class Model(nn.Module):
  def __init__(self,encoding_dim):
    super(Model, self).__init__()
    self.encoding_dim = encoding_dim
    # Encode
    self.binary_filters = BinaryFilterLayer(self.encoding_dim)

    # Decode
    self.decoder = nn.Sequential(
                    nn.LayerNorm(self.encoding_dim),
                    nn.Linear(self.encoding_dim,128*128),
                    nn.LayerNorm(128*128),
                    nn.ReLU(),
                    Reshape(128,128),
                    nn.Conv2d(1,64,kernel_size= 9,padding=2),
                    nn.BatchNorm2d(64),
                    nn.ReLU(),
                    nn.Conv2d(64,32,kernel_size= 1,padding=2),
                    nn.BatchNorm2d(32),
                    nn.ReLU(),
                    nn.Conv2d(32,1,kernel_size= 5,padding=2),
                    nn.BatchNorm2d(1),
                    nn.Sigmoid()
                    )

  def sumfunc(self,x):
    x = torch.sum(x,dim=3)     # maybe there is an efficient way but this should work well also i guess
    x = torch.sum(x,dim=2)
    return x/(128*128)            # not sure if i need to normalize here but probably not
  
  def forward(self, x):
    x = self.binary_filters(x)  # multiply with binary layers
    x = self.sumfunc(x)        # get the sum of each layer
    x = self.decoder(x)        # decode
    return x 
  
</code></pre>
<p>Can somebody help me to optimize these binary filters? Or is there a better way to make this network working better?</p>

anantguptadbl: <p>Here is a slight modification of the code</p>
<pre><code class="lang-auto">import torch
from torch import nn
from torch.autograd import Variable
from torch.utils.data import DataLoader
from torchvision import transforms
import torchvision.models as models
import torch.nn.functional as F

class BinaryFilterLayer(nn.Module):
  def __init__(self,number_of_features):
    super(BinaryFilterLayer, self).__init__()
    self.number_of_features = number_of_features
    # Generating the initial binary filters (only 0 and 1)
    self.weight = nn.Parameter(torch.randint(high = 2,size=(128,128)).float())
    self.sigm=nn.Sigmoid()

  def forward(self, input):
      #print("size of input is {0} and size of weight is {1}".format(input.size(),self.weight.size()))
      self.weight
      output = input * torch.round(self.sigm(self.weight))
      return output

class Flatten(nn.Module):
    def forward(self, input):
        return input.view(128,128)


class Model(nn.Module):
  def __init__(self,encoding_dim):
    super(Model, self).__init__()
    self.encoding_dim = encoding_dim
    # Encode
    self.binary_filters = BinaryFilterLayer(self.encoding_dim)
    # Decode
    self.decoder = nn.Sequential(
                    #nn.LayerNorm(self.encoding_dim),
                    #nn.Linear(self.encoding_dim,128*128),
                    #nn.LayerNorm(128*128),
                    #nn.ReLU(),
                    #Flatten(),
                    nn.Conv2d(1,64,kernel_size= 9,padding=2),
                    nn.BatchNorm2d(64),
                    nn.ReLU(),
                    nn.Conv2d(64,32,kernel_size= 1,padding=2),
                    nn.BatchNorm2d(32),
                    nn.ReLU(),
                    nn.Conv2d(32,1,kernel_size= 5,padding=2),
                    nn.BatchNorm2d(1),
                    nn.Sigmoid()
                    )

  def sumfunc(self,x):
    x = torch.sum(x,dim=3)     # maybe there is an efficient way but this should work well also i guess
    x = torch.sum(x,dim=2)
    return x/(128*128)            # not sure if i need to normalize here but probably not
  
  def forward(self, x):
    x = self.binary_filters(x)  # multiply with binary layers
    #print("Post binary filters, the size of x is {0}".format(x.size()))
    #x = self.sumfunc(x)        # get the sum of each layer
    x = self.decoder(x.view(1,1,128,128))        # decode
    #print("After decoder the size of x is {0}".format(x.size()))
    return x 

model=Model(125)
criterion=torch.nn.MSELoss()
optimizer=torch.optim.Adam(model.parameters(),lr=0.001)
epochs=1000
from skimage import io
import cv2
labrador = io.imread("gdrive/My Drive/Colab Notebooks/YellowLabradorLooking_new.jpg", as_gray=True)
labrador=cv2.resize(labrador,(128,128))
labrador=Variable(torch.from_numpy(labrador.astype(np.float32)))
print("Input size is {0}".format(labrador.size()))

for curEpoch in range(epochs):
  model.zero_grad()
  output=model(labrador)
  loss=criterion(labrador,output)
  loss.backward()
  optimizer.step()
  if(curEpoch %100==0):
    print("Loss is {0}".format(loss.item()))
</code></pre>
<pre><code class="lang-auto">import matplotlib.pyplot as plt
%matplotlib inline
plt.imshow(labrador.detach().numpy().reshape(128,128))
plt.show()
plt.imshow(model.binary_filters.weight.detach().numpy() * labrador.detach().numpy().reshape(128,128))
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/9/a9b0f1c628cca3d1985588eab10b983201e97cb7.png" alt="image" data-base62-sha1="od9NeTLmkrNyh95cQBdS6ggUdIr" width="257" height="251"><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/6/5/6548e5a1a2f1585c9b5a5fe638042b505ea4c877.png" alt="image" data-base62-sha1="es0p9aO9PLTvVS1KydhKrM35zJZ" width="257" height="251"></p>
<p>Hope this helps</p>

Mate_Nagy: <p>Thank you for your answer but maybe I was not clear with the question. You dont use the number_of_features variable which corresponds to the number of binary masks in the first layer. In the network, we want to generate M pieces of binary masks ((1,M,128,128)) and then the next layer is the sum of each images thus we will get an (1,M) size tensor. And what really confuses me is how to change the binary masks during the training because if they are only 0 and 1 than how can we define the derivative of these masks? At least for my code they wont change but this is the main point of this task to optimize the masks.<br>
I hope now my question is more clear <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

anantguptadbl: <aside class="quote no-group" data-username="anantguptadbl" data-post="2" data-topic="70847">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/anantguptadbl/48/17784_2.png" class="avatar"> anantguptadbl:</div>
<blockquote>
<p>self.weight = nn.Parameter(torch.randint(high = 2,size=(128,128)).float())</p>
</blockquote>
</aside>
<p>In that case you can replace</p>
<p><code>self.weight = nn.Parameter(torch.randint(high = 2,size=(128,128)).float())</code></p>
<p>with</p>
<p><code>self.weight = nn.Parameter(torch.randint(high = 2,size=(M,128,128)).float())</code></p>

Mate_Nagy: <p>This one is also not the right answer for my question. Right know I am trying to make a custom autograd function to optimize the binary masks but I have no idea how to calculate the forward and backward function in this case.</p>