Learning a binary matrix

learner47: <p>Hi,</p>
<p>I am currently working on a code in which I need a matrix (I am currently using <code>nn.Parameter</code>) to be trainable. However I need the said matrix to have entries as either <code>0</code> or <code>1</code>. How can I impose a constraint for the weight updates to happen in such a way that <code>0</code> is flipped to <code>1</code> or the other way around, based on if the back propagation requires the weight to be increased or decreased? It is alright if the solution uses something other than <code>nn.Parameter</code>.</p>
<p>PS: I have tried <code>nn.Linear</code> layers, <code>nn.Conv1d</code> layers as well. I can’t get the matrix/kernel to only have zeros and ones.</p>
<p>Any help is appreciated.</p>

CedricLy: <p>Well on top of my head I guess the closest thing would be to use an activation function like sigmoid. Maybe one which is not as soft as sigmoid.<br>
If you want to have 0 and 1 as acceptable values and the parameter shall remain trrainable, than this question is very tricky.<br>
The backpropagation only works for float variables. If all values are 0 and 1 than you cannot calculate a definite derivative.</p>
<p>I hope I am wrong and someone give a solution here, because this could solve my reverse embedding layer problem.</p>

learner47: <p>Hi <a class="mention" href="/u/cedricly">@CedricLy</a>. I have tried using <code>Sigmoid</code>. I have gone, one step further, to make it <code>Sigmoid(50*input)</code> to have the thresholding done. But this messes up my training. The weight updates do not seem to happen and my network is stuck in the state it is in.</p>
<p>The reason I need the matrix to strictly have <code>0</code>s and <code>1</code>s is because I am working in a binary field (where the available digits are 0 and 1 only).</p>
<p>Thank you for your time!</p>

CedricLy: <p>Your treshold Idea is good, but if you change the input this won’t change much, because the weights of the layer will adapt.</p>

caonv: <p>I don’t know why the matrix values should be binary. You can apply a sequence of sigmoid -&gt; minus 0.9 -&gt; relu to obtain hard-binarized values in a learnable way, but the problem is that your model may not converge.</p>
<p>Any way, to follow the above process, I recommend you to register a tensor as parameters as follows:</p>
<pre><code class="lang-auto">class BinaryLayer(nn.Module):
    def __init__(self, in_channels, out_channels): 
        super(BinaryLayer, self).__init__() 
        self.weight = nn.Parameter(torch.randn(in_channels, out_channels), requires_grad=True)  #
    def forward(self, x): 
        bin_weight = torch.nn.functional.relu(torch.sigmoi(self.weight) - 0.9)
        y = torch.mm(x, bin_weight)
        return y
</code></pre>
<p>The maybe some error in the code snip, but  the flow is clear. After the training is complete, you can get a binary matrix in the same way the <code>bin_weight</code> is computed above.</p>

learner47: <p>Hi <a class="mention" href="/u/caonv">@caonv</a>, thank you for your reply.</p>
<p>I have a doubt. <code>torch.sigmoid</code> ranges between <code>(0, 1)</code> and the operation <code>torch.sigmoid(self.weight) - 0.9</code> brings the range to <code>(-0.9, 0.1)</code>. When <code>relu</code> is introduced, the range is now <code>(0, 0.1)</code>. The threshold on <code>0</code> side is perfectly fine, but what about the threshold on <code>1</code>? Am I missing something?</p>

caonv: <p>Actually that is a kind of thresholding with a threshold of 0.9, and 0.9 is very close to 1. I have just found <code>torch.nn.Hardsigmoid</code> which can give you 0-1 in some case but not all the values.</p>