Autoencoder implementation with different activation function

Josiane_Rodrigues: <p>Hello community,</p>
<p>I need to implement an autoencoder network described in this article <a href="https://arxiv.org/pdf/1803.09065" rel="noopener nofollow ugc">https://arxiv.org/pdf/1803.09065</a>. The model is very simple. Contains only a hidden layer, which corresponds to a binarization layer of the input code. My whole difficulty is the activation function used in the hidden layer is non-differentiable and therefore the same weight matrix of the output layer is used to update the input layer. I would like to know how I can do an activation function  in pytorch that does not need to be derived and that uses the output layer weight matrix to update the weights of the input layer. I will be very grateful if anyone can help me.</p>
<p>The network architecture follows below.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/3/35884438f2398c261e131dfffa2212dab14e6a6b.png" alt="Captura%20de%20tela%20de%202019-05-06%2017-57-54" data-base62-sha1="7DzfQkRloT7UUJPnZMin89SLWSD" width="562" height="393"></p>

ptrblck: <aside class="quote no-group" data-username="Josiane_Rodrigues" data-post="1" data-topic="44551">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/josiane_rodrigues/48/4487_2.png" class="avatar"> Josiane_Rodrigues:</div>
<blockquote>
<p>therefore the same weight matrix of the output layer is used to update the input layer</p>
</blockquote>
</aside>
<p>Could you explain it a bit?<br>
Would you like to use the same gradients from the output layer on your input layer or copy the weights into the input layer after each update?</p>

Josiane_Rodrigues: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>,</p>
<p>I would like to copy the weights into the input layer after each update of the output layer.</p>

ptrblck: <p>Thanks for the information!<br>
In that case this code example might work.<br>
I tried to come up with a similar model architecture, so I just used three linear layers and had to transpose the weight matrix since I used a “bottleneck” layer.<br>
Also, I call <code>x.detach()</code> after the first linear layer to fake your behavior of the gradient loss in the input layer:</p>
<pre><code class="lang-python">class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.fc1 = nn.Linear(10, 5)
        self.fc2 = nn.Linear(5, 5)
        self.fc3 = nn.Linear(5, 10)
        
    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = x.detach()  # Fake your use case so that fc1 cannot be updated
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

data = torch.randn(10, 10)
target = torch.randn(10, 10)
model = MyModel()
criterion = nn.MSELoss()
optimizer = optim.SGD(model.parameters(), lr=1e-3)

for epoch in range(10):
    optimizer.zero_grad()
    output = model(data)
    loss = criterion(output, target)
    loss.backward()
    # print(model.fc1.weight.grad) # Will be None
    optimizer.step()
    print('Epoch {}, loss {}'.format(epoch, loss.item()))
    
    # Copy weights from output layer to input layer
    with torch.no_grad():
        model.fc1.weight.copy_(model.fc3.weight.t())
</code></pre>
<p>Let me know, if this helps or if I misunderstood your use case.</p>

Josiane_Rodrigues: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>,</p>
<p>Thank you. I’ll test and say if it worked <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Josiane_Rodrigues: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>,</p>
<p>It worked! Thank you! I have another question =]. I need to use a heaviside (step) function from the input to the hidden layer, instead of the relu function applied here (x = F.relu (self.fc1 (x))), so that the values of this layer are binary [0,1]. I’ve seen that it has the heaviside function in numpy, but it’s conflicting with the pytorch because of the type. Any idea?</p>

ptrblck: <p>What kind of type issue do you see?</p>

Josiane_Rodrigues: <p>I would have to do something like this:</p>
<blockquote>
<p>def forward(self, x):<br>
x = np.heaviside(self.fc1(x),1)<br>
x = x.detach()<br>
x = F.tanh(self.fc2(x))<br>
x = self.fc3(x)<br>
return x</p>
</blockquote>
<p>This error appears:</p>
<p>RuntimeError: Can’t call numpy() on Variable that requires grad. Use var.detach().numpy() instead.</p>

ptrblck: <p>You could probably indeed detach the output of <code>fc1</code>, since you won’t calculate gradients for it.</p>
<pre><code class="lang-python">x = np.heaviside(self.fc1(x).detach().numpy(), 0)
x = torch.from_numpy(x)
</code></pre>
<p>Alternatively, you could also implement <code>np.heaviside</code> in PyTorch.</p>

Josiane_Rodrigues: <aside class="quote no-group" data-username="ptrblck" data-post="9" data-topic="44551">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>x = np.heaviside(self.fc1(x).detach().numpy(), 0) x = torch.from_numpy(x)</p>
</blockquote>
</aside>
<p>This worked \o/. Thank you <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

Josiane_Rodrigues: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>,</p>
<p>I would like to take one last doubt if possible.<br>
This model uses a loss function that is the sum of two terms that follow.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/8/849a9333d980a0ac71f6eaa3bb0bead12838911a.png" alt="Captura%20de%20tela%20de%202019-05-07%2021-24-48" data-base62-sha1="iV4cuGoyUH3VVR30HMgbNFTkHcm" width="322" height="88"><br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/b/b2b694c9efe3dbd9f19e056d8a6cb01637f6c131.png" alt="Captura%20de%20tela%20de%202019-05-07%2021-24-58" data-base62-sha1="puYaqrHPK3XKNs2qqBzVXqTxHpf" width="263" height="62"></p>
<p>In the second term, W is the weight matrix and I is an identity matrix.<br>
For both functions I have used MSELoss to implement. I tried the following.</p>
<blockquote>
<p>loss = criterion_1(output, target) + criterion_2(torch.mm(model.fc3.weight,model.fc3.weight.t()), Variable((torch.eye(10))))</p>
</blockquote>
<p>It did not make any error, but I was in doubt if the code is actually doing what is described in the two functions of the model.</p>

ptrblck: <p>I’m not sure how <code>W</code> and the matrix norm are defined in the paper.<br>
If they are using the Frobenius norm, you cannot use <code>nn.MSELoss</code>:</p>
<pre><code class="lang-python">w = torch.matmul(model.fc3.weight.t(),model.fc3.weight)
i = torch.eye(w.size(0))

# Frobenius norm
0.5 * (w - i).norm(p=2)
((w - i)**2).sum()**0.5 * 0.5

# MSE
criterion = nn.MSELoss()
criterion(w, i)
((w - i)**2).mean()
</code></pre>
<p>But as I said, I’m not familiar with the paper so I might misunderstand the notation.</p>
<p>The first loss looks alright.</p>

Josiane_Rodrigues: <p>Thanks for the explanation.<br>
In the papers this notation is usually a norm L2 squared. In that case, I cannot use MSELoss, right?<br>
Are there any other pytorch functions that I could use in this case?</p>

ptrblck: <p>Would my first approach work? (<code>.norm(p=2)</code>)</p>

Josiane_Rodrigues: <p>Yes! In the paper, the authors did not specify the norm, but the L2 norm is usually used.</p>

Josiane_Rodrigues: <p>Sorry, I feel a little confused now. How would the norm be applied to the activation function?</p>

ptrblck: <p>I’m a bit confused, too.<br>
Which activation function do you mean and what would you like to do with this function?</p>
<p>Based on the formula and code you’ve posted, it seems that <code>l_reg</code> is calculated based on the weight matrix of the last linear layer.</p>

Josiane_Rodrigues: <p>Sorry. I wrote wrong, I meant loss function and no activation function. The loss function I want to implement is made up of the two functions mentioned above. Would it be this?</p>
<blockquote>
<p>loss = criterion(output, target) + 0.5*(w - i).norm(p=2)</p>
</blockquote>

ptrblck: <p>I think this might work, but you should really check it with someone having more insight into the paper (or the general approach). <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

Josiane_Rodrigues: <p>Really thank you for help!!! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>