Idiomatic way to selectively zero gradients based on input

username: Filipe_Silva
cooked: <p>I am building a recommender system based on an auto-encoder network, therefore, in order to train the network I have to only calculate the gradients for the weights connected to positive ratings - an input is a collection of ratings and a negative rating means that no rating has been provided.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/a/ac47c2d4fdce88aebe0410e75e6c153df4eb7b51.png" alt="09" data-base62-sha1="oA3S53PAGAsWHneFbtCCqcmwE1j" width="332" height="182"><br>
Assuming the input to this networks was [3,-1,2,-1,5], we would get the gradients for those weights removed from the picture above set to zero, so that, they end up being “discarded” when updating the weights, otherwise, the network would learn also the negative ratings and we want to avoid that at any cost because negative ratings are just a placeholder for no rating at all.</p>
<p>I have come up with this model:</p>
<pre><code class="lang-auto"># -----------------------------------------------------------------------------
# Model definition for AutoRec
# -----------------------------------------------------------------------------

class AutoRecModel(nn.Module):

    def __init__(self, D_in, H):
        super(AutoRecModel, self).__init__()
        self.encoder = nn.Linear(D_in, H)
        self.decoder = nn.Linear(H, D_in)
        # tie weights
        self.decoder.weight.data = self.encoder.weight.data.transpose(0,1)
        self.register_buffer('input', torch.zeros(D_in))

    def forward(self, x):
       self.input = x.clone()
       hidden_output = self.encoder(x)
       ratings_pred = self.decoder(hidden_output)
       return ratings_pred
</code></pre>
<p>Then, I coded this training function where some parts were removed for the sake of simplicity:</p>
<pre><code class="lang-auto"># -----------------------------------------------------------------------------
# Training function for AutoRec
# -----------------------------------------------------------------------------

autorec = arm.AutoRecModel(5, 3)
input = torch.tensor([3,-1,2,-1,5], requires_grad=False)
loss_function = nn.MSELoss()
optimizer = optim.SGD(autorec.parameters(), lr=1e-1)

for i in range(1):

    autorec.zero_grad()
    pred_y = autorec(input)
    loss = loss_function(pred_y, input)
    encoder_handle, decoder_handle = add_grad_hooks(zero_encoder_weights, zero_decoder_weights)

    loss.backward()
    optimizer.step()
    encoder_handle.remove()
    decoder_handle.remove()
</code></pre>
<p>Finally, these are the auxiliary functions:</p>
<pre><code class="lang-auto"># -----------------------------------------------------------------------------
# Auxiliary functions for AutoRec
# -----------------------------------------------------------------------------

def add_grad_hooks(encoder_hook, decoder_hook):
    for name, param in autorec.named_parameters():
       if name == "encoder.weight":
           encoder_handle = param.register_hook(encoder_hook)
       if name == "decoder.weight":
           decoder_handle = param.register_hook(decoder_hook)
    return encoder_handle, decoder_handle


def zero_weights(grad, network='ENCODER'):
    column_index = 0
    grad_clone = grad.clone()
    for rating in autorec.state_dict().get("input"):
        if rating &lt; 0:
            if network == 'ENCODER':
                grad_clone[:,column_index] = 0
            else:
                grad_clone[column_index, :] = 0
        column_index += 1
    print(grad_clone)
    return grad_clone

def zero_encoder_weights(grad):
    return zero_weights(grad)

def zero_decoder_weights(grad):
    return zero_weights(grad, 'DECODER')
</code></pre>
<p>Despite the said set up working as expected I would like to know if there is any other way to achieve the same result, something cleaner/more idiomatic for zeroing the grads of some weights based on the input?</p>

username: RahimEntezari
cooked: <p>Hi<br>
I was using sth similar to your code, my problem is that this makes the training very very slow. Have you experienced the same problem?</p>