Custom connections in neural network layers

theQmech: <p>At the moment, I’m experimenting with defining custom sparse connections between two fully connected layers of a neural network.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/1X/7e3dfc25dd2eda83d45adcd3d3d6d10f6c5636c3.png" width="386" height="201"></p>
<p>To accomplish this, right now I’m modifying  <code>nn.Linear(in_features, out_features)</code> to <code>nn.MaskedLinear(in_features, out_features, mask)</code>, where mask is the adjacency matrix of the graph containing the two layers. The module <code>nn.Linear</code> uses a method invoked as <code>self._backend.Linear()</code> defined in <code>nn._functions.linear.py</code>, which I clearly will have to modify as well.</p>
<p>I’m not sure if this is the right way to do it. I have a constant feeling that there should be a better way to do this. Any help/comments on this are are much appreciated,</p>

Kaixhin: <p>If you’re happy using just normal <code>nn.Linear</code>, then you can always zero out the weight matrix for the connections you want to remove. If they are zero then no information can go forwards or backwards through it (and hence the weights won’t change from zero), achieving the sparse connectivity that you want.</p>
<p><strong>EDIT:</strong> This is wrong. See my last comment for more details.</p>

theQmech: <p>Sorry but I don’t get your point. Do you mean to say that I should zero out the required weights  before each forward phase?</p>
<p>That would definitely work for now but I was thinking if there was a more efficient solution.</p>
<p>EDIT: You’re right that no information will flow through those weights, but the weights themselves could change due to gradient descent right?</p>

Kaixhin: <p>Just once at the beginning is enough. No gradients will flow through either, hence the weights shouldn’t change (unless you’re manually adjusting the weights in some other way).</p>

theQmech: <p>I think you’ve got it wrong over here. Weights being zero at the beginning doesn’t guarantee weights being zero throughout.</p>

bzcheeseman: <p>If you have a mask already then you could do an element-wise multiply between the mask and the weights for every forward I would think? I don’t imagine an element-wise multiply would be too slow to do every forward. Alternatively, you could do something like this:</p>
<pre><code class="lang-auto">def zero_grad(self, grad_input, grad_output):
    return grad_input * self.mask

class MaskedLinear(nn.Module):
    def __init__(self, in_features, out_features, mask):
        super(MaskedLinear, self).__init__()
        self.linear = nn.Linear(in_features, out_features)
        self.linear.weights *= mask  # to zero it out first
        self.mask = mask
        self.handle = self.register_backward_hook(zero_grad)  # to make sure gradients won't propagate
</code></pre>
<p>Then that way you can still have a fast forward with minimal overhead on the backward?</p>

Kaixhin: <p><a href="https://www.quora.com/If-one-initializes-a-set-of-weights-in-a-Neural-Network-to-zero-is-it-true-that-in-future-iterations-they-will-not-be-updated-by-gradient-descent-and-backpropagation" class="onebox" target="_blank" rel="nofollow noopener">https://www.quora.com/If-one-initializes-a-set-of-weights-in-a-Neural-Network-to-zero-is-it-true-that-in-future-iterations-they-will-not-be-updated-by-gradient-descent-and-backpropagation</a></p>

theQmech: <p><a class="mention" href="/u/kaixhin">@Kaixhin</a> I don’t understand the notation in the answer. I’m certain that the author either misinterpreted the question or got it wrong. If the derivative of a loss wrt a weight at zero is zero, then it would imply that assigning all weights to zero would lead to a stationary point, which is absurd.</p>
<p>Please note that I’m only assigning a subset of weights to zero and not all of them. I’m certain that what you claimed isn’t true and one needs to zero out the required weights for each forward pass.</p>

Kaixhin: <p>Apologies, I was wrong. For a simple example with scalars, <code>y = w * x</code>,<br>
even if <code>w = 0</code>, <code>dy/dw = x</code>, so the weights can indeed change from 0. I<br>
agree with <a class="mention" href="/u/bzcheeseman">@bzcheeseman</a>, masking in the forward pass seems like a<br>
reasonable way to accomplish what you want.</p>

shirindora: <p><a class="mention" href="/u/theqmech">@theQmech</a> Can you comment if you were able to implement the mask based approach? And can you also say something about the speed-loss, if there was any?</p>
<p>Thank you.</p>

theQmech: <p>As a dirty fix, I made a sub-class of nn.Linear. Initialize a 0-1 mask matrix, and multiply(elementwise) with weights with this mask during each backward pass. This implementation is no faster than a naive Linear transform. Note that this was in an older version of pytorch and I do not know if one can still achieve this.</p>
<p>I believe that pytorch0.2 has support for sparse matrix multiplication, which will make the computations much more efficient. Ideally, this is how it should be done.</p>

mariob6: <p><a class="mention" href="/u/theqmech">@theQmech</a>, I’m trying to implement the same exact thing but using sparse tensors for the weights as the full matrix won’t fit in memory. Do you have any hint on how to use sparse tensors and if cuSPARSE is supported? I made something with Theano but it doesn’t support multiple-CPU or GPU on sparse nets, so I’m looking to migrate to pytorch</p>

tengerye: <p><a class="mention" href="/u/theqmech">@theQmech</a> I am trying the <a href="https://discuss.pytorch.org/t/weights-disconnection-implementation/19314/11">same thing</a>, do you have any progress?</p>

uchida-takumi: <p>I have tried to implement MaskedLinear.<br>
↓↓↓<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://github.com/uchida-takumi/MaskedLinear" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars2.githubusercontent.com/u/20367148?s=400&amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/uchida-takumi/MaskedLinear" target="_blank" rel="nofollow noopener">uchida-takumi/MaskedLinear</a></h3>

<p>This is an extended torch.nn.Linear module which mask connection. - uchida-takumi/MaskedLinear</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>I am glad, if it help someone.</p>

jamesproud: <p>Here, also, is my implementation:</p>
<aside class="quote quote-modified" data-post="5" data-topic="12072">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jamesproud/48/8431_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/gradient-masking-in-register-backward-hook-for-custom-connectivity-efficient-implementation/12072/5">Gradient masking in register_backward_hook for custom connectivity - efficient implementation</a> 
  </div>
  <blockquote>
    <a class="mention" href="/u/jonathanephrath">@jonathanEphrath</a> Here is my implementation which was inspired by <a class="mention" href="/u/jukiewiczm">@jukiewiczm</a>’s code. 
class MaskedLinear(nn.Module):
    def __init__(self, in_dim, out_dim, indices_mask):
        """
       :param in_features: number of input features
       :param out_features: number of output features
       :param indices_mask: list of two lists containing indices for dimensions 0 and 1, used to create the mask
       """
        super(MaskedLinear, self).__init__()
 
        def backward_hook(grad):
      …
  </blockquote>
</aside>

<p><a class="mention" href="/u/uchida-takumi">@uchida-takumi</a> I would be interested in your thoughts on comparisons between the approaches.</p>

theQmech: <p><a class="mention" href="/u/mariob6">@mariob6</a> I did this a long time ago when sparse tensors were just proposed in PyTorch. Since then I haven’t played around with sparse tensors at all. Sorry can’t help you here.</p>

Arvind_Subramaniam: <p>You cant prevent the weights from changing during gradient descent. However, you can introduce a parameter called mask, which multiplies a mask with the weights, and use that to do the forward pass and the backward pass.<br>
After this, you can use m.mask.shape, etc,. to look at the profile of the masked weights in your sparse network.<br>
Here is my implementation:</p>
<pre><code class="lang-auto">class newlinear(Function):
    def __init__(self,mask):
        super(newlinear, self).__init__()
        self.mask = mask

     #extendWeights is the masked weight which is used in the forward and backprop. 
     #Hence, the sparsity is maintained during backprop
    def forward(self, input, weight):
        self.save_for_backward(input, weight)
        extendWeights = weight.clone()
        extendWeights.mul_(self.mask.data)
        output = input.mm(extendWeights.t())
        return output

    def backward(self, grad_output):
        input, weight = self.saved_tensors
        grad_input = grad_weight  = None
        extendWeights = weight.clone()
        extendWeights.mul_(self.mask.data)

        if self.needs_input_grad[0]:
            grad_input = grad_output.mm(extendWeights)
        if self.needs_input_grad[1]:
            grad_weight = grad_output.clone().t().mm(input)
            grad_weight.mul_(self.mask.data)

        return grad_input, grad_weight

 ###############################################################
     
class NewLinear(torch.nn.Module):
    def __init__(self, input_features, output_features,matrix, mode='random'):
        super(NewLinear, self).__init__()
        self.input_features = input_features
        self.output_features = output_features
        self.matrix = matrix

        self.weight = nn.Parameter(data=torch.Tensor(output_features, input_features), requires_grad=True)
        
        '''**********************************Getting the mask**************************************'''

        ##Use your definition of mask here
        self.mask = np.random.randint(2,size=(output_features,input_features))
        kk += self.input_features
        self.mask =  torch.from_numpy(self.mask).cuda()
        
        
        nn.init.kaiming_normal(self.weight.data,mode='fan_in')
        '''Declaring the mask as a parameter'''
        self.mask =  nn.Parameter(self.mask.cuda())
        self.mask.requires_grad = False

    def forward(self, input):
        return newlinear(self.mask)(input, self.weight)
</code></pre>

deadbeef: <p>Note for anyone trying to do this: If you use an optimiser with momentum e.g. Adam, then you will see gradient on each weight regardless of whether you set the weight to 0. You have to mask the actual gradient matrix before stepping your model.</p>

Zeeyuu: <p>Thank you so much for your contribution!</p>

gallo: <p>The novel <a href="https://pytorch.org/tutorials/intermediate/pruning_tutorial.html" rel="noopener nofollow ugc">Prune module</a> also works for this.</p>
<pre><code class="lang-auto">import torch.nn.utils.prune as prune

# define some linear layer
out_features = 5
in_features = 10
linLayer = torch.nn.Linear(in_features, out_features, bias = False) 

# define your maks
mask = np.random.randint(2,size=(out_features,in_features)) 
# apply the mask
prune.custom_from_mask(
      linLayer, name='weight', mask=torch.tensor(mask))

plt.spy(linLayer.weight.data)

# [...] training
# after training you can remove the mask for inference
prune.remove(linLayer, "weight") ```</code></pre>