Reparameterizing a Parameter

ajbrock: Hi guys, W_effective[:,:,::2,::2] = W_base And have a 5x5 filter which is effectively a dilated 3x3 filter, and where only the elements that I indexed in that call are parameters that can be updated. Note that I’m not just trying to dilate a filter (the convNd modules already look to have that on lock) but ideally I’d like to be able to arbitrarily parameterize a tensor by indexing. Any help is appreciated! Best, Andy

apaszke: <SCODE>class MyModule(nn.Module):
    def __init__(self):
        self.large_linear = nn.Linear(100, 100)

    def forward(self):
        x = F.sigmoid(self.large_linear(x))
        W_small = self.large_linear.weight[:10]
        b_small = self.large_linear.bias[:10]
        return F.linear(x, W_small, b_small)
<ECODE>

ajbrock: Thanks! Hmm, so including the paramterization in forward() makes sense so that the new param is a part of the graph…might it instead be easier to initialize a Parameter, set the static values to zero, then specifically index in the backprop to only update the desired indices? Best, Andy

apaszke: 

ajbrock: Alright, so I’ve dug further into this and found some interesting things. TL; DR: I solved it for my use-case, but I might have stumbled onto some bugs. Don’t use in-place operations, bad things happen. I managed to achieve what I was going for by creating the main weight parameter, w, and two intermediate variables, I initialize both W and M in the init method of the module, then call W[M] = w during the forward() method, and convolve using the modified W. This works, but for some reason when I do things this way it results in training starting out fast, then progressively slowing down (starting out at around 5 batches/s and dropping to 0.2 batches/s over the course of the first 1000 batches). It also throws an error about non-leaf variables not yet being serializable when I train to use torch.save, presumably because I’m creating some naughty nodes that I shouldn’t be. My initial suspicion was that I was creating additional subgraphs that weren’t being deleted, or not freeing memory appropriately (more on that in a moment), but the memory usage in this case was constant. Investigating further, I found that if I replaced W[M] = w with W.masked_copy_(M,w), I get an error after the first batch saying that I need to use"retain_variables=True" if I want to backpropagate a second time through the graph. The error method is a bit confusing here, as I am only calling backward() once. My intuition is that the above error occurs because I’m calling an in-place method in forward(), which seems to be against best pytorch practice at the moment, so whatever variables autograd would need to do the backprop aren’t getting saved. Calling backward(retain_variables=True) results in the same behavior as using W[M]=w; it works, but it progressively slows down throughout training. I’m still not sure on what’s causing the slowing–my best guess is that some part of the graph isn’t getting appropriately freed in such a way that rather than creating multiple subgraphs that take up memory, it’s just propping through the same graph element an increasing number of times on each successive iteration. Anyhow, I did manage to get things working for my use-case–rather than trying to do a masked copy or in-place operation, I just instantiate W as a full-rank tensor and drop W*M into the F.conv2d call. Works great and is about twice as fast as using the dilation parameter (presumably because it allows for the use of the cuDNN backend). Here’s a code snippet with which I’m currently getting  ~80-100% speedup over using the dilation keyword for my use-case. Note that this currently prevents you from saving with torch.save due to a “can’t serialize non-leaf variables yet” dealio. in init: in forward: Sorry for the wall of text, but hopefully this will prove enlightening and thorough if other people come along with similar issues on in-place ops. For the record, I’m using the build provided by conda install and am on python 2.7 (my attempts at building from source crash, sigh). I tested these with Cuda7.5 on a GTX980 and Cuda8.0 on a Titan X. Best, Andy

apaszke: This also causes the serialization failure - you can only save leaf Variables, but your W keeps hold of the history, so it’s not a leaf! That’s probably because PyTorch has to pick a more memory-intensive backend for dilated convolutions. If you can send me a diff that I could test, I could confirm that it’s because of this.

ajbrock: I nominate Adam for king of speed replies, wow! So I tried two variations on repackage_hidden in the forward() method, both of which solve this issue: I was only calling backward() a single time, hence my initial confusion; I mostly found this interesting because it only occurred when switching to masked_copy_ from a direct indexing call. Thanks, Andy

apaszke: