How to split the Variables along Batch Dimension to several Variables?

yikang-li: Hi all, I have got a batch of features (N features). Now I want to pass message among different features. I found that if I manipulate the features along batch dim, then errors will be raised when backward(). The codes are like below: <SCODE>import torch
from torch.autograd import Variable
import numpy as np
import torch.nn as nn

a = Variable(torch.Tensor(3, 5).normal_(), requires_grad=True)
b = Variable(torch.Tensor(3, 5).normal_(), requires_grad=True)

for i in range(b.size()[0]):
    for j in range(a.size()[0]):
        b[i] = b[i] + a[j]
        
b.backward(torch.Tensor(5, 3).normal_())
<ECODE> When running, I got “inconsistent tensor size at /data/users/soumith/miniconda2/conda-bld/pytorch-0.1.7_1485444530918/work/torch/lib/TH/generic/THTensorCopy.c:40”. Is it possible to divid the original Variable into several parts and then process them?

apaszke: <SCODE>import torch
from torch.autograd import Variable
import numpy as np
import torch.nn as nn

a = Variable(torch.Tensor(3, 5).normal_(), requires_grad=True)
b = Variable(torch.Tensor(3, 5).normal_(), requires_grad=True)

a_summed = a.sum(0) # sum along first dimension
result = b + a_summed.expand_as(b)
result.backward(torch.randn(5, 3))
<ECODE>

yikang-li: Yes, that’s the same. However, the ‘backward()’ will lead to an error? Could you tell me I to do that?

apaszke: 

yikang-li: Thank you for your timely reply. I apologize for misleading you with the example. So what should I do if I want to manipulate the feature along batch dim. In addition, what’s the meaning of leaf Variables?

apaszke: I’m sorry but I don’t know what passing a message along a dimension is. I’d recommend you to look for out-of-place functions that can do this, but I can’t recommend any, as I don’t know what are you trying to achieve. Leaf Variables are Variables that are not results of any computation, so: <SCODE>x = Variable(torch.randn(5, 5)) # this is a leaf
y = x + 2 # y is not a leaf
<ECODE>

yikang-li: I have got that! The problem has fixed. Just add an operation will be enough. <SCODE>import torch
from torch.autograd import Variable
import numpy as np
import torch.nn as nn


x = Variable(torch.Tensor(3, 5).normal_(), requires_grad=True)
y= Variable(torch.Tensor(3, 5).normal_(), requires_grad=True)

a = x + 0
b = y + 0

for i in range(b.size()[0]):
    for j in range(a.size()[0]):
        b[i] = b[i] + a[j]
        
b.backward(torch.Tensor(3, 5).normal_())
<ECODE> Thank you very much for your help and patience.

apaszke: In most cases (like in your example), you can avoid these assignments, and I’d recommend doing that. It will likely result in better performance. Nevertheless, we do support them, so if there’s no other way, you can still do that.

yikang-li: Got it. Thank you very much. I really appreciate your works for PyTorch and the CV community!

pranav: I have been using PyTorch for a couple of weeks now, working mostly with complex RNNs models. In such situations, the natural way of accumulating state for me till now has been to create a Tensor and keep filling it with new state at each time step. But I haven’t been able to write code like that in PyTorch since in-place ops on leaf nodes are not supported. I have had to use lists with manual concatenation and indexing is messy. I can use the ugly hack shown above where in a dummy operation is performed to make a node non-leaf. But it would be great to have some implicit adjustment that allows for in-place ops on leaves. If you think this would be a useful feature to have (or conversely if its not something you are intentionally against), and if you can give some pointers as to how to make this happen, I can work on it and submit a PR . I hope I am not missing something trivial here.

smth: Thanks a lot for the feedback.

apaszke: You can fill in a leaf Variable. Just don’t declare it as requiring grad - you don’t need grad w.r.t. the original content that will get overwritten.