How to do mini batch with dynamic computation graph

shijie-wu: Hi all, I am new to framework with dynamic computation graph. I search everywhere but I couldn’t find a reference about how to implement mini-batch with RNN or even tree LSTM with varying length input. So I guess my general problem is how to do mini batch with dynamic computation graph. Thanks.

jekbradbury: TreeRNNs are harder. I’ll add an example soon that does this, but the general idea for TreeRNNs is that batching is up to you as the user, and you should split and concatenate when you need to. So if you use a binary tree structure, you can represent it as a shift-reduce parser (see the SPINN paper from Bowman et al) that means you can process multiple trees in parallel by doing preprocessing like this: <SCODE>input:
 tree1: ((ab)c)
 tree2: (d(ef))
preprocessed input:
        1        2        3        4        5
 tree1: SHIFT(a) SHIFT(b) REDUCE   SHIFT(c) REDUCE
 tree2: SHIFT(d) SHIFT(e) SHIFT(f) REDUCE   REDUCE
<ECODE> and then using advanced indexing to copy all the tokens for SHIFT at each timestep in parallel while concatenating the stack representations for batched REDUCE. Sorry if this is confusing, I promise an example will be up soon. I would add that PyTorch is impressively fast on TreeRNNs even without batching.

shijie-wu: Thank you for the pointer. So it’s actually up to the user to design batching mechanism. Thank you all for building pytorch with amazing flexibility and great tutorial. Just curious, is there any plan to release a technical report about the performance of pytorch compared to other framework with support of dynamic computation graph?

apaszke: 

shijie-wu: That make sense. Thanks for the great work.

rituk: examples/snli/train.py

apaszke: 

supakjk: 

jekbradbury: Yes, this kind of padding is a stopgap until PyTorch has full masked RNN support, which is on its way.

xuehy: What if we are facing arbitrary trees rather than binary ones? This can correspond to the childsum treelstm where each node can have different number of children. Is it still possible to batch with the shift-reduce strategy?

jekbradbury: It’s possible but a lot harder.

xuehy: I hear that tensorflow-fold is able to batch trees of arbitrary shapes. Is there similar implementations in pytorch? Why no body is trying to make a tool?

Andre_Holzner: for what it’s worth, I could install torchtext using: pip install git+https://github.com/pytorch/text.git (in a virtualenv environment, otherwise try with --user)

Brando_Miranda: whats an example for a feedforward NN or CNN? I try to index my torch arrays of data and it says I can’t/shouldn’t be using numpy to index things. As in: <SCODE>def get_batch(X,Y,M):
    N = len(Y)
    valid_indices = np.array( range(N) )
    batch_indices = np.random.choice(valid_indices,size=M,replace=False)
    batch_xs = X[batch_indices,:]
    batch_ys = Y[batch_indices]
    return batch_xs, batch_ys
<ECODE> where X and Y are torch tensors (or variables).

Brando_Miranda: I think my code runs now, but it seems there has to be a better way than doing: <SCODE>def get_batch2(X,Y,M,dtype):
    X,Y = X.data.numpy(), Y.data.numpy()
    N = len(Y)
    valid_indices = np.array( range(N) )
    batch_indices = np.random.choice(valid_indices,size=M,replace=False)
    batch_xs = torch.FloatTensor(X[batch_indices,:]).type(dtype)
    batch_ys = torch.FloatTensor(Y[batch_indices]).type(dtype)
    return Variable(batch_xs, requires_grad=False), Variable(batch_ys, requires_grad=False)
<ECODE> <SCODE>    #valid_indices = torch.arange(0,N).numpy()
    #valid_indices = np.array( range(N) )
    #batch_indices = np.random.choice(valid_indices,size=M,replace=False)
    #indices = torch.LongTensor(batch_indices)
    #batch_xs, batch_ys = torch.index_select(X_mdl, 0, indices), torch.index_select(y, 0, indices)
    #batch_xs,batch_ys = torch.index_select(X_mdl, 0, indices), torch.index_select(y, 0, indices)
<ECODE> wonder if this silly moving from numpy to torch is actually slowing my code down! I hope not.