What is the standard way to batch and do a forward pass through tree structured data (ASTs) in pytorch so to leverage the power of GPUs?

Brando_Miranda: <p>In vision we usually process multiple images at once. I believe this is possible because most images are the same size or we can easily pad them with zeros if they are not (and thus process many at once).</p>
<p>However, I don’t see a simple way to do this for structure data like data (e.g. programs, code, not NLP) in the form of Trees (e.g. while using a TreeLSTM). It seems that the processing of the data and the forward pass of the TreeLSTM are tightly coupled. e.g. if our TreeLSTM is processing the Abstract Syntax Tree (AST) top-down or bottom-up my guess is that the batching has to be different and custom for each of these. What I am imagining very vaguely and abstracting is that we get each node from the AST, generate the tensor for that node and then do the 1 step forward pass. If we were to do this for many ASTs I’d assume to do this style of batching we’d need to collect ASTs of the same topology (not only size, since I assume even if two trees are the same size their branches everywhere have to match somehow…unless we padd them with zeros?). That way I’d imagine the code to be something like this:</p>
<pre><code class="lang-auto">def forward(asts):
    """ asts have already been collated according to their matching topology. """
    # order trees according to how the TreeNN will process it e.g. post order for bottom up
    order_asts_according_to_traversal = oder_asts(asts)  ## e.g. order in a post order
    # do the forward pass
    current_node_embeddings =  torch.zeros(batch_size, D_embedding)
    for batch_node in order_asts_according_to_traversal:
        current_node_embeddings = node_step_forward(batch_node, current_node_embeddings)
    return current_node_embeddings
</code></pre>
<p>in addition, we’d have to collate the asts (pseudo-code that I didn’t write since I’m not sure how to do this). For me, my data is usually already parsed (e.g. in json or s-expressions, but I prefer and plan to use json for each data point).</p>
<p>The problem that I have above does not feel new. TreeNNs (e.g. TreeLSTMs) have been implemented before so I’d rather use a tested method - the <strong>standard</strong> way to process and do forward passes on tree data for pytorch. Is there such a tutorial or library for pytorch one could easily use?</p>
<hr>
<h1>Research I’ve done</h1>
<p>I’ve found the following options:</p>
<ol>
<li>using CoqGym’s example (<a href="https://github.com/princeton-vl/CoqGym/discussions/48" class="inline-onebox" rel="noopener nofollow ugc">How to efficiently process a batch of terms in one go for Tree Neural Networks? · Discussion #48 · princeton-vl/CoqGym · GitHub</a> and <a href="https://github.com/princeton-vl/CoqGym/issues/16" class="inline-onebox" rel="noopener nofollow ugc">How to efficiently process a batch of terms in one go for Tree Neural Networks · Issue #16 · princeton-vl/CoqGym · GitHub</a>) though that general repo feels very hacky and their way of doing things seem very very custom, which worries me.</li>
<li>this option: <a href="https://medium.com/near-ai/pytorch-dynamic-batching-f4df3dbe09ef" class="inline-onebox" rel="noopener nofollow ugc">PyTorch — Dynamic Batching. If you have been reading my blog, you… | by Illia Polosukhin | NEAR AI | Medium</a>
</li>
<li>a pytorch implementation of tree lstms: <a href="https://github.com/unbounce/pytorch-tree-lstm" class="inline-onebox" rel="noopener nofollow ugc">GitHub - unbounce/pytorch-tree-lstm: Pytorch implementation of the child-sum Tree-LSTM model</a>
</li>
<li>yet another link: <a href="https://developer.nvidia.com/blog/recursive-neural-networks-pytorch/" class="inline-onebox" rel="noopener nofollow ugc">Recursive Neural Networks with PyTorch | NVIDIA Developer Blog</a>
</li>
</ol>
<p>what worries me is that none of these seemed to be part of the standard pytorch ecosystem (<a href="https://pytorch.org/ecosystem/" class="inline-onebox" rel="noopener nofollow ugc">Ecosystem | PyTorch</a>), which raises doubt to me which one to use (if any of them). Should I be using any of these above? My guess is that perhaps the 4th one is the one I should be using? Ideally, I’d like to be able to implement a custom data set object that returns ASTs (from my Json file or s-expressions) and create a batch that takes advantage of GPU acceleration.</p>
<hr>
<h2>Another Solution: using Graph Neural Network libraries</h2>
<p>I also realized that tree are really just a special case of graphs (perhaps not in the way we do the forward pass since GNNs seem to have a more complex set of “cycles” to produce embeddings). So with that I found these links:</p>
<ol>
<li>question from the pytorch forum: <a href="https://discuss.pytorch.org/t/what-is-an-efficient-way-to-do-a-forward-pass-of-a-batch-of-data-for-an-graph-neural-network-gnn/107086/2" class="inline-onebox">What is an efficient way to do a forward pass of a batch of data for an Graph Neural Network (GNN)? - #2 by Brando_Miranda</a>
</li>
<li>deep graph library: <a href="https://www.dgl.ai/" rel="noopener nofollow ugc">https://www.dgl.ai/</a>
</li>
<li>geometric library: <a href="https://github.com/rusty1s/pytorch_geometric" class="inline-onebox" rel="noopener nofollow ugc">GitHub - rusty1s/pytorch_geometric: Geometric Deep Learning Extension Library for PyTorch</a>
</li>
</ol>
<p>with the last two being part of the pytorch ecosystem. Though these are built for GNNs, so I am unsure if they are really worth the effort to check out to decide what to use.</p>
<p><strong>Overall: What is the standard/recommended way to generate batches and process batches of tree data in pytorch to leverage the power of GPUs?</strong></p>
<hr>
<p>cross posted:</p>
<ul>
<li><a href="https://stackoverflow.com/questions/66052104/what-is-the-standard-way-to-batch-and-do-a-forward-pass-through-tree-structured" class="inline-onebox" rel="noopener nofollow ugc">python - What is the standard way to batch and do a forward pass through tree structured data (ASTs) in pytorch so to leverage the power of GPUs? - Stack Overflow</a></li>
<li><a href="https://discuss.pytorch.org/t/what-is-the-standard-way-to-batch-and-do-a-forward-pass-through-tree-structured-data-asts-in-pytorch-so-to-leverage-the-power-of-gpus/110939" class="inline-onebox">What is the standard way to batch and do a forward pass through tree structured data (ASTs) in pytorch so to leverage the power of GPUs?</a></li>
</ul>

Brando_Miranda: <p>A possible solution is to use a graph NN library to do the heavy work:</p>
<blockquote>
<p>The challenge in training Tree-LSTMs is batching — a standard technique in machine learning to accelerate optimization. However, since trees generally have different shapes by nature, parallization is non-trivial. DGL offers an alternative. Pool all the trees into one single graph then induce the message passing over them, guided by the structure of eac</p>
</blockquote>
<p>e.g. from dgl: <a href="https://docs.dgl.ai/en/0.4.x/tutorials/models/2_small_graph/3_tree-lstm.html" class="inline-onebox" rel="noopener nofollow ugc">Tutorial: Tree-LSTM in DGL — DGL 0.4.3post2 documentation</a></p>