PyTorch graphs explanation

JuanFMontesinos: <p>Hi, I would like to ask some questions about how graphs are computed.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/0/0b3703dd2a3bf9c77dd4e84dd0984ba045d9345e.png" data-download-href="https://discuss.pytorch.org/uploads/default/0b3703dd2a3bf9c77dd4e84dd0984ba045d9345e" title="Untitled%20Diagram"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/0/0b3703dd2a3bf9c77dd4e84dd0984ba045d9345e_2_690x336.png" alt="Untitled%20Diagram" data-base62-sha1="1Bd7wVtMQJjLWIpJ2gyP4zqmAfs" width="690" height="336" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/0/0b3703dd2a3bf9c77dd4e84dd0984ba045d9345e_2_690x336.png, https://discuss.pytorch.org/uploads/default/original/2X/0/0b3703dd2a3bf9c77dd4e84dd0984ba045d9345e.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/0/0b3703dd2a3bf9c77dd4e84dd0984ba045d9345e.png 2x" data-dominant-color="CFCBD1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Untitled%20Diagram</span><span class="informations">782×381 12.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Imagine a simple multimodal model where we have 3 sub-networks, A,B,C.</p>
<p>INa and INb are torch variables, with whatever dimensionality but the dimension corresponding to the batch is dim 0, and both have the same amount of samples.</p>
<p>If the proper workflow for one sample  were:<br>
A process a sample of INa and B process a sample of INb, then C takes as input OUTa and OUTb to compute OUT.<br>
and so on through the whole batch. When the batch is already processed compute loss and backprop</p>
<p>Is the previously mentioned graph equal to a graph generated by the following process?:<br>
A process the whole batch INa, B process the whole batch INb, then C process the the whole batch OUTa, OUTb and loss-backward?</p>
<p>in pseudo-code:</p>
<pre><code class="lang-auto">for i in range(batch_size):
     outa_i = model_A(INa_i)
     outb_i = model_B(INb_i)
     out_i = model_C(outa_i,outb_i)s
stack(out)
loss
backward
</code></pre>
<pre><code class="lang-auto">for i in range(batch_size):
     outa_i = model_A(INa_i)
outa=stack(outa_i)
for i in range(batch_size):
     outb_i = model_B(INb_i)
outb=stack(outb_i)
for i in range(batch_size):
     out_i = model_C(outa,outb)
out=stack(out_i)

loss
backward
</code></pre>
<p>Is there a good explanation about pytorch graphs?</p>

smth: <p>They are equivalent graphs.<br>
However, in PyTorch, if you use <code>torch.nn</code> layers, they only know how to do batch processing.</p>
<p>So, your <code>for</code> loop over batch size is redundant, and infact incorrect, because <code>model_A</code> probably doesn’t know how to handle <code>INa_i</code> to give <code>outa_i</code>, but directly does: <code>outa = model_A(ina)</code>.<br>
You can fake batch processing by making <code>Ina_i</code> have a batch dimension of size <code>1</code>.</p>

JuanFMontesinos: <p>Sorry for the late reply.</p>
<p>I agree with you in case of using by-default torch.nn layers like 2d convolution, which expects to have an extra dimension for the batch. I was wondering what about custom nn.modules</p>
<p>Toy example here</p>
<pre><code class="lang-auto">import torch.nn as nn
import torch
import numpy as np

class trial(nn.Module):
    def __init__(self):
        super(trial, self).__init__()
        self.conv = nn.Conv2d(1, 4,kernel_size=3)
    def forward(self,input):
        dims = input.size()
        x = []
        for i in range(dims[0]):
            x.append(self.conv(input[i,:,:,:,:]))

        return dims, torch.stack(x)
class trial2(nn.Module):
    def __init__(self):
        super(trial2, self).__init__()
        self.conv = nn.Conv2d(1, 4,kernel_size=3)
    def forward(self,input):
        dims = input.size()
        x = self.conv(input)
        return x
x = torch.rand(10,1,50,50)
print 'X size {0}'.format(x.size())
m = trial()

y = m(x)
print 'Y size {0}'.format(y[0])
</code></pre>
<p>Imagine I have data of 4 dimensions + 1 for batch = 5D<br>
So lets imagine i have groups of 10 images to be processed:<br>
Batch x group  x channels x H x W<br>
[5,2,3,50,50]</p>
<p>In this case I have some questions:<br>
is using a for loop over dim 0 the only way to process this batch  ?</p>
<p>is the computational graph  the same doing m(input) for an input of size [5,2,3,50,50] than doing conv2d(input) for an input size [2x5,3,50,50]?  (This is, reshaping all the samples in one dimension)</p>
<p>I’ve seen that in case 2) the graph is<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/5/5f0933f783e74fe5f42fb548e3491bed23af3af2.png" data-download-href="https://discuss.pytorch.org/uploads/default/5f0933f783e74fe5f42fb548e3491bed23af3af2" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/5/5f0933f783e74fe5f42fb548e3491bed23af3af2_2_345x212.png" alt="image" data-base62-sha1="dyJ4M3WhNbInkXvKckTzt7jBXWi" width="345" height="212" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/5/5f0933f783e74fe5f42fb548e3491bed23af3af2_2_345x212.png, https://discuss.pytorch.org/uploads/default/optimized/2X/5/5f0933f783e74fe5f42fb548e3491bed23af3af2_2_517x318.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/5/5f0933f783e74fe5f42fb548e3491bed23af3af2_2_690x424.png 2x" data-dominant-color="D2DCDF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1141×703 35.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Meanwhile in case of using for loop and more dimensions the graph is<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/e/ec5115371d8a36cee1c86db3298a014829b22203.png" data-download-href="https://discuss.pytorch.org/uploads/default/ec5115371d8a36cee1c86db3298a014829b22203" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/e/ec5115371d8a36cee1c86db3298a014829b22203_2_345x118.png" alt="image" data-base62-sha1="xIyoWmfo4lD5D3YpOVQz9qdhy2T" width="345" height="118" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/e/ec5115371d8a36cee1c86db3298a014829b22203_2_345x118.png, https://discuss.pytorch.org/uploads/default/optimized/2X/e/ec5115371d8a36cee1c86db3298a014829b22203_2_517x177.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/e/ec5115371d8a36cee1c86db3298a014829b22203_2_690x236.png 2x" data-dominant-color="EAEBEC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1473×498 39.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Well obviously it is not, what is the proper way of dealing with data of custom dimensionality not to get improper graphs? Are there some rules about how to proceed at the time of working with multimodal NN?</p>

David_Choqueluque: <p>Hi, did you solve this question? I have the same doubt.</p>

JuanFMontesinos: <p>Hi,<br>
Gradients exist at element level. If you want to run a network several times without duplicating the graph you just need to forward once stacking everything in the batch dimension.</p>