Confusion about computational graph and gradient calculation in multitask learning setting

Mainul: <p>Let’s say following multi-task learning setting like <strong>Fig:1</strong>, where we have two related datasets (dataset 1 and dataset 2). For each iteration creat a batch of 64 (32+32) that goes through some layers. Each dataset has different classifiers. Next to those common middle layers, there are two classifiers for each dataset. Both losses affect on middle layers.</p>
<p>Now, let’s say another experimental situation <strong>(fig: 2)</strong>, where I don’t calculate loss 2 and its gradients, But both datasets go through the middle layers. Here, I only calculate loss 1 and it optimizes using this loss.</p>
<p>My question is, should not the gradient calculation and optimization be the same of if I don’t use the second dataset at all like <strong>fig:3</strong>?  Because, I think, the computation graph of loss 1 (<strong>fig:2</strong>) does not contain any information about the second dataset.</p>
<p>But, this is not happening in my case. <strong>Fig:3</strong> model optimizes much better and give better performance than <strong>fig: 2</strong><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/3/d31ea99bbdf1da224b88bac6174bf6cc5adfa4c6.png" data-download-href="https://discuss.pytorch.org/uploads/default/d31ea99bbdf1da224b88bac6174bf6cc5adfa4c6" title="Screen Shot 2020-10-08 at 6.34.23 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/3/d31ea99bbdf1da224b88bac6174bf6cc5adfa4c6_2_690x392.png" alt="Screen Shot 2020-10-08 at 6.34.23 PM" data-base62-sha1="u7Erj2Izqv6UbOOVSpBmlBOH6Ky" width="690" height="392" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/3/d31ea99bbdf1da224b88bac6174bf6cc5adfa4c6_2_690x392.png, https://discuss.pytorch.org/uploads/default/original/3X/d/3/d31ea99bbdf1da224b88bac6174bf6cc5adfa4c6.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/d/3/d31ea99bbdf1da224b88bac6174bf6cc5adfa4c6.png 2x" data-dominant-color="F6F7F9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2020-10-08 at 6.34.23 PM</span><span class="informations">780×444 25.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/d/9/d9fbd05af168028f1e693aacefafabab07e4977f.png" alt="Screen Shot 2020-10-08 at 6.28.03 PM" data-base62-sha1="v6n7qIte6GobUifQdeRrMnx2w9N" width="354" height="402"></p>

albanD: <p>Hi,</p>
<p>The major differences I can think of are:</p>
<ul>
<li>The batch size might be half of what you expect?</li>
<li>Layers like batchnorm will behave differently as they won’t see samples from the second dataset.</li>
</ul>

Mainul: <ul>
<li>
<p>For calculating loss 1, the batch size is  32 for both models.</p>
</li>
<li>
<p>There is no batch norm layer. The middle layer actually is a CNN layer</p>
</li>
</ul>

albanD: <p>If there is only a CNN layer, then all will be the same from pytorch point of view.<br>
Most likely a different config in your code between the two no?</p>

Mainul: <p>So, I did some debugging and inspections. I fixed the input and weights of the models.<br>
Based on some experiments, I think that there is an effect of dataset 2.<br>
Here are the experiments I did:</p>
<ol>
<li>
<p>I found that problem (the output dissimilarities) is not in optimization, the problem occurs from the very first prediction (before even first call of <code>.backward()</code>). The predictions of first batch of dataset 1 is different in figure 2 and figure 3.</p>
</li>
<li>
<p>At fig 2 model, I stop passing the dataset 2 through CNN and now the prediction is exactly the same as the model of fig 3. This already showing the effect of dataset 2. But I did one further experiment for more confidence.</p>
</li>
<li>
<p>If I change the examples of dataset 2 (by changing random seed) for creating one batch (examples of dataset 1 + dataset 2 ) for model of fig:2, the prediction for the example of dataset 1 also change each time. If I fix the dataset 2 examples, the prediction of dataset 1 examples is always the same.</p>
</li>
</ol>
<p>Now, the only thing I can think of is changing due to the addition of dataset 2, is the padding. When everything is fixed (weights and examples), the addition of another dataset creates different padding (The padded dataset 1 input will be different between fig 2 and fig 3). I used the default zero padding and the data is natural language not image. Pdding is used because sentences and documents that are of variable sizes.</p>
<p>Do you think, this might be the issue?</p>

albanD: <p>Well if there is only a single conv layer in the middle, it won’t change anything.<br>
So it will depend on what other code you have the behaves differently in both cases indeed.</p>