Separating batch of images before or after embedding causes massively different results. Why?

Valerio_Biscione: <p>Hello.<br>
I am trying to understand a weird phenomenon and I really hope you can help me, as it seems to underpin something quite fundamental that I am missing here.</p>
<p>I have a batch of images. The first <code>k</code> are passed through an embedding network, the other k*n are passed through the same network and also through an RNN network which outputs k hidden states(I believe the details are not important). I then input the <code>k</code> hidden states and the <code>k</code> embedding from the first group to a relation networks which compares them.</p>
<p>Everything else equal, I noticed that the order of embedding at the beginning of the training step is important in such a way that, in one case, the gradient rapidly goes to zero and the network doesn’t learn anything, and in the other it learns rapidly.<br>
<code>x</code> is the batch of images.<br>
This is the one that doesn’t work.</p>
<pre><code class="lang-auto">A = x[:k]
B = x[k:]
emb_A = model.image_embedding(A)
emb_B = model.image_embedding(B)
...
</code></pre>
<p>I have also tried to use two different networks (with same architecture) for <code>emb_A</code> and <code>emb_B</code>, with the same unsatisfying results.</p>
<p>this is the one that works:</p>
<pre><code class="lang-auto">emb_all = model.image_embedding(x)
emb_A = emb_all[:k]
emb_B = emb_all[k:]
...
</code></pre>
<p>I don’t understand why it should matter whether I separated the images before or after doing the embedding. I do get a hint that by reusing the same network on the two groups the gradient may have trouble converging - but as I said the same thing happens when 2 networks are used, which makes me believe that its due to the slicing operation.<br>
Does anyone have any idea?<br>
Thanks</p>

ptrblck: <p>The operations should be equal up to the error created by the limited floating point precision as seen in this example:</p>
<pre><code class="lang-python"># setup
x = torch.randint(0, 100, (100, 10))
image_embedding = nn.Embedding(100, 300)
k = 50

# check forward
A = x[:k]
B = x[k:]
emb_A1 = image_embedding(A)
emb_B1 = image_embedding(B)

emb_all = image_embedding(x)
emb_A2 = emb_all[:k]
emb_B2 = emb_all[k:]

print((emb_A1 - emb_A2).abs().max())
print((emb_B1 - emb_B2).abs().max())

# check backward
emb_A1.mean().backward()
grad_A1 = image_embedding.weight.grad.clone()

image_embedding.zero_grad()
emb_A2.mean().backward(retain_graph=True)
grad_A2 = image_embedding.weight.grad.clone()
print((grad_A1 - grad_A2).abs().max())

emb_B1.mean().backward()
grad_B1 = image_embedding.weight.grad.clone()

image_embedding.zero_grad()
emb_B2.mean().backward()
grad_B2 = image_embedding.weight.grad.clone()
print((grad_B1 - grad_B2).abs().max())
</code></pre>

Valerio_Biscione: <p>Thank you, so the error must be somewhere else</p>

InnovArul: <aside class="quote no-group" data-username="Valerio_Biscione" data-post="1" data-topic="105988">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/valerio_biscione/48/22759_2.png" class="avatar"> Valerio_Biscione:</div>
<blockquote>
<pre><code class="lang-auto">emb_all = model.image_embedding(x)
emb_A = emb_all[:k]
emb_B = emb_all[k:]
</code></pre>
</blockquote>
</aside>
<p>Just a wild guess. Do you use <code>batchnorm</code> in your network?<br>
If so, if <code>A</code> belongs to a different distribution of images than <code>B</code>, then <code>batchnorm</code> statistics may be widely different and they would have a tough time learning statistics, when you pass them individually I guess. In the case that you pass them together, they might get normalized properly.</p>