Why does the size of a batch affect the features extracted from a pre-trained model in eval mode?

username: cvenoburi
cooked: <p>Hi all,<br>
I realized that when I use different <code>batch_size</code> values in <code>torch.utils.data.DataLoader</code>, I end up with slightly different feature vectors (which sometimes affects the model predictions) although I use both <code>model.eval()</code> and <code>torch.no_grad()</code> while extracting features.</p>
<p>To replicate this issue, I created the following test scenario. I try to extract features for 200 images where the first 100 images are exactly the same as the last 100 (in the exact same order). I simply use default pre-trained VGG16 network for this experiment. I test with batch sizes of 20, 50, and 180.</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/cvenoburi/embeddings_batchsizes/blob/master/compare_embeddings_and_batch_size.ipynb" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/cvenoburi/embeddings_batchsizes/blob/master/compare_embeddings_and_batch_size.ipynb" target="_blank" rel="nofollow noopener">cvenoburi/embeddings_batchsizes/blob/master/compare_embeddings_and_batch_size.ipynb</a></h4>
<pre><code class="lang-ipynb">{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "PyTorch Version:  1.3.0\n",
      "Torchvision Version:  0.4.1a0+d94043a\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "os.environ[\"CUDA_VISIBLE_DEVICES\"]=\"1\"\n",
    "import sys\n",
</code></pre>

  This file has been truncated. <a href="https://github.com/cvenoburi/embeddings_batchsizes/blob/master/compare_embeddings_and_batch_size.ipynb" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>The daunting observations are in cells 12 onwards. Here are my quick notes:</p>
<ul>
<li>Cell 12 shows that the input tensors for images <code>i</code> and <code>i+100</code> are the same.</li>
<li>Cell 13 shows that output tensors (obtained with different batch sizes in dataloader) for image <code>i</code> are different.</li>
<li>Cell 14 shows that output tensors for images <code>i</code> and <code>i+100</code> are the same for <code>batch_size=20</code> and <code>batch_size=50</code> whereas the tensors differ for <code>batch_size=180</code>.</li>
<li>Cell 14 also shows that the output tensor for image <code>i</code> when <code>batch_size=20</code> is equal to the output tensor for image <code>i+100</code> when <code>batch_size=180</code> (probably because there are only 20 images left for the second batch of <code>dataloader180</code> although <code>batch_size</code> was set to 180).</li>
</ul>
<p>I think the last point is the most important one because it shows that the issue is probably not exactly about the actual value of <code>batch_size</code> but it is more about how many images there are actually in a batch waiting to be processed.</p>
<p>Unfortunately, these subtle-looking variations in feature values may yield different predictions at test time. I am not sure if there is a fix to this issue, but what should be the rule of thumb?<br>
Would it be better to always use a <code>batch_size=1</code> while testing a model or extracting features?</p>
<p>Thanks!</p>