Model giving different test accuracies for different batch sizes

username: DL_aspirant
cooked: <p>I am running the official code for the pyramidal vision transformer on CIFAR-10.</p>
<p>For a batchsize of 128, my test accuracy starts at 40%,<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/7/7/77d24ff1996c9340fcbb93f3712751b63beaa19c.png" alt="image" data-base62-sha1="h5ZptKEDz7rOogRPT0exeidrYAY" width="317" height="102"></p>
<p>but for a batch size of 4, my model starts at a test accuracy of 44-45%.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/0/5/054bca43bb7e00f51b124c3a7c4ed798d869a3fd.png" alt="image" data-base62-sha1="KQLxTWGo8ifigFwKMa9kBOhVj7" width="327" height="499"></p>
<p>Here’s a high level visualization of the PVT model for a quick understanding:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/6/1646903fe89b178cfc800a18e4acac5bd6857dea.png" data-download-href="https://discuss.pytorch.org/uploads/default/1646903fe89b178cfc800a18e4acac5bd6857dea" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/6/1646903fe89b178cfc800a18e4acac5bd6857dea_2_690x159.png" alt="image" data-base62-sha1="3b3GwAK2GqhonWgEIES3TGw5jIu" width="690" height="159" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/6/1646903fe89b178cfc800a18e4acac5bd6857dea_2_690x159.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/6/1646903fe89b178cfc800a18e4acac5bd6857dea_2_1035x238.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/1/6/1646903fe89b178cfc800a18e4acac5bd6857dea_2_1380x318.png 2x" data-dominant-color="E1E8EF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1759×406 70.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I have attached my colab code below.</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://colab.research.google.com/drive/1tB7vnvQQn7kVnm6AqCV446ddk91TvVtn?usp=sharing">
  <header class="source">
      <img src="https://ssl.gstatic.com/colaboratory-static/common/f6c736cbde16632a9fcd16d9ab1970b1/img/favicon.ico" class="site-icon" width="128" height="128">

      <a href="https://colab.research.google.com/drive/1tB7vnvQQn7kVnm6AqCV446ddk91TvVtn?usp=sharing" target="_blank" rel="noopener nofollow ugc">colab.research.google.com</a>
  </header>

  <article class="onebox-body">
    <img width="260" height="260" src="https://colab.research.google.com/img/colab_favicon_256px.png" class="thumbnail onebox-avatar">

<h3><a href="https://colab.research.google.com/drive/1tB7vnvQQn7kVnm6AqCV446ddk91TvVtn?usp=sharing" target="_blank" rel="noopener nofollow ugc">Google Colaboratory</a></h3>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>The code for the model has been directly taken from the official PVT code:</p><aside class="onebox githubblob" data-onebox-src="https://github.com/whai362/PVT/blob/962734be5c95d61bce809c252caa06d00618b26a/classification/pvt_v2.py">
  <header class="source">

      <a href="https://github.com/whai362/PVT/blob/962734be5c95d61bce809c252caa06d00618b26a/classification/pvt_v2.py" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/whai362/PVT/blob/962734be5c95d61bce809c252caa06d00618b26a/classification/pvt_v2.py" target="_blank" rel="noopener nofollow ugc">whai362/PVT/blob/962734be5c95d61bce809c252caa06d00618b26a/classification/pvt_v2.py</a></h4>


      <pre><code class="lang-py">import torch
import torch.nn as nn
import torch.nn.functional as F
from functools import partial

from timm.models.layers import DropPath, to_2tuple, trunc_normal_
from timm.models.registry import register_model
from timm.models.vision_transformer import _cfg
import math


class Mlp(nn.Module):
    def __init__(self, in_features, hidden_features=None, out_features=None, act_layer=nn.GELU, drop=0., linear=False):
        super().__init__()
        out_features = out_features or in_features
        hidden_features = hidden_features or in_features
        self.fc1 = nn.Linear(in_features, hidden_features)
        self.dwconv = DWConv(hidden_features)
        self.act = act_layer()
        self.fc2 = nn.Linear(hidden_features, out_features)
</code></pre>



  This file has been truncated. <a href="https://github.com/whai362/PVT/blob/962734be5c95d61bce809c252caa06d00618b26a/classification/pvt_v2.py" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>I am not sure why I am getting different accuracies for different batch sizes in Pytorch. I have converted the code to keras and am getting the same test accuracy for different batchsizes in keras, so  I’m not sure where I’m going wrong. Will be glad if someone could help me with this! Thanks!</p>

username: ptrblck
cooked: <p>Changing the validation batch size alone should not change the validation accuracy.<br>
However, in your current code you are changing the training and validation batch sizes together, which is expected to potentially change the accuracy, since different training batch sizes would change the convergence of the model.<br>
I don’t know why you are not seeing the same issue in Keras, but it’s surprising to see the same accuracy for different training batch sizes.</p>

username: DL_aspirant
cooked: <p>Thanks for the reply! I’ll look into my keras code just to see if I am missing out on anything. Also, In my keras model, the model accuracy is stagnating after 10-15 epochs. Is there anything I can try to ensure I am getting the same accuracy as the pytorch model?</p>