Network doesn't learn when increasing the dataset size

priscacheese: <p>Hi,</p>
<p>I am trying to re-implement <a href="https://github.com/ryanzhangfan/NeuroSAT" rel="noopener nofollow ugc">this model</a> using Pytorch Geometric, the model is based on the original one written with TensorFlow. The goal is to train a neural network to classify SAT formulas represented as graphs into two different classes (satisfiable and unsatisfiable formulas).</p>
<p>I’ve created a dataset of 200000 samples, based on the original code, and started training with a small subset of 2000 samples to check whether the network would work. What happens is that the training works when using a small subset of the dataset, but as soon as I increase the dimension of the dataset too much, the loss function doesn’t decrease anymore and the accuracy on the train dataset remains stable around 0.5. (The accuracy on the validation dataset is also always around 0.5 for all dataset dimension I’ve tested so far)</p>
<p>Here are a few plots of the loss function evolution with respect to a few different dataset sizes:</p>
<p>2K samples:<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/d/6d4d1f5b3b05f5639850786c9d38ed2c70553cd3.png" data-download-href="https://discuss.pytorch.org/uploads/default/6d4d1f5b3b05f5639850786c9d38ed2c70553cd3" title="2000_samples"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/d/6d4d1f5b3b05f5639850786c9d38ed2c70553cd3_2_690x345.png" alt="2000_samples" data-base62-sha1="fAVhgkHM2I8hMFt2ORZvEgVLDzB" width="690" height="345" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/d/6d4d1f5b3b05f5639850786c9d38ed2c70553cd3_2_690x345.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/d/6d4d1f5b3b05f5639850786c9d38ed2c70553cd3_2_1035x517.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/d/6d4d1f5b3b05f5639850786c9d38ed2c70553cd3_2_1380x690.png 2x" data-dominant-color="FDFCFC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">2000_samples</span><span class="informations">2400×1200 71.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
10K samples:<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/c/3c7a8e86c3a8c246e06a274a7a18c0afa8f15429.png" data-download-href="https://discuss.pytorch.org/uploads/default/3c7a8e86c3a8c246e06a274a7a18c0afa8f15429" title="10K_samples"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/c/3c7a8e86c3a8c246e06a274a7a18c0afa8f15429_2_690x345.png" alt="10K_samples" data-base62-sha1="8D1dWhzUS5vd09lNmbUZMSxYoZj" width="690" height="345" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/c/3c7a8e86c3a8c246e06a274a7a18c0afa8f15429_2_690x345.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/c/3c7a8e86c3a8c246e06a274a7a18c0afa8f15429_2_1035x517.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/c/3c7a8e86c3a8c246e06a274a7a18c0afa8f15429_2_1380x690.png 2x" data-dominant-color="FDFCFD"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">10K_samples</span><span class="informations">2400×1200 81.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
20K samples:<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/a/eaf8003a9f9d79e97da553a5be3486eefd938016.png" data-download-href="https://discuss.pytorch.org/uploads/default/eaf8003a9f9d79e97da553a5be3486eefd938016" title="20K_samples"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/a/eaf8003a9f9d79e97da553a5be3486eefd938016_2_690x345.png" alt="20K_samples" data-base62-sha1="xwD4bFjvHFONAjcNZweRjcdKYt0" width="690" height="345" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/a/eaf8003a9f9d79e97da553a5be3486eefd938016_2_690x345.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/a/eaf8003a9f9d79e97da553a5be3486eefd938016_2_1035x517.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/a/eaf8003a9f9d79e97da553a5be3486eefd938016_2_1380x690.png 2x" data-dominant-color="FDFAFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">20K_samples</span><span class="informations">2400×1200 96.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
80K samples:<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/3/63b701e434ac9b3f403f5b7cbe1c3b35685a4050.png" data-download-href="https://discuss.pytorch.org/uploads/default/63b701e434ac9b3f403f5b7cbe1c3b35685a4050" title="80K_samples"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/3/63b701e434ac9b3f403f5b7cbe1c3b35685a4050_2_690x345.png" alt="80K_samples" data-base62-sha1="ee7mo3eC1oaoKjvVgWzPSsEqfwA" width="690" height="345" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/3/63b701e434ac9b3f403f5b7cbe1c3b35685a4050_2_690x345.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/3/63b701e434ac9b3f403f5b7cbe1c3b35685a4050_2_1035x517.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/3/63b701e434ac9b3f403f5b7cbe1c3b35685a4050_2_1380x690.png 2x" data-dominant-color="FAFCFC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">80K_samples</span><span class="informations">2400×1200 102 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>(when training on the whole dataset the loss function is basically constant at 0.69)</p>
<p>I am using the Adam optimizer with decreasing learning rate and binary cross entropy loss, as in the original code:</p>
<pre><code class="lang-auto">criterion = nn.BCEWithLogitsLoss()

lr_start = 2e-4
l2_weight = 1e-10 # L2 regularization weight
optimizer = optim.Adam(net.parameters(), lr=lr_start, weight_decay=l2_weight)

lr_decay = torch.tensor(0.99, dtype=torch.float64)
lr_decay_steps = torch.tensor(5, dtype=torch.float64)
gamma = torch.pow(lr_decay, torch.div(1, lr_decay_steps))

scheduler = optim.lr_scheduler.ExponentialLR(optimizer, gamma=gamma, last_epoch=-1)
</code></pre>
<p>Does anybody have an idea of what could cause this strange behavior? I am happy to post any additional information about my model, if needed.</p>

Jan_Hula: <p>Hi <a class="mention" href="/u/priscacheese">@priscacheese</a>, have you managed to solve this issue in the end? I’m now trying to implement the NeuroSAT architecture in PyG. Would you be able to share the code and your experience?</p>

priscacheese: <p>Hi <a class="mention" href="/u/jan_hula">@Jan_Hula</a>, I’ve never managed to solve this issue, unfortunately, and I ended up giving up working on it. Did you encounter the same problem?<br>
Nevertheless, I can share my code on GitHub with you if you think it could be helpful.</p>

Jan_Hula: <p><a class="mention" href="/u/priscacheese">@priscacheese</a> I’m not sure whether it was the same problem, but it seems to be…The accuracy gets stuck at 0.5…Sharing your code would be great, I could try to play with it for a while and let you know if I find something… my GH name is Jan21</p>

priscacheese: <p>Alright, that sounds good to me! I’m going to give you access to my private repository</p>

Jan_Hula: <p>Thanks a lot!..Will let you know.</p>