GNN does not predict/generalize well on test set

sk9292: <p>Hi guys,</p>
<p>I am not generally an expert on neural networks, but I am trying to learn.</p>
<p>I am dealing with 5-dimensional data; each feature has 2D and 3D parts. I stack them together and normalize subtracting the mean and dividing by the standard deviation. I am trying to solve a problem of binary classification. I used GNN to classify each graph, which should include these 5D features.</p>
<p>I tried a few representations of the graphs:</p>
<ul>
<li>Tree-like, a general root connects all the features (1 edge root to features, 1 edge feature to root)</li>
<li>Nearest Neighbor style, I connect the N features are closer in the 2d space; I tried N = 1, 5</li>
</ul>
<p>I am using training, validation, and test set. The validation is sliced from the training set, and the test set instead from different data, but it should be pretty similar in terms of distribution.</p>
<p>Losses and accuracies are “good” for training and validation; however, I usually have trends of increasing loss and stable (low) accuracy in the test set. So basically, my network overfits.</p>
<p>To start, I am using <a href="https://colab.research.google.com/drive/1I8a0DfQ3fI7Njc62__mVXUlcAleUclnb?usp=sharing#scrollTo=1tBMhOrq4JKw" rel="noopener nofollow ugc">this GNN</a> sample, which should be pretty basic, taken from torch_geometric tutorial. The model is the same. I use a batch_size of 128 and Adam with lr=1e-4</p>
<p>I have tried multiple things, but nothing seems to show substantial improvements:</p>
<ul>
<li>some data augmentation of the training set</li>
<li>try changing some hyperparameters, learning rate, optimizer, batch size</li>
<li>change values of noise in the training set</li>
</ul>
<p>Attached are some plots of losses and accuracies over epochs, a super small grid search:<br>
cyan) original model as in the link with 64 cells each<br>
magenta) just first two layers of original model with 64 cells each<br>
yellow) two layers with 32 cells each</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/7/578bb67f45ee800ca0c7aa7573de0092077f04d7.png" data-download-href="https://discuss.pytorch.org/uploads/default/578bb67f45ee800ca0c7aa7573de0092077f04d7" title="Screenshot from 2024-04-22 23-11-51"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/7/578bb67f45ee800ca0c7aa7573de0092077f04d7_2_690x493.png" alt="Screenshot from 2024-04-22 23-11-51" data-base62-sha1="cusS9jZP2abcs2MMJALsTVcVf8j" width="690" height="493" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/7/578bb67f45ee800ca0c7aa7573de0092077f04d7_2_690x493.png, https://discuss.pytorch.org/uploads/default/original/3X/5/7/578bb67f45ee800ca0c7aa7573de0092077f04d7.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/7/578bb67f45ee800ca0c7aa7573de0092077f04d7.png 2x" data-dominant-color="393838"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2024-04-22 23-11-51</span><span class="informations">811×580 137 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I am using just the first tree-like graph representation, If I connect nodes based on their “vicinity” my network predictions are horrible.</p>
<p>I don’t know what to do next.<br>
It is the data? It can be just a matter of grid search?</p>
<p>Probably I don’t have enough methodology. How would you start a deep learning project? How would you proceed? It is there a sort of best-practice guide that one can follow to make it work, isolating problems step-by-step?</p>
<p>Any help, suggestion, reference, will be very appreciated.</p>