How to improve my AI model (numbers layers/neurons and layers' type)

username: VictorVidigal
cooked: <p>I’m studying AI in practice using pytorch, so I would like some tips to improve the model I created.</p>
<p>The model’s input/target follow the role:</p>
<pre data-code-wrap="python"><code class="lang-python">if input[0] + input[1] &gt;= 0:
  target = 10
else:
  target = -10
</code></pre>
<pre data-code-wrap="python"><code class="lang-python">inputs = torch.rand(1000000, 2).sub(.5).mul(200)

# tensor([[ 25.8606, -40.9372],
#        [ 79.3800,  44.6780],
#        [ 76.0107, -77.0862],
#        [-21.3406, -27.1793],
#        [ 71.2402, -81.8930],
#        [ 37.5089,  73.1345],
#        [ 73.6303,  69.5081],
#        [-52.1048, -70.0359],
#        [ 45.5813, -86.4757],
#        [ 96.3207,   5.4885]])
</code></pre>
<pre data-code-wrap="python"><code class="lang-python">targets = (inputs.sum(1, keepdim=True) &gt; 10).float().sub(.5).mul(20)

# tensor([[-10.],
#        [ 10.],
#        [-10.],
#        [-10.],
#        [-10.],
#        [ 10.],
#        [ 10.],
#        [-10.],
#        [-10.],
#        [ 10.]])
</code></pre>
<p>The model</p>
<pre data-code-wrap="python"><code class="lang-python">import os
from pprint import pprint

import torch
from torch import nn


class Model2D(nn.Module):
    CHECKPOINT_DIR = './checkpoints'

    def __init__(self):
        super().__init__()
        self.layer_1 = nn.Linear(in_features=2, out_features=4)
        self.layer_2 = nn.Linear(in_features=4, out_features=2)
        self.layer_3 = nn.Linear(in_features=2, out_features=1)

    def forward(self, x: torch.Tensor) -&gt; torch.Tensor:
        return self.layer_3(self.layer_2(self.layer_1(x)))

    def save(self):
        torch.save(obj=self.state_dict(),
                   f=f"{self.CHECKPOINT_DIR}/Model2D.pth")

    def load(self):
        if (os.path.isfile(f"{self.CHECKPOINT_DIR}/Model2D.pth")):
            self.load_state_dict(torch.load(f"{self.CHECKPOINT_DIR}/Model2D.pth"))
            print("The checkpoints loaded with the following values for weights and bias: ")
            pprint(self.state_dict())

</code></pre>
<p>It is the Train loss and Test loss</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/3/53a7f0a11a61475d4034c855db07ea0af0634767.png" data-download-href="https://discuss.pytorch.org/uploads/default/53a7f0a11a61475d4034c855db07ea0af0634767" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/3/53a7f0a11a61475d4034c855db07ea0af0634767_2_690x340.png" alt="image" data-base62-sha1="bW3rgolG2iMjjH3qtqoK1chmokf" width="690" height="340" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/3/53a7f0a11a61475d4034c855db07ea0af0634767_2_690x340.png, https://discuss.pytorch.org/uploads/default/original/3X/5/3/53a7f0a11a61475d4034c855db07ea0af0634767.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/3/53a7f0a11a61475d4034c855db07ea0af0634767.png 2x" data-dominant-color="F9EFEB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">883×436 84.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The problem is that my model does not learn more. I think it is because the number/type of layers.</p>
<p>Somebody could help me with some tips to improve my model?</p>
<p>I shared the code on <a href="https://colab.research.google.com/drive/1scRvwgcsKWR-D4uzzUCJMiUy8pfdXyVu?usp=sharing" class="inline-onebox" rel="noopener nofollow ugc">Google Colab</a></p>

username: copythatpasta
cooked: <p>use non linear functions (activations) after your linear layers, torch.relu() could be a good start</p>
<p>def forward(self, x: torch.Tensor) → torch.Tensor:<br>
x =  torch.relu(self.layer_1(x))<br>
… rest of layers</p>

username: Eduardo_Lawson
cooked: <p>Maybe you can get better results if you use non-linearity. In this case use a activation function between the layers. I would suggest to use nn.ReLU()](<a href="https://pytorch.org/docs/stable/generated/torch.nn.ReLU.html" class="inline-onebox" rel="noopener nofollow ugc">ReLU — PyTorch 2.3 documentation</a>). My suggestion is to change for this:</p>
<pre><code class="lang-auto">def __init__(self):
        super().__init__()
        self.layer_1 = nn.Linear(in_features=2, out_features=4)
        self.relu = nn.ReLU()
        self.layer_2 = nn.Linear(in_features=4, out_features=2)
        self.relu = nn.ReLU()
        self.layer_3 = nn.Linear(in_features=2, out_features=1)
</code></pre>

username: VictorVidigal
cooked: <p>Thanks for the suggestions.<br>
I changed my model to include nn.ReLU activations.<br>
I also changed my labels from (-10 or 10) to (0 or 1)</p>
<p>Now, I’m having better inferences.</p>
<p>My model changed:</p>
<pre data-code-wrap="python"><code class="lang-python">    def __init__(self):
        super().__init__()
        self.layer_1 = nn.Linear(in_features=2, out_features=4)
        self.act_1 = nn.ReLU()
        self.layer_2 = nn.Linear(in_features=4, out_features=2)
        self.act_2 = nn.ReLU()
        self.layer_3 = nn.Linear(in_features=2, out_features=1)
        # self.act_3 = nn.Sigmoid()

    def forward(self, x: torch.Tensor) -&gt; torch.Tensor:
        x = self.act_1(self.layer_1(x))
        x = self.act_2(self.layer_2(x))
        # x = self.act_3(self.layer_3(x))
        x = self.layer_3(x)
        return x
</code></pre>
<p>I’m using nn.L1Loss() as loss functions. Do you suggest a more appropriate loss function for this kind of problem?</p>

username: Eduardo_Lawson
cooked: <p>I think it is a good loss function for the problem you are trying to solve. But if you want to make some experiments to check, i would give a try on the <a href="https://pytorch.org/docs/stable/generated/torch.nn.MSELoss.html" rel="noopener nofollow ugc">L2 loss or Mse Loss</a> as well.</p>