Non-linear regression with 2 inputs (Flat behavior)

username: Willian_Dimas_Pompeo
cooked: <p>Hello, everyone. I am programming a neural network to predict benzene composition in distillate stream (1 output) according to reflux and boilup ratio (2 inputs) but the predict values are in a flat while the real behavior is not linear. I tried to use just one input to verify the NN performance in this case and the data fitted well, however when I set one property (for example reflux ratio = 2 for all boilup ratios), I obtained the behavior of straight line, unlike the real non linear behavior. Anyone has already faced this? Below, there is the picture with the predictions (red) and real values (green) and the model of my neural network (code)… I will be very grateful if someone can help me…</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/b/2b353eddcc507f61cfb3cdb10cfa5bba25de046d.png" data-download-href="https://discuss.pytorch.org/uploads/default/2b353eddcc507f61cfb3cdb10cfa5bba25de046d" title="Screenshot 2023-12-12 171439"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/b/2b353eddcc507f61cfb3cdb10cfa5bba25de046d_2_542x500.png" alt="Screenshot 2023-12-12 171439" data-base62-sha1="6aeBMj09V2pyW2b2OC39y1HEMKV" width="542" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/b/2b353eddcc507f61cfb3cdb10cfa5bba25de046d_2_542x500.png, https://discuss.pytorch.org/uploads/default/original/3X/2/b/2b353eddcc507f61cfb3cdb10cfa5bba25de046d.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/2/b/2b353eddcc507f61cfb3cdb10cfa5bba25de046d.png 2x" data-dominant-color="C3A78C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2023-12-12 171439</span><span class="informations">687×633 176 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<hr>
<p>class Model1(nn.Module):<br>
def <strong>init</strong>(self):<br>
super().<strong>init</strong>()</p>
<pre><code>self.layer1 = nn.Linear(in_features=2,out_features=64)
self.layer2 = nn.Linear(in_features=64,out_features = 256)
self.layer3 = nn.Linear(in_features=256,out_features = 512)
self.layer4 = nn.Linear(in_features=512,out_features=1024)
self.layer5 = nn.Linear(in_features=1024,out_features=300)
self.layer6 = nn.Linear(in_features=300,out_features=1)
#self.activation_function = nn.ReLU()
</code></pre>
<p>def forward(self,x):</p>
<pre><code>x = self.layer1(x)
x = F.relu(x)
x = self.layer2(x)
x = F.relu(x)
x = self.layer3(x)
x = F.relu(x)
x = self.layer4(x)
x = F.relu(x)
x = self.layer5(x)
x = F.relu(x)
x = self.layer6(x)
</code></pre>
<hr>

username: SibYz
cooked: <aside class="onebox allowlistedgeneric" data-onebox-src="https://www.joeltok.com/posts/2020-04-pytorch-f-relu-to-nn-relu/">
  <header class="source">
      <img src="https://www.joeltok.com/img/emoji_people.svg" class="site-icon" width="48" height="48">

      <a href="https://www.joeltok.com/posts/2020-04-pytorch-f-relu-to-nn-relu/" target="_blank" rel="noopener nofollow ugc" title="12:00AM - 28 April 2020">joeltok.com – 28 Apr 20</a>
  </header>

  <article class="onebox-body">
    

<h3><a href="https://www.joeltok.com/posts/2020-04-pytorch-f-relu-to-nn-relu/" target="_blank" rel="noopener nofollow ugc">Converting F.relu() to nn.ReLU() in PyTorch</a></h3>

  <p>I have been using PyTorch extensively in some of my projects lately, and one of the things that has confused me was how to go about implementing a hidden layer of Rectified Linear Units (ReLU) using the nn.ReLU() syntax. I was already using the...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Checkout this link.<br>
I never used nn.functional on any model, i Usually use nn.ReLU()</p>
<p>Have you tried using just nn.ReLU()?</p>

username: Ashwath_S
cooked: <p>Could you please provide some metrics: How many epochs of training, loss curve trend, accuracy,  etc. It could be because the model still hasn’t done enough learning.</p>

username: Willian_Dimas_Pompeo
cooked: <p>Yes, I’ve tried nn.ReLU() but I obtained the same behavior…<br>
If it is better, here is my code: <a href="https://colab.research.google.com/drive/1DW7PZbdOxBetfcSd8lt5_ra6gMddxUsN?authuser=1#scrollTo=qnWW6sLkFYge" class="inline-onebox" rel="noopener nofollow ugc">Google Colab</a><br>
And here the data that I am using: <a href="https://docs.google.com/spreadsheets/d/1B1NZv5M6nX2B-6ke89QvQc8LoaL5hLfx/edit#gid=706732595" class="inline-onebox" rel="noopener nofollow ugc">Ex1_Cab.xlsx - Google Sheets</a><br>
(There are 4 models of NN but I am using just the Model1 and predict just the first output of the spreadsheet - Fracd_Bz)<br>
I will be very grateful, if you have some contribution</p>

username: Willian_Dimas_Pompeo
cooked: <p>I used 300 epoches and the loss curve trend is below… If it is better to understand what I did, here is my code: <a href="https://colab.research.google.com/drive/1DW7PZbdOxBetfcSd8lt5_ra6gMddxUsN?authuser=1#scrollTo=qnWW6sLkFYge" class="inline-onebox" rel="noopener nofollow ugc">Google Colab</a> and the data that I am using: <a href="https://docs.google.com/spreadsheets/d/1B1NZv5M6nX2B-6ke89QvQc8LoaL5hLfx/edit#gid=706732595" class="inline-onebox" rel="noopener nofollow ugc">Ex1_Cab.xlsx - Google Sheets</a>. I will be very grateful, if you have some contribution…</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/2/222462161e54a648510b3c49073bed85952be630.png" data-download-href="https://discuss.pytorch.org/uploads/default/222462161e54a648510b3c49073bed85952be630" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/2/222462161e54a648510b3c49073bed85952be630_2_624x500.png" alt="image" data-base62-sha1="4S2brlLRAofmFJPF9W18mjyb6lq" width="624" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/2/222462161e54a648510b3c49073bed85952be630_2_624x500.png, https://discuss.pytorch.org/uploads/default/original/3X/2/2/222462161e54a648510b3c49073bed85952be630.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/2/2/222462161e54a648510b3c49073bed85952be630.png 2x" data-dominant-color="FBFAFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">693×555 35.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: Ashwath_S
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/b/fb6eebbed26f6614a1fdcf8b9344fcfd214c812e.png" data-download-href="https://discuss.pytorch.org/uploads/default/fb6eebbed26f6614a1fdcf8b9344fcfd214c812e" title="30_epochs"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/b/fb6eebbed26f6614a1fdcf8b9344fcfd214c812e_2_491x500.png" alt="30_epochs" data-base62-sha1="zShuv8XEypUUARgxctycffpuEW2" width="491" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/b/fb6eebbed26f6614a1fdcf8b9344fcfd214c812e_2_491x500.png, https://discuss.pytorch.org/uploads/default/original/3X/f/b/fb6eebbed26f6614a1fdcf8b9344fcfd214c812e.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/b/fb6eebbed26f6614a1fdcf8b9344fcfd214c812e.png 2x" data-dominant-color="D8B6A6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">30_epochs</span><span class="informations">571×581 117 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I trained for 30 epochs and i got a more non-linear graph. I think this can just be an issue of overfitting. Your model stops learning by the 50th epoch.</p>
<p>Edit:</p>
<p>You seem to be giving your entire training set in one pass. Try breaking it into batches, and try a lighter model.</p>
<p>For example, i broke the dataset into 96 batches of size 69, and i trained with the same parameters you used and the curve is as follows:</p>
<p>Model:</p>
<pre><code>  class Model1(nn.Module):
      def __init__(self):
         super().__init__()

         self.layer1 = nn.Linear(in_features=2,out_features=64)
         self.layer2 = nn.Linear(in_features=64,out_features=1)
         self.activation_function = nn.ReLU()

      def forward(self,x):

         x = self.activation_function(self.layer1(x))
         x = self.layer2(x)
         return x
</code></pre>
<p>Epochs: 1</p>
<p>Training:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/b/8b63d3c3544b2a3fa71ace3f8df3affdc7285ff4.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/8b63d3c3544b2a3fa71ace3f8df3affdc7285ff4" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/b/8b63d3c3544b2a3fa71ace3f8df3affdc7285ff4_2_490x500.jpeg" alt="image" data-base62-sha1="jT6fka0sC41we8AGs5jEOQ1veYs" width="490" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/b/8b63d3c3544b2a3fa71ace3f8df3affdc7285ff4_2_490x500.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/8/b/8b63d3c3544b2a3fa71ace3f8df3affdc7285ff4.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/b/8b63d3c3544b2a3fa71ace3f8df3affdc7285ff4.jpeg 2x" data-dominant-color="E6ADA4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">570×581 58.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Test Samples:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/a/4ae76d7537c8acf8539f6202c921cb209c1a7bac.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/4ae76d7537c8acf8539f6202c921cb209c1a7bac" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/a/4ae76d7537c8acf8539f6202c921cb209c1a7bac_2_490x500.jpeg" alt="image" data-base62-sha1="aGDaaMpfsyqMvoe3pVvAczUkpfC" width="490" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/a/4ae76d7537c8acf8539f6202c921cb209c1a7bac_2_490x500.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/4/a/4ae76d7537c8acf8539f6202c921cb209c1a7bac.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/4/a/4ae76d7537c8acf8539f6202c921cb209c1a7bac.jpeg 2x" data-dominant-color="E0C8BD"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">570×581 125 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>