Strange behaviour in Pytorch Net

username: Jona
cooked: <p>Hi,</p>
<p>I’ve been trying to create my first NN with Pytorch. I want to model a simple univariate Linear Regression.</p>
<p>However, my results are strange, if not saying erronous or weirds. This is the notebook (feel free to modify it)</p>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://console.paperspace.com/images/favicon-32x32.png" class="site-icon" width="32" height="32">
      <a href="https://console.paperspace.com/te93poq4q/notebook/pr5v5opas" target="_blank" rel="nofollow noopener">console.paperspace.com</a>
  </header>
  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/433;"><img src="https://blog.paperspace.com/content/images/size/w2000/2020/05/5eb8ce053cb50746e44069c0_cluster-bg--1-.jpg" class="thumbnail" width="690" height="433"></div>

<h3><a href="https://console.paperspace.com/te93poq4q/notebook/pr5v5opas" target="_blank" rel="nofollow noopener">Paperspace</a></h3>

<p>Cloud Machine Learning, AI, and effortless GPU infrastructure</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>The NB’s name is <strong>(Pytorch NN) Linear Regression</strong></p>
<p>I watched Janani Ravi’s introductory courses about Pytorch (at the same time with fast AI v3 course)</p>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://www.pluralsight.com/etc/clientlibs/pluralsight/main/images/favicons/favicon-32x32.png" class="site-icon" width="32" height="32">
      <a href="https://www.pluralsight.com/courses/pytorch-building-deep-learning-models" target="_blank" rel="nofollow noopener">pluralsight.com</a>
  </header>
  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/388;"><img src="https://pluralsight.imgix.net/course-images/pytorch-building-deep-learning-models-v1.png" class="thumbnail" width="690" height="388"></div>

<h3><a href="https://www.pluralsight.com/courses/pytorch-building-deep-learning-models" target="_blank" rel="nofollow noopener">Building Deep Learning Models Using PyTorch</a></h3>

<p>PyTorch is an open source deep learning framework originally developed by the AI teams at Facebook. PyTorch offers high-level APIs which make it easy to build neural networks and great support for distributed training and prediction.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://www.pluralsight.com/etc/clientlibs/pluralsight/main/images/favicons/favicon-32x32.png" class="site-icon" width="32" height="32">
      <a href="https://www.pluralsight.com/courses/building-your-first-pytorch-solution" target="_blank" rel="nofollow noopener">pluralsight.com</a>
  </header>
  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/388;"><img src="https://pluralsight.imgix.net/course-images/building-your-first-pytorch-solution-v1.jpg" class="thumbnail" width="690" height="388"></div>

<h3><a href="https://www.pluralsight.com/courses/building-your-first-pytorch-solution" target="_blank" rel="nofollow noopener">Building Your First PyTorch Solution</a></h3>

<p>This course covers the important practical aspects of installing PyTorch from scratch on a variety of different platforms and getting going with classification and regression models.&nbsp;</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>In both courses, she uses almost the same net design and hyperparams as I use, but my result are … I don’t know how to describe them <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"></p>
<p>Pls, help me figure out what is going on here</p>
<p>Best regards</p>
<p>Jonathan</p>

username: ptrblck
cooked: <p>I would recommend to scale down the problem a bit and try to overfit a small data snippet (e.g. just 10 samples) to make sure your training code doesn’t have any obvious errors.<br>
If that’s not working out of the box, you could play around with some hyperparameters.</p>
<p>Let us know, if you get stuck. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: Jona
cooked: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> … I’ll do it.</p>
<p>Here I have my NB’s updated and working all of them. What I realized, is that when I try a higher LR, my seems to stop learning (the plots show that situation)</p>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://console.paperspace.com/images/favicon-32x32.png" class="site-icon" width="16" height="16">
      <a href="https://console.paperspace.com/tegoxhoz0/notebook/pryj29evg" target="_blank" rel="nofollow noopener">console.paperspace.com</a>
  </header>
  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:116/73;"><img src="https://blog.paperspace.com/content/images/size/w2000/2020/05/5eb8ce053cb50746e44069c0_cluster-bg--1-.jpg" class="thumbnail" width="116" height="73"></div>

<h3><a href="https://console.paperspace.com/tegoxhoz0/notebook/pryj29evg" target="_blank" rel="nofollow noopener">Paperspace</a></h3>

<p>Cloud Machine Learning, AI, and effortless GPU infrastructure</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Do you have any resource where I can learn from the following topics?</p>
<ol>
<li>How many layer and neurons to use in certain context?</li>
<li>Which activaton function should I use en each layer?</li>
<li>What is the rationale of using certain activation function on a layer?</li>
</ol>
<p>Basically, I want to learn how to desing a NN.</p>
<p>Best regard my friend</p>

username: ptrblck
cooked: <p>To learn more about these points you could:</p>
<ul>
<li>join a course, such as <a href="https://course.fast.ai/">FastAI - Practical Deep Learning for Coders</a>, which would walk you through the best practices (CC <a class="mention" href="/u/jphoward">@jphoward</a> for more information)</li>
<li>track the current literature and see what kind of architectures are used (<a href="https://paperswithcode.com/">paperswithcode.com</a> might be a good place to have a look)</li>
<li>participate in competitions such as <a href="https://www.drivendata.org/">DrivenData</a> or Kaggle and try out your ideas <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:">
</li>
</ul>