Save and load models

snip3r77: <aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/udacity/deep-learning-v2-pytorch/blob/master/intro-to-pytorch/Part%206%20-%20Saving%20and%20Loading%20Models.ipynb" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/udacity/deep-learning-v2-pytorch/blob/master/intro-to-pytorch/Part%206%20-%20Saving%20and%20Loading%20Models.ipynb" target="_blank" rel="nofollow noopener">udacity/deep-learning-v2-pytorch/blob/master/intro-to-pytorch/Part 6 - Saving and Loading Models.ipynb</a></h4>
<pre><code class="lang-ipynb">{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Saving and Loading Models\n",
    "\n",
    "In this notebook, I'll show you how to save and load models with PyTorch. This is important because you'll often want to load previously trained models to use in making predictions or to continue training on new data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "%config InlineBackend.figure_format = 'retina'\n",
    "\n",
</code></pre>

  This file has been truncated. <a href="https://github.com/udacity/deep-learning-v2-pytorch/blob/master/intro-to-pytorch/Part%206%20-%20Saving%20and%20Loading%20Models.ipynb" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Question</p>
<ol>
<li>
<p>So when we save the model and if we decided to tweak the hidden layers, we can just adjust the hidden layers while using the weights from model.state_dict() ? ( the bunch of codes towards the end )</p>
</li>
<li>
<p>Do we usually use this utility script to create a new NN config?<br>
<a href="https://github.com/udacity/deep-learning-v2-pytorch/blob/master/intro-to-pytorch/helper.py" rel="nofollow noopener">https://github.com/udacity/deep-learning-v2-pytorch/blob/master/intro-to-pytorch/helper.py</a></p>
</li>
</ol>
<p>Thanks</p>

ptrblck: <ol>
<li>
<p>The model parameters should match the stored parameters in the <code>state_dict</code>. Otherwise you’ll get a size mismatch error. You could store the original shapes in the checkpoint, if you think it’s convenient.</p>
</li>
<li>
<p>You could adapt the code to your use case, but note that the code hasn’t been updated in a while and still uses <code>Variable</code>s, which were deprecated in PyTorch <code>0.4</code>.</p>
</li>
</ol>

snip3r77: <pre><code class="lang-auto">def load_checkpoint(filepath):
    checkpoint = torch.load(filepath)
    model = fc_model.Network(checkpoint['input_size'],
                             checkpoint['output_size'],
                             checkpoint['hidden_layers'])
    model.load_state_dict(checkpoint['state_dict'])
    
    return model
</code></pre>
<p>Thank you for the reply.</p>
<p>So basically , we can re-use the weights but we need to change the architecture based on this function</p>
<p>What’s the difference if we re-train again when we define the network? In what way we’re re-using<br>
<code>checkpoint['state_dict']</code> and how it differs with transfer learning?</p>
<p>side track,</p>
<p>I’ve been going from coursera deeplearning.ai , fast.ai and now udacity’s pytorch. It seems like I prefer pytorch ( or fast.ai  ) and I love the community here.</p>
<p>My question is how do I get used to the syntax<br>
( i.e turn off gradient and dropout during inferencing ).<br>
Thank you for the reply.</p>
<p>What I’m trying to say is there is a lot of steps that are consistent I think, so how does one remember these steps OR usually we do not need to remember ( just keep it in your notepad) and understand what it is trying to do. Because DL is definitely not as straight forward as sklearn’s .fit and .predict.</p>
<p>Thank you.</p>

ptrblck: <aside class="quote no-group" data-username="snip3r77" data-post="3" data-topic="76612">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/65b543/48.png" class="avatar"> snip3r77:</div>
<blockquote>
<p>So basically , we can re-use the weights but we need to change the architecture based on this function</p>
</blockquote>
</aside>
<p>You shouldn’t change the architecture, but recreate the model using the same arguments as during the initial training.</p>
<aside class="quote no-group" data-username="snip3r77" data-post="3" data-topic="76612">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/65b543/48.png" class="avatar"> snip3r77:</div>
<blockquote>
<p>What’s the difference if we re-train again when we define the network? In what way we’re re-using<br>
<code>checkpoint['state_dict']</code> and how it differs with transfer learning?</p>
</blockquote>
</aside>
<p>Loading the <code>state_dict</code> is used to load all previously stored (and trained) parameters and buffers into the model. It’s the first step when you want to fine tune the model (transfer learning).</p>
<aside class="quote no-group quote-modified" data-username="snip3r77" data-post="3" data-topic="76612">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/65b543/48.png" class="avatar"> snip3r77:</div>
<blockquote>
<p>I’ve been going from coursera deeplearning.ai , fast.ai and now udacity’s pytorch. It seems like I prefer pytorch ( or fast.ai - abstract short codes ) and I love the community here.</p>
<p>My question is how do I get used to the syntax<br>
( i.e turn off gradient and dropout during inferencing ).<br>
Thank you for the reply.</p>
<p>What I’m trying to say is there is a lot of steps that are consistent I think, so how does one remember these steps OR usually we do not need to remember ( just keep it in your notepad) and understand what it is trying to do. Because DL is definitely not as straight forward as sklearn’s .fit and .predict.</p>
</blockquote>
</aside>
<p>That’s great to hear!</p>
<p>Once you’ve created a training, validation and testing loop, you might realize that there are not that many things to remember.<br>
Some common issues are, if you forget to call <code>optimizer.zero_grad()</code> (which still happens to me after some time using PyTorch <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"> ), or to forget to call <code>model.train()/.eval()</code>.<br>
That being said, I’m not sure if there are any gotchas, but let us know, if you think I’m missing something.</p>

snip3r77: <p>I’ve now reached ‘transfer learning’</p>
<p>It seems like the code is even easier as compared to training from scratch.</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/udacity/deep-learning-v2-pytorch/blob/master/intro-to-pytorch/Part%208%20-%20Transfer%20Learning%20(Solution).ipynb" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/udacity/deep-learning-v2-pytorch/blob/master/intro-to-pytorch/Part%208%20-%20Transfer%20Learning%20(Solution).ipynb" target="_blank" rel="nofollow noopener">udacity/deep-learning-v2-pytorch/blob/master/intro-to-pytorch/Part 8 - Transfer Learning (Solution).ipynb</a></h4>
<pre><code class="lang-ipynb">{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Transfer Learning\n",
    "\n",
    "In this notebook, you'll learn how to use pre-trained networks to solved challenging problems in computer vision. Specifically, you'll use networks trained on [ImageNet](http://www.image-net.org/) [available from torchvision](http://pytorch.org/docs/0.3.0/torchvision/models.html). \n",
    "\n",
    "ImageNet is a massive dataset with over 1 million labeled images in 1000 categories. It's used to train deep neural networks using an architecture called convolutional layers. I'm not going to get into the details of convolutional networks here, but if you want to learn more about them, please [watch this](https://www.youtube.com/watch?v=2-Ol7ZB0MmU).\n",
    "\n",
    "Once trained, these models work astonishingly well as feature detectors for images they weren't trained on. Using a pre-trained network on images not in the training set is called transfer learning. Here we'll use transfer learning to train a network that can classify our cat and dog photos with near perfect accuracy.\n",
    "\n",
    "With `torchvision.models` you can download these pre-trained networks and use them in your applications. We'll include `models` in our imports now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
</code></pre>

  This file has been truncated. <a href="https://github.com/udacity/deep-learning-v2-pytorch/blob/master/intro-to-pytorch/Part%208%20-%20Transfer%20Learning%20(Solution).ipynb" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Some questions</p>
<ol>
<li>My batch size is 64 and if I print every 5 steps.<br>
With the following 4 steps below, does it mean I have already gone through 4x5steps = 20 batches? Or 20 x 64 = 1280 images?</li>
</ol>
<pre><code class="lang-auto">Epoch 1/1..Train loss: 2.411.. Test loss: 0.593.. Test accuracy: 0.702
Epoch 1/1..Train loss: 1.049.. Test loss: 0.317.. Test accuracy: 0.873
Epoch 1/1..Train loss: 0.433.. Test loss: 0.191.. Test accuracy: 0.914
Epoch 1/1..Train loss: 0.302.. Test loss: 0.105.. Test accuracy: 0.963
</code></pre>
<ol start="2">
<li>
<p>I think I ran the full 1 epoch and the improvement is 1 or 2%. Do you usually suggest early stopping?</p>
</li>
<li>
<p>If I run this using fast.ai , will it will be significantly faster using the LR-finder ? Is there a similar one for PyTorch? I think Pytorch Lightning already released recently.</p>
</li>
<li>
<p>So usually how does one tweak the hyperparameter for transfer learning assuming I only obtain 75% instead of 96%?</p>
</li>
</ol>
<p>Thanks</p>

ptrblck: <ol>
<li>
<p>Each iteration uses the specified batch size, so your calculation should be correct.</p>
</li>
<li>
<p>The accuracy improvement in the first iteration seems to come close to 26% in the first steps or are you referring to a later epoch? If you don’t see much improvement, I would suggest to stop the training.</p>
</li>
<li>
<p>This might be the case. FastAI uses a lot of utility functions and good practices to achieve really good results in less training time. I’m not sure which higher-level library has implemented the lr finder.</p>
</li>
<li>
<p>I would recommend to take a look at some FastAI lectures, as <a class="mention" href="/u/jphoward">@jphoward</a> explains a lot of techniques to properly fine tune a model. Also, you could use a Kaggle competition to try our some techniques and learn from others during the competition.</p>
</li>
</ol>

snip3r77: <p>Hi,</p>
<p>I’m just wondering if SubsetRandomSampler include <em>stratify</em> whereby % of classes are consistent in train and valid.</p>
<p>Last but not least, is this the usual way to split 80/20 between train and test(codewise) or is there a train-test-split( apologies as I’m looking from the sklearn angle)? Thanks</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/5/f50bbba0179fd2ebd02badd84d401cf87284e8ad.png" data-download-href="https://discuss.pytorch.org/uploads/default/f50bbba0179fd2ebd02badd84d401cf87284e8ad" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/5/f50bbba0179fd2ebd02badd84d401cf87284e8ad.png" alt="image" data-base62-sha1="yXM7jez5xkK91swj8neso1Vei5L" width="475" height="500" data-dominant-color="EDEEEE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">638×671 23.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <aside class="quote no-group" data-username="snip3r77" data-post="7" data-topic="76612">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/65b543/48.png" class="avatar"> snip3r77:</div>
<blockquote>
<p>I’m just wondering if SubsetRandomSampler include <em>stratify</em> whereby % of classes are consistent in train and valid.</p>
</blockquote>
</aside>
<p>No, <code>SubsetRandomSampler</code> will just randomly shuffle the passed indices.</p>
<aside class="quote no-group" data-username="snip3r77" data-post="7" data-topic="76612">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/65b543/48.png" class="avatar"> snip3r77:</div>
<blockquote>
<p>Last but not least, is this the usual way to split 80/20 between train and test(codewise) or is there a train-test-split( apologies as I’m looking from the sklearn angle)?</p>
</blockquote>
</aside>
<p>This would be a valid way, but I’m often using <code>sklearn</code> methods, in case I need to create stratified splits etc. If you are OK with adding <code>sklearn</code> as a dependency to your script, I would just use these methods.</p>

snip3r77: <p>so OpenCV is like an utility library to assist PyTorch in manipulating image?<br>
I’m sure there is a few of such library. Will it be ok if I just still to one or is there a better library than OpenCV?</p>
<p>At the kernel level , when we apply a -ve to the pixel it will turn it to white(1) if it’s dark (255) and vice-versa ?</p>
<p>Thanks.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/e/ced1fed587040158be3604865ccf1241db0f6edd.png" data-download-href="https://discuss.pytorch.org/uploads/default/ced1fed587040158be3604865ccf1241db0f6edd" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/e/ced1fed587040158be3604865ccf1241db0f6edd_2_454x499.png" alt="image" data-base62-sha1="tvCgRSfEeXvmDKwG9yyS5TNr3C5" width="454" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/e/ced1fed587040158be3604865ccf1241db0f6edd_2_454x499.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/e/ced1fed587040158be3604865ccf1241db0f6edd_2_681x748.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/e/ced1fed587040158be3604865ccf1241db0f6edd.png 2x" data-dominant-color="DCDCDC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">762×838 105 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <aside class="quote no-group" data-username="snip3r77" data-post="9" data-topic="76612">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/65b543/48.png" class="avatar"> snip3r77:</div>
<blockquote>
<p>Will it be ok if I just still to one or is there a better library than OpenCV?</p>
</blockquote>
</aside>
<p>You can use every library you like during data loading and processing, as long as you don’t need to use this operation inside the model and need Autograd to calculate the backward pass through it.<br>
<code>torchvision</code> uses <code>PIL</code> as the default image library, but you can of course use OpenCV.<br>
Note that the color format of OpenCV is BGR, while <code>PIL</code> and thus all pretrained models in <code>torchvision</code> use RGB, so you should transform the color channels if you want to feed these images to a pretrained model.</p>
<aside class="quote no-group" data-username="snip3r77" data-post="9" data-topic="76612">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/65b543/48.png" class="avatar"> snip3r77:</div>
<blockquote>
<p>At the kernel level , when we apply a -ve to the pixel it will turn it to white(1) if it’s dark (255) and vice-versa ?</p>
</blockquote>
</aside>
<p>A <code>-1</code> will invert the pixel values, so dark colors in a grayscale image should get bright and vice versa, yes.</p>

snip3r77: <p>So basically we’re doing an <code>.unsqueeze(1)</code> is because we wanted to match the shape. It’s grayscale hence we need to add the channel at <code>axis =1</code> ?</p>
<p>BTW, I’m doing Udacity’s Pytorch course I think I’m now starting to feel uncomfortable when we reach CNN because there’s not much practice.</p>
<p>Also, usually when I code in Python, for things that I do not understand I can break it down piece by piece to debug and understand and for Pytorch I have ‘debugging skills’ issue. The codes are bundled and I can’t run step by step. Is there a tutorial on this?</p>
<p>Appreciate your help. Thanks.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/6/86db8662053c5532e1157236b3a729e5f51813b1.png" data-download-href="https://discuss.pytorch.org/uploads/default/86db8662053c5532e1157236b3a729e5f51813b1" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/6/86db8662053c5532e1157236b3a729e5f51813b1_2_356x499.png" alt="image" data-base62-sha1="jf0jhDCQpGXc39V8rey50Ywu4iR" width="356" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/6/86db8662053c5532e1157236b3a729e5f51813b1_2_356x499.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/6/86db8662053c5532e1157236b3a729e5f51813b1_2_534x748.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/6/86db8662053c5532e1157236b3a729e5f51813b1_2_712x998.png 2x" data-dominant-color="F6F7F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">805×1128 39.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>More <a href="https://github.com/udacity/deep-learning-v2-pytorch/blob/master/convolutional-neural-networks/conv-visualization/conv_visualization.ipynb" rel="noopener nofollow ugc">details</a></p>

ptrblck: <aside class="quote no-group" data-username="snip3r77" data-post="11" data-topic="76612">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/65b543/48.png" class="avatar"> snip3r77:</div>
<blockquote>
<p>So basically we’re doing an <code>.unsqueeze(1)</code> is because we wanted to match the shape. It’s grayscale hence we need to add the channel at <code>axis =1</code> ?</p>
</blockquote>
</aside>
<p>It depends, what you want to do with this tensor.<br>
If you want to pass it to a 2D CNN, then you would have to provide an input shape of <code>[batch_size, channels, height, width]</code>. Since the channel dimension might be missing, you would have to unsqueeze it.</p>
<aside class="quote no-group" data-username="snip3r77" data-post="11" data-topic="76612">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/65b543/48.png" class="avatar"> snip3r77:</div>
<blockquote>
<p>The codes are bundled and I can’t run step by step. Is there a tutorial on this?</p>
</blockquote>
</aside>
<p>Could you run the cells separately for debugging purposes?<br>
I’m not familiar with the course, but where are you stuck?</p>

snip3r77: <p>I have not start coding yet, I will bring this up when start coding CNN. Thanks</p>
<p>CNN Question</p>
<pre><code class="lang-auto">#Input is 130 x 130 x RGB
nn.Conv2d(3, 10, 3) &lt;- line 1
nn.MaxPool2d(4, 4) &lt;- line 2 
nn.Conv2d(10, 20, 5, padding=2) &lt;- line 3
nn.MaxPool2d(2, 2) &lt;- line 4
</code></pre>
<p>so basically at line 1 the input depth is 3 and the output is 10<br>
and we use this formula</p>
<p><code>( input_depth + 2*pad - kernel )/ stride + 1</code></p>
<p>to determine the image size</p>
<p>example 130 - 3 / 1 + 1 = 128</p>
<p>From what I read the depth will be increasing ( from 3 to 20 ) and the image size will reduce ( divide by 4 and 2 )<br>
Image will reduce due to MaxPool downscaling.</p>
<p>So when we see  <code>nn.MaxPool2d(4, 4)</code> , the image will be divided by 4 . Is there a chance that the tuple of (4,4) will be a different value ?</p>
<p>So at line 2 will shrink by <code>factor of 4</code> and at line 4 will <code>shrink by a factor 2</code> .</p>
<p>Side tack<br>
Also, if we want to maintain the image size with known kernel size and use padding to compensate we just use the following formula right?</p>
<p><code>( input_depth + 2*pad - kernel )/ stride + 1 = input_depth </code> then re-arrange to find pad unknown</p>

ptrblck: <p>The output shape formula uses the input height and width, not the depth.<br>
Also some minor errors are in the formula, i.e. the <code>dilation</code> is missing as well as some constant factors.<br>
Have a look at the <a href="https://pytorch.org/docs/stable/nn.html?highlight=conv2d#torch.nn.Conv2d">docs</a> for the complete formula.</p>
<aside class="quote no-group" data-username="snip3r77" data-post="13" data-topic="76612">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/65b543/48.png" class="avatar"> snip3r77:</div>
<blockquote>
<p>so basically at line 1 the input depth is 3 and the output is 10</p>
</blockquote>
</aside>
<p>Yes, this conv uses 3 input channels and 10 filters, which create one output channel each.</p>
<aside class="quote no-group" data-username="snip3r77" data-post="13" data-topic="76612">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/65b543/48.png" class="avatar"> snip3r77:</div>
<blockquote>
<p>From what I read the depth will be increasing ( from 3 to 20 ) and the image size will reduce ( divide by 4 and 2 )<br>
Image will reduce due to MaxPool downscaling.</p>
</blockquote>
</aside>
<p>Yes, that would be the case.</p>
<aside class="quote no-group" data-username="snip3r77" data-post="13" data-topic="76612">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/65b543/48.png" class="avatar"> snip3r77:</div>
<blockquote>
<p>So when we see <code>nn.MaxPool2d(4, 4)</code> , the image will be divided by 4 . Is there a chance that the tuple of (4,4) will be a different value ?</p>
</blockquote>
</aside>
<p>I don’t understand this question. You could pass non-square kernels, e.g. <code>(4, 2)</code>, if you want.</p>
<aside class="quote no-group" data-username="snip3r77" data-post="13" data-topic="76612">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/65b543/48.png" class="avatar"> snip3r77:</div>
<blockquote>
<p>Also, if we want to maintain the image size with known kernel size and use padding to compensate we just use the following formula right?</p>
</blockquote>
</aside>
<p>Yeah, but with the formula from the docs.</p>

snip3r77: <p>Thanks for the prompt reply.</p>
<p>I got this</p>
<p><code>( input_depth + 2*pad - kernel )/ stride + 1 = input_depth</code></p>
<p>when I was going through some of Andrew Ng’s video.<br>
I think he is trying to teach us the intuition but I guess there are more parameters like<br>
<code>dilation</code> that I’m not exposed  to.</p>
<aside class="quote no-group" data-username="ptrblck" data-post="14" data-topic="76612">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>I don’t understand this question. You could pass non-square kernels, e.g. <code>(4, 2)</code> , if you want.</p>
</blockquote>
</aside>
<p>I understood that (4,4) or (2,2)  is reducing by a factor of 4 or 2 ?<br>
Is it uncommon to use non square kernels?</p>
<p>So do we usually hand calculate these outputs or do we leave it for Pytorch to do so? Or do we need to understand some of these things under hood but let Pytorch to do the heavy lifting.</p>
<p>Cheers.</p>

ptrblck: <aside class="quote no-group" data-username="snip3r77" data-post="15" data-topic="76612">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/65b543/48.png" class="avatar"> snip3r77:</div>
<blockquote>
<p>I understood that (4,4) or (2,2) is reducing by a factor of 4 or 2 ?<br>
Is it uncommon to use non square kernels?</p>
</blockquote>
</aside>
<p>Yes, the spatial reduction would be 4x and 2x for these kernel shapes and strides.<br>
It might be uncommon to use non-square kernels, but this doesn’t mean that you shouldn’t use them in your model, if you think they make sense for whatever reason.</p>
<aside class="quote no-group" data-username="snip3r77" data-post="15" data-topic="76612">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/65b543/48.png" class="avatar"> snip3r77:</div>
<blockquote>
<p>So do we usually hand calculate these outputs or do we leave it for Pytorch to do so? Or do we need to understand some of these things under hood but let Pytorch to do the heavy lifting.</p>
</blockquote>
</aside>
<p>You could calculate it manually, but for complicated use cases with an uncommon stride, dilation etc., I just print the output shape and adapt the parameters to match my desired output.</p>

snip3r77: <p>Questions</p>
<ol>
<li>I manually calculated all the conv2d values<br>
and also the FC1 and FC2 dimensions. Is there a way that it can be calculated by pytorch?</li>
</ol>
<p>**Coincidentally, here with the use of padding 1, input equates to output before Pooling, hence<br>
it’s slightly easier over here.</p>
<ol start="2">
<li>
<p>At line 41, is -1 referring to the batch size since we converted the rest to 4W * 4H * 64 final depth ?</p>
</li>
<li>
<p>I understand that I did a resnet transfer learning before and it can achieve &gt;90% but when I use CNN, I only managed to obtain ~70% based on this network. Conceptually would it be better to just use transfer learning instead OR this is just an ‘entry level’ CNN that will lead to a much better one that may or may not be better than transfer learning?</p>
</li>
</ol>
<p>Thanks.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/e/6e556b4244023cd87b16cf2804a23c86c2dd465d.png" data-download-href="https://discuss.pytorch.org/uploads/default/6e556b4244023cd87b16cf2804a23c86c2dd465d" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/e/6e556b4244023cd87b16cf2804a23c86c2dd465d_2_319x500.png" alt="image" data-base62-sha1="fK3wVqNjoJjnbQu4QHXBCHPe23X" width="319" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/e/6e556b4244023cd87b16cf2804a23c86c2dd465d_2_319x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/e/6e556b4244023cd87b16cf2804a23c86c2dd465d_2_478x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/e/6e556b4244023cd87b16cf2804a23c86c2dd465d_2_638x1000.png 2x" data-dominant-color="F0F1F0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">730×1144 60 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <ol>
<li>
<p>There might be utility functions in the wild, but I would recommend to use a dummy input tensor, print all necessary activation shapes, and set the input features of linear layers in this way.<br>
Since each forward pass is dynamic in PyTorch and the computation graph will be created during the forward pass, the shape inference is not straightforward to calculate in eager mode.</p>
</li>
<li>
<p>Yes, the activation will be flattened to <code>[batch_size, 4*4*64]</code>. I would generally recommend to use <code>x.view(x.size(0), -1)</code> to keep the batch size and run into shape mismatch errors instead of changing the batch size, which would be harder to debug.</p>
</li>
<li>
<p>That might be expected, as a pretrained ResNet is quite powerful for a variety of tasks.</p>
</li>
</ol>

snip3r77: <p>Also I realise that if my input is RGB the input/yellow needs to be 3 and if it’s grayscale it needs to be 1. Slowly trying to provide intuition.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/6/76f56672f457a8aefdf57f31cf0fcec2f3809c60.png" alt="image" data-base62-sha1="gYm6KZq0gHv3KFZhYLXq1RoQU8g" width="648" height="161"></p>
<aside class="quote no-group" data-username="ptrblck" data-post="19" data-topic="76612">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>There might be utility functions in the wild, but I would recommend to use a dummy input tensor, print all necessary activation shapes, and set the input features of linear layers in this way.<br>
Since each forward pass is dynamic in PyTorch and the computation graph will be created during the forward pass, the shape inference is not straightforward to calculate in eager mode.</p>
</blockquote>
</aside>
<p>I don’t quite understand how to do this. As mentioned, I think this is the portion that I’m kinda lost ( I do not know how to debug/go to manual mode ). Is there any tutorial that explain on this?</p>
<p>But I did google and manage to find from <a href="https://pytorch.org/tutorials/beginner/blitz/neural_networks_tutorial.html" rel="noopener nofollow ugc">Pytorch 60mins blitz from the documentation</a>.</p>
<p>which I adapted to my code <code>[51]</code> . but I’m not sure how to proceed from here.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/f/df28940083d819008eb89ce8fa57cdea204af0e9.png" data-download-href="https://discuss.pytorch.org/uploads/default/df28940083d819008eb89ce8fa57cdea204af0e9" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/f/df28940083d819008eb89ce8fa57cdea204af0e9.png" alt="image" data-base62-sha1="vQ9pG1E4P4ucFOhLh4lwShpBqIh" width="621" height="500" data-dominant-color="F7F7F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">759×611 16.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Also, I find this is pretty neat and with this I do not need to calculate the FC.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/4/549f9358c7908c3069c9022258e5c14ffa37e0f6.png" data-download-href="https://discuss.pytorch.org/uploads/default/549f9358c7908c3069c9022258e5c14ffa37e0f6" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/4/549f9358c7908c3069c9022258e5c14ffa37e0f6.png" alt="image" data-base62-sha1="c4BZMXlNENjO8QYNm4EubRh9MKG" width="524" height="500" data-dominant-color="EDEEEB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">672×641 35.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Thanks</p>

ptrblck: <aside class="quote no-group" data-username="snip3r77" data-post="20" data-topic="76612">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/65b543/48.png" class="avatar"> snip3r77:</div>
<blockquote>
<p>I don’t quite understand how to do this.</p>
</blockquote>
</aside>
<p>You would add <code>print(x.shape)</code> statements into the <code>forward</code> method to all places, where you don’t know the activation shape or don’t want to calculate it manually for the current input shape.</p>
<p>The <code>num_flatten_features</code> don’t avoid calculating the <code>in_features</code> as <code>16*6*6</code>, but help you to flatten the tensor. I would still recommend to use <code>x.view(x.size(0), -1)</code>.</p>