Transfer Learning Feature Extraction confusion

brunoj: <p>Hello. So I have been doing transfer learning and followed a youtube tutorial but now when I look around to other documents and examples there is this thing very confusing and budging me. I used EfficientNet v1 B0 model and the default model when loaded as<br>
model = timm.create_model(CFG.model_name,pretrained=True)  and when i print model  the last few layers are as: (conv_head): Conv2d(320, 1280, kernel_size=(1, 1), stride=(1, 1), bias=False)<br>
(bn2): BatchNormAct2d(<br>
1280, eps=0.001, momentum=0.1, affine=True, track_running_stats=True<br>
(drop): Identity()<br>
(act): SiLU(inplace=True)<br>
)<br>
(global_pool): SelectAdaptivePool2d (pool_type=avg, flatten=Flatten(start_dim=1, end_dim=-1))<br>
(classifier): Linear(in_features=1280, out_features=1000, bias=True)<br>
)</p>
<p>Now I have 6 class problem and I want to change last layer features =1000 to 6.<br>
I did this by:</p>
<p>print(model.conv_stem)<br>
model.conv_stem = nn.Conv2d(1, 32, kernel_size=(3, 3), stride=(2, 2), bias=False)<br>
model.state_dict()[‘conv_stem.weight’] = model.state_dict()[‘conv_stem.weight’].sum(dim=1, keepdim=True)<br>
print(model.conv_stem)<br>
<span class="hashtag">#let</span>’s update the pretrained model:<br>
for param in model.parameters():<br>
param.requires_grad=False</p>
<p><span class="hashtag">#orginally</span>, it was:<br>
#(classifier): Linear(in_features=1280, out_features=1000, bias=True)<br>
model.classifier = nn.Linear(1280, 6)</p>
<p>But this is what I learned but the youtube tutorial says:<br>
model.classifier = nn.Sequential(<br>
nn.Linear(in_features=1792, out_features=625), <span class="hashtag">#1792</span> is the orginal in_features<br>
nn.ReLU(), <span class="hashtag">#ReLu</span> to be the activation function<br>
nn.Dropout(p=0.3),<br>
nn.Linear(in_features=625, out_features=256),<br>
nn.ReLU(),<br>
nn.Linear(in_features=256, out_features=6)<br>
)<br>
I wonder which is correct and what is feature extractor and what significance sequential thing has? And how one can calculate or decide out_features=625???</p>
<p>Which method is correct whats the pros and cons?</p>

thecho7: <p>Both correct.<br>
Here’s my opinion,</p>
<p>Single <code>nn.Linear</code> has less parameters compared to multiple <code>nn.Linear</code>.<br>
If the classification task is bit more complex(difficult), may consider multiple one.</p>
<p>Additionally, if the output dimension of feature extractor is quite bigger than the number of classes,<br>
often use multiple.</p>

brunoj: <p>alright so I’m doing ist method not the multiples one and got these plots for 10 epochs:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/b/bbfca1ca314dbf02c67584c3211c883649b839fd.png" data-download-href="https://discuss.pytorch.org/uploads/default/bbfca1ca314dbf02c67584c3211c883649b839fd" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/b/bbfca1ca314dbf02c67584c3211c883649b839fd_2_690x388.png" alt="image" data-base62-sha1="qP0y0vx9w2vMrSUdioSCmJMKzFj" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/b/bbfca1ca314dbf02c67584c3211c883649b839fd_2_690x388.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/b/bbfca1ca314dbf02c67584c3211c883649b839fd_2_1035x582.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/b/bbfca1ca314dbf02c67584c3211c883649b839fd_2_1380x776.png 2x" data-dominant-color="F1F1F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1920×1080 349 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/a/5a5e1aec8c0f31d822daac05fafcb15c6dda05f1.png" data-download-href="https://discuss.pytorch.org/uploads/default/5a5e1aec8c0f31d822daac05fafcb15c6dda05f1" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/a/5a5e1aec8c0f31d822daac05fafcb15c6dda05f1_2_690x388.png" alt="image" data-base62-sha1="cTqAGExKqkU4cglDmRXPndeD3j3" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/a/5a5e1aec8c0f31d822daac05fafcb15c6dda05f1_2_690x388.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/a/5a5e1aec8c0f31d822daac05fafcb15c6dda05f1_2_1035x582.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/a/5a5e1aec8c0f31d822daac05fafcb15c6dda05f1_2_1380x776.png 2x" data-dominant-color="F0F0F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1920×1080 310 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>my dataset has imbalance but is it due to only that reason or some other???</p>
<p>(i had train nd test folders in my dataset so i used splitfolders library to split train into 90/10 train validation set. )</p>

thecho7: <p>Val loss seems to fluctuate hard but the accuracy is not that weird.<br>
Decrease the learning rate and see how it changes for more epoch.</p>

brunoj: <p>so i actually trained efficientnet b1 from version 1 which results in the above graphs but for b0 i got<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/3/032a141394fddca39adff66d6d09ad5e57f239b6.png" data-download-href="https://discuss.pytorch.org/uploads/default/032a141394fddca39adff66d6d09ad5e57f239b6" title="accuracy_plot"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/3/032a141394fddca39adff66d6d09ad5e57f239b6_2_640x500.png" alt="accuracy_plot" data-base62-sha1="rZAgOiILjKUEuPKDlN0pecQ38q" width="640" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/3/032a141394fddca39adff66d6d09ad5e57f239b6_2_640x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/3/032a141394fddca39adff66d6d09ad5e57f239b6_2_960x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/0/3/032a141394fddca39adff66d6d09ad5e57f239b6_2_1280x1000.png 2x" data-dominant-color="F7F7F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">accuracy_plot</span><span class="informations">1761×1374 153 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/5/55a2eac854e66253da89c655a99ec8a35b02f57b.png" data-download-href="https://discuss.pytorch.org/uploads/default/55a2eac854e66253da89c655a99ec8a35b02f57b" title="loss_plot"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/5/55a2eac854e66253da89c655a99ec8a35b02f57b_2_631x500.png" alt="loss_plot" data-base62-sha1="cdzDgqFRlGYO9FwvUZ7b1LOZa55" width="631" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/5/55a2eac854e66253da89c655a99ec8a35b02f57b_2_631x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/5/55a2eac854e66253da89c655a99ec8a35b02f57b_2_946x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/5/55a2eac854e66253da89c655a99ec8a35b02f57b_2_1262x1000.png 2x" data-dominant-color="F8F7F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">loss_plot</span><span class="informations">1734×1374 119 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>i only ran 10 epochs for both.</p>

thecho7: <p>Looks better. The accuracy is even high than the previous exp.<br>
Try more:)</p>

brunoj: <p>I think you didnt get my point. The plots I posted at first were of enet b0 size64x64 and the plots i just posted are of enetb1 size64x64. Both are trained on 10 epochs. I have 6 classes out f which 2 are imbalanced besides my training data is 89k, val is 9k nd test is 24k. My confusion is am i doing it right? like i will deal with imbalance later on but right now are my both plots fine? if not then why? as you said val loss in first posted plots is fluctuating why it happens?  btw learning rate is 0.001 batch size 16 nd optimizer is adam.</p>

thecho7: <p>I recommend you to read</p><aside class="onebox stackexchange" data-onebox-src="https://stats.stackexchange.com/questions/255105/why-is-the-validation-accuracy-fluctuating">
  <header class="source">

      <a href="https://stats.stackexchange.com/questions/255105/why-is-the-validation-accuracy-fluctuating" target="_blank" rel="noopener nofollow ugc">stats.stackexchange.com</a>
  </header>

  <article class="onebox-body">
      <a href="https://stats.stackexchange.com/users/103055/pseudomonas" target="_blank" rel="noopener nofollow ugc">
    <img alt="pseudomonas" src="https://www.gravatar.com/avatar/ae52e7465a397c7740bc8de66af2b9ee?s=256&amp;d=identicon&amp;r=PG" class="thumbnail onebox-avatar" width="256" height="256">
  </a>

<h4>
  <a href="https://stats.stackexchange.com/questions/255105/why-is-the-validation-accuracy-fluctuating" target="_blank" rel="noopener nofollow ugc">Why is the validation accuracy fluctuating?</a>
</h4>

<div class="tags">
  <strong>machine-learning, python, deep-learning</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stats.stackexchange.com/users/103055/pseudomonas" target="_blank" rel="noopener nofollow ugc">
    pseudomonas
  </a>
  on <a href="https://stats.stackexchange.com/questions/255105/why-is-the-validation-accuracy-fluctuating" target="_blank" rel="noopener nofollow ugc">02:37AM - 08 Jan 17 UTC</a>
</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://www.analyticsvidhya.com/blog/2020/07/10-techniques-to-deal-with-class-imbalance-in-machine-learning/">
  <header class="source">
      <img src="https://www.analyticsvidhya.com/wp-content/uploads/2015/02/logo_square_v2.jpg" class="site-icon" width="" height="">

      <a href="https://www.analyticsvidhya.com/blog/2020/07/10-techniques-to-deal-with-class-imbalance-in-machine-learning/" target="_blank" rel="noopener nofollow ugc" title="08:27AM - 23 July 2020">Analytics Vidhya – 23 Jul 20</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/341;"><img src="https://cdn.analyticsvidhya.com/wp-content/uploads/2020/07/How-to-treat-Class-Imbalance-in-Machine-Learning.jpg" class="thumbnail" width="" height=""></div>

<h3><a href="https://www.analyticsvidhya.com/blog/2020/07/10-techniques-to-deal-with-class-imbalance-in-machine-learning/" target="_blank" rel="noopener nofollow ugc">10 Techniques to Solve Imbalanced Classes in Machine Learning (Updated 2023)</a></h3>

  <p>Learn how to use data augmentation, resampling techniques, and cost-sensitive learning for solving class imbalance in machine learning.</p>

  <p>
    <span class="label1">Est. reading time: 14 minutes</span>
  </p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>BTW, I understood your message</p>
<blockquote>
<p>so i actually trained efficientnet b1 from version 1 which results in the above graphs but for b0 i got</p>
</blockquote>
<p>Isn’t it mean that the second post is b0?</p>

brunoj: <p>Yes 2nd post is B0.<br>
Alright I read both of them. But reading learning curve is still confusing. And for imbalance Im thinking to just oversample 2 classes which have too much imbalance.<br>
But I read about weightedSampler. Was reading its documentation and ptrblck mentioned some sources and code snippets but I wonder how Im gonna apply that like i dont feel like I understood how to implement in pytorch like im newbie using pytorch although i understood the intuition.</p>
<p>my class imbalance is this way:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/f/4fcedd0b4f92a4a6226f4dd206d8531c5b1b944f.png" data-download-href="https://discuss.pytorch.org/uploads/default/4fcedd0b4f92a4a6226f4dd206d8531c5b1b944f" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/f/4fcedd0b4f92a4a6226f4dd206d8531c5b1b944f_2_690x388.png" alt="image" data-base62-sha1="bo0VehJD7tJ8nArTEkgUNBpqomX" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/f/4fcedd0b4f92a4a6226f4dd206d8531c5b1b944f_2_690x388.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/f/4fcedd0b4f92a4a6226f4dd206d8531c5b1b944f_2_1035x582.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/f/4fcedd0b4f92a4a6226f4dd206d8531c5b1b944f_2_1380x776.png 2x" data-dominant-color="F4EDEC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1920×1080 187 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

brunoj: <p>I wonder can i just copy a class folder say 4 and paste images in that folder or duplicate the folder images? like it will get balanced and equally i mean each image will have 1 similar image?</p>