Training loss plateau ? fix?

Sourabh_Yadav: <p>my training looks like</p>
<div class="md-table">
<table>
<thead>
<tr>
<th>Step</th>
<th>Training Loss</th>
</tr>
</thead>
<tbody>
<tr>
<td>50</td>
<td>16.211700</td>
</tr>
<tr>
<td>100</td>
<td>14.648300</td>
</tr>
<tr>
<td>150</td>
<td>11.676700</td>
</tr>
<tr>
<td>200</td>
<td>8.995300</td>
</tr>
<tr>
<td>250</td>
<td>8.248700</td>
</tr>
<tr>
<td>300</td>
<td>7.802700</td>
</tr>
<tr>
<td>350</td>
<td>7.363100</td>
</tr>
<tr>
<td>400</td>
<td>7.255300</td>
</tr>
<tr>
<td>450</td>
<td>7.258900</td>
</tr>
<tr>
<td>500</td>
<td>7.198600</td>
</tr>
<tr>
<td>550</td>
<td>7.236600</td>
</tr>
<tr>
<td>600</td>
<td>7.211900</td>
</tr>
<tr>
<td>650</td>
<td>7.203300</td>
</tr>
<tr>
<td>700</td>
<td>7.225800</td>
</tr>
<tr>
<td>750</td>
<td>7.187400</td>
</tr>
<tr>
<td>800</td>
<td>7.212800</td>
</tr>
<tr>
<td>850</td>
<td>7.229400</td>
</tr>
<tr>
<td>900</td>
<td>7.201000</td>
</tr>
<tr>
<td>950</td>
<td>7.176700</td>
</tr>
<tr>
<td>1000</td>
<td>7.199600</td>
</tr>
<tr>
<td>1050</td>
<td>7.169000</td>
</tr>
<tr>
<td>1100</td>
<td>7.237900</td>
</tr>
<tr>
<td>1150</td>
<td>7.224100</td>
</tr>
<tr>
<td>1200</td>
<td>7.103900</td>
</tr>
<tr>
<td>1250</td>
<td>7.191000</td>
</tr>
<tr>
<td>1300</td>
<td>7.173000</td>
</tr>
<tr>
<td>1350</td>
<td>7.203800</td>
</tr>
<tr>
<td>1400</td>
<td>7.213800</td>
</tr>
<tr>
<td>1450</td>
<td>7.243000</td>
</tr>
</tbody>
</table>
</div><p>learning rate 1e-5 , AdamW , batch=2,  weight decay =0.01</p>

anantguptadbl: <p><a class="mention" href="/u/sourabh_yadav">@Sourabh_Yadav</a>  unfortunately there is no one solution that will solve your problem. You have to create creative and experiment a few things</p>
<ol>
<li>Start with lr=0.01 and then decrease it by order of 10 gradually to understand whether the minima is being found</li>
<li>batch_size=2 is a big red flag. Your model is not getting enough data points to update weights in the back prop. You can start with a higher batch size and slowly decrease it once you have smoothened out the convergence</li>
</ol>
<p>There are many more factors and you can find many articles and discussion threads for the same</p><aside class="quote quote-modified" data-post="1" data-topic="15924">
  <div class="title">
    <div class="quote-controls"></div>
    <img alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/z/9fc29f/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/sloved-why-my-loss-not-decreasing/15924">[Sloved] Why my loss not decreasing</a> 
  </div>
  <blockquote>
    Hi, I am new to deeplearning and pytorch, I write a very simple demo, but the loss can’t decreasing when training. 
Any comments are highly appreciated! 
I want to use one hot to represent group and resource, there are 2 group and 4 resouces in training data: 
group1 (1, 0) can access resource 1(1, 0, 0, 0) and resource2(0, 1, 0, 0) 
group2(0, 1) can access resource3(0, 0, 1, 0) and resource4(0, 0, 0, 1) 
result is true(0) and false(1) 
So the first line in input [1, 0, 1, 0, 0, 0] mean group1 c…
  </blockquote>
</aside>


Sourabh_Yadav: <p>i increased the batch size to 4 , lr = 2e-6 and grad. accum. steps = 2., epochs = 3</p>
<div class="md-table">
<table>
<thead>
<tr>
<th>Step</th>
<th>Training Loss</th>
</tr>
</thead>
<tbody>
<tr>
<td>50</td>
<td>16.398500</td>
</tr>
<tr>
<td>100</td>
<td>15.975300</td>
</tr>
<tr>
<td>150</td>
<td>15.603200</td>
</tr>
<tr>
<td>200</td>
<td>14.988700</td>
</tr>
<tr>
<td>250</td>
<td>14.447100</td>
</tr>
<tr>
<td>300</td>
<td>13.770100</td>
</tr>
<tr>
<td>350</td>
<td>13.009500</td>
</tr>
<tr>
<td>400</td>
<td>12.087300</td>
</tr>
<tr>
<td>450</td>
<td>11.153500</td>
</tr>
<tr>
<td>500</td>
<td>10.203200</td>
</tr>
<tr>
<td>550</td>
<td>9.561700</td>
</tr>
<tr>
<td>600</td>
<td>9.059600</td>
</tr>
<tr>
<td>650</td>
<td>8.952400</td>
</tr>
<tr>
<td>700</td>
<td>8.840600</td>
</tr>
<tr>
<td>750</td>
<td>8.856600</td>
</tr>
<tr>
<td>800</td>
<td>8.853000</td>
</tr>
<tr>
<td>850</td>
<td>8.733600</td>
</tr>
<tr>
<td>900</td>
<td>8.694100</td>
</tr>
<tr>
<td>950</td>
<td>8.581500</td>
</tr>
<tr>
<td>1000</td>
<td>8.483000</td>
</tr>
<tr>
<td>1050</td>
<td>8.385400</td>
</tr>
<tr>
<td>1100</td>
<td>8.373000</td>
</tr>
<tr>
<td>1150</td>
<td>8.319600</td>
</tr>
<tr>
<td>1200</td>
<td>8.273000</td>
</tr>
<tr>
<td>1250</td>
<td>8.174100</td>
</tr>
<tr>
<td>1300</td>
<td>8.151100</td>
</tr>
<tr>
<td>1350</td>
<td>8.147100</td>
</tr>
<tr>
<td>1400</td>
<td>8.004400</td>
</tr>
<tr>
<td>1450</td>
<td>7.921600</td>
</tr>
<tr>
<td>1500</td>
<td>7.688300</td>
</tr>
<tr>
<td>1550</td>
<td>7.429600</td>
</tr>
<tr>
<td>1600</td>
<td>7.525600</td>
</tr>
<tr>
<td>1650</td>
<td>7.478100</td>
</tr>
<tr>
<td>1700</td>
<td>7.479400</td>
</tr>
<tr>
<td>1750</td>
<td>7.489100</td>
</tr>
<tr>
<td>1800</td>
<td>7.443700</td>
</tr>
<tr>
<td>1850</td>
<td>7.376700</td>
</tr>
<tr>
<td>1900</td>
<td>7.426600</td>
</tr>
<tr>
<td>1950</td>
<td>7.405100</td>
</tr>
<tr>
<td>2000</td>
<td>7.407100</td>
</tr>
<tr>
<td>2050</td>
<td>7.402200</td>
</tr>
</tbody>
</table>
</div><p>should I increase more but then I m also restricted on computation power and its multimodal model (image_text to text)<br>
 → Also what will be good loss to stop<br>
 → and will increasing training data be helpful ?</p>

anantguptadbl: <ul>
<li>Good loss to stop would be determined by your accuracy metrics on a separate test data<br>
It is quite possible with the loss at 7, you are actually getting good predictions on your test dta</li>
<li>Increasing training data is always useful, but make sure that you are not over fitting</li>
</ul>