The gradient descent don't work

ElderCharlie: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/b/0b0abd48e10d20f8dff2185206801e5d0ad22c4a.png" data-download-href="https://discuss.pytorch.org/uploads/default/0b0abd48e10d20f8dff2185206801e5d0ad22c4a" title="微信图片_20200112111000"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/b/0b0abd48e10d20f8dff2185206801e5d0ad22c4a_2_690x303.png" alt="微信图片_20200112111000" data-base62-sha1="1zGgc7N96D1N93e0EXhtjolhDzI" width="690" height="303" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/b/0b0abd48e10d20f8dff2185206801e5d0ad22c4a_2_690x303.png, https://discuss.pytorch.org/uploads/default/original/3X/0/b/0b0abd48e10d20f8dff2185206801e5d0ad22c4a.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/b/0b0abd48e10d20f8dff2185206801e5d0ad22c4a.png 2x" data-dominant-color="FDFCFC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">微信图片_20200112111000</span><span class="informations">965×425 15.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div> <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/b/0b23c7db85257dc77b00901b9a251773a82ac0ce.png" data-download-href="https://discuss.pytorch.org/uploads/default/0b23c7db85257dc77b00901b9a251773a82ac0ce" title="微信图片_20200112111111"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/b/0b23c7db85257dc77b00901b9a251773a82ac0ce_2_690x360.png" alt="微信图片_20200112111111" data-base62-sha1="1AxUxyRMyH8OZz9bRMoCr6fBHm6" width="690" height="360" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/b/0b23c7db85257dc77b00901b9a251773a82ac0ce_2_690x360.png, https://discuss.pytorch.org/uploads/default/original/3X/0/b/0b23c7db85257dc77b00901b9a251773a82ac0ce.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/b/0b23c7db85257dc77b00901b9a251773a82ac0ce.png 2x" data-dominant-color="FDFCFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">微信图片_20200112111111</span><span class="informations">821×429 17 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I made a inception model and train it on my image datasets.Using a learnning rate at 0.1 and using CrossEntropy as a loss function.The optimizer is SGD with momentum.<br>
As shown in the figure above，the weight gradient drops quickly, but the loss don’t go the same.<br>
I got confuse about it does the network trap in local optimum?<br>
And what can I do about it? Using a larger lr or something else？</p>

ptrblck: <p>The original Inception model was trained using the auxiliary outputs (which are also provided in the <code>torchvision</code> implementation) to “increase the gradient signal” as explained in the <a href="https://arxiv.org/abs/1409.4842">paper</a>.<br>
If you are not using it, I would guess the gradient might vanish at some point.</p>

crowsonkb: <p>0.1 is a high learning rate, especially if you are using momentum. I looked at how googlenet (inception v1) <a href="https://github.com/BVLC/caffe/blob/master/models/bvlc_googlenet/quick_solver.prototxt" rel="nofollow noopener">was trained</a> and they used SGD with initial lr 0.01 and momentum 0.9 (<code>quick_solver.prototxt</code> also contains the details of their learning rate decay schedule).</p>