Why my lost function always too high

username: Archy_dragon
cooked: <p>Hi can anyone check my code im not sure what wrong but when i train my module my loss always high with out going down</p>
<pre><code class="lang-auto">def train14(dataloader,net): 
      net = load_net(net, 'gpu')
      net = net.cuda()   
      epoch = 30
      criterion = nn.CrossEntropyLoss()
      optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)
      train_loss = list()
      #set_trace()
      for i in range(epoch):
        for i, data in enumerate(dataloader):
        
          inps, labs = data
          inps, labs = inps.cuda(args['device']), labs.cuda(args['device'])

          inps = Variable(inps).cuda(args['device'])
          labs = Variable(labs).cuda(args['device'])
          optimizer.zero_grad()
          outs = net(inps.permute(0, 3, 1, 2).float())
          soft_outs = F.softmax(outs, dim=1)
          prds = soft_outs.data.max(1)[1]
          loss = criterion(outs, labs)
          loss.backward()
          optimizer.step()
          prds = prds.cpu().numpy()
          inps_np = inps.detach().cpu().numpy()
          labs_np = labs.detach().cpu().numpy()
          train_loss.append(loss.data.item
                          ())
          print('[epoch %d], [iter %d / %d], [train loss %.5f]' % (epoch, i + 1, len(dataloader), np.asarray(train_loss).mean()))
        return net
</code></pre>
<pre><code class="lang-auto">x=train14(dataloadertrain,net='mobiface')
</code></pre>
<pre><code class="lang-auto">/content/drive/My Drive/recfaces13/recfaces/preprocessing/mtcnn_network/first_stage.py:32: UserWarning: volatile was removed and now has no effect. Use `with torch.no_grad():` instead.
  img = Variable(torch.FloatTensor(_preprocess(img)), volatile=True)
/content/drive/My Drive/recfaces13/recfaces/preprocessing/mtcnn_network/first_stage.py:32: UserWarning: volatile was removed and now has no effect. Use `with torch.no_grad():` instead.
  img = Variable(torch.FloatTensor(_preprocess(img)), volatile=True)
/content/drive/My Drive/recfaces13/recfaces/preprocessing/mtcnn_network/get_nets.py:74: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.
  a = F.softmax(a)
/content/drive/My Drive/recfaces13/recfaces/preprocessing/mtcnn_network/get_nets.py:74: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.
  a = F.softmax(a)
/content/drive/My Drive/recfaces13/recfaces/preprocessing/mtcnn_network/detector.py:79: UserWarning: volatile was removed and now has no effect. Use `with torch.no_grad():` instead.
  img_boxes = Variable(torch.FloatTensor(img_boxes), volatile=True)
/content/drive/My Drive/recfaces13/recfaces/preprocessing/mtcnn_network/detector.py:79: UserWarning: volatile was removed and now has no effect. Use `with torch.no_grad():` instead.
  img_boxes = Variable(torch.FloatTensor(img_boxes), volatile=True)
/content/drive/My Drive/recfaces13/recfaces/preprocessing/mtcnn_network/get_nets.py:120: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.
  a = F.softmax(a)
/content/drive/My Drive/recfaces13/recfaces/preprocessing/mtcnn_network/detector.py:100: UserWarning: volatile was removed and now has no effect. Use `with torch.no_grad():` instead.
  img_boxes = Variable(torch.FloatTensor(img_boxes), volatile=True)
/content/drive/My Drive/recfaces13/recfaces/preprocessing/mtcnn_network/get_nets.py:120: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.
  a = F.softmax(a)
/content/drive/My Drive/recfaces13/recfaces/preprocessing/mtcnn_network/detector.py:100: UserWarning: volatile was removed and now has no effect. Use `with torch.no_grad():` instead.
  img_boxes = Variable(torch.FloatTensor(img_boxes), volatile=True)
/content/drive/My Drive/recfaces13/recfaces/preprocessing/mtcnn_network/get_nets.py:174: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.
  a = F.softmax(a)
/content/drive/My Drive/recfaces13/recfaces/preprocessing/matlab_cp2tform.py:312: FutureWarning: `rcond` parameter will change to the default of machine precision times ``max(M, N)`` where M and N are the input matrix dimensions.
To use the future default and silence this warning we advise to pass `rcond=None`, to keep using the old, explicitly pass `rcond=-1`.
  r, _, _, _ = lstsq(X, U)
/content/drive/My Drive/recfaces13/recfaces/preprocessing/mtcnn_network/get_nets.py:174: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.
  a = F.softmax(a)
/content/drive/My Drive/recfaces13/recfaces/preprocessing/matlab_cp2tform.py:312: FutureWarning: `rcond` parameter will change to the default of machine precision times ``max(M, N)`` where M and N are the input matrix dimensions.
To use the future default and silence this warning we advise to pass `rcond=None`, to keep using the old, explicitly pass `rcond=-1`.
  r, _, _, _ = lstsq(X, U)
[epoch 30], [iter 1 / 182], [train loss 13.67538]
[epoch 30], [iter 2 / 182], [train loss 13.93409]
[epoch 30], [iter 3 / 182], [train loss 9.88422]
[epoch 30], [iter 4 / 182], [train loss 7.42359]
[epoch 30], [iter 5 / 182], [train loss 9.99569]
[epoch 30], [iter 6 / 182], [train loss 10.49389]
[epoch 30], [iter 7 / 182], [train loss 8.99476]
[epoch 30], [iter 8 / 182], [train loss 7.91546]
[epoch 30], [iter 9 / 182], [train loss 7.10814]
[epoch 30], [iter 10 / 182], [train loss 7.86791]
[epoch 30], [iter 11 / 182], [train loss 7.16874]
[epoch 30], [iter 12 / 182], [train loss 7.92612]
[epoch 30], [iter 13 / 182], [train loss 8.45253]
[epoch 30], [iter 14 / 182], [train loss 7.84907]
[epoch 30], [iter 15 / 182], [train loss 7.39011]
[epoch 30], [iter 16 / 182], [train loss 6.95442]
[epoch 30], [iter 17 / 182], [train loss 6.97154]
[epoch 30], [iter 18 / 182], [train loss 7.10896]
[epoch 30], [iter 19 / 182], [train loss 6.73483]
[epoch 30], [iter 20 / 182], [train loss 6.55003]
[epoch 30], [iter 21 / 182], [train loss 6.23812]
[epoch 30], [iter 22 / 182], [train loss 5.95457]
[epoch 30], [iter 23 / 182], [train loss 7.43152]
[epoch 30], [iter 24 / 182], [train loss 8.65758]
[epoch 30], [iter 25 / 182], [train loss 9.61982]
[epoch 30], [iter 26 / 182], [train loss 9.24982]
[epoch 30], [iter 27 / 182], [train loss 9.32092]
[epoch 30], [iter 28 / 182], [train loss 9.76417]
[epoch 30], [iter 29 / 182], [train loss 10.52662]
[epoch 30], [iter 30 / 182], [train loss 10.17573]
[epoch 30], [iter 31 / 182], [train loss 9.84750]
[epoch 30], [iter 32 / 182], [train loss 9.54338]
[epoch 30], [iter 33 / 182], [train loss 9.25418]
[epoch 30], [iter 34 / 182], [train loss 9.64890]
[epoch 30], [iter 35 / 182], [train loss 9.79242]
[epoch 30], [iter 36 / 182], [train loss 9.52817]
[epoch 30], [iter 37 / 182], [train loss 9.99964]
[epoch 30], [iter 38 / 182], [train loss 9.74174]
[epoch 30], [iter 39 / 182], [train loss 9.49257]
[epoch 30], [iter 40 / 182], [train loss 9.25540]
[epoch 30], [iter 41 / 182], [train loss 9.02966]
[epoch 30], [iter 42 / 182], [train loss 9.11816]
[epoch 30], [iter 43 / 182], [train loss 9.05369]
[epoch 30], [iter 44 / 182], [train loss 8.84793]
[epoch 30], [iter 45 / 182], [train loss 8.90156]
[epoch 30], [iter 46 / 182], [train loss 8.70805]
[epoch 30], [iter 47 / 182], [train loss 8.52278]
[epoch 30], [iter 48 / 182], [train loss 8.90073]
[epoch 30], [iter 49 / 182], [train loss 8.71908]
[epoch 30], [iter 50 / 182], [train loss 8.82156]
[epoch 30], [iter 51 / 182], [train loss 9.41688]
[epoch 30], [iter 52 / 182], [train loss 9.23589]
[epoch 30], [iter 53 / 182], [train loss 9.27772]
[epoch 30], [iter 54 / 182], [train loss 9.30042]
[epoch 30], [iter 55 / 182], [train loss 9.18761]
[epoch 30], [iter 56 / 182], [train loss 9.02356]
[epoch 30], [iter 57 / 182], [train loss 8.86526]
[epoch 30], [iter 58 / 182], [train loss 8.87040]
[epoch 30], [iter 59 / 182], [train loss 8.86168]
[epoch 30], [iter 60 / 182], [train loss 9.05313]
[epoch 30], [iter 61 / 182], [train loss 9.44328]
[epoch 30], [iter 62 / 182], [train loss 9.29097]
[epoch 30], [iter 63 / 182], [train loss 9.14349]
[epoch 30], [iter 64 / 182], [train loss 9.19337]
[epoch 30], [iter 65 / 182], [train loss 9.81524]
[epoch 30], [iter 66 / 182], [train loss 9.78365]
[epoch 30], [iter 67 / 182], [train loss 10.09947]
[epoch 30], [iter 68 / 182], [train loss 10.36468]
[epoch 30], [iter 69 / 182], [train loss 10.43877]
[epoch 30], [iter 70 / 182], [train loss 10.39080]
[epoch 30], [iter 71 / 182], [train loss 10.24445]
[epoch 30], [iter 72 / 182], [train loss 10.10217]
[epoch 30], [iter 73 / 182], [train loss 9.96378]
[epoch 30], [iter 74 / 182], [train loss 10.02074]
[epoch 30], [iter 75 / 182], [train loss 10.24865]
[epoch 30], [iter 76 / 182], [train loss 10.32667]
[epoch 30], [iter 77 / 182], [train loss 10.39449]
[epoch 30], [iter 78 / 182], [train loss 10.33688]
[epoch 30], [iter 79 / 182], [train loss 10.86841]
[epoch 30], [iter 80 / 182], [train loss 10.74629]
[epoch 30], [iter 81 / 182], [train loss 10.61362]
[epoch 30], [iter 82 / 182], [train loss 10.50374]
Exception ignored in: &lt;bound method _MultiProcessingDataLoaderIter.__del__ of &lt;torch.utils.data.dataloader._MultiProcessingDataLoaderIter object at 0x7fce549480b8&gt;&gt;
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 1101, in __del__
Exception ignored in: &lt;bound method _MultiProcessingDataLoaderIter.__del__ of &lt;torch.utils.data.dataloader._MultiProcessingDataLoaderIter object at 0x7fce549480b8&gt;&gt;
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 1101, in __del__
    self._shutdown_workers()
    self._shutdown_workers()
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 1075, in _shutdown_workers
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 1075, in _shutdown_workers
    w.join(timeout=_utils.MP_STATUS_CHECK_INTERVAL)
  File "/usr/lib/python3.6/multiprocessing/process.py", line 122, in join
    w.join(timeout=_utils.MP_STATUS_CHECK_INTERVAL)
  File "/usr/lib/python3.6/multiprocessing/process.py", line 122, in join
    assert self._parent_pid == os.getpid(), 'can only join a child process'
AssertionError: can only join a child process
    assert self._parent_pid == os.getpid(), 'can only join a child process'
AssertionError: can only join a child process
Exception ignored in: &lt;bound method _MultiProcessingDataLoaderIter.__del__ of &lt;torch.utils.data.dataloader._MultiProcessingDataLoaderIter object at 0x7fce54338400&gt;&gt;
Exception ignored in: &lt;bound method _MultiProcessingDataLoaderIter.__del__ of &lt;torch.utils.data.dataloader._MultiProcessingDataLoaderIter object at 0x7fce54338400&gt;&gt;
Traceback (most recent call last):
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 1101, in __del__
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 1101, in __del__
    self._shutdown_workers()
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 1075, in _shutdown_workers
    w.join(timeout=_utils.MP_STATUS_CHECK_INTERVAL)
    self._shutdown_workers()
  File "/usr/lib/python3.6/multiprocessing/process.py", line 122, in join
    assert self._parent_pid == os.getpid(), 'can only join a child process'
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 1075, in _shutdown_workers
    w.join(timeout=_utils.MP_STATUS_CHECK_INTERVAL)
  File "/usr/lib/python3.6/multiprocessing/process.py", line 122, in join
AssertionError: can only join a child process
Exception ignored in: &lt;bound method _MultiProcessingDataLoaderIter.__del__ of &lt;torch.utils.data.dataloader._MultiProcessingDataLoaderIter object at 0x7fce54115ba8&gt;&gt;
Traceback (most recent call last):
    assert self._parent_pid == os.getpid(), 'can only join a child process'
AssertionError: can only join a child process
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 1101, in __del__
    self._shutdown_workers()
Exception ignored in: &lt;bound method _MultiProcessingDataLoaderIter.__del__ of &lt;torch.utils.data.dataloader._MultiProcessingDataLoaderIter object at 0x7fce54115ba8&gt;&gt;
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 1101, in __del__
    self._shutdown_workers()
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 1075, in _shutdown_workers
    w.join(timeout=_utils.MP_STATUS_CHECK_INTERVAL)
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 1075, in _shutdown_workers
  File "/usr/lib/python3.6/multiprocessing/process.py", line 122, in join
    w.join(timeout=_utils.MP_STATUS_CHECK_INTERVAL)
  File "/usr/lib/python3.6/multiprocessing/process.py", line 122, in join
    assert self._parent_pid == os.getpid(), 'can only join a child process'
    assert self._parent_pid == os.getpid(), 'can only join a child process'
AssertionError: can only join a child process
AssertionError: can only join a child process
[epoch 30], [iter 83 / 182], [train loss 10.62443]
[epoch 30], [iter 84 / 182], [train loss 10.49795]
[epoch 30], [iter 85 / 182], [train loss 10.37444]
[epoch 30], [iter 86 / 182], [train loss 10.78173]
[epoch 30], [iter 87 / 182], [train loss 10.94274]
[epoch 30], [iter 88 / 182], [train loss 11.21831]
[epoch 30], [iter 89 / 182], [train loss 11.64633]
[epoch 30], [iter 90 / 182], [train loss 11.78724]
[epoch 30], [iter 91 / 182], [train loss 11.65771]
[epoch 30], [iter 92 / 182], [train loss 11.91948]
[epoch 30], [iter 93 / 182], [train loss 11.92936]
[epoch 30], [iter 94 / 182], [train loss 11.80245]
[epoch 30], [iter 95 / 182], [train loss 11.67822]
[epoch 30], [iter 96 / 182], [train loss 11.55657]
[epoch 30], [iter 97 / 182], [train loss 12.20565]
[epoch 30], [iter 98 / 182], [train loss 12.65326]
[epoch 30], [iter 99 / 182], [train loss 12.79071]
[epoch 30], [iter 100 / 182], [train loss 12.75293]
[epoch 30], [iter 101 / 182], [train loss 12.98185]
[epoch 30], [iter 102 / 182], [train loss 13.01299]
[epoch 30], [iter 103 / 182], [train loss 13.11844]
[epoch 30], [iter 104 / 182], [train loss 13.23922]
[epoch 30], [iter 105 / 182], [train loss 13.16908]
[epoch 30], [iter 106 / 182], [train loss 13.09564]
[epoch 30], [iter 107 / 182], [train loss 12.97327]
[epoch 30], [iter 108 / 182], [train loss 12.85315]
[epoch 30], [iter 109 / 182], [train loss 12.73523]
[epoch 30], [iter 110 / 182], [train loss 12.62138]
[epoch 30], [iter 111 / 182], [train loss 12.58768]
[epoch 30], [iter 112 / 182], [train loss 12.92546]
[epoch 30], [iter 113 / 182], [train loss 13.38524]
[epoch 30], [iter 114 / 182], [train loss 13.70754]
[epoch 30], [iter 115 / 182], [train loss 13.99418]
[epoch 30], [iter 116 / 182], [train loss 14.29291]
[epoch 30], [iter 117 / 182], [train loss 14.17113]
[epoch 30], [iter 118 / 182], [train loss 14.09050]
[epoch 30], [iter 119 / 182], [train loss 14.19034]
[epoch 30], [iter 120 / 182], [train loss 14.28804]
[epoch 30], [iter 121 / 182], [train loss 14.36596]
[epoch 30], [iter 122 / 182], [train loss 14.56526]
[epoch 30], [iter 123 / 182], [train loss 14.48954]
[epoch 30], [iter 124 / 182], [train loss 14.59177]
[epoch 30], [iter 125 / 182], [train loss 14.53667]
[epoch 30], [iter 126 / 182], [train loss 14.59556]
[epoch 30], [iter 127 / 182], [train loss 14.53797]
[epoch 30], [iter 128 / 182], [train loss 14.42439]
[epoch 30], [iter 129 / 182], [train loss 14.48302]
[epoch 30], [iter 130 / 182], [train loss 14.41163]
[epoch 30], [iter 131 / 182], [train loss 14.56384]
[epoch 30], [iter 132 / 182], [train loss 14.45351]
[epoch 30], [iter 133 / 182], [train loss 14.39059]
[epoch 30], [iter 134 / 182], [train loss 14.32216]
[epoch 30], [iter 135 / 182], [train loss 14.37767]
[epoch 30], [iter 136 / 182], [train loss 14.31511]
[epoch 30], [iter 137 / 182], [train loss 14.38090]
[epoch 30], [iter 138 / 182], [train loss 14.51481]
[epoch 30], [iter 139 / 182], [train loss 14.60453]
[epoch 30], [iter 140 / 182], [train loss 14.51285]
[epoch 30], [iter 141 / 182], [train loss 14.62092]
[epoch 30], [iter 142 / 182], [train loss 14.51796]
[epoch 30], [iter 143 / 182], [train loss 14.56977]
[epoch 30], [iter 144 / 182], [train loss 14.85173]
[epoch 30], [iter 145 / 182], [train loss 14.74931]
[epoch 30], [iter 146 / 182], [train loss 14.64830]
[epoch 30], [iter 147 / 182], [train loss 14.57249]
[epoch 30], [iter 148 / 182], [train loss 14.47403]
[epoch 30], [iter 149 / 182], [train loss 14.46919]
[epoch 30], [iter 150 / 182], [train loss 14.55256]
[epoch 30], [iter 151 / 182], [train loss 14.45618]
[epoch 30], [iter 152 / 182], [train loss 14.36221]
[epoch 30], [iter 153 / 182], [train loss 14.59301]
[epoch 30], [iter 154 / 182], [train loss 14.52462]
[epoch 30], [iter 155 / 182], [train loss 14.43091]
[epoch 30], [iter 156 / 182], [train loss 14.33841]
[epoch 30], [iter 157 / 182], [train loss 14.24708]
[epoch 30], [iter 158 / 182], [train loss 14.46720]
[epoch 30], [iter 159 / 182], [train loss 14.49228]
[epoch 30], [iter 160 / 182], [train loss 14.40170]
[epoch 30], [iter 161 / 182], [train loss 14.31225]
[epoch 30], [iter 162 / 182], [train loss 14.22391]
[epoch 30], [iter 163 / 182], [train loss 14.33623]
[epoch 30], [iter 164 / 182], [train loss 14.24881]
[epoch 30], [iter 165 / 182], [train loss 14.16246]
[epoch 30], [iter 166 / 182], [train loss 14.07714]
[epoch 30], [iter 167 / 182], [train loss 14.10332]
[epoch 30], [iter 168 / 182], [train loss 14.04919]
[epoch 30], [iter 169 / 182], [train loss 14.00492]
[epoch 30], [iter 170 / 182], [train loss 13.92254]
[epoch 30], [iter 171 / 182], [train loss 13.84112]
[epoch 30], [iter 172 / 182], [train loss 13.76065]
[epoch 30], [iter 173 / 182], [train loss 13.68111]
[epoch 30], [iter 174 / 182], [train loss 13.67412]
[epoch 30], [iter 175 / 182], [train loss 13.59598]
[epoch 30], [iter 176 / 182], [train loss 13.51873]
[epoch 30], [iter 177 / 182], [train loss 13.44235]
[epoch 30], [iter 178 / 182], [train loss 13.41863]
[epoch 30], [iter 179 / 182], [train loss 13.35066]
[epoch 30], [iter 180 / 182], [train loss 13.27877]
[epoch 30], [iter 181 / 182], [train loss 13.37068]
[epoch 30], [iter 182 / 182], [train loss 13.29721]
</code></pre>

username: ptrblck
cooked: <p>Based on the loss you’ve posted, it seems the model is initially learning but diverges eventually.<br>
You could try to play around with some hyperparameters (such as the learning rate or trying another optimizer).<br>
If that doesn’t help, you could try to overfit a small data sample first (again by changing some hyperparameters) and make sure your model is able to do so. Once this is done, you could carefully scale up the problem again.</p>