Model is not Training

username: Stanley_C
cooked: <p>Here is my training loop code:</p>
<pre><code class="lang-auto">train_loader = torch.utils.data.DataLoader(MeshLoader("./train_mesh/", input_npoints=1024, output_npoints=2048), batch_size=batch_size, num_workers=num_workers)
test_loader = torch.utils.data.DataLoader(MeshLoader("./test_mesh/", input_npoints=1024, output_npoints=2048), batch_size=batch_size, num_workers=num_workers)
model =  PointNet_Upsample(npoint=1024, up_ratio_per_expansion=2, nstage=1).cuda()#PointNet(up_ratio=2, use_normal=False).cuda()
optimizer = torch.optim.Adam(model.parameters(),lr=0.0001)
count_train = 1
chamferDist = ChamferDistance()
model.train()

loss_array = []
for epoch in range(100):
	for step, (input_pt, target_pt) in enumerate(train_loader):
		optimizer.zero_grad()
		input_pt = input_pt.cuda().float()
		target_pt = target_pt.cuda().float()
		preds = model(input_pt)
		loss = chamferDist(target_pt, preds, bidirectional=True)	
		print("Epoch", epoch+1, ":", loss.detach().cpu().item()//input_pt.shape[0])
		loss.backward()
		optimizer.step()
		count_train = count_train + input_pt.shape[0]
</code></pre>
<p>However, when training, here is my training loss.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/4/949621b82f3aa3a45007d78b751561318031bf74.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/949621b82f3aa3a45007d78b751561318031bf74" title="IMG_2293"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/4/949621b82f3aa3a45007d78b751561318031bf74_2_333x250.jpeg" alt="IMG_2293" data-base62-sha1="lcsjVfKl44fe1pDPdmWVQLblthW" width="333" height="250" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/4/949621b82f3aa3a45007d78b751561318031bf74_2_333x250.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/9/4/949621b82f3aa3a45007d78b751561318031bf74_2_499x375.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/9/4/949621b82f3aa3a45007d78b751561318031bf74_2_666x500.jpeg 2x" data-dominant-color="AEA797"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">IMG_2293</span><span class="informations">3264×2448 2.54 MB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
The repeated pattern in loss seems to indicate that the model is <strong>not training</strong> and that their is an issue with the training loop in particular, because even an incorrect model would still see variance in the loss over time. I am using chamfer’s distance from this repo <a href="https://github.com/krrish94/chamferdist" rel="noopener nofollow ugc">chamfer’s distance repo link</a>. It returns the sum of the distance between the closest points for each point, and I tried to use it the same way that MSE loss would be used, but it seems like the chamfers distance is not interchangeable. The examples.py in the repo: <a href="https://github.com/krrish94/chamferdist/blob/master/example.py" class="inline-onebox" rel="noopener nofollow ugc">chamferdist/example.py at master · krrish94/chamferdist · GitHub</a>, seems to indicate that it should be used the same way:</p>
<pre><code class="lang-auto"># Backprop using this loss!
cdist.backward()
</code></pre>
<p>but, it does not seem to work. Could someone help me with this issue? I have tried many ways to fix this, but the issue still persists, so I was hoping someone here could help me. Thank you!</p>

username: ptrblck
cooked: <p>The repeated pattern indicates that you are not shuffling the data, which also shows your code, so could you enable shuffling and see if it would help in any way?<br>
Also, since the loss seems to be exactly the same, your computation graph might be detached, so check if all parameters get valid gradients after the <code>loss.backward()</code> operation via:</p>
<pre><code class="lang-python">for name, param in model.named_parameters():
    print(name, param.grad)
</code></pre>

username: Stanley_C
cooked: <p>Hi, thanks for responding! I added your code and got results like this:</p>
<pre><code class="lang-auto">FE_layer.0.weight tensor([[[[  43103.2578]],

         [[-176214.8594]],

         [[ 193186.0156]]],


        [[[ -65900.7109]],

         [[   9959.4365]],

         [[   7424.0977]]],


        [[[  -2614.8594]],

         [[ -31229.7559]],

         [[ -17630.7383]]],


        [[[  46508.3984]],

         [[  66506.5938]],

         [[ -16597.5312]]],


        [[[ -29938.4004]],

         [[  20681.8320]],

         [[  16445.1836]]],


</code></pre>
<p>I’m not sure if I’m interpreting it correctly, but it seems like the gradients are flowing, it is just the model doesn’t learn.</p>

username: ptrblck
cooked: <p>If you get valid gradient values for all parameters, then the computation graph is at least not detached.<br>
Note however, that the gradients seem to be quite large. I’m not familiar with your exact use case, but maybe try to scale the loss down a bit so that the gradient magnitudes will also be reduced.</p>

username: Stanley_C
cooked: <p>Okay, so I should divide the loss, so I don’t have “exploding gradients”? So my training loop is fine, is it just the loss is too large, causing the gradient magnitudes to increase to a point where the model does not learn anything?</p>

username: ptrblck
cooked: <p>That would be one possible approach, the other one would be to lower the learning rate.</p>

username: Stanley_C
cooked: <p>I tried scaling my inputs between -1 and 1, and then dividing the loss by a bit, (although it is still in the 6 digits). I still see gradient explosions, later in the neural network. Should I try to future scale down the loss and should I try gradient clipping if that doesn’t work?<br>
Example:</p>
<pre><code class="lang-auto">coordReconstruction_layer.0.10.bias tensor([  29565.5938,  107157.0859, -415445.0000,  -52302.0312, -174125.9688,
          91306.4375,  178269.4062, -365361.7500,  573210.0000,  207930.5625,
         -18185.3281,   58567.1094, -360612.0625,  128182.8359,   29353.1719,
        -199463.4062,  -17777.7500, -110101.8203, -532024.7500,  160549.6250,
        -576055.6875,  189424.9844,  157812.2344, -245538.3906,   85801.1406,
         176016.4375,  224826.5312,   23766.4766, -619454.1875, -294117.7500,
        -325290.5312,   60833.8672, -107854.8438,  -57861.6602, -264200.2500,
          83863.8906,  172188.1406,  385307.7500, -346806.8125,  108188.7266,
         440669.6250,  -97171.9609, -602740.8125,  129569.0156, -736367.2500,
          20825.3438,  -35092.2891,   22432.6875, -297975.8438, -282776.5938,
         226446.8125,   77869.7969,  -57261.6875, -323141.1875,  362888.2500,
        -118788.5781,   48330.5234, -223928.7188,  507906.5000,  -31208.7812,
        -106249.6562,  274496.8750, -547059.8750,  354355.5000],
       device='cuda:0')
coordReconstruction_layer.1.0.weight tensor([[[[ 1.2466e+06]],

         [[ 1.6526e+06]],

         [[ 5.6299e+06]],

         [[ 3.0220e+06]],
</code></pre>

username: ptrblck
cooked: <p>Yes, I would try to compare the gradient stats to a working model (e.g. use your model in a standard classification use case and just plot the gradients magnitudes for a few iterations/epochs).<br>
I don’t know why your loss is that large, but from my naive point of view the gradient values look too large to allow a proper training (disclaimer: I’m not a researcher, so take it with a grain of salt).</p>