Model is not training @ PyTorch

Zohaib: <p>Hi,<br>
I am using Pytorch for train my model however it is not learning. I have written my own loss function. Can anyone please review the code and the steps I am taking for training? A plot showing loss and accuracy is attached in the end. Thank you.</p>
<pre><code class="lang-auto">def train(EPOCHS):
	    BATCH_SIZE = 5
	    best_acc = 0.0
	    train_dataset, train_data_loader, test_dataset, test_data_loader = load_data_set(BATCH_SIZE)  # receiving only second parameter
	    print("Number of train samples: ", len(train_dataset))
	    print("Number of test samples: ", len(test_dataset))
	    print("Epoch       Train Accuracy         TrainLoss          Test Accuracy          Test Loss")
	    for epoch in range(EPOCHS):

		#print("epoch: ", epoch)
		model.train()

		train_acc = 0.0
		train_loss = 0.0
		# 25 random points (x,y) on image for evaluation
		points = torch.randint(0, 32, ([100, 2]), dtype=torch.int)

		batch_count = 0
		for i, (images, gt) in enumerate(train_data_loader):
		    batch_count += 1

		    # Move images and labels to gpu if available
		    if cuda_avail:
		        images_tensor = Variable(images[0].cuda())

		    gt_tensor = Variable(gt[0])


		    # Clear all accumulated gradients
		    optimizer.zero_grad()
		    # Predict classes using images from the train set
		    output_lines = model(images_tensor)
		    # pdb.set_trace()  #TODO need to remove
		    h_matrix = output_lines.view(len(images_tensor), 25, 3)
		    gt_no = 0
		    batch_loss = 0  # calculate loss of every batch
		    batch_acc = 0   # calculate accuracy of every batch

		    for hyper_lines in h_matrix:    # pick all lines of single image

		        gt_img = np.array(gt_tensor[gt_no][0]) # select GT w.r.t. selected image
		        gt_no += 1   # for next GT
		        image_loss = 0
		        image_acc = 0
		        for point in range(points.size()[0]):   # pick one test point at a time
		            x = points[point]  # considering 1 point
		            estimated_value = indicator_functions.estimatedPosition(hyper_lines, x.double())
		            calculated_value = indicator_functions.calculated_Position(gt_img, x)

		            loss = (calculated_value - estimated_value) ** 2

		            # Calculate training loss and training accuracy
		            image_loss += loss.cpu().item()
		            if loss.cpu().item() &lt; 0.5:
		                image_acc += 1.0
		            # pdb.set_trace() #TODO need to remove

		        # Compute the average acc and loss over all 25 lines of a single convex image and add in batch loss/acc
		        batch_loss += image_loss / points.size()[0]
		        batch_acc  += image_acc / points.size()[0]
		        # pdb.set_trace() #TODO need to remove

		    train_loss += batch_loss / h_matrix.size()[0]
		    train_acc += batch_acc / h_matrix.size()[0]

		    ### Making the loss tensor
		    train_loss_tensor = Variable(torch.tensor(train_loss), requires_grad=True)

		    # Backpropagate the loss
		    train_loss_tensor.backward()		 

		    # Adjust parameters according to the computed gradients
		    optimizer.step()
		    # pdb.set_trace()  # TODO need to remove

		# Compute the average acc and loss over all 50000 training images
		train_loss = train_loss / batch_count
		train_acc = train_acc / batch_count
		# pdb.set_trace()  # TODO need to remove

		### Evaluate on the test set
		test_acc, test_loss = test(test_data_loader, test_dataset)

		### Save the model if the test acc is greater than our current best
		if test_acc &gt; best_acc:
		    save_models(epoch)
		    best_acc = test_acc

		# Print the metrics
		print("{}   {}   {}    {}     {}".format(epoch, train_acc, train_loss, test_acc, test_loss))
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/1/51c6d04c0ac715842e75f6d905415db27d058837.png" data-download-href="https://discuss.pytorch.org/uploads/default/51c6d04c0ac715842e75f6d905415db27d058837" title="v1_50epochs"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/1/51c6d04c0ac715842e75f6d905415db27d058837_2_690x232.png" alt="v1_50epochs" data-base62-sha1="bFqD98sZT6mfqDmTfUj0xr0ZQXB" width="690" height="232" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/1/51c6d04c0ac715842e75f6d905415db27d058837_2_690x232.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/1/51c6d04c0ac715842e75f6d905415db27d058837_2_1035x348.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/1/51c6d04c0ac715842e75f6d905415db27d058837_2_1380x464.png 2x" data-dominant-color="FBFAFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">v1_50epochs</span><span class="informations">1782×600 81.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Krish: <p>Here is the problem:</p>
<pre><code class="lang-auto"># Backpropagate the loss
train_loss_tensor = Variable(torch.tensor(train_loss), requires_grad=True)
</code></pre>
<p>When you are making a <code>Variable</code>, you are removing all gradient information from the tensor, hence you see no training improvement, since it doesn’t know its origins.</p>
<p>Try this code:</p>
<pre><code class="lang-auto">a = torch.randn(2, 3, requires_grad=True)
a = a*2
print(a.grad_fn)
# &lt;MulBackward0 object at 0x7f1819cf3518&gt;
a = Variable(a, requires_grad=True)
print(a.grad_fn)
# None
</code></pre>

Zohaib: <aside class="quote no-group quote-modified" data-username="Krish" data-post="2" data-topic="85346">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/krish/48/25533_2.png" class="avatar"> Krish:</div>
<blockquote></blockquote>
</aside>
<p>Thanks for reply Krish. Previously I was calculating loss in float variable. I have updated the a part of code such that now loss is a tensor. Here are the details of the loss updates;</p>
<ol>
<li>Initially I am calculating <strong>loss</strong> which is a tensor;</li>
</ol>
<pre><code class="lang-auto">tensor(1.9175e-11, dtype=torch.float64, grad_fn=&lt;PowBackward0&gt;)
</code></pre>
<ol start="2">
<li>I am averaging 100 losses in variable <strong>batch_loss</strong>,</li>
</ol>
<pre><code class="lang-auto">tensor(3.7276, dtype=torch.float64, grad_fn=&lt;AddBackward0&gt;)
</code></pre>
<ol start="3">
<li>Then I am averaging 25 batch_loss in variable <strong>train_loss</strong>;</li>
</ol>
<pre><code class="lang-auto">tensor(0.7455, dtype=torch.float64, grad_fn=&lt;AddBackward0&gt;)
</code></pre>
<p>When I use <strong>train_loss.backward()</strong>, it shows error.</p>
<pre><code class="lang-auto">RuntimeError: Function AddBackward0 returned an invalid gradient at index 1 - expected type TensorOptions(dtype=float, device=cuda:0, layout=Strided, requires_grad=false) but got TensorOptions(dtype=float, device=cpu, layout=Strided, requires_grad=false) (validate_outputs at /opt/conda/conda-bld/pytorch_1587428398394/work/torch/csrc/autograd/engine.cpp:484)
frame #0: c10::Error::Error(c10::SourceLocation, std::string const&amp;) + 0x4e (0x7f5692314b5e in /home/mz/anaconda3/envs/semSeg/lib/python3.7/site-packages/torch/lib/libc10.so)
frame #1: &lt;unknown function&gt; + 0x2ae2834 (0x7f56bbfd8834 in /home/mz/anaconda3/envs/semSeg/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #2: torch::autograd::Engine::evaluate_function(std::shared_ptr&lt;torch::autograd::GraphTask&gt;&amp;, torch::autograd::Node*, torch::autograd::InputBuffer&amp;) + 0x548 (0x7f56bbfda368 in /home/mz/anaconda3/envs/semSeg/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #3: torch::autograd::Engine::thread_main(std::shared_ptr&lt;torch::autograd::GraphTask&gt; const&amp;, bool) + 0x3d2 (0x7f56bbfdc2f2 in /home/mz/anaconda3/envs/semSeg/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #4: torch::autograd::Engine::thread_init(int) + 0x39 (0x7f56bbfd4969 in /home/mz/anaconda3/envs/semSeg/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
frame #5: torch::autograd::python::PythonEngine::thread_init(int) + 0x38 (0x7f56bf31b558 in /home/mz/anaconda3/envs/semSeg/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #6: &lt;unknown function&gt; + 0xc819d (0x7f56c1d7e19d in /home/mz/anaconda3/envs/semSeg/lib/python3.7/site-packages/torch/lib/../../../.././libstdc++.so.6)
frame #7: &lt;unknown function&gt; + 0x76db (0x7f56dac2e6db in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #8: clone + 0x3f (0x7f56da95788f in /lib/x86_64-linux-gnu/libc.so.6)
</code></pre>
<ol start="4">
<li>Then I use the Variable() as given blow;</li>
</ol>
<pre><code class="lang-auto">train_loss_tensor = Variable(train_loss, requires_grad=True)
#tensor(0.7810, dtype=torch.float64, requires_grad=True)

print(train_loss_tensor.grad_fn)
#None

train_loss_tensor.backward()
</code></pre>
<p>It is executing (no error), but still model is not training. Will you please elaborate, how could I make it working? Thank you.</p>

Krish: <p>The first line of the error suggests that there is a device mismatch. Are you moving the loss to the cpu midway?</p>
<p>And make sure the loss is a result of differentiable functions on the input, else the training won’t work. I don’t know if the indicator_functions are.</p>

Zohaib: <p>Yes, I was moving loss to cpu, but now I am remaining it to gpu.</p>
<p>Here are the indicator_functions:</p>
<pre><code class="lang-auto">
# signed distance function
# H_h(x) = n_h . x + d_h
def sdf(hyperlines, x):
    dist_vector = torch.zeros([hyperlines.size()[0]], dtype=torch.double)
    i = 0
    for x_n, y_n, d_n in hyperlines:
        n_h = torch.tensor([x_n, y_n], dtype=torch.double)
        dist_vector[i] = torch.matmul(n_h, x) + d_n  # n_h[0]*x[0] + n_h[1]*x[1] + d_n
        i += 1
    return dist_vector

def logSumExp(dist_vector, delta):
    return torch.logsumexp(dist_vector, 0)*delta


def sigmoid(sigma, fai_x):
    return torch.sigmoid(sigma * fai_x)

# the Function uses softmax, sigmoid and returns probability of the point x inside object
def estimatedPosition(lines, x):
    dist_vector = sdf(lines, x)
    delta = torch.tensor(1) # 0.006
    fai_x = logSumExp(dist_vector, delta)  # softmax
    sigma = -1 #-75
    estimation = sigmoid(sigma, fai_x)
    return estimation

# function returns 1 if point (x) is inside GT object, returns 0 other wise
def calculated_Position(train_GT_image,x):
    if train_GT_image[x[0].item(), x[1].item()] == 0: return torch.tensor(1.0, dtype=torch.double)
    return torch.tensor(0.0, dtype=torch.double)
</code></pre>
<p>One more thing, after <strong>loss.backward()</strong> when I try <strong>print(model.weight.grad)</strong> it produces the error as shown below.</p>
<pre><code class="lang-auto">(Pdb) print(model.weight.grad)
*** AttributeError: 'SimpleNet' object has no attribute 'weight'
</code></pre>

Krish: <p>Sorry for the late reply. Hope you have already solved the problem.</p>
<p>The indicator functions are perfect and shouldn’t cause any problems in optimization.</p>
<p>Regarding <code>model.weight.grad</code>, your model doesn’t have a weight matrix attached, instead you should call the module object inside the model.</p>
<pre><code class="lang-auto">class SimpleModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(2, 1)

...

model = SimpleModel()
</code></pre>
<p>You should call:</p>
<pre><code class="lang-auto">model.fc1.weight.grad
</code></pre>

Zohaib: <aside class="quote no-group" data-username="Zohaib" data-post="5" data-topic="85346">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/z/3be4f8/48.png" class="avatar"> Zohaib:</div>
<blockquote>
<p>after <strong>loss.backward()</strong> when I try <strong>print(model.weight.grad)</strong> it produces the error as shown below.</p>
</blockquote>
</aside>
<p>Hi Krish,<br>
Thanks for your reply.<br>
Yes, the problem has been slightly solved. Now the model is training but it is not achieving the desired results. This is not related to the model or learning.<br>
As you suggested before, the error was in gradients as they were removing before back tracking. Now, after creating all the torch variable on cuda, the model has started training. Thanks for your help. <img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=12" title=":blush:" class="emoji" alt=":blush:" loading="lazy" width="20" height="20"></p>
<p>-Zohaib</p>

Krish: <p>Great. Hope you get your results soon.<br>
Good luck. <img src="https://discuss.pytorch.org/images/emoji/apple/+1.png?v=9" title=":+1:" class="emoji" alt=":+1:"></p>
<p>P.S. You might want to mark the solution. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>