Error in loss function

Mike2004: <p>Hello, i’m making some changes on a normal CNN, to be compatible with other model, that i create.<br>
I get the following error:</p>
<p>File “C:/Users/user/.spyder-py3/cnn.py”, line 105, in <br>
loss = criterion(output,img)</p>
<p>RuntimeError: The size of tensor a (10) must match the size of tensor b (28) at non-singleton dimension 3</p>
<p>This is the CNN:</p>
<p>import time<br>
import torch<br>
import torch.nn as nn<br>
import torchvision.transforms as transforms<br>
import torchvision.datasets as dsets<br>
from torch.autograd import Variable</p>
<p>GPU = 0</p>
<p>def to_img(x):<br>
x = x.view(x.size(0), 1, 28, 28)<br>
return x</p>
<p>train_dataset = dsets.MNIST(root=’./data’,<br>
train=True,<br>
transform=transforms.ToTensor(),<br>
download=True)</p>
<p>test_dataset = dsets.MNIST(root=’./data’,<br>
train=False,<br>
transform=transforms.ToTensor())</p>
<p>batch_size =251<br>
n_iters = 71800</p>
<p>num_epochs = n_iters / (len(train_dataset) / batch_size)<br>
num_epochs = int(num_epochs)</p>
<p>train_loader = torch.utils.data.DataLoader(dataset=train_dataset,<br>
batch_size=batch_size,<br>
shuffle=True)</p>
<p>test_loader = torch.utils.data.DataLoader(dataset=test_dataset,<br>
batch_size=batch_size,<br>
shuffle=False)</p>
<p>class CNNModel(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(CNNModel, self).<strong>init</strong>()</p>
<pre><code>    self.cnn1 = nn.Conv2d(in_channels=1, out_channels=16, kernel_size=5, stride=1, padding=2)
    self.relu1 = nn.ReLU()

    self.maxpool1 = nn.MaxPool2d(kernel_size=2)

    self.cnn2 = nn.Conv2d(in_channels=16, out_channels=32, kernel_size=5, stride=1, padding=2)
    self.relu2 = nn.ReLU()

    self.maxpool2 = nn.MaxPool2d(kernel_size=2)

    self.fc1 = nn.Linear(32 * 7 * 7, 10)

def forward(self, x):
    out = self.cnn1(x)
    out = self.relu1(out)
   
    out = self.maxpool1(out)
   
    out = self.cnn2(out)
    out = self.relu2(out)

    out = self.maxpool2(out)

    out = out.view(out.size(0), -1)

    out = self.fc1(out)
   
    return out
</code></pre>
<p>model = CNNModel()</p>
<p>if torch.cuda.is_available():<br>
torch.cuda.set_device(GPU)<br>
model.cuda()<br>
print(“cuda OK”)</p>
<p>criterion = nn.MSELoss()</p>
<p>learning_rate = 1e-3</p>
<p>optimizer = torch.optim.Adam(<br>
model.parameters(), lr=learning_rate, weight_decay=0)</p>
<p>iter = 0<br>
start = time.time()<br>
for epoch in range(num_epochs):<br>
for data in train_loader:<br>
img, _ = data<br>
img = Variable(img).cuda()<br>
nova=to_img(img)</p>
<pre><code>    st = time.time()

    optimizer.zero_grad()

    output = model(img)

    loss = criterion(output,img)

    loss.backward()

    optimizer.step()

    iter += 1

       
print('epoch [{}/{}], loss:{:.4f}'
              .format(epoch + 1, num_epochs, loss.item()))
</code></pre>
<p>Anyone knows what i am doing wrong?</p>
<p>Thank you,<br>
Mike</p>

kunalpratap: <p>The criterion takes the output and the corresponding label as it arguments, your code is passing output of shape 10 and input img of (3,28,28)<br>
Try these changes</p>
<ol>
<li>img, target = data</li>
<li>loss = criterion(target, label)</li>
</ol>

kunalpratap: <p>in the for loop it should be,</p>
<p>from data in train_loader:<br>
image,target  = data</p>
<p>data is basically a tuple (image,target)</p>
<p>Try this once</p>

Mike2004: <p>So, how should i define the label variable?</p>

kunalpratap: <p>data is a tuple,<br>
so it should be like<br>
target and label are the same thing, I just used a different variable for it.<br>
for data in train_loader:<br>
image, label = data</p>
<p>and then  loss = criterion(label,output)</p>

Mike2004: <p>Oh, ok, i didnt understand that they are the same thing, thank you. I already made, your suggestions, then i get the following errors:</p>
<p>File “C:/Users/user/.spyder-py3/my_cnn.py”, line 109, in <br>
loss = criterion(label,output)</p>
<p>File “C:\Users\user\Anaconda3\lib\site-packages\torch\nn\modules\loss.py”, line 431, in forward<br>
return F.mse_loss(input, target, reduction=self.reduction)</p>
<p>File “C:\Users\user\Anaconda3\lib\site-packages\torch\nn\functional.py”, line 2199, in mse_loss<br>
ret = (input - target) ** 2</p>
<p>RuntimeError: The size of tensor a (251) must match the size of tensor b (10) at non-singleton dimension 1</p>
<p>This is the alterations:</p>
<p>criterion = nn.MSELoss()</p>
<p>best= 12311242</p>
<p>learning_rate = 1e-3</p>
<p>optimizer = torch.optim.Adam(<br>
model.parameters(), lr=learning_rate, weight_decay=0)</p>
<p>iter = 0<br>
start = time.time()<br>
for epoch in range(num_epochs):<br>
for data in train_loader:<br>
img, label= data<br>
label = Variable(label).cuda()<br>
img = Variable(img).cuda()<br>
nova=to_img(img)</p>
<pre><code>    st = time.time()

    optimizer.zero_grad()

    output = model(img)

    loss = criterion(label,output)

    loss.backward()

    optimizer.step()

    iter += 1

print('epoch [{}/{}], loss:{:.4f}'
              .format(epoch + 1, num_epochs, loss.item()))

if epoch % 1 == 0:
    pic = to_img(output.cpu().data)
    if loss.item()&lt;best:
        best=loss.item()</code></pre>

kunalpratap: <p>Can you print the shapes of label and output and send here?</p>

Mike2004: <p>Of course, its here:</p>
<p>label- torch.Size([251])<br>
output- torch.Size([251, 10])</p>

kunalpratap: <p>Okay, I got your problem. Label is giving out the exact classes for each element in the mini batch.<br>
Output consists of the probability over 10 classes for all minibatches. You can take an argmax over output to get the predicted class and then pass it into criterion.</p>
<p>You can make this change:</p>
<p>loss = criterion(label,torch.argmax(output,dim=1))</p>

jmandivarapu1: <aside class="quote no-group quote-post-not-found" data-username="Mike2004" data-post="11" data-topic="81228">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mike2004/48/24981_2.png" class="avatar"> Mike2004:</div>
<blockquote>
<p>thnn_mse_loss_forward not supported on CUDAType for Long</p>
</blockquote>
</aside>
<p>Yeah it’s saying you are having torch.Longtensors while passing to the MSE function.</p>
<p>So change the type of tensors to float .type(torch.FloatTensor) then pass</p>

Mike2004: <p>Is this, that you are suggesting?</p>
<pre><code>    label=float.type(torch.FloatTensor)
    output=float.type(torch.FloatTensor)

    loss=criterion(label,torch.argmax(output,dim=1))
</code></pre>
<p>Now i get this:</p>
<p>File “C:/Users/user/.spyder-py3/original_cnn.py”, line 109, in <br>
label=float.type(torch.FloatTensor)</p>
<p>AttributeError: type object ‘float’ has no attribute ‘type’</p>

jmandivarapu1: <pre><code class="lang-auto">label=label.type(torch.FloatTensor)
output=output.type(torch.FloatTensor)
</code></pre>

Mike2004: <p>Ok, thank you, i made the alteration that you suggest. I´m stuck here now. I get this errors:</p>
<p>File “C:/Users/user/.spyder-py3/original_cnn.py”, line 112, in <br>
loss = criterion(label,torch.argmax(output,dim=1))</p>
<p>File “C:\Users\user\Anaconda3\lib\site-packages\torch\nn\modules\module.py”, line 541, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)</p>
<p>File “C:\Users\user\Anaconda3\lib\site-packages\torch\nn\modules\loss.py”, line 431, in forward<br>
return F.mse_loss(input, target, reduction=self.reduction)</p>
<p>File “C:\Users\user\Anaconda3\lib\site-packages\torch\nn\functional.py”, line 2204, in mse_loss<br>
ret = torch._C._nn.mse_loss(expanded_input, expanded_target, _Reduction.get_enum(reduction))</p>
<p>RuntimeError: Expected object of scalar type Float but got scalar type Long for argument <span class="hashtag">#2</span> ‘target’ in call to _thnn_mse_loss_forward</p>

jmandivarapu1: <aside class="quote no-group" data-username="Mike2004" data-post="15" data-topic="81228">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mike2004/48/24981_2.png" class="avatar"> Mike2004:</div>
<blockquote>
<p>torch.argmax(output,dim=1)</p>
</blockquote>
</aside>
<p>It’s saying that this output <code>torch.argmax(output,dim=1)</code> is still long. can you print those tensors at the end you can actually see the type of the tensor or <code>type(torch.argmax(output,dim=1))</code></p>

Mike2004: <p>If you are saying me to do this:<br>
print(type(torch.argmax(output,dim=1)))<br>
print(type(label))</p>
<p>The output is: &lt;class ‘torch.Tensor’&gt;<br>
&lt;class ‘torch.Tensor’&gt;</p>
<p>Is this what you told me to do?</p>

jmandivarapu1: <aside class="quote no-group" data-username="Mike2004" data-post="7" data-topic="81228">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mike2004/48/24981_2.png" class="avatar"> Mike2004:</div>
<blockquote>
<p>criterion = nn.MSELoss()</p>
</blockquote>
</aside>
<p>The below is the best I came up to reproduce your error. Adding <code>.type(torch.FloatTensor)</code> should fix your error.</p>
<pre><code class="lang-auto">label=torch.Tensor([0,1,2])
output=torch.Tensor(3,3)
criterion(label,torch.argmax(output,dim=1).type(torch.FloatTensor))

Output : tensor(1.)
</code></pre>
<p>If you remove that then it’s your error<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/8/486aa4182469c140618bad0208fa590212fe3047.png" data-download-href="https://discuss.pytorch.org/uploads/default/486aa4182469c140618bad0208fa590212fe3047" title="Screen Shot 2020-05-17 at 12.07.03 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/8/486aa4182469c140618bad0208fa590212fe3047_2_690x493.png" alt="Screen Shot 2020-05-17 at 12.07.03 PM" data-base62-sha1="akCR1bsiyHodWZ8EosJPpHkoJMj" width="690" height="493" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/8/486aa4182469c140618bad0208fa590212fe3047_2_690x493.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/8/486aa4182469c140618bad0208fa590212fe3047_2_1035x739.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/8/486aa4182469c140618bad0208fa590212fe3047_2_1380x986.png 2x" data-dominant-color="F3F6F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2020-05-17 at 12.07.03 PM</span><span class="informations">1862×1332 365 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Mike2004: <p>Thank you for your patient and time, you are teaching me a lot. After those changes i run my code and then i get the following error:</p>
<p>File “C:/Users/user/.spyder-py3/original_cnn.py”, line 115, in <br>
loss.backward()</p>
<p>File “C:\Users\user\Anaconda3\lib\site-packages\torch\tensor.py”, line 166, in backward<br>
torch.autograd.backward(self, gradient, retain_graph, create_graph)</p>
<p>File “C:\Users\user\Anaconda3\lib\site-packages\torch\autograd_<em>init</em>_.py”, line 99, in backward<br>
allow_unreachable=True)  # allow_unreachable flag</p>
<p>RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn</p>
<p>Loop code:</p>
<pre><code class="lang-python">for epoch in range(num_epochs):
   for data in train_loader: 
       img, label= data
       label = Variable(label).cuda()
       img = Variable(img).cuda()
       nova=to_img(img)
       
       st = time.time()

       optimizer.zero_grad()

       output = model(img)

       label=torch.Tensor([0,1,2])
       output=torch.Tensor(3,3)
       loss=criterion(label,torch.argmax(output,dim=1).type(torch.FloatTensor))

       loss.backward()

       optimizer.step()

       iter += 1
</code></pre>
<p>I searched about this and tried a few changes in my code, like loss.backward(requires_grad=True) but still not working: TypeError: backward() got an unexpected keyword argument ‘requires_grad’<br>
Or this : label.requires_grad=True<br>
(torch.argmax(output,dim=1).type(torch.FloatTensor)).requires_grad=True</p>
<p>But i can´t find a solution, can you help me in this?</p>

Isaac_Kargar: <p>I modified your code to this in the for loop:</p>
<pre><code class="lang-auto">        label=torch.tensor([[0, 0, 0], [0, 1, 0], [0, 0, 2]])
		output=torch.randn(3, 3, requires_grad=True)
		loss=criterion(output, label.float())
</code></pre>
<p>And it works.</p>
<p>If you want to do classification you need to use cross entropy loss. Remember to check shape of tensors you pass into the loss. The complete code for mnist is this:</p>
<pre><code class="lang-auto">import time
import torch
import torch.nn as nn
import torchvision.transforms as transforms
import torchvision.datasets as dsets
from torch.autograd import Variable
from torchvision.utils import save_image

GPU = 0

def to_img(x):
	x = x.view(x.size(0), 1, 28, 28)
	return x

train_dataset = dsets.MNIST(root='./data',
	train=True,
	transform=transforms.ToTensor(),
	download=True)

test_dataset = dsets.MNIST(root='./data',
	train=False,
	transform=transforms.ToTensor())

batch_size =64
n_iters = 71800

num_epochs = n_iters / (len(train_dataset) / batch_size)
num_epochs = int(num_epochs)

train_loader = torch.utils.data.DataLoader(dataset=train_dataset,
		batch_size=batch_size,
		shuffle=True)

test_loader = torch.utils.data.DataLoader(dataset=test_dataset,
		batch_size=batch_size,
		shuffle=False)

class CNNModel(nn.Module):
	def  __init__(self):
		super().__init__()
		self.cnn1 = nn.Conv2d(in_channels=1, out_channels=16, kernel_size=5, stride=1, padding=2)
		self.relu1 = nn.ReLU()
		self.maxpool1 = nn.MaxPool2d(kernel_size=2)
		
		self.cnn2 = nn.Conv2d(in_channels=16, out_channels=32, kernel_size=5, stride=1, padding=2)
		self.relu2 = nn.ReLU()
		
		self.maxpool2 = nn.MaxPool2d(kernel_size=2)
		
		self.fc1 = nn.Linear(32 * 7 * 7, 10)

	def forward(self, x):
	    out = self.cnn1(x)
	    out = self.relu1(out)
	   
	    out = self.maxpool1(out)
	   
	    out = self.cnn2(out)
	    out = self.relu2(out)

	    out = self.maxpool2(out)

	    out = out.view(out.size(0), -1)

	    out = self.fc1(out)
	   
	    return out


model = CNNModel()

if torch.cuda.is_available():
	torch.cuda.set_device(GPU)
	model.cuda()
	print("cuda OK")

criterion = nn.CrossEntropyLoss()

best= 12311242

learning_rate = 1e-3

optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate, weight_decay=0)

iter = 0
start = time.time()
for epoch in range(2):
	for data in train_loader:
		img, label= data
		label = label.cuda()
		img = img.cuda()
		# nova=to_img(img)

    	# st = time.time()
		optimizer.zero_grad()
		output = model(img)
		
		# label=torch.tensor([[0, 0, 0], [0, 1, 0], [0, 0, 2]])
		# output=torch.randn(3, 3, requires_grad=True)

		loss=criterion(output, label.long())
		loss.backward()
    	
		optimizer.step()
		iter += 1

print('done')
</code></pre>

Mike2004: <p>Thank you for your time, could you just explain me more, about this:</p>
<pre><code class="lang-python"> label=torch.tensor([[0, 0, 0], [0, 1, 0], [0, 0, 2]])
		output=torch.randn(3, 3, requires_grad=True)
		loss=criterion(output, label.float())
</code></pre>

Isaac_Kargar: <p>For cross entropy loss you need two tensors with the same shape. I just converted the label to one hot vector.</p>