I don't know why my model's loss will not decrease on GPU

username: shazhongcheng
cooked: <p>i am a new user of pytorch. During my study, i found my model work on CPU, but when i try to train it on<br>
GPU, it does<code>t work, and i don</code>t know why…  Can anybody help me?<br>
Following issues:</p>
<pre><code class="lang-python">class Net(nn.Module):
    def __init__(self):
        super(Net,self).__init__()
        self.conv1=nn.Conv2d(3,6,5)
        self.conv2=nn.Conv2d(6,16,5)
        self.fc1=nn.Linear(16*5*5,120)
        self.fc2=nn.Linear(120,84)
        self.fc3=nn.Linear(84,10)
    
    def forward(self,x):
        x=F.max_pool2d(F.relu(self.conv1(x)),(2,2))
        x=F.max_pool2d(F.relu(self.conv2(x)),2)
        x=x.view(x.size()[0],-1)
        x=F.relu(self.fc1(x))
        x=F.relu(self.fc2(x))
        x=self.fc3(x)
        return x
net=Net()
print(net)

for epoch in range(2):
    running_loss=0.0
    for i,data in enumerate(trainloader,0):
        inputs,labels=data
        inputs,labels=Variable(inputs),Variable(labels)
        
        optimizer.zero_grad()
        
        outputs=net(inputs)
        loss=criterion(outputs,labels)
        loss.backward()
        optimizer.step()
        
        running_loss+=loss.item()
        if i%2000==1999:
            print('[%5d %5d] loss: %.3f'%(epoch+1,i+1,running_loss/2000))
            running_loss=0.0
print('Finished Training')
</code></pre>
<p>It train work,and results:</p>
<pre><code class="lang-auto">[    1  2000] loss: 2.198
[    1  4000] loss: 1.879
[    1  6000] loss: 1.690
[    1  8000] loss: 1.579
[    1 10000] loss: 1.514
[    1 12000] loss: 1.476
[    2  2000] loss: 1.405
[    2  4000] loss: 1.384
[    2  6000] loss: 1.340
[    2  8000] loss: 1.352
[    2 10000] loss: 1.296
[    2 12000] loss: 1.302
Finished Training
</code></pre>
<p>but when on GPU:</p>
<pre><code class="lang-python">device=t.device("cuda:0"if t.cuda.is_available() else "cpu")
net=net.to(device)
for epoch in range(2):
    running_loss=0.0
    for i,data in enumerate(trainloader,0):
        inputs,labels=data
        inputs, labels= inputs.to(device), labels.to(device)
        
        optimizer.zero_grad()
        
        outputs=net(inputs)
        loss=criterion(outputs,labels)
        loss.backward()
        optimizer.step()
        
        running_loss+=loss.item()
        if i%2000==1999:
            print('[%5d %5d] loss: %.3f'%(epoch+1,i+1,running_loss/2000))
            running_loss=0.0
print('Finished Training')
</code></pre>
<p>It doesn`t work,and results:</p>
<pre><code class="lang-auto">[    1  2000] loss: 2.304
[    1  4000] loss: 2.305
[    1  6000] loss: 2.306
[    1  8000] loss: 2.304
[    1 10000] loss: 2.304
[    1 12000] loss: 2.304
[    2  2000] loss: 2.305
[    2  4000] loss: 2.305
[    2  6000] loss: 2.305
[    2  8000] loss: 2.305
[    2 10000] loss: 2.303
[    2 12000] loss: 2.305
Finished Training
</code></pre>
<p>And i don`t know why.</p>

username: phan_phan
cooked: <p>It seems that in your cpu code you use <code>Variable()</code>, but you don’t use it in your gpu code. This might be the source of the problem.</p>
<p>Which version of PyTorch do you use ? <code>Variable()</code> is deprecated since v0.4</p>

username: shazhongcheng
cooked: <p>Thank you! I will try.</p>
<p>My version is 1.0.0</p>

username: mailcorahul
cooked: <p><a class="mention" href="/u/phan_phan">@phan_phan</a> Why do we need gradients for inputs and labels?</p>

username: phan_phan
cooked: <p><a class="mention" href="/u/mailcorahul">@mailcorahul</a>  <code>Variable</code> did not necessarily mean <code>requires_grad=True</code>.<br>
<a class="mention" href="/u/shazhongcheng">@shazhongcheng</a> But as you are working with v1.0.0, I suggest you get rid of it in the cpu code.</p>
<p>I realize I didn’t answer at all your main question which was about you gpu code … sorry.</p>
<p>In your gpu code, you have :</p>
<aside class="quote no-group" data-username="shazhongcheng" data-post="1" data-topic="55212">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/e79b87/48.png" class="avatar"> shazhongcheng:</div>
<blockquote>
<p>net=net.to(device)</p>
</blockquote>
</aside>
<p>Does this mean you defined <code>net</code> before ?<br>
You could try to define <code>optimizer</code> on <code>net.parameters()</code> <strong>after</strong> calling <code>net=net.to(device)</code>.</p>
<p>If this still doesn’t work, show us your whole code ! So that we can see what’s what.</p>

username: mailcorahul
cooked: <aside class="quote no-group" data-username="phan_phan" data-post="5" data-topic="55212">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/phan_phan/48/51960_2.png" class="avatar"> phan_phan:</div>
<blockquote>
<p><a class="mention" href="/u/mailcorahul">@mailcorahul</a> <code>Variable</code> did not necessarily mean <code>requires_grad=True</code> .</p>
</blockquote>
</aside>
<p>Okay.</p>
<aside class="quote no-group" data-username="phan_phan" data-post="5" data-topic="55212">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/phan_phan/48/51960_2.png" class="avatar"> phan_phan:</div>
<blockquote>
<p>You could try to define <code>optimizer</code> on <code>net.parameters()</code> <strong>after</strong> calling <code>net=net.to(device)</code> .</p>
</blockquote>
</aside>
<p>Yes, I tried doing this. Seems to work fine both ways(which is not intuitive).<br>
I guess it still isn’t the cause.</p>

username: shazhongcheng
cooked: <p>Thank you！ you are really kind to reply me! Although i try use .cuda replace Variable and .to(device), it doesn`t work. But when I change my optimizer form SGD to Adam, the loss decrease ! And i am really really Happy!<br>
But my most happy thing is you can reply me. It make me fell warm.<br>
Last,Thank you! I finished it!<br>
I know, maybe the learning_rate is to small, but it work on cpu…so strange.<br>
my net(i run on jupyter):</p>
<pre><code class="lang-python">class Net(nn.Module):
    def __init__(self):
        super(Net,self).__init__()
        self.conv1=nn.Conv2d(3,6,5)
        self.conv2=nn.Conv2d(6,16,5)
        self.fc1=nn.Linear(16*5*5,120)
        self.fc2=nn.Linear(120,84)
        self.fc3=nn.Linear(84,10)
    
    def forward(self,x):
        x=F.max_pool2d(F.relu(self.conv1(x)),(2,2))
        x=F.max_pool2d(F.relu(self.conv2(x)),2)
        x=x.view(x.size()[0],-1)
        x=F.relu(self.fc1(x))
        x=F.relu(self.fc2(x))
        x=self.fc3(x)
        return x
net=Net()

if t.cuda.is_available():
    net=net.cuda()

from torch import optim
criterion=nn.CrossEntropyLoss()
optimizer=optim.Adam(net.parameters(),lr=0.001)

for epoch in range(2):
    running_loss=0.0
    for i,data in enumerate(trainloader,0):
        inputs,labels=data
        if t.cuda.is_available():
            inputs=inputs.cuda()
            labels=labels.cuda()
        
        outputs=net(inputs)
        loss=criterion(outputs,labels)
        
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        
        running_loss+=loss.item()
        if (i+1)%2000==1999:
            print('[%5d %5d] loss: %.3f'%(epoch+1,i+1,running_loss/2000))
            running_loss=0.0
print('Finished Training')

correct=0
total=0
for data in testloader:
    images,labels=data
    with t.no_grad():
        outputs=net(images.cuda()).cpu()
    _,predicted=t.max(outputs.data,1)
    total+=labels.size(0)
    correct+=(predicted==labels).sum()
print('test acc:%d %%'%(100*correct/total))
</code></pre>
<p>and results:</p>
<pre><code class="lang-auto">[    1  1999] loss: 1.874
[    1  3999] loss: 1.605
[    1  5999] loss: 1.514
[    1  7999] loss: 1.458
[    1  9999] loss: 1.422
[    1 11999] loss: 1.400
[    2  1999] loss: 1.316
[    2  3999] loss: 1.298
[    2  5999] loss: 1.280
[    2  7999] loss: 1.283
[    2  9999] loss: 1.274
[    2 11999] loss: 1.242
Finished Training

test acc:57 %
</code></pre>

username: shazhongcheng
cooked: <p>Thank you! I have finished it! My solutions have replied.<br>
Last,reall thank you!</p>

username: mailcorahul
cooked: <p>Are you saying the same code with SGD instead of Adam optimizer doesn’t work?</p>

username: AlittleBrave
cooked: <p>Thanks. Define optimizer after net.to(device) works.</p>