GPU based CNN training

username: achyut_mani_Tripathi
cooked: <p>Dear All,</p>
<h2>I am new to pytorch and training my model with CNN and attention mechanism. The model works fine when run on CPU but gives training and validation accuracy as 0, when code runs on GPU. I tried to solve the problem, but unable to do so. Here is my  code</h2>
<p>#-------------------------------------------------&gt;Import all the required libraries--------------------------------------&gt;<br>
import torch # Import Torch<br>
import torch.nn as nn # Import NN module from Torch<br>
from torchvision.datasets import CIFAR10 # Import CIFAR10 datset from torchvision<br>
from torchvision.transforms import transforms# Import transform module from torchvision<br>
from torch.utils.data import DataLoader # Import dataloader from torch<br>
from torch.optim import Adam # import optimizer module from torch<br>
from torch.autograd import Variable # Import autograd from torch<br>
import numpy as np # Import numpy module<br>
import torchvision.datasets as datasets <span class="hashtag">#Import</span> dataset from torch<br>
from torchvision import models # import pretrained models from torch<br>
from Attention import PAM_Module # import position attention module<br>
from Attention import CAM_Module # import channel attention module<br>
from torch import optim, cuda # import optimizer<br>
import os<br>
import random<br>
import torch.nn.functional as F<br>
import random<br>
SEED = 1234<br>
random.seed(SEED)<br>
np.random.seed(SEED)<br>
torch.manual_seed(SEED)<br>
torch.cuda.manual_seed(SEED)<br>
torch.backends.cudnn.deterministic = True<br>
device = torch.device(‘cuda’)</p>
<p>Training_Path="/home/mani/Desktop/Binary Attention Based ASC/DCASE 2018 Dataset/Training" # path to folder contains traning images<br>
Test_Path="/home/mani/Desktop/Binary Attention Based ASC/DCASE 2018 Dataset/Test" # path to folder contains test images<br>
#----------------------------------------------&gt; Define Training Transformation ---------------------------------------&gt;<br>
train_transformations = transforms.Compose([<br>
<span class="hashtag">#transforms</span>.RandomHorizontalFlip(),<br>
<span class="hashtag">#transforms</span>.RandomCrop(32,padding=4),<br>
transforms.ToTensor(),<br>
transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])])<br>
#----------------------------------------------&gt; Define Test Transformation --------------------------------------------&gt;<br>
test_transformations = transforms.Compose([<br>
<span class="hashtag">#transforms</span>.RandomHorizontalFlip(),<br>
<span class="hashtag">#transforms</span>.RandomCrop(32,padding=4),<br>
transforms.ToTensor(),<br>
transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])])<br>
Validation_transformations = transforms.Compose([<br>
<span class="hashtag">#transforms</span>.RandomHorizontalFlip(),<br>
<span class="hashtag">#transforms</span>.RandomCrop(32,padding=4),<br>
transforms.ToTensor(),<br>
transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])])<br>
#----------------------------------------------&gt; Define and Load Training and Test Set ----------------------------------&gt;<br>
<span class="hashtag">#Load</span> the training set<br>
train_set =datasets.ImageFolder(root=Training_Path,transform=train_transformations)<br>
validation_set =datasets.ImageFolder(root=Training_Path,transform=Validation_transformations)<br>
<span class="hashtag">#print</span>(train_set) # prints dettails of training set along with number of images, type of transformation etc.<br>
test_set =datasets.ImageFolder(root=Training_Path,transform=test_transformations)<br>
<span class="hashtag">#print</span>(test_set) # prints dettails of test set along with number of images, type of transformation etc.<br>
BATCH_SIZE=32 # Define the batch size<br>
#----------------------------------------------&gt; Use dataloader to create batches of data --------------------------------&gt;<br>
train_loader = DataLoader(train_set,batch_size=BATCH_SIZE,shuffle=True)<span class="hashtag">#Create</span> a loder for the training set<br>
test_loader = DataLoader(test_set,batch_size=BATCH_SIZE,shuffle=True)<span class="hashtag">#Create</span> a loder for the test set<br>
validation_loader = DataLoader(validation_set,batch_size=BATCH_SIZE,shuffle=True)<span class="hashtag">#Create</span> a loder for the test set<br>
<span class="hashtag">#print</span>(len(train_loader)) # Given total number of iterations required to complete one training epoch<br>
#----------------------------------------------&gt; Check of GPU and get the pretrained model -----------------------------------------&gt;<br>
device = torch.device(“cuda” if torch.cuda.is_available()<br>
else “cpu”)<br>
print(device)<br>
<span class="hashtag">#model</span> = models.resnet18(pretrained=True) # Import the resnet model<br>
<span class="hashtag">#print</span>(model)<br>
num_classes=10</p>
<p>class BACNN(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(BACNN, self).<strong>init</strong>()<br>
Pre_Trained_Layers = list(models.resnet18().children())[:-2] <span class="hashtag">#all</span> layer expect last layer<br>
print(“pre trained layers after removing the top layers----------------------&gt;”+’\n’)<br>
<span class="hashtag">#print</span>(Pre_Trained_Layers)<br>
<span class="hashtag">#self</span>.count=count<br>
self.features=nn.Sequential(*Pre_Trained_Layers)<br>
self.PAM=PAM_Module(512)<br>
self.CAM=CAM_Module(512)<br>
self.conv1 = nn.Conv2d(512,10,3,bias=True,padding=0)<br>
self.fc1=nn.Linear(18, num_classes)<br>
for p in self.features.parameters():<br>
p.requires_grad=False<br>
<span class="hashtag">#print</span> (self.features) 	<br>
def forward(self, image):<br>
x = self.features(image)<br>
x=F.relu(x)<br>
x_1=self.PAM(x)<br>
x_2=self.CAM(x)<br>
x_3=x_1+x_2<br>
x=self.conv1(x_3)<br>
<span class="hashtag">#print</span>(a)<br>
x = F.relu(x)<br>
x=self.fc1(x)<br>
x= x.view(x.size(0),-1)<br>
<span class="hashtag">#print</span>(x.size)<br>
#count=count+1<br>
<span class="hashtag">#print</span>(count)<br>
<span class="hashtag">#print</span>(“Completed”)<br>
return x</p>
<p>model=BACNN().to(device)<br>
<span class="hashtag">#optimizer</span> = optim.Adam(model.parameters())<br>
optimizer = optim.SGD(model.parameters(), lr=0.001, momentum=0.9)</p>
<h1>Define the cross entropy loss function</h1>
<p>criterion = nn.CrossEntropyLoss()<br>
def calculate_accuracy(fx, y):<br>
preds = fx.max(1, keepdim=True)[1]<br>
correct = preds.eq(y.view_as(preds)).sum()<br>
acc = correct.float()/preds.shape[0]<br>
return acc<br>
def train(model,device,iterator, optimizer, criterion):<br>
print(“Training Starts”)<br>
epoch_loss = 0<br>
epoch_acc = 0<br>
count=0<br>
model.train()</p>
<pre><code>for (x, y) in iterator:
	x=x.to(device)
	y=y.to(device)
	optimizer.zero_grad()
	count=count+1
	Predicted_Train_Label = model(x)
	loss = criterion(Predicted_Train_Label, y)
	acc = calculate_accuracy(Predicted_Train_Label, y)
	print("Training Iteration Number=",count)
	loss.backward()
	optimizer.step()
	epoch_loss += loss.item()
	epoch_acc += acc.item()
return epoch_loss / len(iterator), epoch_acc / len(iterator)
</code></pre>
<p>def evaluate(model,device,iterator, criterion):<br>
print(“Validation Starts”)<br>
epoch_loss = 0<br>
epoch_acc = 0<br>
count=0<br>
model.eval()</p>
<pre><code>with torch.no_grad():
    for (x, y) in iterator:
    	x=x.to(device)
    	y=y.to(device)
    	count=count+1
    	Predicted_Label = model(x)
    	loss = criterion(Predicted_Label, y)
    	acc = calculate_accuracy(Predicted_Label, y)
    	print("Validation Iteration Number=",count)
    	epoch_loss += loss.item()
    	epoch_acc += acc.item()
    
return epoch_loss / len(iterator), epoch_acc / len(iterator) 
</code></pre>
<p>EPOCHS = 250<br>
SAVE_DIR = ‘models’<br>
MODEL_SAVE_PATH = os.path.join("/home/mani/Desktop/Binary Attention Based ASC/", ‘BACNN.pt’)</p>
<p>best_valid_loss = float(‘inf’)</p>
<p><span class="hashtag">#if</span> not os.path.isdir(f’{SAVE_DIR}’):<br>
<span class="hashtag">#os</span>.makedirs(f’{SAVE_DIR}’)<br>
<span class="hashtag">#print</span>(model)<br>
for epoch in range(EPOCHS):<br>
print(“Start Training and Validation For Epoch Number=”,epoch)<br>
train_loss, train_acc = train(model,device,train_loader, optimizer, criterion)<br>
valid_loss, valid_acc = evaluate(model,device,validation_loader,criterion)<br>
if valid_loss &lt; best_valid_loss:<br>
best_valid_loss = valid_loss<br>
torch.save(model.state_dict(), MODEL_SAVE_PATH)<br>
print(“Epoch Number=”,epoch,“Train Loss=”,train_loss,“Training Accuracy=”,train_acc,’\n’)<br>
print(“Epoch Number=”,epoch,“Validation Loss=”,valid_loss,“Validation Accuracy=”,valid_acc,’\n’)</p>
<pre><code>--------------------------------------------------------------------------------------------------------------------------------
</code></pre>
<p>Please see the code and suggest the modifications that will solve the problem.</p>
<p>Thanks<br>
Achyut</p>

username: charan_Vjy
cooked: <ol>
<li>What was the training and validation accuracy when run on the cpu?</li>
<li>Also, print out the loss in both the cpu and gpu . A different behavior in the cpu and gpu is actually very interesting given that most operations are the same.</li>
</ol>

username: achyut_mani_Tripathi
cooked: <p>For CPU training accuracy and validation accuracy go like<br>
Epoch 1 (Training accuracy=23.45, Validation Accuracy=24.38)<br>
.<br>
.<br>
.<br>
Epoch 20 (Training Accuracy=55.67, Validation Accuracy=57.29)</p>
<p>But in GPU both the values come as 0 for each epoch.</p>
<p>Thanks<br>
Achyut</p>

username: charan_Vjy
cooked: <p>What about the loss in both the instances?</p>

username: achyut_mani_Tripathi
cooked: <p>Loss  decreases in case of CPU but remain same in case of GPU.</p>
<p>Following is an Example of loss coming in GPU<br>
Epoch Number= 0 Train Loss= 7.179644451865667 Training Accuracy= 0.004746835443037975<br>
Epoch Number= 0 Validation Loss= 7.178059632265115 Validation Accuracy= 0.0</p>
<p>Epoch Number= 1 Train Loss= 7.177682260923747 Training Accuracy= 0.0</p>
<p>Epoch Number= 1 Validation Loss= 7.176880444152446 Validation Accuracy= 0.0</p>
<p>Epoch Number= 2 Train Loss= 7.176005357428442 Training Accuracy= 0.0</p>
<p>Epoch Number= 2 Validation Loss= 7.17541601688047 Validation Accuracy= 0.0<br>
.<br>
.<br>
.</p>
<p>Thanks<br>
Achyut</p>

username: charan_Vjy
cooked: <p>Since you seem to have ensured deterministicity, try tracking the model ouputs when run on either device for the first epoch. My hunch says, they both should be the same (only the first epoch).</p>

username: achyut_mani_Tripathi
cooked: <p>Ok will see, Thanks!</p>