CUDA alignment error when using DataParallel

Kalamaya: self.conv1 = nn.Conv2d(N.inputChannels, N.outputChannels, N.kernelSquareSize, stride = (1,1), padding = (1,1)); and now, as per the DataParallel documentation, I have: self.conv1 = nn.Conv2d(N.inputChannels, N.outputChannels, N.kernelSquareSize, stride = (1,1), padding = (1,1)); self.conv1 = torch.nn.DataParallel(self.conv1, device_ids = [1, 2]) With this new code however, I am unable to get it to run, and I get the following error: And it is occurring during the (attempted) forward prop in my code… thanks…

apaszke: Can you please give us the parameters of the conv so we can try to reproduce the issue?

Kalamaya: Sure, here is my complete snippet: <SCODE> class Net(nn.Module):
  def __init__(self):
    super(Net, self).__init__()
         
    self.bn1 = nn.BatchNorm2d(8)
    self.bn2 = nn.BatchNorm2d(16)

    self.conv1 = nn.Conv2d(1, 8, 3 ,stride = (1,1), padding = (1,1))
    self.conv1 = torch.nn.DataParallel(self.conv1, device_ids = [1, 2])           
    self.conv2 = nn.Conv2d(8, 16 ,3, stride = (1,1), padding = (1,1))
    //etc ...

  def forward_prop(self, x):
 
    x = F.max_pool2d(F.relu(self.bn1(self.conv1(x))), (2,2))
    x = F.max_pool2d(F.relu(self.bn2(self.conv2(x))), (2,2))    
<ECODE> This is my only change. I will also mention that if I simply remove the torch.nn.DataParallel line in the above, my code runs and trains fine. Thanks,

apaszke: 

Kalamaya: For this question, I am not sure what the relevance is as far as the name goes?.. Perhaps I am missing something - but to be honest I had defined it before in my class and it worked - perhaps I am missing something deeper here? It’s just a name of the forward propagation function that I give… This one is a 100x100 image, single channel, minibatch size of 16.

fmassa: 

apaszke: Ah right. The problem is that we’re numbering GPUs starting from 0. So all the modules and data is on GPU0, but you’re telling the DataParallel to run on GPU1 and GPU2 (i.e. 2nd and 3rd GPU). Can you change that and see if it helps? If that’s it, then we need to improve the error message.

Kalamaya: <SCODE>class Net(nn.Module):
 
  def __init__(self):
    super(Net, self).__init__()

    # Define the network
    self.bn1 = nn.BatchNorm2d(8)
    self.bn2 = nn.BatchNorm2d(16)        
    self.conv1 = nn.Conv2d(1, 8, 3, stride = (1,1), padding = (1,1))
    self.conv2 = nn.Conv2d(8, 16, 3, stride = (1,1), padding = (1,1))    
    self.conv3 = nn.Conv2d(16, 32, 2, stride = (1,1), padding = (0,0))
    self.conv4 = nn.Conv2d(32, 64, 3, stride = (1,1), padding = (1,1))
    self.conv5 = nn.Conv2d(64, 64, 3, stride = (3,3), padding = (0,0))
    self.fc1   = nn.Linear(256, 32) 
    self.fc2   = nn.Linear(32, 16)
    self.fc3   = nn.Linear(16, 2)

  def forward_prop(self, x):
    
    # Conv1 with batch norm
    x = F.max_pool2d(F.relu(self.bn1(self.conv1(x))), (2,2))

    # Conv2 with batch norm.
    x = F.max_pool2d(F.relu(self.bn2(self.conv2(x))), (2,2))    

    # Conv3
    x = F.max_pool2d(F.relu(self.conv3(x)), (2,2))    

    # Conv4
    x = F.max_pool2d(F.relu(self.conv4(x)), (2,2))

    # Conv5
    x = F.relu(self.conv5(x))
    
    # Flatten the feature map thus far for use in the fully connected layer.
    x = x.view(-1, self.num_flat_features(x))
        
    # Fully connected 1    
    x = F.relu(self.fc1(x))

    # Fully connected 2    
    x = F.relu(self.fc2(x))

    # Final layer
    x = self.fc3(x)

    return x

  def num_flat_features(self, x):    
    # all dimensions except the batch dimension
    size = x.size()[1:] 
    num_features = 1
    for s in size:
        num_features *= s
    return num_features
<ECODE> So real quick, in the rest of my file, I basically have: <SCODE>net = Net().cuda()
net.forward_prop(trainingBatch)
<ECODE> So instead I wrote a simple net (as above), and would like to use the DataParallel capability here. What would I do differently on this setup as above? thanks again!

Kalamaya: 

apaszke: So the problem was 1 vs 0-based GPU indexing right? We need to fix that, it should never give you an invalid memory access.

Kalamaya: <SCODE>myNet = Net()
myNet = torch.nn.DataParallel(myNet, device_ids=[0,1])
myNet.cuda()
output = myNet(input)
<ECODE> Is this correct? However there is a subtlety that still confuses me: In the DCGAN example, we have: 

apaszke: