Error training distributed Model

Fernando_Gallego: <p>Hi, I am working to distribute the layers a neural network like alexNet in three devices (Edge, Fog, and Cloud), sending the results of the inferences to the other device. I’m currently trying to train it as a single model and then break it down into sub-models.</p>
<p>The neural network has the following model:</p>
<pre><code class="lang-auto"># AlexNet
# EDGE MODEL

edge_layers1 = nn.Sequential(nn.Conv2d(3, 64, kernel_size=3, stride=2, padding=2),nn.ReLU(inplace=True), nn.MaxPool2d(kernel_size=3, stride=2))
edge_layers2 = nn.Sequential(nn.Linear(8, 10), nn.ReLU(inplace=True))


# FOG MODEL

fog_layers1 = nn.Sequential(nn.Conv2d(64, 192, kernel_size=3, padding=2), nn.ReLU(inplace=True), nn.MaxPool2d(kernel_size=3, stride=2))
fog_layers2 = nn.Sequential(nn.Linear(4, 10), nn.ReLU(inplace=True))

# CLOUD MODEL 

cloud_layers1 = nn.Sequential(nn.Conv2d(192, 384, kernel_size=3, padding=1), nn.ReLU(inplace=True), nn.Conv2d(384, 512, kernel_size=3, padding=1), nn.ReLU(inplace=True), 
                              nn.Conv2d(512, 4096, kernel_size=3, padding=1), nn.ReLU(inplace=True))
cloud_layers2 = nn.Sequential(nn.Linear(4, 1024), nn.ReLU(inplace=True), nn.Dropout(), nn.Linear(1024, 4096), nn.ReLU(inplace=True), nn.Linear(4096, 10))


class AlexNet_Distributed(nn.Module):

    def __init__(self, num_classes=10):
        super(AlexNet_Distributed, self).__init__()
        self.layers1 = edge_layers1
        self.layers2 = edge_layers2
        self.layers3 = fog_layers1
        self.layers4 = fog_layers2
        self.layers5 = cloud_layers1
        self.layers6 = cloud_layers2

    def forward(self, x):
        x = self.layers1(x)
        y = self.layers2(x)
        
        x = self.layers3(x)
        z = self.layers4(x)
        
        x = self.layers5(x)
        x = self.layers6(x)
        return x
    
net = AlexNet_Distributed()
</code></pre>
<p>After training it with the next implementation, I get this error. Does anyone know why?</p>
<pre><code class="lang-auto">criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)
for epoch in range(2):  # loop over the dataset multiple times

    running_loss = 0.0
    for i, data in enumerate(trainloader, 0):
        # get the inputs; data is a list of [inputs, labels]
        inputs, labels = data

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = net(inputs)
        print(outputs[3].size())
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        # print statistics
        running_loss += loss.item()
        if i % 2000 == 1999:    # print every 2000 mini-batches
            print('[%d, %5d] loss: %.3f' %
                  (epoch + 1, i + 1, running_loss / 2000))
            running_loss = 0.0

print('Finished Training')
</code></pre>
<pre><code class="lang-auto">---&gt; 16         loss = criterion(outputs, labels)
RuntimeError: only batches of spatial targets supported (3D tensors) but got targets of dimension: 1
</code></pre>

mrshenli: <p>Based on discussions in <a href="https://discuss.pytorch.org/t/only-batches-of-spatial-targets-supported-non-empty-3d-tensors-but-got-targets-of-size-1-1-256-256/49134" class="inline-onebox">Only batches of spatial targets supported (non-empty 3D tensors) but got targets of size: : [1, 1, 256, 256]</a>. It might be due to the size of <code>outputs</code> or <code>labels</code>. Could you please print out their sizes?</p>