Asynchronous execution of ResNet but not DenseNet

curiousCoder: <p>I’m having difficulty getting a dense-net like architecture to run asynchronously with some CPU tasks (e.g. a data loader). I subsequently noticed that even the pytorch-provided dense-net model has the same problem (code and results below).</p>
<p>The main difference between the layers contained in the two architectures is the amount of residual connectivity and the transition layers. ResNet uses strided convolution as opposed to pooling. So I’m wondering if perhaps it is the pooling causing the issue.</p>
<p>In my example below I’ve rigged the parameters so that the CPU job can be finished in about the same time as the GPU job, and so the run-time with CPU should be minimally impacted (if at all). However, the denseNet example runs in about the run-time of the CPU job + the GPU job, implying that it is not running asynchronously.</p>
<p>Edit: I tried replacing the pooling in my network with a strided convolution and it did not help.</p>
<pre><code class="lang-auto">import torch
import time as tm

batchSize = 48
device = 'cuda:0'
rank = 0
img = torch.rand(batchSize,3,260,192,device=device)
n = 3500
B = torch.rand(n,n, device = 'cpu')
A = torch.rand(n,n, device = device)

model = torch.hub.load('pytorch/vision:v0.6.0', 'resnet50', pretrained=True)
model.to(device)
model.train()
ext = ''
for epoch in range(10):
    epochTime = tm.time()
    t = tm.time()
    sampleCount = 0
    sampleTotal = 0
  
    for i in range(0,45):#adjusted to run in approx 10 seconds per epoch
        
        for q in range(0,4):#adjusted to run in approx 10 seconds per epoch
            out = model(img)

        if(epoch&gt;5):#After 5 epochs start doing CPU activity
            ext = ' with CPU'
            for q in range(0,2):#adjusted to run in approx 10 seconds per epoch
                D = torch.mm(B, B)
    torch.cuda.synchronize()
    print('ResNET Rank %i train epoch %i total time is %f %s' %(rank,epoch + 1,tm.time()-epochTime,ext))

model = torch.hub.load('pytorch/vision:v0.6.0', 'densenet121', pretrained=True)
model.to(device)
model.train()

ext = ''
for epoch in range(10):
    epochTime = tm.time()
    t = tm.time()
    sampleCount = 0
    sampleTotal = 0
  
    for i in range(0,45):#adjusted to run in approx 10 seconds per epoch
        
        for q in range(0,4):#adjusted to run in approx 10 seconds per epoch
            out = model(img)

        if(epoch&gt;5):#After 5 epochs start doing CPU activity
            for q in range(0,2):#adjusted to run in approx 10 seconds per epoch
                ext = ' with CPU'
                D = torch.mm(B, B)
    torch.cuda.synchronize()
    print('DenseNET Rank %i train epoch %i total time is %f %s' %(rank,epoch + 1,tm.time()-epochTime,ext))
</code></pre>
<p>ResNET Rank 0 train epoch 1 total time is 11.131998<br>
ResNET Rank 0 train epoch 2 total time is 9.522481<br>
ResNET Rank 0 train epoch 3 total time is 9.522765<br>
ResNET Rank 0 train epoch 4 total time is 9.518584<br>
ResNET Rank 0 train epoch 5 total time is 9.519885<br>
ResNET Rank 0 train epoch 6 total time is 9.525411<br>
ResNET Rank 0 train epoch 7 total time is 10.564333  with CPU<br>
ResNET Rank 0 train epoch 8 total time is 11.599154  with CPU<br>
ResNET Rank 0 train epoch 9 total time is 10.451248  with CPU<br>
ResNET Rank 0 train epoch 10 total time is 10.926621  with CPU</p>
<p>DenseNET Rank 0 train epoch 1 total time is 8.739759<br>
DenseNET Rank 0 train epoch 2 total time is 8.741329<br>
DenseNET Rank 0 train epoch 3 total time is 8.739949<br>
DenseNET Rank 0 train epoch 4 total time is 8.737826<br>
DenseNET Rank 0 train epoch 5 total time is 8.736841<br>
DenseNET Rank 0 train epoch 6 total time is 8.725820<br>
DenseNET Rank 0 train epoch 7 total time is 15.354975  with CPU<br>
DenseNET Rank 0 train epoch 8 total time is 15.407053  with CPU<br>
DenseNET Rank 0 train epoch 9 total time is 15.394043  with CPU<br>
DenseNET Rank 0 train epoch 10 total time is 15.323986  with CPU</p>