Torchvision model speed differs inside/outside docker environment

JingyuQian: <p>Hello, I’m trying to make my model (deeplabv3-resnet101) do inference inside a Docker container. I tested the inference code both inside/outside the container but got a significant time difference at <code>model.cpu()</code> call.<br>
Here’s the basic code snippet:</p>
<pre><code class="lang-auto">model=torch.load('path_to_model.pt')
model.eval()
for image in images:
    image = image.reshape(1, 3, 640, 640)
    res = model(torch.from_numpy(image).type(torch.cuda.FloatTensor) / 255.0)
    pred = res['out'].cpu().detach().numpy()[0][0]
    # do other stuff...
</code></pre>
<ul>
<li>Docker base: <code>pytorch/pytorch:1.6.0-cuda10.1-cudnn7-runtime</code>
</li>
<li>GPU: GeForce RTX 2080Ti</li>
<li>I made sure model inference happens on <code>cuda:0</code>. So the inference time inside/outside docker is almost the same</li>
<li>Calling <code>res['out'].cpu()</code> (a [1,1,640,640] tensor) takes 0.08s outside container and 10s inside container. I tried adding <code>torch.cuda.sychronize()</code> as in <a href="https://discuss.pytorch.org/t/copy-tensor-from-cuda-to-cpu-is-too-slow/13056/2">here</a> but didn’t make a difference.</li>
</ul>
<p>So what could be the reason causing this difference and how do I solve it? Thanks!</p>

ptrblck: <p>Could you share the profiling code, please?<br>
You’ve already mentioned that you are synchronizing the code, but I would like to make sure you are synchronizing it before starting and stopping the timer (not only in the latter case), as it would report wrong timings.</p>

JingyuQian: <p>Thanks for your reply. Here’s the profiling code:</p>
<pre><code class="lang-auto">model = torch.load('path_to_model.pt')
model.eval()

for image in images:
    image = image.reshape(1, 3, 640, 640)
    start = time.time()
    res = model(torch.from_numpy(img).type(torch.cuda.FloatTensor) / 255)
    end = time.time()
    print(f'Run time: {end - start}')     # About 0.013s

    t0 = time.time()
    torch.cuda.synchronize()
    t1 = time.time()
    pred = res['out'].cpu().detach().numpy()[0][0]
    t2 = time.time()
    torch.cuda.synchronize()
    t3 = time.time()
    print(f'First sync time: {t1 - t0}')             # About 10s inside container, 0.08s outside
    print(f'CPU copy time: {t2 - t1}')               # About 6e-4s both in/out
    print(f'Second sync time: {t3 - t2}')          # About 3e-5s both in/out
</code></pre>
<p>Looks like I made a mistake and used <code>torch.cuda.synchronize()</code> in the wrong place previously. Now it seems CPU copy doesn’t take much time, but the first synchronization call takes very long inside the container.</p>

ptrblck: <p>Thanks for the code. You are not synchronizing before each timer, so I would assume your profile might be still wrong.<br>
Synchronize before each timer via:</p>
<pre><code class="lang-python">torch.cuda.synchronize()
tX = time.perf_counter()
</code></pre>

JingyuQian: <p>Thanks for the suggestion. I’ve changed the profiling code to the following:</p>
<pre><code class="lang-auto">### Profiling model run time

torch.cuda.synchronize()
start = time.perf_counter()

res = model(torch.from_numpy(img).type(torch.cuda.FloatTensor) / 255)

torch.cuda.synchronize()
end = time.perf_counter()
print(f'Run time: {end - start}')

### Profiling GPU-CPU copy time
torch.cuda.synchronize()
t1 = time.perf_counter()

pred = res['out'].cpu().detach().numpy()[0][0]
                    
torch.cuda.synchronize()
t2 = time.perf_counter()
print(f'CPU copy time: {t2 - t1}')
</code></pre>
<p>The result is (inside docker):<br>
Run time: 10.947372228954919<br>
CPU copy time: 0.0007527889683842659</p>
<p>I did another profiling with the following code to see how long the synchronization takes:</p>
<pre><code class="lang-auto">torch.cuda.synchronize()
start = time.perf_counter()

res = model(torch.from_numpy(img).type(torch.cuda.FloatTensor) / 255)

mid = time.perf_counter()

torch.cuda.synchronize()
end = time.perf_counter()
print(f'Run time: {end - start}')
print(f'Second sync call time: {end - mid}')
</code></pre>
<p>And got:<br>
Second sync call time: 10.835095343994908</p>
<p>From my perspective the 2nd <code>torch.cuda.synchronize()</code> is taking a long time in the docker. Do you have any suggestions? Thanks.</p>

ptrblck: <p>Both profiles show the same model runtime as <code>~10s</code>, don’t they?<br>
Note that <code>mid</code> in the second example cannot be used, as you are again not synchronizing.</p>
<aside class="quote no-group" data-username="JingyuQian" data-post="5" data-topic="126894">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jingyuqian/48/40332_2.png" class="avatar"> JingyuQian:</div>
<blockquote>
<p>From my perspective the 2nd <code>torch.cuda.synchronize()</code> is taking a long time in the docker.</p>
</blockquote>
</aside>
<p>I’m not sure, if I understand the question correctly, but the synchronization “takes a long time”, as the previous GPU operations aren’t finished and which is the reason you have to synchronize the code to properly profile the code.</p>

JingyuQian: <p>Yes, I agree with you. As you mentioned in some other posts, GPU calls are asynchronous so syncing forces them to finish.</p>
<p>But back to my first post - I want to know why this 10 seconds happens only in the docker container? <img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=9" title=":thinking:" class="emoji" alt=":thinking:"></p>

ptrblck: <p>Could you post a code snippet including the synchronizations, which shows the 10s execution in the docker and the faster one outside, please?</p>

JingyuQian: <p>Hi, sorry for the late reply. After a discussion with the team  members I decided to ditch Docker. So this problem is not a concern any more. Thanks for you help!</p>