Pytorch 1.6 (and 1.5) is slower than 1.4 using torch.autograd.profiler

Hengck: <p>my profile code is as follows</p>
<pre><code class="lang-auto">    net = torchvision.models.resnext50_32x4d(pretrained=True).cuda()
    net.train()
    with torch.autograd.profiler.profile(use_cuda=True) as prof:
        predict = net(input)
    print(prof)
</code></pre>
<p>code and results can be found at: <a href="https://drive.google.com/open?id=1vyTkqBpQwUvCSnGjjJPdzIzLTEfJF8Sr" rel="nofollow noopener">https://drive.google.com/open?id=1vyTkqBpQwUvCSnGjjJPdzIzLTEfJF8Sr</a></p>
<p>here are the snippet results for pytorch 1.4 and 1.6 :</p>
<pre><code class="lang-auto">...
is_leaf                      0.00%            1.850us          0.00%            1.850us          1.850us          0.00%            1.024us          1.024us          1                []                                   
is_leaf                      0.00%            2.119us          0.00%            2.119us          2.119us          0.00%            2.048us          2.048us          1                []                                   
max_pool2d_with_indices      0.05%            479.919us        0.05%            479.919us        479.919us        0.19%            2.061ms          2.061ms          1                []                                   
---------------------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  -----------------------------------  
Self CPU time total: 1.034s
CUDA time total: 1.097s

accuracy 0.687500

pytorch
	torch.__version__              = 1.6.0.dev20200516
	torch.version.cuda             = 10.2
	torch.backends.cudnn.version() = 7605
		torch.backends.cudnn.benchmark      = False
		torch.backends.cudnn.enabled        = True
		torch.backends.cudnn.deterministic  = False
	torch.cuda.device_count()      = 1
		torch.cuda.get_device_properties()  = _CudaDeviceProperties(name='TITAN X (Pascal)', major=6, minor=1, total_memory=12192MB, multi_processor_count=28)
		torch.cuda.memory_allocated()       = 0 GB
		torch.cuda.memory_reserved()        = 8 GB

</code></pre>
<pre><code class="lang-auto">cudnn_convolution            0.12%            533.067us        0.12%            533.067us        533.067us        0.63%            4.750ms          4.750ms          1                []                                   
add                          0.00%            21.091us         0.00%            21.091us         21.091us         0.00%            9.215us          9.215us          1                []                                   
batch_norm                   0.07%            324.928us        0.07%            324.928us        324.928us        0.23%            1.723ms          1.723ms          1                []                                   
_batch_norm_impl_index       0.07%            318.668us        0.07%            318.668us        318.668us        0.23%            1.721ms          1.721ms          1                []                                   
---------------------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  -----------------------------------  
Self CPU time total: 457.948ms
CUDA time total: 750.666ms

accuracy 0.687500

pytorch
	torch.__version__              = 1.4.0
	torch.version.cuda             = 10.1
	torch.backends.cudnn.version() = 7603
		torch.backends.cudnn.benchmark      = False
		torch.backends.cudnn.enabled        = True
		torch.backends.cudnn.deterministic  = False
	torch.cuda.device_count()      = 1
		torch.cuda.get_device_properties()  = _CudaDeviceProperties(name='TITAN X (Pascal)', major=6, minor=1, total_memory=12192MB, multi_processor_count=28)
		torch.cuda.memory_allocated()       = 0 GB
		torch.cuda.memory_reserved()        = 8 GB


</code></pre>
<p>I also have similar observations for another machine using GTX 1080Ti.<br>
Why is there a huge difference in the results?</p>

ptrblck: <p>Could you please post the slower layers here?<br>
To isolate the issue further, I would also recommend to use the same CUDA and cudnn versions for the different PyTorch versions. Otherwise you are facing a lot of variables, which might affect the performance.</p>

Hengck: <p>Now i change to the same cuda and cudnn version. the results are similar as before. The network is standard torchvision resnet50 model. i also incoude the profiler chrome tracking</p>
<p>here are the results:</p>
<p>using pytorch1.6<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/7/a7732290243da3e1e61dc9f133a851bb203c5ac7.png" data-download-href="https://discuss.pytorch.org/uploads/default/a7732290243da3e1e61dc9f133a851bb203c5ac7" title="1.6_tracing"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/7/a7732290243da3e1e61dc9f133a851bb203c5ac7_2_690x129.png" alt="1.6_tracing" data-base62-sha1="nTkpCiPZmiF5qVobmux0SBDNDdZ" width="690" height="129" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/7/a7732290243da3e1e61dc9f133a851bb203c5ac7_2_690x129.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/7/a7732290243da3e1e61dc9f133a851bb203c5ac7_2_1035x193.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/7/a7732290243da3e1e61dc9f133a851bb203c5ac7_2_1380x258.png 2x" data-dominant-color="D8DBDA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">1.6_tracing</span><span class="informations">1647×308 61.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<pre><code class="lang-auto">output_nr                    0.00%            1.705us          0.00%            1.705us          1.705us          0.00%            2.048us          2.048us          1                []                                   
is_leaf                      0.00%            1.823us          0.00%            1.823us          1.823us          0.00%            1.023us          1.023us          1                []                                   
is_leaf                      0.00%            1.717us          0.00%            1.717us          1.717us          0.00%            1.024us          1.024us          1                []                                   
max_pool2d_with_indices      0.05%            486.468us        0.05%            486.468us        486.468us        0.17%            1.756ms          1.756ms          1                []                                   
---------------------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  -----------------------------------  
Self CPU time total: 987.386ms
CUDA time total: 1.046s

pytorch
	torch.__version__              = 1.6.0.dev20200516+cu101
	torch.version.cuda             = 10.1
	torch.backends.cudnn.version() = 7603
		torch.backends.cudnn.benchmark      = False
		torch.backends.cudnn.enabled        = True
		torch.backends.cudnn.deterministic  = False
	torch.cuda.device_count()      = 1
		torch.cuda.get_device_properties()  = _CudaDeviceProperties(name='TITAN X (Pascal)', major=6, minor=1, total_memory=12192MB, multi_processor_count=28)
		torch.cuda.memory_allocated()       = 0 GB
		torch.cuda.memory_reserved()        = 8 GB
</code></pre>
<p>using pytorch1.4<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/4/44f1d7076b5f473db0c14d7462c2a9a6e6e9b355.png" data-download-href="https://discuss.pytorch.org/uploads/default/44f1d7076b5f473db0c14d7462c2a9a6e6e9b355" title="1.4_tracing"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/4/44f1d7076b5f473db0c14d7462c2a9a6e6e9b355_2_690x162.png" alt="1.4_tracing" data-base62-sha1="9PUBG1ZNOn17i9xLYQ65aC2aiVL" width="690" height="162" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/4/44f1d7076b5f473db0c14d7462c2a9a6e6e9b355_2_690x162.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/4/44f1d7076b5f473db0c14d7462c2a9a6e6e9b355_2_1035x243.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/4/44f1d7076b5f473db0c14d7462c2a9a6e6e9b355_2_1380x324.png 2x" data-dominant-color="D8D7D6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">1.4_tracing</span><span class="informations">1619×381 63.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<pre><code class="lang-auto">cudnn_convolution            0.16%            842.643us        0.16%            842.643us        842.643us        0.98%            8.491ms          8.491ms          1                []                                   
add                          0.00%            24.037us         0.00%            24.037us         24.037us         0.00%            9.215us          9.215us          1                []                                   
batch_norm                   0.06%            335.818us        0.06%            335.818us        335.818us        0.15%            1.270ms          1.270ms          1                []                                   
_batch_norm_impl_index       0.06%            329.220us        0.06%            329.220us        329.220us        0.15%            1.268ms          1.268ms          1                []                                   
---------------------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  -----------------------------------  
Self CPU time total: 534.585ms
CUDA time total: 863.694ms

accuracy 0.687500

pytorch
	torch.__version__              = 1.4.0
	torch.version.cuda             = 10.1
	torch.backends.cudnn.version() = 7603
		torch.backends.cudnn.benchmark      = False
		torch.backends.cudnn.enabled        = True
		torch.backends.cudnn.deterministic  = False
	torch.cuda.device_count()      = 1
		torch.cuda.get_device_properties()  = _CudaDeviceProperties(name='TITAN X (Pascal)', major=6, minor=1, total_memory=12192MB, multi_processor_count=28)
		torch.cuda.memory_allocated()       = 0 GB
		torch.cuda.memory_reserved()        = 8 GB

</code></pre>
<p>again, all code, results can be find at : <a href="https://drive.google.com/drive/u/0/folders/1vyTkqBpQwUvCSnGjjJPdzIzLTEfJF8Sr" class="inline-onebox" rel="noopener nofollow ugc">pytorch_profile - Google Drive</a></p>

ptrblck: <p>Thanks for the additional run with the fixed libs!<br>
We’ll triage this issue and check, where the difference is coming from.</p>
<p><a class="mention" href="/u/alband">@albanD</a> do you have any quick idea, where the performance regression might be coming from?</p>

albanD: <p>Not really.<br>
Note that with matching versions, the difference is not as big.<br>
Not sure why the conv does not show up any more on the profiling…</p>

arnowaczynski: <p><a class="mention" href="/u/hengck">@Hengck</a><br>
I’ve encountered similar problem and discovered that it’s about contiguity of the input.</p>
<p>Try this:</p>
<pre><code class="lang-python">prediction = net(input.contiguous())
</code></pre>
<p>You can read about breaking changes in release notes for 1.5.0 version (look for “contiguous”): <a href="https://github.com/pytorch/pytorch/releases/tag/v1.5.0" rel="nofollow noopener">https://github.com/pytorch/pytorch/releases/tag/v1.5.0</a></p>
<p>Script for reproducing slower runtime in Pytorch &gt;= 1.5:</p>
<pre><code class="lang-python">import numpy as np
import torch
import torchvision

net = torchvision.models.resnet50()
net.train()
net.cuda()

images_array = np.random.randn(4, 224, 224, 3).astype(np.float32)
images_array = np.rollaxis(images_array, 3, 1)
images_tensor = torch.from_numpy(images_array)
images_tensor = images_tensor.to(device="cuda")
print(images_tensor.is_contiguous())  # False

with torch.autograd.profiler.profile(use_cuda=True) as prof:
    prediction = net(images_tensor)  # slower
    # prediction = net(images_tensor.contiguous())  # faster
print(prof)
</code></pre>

albanD: <p>Thanks for the details !</p>
<p>cc <a class="mention" href="/u/vitalyfedyunin">@VitalyFedyunin</a>  who worked on this.</p>

VitalyFedyunin: <p>Thanks <a class="mention" href="/u/arnowaczynski">@arnowaczynski</a> for the accurate description of the case. Indeed you can end up having channels last inputs and not optimized for it hardware.</p>
<p>The only thing I can add here is that <code>images_tensor = images_tensor.to(device="cuda", memory_format=torch.contiguous_format)</code> will work faster than <code>images_tensor.contiguous()</code></p>

FelixPetersen: <p>Hi,<br>
I also observed a slowdown, for me, it was a factor of 2.5 on CPU on an Intel Mac.<br>
In my case, it was for a matrix multiplication-based application.<br>
(Just as an additional input to whom it may concern.)</p>