Conv2d is slow in pytorch build from source

sgw: <p>I built from source code(v1.0.0), but Conv2 speed is slower than that installed with conda install command.</p>
<p>conv2d      cpu time - 709909.681us   cuda time - 709897.758us<br>
conv2d      cpu time - 3431.992us       cuda time - 3430.720us</p>
<p><strong>source code</strong></p>
<hr>
<p>class convT(nn.Module):</p>
<pre><code>def __init__(self):
     super(convT, self).__init__()
     self.conv1=nn.Conv2d(1,1,5)

def forward(self, input):
     x = self.conv1(input)
     return x
</code></pre>
<p>if <strong>name</strong> == ‘<strong>main</strong>’:</p>
<pre><code>x = torch.randn(1,1,200,200)
net = convT()
net.to(device)
x= x.cuda()

torch.cuda.synchronize()
with torch.autograd.profiler.profile(use_cuda=True) as prof:
    out = net(x)
torch.cuda.synchronize()

print(prof.key_averages().table(sort_by='cuda_time_total'))
print(out.size())
</code></pre>
<hr>
<p><strong>environment config</strong></p>
<p>PyTorch version: 1.0.0a0+bb15580<br>
Is debug build: No<br>
CUDA used to build PyTorch: 9.0.176</p>
<p>OS: Ubuntu 16.04.6 LTS<br>
GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.11) 5.4.0 20160609<br>
CMake version: version 3.14.0</p>
<p>Python version: 3.7<br>
Is CUDA available: Yes<br>
CUDA runtime version: 9.0.176<br>
GPU models and configuration:<br>
GPU 0: GeForce GTX TITAN X<br>
GPU 1: GeForce GTX TITAN X<br>
GPU 2: GeForce GTX TITAN X<br>
GPU 3: GeForce GTX TITAN X<br>
GPU 4: GeForce GTX TITAN X<br>
GPU 5: GeForce GTX TITAN X<br>
GPU 6: GeForce GTX TITAN X<br>
GPU 7: GeForce GTX TITAN X</p>
<p>Nvidia driver version: 384.81<br>
cuDNN version: Probably one of the following:<br>
/usr/lib/x86_64-linux-gnu/libcudnn.so<br>
/usr/lib/x86_64-linux-gnu/libcudnn.so.7<br>
/usr/lib/x86_64-linux-gnu/libcudnn.so.7.4.1<br>
/usr/lib/x86_64-linux-gnu/libcudnn_static.a<br>
/usr/lib/x86_64-linux-gnu/tmp/libcudnn.so.7.1.3<br>
/usr/local/cuda-9.0/lib64/libcudnn.so<br>
/usr/local/cuda-9.0/lib64/libcudnn.so.7<br>
/usr/local/cuda-9.0/lib64/libcudnn.so.7.4.1<br>
/usr/local/cuda-9.0/lib64/libcudnn_static.a</p>
<p>Versions of relevant libraries:<br>
[pip] Could not collect<br>
[conda] magma-cuda90              2.5.0                         1    pytorch<br>
[conda] torch                     1.0.0a0+bb15580          pypi_0    pypi<br>
[conda] torchfile                 0.1.0                    pypi_0    pypi</p>

rasbt: <pre><code class="lang-auto">I built from source code(v1.0.0), but Conv2 speed is slower than that installed with conda install command.
conv2d cpu time - 709909.681us cuda time - 709897.758us
conv2d cpu time - 3431.992us cuda time - 3430.720us
</code></pre>
<p>I know that in general, for intel cpus, conda uses MKL (Math Kernel Library), which is a highly tuned a BLAS implementation for Intel CPUs. That could be one of the reasons explaining the CPU performance difference (you typically find similar performance differences when using NumPy when comparing conda and source/pip installations). However, I think on a GPU, you won’t notice that much of a difference.</p>

shaik0440: <p>Facing similar issue. I compared only GPU number. Are you able to locate the source of high latency?</p>
<p>Another observation from my end.</p>
<pre><code>If you do a dummy convolution operation with the input and carry on with the network flow. You will get correct timing for the network.
def forward(self, x):
start0 = time.time()
x1 = self.conv2d(x)
end0 = time.time()
time0 = (end0 - start0)*1000
print(‘Dummy conv2d-depth - %f’%time0)

 start0 = time.time()
 x1 = self.conv2d(x)
 end0 = time.time()
 time0 = (end0 - start0)*1000
 print('conv2d-depth - %f'%time0)

i doubt the latency spike is something related to GPU initialization in native pytorch code.
</code></pre>
<p><a class="mention" href="/u/smth">@smth</a>  Need some pointers to locate the origin of latency increase</p>
<p>Thanks in advance</p>

ptrblck: <p>Note that CUDA operations are called asynchronously, so that you should add manual synchronization points before starting and stopping the timer using <code>torch.cuda.synchronize()</code>.<br>
Otherwise you might just time the kernel launch.</p>

shaik0440: <p>Made the changes… Problem stands</p>
<p>the first call takes 420 ms and the second call (same operations)1.96ms…</p>

ptrblck: <p>Could you post the code you are using to profile these operations?</p>

shaik0440: <p>Hi ptrblck,</p>
<p>Pytorch is installed from the source.<br>
During the first call cublas handle creation is taking time, the same handle is used in the second call. I feel this is the reason for higher latency in the first call.</p>