Tensorflow vs. PyTorch ConvNet benchmark

username: alireza_davoudi
cooked: <p>I created a benchmark to compare the performances of Tensorflow and PyTorch for fully convolutional neural networks in <a href="https://github.com/alirezadavoudi/tensorflow-vs-pytorch" rel="nofollow noopener">this github repository</a>:</p>
<p>I need to make sure if these two implementations are identical. If it is, then the results show that Tensorflow is about %5 faster in one of the experiments and about %20 faster in another experiment.</p>
<p>Thanks in advance</p>

username: colesbury
cooked: <p>Your benchmarking code isn’t accurate. CUDA kernels are launched asynchronously, but you’re measuring the time using <code>time.time()</code> even though the forward pass won’t have completed at that point.</p>
<p>To my knowledge, both PyTorch and TensorFlow use same implementation (cuDNN) for convolutions. The speed of the convolution implementation is going to dominate if you have sufficient batch size (which you do). The other differences you see are probably differences in measurement (see point above) and how you’re passing in data.</p>
<p>There’s already a bunch of accurate benchmarks of CNNs:</p>
<p><aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://assets-cdn.github.com/favicon.ico" class="site-icon" width="32" height="32">
      <a href="https://github.com/soumith/convnet-benchmarks" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars0.githubusercontent.com/u/1310570?s=400&amp;v=4" class="thumbnail onebox-avatar" width="400" height="400">

<h3><a href="https://github.com/soumith/convnet-benchmarks" target="_blank" rel="nofollow noopener">soumith/convnet-benchmarks</a></h3>

<p>convnet-benchmarks - Easy benchmarking of all publicly accessible implementations of convnets</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://assets-cdn.github.com/favicon.ico" class="site-icon" width="32" height="32">
      <a href="https://github.com/jcjohnson/cnn-benchmarks" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars0.githubusercontent.com/u/2718714?s=400&amp;v=4" class="thumbnail onebox-avatar" width="400" height="400">

<h3><a href="https://github.com/jcjohnson/cnn-benchmarks" target="_blank" rel="nofollow noopener">jcjohnson/cnn-benchmarks</a></h3>

<p>cnn-benchmarks - Benchmarks for popular CNN models</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>

username: alireza_davoudi
cooked: <p>‌But the python API is synchronous, otherwise, it won’t work at all. would you please explain the asynchronous API more clearly?</p>
<p>Afterall I measure the time of 10 iterations.</p>

username: mratsim
cooked: <p>Cuda kernels are async by default, that means that as soon as the cuda kernel is launched, the program get the control back even though computation is not finished.</p>
<p>To make sure computation is finished you should call “cudaDeviceSynchronize”, your best bet would be through Cupy.</p>
<p>Furthermore there might be a difference due to the Tensor layouts:</p>
<p>PyTorch use NCHW and Tensorflow uses NHWC, NCHW was the first layout supported by CuDNN but presents a big challenge for optimization (due to access patterns in convolutions, memory coalescing and such …).<br>
NHWC is easier to optimize for convolutions but suffer in linear layers iirc because you have to physically transpose/permute the dimensions.</p>
<p>Furthermore, due to it’s dynamic nature, PyTorch allocate new memory at each new batch while Tensorflow can just reuse previous memory locations since size is known in advance.</p>
<p>Memory is THE bottleneck in Deep Learning not CPU, the big challenge is how to feed data fast enough to the CPU and GPU to get the maximum GFLOPS throughput.</p>
<p>So I think the benchmark is worth it.</p>

username: Vardaan_Gangal
cooked: <p>I know that that tensorflow has the tf_cnn_benchmarks script. Does PyTorch have anything similar?</p>