What happens at 33 filters in Conv2d?

makslevental: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/f/8ff791726691f82a6fa45ee8519e877430244478.png" data-download-href="https://discuss.pytorch.org/uploads/default/8ff791726691f82a6fa45ee8519e877430244478" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/f/8ff791726691f82a6fa45ee8519e877430244478_2_690x486.png" alt="image" data-base62-sha1="kxAGODJoEXUzwJqHpWoHvHsZUN2" width="690" height="486" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/f/8ff791726691f82a6fa45ee8519e877430244478_2_690x486.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/f/8ff791726691f82a6fa45ee8519e877430244478_2_1035x729.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/f/8ff791726691f82a6fa45ee8519e877430244478_2_1380x972.png 2x" data-dominant-color="EFEEEB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1984×1400 420 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>you can disregard the CPU traces.</p>
<p>this is a plot of runtime vs number of filters (<code>n_bins</code>) and kernel width (<code>max_sigma</code>) in a Conv2d layer. what happens at 33 filters? obviously a different convolution strategy is being chosen for some optimization reason but does anyone know exactly what the switch is and where it happens in the code?</p>

albanD: <p>Hi,</p>
<p>I would say this is due to cudnn internals.</p>
<ul>
<li>If you disable cudnn, you should use only our implementation and should not see this jump</li>
<li>cudnn has it’s custom algorithms depending on the input size</li>
<li>You can enable cudnn benchmark (torch.backends.cudnn.benchmark=True) and see if you see better behavior, it should pick the best algorithm and so remove such artifacts.</li>
</ul>

makslevental: <blockquote>
<p>I would say this is due to cudnn internals.</p>
</blockquote>
<p>i’m guessing there’s no way to find out what cudnn is doing because it’s closed source? or is there a way to get nvprof (or something like that) to tell me which conv strategy it’s using?</p>

albanD: <p>I think there is a verbose mode for cudnn that gives you more info. <a class="mention" href="/u/ptrblck">@ptrblck</a> ?</p>
<p>Hopefully using benchmark mode should remove these hiccups that we can’t control <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

makslevental: <p>thanks <a class="mention" href="/u/alband">@albanD</a>. i’m actually not that worried about this - i’m just interested to know which choices are made (and maybe why they’re made).</p>

ptrblck: <p>Yes, you can use <code>nvprof</code> to get the cudnn calls and check, which kernel is called for your current workload.</p>

Adam_Dziedzic: <p>Yes, you can use nvprof to find out which convolution algorithm was used.<br>
On your gpu machine: <code>nvprof -o prof.nvvp python train_mnist.py</code><br>
Then copy the prof.nvvp to your local machine and run: <code>nvvp prof.nvvp</code><br>
More details can be found here: <a href="https://gist.github.com/sonots/5abc0bccec2010ac69ff74788b265086" rel="nofollow noopener">https://gist.github.com/sonots/5abc0bccec2010ac69ff74788b265086</a></p>
<p>Indeed, it seem like a change in the convolution algorithm. By the way, you cannot select/check which convolution is used in PyTorch, as far as I know. Could anybody confirm that or give some more details how to do it? Moreover, there is an optimizer run by NVidia to select the most optimal convolution algorithm, and sometimes the optimizer tests a few convolutions before making the final choice, see: <a href="https://arxiv.org/pdf/1602.08124.pdf" rel="nofollow noopener">https://arxiv.org/pdf/1602.08124.pdf</a> You can also control which convolution algorithm is used on the CUDA level: <a href="http://www.goldsborough.me/cuda/ml/cudnn/c++/2017/10/01/14-37-23-convolutions_with_cudnn/" rel="nofollow noopener">http://www.goldsborough.me/cuda/ml/cudnn/c++/2017/10/01/14-37-23-convolutions_with_cudnn/</a></p>

albanD: <p>Yes pytorch uses the default algorithm if you do nothing. (and you can’t specify it directly).<br>
If you set <code>torch.backends.cudnn.deterministic</code> it will use the default deterministic algorithm.<br>
And if you set <code>torch.backends.cudnn.benchmark</code> it will try different algorithm to pick the best one.</p>

Adam_Dziedzic: <p>To my mind, it’d be great if PyTorch supported a manual (and deterministic) selection of the convolution algorithm.</p>

albanD: <p><a class="mention" href="/u/adam_dziedzic">@Adam_Dziedzic</a> I think there is an issue open for that feature <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:">   I am pretty sure we would be happy to accept a PR adding this !</p>
<p>Edit: looking at it, it is actually issue number 88, quite an old one <img src="https://discuss.pytorch.org/images/emoji/apple/stuck_out_tongue.png?v=9" title=":stuck_out_tongue:" class="emoji" alt=":stuck_out_tongue:"> <a href="https://github.com/pytorch/pytorch/issues/88">https://github.com/pytorch/pytorch/issues/88</a></p>

makslevental: <blockquote>
<p><code>torch.backends.cudnn.deterministic = True</code></p>
</blockquote>
<p>using this for some reason gives me</p>
<p><code>Process finished with exit code 139 (interrupted by signal 11: SIGSEGV)</code></p>
<p>for a long running job, after a certain size Conv2d (50x80x80)</p>
<p>I’m running</p>
<pre><code class="lang-auto">torch==1.4.0
torchvision==0.5.0
Driver Version: 440.33.01    
CUDA Version: 10.2  
</code></pre>
<p>on a Tesla V100.</p>

ptrblck: <p>Could you post the conv setup as well as the input, which would reproduce this error?<br>
Also, I would recommend to update to the latest stable version (or nightly), as it’ll include the latest bug fixes (besides new features) <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

makslevental: <p>it’s quite hard to give you a MWE because of how involved the code is. I have a conda env and script here</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/makslevental/merf/blob/master/figures/run_times.py">
  <header class="source">

      <a href="https://github.com/makslevental/merf/blob/master/figures/run_times.py" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/makslevental/merf/blob/master/figures/run_times.py" target="_blank" rel="noopener nofollow ugc">makslevental/merf/blob/master/figures/run_times.py</a></h4>


      <pre><code class="lang-py">import csv
import os
import time
from pathlib import Path

import numpy as np
import torch
from torch.backends import cudnn

cudnn.deterministic = False
from torch.utils.data import DataLoader
from skimage import io, img_as_float
from skimage.filters import gaussian
import torch.multiprocessing as mp
from nn_dog import PIN_MEMORY, DEVICE, NUM_GPUS
from nn_dog.data import SimulPLIF
from nn_dog.model import DifferenceOfGaussiansFFT, DifferenceOfGaussiansStandardConv
from sk_image.blob import cpu_blob_dog
from sk_image.enhance_contrast import stretch_composite_histogram

</code></pre>



  This file has been truncated. <a href="https://github.com/makslevental/merf/blob/master/figures/run_times.py" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>i suspect it’s a memory leak because if i make shorter runs (i’m iterating over a set of hyperparameters) then there is no segfault. is it possible to use something like valgrind to investigate this?</p>
<p>also btw <a class="mention" href="/u/alband">@albanD</a> setting <code>torch.backends.cudnn.deterministic = True</code> did not actually fix the convolution strategy (i.e. force a fixed strategy); here is what i see</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/9/f91102dcfd01c3d7c7ab954350eed239a7c9590e.png" alt="image" data-base62-sha1="zxlkRgfssw5S40zQ5dIN7MbC4Gi" width="640" height="480"></p>
<p>edit: <a class="mention" href="/u/adam_dziedzic">@Adam_Dziedzic</a> i also can’t see which cuda conv strategy is being used after running nvprof. only thing i can see is implicit_sgemm.</p>

ptrblck: <p>For <code>cudnn.deterministic = True</code> <code>CUDNN_CONVOLUTION_FWD_ALGO_IMPLICIT_PRECOMP_GEMM</code> will be used in the forward pass.</p>
<p>Given that the code includes multiple files, I would recommend to first disable the <code>multiprocessing.pool</code>, and try to remove everything unnecessary until you could narrow down the segmentation fault to a single model with some dummy data.</p>