Loss.backword() shows RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED

MD_TARIQUL_ISLAM: <p>Hello there,<br>
I’m getting <strong>RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED</strong> in loss.backward() stage, which has been triggered from autograd module of torch. What’s the reason behind it?</p>
<p>The executed script is: <a href="https://github.com/bshall/ZeroSpeech/blob/0ef25e4daac07888d22b802c649480dc70b719fe/train.py" class="inline-onebox" rel="noopener nofollow ugc">ZeroSpeech/train.py at 0ef25e4daac07888d22b802c649480dc70b719fe · bshall/ZeroSpeech · GitHub</a></p>
<p>My specs are:<br>
GPU 1660s<br>
CUDA 10.2<br>
cudnn 7.6.5</p>
<p>Thanks in advance.</p>

ptrblck: <p>Could you create a minimal executable code snippet, which would reproduce the issue?<br>
I.e. could you post the model configs or how the modules are initialized as well as the used input shapes?</p>

MD_TARIQUL_ISLAM: <p>Here is code snippet I’m using and this is the input part of the model:</p><aside class="onebox githubblob" data-onebox-src="https://github.com/bshall/ZeroSpeech/blob/0ef25e4daac07888d22b802c649480dc70b719fe/model.py#L20">
  <header class="source">

      <a href="https://github.com/bshall/ZeroSpeech/blob/0ef25e4daac07888d22b802c649480dc70b719fe/model.py#L20" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/bshall/ZeroSpeech/blob/0ef25e4daac07888d22b802c649480dc70b719fe/model.py#L20" target="_blank" rel="noopener nofollow ugc">bshall/ZeroSpeech/blob/0ef25e4daac07888d22b802c649480dc70b719fe/model.py#L20</a></h4>



  <pre class="onebox">    <code class="lang-py">
      <ol class="start lines" start="10" style="counter-reset: li-counter 9 ;">
          <li>
          </li>
<li>def get_gru_cell(gru):</li>
          <li>    gru_cell = nn.GRUCell(gru.input_size, gru.hidden_size)</li>
          <li>    gru_cell.weight_hh.data = gru.weight_hh_l0.data</li>
          <li>    gru_cell.weight_ih.data = gru.weight_ih_l0.data</li>
          <li>    gru_cell.bias_hh.data = gru.bias_hh_l0.data</li>
          <li>    gru_cell.bias_ih.data = gru.bias_ih_l0.data</li>
          <li>    return gru_cell</li>
          <li>
          </li>
<li>
          </li>
<li class="selected">class Encoder(nn.Module):</li>
          <li>    def __init__(self, in_channels, channels, n_embeddings, embedding_dim, jitter=0):</li>
          <li>        super(Encoder, self).__init__()</li>
          <li>        self.encoder = nn.Sequential(</li>
          <li>            nn.Conv1d(in_channels, channels, 3, 1, 0, bias=False),</li>
          <li>            nn.BatchNorm1d(channels),</li>
          <li>            nn.ReLU(True),</li>
          <li>            nn.Conv1d(channels, channels, 3, 1, 1, bias=False),</li>
          <li>            nn.BatchNorm1d(channels),</li>
          <li>            nn.ReLU(True),</li>
          <li>            nn.Conv1d(channels, channels, 4, 2, 1, bias=False),</li>
      </ol>
    </code>
  </pre>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>The argument values which have been passed to this object is:<br>
in_channels: 80<br>
channels: 768<br>
n_embeddings: 512<br>
embedding_dim: 64<br>
jitter: 0.5</p>
<p>The models are initialised here:</p><aside class="onebox githubblob" data-onebox-src="https://github.com/bshall/ZeroSpeech/blob/0ef25e4daac07888d22b802c649480dc70b719fe/train.py#L39">
  <header class="source">

      <a href="https://github.com/bshall/ZeroSpeech/blob/0ef25e4daac07888d22b802c649480dc70b719fe/train.py#L39" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/bshall/ZeroSpeech/blob/0ef25e4daac07888d22b802c649480dc70b719fe/train.py#L39" target="_blank" rel="noopener nofollow ugc">bshall/ZeroSpeech/blob/0ef25e4daac07888d22b802c649480dc70b719fe/train.py#L39</a></h4>



  <pre class="onebox">    <code class="lang-py">
      <ol class="start lines" start="29" style="counter-reset: li-counter 28 ;">
          <li>    print("Saved checkpoint: {}".format(checkpoint_path.stem))</li>
          <li>
          </li>
<li>
          </li>
<li>@hydra.main(config_path="config/train.yaml")</li>
          <li>def train_model(cfg):</li>
          <li>    tensorboard_path = Path(utils.to_absolute_path("tensorboard")) / cfg.checkpoint_dir</li>
          <li>    checkpoint_dir = Path(utils.to_absolute_path(cfg.checkpoint_dir))</li>
          <li>    writer = SummaryWriter(tensorboard_path)</li>
          <li>    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")</li>
          <li>
          </li>
<li class="selected">    encoder = Encoder(**cfg.model.encoder)</li>
          <li>    decoder = Decoder(**cfg.model.decoder)</li>
          <li>    encoder.to(device)</li>
          <li>    decoder.to(device)</li>
          <li>
          </li>
<li>    optimizer = optim.Adam(</li>
          <li>        chain(encoder.parameters(), decoder.parameters()),</li>
          <li>        lr=cfg.training.optimizer.lr)</li>
          <li>    [encoder, decoder], optimizer = amp.initialize([encoder, decoder], optimizer, opt_level="O1")</li>
          <li>    scheduler = optim.lr_scheduler.MultiStepLR(</li>
          <li>        optimizer, milestones=cfg.training.scheduler.milestones,</li>
      </ol>
    </code>
  </pre>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Let me know if anything you need to assure.</p>
<p>By the way, this is a appreciation for you <a class="mention" href="/u/ptrblck">@ptrblck</a>. Whenever I face any pytorch issues and look for the solution, it’s often you who comes with the best solution. Thanks for everything bud!</p>