CUDA Invalid Configuration Error on GPU Only

username: akernel
cooked: <p>Hi everyone, long-time reader, first-time poster!</p>
<p>I seem to be running into an odd bug when training my model.  I am trying to train an embedding model using the Resnet18 architecture, which I have essentially cut off the last linear layer of (I can post the architecture if needed).  I have written my own loss function which implements the “Batch-Hard” loss found in <a href="https://arxiv.org/pdf/1703.07737.pdf" rel="nofollow noopener">this paper</a>.  The training loop (standard <code>zero_grad</code>, <code>output</code>, <code>loss</code>, <code>backward</code>, <code>step</code>) works fine when running on CPU, but throws the following error when running on GPU:</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-11-8cf2cfddbcc6&gt; in &lt;module&gt;()
      3     output = learner.model(imgs)
      4     loss = criterion(output, labels)
----&gt; 5     loss.backward()
      6     optimizer.step()
      7     break

.../anaconda/anaconda3-2019a/lib/python3.6/site-packages/torch/tensor.py in backward(self, gradient, retain_graph, create_graph)
    105                 products. Defaults to ``False``.
    106         """
--&gt; 107         torch.autograd.backward(self, gradient, retain_graph, create_graph)
    108 
    109     def register_hook(self, hook):

.../anaconda/anaconda3-2019a/lib/python3.6/site-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
     91     Variable._execution_engine.run_backward(
     92         tensors, grad_tensors, retain_graph, create_graph,
---&gt; 93         allow_unreachable=True)  # allow_unreachable flag
     94 
     95 

RuntimeError: CUDA error: invalid configuration argument
</code></pre>
<p>Does anyone know why a backwards pass would work on a CPU but not a GPU?  I can confirm that my GPU works generally, I have trained other models with other custom  loss functions.</p>
<p>For reference, this is the implementation of the “Batch-Hard” loss that I’ve written:</p>
<pre><code class="lang-auto">class BatchHardLoss(torch.nn.modules.loss._Loss):
    def __init__(self, margin, k=5, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.margin = margin
        self._k = k

    def forward(self, x, targets):
        distances = torch.cdist(x, x)  # Get the distances
        losses = []
        for t in torch.unique(targets):
            pos_idxs = (targets == t).nonzero().reshape(-1)  # Positive indices
            neg_idxs = (targets != t).nonzero().reshape(-1)  # Negative indices

            choice = np.random.choice(
                len(pos_idxs), min(self._k, len(pos_idxs))
            ).tolist()
            anchors = pos_idxs[choice]  # Chosen anchors

            # Only compute loss if we can do it for the full batch, for simplicity
            loss = 0.0
            if len(anchors) &gt; 1 and len(neg_idxs) &gt; 0:
                pos_distances = torch.index_select(
                    torch.index_select(distances, 0, anchors), 1, pos_idxs
                )
                pos_loss = pos_distances.max(dim=1)[0].sum()

                neg_distances = torch.index_select(
                    torch.index_select(distances, 0, anchors), 1, neg_idxs
                )
                neg_loss = neg_distances.min(dim=1)[0].sum()

                marg_loss = self.margin * len(anchors)

                losses.append(marg_loss + pos_loss - neg_loss)
            else:
                losses.append(torch.tensor(0.0, device=x.device, requires_grad=True))
        return torch.stack(losses).sum()
</code></pre>
<p>If anyone sees any glaring errors or has any insight, I’d really appreciate it.</p>
<p>Thanks!</p>
<p>P.S. I have seen <a href="https://discuss.pytorch.org/t/runtimeerror-cuda-runtime-error-9-invalid-configuration-argument/25373/3">this post</a> but can’t seem to find a similar issue when debugging.</p>

username: ptrblck
cooked: <p>I don’t see anything obviously wrong in your code.<br>
Could you print the <code>output</code> and <code>labels</code> tensors before calling <code>loss.backward()</code>?<br>
Maybe we could isolate this issue to these values and reproduce the error.</p>
<p>If these tensors do not yield the same error, could you additionally also store the <code>state_dict</code> of your model and the current <code>imgs</code>?</p>

username: akernel
cooked: <p>Thanks for replying <a class="mention" href="/u/ptrblck">@ptrblck</a>!  While trying to reproduce the error at home (I was at work when I first got the error) I started with a smaller batch size (I was using 1024), and lo and behold it worked!  I did some investigating, and apparently 1023 works, but any <code>batch_size &gt;= 1024</code> causes that error.  It seems to me that it’s a GPU memory issue, but I don’t know why I’m not getting the traditional <code>RuntimeError: CUDA out of memory</code>.  I’ll double-check that this is the case on my work setup as well, and will reply back tomorrow.</p>

username: akernel
cooked: <p>That does seem to have been the problem - still not sure why the error wasn’t the usual <code>out of memory</code>, but glad it’s fixed!</p>

username: ptrblck
cooked: <p>Good to hear it’s fixed now!<br>
That’s indeed a bit weird, but sometimes OOM errors are masked by e.g. cudnn errors.<br>
Haven’t seen this invalid config error yet.</p>

username: rohaldbUni
cooked: <p>I’m receiving the same error and can’t seem to debug it. I don’t think it has to do with the memory of my GPU either - the computations aren’t that large and it still fails with a batch size of 1. When I add an extra component to my loss function I get the same error thrown on the <code>loss.backward()</code> step. Here is an excerpt from my loss function</p>
<pre><code class="lang-auto">def forward(self, anchor, positive, negative, model, size_average=True):
    #regular triplet loss. This works on GPU and CPU
    distance_positive = (anchor - positive).pow(2).sum(1)  # .pow(.5)
    distance_negative = (anchor - negative).pow(2).sum(1)  # .pow(.5)
    losses = F.relu(distance_positive - distance_negative + self.margin)

    #the additional component that causes the error. This will run on CPU but fails on GPU
    anchor_dists = torch.cdist(model.embedding_net.anchor_net.anchors, model.embedding_net.anchor_net.anchors)
    t = (self.beta * F.relu(self.rho - anchor_dists))
    regularization = t.sum() - torch.diag(t).sum()

    return regularization + losses.mean() if size_average else losses.sum()
</code></pre>
<p>Here is my model:</p>
<pre><code class="lang-auto">TripletNet(
  (embedding_net): EmbeddingNet(
    (anchor_net): AnchorNet(anchors torch.Size([128, 192]), biases torch.Size([128]))
    (embedding): Sequential(
      (0): AnchorNet(anchors torch.Size([128, 192]), biases torch.Size([128]))
      (1): Tanh()
    )
  )
)
</code></pre>
<p>Which is much smaller than the memory capacity of my GPU (8GB).</p>
<p>EDIT: Monitoring the GPU memory usage shows that I’m well under the memory limit when it crashes.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/9/19d7f66482f29c29419782b1056fb455101b3ddf.png" alt="20%20pm" data-base62-sha1="3GCDjWOpYBde7EwEvi7KIuXNvwX" width="568" height="266"></p>
<p>Any ideas <a class="mention" href="/u/ptrblck">@ptrblck</a> ? Thanks!</p>

username: ptrblck
cooked: <p>Could you post a code snippet to reproduce this issue so that we could have a look?</p>

username: rohaldbUni
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, I’d be happy to. Here is an excerpt from the file Runner.ipynb that generates the error:</p>
<pre><code class="lang-auto">cuda = torch.cuda.is_available()
#define datasets
triplet_train_dataset = TripletAudio(True, K, MAX_CLOSE_NEG, P_STRONG_NEG)
triplet_test_dataset = TripletAudio(False, K, MAX_CLOSE_NEG, P_STRONG_NEG)
triplet_train_loader = torch.utils.data.DataLoader(triplet_train_dataset, batch_size=BATCH_SIZE, shuffle=True)
triplet_test_loader = torch.utils.data.DataLoader(triplet_test_dataset, batch_size=BATCH_SIZE, shuffle=False)
#define model 
anchor_net = AnchorNet(triplet_train_dataset.get_dataset(), INPUT_D, OUTPUT_D)
embedding_net = EmbeddingNet(anchor_net)
model = TripletNet(embedding_net)
if cuda:
    model.cuda()
loss_fn = TripletLoss(MARGIN, RHO, BETA)
optimizer = optim.Adam(model.parameters(), lr=1e-3, weight_decay=1e-4)
scheduler = lr_scheduler.StepLR(optimizer, step_size=30, gamma=0.1, last_epoch=-1)
#run the model
train_loss, val_loss = fit(triplet_train_loader, triplet_test_loader, model, loss_fn, optimizer, scheduler, N_EPOCHS, cuda, LOG_INT)
# ^^^^^^ parent function call that generates the error  ^^^^^ 
</code></pre>
<p>And the error:</p>
<pre><code class="lang-auto">/opt/anaconda3/lib/python3.7/site-packages/torch/optim/lr_scheduler.py:82: UserWarning: Detected call of `lr_scheduler.step()` before `optimizer.step()`. In PyTorch 1.1.0 and later, you should call them in the opposite order: `optimizer.step()` before `lr_scheduler.step()`.  Failure to do this will result in PyTorch skipping the first value of the learning rate schedule.See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate
  "https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate", UserWarning)
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-3-bda3ed742c36&gt; in &lt;module&gt;
     16 scheduler = lr_scheduler.StepLR(optimizer, step_size=30, gamma=0.1, last_epoch=-1)
     17 #run the model
---&gt; 18 train_loss, val_loss = fit(triplet_train_loader, triplet_test_loader, model, loss_fn, optimizer, scheduler, N_EPOCHS, cuda, LOG_INT)

~/thesis/trainer.py in fit(train_loader, val_loader, model, loss_fn, optimizer, scheduler, n_epochs, cuda, log_interval, metrics, start_epoch)
     24         scheduler.step()
     25         # Train stage
---&gt; 26         train_loss, metrics, writer_train_index = train_epoch(train_loader, model, loss_fn, optimizer, cuda, log_interval, metrics, writer, writer_train_index)
     27 
     28         message = 'Epoch: {}/{}. Train set: Average loss: {:.4f}'.format(epoch + 1, n_epochs, train_loss)

~/thesis/trainer.py in train_epoch(train_loader, model, loss_fn, optimizer, cuda, log_interval, metrics, writer, writer_train_index)
     80         losses.append(loss.item())
     81         total_loss += loss.item()
---&gt; 82         loss.backward()
     83         optimizer.step()
     84 

/opt/anaconda3/lib/python3.7/site-packages/torch/tensor.py in backward(self, gradient, retain_graph, create_graph)
    116                 products. Defaults to ``False``.
    117         """
--&gt; 118         torch.autograd.backward(self, gradient, retain_graph, create_graph)
    119 
    120     def register_hook(self, hook):

/opt/anaconda3/lib/python3.7/site-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
     91     Variable._execution_engine.run_backward(
     92         tensors, grad_tensors, retain_graph, create_graph,
---&gt; 93         allow_unreachable=True)  # allow_unreachable flag
     94 
     95 

RuntimeError: CUDA error: invalid configuration argument
</code></pre>
<p>The <code>fit</code> function in trainer.py essentially just calls this sub-function:</p>
<pre><code class="lang-auto">def train_epoch(train_loader, model, loss_fn, optimizer, cuda, log_interval, metrics, writer, writer_train_index):
    for metric in metrics:
        metric.reset()

    model.train()
    losses = []
    total_loss = 0

    for batch_idx, (data, target, index) in enumerate(train_loader):
        target = target if len(target) &gt; 0 else None
        if not type(data) in (tuple, list):
            data = (data,)
        if cuda:
            data = tuple(d.cuda() for d in data)
            if target is not None:
                target = target.cuda()

        optimizer.zero_grad()
        outputs = model(*data,)
        if type(outputs) not in (tuple, list):
            outputs = (outputs,)

        loss_inputs = outputs
        if target is not None:
            target = (target,)
            loss_inputs += target

        loss_outputs = loss_fn(*loss_inputs, model)

        loss = loss_outputs[0] if type(loss_outputs) in (tuple, list) else loss_outputs
        losses.append(loss.item())
        total_loss += loss.item()
        loss.backward() # this is the line that generates the error 
        optimizer.step()
</code></pre>
<p>And the loss function is supplied above. The full files (Runner.ipynb, trainer.py, losses.py) can be found <a href="https://github.com/rohaldbUni/thesis/tree/regularization-demo" rel="nofollow noopener">here</a> if desired.</p>
<p>Thanks for your help by the way! This has had me very, very stuck. If there is anything else that could help, please let me know</p>

username: ptrblck
cooked: <p>Thanks for the code!<br>
It looks like the error is thrown in the <code>backward</code> call, so we could probably just raise the error using some dummy input and target data without creating the real dataset.<br>
Could you post the shapes and types of <code>data</code> and <code>target</code>?</p>

username: rohaldbUni
cooked: <p><code>target</code> is actually None (the logic is generalizable and happens to not use target) and  <code>data</code> is an array containing three elements (a triplet), each of which is of size <code>torch.Size([128, 192, 1])</code> and of type <code>torch.FloatTensor</code></p>

username: ptrblck
cooked: <p>I still can’t run the code, as <code>AnchorNet</code> relies on some data.<br>
Could you post some dummy input shapes to <code>train_data</code> (or an executable code snippet)?</p>

username: rohaldbUni
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, here is a runnable <a href="https://colab.research.google.com/drive/1c43ePV0Gjz0GAikH5Nn9wGfVWzGpiAi2" rel="nofollow noopener">colab document</a> with everything stripped but the bare requirements to produce the error. The data loaders contain randomly generated tensors of the appropriate dimension. I hope this helps.</p>

username: Naruto-Sasuke
cooked: <p>The same problem for <code>torch.cdist</code> function.</p>

username: titaneric
cooked: <p>I also have the same error message. BTW, I use the <code>torch.pdist</code> either, which seems the same problem as <a class="mention" href="/u/naruto-sasuke">@Naruto-Sasuke</a>.<br>
The CPU works fine, but GPU will have this error.</p>
<p>torch version: 1.4.0<br>
CUDA: 10.1</p>

username: ptrblck
cooked: <p>Could you update to the latest stable PyTorch version (<code>1.5.0</code>), as we’ve fixed some issues in <code>cdist</code> and <code>pdist</code> regarding these invalid configurations?</p>

username: titaneric
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> thanks for the reply.<br>
I’ve updated the pytorch from <code>1.4.0</code> to <code>1.5.0</code>. Work perfectly.<br>
Thanks again.</p>

username: wxjtwxjt
cooked: <p>Same issue here with pytorch 1.5.0<br>
Works on batch size = 512, crashes on batch size = 1024 reporting CUDA Invalid Config Error related to cdist</p>