"invalid configuration" error when I try to run on GPU

agt: <p>My code works fine on the CPU, but when I move everything to GPU and try to run it, I get an error on this line:<br>
<code>print(ctx["input"])</code></p>
<p>The shape of <code>ctx["input"]</code> is <code>torch.Size([1, 50])</code> and its dtype is <code>torch.float32</code>.</p>
<p>The error says: “RuntimeError: CUDA error: invalid configuration argument”.</p>
<p>Actually, the error appears whenever I try to do anything with <code>ctx["input"]</code>.</p>
<p>How can I fix this?</p>
<p>This is the entire traceback: <a href="https://dpaste.org/qDRq" class="inline-onebox" rel="noopener nofollow ugc">dpaste/qDRq (Python)</a></p>

ptrblck: <p>Could you rerun your code via <code>CUDA_LAUNCH_BLOCKING=1 python script.py args</code> and post the stack trace here?<br>
The error points to a wrong kernel launch config. Are you using any custom CUDA code in your application? If not, which PyTorch and CUDA version as well as GPU are you using?</p>

agt: <p>If I’m in Google Colab, can I just put <code>!CUDA_LAUNCH_BLOCKING=1</code> at the beginning of the cell?</p>
<p>I think the root of the problem is in this module:</p>
<pre><code class="lang-auto">class MessageMaker(torch.nn.Module):
    def __init__(self, embed_size, hidden_size):
        super().__init__()
        
        node_edge_concat_size = embed_size * 2
        self.rnn = torch.nn.LSTM(node_edge_concat_size, hidden_size)

    def forward(self, edges):
        node_reps, edge_reps = edges.src["rep"], edges.data["rep"]
        inputs = torch.cat([node_reps, edge_reps], 1)
        inputs = inputs.unsqueeze(0) #Since we're processing only 1 seq element at a time.
        rnn_state = edges.src["sum_incoming_hidden_and_cell"]
        hiddens, cells = rnn_state[:, 0].contiguous(), rnn_state[:, 1].contiguous()
        hiddens, cells = hiddens.unsqueeze(0), cells.unsqueeze(0) #Since it's only 1 layer+direction.
        outputs, (updated_hiddens, updated_cells) = self.rnn(inputs, (hiddens, cells))
        updated_hiddens, updated_cells = updated_hiddens.squeeze(0), updated_cells.squeeze(0)
        updated_rnn_state = torch.stack([updated_hiddens, updated_cells], 1)
        new_features = { "hidden_and_cell": updated_rnn_state }

        return new_features
</code></pre>
<p>After moving to GPU I got an error “rnn: hx is not contiguous”, so I added <code>.contiguous()</code> to the 5th line of <code>forward()</code> to fix it. Could the CUDA error have something to do with that?</p>

ptrblck: <p>It could potentially be related. Could you post an executable code snippet, which raises the initial error using this module?</p>
<aside class="quote no-group" data-username="agt" data-post="3" data-topic="115748">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/7feea3/48.png" class="avatar"> agt:</div>
<blockquote>
<p>If I’m in Google Colab, can I just put <code>!CUDA_LAUNCH_BLOCKING=1</code> at the beginning of the cell?</p>
</blockquote>
</aside>
<p>It could work, if you set this environment variable before importing any other library, which might initialize the CUDA context. As it’s often not straightforward to use it properly in a Jupyter notebook, I usually recommend to run it in a terminal instead.</p>

agt: <p>I don’t have a personal GPU so I have to use a Colab notebook. Will try setting it at the beginning.</p>