Runtime Error : CUDA Error

Suraj520: <p>RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling `cublasLtMatmul( ltHandle, computeDesc.descriptor(), &amp;alpha_val, mat1_ptr, Adesc.descriptor(), mat2_ptr, Bdesc.descriptor(), &amp;beta_val, result_ptr, Cdesc.descriptor(), result_ptr, Cdesc.descriptor(), &amp;heuristicResult.algo, workspace.data_ptr(), workspaceSize, at::cuda::getCurrentCUDAStream())</p>
<p>or<br>
RuntimeError: philox_cuda_state for an unexpected CUDA generator used during capture. In regions captured by CUDA graphs, you may only use the default CUDA RNG generator on the device that’s current when capture begins. If you need a non-default (user-supplied) generator, or a generator on another device, please file an issue.</p>
<p>This error is popping up while trying to train a transformer model from scratch in Colab. It runs on CPU, Though!</p>
<p>How to resolve ?</p>

Yuyao_Huang: <p>I am not familiar with Colab, but maybe you could post some potentially relevant code as well so it’s easier for us to diagnosis the cause of your issue?</p>

Suraj520: <p>Hi <a class="mention" href="/u/yuyao_huang">@Yuyao_Huang</a><br>
<a href="https://github.com/Suraj520/nlp-basics-to-advanced/blob/main/23_transformer-machine-translation/notebook.ipynb" class="inline-onebox" rel="noopener nofollow ugc">nlp-basics-to-advanced/notebook.ipynb at main · Suraj520/nlp-basics-to-advanced · GitHub</a> 's Cell 102</p>
<p>When I move the tensor tgt_mask to device other than cpu i.e cuda, The error pops up !</p>
<p>Snippet being</p>
<pre><code class="lang-auto">step=0
num_epochs=10
def fit(num_epochs,train_loader,val_loader, model, optimizer,criterion):
    model.train()
    for epoch in range(num_epochs):
        checkpoint = {'state_dict':model.state_dict(), 'optimizer':optimizer.state_dict()}
        if epoch%10==0:
            torch.save(checkpoint,"checkpoint.pth.tar")
        #loading batch
        for i,batch in enumerate(train_loader):
            input_seq = batch['input'].to(device)
            #input_mask = batch['input_mask'].to(device)
            output_seq = batch['output'].to(device)
            #output_mask = batch['output_mask'].to(device)
            input_padding_mask = input_seq == PAD_IDX
            output_padding_mask = output_seq == PAD_IDX
            #input_mask, output_mask, input_padding_mask, output_padding_mask = create_mask(input_seq, output_seq)
            memory_key_padding_mask = input_padding_mask.clone()
            input_padding_mask = rearrange(input_padding_mask, 'n s -&gt; s n')
            output_padding_mask = rearrange(output_padding_mask, 'n s -&gt; s n')
            memory_key_padding_mask = rearrange(memory_key_padding_mask, 'n s -&gt; s n')
            
            tgt_sentence_len = output_seq.shape[0] - torch.sum(output_padding_mask,axis=1)
            tgt_inp, tgt_out = output_seq[:,:], output_seq[:,:]
            tgt_mask = gen_nopeek_mask(output_seq.shape[0])#.to('cuda') #('cuda)
            tgt_mask = tgt_mask.to(device)
            output = model(input_seq,tgt_inp,0,input_padding_mask, output_padding_mask,memory_key_padding_mask,tgt_mask)

            #generating one hot
            from_one_hot = torch.argmax(output,dim=2)

            output = output.view(-1, output.shape[-1])
            tgt_out = tgt_out.view(-1)
            loss = criterion(output,tgt_out)
            loss.backward()
            optimizer.step()

        #validation
        model.eval()
        with torch.no_grad():
            for i, batch in enumerate(val_loader):
                input_seq = batch['input'].to(device)
                output_seq = batch['output'].to(device)
                input_padding_mask = input_seq == PAD_IDX
                output_padding_mask = output_seq == PAD_IDX
                #input_mask, output_mask, input_padding_mask, output_padding_mask = create_mask(input_seq, output_seq)
                memory_key_padding_mask = input_padding_mask.clone()
                input_padding_mask = rearrange(input_padding_mask, 'n s -&gt; s n')
                output_padding_mask = rearrange(output_padding_mask, 'n s -&gt; s n')
                memory_key_padding_mask = rearrange(memory_key_padding_mask, 'n s -&gt; s n')
                
                tgt_sentence_len = output_seq.shape[0] - torch.sum(output_padding_mask,axis=1)
                tgt_inp, tgt_out = output_seq[:,:], output_seq[:,:]
                tgt_mask = gen_nopeek_mask(output_seq.shape[0])#.to('cuda') #('cuda)
                tgt_mask = tgt_mask.to(device)
                output = model(input_seq,tgt_inp,0,input_padding_mask, output_padding_mask,memory_key_padding_mask,tgt_mask)

                #generating one hot
                from_one_hot = torch.argmax(output,dim=2)

                output = output.view(-1, output.shape[-1])
                src_words = indices_to_string("ENGLISH",input_seq)
                predicted_words = indices_to_string("GERMAN",from_one_hot)
                tgt_words = indices_to_string("GERMAN",output_seq)
                print("Input English word - {}".format(src_words))
                print('Output German word - {}'.format(predicted_words))
                print("Actual German Word - {}".format(tgt_words))
                output_seq = output_seq.view(-1)
                val_loss = criterion(output,output_seq)

        print("Epoch - {}, Train Loss - {}, Val Loss - {}".format(epoch,loss.item(),val_loss.item()))
        writer.add_scalar("Train loss",loss, global_step=step)
        step+=1    

</code></pre>

ptrblck: <p>Your code seems to be running into a sticky error, which wasn’t raised properly. Could you update to the latest nightly binary and rerun your code again to check if the error message improves, please?</p>

Suraj520: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> : Thanks a lot ! It got resolved. <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"></p>

ptrblck: <p>That’s great! Would you mind posting the issue and the solution as I would be interested in learning more about it?</p>

Suraj520: <p>Actually I was solving the problem of Machine Translation via Seq2Seq LSTM encoder-decoder model and via Seq2Seq Transformer architecture.</p>
<p>I swapped the Transformer model with Seq2Seq LSTM encoder-decoder model for the dataloaders to check if it’s caused by the dimensionality errors related to I/O of layers inside the model. Turns out, It didn’t work out for Seq2Seq LSTM encoder-decoder too. To be Noted, The Seq2Seq model trained fine on Colab previously.</p>
<p>Haven’t been able to ensure that it runs on Colab, yet!</p>
<p>Colab has lately been throwing the following error too ! Restricting pip installs within runtime.</p>
<pre><code class="lang-auto">NotImplementedError: A UTF-8 locale is required. Got ANSI_X3.4-1968
</code></pre>
<p>Resolution -</p>
<pre><code class="lang-auto">import locale
def getpreferredencoding(do_setlocale = True):
    return "UTF-8"
locale.getpreferredencoding = getpreferredencoding
</code></pre>
<p>After doing this, I upgraded to the latest nightly release too  via the following command @</p>
<pre><code class="lang-auto">!pip install --pre torch torchvision -f https://download.pytorch.org/whl/nightly/cu117/torch_nightly.html -U --force
</code></pre>
<p>The error’s still not resolved !</p>
<p>Maybe, An error related to Google Colab !</p>
<blockquote>
<p>Reference Notebooks</p>
</blockquote>
<ol>
<li>
<p>Seq2Seq LSTM encoder-decoder - <a href="https://github.com/Suraj520/nlp-basics-to-advanced/blob/main/21_machine-translation/notebook.ipynb" rel="noopener nofollow ugc">https://github.com/Suraj520/nlp-basics-to-advanced/blob/main/21_machine-translation/notebook.ipynb</a></p>
</li>
<li>
<p>Seq2Seq Transformer - <a href="https://github.com/Suraj520/nlp-basics-to-advanced/blob/main/23_transformer-machine-translation/notebook.ipynb" rel="noopener nofollow ugc">https://github.com/Suraj520/nlp-basics-to-advanced/blob/main/23_transformer-machine-translation/notebook.ipynb</a></p>
</li>
</ol>

ptrblck: <p>I’m unsure if you are still running into the same issue or a new one using the latest nightly binaries.<br>
In any case, could you rerun the code with <code>CUDA_LAUNCH_BLOCKING=1</code> and check the stacktrace to see what exactly is failing?</p>

Suraj520: <aside class="quote no-group" data-username="ptrblck" data-post="8" data-topic="172376">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>CUDA_LAUNCH_BLOCKING=1</p>
</blockquote>
</aside>
<p>Tried !</p>
<p>No resolve, Yet ! I tried to set <strong>CUDA_LAUNCH_BLOCKING=1</strong> discretely without ensuring force nightly install of PyTorch since it’s giving an error</p>
<pre><code class="lang-auto">RuntimeError: cuDNN error: CUDNN_STATUS_MAPPING_ERROR
</code></pre>
<p>which I believe was related to mismatch of the version between CUDA and CUDNN on host instance(Colab)</p>
<p>The persistent error log remains to be -</p>
<pre><code class="lang-auto">RuntimeError: philox_cuda_state for an unexpected CUDA generator used during capture. In regions captured by CUDA graphs, you may only use the default CUDA RNG generator on the device that's current when capture begins. If you need a non-default (user-supplied) generator, or a generator on another device, please file an issue.
</code></pre>
<p>P.S : To ensure, Dataloader’s multiprocessing was not serving the cause - Turned num_workers =1, too.<br>
Also tried with <strong>torch.backends.cudnn.benchmark = True</strong></p>

ptrblck: <p>Since you are still getting random error messages, the env variable was most likely not set correctly.<br>
Could you post a minimal and executable code snippet to reproduce the issue, please?</p>

Suraj520: <p>Thanks for your patience, I already set the environment variable using the following command</p>
<pre><code class="lang-auto">import os
os.environ['CUDA_LAUNCH_BLOCKING'] = "1"
</code></pre>
<p>An observation that I noticed was with or without the aforementioned snippet of setting environment variable, When I run the following training script of a simple sequential model in PyTorch. It runs effectively.</p>
<pre><code class="lang-auto">import torch
import math
import os
os.environ['CUDA_LAUNCH_BLOCKING'] = "1"

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print(device)

x = torch.linspace(-math.pi, math.pi, 2000)
y = torch.sin(x)
p = torch.tensor([1, 2, 3])
xx = x.unsqueeze(-1).pow(p)

model = torch.nn.Sequential(
    torch.nn.Linear(3, 1),
    torch.nn.Flatten(0, 1)
)
model = model.to(device)

loss_fn = torch.nn.MSELoss(reduction='sum')

learning_rate = 1e-6
for t in range(2000):
    xx = xx.to(device)
    y = y.to(device)
    y_pred = model(xx)
    loss = loss_fn(y_pred, y)
    if t % 100 == 99:
        print(t, loss.item())

    # Zero the gradients before running the backward pass.
    model.zero_grad()
    loss.backward()
</code></pre>
<blockquote>
<p>STDOUT<br>
cuda<br>
99 17250.890625<br>
199 17250.890625<br>
299 17250.890625<br>
399 17250.890625<br>
499 17250.890625<br>
599 17250.890625<br>
699 17250.890625<br>
799 17250.890625<br>
899 17250.890625<br>
999 17250.890625<br>
1099 17250.890625<br>
1199 17250.890625<br>
1299 17250.890625<br>
1399 17250.890625<br>
1499 17250.890625<br>
1599 17250.890625<br>
1699 17250.890625<br>
1799 17250.890625<br>
1899 17250.890625<br>
1999 17250.890625</p>
</blockquote>
<p>But when I run the model with my data loader for the quoted task beforehand, neither it’s training loop starts, nor the simple sequential model’s training loop quoted above. In turn, both yield either the following Debug Log. It gets resolved when I restart the runtime and run the simple sequential model.</p>
<pre><code class="lang-auto">RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling `cublasSgemm( handle, opa, opb, m, n, k, &amp;alpha, a, lda, b, ldb, &amp;beta, c, ldc)`
</code></pre>
<p>I am still looking into what my model (Seq2Seq model or dataloader) is breaking !</p>

ptrblck: <p>Let me know if you can share a minimal and executable code snippet to reproduce the issue, so that I could try to debug it.</p>

Suraj520: <p>Here are the list of few things, I tried !</p>
<ol>
<li>Changed the transformer model to a simpler version. Snippet of which is quoted below</li>
</ol>
<pre><code class="lang-auto">import torch 
import torch.nn as nn

INPUT_DIM = 358#len(eng_vocab.stoi) 3
OUTPUT_DIM = 617#len(ger_vocab.stoi)
dim_model = 256
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        
        self.english_embedding = nn.Embedding(INPUT_DIM, dim_model)
        self.german_embedding = nn.Embedding(OUTPUT_DIM, dim_model)
        self.transformer = nn.Transformer(d_model=dim_model, 
            num_encoder_layers=2, num_decoder_layers=2, 
            dropout=0.5, dim_feedforward=2048)
        self.fc1 = nn.Linear(dim_model, OUTPUT_DIM)
    
    def forward(self, inputs, targets):
        x = self.english_embedding(inputs)
        y = self.german_embedding(targets)
        tgt_mask = torch.triu(torch.ones(targets.size(0), targets.size(0)), diagonal=1).bool().to(device)
        out = self.transformer(x, y, tgt_mask=tgt_mask)
        out = self.fc1(out.permute(1, 0, 2)) # (batch, sequence, feature)
        return out.permute(1, 0, 2).reshape(-1, OUTPUT_DIM) # (sequence, batch, feature)

model = Net().to(device)
</code></pre>
<ol start="2">
<li>Passed in a random tensor of size yielded by the data loader generator function where input_seq= english_language_tokens, output_seq = german_language_tokens. Snippet of which is quoted below</li>
</ol>
<pre><code class="lang-auto">num_epochs = 1000
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
criterion = nn.CrossEntropyLoss(ignore_index=0)
step=0

#Training
for epoch in range(num_epochs):
    model.train()
    input_seq = torch.randint(5, (4,2))
    output_seq = torch.randint(6,(6,2))
    input_seq = input_seq.to(device)
    output_seq = output_seq.to(device)
    pred = model(input_seq.to(device), output_seq[:-1,].to(device))
    loss = criterion(pred, output_seq[1:,].view(-1))
    loss.backward()
    optimizer.step()
    print("Epoch- {}, Loss- {}".format(epoch, loss))
    
</code></pre>
<p>Refer Image to see the dataloader’s I/O shape<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/e/eef106899b18a263f260a96ae68298dc3fd127af.png" data-download-href="https://discuss.pytorch.org/uploads/default/eef106899b18a263f260a96ae68298dc3fd127af" title="Screenshot from 2023-02-15 05-04-26"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/e/eef106899b18a263f260a96ae68298dc3fd127af_2_690x388.png" alt="Screenshot from 2023-02-15 05-04-26" data-base62-sha1="y5M224R83B5R24Un0cuotj9hD0H" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/e/eef106899b18a263f260a96ae68298dc3fd127af_2_690x388.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/e/eef106899b18a263f260a96ae68298dc3fd127af_2_1035x582.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/e/eef106899b18a263f260a96ae68298dc3fd127af_2_1380x776.png 2x" data-dominant-color="FBFBFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2023-02-15 05-04-26</span><span class="informations">1920×1080 149 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>If you run the snippets 1 and 2, You’ll see it runs perfectly !</p>
<ol start="3">
<li>Reduced the batch size to 1.<br>
However, When I pass in the data loader for english and german language tokens(original ones from the dataset), They run for an hour or so before the error whose log is quoted below is generated</li>
</ol>
<pre><code class="lang-auto">RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling `cublasLtMatmul( ltHandle, computeDesc.descriptor(), &amp;alpha_val, mat1_ptr, Adesc.descriptor(), mat2_ptr, Bdesc.descriptor(), &amp;beta_val, result_ptr, Cdesc.descriptor(), result_ptr, Cdesc.descriptor(), &amp;heuristicResult.algo, workspace.data_ptr(), workspaceSize, at::cuda::getCurrentCUDAStream())`
</code></pre>
<p>Refer Image -<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/f/3ff4bed1e06847e79b9d67766b089d5806057c35.png" data-download-href="https://discuss.pytorch.org/uploads/default/3ff4bed1e06847e79b9d67766b089d5806057c35" title="Screenshot from 2023-02-15 05-13-57"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/f/3ff4bed1e06847e79b9d67766b089d5806057c35_2_690x388.png" alt="Screenshot from 2023-02-15 05-13-57" data-base62-sha1="97MrBpTTRF2t8W6my3AsXyN37j7" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/f/3ff4bed1e06847e79b9d67766b089d5806057c35_2_690x388.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/f/3ff4bed1e06847e79b9d67766b089d5806057c35_2_1035x582.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/f/3ff4bed1e06847e79b9d67766b089d5806057c35_2_1380x776.png 2x" data-dominant-color="F9F9FA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2023-02-15 05-13-57</span><span class="informations">1920×1080 317 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<ol start="4">
<li>Did a sanity check of the dataloader to see if it breaks at some point or not !</li>
</ol>
<p>Refer Image<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/b/bb3377e2943a23b6310b4736eb6f67d4c6c1b10f.png" data-download-href="https://discuss.pytorch.org/uploads/default/bb3377e2943a23b6310b4736eb6f67d4c6c1b10f" title="Screenshot from 2023-02-15 05-22-07"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/b/bb3377e2943a23b6310b4736eb6f67d4c6c1b10f_2_690x388.png" alt="Screenshot from 2023-02-15 05-22-07" data-base62-sha1="qI3yygiNUUhJHtPlkWlSHlMRc9V" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/b/bb3377e2943a23b6310b4736eb6f67d4c6c1b10f_2_690x388.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/b/bb3377e2943a23b6310b4736eb6f67d4c6c1b10f_2_1035x582.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/b/bb3377e2943a23b6310b4736eb6f67d4c6c1b10f_2_1380x776.png 2x" data-dominant-color="F1F1F0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2023-02-15 05-22-07</span><span class="informations">1920×1080 144 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The entire source code can be found on -<br>
<a href="https://github.com/Suraj520/nlp-basics-to-advanced/blob/main/23_transformer-machine-translation/notebook.ipynb" class="onebox" target="_blank" rel="noopener nofollow ugc">https://github.com/Suraj520/nlp-basics-to-advanced/blob/main/23_transformer-machine-translation/notebook.ipynb</a></p>
<h3><a name="remarks-1" class="anchor" href="#remarks-1"></a>Remarks</h3>
<p>I found a similar issue - <a href="https://github.com/coqui-ai/TTS/issues/1517" class="inline-onebox" rel="noopener nofollow ugc">[Bug] RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling `cublasSgemm( handle, opa, opb, m, n, k, &amp;alpha, a, lda, b, ldb, &amp;beta, c, ldc)` when training tacotron2 · Issue #1517 · coqui-ai/TTS · GitHub</a><br>
Seems like it’s an issue related to Cuda Toolkit and PyTorch on Google Colab.</p>

ptrblck: <aside class="quote no-group" data-username="Suraj520" data-post="13" data-topic="172376">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/suraj520/48/56970_2.png" class="avatar"> Suraj520:</div>
<blockquote>
<p>However, When I pass in the data loader for english and german language tokens(original ones from the dataset), They run for an hour or so before the error whose log is quoted below is generated</p>
</blockquote>
</aside>
<p>Unfortunately, your current code does not reproduce the issue, so that I won’t be able to debug it.</p>
<aside class="quote no-group" data-username="Suraj520" data-post="13" data-topic="172376">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/suraj520/48/56970_2.png" class="avatar"> Suraj520:</div>
<blockquote>
<p>Seems like it’s an issue related to Cuda Toolkit and PyTorch on Google Colab.</p>
</blockquote>
</aside>
<p>I doubt it and would claim your input tensors might contain vocabulary indices which might be out of bounds for e.g. your embeddings.<br>
The error reporting might be broken if you are using <code>torch==1.13.1</code> and the blocking launches env variable might be set too late so that you are running into CUDA errors which look random.</p>

Suraj520: <p>Thanks ! I think I missed the OOV&gt; token while vocabulary building with the assumption that just the START&gt; and END&gt; should suffice. I did that with the assumption that vocabulary was built on the entire data frame content hence during training there won’t be any OOV&gt; token encountered. Though, It maybe encountered during inference.</p>
<p>This is one of the error that maybe causing the highlighted error ! Will recheck the code.</p>
<p>The explanation looks intuitive enough, especially looking at how code breaks at 19k steps of epoch 0 !</p>
<p>Thanks for your time and patience in the thread ! <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"></p>

ptrblck: <p>Sure, let me know once you’ve isolated the issue.<br>
For the sake of clarity: the error reporting on <code>torch==1.13.1</code> was not properly capturing all CUDA asserts and was thus causing error messages, which are not really helpful.<br>
Since you are seeing seemingly “random” CUDA errors, it seems you are also running into such a case.<br>
The current nightly binaries would fix it, so you could also consider updating, which should then yield a proper error message again.</p>

Suraj520: <p>Sure, Noted !</p>
<p>P.S: The previous reply somehow discarded tags OOV, START and END mentions. Maybe, due to closed &lt; pairs &gt;.</p>

bgoldfe2: <p>What does this code do and why is it necessary?</p>
<p>import os<br>
os.environ[‘CUDA_LAUNCH_BLOCKING’] = “1”</p>

ptrblck: <p><code>CUDA_LAUNCH_BLOCKING=1</code> is used for debugging and disables asynchronous kernel launches.<br>
I would not recommend setting it inside your Python script but to properly <code>export</code> this environment variable in your terminal.</p>