CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling `cublasCreate(handle)`

frisayl: <p>Hi, i’m having a tough time understanding what causes the indicated error. I am trying to develop a token classification model for ABSA with pytorch lightning and BERT, on colab, but got stuck at this error when calling the trainer. Can anybody give me some insight? Thanks in advance!</p>
<p>Here is the error traceback, if it can be useful.</p>
<p>/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py in fit(self, model, train_dataloaders, val_dataloaders, datamodule, train_dataloader, ckpt_path)<br>
736             train_dataloaders = train_dataloader<br>
737         self._call_and_handle_interrupt(<br>
 → 738             self._fit_impl, model, train_dataloaders, val_dataloaders, datamodule, ckpt_path<br>
739         )<br>
740</p>
<p>/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py in _call_and_handle_interrupt(self, trainer_fn, *args, **kwargs)<br>
680         “”"<br>
681         try:<br>
 → 682             return trainer_fn(*args, **kwargs)<br>
683         # TODO: treat KeyboardInterrupt as BaseException (delete the code below) in v1.7<br>
684         except KeyboardInterrupt as exception:</p>
<p>/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py in _fit_impl(self, model, train_dataloaders, val_dataloaders, datamodule, ckpt_path)<br>
770         # TODO: ckpt_path only in v1.7<br>
771         ckpt_path = ckpt_path or self.resume_from_checkpoint<br>
 → 772         self._run(model, ckpt_path=ckpt_path)<br>
773<br>
774         assert self.state.stopped</p>
<p>/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py in _run(self, model, ckpt_path)<br>
1193<br>
1194         # dispatch <code>start_training</code> or <code>start_evaluating</code> or <code>start_predicting</code><br>
 → 1195         self._dispatch()<br>
1196<br>
1197         # plugin will finalized fitting (e.g. ddp_spawn will load trained model)</p>
<p>/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py in _dispatch(self)<br>
1272             self.training_type_plugin.start_predicting(self)<br>
1273         else:<br>
 → 1274             self.training_type_plugin.start_training(self)<br>
1275<br>
1276     def run_stage(self):</p>
<p>/usr/local/lib/python3.7/dist-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py in start_training(self, trainer)<br>
200     def start_training(self, trainer: “pl.Trainer”) → None:<br>
201         # double dispatch to initiate the training loop<br>
 → 202         self._results = trainer.run_stage()<br>
203<br>
204     def start_evaluating(self, trainer: “pl.Trainer”) → None:</p>
<p>/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py in run_stage(self)<br>
1282         if self.predicting:<br>
1283             return self._run_predict()<br>
 → 1284         return self._run_train()<br>
1285<br>
1286     def _pre_training_routine(self):</p>
<p>/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py in _run_train(self)<br>
1304             self.progress_bar_callback.disable()<br>
1305<br>
 → 1306         self._run_sanity_check(self.lightning_module)<br>
1307<br>
1308         # enable train mode</p>
<p>/usr/local/lib/python3.7/dist-packages/pytorch_lightning/trainer/trainer.py in _run_sanity_check(self, ref_model)<br>
1368             # run eval step<br>
1369             with torch.no_grad():<br>
 → 1370                 self._evaluation_loop.run()<br>
1371<br>
1372             self.call_hook(“on_sanity_check_end”)</p>
<p>/usr/local/lib/python3.7/dist-packages/pytorch_lightning/loops/base.py in run(self, *args, **kwargs)<br>
143             try:<br>
144                 self.on_advance_start(*args, **kwargs)<br>
 → 145                 self.advance(*args, **kwargs)<br>
146                 self.on_advance_end()<br>
147                 self.restarting = False</p>
<p>/usr/local/lib/python3.7/dist-packages/pytorch_lightning/loops/dataloader/evaluation_loop.py in advance(self, *args, **kwargs)<br>
107         dl_max_batches = self._max_batches[dataloader_idx]<br>
108<br>
 → 109         dl_outputs = self.epoch_loop.run(dataloader, dataloader_idx, dl_max_batches, self.num_dataloaders)<br>
110<br>
111         # store batch level output per dataloader</p>
<p>/usr/local/lib/python3.7/dist-packages/pytorch_lightning/loops/base.py in run(self, *args, **kwargs)<br>
143             try:<br>
144                 self.on_advance_start(*args, **kwargs)<br>
 → 145                 self.advance(*args, **kwargs)<br>
146                 self.on_advance_end()<br>
147                 self.restarting = False</p>
<p>/usr/local/lib/python3.7/dist-packages/pytorch_lightning/loops/epoch/evaluation_epoch_loop.py in advance(self, data_fetcher, dataloader_idx, dl_max_batches, num_dataloaders)<br>
120         # lightning module methods<br>
121         with self.trainer.profiler.profile(“evaluation_step_and_end”):<br>
 → 122             output = self._evaluation_step(batch, batch_idx, dataloader_idx)<br>
123             output = self._evaluation_step_end(output)<br>
124</p>
<p>/usr/local/lib/python3.7/dist-packages/pytorch_lightning/loops/epoch/evaluation_epoch_loop.py in _evaluation_step(self, batch, batch_idx, dataloader_idx)<br>
215             self.trainer.lightning_module._current_fx_name = “validation_step”<br>
216             with self.trainer.profiler.profile(“validation_step”):<br>
 → 217                 output = self.trainer.accelerator.validation_step(step_kwargs)<br>
218<br>
219         return output</p>
<p>/usr/local/lib/python3.7/dist-packages/pytorch_lightning/accelerators/accelerator.py in validation_step(self, step_kwargs)<br>
234         “”"<br>
235         with self.precision_plugin.val_step_context():<br>
 → 236             return self.training_type_plugin.validation_step(*step_kwargs.values())<br>
237<br>
238     def test_step(self, step_kwargs: Dict[str, Union[Any, int]]) → Optional[STEP_OUTPUT]:</p>
<p>/usr/local/lib/python3.7/dist-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py in validation_step(self, *args, **kwargs)<br>
217<br>
218     def validation_step(self, *args, **kwargs):<br>
 → 219         return self.model.validation_step(*args, **kwargs)<br>
220<br>
221     def test_step(self, *args, **kwargs):</p>
<p> in validation_step(self, batch, batch_idx)<br>
46     attention_mask = batch[“attention_mask”]<br>
47     labels = batch[“labels”]<br>
—&gt; 48     loss, outputs = self(input_ids, attention_mask, labels)<br>
49     <span class="hashtag">#self</span>.log(“val_loss”, loss, prog_bar=True, logger=True)<br>
50     return loss</p>
<p>/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)<br>
1100         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks<br>
1101                 or _global_forward_hooks or _global_forward_pre_hooks):<br>
 → 1102             return forward_call(*input, **kwargs)<br>
1103         # Do not call functions when jit is used<br>
1104         full_backward_hooks, non_full_backward_hooks = [], []</p>
<p> in forward(self, input_ids, attention_mask, y)<br>
12   def forward(self, input_ids, attention_mask, y=None):<br>
13     print(‘input:’, input_ids.size(), ‘att’, attention_mask.size(), ‘y’, y.size())<br>
—&gt; 14     output = self.bert(input_ids, attention_mask, y)<br>
15     logits = output.logits<br>
16</p>
<p>/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)<br>
1118             input = bw_hook.setup_input_hook(input)<br>
1119<br>
 → 1120         result = forward_call(<em>input, **kwargs)<br>
1121         if _global_forward_hooks or self._forward_hooks:<br>
1122             for hook in (</em>_global_forward_hooks.values(), *self._forward_hooks.values()):</p>
<p>/usr/local/lib/python3.7/dist-packages/transformers/models/bert/modeling_bert.py in forward(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, labels, output_attentions, output_hidden_states, return_dict)<br>
1733             output_attentions=output_attentions,<br>
1734             output_hidden_states=output_hidden_states,<br>
 → 1735             return_dict=return_dict,<br>
1736         )<br>
1737</p>
<p>/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)<br>
1118             input = bw_hook.setup_input_hook(input)<br>
1119<br>
 → 1120         result = forward_call(<em>input, **kwargs)<br>
1121         if _global_forward_hooks or self._forward_hooks:<br>
1122             for hook in (</em>_global_forward_hooks.values(), *self._forward_hooks.values()):</p>
<p>/usr/local/lib/python3.7/dist-packages/transformers/models/bert/modeling_bert.py in forward(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, encoder_hidden_states, encoder_attention_mask, past_key_values, use_cache, output_attentions, output_hidden_states, return_dict)<br>
1004             output_attentions=output_attentions,<br>
1005             output_hidden_states=output_hidden_states,<br>
 → 1006             return_dict=return_dict,<br>
1007         )<br>
1008         sequence_output = encoder_outputs[0]</p>
<p>/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)<br>
1100         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks<br>
1101                 or _global_forward_hooks or _global_forward_pre_hooks):<br>
 → 1102             return forward_call(*input, **kwargs)<br>
1103         # Do not call functions when jit is used<br>
1104         full_backward_hooks, non_full_backward_hooks = [], []</p>
<p>/usr/local/lib/python3.7/dist-packages/transformers/models/bert/modeling_bert.py in forward(self, hidden_states, attention_mask, head_mask, encoder_hidden_states, encoder_attention_mask, past_key_values, use_cache, output_attentions, output_hidden_states, return_dict)<br>
588                     encoder_attention_mask,<br>
589                     past_key_value,<br>
 → 590                     output_attentions,<br>
591                 )<br>
592</p>
<p>/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)<br>
1100         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks<br>
1101                 or _global_forward_hooks or _global_forward_pre_hooks):<br>
 → 1102             return forward_call(*input, **kwargs)<br>
1103         # Do not call functions when jit is used<br>
1104         full_backward_hooks, non_full_backward_hooks = [], []</p>
<p>/usr/local/lib/python3.7/dist-packages/transformers/models/bert/modeling_bert.py in forward(self, hidden_states, attention_mask, head_mask, encoder_hidden_states, encoder_attention_mask, past_key_value, output_attentions)<br>
473             head_mask,<br>
474             output_attentions=output_attentions,<br>
 → 475             past_key_value=self_attn_past_key_value,<br>
476         )<br>
477         attention_output = self_attention_outputs[0]</p>
<p>/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)<br>
1100         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks<br>
1101                 or _global_forward_hooks or _global_forward_pre_hooks):<br>
 → 1102             return forward_call(*input, **kwargs)<br>
1103         # Do not call functions when jit is used<br>
1104         full_backward_hooks, non_full_backward_hooks = [], []</p>
<p>/usr/local/lib/python3.7/dist-packages/transformers/models/bert/modeling_bert.py in forward(self, hidden_states, attention_mask, head_mask, encoder_hidden_states, encoder_attention_mask, past_key_value, output_attentions)<br>
405             encoder_attention_mask,<br>
406             past_key_value,<br>
 → 407             output_attentions,<br>
408         )<br>
409         attention_output = self.output(self_outputs[0], hidden_states)</p>
<p>/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)<br>
1100         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks<br>
1101                 or _global_forward_hooks or _global_forward_pre_hooks):<br>
 → 1102             return forward_call(*input, **kwargs)<br>
1103         # Do not call functions when jit is used<br>
1104         full_backward_hooks, non_full_backward_hooks = [], []</p>
<p>/usr/local/lib/python3.7/dist-packages/transformers/models/bert/modeling_bert.py in forward(self, hidden_states, attention_mask, head_mask, encoder_hidden_states, encoder_attention_mask, past_key_value, output_attentions)<br>
264         output_attentions=False,<br>
265     ):<br>
 → 266         mixed_query_layer = self.query(hidden_states)<br>
267<br>
268         # If this is instantiated as a cross-attention module, the keys</p>
<p>/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)<br>
1100         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks<br>
1101                 or _global_forward_hooks or _global_forward_pre_hooks):<br>
 → 1102             return forward_call(*input, **kwargs)<br>
1103         # Do not call functions when jit is used<br>
1104         full_backward_hooks, non_full_backward_hooks = [], []</p>
<p>/usr/local/lib/python3.7/dist-packages/torch/nn/modules/linear.py in forward(self, input)<br>
101<br>
102     def forward(self, input: Tensor) → Tensor:<br>
 → 103         return F.linear(input, self.weight, self.bias)<br>
104<br>
105     def extra_repr(self) → str:</p>
<p>/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py in linear(input, weight, bias)<br>
1846     if has_torch_function_variadic(input, weight, bias):<br>
1847         return handle_torch_function(linear, (input, weight, bias), input, weight, bias=bias)<br>
 → 1848     return torch._C._nn.linear(input, weight, bias)<br>
1849<br>
1850</p>
<p>RuntimeError: CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling <code>cublasCreate(handle)</code></p>

ptrblck: <p>This issue might be caused if you are running out of memory and cublas isn’t able to create its handle.<br>
Reduce the batch size (or try to reduce the memory usage otherwise) and rerun the code.</p>

frisayl: <p>Thanks for your reply. Unfortunately, these measures aren’t solving my problem <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=10" title=":confused:" class="emoji" alt=":confused:"></p>