Failed to create cublas handle: cublas error
Author: cosmo3769
Pull request: no

### System Info

Kaggle with Accelerator **GPU P100**

### Who can help?

@amyeroberts 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Steps to Reproduce the behaviour: 

1. Go to Kaggle. Create a new notebook. Switch to **GPU P100** accelerator.
2. Use `TFSegformerForSemanticSegmentation`: 

```
from transformers import TFSegformerForSemanticSegmentation

model_checkpoint = "nvidia/mit-b0"
id2label = {0: "outer", 1: "inner", 2: "border"}
label2id = {label: id for id, label in id2label.items()}
num_labels = len(id2label)
model = TFSegformerForSemanticSegmentation.from_pretrained(
    model_checkpoint,
    num_labels=num_labels,
    id2label=id2label,
    label2id=label2id,
    ignore_mismatched_sizes=True,
)
```

While running this code block, I get this `cublas` error: 

```
2023-04-12 02:44:14.603089: E tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:219] failed to create cublas handle: cublas error
2023-04-12 02:44:14.603208: E tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:221] Failure to initialize cublas may be due to OOM (cublas needs some free memory when you initialize it, and your deep-learning framework may have preallocated more than its fair share), or may be because this binary was not built with support for the GPU in your machine.
---------------------------------------------------------------------------
NotFoundError                             Traceback (most recent call last)
/tmp/ipykernel_23/279563332.py in <module>
     10     id2label=id2label,
     11     label2id=label2id,
---> 12     ignore_mismatched_sizes=True,
     13 )

/opt/conda/lib/python3.7/site-packages/transformers/modeling_tf_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
   2764                 model(model.dummy_inputs)  # build the network with dummy inputs
   2765         else:
-> 2766             model(model.dummy_inputs)  # build the network with dummy inputs
   2767 
   2768         if safetensors_from_pt:

/opt/conda/lib/python3.7/site-packages/keras/utils/traceback_utils.py in error_handler(*args, **kwargs)
     68             # To get the full stack trace, call:
     69             # `tf.debugging.disable_traceback_filtering()`
---> 70             raise e.with_traceback(filtered_tb) from None
     71         finally:
     72             del filtered_tb

/opt/conda/lib/python3.7/site-packages/transformers/modeling_tf_utils.py in run_call_with_unpacked_inputs(self, *args, **kwargs)
    430 
    431         unpacked_inputs = input_processing(func, config, **fn_args_and_kwargs)
--> 432         return func(self, **unpacked_inputs)
    433 
    434     # Keras enforces the first layer argument to be passed, and checks it through `inspect.getfullargspec()`. This

/opt/conda/lib/python3.7/site-packages/transformers/models/segformer/modeling_tf_segformer.py in call(self, pixel_values, labels, output_attentions, output_hidden_states, return_dict)
    859             output_attentions=output_attentions,
    860             output_hidden_states=True,  # we need the intermediate hidden states
--> 861             return_dict=return_dict,
    862         )
    863 

/opt/conda/lib/python3.7/site-packages/transformers/modeling_tf_utils.py in run_call_with_unpacked_inputs(self, *args, **kwargs)
    430 
    431         unpacked_inputs = input_processing(func, config, **fn_args_and_kwargs)
--> 432         return func(self, **unpacked_inputs)
    433 
    434     # Keras enforces the first layer argument to be passed, and checks it through `inspect.getfullargspec()`. This

/opt/conda/lib/python3.7/site-packages/transformers/models/segformer/modeling_tf_segformer.py in call(self, pixel_values, output_attentions, output_hidden_states, return_dict, training)
    484             output_hidden_states=output_hidden_states,
    485             return_dict=return_dict,
--> 486             training=training,
    487         )
    488         sequence_output = encoder_outputs[0]

/opt/conda/lib/python3.7/site-packages/transformers/models/segformer/modeling_tf_segformer.py in call(self, pixel_values, output_attentions, output_hidden_states, return_dict, training)
    414             embedding_layer, block_layer, norm_layer = x
    415             # first, obtain patch embeddings
--> 416             hidden_states, height, width = embedding_layer(hidden_states)
    417 
    418             # second, send embeddings through blocks

/opt/conda/lib/python3.7/site-packages/transformers/models/segformer/modeling_tf_segformer.py in call(self, pixel_values)
     87 
     88     def call(self, pixel_values: tf.Tensor) -> Tuple[tf.Tensor, int, int]:
---> 89         embeddings = self.proj(self.padding(pixel_values))
     90         height = shape_list(embeddings)[1]
     91         width = shape_list(embeddings)[2]

NotFoundError: Exception encountered when calling layer 'proj' (type Conv2D).

{{function_node __wrapped__Conv2D_device_/job:localhost/replica:0/task:0/device:GPU:0}} No algorithm worked!  Error messages:
  Profiling failure on CUDNN engine 1: UNKNOWN: CUDNN_STATUS_EXECUTION_FAILED
in tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc(4294): 'cudnnConvolutionForward( cudnn.handle(), alpha, input_nd_.handle(), input_data.opaque(), filter_.handle(), filter_data.opaque(), conv_.handle(), ToConvForwardAlgo(algo), scratch_memory.opaque(), scratch_memory.size(), beta, output_nd_.handle(), output_data.opaque())'
  Profiling failure on CUDNN engine 0: UNKNOWN: CUDNN_STATUS_EXECUTION_FAILED
in tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc(4294): 'cudnnConvolutionForward( cudnn.handle(), alpha, input_nd_.handle(), input_data.opaque(), filter_.handle(), filter_data.opaque(), conv_.handle(), ToConvForwardAlgo(algo), scratch_memory.opaque(), scratch_memory.size(), beta, output_nd_.handle(), output_data.opaque())'
  Profiling failure on CUDNN engine 2: UNKNOWN: CUDNN_STATUS_EXECUTION_FAILED
in tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc(4294): 'cudnnConvolutionForward( cudnn.handle(), alpha, input_nd_.handle(), input_data.opaque(), filter_.handle(), filter_data.opaque(), conv_.handle(), ToConvForwardAlgo(algo), scratch_memory.opaque(), scratch_memory.size(), beta, output_nd_.handle(), output_data.opaque())' [Op:Conv2D]

Call arguments received by layer 'proj' (type Conv2D):
  • inputs=tf.Tensor(shape=(3, 518, 518, 3), dtype=float32)
```


### Expected behavior

When I run the PyTorch version of `Segformer`, it loads the model successfully.

```
from transformers import SegformerForSemanticSegmentation

model_checkpoint = "nvidia/mit-b0"
id2label = {0: "outer", 1: "inner", 2: "border"}
label2id = {label: id for id, label in id2label.items()}
num_labels = len(id2label)
model = SegformerForSemanticSegmentation.from_pretrained(
    model_checkpoint,
    num_labels=num_labels,
    id2label=id2label,
    label2id=label2id,
    ignore_mismatched_sizes=True,
)
```

Comments:
Hi @cosmo3769, thanks for reporting this issue. I'll look into it. 

p.s. I think you might have tagged the wrong Amy :) 

> Hi @cosmo3769, thanks for reporting this issue. I'll look into it.

Sure.

> p.s. I think you might have tagged the wrong Amy :)

Oh yeah, sorry for that. 😅


@cosmo3769 It seems this issue might be coming from the kaggle notebook setup and/or hardware. I'm able to run the snippet loading `TFSegformerForSemanticSegmentation` without issue on a linux machine with 2 GPUs. 

Could you share some more information about the running environment: copy-paste the output of running `! transformers-cli env` in a cell. 

Looking up the error online, some other users have reported similar issues (e.g. [here](https://stackoverflow.com/questions/53698035/failed-to-get-convolution-algorithm-this-is-probably-because-cudnn-failed-to-in)) which were resolved with setting `os.environ['TF_FORCE_GPU_ALLOW_GROWTH'] = 'true'` which is simple enough to warrant a try :) 



Yes, it solves the issue. Thanks.