question: RuntimeError: CUDA error: CUBLAS_STATUS_ALLOC_FAILED when calling `cublasCreate(handle)`
context: Hi, everyone.
  I have a repo whose link is https://github.com/zhoufengfan/pytorch-lightning-cifar10.
  When I run python train.py, it returns this error:
  Traceback (most recent call last):
    File "train.py", line 21, in <module>
      trainer.fit(model, cifar10_dm)
    File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 458, in fit
      self._run(model)
    File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 756, in _run
      self.dispatch()
    File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 797, in dispatch
      self.accelerator.start_training(self)
    File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 96, in start_training
      self.training_type_plugin.start_training(trainer)
    File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 144, in start_training
      self._results = trainer.run_stage()
    File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 807, in run_stage
      return self.run_train()
    File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 842, in run_train
      self.run_sanity_check(self.lightning_module)
    File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1107, in run_sanity_check
      self.run_evaluation()
    File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 962, in run_evaluation
      output = self.evaluation_loop.evaluation_step(batch, batch_idx, dataloader_idx)
    File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/evaluation_loop.py", line 174, in evaluation_step
      output = self.trainer.accelerator.validation_step(args)
    File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 226, in validation_step
      return self.training_type_plugin.validation_step(*args)
    File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/ddp.py", line 322, in validation_step
      return self.model(*args, **kwargs)
    File "/root/miniconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
      result = self.forward(*input, **kwargs)
    File "/root/miniconda3/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 619, in forward
      output = self.module(*inputs[0], **kwargs[0])
    File "/root/miniconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
      result = self.forward(*input, **kwargs)
    File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/overrides/base.py", line 57, in forward
      output = self.module.validation_step(*inputs, **kwargs)
    File "/root/code/test-of-pytorch-lightning/backbone.py", line 42, in validation_step
      self.evaluate(batch, 'val')
    File "/root/code/test-of-pytorch-lightning/backbone.py", line 32, in evaluate
      logits = self(x)
    File "/root/miniconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
      result = self.forward(*input, **kwargs)
    File "/root/code/test-of-pytorch-lightning/backbone.py", line 20, in forward
      out = self.model(x)
    File "/root/miniconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
      result = self.forward(*input, **kwargs)
    File "/root/miniconda3/lib/python3.8/site-packages/torchvision/models/resnet.py", line 220, in forward
      return self._forward_impl(x)
    File "/root/miniconda3/lib/python3.8/site-packages/torchvision/models/resnet.py", line 215, in _forward_impl
      x = self.fc(x)
    File "/root/miniconda3/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
      result = self.forward(*input, **kwargs)
    File "/root/miniconda3/lib/python3.8/site-packages/torch/nn/modules/linear.py", line 93, in forward
      return F.linear(input, self.weight, self.bias)
    File "/root/miniconda3/lib/python3.8/site-packages/torch/nn/functional.py", line 1690, in linear
      ret = torch.addmm(bias, input, weight.t())
  RuntimeError: CUDA error: CUBLAS_STATUS_ALLOC_FAILED when calling `cublasCreate(handle)`
  
  But after I change the AVAIL_GPUS  in the hyper_var.py file to min(1, torch.cuda.device_count()), the bug disappear()(see the comment in the hyper_var.py file).
  When I change the  AVAIL_GPUS  in the hyper_var.py file to "1,2", it returns this error:
  Traceback (most recent call last):
    File "train.py", line 21, in <module>
      trainer.fit(model, cifar10_dm)
    File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 458, in fit
      self._run(model)
    File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 753, in _run
  Traceback (most recent call last):
    File "/root/code/test-of-pytorch-lightning/train.py", line 21, in <module>
      self.pre_dispatch()
    File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 778, in pre_dispatch
      trainer.fit(model, cifar10_dm)
    File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 458, in fit
      self.accelerator.pre_dispatch(self)
    File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 108, in pre_dispatch
      self.training_type_plugin.pre_dispatch()
    File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/ddp.py", line 279, in pre_dispatch
      self._run(model)
    File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 753, in _run
      self.barrier()
    File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/ddp.py", line 286, in barrier
      torch_distrib.barrier()
    File "/root/miniconda3/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 1960, in barrier
      self.pre_dispatch()
    File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 778, in pre_dispatch
      self.accelerator.pre_dispatch(self)
    File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/accelerators/accelerator.py", line 108, in pre_dispatch
      work = _default_pg.barrier()
      self.training_type_plugin.pre_dispatch()
    File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/ddp.py", line 279, in pre_dispatch
  RuntimeError: NCCL error in: /opt/conda/conda-bld/pytorch_1607370172916/work/torch/lib/c10d/ProcessGroupNCCL.cpp:784, unhandled system error, NCCL version 2.7.8
      self.barrier()
    File "/root/miniconda3/lib/python3.8/site-packages/pytorch_lightning/plugins/training_type/ddp.py", line 286, in barrier
      torch_distrib.barrier()
    File "/root/miniconda3/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 1960, in barrier
      work = _default_pg.barrier()
  RuntimeError: NCCL error in: /opt/conda/conda-bld/pytorch_1607370172916/work/torch/lib/c10d/ProcessGroupNCCL.cpp:784, unhandled system error, NCCL version 2.7.8
  
  The hardware parameters of my server are:
  NVIDIA GeForce 3090
  CUDA Version: 11.2
  Driver Version: 460.39
  
  Can anyone help me to fix the bug?
  Thanks.
answer: OK, I know the reason. The reason is that I haven't installed NCCL on my machine.
  After I installed NCCL on my machine, the errors disappeared.
  Thanks to all the people who have replied to me. @awaelchli @SkafteNicki