Change:
After setting up faster-whisper for my Home Assistant instance, I learned that my CPU (i5-6500) isn't quite as fast as I'd like for this, so I am now trying to test a GPU (NVIDIA Quadro M2000). My configuration for it is the following: I expected it to then run on my GPU using CUDA, but unless my GPU and CPU are continually equally performant, it is not working. (Also, no processes are shown in ) Steps to reproduce the behavior: NVIDIA GPU, and then configure it accordingly in , add , , , and to faster-whisper in configuration, configure it to work with device = "cuda" for STT to be faster, instead same speed I'm not totally sure what to expect because this is my first time using this, but I expected a Python3 process to appear in and for my STT to be quicker. My config is the following: The following (seemingly) NVIDIA related items are at /dev/ ! ! works, but I am not totally sure that my CUDA is configured correctly, for all I know possibly CUDA isn't configured properly, but I think I have everything correct.
I eyeballed CUDA support, but have no hardware to test or reproduce. I appreciate help in fixing the module to make it work with CUDA.
I am not overly proficient in Nix or anything like this, but I would be willing to help if you have ideas.
Well, for one we're not passing in . Can you try setting ?
Ok, I that and now when looking at the logs, I get this: It then exits.
That looks actionable. We did indeed not build ctranslate2 with CUDA support. I can try to set that up, but it won't be the default, because that would make the package unfree. We possibly need multiple ctranslate2 attributes.
Is it possible to have the package be free unless an option is passed that enables CUDA and then makes it unfree, or would two different packages be needed. There could be a package, I think that could work.
Yeah, that would also work.
If you want to try and build it with CUDA support, I can test it out and report back.
fixes the module, so it passes as expected. It also adds CUDA support to . Check the description of the PR on how to opt into CUDA support in nixpkgs.
Hi Just to clarify, you only need in your (or in your 's ) if you're setting up a development environment to write cuda programs. If you only want to use packages with cuda support enabled, you just use them directly (e.g. ), no need to add anywhere manually. If you want to enable cuda in every package that supports it, you can set in your (nb: this triggers a massive rebuild). Instead of adding , you want to use (this way if you switch e.g. -, you'd be still using for the correct kernel), but also instead of you can take (keeps working if you switch e.g. -) Sorry we didn't get this documented yet
Wow, thank you! I assume I still need in my , right? About enabling , I've already enabled that and let it go through that whole process, so I think I'll just leave it enabled. About changing the , how is that to be done exactly? Do I just swap for in the exact same spot?
Also, is there an easy way for me to test out this updated package before it gets merged?
Depends on how you deploy your machine and what nixos version you are on.
I'm on Unstable and use Flakes.
You could replace your nixpkgs input with a fork of nixos-unstable and pick the commits from that PR on top. I'm sure there is a better solution, but I'm not well versed in flakes.
No, is the old version of , and you don't need it in Yes, except it's just , without at the end. This refers to "whatever graphics driver is in use". But also you can just remove this if you use and (both options are badly named, they're not really specific to neither of opengl or x11)
I have , but I don't have xserver enabled because this is a headless server. Do I even need nvidia_x11 if it's headless with no DE?
is where comes from; I don't recall now if it's ok to use on a headless machine, but if you like to just be explicit, you can keep P.S. Discourse and NixOS matrix channels provide venues for Q&A discussions
With the changes made to ctranslate2, and the fixes to th , I fully expect this could work. If not, please get back to me with some error messages, e.g. the hardening of the systemd unit might cause problems.
No problem. I attempted to use just the new packages, but it became a little difficult because my old nixpkgs didn't think that ctranslate2 could have CUDA enabled, so I'll just wait for master to get pushed to NixOS-Unstable (Hopefully soon!)
The package now fails with . My system may not be correctly configured for CUDA support, or the SystemD Service is not passing through the NVIDIA device correctly. Just a reminder, the following devices are available at : (I'm not sure if is an NVIDIA thing, but it exists if that's of importance.)
Then we're at least missing . What's the content of ? nix DeviceAllow = if [ "cuda" "auto" ] then [ # "/dev/nvidia1" "/dev/nvidia2" "/dev/nvidia3" "/dev/nvidia4" "/dev/nvidia-caps/nvidia-cap1" "/dev/nvidia-caps/nvidia-cap2" "/dev/nvidiactl" "/dev/nvidia-modeset" "/dev/nvidia-uvm" "/dev/nvidia-uvm-tools" ] else "";
I modified the SystemD service to include , and it didn't help at all. The contents of are . Also, I just discovered something new. When running I get So possibly updating this system now broke NVIDIA support and it just so happened to be at the same time I tried to get Whisper working.
Did you also check which is in ? There are two things you need to maintain: 1) your driver should be newer than the cuda libraries used to build your program, 2) your should come from the same (version of) package as the kernel modules (in other words, currently you have to reboot if you update your nixos and get a newer - the new kernel module is only loaded on the next boot)
shows the following for Also, I just rebooted and is working.
from what I listed is accessible from , should my system have functional CUDA support? I tried using some testing parts from , but it seems to be out of date. PS: Should this conversation be continued somewhere else, or is it fine to continue it here because it is somewhat pertinent to the issue?
Yes please, e.g. in or , or on You can just post the final update here once you get things to work, for discoverability purposes. Thanks!

Discussion:
--data-dir $STATE_DIRECTORY  --download-dir $STATE_DIRECTORY  --uri ${options.uri}  <ins> --device ${options.device}  </ins> --model ${options.model}  --language ${options.language}  --beam-size ${options.beamSize} ${options.extraArgs}

, darwin # Accelerate , llvmPackages # openmp , withMkl ? false, mkl <ins> , withCUDA ? false , withCuDNN ? false , cudaPackages </ins> # Enabling both withOneDNN and withOpenblas is broken # https://github.com/OpenNMT/CTranslate2/issues/1294 , withOneDNN ? false, oneDNN

nativeBuildInputs = [ cmake <ins> ] ++ lib.optionals withCUDA [ cudaPackages.cuda_nvcc </ins> ]; cmakeFlags = [

# https://github.com/OpenNMT/CTranslate2/blob/54810350e662ebdb01ecbf8e4a746f02aeff1dd7/python/tools/prepare_build_environment_linux.sh#L53 # https://github.com/OpenNMT/CTranslate2/blob/59d223abcc7e636c1c2956e62482bc3299cc7766/python/tools/prepare_build_environment_macos.sh#L12 "-DOPENMP_RUNTIME=COMP" <ins> "-DWITH_CUDA=${cmakeBool withCUDA}" "-DWITH_CUDNN=${cmakeBool withCuDNN}" </ins> "-DWITH_DNNL=${cmakeBool withOneDNN}" "-DWITH_OPENBLAS=${cmakeBool withOpenblas}" "-DWITH_RUY=${cmakeBool withRuy}"

buildInputs = lib.optionals withMkl [ mkl <ins> ] ++ lib.optionals withCUDA [ cudaPackages.cuda_cudart cudaPackages.libcublas cudaPackages.libcurand ] ++ lib.optionals withCuDNN [ cudaPackages.cudnn </ins> ] ++ lib.optionals withOneDNN [ oneDNN ] ++ lib.optionals withOpenblas [

cpp-jwt = callPackage ../development/libraries/cpp-jwt { }; <del> ctranslate2 = callPackage ../development/libraries/ctranslate2 { }; </del> <ins> ctranslate2 = callPackage ../development/libraries/ctranslate2 { stdenv = if pkgs.config.cudaSupport then gcc11Stdenv else stdenv; withCUDA = pkgs.config.cudaSupport; withCuDNN = pkgs.config.cudaSupport; }; </ins> ubus = callPackage ../development/libraries/ubus { };