Change:
singularity cannot use nvidia gpu because it does not access 'nvidia-container-cli' Steps to reproduce the behavior: return result of ! This is because the path is not listed in , as also in a. It is caused by neither the option nor . I tried to handle it with override, but since it was called in , I think it is impossible to handle. Also, I created a conf file in as shown in the singularity docs, but it was not read. <!-- Please @ people who are in the list of the offending package or module. If in doubt, check for whoever last touched something. --Please run and paste the result.
Thank you for reporting and sorry for the trouble. I'll look at the SingularityCE documentation again. (Those configuration options has been deprecated by Apptainer.) Also, I created a conf file in /usr/local/etc/singularity/ as shown in the singularity docs, but it was not read.  shold be what you are looking for. The overriden package can be assigned to . (If you would like to bypass all the overriding done by the module and directly specify the package to install for some special reason, you can assign it to .)
Thanks for the reply. I followed your advice and tried the override and was able to use the GPU. Here is the code This is simply adding a process to append the nvidia-docker path to the I mentioned earlier that when conf was entered in , it was not reflected, and the same is true for '/etc/singularity'. I referred to .
At least we have confirmed that in works and the path is replaced by , but the product does not seem to be recognizable.
Does setting the to make ldconfig search in help? This is modified from the configuration proposed by (). There's also (probably) related to this topic.
Adding the path of to works well. This once removed from consideration. Next we have a patched version of ldconfig. This does not append the path of to the config. The result was an error that could not be found, and it could not be started.
Oh, I already forgot about this issue. Absolutely coincidentally, but maybe efforts such as might open up a way to implement this hack more reliably. Meanwhile, in my config I'm still abusing You probably already have guessed, but singularity in nixpkgs is configured to read this config from the package-specific directory, rather than from the global . You could in principle override this behaviour, but there maybe isn't much value in supporting that
I tried to enable the use of external sysconfdir (), but have encountered issue trying to specify a directory under with the NixOS module ().
Have you tried to apply both at the same time?
The SingularityCE Admin Guide says If the value of nvidia-container-cli path is empty, SingularityCE will look for the binary on $PATH at runtime. It should work as long as is found in the run-time , even when is not set. As nvidia-container-cli is alredy included in the PATH-prefixing wrapper, we shouldn't have to set the .
Update: I found a bug in the prefixed PATH of the wrapper introduced in . The prefixing should be specified as (in Bash) I forgot to add , causing the trailing to be missing.
The statement in the SingularityCE Admin Guide turns out to be wrong / irrelevant. The source code clearly states that nvidia-container-cli will only be read from the configuration.
(we need to patch nvidia-container-cli, for many reasons )
Eh, actually, closed semi-automatically. Apptainer with is fixed (on NixOS), but not with . The fix doesn't concern singularity-ce either, unfortunately. Shouldn't be hard to make this work with singularity too, but I'm out of bandwidth
Hi folks, I've been trying to get apptainer to run this (which I've done previously successfully with a slight ldconfig hack -- that doesn't work anymore -- on NixOS 22.11), that will make use of the machine's Tesla V100 to do some simulation stuff. The machine is running NixOS 23.11. Using the workaround from 9th of May post further up, I was able to get past the ldconfig issue, but now I'm getting an error saying the driver and CUDA are not in sync. Before I start posting any details (I don't want to re-open this issue, but I'm also not sure if I'm just doing something wrong) I was wondering if the current implementation of singularity/apptainer on NixOS is expected to work, and I'm just being an idiot, or if there are known breaking issues? I've seen the other issues about singularity/apptainer here, but I'm not knowledgeable enough to make a judgement call if they are responsible for my problem or not. Thanks very much in advance!
(If I understood the message correctly) That usually means you need a reboot, and you should experience the same issue outside the container
No, the reboot didn't help either. For some reason, though, after upgrading to 24.05, everything "just works[tm]" again. So, whatever the issue was, thanks for fixing it (probably)! :)
Thanks to Nixpkgs 24.05 includes GPU-related changes/fixes to the package and the module.
Well more like thanks to ShamrockLee

Discussion:
}; enableSuid = mkOption { type = types.bool; <del> default = true; </del> <ins> # SingularityCE requires SETUID for most things. Apptainer prefers user # namespaces, e.g. `apptainer exec --nv` would fail if built # `--with-suid`: # > `FATAL: nvidia-container-cli not allowed in setuid mode` default = cfg.package.projectName != "apptainer"; defaultText = literalExpression ''config.services.singularity.package.projectName != "apptainer"''; </ins> example = false; description = mdDoc '' Whether to enable the SUID support of Singularity/Apptainer.

<ins> From 783ec26c0d83013baf04579a6a415d7f8776ac93 Mon Sep 17 00:00:00 2001 From: Someone Serge <sergei.kozlukov@aalto.fi> Date: Sun, 7 Jan 2024 11:48:24 +0000 Subject: [PATCH] ldCache(): patch for @driverLink@ --- internal/pkg/util/paths/resolve.go | 41 +++++++++++++++++++++++++++--- 1 file changed, 38 insertions(+), 3 deletions(-) diff --git a/internal/pkg/util/paths/resolve.go b/internal/pkg/util/paths/resolve.go index db45d9db1..9d0110b6b 100644 --- a/internal/pkg/util/paths/resolve.go +++ b/internal/pkg/util/paths/resolve.go @@ -14,6 +14,7 @@ import ( "fmt" "os" "os/exec" +	"path" "path/filepath" "regexp" "strings" @@ -154,14 +155,49 @@ func Resolve(fileList []string) ([]string, []string, error) { // lists three variants of libEGL.so.1 that are in different locations, we only // report the first, highest priority, variant. func ldCache() (map[string]string, error) { +    driverDirs := strings.Split("@driverLink@/lib", ":") +    if machine, err := elfMachine(); err == nil && machine == elf.EM_386 { +        driverDirs = strings.Split("@driverLink@-32/lib", ":") +    } + +    soPattern, err := regexp.Compile(`[^s]+.so(.d+(.d+(.d+)?)?)?$`) +    if err != nil { +		return nil, fmt.Errorf("could not compile ldconfig regexp: %v", err) +    } + +	ldCache := make(map[string]string) +    for _, dirPath := range driverDirs { +        dir, err := os.Open(dirPath) +        if err != nil { +            /* Maybe we're not running under NixOS */ +            continue +        } +        files, err := dir.ReadDir(-1) +        if err != nil { +            continue +        } +        for _, f := range files { +            if !soPattern.MatchString(f.Name()) { +                continue +            } +            libName := f.Name() +            libPath := path.Join(dirPath, f.Name()) +			if _, ok := ldCache[libName]; !ok { +				ldCache[libName] = libPath +			} +        } +    } + // walk through the ldconfig output and add entries which contain the filenames // returned by nvidia-container-cli OR the nvliblist.conf file contents ldconfig, err := bin.FindBin("ldconfig") -	if err != nil { +	if err != nil && len(ldCache) == 0 { +        // Note that missing ldconfig is only an "error" as long +        // as there's no driverLink return nil, err } out, err := exec.Command(ldconfig, "-p").Output() -	if err != nil { +	if err != nil && len(ldCache) == 0 { return nil, fmt.Errorf("could not execute ldconfig: %v", err) } @@ -173,7 +209,6 @@ func ldCache() (map[string]string, error) { } // store library name with associated path -	ldCache := make(map[string]string) for _, match := range r.FindAllSubmatch(out, -1) { if match != nil { // libName is the "libnvidia-ml.so.1" (from the above example) -- 2.42.0 </ins>

, buildGoModule , runCommandLocal # Native build inputs <ins> , addDriverRunpath </ins> , makeWrapper , pkg-config , util-linux , which # Build inputs , bash <ins> , callPackage </ins> , conmon , coreutils , cryptsetup

, hello # Overridable configurations , enableNvidiaContainerCli ? true <ins> # --nvccli currently requires extra privileges: # https://github.com/apptainer/apptainer/issues/1893#issuecomment-1881240800 , forceNvcCli ? false </ins> # Compile with seccomp support # SingularityCE 3.10.0 and above requires explicit --without-seccomp when libseccomp is not available. , enableSeccomp ? true

# Whether to compile with SUID support , enableSuid ? false , starterSuidPath ? null <ins> , substituteAll </ins> # newuidmapPath and newgidmapPath are to support --fakeroot # where those SUID-ed executables are unavailable from the FHS system PATH. # Path to SUID-ed newuidmap executable

(buildGoModule { inherit pname version src; <ins> patches = lib.optionals (projectName == "apptainer") [ (substituteAll { src = ./apptainer/0001-ldCache-patch-for-driverLink.patch; inherit (addDriverRunpath) driverLink; }) ]; </ins> # Override vendorHash with the output got from # nix-prefetch -E "{ sha256 }: ((import ./. { }).apptainer.override { vendorHash = sha256; }).goModules" # or with `null` when using vendored source tarball.

if [[ ! -e .git || ! -e VERSION ]]; then echo "${version}" > VERSION fi # Patch shebangs for script run during build patchShebangs --build "$configureScript" makeit e2e scripts mlocal/scripts # Patching the hard-coded defaultPath by prefixing the packages in defaultPathInputs substituteInPlace cmd/internal/cli/actions.go  --replace "defaultPath = "${defaultPathOriginal}"" "defaultPath = "''${defaultPathInputs// //bin:}''${defaultPathInputs:+/bin:}${defaultPathOriginal}"" <ins> substituteInPlace internal/pkg/util/gpu/nvidia.go  --replace  'return fmt.Errorf("/usr/bin not writable in the container")'  "" </ins> ''; postConfigure = ''

wrapProgram "$out/bin/${projectName}"  --prefix PATH : "''${defaultPathInputs// //bin:}''${defaultPathInputs:+/bin:}" # Make changes in the config file <del> ${lib.optionalString enableNvidiaContainerCli '' </del> <ins> ${lib.optionalString forceNvcCli '' </ins> substituteInPlace "$out/etc/${projectName}/${projectName}.conf"  --replace "use nvidia-container-cli = no" "use nvidia-container-cli = yes" ''}

singularity = finalAttrs.finalPackage; }; }; <ins> gpuChecks = lib.optionalAttrs (projectName == "apptainer") { # Should be in tests, but Ofborg would skip image-hello-cowsay because # saxpy is unfree. image-saxpy = callPackage ({ singularity-tools, cudaPackages }: singularity-tools.buildImage { name = "saxpy"; contents = [ cudaPackages.saxpy ]; memSize = 2048; diskSize = 2048; singularity = finalAttrs.finalPackage; }) { }; saxpy = callPackage ({ runCommand, writeShellScriptBin }: let unwrapped = writeShellScriptBin "apptainer-cuda-saxpy" '' ${lib.getExe finalAttrs.finalPackage} exec --nv $@ ${finalAttrs.passthru.tests.image-saxpy} saxpy ''; in runCommand "run-apptainer-cuda-saxpy" { requiredSystemFeatures = [ "cuda" ]; nativeBuildInputs = [ unwrapped ]; passthru = { inherit unwrapped; }; } '' apptainer-cuda-saxpy '') { }; }; </ins> }; })

buildInputs = lib.optionals (lib.versionOlder cudaVersion "11.4") [cudatoolkit] ++ lib.optionals (lib.versionAtLeast cudaVersion "11.4") [ <del> libcublas </del> <ins> libcublas.dev libcublas.lib libcublas.static </ins> cuda_cudart ] ++ lib.optionals (lib.versionAtLeast cudaVersion "12.0") [cuda_cccl];