Change:
Steps to reproduce the behavior: magma From : Also may be interested.
Is this related to the race condition error you saw before?
can you tell me more about what hardware you're using, and what your looks like? For reference, runs without issue for me with using a 4090 and an i9 13900k: EDIT: Would you also try building with I made some changes there that moved the CUDA runtime stub and NVCC into . I'm curious if that helps at all.
I don't think so -- closest I can remember to this was the linking error AMD HIP had with 2.7.x (which is why its stuck on magma 2.6.x), but that was a different inscrutable error from .
CPU is a Xeon Gold 5218 with Quadro RTX 8000 GPUs. I haven't set any configuration other than and .
I don't know why this issue isn't affecting your builds but most likely the fix is to set :
Haven't tried yet either, sorry.
Any idea when this failure started occurring?
magma 2.6.2 I believe works but 2.7.1 fails. I can try to bisect at some point if needed.
What channel were you building from by the way? I'm gonna try to find a way to match your setup but I need to know exactly which version of nixpkgs you're using to build.
Okay, I was able to reproduce it. It happened while a Nixpkgs-review for that magma PR I linked earlier. I suspect it has something to do with either my disabling or restricting . Time to investigate ‍
Something for me to look at later tonight; if it is the case that the increased number of cuda capabilities being targeted is causing the binary to bloat, check that we're using , like PyTorch : IIRC Magma doesn't set that flag. Also, I remember being unable to find specifically in the NVCC docs. Since PyTorch also ships binaries targeting a bunch of different CUDA capabilities, their configs are a goldmine of flags we might need to look into. EDIT: Haven't built with it yet, but it seems like it should fix it. Apache MXNet had the same issue: EDIT2: Reminder to self: if that is the fix, add here: and here:
I know it's still a draft but can you try building again with I think it should fix your issue. Unrelated, but if you're building anything from source and you want faster builds I highly recommend specifying the single compute capability you need to build for (like I do here ) because it results in massively faster builds. HEAD right now builds for 14 different capabilities and that PR gets it down to 8, but it's much faster to build for just one.
is building for me from that branch (without any configuring compute capabilities yet), thanks.

Discussion:
# Refer to comments in the overrides for cuda_nvcc for explanation # CUDA_TOOLKIT_ROOT_DIR is legacy, # Cf. https://cmake.org/cmake/help/latest/module/FindCUDA.html#input-variables <ins> # NOTE: We unconditionally set -Xfatbin=-compress-all, which reduces the size of the compiled #   binaries. If binaries grow over 2GB, they will fail to link. This is a problem for us, as #   the default set of CUDA capabilities we build can regularly cause this to occur (for #   example, with Magma). </ins> '' mkdir -p $out/nix-support cat <<EOF >> $out/nix-support/setup-hook

if [ -z "''${CUDAHOSTCXX-}" ]; then export CUDAHOSTCXX=${backendStdenv.cc}/bin; fi <del> export NVCC_PREPEND_FLAGS+=' --compiler-bindir=${backendStdenv.cc}/bin' </del> <ins> export NVCC_PREPEND_FLAGS+=' --compiler-bindir=${backendStdenv.cc}/bin -Xfatbin=-compress-all' </ins> EOF # Move some libraries to the lib output so that programs that

}: # Type aliases <del> # Gpu = { #   archName: String, # e.g., "Hopper" #   computeCapability: String, # e.g., "9.0" #   minCudaVersion: String, # e.g., "11.8" #   maxCudaVersion: String, # e.g., "12.0" # } </del> <ins> # Gpu :: AttrSet #   - See the documentation in ./gpus.nix. </ins> let inherit (lib) attrsets lists strings trivial versions;

# gpus :: List Gpu gpus = builtins.import ./gpus.nix; <del> # isVersionIn :: Gpu -> Bool </del> <ins> # isSupported :: Gpu -> Bool </ins> isSupported = gpu: let inherit (gpu) minCudaVersion maxCudaVersion; lowerBoundSatisfied = strings.versionAtLeast cudaVersion minCudaVersion; <del> upperBoundSatisfied = !(strings.versionOlder maxCudaVersion cudaVersion); </del> <ins> upperBoundSatisfied = (maxCudaVersion == null) || !(strings.versionOlder maxCudaVersion cudaVersion); </ins> in lowerBoundSatisfied && upperBoundSatisfied; <ins> # isDefault :: Gpu -> Bool isDefault = gpu: let inherit (gpu) dontDefaultAfter; newGpu = dontDefaultAfter == null; recentGpu = newGpu || strings.versionAtLeast dontDefaultAfter cudaVersion; in recentGpu; </ins> # supportedGpus :: List Gpu # GPUs which are supported by the provided CUDA version. supportedGpus = builtins.filter isSupported gpus; <ins> # defaultGpus :: List Gpu # GPUs which are supported by the provided CUDA version and we want to build for by default. defaultGpus = builtins.filter isDefault supportedGpus; </ins> # supportedCapabilities :: List Capability supportedCapabilities = lists.map (gpu: gpu.computeCapability) supportedGpus; <ins> # defaultCapabilities :: List Capability # The default capabilities to target, if not overridden by the user. defaultCapabilities = lists.map (gpu: gpu.computeCapability) defaultGpus; </ins> # cudaArchNameToVersions :: AttrSet String (List String) # Maps the name of a GPU architecture to different versions of that architecture. # For example, "Ampere" maps to [ "8.0" "8.6" "8.7" ].

# dropDot :: String -> String inherit dropDot; } // formatCapabilities { <del> cudaCapabilities = config.cudaCapabilities or supportedCapabilities; </del> <ins> cudaCapabilities = config.cudaCapabilities or defaultCapabilities; </ins> enableForwardCompat = config.cudaForwardCompat or true; }

[ <ins> # Type alias # Gpu = { #   archName: String #     - The name of the microarchitecture. #   computeCapability: String #     - The compute capability of the GPU. #   minCudaVersion: String #     - The minimum (inclusive) CUDA version that supports this GPU. #   dontDefaultAfter: null | String #     - The CUDA version after which to exclude this GPU from the list of default capabilities #       we build. null means we always include this GPU in the default capabilities if it is #       supported. #   maxCudaVersion: null | String #     - The maximum (exclusive) CUDA version that supports this GPU. null means there is no #       maximum. # } </ins> { archName = "Kepler"; computeCapability = "3.0"; minCudaVersion = "10.0"; <ins> dontDefaultAfter = "10.2"; </ins> maxCudaVersion = "10.2"; } { archName = "Kepler"; computeCapability = "3.2"; minCudaVersion = "10.0"; <ins> dontDefaultAfter = "10.2"; </ins> maxCudaVersion = "10.2"; } { archName = "Kepler"; computeCapability = "3.5"; minCudaVersion = "10.0"; <ins> dontDefaultAfter = "11.0"; </ins> maxCudaVersion = "11.8"; } { archName = "Kepler"; computeCapability = "3.7"; minCudaVersion = "10.0"; <ins> dontDefaultAfter = "11.0"; </ins> maxCudaVersion = "11.8"; } { archName = "Maxwell"; computeCapability = "5.0"; minCudaVersion = "10.0"; <del> maxCudaVersion = "12.0"; </del> <ins> dontDefaultAfter = "11.0"; maxCudaVersion = null; </ins> } { archName = "Maxwell"; computeCapability = "5.2"; minCudaVersion = "10.0"; <del> maxCudaVersion = "12.0"; </del> <ins> dontDefaultAfter = "11.0"; maxCudaVersion = null; </ins> } { archName = "Maxwell"; computeCapability = "5.3"; minCudaVersion = "10.0"; <del> maxCudaVersion = "12.0"; </del> <ins> dontDefaultAfter = "11.0"; maxCudaVersion = null; </ins> } { archName = "Pascal"; computeCapability = "6.0"; minCudaVersion = "10.0"; <del> maxCudaVersion = "12.0"; </del> <ins> dontDefaultAfter = null; maxCudaVersion = null; </ins> } { archName = "Pascal"; computeCapability = "6.1"; minCudaVersion = "10.0"; <del> maxCudaVersion = "12.0"; </del> <ins> dontDefaultAfter = null; maxCudaVersion = null; </ins> } { archName = "Pascal"; computeCapability = "6.2"; minCudaVersion = "10.0"; <del> maxCudaVersion = "12.0"; </del> <ins> dontDefaultAfter = null; maxCudaVersion = null; </ins> } { archName = "Volta"; computeCapability = "7.0"; minCudaVersion = "10.0"; <del> maxCudaVersion = "12.0"; </del> <ins> dontDefaultAfter = null; maxCudaVersion = null; </ins> } { archName = "Volta"; computeCapability = "7.2"; minCudaVersion = "10.0"; <del> maxCudaVersion = "12.0"; </del> <ins> dontDefaultAfter = null; maxCudaVersion = null; </ins> } { archName = "Turing"; computeCapability = "7.5"; minCudaVersion = "10.0"; <del> maxCudaVersion = "12.0"; </del> <ins> dontDefaultAfter = null; maxCudaVersion = null; </ins> } { archName = "Ampere"; computeCapability = "8.0"; minCudaVersion = "11.2"; <del> maxCudaVersion = "12.0"; </del> <ins> dontDefaultAfter = null; maxCudaVersion = null; </ins> } { archName = "Ampere"; computeCapability = "8.6"; minCudaVersion = "11.2"; <del> maxCudaVersion = "12.0"; </del> <ins> dontDefaultAfter = null; maxCudaVersion = null; </ins> } { archName = "Ampere"; computeCapability = "8.7"; minCudaVersion = "11.5"; <del> maxCudaVersion = "12.0"; </del> <ins> # NOTE: This is purposefully before 11.5 to ensure it is never a capability we target by #   default. 8.7 is the Jetson Orin series of devices which are a very specific platform. #   We keep this entry here in case we ever want to target it explicitly, but we don't #   want to target it by default. dontDefaultAfter = "11.4"; maxCudaVersion = null; </ins> } { archName = "Ada"; computeCapability = "8.9"; minCudaVersion = "11.8"; <del> maxCudaVersion = "12.0"; </del> <ins> dontDefaultAfter = null; maxCudaVersion = null; </ins> } { archName = "Hopper"; computeCapability = "9.0"; minCudaVersion = "11.8"; <del> maxCudaVersion = "12.0"; </del> <ins> dontDefaultAfter = null; maxCudaVersion = null; </ins> } ]

# uses the last --compiler-bindir it gets on the command line. # FIXME: this results in "incompatible redefinition" warnings. # https://docs.nvidia.com/cuda/cuda-compiler-driver-nvcc/index.html#compiler-bindir-directory-ccbin <ins> # NOTE: We unconditionally set -Xfatbin=-compress-all, which reduces the size of the #   compiled binaries. If binaries grow over 2GB, they will fail to link. This is a problem #   for us, as the default set of CUDA capabilities we build can regularly cause this to #   occur (for example, with Magma). </ins> postInstall = (oldAttrs.postInstall or "") + '' mkdir -p $out/nix-support cat <<EOF >> $out/nix-support/setup-hook

if [ -z "''${CUDAHOSTCXX-}" ]; then export CUDAHOSTCXX=${cc}/bin; fi <del> export NVCC_PREPEND_FLAGS+=' --compiler-bindir=${cc}/bin' </del> <ins> export NVCC_PREPEND_FLAGS+=' --compiler-bindir=${cc}/bin -Xfatbin=-compress-all' </ins> EOF ''; });