Repository: NixOS/nixpkgs
Pull request #395147: nix-prefetch-scripts: just don't depend on Nix
Author: login: K900
Labels: name: 6.topic: rust
description: General-purpose programming language emphasizing performance, type safety, and concurrency.
name: 6.topic: ruby
description: A dynamic, open source programming language with a focus on simplicity and productivity.
name: 10.rebuild-linux: 101-500
description: This PR causes between 101 and 500 packages to rebuild on Linux.
name: 10.rebuild-darwin: 101-500
description: This PR causes between 101 and 500 packages to rebuild on Darwin.
Created: 2025-04-01T11:24:15Z | Merged: 2025-04-02T09:08:35Z

If we're running it from a build (fetchgit, fetchCargoVendor, etc), none of the code paths that actually call Nix are hit.

If we're _not_ running it from a build, the user can reasonably be expected to have a Nix present ambiently.

This avoids pulling Nix into the build closure of fetchCargoVendor, and thus causing infrecs when Nix depends on Rust things.


<!--
^ Please summarise the changes you have done and explain why they are necessary here ^

For package updates please link to a changelog or describe changes, this helps your fellow maintainers discover breaking updates.
For new packages please briefly describe the package or provide a link to its homepage.
-->

## Things done

<!-- Please check what applies. Note that these are not hard requirements but merely serve as information for reviewers. -->

- Built on platform(s)
  - [x] x86_64-linux
  - [ ] aarch64-linux
  - [ ] x86_64-darwin
  - [ ] aarch64-darwin
- For non-Linux: Is sandboxing enabled in `nix.conf`? (See [Nix manual](https://nixos.org/manual/nix/stable/command-ref/conf-file.html))
  - [ ] `sandbox = relaxed`
  - [ ] `sandbox = true`
- [ ] Tested, as applicable:
  - [NixOS test(s)](https://nixos.org/manual/nixos/unstable/index.html#sec-nixos-tests) (look inside [nixos/tests](https://github.com/NixOS/nixpkgs/blob/master/nixos/tests))
  - and/or [package tests](https://github.com/NixOS/nixpkgs/blob/master/pkgs/README.md#package-tests)
  - or, for functions and "core" functionality, tests in [lib/tests](https://github.com/NixOS/nixpkgs/blob/master/lib/tests) or [pkgs/test](https://github.com/NixOS/nixpkgs/blob/master/pkgs/test)
  - made sure NixOS tests are [linked](https://github.com/NixOS/nixpkgs/blob/master/pkgs/README.md#linking-nixos-module-tests-to-a-package) to the relevant packages
- [ ] Tested compilation of all packages that depend on this change using `nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD"`. Note: all changes have to be committed, also see [nixpkgs-review usage](https://github.com/Mic92/nixpkgs-review#usage)
- [ ] Tested basic functionality of all binary files (usually in `./result/bin/`)
- [25.05 Release Notes](https://github.com/NixOS/nixpkgs/blob/master/nixos/doc/manual/release-notes/rl-2505.section.md) (or backporting [24.11](https://github.com/NixOS/nixpkgs/blob/master/nixos/doc/manual/release-notes/rl-2411.section.md) and [25.05](https://github.com/NixOS/nixpkgs/blob/master/nixos/doc/manual/release-notes/rl-2505.section.md) Release notes)
  - [ ] (Package updates) Added a release notes entry if the change is major or breaking
  - [ ] (Module updates) Added a release notes entry if the change is significant
  - [ ] (Module addition) Added a release notes entry if adding a new NixOS module
- [x] Fits [CONTRIBUTING.md](https://github.com/NixOS/nixpkgs/blob/master/CONTRIBUTING.md).

<!--
To help with the large amounts of pull requests, we would appreciate your
reviews of other pull requests, especially simple package updates. Just leave a
comment describing what you have tested in the relevant package/service.
Reviewing helps to reduce the average time-to-merge for everyone.
Thanks a lot if you do!

List of open PRs: https://github.com/NixOS/nixpkgs/pulls
Reviewing guidelines: https://github.com/NixOS/nixpkgs/blob/master/pkgs/README.md#reviewing-contributions
-->

---

Add a :+1: [reaction] to [pull requests you find important].

[reaction]: https://github.blog/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/
[pull requests you find important]: https://github.com/NixOS/nixpkgs/pulls?q=is%3Aopen+sort%3Areactions-%2B1-desc



Diff:
diff --git a/pkgs/applications/version-management/mercurial/default.nix b/pkgs/applications/version-management/mercurial/default.nix
index dac493910d9aa..3b6270bf8c65e 100644
--- a/pkgs/applications/version-management/mercurial/default.nix
+++ b/pkgs/applications/version-management/mercurial/default.nix
@@ -55,7 +55,6 @@ let
         rustPlatform.fetchCargoVendor {
           inherit src;
           name = "mercurial-${version}";
-          allowGitDependencies = false;
           hash = "sha256-k/K1BupCqnlB++2T7hJxu82yID0jG8HwLNmb2eyx29o=";
           sourceRoot = "mercurial-${version}/rust";
         }
diff --git a/pkgs/build-support/rust/fetch-cargo-vendor.nix b/pkgs/build-support/rust/fetch-cargo-vendor.nix
index ce015b099382e..5f5a164407785 100644
--- a/pkgs/build-support/rust/fetch-cargo-vendor.nix
+++ b/pkgs/build-support/rust/fetch-cargo-vendor.nix
@@ -35,9 +35,6 @@ in
   name ? if args ? pname && args ? version then "${args.pname}-${args.version}" else "cargo-deps",
   hash ? (throw "fetchCargoVendor requires a `hash` value to be set for ${name}"),
   nativeBuildInputs ? [ ],
-  # This is mostly for breaking infinite recursion where dependencies
-  # of nix-prefetch-git use fetchCargoVendor.
-  allowGitDependencies ? true,
   ...
 }@args:
 
@@ -58,15 +55,13 @@ let
 
       impureEnvVars = lib.fetchers.proxyImpureEnvVars;
 
-      nativeBuildInputs =
-        [
-          fetchCargoVendorUtil
-          cacert
-        ]
-        ++ lib.optionals allowGitDependencies [
-          nix-prefetch-git
-        ]
-        ++ nativeBuildInputs;
+      nativeBuildInputs = [
+        fetchCargoVendorUtil
+        cacert
+        # break loop of nix-prefetch-git -> git-lfs -> asciidoctor -> ruby (yjit) -> fetchCargoVendor -> nix-prefetch-git
+        # Cargo does not currently handle git-lfs: https://github.com/rust-lang/cargo/issues/9692
+        (nix-prefetch-git.override { git-lfs = null; })
+      ] ++ nativeBuildInputs;
 
       buildPhase = ''
         runHook preBuild
diff --git a/pkgs/by-name/md/mdbook/package.nix b/pkgs/by-name/md/mdbook/package.nix
index cd6c6ebe79b91..21d3dd72fba71 100644
--- a/pkgs/by-name/md/mdbook/package.nix
+++ b/pkgs/by-name/md/mdbook/package.nix
@@ -22,11 +22,8 @@ rustPlatform.buildRustPackage rec {
     hash = "sha256-XTvC2pGRVat0kOybNb9TziG32wDVexnFx2ahmpUFmaA=";
   };
 
-  cargoDeps = rustPlatform.fetchCargoVendor {
-    inherit pname version src;
-    allowGitDependencies = false;
-    hash = "sha256-ASPRBAB+elJuyXpPQBm3WI97wD3mjoO1hw0fNHc+KAw=";
-  };
+  useFetchCargoVendor = true;
+  cargoHash = "sha256-ASPRBAB+elJuyXpPQBm3WI97wD3mjoO1hw0fNHc+KAw=";
 
   nativeBuildInputs = [ installShellFiles ];
 
diff --git a/pkgs/development/compilers/rust/cargo-auditable.nix b/pkgs/development/compilers/rust/cargo-auditable.nix
index a8cd829c59d0a..95143226b041d 100644
--- a/pkgs/development/compilers/rust/cargo-auditable.nix
+++ b/pkgs/development/compilers/rust/cargo-auditable.nix
@@ -21,7 +21,6 @@ let
 
     cargoDeps = rustPlatform.fetchCargoVendor {
       inherit pname version src;
-      allowGitDependencies = false;
       hash = "sha256-oTPGmoGlNfPVZ6qha/oXyPJp94fT2cNlVggbIGHf2bc=";
     };
 
diff --git a/pkgs/development/interpreters/ruby/default.nix b/pkgs/development/interpreters/ruby/default.nix
index befb945e1fe42..20aee447d6aa4 100644
--- a/pkgs/development/interpreters/ruby/default.nix
+++ b/pkgs/development/interpreters/ruby/default.nix
@@ -224,7 +224,6 @@ let
               rustPlatform.fetchCargoVendor {
                 inherit (finalAttrs) src;
                 sourceRoot = "${finalAttrs.pname}-${version}/${finalAttrs.cargoRoot}";
-                allowGitDependencies = false;
                 hash =
                   assert cargoHash != null;
                   cargoHash;
diff --git a/pkgs/tools/package-management/lix/default.nix b/pkgs/tools/package-management/lix/default.nix
index 5c351b2542a28..d6b44473fd9c9 100644
--- a/pkgs/tools/package-management/lix/default.nix
+++ b/pkgs/tools/package-management/lix/default.nix
@@ -89,7 +89,6 @@ lib.makeExtensible (self: {
       docCargoDeps = rustPlatform.fetchCargoVendor {
         name = "lix-doc-${version}";
         inherit src;
-        allowGitDependencies = false;
         sourceRoot = "${src.name or src}/lix-doc";
         hash = "sha256-VPcrf78gfLlkTRrcbLkPgLOk0o6lsOJBm6HYLvavpNU=";
       };
@@ -120,7 +119,6 @@ lib.makeExtensible (self: {
       docCargoDeps = rustPlatform.fetchCargoVendor {
         name = "lix-doc-${version}";
         inherit src;
-        allowGitDependencies = false;
         sourceRoot = "${src.name or src}/lix-doc";
         hash = "sha256-U820gvcbQIBaFr2OWPidfFIDXycDFGgXX1NpWDDqENs=";
       };
diff --git a/pkgs/tools/package-management/nix-prefetch-scripts/default.nix b/pkgs/tools/package-management/nix-prefetch-scripts/default.nix
index 059b06362f16a..040535f63ed80 100644
--- a/pkgs/tools/package-management/nix-prefetch-scripts/default.nix
+++ b/pkgs/tools/package-management/nix-prefetch-scripts/default.nix
@@ -13,18 +13,6 @@
   git-lfs,
   gnused,
   mercurial,
-  # FIXME: These scripts should not depend on Nix, they should depend on a
-  # `.nar` hasher compatible with Nix.
-  #
-  # The fact that these scripts depend on Nix means that e.g. Chromium depends
-  # on Nix.
-  #
-  # Also should be fixed:
-  # - prefetch-yarn-deps
-  # - nurl, nix-init
-  #
-  # Gridlock is one such candidate: https://github.com/lf-/gridlock
-  nixForLinking,
   subversion,
 }:
 
@@ -49,7 +37,6 @@ let
               ++ [
                 coreutils
                 gnused
-                nixForLinking
               ]
             )
           } \
@@ -66,6 +53,9 @@ let
     };
 in
 rec {
+  # No explicit dependency on Nix, as these can be used inside builders,
+  # and thus will cause dependency loops. When used _outside_ builders,
+  # we expect people to have a Nix implementation available ambiently.
   nix-prefetch-bzr = mkPrefetchScript "bzr" ../../../build-support/fetchbzr/nix-prefetch-bzr [
     breezy
   ];
diff --git a/pkgs/tools/text/mdbook-linkcheck/default.nix b/pkgs/tools/text/mdbook-linkcheck/default.nix
index 3d544fd4068db..474f4f5004c54 100644
--- a/pkgs/tools/text/mdbook-linkcheck/default.nix
+++ b/pkgs/tools/text/mdbook-linkcheck/default.nix
@@ -21,11 +21,8 @@ rustPlatform.buildRustPackage rec {
     sha256 = "sha256-ZbraChBHuKAcUA62EVHZ1RygIotNEEGv24nhSPAEj00=";
   };
 
-  cargoDeps = rustPlatform.fetchCargoVendor {
-    inherit pname version src;
-    allowGitDependencies = false;
-    hash = "sha256-Tt7ljjWv2CMtP/ELZNgSH/ifmBk/42+E0r9ZXQEJNP8=";
-  };
+  useFetchCargoVendor = true;
+  cargoHash = "sha256-Tt7ljjWv2CMtP/ELZNgSH/ifmBk/42+E0r9ZXQEJNP8=";
 
   buildInputs = if stdenv.hostPlatform.isDarwin then [ Security ] else [ openssl ];
 


Reviews:
user: login: roberth
state: APPROVED
author_association: MEMBER
submitted_at: 2025-04-01T12:30:02Z
user: login: lf-
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-01T16:03:27Z
user: login: K900
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-01T16:04:27Z
user: login: lf-
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-02T06:26:47Z
user: login: TomaSajt
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-02T06:43:45Z
user: login: K900
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-02T06:45:01Z
user: login: TomaSajt
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-02T07:38:54Z
user: login: TomaSajt
body: LGTM

Hopefully we don't break anything.
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-02T07:39:57Z

Review comments:
diff_hunk: @@ -35,9 +35,6 @@ in
   name ? if args ? pname && args ? version then "${args.pname}-${args.version}" else "cargo-deps",
   hash ? (throw "fetchCargoVendor requires a `hash` value to be set for ${name}"),
   nativeBuildInputs ? [ ],
-  # This is mostly for breaking infinite recursion where dependencies
-  # of nix-prefetch-git use fetchCargoVendor.
-  allowGitDependencies ? true,
path: pkgs/build-support/rust/fetch-cargo-vendor.nix
user: login: lf-
body: does this blow up the lix derivation lmao
created_at: 2025-04-01T16:03:27Z
author_association: MEMBER
diff_hunk: @@ -35,9 +35,6 @@ in
   name ? if args ? pname && args ? version then "${args.pname}-${args.version}" else "cargo-deps",
   hash ? (throw "fetchCargoVendor requires a `hash` value to be set for ${name}"),
   nativeBuildInputs ? [ ],
-  # This is mostly for breaking infinite recursion where dependencies
-  # of nix-prefetch-git use fetchCargoVendor.
-  allowGitDependencies ? true,
path: pkgs/build-support/rust/fetch-cargo-vendor.nix
user: login: K900
body: In the flake? Yes
created_at: 2025-04-01T16:04:27Z
author_association: CONTRIBUTOR
diff_hunk: @@ -35,9 +35,6 @@ in
   name ? if args ? pname && args ? version then "${args.pname}-${args.version}" else "cargo-deps",
   hash ? (throw "fetchCargoVendor requires a `hash` value to be set for ${name}"),
   nativeBuildInputs ? [ ],
-  # This is mostly for breaking infinite recursion where dependencies
-  # of nix-prefetch-git use fetchCargoVendor.
-  allowGitDependencies ? true,
path: pkgs/build-support/rust/fetch-cargo-vendor.nix
user: login: lf-
body: fuck lmao. we should probably get a change started to be compatible with this being present. idk why we have these bright ideas to workaround a problem without imagining what will happen when we properly fix it in nixpkgs but, well, oops!
created_at: 2025-04-02T06:26:46Z
author_association: MEMBER
diff_hunk: @@ -35,9 +35,6 @@ in
   name ? if args ? pname && args ? version then "${args.pname}-${args.version}" else "cargo-deps",
   hash ? (throw "fetchCargoVendor requires a `hash` value to be set for ${name}"),
   nativeBuildInputs ? [ ],
-  # This is mostly for breaking infinite recursion where dependencies
-  # of nix-prefetch-git use fetchCargoVendor.
-  allowGitDependencies ? true,
path: pkgs/build-support/rust/fetch-cargo-vendor.nix
user: login: TomaSajt
body: It shouldn't blow it up: this attr will just be a NOOP.
(We don't have to have it explicitly be destructured from the function input attrset)
created_at: 2025-04-02T06:43:44Z
author_association: CONTRIBUTOR
diff_hunk: @@ -35,9 +35,6 @@ in
   name ? if args ? pname && args ? version then "${args.pname}-${args.version}" else "cargo-deps",
   hash ? (throw "fetchCargoVendor requires a `hash` value to be set for ${name}"),
   nativeBuildInputs ? [ ],
-  # This is mostly for breaking infinite recursion where dependencies
-  # of nix-prefetch-git use fetchCargoVendor.
-  allowGitDependencies ? true,
path: pkgs/build-support/rust/fetch-cargo-vendor.nix
user: login: K900
body: Oh yeah, true, it takes `...`, so we should be good.
created_at: 2025-04-02T06:45:01Z
author_association: CONTRIBUTOR
diff_hunk: @@ -66,6 +53,9 @@ let
     };
 in
 rec {
+  # No explicit dependency on Nix, as these can be used inside builders,
+  # and thus will cause dependency loops. When used _outside_ builders,
+  # we expect people to have a Nix implementation available ambiently.
path: pkgs/tools/package-management/nix-prefetch-scripts/default.nix
user: login: TomaSajt
body: Only `nix-prefetch-git` can be used.
Though we can keep the comment in this place, it's fine.
created_at: 2025-04-02T07:38:54Z
author_association: CONTRIBUTOR

PR comments:
user: login: TomaSajt
created_at: 2025-04-01T12:23:22Z
body: btw `fetchgit` doesn't depend on `nix-prefetch-git`-the-package, but on `nix-prefetch-git`-the-script-file.

`fetchgit` provides its own `git` and `git-lfs` to the build environment the script is run in (but not `nix`).

Personally, I feel like `nix-prefetch-git`-the-package should declare its dependencies...
But I don't care *that* much.

In any case, my other PR would still have to remove `git-lfs`...

author_association: CONTRIBUTOR
user: login: K900
created_at: 2025-04-01T12:25:42Z
body: Ideally we'd refactor this into two scripts - nix-fetch-git or whatever for the actual fetching, and then a second wrapper that just does the store path munging. That way tools can depend on the former, and the latter can depend on Nix. 
author_association: CONTRIBUTOR
user: login: alois31
created_at: 2025-04-01T12:52:12Z
body: Can we also remove the `allowGitDependencies` hack then?
author_association: CONTRIBUTOR
user: login: TomaSajt
created_at: 2025-04-01T12:54:10Z
body: Nope, because `git-lfs` depends on `asciidoctor` (which depends on `ruby` (which depends on `fetchCargoVendor`))
author_association: CONTRIBUTOR
user: login: alois31
created_at: 2025-04-01T13:05:49Z
body: So what exactly are we gaining here then? Unless we want to make only git-lfs optional and not the entirety of git.
author_association: CONTRIBUTOR
user: login: K900
created_at: 2025-04-01T13:43:49Z
body: The only source of the infrec on git-lfs is Ruby, as far as I can tell. We can probably special case that?
author_association: CONTRIBUTOR
user: login: K900
created_at: 2025-04-01T14:11:04Z
body: Some looking later, Cargo does not support git-lfs at all, so stubbing it out is a non-issue, at least for now.
author_association: CONTRIBUTOR
user: login: TomaSajt
created_at: 2025-04-01T14:25:58Z
body: So: we're dropping nix from the `nix-prefetch-git` script because in reality, no nixpkgs fetcher should ever use the nix part of the script.
And: this is better than the override method because we don't have to do it everywhere.
But: this will reduce the "pureness" of the package.
However: it's not that bad, since we're expected to have nix anyway.

Should the dropping of `allowGitDependencies` be part of this PR?

author_association: CONTRIBUTOR
user: login: K900
created_at: 2025-04-01T14:34:59Z
body: > So: we're dropping nix from the nix-prefetch-git script because in reality, no nixpkgs fetcher should ever use the nix part of the script.

Yes.

> And: this is better than the override method because we don't have to do it everywhere.

Also yes.

> But: this will reduce the "pureness" of the package.

I guess so, but ideally the next step would be to split it into two scripts, one that depends on just git (and maybe git-lfs) and another that also depends on Nix. It's not trivial with the way the script is written though, so I feel like we can start here.

> However: it's not that bad, since we're expected to have nix anyway.

For the running it out of nix case, yes.

> Should the dropping of `allowGitDependencies` be part of this PR?

I don't really see why not?
author_association: CONTRIBUTOR
user: login: lf-
created_at: 2025-04-01T16:06:23Z
body: > But: this will reduce the "pureness" of the package.
However: it's not that bad, since we're expected to have nix anyway.

Sorta. It'll also increase the amount of it actually doing what the user asked for by not using a random nix it likes rather than the nix the user put in nix.package. 
author_association: MEMBER
user: login: K900
created_at: 2025-04-02T06:45:10Z
body: Do we all agree on this then? Should I send it?
author_association: CONTRIBUTOR
user: login: TomaSajt
created_at: 2025-04-02T06:47:52Z
body: I still want to check a few things with update scripts and other places `nix-prefetch-git` is used.
author_association: CONTRIBUTOR
user: login: K900
created_at: 2025-04-02T06:50:52Z
body: I've looked around and have not found any other in-sandbox invocations of nix-prefetch-git without `--builder`.
author_association: CONTRIBUTOR
user: login: TomaSajt
created_at: 2025-04-02T07:26:56Z
body: Okay, everything looks OK...
I saw that the other `fetch*` scripts don't even use `nix-prefetch-*`.

IMO there should be a comment in the file stating why nix isn't declared as an input. (For clarity's sake and so that people don't add it back.)

Also small question: do dependency cycles usually go in the direction?
What you wrote:
`nix-prefetch-git -> fetchCargoVendor -> ruby (yjit) -> asciidoctor -> git-lfs -> nix-prefetch-git`
What I think is more readable (going left to right, dependents pointing at dependencies)
`nix-prefetch-git -> git-lfs -> asciidoctor -> ruby (yjit) -> fetchCargoVendor -> nix-prefetch-git`

I wasn't able to find other chains like this in tree (because I guess I'm bad at grepping) so I don't know other examples to see which is usually used.

author_association: CONTRIBUTOR
user: login: K900
created_at: 2025-04-02T07:28:28Z
body: I don't think there's a hard rule, but yeah, it makes more sense to me the other way around too. I'll update the comments.
author_association: CONTRIBUTOR