Change:
The that's currently in nixpkgs uses , the official stack binary release uses For more details, I think the core issue is basically the same as I think the fix is to change the version of which the in nixpkgs is built with from (which is the version of the package in nixpkgs) to .
It would also be great to have some automation to make sure this doesn't regress. I guess that would involve downloading an official matching version stack binary and checking what the version is there (via ), if there is no better way to get the upstream hpack version.
In what way does this issue present? Supposedly stack has become more resistant to this problem: It seems ridiculous that we'd have to track what stack binaries ship for all eternity.
The concrete issue is that get stack from colleagues get stack from the official bindist add new source files run file gets regenerated to include the newer files, but also potentially other changes happen, because my hpack version is different (at the very least, the hpack version comment changes) commit the file, as recommended by the issue you linked my colleague pulls in my changes, they use an official stack release with a different hpack version, so they get arbitrary issues, depending on the different version. At the very least they get some weird mismatch, or the file doesn't get regenerated, etc. I don't really understand this - for dev facing tools, isn't it expected that would track the official releases? Otherwise this introduces reproducibility issues between getting your tools from and from official releases, hence I am now arbitrarily going to get breakage if I choose to use NixOS when collaborating with non-NixOS users (or I have to potentially apply some workarounds to be able to use official releases because official releases don't always work on NixOS).
But the real problem here is not an incompatibility between stack and hpack versions. It is like with very many tools, that you using another hpack version than your coworkers does not work. I am uncertain what the best solution for that is. But in general nixpkgs downgrading software because other distribution methods are not as fast is a bit unsatisfying. Additionally, I find it mighty confusing, that the stack binaries are not build with hpack version from stackage. Using the hpack version from stackage LTS, which we do, is the most blessing one can generally achieve for a package version. So all of this is me shouting like a five year old: “Not our fault!” The question is what to do about it. Have you considered using an older nixpkgs version which uses the right hpack version? Apparently the whole process is super sensitive to the hpack version, so if we downgrade hpack now we probably annoy other people. If there is no better solution maybe we need to supply different hpack versions and stack with different hpack versions and the user can choose?
I agree I agree I would guess that if you bless the version that the official stack bindist ships, nobody would complain, except maybe a single instance of other nixpkgs stack users, after which it should be fine? I guess a more principled approach would be to instead live with this for now and wait until stack bindists bump hpack version, at which point start tracking the hpack version provided with bindists. Of course I can work (and actively am working) around this. And also I do not expect that miraculously software from nixpkgs will be bug for bug compatible with official bindists. But this feels like a relatively simple(perhaps? I do not know how easy it is to have an additional version in nixpkgs) change to make, which results in actual observable incompatibility being reduced.
In the past we have overridden the version of used by to work around this issue. Just like is saying, it is very annoying to use from Nixpkgs with a team of people who otherwise get stack from upstream. I guess a newer hpack version was recently released and it slipped through. would you be able to send a PR overriding the hpack version used by stack so that it is in sync with upstream? If you could figure out a simple test to add in , that would be ideal as well.
I've created to adding a test. However, that requires an internet connection, and from what I understand, nixpkgs tests run in a sandbox, and that won't be available(?) Should I: somehow allow connection to the stack official bindist urls in CI (not sure how to do this) instead manually hardcode a mapping , which must have an entry for the stack version that's currently in nixpkgs, i.e. when a new stack version comes in, whoever is doing updates will be forced to update the mapping (not sure where to do this either) Additionally, I couldn't find how to make build with a specific version of - could you give me a pointer to where I could see an example/read up on this? Currently has , but I don't seem to understand where that is used explicitly (if it is at all).
That's correct. I like what you currently have in your commit, but you'll have to pin the upstream (non-nixpkgs) stack version with a commit hash (and ideally fetch it with something like or or something). Ideally you'd be able to write it so that when the stack version is bumped in Nixpkgs, the test will automatically start failing. Also, could you add yourself as a maintainer to the test, so that you'll be pinged when it starts failing? That would help with the maintenance of the test. A lot of these types of overrides are in . Here's an example of overriding the version of a direct dependency for a package: You'll likely have to do something similar for hpack with stack.
Currently / provides stack-2.9.1 as the top-level attribute (even though the latest release of is 2.9.3), and it uses hpack-0.35.2. Upstream uses the following hpack versions: stack-2.9.1: hpack-0.35.0 stack-2.9.3: hpack-0.35.1 I've just pushed in , which drops 's hpack dependency to 0.35.0, in-line with upstream . This should make sure people getting from Nixpkgs use a compatible version with people that get stack from upstream. I'm going to keep this issue open in case anyone is able to send a PR implementing some sort of test in Nixpkgs that upstream stack and our stack use the same hpack version.
Just so I understand this right: If someone's team uses “stack-2.9.3: hpack-0.35.1” but they themselves get our “stack-2.9.1: hpack-0.35.0” do they still have this problem?
Thanks and sorry for not getting back to this earlier. I'll still have in mind trying to write a test for this. It did somehow become an issue for my workplace, because this causes the original issue but in reverse - I was using a newer hpack(from nix) and someone was using the older one, causing some kind of issues for the other person. However, I currently can't dig out the exact complaints.
My understanding is that this is correct. If you're using + in a team setting, you have to make sure everyone uses the same version of , compiled with the same version of . (Although it might be the case that you only have to make sure the version is the same, you don't all necessarily have to use the same version of .)
What's the state of this issue?
I fixed the underlying problem in . But we don't have any assurance that the same problem won't crop up again the next time Stack releases a new version. I was leaving this issue open so that hopefully someone would send a PR adding some sort of test that the hpack version used by Stack in Nixpkgs is the same as the hpack version used by the upstream Stack release. Although I don't have a good idea of how to write this test in such a way that we won't have to manually bump a version/URL/hash every time there is a new release of Stack, which is quite annoying. It is the git-annex problem all over again.
I implemented the check for the upstream stack/hpack version in

Discussion:
setBuildTarget = callPackage ./setBuildTarget { }; writers = callPackage ./writers { }; incremental = callPackage ./incremental { }; <ins> upstreamStackHpackVersion = callPackage ./upstreamStackHpackVersion { }; </ins> }

<ins> # This derivation confirms that the version of hpack used by stack in Nixpkgs # is the exact same version as the upstream stack release. # # It is important to make sure the version of hpack used by stack in Nixpkgs # matches with the version of hpack used by the upstream stack release.  This # is because hpack works slightly differently based on the version, and it can # be frustrating to use hpack in a team setting when members are using different # versions. See for more info: https://github.com/NixOS/nixpkgs/issues/223390 # # This test is written as a fixed-output derivation, because we need to access # accesses the internet to download the upstream stack release. { cacert, curl, lib, stack, stdenv }: let # Find the hpack derivation that is a dependency of stack.  Throw exception # if hpack cannot be found. hpack = lib.findFirst (v: v.pname or "" == "hpack") (throw "could not find stack's hpack dependency") stack.passthru.getCabalDeps.executableHaskellDepends; # This is a statically linked version of stack, so it should be usable within # the Nixpkgs builder (at least on x86_64-linux). stackDownloadUrl = "https://github.com/commercialhaskell/stack/releases/download/v${stack.version}/stack-${stack.version}-linux-x86_64-static.tar.gz"; # This test code has been explicitly pulled out of the derivation below so # that it can be hashed and added to the `name` of the derivation.  This is # so that this test derivation won't be cached if the body of the test is # modified. # # WARNING: When modifying this script, make sure you don't introduce any # paths to the Nix store within it.  We only want this derivation to be re-run # when the stack version (or the version of its hpack dependency) changes in # Nixpkgs. testScript = '' curl=( curl --location --max-redirs 20 --retry 3 --disable-epsv --cookie-jar cookies --user-agent "curl " --insecure ) # Fetch the statically-linked upstream Stack binary. "''${curl[@]}" "${stackDownloadUrl}" > ./stack.tar.gz tar xf ./stack.tar.gz upstream_stack_version_output="$(./stack-${stack.version}-linux-x86_64-static/stack --version)" echo "upstream `stack --version` output: $upstream_stack_version_output" nixpkgs_stack_version_output="$(stack --version)" echo "nixpkgs `stack --version` output: $nixpkgs_stack_version_output" # Confirm that the upstream stack version is the same as the stack version # in Nixpkgs. This check isn't strictly necessary, but it is a good sanity # check. if [[ "$upstream_stack_version_output" =~ "Version "([0-9]+((.[0-9]+)+)) ]]; then upstream_stack_version="''${BASH_REMATCH[1]}" echo "parsed upstream stack version: $upstream_stack_version" echo "stack version from nixpkgs: ${stack.version}" if [[ "${stack.version}" != "$upstream_stack_version" ]]; then echo "ERROR: stack version in Nixpkgs (${stack.version}) does not match the upstream version for some reason: $upstream_stack_version" exit 1 fi else echo "ERROR: Upstream stack version cannot be found in --version output: $upstream_stack_version" exit 1 fi # Confirm that the hpack version used in the upstream stack release is the # same as the hpack version used by the Nixpkgs stack binary. if [[ "$upstream_stack_version_output" =~ hpack-([0-9]+((.[0-9]+)+)) ]]; then upstream_hpack_version="''${BASH_REMATCH[1]}" echo "parsed upstream stack's hpack version: $upstream_hpack_version" echo "Nixpkgs stack's hpack version: ${hpack.version}" if [[ "${hpack.version}" != "$upstream_hpack_version" ]]; then echo "ERROR: stack's hpack version in Nixpkgs (${hpack.version}) does not match the upstream stack's hpack version: $upstream_hpack_version" echo "The stack derivation in Nixpkgs needs to be fixed up so that it depends on hpack-$upstream_hpack_version, instead of ${hpack.name}" exit 1 fi else echo "ERROR: Upstream stack's hpack version cannot be found in --version output: $upstream_hpack_version" exit 1 fi # Output a string with a known hash. echo "success" > $out ''; testScriptHash = builtins.hashString "sha256" testScript; in stdenv.mkDerivation { # This name is very important. # # The idea here is that want this derivation to be re-run everytime the # version of stack (or the version of its hpack dependency) changes in # Nixpkgs.  We also want to re-run this derivation whenever the test script # is changed. # # Nix/Hydra will re-run derivations if their name changes (even if they are a # FOD and they have the same hash). # # The name of this derivation contains the stack version string, the hpack # version string, and a hash of the test script.  So Nix will know to # re-run this version when (and only when) one of those values change. name = "upstream-stack-hpack-version-test-${stack.name}-${hpack.name}-${testScriptHash}"; # This is the sha256 hash for the string "success", which is output upon this # test succeeding. outputHash = "sha256-gbK9TqmMjbZlVPvI12N6GmmhMPMx/rcyt1yqtMSGj9U="; outputHashMode = "flat"; outputHashAlgo = "sha256"; nativeBuildInputs = [ curl stack ]; impureEnvVars = lib.fetchers.proxyImpureEnvVars; buildCommand = '' # Make sure curl can access HTTPS sites, like GitHub. # # Note that we absolutely don't want the Nix store path of the cacert # derivation in the testScript, because we don't want to rebuild this # derivation when only the cacert derivation changes. export SSL_CERT_FILE="${cacert}/etc/ssl/certs/ca-bundle.crt" '' + testScript; meta = with lib; { description = "Test that the stack in Nixpkgs uses the same version of Hpack as the upstream stack release"; maintainers = with maintainers; [ cdepillabout ]; # This derivation internally runs a statically-linked version of stack from # upstream.  This statically-linked version of stack is only available for # x86_64-linux, so this test can only be run on x86_64-linux. platforms = [ "x86_64-linux" ]; }; } </ins>