Change:
```ShellSession $ git diff > nixpkgs/pkgs/the/package/0001-changes.patch ``` <ins> ## Package tests {#sec-package-tests} Tests are important to ensure quality and make reviews and automatic updates easy. Nix package tests are a lightweight alternative to [NixOS module tests](https://nixos.org/manual/nixos/stable/#sec-nixos-tests). They can be used to create simple integration tests for packages while the module tests are used to test services or programs with a graphical user interface on a NixOS VM. Unittests that are included in the source code of a package should be executed in the `checkPhase`. ### Writing package tests {#ssec-package-tests-writing} This is an example using the `phoronix-test-suite` package with the current best practices. Add the tests in `passthru.tests` to the package definition like this: ```nix { stdenv, lib, fetchurl, callPackage }: stdenv.mkDerivation { … passthru.tests = { simple-execution = callPackage ./tests.nix { }; }; meta = { … }; } ``` Create `tests.nix` in the package directory: ```nix { runCommand, phoronix-test-suite }: let inherit (phoronix-test-suite) pname version; in runCommand "${pname}-tests" { meta.timeout = 3; } '' # automatic initial setup to prevent interactive questions ${phoronix-test-suite}/bin/phoronix-test-suite enterprise-setup >/dev/null # get version of installed program and compare with package version if [[ `${phoronix-test-suite}/bin/phoronix-test-suite version` != *"${version}"*  ]]; then echo "Error: program version does not match package version" exit 1 fi # run dummy command ${phoronix-test-suite}/bin/phoronix-test-suite dummy_module.dummy-command >/dev/null # needed for Nix to register the command as successful touch $out '' ``` ### Running package tests {#ssec-package-tests-running} You can run these tests with: ```ShellSession $ cd path/to/nixpkgs $ nix-build -A phoronix-test-suite.tests ``` ### Examples of package tests {#ssec-package-tests-examples} Here are examples of package tests: - [Jasmin compile test](https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/compilers/jasmin/test-assemble-hello-world/default.nix) - [Lobster compile test](https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/compilers/lobster/test-can-run-hello-world.nix) - [Spacy annotation test](https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/python-modules/spacy/annotation-test/default.nix) - [Libtorch test](https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/libraries/science/math/libtorch/test/default.nix) - [Multiple tests for nanopb](https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/libraries/nanopb/default.nix) </ins>

Discussion:
We have documentation on how to write NixOS (module) tests. But how to write tests for packages? I asked on IRC and got some helpful informations: So my goal with this issue is to get this information documented. explain the difference between NixOS module tests and package tests explain when to use which link to good examples for both create one example for a module test with nginx to test a package that needs internet access to download stuff We maybe have to decide on a policy when a test should be and when not (e.g. it's not important enough to use costly module tests) Related issues:
There is also to link to test derivations from a package. It should be called initially, but it's not possible due to technical limitations.
commented in So this looks like the way to go for now!
Thank you for your contributions. This has been automatically marked as stale because it has had no activity for 180 days. If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity. Here are suggestions that might help resolve this more quickly: for maintainers and people that previously touched the related code and @ mention them in a comment. on the . 3. Ask on the on .
i still want to do it waiting for markdown docs
This issue has been mentioned on NixOS Discourse. There might be relevant details there:
This issue has been mentioned on NixOS Discourse. There might be relevant details there:

While the system we have right now is good for contributing, I think each new nix expression should have a test in written along with it in each new PR. We should do this retroactively where possible for all currently existing packages too. This way, we can ensure everything works with hydra, and be alerted quickly for broken, but successfully built expressions that may not be actively maintained and (user) tested anymore. Would it be possible to change the policy in to do this? The flow I see is this: new expression for program X locally to ensure it works test in nixos/tests for it so hydra can do this automatically PR after everything fits . Is this a good idea, something we should implement? I see more benefits that drawbacks.
This will also be really great for stable releases, since if implemented in full, we can ensure all packages are at least minimally tested.
Some packages offer installed tests
NixOS tests for simple things are currently very frustrating to debug, because the overhead of an empty NixOS test is larger than the runtime of a non-NixOS-test LibreOffice functional test. And naive approaches to non-NixOS-based tests got rejected because launching DBus and X requires a minor duplication of functionality (and so it was rejected). Installed tests shipped with packages come pre-debugged (hopefully), so it should be less of a problem there. Of course, some might depend on network access, but that should be easy to detect and block the coresponding tests. So if you want to see more NixOS tests, your best bet probably includes something about a NixOS profile with minimal boot time. Maybe even with some code to boot it in a Nix build sandbox (used as a container).
Thank you for your contributions. This has been automatically marked as stale because it has had no activity for 180 days. If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity. Here are suggestions that might help resolve this more quickly: for maintainers and people that previously touched the related code and @ mention them in a comment. on the . on the on .

Unless I got lost in the package tree, I don't see any tests other than those in , which are effectively for the NixOS modules. I want to start adding package tests, primarily for all those package I regularly update/maintain. a) Where should I put those b) How would I bootstrap them? (I want them to be run as part of the channel updates, not unlike the files) c) Do we have concerns with Hydra build times if/when people were to start adding lots of tests? /cc (just pinging people who usually have good opinions on these things, anybody is welcome to provide feedback)
I know that has already put a lot of thought into package tests and has some code ready. Not sure where, though. :)
Yes! It’s nearly finished! if you want we can chat (Profpatsch on freenode, on Jabber/XMPP) and I can show you what I got so far.
Very good, I will wait for the PR. Ping me when you have something reviewable :) A little while ago I a test for , not realizing that tests need to be explicitly bootstrapped. So I believe my test does not run at all right now. I just want a simple solution that lets us test packages, since we do have a good amount of breakage on a regular basis...
For people who already want to write package tests, I’ve written a short overview on how to do it until the implementation gets merged:
PING :D Any status updates? :)
Not yet. I will be working on it some more over the next month.
Ping? :innocent:
any progress? :)
essentially implements this right?
Parts of it. I should really merge my code. Wouldn’t comment on this thread too much, since it mentions so many people.
what is the status of this effort? Was the whole idea abandoned after was rejected? I'm still waiting for a good way to annotate packages with simple tests. Do you agree that checkPhase inside the derivation is currently the best way to do it? Would it be useful if package maintainers add it to their packages? Especially in the context of and possibly duplicated effort.
IMHO is not a good place since it cannot be run independently of the package build.

Tweaking the test would mean rebuilding the whole package, and so would disabling flaky test. Me and are adding new nixos tests to (). r-ryantm runs derivations listed in that attribute () and ofborg might do that as well (). I think using in there might also work nicely.
Yes! For simple programs an entire VM test is overkill and a derivation running a few commands is enough. See for example:
I join the recommendation to use over for tests we add in Nixpkgs without relying on the upstream tests. I think we also have some support for upstream's installed-tests. the worst part is that even for LibreOffice NixOS VM tests are a bit of an overkill (VM setup takes longer than the test itself, starting LibreOffice and clicking on stuff and checking menus actually appear via OCR), but there we need to refactor the necessary functionality, such as launching display server and D-Bus.
The downside of is that test failures won't fail the build and tests are not run automatically. Linking nixos tests has also a discoverability problem.
that's just a tooling problem. We have to teach our tools to do this. Hydra, r-ryantm, ofBorg and maybe . I think it would even be useful to extract the checkPhase to its own derivation, so that changing the test doesn't cause a rebuild for the program's derivation.
I don't think tooling is a nice solution for this. As long as running tests is optional, many people won't run them when making changes. Also many people will install broken packages from hydra without knowing they are broken. I like approach much better (inverting the dependencies so that the packages depend on their tests, IIRC). I'm not saying that is bad, just that its not optimal in my view.
Unavoidable tests, though, create a pressure to reduce them (to make installation faster) — so realistically we could switch from opt-in to opt-out, but mandatory will lead to a different conflict of interest.
Yeah, the reverse-dependency approach would essentially be opt-out. Each package could provide an attribute that skips the tests. I think opt-in would lead to a lot of failing tests that are just accepted.

I guess we end up having an untested set like we have , but yeah, not having it as a default might be a good idea. I do not think that including the tests in the main build itself leads to a good balance of incentives, so I guess putting them into for now is an OK idea. (I guess in the long run we could teach that sequentialisation trick by and give it a list of functions from untested outputs to installed tests)
Thank you for your contributions. This has been automatically marked as stale because it has had no activity for 180 days. If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity. Here are suggestions that might help resolve this more quickly: for maintainers and people that previously touched the related code and @ mention them in a comment. on the . 3. Ask on the on .