Repository: tweag/nickel
Pull request #954 (2022-11-28 18:22:30)

Restructure integration tests into a single crate
This PR replaces our multi-crate integration test setup with a single crate that contains all tests, inspired by [this blog](https://matklad.github.io/2021/02/27/delete-cargo-integration-tests.html).

This has dual benefits:
1. it should reduce the amount of time tests take to compile, as we're only producing a single binary,
2. it should reduce the amount of time tests take to run, as cargo schedules individual test binaries to run sequentially whereas tests within a binary can run in parallel.

I don't have hugely strong feelings about this, and totally understand if people want to keep the existing structure, but at least on my machine this seemed to reduce the time of `cargo test` by about 50%.


Patch:
diff --git a/.github/workflows/update-flake-lock.yml b/.github/workflows/update-flake-lock.yml
index 9539ee85..e3da5b7d 100644
--- a/.github/workflows/update-flake-lock.yml
+++ b/.github/workflows/update-flake-lock.yml
@@ -16,7 +16,7 @@ jobs:
           extra_nix_config: |
             access-tokens = github.com=${{ secrets.GITHUB_TOKEN }}
       - name: Update flake.lock
-        uses: DeterminateSystems/update-flake-lock@v14
+        uses: DeterminateSystems/update-flake-lock@v15
         with:
           # Providing a secret token is necessary for the PRs to run CI
           # See https://github.com/DeterminateSystems/update-flake-lock#with-a-personal-authentication-token
diff --git a/HACKING.md b/HACKING.md
index d013fc82..8e766f12 100644
--- a/HACKING.md
+++ b/HACKING.md
@@ -117,22 +117,23 @@ LICENSE  package.json nickel_lang_bg.js  nickel_lang_bg.wasm [..]
 Tests are run via `cargo test`. They are two types of tests:
 
 - Unit tests, located directly in the corresponding module.
-- Integration tests, located in the dedicated crate `tests`.
+- Integration tests, located in the dedicated crate `tests/integration`.
 
 You can take inspiration from the existing tests to add your own. By convention,
-tests expected to pass are written in a standalone Nickel file in `tests/pass/`.
-Each `.ncl` file defines a list of expressions that must individually evaluate
-to the boolean `true`. The whole file is an expression that returns true if and
-only if every tests pass, or fail with a contract failure to help locating the
-failing test (instead of returning just `false`).
+tests expected to pass are written in a standalone Nickel file in
+`tests/integration/pass/`. Each `.ncl` file defines a list of expressions that
+must individually evaluate to the boolean `true`. The whole file is an
+expression that returns true if and only if every tests pass, or fail with a
+contract failure to help locating the failing test (instead of returning just
+`false`).
 
-If a test expected to pass is failing, run it directly using nickel with `nickel
--f tests/pass/test_that_doesnt_pass.ncl` to get better error messages than
-`cargo test`.
+If a test expected to pass is failing, run it directly using nickel with
+`nickel -f tests/integration/pass/test_that_doesnt_pass.ncl` to get better error
+messages than `cargo test`.
 
 Tests expected to fail are often embedded directly into rust source code,
 because you usually want to additionally check that the error is the one you
-expect. For example ([`tests/records_fail.rs`](./tests/records_fail.rs)):
+expect. For example ([`tests/integration/records_fail.rs`](./tests/records_fail.rs)):
 
 ```rust
 #[test]


Test patch:
diff --git a/tests/README.md b/tests/README.md
new file mode 100644
index 00000000..f799f76d
--- /dev/null
+++ b/tests/README.md
@@ -0,0 +1,16 @@
+# A quick note on how tests are structured
+
+By default, `cargo` encourages developers to put integration test files directly
+in the `/tests` directory. Doing so means that a different test crate (and
+associated binary) is built for each file.
+
+There is both a compile-time and a runtime cost to doing this, as (1) multiple
+different binaries must be built to run all tests and (2) while `cargo` can
+run multiple tests in parallel, it runs multiple binaries sequentially.
+
+By instead having a single crate which contains all tests (as we do in
+`/tests/integration`), we save on compilation time, as well as being able to
+fully parallelise all of our test runs.
+
+See [this blog post](https://matklad.github.io/2021/02/27/delete-cargo-integration-tests.html)
+for more information.
diff --git a/tests/basics_fail.rs b/tests/integration/basics_fail.rs
similarity index 100%
rename from tests/basics_fail.rs
rename to tests/integration/basics_fail.rs
diff --git a/tests/contracts_fail.rs b/tests/integration/contracts_fail.rs
similarity index 100%
rename from tests/contracts_fail.rs
rename to tests/integration/contracts_fail.rs
diff --git a/tests/destructuring.rs b/tests/integration/destructuring.rs
similarity index 100%
rename from tests/destructuring.rs
rename to tests/integration/destructuring.rs
diff --git a/tests/destructuring/assign.ncl b/tests/integration/destructuring/assign.ncl
similarity index 100%
rename from tests/destructuring/assign.ncl
rename to tests/integration/destructuring/assign.ncl
diff --git a/tests/destructuring/assign_fail.ncl b/tests/integration/destructuring/assign_fail.ncl
similarity index 100%
rename from tests/destructuring/assign_fail.ncl
rename to tests/integration/destructuring/assign_fail.ncl
diff --git a/tests/destructuring/atbind.ncl b/tests/integration/destructuring/atbind.ncl
similarity index 100%
rename from tests/destructuring/atbind.ncl
rename to tests/integration/destructuring/atbind.ncl
diff --git a/tests/destructuring/closed_fail.ncl b/tests/integration/destructuring/closed_fail.ncl
similarity index 100%
rename from tests/destructuring/closed_fail.ncl
rename to tests/integration/destructuring/closed_fail.ncl
diff --git a/tests/destructuring/default.ncl b/tests/integration/destructuring/default.ncl
similarity index 100%
rename from tests/destructuring/default.ncl
rename to tests/integration/destructuring/default.ncl
diff --git a/tests/destructuring/fun.ncl b/tests/integration/destructuring/fun.ncl
similarity index 100%
rename from tests/destructuring/fun.ncl
rename to tests/integration/destructuring/fun.ncl
diff --git a/tests/destructuring/mixed.ncl b/tests/integration/destructuring/mixed.ncl
similarity index 100%
rename from tests/destructuring/mixed.ncl
rename to tests/integration/destructuring/mixed.ncl
diff --git a/tests/destructuring/nested.ncl b/tests/integration/destructuring/nested.ncl
similarity index 100%
rename from tests/destructuring/nested.ncl
rename to tests/integration/destructuring/nested.ncl
diff --git a/tests/destructuring/open.ncl b/tests/integration/destructuring/open.ncl
similarity index 100%
rename from tests/destructuring/open.ncl
rename to tests/integration/destructuring/open.ncl
diff --git a/tests/destructuring/rest.ncl b/tests/integration/destructuring/rest.ncl
similarity index 100%
rename from tests/destructuring/rest.ncl
rename to tests/integration/destructuring/rest.ncl
diff --git a/tests/destructuring/rest_fail.ncl b/tests/integration/destructuring/rest_fail.ncl
similarity index 100%
rename from tests/destructuring/rest_fail.ncl
rename to tests/integration/destructuring/rest_fail.ncl
diff --git a/tests/destructuring/simple.ncl b/tests/integration/destructuring/simple.ncl
similarity index 100%
rename from tests/destructuring/simple.ncl
rename to tests/integration/destructuring/simple.ncl
diff --git a/tests/destructuring/typecontract.ncl b/tests/integration/destructuring/typecontract.ncl
similarity index 100%
rename from tests/destructuring/typecontract.ncl
rename to tests/integration/destructuring/typecontract.ncl
diff --git a/tests/destructuring/typecontract_fail.ncl b/tests/integration/destructuring/typecontract_fail.ncl
similarity index 100%
rename from tests/destructuring/typecontract_fail.ncl
rename to tests/integration/destructuring/typecontract_fail.ncl
diff --git a/tests/examples.rs b/tests/integration/examples.rs
similarity index 100%
rename from tests/examples.rs
rename to tests/integration/examples.rs
diff --git a/tests/free_vars.rs b/tests/integration/free_vars.rs
similarity index 100%
rename from tests/free_vars.rs
rename to tests/integration/free_vars.rs
diff --git a/tests/imports.rs b/tests/integration/imports.rs
similarity index 100%
rename from tests/imports.rs
rename to tests/integration/imports.rs
diff --git a/tests/imports/contract-fail.ncl b/tests/integration/imports/contract-fail.ncl
similarity index 100%
rename from tests/imports/contract-fail.ncl
rename to tests/integration/imports/contract-fail.ncl
diff --git a/tests/imports/cycle.ncl b/tests/integration/imports/cycle.ncl
similarity index 100%
rename from tests/imports/cycle.ncl
rename to tests/integration/imports/cycle.ncl
diff --git a/tests/imports/cycle_b.ncl b/tests/integration/imports/cycle_b.ncl
similarity index 100%
rename from tests/imports/cycle_b.ncl
rename to tests/integration/imports/cycle_b.ncl
diff --git a/tests/imports/multi_imports.ncl b/tests/integration/imports/multi_imports.ncl
similarity index 100%
rename from tests/imports/multi_imports.ncl
rename to tests/integration/imports/multi_imports.ncl
diff --git a/tests/imports/nested.ncl b/tests/integration/imports/nested.ncl
similarity index 100%
rename from tests/imports/nested.ncl
rename to tests/integration/imports/nested.ncl
diff --git a/tests/imports/record.ncl b/tests/integration/imports/record.ncl
similarity index 100%
rename from tests/imports/record.ncl
rename to tests/integration/imports/record.ncl
diff --git a/tests/imports/root_path.ncl b/tests/integration/imports/root_path.ncl
similarity index 100%
rename from tests/imports/root_path.ncl
rename to tests/integration/imports/root_path.ncl
diff --git a/tests/imports/root_path/fourtytwo.ncl b/tests/integration/imports/root_path/fourtytwo.ncl
similarity index 100%
rename from tests/imports/root_path/fourtytwo.ncl
rename to tests/integration/imports/root_path/fourtytwo.ncl
diff --git a/tests/imports/root_path/import.ncl b/tests/integration/imports/root_path/import.ncl
similarity index 100%
rename from tests/imports/root_path/import.ncl
rename to tests/integration/imports/root_path/import.ncl
diff --git a/tests/imports/two.ncl b/tests/integration/imports/two.ncl
similarity index 100%
rename from tests/imports/two.ncl
rename to tests/integration/imports/two.ncl
diff --git a/tests/imports/typecheck-fail.ncl b/tests/integration/imports/typecheck-fail.ncl
similarity index 100%
rename from tests/imports/typecheck-fail.ncl
rename to tests/integration/imports/typecheck-fail.ncl
diff --git a/tests/imports/unexpected_token.ncl b/tests/integration/imports/unexpected_token.ncl
similarity index 100%
rename from tests/imports/unexpected_token.ncl
rename to tests/integration/imports/unexpected_token.ncl
diff --git a/tests/imports/unexpected_token_buried.ncl b/tests/integration/imports/unexpected_token_buried.ncl
similarity index 100%
rename from tests/imports/unexpected_token_buried.ncl
rename to tests/integration/imports/unexpected_token_buried.ncl
diff --git a/tests/imports/unexpected_token_in_record.ncl b/tests/integration/imports/unexpected_token_in_record.ncl
similarity index 100%
rename from tests/imports/unexpected_token_in_record.ncl
rename to tests/integration/imports/unexpected_token_in_record.ncl
diff --git a/tests/infinite_rec.rs b/tests/integration/infinite_rec.rs
similarity index 100%
rename from tests/infinite_rec.rs
rename to tests/integration/infinite_rec.rs
diff --git a/tests/integration/lib.rs b/tests/integration/lib.rs
new file mode 100644
index 00000000..fb055e5a
--- /dev/null
+++ b/tests/integration/lib.rs
@@ -0,0 +1,17 @@
+mod basics_fail;
+mod contracts_fail;
+mod destructuring;
+mod examples;
+mod free_vars;
+mod imports;
+mod infinite_rec;
+mod merge_fail;
+mod parse_fail;
+mod pass;
+mod pretty;
+mod query;
+mod records_fail;
+mod stdlib_arrays_fails;
+mod stdlib_typecheck;
+mod typecheck_fail;
+mod unbound_type_variables;
\ No newline at end of file