Repository: nodejs/node
Pull request: doc: recommend writing tests in new files and including comments

The previous phrasing encouraged or did not discourage appending new test cases to existing files - a practice that can reduce the debuggability of the tests over time as they get bigger and bigger, some times thousands of lines long with hundreds of test cases, and make the CI output increasingly difficult to read when one of the test cases fail in a very long test.

This patch updates the guideline to explicitly discourage appending test cases this way. Also recommend including an opening comment to describe what the test does to optimize the test towards the scenario when it fails.

For some concrete examples:

1. Previously I had to split [`test-crypto-dh.js`](https://github.com/nodejs/node/pull/49492) and [`test-wasi.js`](https://github.com/nodejs/node/pull/51836) because they were timing out in the CI, some might be related to actual bugs but there is no way to figure out which test specifically is timing out in a long test, so it had to be split to isolate them.
2. When the test crashes like this https://ci.nodejs.org/job/node-test-commit-arm-debug/17147/nodes=ubuntu2204_debug-arm64/testReport/junit/(root)/parallel/test_child_process_exec_maxbuf/ it's difficult to figure out exactly which test case is crashing

To avoid losing git history I would not recommend splitting existing tests unless there's a ongoing motivation to do so (e.g. it's flaking in the CI). But I think we should at least discourage new tests to be written in a way that exacerbate the problem.

<!--
Before submitting a pull request, please read:

- the CONTRIBUTING guide at https://github.com/nodejs/node/blob/HEAD/CONTRIBUTING.md
- the commit message formatting guidelines at
  https://github.com/nodejs/node/blob/HEAD/doc/contributing/pull-requests.md#commit-message-guidelines

For code changes:
1. Include tests for any bug fixes or new features.
4. Update documentation if relevant.
5. Ensure that `make -j4 test` (UNIX), or `vcbuild test` (Windows) passes.

If you believe this PR should be highlighted in the Node.js CHANGELOG
please add the `notable-change` label.

Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I
    have the right to submit it under the open source license
    indicated in the file; or

(b) The contribution is based upon previous work that, to the best
    of my knowledge, is covered under an appropriate open source
    license and I have the right under that license to submit that
    work with modifications, whether created in whole or in part
    by me, under the same open source license (unless I am
    permitted to submit under a different license), as indicated
    in the file; or

(c) The contribution was provided directly to me by some other
    person who certified (a), (b) or (c) and I have not modified
    it.

(d) I understand and agree that this project and the contribution
    are public and that a record of the contribution (including all
    personal information I submit with it, including my sign-off) is
    maintained indefinitely and may be redistributed consistent with
    this project or the open source license(s) involved.
-->


Changed files:
filename: doc/contributing/writing-tests.md
patch: @@ -21,9 +21,22 @@ Add tests when:
 ## Test directory structure
 
 See [directory structure overview][] for outline of existing test and locations.
-When deciding on whether to expand an existing test file or create a new one,
-consider going through the files related to the subsystem.
-For example, look for `test-streams` when writing a test for `lib/streams.js`.
+
+## How to write a good test
+
+A good test should be written in a style that is optimial for debugging
+when it fails.
+
+In principle, when adding a new test, it should be placed in a new file.
+Unless there is strong motivation to do so, refrain from appending
+new test cases to an existing file. Similar to the reproductions we ask
+for in the issue tracker, a good test should be as minimal and isolated as
+possible to facilitate debugging.
+
+A good test should come with comments explaining what it tries to test,
+so that when it fails, other contributors can fix it with the full context
+of its intention, and be able to modify it in good confidence when the context
+changes.
 
 ## Test structure