Repository: EC-USGS/pywatershed (language: python)

Issue:
improve test data generation
* Fix stale description
* Provide a way for autotest to tell if the test data exist or are stale
* Streamline the test data creation, preferably from autotest rather than from test_data/generate



Discussion / hints:

Fix patch:
diff --git a/.github/workflows/ci.yaml b/.github/workflows/ci.yaml
index a885a8bc..b546657f 100644
--- a/.github/workflows/ci.yaml
+++ b/.github/workflows/ci.yaml
@@ -188,11 +188,10 @@ jobs:
           pip list
 
       - name: hru_1 - generate and manage test data domain, run PRMS and convert csv output to NetCDF
-        working-directory: test_data/generate
+        working-directory: autotest
         run: |
-          pytest -vv -n=2 --durations=0 run_prms_domains.py --domain=hru_1
-          pytest -vv -n=auto --durations=0 convert_prms_output_to_nc.py  --domain=hru_1
-          pytest -vv -n=auto --durations=0 remove_prms_csvs.py
+          python generate_test_data.py \
+            -n=auto --domain=hru_1 --remove_prms_csvs --remove_prms_output_dirs
 
       - name: hru_1 - list netcdf input files
         working-directory: test_data
@@ -212,12 +211,10 @@ jobs:
 
 
       - name: drb_2yr - generate and manage test data
-        working-directory: test_data/generate
+        working-directory: autotest
         run: |
-          pytest -vv remove_output_dirs.py --domain=hru_1
-          pytest -vv -n=2 run_prms_domains.py --domain=drb_2yr
-          pytest -vv -n=auto convert_prms_output_to_nc.py --domain=drb_2yr
-          pytest -vv -n=auto remove_prms_csvs.py
+          python generate_test_data.py \
+            -n=auto --domain=drb_2yr --remove_prms_csvs --remove_prms_output_dirs
 
       - name: drb_2yr - list netcdf input files
         working-directory: test_data
@@ -236,12 +233,10 @@ jobs:
           --junitxml=pytest_drb_2yr.xml
 
       - name: ucb_2yr - generate and manage test data
-        working-directory: test_data/generate
+        working-directory: autotest
         run: |
-          pytest -vv remove_output_dirs.py --domain=drb_2yr
-          pytest -vv -n=2 run_prms_domains.py --domain=ucb_2yr
-          pytest -vv -n=auto convert_prms_output_to_nc.py --domain=ucb_2yr
-          pytest -vv -n=auto remove_prms_csvs.py
+          python generate_test_data.py \
+            -n=auto --domain=ucb_2yr --remove_prms_csvs --remove_prms_output_dirs
 
       - name: ucb_2yr - list netcdf input files
         working-directory: test_data
diff --git a/DEVELOPER.md b/DEVELOPER.md
index 2ade21f6..602e6d2d 100644
--- a/DEVELOPER.md
+++ b/DEVELOPER.md
@@ -141,28 +141,34 @@ all formally encoded in `.github/workflows/ci.yaml` and
 
 ## Testing
 Once the dependencies are available, we want to verify the software by running
-its test suite. The following testing procedures are also covered in the
-notebook `examples/01_automated_testing.ipynb`.  To run the tests, we first need
-to generate the test data. This consists of running PRMS and then converting the
-output to netcdf. From `test_data/scripts`:
+its test suite. However, we first need to generate the test data. This consists
+of running binaries (PRMS) and then converting the output to netcdf files used
+by autotest as the answers or reference results. In the `autotest/` directory,
+test data is generated using the following command.
 
+```shell
+python generate_test_data.py -n=auto
 ```
-pytest -v -n=4 test_run_domains.py
-pytest -v -n=8 test_nc_domains.py
-```
+
+Additional options may be supplied to `generate_test_data.py`. For more details
+on generating the test data, see [`test_data/README.md`](test_data/README.md).
 
 Finally, the tests can be run from the `autotest` directory:
 
-``` pytest -v -n=8 ```
+```shell
+pytest -v -n=auto
+```
 
 All tests should pass, XPASS, or XFAIL. XFAIL is an expected
-failure. Substitute `-n auto` to automatically use all available cores on your
+failure. The flag `-n auto` to automatically use all available cores on your
 machine.
 
+For more details on the autotests, see [`autotest/README.md`](autotest/README.md).
+
 
 ## Linting
 Automated linting procedures are performed in CI and enforced, these are
-```
+```shell
 isort ./autotest ./pywatershed
 black ./autotest ./pywatershed
 flake8 --count --show-source --exit-zero ./pywatershed ./autotest


Test patch:
diff --git a/autotest/README.md b/autotest/README.md
index 3d341cb3..f961947e 100644
--- a/autotest/README.md
+++ b/autotest/README.md
@@ -1,21 +1,22 @@
-# Autotest
-
-## Usage
-
-```
-cd autotest
-pytest
-```
-Pytest options can be explored via `pytest --help`.
+<!-- START doctoc generated TOC please keep comment here to allow auto update -->
+<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
+**Table of Contents**
 
+- [Autotest](#autotest)
+  - [Test data](#test-data)
+  - [Usage](#usage)
+  - [domain_yaml details](#domain_yaml-details)
+    - [Path resolution](#path-resolution)
+    - [Answers for domain tests](#answers-for-domain-tests)
 
-## Developer
+<!-- END doctoc generated TOC please keep comment here to allow auto update -->
 
-This is how the pywatershed package tests itself.
+# Autotest
 
-The test suite consists both
-	* stand alone tests, and
-	* domain tests
+Autotest provides the test suite for pywatershed. The tests include
+both
+  * stand alone tests, and
+  * domain tests.
 
 Stand alone tests do not require any input files or output files besides what
 is supplied in the testing framework. These tests generally run quickly
@@ -23,60 +24,103 @@ is supplied in the testing framework. These tests generally run quickly
 
 Domain tests require input or output files for an NHM domain. These tests
 scale with the size or number of HRUs in a domain. The test suite takes
-arguments related to domain tests:
+arguments related to domain tests which are described in sthe usage section below.
+
+## Test data
+The majority of tests are domain tests (as of writing this). And the vast majority of the
+domain test data must be generated by running binaries included in the repository BEFORE
+running the autotests. 
+
+To generate the test data, run the following command from `autotest/`:
+
+```shell
+python generate_test_data.py -n=auto
+```
+
+This command should generally be sufficient but more options are available, including
+passing of options to pytest. Run `python generate_test_data.py --help` for more details.
+For example, if you are in a hurry to run and test a single domain you may restrict
+(or expand) the domains processed with the `--domains` option. To get verbose output
+from pytest, pass `-vv`. 
 
+Using the above command allows the autotests to provide checks that the test data in
+`test_data/` have been generated and have been generated by the current version of
+pywatershed. These checks are meant to avoid gross errors with autotesting but will
+not cover ever possible testing situation. When in doubt about tests, it's always best
+practice to start over by re-generating the test data.
+
+There are temporary situations where errors caused by checking for test data are
+unwarranted and in those cases, disabling the errors by editing `autotest/conftest.py`
+is the temporary solution.
+
+Please see [`test_data/README.md`](../test_data/README.md) for additional details on how
+to generate the test data.
+
+## Usage
+
+```
+cd autotest
+pytest -n=auto -vv
 ```
- --domain_yaml=DOMAIN_YAML
-                        YAML file(s) for indiv domain tests. You can pass multiples of this argument. Default
-                        value (not shown here) is --domain_yaml=../test_data/drb_2yr/drb_2yr.yaml.
+
+Pytest options can be explored via `pytest --help`. Custom options for
+pywatershed are buried down in the output of this call for help, these are:
+
+```
+Custom options:
+  --domain_yaml=DOMAIN_YAML
+                        YAML file(s) for indiv domain tests. You can pass multiples of this
+                        argument. Default value (not shown here) is
+                        --domain_yaml=../test_data/drb_2yr/drb_2yr.yaml
   --print_ans           Print results and assert False for all domain tests
+  --all_domains         Run all test domains
 ```
 
-Note that the `--domain_yaml` argument may be repeated within a single call to test multiple
-domains in a single pytest.
+The default domain tested is `drb_2yr`. All domains present in `test_data/`
+can be tested using `--all_domains`. Requesting a specific domain or multiple
+domains is done by passing one or more `--domain_yaml` arguments. 
 
-The `domain_yaml` file is an *evolving* set of data that includes static domain
-inputs (e.g. CBH forcing files, parameter files), static or reference model
-output (from PRMS/NHM), and the answers to domain tests.
+## domain_yaml details
 
-Examples of `domain_yaml` files can be found in, for example, in
-`pywatershed/test_data/drb_2yr/drb_2yr.yaml`
-and
-`pywatershed/test_data/conus_2yr/conus_2yr.yaml`.
+This section is about creating or working with the domain_yaml file when
+writing tests.
 
+The `domain_yaml` file provides information to the autotests. The domain_yaml
+files for a domain directory `somewhere/` will be `somewhere/somewhere.yaml`. 
+This YAML file includes paths to static domain inputs (e.g. CBH forcing files,
+parameter files), paths to static or reference model output (from PRMS/NHM),
+and the answers to domain tests. Examples of `domain_yaml` files can be found
+in, each domain.
 
-### Domain inputs
-These are specific files for a specific model domain (e.g. the Delaware River Basin,
-or the CONUS NHM, etc). The files are paths and are to be specified as either
-  * relative paths: relative to the location of the yaml file
-    in which the path is appearing
-  * absolute paths: use absolute paths (for some machine and potentially user)
-The examples listed above demonstrate use of both relative and absolute paths.
+Some details on the contents of the YAML file are given below.
+
+### Path resolution
+The test configuration for autotest (`autotest/conftest.py`) provides special
+path resolution relative to the `domain_yaml` file for paths specified in a
+list in that file:
+
+```  
+    ["param_file", "control_file", "cbh_nc", "prms_run_dir", "prms_output_dir",]
+```
 
-Domain inputs are generally created by scripts run in `test_data/scripts`. The
-`drb_2yr` case shows the relationship between the prms binary, the input files,
-and the output files. These scripts help maintain sanity when generating new
-files for domain tests.
+Additional fields can be added to this list to provide path resolution for new
+fields in the YAML file. 
 
 
 ### Answers for domain tests
-Tests have objectively correct answers. The answer key is stored in the yaml in
-top-level attribute/key called `test_ans`. This can be seen in the examples
-listed above. This attibute has a sub-attributes for each file with domain
-tests in `autotest/test_*py` (the text `test_` and `.py` is dropped in the
-yaml reference). Below this, each file has named tests attributes which in turn
-potentially have named cases/keys when iterated with other fixtures (such as
-variables or types).
-
-The tests answers for a given test are output of some kind of summary statistic
-or other kind of reduction performed on data in memory. The answers vary with
-domain and are most easily collected by running the tests themselves while
-verifying accuracy manually when putting the answers into the domain yaml file.
-
-For convenience, a user can select to print the answers at run/test time using
-the `--print_ans` argument when running ptest. This prints the answer for
-domain tests and always asserts False after doing so for each test in a file.
-This option aids in conctruction of the tests_ans section of the domain
-yaml file for new domains or when extending tests on existing domains. Running
-`pytest -s ...` prints output in a format that can be copied to the yaml and
-edited quickly to get new test results enshrined in the answer key.
+Certain tests have answers stored in the domain YAML. This "answer key" is
+stored in the top-level key `test_ans`. This can be seen in the examples
+listed above. Generally, for a autotest `test_x.py` the key "x" will be
+provided below `test_ans` and below it will be what ever data is used in
+that test.
+
+These tests answers for a given test are typically a summary statistic
+performed on model data in memory. The answers vary with domain and
+are most easily collected by running the tests themselves while
+verifying accuracy of the test through expert judgement. The answers
+enshrined in the domain YAML indicate when test results change. Because
+test results change from time to time, a convenience utility function,
+`assert_or_print` is provided in `utils.py`. Using this function allows
+the option `--print_ans -s` (note that `-s` prints output to the terminal)
+to be passed at run time to print all the new values that should be
+updated into the domain YAML.
diff --git a/autotest/conftest.py b/autotest/conftest.py
index db2a7134..ae82f178 100644
--- a/autotest/conftest.py
+++ b/autotest/conftest.py
@@ -1,4 +1,4 @@
-import pathlib
+import pathlib as pl
 
 import yaml
 
@@ -34,6 +34,49 @@ def pytest_addoption(parser):
 
 
 def pytest_generate_tests(metafunc):
+    # check domain test data regardless of what fixtures are being requested
+    if metafunc.config.getoption("all_domains"):
+        domain_file_list = [
+            "../test_data/hru_1/hru_1.yaml",
+            "../test_data/drb_2yr/drb_2yr.yaml",
+            "../test_data/ucb_2yr/ucb_2yr.yaml",
+        ]
+    else:
+        domain_file_list = metafunc.config.getoption("domain_yaml")
+        if len(domain_file_list) == 0:
+            domain_file_list = ["../test_data/drb_2yr/drb_2yr.yaml"]
+
+    # check that the test_data has been generated and is up-to-date
+    # for each domain
+    for dd in domain_file_list:
+        domain_name = pl.Path(dd).parent.name
+        test_data_version_file = pl.Path(
+            f"../test_data/.test_data_version_{domain_name}.txt"
+        )
+        if not test_data_version_file.exists():
+            msg = (
+                f"Test data for domain {domain_name} do not appear to have\n"
+                "been generated. Please see DEVELOPER.md for information on\n"
+                "generating test data.\n"
+            )
+            raise ValueError(msg)
+
+        repo_version_file = pl.Path("../version.txt")
+
+        with open(test_data_version_file) as ff:
+            test_version = ff.read()
+        with open(repo_version_file) as ff:
+            repo_version = ff.read()
+
+        if test_version != repo_version:
+            msg = (
+                f"Test data for domain {domain_name} do not appear to\n"
+                "have been generated by the current version of the\n"
+                "pywatershed repository. Please see DEVELOPER.md for\n"
+                "information on generating test data.\n"
+            )
+            raise ValueError(msg)
+
     if "domain" in metafunc.fixturenames:
         # Put --print_ans in the domain fixture as it applies only to the
         # domain tests. It is a run time attribute, not actually an attribute
@@ -42,22 +85,11 @@ def pytest_generate_tests(metafunc):
         # Not sure I love this, maybe have a domain_opts fixture later?
         print_ans = metafunc.config.getoption("print_ans")
 
-        if metafunc.config.getoption("all_domains"):
-            domain_file_list = [
-                "../test_data/hru_1/hru_1.yaml",
-                "../test_data/drb_2yr/drb_2yr.yaml",
-                "../test_data/ucb_2yr/ucb_2yr.yaml",
-            ]
-        else:
-            domain_file_list = metafunc.config.getoption("domain_yaml")
-            if len(domain_file_list) == 0:
-                domain_file_list = ["../test_data/drb_2yr/drb_2yr.yaml"]
-
         # open and read in the yaml and
-        domain_ids = [pathlib.Path(ff).stem for ff in domain_file_list]
+        domain_ids = [pl.Path(ff).stem for ff in domain_file_list]
         domain_list = []
         for dd in domain_file_list:
-            dd_file = pathlib.Path(dd)
+            dd_file = pl.Path(dd)
             with dd_file.open("r") as yaml_file:
                 domain_dict = yaml.safe_load(yaml_file)
 
@@ -78,15 +110,15 @@ def pytest_generate_tests(metafunc):
                 "prms_run_dir",
                 "prms_output_dir",
             ]:
-                domain_dict[ff] = pathlib.Path(domain_dict[ff])
+                domain_dict[ff] = pl.Path(domain_dict[ff])
                 if not domain_dict[ff].is_absolute():
                     domain_dict[ff] = domain_dict["dir"] / domain_dict[ff]
 
             for fd_key in ["cbh_inputs"]:
                 domain_dict[fd_key] = {
                     key: (
-                        pathlib.Path(val)
-                        if pathlib.Path(val).is_absolute()
+                        pl.Path(val)
+                        if pl.Path(val).is_absolute()
                         else domain_dict["dir"] / val
                     )
                     for key, val in domain_dict[fd_key].items()
diff --git a/autotest/generate_test_data.py b/autotest/generate_test_data.py
new file mode 100755
index 00000000..c3b6d14a
--- /dev/null
+++ b/autotest/generate_test_data.py
@@ -0,0 +1,212 @@
+#!/usr/bin/env python
+"""Generate pywatershed test data.
+
+usage examples:
+    python:  `python generate_test_data.py --domains=hru_1,drb_2yr -n5`
+    ipython: `ipython generate_test_data.py -- -n=5 --domains=hru_1,drb_2yr`
+    command line: `./generate_test_data.py --domains=hru_1,drb_2yr -n5`
+
+For help, call with -h or --help. Note that arguments not listed by
+generate_test_data.py are passed directly to pytest. See `pytest --help`
+for its extensive list of options.
+
+For details look at DEVELOPER.md, autotest/README.md, and test_data/README.md.
+
+This wrapper calls pytest to create and destroy test data in the test_data
+directory. It also creates and destroys dot files which are looked for and at
+by autotest to get a rough sense if the test data are up to date. These dot
+files are `test_data/.test_data_version_{domain_name}.txt` for each domain.
+
+When in doubt about the freshness of test data, remake it!
+
+Note that this wrapper was create because the test version file
+tracking was practically impossible with pytest-xdist.
+"""
+
+import argparse
+import os
+import pathlib as pl
+import shutil
+import sys
+from copy import deepcopy
+
+import pytest
+
+
+def parse_args():
+    """Parse the arguments.
+
+    Args: None.
+    Returns: A list of arguments parsed in a way to pass to run_prms_domains.py
+        and convert_prms_output_to_nc.py.
+
+    Special handling of --domains here to kind of cleanup inconsistencies
+    betweenrun_prms_domains.py and convert_prms_output_to_nc.py and also to
+    control recording of which domains have available test_data.
+    """
+    desc = (
+        "Generate pywatershed test data. "
+        "*** Arguments not listed by generate_test_data.py are passed "
+        "directly to pytest. See `pytest --help` for its extensive list of "
+        "options. ***"
+    )
+    parser = argparse.ArgumentParser(
+        description=desc,
+        add_help=False,
+        formatter_class=argparse.ArgumentDefaultsHelpFormatter,
+    )
+    parser.add_argument(
+        "-h",
+        "--help",
+        action="help",
+        help="Help on calling generate_test_data.py.",
+    )
+    parser.add_argument(
+        "--domains",
+        default="hru_1,drb_2yr,ucb_2yr",
+        help=(
+            "Domain name values for which to generate test data, comma "
+            "separate multiple values."
+        ),
+        type=str,
+    )
+    parser.add_argument(
+        "--remove_prms_csvs",
+        help=(
+            "Option to remove PRMS csv output in domain/output/ directories "
+            "after running"
+        ),
+        action="store_true",
+    )
+    parser.add_argument(
+        "--remove_prms_output_dirs",
+        help=(
+            "Option to remove existing PRMS output directories before running"
+        ),
+        action="store_true",
+    )
+
+    known, unknown = parser.parse_known_args()
+
+    domains = known.domains
+    if "," in domains:
+        domains_split = domains.split(",")
+        domain_list = []
+        for dd in domains_split:
+            domain_list += [f"--domain={dd}"]
+    else:
+        domain_list = [f"--domain={domains}"]
+
+    remove_prms_csvs = [f"--remove_prms_csvs={known.remove_prms_csvs}"]
+    remove_prms_output_dirs = [
+        f"--remove_prms_output_dirs={known.remove_prms_output_dirs}"
+    ]
+
+    arg_list = (
+        domain_list + unknown + remove_prms_csvs + remove_prms_output_dirs
+    )
+
+    return arg_list
+
+
+def main():
+    arg_list = parse_args()
+
+    print("\nGenerating test data. Running in ../test_data/generate")
+    os.chdir("../test_data/generate")
+
+    # collect info on domains to be run_domains
+    domains_to_run = []
+    remove_prms_csvs = False
+    remove_prms_output_dirs = False
+    rm_vals = []
+    for ii in arg_list:
+        if "--domain" in ii:
+            domains_to_run += [ii.split("--domain")[-1].split("=")[-1]]
+        elif "--remove_prms_csvs" in ii:
+            remove_prms_csvs = ii.split("=")[-1] == "True"
+            rm_vals += [ii]
+        elif "--remove_prms_output_dirs" in ii:
+            remove_prms_output_dirs = ii.split("=")[-1] == "True"
+            rm_vals += [ii]
+
+    for ii in rm_vals:
+        arg_list.remove(ii)
+
+    n_domains = len(domains_to_run)
+
+    print(f"\nGenerating test data for domains: {domains_to_run}")
+    for dd in domains_to_run:
+        test_data_version_file = pl.Path(f"../.test_data_version_{dd}.txt")
+        test_data_version_file.unlink(missing_ok=True)
+
+    if remove_prms_output_dirs:
+        # todo: make this take the domain argument
+        print("\nRemoving existing PRMS output directories ...")
+        rm_output_dirs_list = arg_list + ["remove_output_dirs.py"]
+        retcode_output_dirs = pytest.main(rm_output_dirs_list)
+        if retcode_output_dirs != pytest.ExitCode.OK:
+            print("\nRemoving existing PRMS output dirs failed.")
+            sys.exit(retcode_output_dirs.value)
+
+    print("\nRunning PRMS domains ...")
+    # if -n is in the list and numeric and > ndomains+1
+    #     then set it to ndomains + 1 just for running the domains
+    n_orig = None
+    for ii in arg_list:
+        if "-n" in ii:
+            arg_list_no_n = deepcopy(arg_list)
+            _ = arg_list_no_n.remove(ii)
+            n_orig = ii.split("-n")[-1].split("=")[-1]
+            n_orig = int(n_orig) if n_orig.isdigit() else n_orig
+
+    if (
+        (n_orig is None)
+        or (not isinstance(n_orig, int))
+        or (n_orig <= (n_domains + 1))
+    ):
+        run_arg_list = arg_list + ["run_prms_domains.py"]
+        conv_arg_list = arg_list + ["convert_prms_output_to_nc.py"]
+    else:
+        run_arg_list = arg_list_no_n + [
+            f"-n={n_domains + 1}",
+            "run_prms_domains.py",
+        ]
+        conv_arg_list = arg_list_no_n + [
+            f"-n={n_orig}",
+            "convert_prms_output_to_nc.py",
+        ]
+
+    retcode_run = pytest.main(run_arg_list)
+    if retcode_run != pytest.ExitCode.OK:
+        print("Running PRMS domains failed.")
+        sys.exit(retcode_run.value)
+
+    print("\nConverting PRMS output to NetCDF ...")
+    retcode_conv = pytest.main(conv_arg_list)
+    if retcode_conv != pytest.ExitCode.OK:
+        print("\nConverting PRMS output to NetCDF failed.")
+        sys.exit(retcode_run.value)
+
+    for dd in domains_to_run:
+        test_data_version_file = pl.Path(f"../.test_data_version_{dd}.txt")
+        shutil.copy("../../version.txt", test_data_version_file)
+
+    if remove_prms_csvs:
+        # todo: make this take the domain argument
+        print("\nRemoving PRMS CSV output file ...")
+        rm_csv_arg_list = arg_list + ["remove_prms_csvs.py"]
+        retcode_rm_csvs = pytest.main(rm_csv_arg_list)
+        if retcode_rm_csvs != pytest.ExitCode.OK:
+            print("\nRemoving PRMS CSV output files failed.")
+            sys.exit(retcode_rm_csvs.value)
+
+    print(
+        "\nSuccess generating pywatershed test data for domains "
+        f"{domains_to_run}\n"
+    )
+    sys.exit(0)
+
+
+if __name__ == "__main__":
+    main()
diff --git a/test_data/README.md b/test_data/README.md
index c7cf183c..5cff458b 100644
--- a/test_data/README.md
+++ b/test_data/README.md
@@ -1,31 +1,94 @@
+<!-- START doctoc generated TOC please keep comment here to allow auto update -->
+<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
+**Table of Contents**
+
+- [Test data](#test-data)
+  - [Generating data](#generating-data)
+  - [Domain directories](#domain-directories)
+  - [Domain YAML file](#domain-yaml-file)
+
+<!-- END doctoc generated TOC please keep comment here to allow auto update -->
+
 # Test data
 
-The `test_data` directory contains
-  * domain directories for domain tests
-  * `scripts/`, which contains scripts to run domain simulations and generate data.
+This directory contains data to be used by the tests in `../autotest`. The
+`generate/` directory provides programs to generate test data not included
+in the repository but which are critical to the autotests.
 
-## Domain directories
+## Generating data
 
-Some domain directories only contain `control.test` and `domain.yaml` files. These domains 
-are not to be run in a self-contained way within the repository. They rely on other files on 
-other systems. The main example is `conus_2yr/`. It is to be run on Denali/Tallgrass but its
-input files are too large to be included in this repo and it is desirable to write model 
-output to a disk faster than the one where the repository lives. However, we do want to 
-version control certain files in the directory. See the `conus_2yr/README.md` for details 
-on how it is setup.
+Generating test data should now be done from the `autotest/` directory with
+the following command:
 
-Other domain directories can be run from pytest given the repository. The pytest 
-flag `--all_domains` will detect these repos and run them.
+```shell
+python generate_test_data.py -n=auto
+```
 
-Original source data can be found on Denali for most (all?) domains in `/home/jmccreight/pywatershed_data`.
+or as described in the [`autotest/README.md`](../autotest/README.md).Should
+problems arise with the above, additional details are provided below.
 
-# Generating data
+In general, `autotest/generate_test_data.py` is a wrapper that calls the
+scripts in `test_data/generate`. It as a few of its own arguments and passes
+all other arguments along to pytest. The only additional thing that
+`generate_test_data.py` handles are dot files like
+`test_data/.test_data_version_{domain_name}.txt` which
+indicate by their existence that test data has been generated for a domain
+and by their contents the `version.txt` file from the root of the repository
+which was available when created.
 
-The `test_data/scripts` subdirectory contains code for reproducing test data in the domains. Importantly, `test_run_domains.py` should be run occasionally to update the test data.  After running the domains, NetCDF files can be created from simulation outputs by running the tests in `test_nc_domains.py`. E.g.,
+Scripts in the `test_data/generate/` subdirectory, which create and destroy
+test data are: 
+
+* `remove_output_dirs.py`
+* `run_prms_domains.py`
+* `convert_prms_output_to_nc.py`
+* `remove_prms_csvs.py`
+
+The first and last remove data and are generally only used to support testing
+in CI. The "run" and "convert" are the core scripts for generating the
+test data.
+
+The `run_prms_domains.py` scripts uses binaries provided in the pywatershed
+repository to run reference models, specifically the PRMS v5.2.1 binary. All
+domains included in the the 'test_data/' directory will be run with
+the binary (unless the `--domain` option is exercised) and the output
+generated by the binary then becomes test answers or reference data for
+pytest autotests. The `convert_prms_output_to_nc.py` simply converts
+the output from the first step to NetCDF format and also calculates a few
+derived variables, providing the actual data sources used by `autotest`.
+
+It is possible that the included binaries will not work on your system.
+For a failing PRMS 5.2.1 binary, please see [`prms_src/prms5.2.1/README.md`](../prms_src/prms5.2.1/README.md).
+If you compile yourself, you'll need to copy `prms_src/prms5.2.1/bin/prms`
+to `bin/` while renaming it to one of the existing files found therein.
+Currently, supplied binaries are all compiled with gfortran except on
+Mac ARM architecture where ifort is used.
+
+## Domain directories
+
+Additional domains may be supplied by the user for autotesting by
+pywatershed. Also note that evaluation of pywatershed against PRMS in
+the notebook `examples/03_compare_pws_prms.ipynb` uses this approach
+as well. To supply your own domain, you must generate the required
+files in `test_data/your_domain` given in this listing:
 
-```shell
-pytest -v -n auto test_run_domains.py
-pytest -v -n auto test_nc_domains.py
 ```
+control.test  prcp.cbh      sf_data       tmax.nc       tmin.nc
+myparam.param prcp.nc       tmax.cbh      tmin.cbh
+```
+
+The `*.cbh` files must be pre-converted to netcdf for `prcp`, `tmin`,
+and `tmax`. How to do this can be found near the top of notebook
+`examples/02_prms_legacy_models.ipynb`. The `control.test` and
+`myparam.param` files are used by both PRMS and PWS. The `control.test`
+files in the repository are specific for being able to run sub-models in
+pywatershed and include a nearly maximal amount of model output
+(time-inefficient for both PRMS and PWS but necessary or pywatershed
+autotests). The stock control files can be found in `test_data/common`.
+There is a file for single-HRU domains and multi-HRU domains and these
+are identical (as appropriate) for the domains included in the
+repository.
 
-NetCDF dependencies are encoded implicitly into the `pytest` fixture system: `test_nc_domains.py` uses a custom test parametrization with `pytest_generate_tests` to map each CSV file created by the domain simulation to one or more NetCDF files, which are then aggregated into further files on session teardown by [yield fixtures](https://docs.pytest.org/en/7.2.x/how-to/fixtures.html#teardown-cleanup-aka-fixture-finalization). A [filelock](https://pytest-xdist.readthedocs.io/en/latest/how-to.html#making-session-scoped-fixtures-execute-only-once) is used to ensure aggregate files are only created once, even with multiple `pytest-xdist` workers.
+## Domain YAML file
+The contents of the `domain/domain.yaml` file is described in the
+[`autotest/README.md`](../autotest/README.md)
\ No newline at end of file
diff --git a/test_data/generate/conftest.py b/test_data/generate/conftest.py
index a8481217..539661e2 100644
--- a/test_data/generate/conftest.py
+++ b/test_data/generate/conftest.py
@@ -3,11 +3,25 @@
 import sys
 from fnmatch import fnmatch
 from platform import processor
+from shutil import copy2
+import time
 from typing import List
 from warnings import warn
 
 import pytest
 
+rootdir = ".."
+
+# Subset this to speed up tests by eliminating some domains
+test_dirs = sorted(
+    [path for path in pl.Path(rootdir).iterdir() if path.is_dir()]
+)
+
+# This would change to handle other/additional schedulers
+domain_globs_schedule = ["*conus*"]
+
+final_var_names = ["through_rain", "seg_lateral_inflow"]
+
 
 def pytest_addoption(parser):
     parser.addoption(
@@ -47,16 +61,6 @@ def exe():
     return exe_pth
 
 
-rootdir = ".."
-test_dirs = sorted(
-    [path for path in pl.Path(rootdir).iterdir() if path.is_dir()]
-)
-
-
-# This would change to handle other/additional schedulers
-domain_globs_schedule = ["*conus*"]
-
-
 def scheduler_active():
     slurm = os.getenv("SLURM_JOB_ID") is not None
     pbs = os.getenv("PBS_JOBID") is not None
@@ -178,7 +182,6 @@ def pytest_generate_tests(metafunc):
         metafunc.parametrize("simulation", sims, ids=ids, scope="session")
 
     if "final_file" in metafunc.fixturenames:
-        final_var_names = ["through_rain", "seg_lateral_inflow"]
         final_files = [
             pl.Path(kk) / var
             for kk in simulations.keys()