Repository: FredHutch/wdl-unit-tests
Pull request #173: Validate WDL dir contents
Author: login: sckott
Labels: name: infrastructure
description: Infrastructure fix to execute WDL GitHub Actions
Created: 2025-05-14T23:03:09Z | Merged: 2025-05-21T19:15:13Z

fix #76

- adds script for validating WDL folder contents 
- run with `test_api_cached` for now

This isn't my final pitch here, just to start the conversation. 

I'm using logging for now, but it could instead be an exception so the script or test run stops. or we could use a combination. 

Please can I have some feedback on this then i'll iterate, thanks

Linked issue #76: Validate WDL dir contents in tests
So there's some expectations that we have of contents in each WDL dir (what others call unit tests) that are not checked right now in code, but should be:

- [ ] validate that the `.wdl` file in the dir has the same exact name as the dir name
- [ ] inputs.json and options.json: we look for these, but at least in 1 case, there was an inputs file that was a different name and so we just silently didn't load it, leading to confusion. probably warn if there's some other files (json or otherwise) so the program doesn't stop but there's a warning at least - or maybe error since a warning could be missed?
- [ ] validate that if there's an inputs.json file there's only 1, same for options.json
- [ ] anything else?


Linked issue comments: user: login: tefirman
created_at: 2025-01-23T18:51:59Z
body: Love this idea. And I'm assuming we would want all of the other GitHub actions to "depend" on this one? i.e. they will only run if this check completes successfully first?

Might also want to check for the `data` subdirectory in case each test needs example data (see [arrayOperations](https://github.com/FredHutch/wdl-unit-tests/tree/main/arrayOperations))

One other thing we might eventually want to check for is a "cromwell.conf" configuration file, but not now. 
author_association: COLLABORATOR
user: login: sitapriyamoorthi
created_at: 2025-01-23T19:57:53Z
body: Maybe relevant but from what I know about womtools it only accepts 1 JSON file so if there are workflows that have two input jsons (for example sometime one may write a json that specifes parameters as a separate json) both of those wont be parsed and the validate will fail.
author_association: COLLABORATOR
user: login: sckott
created_at: 2025-01-23T20:19:53Z
body: @tefirman 

> And I'm assuming we would want all of the other GitHub actions to "depend" on this one? 

No, I'm not proposing a new gh action file. The logic would be in python code for both the api and java classes https://github.com/FredHutch/wdl-unit-tests/blob/main/tests/cromwellapi/cromwell.py and https://github.com/FredHutch/wdl-unit-tests/blob/main/tests/cromwelljava/utils.py
author_association: COLLABORATOR
user: login: sckott
created_at: 2025-01-23T20:53:14Z
body: thanks @sitapriyamoorthi 

the way the `Womtool` class works in https://github.com/FredHutch/wdl-unit-tests/blob/main/tests/cromwelljava/utils.py is it look for a file called `inputs.json` and only uses that as input in addition to the wdl file, so any additional files would be ignored. this does mean (just as for the `CromwellJava` class) that we need to check that files names are in the accepted set - which we'll do as discussed here
author_association: COLLABORATOR

Diff:
diff --git a/Makefile b/Makefile
index 6c20c9988..ed688b367 100644
--- a/Makefile
+++ b/Makefile
@@ -33,8 +33,11 @@ check_env_vars:
 	$(call check_defined, PATH_ROOTS, env var for which paths to scrub in vcr cassettes)
 	$(call check_defined, PROOF_API_TOKEN_DEV, env var for test PROOF user)
 
-test_api_cached: check_env_vars
-	op run -- uv run pytest -n $(WORKERS) \
+check_wdl_dirs:
+	@uv run tests/validate_wdls.py
+
+test_api_cached: check_env_vars check_wdl_dirs
+	@op run -- uv run pytest -n $(WORKERS) \
 	--color=yes --record-mode=once --verbose \
 	tests/cromwellapi/
 
diff --git a/pyproject.toml b/pyproject.toml
index 307b328db..acfaf3a47 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -15,6 +15,7 @@ dependencies = [
   "vcrpy>=7.0.0",
   "pytest-xdist>=3.6.1",
   "pytest-json-report>=1.5.0",
+  "rich>=14.0.0",
 ]
 
 [dependency-groups]
diff --git a/tests/common_utils.py b/tests/common_utils.py
index 0b8bcb238..d6e7899b0 100644
--- a/tests/common_utils.py
+++ b/tests/common_utils.py
@@ -1,6 +1,7 @@
 import json
 import sys
 from pathlib import Path
+from typing import Optional
 
 
 def is_interactive() -> bool:
@@ -29,3 +30,51 @@ def conditions(wdl_path: Path) -> dict:
     else:
         conditions = {"badVal": False, "badRunJava": False, "badRunAPI": False}
     return conditions
+
+
+def check_one(x: str):
+    allowed = ["badRun", "badRunAPI", "badRunJava", "badVal"]
+    if x not in allowed:
+        raise ValueError(f"include/exclude need to be in the set: {allowed}")
+
+
+def check_exclude_include(x: list):
+    [check_one(item) for item in x]
+
+
+def fetch_wdl_paths(
+    exclude: Optional[list] = None, include: Optional[list] = None
+) -> list:
+    """
+    Fetch WDL paths, with optional include or exclude
+
+    Examples:
+      fetch_wdl_paths()
+      fetch_wdl_paths(exclude=["badRunAPI"])
+      fetch_wdl_paths(exclude=["badRunAPI", "badRunJava"])
+      fetch_wdl_paths(include=["badVal", "badRunAPI"])
+    """
+    root = get_root()
+
+    paths = list(root.glob("**/*.wdl"))
+
+    if not exclude and not include:
+        return paths
+
+    paths_keep = []
+
+    if exclude:
+        check_exclude_include(exclude)
+        for path in paths:
+            bools = [conditions(path).get(item, False) for item in exclude]
+            if not any(bools):
+                paths_keep.append(path)
+
+    if include:
+        check_exclude_include(include)
+        for path in paths:
+            bools = [conditions(path).get(item, True) for item in include]
+            if any(bools):
+                paths_keep.append(path)
+
+    return paths_keep
diff --git a/tests/cromwellapi/test-call.py b/tests/cromwellapi/test-call.py
index 10feba7e2..ab5b67e41 100644
--- a/tests/cromwellapi/test-call.py
+++ b/tests/cromwellapi/test-call.py
@@ -2,8 +2,9 @@
 
 import pytest
 
+from ..common_utils import fetch_wdl_paths
 from .submit_wdl import submit_wdl
-from .utils import fetch_wdl_paths, metadata_response_keys
+from .utils import metadata_response_keys
 
 params = {"expandSubWorkflows": True}
 
diff --git a/tests/cromwellapi/test-failures.py b/tests/cromwellapi/test-failures.py
index 1eb0e7037..64289a1fb 100644
--- a/tests/cromwellapi/test-failures.py
+++ b/tests/cromwellapi/test-failures.py
@@ -3,8 +3,8 @@
 
 import pytest
 
+from ..common_utils import fetch_wdl_paths
 from .submit_wdl import submit_wdl
-from .utils import fetch_wdl_paths
 
 params = {
     "includeKey": [
diff --git a/tests/cromwellapi/test-metadata.py b/tests/cromwellapi/test-metadata.py
index f019904f2..00ed4020d 100644
--- a/tests/cromwellapi/test-metadata.py
+++ b/tests/cromwellapi/test-metadata.py
@@ -1,7 +1,8 @@
 import pytest
 
+from ..common_utils import fetch_wdl_paths
 from .submit_wdl import submit_wdl
-from .utils import fetch_wdl_paths, metadata_response_keys
+from .utils import metadata_response_keys
 
 params = {"expandSubWorkflows": False, "excludeKey": "calls"}
 
diff --git a/tests/cromwellapi/test-outputs.py b/tests/cromwellapi/test-outputs.py
index 229889f70..01aa1e1e4 100644
--- a/tests/cromwellapi/test-outputs.py
+++ b/tests/cromwellapi/test-outputs.py
@@ -1,7 +1,7 @@
 import pytest
 
+from ..common_utils import fetch_wdl_paths
 from .submit_wdl import submit_wdl
-from .utils import fetch_wdl_paths
 
 wdl_paths = fetch_wdl_paths()
 
diff --git a/tests/cromwellapi/test-successes.py b/tests/cromwellapi/test-successes.py
index ff56f7915..73ce504b8 100644
--- a/tests/cromwellapi/test-successes.py
+++ b/tests/cromwellapi/test-successes.py
@@ -2,8 +2,8 @@
 
 import pytest
 
+from ..common_utils import fetch_wdl_paths
 from .submit_wdl import submit_wdl
-from .utils import fetch_wdl_paths
 
 # Parameters included during the PROOF metadata query
 # Dictates which fields come through in the response
diff --git a/tests/cromwellapi/test-validate.py b/tests/cromwellapi/test-validate.py
index fe176c062..b3e1c252b 100644
--- a/tests/cromwellapi/test-validate.py
+++ b/tests/cromwellapi/test-validate.py
@@ -1,6 +1,6 @@
 import pytest
 
-from .utils import fetch_wdl_paths
+from ..common_utils import fetch_wdl_paths
 
 wdl_paths_not_bad_val = fetch_wdl_paths(exclude=["badVal"])
 wdl_paths_bad_val = fetch_wdl_paths(include=["badVal"])
diff --git a/tests/cromwellapi/utils.py b/tests/cromwellapi/utils.py
index 5eb3f6e5b..5ed244dd2 100644
--- a/tests/cromwellapi/utils.py
+++ b/tests/cromwellapi/utils.py
@@ -1,9 +1,6 @@
 import os
 from datetime import datetime, timedelta
 from pathlib import Path
-from typing import Optional
-
-from ..common_utils import conditions, get_root
 
 metadata_response_keys = {
     "running": [
@@ -128,54 +125,6 @@ def before_sleep_message(state):
     )
 
 
-def check_one(x: str):
-    allowed = ["badRun", "badRunAPI", "badRunJava", "badVal"]
-    if x not in allowed:
-        raise ValueError(f"include/exclude need to be in the set: {allowed}")
-
-
-def check_exclude_include(x: list):
-    [check_one(item) for item in x]
-
-
-def fetch_wdl_paths(
-    exclude: Optional[list] = None, include: Optional[list] = None
-) -> list:
-    """
-    Fetch WDL paths, with optional include or exclude
-
-    Examples:
-      fetch_wdl_paths()
-      fetch_wdl_paths(exclude=["badRunAPI"])
-      fetch_wdl_paths(exclude=["badRunAPI", "badRunJava"])
-      fetch_wdl_paths(include=["badVal", "badRunAPI"])
-    """
-    root = get_root()
-
-    paths = list(root.glob("**/*.wdl"))
-
-    if not exclude and not include:
-        return paths
-
-    paths_keep = []
-
-    if exclude:
-        check_exclude_include(exclude)
-        for path in paths:
-            bools = [conditions(path).get(item, False) for item in exclude]
-            if not any(bools):
-                paths_keep.append(path)
-
-    if include:
-        check_exclude_include(include)
-        for path in paths:
-            bools = [conditions(path).get(item, True) for item in include]
-            if any(bools):
-                paths_keep.append(path)
-
-    return paths_keep
-
-
 def find_project_root(marker_file: str = "pyproject.toml") -> str:
     """
     A hack to avoid using __file__ as that does not work when using a
diff --git a/tests/validate_wdls.py b/tests/validate_wdls.py
new file mode 100644
index 000000000..b06fe2065
--- /dev/null
+++ b/tests/validate_wdls.py
@@ -0,0 +1,103 @@
+import logging
+
+from common_utils import fetch_wdl_paths
+from rich import print as rich_print
+from rich.logging import RichHandler
+
+logging.basicConfig(
+    level=logging.WARNING,
+    format="%(message)s",
+    handlers=[RichHandler(show_time=False, markup=True)],
+)
+log = logging.getLogger("rich")
+
+
+class LogCapture:
+    def __init__(self):
+        self.records = []
+        self.handler = None
+
+    def setup(self):
+        self.handler = logging.Handler()
+        self.handler.setLevel(logging.WARNING)
+        self.handler.emit = lambda record: self.records.append(record)
+        log.addHandler(self.handler)
+
+    def teardown(self):
+        if self.handler:
+            log.removeHandler(self.handler)
+
+    def has_logs(self):
+        return len(self.records) > 0
+
+
+def validate_wdls(wdl_paths, raise_on_logs=True):
+    """Validate WDL files and optionally raise exception if any issues are found.
+
+    Args:
+        wdl_paths: List of paths to WDL files to validate
+        raise_on_logs: If True, raise exception if any validation logs are produced
+
+    Raises:
+        ValueError: If raise_on_logs is True and any validation logs are produced
+    """
+    log_capture = LogCapture()
+    try:
+        log_capture.setup()
+        [check_wdl_name_matches(w) for w in wdl_paths]
+        [check_for_other_json_files(w) for w in wdl_paths]
+        [check_data_dirs(w) for w in wdl_paths]
+
+        if raise_on_logs and log_capture.has_logs():
+            raise ValueError(
+                f"validation failed with {len(log_capture.records)} issues. "
+                "See logs above for details."
+            )
+    finally:
+        log_capture.teardown()
+
+
+def check_wdl_name_matches(path):
+    if not path.name.replace(".wdl", "") == path.parent.name:
+        log.warning(
+            f"WDL dir name [u]{path.parent.name}[/] and file name"
+            f" [b yellow]{path.name}[/] do not match"
+        )
+
+
+def check_for_other_json_files(path):
+    jsons = list(path.parent.glob("*.json"))
+    if jsons:
+        for json in jsons:
+            if json.name not in [
+                "options.json",
+                "inputs.json",
+                "conditions.json",
+            ]:
+                log.warning(
+                    f"WDL dir [u]{json.parent.name}[/] has additional json"
+                    f" files beyond inputs/options: [b yellow]{json.name}[/]"
+                )
+
+
+def check_data_dirs(path):
+    dirs = [x for x in path.parent.iterdir() if x.is_dir()]
+    if dirs:
+        for ele in dirs:
+            if ele.name not in ["data"]:
+                log.warning(
+                    f"WDL dir [u]{path.parent.name}[/] has subdir "
+                    f"[b yellow]{ele.name}[/]; only dir allowed is "
+                    f"[b green]data[/]"
+                )
+
+
+if __name__ == "__main__":
+    rich_print("Validating WDLs...")
+    wdl_paths = fetch_wdl_paths()
+    try:
+        validate_wdls(wdl_paths)
+        rich_print("Validation completed successfully!")
+    except ValueError as e:
+        rich_print(f"Error: {e}")
+        exit(1)


Reviews:
user: login: tefirman
body: @sckott -- Works great on my end with `make test_api_cached`, really like this direction. My only wonder is whether we also want to check for the `data` subdirectory like in the [arrayOperations](https://github.com/FredHutch/wdl-unit-tests/tree/main/arrayOperations) unit test. In a similar fashion, making sure that the only subdirectory allowed is `data`. Thoughts?
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-15T23:32:40Z
user: login: seankross
body: Otherwise lgtm but I want @tefirman's feedback and approval before merging.
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-19T15:35:21Z
user: login: sckott
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-19T16:58:04Z
user: login: sckott
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-19T17:22:39Z
user: login: tefirman
body: Looks awesome, thanks @sckott !!
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-05-21T18:25:02Z

Review comments:
diff_hunk: @@ -0,0 +1,37 @@
+import logging
+
+from utils import fetch_wdl_paths
+
+logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
+
+
+def validate_wdls(wdl_paths):
+    [check_wdl_name_matches(w) for w in wdl_paths]
+    [check_for_other_json_files(w) for w in wdl_paths]
+
+
+def check_wdl_name_matches(path):
+    # print(f"Checking {path}")
+    if not path.name.replace(".wdl", "") == path.parent.name:
+        logging.info(
+            f"WDL file name {path.name} and dir name {path.parent.name} do not match"
+        )
+
+
+def check_for_other_json_files(path):
+    jsons = list(path.parent.glob("*.json"))
+    if jsons:
+        for json in jsons:
+            if json.name not in ["options.json", "inputs.json"]:
+                logging.info(
path: tests/cromwellapi/validate_wdls.py
user: login: seankross
body: Instead of info() can we use warning() so it shows up as WARNING in the output?
created_at: 2025-05-19T15:34:03Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,37 @@
+import logging
+
+from utils import fetch_wdl_paths
+
+logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
+
+
+def validate_wdls(wdl_paths):
+    [check_wdl_name_matches(w) for w in wdl_paths]
+    [check_for_other_json_files(w) for w in wdl_paths]
+
+
+def check_wdl_name_matches(path):
+    # print(f"Checking {path}")
+    if not path.name.replace(".wdl", "") == path.parent.name:
+        logging.info(
+            f"WDL file name {path.name} and dir name {path.parent.name} do not match"
+        )
+
+
+def check_for_other_json_files(path):
+    jsons = list(path.parent.glob("*.json"))
+    if jsons:
+        for json in jsons:
+            if json.name not in ["options.json", "inputs.json"]:
+                logging.info(
path: tests/cromwellapi/validate_wdls.py
user: login: sckott
body: @seankross just for this check or do you mean for all checks?
created_at: 2025-05-19T16:58:04Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,37 @@
+import logging
+
+from utils import fetch_wdl_paths
+
+logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
+
+
+def validate_wdls(wdl_paths):
+    [check_wdl_name_matches(w) for w in wdl_paths]
+    [check_for_other_json_files(w) for w in wdl_paths]
+
+
+def check_wdl_name_matches(path):
+    # print(f"Checking {path}")
+    if not path.name.replace(".wdl", "") == path.parent.name:
+        logging.info(
+            f"WDL file name {path.name} and dir name {path.parent.name} do not match"
+        )
+
+
+def check_for_other_json_files(path):
+    jsons = list(path.parent.glob("*.json"))
+    if jsons:
+        for json in jsons:
+            if json.name not in ["options.json", "inputs.json"]:
+                logging.info(
path: tests/cromwellapi/validate_wdls.py
user: login: sckott
body: ![Screenshot 2025-05-19 at 10 19 02 AM](https://github.com/user-attachments/assets/40086cbf-f9a2-4531-ba99-d78e70a755f6)

@seankross changed to warnings for both checks. using `rich` for colored logging and it automatically shows line numbers 
created_at: 2025-05-19T17:22:38Z
author_association: COLLABORATOR

PR comments:
user: login: sckott
created_at: 2025-05-15T16:47:56Z
body: This is what a run looks like where I purposefully caused an issue (there aren't any right now). It's the first two lines of the output that I want you to look at. 

```
Validating WDLs...
INFO: WDL file name testNestedJsonArrayadfadsf.wdl and dir name testNestedJsonArray do not match
op run -- uv run pytest -n auto \
	--color=yes --record-mode=once --verbose -s \
	tests/cromwellapi/test-abort.py
============================= test session starts ==============================
platform darwin -- Python 3.13.3, pytest-8.3.4, pluggy-1.5.0 -- /Users/schambe3/github/FredHutch/wdl-unit-tests/.venv/bin/python
cachedir: .pytest_cache
metadata: {'Python': '3.13.3', 'Platform': 'macOS-15.4.1-arm64-arm-64bit-Mach-O', 'Packages': {'pytest': '8.3.4', 'pluggy': '1.5.0'}, 'Plugins': {'recording': '0.13.2', 'json-report': '1.5.0', 'metadata': '3.1.1', 'anyio': '4.7.0', 'xdist': '3.6.1'}, 'AWS_REGION': 'us-west-2'}
vcr recording mode: once (ignore Retry attempt messages)
vcr cassettes last recorded approx.: April 16, 2025 09:25
rootdir: /Users/schambe3/github/FredHutch/wdl-unit-tests
configfile: pytest.ini
plugins: recording-0.13.2, json-report-1.5.0, metadata-3.1.1, anyio-4.7.0, xdist-3.6.1
created: 8/8 workers
8 workers [2 items]

scheduling tests via LoadScheduling

tests/cromwellapi/test-abort.py::test_abort[helloDockerHostname]
tests/cromwellapi/test-abort.py::test_abort[helloHostname]
[gw0] PASSED tests/cromwellapi/test-abort.py::test_abort[helloHostname]
[gw1] PASSED tests/cromwellapi/test-abort.py::test_abort[helloDockerHostname]
```
author_association: COLLABORATOR
user: login: sckott
created_at: 2025-05-16T17:08:17Z
body: @tefirman sure we can do the check for data dir. but is that something that's decided or still up in the air? if decided i'll do it. 

please can you comment on whether you think these checks should be logged as happens now, or do you want errors to stop test runs or logs for some checks and errors for others?
author_association: COLLABORATOR
user: login: tefirman
created_at: 2025-05-19T17:37:28Z
body: @sckott -- Sorry for the delayed response!

> @tefirman sure we can do the check for data dir. but is that something that's decided or still up in the air? if decided i'll do it.

That's how we outline it in the [CONTRIBUTING.md file](https://github.com/FredHutch/wdl-unit-tests/blob/81aae11361e61e6ca175d9bf0a2149fe199193dc/.github/CONTRIBUTING.md?plain=1#L27), so my assumption was that it was decided. It's not a regular occurrence, I just want to make sure people aren't putting in rogue subdirectories.

> please can you comment on whether you think these checks should be logged as happens now, or do you want errors to stop test runs or logs for some checks and errors for others?

My gut is that a content validation failure should fail the entire "API Tests" check, and therefore stop us from merging to main until that's resolved. I think it's fine to let the subsequent test runs continue after a content validation failure to make sure everything else looks good, but I just want to make sure we avoid merging content failures into main. Would that just look like adding some assertions to the `validate_wdls.py` script?
author_association: COLLABORATOR
user: login: sckott
created_at: 2025-05-19T18:18:17Z
body: thanks @tefirman - can you clarify what a "content validation failure" is? all of the checks in this PR? or something else?
author_association: COLLABORATOR
user: login: tefirman
created_at: 2025-05-19T18:20:27Z
body: > thanks @tefirman - can you clarify what a "content validation failure" is? all of the checks in this PR? or something else?

Yes, the checks performed in `validate_wdls.py` in this PR. Am I understanding right that currently they just print a warning?
author_association: COLLABORATOR
user: login: sckott
created_at: 2025-05-19T18:24:27Z
body: > Yes, the checks performed in validate_wdls.py in this PR

thanks

> Am I understanding right that currently they just print a warning?

yes, a warning. we could even log an error type but that doesn't stop execution at all. i'll play with stopping execution and see what that looks like
author_association: COLLABORATOR
user: login: sckott
created_at: 2025-05-19T19:13:31Z
body: @tefirman with last commit pushed, here's an example of what the 3 different checks could output:

![Screenshot 2025-05-19 at 12 13 22 PM](https://github.com/user-attachments/assets/3839661d-6c75-45e6-ad7c-8d2dc35276e8)


I opened a PR (https://github.com/FredHutch/wdl-unit-tests/pull/177) on a new branch for the option to raise an exception instead - does that seem better or stick with no exceptions raised?
author_association: COLLABORATOR