Repository: facebookresearch/CompilerGym

Issue:
[validation] Assert in validate_sha_output() causes infinite hang on failure
## 🐛 Bug

Failures in the validation script from this assertion cause the script to hang forever. Stack trace:

```
❌  cBench-v0/rijndael  Failed 1 of 20 validators: [1/1] Benchmark output differs from expected., Expected: E0227 14:31:39.739494155  442433 backup_poller.cc:133]       Run client channel backup poller: {"created":"@1614465099.739425805","description":"pollset_work","file":"src/core/lib/iomgr/ev_epollex_linux.cc","file_line":321,"referenced_errors":[{"created":"@1614465099.739423909","description":"Bad file descriptor","errno":9,"file":"src/core/lib/iomgr/ev_epollex_linux.cc","file_line":952,"os_error":"Bad file descriptor","syscall":"epoll_wait"}]}, , Actual:
121 remaining states to validate ... multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "/private/home/cummins/.conda/envs/compiler_gym/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/private/home/cummins/.conda/envs/compiler_gym/lib/python3.8/site-packages/compiler_gym-0.1.3-py3.8.egg/compiler_gym/validate.py", line 22, in _validate_states_worker
    result = env.validate(state)
  File "/private/home/cummins/.conda/envs/compiler_gym/lib/python3.8/site-packages/compiler_gym-0.1.3-py3.8.egg/compiler_gym/envs/compiler_env.py", line 1009, in validate
    semantics_error = validate_semantics(self)
  File "/private/home/cummins/.conda/envs/compiler_gym/lib/python3.8/site-packages/compiler_gym-0.1.3-py3.8.egg/compiler_gym/envs/llvm/datasets.py", line 469, in composed
    errors = [result for result in results if result is not None]
  File "/private/home/cummins/.conda/envs/compiler_gym/lib/python3.8/site-packages/compiler_gym-0.1.3-py3.8.egg/compiler_gym/envs/llvm/datasets.py", line 469, in <listcomp>
    errors = [result for result in results if result is not None]
  File "/private/home/cummins/.conda/envs/compiler_gym/lib/python3.8/site-packages/compiler_gym-0.1.3-py3.8.egg/compiler_gym/envs/llvm/datasets.py", line 468, in <genexpr>
    results = (future.result() for future in as_completed(futures))
  File "/private/home/cummins/.conda/envs/compiler_gym/lib/python3.8/concurrent/futures/_base.py", line 432, in result
    return self.__get_result()
  File "/private/home/cummins/.conda/envs/compiler_gym/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
    raise self._exception
  File "/private/home/cummins/.conda/envs/compiler_gym/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/private/home/cummins/.conda/envs/compiler_gym/lib/python3.8/site-packages/compiler_gym-0.1.3-py3.8.egg/compiler_gym/envs/llvm/datasets.py", line 353, in validator_cb
    validate_result(outcome)
  File "/private/home/cummins/.conda/envs/compiler_gym/lib/python3.8/site-packages/compiler_gym-0.1.3-py3.8.egg/compiler_gym/envs/llvm/datasets.py", line 502, in validate_sha_output
    assert re.match(
AssertionError
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "e_greedy.py", line 115, in <module>
    eval_policy(e_greedy_search)
  File "/private/home/cummins/src/CompilerGym/leaderboard/llvm_codesize/e_greedy/../eval_policy.py", line 233, in eval_policy
    app.run(main)
  File "/private/home/cummins/.conda/envs/compiler_gym/lib/python3.8/site-packages/absl/app.py", line 303, in run
    _run_main(main, args)
  File "/private/home/cummins/.conda/envs/compiler_gym/lib/python3.8/site-packages/absl/app.py", line 251, in _run_main
    sys.exit(main(argv))
  File "/private/home/cummins/src/CompilerGym/leaderboard/llvm_codesize/e_greedy/../eval_policy.py", line 231, in main
    validate(["argv0", FLAGS.logfile])
  File "/private/home/cummins/.conda/envs/compiler_gym/lib/python3.8/site-packages/compiler_gym-0.1.3-py3.8.egg/compiler_gym/bin/validate.py", line 261, in main
    for i, result in enumerate(validation_results, start=1):
  File "/private/home/cummins/.conda/envs/compiler_gym/lib/python3.8/site-packages/compiler_gym-0.1.3-py3.8.egg/compiler_gym/validate.py", line 67, in validate_states
    yield from map_func(
  File "/private/home/cummins/.conda/envs/compiler_gym/lib/python3.8/multiprocessing/pool.py", line 868, in next
    raise value
AssertionError
```

## To Reproduce

Write a unit test that mocks a failing SHA output.

## Expected behavior

The error is reported as `not ValidationResult.okay()`, not an assertion.

## Environment

Please fill in this checklist:

- CompilerGym: v0.1.3
- How you installed CompilerGym (conda, pip, source): source
- OS: Ubuntu 20.04
- Python version: 3.8


Discussion:

Resolving patch:
diff --git a/compiler_gym/envs/llvm/datasets.py b/compiler_gym/envs/llvm/datasets.py
--- a/compiler_gym/envs/llvm/datasets.py
+++ b/compiler_gym/envs/llvm/datasets.py
@@ -501,15 +501,19 @@ def composed(env):
 # ===============================
 
 
-def validate_sha_output(result: BenchmarkExecutionResult):
+def validate_sha_output(result: BenchmarkExecutionResult) -> Optional[str]:
     """SHA benchmark prints 5 random hex strings. Normally these hex strings are
     16 characters but occasionally they are less (presumably becuase of a
     leading zero being omitted).
     """
-    assert re.match(
-        r"[0-9a-f]{0,16} [0-9a-f]{0,16} [0-9a-f]{0,16} [0-9a-f]{0,16} [0-9a-f]{0,16}",
-        result.output.decode("utf-8").rstrip(),
-    )
+    try:
+        if not re.match(
+            r"[0-9a-f]{0,16} [0-9a-f]{0,16} [0-9a-f]{0,16} [0-9a-f]{0,16} [0-9a-f]{0,16}",
+            result.output.decode("utf-8").rstrip(),
+        ):
+            return "Failed to parse hex output"
+    except UnicodeDecodeError:
+        return "Failed to parse unicode output"
 
 
 def setup_ghostscript_library_files(cwd: Path):


Test patch:
diff --git a/tests/llvm/BUILD b/tests/llvm/BUILD
--- a/tests/llvm/BUILD
+++ b/tests/llvm/BUILD
@@ -83,6 +83,16 @@ py_test(
     ],
 )
 
+py_test(
+    name = "datasets_test",
+    timeout = "short",
+    srcs = ["datasets_test.py"],
+    deps = [
+        "//compiler_gym/envs/llvm:datasets",
+        "//tests:test_main",
+    ],
+)
+
 py_test(
     name = "fork_env_test",
     timeout = "short",
diff --git a/tests/llvm/datasets_test.py b/tests/llvm/datasets_test.py
new file mode 100644
--- /dev/null
+++ b/tests/llvm/datasets_test.py
@@ -0,0 +1,29 @@
+# Copyright (c) Facebook, Inc. and its affiliates.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+"""Tests for //compiler_gym/envs/llvm:datasets."""
+
+from compiler_gym.envs.llvm import datasets
+from tests.test_main import main
+
+
+def test_validate_sha_output_okay():
+    output = datasets.BenchmarkExecutionResult(
+        walltime_seconds=0,
+        output="1234567890abcdef 1234567890abcd 1234567890abc 1234567890 12345".encode(
+            "utf-8"
+        ),
+    )
+    assert datasets.validate_sha_output(output) is None
+
+
+def test_validate_sha_output_invalid():
+    output = datasets.BenchmarkExecutionResult(
+        walltime_seconds=0, output="abcd".encode("utf-8")
+    )
+    assert datasets.validate_sha_output(output)
+
+
+if __name__ == "__main__":
+    main()