Repository: facebookresearch/CompilerGym

Issue:
CMake build gets stuck on ProGraML bazel step
## 🐛 Bug

I have encountered a strange error where the CMake build appears to get "stuck" after the bazel build step of ProGraML. The build appears to hang indefinitely. Cancelling and re-running the CMake build works fine.


## To Reproduce

Steps to reproduce the behavior:

```sh
$ cmake ....... # lots of flags (see below)

# <snip>
[3/9] Performing build_programl step for 'programl'
Starting local Bazel server and connecting to it...
Loading:
Loading: 0 packages loaded
Analyzing: 12 targets (5 packages loaded, 0 targets configured)
Analyzing: 12 targets (35 packages loaded, 209 targets configured)
Analyzing: 12 targets (50 packages loaded, 3390 targets configured)
INFO: Analyzed 12 targets (50 packages loaded, 6035 targets configured).
INFO: Found 12 targets...
[3 / 157] [Prepa] BazelWorkspaceStatusAction stable-status.txt
[84 / 227] Compiling libs/exception/src/clone_current_exception_non_intrusive.cpp; 0s linux-sandbox ... (32 actions, 31 running)
[94 / 227] Compiling absl/base/internal/sysinfo.cc; 1s linux-sandbox ... (32 actions, 31 running)
[111 / 227] Compiling libs/regex/src/c_regex_traits.cpp; 2s linux-sandbox ... (32 actions, 31 running)
[115 / 227] Compiling libs/regex/src/c_regex_traits.cpp; 3s linux-sandbox ... (32 actions running)
[119 / 227] Compiling libs/regex/src/c_regex_traits.cpp; 5s linux-sandbox ... (32 actions running)
[140 / 237] Compiling libs/regex/src/c_regex_traits.cpp; 6s linux-sandbox ... (32 actions running)
[156 / 243] Compiling libs/regex/src/c_regex_traits.cpp; 8s linux-sandbox ... (32 actions, 31 running)
[174 / 250] Compiling libs/regex/src/cregex.cpp; 10s linux-sandbox ... (32 actions, 31 running)
[220 / 270] Compiling libs/regex/src/cregex.cpp; 12s linux-sandbox ... (32 actions, 31 running)
INFO: From Compiling labm8/cpp/string.cc:
external/labm8/labm8/cpp/string.cc:135:22: warning: implicit conversion from 'int' to 'char' changes value from 226 to -30 [-Wconstant-conversion]
        to.append(1, 226);
           ~~~~~~    ^~~
external/labm8/labm8/cpp/string.cc:136:22: warning: implicit conversion from 'int' to 'char' changes value from 130 to -126 [-Wconstant-conversion]
        to.append(1, 130);
           ~~~~~~    ^~~
external/labm8/labm8/cpp/string.cc:137:22: warning: implicit conversion from 'int' to 'char' changes value from 172 to -84 [-Wconstant-conversion]
        to.append(1, 172);
           ~~~~~~    ^~~
3 warnings generated.
[269 / 272] Compiling programl/ir/llvm/internal/text_encoder.cc; 5s linux-sandbox
INFO: Elapsed time: 24.341s, Critical Path: 14.47s
INFO: 146 processes: 1 internal, 145 linux-sandbox.
INFO: Build completed successfully, 146 total actions
INFO: Build completed successfully, 146 total actions
```

Cancel, re-run CMake and the build proceeds as normal:

```sh
$ cmake .... # same as before
# <snip>
[0/2] Performing build_programl step for 'programl'
Loading:
Loading: 0 packages loaded
Analyzing: 12 targets (0 packages loaded, 0 targets configured)
INFO: Analyzed 12 targets (0 packages loaded, 0 targets configured).
INFO: Found 12 targets...
[0 / 1] [Prepa] BazelWorkspaceStatusAction stable-status.txt
INFO: Elapsed time: 0.425s, Critical Path: 0.01s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
INFO: Build completed successfully, 1 total action
[2/2] Completed 'programl'
# ....
```


## Environment

Please fill in this checklist:

- CompilerGym: development (0.2.2)
- How you installed CompilerGym (conda, pip, source): source
- OS: Ubuntu 20.04
- Python version: 3.8
- Build command you used (if compiling from source): `cmake -GNinja -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ -DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_CXX_COMPILER_LAUNCHER=ccache -DCMAKE_EXE_LINKER_FLAGS_INIT="-fuse-ld=lld" -DCMAKE_MODULE_LINKER_FLAGS_INIT="-fuse-ld=lld" -DCMAKE_SHARED_LINKER_FLAGS_INIT="-fuse-ld=lld" -DPython3_FIND_VIRTUALENV=FIRST -DCMAKE_BUILD_WITH_INSTALL_RPATH=true -S . -B build`
- GCC/clang version (if compiling from source): clang 10.0.0
- Bazel version (if compiling from source): 4.2.2
- Versions of any other relevant libraries:


Discussion:
cc @sogartar have you run into this before?
@ChrisCummins, I have noticed that sometimes if CMake need to do a reconfiguration before doing the build in the same execution it may get suck after the configuration. When you run the build again it would do only the build step, which points to that the config step has completed successfully. I have not figured out why this happens.

The strange thing here is that in your case it got stuck during configuration, when thirdparty stuff are getting built. Internally ProGraML is build in an proxy/wrapper CMake process. It is possible, that this same thing happens there. The difference there is that the configure command is invoked separately. In this use-case I have not seen CMake get stuck.

To debug this it is helpful to call CMake with the `--trace-expand` option to see where it gets stuck during configuration. Also add verbose flags to Ninja to see if it gets stuck during the build step. This should be done when executing the wrapper CMake processes https://github.com/facebookresearch/CompilerGym/blob/22ac8444ae7e0d589f6d2cee3a84159132b2adf7/build_tools/cmake/build_external_cmake_project.cmake#L29

If we are unable to fix this we may resort to putting a generous timeout and retry.
Thanks for the debugging tips! It is definitely getting stuck during the configuration stage. It's not a particularly large problem for now, and may be a total non-issue if we end up porting ProGraML's build to CMake to achieve #568.

Cheers,
Chris

Resolving patch:
diff --git a/compiler_gym/compiler_env_state.py b/compiler_gym/compiler_env_state.py
--- a/compiler_gym/compiler_env_state.py
+++ b/compiler_gym/compiler_env_state.py
@@ -26,7 +26,10 @@ class CompilerEnvState(BaseModel):
 
     benchmark: str = Field(
         allow_mutation=False,
-        examples=["benchmark://cbench-v1/crc32", "generator://csmith-v0/0",],
+        examples=[
+            "benchmark://cbench-v1/crc32",
+            "generator://csmith-v0/0",
+        ],
     )
     """The URI of the benchmark used for this episode."""
 
@@ -37,7 +40,9 @@ class CompilerEnvState(BaseModel):
     """The walltime of the episode in seconds. Must be non-negative."""
 
     reward: Optional[float] = Field(
-        required=False, default=None, allow_mutation=True,
+        required=False,
+        default=None,
+        allow_mutation=True,
     )
     """The cumulative reward for this episode. Optional."""
 
diff --git a/compiler_gym/service/client_service_compiler_env.py b/compiler_gym/service/client_service_compiler_env.py
--- a/compiler_gym/service/client_service_compiler_env.py
+++ b/compiler_gym/service/client_service_compiler_env.py
@@ -814,7 +814,7 @@ def _call_with_error(
 
         # If the action space has changed, update it.
         if reply.HasField("new_action_space"):
-            self.action_space = self.service_message_converters.action_space_converter(
+            self._action_space = self.service_message_converters.action_space_converter(
                 reply.new_action_space
             )
 
@@ -931,7 +931,7 @@ def raw_step(
 
         # If the action space has changed, update it.
         if reply.HasField("new_action_space"):
-            self.action_space = self.service_message_converters.action_space_converter(
+            self._action_space = self.service_message_converters.action_space_converter(
                 reply.new_action_space
             )
 


Test patch:
diff --git a/tests/requirements.txt b/tests/requirements.txt
--- a/tests/requirements.txt
+++ b/tests/requirements.txt
@@ -1,3 +1,4 @@
+coverage<6.0  # Implicit dependency of pytest-cov, fix version < 6.x for Py3.6 support.
 flaky==3.7.0
 psutil==5.8.0  # Implicit dependency of pytest-xdist
 pytest==6.2.5