Repository: facebookresearch/CompilerGym

Issue:
Is running CompilerGym intended to leave cache directories behind?
## ❓ Questions and Help

Not sure if this is a bug or not, so submitting as a question. Running a CompilerGym experiment leaves behind many cache directories. When running a large experiment, this can create problems through the sheer number of directories in `COMPILER_GYM_CACHE`. I expected the `COMPILER_GYM_CACHE` to not have anything after the experiment exited cleanly.

Is there a way to avoid the experiments leaving the directories behind?

## Steps to reproduce
Running the following on my machine leaves behind about 270 cache directories.

```python
import compiler_gym
import compiler_gym.wrappers
from ray import tune
from ray.rllib.agents.ppo import PPOTrainer


def make_env(env_config):
    env = compiler_gym.make(env_config['cgym_id'])
    env = compiler_gym.wrappers.TimeLimit(env, env_config['timelimit'])
    dataset = env.datasets[env_config['dataset']]
    env = compiler_gym.wrappers.CycleOverBenchmarks(
        env, dataset.benchmarks())
    return env


config = {
    "env_config": {
        "cgym_id": "llvm-autophase-ic-v0",
        "timelimit": 45,
        "dataset": "benchmark://cbench-v1",
    },
    "env": "CompilerGym",
}

stop = {
    "timesteps_total": 10_000,
}

tune.register_env("CompilerGym", make_env)
tune.run(
    PPOTrainer,
    config=config,
    stop=stop,
    name='cgym_cache_dir_demo',
)
```

## Environment

Please fill in this checklist:

- CompilerGym: 0.2.2
- How you installed CompilerGym (conda, pip, source): pip
- OS: Ubuntu 20.04.1 LTS (x86_64)
- Python version: 3.9.7
- Build command you used (if compiling from source): N/A
- GCC/clang version (if compiling from source): N/A
- Bazel version (if compiling from source): N/A
- Versions of any other relevant libraries: ray: 1.10.0, gym: 0.20.0



Discussion:
Hi @vuoristo,

Thanks for the report, google question!

> Running a CompilerGym experiment leaves behind many cache directories. When running a large experiment, this can create problems through the sheer number of directories in COMPILER_GYM_CACHE.

This sounds like something is wrong.

There are three locations where CompilerGym creates files at runtime. The first is `~/.local/share/compiler_gym`, which is used to store data files downloaded from the net. When you use a new environment or new dataset, CompilerGym may need to download some data and it will be cached here. After running your example script on my machine, my `~/.local/share/compiler_gym` has ~600 files in it:

```
$ tree ~/.local/share/compiler_gym -a 
/home/cummins/.local/share/compiler_gym
└── llvm-v0
   ...

25 directories, 618 files
```

Then there is an on-disk cache, located at `~/.cache/compiler_gym`. This is used by CompilerGym for a handful of purposes like caching downloads from the web. Here is mine:

```
$ tree ~/.cache/compiler_gym/ -a
/home/cummins/.cache/compiler_gym/
├── .cbench-v1-runtime-data.LOCK
├── downloads
│   ├── 59c3f328efd51994a11168ca15e43a8d422233796c6bc167c9eb771c7bd6b57e
│   ├── .59c3f328efd51994a11168ca15e43a8d422233796c6bc167c9eb771c7bd6b57e.lock
│   ├── 601fff3944c866f6617e653b6eb5c1521382c935f56ca1f36a9f5cf1a49f3de5
│   ├── .601fff3944c866f6617e653b6eb5c1521382c935f56ca1f36a9f5cf1a49f3de5.lock
│   ├── a1b5b5d6b115e5809ccaefc2134434494271d184da67e2ee43d7f84d07329055
│   ├── .a1b5b5d6b115e5809ccaefc2134434494271d184da67e2ee43d7f84d07329055.lock
│   ├── eeffd7593aeb696a160fd22e6b0c382198a65d0918b8440253ea458cfe927741
│   └── .eeffd7593aeb696a160fd22e6b0c382198a65d0918b8440253ea458cfe927741.lock
└── .llvm-v0-install.LOCK
```

Finally, there is a "transient cache". On your linux machine this will default to `/dev/shm/compiler_gym_${USER}`, stored in RAM. Every time you create an environment, it gets its own dedicated directory which it uses to write a couple of needed files and also to use as a scratch space. Importantly, when the environment is closed (by calling `env.close()`), the directory is tidied up. So after running your script, the transient cache should contain only a single empty directory:

```
$ tree /dev/shm/compiler_gym_cummins/ -a
/dev/shm/compiler_gym_cummins/
└── s
```

My guess is that in your case, environments are not shutting down correctly, leaving behind files in the transient cache. Can you confirm what directory(s) are filling up?

BTW, the locations of these directories can be overriden by environment variables. See [source](https://github.com/facebookresearch/CompilerGym/blob/1d095e2b36aec2c43bd99cab60fbc0d36bc62a78/compiler_gym/util/runfiles_path.py#L58-L132).

Cheers,
Chris
Hi Chris, thanks for the quick response!

I'm seeing many lingering directories specifically in the `$COMPILER_GYM_CACHE` path, which I have set through an environment variable. The directories are named `benchmark-scratch-0030-0ce2`, `benchmark-scratch-0035-3536`, etc. There does not appear to be anything in the directories. The number of these directories grows each time I run the script.

```bash
$ find $COMPILER_GYM_CACHE -print
/home/risrio/cgym_test_cache2
/home/risrio/cgym_test_cache2/benchmark-scratch-4b26-fc6d
/home/risrio/cgym_test_cache2/benchmark-scratch-f112-9fd6
/home/risrio/cgym_test_cache2/benchmark-scratch-795a-911f
/home/risrio/cgym_test_cache2/benchmark-scratch-3046-29da
/home/risrio/cgym_test_cache2/benchmark-scratch-4d02-023a
/home/risrio/cgym_test_cache2/benchmark-scratch-7f82-da00
/home/risrio/cgym_test_cache2/benchmark-scratch-39e7-584e
/home/risrio/cgym_test_cache2/benchmark-scratch-b305-9218
/home/risrio/cgym_test_cache2/benchmark-scratch-9347-2109
/home/risrio/cgym_test_cache2/benchmark-scratch-6136-80fc
/home/risrio/cgym_test_cache2/benchmark-scratch-48c0-c0b4
/home/risrio/cgym_test_cache2/benchmark-scratch-825f-54f3
/home/risrio/cgym_test_cache2/benchmark-scratch-062c-0933
/home/risrio/cgym_test_cache2/benchmark-scratch-81d0-9408
...
```

The number of items in `~/.local/share/compiler_gym` does not change if I run the same script repeatedly.

As far as I can tell my script exited cleanly. `tune` exits with message that I believe to be sign of success `2022-02-22 13:37:20,960 INFO tune.py:636 -- Total run time: 25.84 seconds (25.27 seconds for the tuning loop).` and there are no errors in the tune log directory.
Thanks for the extra details.

> As far as I can tell my script exited cleanly. tune exits with message that I believe to be sign of success 2022-02-22 13:37:20,960 INFO tune.py:636 -- Total run time: 25.84 seconds (25.27 seconds for the tuning loop). and there are no errors in the tune log directory.

Yep, your code looks good. The problem is in CompilerGym, specifically, [here](https://github.com/facebookresearch/CompilerGym/blob/d70408f82599a7ef727de5caa87b41e5f12e9693/compiler_gym/envs/llvm/service/Benchmark.cc#L182-L186). Either that code is not being called, or it is failing. Given that I can't repro the problem locally, it might take a bit more work to track down the cause.

One thing that would be useful to me is to know if your environments are crashing. If an environment crashes, the `done` value returned by `step()` is True, and there is a string `error_details` in the info dict. Could you perhaps collect them from your script and see if there are any messages?

```py
obs, reward, done, info = env.step(...)
if done:
     print("Error details:", info.get("error_details"))
     env.reset()
```

Cheers,
Chris
Right, some problem with `Benchmark::Close` not getting called looks like a probable cause.

I ran 
```python
env = make_env(env_config)
env.reset()
for i in range(100):
    for j in range(45):
        obs, reward, done, info = env.step(env.action_space.sample())
        if done:
            print("Error details:", info.get("error_details"))
            env.reset()
```
With `make_env` and `env_config` as defined above. The number of cache directories is increasing, confirming this is not related to ray. The error details are consistently `None`. I also tried dataset `generator://llvm-stress-v0` with the same results.
Thanks so much for following up @vuoristo. One last thing, could you please prepend the following to your script above and attaching the full output that running it generates?

```py
import logging
import os
os.environ["COMPILER_GYM_DEBUG"] = "4"
logging.basicConfig(level=logging.DEBUG)
```

Cheers,
Chris
Of course.

Here's the output for otherwise the same script, except it's running for only 40 steps total and the timelimit is set to 10. The same problem persists.

```
DEBUG:compiler_gym.service.connection:Exec ['./compiler_gym-llvm-service', '--working_dir=/dev/shm/compiler_gym_risrio/s/0302T125623-657646-1371', '--alsologtostderr', '-v=3', '--logbuflevel=-1']
I0302 12:56:23.693117 318122 CreateAndRunCompilerGymServiceImpl.h:123] Service "/dev/shm/compiler_gym_risrio/s/0302T125623-657646-1371" listening on 42835, PID = 318122
I0302 12:56:23.765956 318126 CompilerGymServiceImpl.h:41] GetSpaces()
DEBUG:compiler_gym.envs.compiler_env:Setting benchmark by name: benchmark://cbench-v1/qsort
DEBUG:compiler_gym.datasets.tar_dataset:Read benchmark://cbench-v1 manifest, 23 entries
DEBUG:compiler_gym.envs.compiler_env:Setting benchmark: benchmark://cbench-v1/adpcm
I0302 12:56:23.769544 318126 CompilerGymServiceImpl.h:60] StartSession(id=0, benchmark=benchmark://cbench-v1/adpcm), 1 active sessions
I0302 12:56:23.769908 318135 CompilerGymServiceImpl.h:192] AddBenchmark()
I0302 12:56:23.769939 318135 BenchmarkCache.cc:54] Cached benchmark benchmark://cbench-v1/adpcm (9535 bytes). Cache size = 9535 bytes, 1 items
I0302 12:56:23.770071 318126 CompilerGymServiceImpl.h:60] StartSession(id=0, benchmark=benchmark://cbench-v1/adpcm), 1 active sessions
I0302 12:56:23.770093 318126 BenchmarkFactory.cc:38] BenchmarkFactory initialized
I0302 12:56:23.770099 318126 BenchmarkFactory.cc:64] LLVM benchmark cache miss, add bitcode: benchmark://cbench-v1/adpcm
I0302 12:56:23.770124 318126 Benchmark.cc:123] llvm::parseBitcodeFile(9500 bits)
I0302 12:56:23.824918 318126 BenchmarkFactory.cc:124] Cached LLVM benchmark: benchmark://cbench-v1/adpcm. Cache size = 1 items
I0302 12:56:23.824942 318126 BenchmarkFactory.cc:55] LLVM benchmark cache hit: benchmark://cbench-v1/adpcm
I0302 12:56:23.825352 318126 Benchmark.cc:123] llvm::parseBitcodeFile(9352 bits)
I0302 12:56:23.827518 318135 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 12:56:23.828070 318126 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 12:56:23.828230 318135 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 12:56:23.828383 318126 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 12:56:23.828557 318135 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 12:56:23.828967 318126 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 12:56:23.829197 318135 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 12:56:23.830883 318126 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 12:56:23.831135 318135 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 12:56:23.831698 318126 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 12:56:23.832126 318135 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 12:56:23.832520 318126 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 12:56:23.833009 318135 CompilerGymServiceImpl.h:144] Session 0 Step()
DEBUG:compiler_gym.envs.compiler_env:Ending session 0
I0302 12:56:23.833914 318126 CompilerGymServiceImpl.h:120] EndSession(id=0), 0 sessions remaining
DEBUG:compiler_gym.envs.compiler_env:Setting benchmark: benchmark://cbench-v1/bitcount
I0302 12:56:23.834303 318135 CompilerGymServiceImpl.h:60] StartSession(id=1, benchmark=benchmark://cbench-v1/bitcount), 1 active sessions
I0302 12:56:23.834545 318126 CompilerGymServiceImpl.h:192] AddBenchmark()
I0302 12:56:23.834574 318126 BenchmarkCache.cc:54] Cached benchmark benchmark://cbench-v1/bitcount (15665 bytes). Cache size = 25200 bytes, 2 items
I0302 12:56:23.834695 318135 CompilerGymServiceImpl.h:60] StartSession(id=1, benchmark=benchmark://cbench-v1/bitcount), 1 active sessions
I0302 12:56:23.834713 318135 BenchmarkFactory.cc:64] LLVM benchmark cache miss, add bitcode: benchmark://cbench-v1/bitcount
I0302 12:56:23.834733 318135 Benchmark.cc:123] llvm::parseBitcodeFile(15528 bits)
I0302 12:56:23.917367 318135 BenchmarkFactory.cc:124] Cached LLVM benchmark: benchmark://cbench-v1/bitcount. Cache size = 2 items
I0302 12:56:23.917392 318135 BenchmarkFactory.cc:55] LLVM benchmark cache hit: benchmark://cbench-v1/bitcount
I0302 12:56:23.918035 318135 Benchmark.cc:123] llvm::parseBitcodeFile(15216 bits)
I0302 12:56:23.919950 318135 CompilerGymServiceImpl.h:144] Session 1 Step()
I0302 12:56:23.920950 318126 CompilerGymServiceImpl.h:144] Session 1 Step()
I0302 12:56:23.921136 318135 CompilerGymServiceImpl.h:144] Session 1 Step()
I0302 12:56:23.921278 318126 CompilerGymServiceImpl.h:144] Session 1 Step()
I0302 12:56:23.921450 318135 CompilerGymServiceImpl.h:144] Session 1 Step()
I0302 12:56:23.921826 318126 CompilerGymServiceImpl.h:144] Session 1 Step()
I0302 12:56:23.922421 318135 CompilerGymServiceImpl.h:144] Session 1 Step()
I0302 12:56:23.922672 318126 CompilerGymServiceImpl.h:144] Session 1 Step()
I0302 12:56:23.922906 318135 CompilerGymServiceImpl.h:144] Session 1 Step()
I0302 12:56:23.923146 318126 CompilerGymServiceImpl.h:144] Session 1 Step()
I0302 12:56:23.925016 318135 CompilerGymServiceImpl.h:144] Session 1 Step()
I0302 12:56:23.925307 318126 CompilerGymServiceImpl.h:144] Session 1 Step()
I0302 12:56:23.925541 318135 CompilerGymServiceImpl.h:144] Session 1 Step()
DEBUG:compiler_gym.envs.compiler_env:Ending session 1
I0302 12:56:23.926357 318126 CompilerGymServiceImpl.h:120] EndSession(id=1), 0 sessions remaining
DEBUG:compiler_gym.envs.compiler_env:Setting benchmark: benchmark://cbench-v1/blowfish
I0302 12:56:23.926792 318135 CompilerGymServiceImpl.h:60] StartSession(id=2, benchmark=benchmark://cbench-v1/blowfish), 1 active sessions
I0302 12:56:23.927074 318126 CompilerGymServiceImpl.h:192] AddBenchmark()
I0302 12:56:23.927119 318126 BenchmarkCache.cc:54] Cached benchmark benchmark://cbench-v1/blowfish (39349 bytes). Cache size = 64549 bytes, 3 items
I0302 12:56:23.927286 318135 CompilerGymServiceImpl.h:60] StartSession(id=2, benchmark=benchmark://cbench-v1/blowfish), 1 active sessions
I0302 12:56:23.927304 318135 BenchmarkFactory.cc:64] LLVM benchmark cache miss, add bitcode: benchmark://cbench-v1/blowfish
I0302 12:56:23.927333 318135 Benchmark.cc:123] llvm::parseBitcodeFile(39052 bits)
I0302 12:56:24.096865 318135 BenchmarkFactory.cc:124] Cached LLVM benchmark: benchmark://cbench-v1/blowfish. Cache size = 3 items
I0302 12:56:24.096895 318135 BenchmarkFactory.cc:55] LLVM benchmark cache hit: benchmark://cbench-v1/blowfish
I0302 12:56:24.098310 318135 Benchmark.cc:123] llvm::parseBitcodeFile(38564 bits)
I0302 12:56:24.101995 318126 CompilerGymServiceImpl.h:144] Session 2 Step()
I0302 12:56:24.102535 318135 CompilerGymServiceImpl.h:144] Session 2 Step()
I0302 12:56:24.102685 318126 CompilerGymServiceImpl.h:144] Session 2 Step()
I0302 12:56:24.102814 318135 CompilerGymServiceImpl.h:144] Session 2 Step()
I0302 12:56:24.102977 318126 CompilerGymServiceImpl.h:144] Session 2 Step()
I0302 12:56:24.104717 318135 CompilerGymServiceImpl.h:144] Session 2 Step()
I0302 12:56:24.105562 318126 CompilerGymServiceImpl.h:144] Session 2 Step()
I0302 12:56:24.105985 318135 CompilerGymServiceImpl.h:144] Session 2 Step()
I0302 12:56:24.110517 318126 CompilerGymServiceImpl.h:144] Session 2 Step()
I0302 12:56:24.111014 318135 CompilerGymServiceImpl.h:144] Session 2 Step()
I0302 12:56:24.111496 318126 CompilerGymServiceImpl.h:144] Session 2 Step()
I0302 12:56:24.113107 318135 CompilerGymServiceImpl.h:144] Session 2 Step()
I0302 12:56:24.113476 318126 CompilerGymServiceImpl.h:144] Session 2 Step()
DEBUG:compiler_gym.envs.compiler_env:Ending session 2
I0302 12:56:24.114728 318135 CompilerGymServiceImpl.h:120] EndSession(id=2), 0 sessions remaining
DEBUG:compiler_gym.envs.compiler_env:Setting benchmark: benchmark://cbench-v1/bzip2
I0302 12:56:24.115345 318126 CompilerGymServiceImpl.h:60] StartSession(id=3, benchmark=benchmark://cbench-v1/bzip2), 1 active sessions
I0302 12:56:24.115957 318135 CompilerGymServiceImpl.h:192] AddBenchmark()
I0302 12:56:24.116179 318135 BenchmarkCache.cc:54] Cached benchmark benchmark://cbench-v1/bzip2 (287991 bytes). Cache size = 352540 bytes, 4 items
I0302 12:56:24.116323 318126 CompilerGymServiceImpl.h:60] StartSession(id=3, benchmark=benchmark://cbench-v1/bzip2), 1 active sessions
I0302 12:56:24.116341 318126 BenchmarkFactory.cc:64] LLVM benchmark cache miss, add bitcode: benchmark://cbench-v1/bzip2
I0302 12:56:24.116487 318126 Benchmark.cc:123] llvm::parseBitcodeFile(287744 bits)
I0302 12:56:26.235937 318126 BenchmarkFactory.cc:124] Cached LLVM benchmark: benchmark://cbench-v1/bzip2. Cache size = 4 items
I0302 12:56:26.235983 318126 BenchmarkFactory.cc:55] LLVM benchmark cache hit: benchmark://cbench-v1/bzip2
I0302 12:56:26.246497 318126 Benchmark.cc:123] llvm::parseBitcodeFile(265480 bits)
I0302 12:56:26.267096 318126 CompilerGymServiceImpl.h:144] Session 3 Step()
I0302 12:56:26.281613 318135 CompilerGymServiceImpl.h:144] Session 3 Step()
I0302 12:56:26.281780 318126 CompilerGymServiceImpl.h:144] Session 3 Step()
I0302 12:56:26.281921 318135 CompilerGymServiceImpl.h:144] Session 3 Step()
I0302 12:56:26.282096 318126 CompilerGymServiceImpl.h:144] Session 3 Step()
I0302 12:56:26.315001 318135 CompilerGymServiceImpl.h:144] Session 3 Step()
I0302 12:56:26.317267 318126 CompilerGymServiceImpl.h:144] Session 3 Step()
I0302 12:56:26.319507 318135 CompilerGymServiceImpl.h:144] Session 3 Step()
I0302 12:56:26.321929 318126 CompilerGymServiceImpl.h:144] Session 3 Step()
I0302 12:56:26.330498 318135 CompilerGymServiceImpl.h:144] Session 3 Step()
I0302 12:56:26.332695 318126 CompilerGymServiceImpl.h:144] Session 3 Step()
I0302 12:56:26.469568 318135 CompilerGymServiceImpl.h:144] Session 3 Step()
I0302 12:56:26.473263 318126 CompilerGymServiceImpl.h:144] Session 3 Step()
DEBUG:compiler_gym.envs.compiler_env:Ending session 3
I0302 12:56:26.475675 318135 CompilerGymServiceImpl.h:120] EndSession(id=3), 0 sessions remaining
DEBUG:compiler_gym.envs.compiler_env:Setting benchmark: benchmark://cbench-v1/crc32
I0302 12:56:26.479002 318126 CompilerGymServiceImpl.h:60] StartSession(id=4, benchmark=benchmark://cbench-v1/crc32), 1 active sessions
I0302 12:56:26.479251 318135 CompilerGymServiceImpl.h:192] AddBenchmark()
I0302 12:56:26.479286 318135 BenchmarkCache.cc:54] Cached benchmark benchmark://cbench-v1/crc32 (8135 bytes). Cache size = 360675 bytes, 5 items
I0302 12:56:26.479410 318126 CompilerGymServiceImpl.h:60] StartSession(id=4, benchmark=benchmark://cbench-v1/crc32), 1 active sessions
I0302 12:56:26.479432 318126 BenchmarkFactory.cc:64] LLVM benchmark cache miss, add bitcode: benchmark://cbench-v1/crc32
I0302 12:56:26.479449 318126 Benchmark.cc:123] llvm::parseBitcodeFile(7904 bits)
I0302 12:56:26.532228 318126 BenchmarkFactory.cc:124] Cached LLVM benchmark: benchmark://cbench-v1/crc32. Cache size = 5 items
I0302 12:56:26.532276 318126 BenchmarkFactory.cc:55] LLVM benchmark cache hit: benchmark://cbench-v1/crc32
I0302 12:56:26.532634 318126 Benchmark.cc:123] llvm::parseBitcodeFile(7780 bits)
Error details: None
Error details: None
Error details: None
Error details: None
I0302 12:56:26.567595 318135 CompilerGymServiceImpl.h:120] EndSession(id=4), 0 sessions remaining
I0302 12:56:26.567958 318122 CreateAndRunCompilerGymServiceImpl.cc:19] Service received signal: 15
I0302 12:56:26.567991 318122 CreateAndRunCompilerGymServiceImpl.h:131] Shutting down the RPC service
I0302 12:56:26.568418 318122 BenchmarkFactory.cc:44] BenchmarkFactory closing with 5 entries

```
Thanks! I see that you're different benchmarks, what does your `make_env()` function do?

Here's my repro script:

```py
import logging
import os
import compiler_gym

os.environ["COMPILER_GYM_DEBUG"] = "4"
logging.basicConfig(level=logging.DEBUG)

env = compiler_gym.make("llvm-v0")
env.reset()
for i in range(10):
    for j in range(10):
        obs, reward, done, info = env.step(env.action_space.sample())
        if done:
            print("Error details:", info.get("error_details"))
            env.reset()
```

Which produces:

```
$ python issue582.py
DEBUG:compiler_gym.service.connection:Exec ['./compiler_gym-llvm-service', '--working_dir=/dev/shm/compiler_gym_cummins/s/0302T150137-648277-6fe5', '--alsologtostderr', '-v=3', '--logbuflevel=-1']
I0302 15:01:37.674084 34047 CreateAndRunCompilerGymServiceImpl.h:121] Service "/dev/shm/compiler_gym_cummins/s/0302T150137-648277-6fe5" listening on 46247, PID = 34047
I0302 15:01:37.755614 34051 CompilerGymServiceImpl.h:41] GetSpaces()
DEBUG:compiler_gym.envs.compiler_env:Setting benchmark by name: benchmark://cbench-v1/qsort
I0302 15:01:37.757656 34051 CompilerGymServiceImpl.h:60] StartSession(id=0, benchmark=benchmark://cbench-v1/qsort), 1 active sessions
I0302 15:01:37.757898 34060 CompilerGymServiceImpl.h:192] AddBenchmark()
I0302 15:01:37.757939 34060 BenchmarkCache.cc:54] Cached benchmark benchmark://cbench-v1/qsort (11512 bytes). Cache size = 11512 bytes, 1 items
I0302 15:01:37.758057 34051 CompilerGymServiceImpl.h:60] StartSession(id=0, benchmark=benchmark://cbench-v1/qsort), 1 active sessions
I0302 15:01:37.758086 34051 BenchmarkFactory.cc:41] BenchmarkFactory initialized
I0302 15:01:37.758096 34051 BenchmarkFactory.cc:67] LLVM benchmark cache miss, add bitcode: benchmark://cbench-v1/qsort
I0302 15:01:37.758132 34051 Benchmark.cc:99] llvm::parseBitcodeFile(10840 bits)
I0302 15:01:37.956909 34051 BenchmarkFactory.cc:142] Cached LLVM benchmark: benchmark://cbench-v1/qsort. Cache size = 1 items
I0302 15:01:37.956939 34051 BenchmarkFactory.cc:58] LLVM benchmark cache hit: benchmark://cbench-v1/qsort
I0302 15:01:37.957269 34051 Benchmark.cc:99] llvm::parseBitcodeFile(10512 bits)
I0302 15:01:37.960049 34060 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.960240 34051 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.961112 34060 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.961535 34051 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.962123 34060 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.962255 34051 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.962410 34060 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.962633 34051 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.962759 34060 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.963068 34051 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.963335 34060 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.963572 34051 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.963943 34060 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.964172 34051 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.964524 34060 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.964725 34051 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.965195 34060 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.965418 34051 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.966009 34060 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.966243 34051 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.966459 34060 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.967025 34051 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.967223 34060 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.967447 34051 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.967886 34060 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.968721 34051 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.968973 34060 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.969517 34051 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.970172 34060 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.970363 34051 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.970932 34060 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.971122 34051 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.971603 34060 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.972018 34051 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.972709 34060 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.973260 34051 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.973718 34060 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.974037 34051 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.974237 34060 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.974457 34051 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.974759 34060 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.975090 34051 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.975286 34060 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.975500 34051 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.976570 34060 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.977210 34051 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.977418 34060 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.977775 34051 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.977962 34060 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.978170 34051 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.978427 34060 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.978610 34051 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.978794 34060 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.979312 34051 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.979883 34060 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.980285 34051 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.980473 34060 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.980655 34051 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.980875 34060 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.981304 34051 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.981503 34060 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.981750 34051 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.981932 34060 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.982110 34051 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.982425 34060 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.982700 34051 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.982897 34060 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.983083 34051 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.983376 34060 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.983916 34051 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.984143 34060 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.984426 34051 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.984840 34060 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.985386 34051 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.985602 34060 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.985816 34051 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.986021 34060 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.986577 34051 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.986960 34060 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.987284 34051 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.987673 34060 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.987860 34051 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.988083 34060 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.988430 34051 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.988972 34060 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.989282 34051 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.989516 34060 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.989845 34051 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.990355 34060 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.990554 34051 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.991760 34060 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.992101 34051 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.992332 34060 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.992516 34051 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.993172 34060 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.993387 34051 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.993604 34060 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.994174 34051 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.994462 34060 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:37.994704 34051 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:01:38.038002 34060 CompilerGymServiceImpl.h:120] EndSession(id=0), 0 sessions remaining
I0302 15:01:38.038374 34047 CreateAndRunCompilerGymServiceImpl.cc:19] Service received signal: 15
I0302 15:01:38.038400 34047 CreateAndRunCompilerGymServiceImpl.h:129] Shutting down the RPC service
I0302 15:01:38.038659 34047 BenchmarkFactory.cc:47] BenchmarkFactory closing with 1 entries
```

Cheers,
Chris
Here's the whole script

```python
import compiler_gym
import compiler_gym.wrappers

import logging
import os
os.environ["COMPILER_GYM_DEBUG"] = "4"
logging.basicConfig(level=logging.DEBUG)


def make_env(env_config):
    env = compiler_gym.make(env_config['cgym_id'])
    env = compiler_gym.wrappers.TimeLimit(env, env_config['timelimit'])
    dataset = env.datasets[env_config['dataset']]
    env = compiler_gym.wrappers.CycleOverBenchmarks(
        env, dataset.benchmarks())
    return env

env_config = {
    "cgym_id": "llvm-autophase-ic-v0",
    "timelimit": 10,
    "dataset": "benchmark://cbench-v1",
}
env = make_env(env_config)

env.reset()
for j in range(40):
    obs, reward, done, info = env.step(env.action_space.sample())
    if done:
        print("Error details:", info.get("error_details"))
        env.reset()
```
I also ran your script and on a quick inspection it looks to be producing similar results.

Hmm, it also looks like your script hangs on my machine after the last line of output. My script doesn't.

```
DEBUG:compiler_gym.service.connection:Exec ['./compiler_gym-llvm-service', '--working_dir=/dev/shm/compiler_gym_risrio/s/0302T150515-812567-c81f', '--alsologtostderr', '-v=3', '--logbuflevel=-1']
I0302 15:05:15.849172 391991 CreateAndRunCompilerGymServiceImpl.h:123] Service "/dev/shm/compiler_gym_risrio/s/0302T150515-812567-c81f" listening on 44479, PID = 391991
I0302 15:05:15.920815 391995 CompilerGymServiceImpl.h:41] GetSpaces()
DEBUG:compiler_gym.envs.compiler_env:Setting benchmark by name: benchmark://cbench-v1/qsort
I0302 15:05:15.923563 391995 CompilerGymServiceImpl.h:60] StartSession(id=0, benchmark=benchmark://cbench-v1/qsort), 1 active sessions
I0302 15:05:15.923972 392004 CompilerGymServiceImpl.h:192] AddBenchmark()
I0302 15:05:15.924013 392004 BenchmarkCache.cc:54] Cached benchmark benchmark://cbench-v1/qsort (11104 bytes). Cache size = 11104 bytes, 1 items
I0302 15:05:15.924160 391995 CompilerGymServiceImpl.h:60] StartSession(id=0, benchmark=benchmark://cbench-v1/qsort), 1 active sessions
I0302 15:05:15.924183 391995 BenchmarkFactory.cc:38] BenchmarkFactory initialized
I0302 15:05:15.924188 391995 BenchmarkFactory.cc:64] LLVM benchmark cache miss, add bitcode: benchmark://cbench-v1/qsort
I0302 15:05:15.924217 391995 Benchmark.cc:123] llvm::parseBitcodeFile(10840 bits)
I0302 15:05:16.029469 391995 BenchmarkFactory.cc:124] Cached LLVM benchmark: benchmark://cbench-v1/qsort. Cache size = 1 items
I0302 15:05:16.029495 391995 BenchmarkFactory.cc:55] LLVM benchmark cache hit: benchmark://cbench-v1/qsort
I0302 15:05:16.029983 391995 Benchmark.cc:123] llvm::parseBitcodeFile(10512 bits)
I0302 15:05:16.032155 391995 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.032637 392004 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.033224 391995 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.033984 392004 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.034288 391995 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.034648 392004 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.035384 391995 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.035591 392004 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.035796 391995 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.037271 392004 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.037470 391995 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.037636 392004 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.037778 391995 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.037969 392004 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.038117 391995 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.038278 392004 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.038457 391995 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.038632 392004 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.039322 391995 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.039465 392004 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.039616 391995 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.039748 392004 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.040328 391995 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.040783 392004 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.041045 391995 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.041198 392004 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.041615 391995 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.041754 392004 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.042027 391995 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.042438 392004 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.042594 391995 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.043009 392004 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.043221 391995 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.043455 392004 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.043920 391995 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.044070 392004 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.044209 391995 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.044509 392004 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.044759 391995 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.044937 392004 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.045099 391995 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.045565 392004 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.045706 391995 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.045840 392004 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.045981 391995 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.046136 392004 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.046418 391995 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.046819 392004 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.046955 391995 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.047240 392004 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.047382 391995 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.047523 392004 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.047976 391995 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.048111 392004 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.048300 391995 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.048584 392004 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.048806 391995 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.049480 392004 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.050238 391995 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.050395 392004 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.050726 391995 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.051934 392004 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.052244 391995 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.052835 392004 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.053035 391995 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.053191 392004 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.053908 391995 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.054080 392004 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.054754 391995 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.054911 392004 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.055052 391995 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.055649 392004 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.055788 391995 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.055991 392004 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.056182 391995 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.056370 392004 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.056500 391995 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.056628 392004 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.056798 391995 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.057271 392004 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.057454 391995 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.057644 392004 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.057813 391995 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.057950 392004 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.058454 391995 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.058637 392004 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.060055 391995 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.060211 392004 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.060364 391995 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.060519 392004 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.061035 391995 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.061327 392004 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.061688 391995 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.062019 392004 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.062213 391995 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.062355 392004 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.062891 391995 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.063369 392004 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.063635 391995 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.065101 392004 CompilerGymServiceImpl.h:144] Session 0 Step()
I0302 15:05:16.098289 391995 CompilerGymServiceImpl.h:120] EndSession(id=0), 0 sessions remaining
I0302 15:05:16.099086 391991 CreateAndRunCompilerGymServiceImpl.cc:19] Service received signal: 15
I0302 15:05:16.099114 391991 CreateAndRunCompilerGymServiceImpl.h:131] Shutting down the RPC service
I0302 15:05:16.099371 391991 BenchmarkFactory.cc:44] BenchmarkFactory closing with 1 entries
```
> Hmm, it also looks like your script hangs on my machine after the last line of output. My script doesn't.

This is because you are not closing `env.close()`. You must always call `env.close()` to ensure that temporary directories are tidied up and subprocesses are killed. You can either do this:

```py
env = make_env(env_config)
try:
  # use env...
finally:
  env.close()
```

or you can use a `with` statement so you don't have to remember to call `close()`:

```py
import compiler_gym
import compiler_gym.wrappers

import logging
import os
os.environ["COMPILER_GYM_DEBUG"] = "4"
logging.basicConfig(level=logging.DEBUG)


def make_env(env_config):
    env = compiler_gym.make(env_config['cgym_id'])
    env = compiler_gym.wrappers.TimeLimit(env, env_config['timelimit'])
    dataset = env.datasets[env_config['dataset']]
    env = compiler_gym.wrappers.CycleOverBenchmarks(
        env, dataset.benchmarks())
    return env

env_config = {
    "cgym_id": "llvm-autophase-ic-v0",
    "timelimit": 10,
    "dataset": "benchmark://cbench-v1",
}

with make_env(env_config) as env:
  env.reset()
  for j in range(40):
      obs, reward, done, info = env.step(env.action_space.sample())
      if done:
          print("Error details:", info.get("error_details"))
          env.reset()
```

Cheers,
Chris
Ah gotcha, I missed that. Running your latest script on my machine still leaves cache directories behind though. So calling `env.close` does not seem to be enough to fix that.

Resolving patch:
diff --git a/compiler_gym/service/runtime/create_and_run_compiler_gym_service.py b/compiler_gym/service/runtime/create_and_run_compiler_gym_service.py
--- a/compiler_gym/service/runtime/create_and_run_compiler_gym_service.py
+++ b/compiler_gym/service/runtime/create_and_run_compiler_gym_service.py
@@ -130,6 +130,7 @@ def main(argv):
         logging.info("Shutting down the RPC service")
         server.stop(60).wait()
         server_thread.join()
+        logging.info("Service closed")
 
         if len(service.sessions):
             print(


Test patch: