Repository: facebookresearch/CompilerGym

Issue:
--benchmark flag not working as expected
## 🐛 Bug

--benchmark flag not working correctly.  Results in error: `AssertionError: Must call reset() before step()`

## To Reproduce

Steps to reproduce the behavior:

Pass in a `--benchmark` and then run any action

```
(compiler_gym) bwasti@devfair0187:~/compiler_gym$ COMPILER_GYM_DEBUG=3 python -m compiler_gym.bin.manual_env --local_service_binary CompilerGym/examples/example_compiler_gym_service/service_py/example_service.py --benchmark bar
I0322 12:18:25.167602 140345274726208 example_service.py:215] Starting service on 42673 with working dir /dev/shm/compiler_gym/service-2021-03-22T12:18:24.112461-60ea
I0322 12:18:25.423372 140342182409984 example_service.py:146] GetSpaces()
Initialized environment in 1.314s
I0322 12:18:25.425686 140342182409984 example_service.py:157] GetBenchmarks()
I0322 12:18:25.426420 140342182409984 example_service.py:157] GetBenchmarks()
Welcome to the CompilerGym Shell!
---------------------------------
Type help or ? for more information.
The 'tutorial' command will give a step by step guide.

compilergym:bar> action -
Action b
Traceback (most recent call last):
  File "/private/home/bwasti/.conda/envs/compiler_gym/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/private/home/bwasti/.conda/envs/compiler_gym/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/private/home/bwasti/.conda/envs/compiler_gym/lib/python3.8/site-packages/compiler_gym-0.1.3-py3.8.egg/compiler_gym/bin/manual_env.py", line 909, in <module>
    main(sys.argv)
  File "/private/home/bwasti/.conda/envs/compiler_gym/lib/python3.8/site-packages/compiler_gym-0.1.3-py3.8.egg/compiler_gym/bin/manual_env.py", line 904, in main
    shell.cmdloop()
  File "/private/home/bwasti/.conda/envs/compiler_gym/lib/python3.8/cmd.py", line 138, in cmdloop
    stop = self.onecmd(line)
  File "/private/home/bwasti/.conda/envs/compiler_gym/lib/python3.8/cmd.py", line 217, in onecmd
    return func(arg)
  File "/private/home/bwasti/.conda/envs/compiler_gym/lib/python3.8/site-packages/compiler_gym-0.1.3-py3.8.egg/compiler_gym/bin/manual_env.py", line 495, in do_action
    observation, reward, done, info = self.env.step(index)
  File "/private/home/bwasti/.conda/envs/compiler_gym/lib/python3.8/site-packages/compiler_gym-0.1.3-py3.8.egg/compiler_gym/envs/compiler_env.py", line 761, in step
    assert self.in_episode, "Must call reset() before step()"
AssertionError: Must call reset() before step()
```



## Expected behavior

Same behavior as `set_benchmark`:

```
(compiler_gym) bwasti@devfair0187:~/compiler_gym$ COMPILER_GYM_DEBUG=3 python -m compiler_gym.bin.manual_env --local_service_binary CompilerGym/examples/example_compiler_gym_service/service_py/example_service.py
I0322 12:17:23.826539 140646749136704 example_service.py:215] Starting service on 41035 with working dir /dev/shm/compiler_gym/service-2021-03-22T12:17:22.682603-4f26
I0322 12:17:23.992188 140643586266880 example_service.py:146] GetSpaces()
Initialized environment in 1.311s
I0322 12:17:23.994066 140643586266880 example_service.py:157] GetBenchmarks()
I0322 12:17:23.994649 140643586266880 example_service.py:157] GetBenchmarks()
Welcome to the CompilerGym Shell!
---------------------------------
Type help or ? for more information.
The 'tutorial' command will give a step by step guide.

compilergym:NO-BENCHMARK> list_benchmarks
bar, foo

compilergym:NO-BENCHMARK> set_benchmark bar
I0322 12:17:38.067934 140643586266880 example_service.py:164] StartSession(benchmark=bar)
Reset bar environment in 1.4ms

compilergym:bar> action -
Action c
I0322 12:17:40.889100 140643586266880 example_service.py:194] Step()
I0322 12:17:40.889228 140643586266880 example_service.py:101] Apply action 2
Actions c in 4.1ms with reward 0.

compilergym:bar>
```

## Environment

Please fill in this checklist:

- CompilerGym: 2623458424faa553df43e63f7b00bf5e45aade3f (Mon Mar 22 11:13:18 2021)
- How you installed CompilerGym (conda, pip, source): source
- OS: linux
- Python version: Python 3.8.8
- Build command you used (if compiling from source): make install; python setup.py install
- GCC/clang version (if compiling from source): gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
- Bazel version (if compiling from source): not sure

--benchmark flag not working as expected
## 🐛 Bug

--benchmark flag not working correctly.  Results in error: `AssertionError: Must call reset() before step()`

## To Reproduce

Steps to reproduce the behavior:

Pass in a `--benchmark` and then run any action

```
(compiler_gym) bwasti@devfair0187:~/compiler_gym$ COMPILER_GYM_DEBUG=3 python -m compiler_gym.bin.manual_env --local_service_binary CompilerGym/examples/example_compiler_gym_service/service_py/example_service.py --benchmark bar
I0322 12:18:25.167602 140345274726208 example_service.py:215] Starting service on 42673 with working dir /dev/shm/compiler_gym/service-2021-03-22T12:18:24.112461-60ea
I0322 12:18:25.423372 140342182409984 example_service.py:146] GetSpaces()
Initialized environment in 1.314s
I0322 12:18:25.425686 140342182409984 example_service.py:157] GetBenchmarks()
I0322 12:18:25.426420 140342182409984 example_service.py:157] GetBenchmarks()
Welcome to the CompilerGym Shell!
---------------------------------
Type help or ? for more information.
The 'tutorial' command will give a step by step guide.

compilergym:bar> action -
Action b
Traceback (most recent call last):
  File "/private/home/bwasti/.conda/envs/compiler_gym/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/private/home/bwasti/.conda/envs/compiler_gym/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/private/home/bwasti/.conda/envs/compiler_gym/lib/python3.8/site-packages/compiler_gym-0.1.3-py3.8.egg/compiler_gym/bin/manual_env.py", line 909, in <module>
    main(sys.argv)
  File "/private/home/bwasti/.conda/envs/compiler_gym/lib/python3.8/site-packages/compiler_gym-0.1.3-py3.8.egg/compiler_gym/bin/manual_env.py", line 904, in main
    shell.cmdloop()
  File "/private/home/bwasti/.conda/envs/compiler_gym/lib/python3.8/cmd.py", line 138, in cmdloop
    stop = self.onecmd(line)
  File "/private/home/bwasti/.conda/envs/compiler_gym/lib/python3.8/cmd.py", line 217, in onecmd
    return func(arg)
  File "/private/home/bwasti/.conda/envs/compiler_gym/lib/python3.8/site-packages/compiler_gym-0.1.3-py3.8.egg/compiler_gym/bin/manual_env.py", line 495, in do_action
    observation, reward, done, info = self.env.step(index)
  File "/private/home/bwasti/.conda/envs/compiler_gym/lib/python3.8/site-packages/compiler_gym-0.1.3-py3.8.egg/compiler_gym/envs/compiler_env.py", line 761, in step
    assert self.in_episode, "Must call reset() before step()"
AssertionError: Must call reset() before step()
```



## Expected behavior

Same behavior as `set_benchmark`:

```
(compiler_gym) bwasti@devfair0187:~/compiler_gym$ COMPILER_GYM_DEBUG=3 python -m compiler_gym.bin.manual_env --local_service_binary CompilerGym/examples/example_compiler_gym_service/service_py/example_service.py
I0322 12:17:23.826539 140646749136704 example_service.py:215] Starting service on 41035 with working dir /dev/shm/compiler_gym/service-2021-03-22T12:17:22.682603-4f26
I0322 12:17:23.992188 140643586266880 example_service.py:146] GetSpaces()
Initialized environment in 1.311s
I0322 12:17:23.994066 140643586266880 example_service.py:157] GetBenchmarks()
I0322 12:17:23.994649 140643586266880 example_service.py:157] GetBenchmarks()
Welcome to the CompilerGym Shell!
---------------------------------
Type help or ? for more information.
The 'tutorial' command will give a step by step guide.

compilergym:NO-BENCHMARK> list_benchmarks
bar, foo

compilergym:NO-BENCHMARK> set_benchmark bar
I0322 12:17:38.067934 140643586266880 example_service.py:164] StartSession(benchmark=bar)
Reset bar environment in 1.4ms

compilergym:bar> action -
Action c
I0322 12:17:40.889100 140643586266880 example_service.py:194] Step()
I0322 12:17:40.889228 140643586266880 example_service.py:101] Apply action 2
Actions c in 4.1ms with reward 0.

compilergym:bar>
```

## Environment

Please fill in this checklist:

- CompilerGym: 2623458424faa553df43e63f7b00bf5e45aade3f (Mon Mar 22 11:13:18 2021)
- How you installed CompilerGym (conda, pip, source): source
- OS: linux
- Python version: Python 3.8.8
- Build command you used (if compiling from source): make install; python setup.py install
- GCC/clang version (if compiling from source): gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
- Bazel version (if compiling from source): not sure



Discussion:

Resolving patch:
diff --git a/compiler_gym/util/flags/env_from_flags.py b/compiler_gym/util/flags/env_from_flags.py
--- a/compiler_gym/util/flags/env_from_flags.py
+++ b/compiler_gym/util/flags/env_from_flags.py
@@ -124,13 +124,15 @@ def env_from_flags(benchmark: Optional[Union[str, Benchmark]] = None) -> Compile
         local_service_binary = (
             Path(FLAGS.local_service_binary) if FLAGS.local_service_binary else None
         )
-        return CompilerEnv(
+        env = CompilerEnv(
             service=local_service_binary or FLAGS.service,
             connection_settings=connection_settings,
             benchmark=benchmark,
             observation_space=FLAGS.observation,
             reward_space=FLAGS.reward,
         )
+        env.reset()
+        return env
     else:
         raise app.UsageError("Neither --env or --local_service_binary is set")
 


Test patch: