Repository: facebookresearch/hydra

Issue:
[Bug] Argument parsing fails for multirun with `range` if `PYTHONOPTIMIZE=1`
# 🐛 Bug
## Description
Hydra will fail to parse a `range` statement in multi-run mode if `PYTHONOPTIMIZE=1`.

## Checklist
- [x] I checked on the latest version of Hydra
- [x] I created a minimal repro (See [this](https://stackoverflow.com/help/minimal-reproducible-example) for tips).

## To reproduce

Create a simple hydra app like this:
```python
# run.py
from dataclasses import dataclass

import hydra
from hydra.core.config_store import ConfigStore

@dataclass
class MySQLConfig:
    host: str = "localhost"
    port: int = 3306

cs = ConfigStore.instance()
cs.store(name="config", node=MySQLConfig)

@hydra.main(version_base=None, config_name="config")
def my_app(cfg: MySQLConfig) -> None:
    if cfg.port == 80:
        print("Is this a webserver?!")

if __name__ == "__main__":
    my_app()
```

Now run `PYTHONOPTIMIZE=1 python run.py -m port=80,81`.
Hydra should handle this correctly, outputting something like:
```
[2023-02-02 10:05:07,993][HYDRA] Launching 2 jobs locally
[2023-02-02 10:05:07,993][HYDRA]        #0 : port=80
Is this a webserver?!
[2023-02-02 10:05:08,294][HYDRA]        #1 : port=81
```

If we instead run `PYTHONOPTIMIZE=1 python run.py -m port="range(80,82)"`, which should be equivalent, we get an error:
```
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/hydra/_internal/utils.py", line 219, in run_and_report
    return func()
  File "/usr/local/lib/python3.8/dist-packages/hydra/_internal/utils.py", line 466, in <lambda>
    lambda: hydra.multirun(
  File "/usr/local/lib/python3.8/dist-packages/hydra/_internal/hydra.py", line 143, in multirun
    cfg = self.compose_config(
  File "/usr/local/lib/python3.8/dist-packages/hydra/_internal/hydra.py", line 594, in compose_config
    cfg = self.config_loader.load_configuration(
  File "/usr/local/lib/python3.8/dist-packages/hydra/_internal/config_loader_impl.py", line 142, in load_configuration
    return self._load_configuration_impl(
  File "/usr/local/lib/python3.8/dist-packages/hydra/_internal/config_loader_impl.py", line 244, in _load_configuration_impl
    parsed_overrides, caching_repo = self._parse_overrides_and_create_caching_repo(
  File "/usr/local/lib/python3.8/dist-packages/hydra/_internal/config_loader_impl.py", line 228, in _parse_overrides_and_create_caching_repo
    parsed_overrides = parser.parse_overrides(overrides=overrides)
  File "/usr/local/lib/python3.8/dist-packages/hydra/core/override_parser/overrides_parser.py", line 82, in parse_overrides
    parsed = self.parse_rule(override, "override")
  File "/usr/local/lib/python3.8/dist-packages/hydra/core/override_parser/overrides_parser.py", line 67, in parse_rule
    ret = visitor.visit(tree)
  File "/usr/local/lib/python3.8/dist-packages/antlr4/tree/Tree.py", line 34, in visit
    return tree.accept(self)
  File "/usr/local/lib/python3.8/dist-packages/hydra/grammar/gen/OverrideParser.py", line 195, in accept
    return visitor.visitOverride(self)
  File "/usr/local/lib/python3.8/dist-packages/hydra/core/override_parser/overrides_visitor.py", line 174, in visitOverride
    value = self.visitValue(ctx.value())  # type: ignore[no-untyped-call]
  File "/usr/local/lib/python3.8/dist-packages/hydra/core/override_parser/overrides_visitor.py", line 132, in visitValue
    return self.visitElement(ctx.element())  # type: ignore[no-untyped-call]
  File "/usr/local/lib/python3.8/dist-packages/hydra/core/override_parser/overrides_visitor.py", line 118, in visitElement
    return self.visitFunction(ctx.function())  # type: ignore
  File "/usr/local/lib/python3.8/dist-packages/hydra/core/override_parser/overrides_visitor.py", line 233, in visitFunction
    value = self.visitElement(cur)
  File "/usr/local/lib/python3.8/dist-packages/hydra/core/override_parser/overrides_visitor.py", line 117, in visitElement
    if ctx.function():  # type: ignore[no-untyped-call]
AttributeError: 'TerminalNodeImpl' object has no attribute 'function'
```


## Expected Behavior
Expected the output of  `PYTHONOPTIMIZE=1 python run.py -m port="range(80,82)"` to be identical to the output of `PYTHONOPTIMIZE=1 python run.py -m port=80,81`.

## System information
- **Hydra Version** :  1.3.1
- **Python version** :  3.8.10
- **Virtual environment type and version** : System install inside docker
- **Operating system** :  Ubuntu 20.04 (docker) / Ubuntu 22.04 (host)



Discussion:
Thanks for the report, @EspenHa. I'm able to reproduce the error.
The traceback looks like it's related to antlr (which Hydra uses for parsing CLI overrides).

Resolving patch:
diff --git a/hydra/core/override_parser/overrides_visitor.py b/hydra/core/override_parser/overrides_visitor.py
--- a/hydra/core/override_parser/overrides_visitor.py
+++ b/hydra/core/override_parser/overrides_visitor.py
@@ -107,7 +107,8 @@ def visitDictKeyValuePair(
         item = next(children)
         assert isinstance(item, OverrideParser.DictKeyContext)
         pkey = self.visitDictKey(item)
-        assert self.is_matching_terminal(next(children), OverrideLexer.COLON)
+        colon = next(children)
+        assert self.is_matching_terminal(colon, OverrideLexer.COLON)
         value = next(children)
         assert isinstance(value, OverrideParser.ElementContext)
         return pkey, self.visitElement(value)
@@ -212,7 +213,8 @@ def visitFunction(self, ctx: OverrideParser.FunctionContext) -> Any:
         kwargs = {}
         children = ctx.getChildren()
         func_name = next(children).getText()
-        assert self.is_matching_terminal(next(children), OverrideLexer.POPEN)
+        popen = next(children)
+        assert self.is_matching_terminal(popen, OverrideLexer.POPEN)
         in_kwargs = False
         while True:
             cur = next(children)


Test patch: