Repository: hylang/hy

Issue:
A gotcha in keyword mangling
I was a bit blindsided today by a call to `requests.get` which looked like `(requests.get URL (dict :param1 1 :param2 2 :from 3))`. The results of the call were subtly wrong. It took me some digging to realize that instead of putting `&from=3` in my URL, I was putting `&hyx_from=3`, because `from` is a Python reserved word and hence needs to be mangled. In Python, `dict(from = 3)` is simply a syntax error, which is annoying, but doesn't lead to subtle bugs like the one I got here.

I can imagine a few different responses to this:

1. Live with it. Besides, I should've checked the produced URL carefully, because Web services tend not to produce an error for an unrecognized parameter, so any typo in a parameter name would've caused this sort of problem.
2. Don't use mangling to make Python reserved words legal identifiers. Instead, forbid such variable names, as in Python itself. It's cute that you can use `def` as an ordinary variable name and still produce valid Python code, but maybe it's not worth headaches like this.
3. Raise a warning in the specific case of using a reserved word as a keyword argument, because this is unlikely to do what the programmer is expecting.
4. As (3), but raise an error instead.

What do you guys think?


Discussion:
Personally I like option 2 the best; this isn't really important enough to have special cases for.
We should beware that option 2 will put us at the mercy of any flaws in our ability to detect illegal assignments (e.g., #1780).
@Kodiologist now that you've fixed #1780 in #1983, I think we should look into this again. I'm also for option #2. Being able to shadow python reserved words is a weird inconsistency with Python and leads to weird behavior like this. changing `_nonconst` to reference python's builtin `keyword.kwlist` instead of `("None", "True", "False")` seems to be a good start, but causes issues in `core.shadow` (special case this module maybe?)
I see your point. It's not a priority for me right now, but I would be open to such a change.
Removing mangling on python reserved words breaks several names in `hy.core.shadow`, but given that the plan is to
> Rename hy.core.shadow to something more logical like hy.pyops and encourage users to star-import it when they want first-class Python operators, as in (import [hy.pyops [*]]).

maybe it's time to rename those operators to be more in line with python's `operator` module? Python adds an underscore at the end of methods shadowing reserved words so the operator equivalent of `and` becomes `operator.and_`. What do we think about this? I think we should add a `-` to reserved words then move `hy.core.shadow` to `hy.operators` so they can be accessed easily at any time akin to `hy.models`
I'd rather be able to use the same name for `not` whether I'm using the special operator or the shadow function. I suggested `pyops` instead of `operators` to reduce confusion with the Lisp sense of "operator".
So, clearly I didn't think option (2) through. I thought an exception would be needed for *defining* the (purely alphabetic) shadow functions at most, but in reality, you'd need an exception for every use of them.
yeah, to make an exception just for `pyops` would be pretty complex as far as i can tell. maybe this should be relabeled to `docme` as a known quirk of the language, unless someone can come up with an elegant solution. 
We can also just not do anything at all to Python reserved keywords. If we really want to we can raise a warning (or error) if we detect a reserved word when producing `hy2py` output, but otherwise reserved keywords are valid Python names at the AST level:

```python
>>> print(ast.dump(and_symbol, indent=2))
Module(
  body=[
    Assign(
      targets=[
        Name(id='and', ctx=Store())],
      value=Constant(value=3)),
    Expr(
      value=Call(
        func=Name(id='print', ctx=Load()),
        args=[
          Name(id='and', ctx=Load())],
        keywords=[]))],
  type_ignores=[])
>>> exec(compile(and_symbol, 'and_symbol', 'exec'))
3
>>> globals()['and']
3
```
Of note: Python itself doesn't check for reserved words when unparsing:
```python
>>> print(ast.unparse(and_symbol))
and = 3
print(and)
>>> exec(ast.unparse(and_symbol))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<string>", line 1
    and = 3
    ^
SyntaxError: invalid syntax
```
I've gone to some lengths to ensure that Hy will only produce AST that can be rendered as syntactically valid Python code. Let's not break that.
Yeah, that's a fair point. I'm coming around to option (4) in the first post.

We can check both call sites and `lambda_list`s for python reserved words at compile time, but otherwise let symbols use (mangled) reserved words so that `pyops` still work as expected.
I think thats probably the best solution. I'll give it shot now that #2077 has been merged and see how it feels/how intrusive it is

Resolving patch:
diff --git a/hy/compiler.py b/hy/compiler.py
--- a/hy/compiler.py
+++ b/hy/compiler.py
@@ -30,6 +30,7 @@
 import copy
 import builtins
 import __future__
+import keyword
 
 from collections import defaultdict
 from functools import reduce
@@ -411,6 +412,11 @@ def _compile_collect(self, exprs, with_kwargs=False, dict_display=False):
                         expr, arg=None, value=ret.force_expr))
 
             elif with_kwargs and isinstance(expr, Keyword):
+                if keyword.iskeyword(expr.name):
+                    raise self._syntax_error(
+                        expr, "keyword argument cannot be Python reserved word"
+                    )
+
                 try:
                     value = next(exprs_iter)
                 except StopIteration:
@@ -1528,6 +1534,26 @@ def compile_macro_def(self, expr, root, name, params, body):
     def _compile_lambda_list(self, params, ret):
         posonly_parms, args_parms, rest_parms, kwonly_parms, kwargs_parms = params
 
+        py_reserved_param = next(
+            (
+                sym
+                for param in (
+                    (posonly_parms or [])
+                    + args_parms
+                    + kwonly_parms
+                    + [rest_parms, kwargs_parms]
+                )
+                if isinstance(param, tuple)
+                for sym in [param[1][0] if isinstance(param[1], List) else param[1]]
+                if keyword.iskeyword(str(sym))
+            ),
+            None,
+        )
+        if py_reserved_param:
+            raise self._syntax_error(
+                py_reserved_param, "parameter name cannot be Python reserved word"
+            )
+
         if not (posonly_parms or posonly_parms is None):
             raise self._syntax_error(
                 params, "positional only delimiter '/' must have an argument"


Test patch:
diff --git a/tests/native_tests/language.hy b/tests/native_tests/language.hy
--- a/tests/native_tests/language.hy
+++ b/tests/native_tests/language.hy
@@ -139,7 +139,6 @@
       (setv True 1)
       (defn None [] (print "hello"))
       (defn True [] (print "hello"))
-      (defn f [True] (print "hello"))
       (for [True [1 2 3]] (print "hello"))
       (lfor  True [1 2 3]  True)
       (lfor  :setv True 1  True)
@@ -148,7 +147,14 @@
       (defclass True [])]]
     (with [e (pytest.raises HyLanguageError)]
       (hy.eval form))
-    (assert (in "Can't assign" e.value.msg))))
+    (assert (in "Can't assign" e.value.msg)))
+
+  (for [form '[(defn f [True] (print "hello"))
+               (defn f [a b break])
+               (defn f [a [from 1]])]])
+  (with [e (pytest.raises HySyntaxError)]
+    (hy.eval form))
+  (assert (in "parameter name cannot be" e.value.msg)))
 
 
 (defn test-no-str-as-sym []