Repository: hylang/hy

Issue:
Metaclasses under Python 3
When pondering #850, it occurred to me that I don't know how to set a metaclass in Hy. (It doesn't come up much.) I mean in Python2 we can do it with the old syntax (`__metaclass__ = Meta` in the class body), which is straightforward in Hy. But this syntax isn't allowed in Python3. After reading the docs, I don't think Hy can do this. That or the docs need some updating. It's an issue either way.



Discussion:
Good observation; I agree that this is currently impossible. With the Python 3 metaclass-definition syntax, the metaclass keyword argument ends up in the `keywords` attribute of `ast.ClassDef`, which is [unconditionally empty](https://github.com/hylang/hy/blob/d95bfbaa/hy/compiler.py#L2209) in the classdefs we produce.

```
in [2]: class the_trivial_metaclass(type): pass
in [3]: ast.parse("class Foo(metaclass=the_trivial_metaclass): pass").body[0].keywords[0].value.id
Out[3]: 'the_trivial_metaclass'
```

I guess the most obvious syntax would be.

```
(defclass Foo [Parent :metaclass Meta :spam "spam"]
...)
```

How about this:

``` hy
(defclass Foo [Parent1 Parent2 [:meta MyMeta]])
  ...
```

The reason for the extra brackets is that this makes it stand out a bit more, which makes it easier (for the human eye) to notice, especially when one adds more parent classes after:

``` hy
(defclass Foo [P1 [:meta Meta] P2]
  ...)
```

vs

``` hy
(defclass Foo [P1 :meta Meta P2]
  ...)
```

Negligible difference, mind you, but I believe bracket-grouping these kinds of things is a useful thing. Also, this way I can just iterate over the members of the parent list, and if I find a list, I'll handle that specially. If we had it inlined, I'd have to check each element and sometimes pop one off, sometimes two. With a bracket, we pop one off every time, and maybe treat it specially. Again, not much of a difference, but still.

Since this is not likely to be used all that much, the extra brackets shouldn't be much of an issue to type.

No, you can't put the args after the kwargs in a Python class definition, because it uses the same grammar rule as a function call. Why would we want to allow this in Hy? This would get even more confusing when you use more keywords. Let's just be consistent with Hy's function call syntax and do it the same way.

Maybe I should explain myself better. Metaclasses are really not that hard, but they're perhaps an arcane topic.

In Python, a class definition is just syntactic sugar for a function call:

``` Python
X = type('X', (object,), dict(a=1))
# same as
class X(object):
    a = 1
```

Like everything else in Python, classes are themselves objects, and they are returned by calls to `type()` (the 3-arg version; 1-arg just names the class).

Metaclasses just let you change which function you're calling. Here we've changed the default metaclass--from `type`, to `my_meta`:

``` Python
>>> def my_meta(name,bases,namespace,**kwargs):
    print("in my_meta!")
    print(name,bases,namespace,kwargs)
    return "Not even a class!"

>>> class foo(metaclass=my_meta):pass

in my_meta!
foo () {'__module__': '__main__', '__qualname__': 'foo'} {}
>>> foo
'Not even a class!'
>>> class foo(str, int, spam="spam", metaclass=my_meta):pass

in my_meta!
foo (<class 'str'>, <class 'int'>) {'__module__': '__main__', '__qualname__': 'foo'} {'spam': 'spam'}
>>> # mixing order not allowed!
>>> class foo(str, metaclass=my_meta, int):pass
SyntaxError: non-keyword arg after keyword arg
```

See, just a function call. There is one more trick with the `__prepare__` attribute:

``` Python
>>> my_meta.__prepare__ = my_prepare_func
```

This lets you call an additional function to pre-populate the class attributes before the class body statements are executed. This function also gets the same kwargs that the metaclass gets.

Hrm. I brooded over this last night, and your reasoning makes sense. I'll prep a PR with a `(defclass Foo [a b :meta m] ...)`  syntax.

> No, you can't put the args after the kwargs in a Python class definition, because it uses the same grammar rule as a function call. Why would we want to allow this in Hy?

To answer my own question, it appears that Hy actually _does_ allow positional arguments after keyword arguments.

``` Hy
=> (defn test [arg &kwargs kwargs])
def test(arg, **kwargs):
    pass
=> (test :foo "bar" "before")
test('before', foo='bar')
```

Even though this is expressly forbidden in Python:

``` Python
>>> def test(arg, **kwargs):
    pass

>>> test(foo="bar", "before")
SyntaxError: non-keyword arg after keyword arg
>>> 
```

I wonder if this was intentional. This strange behavior is potentially useful given the tail-threading macro `->>`, but also potentially confusing.

The helper function responsible for handling kwargs in function calls begins on line 445 in compiler.py. This might be a helpful reference for kwarg support in defclass.


Resolving patch:
diff --git a/hy/compiler.py b/hy/compiler.py
--- a/hy/compiler.py
+++ b/hy/compiler.py
@@ -2057,11 +2057,12 @@ def rewire_init(expr):
 
         bases_expr = []
         bases = Result()
+        keywords = []
         if expressions:
             base_list = expressions.pop(0)
             if not isinstance(base_list, HyList):
                 raise HyTypeError(base_list, "Base classes must be a list.")
-            bases_expr, bases, _ = self._compile_collect(base_list)
+            bases_expr, bases, keywords = self._compile_collect(base_list, with_kwargs=PY3)
 
         body = Result()
 
@@ -2092,7 +2093,7 @@ def rewire_init(expr):
             expressions,
             decorator_list=[],
             name=ast_str(class_name),
-            keywords=[],
+            keywords=keywords,
             starargs=None,
             kwargs=None,
             bases=bases_expr,


Test patch:
diff --git a/tests/compilers/test_ast.py b/tests/compilers/test_ast.py
--- a/tests/compilers/test_ast.py
+++ b/tests/compilers/test_ast.py
@@ -214,6 +214,13 @@ def test_ast_good_defclass():
     can_compile("(defclass a [])")
 
 
+@pytest.mark.skipif(not PY3, reason="Python 3 supports class keywords")
+def test_ast_good_defclass_with_metaclass():
+    "Make sure AST can compile valid defclass with keywords"
+    can_compile("(defclass a [:metaclass b])")
+    can_compile("(defclass a [:b c])")
+
+
 def test_ast_bad_defclass():
     "Make sure AST can't compile invalid defclass"
     cant_compile("(defclass)")
diff --git a/tests/native_tests/py36_only_tests.hy b/tests/native_tests/py36_only_tests.hy
--- a/tests/native_tests/py36_only_tests.hy
+++ b/tests/native_tests/py36_only_tests.hy
@@ -37,3 +37,11 @@
         (setv x (+ x a))
         (else (setv x (+ x 50))))
       (assert (= x 53)))))
+
+(defn test-pep-487 []
+  (defclass QuestBase []
+    [--init-subclass-- (fn [cls swallow &kwargs kwargs]
+                         (setv cls.swallow swallow))])
+
+  (defclass Quest [QuestBase :swallow "african"])
+  (assert (= (. (Quest) swallow) "african")))
diff --git a/tests/native_tests/py3_only_tests.hy b/tests/native_tests/py3_only_tests.hy
--- a/tests/native_tests/py3_only_tests.hy
+++ b/tests/native_tests/py3_only_tests.hy
@@ -84,3 +84,26 @@
        (assert (= (foo :b 20 :a 10 :c 30)
                   (, 10 20 30)))))
 
+(defn test-pep-3115 []
+  (defclass member-table [dict]
+    [--init-- (fn [self] (setv self.member-names []))
+
+     --setitem-- (fn [self key value]
+                   (if (not-in key self)
+                       (.append self.member-names key))
+                   (dict.--setitem-- self key value))])
+
+  (defclass OrderedClass [type]
+    [--prepare-- (classmethod (fn [metacls name bases] (member-table)))
+
+     --new-- (fn [cls name bases classdict]
+               (setv result (type.--new-- cls name bases (dict classdict)))
+               (setv result.member-names classdict.member-names)
+               result)])
+
+  (defclass MyClass [:metaclass OrderedClass]
+    [method1 (fn [self] (pass))
+     method2 (fn [self] (pass))])
+
+  (assert (= (. (MyClass) member-names)
+             ["__module__" "__qualname__" "method1" "method2"])))
diff --git a/tests/test_bin.py b/tests/test_bin.py
--- a/tests/test_bin.py
+++ b/tests/test_bin.py
@@ -12,7 +12,7 @@
 
 import pytest
 
-from hy._compat import PY3, PY35, builtins
+from hy._compat import PY3, PY35, PY36, builtins
 from hy.importer import get_bytecode_path
 
 
@@ -229,9 +229,11 @@ def test_hy2py():
     for dirpath, dirnames, filenames in os.walk("tests/native_tests"):
         for f in filenames:
             if f.endswith(".hy"):
-                if f == "py3_only_tests.hy" and not PY3:
+                if "py3_only" in f and not PY3:
                     continue
-                if f == "py35_only_tests.hy" and not PY35:
+                if "py35_only" in f and not PY35:
+                    continue
+                if "py36_only" in f and not PY36:
                     continue
                 i += 1
                 output, err = run_cmd("hy2py -s -a " + quote(os.path.join(dirpath, f)))