Repository: hylang/hy

Issue:
Triple-quoted docstrings trigger internal compiler bug
If I accidentally use triple quotes in my docstrings (say, when playing with the Hy tutorial and assuming too much similarity to Python class docstrings), I trigger an internal compiler bug (`IndexError: string index out of range`), rather than a more readable Hy syntax error.

I first saw this in v0.14 and am able to replicate it in the bleeding-edge commit b023ebd0b5340a666df83e3fc0b72d7734636e04.

(Maybe the discussion in #1527 is relevant.)


Discussion:
I am unable to verify. Can you provide a minimal example? Does this appear in the repl, or just .hy files?

We've noticed some issues with stale .pyc files after upgrades. Can you reproduce in a bleeding-edge install in a fresh virtual environment?
No wait, I see it. It's only class docstrings, not function docstrings.
Minimal example:
```
(defclass Foo [] "doc" "")
```
That's not supposed to fail.
I don't think that has anything to do with docstrings though, this fails the same way:

```clojure
(defclass Foo [] "doc" 42)
```

And then there's some other weirdness in how classes compile:

```
$ hy - <<<'(defclass Foo [] "test" test)'
class Foo:
    __doc__ = 'test'

```

Where did `test` go?
This might be the root of the problem:

```
=> """test"""
""
```
>     => """test"""
>     ""

I don't know if you meant to imply otherwise, but just to be clear, this example is correct. The input is parsed as `"" "test" ""` and the REPL prints only the final value.
Oh that makes sense. I expected it to fail (or work like Python), but I guess there's no requirement for white space between expressions.
It seems like the issue is with atoms in the body of `defclass` expressions, and has nothing to do with triple quotes as I had initially thought, as @vodik insightfully noted.

For example, if I just skip the `macroexpand` as in the [attached patch](https://github.com/hylang/hy/files/1813350/defclass_atom_body.patch.txt) (which probably breaks the compiler logic, so isn't a fix!), then the atoms don't choke the `defclass` expansion.

The offending `defclass`es with atoms in their bodies, like `(defclass Foo [] "doc" 42)`,  then compile just fine.

I just started looking at Hy today, so I don't yet have an intuition for what might be breaking — perhaps the iterated `macroexpand_1`?
@saketh I think I know. The error gives a bit more information:

    TypeError: 'HyInteger' object does not support indexing

And its tripping up here:

    if expr[0] == HySymbol("setv"):

In https://github.com/hylang/hy/blob/b023ebd0b5340a666df83e3fc0b72d7734636e04/hy/compiler.py#L2033-L2048

Which is probably called from here lower in the `defclass` compiler:

https://github.com/hylang/hy/blob/b023ebd0b5340a666df83e3fc0b72d7734636e04/hy/compiler.py#L2085-L2086

`macroexpand` is probably a red herring as it should have no effect in this case:

```clojure
=> (macroexpand "test")
"test"
=> (macroexpand 42)
42
```

The problem is likely that we expect expressions inside classes to be all `HyExpressions`, but don't actually check for it. Note we dig right into `expr[0]` without first checking of `isinstance(expr, HyExpression) and len(expr) >= 1` as you'll see it lots of other places in the code base.

Resolving patch:
diff --git a/hy/compiler.py b/hy/compiler.py
--- a/hy/compiler.py
+++ b/hy/compiler.py
@@ -20,10 +20,8 @@
 
 import traceback
 import importlib
-import codecs
 import ast
 import sys
-import keyword
 import copy
 
 from collections import defaultdict
@@ -2038,7 +2036,10 @@ def compile_return(self, expr):
     def compile_class_expression(self, expressions):
         def rewire_init(expr):
             new_args = []
-            if expr[0] == HySymbol("setv"):
+            if (isinstance(expr, HyExpression)
+                and len(expr) > 1
+                and isinstance(expr[0], HySymbol)
+                and expr[0] == HySymbol("setv")):
                 pairs = expr[1:]
                 while len(pairs) > 0:
                     k, v = (pairs.pop(0), pairs.pop(0))
@@ -2070,13 +2071,12 @@ def rewire_init(expr):
         body = Result()
 
         # grab the doc string, if there is one
+        docstring = None
         if expressions and isinstance(expressions[0], HyString):
             docstring = expressions.pop(0)
-            symb = HySymbol("__doc__")
-            symb.start_line = docstring.start_line
-            symb.start_column = docstring.start_column
-            body += self._compile_assign(symb, docstring)
-            body += body.expr_as_stmt()
+            if not PY37:
+                body += self.compile(docstring).expr_as_stmt()
+                docstring = None
 
         if expressions and isinstance(expressions[0], HyList) \
            and not isinstance(expressions[0], HyExpression):
@@ -2087,7 +2087,8 @@ def rewire_init(expr):
             body += self.compile(rewire_init(expr))
 
         for expression in expressions:
-            body += self.compile(rewire_init(macroexpand(expression, self)))
+            e = self.compile(rewire_init(macroexpand(expression, self)))
+            body += e + e.expr_as_stmt()
 
         if not body.stmts:
             body += asty.Pass(expressions)
@@ -2100,7 +2101,8 @@ def rewire_init(expr):
             starargs=None,
             kwargs=None,
             bases=bases_expr,
-            body=body.stmts)
+            body=body.stmts,
+            docstring=(None if docstring is None else str_type(docstring)))
 
     @builds("dispatch-tag-macro")
     @checkargs(exact=2)


Test patch:
diff --git a/tests/compilers/test_ast.py b/tests/compilers/test_ast.py
--- a/tests/compilers/test_ast.py
+++ b/tests/compilers/test_ast.py
@@ -212,6 +212,9 @@ def test_ast_good_defclass():
     "Make sure AST can compile valid defclass"
     can_compile("(defclass a)")
     can_compile("(defclass a [])")
+    can_compile("(defclass a [] None 42)")
+    can_compile("(defclass a [] None \"test\")")
+    can_compile("(defclass a [] None (print \"foo\"))")
 
 
 @pytest.mark.skipif(not PY3, reason="Python 3 supports class keywords")
diff --git a/tests/native_tests/defclass.hy b/tests/native_tests/defclass.hy
--- a/tests/native_tests/defclass.hy
+++ b/tests/native_tests/defclass.hy
@@ -127,3 +127,14 @@
   (setv b (B))
   (assert (= a.x 1))
   (assert (= b.x 2)))
+
+(defn test-class-sideeffects []
+  "NATIVE: test that defclass runs all expressions"
+  (defn set-sentinel []
+    (setv set-sentinel.set True))
+  (setv set-sentinel.set False)
+
+  (defclass A []
+    (set-sentinel))
+
+  (assert set-sentinel.set))