Repository: hylang/hy

Issue:
Nested unquotes leave behind extra unquote forms
In light of #2251, consider

    (setv c '(f))
    ```[~a ~~b ~~~c]

This returns

    ``[~a ~~b ~~(f)]

but a more useful, and perhaps more logical, behavior would be

    ``[~a ~~b (f)]

It's hard to say exactly what should happen in these cases because we usually think of nested backticks with intervening unquotes. Nested backticks without an intervening unquote only come up when you're trying to return a backticked form for later expansion.


Discussion:
I took a survey of Lisps to see if there was a standard approach to this issue, and it looks like there isn't a great deal of consensus. Emacs and Scheme keep the extra unquotes like Hy does, whereas Common Lisp and Clojure get rid of them.

Emacs Lisp:

```
(let ((c 17))
  (nth 1 (nth 1 ```(,a ,,b ,,,c))))
=> ((\, a) (\, (\, b)) (\, (\, 17)))
```

Common Lisp:

```
(let ((c 17)) (princ
  ```(,a ,,b ,,,c)))
=> (LIST 'LIST 'A B 17)
```

Chicken Scheme:

```
(let ((c 17)) (print
  ```(,a ,,b ,,,c)))
=> (quasiquote (quasiquote ((unquote a) (unquote (unquote b)) (unquote (unquote 17)))))
```

Clojure:

<details>

```
(ns x (:require [clojure.pprint :as pp]))

(let [c 17] (pp/pprint
  ```[~a ~~b ~~~c 18]))
=> (clojure.core/seq
 (clojure.core/concat
  (clojure.core/list 'clojure.core/apply)
  (clojure.core/list 'clojure.core/vector)
  (clojure.core/list
   (clojure.core/seq
    (clojure.core/concat
     (clojure.core/list 'clojure.core/seq)
     (clojure.core/list
      (clojure.core/seq
       (clojure.core/concat
        (clojure.core/list 'clojure.core/concat)
        (clojure.core/list
         (clojure.core/seq
          (clojure.core/concat
           (clojure.core/list 'clojure.core/list)
           (clojure.core/list 'x/a))))
        (clojure.core/list
         (clojure.core/seq
          (clojure.core/concat
           (clojure.core/list 'clojure.core/list)
           (clojure.core/list x/b))))
        (clojure.core/list
         (clojure.core/seq
          (clojure.core/concat
           (clojure.core/list 'clojure.core/list)
           (clojure.core/list 17))))
        (clojure.core/list
         (clojure.core/seq
          (clojure.core/concat
           (clojure.core/list 'clojure.core/list)
           (clojure.core/list 18))))))))))))
```
</details>
I think that our code for rendering quoted forms (`render_quoted_form`) is not quite clever enough, because it doesn't give backticks sufficiently special treatment. The [Common Lisp HyperSpec](http://www.lispworks.com/documentation/lw70/CLHS/Body/02_df.htm) says

> If the backquote syntax is nested, the innermost backquoted form should be expanded first. This means that if several commas occur in a row, the leftmost one belongs to the innermost backquote.

and I suspect that's the best way to do it.

Here are some more comparisons of Common Lisp with Hy. Remember that `hy.repr` always shows one more quote level than Common Lisp's printing functions would, in order for the output to round-trip.

```
Common Lisp: ``,a    =>  A
Hy:          ``~a    =>  '`~a
Common Lisp: '`,a    =>  `,A
Hy:          '`~a    =>  '`~a
Common Lisp: ``,``a  =>  ''A
Hy:          ``~``a  =>  '`~``a
```

Resolving patch:
diff --git a/hy/core/result_macros.py b/hy/core/result_macros.py
--- a/hy/core/result_macros.py
+++ b/hy/core/result_macros.py
@@ -176,11 +176,9 @@ def compile_inline_python(compiler, expr, root, code):
 
 @pattern_macro(["quote", "quasiquote"], [FORM])
 def compile_quote(compiler, expr, root, arg):
-    level = Inf if root == "quote" else 0  # Only quasiquotes can unquote
-    stmts, _ = render_quoted_form(compiler, arg, level)
-    ret = compiler.compile(stmts)
-    return ret
-
+    return compiler.compile(render_quoted_form(compiler, arg,
+        level = Inf if root == "quote" else 0)[0])
+          # Only quasiquotes can unquote
 
 def render_quoted_form(compiler, form, level):
     """
@@ -198,21 +196,19 @@ def render_quoted_form(compiler, form, level):
 
     op = None
     if isinstance(form, Expression) and form and isinstance(form[0], Symbol):
-        op = unmangle(mangle(form[0]))
-    if level == 0 and op in ("unquote", "unquote-splice"):
-        if len(form) != 2:
-            raise HyTypeError(
-                "`%s' needs 1 argument, got %s" % op,
-                len(form) - 1,
-                compiler.filename,
-                form,
-                compiler.source,
-            )
-        return form[1], op == "unquote-splice"
-    elif op == "quasiquote":
-        level += 1
-    elif op in ("unquote", "unquote-splice"):
-        level -= 1
+        op = mangle(form[0]).replace('_', '-')
+        if op in ("unquote", "unquote-splice", "quasiquote"):
+            if level == 0 and op != "quasiquote":
+                if len(form) != 2:
+                    raise HyTypeError(
+                        "`%s' needs 1 argument, got %s" % op,
+                        len(form) - 1,
+                        compiler.filename,
+                        form,
+                        compiler.source,
+                    )
+                return form[1], op == "unquote-splice"
+            level += 1 if op == "quasiquote" else -1
 
     name = form.__class__.__name__
     body = [form]
diff --git a/hy/reader/mangling.py b/hy/reader/mangling.py
--- a/hy/reader/mangling.py
+++ b/hy/reader/mangling.py
@@ -12,7 +12,7 @@ def mangle(s):
     to :ref:`Hy's mangling rules <mangling>`. ::
 
         (hy.mangle 'foo-bar)   ; => "foo_bar"
-        (hy.mangle "🦑")       ; => "hyx_squid"
+        (hy.mangle "🦑")       ; => "hyx_XsquidX"
 
     If the stringified argument is already both legal as a Python identifier
     and normalized according to Unicode normalization form KC (NFKC), it will


Test patch:
diff --git a/tests/native_tests/functions.hy b/tests/native_tests/functions.hy
--- a/tests/native_tests/functions.hy
+++ b/tests/native_tests/functions.hy
@@ -299,12 +299,17 @@
 
 
 (defn test-yield-in-try []
+  (setv hit-finally False)
   (defn gen []
     (setv x 1)
-    (try (yield x)
-         (finally (print x))))
+    (try
+      (yield x)
+      (finally
+        (nonlocal hit-finally)
+        (setv hit-finally True))))
   (setv output (list (gen)))
-  (assert (= [1] output)))
+  (assert (= [1] output))
+  (assert hit-finally))
 
 
 (defn test-midtree-yield []
diff --git a/tests/native_tests/model_patterns.hy b/tests/native_tests/model_patterns.hy
--- a/tests/native_tests/model_patterns.hy
+++ b/tests/native_tests/model_patterns.hy
@@ -39,7 +39,6 @@
   (defn f [loopers]
     (setv head (if loopers (get loopers 0) None))
     (setv tail (cut loopers 1 None))
-    (print head)
     (cond
        (is head None)
         `(do ~@body)
diff --git a/tests/native_tests/quote.hy b/tests/native_tests/quote.hy
--- a/tests/native_tests/quote.hy
+++ b/tests/native_tests/quote.hy
@@ -1,95 +1,163 @@
+"Tests of `quote` and `quasiquote`."
+
+
 (import
   pytest)
 
-
-(defn test-quote []
-  (setv q (quote (a b c)))
-  (assert (= (len q) 3))
-  (assert (= q (hy.models.Expression [(quote a) (quote b) (quote c)]))))
+(setv E hy.models.Expression)
+(setv S hy.models.Symbol)
 
 
-(defn test-basic-quoting []
-  (assert (= (type (quote (foo bar))) hy.models.Expression))
-  (assert (= (type (quote foo)) hy.models.Symbol))
-  (assert (= (type (quote "string")) hy.models.String))
-  (assert (= (type (quote b"string")) hy.models.Bytes)))
+(defn test-quote-basic []
+  (assert (= '3 (hy.models.Integer 3)))
+  (assert (= 'a (S "a")))
+  (assert (= 'False (S "False")))
+  (assert (= '"hello" (hy.models.String "hello")))
+  (assert (= 'b"hello" (hy.models.Bytes b"hello")))
+  (assert (= '(a b) (E [(S "a") (S "b")])))
+  (assert (=
+    '{foo bar baz quux}
+    (hy.models.Dict (map S ["foo" "bar" "baz" "quux"])))))
 
 
 (defn test-quoted-hoistable []
-  (setv f (quote (if True True True)))
-  (assert (= (get f 0) (quote if)))
-  (assert (= (cut f 1 None) (quote (True True True)))))
+  (setv f '(if True True True))
+  (assert (= (get f 0) 'if))
+  (assert (= (cut f 1 None) '(True True True))))
+
+
+(defn test-quasiquote-trivial []
+  "Quasiquote and quote are equivalent for simple cases."
+  (assert (= `(a b c) '(a b c))))
 
 
 (defn test-quoted-macroexpand []
   "Don't expand macros in quoted expressions."
   (require tests.resources.macros [test-macro])
-  (setv q1 (quote (test-macro)))
-  (setv q2 (quasiquote (test-macro)))
+  (setv q1 '(test-macro))
+  (setv q2 `(test-macro))
   (assert (= q1 q2))
-  (assert (= (get q1 0) (quote test-macro))))
-
-
-(defn test-quote-dicts []
-  (setv q (quote {foo bar baz quux}))
-  (assert (= (len q) 4))
-  (assert (= (get q 0) (quote foo)))
-  (assert (= (get q 1) (quote bar)))
-  (assert (= (get q 2) (quote baz)))
-  (assert (= (get q 3) (quote quux)))
-  (assert (= (type q) hy.models.Dict)))
+  (assert (= (get q1 0) 'test-macro)))
 
 
 (defn test-quote-expr-in-dict []
-  (setv q (quote {(foo bar) 0}))
-  (assert (= (len q) 2))
-  (setv qq (get q 0))
-  (assert (= qq (quote (foo bar)))))
-
-
-(defn test-quasiquote []
-  "Quasiquote and quote are equivalent for simple cases."
-  (setv q (quote (a b c)))
-  (setv qq (quasiquote (a b c)))
-  (assert (= q qq)))
+  (assert (=
+    '{(foo bar) 0}
+    (hy.models.Dict [(E [(S "foo") (S "bar")]) (hy.models.Integer 0)]))))
 
 
 (defn test-unquote []
-  (setv q (quote (unquote foo)))
+  (setv q '~foo)
   (assert (= (len q) 2))
-  (assert (= (get q 1) (quote foo)))
-  (setv qq (quasiquote (a b c (unquote (+ 1 2)))))
-  (assert (= (len qq) 4))
-  (assert (= (hy.as-model qq) (quote (a b c 3)))))
+  (assert (= (get q 1) 'foo))
+  (setv qq `(a b c ~(+ 1 2)))
+  (assert (= (hy.as-model qq) '(a b c 3))))
 
 
 (defn test-unquote-splice []
-  (setv q (quote (c d e)))
+  (setv q '(c d e))
   (setv qq `(a b ~@q f ~@q ~@0 ~@False ~@None g ~@(when False 1) h))
-  (assert (= (len qq) 11))
-  (assert (= qq (quote (a b c d e f c d e g h)))))
+  (assert (= qq '(a b c d e f c d e g h))))
 
 
-(defn test-nested-quasiquote []
-  (setv qq (hy.as-model `(1 `~(+ 1 ~(+ 2 3) ~@None) 4)))
-  (setv q (quote (1 `~(+ 1 5) 4)))
-  (assert (= (len q) 3))
-  (assert (= (get qq 1) (quote `~(+ 1 5))))
-  (assert (= q qq)))
-
+(defmacro doodle [#* args]
+  `[1 ~@args 2])
 
-(defmacro doodle [#* body]
-  `(do ~@body))
-
-(defn test-unquote-splice []
+(defn test-unquote-splice-in-mac []
   (assert (=
     (doodle
-      [1 2 3]
-      [4 5 6])
-    [4 5 6])))
+      (setv x 5)
+      (+= x 1)
+      x)
+    [1 None None 6 2])))
 
 
 (defn test-unquote-splice-unpack []
   ; https://github.com/hylang/hy/issues/2336
   (with [(pytest.raises hy.errors.HySyntaxError)]
     (hy.eval '`[~@ #* [[1]]])))
+
+
+(defn test-nested-quasiquote []
+  (setv qq (hy.as-model `(1 `~(+ 1 ~(+ 2 3) ~@None) 4)))
+  (setv q '(1 `~(+ 1 5) 4))
+  (assert (= (len q) 3))
+  (assert (= (get qq 1) '`~(+ 1 5)))
+  (assert (= qq q)))
+
+
+(defn test-nested-quasiquote--nested-struct []
+  (assert (=
+    (hy.as-model `(try
+      ~@(lfor
+        i [1 2 3]
+        `(setv ~(S (+ "x" (str i))) (+ "x" (str ~i))))
+      (finally
+        (print "done"))))
+    '(try
+      (setv x1 (+ "x" (str 1)))
+      (setv x2 (+ "x" (str 2)))
+      (setv x3 (+ "x" (str 3)))
+      (finally
+        (print "done"))))))
+
+
+(defmacro macroify-programs [#* names]
+  `(do
+    ~@(lfor name names
+      `(defmacro ~name [#* args]
+        `(.append imaginary-syscalls #(~'~(str name) ~@(map str args)))))))
+
+(defn test-nested-quasiquote--macro-writing-macro-1 []
+  "A test of the construct ~'~ (to substitute in a variable from a
+  higher-level quasiquote) inspired by
+  https://github.com/hylang/hy/discussions/2251"
+
+  (setv imaginary-syscalls [])
+  (macroify-programs ls mkdir touch)
+  (mkdir mynewdir)
+  (touch mynewdir/file)
+  (ls -lA mynewdir)
+  (assert (= imaginary-syscalls [
+    #("mkdir" "mynewdir")
+    #("touch" "mynewdir/file")
+    #("ls" "-lA" "mynewdir")])))
+
+
+(defmacro def-caller [abbrev proc]
+  `(defmacro ~abbrev [var form]
+    `(~'~proc
+      (fn [~var] ~form))))
+(def-caller smoo-caller smoo)
+
+(defn test-nested-quasiquote--macro-writing-macro-2 []
+  "A similar test to the previous one, based on section 3.2 of
+  Bawden, A. (1999). Quasiquotation in Lisp. ACM SIGPLAN Workshop on Partial Evaluation and Program Manipulation. Retrieved from http://web.archive.org/web/20230105083805id_/https://3e8.org/pub/scheme/doc/Quasiquotation%20in%20Lisp%20(Bawden).pdf"
+
+  (setv accum [])
+  (defn smoo [f]
+    (.append accum "entered smoo")
+    (f "in smoo")
+    (.append accum "exiting smoo"))
+  (smoo-caller arg
+    (.append accum (+ "in the lambda: " arg)))
+  (assert (= accum [
+    "entered smoo"
+    "in the lambda: in smoo"
+    "exiting smoo"])))
+
+
+(defn test-nested-quasiquote--triple []
+  "N.B. You can get the same results with an analogous test in Emacs
+  Lisp or Common Lisp."
+
+  (setv
+    a 1  b 1  c 1
+    x ```[~a ~~b ~~~c]
+    ; `x` has been implicitly evaluated once. Let's explicitly
+    ; evaluate it twice more, so no backticks are left.
+    a 2  b 2  c 2
+    x (hy.eval x)
+    a 3  b 3  c 3
+    x (hy.eval x))
+  (assert (= (hy.as-model x) '[3 2 1])))
diff --git a/tests/resources/importer/circular.hy b/tests/resources/importer/circular.hy
--- a/tests/resources/importer/circular.hy
+++ b/tests/resources/importer/circular.hy
@@ -2,4 +2,3 @@
 (defn f []
   (import circular)
   circular.a)
-(print (f))
diff --git a/tests/resources/importer/foo/__init__.hy b/tests/resources/importer/foo/__init__.hy
--- a/tests/resources/importer/foo/__init__.hy
+++ b/tests/resources/importer/foo/__init__.hy
@@ -1,2 +1 @@
-(print "This is __init__.hy")
 (setv ext "hy")
diff --git a/tests/resources/importer/foo/__init__.py b/tests/resources/importer/foo/__init__.py
--- a/tests/resources/importer/foo/__init__.py
+++ b/tests/resources/importer/foo/__init__.py
@@ -1,2 +1 @@
-print("This is __init__.py")
 ext = "py"
diff --git a/tests/resources/importer/foo/some_mod.hy b/tests/resources/importer/foo/some_mod.hy
--- a/tests/resources/importer/foo/some_mod.hy
+++ b/tests/resources/importer/foo/some_mod.hy
@@ -1,2 +1 @@
-(print "This is test_mod.hy")
 (setv ext "hy")
diff --git a/tests/resources/importer/foo/some_mod.py b/tests/resources/importer/foo/some_mod.py
--- a/tests/resources/importer/foo/some_mod.py
+++ b/tests/resources/importer/foo/some_mod.py
@@ -1,2 +1 @@
-print("This is test_mod.py")
 ext = "py"
diff --git a/tests/test_bin.py b/tests/test_bin.py
--- a/tests/test_bin.py
+++ b/tests/test_bin.py
@@ -131,7 +131,6 @@ def test_error_parts_length():
     # Make sure lines are printed in between arrows separated by more than one
     # character.
     _, err = run_cmd("hy -i", prg_str.format(3, 3, 1, 6))
-    print(err)
 
     msg_idx = err.rindex("HyLanguageError:")
     assert msg_idx
@@ -420,7 +419,7 @@ def req_err(x):
     assert len(error_lines) <= 4
     req_err(error_lines[-1])
 
-    output, error = run_cmd('hy -i -c "(require not-a-real-module)"')
+    output, error = run_cmd('hy -i -c "(require not-a-real-module)"', '')
     assert output.startswith("=> ")
     req_err(error.splitlines()[2])