Repository: hylang/hy

Issue:
Consider adding Common Lisp's Sharpsign-dot (#.) reader macro
Pushing the eval of certain forms back to readtime has various uses. A big one is to swap out code based on various conditions, like the `#if` and related preprocessor directives from C.

``` Hy
(#.(if (< *version* 2)  old-func  new-func) arg)
```

This would probably fit well with #740.

Another is to substitute for literals that don't have a literal notation. Compare:

``` Hy
=> (int "1a" 32)  ; base 32 number
int('1a', 32)
42
=> #.(int "1a" 32)  ; expands directly into 42 at readtime, like a literal
42
42
=> (+ "A line that is much too long to fit on one line "
... "doesn't actually have to fit on one line.")
('A line that is much too long to fit on one line ' + "doesn't actually have to fi
t on one line.")
"A line that is much too long to fit on one line doesn't actually have to fit on
 one line."
=> #.(+ "A line that is much too long to fit on one line "
... "doesn't actually have to fit on one line.")
"A line that is much too long to fit on one line doesn't actually have to fit on
 one line."
"A line that is much too long to fit on one line doesn't actually have to fit on
 one line."
```

Because evaluation happens at read time, it's safe to use in loops and such, without a performance hit for recomputing it every time.

I thought it would be a simple implementation in Hy:

``` Hy
#@((hy.macros.reader ".") (fn [expr] (eval expr)))
```

And this does seem to work for the above examples, but I'm running into `HyModel` issues again:

``` Hy
=> #.(lambda [] nil)
```

[long stack trace including `hy.errors.HyCompileError: Internal Compiler Bug \U0001f631\u2937 TypeError: Don't know how to wrap a <class 'function'> object to a HyObject`]

But this works fine in Common Lisp:

``` Lisp
* #.(lambda () nil)

#<FUNCTION (LAMBDA ()) {1002E222DB}>
```

Is there a simple change in the macro definition that would fix this? Can we perhaps make everything a HyObject by default if it doesn't already have a model?



Discussion:
Is `eval-when-compile`, which we now have, equivalent to what you asked for here?
Maybe. It's not documented, so I'm not sure. And many of my use cases would need a shorter alias.
I was wrong; `eval-when-compile` doesn't expand to anything, so it doesn't do this. It's only for changing the state of the compiler, presumably to affect macro expansions.
[The Emacs docs](https://www.gnu.org/software/emacs/manual/html_node/elisp/Eval-During-Compile.html) on `eval-when-compile` says it's kind of similar.
> Elsewhere, the Common Lisp `‘#.’` reader macro (but not when interpreting) is closer to what `eval-when-compile` does.

Unlike Hy, in Emacs
> The result of evaluation by the compiler becomes a constant which appears in the compiled program. 

But since Hy's version doesn't expand to anything, it can't be used that way.

But how can we make a constant appear in the compiled program? As my first attempt demonstrated, only certain data types have a Hy model. To get an arbitrary object into the compiled file, it would have to be serialized somehow. Sounds like a job for `pickle`.

As a first attempt,
```Hy
(deftag "." [expr]
  (import pickle)
  `((. (__import__ "pickle")
       loads)
    ~(pickle.dumps (eval expr))))
```
This does have some runtime overhead. It's not a big deal at the toplevel, but in a nested loop, that would be pretty bad--and that was a major motivation for doing `#.` in the first place.

I think we can overcome this problem with memoization.
```Hy
(import pickle)

;; you can memoize with functools in Python3...
(defn loadstatic [bytestr &optional [_memo {}]]
  (try
    (get _memo bytestr)
    (except [KeyError]
      (assoc _memo bytestr (pickle.loads bytestr))
      (loadstatic bytestr))))

(deftag "." [expr]
  `(loadstatic ~(pickle.dumps (eval expr))))
```
This way it would only get unpickled once. `loadstatic` could be a core function, or we could have the compiler do it with a special form or a new `HyStatic` Hy model type or something.

There's two more problems with this though.

First, if you pickle two equal mutable objects, then the memoization will make them point to the same instance. We can avoid this by adding a gensym to the lookup key, e.g. `(loadstatic ~(pickle.dumps (, (gensym) (eval expr)))))`. This gives a different instance per expansion.

And secondly, `pickle` is also limited in the data types it can serialize. It can't even do lambdas. But there are more capable third-party serializers, like [dill](https://pypi.org/project/dill/). But that's another dependency. Maybe we could try importing that, and fall back to `pickle` if it's not available.

I think you want `eval-and-compile`, not `eval-when-compile`.

Resolving patch:
diff --git a/conftest.py b/conftest.py
--- a/conftest.py
+++ b/conftest.py
@@ -1,38 +1,15 @@
 import importlib
 import os
-import sys
-from functools import reduce
-from operator import or_
 from pathlib import Path
 
 import pytest
 
-import hy
-from hy._compat import PY3_8, PY3_10, PY3_11
-
 NATIVE_TESTS = os.path.join("", "tests", "native_tests", "")
 
 # https://github.com/hylang/hy/issues/2029
 os.environ.pop("HYSTARTUP", None)
 
 
-def pytest_ignore_collect(path, config):
-    versions = [
-        (sys.version_info < (3, 8), "sub_py3_7_only"),
-        (PY3_8, "py3_8_only"),
-        (PY3_10, "py3_10_only"),
-        (PY3_11, "py3_11_only"),
-    ]
-
-    return (
-        reduce(
-            or_,
-            (name in path.basename and not condition for condition, name in versions),
-        )
-        or None
-    )
-
-
 def pytest_collect_file(parent, path):
     if (
         path.ext == ".hy"
diff --git a/hy/core/result_macros.py b/hy/core/result_macros.py
--- a/hy/core/result_macros.py
+++ b/hy/core/result_macros.py
@@ -54,6 +54,7 @@
     String,
     Symbol,
     Tuple,
+    as_model,
     is_unpack,
 )
 from hy.reader import mangle, unmangle
@@ -88,12 +89,12 @@ def compile_do(self, expr, root, body):
     return self._compile_branch(body)
 
 
-@pattern_macro(["eval-and-compile", "eval-when-compile"], [many(FORM)])
-def compile_eval_and_compile(compiler, expr, root, body):
+@pattern_macro(["eval-and-compile", "eval-when-compile", "do-mac"], [many(FORM)])
+def compile_eval_foo_compile(compiler, expr, root, body):
     new_expr = Expression([Symbol("do").replace(expr[0])]).replace(expr)
 
     try:
-        hy_eval(
+        value = hy_eval(
             new_expr + body,
             compiler.module.__dict__,
             compiler.module,
@@ -115,7 +116,9 @@ def compile_eval_and_compile(compiler, expr, root, body):
         raise HyEvalError(str(e), compiler.filename, body, compiler.source)
 
     return (
-        compiler._compile_branch(body)
+        compiler.compile(as_model(value))
+        if mangle(root) == "do_mac"
+        else compiler._compile_branch(body)
         if mangle(root) == "eval_and_compile"
         else Result()
     )


Test patch:
diff --git a/tests/native_tests/eval_foo_compile.hy b/tests/native_tests/eval_foo_compile.hy
--- a/tests/native_tests/eval_foo_compile.hy
+++ b/tests/native_tests/eval_foo_compile.hy
@@ -1,4 +1,4 @@
-;; Tests of `eval-when-compile` and `eval-and-compile`
+;; Tests of `eval-when-compile`, `eval-and-compile`, and `do-mac`
 
 
 (defn test-eval-foo-compile-return-values []
@@ -15,3 +15,33 @@
   (setv derrick (eval-when-compile (+= jim 1) 2))
   (assert (= jim 1))
   (assert (is derrick None)))
+
+
+(defn test-do-mac []
+
+  (assert (is (do-mac) None))
+
+  (setv x 2)
+  (setv x-compile-time (do-mac
+    (setv x 3)
+    x))
+  (assert (= x 2))
+  (assert (= x-compile-time 3))
+
+  (eval-when-compile (setv x 4))
+  (assert (= x 2))
+  (assert (= (do-mac x) 4))
+
+  (defmacro m []
+    (global x)
+    (setv x 5))
+  (m)
+  (assert (= x 2))
+  (assert (= (do-mac x) 5))
+
+  (setv l [])
+  (do-mac `(do ~@(* ['(.append l 1)] 5)))
+  (assert (= l [1 1 1 1 1]))
+
+  (do-mac `(setv ~(hy.models.Symbol (* "x" 5)) "foo"))
+  (assert (= xxxxx "foo")))
diff --git a/tests/native_tests/py3_10_only_tests.hy b/tests/native_tests/match.hy
similarity index 99%
rename from tests/native_tests/py3_10_only_tests.hy
rename to tests/native_tests/match.hy
--- a/tests/native_tests/py3_10_only_tests.hy
+++ b/tests/native_tests/match.hy
@@ -1,3 +1,6 @@
+(do-mac (when hy._compat.PY3_10 '(do
+
+
 (import pytest
         dataclasses [dataclass]
         hy.errors [HySyntaxError])
@@ -260,3 +263,6 @@
                (match y
                       {"b" b #**a} a)))
     (assert (= b 2))))
+
+
+)))
diff --git a/tests/native_tests/py3_11_only_tests.hy b/tests/native_tests/py3_11_only_tests.hy
deleted file mode 100644
--- a/tests/native_tests/py3_11_only_tests.hy
+++ /dev/null
@@ -1,23 +0,0 @@
-(defn test-except* []
-  (setv got "")
-
-  (setv return-value (try
-    (raise (ExceptionGroup "meep" [(KeyError) (ValueError)]))
-    (except* [KeyError]
-      (+= got "k")
-      "r1")
-    (except* [IndexError]
-      (+= got "i")
-      "r2")
-    (except* [ValueError]
-      (+= got "v")
-      "r3")
-    (else
-      (+= got "e")
-      "r4")
-    (finally
-      (+= got "f")
-      "r5")))
-
-  (assert (= got "kvf"))
-  (assert (= return-value "r3")))
diff --git a/tests/native_tests/py3_8_only_tests.hy b/tests/native_tests/setx.hy
similarity index 70%
rename from tests/native_tests/py3_8_only_tests.hy
rename to tests/native_tests/setx.hy
--- a/tests/native_tests/py3_8_only_tests.hy
+++ b/tests/native_tests/setx.hy
@@ -1,9 +1,13 @@
-;; Tests where the emitted code relies on Python ≥3.8.
-;; conftest.py skips this file when running on Python <3.8.
-
 (import pytest)
 
-(defn test-setx []
+(defn
+    [(pytest.mark.skipif hy._compat.PY3_8 :reason "Python ≥ 3.8")]
+    test-cant-setx []
+  (with [e (pytest.raises hy.errors.HySyntaxError)]
+    (hy.eval '(setx x 1)))
+  (assert (= "setx requires Python 3.8 or later")))
+
+(do-mac (when hy._compat.PY3_8 '(defn test-setx []
   (setx y (+ (setx x (+ "a" "b")) "c"))
   (assert (= x "ab"))
   (assert (= y "abc"))
@@ -22,9 +26,9 @@
   (assert (= filtered ["apple" "banana"]))
   (assert (= v "banana"))
   (with [(pytest.raises NameError)]
-    i))
+    i))))
 
-(defn test-setx-generator-scope []
+(do-mac (when hy._compat.PY3_8 '(defn test-setx-generator-scope []
   ;; https://github.com/hylang/hy/issues/1994
   (setv x 20)
   (lfor n (range 10) (setx x n))
@@ -47,11 +51,11 @@
 
   (lfor n (range 0) :do x (setx z n))
   (with [(pytest.raises UnboundLocalError)]
-    z))
+    z))))
 
-(defn test-let-setx []
+(do-mac (when hy._compat.PY3_8 '(defn test-let-setx []
   (let [x 40
         y 13]
     (setv y (setx x 2))
     (assert (= x 2))
-    (assert (= y 2))))
+    (assert (= y 2))))))
diff --git a/tests/native_tests/sub_py3_7_only.hy b/tests/native_tests/sub_py3_7_only.hy
deleted file mode 100644
--- a/tests/native_tests/sub_py3_7_only.hy
+++ /dev/null
@@ -1,9 +0,0 @@
-;; Tests where the emitted code relies on Python ≤3.7.
-;; conftest.py skips this file when running on Python ≥3.8.
-
-(import pytest)
-
-(defn test-setx []
-  (with [e (pytest.raises hy.errors.HySyntaxError)]
-    (hy.eval '(setx x 1)))
-  (assert (= "setx requires Python 3.8 or later")))
diff --git a/tests/native_tests/try.hy b/tests/native_tests/try.hy
--- a/tests/native_tests/try.hy
+++ b/tests/native_tests/try.hy
@@ -211,6 +211,31 @@
   (assert (= y 1)))
 
 
+(do-mac (when hy._compat.PY3_11 '(defn test-except* []
+  (setv got "")
+
+  (setv return-value (try
+    (raise (ExceptionGroup "meep" [(KeyError) (ValueError)]))
+    (except* [KeyError]
+      (+= got "k")
+      "r1")
+    (except* [IndexError]
+      (+= got "i")
+      "r2")
+    (except* [ValueError]
+      (+= got "v")
+      "r3")
+    (else
+      (+= got "e")
+      "r4")
+    (finally
+      (+= got "f")
+      "r5")))
+
+  (assert (= got "kvf"))
+  (assert (= return-value "r3")))))
+
+
 (defn test-raise-from []
   (assert (is NameError (type (.
     (try