Repository: hylang/hy

Issue:
list-comp only iterates over the last expression when it is passed a do
Only the last expression in do is interpreted in context of the list-comp:

```
=> (list-comp (do (print n) (print n)) [n [1 2]])
print(n)
[print(n) for n in [1L, 2L]]
Traceback (most recent call last):
  File "<input>", line 1, in <module>
NameError: name 'n' is not defined
```

Workaround (and the expected result of the above function):

```
=> (list-comp ((fn [a] (do (print a) (print a))) n) [n [1 2]])
def _hy_anon_fn_1(a):
    print(a)
    return print(a)
[_hy_anon_fn_1(n) for n in [1L, 2L]]
1
1
2
2
[None, None]
```



Discussion:
Here's a similar example:

``` Hy
=> (print (list-comp (assert x) [x [1 2 3]]))
assert x
print([None for x in [1, 2, 3]])
Traceback (most recent call last):
  File "<input>", line 1, in <module>
NameError: name 'x' is not defined
```

Surprising behavior like this is unacceptable, but what should this even compile to?
Maybe something like this:

``` Python
_hy_anon_var_1 = []
for x in [1, 2, 3]:
    assert x
    _hy_anon_var_1.append(None)
print(_hy_anon_var_1)
```

This leaks `x` into the surrounding scope, which was actually correct for Python 2.

The above won't work for a `genexpr`, which is also broken:

``` Hy
=> (list (genexpr (assert x) [x [1 2 3]]))
assert x
list((None for x in [1, 2, 3]))
Traceback (most recent call last):
  File "<input>", line 1, in <module>
NameError: name 'x' is not defined
```

It must be lazy to work correctly. Maybe something like this:

``` Python
_hy_anon_var_1 = [1, 2, 3]
def _hy_anon_fn_1():
    for x in _hy_anon_var_1:
        assert x
        yield None
list(_hy_anon_fn_1())
```

No leak this time. You can use `genexpr` to implement all the other comprehensions. This seems like an improvment over the status quo, but compiling to a function like this introduces the same problems we had with `let`.

Probably relevant: https://stackoverflow.com/questions/32139885/yield-in-list-comprehensions-and-generator-expressions#32139977

Basically, Python compiles generator expressions to generator function objects anyway. Therefore, it won't cause problems (beyond what problems Python already has) if we do the same in Hy, as I suggested in my previous comment.

Here's an even simpler example.
```Python
>>> [(lambda: x)() for x in range(5)]
[0, 1, 2, 3, 4]
```
```Python
=> (list-comp ((fn [] x)) [x (range 5)])
[(lambda : x)() for x in range(5)]
[0, 1, 2, 3, 4]
=> (list-comp ((fn [] "boo!" x)) [x (range 5)])
def _hy_anon_fn_1():
    'boo!'
    return x
[_hy_anon_fn_1() for x in range(5)]
Traceback (most recent call last):
  File "c:\users\me\documents\github\hy\hy\importer.py", line 198, in hy_eval
    return eval(ast_compile(expr, "<eval>", "eval"), namespace)
  File "<eval>", line 1, in <module>
  File "<eval>", line 1, in <listcomp>
  File "<eval_body>", line 1, in _hy_anon_fn_1
NameError: name 'x' is not defined
```
Digging this up because of the recent #1342 and #1362 which have similar problems.

I think we have a good solution. I'm proposing that generally, if there are any expressions we have to pull out, that
```Hy
(genexpr <expression> [<binding> <sequence>] <condition>)
```
Should compile to
```Python
# <sequence> statements naturally go here.
_hy_anon_var_1 = <sequence>
def _hy_anon_fn_1():
    for <binding> in _hy_anon_var_1:
        # Pull out any <condition> statements here.
        if <condition>:
            # Pull out any <expression> statements here.
            yield <expression>
_hy_anon_fn_1()
```

For a `list-comp`, change the last line to
```Python
[_hy_anon_var_2 for _hy_anon_var_2 in _hy_anon_fn_1()]
```
We shouldn't use `list(_hy_anon_fn_1())`, because `list` might be shadowed by a local.
Similarly for the `set-comp`, but use `{}` instead of `[]`.

A `dict-comp` is a little different:
```Hy
(dict-comp <key> <value> [<binding> <sequence>] <condition>)
```
```Python
# <sequence> statements.
_hy_anon_var_1 = <sequence>
def _hy_anon_fn_1():
    for <binding> in _hy_anon_var_1:
        # <condition> statements
        if <condition>:
            # Pull out any <key> or <value> statements here.
            yield <key>, <value>
{_hy_anon_var2: _hy_anon_var3 for _hy_anon_var2, _hy_anon_var3 in_hy_anon_fn_1()}
```
But it's the same basic idea.

This list-comp doesn't leak the binding into the local scope in Python2. I'm not sure if there's any point to that. It was undesired behavior due to an optimization. Generator expressions, and set- and dict-comprehensions don't have this problem even in Python2. And we won't be supporting Python2 much longer anyway.

But the leaky Python2 `list-comp` could instead be compiled to
```Python
# Pull out any <sequence> statements here.
_hy_anon_var_1 = <sequence>
_hy_anon_var_2 = []
for <binding> in _hy_anon_var_1:
    # Pull out any <condition> statements here.
    if <condition>:
        # Pull out any <expression> statements here.
        _hy_anon_var_2.append(<expression>)
_hy_anon_var_2
```

I forgot the case with multiple bindings:
```Hy
(genexpr <expression>
         [<binding1> <sequence1>
          <binding2> <sequence2>
            ...
          <bindingN> <sequenceN>])
```
But the compilation is straightforward. Add more loops:
```Python
# <sequence1> statements
_hy_anon_var_1 = <sequence1>
def _hy_anon_fn_1():
    for <binding1> in _hy_anon_var_1:
        # <sequence2> statements
        _hy_anon_var_2 = <sequence2>
        for <binding2> in _hy_anon_var_2:
             ...
            # <sequenceN> statements
            _hy_anon_var_N = <sequenceN>
            for <bindingN> in _hy_anon_var_N:
                # <expression> statements
                yield <expression>
_hy_anon_fn_1()
```

Resolving patch:
diff --git a/hy/compiler.py b/hy/compiler.py
--- a/hy/compiler.py
+++ b/hy/compiler.py
@@ -7,7 +7,7 @@
                        HyString, HyBytes, HySymbol, HyFloat, HyList, HySet,
                        HyDict, HySequence, wrap_value)
 from hy.model_patterns import (FORM, SYM, STR, sym, brackets, whole, notpexpr,
-                               dolike, pexpr, times)
+                               dolike, pexpr, times, Tag, tag)
 from funcparserlib.parser import some, many, oneplus, maybe, NoParseError
 from hy.errors import HyCompileError, HyTypeError
 
@@ -111,13 +111,13 @@ def _dec(fn):
 # ast.Foo(..., lineno=x.lineno, col_offset=x.col_offset)
 class Asty(object):
     def __getattr__(self, name):
-        setattr(Asty, name, lambda self, x, **kwargs: getattr(ast, name)(
+        setattr(Asty, name, staticmethod(lambda x, **kwargs: getattr(ast, name)(
             lineno=getattr(
                 x, 'start_line', getattr(x, 'lineno', None)),
             col_offset=getattr(
                 x, 'start_column', getattr(x, 'col_offset', None)),
-            **kwargs))
-        return getattr(self, name)
+            **kwargs)))
+        return getattr(Asty, name)
 asty = Asty()
 
 
@@ -180,6 +180,22 @@ def expr(self, value):
         self.__used_expr = False
         self._expr = value
 
+    @property
+    def lineno(self):
+        if self._expr is not None:
+            return self._expr.lineno
+        if self.stmts:
+            return self.stmts[-1].lineno
+        return None
+
+    @property
+    def col_offset(self):
+        if self._expr is not None:
+            return self._expr.col_offset
+        if self.stmts:
+            return self.stmts[-1].col_offset
+        return None
+
     def add_imports(self, mod, imports):
         """Autoimport `imports` from `mod`"""
         self.imports[mod].update(imports)
@@ -1004,40 +1020,150 @@ def _compile_generator_iterables(self, trailers):
 
         return gen_res + cond, gen
 
-    @special(["list-comp", "set-comp", "genexpr"], [FORM, FORM, maybe(FORM)])
-    def compile_comprehension(self, expr, form, expression, gen, cond):
-        # (list-comp expr [target iter] cond?)
-
-        if not isinstance(gen, HyList):
-            raise HyTypeError(gen, "Generator expression must be a list.")
-
-        gen_res, gen = self._compile_generator_iterables(
-            [gen] + ([] if cond is None else [cond]))
-
-        if len(gen) == 0:
-            raise HyTypeError(expr, "Generator expression cannot be empty.")
-
-        ret = self.compile(expression)
-        node_class = (
-            asty.ListComp if form == "list-comp" else
-            asty.SetComp if form == "set-comp" else
-            asty.GeneratorExp)
-        return ret + gen_res + node_class(
-            expr, elt=ret.force_expr, generators=gen)
-
-    @special("dict-comp", [FORM, FORM, FORM, maybe(FORM)])
-    def compile_dict_comprehension(self, expr, root, key, value, gen, cond):
-        key = self.compile(key)
-        value = self.compile(value)
-
-        gen_res, gen = self._compile_generator_iterables(
-            [gen] + ([] if cond is None else [cond]))
-
-        return key + value + gen_res + asty.DictComp(
-            expr,
-            key=key.force_expr,
-            value=value.force_expr,
-            generators=gen)
+    _loopers = many(
+        tag('setv', sym(":setv") + FORM + FORM) |
+        tag('if', sym(":if") + FORM) |
+        tag('do', sym(":do") + FORM) |
+        tag('afor', sym(":async") + FORM + FORM) |
+        tag('for', FORM + FORM))
+    @special(["for"], [brackets(_loopers),
+        many(notpexpr("else")) + maybe(dolike("else"))])
+    @special(["lfor", "sfor", "gfor"], [_loopers, FORM])
+    @special(["dfor"], [_loopers, brackets(FORM, FORM)])
+    def compile_comprehension(self, expr, root, parts, final):
+        root = unmangle(ast_str(root))
+        node_class = {
+            "for":  asty.For,
+            "lfor": asty.ListComp,
+            "dfor": asty.DictComp,
+            "sfor": asty.SetComp,
+            "gfor": asty.GeneratorExp}[root]
+        is_for = root == "for"
+
+        orel = []
+        if is_for:
+            # Get the `else`.
+            body, else_expr = final
+            if else_expr is not None:
+                orel.append(self._compile_branch(else_expr))
+                orel[0] += orel[0].expr_as_stmt()
+        else:
+            # Get the final value (and for dictionary
+            # comprehensions, the final key).
+            if node_class is asty.DictComp:
+                key, elt = map(self.compile, final)
+            else:
+                key = None
+                elt = self.compile(final)
+
+        # Compile the parts.
+        if is_for:
+            parts = parts[0]
+        if not parts:
+            return Result(expr=ast.parse({
+                asty.For: "None",
+                asty.ListComp: "[]",
+                asty.DictComp: "{}",
+                asty.SetComp: "{1}.__class__()",
+                asty.GeneratorExp: "(_ for _ in [])"}[node_class]).body[0].value)
+        parts = [
+            Tag(p.tag, self.compile(p.value) if p.tag in ["if", "do"] else [
+                self._storeize(p.value[0], self.compile(p.value[0])),
+                self.compile(p.value[1])])
+            for p in parts]
+
+        # Produce a result.
+        if (is_for or elt.stmts or (key is not None and key.stmts) or
+            any(p.tag == 'do' or (p.value[1].stmts if p.tag in ("for", "afor", "setv") else p.value.stmts)
+                for p in parts)):
+            # The desired comprehension can't be expressed as a
+            # real Python comprehension. We'll write it as a nested
+            # loop in a function instead.
+            contains_yield = []
+            def f(parts):
+                # This function is called recursively to construct
+                # the nested loop.
+                if not parts:
+                    if is_for:
+                        if body:
+                            bd = self._compile_branch(body)
+                            if bd.contains_yield:
+                                contains_yield.append(True)
+                            return bd + bd.expr_as_stmt()
+                        return Result(stmts=[asty.Pass(expr)])
+                    if node_class is asty.DictComp:
+                        ret = key + elt
+                        val = asty.Tuple(
+                            key, ctx=ast.Load(),
+                            elts=[key.force_expr, elt.force_expr])
+                    else:
+                        ret = elt
+                        val = elt.force_expr
+                    return ret + asty.Expr(
+                        elt, value=asty.Yield(elt, value=val))
+                (tagname, v), parts = parts[0], parts[1:]
+                if tagname in ("for", "afor"):
+                    orelse = orel and orel.pop().stmts
+                    node = asty.AsyncFor if tagname == "afor" else asty.For
+                    return v[1] + node(
+                        v[1], target=v[0], iter=v[1].force_expr, body=f(parts).stmts,
+                        orelse=orelse)
+                elif tagname == "setv":
+                    return v[1] + asty.Assign(
+                        v[1], targets=[v[0]], value=v[1].force_expr) + f(parts)
+                elif tagname == "if":
+                    return v + asty.If(
+                        v, test=v.force_expr, body=f(parts).stmts, orelse=[])
+                elif tagname == "do":
+                    return v + v.expr_as_stmt() + f(parts)
+                else:
+                    raise ValueError("can't happen")
+            if is_for:
+                ret = f(parts)
+                ret.contains_yield = bool(contains_yield)
+                return ret
+            fname = self.get_anon_var()
+            # Define the generator function.
+            ret = Result() + asty.FunctionDef(
+                expr,
+                name=fname,
+                args=ast.arguments(
+                    args=[], vararg=None, kwarg=None,
+                    kwonlyargs=[], kw_defaults=[], defaults=[]),
+                body=f(parts).stmts,
+                decorator_list=[])
+            # Immediately call the new function. Unless the user asked
+            # for a generator, wrap the call in `[].__class__(...)` or
+            # `{}.__class__(...)` or `{1}.__class__(...)` to get the
+            # right type. We don't want to just use e.g. `list(...)`
+            # because the name `list` might be rebound.
+            return ret + Result(expr=ast.parse(
+                "{}({}())".format(
+                    {asty.ListComp: "[].__class__",
+                     asty.DictComp: "{}.__class__",
+                     asty.SetComp: "{1}.__class__",
+                     asty.GeneratorExp: ""}[node_class],
+                    fname)).body[0].value)
+
+        # We can produce a real comprehension.
+        generators = []
+        for tagname, v in parts:
+            if tagname in ("for", "afor"):
+                generators.append(ast.comprehension(
+                    target=v[0], iter=v[1].expr, ifs=[],
+                    is_async=int(tagname == "afor")))
+            elif tagname == "setv":
+                generators.append(ast.comprehension(
+                    target=v[0],
+                    iter=asty.Tuple(v[1], elts=[v[1].expr], ctx=ast.Load()),
+                    ifs=[], is_async=0))
+            elif tagname == "if":
+                generators[-1].ifs.append(v.expr)
+            else:
+                raise ValueError("can't happen")
+        if node_class is asty.DictComp:
+            return asty.DictComp(expr, key=key.expr, value=elt.expr, generators=generators)
+        return node_class(expr, elt=elt.expr, generators=generators)
 
     @special(["not", "~"], [FORM])
     def compile_unary_operator(self, expr, root, arg):
@@ -1288,36 +1414,6 @@ def _compile_assign(self, name, result):
 
         return result
 
-    @special(["for*", (PY35, "for/a*")],
-             [brackets(FORM, FORM), many(notpexpr("else")), maybe(dolike("else"))])
-    def compile_for_expression(self, expr, root, args, body, else_expr):
-        target_name, iterable = args
-        target = self._storeize(target_name, self.compile(target_name))
-
-        ret = Result()
-
-        orel = Result()
-        if else_expr is not None:
-            for else_body in else_expr:
-                orel += self.compile(else_body)
-                orel += orel.expr_as_stmt()
-
-        ret += self.compile(iterable)
-
-        body = self._compile_branch(body)
-        body += body.expr_as_stmt()
-
-        node = asty.For if root == 'for*' else asty.AsyncFor
-        ret += node(expr,
-                    target=target,
-                    iter=ret.force_expr,
-                    body=body.stmts or [asty.Pass(expr)],
-                    orelse=orel.stmts)
-
-        ret.contains_yield = body.contains_yield
-
-        return ret
-
     @special(["while"], [FORM, many(notpexpr("else")), maybe(dolike("else"))])
     def compile_while_expression(self, expr, root, cond, body, else_expr):
         cond_compiled = self.compile(cond)
diff --git a/hy/model_patterns.py b/hy/model_patterns.py
--- a/hy/model_patterns.py
+++ b/hy/model_patterns.py
@@ -9,6 +9,7 @@
     some, skip, many, finished, a, Parser, NoParseError, State)
 from functools import reduce
 from itertools import repeat
+from collections import namedtuple
 from operator import add
 from math import isinf
 
@@ -74,3 +75,11 @@ def f(tokens, s):
             end = e.state.max
         return result, State(s.pos, end)
     return f
+
+Tag = namedtuple('Tag', ['tag', 'value'])
+
+def tag(tag_name, parser):
+    """Matches the given parser and produces a named tuple `(Tag tag value)`
+    with `tag` set to the given tag name and `value` set to the parser's
+    value."""
+    return parser >> (lambda x: Tag(tag_name, x))


Test patch:
diff --git a/tests/compilers/test_ast.py b/tests/compilers/test_ast.py
--- a/tests/compilers/test_ast.py
+++ b/tests/compilers/test_ast.py
@@ -531,9 +531,7 @@ def test_for_compile_error():
         can_compile("(fn [] (for)))")
     assert excinfo.value.message == "Ran into a RPAREN where it wasn't expected."
 
-    with pytest.raises(HyTypeError) as excinfo:
-        can_compile("(fn [] (for [x] x))")
-    assert excinfo.value.message == "`for' requires an even number of args."
+    cant_compile("(fn [] (for [x] x))")
 
 
 def test_attribute_access():
@@ -556,14 +554,6 @@ def test_attribute_empty():
     cant_compile('[2].foo')
 
 
-def test_invalid_list_comprehension():
-    """Ensure that invalid list comprehensions do not break the compiler"""
-    cant_compile("(genexpr x [])")
-    cant_compile("(genexpr [x [1 2 3 4]] x)")
-    cant_compile("(list-comp None [])")
-    cant_compile("(list-comp [x [1 2 3]] x)")
-
-
 def test_bad_setv():
     """Ensure setv handles error cases"""
     cant_compile("(setv (a b) [1 2])")
diff --git a/tests/importer/test_importer.py b/tests/importer/test_importer.py
--- a/tests/importer/test_importer.py
+++ b/tests/importer/test_importer.py
@@ -84,5 +84,5 @@ def eval_str(s):
     assert eval_str('(.strip " fooooo   ")') == 'fooooo'
     assert eval_str(
         '(if True "this is if true" "this is if false")') == "this is if true"
-    assert eval_str('(list-comp (pow num 2) [num (range 100)] (= (% num 2) 1))') == [
+    assert eval_str('(lfor num (range 100) :if (= (% num 2) 1) (pow num 2))') == [
         pow(num, 2) for num in range(100) if num % 2 == 1]
diff --git a/tests/native_tests/comprehensions.hy b/tests/native_tests/comprehensions.hy
new file mode 100644
--- /dev/null
+++ b/tests/native_tests/comprehensions.hy
@@ -0,0 +1,226 @@
+(import
+  types
+  pytest
+  [hy._compat [PY3]])
+
+
+(defn test-comprehension-types []
+
+  ; Forms that get compiled to real comprehensions
+  (assert (is (type (lfor x "abc" x)) list))
+  (assert (is (type (sfor x "abc" x)) set))
+  (assert (is (type (dfor x "abc" [x x])) dict))
+  (assert (is (type (gfor x "abc" x)) types.GeneratorType))
+
+  ; Forms that get compiled to loops
+  (assert (is (type (lfor x "abc" :do (setv y 1) x)) list))
+  (assert (is (type (sfor x "abc" :do (setv y 1) x)) set))
+  (assert (is (type (dfor x "abc" :do (setv y 1) [x x])) dict))
+  (assert (is (type (gfor x "abc" :do (setv y 1) x)) types.GeneratorType)))
+
+
+#@ ((pytest.mark.parametrize "specialop" ["for" "lfor" "sfor" "gfor" "dfor"])
+(defn test-fors [specialop]
+
+  (setv cases [
+    ['(f x [] x)
+      []]
+    ['(f j [1 2 3] j)
+      [1 2 3]]
+    ['(f x (range 3) (* x 2))
+      [0 2 4]]
+    ['(f x (range 2) y (range 2) (, x y))
+      [(, 0 0) (, 0 1) (, 1 0) (, 1 1)]]
+    ['(f (, x y) (.items {"1" 1 "2" 2}) (* y 2))
+      [2 4]]
+    ['(f x (do (setv s "x") "ab") y (do (+= s "y") "def") (+ x y s))
+      ["adxy" "aexy" "afxy" "bdxyy" "bexyy" "bfxyy"]]
+    ['(f x (range 4) :if (% x 2) (* x 2))
+      [2 6]]
+    ['(f x "abc" :setv y (.upper x) (+ x y))
+      ["aA" "bB" "cC"]]
+    ['(f x "abc" :do (setv y (.upper x)) (+ x y))
+      ["aA" "bB" "cC"]]
+    ['(f
+        x (range 3)
+        y (range 3)
+        :if (> y x)
+        z [7 8 9]
+        :setv s (+ x y z)
+        :if (!= z 8)
+        (, x y z s))
+      [(, 0 1 7 8) (, 0 1 9 10) (, 0 2 7 9) (, 0 2 9 11)
+        (, 1 2 7 10) (, 1 2 9 12)]]
+    ['(f
+        x [0 1]
+        :setv l []
+        y (range 4)
+        :do (.append l (, x y))
+        :if (>= y 2)
+        z [7 8 9]
+        :if (!= z 8)
+        (, x y (tuple l) z))
+      [(, 0 2 (, (, 0 0) (, 0 1) (, 0 2)) 7)
+        (, 0 2 (, (, 0 0) (, 0 1) (, 0 2)) 9)
+        (, 0 3 (, (, 0 0) (, 0 1) (, 0 2) (, 0 3)) 7)
+        (, 0 3 (, (, 0 0) (, 0 1) (, 0 2) (, 0 3)) 9)
+        (, 1 2 (, (, 1 0) (, 1 1) (, 1 2)) 7)
+        (, 1 2 (, (, 1 0) (, 1 1) (, 1 2)) 9)
+        (, 1 3 (, (, 1 0) (, 1 1) (, 1 2) (, 1 3)) 7)
+        (, 1 3 (, (, 1 0) (, 1 1) (, 1 2) (, 1 3)) 9)]]
+
+    ['(f x (range 4) :do (unless (% x 2) (continue)) (* x 2))
+      [2 6]]
+    ['(f x (range 4) :setv p 9 :do (unless (% x 2) (continue)) (* x 2))
+      [2 6]]
+    ['(f x (range 20) :do (when (= x 3) (break)) (* x 2))
+      [0 2 4]]
+    ['(f x (range 20) :setv p 9 :do (when (= x 3) (break)) (* x 2))
+      [0 2 4]]
+    ['(f x [4 5] y (range 20) :do (when (> y 1) (break)) z [8 9] (, x y z))
+      [(, 4 0 8) (, 4 0 9) (, 4 1 8) (, 4 1 9)
+        (, 5 0 8) (, 5 0 9) (, 5 1 8) (, 5 1 9)]]])
+
+  (for [[expr answer] cases]
+    ; Mutate the case as appropriate for the operator before
+    ; evaluating it.
+    (setv expr (+ (HyExpression [(HySymbol specialop)]) (cut expr 1)))
+    (when (= specialop "dfor")
+      (setv expr (+ (cut expr 0 -1) `([~(get expr -1) 1]))))
+    (when (= specialop "for")
+      (setv expr `(do
+        (setv out [])
+        (for [~@(cut expr 1 -1)]
+          (.append out ~(get expr -1)))
+        out)))
+    (setv result (eval expr))
+    (when (= specialop "dfor")
+      (setv result (.keys result)))
+    (assert (= (sorted result) answer) (str expr)))))
+
+
+(defn test-fors-no-loopers []
+
+  (setv l [])
+  (for [] (.append l 1))
+  (assert (= l []))
+
+  (assert (= (lfor 1) []))
+  (assert (= (sfor 1) #{}))
+  (assert (= (list (gfor 1)) []))
+  (assert (= (dfor [1 2]) {})))
+
+
+(defn test-raise-in-comp []
+  (defclass E [Exception] [])
+  (setv l [])
+  (import pytest)
+  (with [(pytest.raises E)]
+    (lfor
+      x (range 10)
+      :do (.append l x)
+      :do (when (= x 5)
+        (raise (E)))
+      x))
+  (assert (= l [0 1 2 3 4 5])))
+
+
+(defn test-scoping []
+
+  (setv x 0)
+  (for [x [1 2 3]])
+  (assert (= x 3))
+
+  ; An `lfor` that gets compiled to a real comprehension
+  (setv x 0)
+  (assert (= (lfor x [1 2 3] (inc x)) [2 3 4]))
+  (assert (= x (if PY3 0 3)))
+    ; Python 2 list comprehensions leak their variables.
+
+  ; An `lfor` that gets compiled to a loop
+  (setv x 0  l [])
+  (assert (= (lfor x [4 5 6] :do (.append l 1) (inc x)) [5 6 7]))
+  (assert (= l [1 1 1]))
+  (assert (= x 0))
+
+  ; An `sfor` that gets compiled to a real comprehension
+  (setv x 0)
+  (assert (= (sfor x [1 2 3] (inc x)) #{2 3 4}))
+  (assert (= x 0)))
+
+
+(defn test-for-loop []
+  "NATIVE: test for loops"
+  (setv count1 0 count2 0)
+  (for [x [1 2 3 4 5]]
+    (setv count1 (+ count1 x))
+    (setv count2 (+ count2 x)))
+  (assert (= count1 15))
+  (assert (= count2 15))
+  (setv count 0)
+  (for [x [1 2 3 4 5]
+        y [1 2 3 4 5]]
+    (setv count (+ count x y))
+    (else
+      (+= count 1)))
+  (assert (= count 151))
+
+  (setv count 0)
+  ; multiple statements in the else branch should work
+  (for [x [1 2 3 4 5]
+        y [1 2 3 4 5]]
+    (setv count (+ count x y))
+    (else
+      (+= count 1)
+      (+= count 10)))
+  (assert (= count 161))
+
+  ; don't be fooled by constructs that look like else
+  (setv s "")
+  (setv else True)
+  (for [x "abcde"]
+    (+= s x)
+    [else (+= s "_")])
+  (assert (= s "a_b_c_d_e_"))
+
+  (setv s "")
+  (with [(pytest.raises TypeError)]
+    (for [x "abcde"]
+      (+= s x)
+      ("else" (+= s "z"))))
+  (assert (= s "az"))
+
+  (assert (= (list ((fn [] (for [x [[1] [2 3]] y x] (yield y)))))
+             (lfor  x [[1] [2 3]]  y x  y)))
+  (assert (= (list ((fn [] (for [x [[1] [2 3]] y x z (range 5)] (yield z)))))
+             (lfor  x [[1] [2 3]]  y x  z (range 5)  z))))
+
+
+(defn test-nasty-for-nesting []
+  "NATIVE: test nesting for loops harder"
+  ;; This test and feature is dedicated to @nedbat.
+
+  ;; OK. This next test will ensure that we call the else branch exactly
+  ;; once.
+  (setv flag 0)
+  (for [x (range 2)
+        y (range 2)]
+    (+ 1 1)
+    (else (setv flag (+ flag 2))))
+  (assert (= flag 2)))
+
+
+(defn test-empty-for []
+
+  (setv l [])
+  (defn f []
+    (for [x (range 3)]
+      (.append l "a")
+      (yield x)))
+  (for [x (f)])
+  (assert (= l ["a" "a" "a"]))
+
+  (setv l [])
+  (for [x (f)]
+    (else (.append l "z")))
+  (assert (= l ["a" "a" "a" "z"])))
diff --git a/tests/native_tests/contrib/hy_repr.hy b/tests/native_tests/contrib/hy_repr.hy
--- a/tests/native_tests/contrib/hy_repr.hy
+++ b/tests/native_tests/contrib/hy_repr.hy
@@ -144,11 +144,11 @@
 
   (setv x {1 2  3 [4 5]  6 7})
   (setv (get x 3 1) x)
-  (assert (in (hy-repr x) (list-comp
+  (assert (in (hy-repr x) (lfor
     ; The ordering of a dictionary isn't guaranteed, so we need
     ; to check for all possible orderings.
-    (+ "{" (.join "  " p) "}")
-    [p (permutations ["1 2" "3 [4 {...}]" "6 7"])]))))
+    p (permutations ["1 2" "3 [4 {...}]" "6 7"])
+    (+ "{" (.join "  " p) "}")))))
 
 (defn test-matchobject []
   (import re)
diff --git a/tests/native_tests/contrib/walk.hy b/tests/native_tests/contrib/walk.hy
--- a/tests/native_tests/contrib/walk.hy
+++ b/tests/native_tests/contrib/walk.hy
@@ -46,7 +46,7 @@
   (assert (= (macroexpand-all '(with [a 1]))
              '(with* [a 1] (do))))
   (assert (= (macroexpand-all '(with [a 1 b 2 c 3] (for [d c] foo)))
-             '(with* [a 1] (with* [b 2] (with* [c 3] (do (for* [d c] (do foo))))))))
+             '(with* [a 1] (with* [b 2] (with* [c 3] (do (for [d c] foo)))))))
   (assert (= (macroexpand-all '(with [a 1]
                                  '(with [b 2])
                                  `(with [c 3]
diff --git a/tests/native_tests/language.hy b/tests/native_tests/language.hy
--- a/tests/native_tests/language.hy
+++ b/tests/native_tests/language.hy
@@ -176,110 +176,6 @@
   (with [(pytest.raises TypeError)] ("when" 1 2)))    ; A macro
 
 
-(defn test-for-loop []
-  "NATIVE: test for loops"
-  (setv count1 0 count2 0)
-  (for [x [1 2 3 4 5]]
-    (setv count1 (+ count1 x))
-    (setv count2 (+ count2 x)))
-  (assert (= count1 15))
-  (assert (= count2 15))
-  (setv count 0)
-  (for [x [1 2 3 4 5]
-        y [1 2 3 4 5]]
-    (setv count (+ count x y))
-    (else
-      (+= count 1)))
-  (assert (= count 151))
-
-  (setv count 0)
-  ; multiple statements in the else branch should work
-  (for [x [1 2 3 4 5]
-        y [1 2 3 4 5]]
-    (setv count (+ count x y))
-    (else
-      (+= count 1)
-      (+= count 10)))
-  (assert (= count 161))
-
-  ; don't be fooled by constructs that look like else
-  (setv s "")
-  (setv else True)
-  (for [x "abcde"]
-    (+= s x)
-    [else (+= s "_")])
-  (assert (= s "a_b_c_d_e_"))
-
-  (setv s "")
-  (setv else True)
-  (with [(pytest.raises TypeError)]
-    (for [x "abcde"]
-      (+= s x)
-      ("else" (+= s "z"))))
-  (assert (= s "az"))
-
-  (assert (= (list ((fn [] (for [x [[1] [2 3]] y x] (yield y)))))
-             (list-comp y [x [[1] [2 3]] y x])))
-  (assert (= (list ((fn [] (for [x [[1] [2 3]] y x z (range 5)] (yield z)))))
-             (list-comp z [x [[1] [2 3]] y x z (range 5)])))
-
-  (setv l [])
-  (defn f []
-    (for [x [4 9 2]]
-      (.append l (* 10 x))
-      (yield x)))
-  (for [_ (f)])
-  (assert (= l [40 90 20])))
-
-
-(defn test-nasty-for-nesting []
-  "NATIVE: test nesting for loops harder"
-  ;; This test and feature is dedicated to @nedbat.
-
-  ;; let's ensure empty iterating is an implicit do
-  (setv t 0)
-  (for [] (setv t 1))
-  (assert (= t 1))
-
-  ;; OK. This first test will ensure that the else is hooked up to the
-  ;; for when we break out of it.
-  (for [x (range 2)
-        y (range 2)]
-      (break)
-    (else (raise Exception)))
-
-  ;; OK. This next test will ensure that the else is hooked up to the
-  ;; "inner" iteration
-  (for [x (range 2)
-        y (range 2)]
-    (if (= y 1) (break))
-    (else (raise Exception)))
-
-  ;; OK. This next test will ensure that the else is hooked up to the
-  ;; "outer" iteration
-  (for [x (range 2)
-        y (range 2)]
-    (if (= x 1) (break))
-    (else (raise Exception)))
-
-  ;; OK. This next test will ensure that we call the else branch exactly
-  ;; once.
-  (setv flag 0)
-  (for [x (range 2)
-        y (range 2)]
-    (+ 1 1)
-    (else (setv flag (+ flag 2))))
-  (assert (= flag 2))
-
-  (setv l [])
-  (defn f []
-    (for [x [4 9 2]]
-      (.append l (* 10 x))
-      (yield x)))
-  (for [_ (f)])
-  (assert (= l [40 90 20])))
-
-
 (defn test-while-loop []
   "NATIVE: test while loops?"
   (setv count 5)
@@ -934,64 +830,18 @@
 (defn test-for-else []
   "NATIVE: test for else"
   (setv x 0)
-  (for* [a [1 2]]
+  (for [a [1 2]]
     (setv x (+ x a))
     (else (setv x (+ x 50))))
   (assert (= x 53))
 
   (setv x 0)
-  (for* [a [1 2]]
+  (for [a [1 2]]
     (setv x (+ x a))
     (else))
   (assert (= x 3)))
 
 
-(defn test-list-comprehensions []
-  "NATIVE: test list comprehensions"
-  (assert (= (list-comp (* x 2) [x (range 2)]) [0 2]))
-  (assert (= (list-comp (* x 2) [x (range 4)] (% x 2)) [2 6]))
-  (assert (= (sorted (list-comp (* y 2) [(, x y) (.items {"1" 1 "2" 2})]))
-             [2 4]))
-  (assert (= (list-comp (, x y) [x (range 2) y (range 2)])
-             [(, 0 0) (, 0 1) (, 1 0) (, 1 1)]))
-  (assert (= (list-comp j [j [1 2]]) [1 2])))
-
-
-(defn test-set-comprehensions []
-  "NATIVE: test set comprehensions"
-  (assert (instance? set (set-comp x [x (range 2)])))
-  (assert (= (set-comp (* x 2) [x (range 2)]) (set [0 2])))
-  (assert (= (set-comp (* x 2) [x (range 4)] (% x 2)) (set [2 6])))
-  (assert (= (set-comp (* y 2) [(, x y) (.items {"1" 1 "2" 2})])
-             (set [2 4])))
-  (assert (= (set-comp (, x y) [x (range 2) y (range 2)])
-             (set [(, 0 0) (, 0 1) (, 1 0) (, 1 1)])))
-  (assert (= (set-comp j [j [1 2]]) (set [1 2]))))
-
-
-(defn test-dict-comprehensions []
-  "NATIVE: test dict comprehensions"
-  (assert (instance? dict (dict-comp x x [x (range 2)])))
-  (assert (= (dict-comp x (* x 2) [x (range 2)]) {1 2 0 0}))
-  (assert (= (dict-comp x (* x 2) [x (range 4)] (% x 2)) {3 6 1 2}))
-  (assert (= (dict-comp x (* y 2) [(, x y) (.items {"1" 1 "2" 2})])
-             {"2" 4 "1" 2}))
-  (assert (= (dict-comp (, x y) (+ x y) [x (range 2) y (range 2)])
-             {(, 0 0) 0 (, 1 0) 1 (, 0 1) 1 (, 1 1) 2})))
-
-
-(defn test-generator-expressions []
-  "NATIVE: test generator expressions"
-  (assert (not (instance? list (genexpr x [x (range 2)]))))
-  (assert (= (list (genexpr (* x 2) [x (range 2)])) [0 2]))
-  (assert (= (list (genexpr (* x 2) [x (range 4)] (% x 2))) [2 6]))
-  (assert (= (list (sorted (genexpr (* y 2) [(, x y) (.items {"1" 1 "2" 2})])))
-             [2 4]))
-  (assert (= (list (genexpr (, x y) [x (range 2) y (range 2)]))
-             [(, 0 0) (, 0 1) (, 1 0) (, 1 1)]))
-  (assert (= (list (genexpr j [j [1 2]])) [1 2])))
-
-
 (defn test-defn-order []
   "NATIVE: test defn evaluation order"
   (setv acc [])
diff --git a/tests/native_tests/native_macros.hy b/tests/native_tests/native_macros.hy
--- a/tests/native_tests/native_macros.hy
+++ b/tests/native_tests/native_macros.hy
@@ -101,7 +101,7 @@
 (defn test-midtree-yield-in-for []
   "NATIVE: test yielding in a for with a return"
   (defn kruft-in-for []
-    (for* [i (range 5)]
+    (for [i (range 5)]
       (yield i))
     (+ 1 2)))
 
@@ -117,7 +117,7 @@
 (defn test-multi-yield []
   "NATIVE: testing multiple yields"
   (defn multi-yield []
-    (for* [i (range 3)]
+    (for [i (range 3)]
       (yield i))
     (yield "a")
     (yield "end"))
diff --git a/tests/native_tests/py36_only_tests.hy b/tests/native_tests/py36_only_tests.hy
--- a/tests/native_tests/py36_only_tests.hy
+++ b/tests/native_tests/py36_only_tests.hy
@@ -13,7 +13,7 @@
   (.run_until_complete (get-event-loop) (coro)))
 
 
-(defn test-for/a []
+(defn test-for-async []
   (defn/a numbers []
     (for [i [1 2]]
       (yield i)))
@@ -21,11 +21,11 @@
   (run-coroutine
     (fn/a []
       (setv x 0)
-      (for/a [a (numbers)]
+      (for [:async a (numbers)]
         (setv x (+ x a)))
       (assert (= x 3)))))
 
-(defn test-for/a-else []
+(defn test-for-async-else []
   (defn/a numbers []
     (for [i [1 2]]
       (yield i)))
@@ -33,7 +33,7 @@
   (run-coroutine
     (fn/a []
       (setv x 0)
-      (for/a [a (numbers)]
+      (for [:async a (numbers)]
         (setv x (+ x a))
         (else (setv x (+ x 50))))
       (assert (= x 53)))))
diff --git a/tests/native_tests/py3_only_tests.hy b/tests/native_tests/py3_only_tests.hy
--- a/tests/native_tests/py3_only_tests.hy
+++ b/tests/native_tests/py3_only_tests.hy
@@ -49,7 +49,7 @@
 (defn test-yield-from []
   "NATIVE: testing yield from"
   (defn yield-from-test []
-    (for* [i (range 3)]
+    (for [i (range 3)]
       (yield i))
     (yield-from [1 2 3]))
   (assert (= (list (yield-from-test)) [0 1 2 1 2 3])))
@@ -63,7 +63,7 @@
     (yield 3)
     (assert 0))
   (defn yield-from-test []
-    (for* [i (range 3)]
+    (for [i (range 3)]
       (yield i))
     (try
       (yield-from (yield-from-subgenerator-test))
diff --git a/tests/native_tests/tag_macros.hy b/tests/native_tests/tag_macros.hy
--- a/tests/native_tests/tag_macros.hy
+++ b/tests/native_tests/tag_macros.hy
@@ -110,7 +110,7 @@
     ((wraps func)
        (fn [&rest args &kwargs kwargs]
          (func #* (map inc args)
-               #** (dict-comp k (inc v) [[k v] (.items kwargs)])))))
+               #** (dfor [k v] (.items kwargs) [k (inc v)])))))
 
   #@(increment-arguments
      (defn foo [&rest args &kwargs kwargs]
diff --git a/tests/resources/pydemo.hy b/tests/resources/pydemo.hy
--- a/tests/resources/pydemo.hy
+++ b/tests/resources/pydemo.hy
@@ -29,10 +29,10 @@ Call me Ishmael. Some years ago—never mind how long precisely—having little
 (setv myset #{4 5 6})
 (setv mydict {7 8  9 900  10 15})
 
-(setv mylistcomp (list-comp x [x (range 10)] (% x 2)))
-(setv mysetcomp (set-comp x [x (range 5)] (not (% x 2))))
-(setv mydictcomp (dict-comp k (.upper k) [k "abcde"] (!= k "c")))
-(setv mygenexpr (genexpr x [x (cycle [1 2 3])] (!= x 2)))
+(setv mylistcomp (lfor x (range 10) :if (% x 2) x))
+(setv mysetcomp (sfor x (range 5) :if (not (% x 2)) x))
+(setv mydictcomp (dfor k "abcde" :if (!= k "c") [k (.upper k)]))
+(setv mygenexpr (gfor x (cycle [1 2 3]) :if (!= x 2) x))
 
 (setv attr-ref str.upper)
 (setv subscript (get "hello" 2))