Repository: numba/numba
Pull request: Detect and raise unsupported on generator expressions
As title.

Closes #4271

<!--

Thanks for wanting to contribute to Numba :)

First, if you need some help or want to chat to the core developers, please
visit https://gitter.im/numba/numba for real time chat or post to the Numba
mailing list https://groups.google.com/a/continuum.io/forum/#!forum/numba-users.

Here's some guidelines to help the review process go smoothly.

0. Please write a description in this text box of the changes that are being
   made.

1. Please ensure that you have written units tests for the changes made/features
   added.

2. If you are closing an issue please use one of the automatic closing words as
   noted here: https://help.github.com/articles/closing-issues-using-keywords/

3. If your pull request is not ready for review but you want to make use of the
   continuous integration testing facilities please label it with `[WIP]` and
   then remove the label when you'd like it to be reviewed.

4. Once review has taken place please do not add features or make changes out of
   the scope of those requested by the reviewer (doing this just add delays as
   already reviewed code ends up having to be re-reviewed/it is hard to tell
   what is new etc!). Further, please do not rebase your branch on master/force
   push/rewrite history, doing any of these causes the context of any comments
   made by reviewers to be lost. If conflicts occur against master they should
   be resolved by merging master into the branch used for making the pull
   request.

Many thanks in advance for your cooperation!

-->


Problem statement:
Lowering error with Numpy.fromiter
Error message:

```
numba.errors.LoweringError: Failed in object mode pipeline (step: object mode backend)
'NoneType' object has no attribute 'yield_points'

File "code/utility.py", line 96:
def error():
    <source elided>
    sampled = np.zeros(10, dtype=types)
    res = np.fromiter((func(x) for x in sampled), dtype=int)
    ^

[1] During: lowering "$4.5.117 = yield $4.4" at code/utility.py (96)
```


Reproducer:

```
def func(x):
    return x[0]

@jit
def error():
    topo = [chr(ord('a') + i) for i in range(5)]
    types = [(var_name, "int") for var_name in topo]
    sampled = np.zeros(10, dtype=types)
    res = np.fromiter((func(x) for x in sampled), dtype=int)

error()
```

Hints from issue discussion:
Similar error, another reproducer:

```
@jit
def error():
    topo = [chr(ord('a') + i) for i in range(5)]
    types = [(var_name, "int") for var_name in topo]
    sampled = np.zeros(10, dtype=types)
    issue = tuple(sampled[i] for i in top[:3])

error()
```
@yujianll thanks for reporting this! We appreciate you using Numba and taking the time to open issues. I will reformat your comments for legibility and queue this for inspection.

I actually get a slightly different error:

```
vhaenel@parraline [numba] [numba:typed_list:★ ★ ★ ] ~/git/numba
💥 zsh» python foo11.py
Traceback (most recent call last):
  File "foo11.py", line 17, in <module>
    error()
  File "/Users/vhaenel/git/numba/numba/dispatcher.py", line 351, in _compile_for_args
    error_rewrite(e, 'typing')
  File "/Users/vhaenel/git/numba/numba/dispatcher.py", line 316, in error_rewrite
    raise e
  File "/Users/vhaenel/git/numba/numba/dispatcher.py", line 327, in _compile_for_args
    return self.compile(tuple(argtypes))
  File "/Users/vhaenel/git/numba/numba/compiler_lock.py", line 32, in _acquire_compile_lock
    return func(*args, **kwargs)
  File "/Users/vhaenel/git/numba/numba/dispatcher.py", line 659, in compile
    cres = self._compiler.compile(args, return_type)
  File "/Users/vhaenel/git/numba/numba/dispatcher.py", line 83, in compile
    pipeline_class=self.pipeline_class)
  File "/Users/vhaenel/git/numba/numba/compiler.py", line 955, in compile_extra
    return pipeline.compile_extra(func)
  File "/Users/vhaenel/git/numba/numba/compiler.py", line 377, in compile_extra
    return self._compile_bytecode()
  File "/Users/vhaenel/git/numba/numba/compiler.py", line 886, in _compile_bytecode
    return self._compile_core()
  File "/Users/vhaenel/git/numba/numba/compiler.py", line 873, in _compile_core
    res = pm.run(self.status)
  File "/Users/vhaenel/git/numba/numba/compiler_lock.py", line 32, in _acquire_compile_lock
    return func(*args, **kwargs)
  File "/Users/vhaenel/git/numba/numba/compiler.py", line 254, in run
    raise patched_exception
  File "/Users/vhaenel/git/numba/numba/compiler.py", line 245, in run
    stage()
  File "/Users/vhaenel/git/numba/numba/compiler.py", line 501, in stage_nopython_frontend
    self.locals)
  File "/Users/vhaenel/git/numba/numba/compiler.py", line 1104, in type_inference_stage
    infer.build_constraint()
  File "/Users/vhaenel/git/numba/numba/typeinfer.py", line 877, in build_constraint
    self.constrain_statement(inst)
  File "/Users/vhaenel/git/numba/numba/typeinfer.py", line 1156, in constrain_statement
    self.typeof_assign(inst)
  File "/Users/vhaenel/git/numba/numba/typeinfer.py", line 1227, in typeof_assign
    self.typeof_global(inst, inst.target, value)
  File "/Users/vhaenel/git/numba/numba/typeinfer.py", line 1324, in typeof_global
    typ = self.resolve_value_type(inst, gvar.value)
  File "/Users/vhaenel/git/numba/numba/typeinfer.py", line 1248, in resolve_value_type
    raise TypingError(msg, loc=inst.loc)
numba.errors.TypingError: Failed in nopython mode pipeline (step: nopython frontend)
Untyped global name 'chr': cannot determine Numba type of <class 'builtin_function_or_method'>

File "foo11.py", line 11:
def error():
    topo = [chr(ord('a') + i) for i in range(5)]
```

But this is using `nopython` mode.
Using plain `@jit` I can reproduce the error you mentioned.
Yeah I tried both modes. I want to use fromiter to apply function to each row of Numpy array in the first case, and construct a tuple in the second case.
Thanks for the report. The issue in this function (and the other reproducer):
```python
@jit
def error():
    topo = [chr(ord('a') + i) for i in range(5)]
    types = [(var_name, "int") for var_name in topo]
    sampled = np.zeros(10, dtype=types)
    res = np.fromiter((func(x) for x in sampled), dtype=int)

error()
```
is that Numba does not support generator comprehensions, switching the arg to `np.fromiter` to a list comprehension fixes it, this works fine:
```python
from numba import jit
import numpy as np

def func(x):
    return x[0]

@jit
def error():
    topo = [chr(ord('a') + i) for i in range(5)]
    types = [(var_name, "int") for var_name in topo]
    sampled = np.zeros(10, dtype=types)
    res = np.fromiter([func(x) for x in sampled], dtype=int)

error()
```
am going to note this as a bug because the lowering error should not have happened as the unsupported functionality should have been caught earlier.
Thanks! So I cannot use generator comprehensions even in plain `@jit` mode?
@yujianll running the following suggests that you are correct:

```
from numba import jit


@jit
def error():
    gen = (i**2 for i in range(10))
    print(gen)


error.py_func()
error()
```

The first---the original python function---works and prints something about the generator, whereas the second one fails.
Thanks!

Commit message: Handle funcsigs failing to find signature

Patch to review:
diff --git a/numba/ir_utils.py b/numba/ir_utils.py
index ec5fbe231c2..9c4431c2e1b 100644
--- a/numba/ir_utils.py
+++ b/numba/ir_utils.py
@@ -1926,6 +1926,12 @@ def raise_on_unsupported_feature(func_ir, typemap):
                     isinstance(ty, types.DictType)):
                     raise TypingError(msg % (ty, stmt.value.name, ty), loc=stmt.loc)
 
+            # checks for generator expressions (yield in use when func_ir has
+            # not been identified as a generator).
+            if isinstance(stmt.value, ir.Yield) and not func_ir.is_generator:
+                msg = "The use of generator expressions is unsupported."
+                raise UnsupportedError(msg, loc=stmt.loc)
+
     # There is more than one call to function gdb/gdb_init
     if len(gdb_calls) > 1:
         msg = ("Calling either numba.gdb() or numba.gdb_init() more than once "
diff --git a/numba/tests/test_closure.py b/numba/tests/test_closure.py
index 0e7fdb7c611..534d00e1de2 100644
--- a/numba/tests/test_closure.py
+++ b/numba/tests/test_closure.py
@@ -435,10 +435,10 @@ def inner(x):
         msg = "with argument(s) of type(s): (none)"
         self.assertIn(msg, str(raises.exception))
 
-        with self.assertRaises(LoweringError) as raises:
+        with self.assertRaises(UnsupportedError) as raises:
             cfunc = jit(nopython=True)(outer17)
             cfunc(var)
-        msg = "'NoneType' object has no attribute 'yield_points'"
+        msg = "The use of generator expressions is unsupported."
         self.assertIn(msg, str(raises.exception))
 
         with self.assertRaises(TypingError) as raises:
diff --git a/numba/tests/test_errorhandling.py b/numba/tests/test_errorhandling.py
index eda3e1e9e51..337edf999ce 100644
--- a/numba/tests/test_errorhandling.py
+++ b/numba/tests/test_errorhandling.py
@@ -168,6 +168,21 @@ def foo():
                     "has it been imported")
         self.assertIn(expected, str(raises.exception))
 
+    def test_handling_unsupported_generator_expression(self):
+        def foo():
+            y = (x for x in range(10))
+
+        if utils.IS_PY3:
+            expected = "The use of generator expressions is unsupported."
+        else:
+            # funcsigs falls over on py27
+            expected = "Cannot obtain a signature for"
+
+        for dec in jit(forceobj=True), njit:
+            with self.assertRaises(errors.UnsupportedError) as raises:
+                dec(foo)()
+            self.assertIn(expected, str(raises.exception))
+
 
 class TestConstantInferenceErrorHandling(unittest.TestCase):
 
diff --git a/numba/utils.py b/numba/utils.py
index 64ee20f7353..a08ba78580a 100644
--- a/numba/utils.py
+++ b/numba/utils.py
@@ -15,6 +15,7 @@
 import numpy as np
 
 from .six import *
+from .errors import UnsupportedError
 try:
     # preferred over pure-python StringIO due to threadsafety
     # note: parallel write to StringIO could cause data to go missing
@@ -77,11 +78,18 @@ def erase_traceback(exc_value):
     from inspect import Parameter as pyParameter
 except ImportError:
     try:
-        from funcsigs import signature as pysignature
+        from funcsigs import signature as _pysignature
         from funcsigs import Signature as pySignature
         from funcsigs import Parameter as pyParameter
         from funcsigs import BoundArguments
 
+        def pysignature(*args, **kwargs):
+            try:
+                return _pysignature(*args, **kwargs)
+            except ValueError:
+                msg = ("Cannot obtain a signature for:", args)
+                raise UnsupportedError(msg)
+
         # monkey patch `apply_defaults` onto `BoundArguments` cf inspect in py3
         # This patch is from https://github.com/aliles/funcsigs/pull/30/files
         # with minor modifications, and thanks!


Code review comments:
diff_hunk: @@ -77,11 +78,18 @@ def erase_traceback(exc_value):
     from inspect import Parameter as pyParameter
 except ImportError:
     try:
-        from funcsigs import signature as pysignature
+        from funcsigs import signature as _pysignature
         from funcsigs import Signature as pySignature
         from funcsigs import Parameter as pyParameter
         from funcsigs import BoundArguments
 
+        def pysignature(*args, **kwargs):
+            try:
+                return _pysignature(*args, **kwargs)
+            except ValueError:
+                msg = ("Cannot obtain a signature for:", args)
+                raise UnsupportedError(msg)
path: numba/utils.py
text: @user1:
Better to include the original message, which will contain " '.0' is not a valid parameter name". 

Also,  it is weird that python2 is generating that a arg with `.0` as the name, and that `funcsigs` can't handle it.

@author:
> Better to include the original message, which will contain " '.0' is not a valid parameter name".

Good idea

> Also, it is weird that python2 is generating that a arg with `.0` as the name, and that `funcsigs` can't handle it.

Yes, this was weird. That you have seen the `.0` confirms it is not just me. Looking at the byte code disassembly, it's definitely in there!