Repository: cython/cython

Issue:
asyncio.iscoroutinefunction returns false for Cython async function objects
Referring to #2092 

@scoder the implementation in asyncio is ridiculous

```
def iscoroutinefunction(func):
    """Return True if func is a decorated coroutine function."""
    return (inspect.iscoroutinefunction(func) or
            getattr(func, '_is_coroutine', None) is _is_coroutine)
```

So once we agreed `inspect` returns False, it happens to be that CPython just add a simple attribute to the function `_is_coroutine`, I think if Cython does this adding up this attribute it will make everything work

Here: https://github.com/python/cpython/blob/789e359f51d2b27bea01b8c6c3bf090aaedf8839/Lib/asyncio/coroutines.py#L152

And here: 
https://github.com/python/cpython/blob/789e359f51d2b27bea01b8c6c3bf090aaedf8839/Lib/asyncio/coroutines.py#L160

So the solution as it can't be another it results in this, from my test in iPython:

```
In [46]: fut = init_connection()
In [47]: asyncio.iscoroutine(fut)
Out[47]: True

In [49]: init_connection.__class__
Out[49]: cython_function_or_method

In [50]: asyncio.iscoroutinefunction(init_connection)
Out[50]: False

In [56]: from asyncio.coroutines import _is_coroutine

In [57]: init_connection._is_coroutine = _is_coroutine

In [58]: asyncio.iscoroutinefunction(init_connection)
Out[58]: True
```

Dirty but easy. Don't know if this is a 5 seconds issue or a hara-kiri, theoretically it should be easy to plug the attribute runtime from `from asyncio.coroutines import _is_coroutine`, or not.

This is a super annoying problem (I don't blame anyone with the word _bug_) that make cythonized code to be very dramatic in several asyncio frameworks. `asyncio.iscoroutine` work well with cythonized funcs for the silliest reason I think.


Following the `cyfunctions` above, `create_token` is NOT a coroutine, while `init_connection` is an `async` function, **this workaround works in both so it's ok if you can schedule the functions whether are coros or not**, no way to easily fix this with `asyncio.iscoroutinefunction` or at least easier than that:

```
In [59]: async def test():
    ...:     await create_token({})
    ...:
In [61]: asyncio.get_event_loop().run_until_complete(test())
TypeError: object str can't be used in 'await' expression

In [73]: async def test():
    ...:     return await init_connection()
Out[74]: <asyncpg.pool.Pool at 0x7f78e04752c8>
```

Here comes the magic:

```
In [69]: async def test():
    ...:     return await asyncio.coroutine(create_token)({})
In [70]: asyncio.get_event_loop().run_until_complete(test())
Out[70]: 'eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzUxMiJ9.eyJleHAiOjE1MjU1NTk0NTB9.v8wRZGIke6RYizIpJhl4oaypKyvuVARKaiq0KdpJg6XJ0qTB0o76BuTLera6kSQ_5qXnAb7_9DQSadwdRqgPmw'

In [71]: async def test():
    ...:     return await asyncio.coroutine(init_connection)()
In [72]: asyncio.get_event_loop().run_until_complete(test())
Out[72]: <asyncpg.pool.Pool at 0x7f78df48af48>
```

With that the workaround:

```python
def nasty_iscoroutinefunction(func):
      is_coro = asyncio. iscoroutinefunction(func)
      if func.__class__.__name__ == 'cython_function_or_method':
            # It's cythonized, danger!
            is_coro = True  # We'll make it possible
            func = asyncio.coroutine(func)
      return is_coro
```

This works!


Discussion:
Found an interesting, better and working workaround:

https://github.com/timothycrosley/hug/blob/develop/hug/introspect.py#L33

```python
def is_coroutinefunction(function):
    """Returns True if the passed in function is a coroutine"""
    return function.__code__.co_flags & 0x0080 or getattr(function, '_is_coroutine', False)
```

Is there plans to fix this?
@danigosa, AFAIR you suggest to replace `asyncio.iscoroutinefunction` in `asyncio` module, but I think this should be fixed on cython side instead, so the original `asyncio` method will work as expected.
Hm, it looks like [tests](https://github.com/cython/cython/blob/master/tests/run/test_coroutines_pep492.pyx#L876) with code similar to [inspect.py](https://github.com/python/cpython/blob/789e359f51d2b27bea01b8c6c3bf090aaedf8839/Lib/inspect.py#L180) are disabled.
The problem is that if we make Cython coroutines look too much like Python coroutines at the C level (e,g, by setting the `CO_COROUTINE` flag on them), then CPython might start doing things with them that it shouldn't. Even if we find that it doesn't do that right now, it might change at any time, and then people have their Cython modules out there that break. So this is risky business.

Similarly, unconditionally patching `inspect` might have unfortunate side effects as well, even if requires an opt-in. It's impossible to guess what mixture of code end users will have on their systems.
@scoder 
I'm half way to make PR to add `CO_COROUTINE` to coroutine wrappers. But if it will not be accepted anyway, I will stop.

AFAIR, right now c wrapper looks like:
``` python
def c_wrapper(...):
    return coroutine(...)
```
It looks like this can be changed to
``` python
@types.coroutine
def c_wrapper(...):
    return (yield from coroutine(...))
```
In this case all stuff is on python shoulders.
@scoder 
I have made some dirty hack and it looks like it works. Any thoughts about such solution?
https://github.com/pohmelie/cython/commit/5a84869a731c848d135bf098180b5711e18ca6e3
``` python
import asyncio


async def foo():
    pass


print("foo", foo.__code__.co_flags)
print("foo", asyncio.iscoroutinefunction(foo))
print(__cython_coroutine_wrapper)
```
compiled code output:
```
('foo', 319)
('foo', True)
<function coroutine at 0x7f30b9a0e620>
```
Linked code is not working in all cases, it is just quick proof of concept.
By the way, I can't find any problem with add `CO_COROUTINE` flag, since
``` python
async def afoo():
    ...
```
is same as
``` python
def foo():
    return afoo()
```
you can `await foo()` and it will be absolutely same as `await afoo()`.
There is at least this code in CPython, which I'd rather not want to trigger for Cython coroutines:
https://github.com/python/cpython/blob/caba55b3b735405b280273f7d99866a046c18281/Python/ceval.c#L3906-L3949
Can't say right now if it's possible to do that from Python code, probably via eval&friends.

Also, what we are discussing here actually applies to these three functions in `inspect.py`:  `isgeneratorfunction()`, `iscoroutinefunction()` and `isasyncgenfunction()`.

https://github.com/python/cpython/blob/41254ebd5e4f40a2e095d8aaea60bf3973de4647/Lib/inspect.py#L171-L194
But why old-style `yield from` coroutines with `asyncio.coroutine` works fine? Why we can't just wrap new-style coroutines with same decorator on building ast step?
@scoder Can you consider adding new flags like `CO_CYTHON_COROUTINE`?

I'm using Starlette, [it requires asyncio.iscoroutinefunction() to distinguish coroutine functions](https://github.com/encode/starlette/blob/474b499141f2a2b6b7d7c203b8d1a248c45cb160/starlette/routing.py#L35). I can't fix it because Cython hides everything about coroutine functions.
Can't starlette do a type-check against the Coroutine ABC instead?
How can I know a function will return a coroutine without call it?
> How can I know a function will return a coroutine without call it?

You cannot. There is no way to do that. It's called the "halting problem".
https://en.wikipedia.org/wiki/Halting_problem
But CPython can:
```python
>>> async def f():
...   pass
...
>>> import inspect
>>> inspect.iscoroutinefunction(f)
True
>>> import asyncio
>>> asyncio.iscoroutinefunction(f)
True
```
No, it can't:
```python
>>> async def func(): pass
>>> import inspect
>>> inspect.iscoroutinefunction(func)
True
>>> def retfunc(): return func()
>>> inspect.iscoroutinefunction(retfunc)
False
>>> type(func())
<class 'coroutine'>
>>> type(retfunc())
<class 'coroutine'>
```
Is there currrently a way to test if a cython function or method is async? Without that, we can't programmatically fix it even on a case-by-case basis...
There doesn't have to be anything special about a callable that returns an awaitable. It does not have to be an async function. It can be any callable, even including a class. The idea of guessing the return type of a callable, without actually looking at the type of the object that it returns, seems inherently flawed to me.

IMHO, the work-around is to call the callable and look at the value it returns. If that is something to await, then await it. If not, then not.

However, as a clean fix, I would suggest to _always_ require/pass/return something that is awaitable, and not handle the callables differently based on _their_ type.
> the work-around is to call the callable and look at the value it returns

I think it is bad. Since this promotes bad practice of callable returns awaitable vs awaitable is callable. This should not work like this.
I don't really care that a regular function may return an awaitable, I care about compatibility (or a possibility of compatibility) with CPython. For CPython, `iscoroutinefunction` returns true if the function was defined with async. As far as I can tell, Cython does not expose this information in any way.

There may be reasons for `object.__code__.co_flags & CO_COROUTINE` to not be true for async functions in Cython - but at a minimum *there must be a programmatic way to determine that a function was defined with async*.
I'll accept a PR that adds an `_is_coroutine` flag to `CyFunction` and sets it for async functions.

~~There are already tests in `test_coroutine_pep492.pyx` that have their call to `inspect.iscoroutinefunction()` commented out. Note that this file is copied from CPython, so don't change it too much. Just enable those lines for the CPython versions (`sys.version_info`) that support them.~~

EDIT: sorry, we are aiming for `asyncio.iscoroutinefunction()` in this ticket, not `inspect.iscoroutinefunction()`. The test to extend is `tests/run/asyncio_generators.srctree`.
@pohmelie "callable returns awaitable vs awaitable is callable" – note that for async functions, the callable is the function, and the awaitable is the return value of the call. An async function (object) is not awaitable.
> There doesn't have to be anything special about a callable that returns an awaitable.
> (...)
> IMHO, the work-around is to call the callable and look at the value it returns. If that is something to await, then await it. If not, then not.

@scoder for good or bad, Python really does not make a lot of distinctions between async / sync functions. When writing tests though, we do mock things that maybe sync / async and calling them to see if they are sync / async is just not an option. Knowing *before* calling, is extremely useful, as it enables mocks to detect bugs (eg: mocking an async function with a sync function).

This is how I bumped into this bug. I work on [TestSlide](https://github.com/facebookincubator/TestSlide), which Facebook open sourced last year. The gist of it is that its mocks have significantly more interface validation than vanilla `Mock(spec=...)`. When asyncio support was added recently, `iscoroutinefunction` was used to [validate mocking of sync / async methods](https://github.com/facebookincubator/TestSlide), which turned out to be completely broken for [Thrift](https://github.com/facebook/fbthrift) clients, which are all Cython async.

Perhaps having `cython.iscoroutinefunction` would be simpler to implement, so we can start using right on? I have limited bandwidth at the moment to deeply dig into this, but with some nice pointers to where to look, I might be able to craft a PR for this.
Just for tracking, the corresponding python issue:

https://bugs.python.org/issue38225

And a related cython issue: #3143
Repeating my comment above here, so that people know what the status of this ticket is:
I'll accept a PR that adds an _is_coroutine flag to CyFunction and sets it for async functions.

I also agree with the proposal in https://bugs.python.org/issue38225 that there should be an actual protocol for this.

Resolving patch:
diff --git a/Cython/Compiler/ExprNodes.py b/Cython/Compiler/ExprNodes.py
--- a/Cython/Compiler/ExprNodes.py
+++ b/Cython/Compiler/ExprNodes.py
@@ -9436,6 +9436,9 @@ def generate_cyfunction_code(self, code):
         if def_node.local_scope.parent_scope.is_c_class_scope and not def_node.entry.is_anonymous:
             flags.append('__Pyx_CYFUNCTION_CCLASS')
 
+        if def_node.is_coroutine:
+            flags.append('__Pyx_CYFUNCTION_COROUTINE')
+
         if flags:
             flags = ' | '.join(flags)
         else:
diff --git a/Cython/Compiler/Nodes.py b/Cython/Compiler/Nodes.py
--- a/Cython/Compiler/Nodes.py
+++ b/Cython/Compiler/Nodes.py
@@ -1665,6 +1665,8 @@ class FuncDefNode(StatNode, BlockNode):
     needs_outer_scope = False
     pymethdef_required = False
     is_generator = False
+    is_coroutine = False
+    is_asyncgen = False
     is_generator_body = False
     is_async_def = False
     modifiers = []
@@ -4308,9 +4310,7 @@ class GeneratorDefNode(DefNode):
     #
 
     is_generator = True
-    is_coroutine = False
     is_iterable_coroutine = False
-    is_asyncgen = False
     gen_type_name = 'Generator'
     needs_closure = True
 


Test patch:
diff --git a/tests/run/py35_asyncio_async_def.srctree b/tests/run/py35_asyncio_async_def.srctree
--- a/tests/run/py35_asyncio_async_def.srctree
+++ b/tests/run/py35_asyncio_async_def.srctree
@@ -1,5 +1,5 @@
 # mode: run
-# tag: asyncio, gh1685
+# tag: asyncio, gh1685, gh2273
 
 PYTHON setup.py build_ext -i
 PYTHON main.py
@@ -19,6 +19,7 @@ setup(
 
 import asyncio
 import cy_test
+import py_test
 from contextlib import closing
 
 async def main():
@@ -31,6 +32,12 @@ with closing(asyncio.get_event_loop()) as loop:
     print("Running Cython coroutine ...")
     loop.run_until_complete(cy_test.say())
 
+assert asyncio.iscoroutinefunction(cy_test.cy_async_def_example) == True
+assert asyncio.iscoroutinefunction(cy_test.cy_async_def_example) == True
+assert asyncio.iscoroutinefunction(py_test.py_async_def_example) == True
+assert asyncio.iscoroutinefunction(py_test.py_async_def_example) == True
+assert asyncio.iscoroutinefunction(cy_test.cy_def_example) == False
+assert asyncio.iscoroutinefunction(py_test.py_def_example) == False
 
 ######## cy_test.pyx ########
 
@@ -51,8 +58,19 @@ async def cb():
     await asyncio.sleep(0.5)
     print("done!")
 
+async def cy_async_def_example():
+    return 1
+
+def cy_def_example():
+    return 1
 
 ######## py_test.py ########
 
 async def py_async():
     print("- and this one is from Python")
+
+async def py_async_def_example():
+    return 1
+
+def py_def_example():
+    return 1