JukkaL/mypy — pull request #367 by rockneurotiko (2014-08-09 14:53:11+00:00)
Title: Yield from

This is a text to explain how yield from works, and why I took some decisions :-)
(In the commits, are so much "white lines" changed, that's because in the original source code are lines with just spaces, or spaces at the end of the line, and my editor (Subl3) remove that)

Yield From
====

**yield from** can be applied to **Iterables** and **Futures** (A kind of Iterable).

And can be applied in two ways:

* Statement:
```python
def h():
yield from x()
```
* Expression:
```python
def h():
y = yield from x()
```

About Iterables:
------

##### Statement:
Check if the return type of the function is an iterable and the infered of the iterable applied is compatible with the return type.
Ex:
```python
def h() -> Iterator[int]:
yield from [1, 2, 3] # Here, List is an Iterable, and have ints inside.

x = h() # the type will be Iterator[int]
for i in x:
print(i)
```
In the example above, the function return a Iterator of ints, that is a supertype of Iterable.
Here another example with a function Iterator:
```python
def h() -> Iterator[int]:
yield from h2()
def h2() -> Iterator[int]:
yield 1
yield 2
yield 3
x = h() # the type will be Iterator[int]
for i in x:
print(i)
```

##### Expression:

Right now just checks if the applied is an Iterable.
In the future, when other things are added to mypy, here should check the
return type, and check the types (just like in statement).
The things that mypy needs to add this to the expression type check:

* Function with yields have a return.
```python
def h() -> Iterator[int]:
yield 1
return 2 # Here now fail, but this is a valid syntax

def h2() -> Iterator[int]:
x = yield from h() # Here when iterate will send the h() iterator
# and when the iterator is finished,
# will get the return value in x
```
* yield statement can be an expression assignment.
```python
def h() -> Iterator[int]:
y = yield 1
```

This two things have interaction with the new **send()**, **close()** and **throw()** method of the iterables (this should have added too)

About Futures.
--------

I said that Futures are a "kind" of iterables, because it have an \_\_iter\_\_ function but we never iterate over there (we just can do one next(), if try to do any more raises an error). The main loop will catch the futures that had been "yielded from" or that are inserted directly (call_soon, run_until_complete, ...)

So, the idea that I had to write the Future type, and that I've implemented is that one:

* Future type is writed like Iterator: 'Future[Some]'
* But also can be a instance, but have to say the type inside:
```python
x = Future() # type: Future[int]
```
* If the function have a Future type as type returned, that function can be "yielded from".
* If a function have the @coroutine decorator, or **"yield from"** a Future inside, then, that function can be "yielded from", and the type must be 'Future[Some]'. You can always return 'Some', and will be wrapped inside a Future type. (Examples after)
* When yield from a 'Future[Some]', the type returned to a variable assignment will be 'Some', and not the 'Future[Some]'. Right here an example, using **asyncio.sleep(float, T)**, that have this return type: **'Future[T]'**:

```python
x = asyncio.sleep(100, '1') # the function will return the type:'Future[str]'
# (and x will be of that type)
# won't sleep (just called, not "yielded from",
# but we doesn't care about that)
y = yield from x # 'x' is a 'Future[str]',and we do a "yield from" to it
# so, 'y' will be 'str' type (take out the Future)
# (When executed the code, will sleep here, because we
# give the future to the main loop and execute it)
```

So, if "yield from" is applied to a Future, we won't get it (the loop will).
But if we wan't to do a function that can be "yielded from" (this is called coroutine), then the function need to have the return type as Future[Some], and the @asyncio.coroutine decorator (this is explained after)

##### Statement:

We just don't care about it, we won't get the Future, the loop will, so we don't need to check anything.
Ex:
```python
@asyncio.coroutine
def h() -> 'Future[int]':
yield from asyncio.sleep(2)
return 1
```

We set the return type as **Future[int]** because we want to let know that this is a coroutine and can be yielded from but, also, that will return an int.
If you do: **"y = yield from h()"**, the program will sleep 2 seconds, and then y will be 1 (the return value).

Actually, if you just call the function, and not "yield from" it, you will have a generator (but can't iterate over there, that raises an exception)
```python
y = yield from h() # type: int # this is infered
z = h() # type: Future[int] # this is infered too
# now you can "yield from z" because it's a Future
for i in z: # This raises an exception, we can't iterate
print(i)
```

##### Expression:

When we check the expression yield from, if we see that the type applied is a **Future[Some]**, then, the type to return (to assignment_stmt) won't be **Future[Some]**, instead of that, we'll return the type **Some**, because, as said, the Future is just for know where are a coroutine, or where can be "yielded from", but in the assignment expression, the variable get the type returned, not the Future.
Ex:

```python
@asyncio.coroutine # say that is a coroutine
def h() -> 'Future[str]': # this will be a future, so must be yielded from
# and is a coroutine too
return 'hello!' # Here we return a str, but see that
# is a coroutine, and will wrap the
# str in a Future[str]

@asyncio.coroutine
def h2() -> 'Future[None]':
x = yield from h() # Here, we can yield from because h is a Future,
# but 'x' won't be 'Future[str]', it will
# be str
print(x)
```

Other things of Futures
--------

* Can I **"yield from"** a func?.
* If the function returns **Future[Some]**, yes, you can.
* How wrap the return type into a Future in a function.
* The **@asyncio.coroutine** decorator or **yield from** a Future is needed, when some of that happens, the function is setted as coroutine, and the return type will try to wrap it in a Future.
* Count how many **"Futures"** have the function return type against the type returned, and if the difference is up to one, will wrap the type returned.
Ex:
```python
def h() -> Future[int]:
return 1 # Here, the difference of "Futures" is one, so the
# int returned will be wrapped
# inside a Future as Future[int]

def h() -> Future[int]:
return asyncio.Future() # type: Future[int]
# Here nothing is wrapped because the difference is 0
# but the type check will fail, because the first
# Future[] of the return type is removed when "yielded from"
# so, that function would need another Future[] in the function type

def h() -> Future[int]:
a = asyncio.Future() # type: Future[Future[int]]
return a
# Here nothing is wrapped, but the type check will fail
# because a Future with an int was expected
# and a Future with a Future was given.
```
* The **Future** type used comes from **asyncio.futures.Future**, and the type definition is in **"stubs/3.4/asyncio/futures.py"** so, you will need to do **"from asyncio import Future"**, or use **"asyncio.Future"** instead of **"Future"**.
For that reason, the type have to be writted as string ( **'Future[int]'** ), if it's not writted in that way, the script will fail when try to run it, because the real Future type is not subscritable.
I think that in the future, can be a type in **"typing"**, called **Futur** or some like that, to can write the type without the string.
Because of that (**Future** comes from **asyncio**), now the tests to Futures can't be written (fails trying to import all things)
Anyway, in **stubs/3.4/asyncio** I added a directory called **"examples"** where are so much examples about **Futures**.
* What if I want a Future that have a Future inside, that have a Future inside, ...?
You have to do it like if you have a Iterator that return an Iterator, ...:

```python
from typing import Iterator

def h4(x: int) -> Iterator[int]:
yield x
yield x+1

def h3(x: int) -> Iterator[Iterator[int]]:
yield h4(x)
yield h4(x+2)

def h() -> Iterator[Iterator[Iterator[int]]]:
yield h3(1)
yield h3(5)

x = h()
for i in x: # First Iterator
for j in i: # Second Iterator
for k in j: # Third Iterator
print(k) # Here get the ints
```

But with Futures!

```python
import typing
import asyncio
from asyncio import Future

@asyncio.coroutine
def h4() -> 'Future[Future[int]]':
"""
Return a Future[int], and have one more Future anotation saying
that can be "yielded from".
"""
yield from asyncio.sleep(1) # Just to wait a little bit :)
f = asyncio.Future() #type: Future[int]
return f # Here Future[int] is wrapped in Future[Future[int]]

@asyncio.coroutine
def h3() -> 'Future[Future[Future[int]]]':
"""
Same as h4() but with one more Future :P
"""
x = yield from h4() # Get the Future[int] of h4()
x.set_result(42) # set 42 as result
f = asyncio.Future() #type: Future[Future[int]] # Create a new Future
f.set_result(x) # set the result as the Future taked from h4
return f # return it

@asyncio.coroutine
def h() -> 'Future[None]':
print("Before")
x = yield from h3() # Get the Future[Future[int]] from h3()
y = yield from x # Get the Future[int] from the Future[Future[int]]
z = yield from y # Get the int from the Future[int]
print(z) # Prints 42
print(y) # Prints Future
print(x) # Prints Future>

loop = asyncio.get_event_loop()
loop.run_until_complete(h())
loop.close()
```

Summary
--------
To say that a function can be **"yielded from"**, the return type need to have a **Future[Some]**

The first Future of the type, is just to indicate that can be yielded from, so, if you
are going to return an int, the type of the function is Future[int] and if you are going to return a Future
that have inside an int, the type of the function is: Future[Future[int]]

The type in the function have to be writted in string.
```python
def h() -> 'Future[None]':
pass
```

See all the examples in **"stubs/3.4/asyncio/examples"**