# An Apprentice Experiment in Python Programming, Part 2

*[Epistemic status: me trying to recall what happened in a 4.5-hour pair-programming session, during which I was focused on solving problems. This post was reconstructed from scratch code files, written messages (the session was over voice call) and my python shell history.]*

Previously: https://www.lesswrong.com/posts/kv3RG7Ax8sgn2eog7/an-apprentice-experiment-in-python-programming

Three days ago gilch and I had another session on Python Programming, where we continued talking about decorators.

### Lambda decorator

So far, all of the decorators we've seen have been functions that take in another function as parameter, then used by the syntax `@<decorator_name>`. Turns out we can use a lambda function as a decorator too:

```python
@lambda f: print("hi")
def add(a, b):
    return a + b
```

When this file gets run in Python 3.9 interactive mode, the output is

```
hi
```

and when I call the `add` function:

```
>>> add(1, 2)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'NoneType' object is not callable
```

Gilch asked me to write out the desugared version of this code. I wrote `add = print("hi")`, which was not correct. Gilch gave me the correct desugared code:

```python
def add(a, b):
    return a + b

add = (lambda f: print("hi"))(add)
```

### Partial

We did a brief introduction to `partial`. We had code from last time:

```python
def subtract(a, b, c):
    return a - b - c
```

Accessing doc from within python shell:

```
>>> from functools import partial
>>> help(partial)
```

Using partial to create a function with the first and second arguments passed in:

```
>>> partial(subtract, 1, 2)
functools.partial(<function subtract at 0x7f34649af3a0>, 1, 2)
>>> partial_sub = partial(subtract, 1, 2)
```
Gilch pointed out that the line above, `partial_sub = partial(subtract, 1, 2)`, could've been written as `partial_sub = _`  because in python shell, `_` returns the last value that's not None.

Testing the partial function:

```
>>> partial_sub(3)
-4
```
1 - 2 - 3 equals -4, so we're getting what we expected. 

Gilch then pointed out that arguments can be passed into `partial` as keyword arguments:

```
>>> partial(subtract, b=0, c=2)
functools.partial(<function subtract at 0x7f34649af3a0>, b=0, c=2)
>>> _(10)
8
```
Finally, there's a way to implement partial functions using lambda:

```
>>> lambda x: subtract(x, 0, 2)
<function <lambda> at 0x7f346474ae50>
>>> _(10)
8
```

### Running Bash in Python

While running `help(partial)` in the python shell, the topic of using `less` came up. I asked gilch if it was possible to pipe outputs from python shell to bash tools, gilch answered that we could not do it directly, but just like how bash can run python, python can also run bash:

```
>>> import os
>>> os.system("bash")
```

Now we're in a bash shell, run by python:

```
~/code_stuff/python_scratch $ ls | less
~/code_stuff/python_scratch $ exit
exit
0
>>> 
```

At the last line, we're back in python shell again.

### Lambda functions

Gilch gave me a number of lambda expressions in increasing complexity, asked me to explain what they did, then verify:

- `(lambda x: x+x)(3)` evaluates to 6, this one was straightforward 
- `(lambda x: lambda y: x + y)(2)(3)` I assumed that this statement expanded inside out (i.e. `2` was passed in as `y` and `3` was passed in as `x`) so I concluded that this would evaluate to 5. I got the correct answer, but my mistake was exposed by the next example:
- `(lambda x: lambda y: x + y)('a')('b')` Using the same logic, I concluded that the result would be `'ba'`,  but the actual result was `'ab'`. Why did I make this mistake? Gilch gave me the next question:
- `(lambda x: lambda y: x + y)('a')` evaluates to a lambda. A lambda of `x` or `y`? The first part of the expression, `(lambda x: lambda y: x + y)`, is a lambda function that takes in a parameter `x`, and returns a lambda function `lambda y: x + y`. This lambda function that takes in `x` is then called with argument `'a'`.  The return value is `lambda y: 'a' + y`.  And then it made sense why the previous question evaluated to `'ab'` instead of `'ba'`.

I think what gilch was getting at here was the syntax of function calls other than the format of `f(x)`. Gilch also at some point pointed out the difference between a callable and a function.

### Preprint

Gilch gave me the next challenge: write a decorator `preprint` such that it can be used in the following way:

```python
@preprint("hi")
def bob():
    print("Bob")
@preprint("hello")
def alice():
    print("Alice")
```

```
>>> bob()
hi
Bob
>>> alice()
hello
Alice
```

I struggled here. I attempted to write out some vague idea in my head:

```python
def greet_by_name(salutation, name):
    print(salutation)
    print(name)

def bob(salutation):
    from functools import partial
    return partial(greet_by_name, name="Bob")
```

Gilch reminded me that the use case was already specified; functions `alice` and `bob` were not to be modified. They then suggested that I wrote out the desugared version of the code. I struggled and gave an incorrect answer again until gilch provided me with the correct desugaring:

```python
@preprint("hi")
def bob():
    print("Bob")
```

is equivalent to `bob = (preprint("hi"))(bob)`. 

### Understanding decorator de-sugaring

As you can see, at this point, I was struggling with desugaring. So gilch presented me with a formula:

```
@<decorator expression>
def <name>(<args>):
    <body>
```

is equivalent to 

```
def <name>(<args>):
    <body>

<name> = (<decorator expression>)(<name>)
```

### Preprint

My memory of what exactly happened here is spotty, perhaps because I was focused on finding out how to write the `preprint` decorator. Eventually I came up with something like this:

```python
def preprint(greetings):
    from functools import partial
    def greet_by_name(greeting, foo):
        def wrapper():
            print(greeting)
            foo()
        return wrapper
    return partial(greet_by_name, greetings)
```

This worked, but you may be able to see from the way I passed in `greetings` into `greet_by_name`, I was confused about why there needed to be 3 layers of definitions. Gilch gave me an alternative solution, and explained the pattern:

```python
def preprint(greetings): # factory
    def greet_by_name(function): # decorator
        def wrapper(): # replaces the function being decorated
            print(greetings)
            return function()
        return wrapper
    return greet_by_name
```

Gilch explained that, for decorators that take in parameters, they are like a factory that produces different decorators based on input. Thus the first layer of definition processes the input, the second layer of definition is the decorator itself, and the innermost `wrapper` function is what replaces the function that ends up being decorated. This

```python
@preprint("hi")
def bob():
    print("Bob")
@preprint("hello")
def alice():
    print("Alice")
```

is equivalent to 

```python
greet_by_name_hi = preprint("hi")
greet_by_name_hello = preprint("hello")
@greet_by_name_hi
def bob():
   print("Bob")
@greet_by_name_hello
def alice():
    print("Alice")
```

At this point, I was not quite getting the 3-layer structure--I identified that there was a pattern of factory-decorator-function but I was mostly mimicking the pattern. This lack of understanding would be exposed in a later example.

### `wraps`

In the previous example, we had the decorator `preprint`:

```python
def preprint(greetings): # factory
    def greet_by_name(function): # decorator
        def wrapper(): # replaces the function being decorated
            print(greetings)
            return function()
        return wrapper
    return greet_by_name
```

Gilch instructed me to add a doc string to a function being decorated by this decorator. So we have:

```python
@preprint("hi")
def bob():
    """
    prints 'bob'
    """
    print("bob")
```

What happens when we try to access the doc string of `bob`?

```
>>> help(bob)

Help on function wrapper in module __main__:


wrapper()
(END)

```

Gilch then introduced `wraps`.  According to the [documentation](https://docs.python.org/3/library/functools.html#functools.wraps), it calls [update_wrapper](https://docs.python.org/3/library/functools.html#functools.update_wrapper) to reassign some attributes of the wrapped function to the wrapper function. 

```python
from functools import wraps
def preprint(greetings): # factory
    def greet_by_name(function): # decorator
        @wraps(function)
        def wrapper(): # replaces the function being decorated
            print(greetings)
            return function()
        return wrapper
    return greet_by_name

@preprint("hi")
def bob():
    """
    prints 'bob'
    """
    print("bob")
```

```
>>> help(bob)
Help on function bob in module __main__:

bob()
    prints 'bob'
(END)

```



### `map`

Gilch asked me if I knew what `map` in python did. I said `map` took in a function and an array and return the result of the function applied to each element of the array. Gilch responded that it was mostly correct, with the caveat that map took in an iterable instead of an array. 

Gilch gave me a test case for which I needed to implement the decorator:

```python
@<something>
def double(x):
    return x+x
assert list(double) == ['aa','bb','cc','dd']
```

I came up with

```python
@lambda f: map(f, 'abcd')
def double(x):
    return x+x
```

When we run `assert list(double) == ['aa','bb','cc','dd']`, python did not complain. However when we ran `list(double)` again, we saw an empty array:

```
>>> list(double)
[]
>>> double
<map object at 0x7f3ad2d17940>
>>> list(double)
[]
```

Gilch explained that `double` was not an array, it was an iterable. After we had iterated through `double` in the assert statement, it became empty. 

### Callable

Gilch illustrated the difference between a function and a callable:

```python
def my_if(b, then, else_):
   return then if b else else_
```

in which `then if b else else_` is a callable. Two use cases:

```
>>> my_if(1<2, print('yes'), print('no'))
yes
no
```

```
>>> my_if(1<2, lambda:print('yes'), lambda:print('no'))()
yes
>>> my_if(1>2, lambda:print('yes'), lambda:print('no'))()
no
```

In the first case, `print("yes")` and `print("no")` are not callables (they get run and evaluated to `None`) whereas the `lambda:print('yes')` and `lambda:print('no')` in the second case are callables.

Similarly, when we put the print statements in functions, the functions are callables:

```python
def print_yes():
    print("yes")

def print_no():
    print("no")
```

```
>>> my_if(1<2, print_yes, print_no)
<function print_yes at 0x7fa36ffbb280>
>>> my_if(1<2, print_yes, print_no)()
yes
```

### Branching, with 1 branch

Gilch illustrated how we could use decorator for branching:

```python
def when(condition):
    def decorator(function):
        if condition:
            return function()
    return decorator
```

```
>>> @when(1<2):
... def result():
...     print("Yes.")
...     return 42
... 
Yes.
>>> assert result == 42
>>> @when(1>2):
... def result():
...     print("Yes.")
...     return 42
... 
>>> assert result == None
```

### Branching, with 2 branches

Gilch asked me how I would implement branching such that if a given condition is true, one branch gets evaluated; if condition is false, another branch gets evaluated. This would essentially mean we decorate two functions. I came up with this solution:

```python
def if_(condition):
    def decorator(function):
        if condition:
            return function()
    return decorator

def else_(condition):
    def decorator(function):
        if not condition:
            return function()
    return decorator

condition = 1<2

@if_(condition)
def result():
    print("Yes.")
    return 42

@else_(condition)
def print_no():
    print("no!")
```

```
$ python3.9 -i map.py
Yes.
>>> result
42
```

And with `condition = 1>2`:

```
$ python3.9 -i map.py
no!
>>> result
>>>
```

The problem of this solution was too much code repetition. Also, I used 2 decorators instead of 1. Gilch said one way to do this was to decorate a class instead of a function.

### Decorating a Class

Gilch gave me the test case, and asked me to write the decorator:

```python
@if_(1>2)
class Result:
    def then():
        print("Yes.")
        return 42
    def else_():
        print("No.")
        return -1
```

Eventually I came up with the decorator:

```python
def if_(condition):
    def decorator(c):
        if condition:
            return c.then()
        else:
            return c.else_()
    return decorator
```

```
$ python3.9 -i decorating_a_class.py 
Yes.
>>> Result
42
```

with condition reversed (`@if_(1<2)`):

```
$ python3.9 -i decorating_a_class.py 
No.
>>> Result
-1
```

### Decorating a Class, Example 2

Gilch gave me another test case for which I needed to write the decorator:

```python
@sign(x)
class Result:
    def positive():
        print('pos')
    def zero():
        print('zero')
    def negative():
        print('neg')
```

I modified the solution from the previous question to answer this one:

```python
def sign(x):
    def decorator(c):
        if x > 0:
            return c.positive()
        elif x == 0:
            return c.zero()
        else:
            return c.negative()
    return decorator
```

This was basically the same example as the previous one in the sense that I didn't need to change the structure of the decorator, only the specific functions within the innermost layer.

### Making Branching Statements without Using `if`

I asked gilch if there were ways to make if statement without using `if`. Gilch gave me some examples:

- `return [c.then,c.else][not condition]()`
- `return {True:c.then,False:c.else_}[bool(condition)]()`

### A More Difficult Example

Gilch gave me another test case:

```python
@if_(1<2)
def then():
    print('yes')
    return 42
@then
def result():
    print('no')
    return -1
assert result == 42
```

And when the first line is changed to `@if_(1>2)`, `assert result == -1` should pass.

I figured out how to make the first branch work but not the second. At this point, it was getting quite late in my timezone and by brain was slowing down after hours of coding, so I told gilch I'd work on it the next day.

When I picked this problem up the next day, all of the wobbly parts of my understanding came back to bite me. I was running into a bunch of error messages like `TypeError: 'NoneType' object is not callable` and `TypeError: <function> missing 1 required positional argument: 'greetings'` (or `TypeError: <function> takes 1 positional argument but 2 were given`), and since the error messages would point at the line where `result` was defined instead of the line within the decorator where the problem was, I was struggling a lot debugging my code. I poked around the internet a lot to look at other examples as an attempt to understand how to expand decorators in my head, and eventually came across [this slide](https://speakerdeck.com/pycon2019/reuven-m-lerner-practical-decorators?slide=28) from [a PyCon talk](https://www.youtube.com/watch?v=MjHpMCIvwsY&t=837s) that made things a bit more clear to me.

I sent gilch my solution:

```python
def if_(condition):
    def decorator(function):
        if condition:
            def wrapper(*x):
                return function()
            return wrapper
        else:
            def decorator2(function):
                return function()
            return decorator2
    return decorator

# original test cases below
@if_(1<2)
def then():
    print('yes')
    return 42
@then
def result():
    print('no')
    return -1
assert result == 42
```

```
$ python3.9 -i cases.py
yes
>>> result
42
```

With the condition flipped:

```
$ python3.9 -i cases.py
no
>>> result
-1
```

Gilch noticed my confusion: "The way you named things indicates a lack of understanding. Naming should have been more like this."

```python
def if_(condition):
    def decorator(function):
        if condition:
            def decorator2(f):
                return function()
            return decorator2
        else:
            def decorator2(f):
                return f()
            return decorator2
    return decorator
```

Alternatively, gilch provided two more solutions:

```python
def if_(condition):
    def decorator(function):
        if condition:
            return lambda _: function()
        else:
            return lambda f: f()
    return decorator
```

```python
def if_(condition):
    def decorator(then_func):
        def decorator2(else_func):
            return (then_func if condition else else_func)()
        return decorator2
    return decorator
```

"They're both a decorator you apply to the second definition, regardless of which branch is taken. So calling the first one a wrapper is the wrong way to think of it," explained gilch. 



### Observations

1. The concept of decorator being syntactic sugar of calling the decorator on a function then reassigning the result to the variable that used to be the function somehow didn't stick for me. When gilch asked me to write the desugared version of functions, I kept thinking about further expanding the functions.
2. There are a few things that I was not used to thinking about: functions of functions, functions that use variables that are in scope but don't get passed in as parameters, functions that return callables. All these made decorators--especially parameterized decorators--"hard to fit into my head at once," using gilch's remark.
3. Gilch's teaching was primarily puzzle-based, which is quite different from the lecture-based learning that I have done. This is pretty cool, because for most of my life, solving problems has been in the context of some evaluation. Stress sucked the fun out of puzzles for me when I had trouble decoupling my self worth and the outcomes of those evaluations. Now I'm solving puzzles for no other reason beyond learning, I've realized this is also an opportunity to [unlearn](http://paulgraham.com/lesson.html).