ipython/ipython — pull request #2447 by takluyver (2012-09-29 22:34:49+00:00)
Title: Stateful line transformers

The big step for IPEP2 (#2293). I anticipate there may be more work to do on this, but I think it's ready for other people to start playing with it.

What does this solve?

* De-denting indented code, but only by as much as the first line of the cell is indented. See PR #2436.
* Line magics can have line continuations within them (#2164)
* The architecture for transforming cell magics is much cleaner.
* We can clean prompts within multiline strings, e.g.

```
>>> a = """123
... 456"""
```

How does it work?

You can create 3 kinds of transformers:
* Stateless transformers can be defined as a simple function with a decorator `@stateless_input_transformer`. The transformer that recognises `?` at the end of lines is defined like this.
* Stateful transformers can be defined as a generator with a decorator `@coroutine_input_transformer`. Most of our own transformers are written like this, because the problem lends itself neatly to using coroutines. Coroutines can save up input by yielding `None` until they're ready to return it. They're reset (at present) by sending `None` in, but I'm thinking of changing that mechanism, perhaps to throwing&catching an exception.
* Finally you can subclass `IPython.core.inputtransformer.InputTransformer` directly, implementing `push(line)` and `reset()` methods.

Transformers have an attribute `look_in_string`, which if set to True allows them to be called even if the start of the line is within a multiline string (or another multiline expression). This is set for the `leading_indent`, `classic_prompt` and `ipy_prompt` transformers.