kennethreitz/envoy — pull request #43 by paddymul (2012-06-17 19:28:34+00:00)
Title: Ext proc integration

I forked extproc from aht, and I cleaned up a bunch of the implementation internals.
I used my forked extproc to implement envoy functionality in terms of extproc.

Here are the big things I added to extproc

* Cmd objects implement the same interface as Pipe objects
this means that you can construct a Pipe of Pipes, This makes life a lot easier for some things.

* @fork_dec
this lets python functions participate in pipelines, the functions are forked into their own process
Take a look at from https://github.com/paddymul/extproc/blob/ea1bbcc0690a01b679bee1785e5e0aa9d202f4db/test_extproc/extproc_test.py#L23

```python

def test_pipe_proc_decorator(self):
@fork_dec
def echoer(stdin_f, stdout_f, stderr_f):
for line in stdin_f:
stdout_f.write(line + "\n")
pipe_obj = Pipe(Cmd("echo foo"), echoer)
self.assertSh(pipe_obj.capture(1).stdout.read(), 'foo')

@fork_dec
def doubler(stdin_f, stdout_f, stderr_f):
for line in stdin_f:
stdout_f.write(line+line + "\n")

pipe_obj = Pipe(Cmd("echo foo"), doubler)
self.assertSh(pipe_obj.capture(1).stdout.read(), 'foofoo')

```

compared to envoy this fork implements a couple of things. connect_extproc is a replacement for connect. connect_extproc supports connect semantics on pipelines.

I think extproc does a better job of pipelines than envoy. Cmds, Pipes, and fork_decs are connected to each other via file descriptors, whereas in envoy, the first Cmd runs and its output is read into python, then this is written into the second command. extproc functions more like bash and other shells.

extproc does have some deficiencies. I'm not sure how well it works on Windows, I know fork_dec won't work, I don't think it will be a lot of work to make it work on Windows, I'm just not sure. Extproc's implementation is still tough to follow and needs more code cleanup, it is very much a work in progress.

I like the interface of envoy and I wanted to see what you think of this integration. I expect that this will take significant extra work on my part which I am happy to do. I have a couple of questions for how you would handle interface ideas that extproc makes possible.

1. extproc gets away from a lot of shell escaping issues because it makes it easier to compose pipelines programatically, - it gives you access to the parse tree. What do you think of

```python
Cmd('ls').pipe_to(Cmd('grep pyc'))
```
I think you could sidestep a lot of tricky shlex parsing issues by incorporating this into envoy somehow

```python
envoy_pipe('ls | grep pyc').pipe_to(Cmd('xargs incantation with tricky arguments that would tripup shlex'))
```
When I find myself dealing with shell escaping issues, I don't want to try to trick the shell into accepting my input, I want access to the parse tree so I can actually manipulate objects.

2. Job and process management. I like extproc's JOBS global that acts like jobs in bash. I think there is some really neat stuff that could be done with spawned pipes, - imagine if there were an easy to run a cmd after 2 other background processes have both finished.

Any thoughts?