slamdata/slamengine — pull request #368 by sellout (2014-10-06 19:49:12+00:00)
Title: Convert ProjectOp and UnwindOp to map/reduce ops.

This is done via the new `WorkflowOp.finalize` function that performs
potentially-irreversible changes to the workflow prior to converting to
`WorkflowTask`. (Fixes #348)

A lot of other changes were made to support this:

* everywhere we use `Js.Expr` is now parameterized so we can specify the
base of the expression, rather than abusing `this` all the time;
* `ExprOp`s can now _sometimes_ be converted to JS (fixing part of #272);
* map/reduce ops no longer change the WorkflowBuilder `base` to
`ExprVar` (fixes #326);
* `FoldLeftOp` now handles patching each of its sources when necessary,
rather than expecting the creator to do it (projecting into `value`
for the first one, and potentially adding a ReduceOp on the others);
* `WorkflowOp.crush` tracks the `base` so it can patch operations that
follow a `MapReduceTask` (corollary to the fix to #326), which also
gives us some info that is useful for #287; and
* `coalesce` now simplifies some `FlatMapOp`s, eliminating ones with a
single source and flattening nested ones when possible.

`finalize` is also a step toward #227, as that largely implies building
a `Stream` that uses different combinations of `finalize`’s
optimizations.

The one bit of nastiness (there’s always one, isn’t there?) is that I
created a `private FoldLeftOp0` that is generated by `finalize` to
indicate that the `FoldLeftOp` has already been processed. Ideally this
could be noticed in `finalize`, but my attempts were not fruitful.