JuliaStats/DataFramesMeta.jl — pull request #3 by tshort (2014-02-10 22:19:46+00:00)
Title: LINQ-like additions

This pull request adds a number of functions and macros for operations on DataFrames have been defined. See also https://github.com/JuliaStats/DataFrames.jl/issues/381.

Here is a table of equivalents for Hadley's [dplyr](https://github.com/hadley/dplyr) and common [LINQ](http://en.wikipedia.org/wiki/Language_Integrated_Query) functions:

Julia dplyr LINQ
---------------------------------------------
@sub filter Where
@transform mutate Select (?)
@by GroupBy
@groupby group_by
@based_on summarise
orderby arrange OrderBy
select select Select

The main immediate input needed is suggestions on naming. I'm happy with the names and roles of `@transform`, `@by`, `@groupby`, and `select`. The rest are a bit klunky:

- `@select` - overlap with `select` (the function)
- `@sub` - not a sub (yet)
- `@based_on` - just a klunky name
- `orderby` - confusion with `sort`

Chaining operations is a useful way to manipulate data. It's done in dplyr, LINQ, and pandas. There are several ways to do this. This is still in flux in base Julia (https://github.com/JuliaLang/julia/issues/5571). Here is one option adapted from [Lazy.jl](https://github.com/one-more-minute/Lazy.jl) by Mike Innes (@one-more-minute). `@>` is a threading macro that flows results from one operation to the next.

```julia
x_thread = @> begin
df
@transform(y = 10 * :x)
@sub(:a .> 2)
@by(:b, meanX = mean(:x), meanY = mean(:y))
orderby(:meanX)
select([:meanX, :meanY, :b])
end
```

There are several ways to implement chaining. We will probably want to follow whatever Base does (if anything). We might want to try to influence that if we have strong preferences. Here is another [option](https://github.com/JuliaLang/julia/pull/5734) by James Porter (@porterjamesj), another style threading macro that uses a placeholder (`_`).

```julia
x = @as _ begin
df
@sub(_, :a .> 2)
@transform(_, y = 10 * :x)
@by(_, :b, meanX = mean(:x), meanY = mean(:y))
orderby(_, :meanX)
select(_, [:meanX, :meanY, :b])
end
```

I've also partially implemented chaining uses `|>` in Base to chain operations:

```julia
x = df |> @sub(:a .> 2) |> orderby(:x) |> select([:b, :x])
```

I may remove these. The main issue is that you have to manually add function curries for these, and doing that constrains the function form. The main constraint is that arguments cannot be variable.