HarlanH/DataFrames.jl — pull request #73 by johnmyleswhite (2012-11-23 22:14:11+00:00)
Title: [RFC]: New data reading backend

This pull request outlines a new backend for reading delimited data from disk. This new backend design has the potential to be an improvement in several ways:

* It handles escaping-quotes within CSV file appropriately
* It handles CSV, TSV and WSV files appropriately
* It performs automatic metadata inference by making a first pass through a data set to determine which columns can be represented using `Int64`, `Float64` or `ASCIIString`
* It reads data in using minibatches, which has the potential for providing a unified backend for both `DataFrame`'s and `DataStream`'s
* It handles missing values described using either blank-space or with an explicit "NA". It also provides tools for specify other types of missingness indicators

In order to provide this type of functionality, the new backend consists of five parts:

* `src/split.jl`: This implements a simplistic two-state machine that allows the reader to cope with escaping-quotes that delimit fields which can then contain commas
* `src/convert.jl`: This provides a mechanism for sweeping through an array of strings and converting them to a desired set of types using `DataVec`'s to cope with missing values
* `src/inference.jl`: This provides a tool for passing through a data set and determining the tightest type that can be used to store a column of type. The tightest descreases as `Int64` > `Float64` > `ASCIIString`. Ultimately we will want other string types (and maybe other numeric types), but this seemed like a good start
* `src/parse.jl`: This provides a mechanism for parsing a delimited line into a single row `DataFrame` with the appropriate types and missingness
* `src/read.jl`: This provides mechanisms for reading minibatches from disk and then uses this mechanism to read entire tables into a single `DataFrame` for batch processing

Quite a lot of cleanup will be needed before this new backend could be the definitive backend, but I think this current version is finally strong enough for testing and refinement.