liebke/incanter — pull request #227 by dacamo76 (2014-01-24 23:18:21+00:00)
Title: Simplify read-dataset

Removes most options from read-dataset and introduce header, row, and dataset options that operate on the column headers, rows, and the full sequence of rows respectively.

**Justification:**
If new options are always added for a specific scenario, the list will grow unwieldy and it will be difficult to change options not knowing how it will affect other options. In my opinion there is no need to add unneeded complexity to the function for every possible scenario.

We can invert the process, strip out all the options (leaving just a few universally accepted defaults) and have the user pass in functions to manipulate the headers, rows, and collection respectively. This should leave the function clean, and users can specify and behavior they need by passing in the functions. If a user need specific functionality, new helper/utility functions can be added. This also has the advantage of building up a library of commonly used functions that can be composed and easily used to achieve any desired behavior, no matter how complicated. The inspiration comes from using http://clojure.github.io/java.jdbc/#clojure.java.jdbc/query.

* You can transform a column to a number by passing in a row-fn which does just that.
* Read in files too large for RAM? sample using dataset-fn.
* Remove lines with comments? pass in function to dataset-fn
* Column names as lowercase keywords? use :header-fn (comp keyword clojure.string/upper-case)

For example: To remove all lines beginning with "#". Pull request https://github.com/liebke/incanter/pull/219

```clojure
(defn comment?
"True if first element of coll begins with s"
[s row]
(.startsWith (first row) s))

(read-dataset (java.clojure.io/reader "file-with-comments.csv")
:dataset-fn #(remove (partial comment? "#") %))
```

Say you want your headers to be keywords, empty fields converted to :NA, and to sample only 10% of a huge file.

```clojure
(defn sample-percent
"Function take from Clojure Data Analysis Cookbook"
[k coll]
(filter (fn [_] (<= (rand) k)) coll))

(io/read-dataset (java.clojure.io/reader "large-file.csv")
:header true
:header-fn keyword
:row-fn (fn [row] (map #(if (clojure.string/blank? %) :NA %) row))
:dataset-fn (partial sample-percent 0.10))
```

I tried to make small changes so you can cherry pick what is needed.
The first commit makes the change to data.csv and the first few are backward compatible changes
The last commit refactors the whole function and adds unit tests with examples of usage.