incanter/incanter — pull request #268 by dacamo76 (2014-08-22 15:37:16+00:00)
Title: Simplify read-dataset

Removes most options from read-dataset and introduce header, row, and dataset options that operate on the column headers, rows, and the full sequence of rows respectively.

**Justification:**
If new options are always added for a specific scenario, the list will grow unwieldy and it will be difficult to change options not knowing how it will affect other options. In my opinion there is no need to add unneeded complexity to the function for every possible scenario.

We can invert the process, strip out all the options (leaving just a few universally accepted defaults) and have the user pass in functions to manipulate the headers, rows, and collection respectively. This should leave the function clean, and users can specify and behavior they need by passing in the functions. If a user need specific functionality, new helper/utility functions can be added. This also has the advantage of building up a library of commonly used functions that can be composed and easily used to achieve any desired behavior, no matter how complicated. The inspiration comes from using http://clojure.github.io/java.jdbc/#clojure.java.jdbc/query.

* You can transform a column to a number by passing in a row-fn which does just that.
* Read in files too large for RAM? sample using dataset-fn.
* Remove lines with comments? pass in function to dataset-fn
* Column names as lowercase keywords? use :header-fn (comp keyword clojure.string/upper-case)

For example: To remove all lines beginning with "#".

```clojure
(defn comment?
"True if first element of coll begins with s"
[s row]
(.startsWith (first row) s))

(read-dataset (java.clojure.io/reader "file-with-comments.csv")
:dataset-fn #(remove (partial comment? "#") %))
```

Say you want your headers to be keywords, empty fields converted to :NA, and to sample only 10% of a huge file.

```clojure
(defn sample-percent
"Function take from Clojure Data Analysis Cookbook"
[k coll]
(filter (fn [_] (<= (rand) k)) coll))

(io/read-dataset (java.clojure.io/reader "large-file.csv")
:header true
:header-fn keyword
:row-fn (fn [row] (map #(if (clojure.string/blank? %) :NA %) row))
:dataset-fn (partial sample-percent 0.10))
```

I tried to make small changes so you can cherry pick what is needed.
The first commit makes the change to data.csv and the first few are backward compatible changes
The last commit refactors the whole function and adds unit tests with examples of usage.

Eventually, new functionality should be added by virtue of helper functions instead of option parameters. Maintaining helper functions allows greater flexibility via function composition and will allow new users to quickly get up to speed on functionality available in `read-dataset`. At the same time, users with unique requirements can also continue without resorting to a pull request adding new options for said requirements.

Below is example of possible helper functions useful for removing comment characters

Say you want to skip files that begin with ";", "#" or "//", but not lines that begin with a single "/".

```clojure
;;; Helper functions (another namespace?)
(defn comment?
"True if first element of row begins with s"
[s row]
(.startsWith (first row) s))

(def semicolon (partial comment? ";"))
(def pound (partial comment? "#"))
(def double-slash (partial comment? "//"))
;;;
````

````clojure
user=> (require '[incanter.io :as io])
nil
user=> (def test-reader
#_=> (java.io.StringReader. "#firstcomment\n//second comment line\nthis line not skipped\n;skipped\n/single slash"))
#'user/test-reader
user=> (io/read-dataset test-reader
#_=> :dataset-fn #(remove (some-fn semicolon pound double-slash) %))

| 0 |
|-----------------------|
| this line not skipped |
| /single slash |
````