astropy/astropy — pull request #2716 by amras1 (2014-07-09 20:21:52+00:00)
Title: Added a new fast reading/writing engine to io.ascii

This PR adds a Cython/C parsing engine to the infrastructure of `io.ascii` so that new reading classes for simple formats (currently `FastBasic`, `FastTab`, `FastCsv`, `FastNoHeader`, `FastCommentedHeader`, and `FastRdb`) can opt to use this engine instead of the ordinary `BaseReader` inheritance model. The ordinary readers for these formats still remain to maintain flexibility and compatibility. The PR introduces a new parameter ``use_fast_reader`` in `ascii.read()` which can be used to enable or disable the fast engine (by default, `read()` will attempt to use the fast reader and fall back on the slow reader if parsing fails). The fast readers are also on the guess list directly in front of their slower counterparts.

There is also a Cython engine for writing with these formats; a similar parameter ``use_fast_writer`` can enable or disable this engine, and `ascii.write()` uses the engine by default when writing output in a compatible format.

Using randomly generated text files, it seems that the fast reading engine is anywhere from 3 to 7 times faster than the ordinary readers depending on the data type of table columns, while the fast writing engine is anywhere from ~2.5 to 11 times faster than the ordinary writers; both engines have less of a speed gain for floating-point data, which might be worth looking into at some point. [Here](http://amras1.github.io/astropy-benchmarks/#) are some before and after benchmarks, which cover a range of ASCII formats and compare Astropy's performance with that of numpy and Pandas. There is also a short design document [here](https://github.com/amras1/astropy-benchmarks/blob/master/design.md), which gives a quick overview of the parsing implementation in this PR.

This is a pretty big PR, so feel free to make any suggestions or ask me anything.