stan-dev/stan — pull request #542 by mbrubake (2014-02-06 22:19:27+00:00)
Title: Speed up reading R dump formatted files

This branch may not (necessarily) ready to be merged. I'm putting the PR here so that others can comment on it and contribute (if so inclined). That said, it does pass unit tests, the changes are relatively minor and the net result is that it cuts the cost of reading a 150K dump file to ~1/3 of what it was. That is, so far it produces around a 70% decrease in parsing time.

So far, three things have been done:
1. Rewrite ```scan_chars()``` to take a ```const char *``` instead of a ```std::string```. This prevents lots (and lots) of redundant ```std::string``` allocations which simply copy string constants around.
2. Prevent constant reallocations of ```std::string```s by creating a buf_ member variable which is simply cleared before use in ```scan_number()```.
3. Convert to using ```boost::lexical_cast``` instead of stringstream to convert strings to numbers. This is much faster.

Anyone have other suggestions for changes to speed things up? One that, based on profiling, might help is to make larger reads and stop using lots of individual calls to ```get()``` and ```putback()```. For some reason there is a fairly high overhead to these functions (at least on Linux).