JuliaLang/julia — pull request #6718 by tanmaykm (2014-05-02 11:55:25+00:00)
Title: readdlm improvements for reading large files

This PR aims to improve `readdlm`'s ability to read large files. Ref. discussions at: https://groups.google.com/forum/#!topic/julia-dev/qFrrHiKalG8 and https://groups.google.com/forum/#!topic/julia-dev/vLvP4G-68es

1. Speed up reading of large files if dimensions are known. Result is allocated and populated while parsing, without the need of storing offsets or making one more pass over the data. This is exposed a new optional parameter `dims` which can be a `(nrow,ncol)` tuple.
2. Prefer doing two passes when offset storage may take too much memory (current heuristic value: > 20% of ram). This essentially determines `nrow` and `ncol` in the first pass and calls (1). For smaller files, storing offsets during the first pass and just dealing with offsets results in considerable speedup.
3. Moved `datafmt.jl` into a module named `DataFmt` to avoid exposing unrelated symbols in `Base`.