major fix and support for datasets larger than RAM