adding support for large datasets TODO finish and benchmark