tested with full size datasets