tested on full size datasets