tested on full size data sets