code changed to work on full data