now calculating stats about the dataset
merged some teddy changes with will changes