functions to process initial dataset