major work on data processing functions