process three main datasets we need