implemented most of the metagraph and job progress tracking