add infinite processing and publishing loops
move get_job_status to utils, modify db structure to minimize problems with number of files