updates for more robust processing and hooking up to processing jobs