realtime pipeline processing engine