framework that routes data trough a complex real-time/batch processing pipeline