Stream Processing in Action

The world is now information-driven. Many business and government institutions are fully automated, relying on a variety of sensors and infrastructure to collect, store, and analyze incoming data on a continuous basis. Clearly, the need to monitor and act on information from sensors in the field, to drive rapid decisions, to tweak production processes, to optimize logistics choices, and, ultimately, to better manage physical systems is now fundamental. Data stream processing was born from increasingly stringent data management, processing, and analysis requirements of this new breed of business and scientific applications, coupled with the confluence of two major scientific and technological shifts: first, the advances in data management technologies including the software and hardware infrastructure for supporting large distributed deployments; second, the advances in basic techniques in online data mining and real-time data analytics. Stream Processing Applications (SPAs) are used in senseand-respond systems [7] that continuously receive external signals in the form of one or more data streams from multiple sources and employ analytics aimed at detecting and, sometimes, predicting actionable information. Examples of streaming applications and systems abound. Early instances include different Supervisory Control And Data Acquisition (SCADA) systems deployed for monitoring and controlling manufacturing, power distribution, and communication networks, as well as environmental monitoring systems. More recently, SPAs hosted by Stream Processing Systems (SPSs) supporting modern business applications have become common, in areas such as finance engineering, surveillance, and telecommunications. While SPAs are becoming central to a variety of new application domains, designing and developing them is challenging. Not only is it essential to ensure that the application performs its required analytical task correctly, but it is also important to ensure that this is accomplished efficiently, scalably, and reliably, utilizing the available computing infrastructure parsimoniously, while meeting its performance requirements and adapting to unexpected variations in load and in resource availability. To meet these goals, application developers, with the help of the SPS, must carefully decompose the application, mapping it onto a set of operators organized in a Data Flow Graph (DFG), design and implement individual components to perform the desired analytic tasks, distribute the application across the computational infrastructure, and, finally, continuously tune the application so it evolves appropriately. Additionally, developers also need to account for fault tolerance and dynamic adaptation in response to data and processing variations that might arise at runtime. In our experience, these characteristics require a shift in the developer’s thought process to incorporate new analytic and engineering methods during application design, development, and its continuous evolution. The stream processing paradigm represents this shift, and SPSs serve as the software platform technology that enables the implementation of applications that adhere to this new computing paradigm.