Implementation of Failover Handling of Broker Network in Big Active Data CS 237 Project Final Report Team 6 :

While the majority of today’s Big Data systems are passive, there is a widespread need for a scalable system that continuously and reliably captures the Big Data and facilitates the timely and automatic delivery of new information to a large population of interested users. Big Active Data project [1] meets the mentioned requirements by extending the AsterixDB, a scalable open-source Big Data Management System which offers native data storage and indexing as well as querying of datasets in local files that enables efficiency for smaller as well as large queries. In addition, it has an open data model that handles complex nested data as well as flat data, and a full query language that supports declarative querying over multiple data sets. Along with AsterixDB as the data cluster, the BAD system consists of data sources (Data Publishers) and end users (Data Subscribers), and it handles notification management and distribution by utilizing a broker network. The system must have extensible support for a variety of high-volume data feeds so that data from many different Data Publishers can be continuously and reliably ingested and stored for use by BAD applications. Given a set of registered data interests from the universe of Data Subscribers, the BAD platform must capture and monitor their potentially many interests against the evolving state of the stored data – in an efficient, shared, and scalable manner – in order to detect the need to generate and send out new notifications. To that end, the platform can provide a notion of information channels, based on application-defined parameterized queries, that can be subscribed to by users. When new notifications are generated by the BAD Data Cluster, they must be disseminated to their target users. The data distribution responsibilities of the BAD platform are handled by a network of cooperating BAD brokers which manage the system’s connections and communications with end users and their devices. The current BAD platform uses an extended version of AsterixDB which consists AQL A rich and declarative query language, a scalable distributed dataflow platform, rich data types and fast data feeds. Brokers provide a go-between layer between the BAD Data Cluster and the actual, geographically distributed, end users of a given BAD application. Based on the interests of the users, BAD application manager will create a set of channels – continuous queries with parameters. The end users will subscribe to a channel via the broker node they are connected to. The broker nodes are responsible for handling client (called BAD client) registration, managing subscriptions and delivering results for those subscriptions. Each broker has two parts: a “client-facing” part managing the clients and an “Asterixfacing” part handling interactions with the Asterix backend. A simple workflow of interaction between a client and the broker is as follows: The client registers via the broker and logs in. The client then subscribes to one or more available channels using desired parameter values, which are passed to the Asterix backend by the broker. The backend notifies the broker when new results are populated in the subscribed channels. The broker, in turn, notifies the client, and the client acts to fetch the results as desired. Broker nodes are implemented as RESTful servers written in Python using the Tornado web framework.