MONITORING DISTRIBUTED DATA STREAMS

Monitoring distributed streams of data is a basic construct in many distributed systems. Examples include a wireless sensor network, where we would like to receive an alert every time the average of the temperature readings taken by the sensors exceeds a given threshold, or a distributed search engine, where we would like to determine the set of queries whose frequency of use exceeds a given threshold. So far, research has focused on detecting global properties that are de ned by simple aggregates (e.g., sum, average, or minimum) over the distributed data. For these cases, there are usually local conditions for the occurrence of the global property of interest. For example, if we wish our monitoring system to alert when the average temperature is over 50 degrees, we can use the observation that for this to occur, the temperature measurement in at least one node has to be over 50 degrees hence con guring each node to remain silent as long as its reading is below 50 degrees. In many cases, however, global properties of interest are expressed by more complex functions. Examples include detecting when the variance of the sensor readings in a sensor network exceeds a given threshold, determining the set of pairs of queries whose correlation exceeds a given threshold, and performing distributed classication tasks. Monitoring such complex functions is an expensive task in terms of communication, processing and storage access. Almost all the research to date has been restricted to linear scoring functions, to the exclusion of many other important functions, such as variance, mutual information and correlation coe cients. We study global properties of the following form: we assume that each node holds a time varying two dimensional vector. In addition, we assume that we are T ec hn io n C om pu te r Sc ie nc e D ep ar tm en t M . S c. T he si s M SC -2 01 017 2 01 0 given an arbitrary scoring function that is calculated on the weighted average of the local vectors held by the nodes. We are interested in detecting when the value of this function violates a predetermined threshold value. In the general case described above, it is far harder to devise local conditions such as the ones employed for simple aggregates. Even in the apparently trivial case of f (x) = x2, we cannot make any inference on whether f ( x+y 2 ) is above or below the threshold t , when given the position of f (x) and f (y) vis-a-vis a threshold value t ; i.e., if we know that f (x) > t and f (y) > t, we cannot tell whether f ( x+y 2 ) > t or f ( x+y 2 ) < t. As this simple example illustrates, it is generally very di cult to nd local indications for monitoring nonlinear functions. The problem is that it is not enough to know the function's values at the local data vectors; we have to know something about the possible domain which contains the average of the local vectors. Our proposed approach is based on the mathematical and geometric properties of the scoring functions, which have never been used by the data mining community. We plan to apply this approach for di erent data mining algorithms in di erent distributed setups, for which current solutions are either suboptimal or nonexistent. T ec hn io n C om pu te r Sc ie nc e D ep ar tm en t M . S c. T he si s M SC -2 01 017 2 01 0