Achieving Communication Efficiency through Push-Pull Partitioning of Semantic Spaces to Disseminate Dynamic Information Abstract-Many database applications that need to disseminate dynamic information from a server to various clients can suffer from heavy communication costs. Data caching at a client can help mitigate these costs, particularly when individual PUSH-PULL decisions are made for the different semantic regions in the data space. The server is responsible for notifying the client about updates in the PUSH regions. The client needs to contact the server for queries that ask for data in the PULL regions. We call the idea of partitioning the data space into PUSH-PULL regions to minimize communication cost data gerrymandering. In this paper, we present solutions to technical challenges in adopting this simple but powerful idea. We give a provably optimal-cost dynamic programming algorithm for gerrymandering on a single query attribute. We propose a family of efficient heuristics for gerrymandering on multiple query attributes. We handle the dynamic case in which the workloads of queries and updates evolve over time. We validate our methods through extensive experiments on real and synthetic data sets. M ANY emerging applications need to disseminate dynamic information from a server to various clients, especially due to the fact that an increasing amount of information becomes available on the Web, and it can be disseminated to clients in different ways such as the Internet and wireless networks. A typical example is realtime traffic report systems, e.g., the Travel Advisory News Network (TANN) [1] and SIGALERT [2] . In such a system, a server provides real-time traffic information to its subscribers. By using a handheld device, a subscriber can get the traffic conditions of the highways she is interested in. She can make queries such as "tell me the locations of the accidents (if any) between the Jeffrey Road and the Santa Ana Boulevard on Highway I-5 North." She can also register a set of queries to the server, such as "send me an alert on each weekday from 5 p.m. to 6 p.m. whenever the traffic speed between the Jeffrey Road and the Santa Ana Boulevard on Highway I-5 North is below 40 miles per hour." These systems share the following characteristics: 1) The data provided by the server is dynamically changing.. 2) It is critical for the latest information to be disseminated to the subscribers so that they can make timely decisions (e.g., for subscribers who are driving).. 3) The network bandwidth could be limited since the subscribers are relying on a wireless network to send queries and get information from the server.. Two communication paradigms are widely used to disseminate information. In the PUSH paradigm, the server sends updates to a subscriber every time there are changes to its data. Since the data stored at a subscriber (client) is always up to date, every query can be answered on the client side without contacting the server. This approach is preferable when the data is relatively static compared to the frequency of the queries. In the PULL paradigm, a subscriber contacts the server for each query, and the server does not need to push any updates to the client. This approach is more communication efficient when updates are relatively more frequent than queries.. In this paper, we study the following problem: In these systems, how can the PUSH and PULL paradigms be combined to reduce communication costs based on the distributions of the queries of each client and the data updates on the server? To see the motivation, consider the traffic report example. The queries asked by a subscriber could have a repetitive pattern. On each workday, she tends to ask for traffic conditions of the highway segments she needs to take between home and work. If these queries are very frequently asked, relatively to the data updates (changes of traffic conditions) on these segments, then it is communication efficient for the server and the client to use the PUSH paradigm for the data of these segments. On the other hand, the subscriber can ask queries for highway segments not on her list of "frequently asked queries." It is better to use the PULL paradigm for the client to get data for these infrequently asked segments. As a consequence, we can utilize the different distributions of the queries of a client and data updates on the server to partition the entire region of traffic information, and decide a PUSH and PULL paradigm for each area. We call the idea of partitioning the data space into PUSH-PULL regions to minimize communication cost data gerrymandering. The name is inspired by the concept of gerrymandering electoral districts to consolidate support for one political party. Here, we want to gerrymander data, i.e., partition it to combine regions that make the same PUSH-PULL choice.. We study technical challenges when adopting this simple but powerful idea. First, if we knew the pattern of the expected queries at a client and the data-update distribution on the server, what is the best way to partition the space and make a PUSH-PULL decision for each partition? We first study this optimization problem in the case where each client and the server do not have a limitation on how many different regions can be used for the two paradigms. We develop algorithms in Section 3 for the case where the client queries have conditions on a single attribute, such as a segment on a highway. The second challenge is how to do efficient gerrymandering when there are many clients in the system. As a consequence, the server may not be able to have too many semantic regions for a client due to storage and efficient-dissemination reasons. In addition, we also need efficient algorithms for deciding a PUSH=PULL-labeled partition with each client. We study this problem in Section 4, for both the case of a single attribute on query conditions and the general case of multiple attributes. The third challenge is how to do data gerrymandering when the queries on the client and the data updates on the server can change their distributions. The server needs to adapt to these changes by automatically detecting these changes and dynamically deciding a new labeled partition with a client. We present our solution to this problem in Section 5. We also conducted extensive experiments on both synthetic data and real data to evaluate our solutions to these challenges (Section 6).. For multiple-attribute data, it is worth emphasizing that we do not necessarily do gerrymandering on all its attributes because of two reasons: 1) Users may ask queries on a subset of these attributes, and the system returns objects including data of other informational attributes. This is true especially in Web-based applications that provide search forms. Typically, these forms include only some of the attributes for users to specify conditions on. In this case, the query-condition attributes are candidate gerrymandering attributes, and other attributes will not be considered in gerrymandering. 2) Even for the candidate attributes, we do not need to choose all of them to do gerrymandering. We can choose the gerrymandering attributes based on the skew in updates and queries, the complexity of computing a labeled partition, and the corresponding benefits in reducing communication costs. In Section 4.2, we discuss how to choose gerrymandering attributes and compute a labeled partition once they have been decided. We introduced data gerrymandering as a technique for reducing communication cost of queries on dynamic data in client-server environments using client-side caching. Its idea is to partition the semantic space of a subset of the attributes and decide a PUSH=PULL label for each region in the space, based on the distributions of queries and updates. We studied the technical challenges when adopting this simple but powerful idea. We developed a dynamic programming algorithm to find an optimal solution for a single gerrymandering attribute and proposed efficient heuristics to compute labeled partitions. We studied the optimization problem by considering various costs. We discussed how to choose a subset of attributes to do gerrymandering and how to compute labeled partitions for multiple attributes. We studied how to use gerrymandering for evolving workloads of queries and updates. Our extensive experiments on real-life and synthetic workloads show that this technique can significantly reduce communication costs. Semantic data caching was proposed in [3] , in which the client maintains a semantic description of the data in its cache, and decides whether it should contact the server for data not in the cache. The data needed from the server is specified as a remainder query. Dar et al. [3] focused on cache replacement policies, without considering data updates on the server. Our work differs from this previous work since we consider the case where the server data is dynamic, and the client site has enough cache to store data.. Our work is closely related to the predicate-based caching scheme proposed in [4] , which uses possibly overlapping query-based predicates to describe the cached data on the client. The server is responsible for notifying the client about data updates satisfying these predicates. There are two main differences between our work and theirs. First, our work uses the concept of a PUSH=PULL partitioning scheme for the purpose of formalizing the notion of using semantic regions to minimize communication costs. In our work, finding a good partitioning scheme requires a sophisticated communication model based on the distribution of queries and updates. Developing this allows us to systematically study the problem of minimizing communication cost. This problem is not studied thoroughly in the other work. Second, the work in [4] does not include any experimental evaluation to validate their heuristics. We develop efficient algorithms to solve the gerrymandering problem, including algorithms adopted from the heuristics proposed in [4] (Section 4.1). We also conduct an extensive experimental study on these algorithms.. Among other related work is caching in distributed database systems [5] , [6] , [7] , [8] , [9] , [10] . For instance, [8] , [9] studied how to cache pages of queries on dynamic data to reduce workload on database and Web servers. Tomasic et al. [11] and Candan et al. [9] studied how to use cache tables to reduce workload on a database server. In both cases, notification messages are sent from the server to the cache when there are data changes. Pierre et al. [12] proposed that each document at a client should have its own associated strategy for deciding whether to be pushed to a replication site or stay at the server. Other techniques take the capabilities and load at the servers and proxies, and clients' coherency requirements to adaptively decide whether data should be pushed or stored until it is pulled [13] . Some use data affinity to reduce cache misses [14] . Our work differs from these studies since it mainly focuses on how to partition the semantic space of data and compute PUSH=PULL labels in order to minimize communication cost, which is not studied by these works. Thus, our work complements the earlier studies.. The application of the notion of semantic regions to caching seems to be a natural notion which has been studied in other contexts. One example is Amiri et al.'s proposal for Content Distribution Networks [15] . They suggest a three-level architecture-Web server, application server, database-and propose a method for storing useful data at the application server to speed up Web applications. In the context of query processing, Haas et al. [16] consider the situation where queries may need to go back to the database to execute methods on objects they have accessed already. In this situation, they suggest that the client improves performance significantly by fetching and storing data that it might need to completely process the query.. There has also been interest in making broadcast environments adaptive by allowing clients to request data back channels [17] , [18] . The asymmetry inherent in the capabilities of nodes versus base stations in mobile networks makes this kind of capability particularly important [19] . Our approach involves partitioning the space into regions and studying their properties to make intelligent dissemination. One area of research which proceeds on similar lines involves constructing histograms over the range based on different parameters. For example, [20] described a method of forming workload-aware histograms based on the density of queries over the range. Another related area is to answer client queries using cached data with certain errors [21] . The difference is that our work assumes each query should be answered using the latest data. A similar idea of combining PUSH and PULL paradigms is developed in [22] to reduce communication cost to answer queries in sensor networks. A related work is [23] , which proposes techniques to disseminate dynamic information to clients by using cooperating repositories. In our work, we assume the server needs to disseminate information to clients without such cooperating repositories. In this section, we present our experimental results to demonstrate the usefulness of data gerrymandering and compare the performance of the proposed approaches for different scenarios.

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O


A Depth-optimal Low-complexity Distributed Wireless Multicast Algorithm This paper presents a wireless multicast tree construction algorithm, SWIM (Source-initiated WIreless Multicast). SWIM constructs a tree on which each multicast destination has the minimum possible depth (number of hops from the nearest source). It is proved that SWIM is fully distributed, with a worst case complexity upper-bounded by O(N 3 ), and an empirically found average complexity of only O(N 2 ). SWIM forms one shared tree from source(s) to the multicast destinations; yet, as a byproduct, it creates a multicast mesh structure by maintaining alternative branches at every tree node, thus providing robustness to link failures. This makes it suitable for both ad hoc networks and access networks with multiple gateways. In terms of minimizing the number of forwarding nodes, SWIM is optimal for unicast and competitive with the state of the art for multicast, outperforming the best known distributed approaches from the literature except for the multicast ad hoc on demand distance vector (MAODV) algorithm. However, simulations of the MAODV algorithm alongside SWIM on a large set of network instances show that the depth minimality of SWIM leads to lower average delay per multicast destination. It is also shown that the delay performance of SWIM is virtually unaffected by the presence of low mobility in the network. In multi-hop wireless networks, such as mobile ad hoc networks and mesh networks, multicast sessions from a source node, gateway (or set of gateways) to a group of destinations, often occur. Such sessions are sometimes delay sensitive (in the case of, for example, near-real-time applications). Rather than sending the data along unicast routes to each destination, it is desirable, primarily to avoid burdening the network with unnecessary transmissions, to construct a multicast route [1] . A good multicast tree will make efficient use of the bandwidthor energy-constrained wireless links [2, 3] , avoid bottlenecks at multicast source nodes and maximize energy efficiency by keeping the number of forwarding nodes (NFN) low, while maintaining a sufficiently low delay for each destination. To keep the delay low, it is of interest to make the depth of each destination on the multicast tree small. Our aim is to develop an algorithm that keeps the depth of a destination node to a minimum, while keeping the NFN as low as possible. Minimizing NFN requires a solution of the Wireless Multicast Tree (WMT) problem: finding a routing tree with the minimum number of transmissions needed to reach all destinations; the tree needs to dominate, but not necessarily include all destination nodes. An illustration of this is given in Fig. 1 .. The WMT problem is an NP-complete problem [4] . To see this, it suffices to show that any instance of the Steiner Tree problem can be converted to an instance of WMT in polynomial time (see Fig. 2 ): simply add dummy neighbor nodes to each of the nodes to be covered by the Steiner tree (except one,A Depth-optimal Low-complexity Distributed Wireless Multicast Algorithm 989 FIGURE 1. The multicast group is M = {A, B, C}. Node S is the source. For the wired case, the routing tree (i.e. the Steiner Tree) consists of the nodes {S, A, B, C, D}. This is the smallest tree containing {S}∪M. In the wireless case, allowing MAC layer broadcast (a.k.a. the wireless multicast advantage,) one transmission from D suffices to reach both A and B. So, the optimal WMT consists only of the nodes {S, D, B}. Interest in wireless multicast has risen rapidly in the last decade [12] . A number of multicast routing algorithms have been developed, focusing on different priorities such as low latency, energy efficiency and so on [13] . A notable multicast tree formation algorithm, for both weighted and not-weighted 990 A.S. Akyurek and E. U-Biyikoglu graphs, appears in [14] . This solution is based on merging optimum unicast routes and pruning the resulting subgraph. A different approach is presented in [15] : the objective is to select the minimum number of nodes in the network that are 'on' (and keep others turned 'off'), while keeping a communication path from the source to the destinations, by utilizing information about the geographic position of the nodes in the network. A minimum spanning tree is calculated on the final state to further reduce the number of 'on' nodes. Another notable multicast routing heuristic presented in [16] relies on clustering and a certain medium access control (MAC) protocol [17] .. There is a richer literature on wireless broadcast, which is a special case of the multicast problem. Much of the recent work on broadcast has considered energy efficiency, and power control [15, 18, 19] . Iterative Maximum-Branch Minimization, proposed in [20] , constructs an iterative mechanism for reducing power in a source-initiated wireless broadcast tree, with the objective of minimizing the total required power. Another iterative method with an energy-efficiency objective is presented in [21] , where integer programming has been used.. NP-hardness of the minimum energy broadcast problem in metric space was proved in [22] and, later in [23] , it was shown under more general conditions that power-optimal broadcast is NP-complete. Wieselthier et al. [18] proposed several broadcast tree heuristics: Broadcast Incremental Power, broadcast linkbased minimum-cost spanning tree and broadcast least unicast. It is worth noting that multicast versions of these, namely, multicast incremental power, multicast least unicast and multicast link-based minimum-cost spanning tree, have also been proposed. These algorithms have some commonalities with the algorithm proposed in this paper: worst case complexity of O(N 3 ), optimality in the unicast case and containing a sweeping operation to remove unnecessary transmissions. However, not all of these are distributed and, moreover, their closeness to the optimal tree in terms of the number of transmissions, in other words, NFN was not studied. Moreover, as these algorithms are based on power control, which is out of the scope of our treatment, they are not directly comparable with our solution.. More recently, an algorithm specifically developed for voice multicasting with the purpose of minimizing NFN was proposed in [24] ; however, this protocol does not address the multiplesource case.. Minimizing the depth of each destination from the nearest source ideally minimizes the maximum forwarding delay. It is harder to argue about queuing delay, or throughput under imperfect channels, with medium access limitations; however, these are harder to model as they require information about traffic patterns and channel state processes. But realistic performance metrics, such as Delivery Ratio, Average Delay and Maximum Delay, are also studied through a real-time event-based ns3 simulation environment. In addition, issues such as resilience to link failures, distributed implementation, messaging overhead and computational complexity will be addressed as primary concerns in the development of SWIM, which is intended for real-life implementation. The NFN properties of SWIM will also be studied. NFN, in contrast to tree depth, is more directly related to the transmission energy efficiency, rather than delay.. While a full experimental comparison, including realistic channel models and packet arrival models, requires a separate study, one can readily make a conceptual comparison between two well-known protocols and our proposal as follows: Simulations have been done to compare MAODV and SWIM. It should be noted that the simulations in this paper are on a static network model and hence are not conclusive as to which is a better solution under mobility. Our main aim is to find a tree, connecting a source (or set of sources) to a set of destinations, while finding the optimal depth for all destinations and keeping the NFN as low as possible.. We assume that there is a given set of links between nodes, forming a connected graph, and that each node knows about its one-hop neighbors on this graph. An important assumption is that the wireless multicast advantage will be exploited, that is, the MAC layer enables all one-hop neighbors to hear when a node makes a transmission.

O O O O B I O O O O O B I O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O B I I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O


Degraded Compound Multi-Receiver Wiretap Channels We study the degraded compound multi-receiver wiretap channel (DCMRWC). DCMRWC consists of two groups of users and a group of eavesdroppers, where, if we pick an arbitrary user from each group of users and an arbitrary eavesdropper, they satisfy a certain Markov chain. We study two different communication scenarios for this channel. In the first scenario, the transmitter wants to send a confidential message to users in the first (stronger) group and a different confidential message to users in the second (weaker) group, where both messages need to be kept confidential from the eavesdroppers. For this scenario, we assume that there is only one eavesdropper. We obtain the secrecy capacity region for the discrete memoryless channel model, the parallel channel model, and the Gaussian parallel channel model. For the Gaussian multiple-input multiple-output (MIMO) channel model, we obtain the secrecy capacity region when there is only one user in the second group. In the second scenario we study, the transmitter sends a confidential message to users in the first group which needs to be kept confidential from the second group of users and the eavesdroppers. Moreover, the transmitter sends a different confidential message to users in the second group which needs to be kept confidential only from the eavesdroppers. For this scenario, we do not put any restriction on the number of eavesdroppers. As in the first scenario, we obtain the secrecy capacity region for the discrete memoryless channel model, the parallel channel model, and the Gaussian parallel channel model. For the Gaussian MIMO channel model, we establish the secrecy capacity region when there is only one user in the second group.Index Terms-Compound wiretap channels, Gaussian multipleinput multiple-output (MIMO) compound wiretap channels, secrecy capacity region. I NFORMATION-theoretic secrecy was initiated by Wyner in his seminal work [1] , where he considered the degraded wiretap channel and established the capacity-equivocation rate region of this degraded channel model. Later, Csiszar and Korner generalized his result to arbitrary, not necessarily degraded, wiretap channels in [2] . In recent years, multiuser versions of the wiretap channel have attracted a considerable amount of research interest (see references [3] - [21] in [3] ). Among all these extensions, two natural extensions of the wiretap channel to the multiuser setting are particularly of interest here: secure broadcasting and compound wiretap channels.. Secure broadcasting refers to the situation where a transmitter wants to communicate with several legitimate receivers confidentially in the presence of an external eavesdropper. We call this channel model the multi-receiver wiretap channel.. Since the underlying channel model without an eavesdropper is the broadcast channel, which is not understood to the full extent even for the two-user case, most works on secure broadcasting have focused on some special classes of multi-receiver wiretap channels, where these classes are identified by certain degradation orders [4] - [9] . In particular, in [5] - [8] , the authors consider the degraded multi-receiver wiretap channel, where observations of all users and the eavesdropper satisfy a certain Markov chain. In [5] and [6] , the secrecy capacity region is derived for the two-user case, and in [7] and [8] , the secrecy capacity region is established for an arbitrary number of legitimate users. The importance of this result lies in the fact that the Gaussian multi-receiver wiretap channel belongs to this class, and the secrecy capacity region of the degraded multi-receiver wiretap channel serves as a crucial step in establishing the secrecy capacity region of the Gaussian multiple-input multiple-output (MIMO) multi-receiver wiretap channel [3] , [10] , though the latter channel is not necessarily degraded. In [3] , besides proving the secrecy capacity region of the Gaussian MIMO multi-receiver wiretap channel, we also present new optimization results regarding extremal properties of Gaussian random vectors, which we generalize here.. Another extension of the wiretap channel that we are particularly interested in here is the compound wiretap channel. In compound wiretap channels, there are a finite number of channel states determining the channel transition probability. The channel takes a certain fixed state for the entire duration of the transmission, and the transmitter does not have any knowledge about the channel state realization. Thus, the aim of the transmitter is to ensure the secrecy of messages irrespective of the channel state realization. In addition to this definition, the compound wiretap channel admits another interpretation. Consider the multi-receiver wiretap channel with several legitimate users and many eavesdroppers, where the transmitter wants to transmit a common confidential message to legitimate users while keeping all of the eavesdroppers totally ignorant of the message. Since each eavesdropper and legitimate user pair can be regarded as a different channel state realization, this channel is equivalent to a compound wiretap channel. Therefore, one can interpret a compound wiretap channel as multicasting a common confidential message to several legitimate receivers in the presence of one or more eavesdroppers [11] . In this study, we mostly refer to this interpretation, which is also the reason why we classify the compound wiretap channel as an extension of the wiretap channel to a multiuser setting.. Keeping this interpretation in mind, first works about the compound wiretap channel are due to Yamamoto [12] , [13] .. Yamamoto [12] , [13] considers the parallel wiretap channel with two subchannels where each subchannel is wiretapped by a different eavesdropper. [12] and [13] establish capacity-equivocation rate region for the situation where in each subchannel, the legitimate receiver is less noisy with respect to the eavesdropper of this subchannel. Other works which implicitly study the compound wiretap channel are [4] , [7] - [9] , and [14] , where works [4] , [7] , and [8] consider the transmission of a common confidential message to many legitimate receivers in the presence of a single eavesdropper, work in [9] focuses on two legitimate receivers one eavesdropper and one legitimate receiver two eavesdroppers scenarios, and [14] studies the fading wiretap channel with many receivers. Liang et al. [11] consider the general discrete compound wiretap channel and provides inner and outer bounds for the secrecy capacity. In addition to these inner and outer bounds, [11] also establishes the secrecy capacity of the degraded compound wiretap channel as well as its degraded Gaussian MIMO instance. Another work on the compound wiretap channel is [15] where the secrecy capacity of a class of nondegraded Gaussian parallel compound wiretap channels is established.. In this paper, we consider compound broadcast channels from a secrecy point of view, which enables us to study the secure broadcasting problem over compound channels. We note that the current literature regarding the compound wiretap channel considers the transmission of only one confidential message, whereas here, we study the transmission of multiple confidential messages, where each of these messages needs to be delivered to a different group of users in perfect secrecy. Hereafter, we call this channel model the compound multi-receiver wiretap channel to emphasize the presence of more than one confidential message. The compound multi-receiver wiretap channel we study here consists of two groups of users and a group of eavesdroppers, as shown in Fig. 1 . We focus on a special class of compound multi-receiver wiretap channels which exhibits a certain degradation order. If we consider an arbitrary user from each group and an arbitrary eavesdropper, they satisfy a certain Markov chain. In particular, we assume that there exist two fictitious users. The first fictitious user is degraded with respect to any user from the first group, and any user from the second group is degraded with respect to the first fictitious user. There exists a similar degradedness structure for the second fictitious user in the sense that it is degraded with respect to any user from the second group, and any eavesdropper is degraded with respect to it. Without eavesdroppers, this channel model reduces to the degraded compound broadcast channel studied in [16] . Adapting their terminology, we call our channel model the degraded compound multi-receiver wiretap channel (DCMRWC). Here, we consider the general discrete memoryless version of DCMRWC as well as its specializations to the parallel DCMRWC, the Gaussian parallel DCMRWC, and the Gaussian MIMO DCMRWC. We study two different communication scenarios for each version of DCMRWC.. In the first scenario, which is illustrated in Fig. 2 , the transmitter wants to send a confidential message to users in the first group, and a different confidential message to users in the second group, where both messages need to be kept confidential from the eavesdroppers. For this scenario, we assume that there exists only one eavesdropper and obtain the secrecy capacity region in a single-letter form. While obtaining this result, the presence of the fictitious user between the two groups of users plays a crucial role in the converse proof by providing a conditional independence structure in the channel, which enables us to define an auxiliary random variable that yields a tight outer bound. After establishing single-letter expressions for the secrecy capacity region, we consider the parallel DCMRWC. For the parallel DCMRWC, we obtain the secrecy capacity region in a single-letter form as well. Though the general discrete memoryless DCMRWC encompasses the parallel DCMRWC as a special case, we still need a converse proof to establish the optimality of independent signalling in each subchannel. After we obtain the secrecy capacity region of the parallel DCMRWC, we consider the Gaussian parallel DCMRWC. In particular, we evaluate the secrecy capacity region of the parallel DCMRWC for the Gaussian case, which is tantamount to finding the optimal joint distribution of auxiliary random variables and channel inputs, which is shown to be Gaussian. We accomplish this by using Costa's entropy power inequality [17] . Finally, we consider the Gaussian MIMO DCMRWC and evaluate its secrecy capacity region when there is only one user in the second group. We show the optimality of a jointly Gaussian distribution for auxiliary random variables and channel inputs by generalizing our optimization results in [3] .. In the second scenario we study here, which is illustrated in Fig. 3 , the transmitter wants to send a confidential message to users in the first group which needs to be kept confidential from users in the second group and eavesdroppers. Moreover, the transmitter sends a different confidential message to users in the second group, which needs to be kept confidential from the eavesdroppers. If there were only one user in each group and one eavesdropper, this channel model would reduce to the channel model that was studied in [18] . However, here, there are an arbitrary number of users in each group and an arbitrary number of eavesdroppers. Hence, our model can be viewed as a generalization of [18] to a compound setting. Adapting their terminology, we call this channel model the DCMRWC with layered messages. We first obtain the secrecy capacity region in a single-letter form for a general discrete memoryless setting, where again the presence of fictitious users plays a key role in the converse proof. Next, we consider the parallel DCMRWC with layered messages and establish its secrecy capacity region in a single-letter form. In this case as well, we provide the converse proof which is again necessary to show the optimality of independent signalling in each subchannel. After we obtain the secrecy capacity region of the parallel DCMRWC with layered messages, we evaluate it for the Gaussian parallel DCMRWC with layered messages by showing the optimality of a jointly Gaussian distribution for auxiliary random variables and channel inputs. For that purpose, we again use Costa's entropy power inequality [17] . Finally, we consider the Gaussian MIMO DCMRWC with layered messages and evaluate its secrecy capacity region when there is only one user in the second group. To this end, we show that jointly Gaussian auxiliary random variables and channel inputs are optimal by extending our optimization results in [3] . In this paper, we studied two different communication scenarios for the DCMRWC. In the first scenario, the transmitter wants to send a confidential message to users in the first group, and a different confidential message to users in the second group, where both messages are to be kept confidential from an eavesdropper. We establish the secrecy capacity region of the general discrete memoryless channel model, the parallel channel model, and the Gaussian parallel channel model. For the Gaussian MIMO channel model, we obtain the secrecy capacity region when there is only one user in the second group.. In the second scenario we study, the transmitter sends a confidential message to users in the first group which is wiretapped by both users in the second group and eavesdroppers. In addition to this message sent to the first group of users, the transmitter sends a different message to users in the second group which needs to be kept confidential only from the eavesdroppers. In this case, we do not put any restriction on the number of eavesdroppers. As in the first scenario, we establish the secrecy capacity region for the general discrete memoryless channel model, the parallel channel model, and the Gaussian parallel channel model. For the Gaussian MIMO channel model, we obtain the secrecy capacity region when there is only one user in the second group.

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O B I I B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O


Crowdsourcing service-level network event monitoring The user experience for networked applications is becoming a key benchmark for customers and network providers. Perceived user experience is largely determined by the frequency, duration and severity of network events that impact a service. While today's networks implement sophisticated infrastructure that issues alarms for most failures, there remains a class of silent outages (e.g., caused by configuration errors) that are not detected. Further, existing alarms provide little information to help operators understand the impact of network events on services. Attempts to address this through infrastructure that monitors end-to-end performance for customers have been hampered by the cost of deployment and by the volume of data generated by these solutions.We present an alternative approach that pushes monitoring to applications on end systems and uses their collective view to detect network events and their impact on services -an approach we call Crowdsourcing Event Monitoring (CEM). This paper presents a general framework for CEM systems and demonstrates its effectiveness for a P2P application using a large dataset gathered from BitTorrent users and confirmed network events from two ISPs. We discuss how we designed and deployed a prototype CEM implementation as an extension to BitTorrent. This system performs online service-level network event detection through passive monitoring and correlation of performance in end-users' applications. The Internet is increasingly used as a platform for diverse distributed services including online multiplayer gaming, content Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. SIGCOMM'10, August 30-September 3, 2010, New Delhi, India. Copyright 2010 ACM 978-1-4503-0201-2/10/08 ...$10.00. distribution and IPTV. Given the popularity and potential for revenue from these services, their user experience has become an important benchmark for service providers, network providers and end users [17] .. Perceived user experience is in large part determined by the frequency, duration and severity of network events (e.g., outages, route changes or misconfigurations) that impact a service. There is thus a clear need to detect, isolate and determine the root causes of these service-level network events so that operators can address them in a timely manner, minimizing their impact on revenue and reputation. In this work, we develop a practical monitoring approach enables online detection (within seconds or minutes) of network events impacting the user experience for services at the network edge.. While today's networks generally implement sophisticated infrastructure that detects and issues alarms when core network elements fail, there remains a class of events that often go undetected -the so-called silent failures. Configuration errors (e.g, incorrect ACL settings), routing anomalies (e.g., routing loops), and router bugs (simply because routers are incapable of detecting their own internal failures) are common causes for silent failures that can impact performance for services. Beyond these issues, in-network alarms fail to provide information to help operators understand the impact of network events, nor do they assist in detecting events caused by external ISPs. Despite efforts to use infrastructure to monitor end-to-end performance [16, 17, 36] , the cost of deployment, the number of services to monitor and the volume of data generated by these solutions hampers their network visibility and limits their scalability and effectiveness. This paper presents an alternative approach to detecting, isolating and reporting service-level network events -we call this approach CEM, for Crowdsourcing Event Monitoring. The key idea behind CEM is to push service-level event monitoring to the end systems where the services are used. Building on end systems has a number of clear advantages. First, the approach provides flexibility in the types of monitoring software that can be installed inside or alongside services, facilitating immediate and incremental deployments. Second, by leveraging the unique perspective of participating end systems, it offers the potential for broad network visibility into an increasingly opaque Internet. Finally, its collaborative model enables a highly robust and more scalable system by drawing from every node's resources and avoiding any centralized components.. Detecting events from the network edge also poses a number of interesting challenges. First, any practical approach must address scalability constraints imposed by managing information from potentially millions of end systems. Second, to assist operators in addressing problems promptly, events should be detected quickly (i.e., within minutes), isolated to specific networks (e.g., BGP prefixes) and this information should be available to ISPs for troubleshooting and problem mitigation. Last, the approach must facilitate a broad (Internet-scale) deployment of edge-system monitors that ensures user privacy and provides trustworthy event detection. We show how our approach addresses these challenges and present the Network Early Warning System (NEWS), a proofof-concept implementation for online event detection in BitTorrent.. In Sec. 2, we describe the challenges faced by end-system monitoring in more detail and discuss potential solutions. Sec. 3 addresses the general problem of how to detect network performance events from the edge. Specifically, we develop a framework for our CEM approach in which each end system performs a significant portion of event detection locally, then uses a distributed approach to corroborate their findings.. Demonstrating the effectiveness of any edge-based approach is challenging due to the lack of representative testbeds and the sheer scale and diversity of networks worldwide. In Sec. 4, we address this issue using a large dataset of diagnostic information from edge systems, gathered from users running the Ono plugin [8] for the Vuze BitTorrent client. We use our findings to motivate the design and implementation of the Network Early Warning System.. We present results of our evaluation in Sec. 5 . In addition to comparing NEWS-detected events with confirmed ones, we demonstrate that our crowdsourcing approach detects network events worldwide, including events spanning multiple networks. Our approach is robust to various parameter settings and incurs reasonably low overhead.. NEWS has already been installed 45,000 times, demonstrating not only the feasibility of our approach for a real application, but also that there are appropriate incentives for widespread adoption (Sec. 6). We are currently investigating other potential hosting applications and services for our CEM approach. Last, to assist with quickly resolving problems causing detected network events, we have implemented NEWSight 1 -a system that accesses live event information gathered by NEWS and publishes its results. We are currently beta-testing NEWSight interface with ISPs. The user experience for networked applications is becoming an important benchmark for customers and network providers. To assist operators with resolving such issues in a timely manner, we argued that the most appropriate place for monitoring servicelevel events is at the end systems where the services are used. We proposed a new approach, called CEM for Crowdsourcing Event Monitoring, based on pushing end-to-end performance monitoring and event detection to the end systems themselves. We presented a general framework for CEM systems and demonstrated its effectiveness using a large dataset of diagnostic information gathered from peers in the BitTorrent system, along with confirmed network events from two different ISPs. We showed that our crowdsourcing approach enables worldwide event detection, including events spanning multiple networks. Finally, we designed, implemented and deployed a BitTorrent extension that performs online event detection using our approach -currently installed more than 45,000 times. Having demonstrated the feasibility and effectiveness of this approach, we are investigating opportunities for porting it to other host applications such as VoIP and streaming video. The problem of detecting network events (or anomalies) has attracted a large number of research efforts. In this context, CEM is a framework for online detection of network events that impact performance for applications running on end systems. This section classifies key properties of event detection systems and describes how CEM relates to previous work in these areas.. Crowdsourcing. Central to our approach is the idea of crowdsourcing event detection to ensure good coverage and accuracy at the scale of hundreds of thousands of users. This model has successfully enabled projects that include solving intractable [33] or otherwise prohibitively expensive problems [1] using human computation. Unlike these examples, our system passively monitors network activity from each member of a crowd, but it does not require human input. Dash et al. [10] use a similar model to improve the quality of intrusion detection systems in an enterprise network and demonstrate its effectiveness through simulation using traffic data from 37 hosts inside their enterprise network.. Event types. A class of previous work focuses on detecting network events in or near backbone links, using data gathered from layer-3 and below [13, 18, 19, 21, 29] . While these monitors can accurately detect a variety of events, they may miss silent failures (e.g., incompatible QoS/ACL settings) and their impact on performance. Other work focuses on detecting network events from a distributed platform [2, 16, 20, 36] . These solutions do not correlate these events with user-perceived performance, and their detection is limited by their network visibility and/or the overhead for probing large numbers of networks. The goal of CEM is to detect service-level network events and correlate their impact on application performance from the perspective of end users.. Monitoring location. CEM targets events that impact userperceived application performance, by running on the end systems themselves. While several researchers have proposed using endhost probing to identify routing disruptions and their effect on endto-end services [12, 16, 35, 37] , they have focused on GREN [20, 36] or enterprise [10, 15, 26] environments and have not looked at the impact of network events on application performance nor addressed the issues of scalability when running on end systems. Some commercial network monitoring tools generate flows that simulate protocols used by edge systems (e.g., Keynote and IneoQuest 8 ). While these can indeed detect end-to-end performance problems, these tools require controllable, dedicated infrastructure and are inherently limited to relatively small deployments in PoPs. Our CEM approach does not require any new infrastructure, nor control of end systems, and thus can be installed on systems at the edge of the network. Several research efforts have investigated the idea of active and passive network measurement from end users, e.g., DIMES [31] and Neti@home [32] , but have not explored the use of their monitoring information for online network event detection.. Measurement technique. CEM focuses on passive monitoring of popular applications to detect events, which allows our approach to scale to the vast numbers of users at the edge of the network while still detecting events quickly. In a similar vein, previous work has suggested that the volume and breadth of P2P systems' natural traffic could be sufficient to reveal information about the used network paths without requiring any additional measurement overhead [9, 36] . PlanetSeer [36] uses passive monitoring of a CDN deployed on PlanetLab, but relies on active probes to characterize the scope of the detected events. Casado et al. [3] and Isdal et al. [14] use opportunistic measurement to reach these edges of the network, by leveraging spurious traffic or free-riding in BitTorrent. Unlike these efforts, CEM takes advantage of the steady stream of natural, (generally) benign traffic generated by applications. Approaches that use active monitoring (e.g., [2, 16] ) are limited by the overhead for detection, which grows with the number of monitored networks and services. While CEM could be combined with limited active probes to assist in characterizing and localizing network events, it does not require them.

B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Measuring and predicting web login safety Users increasingly entrust websites with their personal and sensitive information. Sites commonly protect this information using user-supplied credentials (i.e., logins). We conducted a measurement study of top websites and surprisingly found that they transmit these credentials in the clear, thus leaving them vulnerable to eavesdropping. To make matters worse, users are often unaware of this threat because sites and browsers reflect little information about how logins are handled.As a first step towards solving this problem, we develop techniques for measuring logins on browsers to predict how logins would be handled before they are submitted. We demonstrate that achieving this goal requires instrumentation at the application layer and inside browsers. Specifically, network traces are not sufficient for determining login safety in general due to application-layer encryption; similarly, application-layer traces are insufficient because login submission logic may be generated in the browser at runtime. Based on a measurement study using login pages gathered from popular sites in addition to those visited by users through normal Web browsing, we found such predictions to be quite challenging due to a lack of any standard formats for Web logins. However, by applying a carefully chosen set of rules when measuring logins, we almost always correctly predict how logins will be handled. Internet users commonly entrust websites with personal and sensitive information pertaining to their identities, finances, and interests. To prevent other parties from accessing this data, websites typically employ login-based access control that entails sending credentials (usually a username and password) to a remote server for authentication. As such, login credentials are arguably the most sensitive information transferred on the Web and should receive strong protection. If compromised, all of the private data that they protect is at risk.. Despite the importance of website logins, their security has generally received little attention. We tested the top 100 websites in the United States according to Alexa [1] , and found that credentials were exposed to eavesdroppers for fully one third of the sites. Surprisingly, a top site wikipedia.org and an insurance site progressive.com transmit logins in the clear. The threat posed by such exposure is real -it is trivial for an attacker with freely available traffic monitoring software to recover user credentials from any unencrypted Wi-Fi connection (e.g., at public hotspots such as airports and cafes). While the technical means to protect logins is clear -encrypting them with HTTPS -it is not widely deployed for a variety of operational reasons.. We argue that transparency is a critical missing feature for improving privacy [16] . Many studies show that websites send personal information to third parties without user awareness or consent. Users can choose to avoid such exposures if armed with knowledge about potential risks by simply not visiting those websites or visiting them on a secured network; further, making Web sites aware of such risks may encourage them to deploy secure authentication schemes. In the context of Web logins, our approach to enabling users to make informed decisions is to measure and predict Web login safety, then make this information available to users in an unobtrusive and intuitive way. Thus even without changing how and when passwords are encrypted, our approach can alleviate the risks of Web logins. Further, this approach only requires browsers to enable third-party code to observe how logins will be sent, enabling an immediately deployable implementation through browser extensions.. The goal of this work is to reduce this risk by notifying users about how their logins are protected before they are sent. Unlike existing interfaces [3] that notify users when the current page has been loaded via an encrypted connection (e.g., the HTTPS padlock icon), our approach is to predict how and where upcoming login requests will be sent by measuring logins in Web pages. We argue that such information will help users make informed decisions about how they share credentials with sites they visit.. While the goal is straightforward, our measurements of Web logins reveal a number of interesting challenges. First, network traces cannot help here because we cannot find logins (to evaluate their security) in traces that are encrypted by higher level protocols; application-level traces also do not work because they do not reliably contain information about how logins are submitted. Second, there is a wide variety of non-standard login implementations because Web pages are constructed not only with HTML, but also with Turing complete languages (e.g., Javascript), suggesting the need for flexible heuristics to identify logins. Third, we find that determining how login credentials are submitted is difficult due to the variety of technologies currently used for clientserver communication. Last, our approach must run in real time while users interact with Web logins, so that we can notify them of how their logins would be transferred before they are submitted.. In this paper, we address these challenges by using a measurement study of 188 popular websites to develop a set of heuristics for determining Web login security. Our main contributions are as follows:. • We identify popular approaches to implementing Web logins, finding that there are more than six kinds of login buttons, only four of which can be identified directly. We also find four ways that logins are submitted, one of which enables perfect prediction of submission URLs at runtime.. • We use these results to build a Firefox extension that detects logins and predicts, in real time, whether logins will be protected during transmission. Based on an early deployment, we find that the rules for detecting login buttons work well for more than 93% sites; rules for predicting whether logins are sent encrypted and the top-level domains are nearly perfectly accurate.. The remainder of the paper is organized as follows. We discuss related work in the next section. In Section 3 we formally define the goals in this work and discuss the associated challenges. Section 4 presents the results from a measurement study to determine how popular Web sites implement logins and discusses the rules that we designed to identify and predict logins. In Section 5 we evaluate our rules. We discuss open issues in Section 6 and conclude in Section 7. This paper demonstrates an important first step toward improving the privacy of Web interactions by informing users whether their Web login credentials are protected before they are transmitted to a server. We used a study of 188 popular Web sites to demonstrate that it is possible to predict, in real time, whether Web logins are transmitted with encryption and where this information will be sent. By making this information available to users when they enter login information, we enable informed decisions about whether to submit login credentials. We implemented this approach as a Firefox extension and made it available to users. Based on results gathered from beta testing, we show that our software can accurately identify 93% of logins and almost always correctly predicts whether login credentails are transmitted securely. Our results indicate that it is feasible to use in-browser measurements to predict Web login safety and reflect this information to users in real time. While we were able to achieve good results for popular websites, there remain a number of open issues.. For one, manually inspecting and identifying Web logins for incorporating into our rule set simply does not scale. To address this limitation, we are currently experimenting with a crowd-sourcing approach that relies on information about Web logins collected from Piigeon. Our goal is to use this information to further extend our current rule set, then deploy these rules to users. An interesting area of future work is determining how to automatically generate and share these rules, e.g., using heuristic, statistical and/or machine learning approaches. We do not plan to use taint analysis, not only because it will introduce high overheads that may not work in real time, but also because literal search works well for tracking passwords.. While crowd-sourcing is promising, there remain some Web logins that our tool cannot identify and thus cannot be collected from users. One potential solution to this problem is to extend the DOM to make identifying Web logins easier. In particular, if the DOM were to tie HTTP requests to information from password boxes, the login submission problem would become trivial. The drawback of this approach is that it requires browser kernel modifications and is thus difficult to deploy broadly and uniformly across browsers.. Another issue is that users have limited options when notified about an unsafe Web login -they can leave the website or submit logins despite the risk. To address this, we plan to automatically detect whether sites support HTTPS and offer users to switch to HTTPS. We also plan to make our ongoing results publicly available through an interactive website, both to provide motivation for unsafe sites to improve how they protect Web logins and to enable non-Piigeon users to make informed choices about which sites to use.. Finally, we plan to extend our approach to the more general problem of privacy in Web interactions. One of the key challenges is identifying private information (e.g., names, addresses and phone numbers) outside the limited scope of logins. Another important research direction is moving be-yond the Web client side of interactions and inferring how private information is stored and shared among sites. Our work is closely related to approaches that improve the transparency of how private data is shared in general, and to approaches that attempt to enforce secure Web browsing.. Informing users of their privacy. Informing users of their exposed privacy is an alternative (to technical solutions, e.g., encryption) that can improve privacy [16] . Unfortunately, both goals (i.e., informing users and reflecting privacy in practice) are rarely met at the same time. Most sites provide users with privacy policies that are often difficult to read [14, 13] and sometimes even incorrect. Browsers provide padlocks and other cues to indicate page encryption. However, these cues are often ignored or misunderstood [17] .. The Web of Trust (WOT) [18] project solicits privacy and trustworthiness ratings of websites from users and reflect this information to users as they browse websites. However, a user's trust in WOT is only as strong as their trust in the users -potentially non-experts -who rate sites. Other work measures the private information flows in practice without reflecting them to users in real time [12, 2, 10] . In addition, these approaches do not predict the privacy of future actions and thus is not sufficient to inform users how their data will be protected before it is transmitted.. To reflect exposed privacy in practice, we argued that there should be low-level support to measure and reflect privacy on Web and mobile applications [16] . This work is a first step towards both measuring and reflecting a piece of private information (whether logins are sent encrypted) in practice. By predicting this information beforehand, our work enables users to take action to protect their own privacy before exposing information.. Adopting HTTPS. A common approach to providing privacy for Web interactions is to encrypt all transmissions between a client and a server using HTTPS. At a minimum, Web traffic related to the most sensitive information (e.g., logins and financial information) should be transmitted using HTTPS. Unfortunately, while all moderns browsers support HTTPS, the protocol requires support at both endpoints and thus the security of Web site interaction largely depends on Web sites. A number of solutions attempt to encourage or enforce encrypted Web interaction. One approach is to exert pressure on websites to support HTTPS, e.g.the Electronic Frontier Foundation (EFF) [4] efforts surrounding the Firesheep [6] extension. Another solution is enforcing HTTPS on browsers, based on the hypothesis that many websites support encryption but use HTTP by default. In this vein, the HTTPS-Everywhere [8] extension manually inspects a list of pages/requests that support encryption and enforces HTTPS for those pages. This approach is not feasible at large scale and is not robust to website changes over time. A similar extension, HTTPS-Finder [9] , automatically detects pages that support encryption and enforces HTTPS on those pages. However, it only enforces HTTPS of loaded Web pages and does not apply to data transmitted to Web servers (e.g.credentials).

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B O O O O O O O B O O O O O O O O O O O O O O O