On the scale and performance of cooperative Web proxy caching While algorithms for cooperative proxy caching have been widely studied, little is understood about the performance of cooperative caching in the Web environment as it scales to large client populations. This paper uses both tracebased analysis and analytic modelling to show the potential advantages and drawbacks of inter-proxy cooperation. With our traces, we evaluate quantitatively the performance-improvement potential of cooperation between 200 small-organization proxies within a university environment, and between two large-organization proxies handling 20,000 and 60,000 clients, respectively. With our model, we go beyond those populations to project the implications for cooperative caching in regions with millions of clients. Overall, we demonstrate that cooperative caching has performance benefits only within limited population bounds. We also use our model to examine the implications of future trends in Web-access behavior and traffic. Cooperative caching -the sharing and coordination of cache state among multiple communicating caches -has been shown to improve the performance of file and virtualmemory systems in a high-speed local-area network environment [3, 16] . For example, when a file-page miss occurs, the local file cache may transfer the page from the file cache on another node. Cooperative caching works in this environment, because network transfer time is much smaller than the disk access time required to service a miss.. In the context of the Internet, proxy caches have become commonplace to improve the performance of client browsers. Typically, the proxy sits above an entire company or organization. By caching requests for a group of users, a proxy can quickly return documents previously accessed by other clients. Ultimately, though, the hit rate of the proxy is a function of the size of the population it manages -a size often dictated by political, organizational, or geographic considerations. An obvious question, then, is whether multiple proxies should cooperate with each other in order to increase total client population, improve hit ratios, and reduce document-access latency. Whether such cooperative proxy caching will work depends on the sharing patterns of documents across organizations and the ratio of inter-proxy communication time to server fetch time.. Several cooperative-caching protocols have been proposed [9, 34, 15, 28, 30] ; however, few studies have examined cooperative Web caching from a systemic viewpoint. As a result, we know neither the environments in which cooperative caching pays (if any) nor its potential performance benefits. Answering such questions has been difficult in the past, because studying proxy cooperation requires simultaneous traces from multiple proxies.. In this paper, we take a two-pronged approach to exploring the limits and potentials of cooperative proxy caching. As the first approach, we collect and analyze traces from two environments: a large university environment and the Microsoft corporation. As a key component of our university trace, we identify each client in terms of its membership in one of about 200 university departments or programs. This gives us the equivalent of a simultaneous trace of 200 diverse, independent organizations, permitting us to analyze document sharing among those organizations, and to measure the potential benefits of cooperation among organization-based proxies. We examine latency and bandwidth benefits of proxy caching for this data as well. We then use the Microsoft trace of employee traffic to the Internet to explore the potentials of cooperation between larger organizations. To do this, we analyze traces from Microsoft and the university that we collected over the same time period and processed with the same anonymization function. This permits a direct computation of the degree of document sharing, and hence the benefit of sharing, between two proxies handling tens of thousands of clients each.. As the second approach, we develop an analytic model of Web behavior in order to extend beyond the limits of our trace results. The model permits us to examine the impact of larger populations sizes, to explore the tradeoffs between various cooperative-caching schemes, and to speculate on the performance implications of future trends.. Our results show the benefits of cooperative caching among collections of small organizations. However, we show that cooperative caching is unlikely to have significant benefits for larger organizations or populations. That is, with current sharing patterns, there is little point in designing scalable cooperative-caching schemes; all reasonable schemes will have similar performance in the low-end population range where cooperative caching works. Thus, the crucial problem that must be solved to improve Web performance is to increase document cacheability.. The paper is organized as follows. The following section describes previous work and its relationship to our study. Section 3 describes our traces, and presents and analyzes data from the traces. Section 4 develops an analytic model of Web behavior and presents results gained from that model. Section 5 summarizes and concludes. In this section, we analyzed traces from two environments to explore the bounds of cooperative-caching performance. From this data, we saw that:. 1. The behavior of cooperative caching is characterized by two different regions of the hit rate vs. population curve. For smaller populations, hit rate increases rapidly with population; it is in this region where cooperative caching can be used effectively. However, these population sizes can be handled by a single proxy. Therefore cooperative caching is only necessary to adapt to proxy assignments made for political or geographical reasons.. 2. For larger populations (beyond the knee of the population vs. hit rate curve), cooperative caching is unlikely to provide significant benefit. We demonstrate this using our simultaneous traces of the Microsoft and university populations: a 4-fold increase to a large university population via cooperative caching netted only a 2.7% increase in cacheable hit rate.. 3. Although there exist organizations with significant sharing rates and locality, such as the Microsoft population, our clustering study shows that cooperative caching specialized to interest groups is unlikely to be effective.. 4. While some of these general trends have been observed before, the inevitable conclusion for cooperative caching has not been explicitly noted.. Despite these negative conclusions, it is still possible that cooperative caching will perform better, either in steady state or for different (possibly future) Web characteristics. In the domain where cooperative caching is effective, we also wish to understand which of the proposed cooperative-caching algorithms will be most effective. We explore these questions in the next section.. In this section, we developed an analytic model and used it to examine the steady-state performance of cooperativecaching schemes. Our model extends the results from our trace for a wider range of parameter values, including for example, document popularity and document rate of change. We show again that relatively small populations achieve most of the performance benefits of cooperative caching.. We also evaluate three alternative approaches to cooperative caching, and show that all three schemes perform well in the region where cooperative caching is advantageous (e.g., at the level of a medium-sized city). However, a hash-based scheme will achieve the best storage efficiency. In a broader area (e.g., the size of the West Coast), the increased latency of inter-proxy communication overshadows the (very limited) benefits of increased population. This paper studied cooperative proxy caching in local-and wide-area environments. We use a combination of tracebased analysis and analytic modelling to evaluate cooperative caching, and proxy caching in general, at a wide range of population sizes, document characteristics, and access patterns. At a high level, our results show that:. 1. In the absence of significant changes in client behavior, there is little point in continuing to expend effort on the design and evaluation of highly-scalable cooperative-caching schemes. The scale at which cooperative caching makes sense, e.g., up to the level of a medium-sized city, is sufficiently small that most reasonable schemes will achieve most of the benefit.. 2. The largest benefit for cooperative caching is achieved for relatively small populations. This is demonstrated by our analysis of cooperation among small organizations within the university environment. Our simultaneous traces of the university and Microsoft confirmed the negligeable benefit of cooperative caching between organizations whose populations are already in the range of 20K clients. Scaling beyond such populations provides only marginal improvement, and therefore makes sense only in very high-bandwidth low-latency environments.. 3. Performance at the population level at which cooperative caching works effectively is basically lim- ited by document cacheability. Therefore, increasing cacheability of documents is the main challenge for research aimed at improving Web cache behavior.. 4. Cluster-based analysis of client access patterns indicates that wide-area cooperative-caching organizations based on mutual interest offer no obvious advantages over randomly-assigned or organizationbased groupings.. 5. Given observed document popularity distributions, hash-based cooperative caching makes the most sense in terms of avoiding unnecessary replication of documents within a small (low-latency) region.. Finally, we feel that in practice, cooperative-caching schemes on the scale of up to a medium-sized city could be possible, with many small organizations cooperating. We believe that this cooperation might be best achieved by using a small number of proxy caches (e.g., ten) at the second level of a cooperative-caching hierarchy, with a hashbased scheme used to distribute the load and storage requirements among these second-level caches. Such a system would achieve near optimal latency, bandwidth, and storage performance. Web tracing and caching are highly active research areas. Recent studies of Web traffic include analyses of Web access traces from the perspective of browsers [11, 25] , proxies [10, 2, 23, 14, 20, 13, 28, 6, 7, 5, 17] , and servers [27, 1, 4] . The earlier tracing studies were rather limited in request rate, number of requests, and diversity of population. The most recent tracing studies have been larger and generally more diverse. In addition to static analysis, some studies have also used trace-driven cache simulation to characterize the locality and sharing properties of these very large traces [2, 23, 14, 20, 6, 17] , and to study the effects of cookies, aborted connections, and persistent connections on the performance of proxy caching [6, 17] .. Researchers have studied the temporal locality of Web proxy traces and examined how hit-ratio depends, asymptotically, on cache size and the number of requests. Several interesting findings have been observed. First, it has been well-documented that for most traces, the relative frequency with which Web pages are requested follows a Zipf-like distribution [5, 19, 11, 2, 8, 31] . Second, for infinite-sized caches, it has been shown empirically and analytically that the hit ratio for a Web proxy grows logarithmically with the client population of the proxy and the number of requests seen by the proxy [5, 8, 31, 20, 21, 14] .. There has also been extensive work on cooperative Web caching as a technique to reduce access latency and bandwidth consumption. Cooperative Web caching proposals include hierarchical schemes like Harvest and Squid [9, 29] , hash-based schemes [33, 22] , directorybased schemes [15, 30, 26] and multicast-based schemes [34, 32] . Although each of these research efforts included a performance evaluation of the protocols proposed and a discussion of algorithm scalability, none of these evaluations presented either empirical or analytical evaluations of the effectiveness of their schemes for large scale client populations as found in a wide-area setting.. In this paper, we expand on these previous research efforts. We use trace-based analysis to quantify the potential advantages and drawbacks of inter-proxy cooperation for small and medium-sized organizations. We use analytic modelling to examine cooperative-caching performance in wider-area environments. Our model is inspired by the model of Breslau et al. [5] . We have augmented their assumptions with a model of document rate of change, client request rate, and a small number of other parameters that allow us to study the steady-state behavior of cooperativecaching schemes as a function of the characteristics of the population they serve.

O O B O B O O B B O O O O O B O O O O O O O O O O B O O O O O B O O O O O O O O O B O O O B O O O O O O O O O O O O B O O B O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O B O O O O B O O O O O O O O B O B O O O O O B O O O B O B O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B O O O O O O B O O O O O B O O O O O O B O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O B B O O B O O O O O B O O O O O O O O O O O O O B O O O O O O O O O B O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O B O O O O O B O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O B B O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O B O O O O O B O B B O O O O O O B O O O O O O O O O O B O O B O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B O O O O B O B O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B B O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B B O O O O O O O O O O O B B O O O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B O B O O O O O O O O O O O B B B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B B O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B O O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O B O B O O O O O O O O O B O O O O O O B O O O O O B O O B O B O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O


New perspectives and methods in link prediction This paper examines important factors for link prediction in networks and provides a general, high-performance framework for the prediction task. Link prediction in sparse networks presents a significant challenge due to the inherent disproportion of links that can form to links that do form. Previous research has typically approached this as an unsupervised problem. While this is not the first work to explore supervised learning, many factors significant in influencing and guiding classification remain unexplored. In this paper, we consider these factors by first motivating the use of a supervised framework through a careful investigation of issues such as network observational period, generality of existing methods, variance reduction, topological causes and degrees of imbalance, and sampling approaches. We also present an effective flow-based predicting algorithm, offer formal bounds on imbalance in sparse network link prediction, and employ an evaluation method appropriate for the observed imbalance. Our careful consideration of the above issues ultimately leads to a completely general framework that outperforms unsupervised link prediction methods by more than 30% AUC. Link prediction is an important task in network science that offers unique ways whereby the study of networks can Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. benefit researchers and organizations in a variety of fields. Security agencies can more precisely focus their efforts based on probable relationships in malicious networks that have heretofore gone unobserved [10] . In social networks, individuals can efficiently and effectively find companions, assistants, or colleagues [9] . In medicine and biology, link prediction can be used to find relationships and associations that exist, but which might otherwise surface only after arduous and expensive research and study on a huge selection of agents. Finally, researchers can easily adapt link prediction methods to identify links that are surprising given their surrounding network, or which may not belong at all [15] . Put simply, any environment that naturally maps to a network probably has an equally coherent mapping from link prediction in that network back to an important question in the environment.. This broad applicability demands a powerful yet general framework, and we promote supervised learning. Unsupervised methods, which receive the most attention in link prediction literature, are fundamentally unable to cope with dynamics, interdependencies, and other properties in networks. We recognize that this is not the first paper to apply supervised learning to the link prediction problem, but there are important differences versus past work. First, in spite of the excellent intentions of past researchers, they have fallen prey to unique pitfalls endemic to problems with highly imbalanced class distributions. In [2] , the holdout test set is undersampled to balance, and the authors of [17] also contribute only a sample of the negative instances to their test set. As researchers familiar with high skew are aware, modifying the data distribution on which testing is performed generates uninterpretable results. The distribution of the resulting testing data no longer presents the same challenges as the real-world distribution, and performance measures in testing no longer reflect the real capabilities and limitations of the model. Additionally, both of these works employ semantic and contextual information that pertains almost exclusively to the bibliographic domain. Finally, these works do not consider the important impact of geodesic distance and the intricacies of class imbalance specific to the task of link prediction.. We demonstrate that decomposition by geodesic distance has important impacts on predictor performance irrespective of the choice of predictor. We also expand the library of unsupervised measures with an intuitive flow-based metric that is over 15% AUC more predictive than baseline methods in certain networks. After illustrating the benefits of supervised learning, we cast link prediction as a problem in class imbalance. The result of these considerations is a framework that improves upon the best baseline unsupervised methods by over 30% AUC in our test networks. Furthermore, the framework is entirely general, operating over any class of network whether it be weighted, unweighted, directed, or undirected. It does not require any node attributes but is capable of accepting them.. In Section 2 we describe the data sources and evaluation measures. Section 3 explains standard unsupervised approaches and defines a new metric, PropFlow. Section 4 lays out the rationale for supervised learning. This leads to a discussion of class imbalance in Section 5. In Section 6 we describe our realization of the framework and Section 7 presents results. Finally, Section 8 provides recommendations and concludes. The general framework we propose in this paper achieves major improvements over existing methods. Although it outperforms such methods by > 30% in terms of AUC, it does not require any domain-specific node attributes to do so. It can be applied in any domain exactly as described or it can accept any number of domain-specific features. It is also highly scalable; feature computation for a single pathbased method requires more time than the entire classification framework. The feature computation itself is embarrassingly parallel.. In addition to the results, the supporting study allows for some recommendations. Unsurprisingly, for networks where topological convergence and saturation may be a concern, the training observation period, τ x, should be as long as possible. The parameter τ y for the static network from which labels are gathered should match the size of the real-world prediction window so that testing and real-world prior distributions are as similar as possible. In link prediction on networks such as the Internet or electricity grids, these concerns are moot since snapshots contain the entire network structure.. Optimal class ratios for undersampling are specific to the problem at hand, but the results we obtained for both networks indicate that undersampling to balance may not be ideal in the link prediction domain. Those employing this classification framework should be aware of this fact and should investigate other ratios as resources permit. For small networks where computational resources are not problematic, we advise the use of skew-insensitive classifiers such as Hellinger trees in an ensemble framework. We encourage the community to consider the link prediction task as a separate problem for each desired neighborhood in domains where local mechanisms are likely to pertain. This not only decreases computational time by considering those links most likely to rank highly regardless but has the potential to sensitize supervised classification to the specific mechanisms and boundaries present for predictions within the target graph distances.. In general, the application of unsupervised methods, at least without due study and consideration, is highly suboptimal. No such method, no matter how high its performance in some subset of our data, provides acceptable performance for the entire range of problems. Where there is some limitation on supervised learning due to the unavailability of labels for training, we hope that the included study of unsupervised performance measures proves helpful in the selection of an appropriate option. For networks where one expects local mechanisms to dominate, especially local mechanisms related to flow or propagation, we highly encourage the use of the unsupervised method, PropFlow, proposed in this paper.. We have published all scripts and source code for prediction and evaluation at http://www.nd.edu/~rlichten/ linkpred along with the condmat data set. We regret that we are unable to make the phone data set available due to non-disclosure agreements.

O O O O O B I O O O O O O B I O B O O O O O O O O O O B I O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O B I O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B I O O O O B I O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


An evolutionary approach to collective communication scheduling In this paper, we describe two evolutionary algorithms aimed at scheduling collective communications on interconnection networks of parallel computers. To avoid contention for links and associated delays, collective communications proceed in synchronized steps. Minimum number of steps is sought for the given network topology, wormhole (pipelined) switching, minimum routing and given sets of sender and/or receiver nodes. Used algorithms are able not only re-invent optimum schedules for known symmetric topologies like hyper-cubes, but they can find schedules even for any asymmetric or irregular topologies in case of general many-to-many collective communications. In most cases does the number of steps reach the theoretical lower bound for the given type of collective communication; if it does not, non-minimum routing can provide further improvement. Optimum schedules may serve for writing high-performance communication routines for application-specific networks on chip or for development of communication libraries in case of generalpurpose interconnection networks. The importance of communication among CPU cores, processors and computers and of related interconnection networks is recently steadily growing. More often than not, processing nodes access the network according to a global, structured communication pattern. The performance of these collective communications (CC for short) has a dramatic impact on the overall processing efficiency, because communication times, software-or hardwarerelated alike, add up to an overhead of parallel processing.. 2. ⎢T ∩ R ⎢≥ 1. Many-to-many communication with overlapping sets of processes.. 3. ⎢T ∩ R ⎢= P. All-to-all communications such as permutation, all-to-all scatter (AAS), broadcast (AAB), reduce (AAR), and others.. In one to all, one process in group T is the sender (transmitter) and all the other processes in group R are receivers. This category of group communication has two distinct services, broadcast (multicast) and scatter. Broadcast (multicast) is when the same message is delivered to all the other processes in the group and scatter delivers a different message to each receiver in the group. For all to one group communication, all the processes in group T are identified as senders and the only one process in group R is identified as the receiver. A service offered in this category is gather, which occurs when different messages from the senders are received by the receiving process. This is the opposing process of scatter. And finally, all to all communication is when each process in group T performs its own one to all message passing. A service provided in this category is all broadcast, which allows for each sender to deliver the same message to all the receivers in the group. This is also known as gossiping. The other service of this group is all scatter, which means a different message is sent to each receiving process from the sending process. All scatter is also referred to as complete exchange.. Some researchers have taken a topology independent approach to the design of deadlock free wormhole routed CC algorithms. E.g. the postal model (similar to sending a batch of letters through the postal service at one time) demonstrates the ability for a sending node to transmit multiple messages before the receiving node receives the first message. In this paper, we will rather take a topology-aware approach and will assume that CC in WH networks proceeds in synchronized steps. In one step of CC, a set of simultaneous message transfers takes place along complete disjoint paths between source-destination node pairs. If the source and destination nodes are not adjacent, the messages go via some intermediate nodes, but processors in these nodes are not aware of it; the messages are routed automatically via routers attached to processors. We will assume that all messages in CC have identical size and are not combined or partitioned by network nodes (the non-combining model).. The paper is structured as follows. In Section 2 we present formally the scheduling problem and recent solutions of its certain instances. Our novel approach based on evolutionary algorithms is explained in Section 3. Solutions to sample communication problems on selected networks are given in Section 4, where the quality of resulting schedules is also discussed. In Conclusion we give the range and scope of scheduling problems solvable by the presented approach and possible future extensions. The evolutionary algorithms such as MBOA and HGSA have been applied successfully to several network topologies with slim or fat nodes and quite general collective communications with overlapping sets of senders and receivers. Scheduling CC in the minimum number of steps without creating a conflict (a common channel in two transfers in the same step) led to optimal solutions (τ CC (G) = τ CC (G)) or nearly optimal solutions. Of course, the fact that the lower bound may not be always reached by presented algorithms is to be expected because it may not be attainable in principle by any algorithm. Sometimes lower bounds can be obtained in schedules with non-minimum routing. However, only minimum routing has been considered in this paper because inclusion of the non-minimum routing would lead to an enormous increase of possible paths from sources to destinations and to the prohibitive computer memory and time requirements.. The results were derived for general case of M-to-N collective communications on WH interconnection networks. The application-oriented CCs of this kind are increasingly important in multiprocessor SoCs (System on Chips). One example is when one group of processors finishes a task and a group of different size continues and needs the intermediate results from the first group. The really obtained upper bounds τ CC (G) were presented for the 2D-mesh, coated mesh and (fat) Octagon topology of small size for illustration only. The presented algorithms are at current form applicable to networks with up to around 64 nodes; however, if we put up with suboptimal schedules (giving the number of steps reasonably close to the lower bound), the network size can be substantially larger. Scalability limits of the both presented algorithms and their possible improvements could be a subject of future research.

O O O O O B I O O O O O O O O O O O B I O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Quantifying and Verifying Reachability for Access Controlled Networks Abstract-Quantifying and querying network reachability is important for security monitoring and auditing as well as many aspects of network management such as troubleshooting, maintenance, and design. Although attempts to model network reachability have been made, feasible solutions to computing network reachability have remained unknown. In this paper, we propose a suite of algorithms for quantifying reachability based on network configurations [mainly Access Control Lists (ACLs)] as well as solutions for querying network reachability. We present a network reachability model that considers connectionless and connection-oriented transport protocols, stateless and stateful routers/ firewalls, static and dynamic NAT, PAT, IP tunneling, etc. We implemented the algorithms in our network reachability tool called Quarnet and conducted experiments on a university network. Experimental results show that the offline computation of reachability matrices takes a few hours, and the online processing of a reachability query takes 0.075 s on average. We make four major contributions in this paper. First, we model and formulate network reachability considering the differences in connectionless and connection-oriented transport protocols, stateless and stateful middleboxes, as well as the presence and absence of various packet transformers. Second, we present algorithms for computing network reachability matrices. Third, we give solutions for expressing and processing reachability queries. Fourth, we implemented our algorithms and conducted experiments on a campus network. Results show that our offline reachability computation is practical and online query processing is very efficient. Active probing tools, which actively test network reachability by sending probing packets and analyzing the response packets, are commonly used by network administrators. Such tools include and , which use ICMP echo request/ reply or ICMP time-exceed packets to test whether a host on target network is reachable. The use of such tools is limited because they cannot verify the reachability of UDP or TCP packets. There are tools such as NMAP and NESSUS that can test the reachability of UDP or TCP packets. However, such tools have significant limitations. First, they cannot perform comprehensive testing due to the amount of packets that have to be generated. Second, the test results of such tools are valid only for the routing state at the time that the testing is performed and may not hold afterwards due to the change of routing states over time. Third, such tools only show the open ports on which a server daemon is listening and does not reveal the open ports with no server listening on them at the time of testing. In comparison, our Quarnet is nonintrusive and comprehensive.. As little work has been done on static network reachability analysis, no concrete algorithm for computing network reachability has ever been proposed previously. Xie et al. presented a model of network reachability for instantaneous reachability and reachability bounds. However, we significantly go beyond Xie et al.'s work along three dimensions: 1) reachability modeling and formulation; 2) algorithms for computing reachability; 3) solutions for reachability queries. Our model differs from Xie et al.'s work in the following aspects. First, we model a network as a graph over both routers and subnets, while Xie et al. model a network as a graph over only routers. Thus, Xie et al.'s model is limited to describing the networks where each subnet connects to only one router, while our model does not have this limitation. Furthermore, we calculate reachability between two subnets, and they calculate reachability between two middleboxes. Second, we distinguish network reachability formulations based on both the properties of transport-layer protocols (namely connectionless and connection-oriented protocols) and the statefulness of routers/firewalls on every path, while Xie et al. did not. Third, our model addresses IP tunneling and three types of packet transformations: static NAT, dynamic NAT, and PAT, while Xie et al.'s model addresses only static NAT. Xie et al. gave no implementable algorithms for computing reachability and no solutions for reachability queries. Recently, in a poster paper [16] , Zhang et al. proposed monitoring and verifying reachability in real time by computing instantaneous reachability. However, they provided no algorithm for computing the reachability from a source to a destination along a given path and no experimental results. Some effort has been made to use binary decision diagrams (BDDs) for reachability analysis [17] , [18] . However, such work can only answer host-to-host queries and cannot answer subnet-to-subnet queries as we do in this paper. We choose FDDs instead of BDDs for several reasons. First, it is extremely difficult to swap the values of packet fields or rearrange packet fields in a BDD calculated over multiple ACLs. Note that our methods for performing these operations on FDDs, which requires generating rules from FDDs and reconstructing FDDs, cannot be applied to BDDs because generating rules from a BDD could easily lead to millions of rules as reported in [8] . Second, a reachability query engine built with BDDs can only process closed queries that demand a yes/no answer. In contrast, our reachability query engine built with FDDs can process both closed queries and open queries. Third, the reachability calculated by BDDs is not human readable. In contrast, every element in our reachability matrix is an FDD, which can be visualized for examination.. In [19] , Mayer et al. proposed Fang, a firewall analysis engine. Fang supports limited queries (over 3-tuples), and each query is compared to every rule in every ACL along every path from a source to a destination, which is very inefficient. There is some work, which is orthogonal to ours, on detecting reachability problems caused by routing faults (e.g., the faults identified in [20] ), instead of ACL misconfiguration (e.g., [21] ). In [22] , Ingols et al. proposed algorithms for creating attack graphs for a network. However, their focus is not on reachability computation. Our proposed work is complementary to [22] .

O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O B I O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B I O O O B I O O O O O O O O O O O O O O O O O O O O O O O O B O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B I O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O B I O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O


An investigation of decompounding for cross-language patent search Decompounding has been found to improve information retrieval (IR) effectiveness in general domains for languages such as German or Dutch. We investigate if cross-language patent retrieval can profit from decompounding. This poses two challenges: i) There may be few resources such as parallel corpora available for training an machine translation system for a compounding language. ii) Patents have a specific writing style and vocabulary ("patentese"), which may affect the performance of decompounding and translation methods. Experiments on data from the CLEF-IP 2010 task show that decompounding patents for translation can overcome out-of-vocabulary problems (OOV) and that decompounding improves IR performance significantly for small training corpora. Compounding languages such as German or Dutch allow combining simple words into complex words by concatenating them. In contrast to English, these compounds are written as single words. Splitting compounds into their constituent parts (i.e. decompounding) has been found to improve IR effectiveness, because it can overcome vocabulary mismatches [1, 2, 6 ].. An area of increasing interest in IR is prior art patent search, which is concerned with finding all relevant patents for a patent application. Since patents are often written in different languages, cross-language information retrieval (CLIR) is usually an essential component of effective patent search. For patent search in compounding languages, the CLIR effectiveness is usually lower than for other language pairs [3, 7] . This can be attributed to the presence of compounds, which leads to higher rates of OOV compound Copyright is held by the author/owner(s). SIGIR'11, July 24-28, 2011, Beijing, China. ACM 978-1-4503-0757-4/11/07. terms. OOV terms cannot be translated, which results in missing some portion of the query text and degrades IR effectiveness.. In this paper we apply decompounding on German patent topics in the cross-language patent search task from the CLEF-IP 2010 track and investigate machine translation (MT) quality by examining retrieval performance. The results show that decompounding improves retrieval performance significantly for small training corpora, or for corpora with high OOV rates. When using smaller MT training corpora or corpora with high OOV rates, decompounding shows higher performance compared to not decompounding. This effect is important for training MT systems for specific domains where only small parallel corpora are available for training MT systems, i.e. less training data is needed for good CLIR performance when decompounding is used to overcome OOV problems, and for training MT systems for languages with few linguistic resources.. The effect of incorrectly translating a constituent word of a compound is similar to a topic drift in blind relevance feedback, when adding query terms may result in a loss of precision. In patent search, where queries are formed by patent documents, incorrectly decompounded or incorrectly translated constituent words may result in a much higher query term frequency (and thus, a higher term weight in the query), causing a loss in retrieval effectiveness. This result is in line with observations made by Koehn and Knight [4] , who conclude that "eager splitting fares abysmally".. Future work includes investigating methods to identify technical terms and chemical formulas in patents to treat them differently for decompounding and/or translation.

O O O B O O O O B O O O O O O O O O O O O O O O O O O O O O O O B I O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O


Multiparty Access Control for Online Social Networks: Model and Mechanisms Abstract-Online social networks (OSNs) have experienced tremendous growth in recent years and become a de facto portal for hundreds of millions of Internet users. These OSNs offer attractive means for digital social interactions and information sharing, but also raise a number of security and privacy issues. While OSNs allow users to restrict access to shared data, they currently do not provide any mechanism to enforce privacy concerns over data associated with multiple users. To this end, we propose an approach to enable the protection of shared data associated with multiple users in OSNs. We formulate an access control model to capture the essence of multiparty authorization requirements, along with a multiparty policy specification scheme and a policy enforcement mechanism. Besides, we present a logical representation of our access control model that allows us to leverage the features of existing logic solvers to perform various analysis tasks on our model. We also discuss a proof-of-concept prototype of our approach as part of an application in Facebook and provide usability study and system evaluation of our method. O NLINE social networks (OSNs) such as Facebook, Googleþ, and Twitter are inherently designed to enable people to share personal and public information and make social connections with friends, coworkers, colleagues, family, and even with strangers. In recent years, we have seen unprecedented growth in the application of OSNs. For example, Facebook, one of representative social network sites, claims that it has more than 800 million active users and over 30 billion pieces of content (web links, news stories, blog posts, notes, photo albums, and so on.) shared each month [3] . To protect user data, access control has become a central feature of OSNs [2] , [4] .. A typical OSN provides each user with a virtual space containing profile information, a list of the user's friends, and webpages, such as wall in Facebook, where users and friends can post content and leave messages. A user profile usually includes information with respect to the user's birthday, gender, interests, education, and work history, and contact information. In addition, users can not only upload a content into their own or others' spaces but also tag other users who appear in the content. Each tag is an explicit reference that links to a user's space. For the protection of user data, current OSNs indirectly require users to be system and policy administrators for regulating their data, where users can restrict data sharing to a specific set of trusted users. OSNs often use user relationship and group membership to distinguish between trusted and untrusted users. For example, in Facebook, users can allow friends, friends of friends (FOF), groups, or public to access their data, depending on their personal authorization and privacy requirements.. Although OSNs currently provide simple access control mechanisms allowing users to govern access to information contained in their own spaces, users, unfortunately, have no control over data residing outside their spaces. For instance, if a user posts a comment in a friend's space, she/he cannot specify which users can view the comment. In another case, when a user uploads a photo and tags friends who appear in the photo, the tagged friends cannot restrict who can see this photo, even though the tagged friends may have different privacy concerns about the photo. To address such a critical issue, preliminary protection mechanisms have been offered by existing OSNs. For example, Facebook allows tagged users to remove the tags linked to their profiles or report violations asking Facebook managers to remove the contents that they do not want to share with the public. However, these simple protection mechanisms suffer from several limitations. On one hand, removing a tag from a photo can only prevent other members from seeing a user's profile by means of the association link, but the user's image is still contained in the photo. Since original access control policies cannot be changed, the user's image continues to be revealed to all authorized users. On the other hand, reporting to OSNs only allows us to either keep or delete the content. Such a binary decision from OSN managers is either too loose or too restrictive, relying on the OSN's administration and requiring several people to report their request on the same content. Hence, it is essential to develop an effective and flexible access control mechanism for OSNs, accommodating the special authorization requirements coming from multiple associated users for managing the shared data collaboratively.. In this paper, we pursue a systematic solution to facilitate collaborative management of shared data in OSNs. We begin by examining how the lack of multiparty access control (MPAC) for data sharing in OSNs can undermine the protection of user data. Some typical data sharing patterns with respect to multiparty authorization in OSNs are also identified. Based on these sharing patterns, an MPAC model is formulated to capture the core features of multiparty authorization requirements that have not been accommodated so far by existing access control systems and models for OSNs (e.g., [11] , [12] , [17] , [18] , [28] ). Our model also contains a multiparty policy specification scheme. Meanwhile, since conflicts are inevitable in multiparty authorization enforcement, a voting mechanism is further provided to deal with authorization and privacy conflicts in our model.. Another compelling feature of our solution is the support of analysis on the MPAC model and systems. The correctness of implementation of an access control model is based on the premise that the access control model is valid. Moreover, while the use of an MPAC mechanism can greatly enhance the flexibility for regulating data sharing in OSNs, it may potentially reduce the certainty of system authorization consequences due to the reason that authorization and privacy conflicts need to be resolved elegantly. Assessing the implications of access control mechanisms traditionally relies on the security analysis technique, which has been applied in several domains (e.g., operating systems [20] , trust management [30] , and role-based access control [6] , [21] ). In our approach, we additionally introduce a method to represent and reason about our model in a logic program. In addition, we provide a prototype implementation of our authorization mechanism in the context of Facebook. Our experimental results demonstrate the feasibility and usability of our approach.. The rest of the paper is organized as follows: In Section 2, we present multiparty authorization requirements and access control patterns for OSNs. We articulate our proposed MPAC model, including multiparty authorization specification and multiparty policy evaluation in Section 3. Section 4 addresses the logical representation and analysis of MPAC. The details about prototype implementation and experimental results are described in Section 5. Section 6 discusses how to tackle collusion attacks followed by the related work in Section 7. Section 8 concludes this paper and discusses our future directions. In this paper, we have proposed a novel solution for collaborative management of shared data in OSNs. An MPAC model was formulated, along with a multiparty policy specification scheme and corresponding policy evaluation mechanism. In addition, we have introduced an approach for representing and reasoning about our proposed model. A proof-of-concept implementation of our solution called MController has been discussed as well, followed by the usability study and system evaluation of our method.. As part of future work, we are planning to investigate more comprehensive privacy conflict resolution approach [23] , [26] and analysis services for collaborative management of shared data in OSNs. Also, we would explore more criteria to evaluate the features of our proposed MPAC model. For example, one of our recent work has evaluated the effectiveness of the MPAC conflict resolution approach based on the tradeoff of privacy risk and sharing loss [25] . In addition, users may be involved in the control of a larger number of shared photos and the configurations of the privacy preferences may become time-consuming and tedious tasks. Therefore, we would study inference-based techniques [37] for automatically configure privacy preferences in MPAC. Besides, we plan to systematically integrate the notion of trust and reputation into our MPAC model and investigate a comprehensive solution to cope with collusion attacks for providing a robust MPAC service in OSNs.

B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


The iDUDE Framework for Grayscale Image Denoising Abstract-We present an extension of the discrete universal denoiser DUDE, specialized for the denoising of grayscale images. The original DUDE is a low-complexity algorithm aimed at recovering discrete sequences corrupted by discrete memoryless noise of known statistical characteristics. It is universal, in the sense of asymptotically achieving, without access to any information on the statistics of the clean sequence, the same performance as the best denoiser that does have access to such information. The DUDE, however, is not effective on grayscale images of practical size. The difficulty lies in the fact that one of the DUDE's key components is the determination of conditional empirical probability distributions of image samples, given the sample values in their neighborhood. When the alphabet is relatively large (as is the case with grayscale images), even for a small-sized neighborhood, the required distributions would be estimated from a large collection of sparse statistics, resulting in poor estimates that would not enable effective denoising. The present work enhances the basic DUDE scheme by incorporating statistical modeling tools that have proven successful in addressing similar issues in lossless image compression. Instantiations of the enhanced framework, which is referred to as iDUDE, are described for examples of additive and nonadditive noise. The resulting denoisers significantly surpass the state of the art in the case of salt and pepper (S&P) and M-ary symmetric noise, and perform well for Gaussian noise.Index Terms-Context-based denoising, discrete universal denoiser (DUDE) algorithm, discrete universal denoising, Gaussian noise, image denoising, impulse noise. We have presented a framework for grayscale image denoising based upon the discrete universal denoiser DUDE of [2] . The framework overcomes the practical limitations, stemming from the model cost issues associated with large alphabets and limited sizes of image data, by exploiting prior knowledge on the structure of images, as previously done in lossless image compression, and confirms an important principle in the practical use of universal schemes: Algorithms should be as universal as necessary for the application at hand but not more-they should not be expected to learn what is already known in advance. In that sense, the full universality of the basic DUDE in the class of stationary sources is excessive for grayscale images. Instantiations of the enhanced iDUDE framework were shown to be effective on a variety of image and noise types, achieving state of the art denoising performance for impulse channels (S&P and -ary symmetric), and performance competitive with modern denoising schemes for the Gaussian channel. Further improvements in performance for the latter is a subject of ongoing research. The examples presented in the paper suggest that the following general steps are required for instantiating iDUDE for a different noise channel , characterized by a channel transition matrix . 1) Determine whether the image assumptions A1-A4 of Section III are effective for images affected by . If not, choose an appropriate prefilter for , for use in a prefiltering and iteration loop as described in Fig. 5 . If the assumptions are still effective for , prefiltering and iteration might not be essential (as in the case of Gaussian noise).. 2) Determine if a numerically stable inverse can be obtained. If so, the inverse can be used to define an estimation matrix as in (12) . Otherwise, an alternative channel inversion method, such as those discussed in Section III-F might be appropriate. 3) Design a context model and aggregation strategy appropriate for images affected by , and possibly prefiltered. For example, if gradient information remains reliable under and prefiltering, a context model similar to WGT (cf. Section IV-A) might be appropriate. Otherwise, as in the low-SNR Gaussian channel case, an LBG-based context model might work better. In addition, some parameters of the context model can be made to depend upon the strength, and not just the type, of the noise (e.g., the parameters , in WGT). Of course, given a specific channel and matrix , further optimizations are likely to emerge as a result of experimentation and analysis. In this section, we present results obtained with the iDUDE on images corrupted by simulated S&P, -ary, and Gaussian noise. For each type of noise, we compare our results with those of a sample of recent denoising algorithms from the literature for which an objective basis for comparison was available, and including in all cases the schemes with the best available published results as of the writing of this paper. Our iDUDE experiments are based upon a research prototype implementation written in C++, and run on a vintage 2007 Intel-based personal computer. 10 For a very rough complexity reference, we measured the running time of one iDUDE iteration in this implementation (using the WGT context model) on the 2048 2560 image Bike at approximately 7 s, for a throughput of approximately 730 Kpixels/s. Running times for a given context model do not vary significantly with the noise type or level. As will be shown in Fig. 8 and its discussion, the number of iterations necessary to approach the best performance is generally small.. The images used in the experiments are listed in Table I . The "very small" heading in the table refers to a set of 24 images of dimensions 384 256 (referred to as ) available at [36], for which results of denoising with the state of the art scheme of [28] at various levels of S&P noise are available. The "small" (512 512) images in the table are from the set traditionally used in the image processing literature. 11 Since the images in either set are rather small by today's standards, we include also larger images from the benchmark set used in the development of the JPEG-LS standard [5] .. We evaluate denoising performance by measuring peak signal-to-noise ratio (PSNR) between the denoised image and the original clean image. Table II summarizes the iteration and model size parameters used for the various experiments and noise types. The parameters, and the general iDUDE configuration for each noise type, were defined in Section IV. We use one set of iteration and model size parameters for each combination of image size category, noise type, and noise level, rather than parameters optimized for each individual image. The fixed predictor parameters and (cf. Appendix B) were 10 Specifically, Intel(R) Xeon(R) 5160 CPU, 3 GHz clock speed, 3 GB RAM, running Linux. 11 [28] . COMPARISON WITH CHN05 DISPLAYED SEPARATELY set as follows: of maximum gradient magnitude in the context, for the S&P channel, for the -ary symmetric channel; for the Gaussian channel, and were optimized to minimize the observable prediction RMSE for each noisy image, with varying between 5% and 17%, and between 0 and 0.05. This is one case where it is "legitimate" to optimize the parameter for each image, since the optimization is based upon observable data.

O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I I I B I I B I B I B I O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Implications of ceiling effects in defect predictors Context: There are many methods that input static code features and output a predictor for faulty code modules. These data mining methods have hit a "performance ceiling"; i.e., some inherent upper bound on the amount of information offered by, say, static code features when identifying modules which contain faults.Objective: We seek an explanation for this ceiling effect. Perhaps static code features have "limited information content"; i.e. their information can be quickly and completely discovered by even simple learners.Method: An initial literature review documents the ceiling effect in other work. Next, using three sub-sampling techniques (under-, over-, and micro-sampling), we look for the lower useful bound on the number of training instances.Results: Using micro-sampling, we find that as few as 50 instances yield as much information as larger training sets.Conclusions: We have found much evidence for the limited information hypothesis. Further progress in learning defect predictors may not come from better algorithms. Rather, we need to be improving the information content of the training data, perhaps with case-based reasoning methods. The current state of the art in learning defect predictors is curiously static. As shown below, better results have not been forthcoming, despite the application of supposedly better data miners.. If better algorithms are not useful, perhaps it is time to better understand the training data. Accordingly, instead of:. • Changing the data miners. . .. • .... this paper changes the training data.. Specifically, we study how throwing training data away effects the performance of our defect predictors. The results are quite surprising. In experiments with under/over-sampling and incremental cross-validation, we found that much of the training data can be discarded without losing effectiveness in defect prediction. This leads to the following notion:. Hypothesis: Static code features have limited information content. The development of fault prediction models has been a very active research area. The reason for such a significant attention to automated quality predictors lays in their practical importance. Cur-rent models are useful, as they allow software project managers to better guide the allocation usually meager quality assurance resources to artifacts which need them the most.. Recent results now indicate that this current research paradigm, which relied on relatively straightforward application of machine learning tools, has reached its limits. Building software quality predictors via data mining is essentially an inductive generalization over past experience. According to Mitchell's classic model of data mining [38] , any inductive generalization explores a version space of possible theories. All data miners hit a performance ceiling effect when they cannot find additional information that better relates software measure with fault occurrence.. To build better quality predictors that break through ceiling effects, we must introduce more topology into the search space. Standard machine learning algorithms lack the business knowledge which characterizes software projects. To add that business knowledge, we propose human-in-the-loop CBR tools. We expect that this approach will allow software managers to safely focus on the application of quality assurance techniques of choice, confident that automated quality predictors will raise alerts about the artifacts where quality issues actually exist.. Three results make us advocate this kind of CBR tool:. • The environment would impractical if the human operator must examine a very large number of cases. The microsampling results suggest that this might not be the case. Indeed, as few as 50 randomly selected instances might suffice.. • The environment would run fast. If 50 instances are enough, then training should be virtually instantaneous. A human user could explore a very large number of features or combinations of features, all the while getting very rapid feedback.. • The environment, or some other new direction, is required.. As argued above, our current generation of AI algorithms have hit a ceiling effect. We doubt that further progress will be made using better algorithms. Instead, we would advocate methods (like the CBR tool briefly described above), that increase the information content of the training data. In summary, we think it is time to change the subject of the which question. Rather than which learner we should focus on which data.. In this context, our future direction is clear-build a human-inthe-loop CBR environment for learning defect predictors, then benchmark that environment against automatic methods.. Other candidates for future work are. • To better understand the attribute weighting schemes explored in Figure 8 .. • To improve on the randomized sampling strategies used in this paper. It may be possible to increase the performance of defect predictors with wiser sampling strategies (e.g. some initially clustering to find a small number of most representative, or most unusual, instances). There exist numerous incremental case-based reasoning tools [7, 23, 24] that ask humans to audit a stochastic sample of real-world cases. Insights gained from those sessions are automatically generalized and applied to another random sample, Experts then review the classifications made on the new sample, and offer further refinements. As far back as 1999, software metrics experts Fenton and Neil [12] postulated that such human-machines-based system might out-perform systems based on on static code measures (since other features/metrics could be accounted for that cannot currently be addressed using static code metrics).. When is case-based reasoning preferable to automatic data mining? While there are many answers to this question, this paper can make specific comments about one particular issue. Case-based reasoning methods require humans to examine and comment on specific cases. This is impractical if learning adequate theories requires examining a very large number of cases.. The results of experiment #1 suggest that, for static code measures, it is not necessary to manually inspect thousands of cases. In fact, just a few hundred may suffice. Consider the 9,466 modules of MC1. This data set has a defective module rate of 0.71% ; i.e. 9.466*0.71/100=67 modules:. • When under-sampling, a data set of 134 modules is created (all the defective, plus 67 others, selected at random).. • When micro-sampling, the results of Experiment #2 suggest that 50 modules would suffice (any 25 defective modules, but any other 25-selected at random).. These results raise the possibility that a human-in-the-loop casebased reasoning environment might perform as well as automatic methods, despite the automatic methods exploring more examples.. Based on other recent research, we go further and hypothesize that such an environment might perform better than automatic methods. Elsewhere, we have explored combining static code measures with other measures that, serendipitously, a particular domain may contain. For example, at ISSRE 2007 [20] , we reported experiments where static code measures were combined with results from an ultra-lightweight text miner. Figure 7 shows how a remarkable improvement in learner performance was achieved by applying combinations of requirements and code features. Other results are analogous to Figure 7: • In the JMLR special issue on feature selection, Guyon and Elisseeff provide simple examples showing that the information content of data can be significantly increased when features are used together rather than individually [17] . In the context of this paper, we would re-express that result as: using features from different sources, e.g. requirements and source code, can significantly increase the information content of SE data. Figure 7 : {pd, pf } curves seen when using code and/or requirements features. Results from CM1 and PC1 after a 10-way cross validation. The ideal spot on these ROC curves is top left; i.e. no false alarms and perfect detection ({pd, pf } = {1, 0}). The dashed lines on those plots show {pd, pf } results when fault prediction models used features mined from requirements text, or features mined from static code measures (in isolation). The solid lines show the results of models which used these two kinds of features in combination. From [20] .. • Face recognition differentiates from other image processing research by using facial properties such as the location of the nose, lips, eyes etc. Combination of facial characteristics from such different sources improves the performance of face detectors that solely use i.e. pixel values of the image [16] . We argue that SE domain should make use of domain specific knowledge (when possible) to differentiate from general data mining tasks.. Note that the lesson of Figure 7 is not "always augment static code features with requirements features". This is impractical ad- of these results, the weighted schemes have higher medians or higher maximum values, or both.. vice since not all domains allow data mining specialists access to source code and the requirements that produced them; e.g. very few open source projects have detailed textual requirements documents.. Rather, the lesson of Figure 7 is that it can be very useful to let experts access and combine features from whatever sources are locally available. Such an "explore whatever" environment is not an automatic black box data miner. Rather, it is a human-in-theloop case-based reasoning (CBR) environment where humans reflect on the specifics of particular cases, connect to different data sources, and (sometimes) run automatic data miners on combinations or subsets of a variety of types of features.. Two prior PROMISE papers [1, 9] suggest how such CBR environments might operate:. • At PROMISE 2006, Amor et.al. [1] describe an incremental classification cycle where, in round i, an expert classifies a random selection of the available data. A classifier learned from this examples is then applied to other, randomly selected examples. Then, in round i + 1, the expert reviews the results of that those classifications to fix any incorrect classifications from round i (formally, this is boosting where humans are used to identify examples that were harder to classify in the proceeding rounds).. • At PROMISE 2007, Dekhtyar et.al. [9] showed improvement in the performance of a traceability environment after several rounds of an expert reviewing results automatically inferred in a prior round. Based on this paper, we would speculate that policies (a) and (b) would significantly decrease the time required by an expert to achieve performance plateaus. Another role for human experts in a CBR environment is to instruct the learner how combinations of attributes can work together to provide solutions. For example a standard Naive Bayes classifier gives equal weights to all attributes then uses frequency counts to learn the relative importance of each attribute. Elsewhere [42, 43] we have allowed an oracle to offer unequal attribute weights. In this scheme, instead of feature subset selection [22] , we have used all attributes with weights assigned to them. We have converted the information gain and gain ratio of features into weight values. The results are compelling such that the performance can improve over standard Naive Bayes, and there is also no need for dealing with feature subset selection. Though the improvements in Figure 8 are not ground-breaking, they provide a hint regarding the value of unequal treatment of information sources. In the case of Figure 8 , weights are assigned by the model. As Fenton and Neil warn us, the weights provided by models may not be meaningful to humans in the process [12] . However, we argue that weights assigned to different information sources by human experts with business knowledge can increase the quality of solutions. For some time now, we have applied data miners to build defect predictors from static code measures [4, 8, 20, [27] [28] [29] [30] [31] [33] [34] [35] [36] [37] . To learn such predictors, tables of historical examples (like those in Figure 1 ) are formed where one column has a boolean value for "faults detected" and the other columns describe software features such as (i) lines of code, (ii) number of unique symbols [18] , or (iii) max. number of possible execution pathways [26] . (# modules Each row in the table holds data from one "module"; i.e. the unit of functionality. Depending on the language, modules may be called "functions", "methods", "procedures" or "files".. The data mining task is to find combinations of features that predict for the value in the defects column. Once such combinations are found, managers can use them to determine where to best focus their QA effort. Better yet, if they have already focused their QA effort on the most critical portions of the system, the detectors can "nudge" them towards areas that need the most attention. Note that these data miners do not predict the total number of defects, just the number of modules containing more than zero defects.. In theory, such defect predictors are not useful. Fenton offers an insightful example where the same functionality is achieved using different programming language constructs resulting in different static measurements for that module [13] .. In practice, they are quite effective, at least for the NASA code we have studied [20, 28, 30, 31, [33] [34] [35] [36] [37] [41] [42] [43] . In those data sets, our fault prediction models find defect predictors [35] with a probability of detection (pd) and probability of false alarm (pf ) of mean(pd, pf ) = (71%, 25%). These values are much higher then known industrial averages for manual defect detection [11, 40] .. In January 2007, we published a study [35] that defined a repeatable experiment in learning defect predictors. The intent of that work was to offer a benchmark in defect prediction that other researchers could repeat/ improve/ refute. That experiment used:. • Public domain data sets (from the the PROMISE repository;. • Open source data mining tools (the WEKA toolkit [44] ); • Randomly sorting training data rows (stops order effects); • 10-way cross-validation (to test on data not used in training);. • Learning via multiple types of machine learning algorithms (rule learners, decision tree learners, Bayes classifiers); • Assessment via multiple criteria such as probability of detection (pd), probability of false alarm (pf ), and balance that combines {pd, pf } (balance is defined in Figure 2 ); • Statistical hypothesis tests over the assessment criteria;. • Novel visualization methods for the results;. • Feature subset selection to find the most important subset of the static code features; Surprisingly, very simple Bayes classifiers (with a simple preprocessor for the numerics) out-performed the other studied methods. Since that study, we have tried to find better data mining algorithms for defect prediction. To date, we have failed. Our recent (as yet, unpublished) experiments have found no additional statistically significant improvement from the application of the following If {A, B, C, D} are the true negatives, false negatives, false positives, and true positives (respectively) found by a defect predictor, then:. All these values range zero to one. Better and larger balances fall closer to the desired zone of no false alarms and 100% detection.. Other measures such as accuracy and precision were not used since, as shown in Figure 1 , the percent of defective examples in our tables was usually very small (median value around 8%). Accuracy and precision are poor indicators of performance for data were the target class is so rare (for more on this issue, see [33, 35] ). data mining methods: logistic regression; average one-dependence estimators [15] ; under-or over-sampling [10] random forests [2] , RIPPER [6] , J48 [39] , OneR [19] and bagging [3] . Only boosting [14] on discretized data offers a statistically better result than a Bayes classifier. However, we cannot recommend boosting: the median improvement is quite negligible and boosting is orders of magnitudes slower than a simple Bayes classifier.. Other researchers have also failed to improve our results. Due to our connection with PROMISE, we are aware of studies by other researchers (currently under review) that tried other data mining methods. Those studies investigated the statistical difference of the results from two dozen learners on the same datasets. The simple Bayesian method discussed above ties in first place along with 15 other methods.. How can we explain all these failed attempts to improve fault prediction models in repeatable experiments using same (PROMISE) data sets? One lesson from the above is that exploring better algorithms may not be productive. Hence, we explored the data that the algorithms were processing.. Elsewhere [32] we checked how little information was required to learn a defect predictor. Defect predictors were learned from N = 100, N = 200, N = 300.. instances then T ested on another 100 instances. For each N , 10 experiments were performed where training was conducted on |T rain| = 90% * N instances and testing on |T est| = 100 (and T rain ∩ T est =). For all experiments, the N, T rain, T est instances were selected at random.. This study was conducted on the twelve data sets of Figure 1 . Space does not permit showing all the results but a representative sample are shown in Figure 3 [32] . In that figure, the X-axis is the size of training set and the Y-axis is the balance measure defined in Figure 2 . Note that the performance does not change much regardless of whether the model is inferred from 100 instances or from up to several thousand instances. In fact, learning from too many training examples was actually detrimental (witness the widening variance as the training set increases). A Mann Whitney U test [25] (95% confidence) confirms the visual pattern apparent in Figure 3 : static code features used as the basis for predicting module's fault content reveal all that they can reveal after as little as 100 instances.. The above results motivated new experiments, reported below. The goal of these experiments was two-fold:. 1. Can we confirm the effects of Figure 3 ; i.e. that ignoring large amounts of the training data is not harmful? 2. Can we exploit that effect to build better defect predictors?. Goal #1 was achieved and we have a promising lead on Goal #2.

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O