Inefficiency in selecting products for submission to national research assessment exercises One of the critical issues in national research assessment exercises concerns the choice of whether to evaluate the entire scientific portfolio of the institutions or a subset composed of the best products. Under the second option, the capacities of the institutions to select the appropriate researchers and their best products (the UK case) or simply the best products of every researcher (the Italian case) becomes critical, both for purposes of correct assessment of the real quality of research in the institutions evaluated, and for the selective funding that follows. In this work, through case studies of three Italian universities, we analyze the efficiency of the product selection that is intended to maximize the universities' scores in the current national research assessment exercise, the results of which will be the basis for assigning an important share of public financing over the coming years. Higher education institutions (HEIs) are an important pillar of national research and innovation systems, thus the policy agendas of many countries now place high priority on strengthening such institutions. One expression of this policy is the increasing diffusion of national research assessment exercises. Beginning from the original British RAE, which has inspired many other countries, national assessment exercises are becoming regular events, particularly in nations wishing to introduce new principles of governance and more innovative management of the research sphere, such as through performance-based research funding systems (PBRFs) -which in turn depend on the results of the national assessments. The intended purpose of PBRFs is to influence improvement in underperforming institutions (Herbst, 2007, p. 90) and to stimulate continuous improvement of productivity in the whole research system (Abramo et al., 2009 ).. An international comparative analysis on the adoption of PBRFs, by Hicks (2012) , indicates that following the example of the RAE, at least 14 other countries (of which 11 in the EU, plus China, Australia, New Zealand) now base some portion of public financing for research institutions on national assessment exercises. The magnitude of the funds assigned on the basis of the evaluations varies from nation to nation, as does the methodology for assigning them. Yet the adoption of PBRFs does not seem immune from serious concerns and risks, as shown in a study by Geuna and Martin (2003) analyzing the experience of twelve countries in Europe and the Asia-Pacific region. Their analysis indicates that while the initial benefits of PBRFs might outweigh the costs, they produce diminishing returns over time. In any case, national evaluations continue, taking on particular importance where the results are used to support important policy decisions or apply incentives that influence the behavior of scientists (Council of Canadian Academies, 2012) . Indeed, a typical assumption for any national evaluation process is that it is intended to produce information and data to "enhance management practices and achieve goals, provide accountability for the stated goals, improve performance, allocate resources, and lead to informed policy decisions" (Jordan and Malone, 2001) . National assessments serve not only to allocate resources in an efficient manner and stimulate improvement in performance, but also to reduce information asymmetry between supply and demand for new knowledge, and further to demonstrate that investment in research is effective and delivers public benefits. Given that assessment exercises are intended to pursue such important objectives, it is clear that they must be executed with maximum methodological rigor, applying the best technologies available.. Planning of national research assessment exercises has always involved a choice between two distinct methodologies: peer review and bibliometry. Until recently, the most commonly adopted method was peer review, where products submitted by institutions are evaluated by panels of appointed experts. But new developments in bibliometric indicators have led many governments to introduce the use of these metrics, either in integration with peer review (known as "informed peer review") or in complete substitution. The use of bibliometric measures is still limited to the hard sciences, where publications in international journals and conference proceedings are the most accepted form for the codification of new knowledge, and where the publications therefore represent a trustworthy proxy of overall research outputs.. Both approaches offer their own strengths and weaknesses, which have been amply discussed in the literature Moxham and Anderson 1992; Horrobin 1990 ). Ultimately, the practical choice in adopting a methodology often occurs through a process of compromise between the owners of the evaluation process (ministries, governing agencies, funding councils) and those representing the interests of the affected academic community. Australia's current national research assessment and the historic series of UK RAE exercises represent the two extreme poles. Australia's ERA (Excellence in Research for Australia ) adopts what can best be termed a "metrics-based expert review" model: here, the expert panels do not examine individual publications, and instead rely on bibliometrics in formulating their judgments. On the other hand the British RAE, since its inception in 1986, has adopted a peer review-based process. Still, the plan for the next UK exercise, scheduled for 2014 and named the Research Excellence Framework (REF) , is to use informed peer review, where assessment outcomes are a product of expert review supported by bibliometric information.. One of the critical issues in any assessment is the choice of whether to evaluate the entire scientific portfolio of the institutions or a subset of the production. The REF, for example, will examine the work of selected research units and will not consider all their outputs -each of the selected units' members will submit a maximum of three or four of their highest quality works. The current Italian assessment, known as the VQR 2 , is also based on a subset of products: universities are required to present, for each of their professors, the best three research works from the 2004-2010 period. On the other hand, in the Australian ERA, the institutions must submit the full research production of their entire research staff. The approach of evaluating all products clearly has advantages (Abramo and D'Angelo, 2011) . However in systems involving peer review, for obvious reasons of costs and time it is clearly unthinkable to evaluate the entire output of a national research system. This fact results in undeniable penalties. First, it prevents providing any measure of productivity, the essential indicator of efficiency for any production system. Second, rankings are then sensitive to the size of whatever subset of output is evaluated, which seriously jeopardizes robustness of the underlying peerreview methodology. Third, it implies the necessity of selecting a subset of products (or researchers), which is not necessarily an efficient process, and which can well introduce elements of distortion that compromise the validity of the peer-review method. Fourth, it limits the functionality of the method, since it cannot be applied to all single researchers or research groups. As a consequence, universities do not receive performance rankings of their research staff, to in turn inform their internal selective funding. Finally, with respect to the bibliometric approach, peer-review implies very high costs and long times for execution, which limits its potential frequency of execution.. In this work we specifically wish to analyze the potential inefficiencies in the institutional process of selecting "best products" to submit for evaluation. The process appears to present significant risks that could nullify achievement of the very objectives of the evaluation. In fact, for institutional scores and rankings that descend from the submitted products to faithfully represent research quality, it is fundamental that the works submitted for evaluation truly be the institutions' best ones. If this were not the case, the evaluation exercises would produce rankings of quality that were distorted by an initial ineffective selection, with consequent inefficient allocation of resources. When the assessment exercises are based on a subset of the best products, as in the UK REF and the Italian VQR, it is thus fundamental that these truly represent the most valid part of the institutions' scientific portfolios, otherwise there is a risk of rewarding universities not for their true excellence, but for their ability or luck in selecting best products. In a preceding examination of the first Italian evaluation exercise (the 2001 (the -2003 3 , Abramo et al. (2009) reported that the institutions demonstrated limited efficiency in selecting their products. The inexperience of the institutions, due to the fact that this was the first ever national evaluation, could have contributed to their difficulties in choosing best products. They would also encounter the objective difficulty of comparing and then choosing research outputs from different fields. In addition, selections could have been influenced by social factors, including the varying negotiating power of persons or groups. In the current VQR evaluation exercise, the risk of ineffective selection seems lessened by two new aspects. First, a prevalently bibliometric approach has been adopted for the hard sciences, meaning that the criteria for evaluating the products could be publicized in advance, and the choice of submissions to evaluation could then be based on quantitative parameters and objective data. In addition, the indication of the products to be submitted was entrusted first to the individual researchers, with each one asked to submit a list of their top products in order of quality, from which the institution could then forward the final number of submissions requested. However in reality, non-specialist academics found that it was not particularly easy to apply bibliometric indicators for support in selecting their best products. In addition, the fact that researchers were called to indicate their best products then gave them the possibility of boycotting or acting against the interests of their home institutions, by selecting worthless products or not providing any list at all (every product shortfall results in penalties to the institutional scores). For the Italian university system, the current national evaluation exercise (2004 is an event of vital importance, since the results will support the process of allocating a "reward" share of core government funding.. One of the critical methodological issues in the VQR, and in national assessment exercises in general, concerns the choice of whether to evaluate the entire scientific portfolio of the institutions or a subset composed of the best products. In the VQR, for each member of their staff, the institutions were asked to submit a certain number of products for evaluation. The risks both for the funding agency and the individual institutions are then that the difficulty in carrying out efficient selections could lead to final scores and relative assignments of selective funding that do not represent the true quality of research conducted in the institutions. For this, in the current work we have examined three case studies of Italian universities. Through a simulation approach that closely corresponds to the VQR procedures, we evaluate the rate of error in the selection of products by the three universities, and the penalization that they experience in terms of overall score and by individual discipline.. We observed that in a small but notable number of cases the researchers recommended submission of products that did not meet the criteria for consideration under the VQR. In a much greater number of cases, the researchers tended to overevaluate some of the works in their lists of recommended scientific products, placing lesser products among those they rated as best for submission. In numerous further cases the researchers recommended lists of products that completely omitted other works, by the same individual, which were superior to the ones proposed. The details of design of national exercises can render the selection of products still more complex: in the case of the Italian VQR, for example, further difficulties (and error) arose from the rule that products coauthored by researchers from the same institution could not be submitted twice. After observing the errors in the actual recommendations of works by the researchers, we then simulated three scenarios corresponding to successively more sophisticated optimization of the university product selection procedures.. In terms of the impact of errors in selection for the final scores of quality, the results of the simulation demonstrate that the universities are particularly penalized in the selection of their products if they adhere exclusively to the proposals from the researchers, instead of choosing from among them on the basis of the criteria defined by the VQR panels. A further significant increase in score can be obtained by the universities if they choose from all the scientific production of the researchers, rather than only the lists of works that the researchers recommend to their home institution.. Given our analysis, we conclude that it is highly probable that the national ranking lists produced by the exercise will be strongly distorted by the types of errors noted above, with negative consequences for the subsequent distribution of financing, and further effects completely contrary to the logic of incentivizing for which the VQR was conceived.. Some universities will certainly be better equipped to deal with such requirements in calls for submissions -indeed, the three case studies indicate clearly different rates of error, depending on the process of selection implemented by the universities. The errors in selection on the part of the universities further undermine the already weak methodological structure at the base of the VQR, as illustrated in the introduction. The only hope is that the distortions introduced in the final rankings by all of these factors will somehow cancel out, perhaps by chance. Only in this way will it be possible to limit the damage caused by allocation of resources based on distorted rankings.. However the rebirth of academic systems such as Italy's depends precisely on implementing incisive and effective reward measures, such as might have been hoped for from the VQR. The authors thus question whether it would have been opportune, at least for the disciplines where bibliometrics are appropriate, to evaluate the entire scientific portfolio of the university. Such a strategy would involve less expenditure of funds and time, would avoid errors due to selection, and obtain real productivity measures on which to base not only financial incentives for the universities, but also internal incentivizing mechanisms, distributing "revenue to revenue generators". The dataset for each of the three universities examined consists of two distinct lists of products:  Products proposed by the researchers (set A);  Publications indexed in the WoS but not proposed by the researchers, attributed to them by analysis of ORP data (set B).. Beginning from these two data sets we develop a further three: -C = A B; the total set of works produced by the university's researchers; -D A; the subset of the three products proposed as "maximum priority" by each researcher 12 ; -E ⊂ C; the subset of the three best products identified from set C, for each researcher. These sets then permit quantification of proxies of the errors of selection in the recommendation of products by the researchers, as follows:. I. First, products "over-valued" by the researcher: D -(D ∩ E). This type of error is committed when the researcher recommends the product as high priority (1, 2 or 3) when in reality it places below third place in their overall scientific portfolio, when ranked under the criteria for the relevant GEV. II.. Products "undervalued" by the researcher: E ∩ (A -D). This second type of error is committed when the researcher proposed lower priority for the product (4, 5, etc.) but it actually places in the first three positions in the ranking for their overall portfolio, under the criteria set by the GEV. III.. Products "omitted" by the researcher: E -(A ∩ E). This third type of error occurs when a publication indexed in the WoS is not proposed by the researcher, while under the criteria for the GEV it would be in the top three places of the individual's 2004-2010 publications ranking. We then apply the criteria defined by the GEVs, proceeding to bibliometric analysis only for those products listed in the WoS and adopting certain assumptions concerning those directed to peer review or "informed peer review", as detailed below.  For products that were directed to informed peer review under the GEV procedures, we assign a score of 0.5. These are the products where the values of the two bibliometric indicators are inconsistent (high impact factor and low number of citations or vice versa). We hypothesize that the reviewers would adopt a judgement with "intermediate" modal value under the scale available.  For products indexed in the WoS but without impact factor (usually conference proceedings) we assign values of: i) nil, if intended for Biology and Medicine (where the GEVs explicitly notified that they would be given nil score); ii) 0.5 if intended for Information engineering (since in this field conference procedings are considered to have value); iii) 0.25 in all other cases.  For products not indexed in the WoS we assign a score of 0.25.. The assumption of 0.25 score for the final two points considers that in the hard sciences there are only a limited number of resarchers that produce less than three publications indexed in the WoS. The resort to submission of non-indexed publications indicates that the researcher is quite unproductive, and we also assume the quality of their works is relatively modest.. We further assume that potential products proposed by the researchers as 1, 2 or 3 priority, which are indexed in Scopus but not in WoS, would not be better than those contained in subset E. Finally, we assume that products not censused in either WoS or Scopus and not submitted by the researcher are not better than those contained in E.. All of the above assumptions should be recalled for proper interpretation of the results of the analysis, as reported in the following sections. Table 3 presents, for each of the three universities: i) the number of WoS-indexed publications proposed by the researchers: ii) the number of WoS publications that were not proposed; iii) the ratio between the "not-proposed" publications and the total WoSindexed publications for the university. Table 3 : Publications* indexed in the WoS that were proposed (P) and not proposed (NP) for VQR submission, for the three universtities examined (in parentheses the ratio of NP to P + NP)

O O O O O O O O B I I O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O B I O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O B I O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O


The design and evaluation of a high-performance soft keyboard The design and evaluation of a high performance soft keyboard for mobile systems are described. Using a model to predict the upper-bound text entry rate for soft keyboards, we designed a keyboard layout with a predicted upper-bound entry rate of 58.2 wpm. This is about 35% faster than the predicted rate for a QWERTY layout. We compared our design ("OPTI") with a QWERTY layout in a longitudinal evaluation using five participants and 20 45-minute sessions of text entry. Average entry rates for OPT1 increased from 17.0 wpm initially to 44.3 wpm at session 20. The average rates exceeded those for the QWERTY layout after the 10"' session (about 4 hours of practice). A regression equation (R' = .997) in the form of the power-law of learning predicts that our upper-bound prediction would be reach at about session 50. Besides handwriting recognition, one popular method of text input for pen-based mobile systems is through a soft keyboard. Users enter text by tapping on the image of a keyboard on the system's display. Although the QWERTY layout is entrenched for physical keyboards, soft keyboards are easy to implement and modify. So, a reasonable goal is to search for an alternate, perhaps better, layout than the venerable QWERTY. This paper describes the design and evaluation of one such layout, which we call OPT1 in reference to our goal of designing an optimal soft keyboard. We have described the design and evaluation ad a highperformance soft keyboard for mobile systems. Our results indicate that after about four hours of practice users' entry rates will be higher with the OPT1 layout than with a QWERTY layout. After about seven hours of practice users achieved a mean entry rate of 44.3 wpm. Our model predicts that entry rates will edge upward, reaching about 58 wpm for expert users.. ' When the character-space-character pattern is symmetrical there would be two or more space keys that create the same shortest path. In an extreme 'case with E-Space-E, all the four space keys are equally optimal. A usability study was undertaken to evaluate the OPTI soft keyboard. Since QWERTY is the most common layout, we included both layouts in the study. Our goal was not just to evaluate the OPTI layout, but to determine its performance relative a QWERTY soft keyboard. We fully expected the QWERTY layout to be better initially wanted to establish the learning trends for both layouts and to determine if and when a crossover point would occur, wherein OPTI would fare better than QWERTY.

O B O B O O B O O O B O B O O O O O O O B I O O O O B O B O O B I O O B I O O O O B O O O O O O O O O O O O O O O O O O O O O B O O O B O O O O B O O O B O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B I O O O O O O O O O O O O O O O O O O B O O O O O O O O B O B O O O B O O O O O O O O O O O O O O O O O O O B O B O O O O O O B I O O O O O O O O O O O O O O O O O O O B O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O


VARIABLE SELECTION IN LINEAR REGRESSION: SEVERAL APPROACHES BASED ON NORMALIZED MAXIMUM LIKELIHOOD In this talk, we discuss the application of the normalized maximum likelihood (NML) for model selection in Gaussian linear regression. All the results which will be presented have been recently published in [1] . The use of the NML for model selection in Gaussian linear regression poses troubles because the normalization coefficient is not finite. The most elegant solution has been proposed by Rissanen and consists in applying a particular constrain for the data space [2] . The resulting criterion is independent of arbitrarily selected hyper-parameters. Surprisingly, for about one decade, it was totally ignored the important fact that the closed-form expression of the criterion depends on the particular constraint which has been involved in its derivation. Only recently, it was shown in [3] that two other criteria can be obtained by employing constraints which are different of the one used in [2] . Hence, novel NML-based criteria can be devised by enforcing various constraints.

O O O O O O O O O B I I O O O O O O O O O B I I O O B I O B I I O O O O O O O O O O O O O O O O O O O O B I O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Leveraging Eclipse for integrated model-based engineering of web service compositions In this paper we detail the design and implementation of an Eclipse plug-in for an integrated, model-based approach, to the engineering of web service compositions. The plug-in allows a designer to specify a service's obligations for coordinated web service compositions in the form of Message Sequence Charts (MSCs) and then generate policies in the form of WS-CDL and services in the form of BPEL4WS.The approach uses finite state machine representations of web service compositions and service choreography rules, and assigns semantics to the distributed process interactions.The move towards implementing web service choreography requires design time verification of these service interactions to ensure that service implementations fulfill requirements for multiple interested partners before such compositions and choreographies are deployed. The plug-in provides a tool for integrated specification, formal modeling, animation and providing verification results from choreographed web service interactions. The LTSA-Eclipse (for Web Services) plug-in is publicly available, along with other plug-ins, at: http://www.doc.ic.ac.uk/ltsa. The practice of engineering web services currently focuses on the technical implementation of service functionality in a number of development language environments (such as Java and .NET). There has been a growing presence of web service tools, aimed mostly at the developer, who can generate web service interfaces and deployment configurations for hosting services on these different environments, yet the service interactions are equally important in the consideration of how the clients (including other services) needs will be incorporated in to the design of service functionality. Our work to date [1] [2] [3] focuses on providing an approach and tool which facilitates the design of appropriate service interaction specifications, verifying implementations of these specifications and generating representations in the standards for web service orchestrations and choreography. We achieve this through the provision of editors and views for analyzing the scenarios in service interactions and by verifying properties of the models produced through interaction specifications. This paper is organized as follows. In section 2 we describe the design goals by considering the aspects of web service compositions in relation to the web service standards. We then describe in section 3, a path of engineering service compositions in the tool from design through to maintenance. Section 4 details the approach to plug-in architecture for Eclipse, with Section 5 providing a brief example. Section 6 is an evaluation of the plug-in and Section 7 concludes the paper with opportunities highlighted as part of this on-going work. Firstly, we wish to continue describing compositional behaviour by elaborating on the wider choreography aspects of partnered services, and are also seeking to model the requirements of distributed resources for service compositions (such as impact of number of instances and type of requests between compositions). This also includes considering fault, compensation and transactional integrity within and between distributed processes. As part of this work we are aligned closely with consortiums, such as the W3C, on their work with choreography architectures and specifications. It is anticipated that the result of their work could be incorporated into our approach and the plug-in to provide an extension to the choreography elements we have considered thus far. We are also evaluating the use of core Eclipse graphical modeling plug-ins, such as the Eclipse Modeling Framework (EMF) [13] that may replace the custom MSC classes we have developed and for translation between graphical and textual notations the Meta Object Facility (MOF) [14] . The authors would like to acknowledge that this research was supported, in part, by the STATUS ESPIRIT project (IST-2001-32298) and by an IBM Eclipse Innovation Grant (EIG 2005). There are a number of initiatives to utilise Eclipse for modeling and then generating web services compositions including [11] and [12] . We focus on modeling web services as reusable components from the perspective of assessing multiple client usage scenarios. We achieve this by using formal software process verification techniques and providing the user with validation analysis to ascertain if requirements have been implemented appropriately. The related work can be seen as opportunities to contribute our plug-in such that their work could potentially benefit from this analysis.

O O O O O O O O O O O O O O O O O O B O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Segmentation-based multi-class semantic object detection Abstract In this paper we study the problem of the detection of semantic objects from known categories in images. Unlike existing techniques which operate at the pixel or at a patch level for recognition, we propose to rely on the categorization of image segments. Recent work has highlighted that image segments provide a sound support for visual object class recognition. In this work, we use image segments as primitives to extract robust features and train detection models for a predefined set of categories. Several segmentation algorithms are benchmarked and their performances for segment recognition are compared. We then propose two methods for enhancing the segments classification, one based on the fusion of the classification results obtained with the different segmentations, the other one based on the optimization of the global labelling by correcting local ambiguities between neighbor segments. We use as a benchmark the Microsoft MSRC-21 image database and show that our method competes with the current state-of-the-art. The object detection and categorization aims to extract two kinds of information from the images: which objects are present in the image and where exactly are those objects located. The detection is closely linked to the task of object segmentation. One of the difficulties of object segmentation is to extract complex, highly structured objects composed of regions visually dissimilar. On the other hand, the recognition of objects is affected by several problems: low quality of images, partial occlusions, visually heterogeneous object categories, cluttered background, . . . The discrepancy between the low level features that can be extracted from the images and the high level semantic interpretation of images is known as the semantic gap [30] . Many research efforts have been carried in the task of global image recognition such as scene classification [6, 14, 23] , but object recognition remains a challenge. This task is even more ambitious when multiple objects are present in the images.. One of the earliest work closely linked to the object detection and recognition topic was proposed by Duygulu et al. [10] . In this work, the authors address the recognition problem as the process of attaching words to image segments considering the task as a translation between one language (English words) to another (visual words, or blobs). A mapping between the keywords and the visual blobs is performed using a method based on Expectation Maximization. The rest of the literature [2, 13, 16, 28, 33] noticeably differs from the original work by Duygulu et al. in the sense that the models built try to exploit the maximum of information that can be extracted from the image: not only low level features (color, texture, etc.), but also local contextual relationships between pixels or image segments, location and even global relevance estimates. He et al. [16] proposed a pixel-wise labelling into a finite set of labels using a multiscale conditional random field formulation(mCRF). The mCRF combines the output of a local pixel-wise classification, a representation of the local geometric relationships between the objects and global label features into a probabilistic framework, trained using supervised labeled image data. Shotton et al. [28] introduced a new approach for learning a discriminative model of object classes based on texture, layout and context information. Their model is based on texturelayout f ilters, a feature which jointly model patterns of texture and their spatial layout. Texture layout filters are combined with lower level image features (color, location) into a Conditional Random Field to provide a high-level discriminative model. They also discuss an efficient learning method of the CRF parameters based on boosting and piecewise training method. Verbeek and Triggs [33] point out that Markov Random Field and aspect models such as Probabilistic Latent Semantic Analysis and Latent Dirichlet Allocation are complementary methods which attempt to improve the coherence of the labelling of image patches. MRF provides crisper local labelling by exploiting neighbourhood-level couplings while PLSA and LDA use global relevance estimates. They studied two spatial extensions of the aspect models, one based on a forest of minimal spanning trees and the other one on a regular 8-neighbor MRF. Galleguillos et al. [13] have shown that introducing contextual information about the co-occurrences and the relative location of image regions with local appearance-based features improves the global labelling. Athanasiadis et al. [2] define a framework for simultaneous image segmentation and object labelling operating at the semantic level. They represent the contextual information as an ontological taxonomy of the set of possible semantic labels and employ fuzzy algebra to adjust the labelling of the regions given by region growing segmentation algorithms.. Our approach has been inspired by the notion of superpixels in image segmentation given by Ren and Malik [26] . As opposed to pixels which are an unnatural arbitrary grid sampling of an image into a very large number of entities, superpixels provide a larger, locally homogeneous and coherent regions that preserve most of the structure necessary for accurate segmentation. We take up the idea that an image over-segmentation is sufficient to perform efficient recognition of small objects or object parts. The visual descriptors extracted at the region level are statistically more robust to noise than descriptors extracted at the pixel level. We will show that a Support Vector Machine classifier trained in an appropriate descriptor space for representing the segments already yields reasonable performances. A similar approach of using image segments was proposed by Hoeim [17] for the recovery of geometry context from single images. Yang et al. integrated appearance based classification of meanshift patches, a bag-of-keypoints model and global shape based Elliptical Fourier descriptors within a unified framework for object-based segmentation. Gould et al. proposed to combine appearance-based features computed on superpixels patches with relative location priors in a two stage classification process [15] . Malisiewicz and Efros have also shown that using image segmentation was efficient to improve the spatial support for object detection and recognition [19] .. In 1976, Rosenfeld et al. have formulated a process to reduce the ambiguities in object identification based on the relationships between the objects [27] . They proposed an ambiguity-reduction process by iterated parallel relaxation operations known as Relaxation Labelling. This approach was widely studied by the researchers in the context of graph matching for object recognition in videos [7] . We show that relaxation labelling improves the global coherence of the segment classification and provides cleaner, regularized object extraction. Our framework for object detection and categorization is depicted in Fig. 1 . Each row going from image segmentation to relaxation labelling describes the different steps to achieve semantic object extraction using a single segmentation method. The image is first segmented, feature descriptors are extracted on the segments and the classification of the segment into one of the semantic categories is performed based on these descriptors. The initial classification is further refined with the relaxation labelling algorithm. As the figure shows, it is possible to run the framework using multiple segmentation algorithms as the initial preprocessing step. When multiple segmentations and detection are available, it is possible to combine them using a late fusion schema, which is depicted by the colored arrows going from the segment classification or relaxation labelling boxes to the late fusion box. Again, it is possible to apply the relaxation labelling at the very end on the combined results as a last regularization step.. In this paper, we provide a detailed contribution for each step of the framework. In order to gain insights about the impact of image segmentation on the segment recognition performances, we applied our framework using four segmentation algorithms which will be introduced in Section 2. In Section 3, we introduce an efficient low level feature representation of the image segments. We propose to model the segments using color, texture and keypoint-based descriptors. For the latter, we propose a new way of sampling the keypoints locations to obtain a robust keypointbased description of the segments. In the same section, we discuss the segment classification method focusing on how to obtain a multi-class probability estimates for each segment using Support Vector Machines (SVM) classifiers. Then, we present our methods for improving the recognition results: the relaxation labelling algorithm is described in Section 4, while the late fusion strategies are introduced in Section 5.. In Section 6 we perform the quantitative evaluation of our approach and compare it with the current state-of-the-art. The experiments are carried using the MSRC-21 class database, a collection of 591 pictures of 21 object categories taken under different viewpoint and illumination conditions. Each image has a corresponding pixel-wise segmentation ground truth enabling training and evaluation (see Fig. 2 ). The goal of this section is to assess the impact of the following parameters in the quality of the object detection and categorization:. -The influence of the quality of the segmentation algorithm -The benefits of the neighborhood regularization method relaxation labelling -The interest of late fusion of multiple classification outputs using different segmentation support. The experiments were conducted on the MSRC-21 class image database. We followed the same evaluation protocol carried by other works on the same dataset [15, 28, 34, 36] : we ignored the void class during training and evaluation. The classes horse and mountain are also ignored due to insufficient amount of representative in the database. The reported results are the pixel-wise accuracy, that is the percentage of pixels correctly classified with respect to the total number of pixels in the segmentation ground truth. We splat the database randomly into 50% for training and 50% for testing.. 6.1 Impact of the segmentation algorithm Figure 5 shows the overall pixel-wise accuracy obtained using the different segmentation algorithms presented in Section 2. The best results are achieved using Edison, which clearly indicates the benefits of an accurate segmentation for the recognition. This is emphasized by the fact that the results obtained by the triangle segmentation, despite the approximate same number of segments per image, are much lower. Graphseg and grid segmentation lead to the same overall pixel-wise accuracy, despite their very different spatial support (large accurate segments versus small, arbitrary patches). It seems that the very limited view of the objects provided by the small square patches is still sufficient for recognition. It is interesting to see the differences of results at the object class level, as shown in Fig. 6 . We can see that no segmentation yields better results for all the classes, but significant differences among the classes appears. For example, we obtain much worse classification results for categories dog and sheep using the graph-based segmentation than using the other algorithms, while the best results are obtained with this segmentation for the categories sky, building and cars. It is hard to tell from these results if one algorithm is particularly suited for some kind of object classes. However, an interesting result to note is the fact that good performances are obtained for some categories (body cow and sheep) with the grid segmentation while we would have expected that accurate segmentation would help for their recognition. The results obtained with the graphbased segmentation seems to suffer a lack of robustness in the recognition, since for some classes the results are very low compared with the other segmentation. The difference in recognition among the classes is due to the fact that the easiest classes are the one with the lowest visual variability (sky, grass) while the hardest classes are the ones with a high visual variability and fewer training examples.. From these results, we can draw the conclusion that each segmentation carry its own advantages that help to recognize some of the particular classes. The fact that the best results are achieved by edison contributes to the idea that an accurate segmentation improves the recognition, as was highlighted by other authors [15, 17, 19 ], Yet very different segmentation such as the high over-segmentation given by the grid and the coarser but accurate graph-based segmentation lead to the same recognition accuracy (overall). Nevertheless, in the case of single segmentation recognition, an over-segmentation should be favored since it appears to be more robust for the recognition of the different classes, despite the partial view of an object provided by the segments. Figure 7 presents the results obtained before and after relaxation labelling. The relaxation labelling step significantly increases the classification performances: 67.4 to 74.3 for triangle segmentation, 67.9 to 74.8 for grid segmentation, 68.7 to 75.1 for edison segmentation. Hence, we see that the regularization of the global labelling taking into account the relationships between neighboring regions is an efficient way of further improvement of the classification results. The probabilities estimates from the SVM model are sufficiently accurate to enable a convergence of the algorithm toward improved results. This is not the case, however, for the results obtained with graphseg, which are actually decreased after relaxation (66.9 to 66.8). Figure 8 presents the detailed classification results for the object classes before and after relaxation labelling. We note that, in fact, relaxation labelling is improving the results for most of the object classes over the direct classification results. However, for some classes which are the most frequent in the database (grass, sky), the results are decreased. Again, the large segments provided by the graph-based segmentation algorithm are a drawback for the relaxation labelling, since a wrong correction of a segment label has a stronger impact on the overall results.

O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O B O B I O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O B I O O O B I O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B I O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O B O O O O O O O O B I O O O B I O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O B I O O O O O O O O O O O B O O O O O B O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B I O O O O O O O O O O O O O O O O


Partitioning and placement for buildable QCA circuits Quantum-dot Cellular Automata (QCA) is a novel computing mechanism that can represent binary information based on spatial distribution of an electron charge configuration in chemical molecules.In this article, we present the first partitioning and placement algorithm for automatic QCA layout. We identify several objectives and constraints that will enhance the buildability of QCA circuits. The results are intended to: (1) define what is computationally interesting and could actually be built within a set of predefined constraints, (2) project what designs will be possible as additional constructs become realizable, and (3) provide a vehicle that we can use to compare QCA systems to silicon-based systems. Nano technology and devices will have revolutionary impact on the computeraided design (CAD) field. Similarly, CAD research at circuit, logic, and architectural levels for nano devices can provide valuable feedback to nano research and illuminate ways for developing new nano devices. It is time for CAD researchers to play an active role in nano research. One approach to computing at the nano-scale is the quantum-dot cellular automata (QCA) concept that represents information in a binary fashion but replaces a current switch with a cell having a bistable charge configuration. QCA devices can be realized in metal [Amlani et al. 1998 ] or with chemical molecules [Lieberman et al. 2002] . A wealth of experiments have been conducted with metal-dot QCA, with individual devices [Amlani et al. 1998 ], logic gates Amlani et al. 1999] , wires , latches [Kummamuru et al. 2002] , and clocked devices [Kummamuru et al. 2002] . These advancements have been followed by various recent efforts in developing CAD tools for QCA-based circuits and systems [Gergel et al. 2003; Bernstein 2003; Walus et al. 2004 ; J. Huang and Lombardi 2004] . A recent work on ILP-based QCA circuit partitioning is presented in Antonelli et al. [2004] , where the authors partition individual gates to timing zones so that the difference in clocking zone heights is minimized.. Our goal in this article is to explain how CAD can help research move from small circuits to small systems of quantum-dot cellular automata (QCA) devices. We leverage our ties to physical scientists who are working to build real QCA devices. Based upon this interaction, a set of near-term buildability constraints has evolved-essentially a list of logical constructs that are viewed as implementable by physical scientists in the near-term. Until recently, most of the design optimizations have been done by hand. These initial attempts to automate the process of removing a single, undesirable, and unimplementable feature from a design were quite successful. We now intend to use CAD, especially physical layout automation, to address all undesirable features of design that could hinder movement toward a "buildability point" in QCA. The net result should be an expanded subset of computationally interesting tasks that can be accomplished within the constraints of a given buildability point. CAD will also be used to project what is possible as the state-of-the-art in physical science expands.. In this article, we present the first partitioning and placement algorithm for automatic QCA layout. The purpose of zone partitioning is to initially partition a given circuit such that a single clock potential modulates the interdot barriers in all of the QCA cells within each zone. We then place these zones as well as individual QCA cells in these zones during our placement step. We identify several objectives and constraints that will enhance the buildability of QCA circuits and use them in our optimization process. The results are intended to: (1) define what is computationally interesting and could actually be built within a set of predefined constraints, (2) project what designs will be possible as additional constructs become realizable, and (3) provide a vehicle that we can use to compare QCA systems to silicon-based systems.

B O B O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Does wireless sensor network scale? A measurement study on GreenOrbs Abstract-In spite of the remarkable efforts the community put to build the sensor systems, an essential question still remains unclear at the system level, motivating us to explore the answer from a point of real-world deployment view. Does the wireless sensor network really scale? We present findings from a large scale operating sensor network system, GreenOrbs, with up to 330 nodes deployed in the forest. We instrument such an operating network throughout the protocol stack and present observations across layers in the network. Based on our findings from the system measurement, we propose and make initial efforts to validate three conjectures that give potential guidelines for future designs of large scale sensor networks. The network dynamics mainly come from the inherent concur rency of network operations instead of environment changes. (3) The environment, although the dynamics are not as significant as we assumed, has an unpredictable impact on the sensor network. We suggest that an event-based routing structure can be trained optimal and thus better adapt to the wild environment when building a large scale sensor network. Recent advances in low-power wireless technologies have enabled us to make use of wireless sensor networks, a new class of networked systems. Researchers have envisioned a wide variety of applications, from environment monitoring [1], scientific observation [2] , to emergency detection [3] , field surveillance [4] , structure monitoring [5] , and etc. In those applications, hundreds or even thousands of sensor nodes are assumed to be deployed in the target field. Beside many algorithmic studies that focus on designing efficient schemes or protocols to coordinate the large scale sensor network, there are also systematic studies that make efforts in optimizing sensor network behaviors in practice, which are usually tested on lab-scale testbeds or small scale deployments. An essential question, however, remains unclear at the system level, mo tivating us to explore from a point of real world deployment VIew:. Does the wireless sensor network really scale to contain hundreds or even thousands of nodes that cooperatively work without depleting the limited physical resources, just as it was expected?. There have been several large-scale sensor network de ployments reported during the past years, including Vigil Net for field surveillance [4] , Motelab that provides an indoor testbed [6] , SensorScope for weather monitoring in the wild [7] , and Trio which enables a large-scale solar powered sensor network [8] . Those deployments, however, are often highly optimized for specific application needs and not fully leveraged as platforms for consistently observing general network behaviors.. In this work, we conduct a measurement study on GreenOrbs, which is a consistently operating sensor net-work system deployed for the aim of forest monitoring. With up to 330 nodes deployed in the wild, GreenOrbs provides us an excellent platform for observing sensor network behaviors at scale. Figure 1 plots the real topology of the sensor network. The sink is deployed at the upper left corner. Each sensor node is depicted according to its 2-D geographical location. We plot all the communicational links through which data packets are delivered. Although such lined links together with the dotted nodes compose a network-wide topology which traditional algorithm or protocol designers used to play with, we highlight a subgraph within the network and exhibit that the concept of "topology" does vary according to the perspective we look at it. [10] when data packets are transmitted. If we consider different conditions or calibrating criterion, there will be more different types of "topologies", and such "topologies" vary from time to time. Indeed, like many large-scale distributed systems have exhibited, in sensor networks, there grow numerous dynamic behaviors with the concurrent and interactive operations inside the system. Such dynamic behaviors can hardly be fully considered before the system is deployed in the field brimming of unpredictable and unexpected operating conditions.. In this work, we conduct the measurement study on the operating sensor network system deployed in the wild, trying to summarize the critical factors that limit the system scale out of the dynamics on the surface. We instrument such an operating network throughout the protocol stack. We vary the system settings, e.g., the network scale, traffic generation, transmission power level, and test the system behaviors under a variety of conditions.. We present findings across different layers that the system works on. At the physical layer, we present measurements on radio signal strength impacted by wild environment. At the link layer, we measure packet drop/reception, link quality, transmitting rate over the entire network and how they are affected by a variety of system settings. At upper layers, we present observations on routing dynamic, traffic distribution, end-to-end delivery, topological features, and etc. Our study reveals that traditional opinions on the "hot area" around sink and the instability of links may not be the major concern for large-scale sensor network systems. The physical resources in such networks may have been underestimated and severely underutilized. There is an urgent need to improve current methods in company with those emerging critical factors when the network scales. Our experimental results also suggest us several guidelines that we should carefully consider in designing future protocols for large-scale sensor networks.. In particular, the designers should take special care of the phe nomena raised from the inherent contention and concurrency of numerous nodes when the sensor network scales, which might be underestimated in existing design concept continued from traditional Wi-LAN or MANET protocol design.. The rest of the paper is organized as follows. In Section II, we describe related work in sensor network deployment and measurement experiences, as well as existing work towards making sensor networks scalable. In Section III, we introduce the background of GreenOrbs, some details of the system implementation, and the measurement methodology. We present our major observations in Section IV. In Section V, we give a comprehensive discussion on how the network is bottlenecked and give guidelines in mitigating such effect. We conclude this work in Section VI. In this work we conduct a measurement study on a large scale operating sensor network system, GreenOrbs, with up to 330 nodes deployed in the wild. We aim to comprehensively understand how the sensor network performs when it scales to contain hundreds or even thousands of nodes. We instrument such an operating network throughout the protocol stack. The contribution of this work is twofold.. First, to the best of our knowledge, we are the first to conduct a long term and large-scale measurement study on an operating sensor network in the wild. We present observations across a variety of layers in the network that provide research community empirical experiences on how practical problems affect when the sensor network scales. Second, based on our basic findings from the system mea surement, we further propose and initially attempt to validate three conjectures that provide guidelines for future algorithm and protocol designs with larger scale sensor networks. In summary, (1) we think it might be very possible that some of the intermediate nodes bottleneck the entire network, and most of currently used indicators may not accurately capture them; (2) most of the wireless links in large scale sensor networks are physically stable. The dynamics mainly come from the inherent concurrency of network operations which should be further investigated and considered in designing scalable network protocols ; (3) the environment, while with insignificant dynamics, has an unpredictable impact on the sensor network under it. We suggest that an event based routing structure can be trained optimized and thus better adapt to the wild environment when building a large-scale sensor network. In this section we summarize the efforts of research commu nity in building large-scale sensor networks and corresponding measurement studies.. A number of practical network deployments have been reported during the last decade [I] , [2] , [7] , [11] . Although the findings from the above deployments are important, the measurements at this scale, usually tens of sensors, can hardly reveal some network behaviors, such as routing dynamics and topology evolution, which exist only in large-scale networks. Researchers have designed and developed indoor medium scale testbed such as MoteLab [6] and Kansei [12] . Those indoor testbeds, however, do not fully capture the nuances in realistic environments.. Deploying sensor networks at scale is important because each order of magnitude increase in network size ushers in a new set of unforeseen challenges. VigilNet [4] is designed to support long-term military surveillance using a sensor network consisting of 200 nodes. ExScal [13] is an attempt to deploy a sensor network at "extreme" scale. The system consists of about 1000 sensor nodes and 200 backbone nodes, covering 1300*300 square meters. Dutta et al. [8] report a network deployment "Trio" of 557 solar-powered motes for multi-target tracking. SenseScope [7] , [14] is a real-world deployment that took place on a rock glacier, consisting of about 100 sensor nodes.. Most of above mentioned systems are not clearly proper for network measurement due to the following two reasons. First, those systems are organized hierarchically. Such hierarchical architecture inherently alleviates the negative impact induced by large-scales, thus hardly reflecting the performance of general and homogeneous ad-hoc sensor networks. Second, no single system has integrated large-scale (e.g., hundreds of nodes) and long-lifetime (e.g., one year) into a cohesive whole. In other words, those deployments have achieved either scale or lifetime, but usually not both.. In the context of wireless sensor networks, a number of em pirical studies present network measurement results in many aspects, with emphasis on understanding the complex and non ideal behavior of low power wireless communications. Link quality is one of the most important indicators for wireless communication and thus attracts many research efforts [15] [20] . Srinivasan et al. [9] conclude from measurements on MicaZ motes with CC2420 radios that RSSI is a good estimate of link quality. The authors in [21] study the transition region and quantify its influence. Studies such as [22] , [23] emphasize the temporal performance dynamics of wireless links and provide important findings about such phenomenon. The study of beta-factor [24] presents a comprehensive study to quantify and characterize link burstiness. The authors in [25] - [28] investigate radio interference and point out the inaccuracy of range-based interference model [25] .. The results presented in those empirical studies are basically obtained from single-purpose and short-lived testbeds of tens of sensors. In contrast, the measurements of GreenOrbs are fairly comprehensive, from low-level radio signal strength and link quality to high-level routing and data traffic issues.

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Automatic generation of persistent formations for multi-agent networks under range constraints Abstract-In this paper we present a collection of graph-based methods for determining if a team of mobile robots, subjected to sensor and communication range constraints, can persistently achieve a specified formation. What we mean by this is that the formation, once achieved, will be preserved by the direct maintenance of the smallest subset of all possible pairwise interagent distances. In this context, formations are defined by sets of points separated by distances corresponding to desired interagent distances. Further, we provide graph operations to describe agent interactions that implement a given formation, as well as an algorithm that, given a persistent formation, automatically generates a sequence of such operations. Experimental results are presented that illustrate the operation of the proposed methods on real robot platforms. Due to recent developments in mobile sensing, computation, and actuation, formation control for multi-agent networks has received significant attention during the last decade. (See for example [1] , [2] , [3] , [4] , [5] , [6] , [7] , [8] , [9] , [10] , [11] , [12] , [13] , [14] for a recent, representative sample.) In fact, recent research suggests the use of graph-theoretic structures to represent formations, where vertices represent agents, and edges represent specific inter-agent distances to be maintained through decentralized control laws (see [15] , [16] , [17] , [18] , [19] , [20] , [21] , [22] , [23] , [24] , [25] , [26] ).. This work is part of a National Aeronautics and Space Administration (NASA) project to implement a multiple-robot system for research in Antarctica. In this project, a team of geologists at NASA should be able to use a mobile sensor network composed of mobile robots to take sensor readings across ice shelves in order to better understand the impacts of global climate change on the ice shelves. According to specifications, the network should be able to automatically deploy and distribute itself across an area of interest with a user-defined resolution, and to achieve specific, user-defined geometric relationships among the members of the network. The pre-Antarctic stages of this project will be implemented with a prototype multiple-robot system, shown in Fig. 1 .. Based on the NASA project as a motivating application, we will study graph-based abstractions of formations, and we define a target formation as a set of pairwise, desired interagent distances associated with a complete graph, i.e., these distances are specified with respect to all pairs of agents. However, it may not be the case that all inter-agent distances Brian S. Smith, Magnus Egerstedt, and Ayanna Howard are with the School of Electrical and Computer Engineering, Georgia Institute of Technology, Atlanta, GA, USA. Email: {brian, magnus, ayanna.howard}@ece.gatech.edu Fig. 1 . The multi-robot network. This network is used in the pre-Antarctic stages of this project. Its mobility and sensor abilities approximate the Antarctic network sufficiently to assemble and deploy formations using the same automatic tools.. are needed, which leads to the study of so-called persistent formations [27] .. In a persistent formation, each agent is assigned a set of constraints, which are specific inter-agent distances to maintain. These constraints are oriented in the sense that each constraint is the responsibility of a single agent rather than two agents. Maintaining a formation may not require all interagent distances to be directly maintained by the control laws. Persistent formations typically involve only a proper subset of all possible inter-agent constraints. In [28] , graph operations are proposed that, through successive applications, produce a graph corresponding to a persistent formation.. We want to use such operations in order to build persistent formations in the presence of constraints on the effective communication and sensing distances. In fact, these types of constraints were not considered in [28] , and the main contribution of this paper is the sequential construction of persistent formations that respect the inter-agent range constraints.. The outline of this paper is as follows: In Section II, we recall some of the basic definitions needed to set up the problem, followed by a method for determining if a specific target formation is rigidly feasible with respect to the range constraints, in Section III. Next, in Section IV, we show how the same method for determining rigid feasibility also determines persistent feasibility. We also present a set of graph operations for building persistent formations with respect to the range constraints, as well as an algorithm for automatically generating a sequence of such operations given a persistent formation graph (Section V). The experimental results are discussed in Section VI, followed by the conclusions in Section VII.

O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Some problems on factorizations with constraints in bipartite graphs Let G = (X; Y; E(G)) be a bipartite graph with vertex set V (G) = X ∪ Y and edge set E(G) and let g and f be two non-negative integer-valued functions deÿned on In our daily life many problems on optimization and network design, e.g., coding design, building blocks, the ÿle transfer problems on computer networks, scheduling problems and so on, are related to the factors, factorizations and orthogonal factorizations of graphs [2] . The ÿle transfer problem can be modeled as (0; f)-factorizations (or f-colorings) of a graph [12] . The designs of Latin squares and Room squares are related to orthogonal factorizations in graphs which are ÿrstly presented by Alspach et al. [2] .. Let G be a graph with vertex set V (G) and edge set E(G). The degree of a vertex x is denoted by d G (x). Let g and f be two non-negative integer-valued functions deÿned on V (G) such that g(x) 6 f(x) for each x ∈ V (G). Then a (g; f)-factor of G is a spanning subgraph F of G satisfying g(x) 6 d F (x) 6 f(x) for each x ∈ V (F). In particular, G is called a (g; f)-graph if G itself is a (g; f)-factor. A subgraph H of G is called an m-subgraph if H has m edges in total. A (g; f)-factorization F={F 1 ; F 2 ; : : : ; F m } of a graph G is a partition of E(G) into edge-disjoint (g; f)-factors F 1 ; F 2 ; : : : ; F m . Let a and b be two non-negative integers with a 6 b. If g(x) = a and f(x) = b for all x ∈ V (G), then a (g; f)-factor is called an [a; b]-factor. Similarly, an [a; b]-factorization can be deÿned. Let H be a km-subgraph of a graph G. A (g; f)-factorization F = {F 1 ; F 2 ; : : : ; F m } is k-orthogonal to H if |E(H ) ∩ E(F i )| = k for 1 6 i 6 m. If for any partition {A 1 ; A 2 ; : : : ; A m } of E(H ) with |A i | = k there is a (g; f)-factorization F = {F 1 ; F 2 ; : : : ; F m } of G such that A i ⊆ F i , 1 6 i 6 m, then we say that G has (g; f)-factorizations randomly k-orthogonal to H . Note that randomly 1-orthogonal is equivalent to 1-orthogonal and 1-orthogonal is also called orthogonal. It is easy to see that the problems on (g; f)-factorizations, k-orthogonal or randomly k-orthogonal factorizations are in NP. Given a graph G, two functions g and f deÿned as the above and a positive integer k, is there a (g; f)-factorization F={F 1 ; F 2 ; : : : ; F m } of G such that m 6 k ? The general problem is NP-complete since the simple version of the problem can be reduced to the edge-coloring problem. By the above argument, clearly, the problem which asks whether a graph has a (g; f)-factorization k-orthogonal (or factorizations randomly k-orthogonal) to a given km-subgraph is also NP-complete as factorizations are the special case of orthogonal (or randomly orthogonal) factorizations. Note that there are polynomial algorithms for deciding whether a graph G has a (g; f)-factor. When g(x) = f(x) for every x ∈ V (G) or G is a bipartite graph, Heinrich et al. [6] gave a very simple existence criterion for a (g; f)-factor which yields an (g; f)-factor algorithm of time complexity O(g(V (G))|E(G)|); Hell and Kirkpatrick gave O( g(V (G))|E(G)|) algorithms [7] . For general (g; f)-factor problems Anstee gave a polynomial algorithm which either ÿnds a (g; f)-factor or shows that one does not exist in O(|V (G)| 3 ) operations [3] . In this paper we consider only bipartite graphs and give algorithms for ÿnding the desired factorizations. The graphs considered in the following will be ÿnite undirected graphs with neither loops nor multiple edges. A graph denoted by G = (X; Y; E(G)) is a bipartite graph with vertex bipartition (X; Y ) and edge set E(G). Alspach et al. [2] posed the following problem: given a subgraph H , does there exist a factorization F of G with some ÿxed type orthogonal to H ? Li and Liu [10] gave a su cient condition for a graph to have a (g; f)-factorization orthogonal to any given m-subgraph. Lam et al. [9] studied orthogonal factorizations of graphs. Kano [8] studied [a; b]-factorizations in graphs. Anstee and Caccetta [4] discussed orthogonal matchings. Recently Tokuda [11] discussed connected [a; b]-factors. The basic results on graph factors and factorizations can be found in [1] . Now we consider the randomly k-orthogonal and k-orthogonal factorizations of bipartite graphs. The purpose of this paper is to solve some problems on orthogonal factorizations for bipartite graphs. It is shown that a bipartite (mg + m − 1; mf − m + 1)-graph G has (g; f)-factorizations randomly k-orthogonal to any given km-subgraph if k 6 g(x) for any x ∈ V (G) and a bipartite (mg+m−1; mf−m+1)-graph G has a (g; f)-factorization k-orthogonal to any given km-subgraph if k − 1 6 g(x) for any x ∈ V (G) and that every bipartite (mg; mf)-graph has a (g; f)-factorization orthogonal to any given m-star if 0 6 g(x) 6 f(x) for every x ∈ V (G). We shall show that the results in this paper are in some sense best possible. Furthermore, we shall give polynomial algorithms for ÿnding the desired (g; f)-factorizations.

O O O O O O O O O O B O O O O O O B I O O O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B O O O O O O O O O O O O O O B O O B O O O O O O O O O O B O O O O O O O O O O B O O B B O B O O O O O O B O O O O O O O O O O O O O O O O B O O O B O O B O O O B B O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B B O O O O O O O O O O O O O O O O O O O O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B B O O O O O O O O O B O B O O B I O O O O O O O O O O O O B B O O O B B B O O O O O O O O O O O O O O O O O B B O O O O O B O O O O B B O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B O O O O O O O O O O O O O O O O B O O O O O B O O B O O O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B O


An empirical evaluation of entropy-based traffic anomaly detection Entropy-based approaches for anomaly detection are appealing since they provide more fine-grained insights than traditional traffic volume analysis. While previous work has demonstrated the benefits of entropy-based anomaly detection, there has been little effort to comprehensively understand the detection power of using entropy-based analysis of multiple traffic distributions in conjunction with each other. We consider two classes of distributions: flow-header features (IP addresses, ports, and flow-sizes), and behavioral features (degree distributions measuring the number of distinct destination/source IPs that each host communicates with). We observe that the timeseries of entropy values of the address and port distributions are strongly correlated with each other and provide very similar anomaly detection capabilities. The behavioral and flow size distributions are less correlated and detect incidents that do not show up as anomalies in the port and address distributions. Further analysis using synthetically generated anomalies also suggests that the port and address distributions have limited utility in detecting scan and bandwidth flood anomalies. Based on our analysis, we discuss important implications for entropy-based anomaly detection. There has been recent interest in the use of entropy-based metrics for traffic analysis [20] and anomaly detection [10, Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. IMC'08, October 20-22, 2008 , Vouliagmeni, Greece. Copyright 2008 ACM 978-1-60558-334-1/08/10 ...$5.00. 4, 7, 5, 12, 18] . The goal of such analysis is to capture finegrained patterns in traffic distributions that simple volume based metrics cannot identify [10] .. Several traffic features (e.g., flow size, ports, addresses) have been suggested as candidates for entropy based anomaly detection. However, there has been little work in understanding the analysis capabilities provided by a set of entropy metrics used in conjunction with one another. For example, it is unknown whether the different features complement each other, or if they detect the same anomalies and are redundant.. The goal of this paper is to provide a better understanding of the use of entropy-based methods in anomaly detection. We consider two types of distributions based on flow-header features and behavioral features. The flow-header features are addresses (source and destination), ports (source and destination), and the flow size distribution (FSD) [10, 5, 9] . The behavioral features are the in and out-degree distributions (degree of an end-host X is the number of distinct IP addresses that X communicates with) that capture the structure of end-host communication patterns.. The key results from our measurement study are: • Port and address distributions are highly correlated, with pairwise correlation scores greater than 0.95. The degree distributions and FSD are weakly correlated with each other and with the port/address distributions.. • The correlation between the source (destination) port and source (destination) address distribution arises due to the nature of the underlying traffic patterns. However, the correlations across the source and destination distributions stem from the uni-directional nature of flow-level measurements available today.. • The anomalies detected by the port and address distributions overlap significantly. In our dataset, almost all the anomalies detected by these distributions are alpha flows [10] . In contrast, host degree distributions and FSD identify anomalous scan, DoS, and P2P activity that are not detected by the port and address distributions.. • Experiments with synthetically generated anomalies show that FSD and the degree distributions detect scanning events that cannot be detected by the port and address distributions. For DDoS-style events, port and degree distributions detect only high-magnitude events that would have appeared as traffic volume anomalies.. These observations have important implications for entropybased analysis. First, we should select candidate distributions with care. While ports and addresses have been commonly suggested [10] as good candidates for entropy-based anomaly detection, our measurements question this rationale. Our results also suggest a natural approach: select traffic distributions that inherently complement one another and thus provide different views into the underlying traffic structure. Second, we need to move beyond the traditional uni-directional flow semantics available today (e.g., [14, 16] ), since they can artificially skew the properties of the underlying distributions. Thus, it is prudent for administrators to use bi-directional flow collection tools whenever possible. Finally, we discuss how to use the correlations to design a better anomaly detection system. Our preliminary results show that using time-series anomaly detection on the correlation scores can expose new anomalies that do not manifest in the raw time-series. Entropy-based methods have recently been suggested as good candidates for fine-grained anomaly detection and traffic classification. The goal of our measurement study is to understand the analysis and detection capabilities provided by different entropy based metrics.. We find that the port and address distributions are strongly correlated both in their entropy timeseries and detection capabilities. The behavioral metrics (in-and out-degree) and the flow size distribution provide detection abilities that are distinct from other distributions. Using synthetic anomalies, we further confirmed that the port and address distributions have limited utility in anomaly detection: they are ineffective for scanning attacks, and the flood anomalies they detect are large enough to be volume anomalies.. Our results have two main implications. First, we should look beyond port and address distributions for fine-grained anomaly detection. In particular, we should consider distributions that complement each other in their detection capabilities. Second, to avoid the biases arising from unidirectional auditing, it is prudent to use bi-directional flow abstractions for computing traffic distributions. The use of entropy and distributions of traffic features has recently received a lot of attention. Feinstein et al. [5] consider the use of entropy of the distribution of source addresses seen at a network ingress point for DDoS detection. Lakhina et al. [10] augment the PCA framework with entropy based metrics and show that this detects anomalies that cannot be identified using volume based analysis alone. These approaches show the promise of entropy-based anomaly detection. Our work studies the selection of traffic feature distributions for entropy based anomaly detection.. Lee and Xiang [12] propose information-theoretic measures for intrusion detection. Entropy has also been to automatically cluster traffic patterns [20] . Wagner et al. [18] use entropy for worm detection by evaluating the compressibility of flow data during attacks.. There is a large body of work related to the accuracy of estimating distributional properties. These include the work on streaming algorithms for estimating the flow size distribution [9] and distribution entropy [11] . Brauckhoff et al. [4] evaluate how packet sampling affects the fidelity of entropy based anomaly detection and show that sampling does not affect the accuracy of detecting the Blaster worm [13] .

O O O O O O B I O O O B I O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B I O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O B O B I O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B O B I O O O O O O O O O O O O O O O O