Formal representation of the SS-DB benchmark and experimental evaluation in EXTASCID Evaluating the performance of scientific data processing systems is a difficult task considering the plethora of application-specific solutions available in this landscape and the lack of a generally-accepted benchmark. The dual structure of scientific data coupled with the complex nature of processing complicate the evaluation procedure further. SS-DB is the first attempt to define a general benchmark for complex scientific processing over raw and derived data. It fails to draw sufficient attention though because of the ambiguous plain language specification and the extraordinary SciDB results. In this paper, we remedy the shortcomings of the original SS-DB specification by providing a formal representation in terms of ArrayQL algebra operators and ArrayQL/SciQL constructs. These are the first formal representations of the SS-DB benchmark. Starting from the formal representation, we give a reference implementation and present benchmark results in EXTASCID, a novel system for scientific data processing. EXTASCID is complete in providing native support both for array and relational data and extensible in executing any user code inside the system by the means of a configurable metaoperator. These features result in significant improvement over SciDB at data loading, extracting derived data, and operations over derived data. In this paper, we present a formal representation of the SS-DB benchmark in terms of array algebra operators and array query language constructs. These are meant to simplify the implementation in other systems and foster the acceptance of SS-DB as the standard benchmark to evaluate scientific data processing applications. Given that no alternatives exist, SS-DB fills an important void in the evaluation of a large class of Big Data applications. To verify the soundness of our formalization, we give a reference implementation and present benchmark results in EXTASCID, a novel system for scientific data processing we have developed. EXTASCID is complete in providing native support both for array and relational data and extensible in executing any user code inside the system by the means of a configurable metaoperator. These features result in considerable improvement over SciDB at data loading, extracting derived data, and operations over derived data. The results prove that the integrated EXTASCID architecture supporting natively both arrays and relations is more suited for complex scientific processing over raw and derived data requiring a high degree of extensibility. Given the algebra representation for all the operations in the benchmark, we analyze what array structures and operators are required for an actual implementation. The 3-D array corresponding to the raw images is given in Eq. (1). Images is represented in the local coordinate system. This representation suffices to answer Q1-Q3 and to execute the cooking. For Q8 and Q9, the representation of images in the global coordinate system is required. This can be obtained by shifting the origin of the images based on image_origin. For grouping, only the observation centers are required. A sparse 3-D array obs_center over the global coordinates containing valid cells only where observation centers are located can do the job. The data inside the cell contain the observation id and a set of attributes (Q4, Q6). All the cells that are part of observations are required for Q5. They can be represented as a sparse 3-D array obs with the observation id as the single attribute. This array is also required to answer Q9. It is important to notice that this complete representation of observations in terms of the member cells eliminates the need to represent bounding polygons as arrays. With this representation, intersections between polygons can be expressed as a combination of the REBOX and REDUCE operators (Eq. (8)). Group centers are required to answer Q7 and Q8. In Q7, the single group centers can be represented as a sparse 3-D array group_center containing the group ids in the valid cells. In Q8, group centers are recorded for each image in the cycle-they can be different from one image to another. Thus, a sparse 4-D array group_center_img with the group ids stored in the valid cells is required. Notice that cycle represents the additional dimension in the arrays corresponding to groups. Table 1 summarizes the arrays required to answer the benchmark queries.. In terms of the ArrayQL algebra operators ( Table 2) , REBOX is used in all the range queries. The generalized form of APPLY+ that is present only in AML is used for cooking, grouping, Q2, Q3, and Q6. The calls differ in terms of the cells where APPLY+ is invoked, the shape of the neighborhood, and the aggregator. REDUCE is applied whenever an aggregate has to be computed, e.g., Q1 and Q4. FILTER, SHIFT, and INNERDJOIN are the other operators used throughout the benchmark queries, with INNERDJOIN appearing only in cooking and grouping. From a critical point of view, the SS-DB benchmark emphasizes range queries and aggregation as the most significant array operations. This is in line with the previous work on array databases. In addition to these standard operators, APPLY+ variations and INNERDJOIN provide SS-DB generality beyond any other array benchmark. . There are multiple conclusions we can draw from the experimental results presented in this section. The most important point to remark is that although EXTASCID and SciDB share common design and implementation features, their performance is quite different for the SS-DB benchmark. EXTASCID outperforms SciDB at data loading and grouping while their performance is similar for cooking. The difference for loading is two orders of magnitude when the reorganization time is included. The relationship between the two systems is more nuanced when considering query performance. SciDB is slightly faster on queries over raw data whenever compression and chunk caching can be applied. For queries over derived data-amenable to a relational representation-EXTASCID provides the better execution time since it supports natively both dense as well as sparse arrays. Since SciDB is targeted at dense arrays, mapping between the two representations is required. These trends manifest both for a single-node configuration as well as in a cluster environment. Whenever processing raw data the two systems exhibit linear scaleup, thus confirming their scalability over large datasets partitioned across many processing nodes. There are three lines of research on scientific and array data processing that we consider related to the topics addressed in this paper-benchmarking, array query algebras and languages, and scientific data processing systems. To the best of our knowledge, Sloan Digital Sky Survey (SDSS) [3, 25] is the single other benchmark targeted specifically at scientific processing. Similar to SS-DB, SDSS is also based on processing astronomical images. Unlike SS-DB though, SDSS operates exclusively on relational data obtained as a bi-product of astronomical observations. SS-DB is more general and contains a full spectrum of operations ranging from raw data processing to the creation and querying of derived observation data. These operations manipulate array-oriented data through relatively sophisticated user-defined functions, not always expressible in SQL. While typical implementations of the SDSS benchmark are SQL-based, e.g., MS SQL Server and MonetDB [26] , SciDB and MySQL [1] are the only two implementations of the SS-DB benchmark we are aware of. With our implementation in EXTASCID, we provide another reference point for a larger adoption of the SS-DB benchmark-the only general benchmark for scientific data processing.. Array algebras and query languages ArrayQL algebra [8] and query language [9] are formalisms introduced in the SciDB context-similar to the SS-DB benchmark. This is the main reason we choose these formalisms to represent the SS-DB operations. They are not commonly accepted though as the representative array algebra and query language. In fact, there is no commonly accepted array algebra and query language. In the following, we discuss other such alternatives proposed in the literature. AQL [27] is a declarative query language for multi-dimensional arrays that treats arrays as functions from index sets to values rather than as collection types. AQL is based on the nested relational calculus with arrays which plays the same role relational calculus and algebra play for the relational data model. The RasDaMan [13] array algebra conceptualizes arrays as functions from rectangular domains to cell values. Three core constructs-MARRAY, COND, and SORT-that can express every array operation when composed together are introduced. In the corresponding RasQL query language [28] , arrays are treated as a composite attribute type with a set of corresponding operators. AML [12] is an algebra consisting of three operators that manipulate dense arrays and take bit patterns as parameters. A significant AML limitation is that it contains only structural operators, i.e., operators that consider the indexes. At query language level, AML is more like an elevated execution plan description than a declarative array query language. RAM [29] and SRAM [30] are array algebras for dense and sparse arrays, respectively, developed in the context of the MonetDB [31] columnar database system. Since the execution happens inside a relational database engine, array queries follow a sequence of transformations that take arrays represented in the comprehension syntax to relational operators through an intermediate array algebra stage. Although a series of rewriting rules and optimizations are applied at each of these two steps, relying on the relational algebra operators to map and process array operations introduces inefficiencies due to the impedance mismatch in representation. SciQL [10] is the most comprehensive extension to the SQL:2003 standard with support for arrays. It provides seamless integration of set, sequence, and array semantics. The goal is to make minimal modifications to the SQL syntax while allowing for maximum expressiveness in the array operations supported by the language. An interesting characteristic of all the array algebras discussed above is their equivalence. In [32] , it is shown that all the array algebras can be reduced to RasQL-both in array representation as well as operations. This is primarily due to the equivalence between comprehensions and the MARRAY operator for creating arrays. We point the interested reader to [23] for a comprehensive discussion on these and other array algebras and query languages.. Array processing systems EXTASCID is part of a long series of parallel systems for scientific data processing. Titan [33] and T2 [34] are the first systems designed with extensibility in mind. They adopt an execution strategy closely related to the Map-Reduce [35] paradigm. More recently, SciHadoop [36] implements array processing on top of the popular Hadoop Map-Reduce framework. The main differences between EXTASCID and these systems are the different execution strategy, i.e., UDA vs. Map-Reduce, and the native support for arrays and relations in EXTASCID. RasDaMan [28] is a general middleware for array processing with array chunks stored as BLOBs in a back-end database. The processing is specified through a limited number of secondorder operators integrated into SQL and executed entirely inside the middleware. RasDaMan is targeted only at array data-relational data have to be processed either at the application level or inside the middleware-and it is not parallel. The RAM [29] and SRAM [30] systems provide support for array processing on top of the Mon-etDB [31] columnar database. They do not provide native support for arrays since arrays are represented as relations and array operations are mapped over relational algebra operators. Moreover, RAM and SRAM are not parallel. RIOT [37] is a prototype system for linear algebra operations over large vectors and matrices mapped into a standard relational database representation. Linear algebra operations are rewritten into SQL views and evaluated lazily. Since a relational database is used both for array storage and processing, RIOT falls in the category of middleware systems similar to RasDaMan. ArrayStore [20] and TrajStore [38] are storage managers optimized for multi-dimensional arrays and trajectories, respectively. They do not provide a query execution strategy and operators to implement it. We point the interested reader to [23] for a comprehensive discussion on array database systems.. SciDB SciDB [11] is the system EXTASCID resembles the most. Both are parallel systems designed to be extensible. Parallelism is obtained through data partition-ing into chunks further optimized by dimension suppression. While SciDB makes heavy use of array decomposition and compression, EXTASCID resorts to composite chunks organized vertically. SciDB supports natively only arrays. EXTASCID provides native support both for arrays and relations. While SciDB adopts a traditional pull-based execution model, EXTASCID is entirely push-driven. The execution strategy in EXTASCID is well-defined through the UDA interface which makes reasoning about parallelism clear. The same is not true in SciDB where a series of UDFs are arbitrarily interconnected and the parallelization strategy is based on chunk overlapping rather than merging. It is important to emphasize that in SciDB UDAs are considered only as a mechanism to implement aggregate computations for user-defined types (UDT). They are never considered a general processing paradigm capable of expressing almost any type of computation, the case in EXTASCID. While there are discussions on the implementation of the SS-DB benchmark in multiple of these systems, the SciDB implementation is the only we are aware of at the time when the paper is written. Consequently, this is our reference for the EXTASCID evaluation on the SS-DB benchmark. As mentioned earlier in the paper, EXTASCID is implemented on top of the GLADE parallel processing system. GLADE [14] in turn uses an extended version of the centralized DataPath [15] relational execution engine for local processing at each node. As a result, EXTASCID inherits the relational algebra operators implemented in DataPath. It also inherits the communication mechanism for parallel aggregation available in GLADE. In this section, we provide more details on how the array-specific ArrayQL algebra operators are implemented in the DataPath execution engine. We consider parallel versions of the operators that process chunked arrays. Chunks are processed independently and in parallel, with minimal data sharing and transfer, in order to maximize the parallelism. Before discussing the array operators, we first look into the parallel implementation of the original relational algebra operators and of the GLA metaoperator.

O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Bias-free hypothesis evaluation in multirelational domains In machine learning one typically assumes that the true classification of an object depends only on the object itself and given the object, is independent of the classification of other objects. In this case, setting aside a sufficiently large and randomly chosen part of the training data as a test set, the observed sample error on the test set is an unbiased estimator of true error. However, in many application settings, those mainstream approaches to model evaluation might be inappropriate. As pointed out by [2], among others, whenever there is autocorrelation, i.e., whenever the target value of one object depends not only on the object itself, but also on other objects' classifications or information that is shared between objects, observed error on a randomly chosen test set may not be an unbiased estimator anymore. We introduce a sampling technique, generalized subgraph sampling, that avoids a bias in error estimation by establishing the required amount of linked objects in the test set.

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B O O O O O B I O O O O O O O O O O O O


IDD: A Supervised Interval Distance-Based Method for Discretization Abstract-This paper introduces a new method for supervised discretization based on interval distances by using a novel concept of neighborhood in the target's space. The proposed method takes into consideration the order of the class attribute, when this exists, so that it can be used with ordinal discrete classes as well as continuous classes, in the case of regression problems. The method has proved to be very efficient in terms of accuracy and faster than the most commonly supervised discretization methods used in the literature. It is illustrated through several examples, and a comparison with other standard discretization methods is performed for three public data sets by using two different learning tasks: a decision tree algorithm and SVM for regression.

O O O O O O O O O O O O O O O B I O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Modelling Multi-Output Stochastic Frontiers Using Copulas The aim of this work is to introduce a new econometric methodology for multioutput production frontiers. In the context of a system of frontier equations, we use a flexible multivariate distribution for the inefficiency error term. This multivariate distribution is constructed through a copula function which allows for separate modelling of the marginal inefficiency distributions and the dependence. We pay specific attention to the elicitation of a sensible (improper) prior and provide a simple sufficient condition for posterior propriety. Inference is conducted through a Markov chain Monte Carlo sampler. We use Bayes factors to compare various copula specifications in the empirical context of Dutch dairy farm data, with two outputs.JEL classification: C11, C15, C23, D24 In the economic literature an important place is taken by efficiency measurement, especially in microeconomics and industrial organization. In order to study this issue, the stochastic frontier model was introduced in Aigner et al. (1977) and Meeusen and van den Broeck (1977) . Fields of application abound and include banking, agriculture, public services and health care. The basic idea of a frontier is a characterization of best-practice technology in a particular sector: the production frontier describes the potential maximum production that an economic unit can achieve with a certain set of inputs. The distance between the hypothetical frontier and the observed production is directly related to efficiency. This distance (usually called "inefficiency") is typically modeled as a one-sided random variable, and different distributions have been proposed in the literature (see for example van den Broeck et al., 1994 , Tsionas, 2000 and Griffin and Steel, 2008 . In addition, we need to infer the frontier from the data and this entails another random component which is usually assumed to be symmetric measurement error. The sum of inefficiency and measurement error terms is often referred to as "composed error". In contrast to the DEA-type approaches to efficiency measurement, extensions of the basic stochastic frontier model to deal with multiple outputs have been particularly challenging. This is why the literature has often dealt with such cases through cost or profit functions. Kumbhakar (1996) and Fernández et al. (2000) provide a more detailed discussion of the particular problems that occur if we want to specify a meaningful statistical model for m > 1 outputs.. The approach of Fernández et al. (2000) is to use a parametric aggregator of the outputs and to model the aggregate output through a univariate frontier. The multivariate model is then completed by specifying a Dirichlet distribution on the output shares. Closer to the approach in this paper, Ferreira and Steel (2007) propose the use of a multivariate skewed distribution to model the composed error term in a system of m equations.. We aim to specify a multivariate model in such a way that we can make separate inference about the two forms of error. In our model each output has its own frontier and its own inefficiency and measurement error. The choice of distribution for the inefficiencies is very critical, as inference on efficiencies is typically of most interest in applications and the sample is often not very informative on this aspect (since each firm has its own inefficiency).. Thus, flexibility is a key requirement for the multivariate modelling of the inefficiencies. For this reason, we use a copula function as a flexible tool to deal with complex multivariate distributions 1 . This approach to multivariate modelling was introduced by Sklar (1959) .. The basic idea of the copula function is to separate the dependence structure from the specification of the marginal distributions. We then separately choose the marginal densities (using a general family of distributions that contains most distributions proposed in the literature) and we select different copula functions to model the dependence between the inefficiencies of the different outputs. Each copula function is characterized by a parameter that represents the dependence (which can be expressed through general association measures like Kendall τ and Spearman ρ which are invariant with respect to the margins and can take into account non-linearity in the dependence).. One problem facing this kind of multivariate models is the estimation technique; the usual frequentist perspective requires maximizing a likelihood with many parameters, which can be challenging in practice 2 . Following the suggestion of Patton (2006) , Joe and Xu (1996) and Newey and McFadden (1994) we can separately estimate the marginal distributions and the copula. Using this kind of technique, as underlined by Newey and McFadden (1994) , we have to calculate robust standard errors in order to conduct valid inference. In addition, separating the estimation leads, generally, to a loss of information, see Patton (2006) and Joe (2005) . Another problem related with the frequentist approach is the estimation of the inefficiency terms. Although it should be possible to extend the work of Jondrow et al. (1982) to the multivariate framework, it is difficult to have useful estimates of the inefficiencies if we have few observations for each unit. Instead, Bayesian methods avoid the problem of loss of information 3 and immediately lead to inference on the unobserved inefficiencies. Finally, we have the possibility to formally introduce reasonable prior information, which can be a major advantage in these highly structured models. We pay particular attention to the prior specification, and propose an improper prior structure which allows us to remain vague about aspects we usually do not have strong prior notions about, while implying "reasonable" priors on efficiencies and imposing economic regularity conditions. We also provide a mild and easily verifiable condition that ensures a proper posterior. In order to make a meaningful comparison between different copulas (through Bayes factors) we will use prior matching based on a common measure of association. Model selection is then straightforward, in principle: we use Bayes factors to find the copula that best describes the dependence in the data. Inference will be conducted through Markov chain Monte Carlo (MCMC) methods, which are developed and explained in the paper.. This work is organized as follows: Section 2 introduces the main concepts of copula functions, while Section 3 presents the multivariate stochastic frontier model proposed here, with discussion of the prior structure, existence of the posterior and brief details of the computational methods. An application to a data set of Dutch dairy farms with two outputs is analysed in Section 4. Finally, Section 5 concludes. In this paper we have proposed a new multivariate stochastic frontier model for multioutput firms. The main innovation is the fact that we introduce a copula function to model in a flexible way the multivariate distribution of the efficiencies. Our model allows for inference on individual firm efficiencies with a general class of marginal efficiency distributions and a choice of copulas to accommodate the dependence between the efficiencies of different outputs. The use of Bayesian methods naturally allows for inference on firm efficiencies and formally deals with regularity conditions and both parameter and model uncertainty. We pay particular attention to the elicitation of a reasonable prior, which is a critical component of stochastic frontier models, and we examine existence of the posterior under a convenient improper benchmark prior. The MCMC algorithm, using partial centring, is found to work well in practically relevant situations. Our framework represents a starting point from where to build more complex models. An extension to allow for dif-ferent explanatory variables in the frontiers for different outputs is straightforward. Using our framework for problems with more than two outputs (m > 2) requires the specification of copulas in more than two dimensions, which are not so readily available. A currently promising direction to explore is the use of the pair-copula decomposition of Aas et al. (2009) , which can construct complex multivariate distributions in higher dimensions. 8 Further natural extensions could be to add explanatory variables to model the inefficiencies or to introduce a more "structural" model, like the decomposition model proposed by Griffin and Steel (2008) , that can take into account multimodality in the marginal distribution of the inefficiencies. Finally, it would be interesting to investigate extensions to deal with situations where some of the outputs are "bads" (such as pollution), as was examined in Fernández et al. (2002) in the context of the same data set.. From Theorem 18.1.6 in Harville (1997) 9 we have that. is an idempotent symmetric matrix so it has eigenvalues equal to 1 or 0 and from its symmetry it admits the decompositionM L = P ΛP ′ , where P is the matrix of eigenvectors. Λ is the diagonal matrix of the eigenvalues. Since the matrix is idempotent the trace is equal to the rank. From Cor. 10.2.3 in Harville (1997) :. with T N − r(L) eigenvalues equal to one and the rest zeros, it selects a sub-matrix of W W ′ corresponding to T N −r(L) observations. If we denote by W i the m×1 vector corresponding to observation i and (without loss of generality) we reorder the observations so that the first T N − r(L) correspond to the unit eigenvalues, we can write . For the inner integral, we use the fact that the integrand is proportional to a matricvariate Student-t distribution on B provided T N > k + m − 1 (see Bauwens et al. 1999, p. 307) , whereas for the outside integral we use the bound. So the posterior exists for any proper h(Z).. In our proof we have used the generalized inverse for the matrices L ′ L and D ′ M X D because in some cases, depending on the choice of D, the matrix (X : D) can have deficient column rank 10 , despite the fact that X has rank k. . where. Since L ′ L is a positive definite matrix of rank T N , it has T N non zero eigenvalues, which will all be positive; without loss of generality, we choose λ T N +1 = . . . = λ k+M = 0.. Assume that B 0 and Z 0 are such that there exists a neighbourhood of ∆ 0 , say D, in which p(∆) K > 0 for some constant K. Note that this defines a set of positive Lebesgue measure for Y in ℜ T N ×m . Then, a lower bound for (20) is proportional to:. for Γ = (γ 1 , . . . , γ k+M ) ′ = Q(∆ − ∆ 0 ) and G = {Γ : ∆ ∈ D}. We have to remember that the matrix Λ selects only the first T N components for each equation. Now the matrix T N i=1 γ i γ ′ i can be written asΓ ′Γ whereΓ ′ is a m × T N matrix with m ≤ T N . Now we can writeΓ ′ as T H 1 where T is a m × m lower triangular matrix while H 1 is a semiorthogonal matrix of dimension m × T N . Thus, R = T T ′ =Γ ′Γ . Now, using Theorem 1.4.10 in Gupta and Nagar (2000) our lower bound is proportional to. whereG = {Γ : ∆ ∈ D}, R = {Γ ′Γ :Γ ∈G} and Γ p (·) is the multivariate gamma function 10 In our case, with constant efficiencies over time, the rank of (X : D) is k + N − 1.. as defined in Gupta and Nagar (2000, p. 18) . The nonintegrability of the integral (23) can be proven as follow: since R = T T ′ we have |R| = m i=1 t 2 ii and the Jacobian of the transformation is:

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Unexpectedness as a Measure of Interestingness in Knowledge Discovery Organizations are taking advantage of "data-mining" techniques to leverage the vast amounts ofdata captured as they process routine transactions. Data-mining is the process of discoveringhidden structure or patterns in data. However several of the pattern discovery methods in dataminingsystems have the drawbacks that they discover too many obvious or irrelevant patternsand that they do not leverage to a full extent valuable prior domain knowledge that managershave. This research addresses these drawbacks by developing ways to generate interestingpatterns by incorporating managers' prior knowledge in the process of searching for patterns indata. Specifically we focus on providing methods that generate unexpected patterns with respectto managerial intuition by eliciting managers' beliefs about the domain and using these beliefs toseed the search for unexpected patterns in data. Our approach should lead to the development ofdecision support systems that provide managers with more relevant patterns from data and aid ineffective decision making. In this paper we proposed a new definition of unexpectedness of a rule with respect to a belief and presented an algorithm that finds unexpected association rules from data using this measure.. We tested our methods on consumer purchase data from a market research firm and found some unexpected patterns with respect to our belief set. We also compared our approach with a standard association rule generating algorithm (Apriori) across the two dimensions:. interestingness of rules and number of rules generated. We conclude that our method discovers, generally, fewer rules and avoids discovering many obvious or irrelevant rules as Apriori does.. This means that our approach provides more focused and, therefore, more efficient search for interesting rules than Apriori.. In future work we plan to extend our algorithm to discover unexpected patterns of a more general nature than association rules. We also plan to apply our method in the context of knowledge refinement based on the discovery of unexpected patterns. More generally, the contribution of this research will be procedures for making data-mining more intelligent and useful for the decision maker. Our approach should lead to the development of decision support systems that provide more relevant patterns in the data to the user, patterns that confirm or challenge the user's beliefs and domain knowledge. We believe that such a breakthrough is required if datamining is to achieve its potential in business applications. We tested our algorithm on consumer purchase data from a major market research firm. We preprocessed this data by combining different data sets, made available to us by this firm, into one table containing 36 different attributes. These attributes pertain to the item purchased by a shopper at a store, together with certain characteristics of the store and the demographic data about the shopper and his or her family 9 . Some demographic attributes include age and sex of the shopper, occupation, income and marital status of the household head and the presence of children in the family and the size of the household. Some transaction-specific attributes include type of item purchased, coupon usage (whether the shopper used any coupons to get a lower price or not), the availability of store coupons or manufacturer's coupons and presence of advertisements for the product purchased in the store.. While we generated this combined data set, we also restricted the purchasing records only to the class of carbonated beverages, i.e., each record in this data set refers to a purchase of some carbonated beverage by a shopper. The resulting dataset had 87437 records, each consisting of 36 discrete fields. The levels of discrete attributes range from 2 to 12 distinct values.

B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Complexity of unique list colorability Given a list L(v) for each vertex v, we say that the graph G is L-colorable if there is a proper vertex coloring of G where each vertex v takes its color from L(v). The graph is uniquelyk-list colorable if there is a list assignment L such that |L(v)|=k for every vertex v and the graph has exactly one L-coloring with these lists. Mahdian and Mahmoodian [M. Mahdian, E.S. Mahmoodian, A characterization of uniquely 2-list colorable graphs, Ars Combin. 51 (1999) 295-305] gave a polynomial-time characterization of uniquely 2-list colorable graphs. Answering an open question from [M. Ghebleh, E.S. Mahmoodian, On uniquely list colorable graphs, Ars Combin. 59 (2001) 307-318; M. Mahdian, E.S. Mahmoodian, A characterization of uniquely 2-list colorable graphs, Ars Combin. 51 (1999) 295-305], we show that uniquely 3-list colorable graphs are unlikely to have such a nice characterization, since recognizing these graphs is @S"2^p-complete. List colorings were introduced in [ERT80, Viz76] as a generalization of ordinary vertex coloring. Given a list assignment L that assigns to each vertex v a set of colors, we say that G is L-colorable (or list colorable with lists L) if there is a proper vertex coloring of G where each vertex receives a color from its set L(v) of available colors. Obviously, if every set L(v) is {1, 2, . . . , k}, then L-colorability is the same as k-colorability.. A k-list assignment is a list assignment in which the list of each vertex has size k. A graph G is k-list colorable (or k-choosable) if it is L-colorable for every k-list assignment L. Rubin [ERT80] gave a polynomial-time characterization of 2-list colorable graphs, while in [Gut96] it is shown that recognizing 3-list colorable graphs is Π The concept of uniquely list colorable graphs was introduced independently in [DM95] and [MM99] . A graph is uniquely k-list colorable if there is a klist assignment L such that G has exactly one L-coloring. Figure 1 shows a uniquely 3-list colorable graph (taken from [EGH02] ) with a uniquely colorable 3-list assignment. Trivially, every graph is uniquely 1-list colorable. Mahdian and Mahmoodian [MM99] characterized uniquely 2-list colorable graphs:. Theorem 1 (Mahdian and Mahmoodian [MM99] ) A graph is uniquely 2-list colorable if and only if it contains a biconnected component which is neither a cycle, a complete graph, nor a complete bipartite graph.. Theorem 1 implies that uniquely 2-list colorable graphs can be recognized in polynomial time. In [MM99] it is asked as an open question to characterize uniquely k-list colorable graphs for k ≥ 3. More specifically, Ghebleh and Mahmoodian [GM01] ask what is the complexity of deciding whether a graph is uniquely 3-list colorable. The main contribution of this paper is showing that recognizing uniquely k-list colorable graphs is Σ p 2 -complete for every k ≥ 3. Essentially, this means that the problem is as hard as possible, and we cannot hope for an NP-or coNP-characterization of these graphs.. The paper is organized as follows. Section 2 introduces notation and some preliminary results. In Section 3, we introduce a special Σ p 2 -complete satisfiability problem that will be used to obtain our hardness result. The reduction uses a number of somewhat complicated gadgets, these gadgets are described in Sections 4-6. The reduction itself is presented in Section 7.

B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


A Syntax-based approach to measuring the degree of inconsistency for belief bases Measuring inconsistency Minimal inconsistent subsets Degree of inconsistency Measuring the degree of inconsistency of a belief base is an important issue in many realworld applications. It has been increasingly recognized that deriving syntax sensitive inconsistency measures for a belief base from its minimal inconsistent subsets is a natural way forward. Most of the current proposals along this line do not take the impact of the size of each minimal inconsistent subset into account. However, as illustrated by the well-known Lottery Paradox, as the size of a minimal inconsistent subset increases, the degree of its inconsistency decreases. Another lack in current studies in this area is about the role of free formulas of a belief base in measuring the degree of inconsistency. This has not yet been characterized well. Adding free formulas to a belief base can enlarge the set of consistent subsets of that base. However, consistent subsets of a belief base also have an impact on the syntax sensitive normalized measures of the degree of inconsistency, the reason for this is that each consistent subset can be considered as a distinctive plausible perspective reflected by that belief base, whilst each minimal inconsistent subset projects a distinctive view of the inconsistency. To address these two issues, we propose a normalized framework for measuring the degree of inconsistency of a belief base which unifies the impact of both consistent subsets and minimal inconsistent subsets. We also show that this normalized framework satisfies all the properties deemed necessary by common consent to characterize an intuitively satisfactory measure of the degree of inconsistency for belief bases. Finally, we use a simple but explanatory example in requirements engineering to illustrate the application of the normalized framework. Interest in measuring inconsistency for belief bases has grown rapidly in recent years. This interest is driven by the desire to understand how measures of inconsistency can be formalized (e.g., [1] [2] [3] ), and by the practical needs of realworld applications (e.g., [4] [5] [6] ). A number of proposals for measuring the degree of inconsistency of a belief base have been presented, including the maximal η-consistency [7, 8] , n-consistency and n-probability [9] , measures based on variables or paraconsistent models [1, [10] [11] [12] [13] , measures based on minimal inconsistent subsets [14, 3] , and the Shapley Inconsistency Value [2] .. In particular, the measures based on minimal inconsistent subsets are attractive for syntax sensitive conflict resolution in some applications such as requirements engineering [3] . As a matter of fact, for conflict resolution, the minimal inconsistent subsets of a belief base can be considered as the purest expression of inconsistency, since an agent needs to remove only one formula from each minimal inconsistent subset to restore consistency [15] . Following this viewpoint, several approaches to measuring inconsistency based on minimal inconsistent subsets have been proposed. The scoring function [14] and the MI measure [3] are the most representative methods of such approaches to measuring the degree of inconsistency. The scoring function assigns each subset of a belief base the number of minimal inconsistent subsets of the base that would be eliminated if the subset was removed from that belief base [14] . The degree, or relative amount of inconsistency of a belief base is captured by scoring function values of all the non-empty subsets of that belief base together. In contrast, the MI measure views the total number of minimal inconsistent subsets of a belief base as a measure of the amount of inconsistency of that belief base [3] . The common idea of the scoring function and the MI measure is to focus on counting the number of minimal inconsistent subsets of a belief base.. Conspicuously, these two methods do not make any explicit distinction between two minimal inconsistent subsets with different sizes. Moreover, the MinInc property presented in [3] states that each minimal inconsistent subset brings the same amount of inconsistency. However, as the cardinality of a minimal inconsistent subset increases, its inconsistency becomes more tolerable [7, 3] , i.e., the bigger the size of a minimal inconsistent subset, the smaller the degree of its inconsistency. To illustrate this in an informal way, let us consider the well-known lottery paradox which motivated Knight proposing his approach [7] . The lottery paradox presented in [16] considered an n-ticket lottery scenario, known to be fair and to have exactly one winner. It is rational to accept that for any individual ticket i, ticket i will not win, since the probability of ticket i being the winner cannot exceed a given, high enough threshold due to the fairness of the lottery. Then K(n) = {¬w 1 , . . . , ¬w n , w 1 ∨ · · · ∨ w n } is a minimal inconsistent belief base (we will define this concept carefully later) for the lottery, where for each i, w i asserts that ticket i is the winner. Intuitively, if there are a sufficiently large number of tickets in the lottery, say 1 million, the belief base K(n) is almost consistent(in our case only 1 in 1 million is wrong), whereas K(n) is highly inconsistent if there are, say, only three tickets. Therefore, considering only the total number of minimal inconsistent subsets is not enough for precisely capturing the degree of inconsistency of a belief base. The size of each minimal inconsistent subset should be considered.. In addition to this, the role of free formulas of a belief base (i.e., formulas not belonging to any minimal inconsistent subset of a belief base) in measuring the degree of inconsistency of that belief base has not yet been characterized well. According to the viewpoint that minimal inconsistent subsets are the purest form of inconsistency, free formulas of a belief base have nothing to do with the conflicts of that belief base because free formulas do not belong to any minimal inconsistent of that belief base. To uphold this viewpoint, the Free Formula Independence property presented in [2, 3] requires that adding a free formula to a belief base should not change the inconsistency measure of that base.. However, the Free Formula Independence property does not capture the impact of free formulas on the degree of inconsistency of a belief base well for syntax-based measures as well as for model-based or variable-based measures. Roughly speaking, model-based or variable-based inconsistency measures are often based on some paraconsistent models, which allow us to assign an inconsistent truth value (i.e., both true and false) to variables involved in inconsistency. Then the degree of inconsistency of a belief base is often captured by the normalized minimum number of inconsistent truth values in such a model. A free formula not belonging to any minimal inconsistent subset cannot guarantee that it is free from confliction with any other formulas. To illustrate this, consider K = M ∪ {¬b}, where ¬b is a free formula of K and M = {a ∧ ¬a ∧ b} is the minimal inconsistent subset of K. Clearly, M conveys contradictory information about a alone, but K conveys contradictory information about b as well as that about a. So variable b must have been involved in inconsistent values in such paraconsistent models. In such cases, adding a free formula to a belief base increases the degree of inconsistency of that belief base. For example, as pointed out in [17] , the I LP m inconsistency measure presented in [2] allows some free formulas of a belief base to increase the existing conflicts in minimal inconsistent subsets of that belief base. On the other hand, for a given belief base, if we add a formula consisting of new variables that do not appear in formulas of that belief base, then the minimum number of inconsistent truth values will not change since the enlarged belief base cannot convey contradictory information about these new variables. But the degree of inconsistency of that base will be diluted by normalization since the new variables enlarge the set of all variables. For example, consider I LP m again, I LP m ({a ∧ ¬a ∧ ¬b}) = 1 2 . After adding. . In summary, in the context of model-based or variable-based inconsistency measures, adding free formulas can either strengthen or weaken the degree of inconsistency. The impact of free formulas of a belief base on the degree of that belief base depends on literals of free formulas. Therefore, model-based or variable-based inconsistency measures may not provide support for the Free Formula Independence property.. Even if we consider syntax-based or syntax sensitive proposals for measuring the degree of inconsistency, the Free Formula Independence property presents problems. It is again not an undisputed property to characterize measures for the degree of inconsistency of belief bases. Intuitively, in syntax sensitive application domains, as the number of free formulas of a belief base increases the degree of its inconsistency will be diluted if its set of minimal inconsistent subsets remains the same. This discussion, nevertheless, is still in harmony with the viewpoint that minimal inconsistent subsets are the purest form of inconsistency. This can be explained by pointing out that each minimal inconsistent subset projects a distinctive viewpoint of the inconsistency, whilst each consistent subset projects a different but completely consistent viewpoint of the same base. It is again intuitive that how inconsistent a belief base is should depend on how much distinctive inconsistent viewpoints account for all the distinctive (inconsistent or plausible) viewpoints reflected by that belief base. Therefore, the degree of inconsistency of a belief base should be influenced by both the consistent subsets as well as the minimal inconsistent subsets. In this sense, adding a free formula will certainly enlarge the set of consistent subsets. Correspondingly, adding or removing free formulas from a belief base may decrease or increase the degree of inconsistency of that belief base. This intuition is also partially supported by most of the inconsistency measures based on variables. To illustrate, let us consider the inconsistency measure I LP m presented in [2] , which is the normalized minimum number of inconsistent truth values in the LP m models [18] . For any belief base K, I LP m (K ∪ {α}) < I LP m (K) if α is a free formula of K ∪ {α} and α does contain only variables that do not appear in formulas of K.. To address these issues, in this paper, we first present a framework for measuring the degree of inconsistency for belief bases, integrating the impact of both consistent and minimal inconsistent subsets. We then discuss the intuitive properties that such measures should satisfy. Our study shows that a measure instantiated by our framework gives more intuitive measurements for the degree of inconsistency for belief bases than other measures.. The rest of this paper is organized as follows: In the next section, we give some necessary notations about minimal inconsistent subsets of a belief base. In Section 3 we briefly recall some closely related work. In Section 4, we define a framework for measuring the degree of inconsistency for a belief base based on both minimal inconsistent subsets and consistent subsets. In Section 5, we propose a set of properties to characterize measures of the degree of inconsistency for belief bases. Then we give some instantiated measures in Section 6. In Section 7, we present a case study to illustrate the application of our approach in the domain of requirements engineering. In Section 8, we compare our measures with some closely related work. Finally, we conclude this paper in Section 9. Viewing minimal inconsistent subsets as the purest form of inconsistency, it is natural to derive syntax sensitive inconsistency measures for a belief base from the minimal inconsistent subsets of that belief base.. We have presented a framework for measuring the degree of inconsistency of a belief base by using its minimal inconsistent subsets along with its consistent subsets. It allows us to consider the impact of the size of each minimal inconsistent subset on the degree of inconsistency as well as to well characterize the role of free formulas in measuring the degree of inconsistency.. We have argued that the number of minimal inconsistent subsets of a belief base is insufficient to capture the degree of inconsistency of that belief base. Moreover, the Free Formula Independence property does not well characterize the role of free formulas in measuring the degree of inconsistency from a syntax sensitive perspective. Then Free Formula Dilution property instead of Free Formula Independence was proposed to characterize the change of the degree of inconsistency of a belief base due to adding or removing free formulas from that belief base. Furthermore, Contradiction, Attenuation, and Almost Consistency were proposed as the essential properties to characterize the degree of inconsistency of each minimal inconsistent subset. Motivated by these intuitive properties, we presented a general framework to define normalized measures for the degree of inconsistency of a belief base from its minimal inconsistent subsets along with its consistent subsets. We also have shown the measure defined from this general framework satisfies all the properties presented in this paper to characterize an intuitive measure of the degree of inconsistency for belief bases. Finally, we presented two instantiated measures based on two different but intuitive rudimentary sequences of the degree of inconsistency. To illustrate the practical potential usage of our measures, we presented a software engineering application. In this section, we give a brief introduction to current inconsistency measures illustrating the roles of minimal inconsistent subsets and free formulas, as well as properties to characterize inconsistency measures, respectively.

O O O O O O B I I O O O B I B I I B I I O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O B I I O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O B I I O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O B I I O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O B I I O O O O O O O B I I O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O


Observing the evolution of internet as topology Characterizing the evolution of Internet topology is important to our understanding of the Internet architecture and its interplay with technical, economic and social forces. A major challenge in obtaining empirical data on topology evolution is to identify real topology changes from the observed topology changes, since the latter can be due to either topology changes or transient routing dynamics. In this paper, we formulate the topology liveness problem and propose a solution based on the analysis of BGP data. We find that the impact of transient routing dynamics on topology observation decreases exponentially over time, and that the real topology dynamics consist of a constant-rate birth process and a constant-rate death process. Our model enables us to infer real topology changes from observation data with a given confidence level. We demonstrate the usefulness of the model by applying it to three applications: providing more accurate views of the topology, evaluating theoretical evolution models, and empirically characterizing the trends of topology evolution. We find that customer networks and provider networks have distinct evolution trends, which can provide an important input to the design of future Internet routing architecture. The Internet has been evolving rapidly over time like a living organism, so has its topology. Characterizing the dynamics and evolution trends of the Internet topology is an important research topic for several reasons. It provides an essential input to the understanding of limitations of existing routing protocols, the evaluations of new designs, as well as the projection of future needs; and it will help advance our understanding of the interplay between networking technology, the resulting topology, and the economic forces behind them.. In recent years there have been a plethora of efforts in understanding the Internet AS topology and its evolution. On the empirical side, most work has focused on examining graph properties of topology snapshots (e.g., node degree distribution [17] , likelihood [22] , and degree correlation [24] ). On the theoretical side, a number of topology evolution models (e.g., [7, 12, 13, 31] ) have been proposed for the addition and removal processes of nodes and links in the Internet AS topology. However there is a missing link between these two categories of work: an empirical description of the topology evolution, that is when and where AS nodes and inter-AS links are added or removed over time in reality. Without this knowledge, it is difficult to judge the accuracy of topology snapshots taken at different times in the graph property computation, or to validate theoretical evolution models. For example, up to now, theoretical evolution models have been evaluated by comparing certain graph properties of the topology derived from the evolution model with that of the AS topology snapshots taken over time. As shown in [22] , however, significantly different topologies can share the same graph properties, such as power-law distribution of node degrees. A more accurate evaluation approach would be to compare the modeling results against empirical topology evolution data directly.. Obtaining empirical topology evolution data is a challenging task. Besides the well known issue that the observed AS topology snapshots only capture a subset of the real Internet topology [10, 33, 25, 28, 32] , a new problem arises when we try to measure topology changes over time: the changes in the observed topology do not necessarily reflect the changes in the real topology and vice versa. Because the observed topology is normally inferred from routing or data paths, its changes can be due to either real topology changes or transient routing dynamics (e.g., caused by link failures or router crashes). Therefore the challenge is, given all the changes in the observed topology over time, how to differentiate those caused by real topology changes from those caused by transient routing dynamics, which we call the liveness problem. Only after solving this problem can we provide empirical topology evolution data such as when and where an AS or an inter-AS link is added or removed from the Internet.. In this paper we developed a solution to the liveness problem based on the analysis of available data. Our analysis shows that the effect of transient routing dynamics on the observed topology decreases exponentially over time, and the real topology changes can be modeled as the combination of a constant-rate birth process and a constant-rate death process. Fitting the model to data, we obtain the values of several key parameters that enable us to identify real topology changes from observed changes with a given confidence level, and to estimate the birth and death rates of links and nodes in the Internet AS topology for the first time.. We show the usefulness of our empirical topology evolution model through three applications. First, it allows us to obtain a more accurate topology by combining snapshots taken at different times and discounting dead links and nodes. Second, it can be used to evaluate theoretical topology evolution models. Third, it reveals evolution trends of the Internet. More specifically, we find that the edge of the Internet and the core have distinct growth patterns. The Internet's growth in size is mainly attributed to customer networks at the edge of the AS topology, where the majority of node births and deaths occur. Provider networks, which occupy the core in the topology, grow slowly in number but adjust their inter-connectivity frequently, and most adjustments involve peer-to-peer AS links.. We would like to be the first to point out an intrinsic limitation of our results. The AS topology we collected and studied is inevitably incomplete, and the numerical results based on this partial topology could be potentially off from that of the real Internet topology. However this inherent limitation is shared by all empirical studies on the Internet topology and it should not stop us from solving the liveness problem. We also make our best efforts to mitigate the problem. We use the data collected from hundreds of publicly available BGP monitors, and we observed diminishing returns by additional monitors once the number reaches a certain level. We develop our model using different numbers and combinations of the monitors to assure that the model has no bias towards any particular set of monitors. Furthermore, we compare the results based on BGP data with that based on data from traceroute and Internet registries. Section 3.5 provides further discussions on the model's limitations.. The rest of the paper is organized as follows. Section 2 defines the liveness problem in studying Internet topology. Section 3 develops and verifies our topology evolution model using three different data sources: BGP logs, traceroute, and Internet registries. It also provides an evaluation of the AS topology data collected from these sources regarding their suitability for solving the liveness problem. Section 4 presents three applications of our model. Section 5 discusses the related work, and Section 6 concludes the paper. In this paper we formulated the topology liveness problem and contrasted it with the completeness problem that has been studied intensively by the previous efforts. We developed an empirical model that characterizes the changes in the observed AS topology by three processes: birth, death, and revelation. Based on the model, we are able to distinguish real topology changes from transient routing changes with a given confidence level. The success of the model calls for further investigations on why the evolution of the Internet, a vast distributed and autonomous system, can be approximated well by such a seemingly over-simplified model.. Besides providing better views of the topology and helping validate topology evolution models, our results of the topology evolution trends can be an important input to the design of future Internet routing architecture. We show that the main contributor to the growth of the Internet size is the fast increasing number of customer networks, which puts a heavy burden on the global routing system's scalability. Our results also show the highly active connectivity adjustment among transit networks and the rapid growth in density among them. A future global routing system may take into account the distinct growth patterns of these two types of networks in order to achieve long-term scalability. The model developed in this section captures the main characteristics of observed AS topology changes by three dynamic processes: birth, death, and revelation. This model can help us obtain key parameters of different dynamic processes and separate real topology changes from transient routing changes with a given confidence. At the same time, we must also understand the limitations of this model.. The model is mainly descriptive and not derived from first principles. It matches the data well and is useful in studying topology dynamics. However, it is possible to have other models that also fit the data well, e.g., node birth may be modeled by an exponential function with a small exponent [20] . Our model does not provide an explanation for why birth and death rates are constant, or why the revelation probability are uniform. Answering these questions is likely to require looking deep into economic, technological, and operational factors behind the Internet evolution, and our model can serve as an important input to such a study.. We may also articulate the reasons that made the model work well. One sound reason relates to the model's macroscopic granularity and the large scale of the Internet. Individual factors influencing the AS topology evolution are probably not constant or uniformly distributed, however, given the large scale of the Internet, and the large number of (perhaps independent) factors in action, fluctuations caused by individual factors may even out when we measure macroscopic properties using data aggregated from many different views. Had we restricted our study to a small geographic area, or to a subgraph of the Internet (e.g., an academic network), we might have obtained very different results. The model also may not apply to very different time scales, e.g., looking at link/node appearances at hourly basis (or over decades) might show a different pattern. We develop the model based on data from recent three years, and we are yet to see whether the model will hold in the future. Our model also makes some simplified assumptions. For instance, in reality different links may have different revelation probabilities (λ and µ), depending on their connectivity and routing polices.. The apparent success of the model should not be taken without caution; rather, it is a call for additional verifications. Besides BGP data, we use registry and traceroute data to show evidence that the observed data is indeed explained by the model. However, it remains a research challenge regarding how to perform a thorough validation, given the lack of public information about ISP connectivity changes. One specific validation step we plan to carry out in the near future is to contact a number of ISPs to verify the accuracy of the link birth/death timestamps inferred by the model. One of the early work in AS topology measurement studies is [19] , in which topology snapshots are constructed by using BGP routing tables and updates within 21 days. Since then many topology characterization results have been obtained through using topology snapshots from either BGP routing tables or traceroute data (e.g., [17, 24] ). However, as pointed by [10] and [28] , the existing snapshot approach may miss a significant number of links, especially the peer-to-peer links between ASes at lower tiers. To collect a more complete AS topology, we can either gather topology information from additional sources (e.g., IRR, looking glasses, exchange points) [33, 25, 32, 10] , or place additional monitors at different places [29] . Different from the above mentioned efforts which focus on the completeness problem, our work reported in this paper focuses on the topology liveness problem. However our results can be used to help capture more complete topologies with a given confidence on link/node liveness.. There are a number of topology evolution models. For example, the work reported in [7, 9, 34 ] models the addition of nodes and links so that the resulting topology can have certain desired graph properties, such as degree distribution, clustering coefficient, richclub connectivity and betweenness centrality. Chang et al. [12] extend the "heuristically optimized trade-off" (HOT) model [16] to allow each AS to have multiple geographical locations; more recently in [13] the authors also identify a set of criteria used in the the decision process when new peering relationships are established. Wang et al. [31] propose an evolution model based on random walks and wealth evolution.. Our empirical model differs from the previous topology evolution models in that we model how fast the topology changes as measured by the birth and death rates of nodes and links, while previous work models where the changes happen. We consider the effects of transient routing dynamics, which has not been addressed by any previous effort. Using our empirical model, we can obtain the dates of individual node or link changes, which can then be used to evaluate theoretical models. Our work is complementary to previous efforts and can help advance realistic modeling of the Internet topology and its evolution.. Chen et al. [14] evaluate the BA model using BGP data. They ignore the first ten days and the last ten days of data to reduce the impact of the topology revelation, instead of addressing the liveness problem directly. The choice of ten days is mainly based on intuitive judgement. Our model provides a means to choose the threshold with a target confidence level. Huston [20] studies the consumption of AS numbers and shows that AS allocation over time can be fitted by either a linear function or an exponential function with a small exponent. This is similar to our linear modeling of node birth. Our model is more complete in that, in addition to birth, it also includes the death process and revelation process, and can be applied to links' birth, death, and revelation as well.

O O B O B O O O O B O B I O O O O O O O B B O O O O O O O B O O O B O O O B O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O B O O O O O O O O O O O O O B O O B O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O B O B O O O O O O O O O B O O O O O O B O O O O B O O O O O B O O O B O O O O O B O O O O O O O O B O O O O O O O O O O B O O O O O O B O O B O B I B B O B O O O O O O O O O O O O O O O O O O B O O O B I O O O O O O O O O O O O B O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B O O O O O O O O O O O O O O O B B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B O O O O B B O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B B O O O O O O O O O O O O O O O O O B O O B O O O O O O O O O O O B O O B O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B O O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B B O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B O O O B O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O B O O O B O O O O O O O O O O O O O O B O O O O O O O O O O O O B I O O O O O O O B B O O O B O O O O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O B O O O O O O O O O O O B O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O B O O O O O O O B O O B O O O O O O O O O O O O O O O B O O O B O O O O O O O B O O O O O O B O O O O O B O O B O B I B B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O B O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B O O B O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B O O O O O O O O O O O O O O O O B O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O B O B O O O B O O O O O O B O O O O O B O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B B O O O O O O B O O B O O O O O B O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O B O O O O O O O B O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B O O B O O O O O O O O O O O O