Intrinsic Statistics on Riemannian Manifolds: Basic Tools for Geometric Measurements In medical image analysis and high level computer vision, there is an intensive use of geometric features like orientations, lines, and geometric transformations ranging from simple ones (orientations, lines, rigid body or affine transformations, etc.) to very complex ones like curves, surfaces, or general diffeomorphic transformations. The measurement of such geometric primitives is generally noisy in real applications and we need to use statistics either to reduce the uncertainty (estimation), to compare observations, or to test hypotheses. Unfortunately, even simple geometric primitives often belong to manifolds that are not vector spaces. In previous works [1, 2], we investigated invariance requirements to build some statistical tools on transformation groups and homogeneous manifolds that avoids paradoxes. In this paper, we consider finite dimensional manifolds with a Riemannian metric as the basic structure. Based on this metric, we develop the notions of mean value and covariance matrix of a random element, normal law, Mahalanobis distance and χ 2 law. We provide a new proof of the characterization of Riemannian centers of mass and an original gradient descent algorithm to efficiently compute them. The notion of Normal law we propose is based on the maximization of the entropy knowing the mean and covariance of the distribution. The resulting family of pdfs spans the whole range from uniform (on compact manifolds) to the point mass distribution. Moreover, we were able to provide tractable approximations (with their limits) for small variances which show that we can effectively implement and work with these definitions. To represent the results of a random experiment, one theoretically consider a probability measure on the space of all events. Although this probabilized space contains all the information about the random experiment, one often have access only to some measurements depending of the outcome of the experiment. The mathematical way to formalize this is to investigate random variables or observables which are maps from the probabilized space into R. One usually further simplify by restricting to random variables that have a probability density function (pdf).. However, from a computational point of view, the pdf is still too informative and we have to restrict the measurements to a few numeric characteristics of a random variable. Thus, one usually approximate a unimodal pdf by a central value and a dispersion value around it. The most used central value is the mean value or expectation of the random variable:x = E[x] = x dPr = R y p x (y) dy. The corresponding dispersion value is the variance σ . In real problems, we can have several simultaneous measurements of the same random experiment. If we arrange these n random variables x i into a vector x = (x 1 . . . x n ), we obtain a random vector. As the Pennec expectation is a linear operator, it is easily generalized to vector or matrix functions in order to define the mean value and the covariance matrix of a random vector: xx = E[(x −x)(x −x) T ] = (y −x)(y − x). T p x (y) dy. If one has to assume a probability distribution, the Gaussian distribution is usually well adapted, as it is completely determined by the mean and the covariance. It is moreover the entropy maximizing distribution knowing only these moments. Then, one can use a statistical distance between distributions such as the Mahalanobis distance and the associated statistical tests.. The problem we investigate in this article is to generalize this framework to measurements in finite dimensional Riemannian manifolds instead of measurements in a vector space. Examples of manifolds we routinely use in medical imaging applications are 3D rotations, 3D rigid transformations, frames (a 3D point and an orthonormal trihedron), semi-or non-oriented frames (where 2 (resp. 3) of the trihedron unit vectors are given up to their sign) [3, 4] , oriented or directed points [5, 6] , positive definite symmetric matrices coming from diffusion tensor imaging [8] [9] [10] [11] or from variability measurements [12] . We have already shown in [2, 13] that this is not an easy problem and that some paradoxes can arise. In particular, we cannot generalize the expectation to give a mean value since it would be an integral with value in the manifold: a new definition of the mean is needed, which implies revisiting an important part of the theory of statistics.. Statistical analysis on manifolds is a relatively new domain at the confluent of several mathematical and application domains. Its goal is to study statistically geometric object living in differential manifolds. It is linked to the theory of statistical manifolds [14, 15] , which aims at providing a Riemannian structure to the the space of parameters of statistical distribution. However, the targeted geometrical objects are usually different. Directional statistics [16] [17] [18] [19] provide a first approach to statistics on manifold. As the manifolds considered in this field are spheres and projective spaces, the tools developed were mostly extrinsic, i.e. relying on the embedding of the manifold in the ambient Euclidean space. More complex objects are obtained when we consider the "shape" of a set of k points, i.e. what remains invariant under the action of a given group of transformation (usually rigid body ones or similarities). The statistics on these shape spaces [20] [21] [22] [23] raised the need for intrinsic tools. However, the link between the tools developed in these works, the metric used and the space structure was not always very clear.. Another mathematical approach was provided by the study of stochastic processes on Lie groups. For instance, [24] derived central limit theorems on different families of groups and semi-groups with specific algebraic properties. Since then, several authors in the area of stochastic differential geometry and stochastic calculus on manifolds proposed results related to mean values [25] [26] [27] [28] [29] [30] . On the applied mathematics and computer science side, people get interested in computing and optimizing in specific manifolds, like rotations and rigid body transformations [4, [31] [32] [33] [34] , Stiefel and Grassmann manifolds [35] , etc.. Over the last years, several groups attempted to federate some of the above approaches in a general statistical framework, with different objectives in mind. For instance, [36] and [15] aimed at characterizing the performances of statistical parametric estimators, like the bias and the mean square error. Hendricks [36] considered extrinsic statistics, based on the Euclidean distance of the embedding space, while [15] considered the intrinsic Riemannian distance, and refined the Cramer-Rao lower bound using bounds on the sectional curvature of the manifold. In [37, 38] , the authors focused on the asymptotic consistency properties of the extrinsic and intrinsic means and variances for large sample sizes, and were able to propose a central limit theorem for flat manifolds. Here, in view of computer vision and medical image analysis applications, our concern is quite different: we aim at developing computational tools that can consistently deal with geometric features, or that provide at least good approximations. As we often have few measurements, we are interested in small sample sizes rather than large one, and we prefer to obtain approximations rather than bounds on the quality of the estimation. Thus, one of our special interest is to develop Taylor expansions with respect to the variance, in order to evaluate the quality of the computations with respect to the curvature of the manifold. In all cases, the chosen framework is the one of geodesically complete Riemannian manifolds, which appears to be powerful enough to support an interesting theory. To ensure a maximal consistency of the theory, we rely in this paper only on intrinsic properties of the Riemannian manifold, thus excluding methods based on the embedding of the manifold in an ambient Euclidean space.. We review in Section 2 some basic notions of differential and Riemannian geometry that will be needed afterward. This synthesis was inspired from [39, chap. 9] , [40] [41] [42] , and the reader can refer to these books to find more details. In the remaining of the paper, we consider that our Riemannian manifold is connected Intrinsic Statistics on Riemannian Manifolds 129 and geodesically complete. We first detail in Section 3 the measure induced by the Riemannian metric on the manifold, which allows to define probability density functions, in particular the uniform one. Then, we turn in Section 4 to the expectation of a random point. We provide a quite comprehensive survey of the definitions that have been proposed. Among them, we focus on the Karcher and Fréchet means, defined as the set of points minimizing locally or globally the variance (the expected Riemannian distance). We provide a new proof of the barycentric characterization theorem and an original Gauss-Newton gradient descent algorithm to practically compute the mean. Once the mean value is determined, one can easily define the covariance matrix of a random point (and possibly higher order moments) using the exponential map at the mean point (Section 4). To generalize the Gaussian distribution, we propose in Section 6 a new family of distributions based on a maximum entropy approach. Under some reasonable hypotheses, we show that is amounts to take a truncated Gaussian distribution in the exponential map at the mean point. We illustrate the properties of this pdf family on the circle, and provide computationally tractable approximations for concentrated distributions. Last but not least, we investigate in Section 7 the generalization of the Mahalanobis distance and the χ 2 law. A careful analysis shows that, with our definition of the generalized Gaussian, the χ 2 law remains independent of the variance and of the manifold curvature, up to the order 3. This demonstrate that the whole framework is computationally sound and particularly consistent. The maximum entropy approach to generalize the normal distribution to Riemannian manifolds is interesting since we obtain a whole family of densities going from the Dirac measure to the uniform distribution (or the uniform measure if the manifold is only locally compact). Unfortunately, this distribution is generally not differentiable at the cut locus, and often even not continuous. However, if the relation between the parameters and the moments of the distribution are not as simple as in the vector case (but can we expect something simpler in the general case of Riemannian manifolds?), the approximation for small covariances turns out to be rather simple. Thus, this approximate distribution can be handled quite easily for computational purposes. It is likely that similar approximations hold for wrapped Gaussian, but this remains to establish.. On a (geodesically complete) Riemannian manifold, it is easy to define probability density functions associated to random points, thanks to the availability of a metric. However, as soon as the expectation is concerned, we may only define the expectation of an observable (a real or vectorial function of the random point). Thus, the definition of a mean value for a random point is much more complex than for the vector case and it requires a distance-based variational formulation: the Riemannian center of mass basically minimize locally the variance. As the mean is now defined through a minimization procedure, its existence and uniqueness are not ensured any more (except for distributions with a sufficiently small compact support). In practice, one mean value almost always exists, and it is unique as soon as the distribution is sufficiently peaked. The properties of the mean are very similar to those of the modes (that can be defined as central values of order 0) in the vector case. We present here a new proof of the barycentric characterization theorem that is valid for distribution with any kind of support. The main difference with the vector case is that we have to ensure a null probability measure of the cut locus.. To compute the mean value, we designed an original Gauss-Newton gradient descent algorithm that essentially alternates the computation of the barycenter in the exponential chart centered at the current estimation, and a re-centering step of the chart at the newly computed barycenter.. To define higher moments of the distribution, we used the exponential chart at the mean point (which may be seen as the development of the manifold onto its tangent space at this point along the geodesics): the random point is thus represented as a random vector with null mean in a star-shaped and symmetric domain. With this representation, there is no more problem to define the covariance matrix and potentially higher order moments. Based on this covariance matrix, we defined a Mahalanobis distance between a random and a deterministic point that basically weights the distance between the deterministic point and the mean point using the inverse of the covariance matrix. Interestingly, the expected Mahalanobis distance of a random point with itself is independent of the distribution and is equal to the dimension of the manifold, as in the vector case.. Like for the mean, we choose a variational approach to generalize the Normal law: we define it as the maximum entropy distribution knowing the mean and the covariance. Neglecting the cut-locus constraints, we show that this amounts to consider a truncated Gaussian distribution on the exponential chart centered at the mean point. However, the relation between the concentration matrix (the "metric" used in the exponential of the pdf) and the covariance matrix is slightly more complex that the simple inversion of the vector case, as it has to be corrected for the curvature of the manifold.. Last but not least, using the Mahalanobis distance of a Normally distributed random point, we can generalize the χ 2 law: we were able to show that is has the same density as in the vector case up to an order 3 in σ . This opens the way to the generalization of many other statistical tests, as we may expect similarly simple approximations for sufficiently centered distributions.. In this paper, we focused on purpose on the theoretical formulation of the statistical framework in geodesically complete Riemannian spaces, eluding applications examples. The interested reader will find practical applications in computer vision to compute the mean rotation [32, 33] or for the generalization of matching algorithms to arbitrary geometric features [65] . In medical image analysis, selected applications cover the validation of the rigid registration accuracy [4, 66, 67] , shape statistics [7] and more recently tensor computing, either for processing and analyzing diffusion tensor images [8] [9] [10] [11] , or to model the brain variability [12] . One can even find applications in rock mechanics with the analysis of fracture geometry [68] .. In the theory presented here, all definitions are derived from the Riemannian metric of the manifold. A natural question is how to chose this metric? Invariance properties requirements provide a partial answer for connected Lie groups and homogeneous manifolds [1, 45] . However, an invariant metric does not always exist on an homogeneous manifolds. Likewise, left and right invariant metric and generally different in non-compact Lie groups, so that we only have a partial consistency between the geometric and statistical operations. Another way to chose the metric could be to estimate it from empirical data.. More generally, we could design other definitions of the mean value using the notion of connector introduced in [29] . This connector formalizes a relationship between the manifold and its tangent space at one point, exactly in the way we used the exponential map of the Riemannian metric. Thus, we could generalize easily the higher order moments and the other statistical operations we defined by replacing everywhere the exponential map with the connector. One important point is that these connectors can model extrinsic distances (like the Euclidean distance on unit vectors), and could lead to very efficient approximations of the mean value for sufficiently peaked distributions. For instance, the "barycenter/re-centering" algorithm we designed will most probably converge toward a first order approximation of the Riemannian mean if we use any chart that is consistent to the exponential chart at the first order (e.g. Euler's angle on re-centered 3D rotations). We believe that this research track may become one of the most productive from the practical point of view.

O B O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Semantics of Higher-Order Recursion Schemes Abstract. Higher-order recursion schemes are recursive equations defining new operations from given ones called "terminals". Every such recursion scheme is proved to have a least interpreted semantics in every Scott's model of λ-calculus in which the terminals are interpreted as continuous operations. For the uninterpreted semantics based on infinite λ-terms we follow the idea of Fiore, Plotkin and Turi and work in the category of sets in context, which are presheaves on the category of finite sets. Fiore et al showed how to capture the type of variable binding in λ-calculus by an endofunctor H λ and they explained simultaneous substitution of λ-terms by proving that the presheaf of λ-terms is an initial H λ -monoid. Here we work with the presheaf of rational infinite λ-terms and prove that this is an initial iterative H λ -monoid. We conclude that every guarded higher-order recursion scheme has a unique uninterpreted solution in this monoid. The present paper is a contribution to the study of the semantics of recursive definitions using category-theoretic tools and methods. Our goal is to present a category-theoretic semantics of higher-order recursion schemes in the sense of W. Damm [8] . To reach this goal we apply the theory of rational monads on a category K , developed in our previous work [2] in order to formalize iteration in algebra, to the category K = Set F (F = finite sets and functions) of sets in context. We use the approach to λ-calculus based on H-monoids in the category of sets in context due to M. Fiore, G. Plotkin and D. Turi [10] . Our main result is a description of the initial iterative H-monoid as the monoid of rational λ-terms, and the fact that in this monoid every higher-order recursion scheme has a unique uninterpreted solution.. We now explain the motivation of our paper in more detail. In the higher-order semantics we assume a given collection Σ of existing programs of given types (that is, a manysorted signature of "terminals"). One recursively defines new typed programs p 1 , . . . , p n The above example is untyped, and indeed we are only treating the untyped case in the present paper since its uninterpreted semantics is technically simpler than the typed case; however, the basic ideas of uninterpreted semantics are similar. In contrast, the interpreted semantics (based on a specified model of λ-calculus with "terminal" symbols interpreted as operations) is more subtle in the untyped case.. Our main result is that every guarded higher-order recursion scheme has a unique uninterpreted solution, and a least interpreted one. This demonstrates that the methods for iteration in locally finitely presentable categories developed in [2] can serve not only for first-order iteration, when applied to endofunctors of Set, but also for higher-order iteration: it is possible to apply these methods to other categories, here the category of sets in context. Related Work. This is an extended and revised version of the conference paper [4] . In addition to the material in that extended abstract we include here the theory of iterative monoids in a monoidal category, see Section 4 below, and we provide detailed proofs. We proved that guarded higher-order recursion schemes have a unique uninterpreted solution, i.e., a solution as a rational λ-Σ-term. And they also have the least interpreted solution for interpretations based on Scott's models of λ-calculus as CPO's with continuous operations for all "terminal" symbols of the recursion scheme.. Following M. Fiore et al [10] we worked in the category Set F of sets in context, that is, covariant presheaves on the category F of finite sets and functions. A presheaf is a set dependent on a context (a finite set of variables). For every signature Σ of "terminal" operation symbols it was proved in [10] that the presheaf F λ,Σ of all finite λ-Σ-terms is the initial H λ,Σ -monoid. This means that F λ,Σ has (i) the λ-operations (of abstraction and application) together with the operations given by Σ rendering an H λ,Σ -algebra, (ii) the operation expressing simultaneous substitution rendering a monoid in the category of presheaves, and (iii) these two structures are canonically related. And F λ,Σ is the initial presheaf with such structure. R. Matthes and T. Uutalu [16] showed that the presheaf T λ,Σ of finite and infinite λ-Σ-terms is also an H λ,Σ -monoid. Here we proved that this is the initial completely iterative H λ,Σ -monoid. And its subpresheaf R λ,Σ of all rational λ-Σ-terms is the initial iterative H λ,Σ -monoid. We used that last presheaf in our uninterpreted semantics of recursion schemes.. Our approach was based on untyped λ-calculus. The ideas in the typed version are quite analogous. If S is the set of all types, then we form the full subcategory F of Set S of finite S-sorted sets and consider presheaves in (Set S ) F -the latter category is equivalent to that of finitary endofunctors of the category Set S . The definition of H λ,Σ is then completely analogous to the untyped case, and one can form the presheaves F λ,Σ (free algebra on V ), T λ,Σ (free completely iterative algebra) and R λ,Σ (free iterative algebra). Each of them is a monoid, in fact, an H λ,Σ -monoid in the sense of [10] . Moreover, every guarded higher-order recursion scheme has a unique solution in R λ,Σ . The interpreted semantics can be built up on a CPO-enriched cartesian closed category (as our model of typed λ-calculus) with additional continuous morphisms for all terminals. The details of the typed version are more involved, and we leave them for future work.. Related results on higher-order substitution can be found e.g. in [16] and [20] . In future work we will, analogously as in [18] , investigate the relation of uninterpreted and interpreted solutions.

O O B I I O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Going Around in Circles Let ε > 0 and let Ω be a disk of sufficiently large radius R in the plane, i. e., R ≥ R(ε). We first show that the set of lattice points inside Ω can be connected by a (possibly selfintersecting) spanning tour (Hamiltonian cycle) consisting of straight line edges such that the turning angle at each point on the tour is at most ε. This statement remains true for any large and evenly distributed point set (suitably defined) in a disk. This is the first result of this kind that suggests far-reaching generalizations to arbitrary regions with a smooth boundary. Our methods are constructive and lead to an efficient algorithm for computing such a tour. On the other hand, it is shown that such a result does not hold for convex regions without a smooth boundary. A spanning tour (spanning path) on n points is a directed Hamiltonian cycle (Hamiltonian path, respectively), drawn with straight line edges. In the Euclidean traveling salesman problem (TSP), given a set of points in the plane, one seeks a shortest spanning tour. Particularly in the last decade, there has been an increased interest in studying tours that optimize objective functions related to angles between consecutive edges in the tour, rather than the length. The problem has applications in motion planning, where restrictions on turning angles have to be enforced. For example, an aircraft or a boat moving at high speed, required to pass through a set of given locations, cannot make sharp turns in its motion. This and other applications to planning curvature-constrained paths for auto-vehicles and aircraft are discussed in [2, 3, 8, 11, 12] .. Consider a spanning tour (or path) on a set of n ≥ 2 points. When three consecutive points, p 1 , p 2 , and p 3 , are traversed in this order, the turning angle at p 2 , denoted by turn(p 1 , p 2 , p 3 ), is the supplement of the angle in [0, π] determined by the segments p 2 p 1 and p 2 p 3 ; observe that turn(p 1 , p 2 , p 3 ) ∈ [0, π]. If p 3 is on the left (resp. right) side of the oriented line − − → p 1 p 2 , we say that the tour (or path) makes a left (resp. right) turn at p 2 . If all of its turning angles are at least π/2, we call it an acute tour (or path). If all turning angles are at most π/2, the tour (or path) is obtuse; see Figure 1 .. Fekete and Woeginger [10] proved that every n-element point set S admits an acute spanning path. It is easy to see that in some cases, such a path cannot be completed to an acute tour: indeed, if all points are on a line and n is odd, then along any (spanning) tour, one of the turning angles must be equal to 0. Most desirable in applications, particularly in motion planning, are spanning paths or tours from the other extreme, that we study here, namely where each turning angle is small. A rough approximation is provided by paths or tours that are obtuse. However not all point sets admit obtuse tours or even obtuse paths. For instance, some point sets require turning angles at least as large as 5π/6 in any spanning path [10] : the 3 vertices of an equilateral triangle and its center make a small 4-point example. Moreover, other point sets (e. g., collinear) require the maximum turning angle possible, namely π, in any spanning cycle.. Related problems and results. Aggarwal et al. [2] studied the following variant of TSP. Given n points in the plane, compute a Hamiltonian tour of the points that minimizes the total turning angle. The total turning angle of a tour is the sum of the turning angles at each of the n points. They proved that this problem is NP-hard. On the positive side, they gave a polynomial-time algorithm with approximation ratio O(log n).. Various angle conditions imposed on geometric graphs (graphs with straight-line edges) drawn on a given set of points have been studied in [4, 5, 6, 7] . For instance, Bárány, Pór, and Valtr [7] proved that any point set admits a (possibly self-intersecting) Hamiltonian path in which each turning angle is at most 8π/9. Fekete and Woeginger [10] had conjectured earlier that this holds for turning angles at most 5π/6, and showed that no smaller value would do. Dumitrescu et al. [9] have recently shown that every set S of n points in the plane (n ≥ 4, even) admits a (possibly self-intersecting) Hamiltonian cycle consisting of n straight line edges such that the turning angle at each point on the cycle is at least π/3.. Aichholzer et al. [4] studied similar questions for planar geometric graphs. For instance, they showed that any point set in general position in the plane admits a non-intersecting Hamiltonian (spanning) path with the property that each turning angle is at least π/4. They also conjectured that this value can be replaced by π/2. Arkin et al. [6] introduced the notion of reflexivity of a point set, as the minimum number of reflex vertices in a polygonization (i. e., simple polygon) of the set. They gave estimates for the maximum reflexivity of an n-element point set. Recently, Ackerman et al. [1] made further progress on this problem. Other variants of Euclidean TSP can be found in the survey article by Mitchell [14] .. Our results. Theorem 1. For any ε > 0, there exists R(ε) = O(ε −3 ) with the following property. Let Ω be a disk of radius R ≥ R(ε) in the plane. Then the set of lattice points Z 2 ∩ Ω admits a Hamiltonian tour with each turning angle at most ε. Such a tour can be computed in O(|Z 2 ∩ Ω|) time.. This statement remains true for any large and evenly distributed point set (suitably defined) in a disk; see Theorem 3 below. This shows that the result is not just an artifact of some grid structure. The grid point example illustrates the ideas of the proof in a cleaner and simpler way, while for applications, the more general Theorem 3 concerning "arbitrary" even-distributed point sets is much more relevant. Further extensions are discussed below. Theorems 1 and 2 suggest far-reaching extensions for regions with a smooth boundary: Conjectures 1 and 2 below. On the other hand, it is obvious that the result does not hold for small (lattice) point sets. Theorem 2 shows that one cannot expect such a result either for convex regions without a smooth boundary.. We think that the statement in Theorem 1 holds for two reasons: (1) the lattice points are evenly distributed in the disk, and (2) the disk has a smooth boundary. We next make these notions precise.. We say that a region, X, bounded by a closed Jordan curve, has a smooth boundary, if there is a unique tangent to the curve at each boundary point. Obviously the disk is the simplest region with a smooth boundary, and that is why we prove Theorem 1 for disks. Theorem 2 below shows that the smooth boundary condition is necessary for such a result to hold for convex regions, irrespective of the size of the point set.. Theorem 2. Let X be a convex region without a smooth boundary in the plane. Then there exist ε 0 > 0, and λ 0 > 0, with the following property: For any λ ≥ λ 0 , and any similar region λX of X, each Hamiltonian cycle on the set Z 2 ∩ λX has a turning angle larger than ε 0 .. Consider a fixed (possibly disconnected) region X in the plane, bounded by finitely many closed smooth pairwise-disjoint (boundary) curves. For brevity, let us refer to this type of region as bounded region with a smooth boundary. Let ρ > 0, and c ≥ 1. Let S be a finite set of points in X. We say that S is (ρ, c)-evenly distributed in X if (i) the disks of radius ρ centered at the points in S are pairwise interior-disjoint, and (ii) the disks of radius cρ centered at the points in S collectively cover X. For instance, the set of lattice points Z 2 ∩ Ω in a disk Ω is (1/2, √ 2)-evenly distributed in Ω. Our result in Theorem 1 remains true for any large and evenly distributed point set in a disk.. Theorem 3. Let ρ > 0, and c ≥ 1 be two constants. For any ε > 0, there exists R(ρ, c, ε) = O(ε −3 ) with the following property. Let Ω be a disk of radius R ≥ R(ρ, c, ε) in the plane. Let S be a (ρ, c)-evenly distributed point set in Ω. Then S admits a Hamiltonian tour with each turning angle at most ε. Such a tour can be computed in O(|S|) time.. Let X be a bounded region with a smooth boundary in the plane. We believe that the statement in Theorem 3 holds for any evenly distributed point set contained in any sufficiently large similar copy λX of X, and likewise for large random point sets, uniformly selected from X: Conjecture 1. Let ρ > 0, and c ≥ 1 be two constants, and X be a bounded region with a smooth boundary in the plane. Let S be a (ρ, c)-evenly distributed point set in a similar copy λX of X. Then, for any ε > 0, the point set S admits a Hamiltonian tour with each turning angle at most ε, provided that λ is sufficiently large.. Conjecture 2. Let X be a bounded region with a smooth boundary in the plane, and let S be a set of n points, randomly and uniformly selected from X. Then, for any ε > 0, the point set S almost surely admits a Hamiltonian tour with each turning angle at most ε, as n tends to infinity.. From the proof of Theorem 2 we obtain the following. Corollary 1. Let P = p 1 . . . p n be a convex polygon with angles β i = π − α i , where α i ∈ (0, π) for i = 1, . . . , n. Let α = max{α i | i = 1, . . . , n}. Then there exists λ 0 > 0, with the following property: For any λ ≥ λ 0 , and any similar region λP of P , each Hamiltonian cycle on the set Z 2 ∩ λP has some turning angle that is Ω(α).. We think that this lower bound is asymptotically tight: Conjecture 3. Let P = p 1 . . . p n be a convex polygon with angles β i = π − α i , where α i ∈ (0, π) for i = 1, . . . , n. Let α = max{α i | i = 1, . . . , n}. Then there exist λ 0 > 0 and c ≥ 1, with the following property: For any λ ≥ λ 0 , and any similar region λP of P , the set Z 2 ∩ λP can be traversed by a Hamiltonian cycle with maximum turning angle at most cα.. Remarks. By placing a vertex of λP which attains the minimum angle, say β i , at a lattice point forces a turning angle at least α i = π − β i . This shows that if Conjecture 3 holds, the inequality c ≥ 1 is needed. Obviously any turning angle is at most π, so the interest in Conjecture 3 is for α close to zero (unbounded from below). Observe also that n i=1 α i = 2π which implies α ≥ 2π/n. In particular, for the regular n-gon, α = 2π/n, and the question is whether the set Z 2 ∩ λP can be traversed by a Hamiltonian cycle with maximum turning angle O(1/n), provided that λ is sufficiently large. By Corollary 1, the maximum turning angle is Ω(1/n), regardless of how λP is placed.. Definitions and notations. For a positive integer m, let [m] = {1, 2, . . . , m}. A lattice point (i, j) is special if both i and j are congruent to 0 modulo 4. For a bounded region R in the plane, let Area(X) and per(X) denote the area and respectively, the perimeter, of X. Let ∂X denote the boundary of X. We will refer to a circular annulus also as a circular ring, or simply ring, and to an annular sector also as a ring sector. A ring sector will be also called block. The length of a block with inner and outer radii r 1 , r 2 and center angle β is r 1 +r 2 2 β. When there is no danger of confusion, a block may also refer to the actual set of lattice points within.. Given a fixed (possibly disconnected) region X in the plane, bounded by finitely many closed curves, and λ > 0, denote by λX any similar copy of X, that is, a possibly rotated copy of X that is scaled by a factor of λ. While our proofs of Theorems 1 and 3 are tailored for the disk case, we believe that the general ideas used there will be applicable for the general case (Conjectures 1 and 2) in constructing Hamiltonian tours with small turning angles. The two basic ideas are:. (i) Partitioning the input region into smaller sub-regions that can be conveniently linked.. (ii) Visiting only some points in a sub-region and going back to that sub-region over and over again, until all points are exhausted while maintaining the turning angle constraint.. We outline two further directions.. 1. The proofs of Theorems 1 and 3 are constructive and lead to linear-time algorithms for computing a Hamiltonian tour with each turning angle at most ε: for an input point set S, such a tour can be computed in O(|S|) time. However, some constant factors in our tour construction are too large and need to be reduced. These constants are of little concern in the existence part of Theorem 1, but are relevant for the algorithmic part, because they impose large disks (regions, in general) as inputs.. 2. Let S be a set of n points in the plane, and α ∈ [0, π]. A necessary condition for S to admit a Hamiltonian cycle with each turning angle at most π − α is the following:. [T] Each point q ∈ S determines a triangle ∆pqr with ∠pqr ≥ α; or equivalently, that the turning angle at q in the sequence p, q, r is at most π − α.. It is easy to construct examples which show that the above condition does not suffice to guarantee a Hamiltonian cycle with each turning angle at most π − α: for instance: (i) an equilateral triangle with its center, for α = π/3, and (ii) a square with its center, for α = π/2. Suppose that S satisfies the above condition [T] . In particular, [T] implies that the interior angle at each point on conv(S) is at least α. Does any such S admit a Hamiltonian cycle with each turning angle at most π − c · α, for some absolute constant c > 0 ? By the above examples, c ≤ 1/2, if it exists.

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O B I O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O B I O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O B I O O O O O O O O O O O O O O O O O O O O O O O O O


The PCG: An Empirical Study Process communication graph (PCG) is the visual formalism used in a graph-based visual language (VL) for parallel programming. It combines control flow and data flow graphs into a single visual formalism, and supports different levels of abstraction at which parallel programs are expressed and moves to compositional programming. Empirical studies allow designers to put their designs to test in a direct and intentional interaction with users. For research projects this may be the only way to assess if their goals have been met. The case study presented here was conducted on programmers (students) solving parallel programming problems using the PCG formalism to construct parallel programs. The results of this evaluation indicate that users benefit from visual programming, even at the beginning of the learning curve.Academic Press THE IDEA OF USING VISUAL rather than textual languages for programming is not new, and we have witnessed numerous efforts. Since the complexity of computer programs is steadily increasing, presenting them in graphic form facilitates their comprehension. By capturing the strategy and the algorithm of a program, the graphics abstracts away implementation details, which makes the program code more understandable. Scanlan [1] found that structured flowcharts had an advantage over pseudo-code in algorithm comprehension even in simple cases. Graphics as windows is an integral part of the modern desktop making it user friendlier [2] . Yet, in the world of computing visual programming is often attacked by skeptical critics inclined to dismiss the idea completely. Usability of visual languages is addressed by empirical studies, which attempt to evaluate and compare them to textual languages in terms of effectiveness and appropriateness [3] . However, case studies about solving problems through parallel computation using visual methods have rarely been conducted. The lack of empirical evidence has been regarded as the biggest open problem in research into and design of visual programming languages [4] .. Even though the idea to use graphics for programming is compelling, the practical experience so far has taught us that graphics suit better program design phase than program implementation phase. As a matter of fact, the presence and appeal of graphics during design is overwhelming. Graphical symbols convey easily and clearly program structure, action and event sequences, entity relationships, and similar. However, they scale badly when used for actual coding.. Through everyday practice, problems, challenges and advantages of visual programming have been identified: z Graphics take up space and the information density can be lower than in text, which can make a visual program less readable. This problem is often referred to as the Deutsch Limit of 50 [5] . z In general, graphics can convey structural information more effectively than text, while text is usually more precise than graphics. Therefore, any effective visual languages must use both text and graphics. z It is in fact not practical or feasible to have a pure graphical language, since programming language semantics and programming construct semantics do not map one to one. Implementation complexity far exceeds the programming language complexity. A programming language is not just a small set of keywords, but also requires a functional library. z Graphical representation of a parallel program is very useful since it can closely resemble program execution on a multiprocessor or a network of workstations. z Practically, a visual approach is very effective when solving visual problems such as building a dialogue box or a menu. Rather than imagining the right color, size and position of an entity, the visual technique materializes the mental image. z Defining a GUI entity is a simple mechanical process of setting up a set of attribute-vale pairs by using, for example, JavaBuilder, Visual Basic, Visual Works. z Generating code from graphics follows the same pattern of reasoning. It is easy to generate GUI code, since the domain is very constrained, as it implies no logic or alternative traversal paths.. So why do we suggest directed graphs for parallel programming? The reason is that graphs excel at displaying certain information that is important for better understanding of parallel programs and obtaining good performance from them. Graphs show parallelism in execution, locality and communication topology, which are among the most important aspects of parallel programming.. The success of a visual language often depends on the editor used for graphics rendering and program composition. The editor enforces consistency and performs syntax checking. The ease of use has been recognized as a more important goal of visual programming environments. Newton conducted a case study of the CODE [6] parallel programming environment, and concluded that though important, ease of use is difficult to evaluate due to its subjective nature [7] . It requires a comparative study of multiple languages and environments, such as the one reported in [8] , conducted on many subjects. Based on these and similar works we have decided, when designing our visual language for parallel programming the approach should be pragmatic. Rather than insisting on a pure graphical environment, we decided to take advantage of different language and programming forms that better meet their goals and serve the purposes.. In the next section, we present the PCG visual parallel-programming environment within the Visper system, a Java-based parallel programming prototype [9] . Section 3 presents an example of the program composition using PCG, and Section 4 describes our empirical study. Section 5 concludes the paper. This paper presents an empirical study conducted to acquire a direct feedback on PCG. The subjects were the students who have been exposed to parallel programming before participating in the study. Therefore, the experience in using a visual programming tool could be compared to the text editor practice without a bias (positive or negative) towards the new technology. While the number of subjects was not large, it was comparable to similar undertaken studies. In summary, it was generally agreed that integrating visual features into any software system was very important, particularly for parallel programming tools. The PCG editor has been found as simple and relatively easy to use. It adopts a systematic approach, and intuitive graphical representation. Although the PCG editor is not mature enough to be used for large-scale programming, using it for teaching parallel programming is expected to be useful.. This empirical study has encouraged us that a visual parallel programming formalism like the PCG can be useful if scalable and easy to use. It also provides us with a first-hand knowledge on what are the most needed features and what are the most common difficulties in visual parallel programming. Although the results are not conclusive, the findings do provide a useful guidance that will help us in improving usability of the PCG environment. While in a study like this one can only skim through relevant issues, it gives us a first-hand experience with the basic aspects regarding our approach. A more comprehensive study such as the one recommended in [18] would require subjects with a relevant background and experience. It would allow us to test more complex constructs in PCG, such as the conditional data flow, the secondary notation, collective communications, etc. Our future work is towards that direction.

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


A simple, fast and scalable non-blocking concurrent FIFO queue for shared memory multiprocessor systems A non-blocking FIFO queue algorithm for m ultiprocessor shared memory systems is presented in this paper. The algorithm is very simple, fast and scales very well in both symmetric and non-symmetric multiprocessor shared memory systems. Experiments on a 64-node SUN Enterprise 10000 ; a symmetric m ultiprocessorsystem ; and on a 64-node SGI Origin 2000 ; a cac he coherent non uniform memory access m ultiprocessorsystem ; indicate that our algorithm considerably outperforms the best of the known alternativ es in both multiprocessors in any lev el of multiprogramming. This work introduces t w o new, simple algorithmic mechanisms. The rst low ers the contention to key variables used by the concurrent enqueue and/or dequeue operations which consequently results in the good performance of the algorithm, the second deals with the pointer recycling problem, an inconsistency problem that all non-blocking algorithms based on the compare-and-swap sync hronisation primitive h a ve to address. In our construction we selected to use compare-and-swap since compare-and-swap is an atomic primitive that scales well under contention and either is supported by modern multiprocessors or can be implemented e cient l y o n t h e m . Concurrent FIFO queue data structures are fundamental data structures used in many applications, algorithms and operating systems for m ultiprocessorsystems. T o protect the integrity of the shared queue, concurrent operations that have been created either by a parallel application or by the operating system have to be sync hronised. T ypically , algorithms for concurrent data structures, including FIFO This w ork is partially supported by: i) the national Swedish Real-Time Systems research initiativ e ARTES (www.artes.uu.se) supported by the Swedish Foundation for Strategic Research and ii) the Swedish Research Council for Engineering Sciences.. queues, use some form of mutual exclusion (locking) to synchronise concurrent operations. Mutual exclusion protects the consistency of the concurrent data structure by a l l o wing only one process (the holder of the lock o f the data structure) at a time to access the data structure and by blocking all the other processes that try to access the concurrent data structure at the same time. Mutual exclusion and, in general, solutions that introduce blocking are penalised by locking that introduces priority i n version, deadlock scenarios and performance bottlenecks. The time that a process can spend blocked while waiting to get access to the critical section can form a substantial part of the algorithm execution time 5, 9, 10, 14]. There are two main reasons that locking is so expensive. The rst reason is the convoying e ect that bloc king synchronisation su ers from: if a process holding the loc k is preempted, any other process waiting for the lock is unable to perform any useful work until the process that hold the locks is scheduled. When we taking into account that the m ultiprocessorrunning our program is used in a m ultiprogrammingenvironment, convoying e ects can become serious. The second is that locking tends to produce a large amount of memory and interconnection network contention, loc ks become hot memory spots.Researchers in the eld rst designed di erent l o c k implementations that low er the con tention when the system is in a high congestion situation, and they give di erent execution times under di erent con tention instances. But on the other hand the overhead due to blocking remained. T o address the problems that arise from blocking researchers have proposed non-blocking implementations of shared data structures. Non-blocking implementation of shared data objects is a new alternative approach to the problem of designing scalable shared data objects for multiprocessor systems. Non-blocking implementations allow m ultiple tasks to access a shared object at the same time, but without enforcing mutual exclusion to accomplish this. Since in non-blocking implementations of shared data structures one process is not allowed to block another process, non-blocking shared data structures have the follo wing signi cant a d v antages over lock-based ones:. 1. they avoid lock c o n voys and contention points (locks). 2. they pro vide high fault tolerance (processor failures will never corrupt shared data objects) and eliminates deadloc k scenarios, where t w o or more tasks are waiting for loc ks held by the other. 3. they do not give priority i n version scenarios. Among all the innovative architectures for multiprocessor systems that have been proposed the last forty y ears shared memory multiprocessor architectures are gaining a c e n tral place in high performance computing. Over the last decade many shared memory multiprocessors have been built and almost all major computer vendors develop and o er shared memory multiprocessor systems nowadays. There are two main classes of shared memory multiprocessors: the CacheCoherent N o n uniform Memory Access multiprocessors (cc-NUMA) and the symmetric or Uniform Memory Access (UMA) multiprocessors, their di erences coming from the architectural philosophy they are based on. In symmetric shared memory multiprocessors every processor has its own cache, and all the processors and memory modules attach t o t h e same interconnect, which is a shared bus. ccNUMA is a relatively new system topology that is the foundation for nextgeneration shared memory multiprocessor systems. As in UMA systems, ccNUMA systems maintain a uni ed, global coherent memory and all resources are managed by a s i ngle copy of the operating system. A hardware-based cache coherency scheme ensures that data held in memory is consistent on a system-wide basis. In contrast to symmetric shared memory multiprocessor systems in which all memory accesses are equal in latency, in ccNUMA systems, memory latencies are not all equal, or uniform (hence, the name -Non-Uniform Memory Access). Accesses to memory addresses located on "far" modules take longer than those made to "local" memory. This paper addresses the problem of designing scalable, practical FIFO queues for shared memory multiprocessor systems. First we present a non-blocking FIFO queue algorithm. The algorithm is very simple, it algorithmically implements the FIFO queue as a circular array and introduces two new algorithmic mechanisms that we believe can be of general use in the design of e cient non-blocking algorithms for multiprocessor systems. The rst mechanism restricts contention to key variables generated by concurrent enqueue and/or dequeue operations in low l e v els contention to shared variables degrades performance not only in memory tanks where the variables are located but also in the processor-memory interconnection network. The second algorithmic mechanism that this paper introduces is a mechanism that deals with the pointer recycling (also known as ABA) problem, a problem that all non-blocking algorithms based on the compare-and-swap primitive h a ve to address. The performance improvements are due to these two mechanisms and to its simplicity that comes from the simplicity and richness of the structure of circular arrays. We have selected to use the compare-and-swap primitive since it scales well under contention and either is supported by modern multiprocessors or can be implemented e ciently on them. Last, we evaluate the performance of our algorithm on a 64-node SUN Enterprise 10000 multiprocessor and a 64-node SGI Origin 2000. The SUN system is a Uniform Memory Access (UMA) multiprocessor system while the SGI system is a Cache-Coherent Nonuniform Memory Access (ccNUMA) one SUN Enterprise 10000 supports the compare-and-swap while SGI Origin 2000 does not. The experiments clearly indicate that our algorithm considerably outperforms the best of the known alternatives in both UMA and ccNUMA machines with respect to both dedicated and multiprogramming workloads. Second, the experimental results also give a better insight into the performance and scalability of non-blocking algorithms in both UMA and ccNUMA large scale multiprocessors with respect to dedicated and multiprogramming workloads, and they con rm that non-blocking algorithms can perform better than blocking on both UMA and ccNUMA large scale multiprocessors, and that their performance and scalability increases as multiprogramming increases.. Concurrent FIFO queue data structures are fundamental data structures used in many m ultiprocessor programs and algorithms and, as can be expected, many researchers have proposed non-blocking implementations for them. Lamport 6] introduced a wait-free queue that does not allow more than one enqueue operation or dequeue operation at a time. Herlihy and Wing in 4] presented an algorithm for a nonblocking linear FIFO queue which requires an in nite array. Prakash, Lee and Johnson in 11] presented a non-blocking and linearisable queue algorithm based on a singly-linked list. Stone describes a non-blocking algorithm based on a circular queue. Massalin and Pu 8] present a non-blocking array-based queue which requires the double-compare-andswap atomic primitive that is available only on some members of the Motorola 68000 family of processors. Valois in 12] presents a non-blocking queue algorithm together with several other non-blocking data structures, his queue is an array-based one. Michael and Scott in 10] presented a nonblocking queue based on a singly-link list, which is the most e cient and scalable non-blocking algorithm compared with the other algorithms mentioned above.. The remainder of the paper is organised as follows. In Section 2 we g i v e a brief introduction to shared memory multiprocessors. Section 3 presents our algorithm together with a proof sketch. In Section 4, the performance evaluation of our algorithm is presented. The paper concludes with Section 5. In this paper we presented a new bounded non-blocking concurrent FIFO queue algorithm for shared memory multiprocessor systems. The algorithm is simple and introduces two new simple algorithmic mechanisms that can be of general use in the design of e cient non-blocking algorithms. The experiments clearly indicate that our algorithm considerably outperforms the best of the known alternatives in both UMA and ccNUMA machines with respect to both dedicated and multiprogramming workloads. The experimental results also give a better insight i n to the performance and scalability of non-blocking algorithms in both UMA and cc-NUMA large scale multiprocessors with respect to dedicated and multiprogramming workloads, and they con rm that non-blocking algorithms can perform better than blocking on both UMA and ccNUMA large scale multiprocessors.

O O O O O O O O O O B I B B O O O O B O O O B I B O O O O B O B O O O O O O O O O O O O O B B I B O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B B O O O O O O O O O B I B O O O O O O O O O O O B O O O O O O B I O O O O O O O O O O O O O O B I B B O O O O O O O B I O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O B I O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I B B O O O B O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B I B B O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O O B I B B O O O O O O O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B B O O O O B O O O O O O O O O O O O O O O O B O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O B O O O O B O B O O O O O O O O O O O O O B O B O O O O O O O O O O B O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O B I O O O O O B O O O O O O O O O B O O O B O O O B O O O O O O B O O O O O O O O O B O B I B B O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Software effort estimation as a multiobjective learning problem Ensembles of learning machines are promising for software effort estimation (SEE), but need to be tailored for this task to have their potential exploited. A key issue when creating ensembles is to produce diverse and accurate base models. Depending on how differently different performance measures behave for SEE, they could be used as a natural way of creating SEE ensembles. We propose to view SEE model creation as a multiobjective learning problem. A multiobjective evolutionary algorithm (MOEA) is used to better understand the tradeoff among different performance measures by creating SEE models through the simultaneous optimisation of these measures. We show that the performance measures behave very differently, presenting sometimes even opposite trends. They are then used as a source of diversity for creating SEE ensembles. A good tradeoff among different measures can be obtained by using an ensemble of MOEA solutions. This ensemble performs similarly or better than a model that does not consider these measures explicitly. Besides, MOEA is also flexible, allowing emphasis of a particular measure if desired. In conclusion, MOEA can be used to better understand the relationship among performance measures and has shown to be very effective in creating SEE models. Estimating the cost of a software project is a task of strategic importance in project management. Both over and underestimations of cost can cause serious problems to a company. For instance, overestimations may result in a company losing contracts or wasting resources, whereas underestimations may result in poor quality, delayed or unfinished softwares. The major contributing factor for software cost is effort [Agarwal et al. 2001] .. Human made effort estimations tend to be strongly affected by effort-irrelevant and misleading information [Jørgensen and Grimstad 2011] . Moreover, developers tend not to improve their effort estimation uncertainty assessment even after feedback about their estimates is provided [Gruschke and Jørgensen 2008 ]. An alternative to human made effort estimations is to use automated effort estimators. Models for estimating software effort can be used as decision support tools, allowing investigation of the impact of certain requirements and development team features on the cost/effort of a project to be developed.. A major difficulty in performing software estimation is the lack of project data that include size and detailed quantitative counts on the elements of UML models used, for example, the number of analysis classes, number of relationship types, number of associated attributes, use-case point, etc. As a result, even though UML models are commonly used, it is still very difficult to do software estimation based on UML models constructed in the early stage of software development. Hence, it is very difficult to further test, evaluate and enhance very promising estimation approaches that are based on the UML models constructed in the early state of requirements analysis [Tan et al. 2006 [Tan et al. , 2009 Mohagheghi et al. 2005] . The analysis of the effectiveness of estimation models based on limited project data thus becomes an important field to be investigated. Different automated methods for software cost or Software Effort Estimation (SEE) have been proposed [Jørgensen and Shepperd 2007] .. In the present work, we look into a type of machine learning method which has recently attracted attention from the SEE community [Braga et al. 2007; Kultur et al. 2009; Kocaguneli et al. 2009; Minku and Yao 2011] , namely ensembles of learning machines. Ensembles are sets of learners trained to perform the same task and combined with the aim of improving predictive performance [Chen and Yao 2009] . When combining learning models in an attempt to get more accurate predictions, it is commonly agreed that these base models should behave differently from each other. Otherwise, the overall prediction will not be better than the individual predictions [Brown et al. 2005; Kuncheva and Whitaker 2003 ]. This behaviour matches intuition: if models that make the same mistakes are combined into an ensemble, the ensemble will make the same mistakes as the individual models and its performance will be no better than the individual performances. On the contrary, ensembles composed of diverse models can compensate the mistakes of certain models through the correct predictions performed by other models. So, diversity refers to the predictions/errors made by the models. Two models are said to be diverse if they make different errors on the same data points [Chandra and Yao 2006] . Different ensemble learning approaches can be seen as different ways to generate diversity among the base models.. Even though ensembles have been shown to be promising for SEE, it is necessary to tailor them for this estimation task. Simply using any general purpose ensemble approach from the literature does not necessarily improve SEE in comparison to single learning machines [Minku and Yao 2011] . Minku and Yao [2013] showed that combining the power of ensembles to local learning through the use of bagging ensembles of regression trees outperforms several other learning machines for SEE in terms of Mean Absolute Error (MAE). This combination is a way to tailor ensembles for SEE, but Minku and Yao [2013] also show that more improvements may still be achieved if additional tailoring is performed. Very recently, Kocaguneli et al. [2012] proposed an ensemble method claimed to outperform single learners for SEE. However, their method is not fully automated, as it needs manual/visual inspection of extensive experiments to create the ensemble. Section 2 explains related work on machine learning and in particular ensembles for SEE.. Much of the SEE work involves empirical evaluation of models and several different measures of performance can be used for that. Examples of measures are two popular measures based on the magnitude of the relative error, namely Mean Magnitude of the Relative Error (MMRE) and percentage of estimations within N% of the actual value (PRED(N)), and a measure based on the logarithm of the residuals recommended by Foss et al. [2003] , namely Logarithmic Standard Deviation (LSD). Different measures can behave differently and it is highly unlikely that there is a "single, simple-touse, universal goodness-of-fit kind of metric" [Foss et al. 2003 ], that is, none of these measures is totally free of problems.. The relationship among different measures is not yet well understood. For example, it is not known to what extent a lower MMRE could be related to a higher LSD and vice versa, or how different are the MMREs associated to the same PRED value. Depending on how differently these performance measures behave, it may be possible to use them as a natural way to generate diversity in ensembles for SEE. With that in mind, this paper aims at answering the following research questions: -RQ1. What is the relationship among different performance measures for SEE? -RQ2. Can we use different performance measures as a source of diversity to create SEE ensembles? In particular, can that improve the performance in comparison to models created without considering these measures explicitly? Existing models do not necessarily consider the performance measures in which we are interested explicitly. For example, Multi-layer Perceptrons (MLPs) are usually trained using Backpropagation, which is based on the mean squared error. So, they can only improve MMRE, PRED, and LSD indirectly. We would like to know whether creating an ensemble considering MMRE, PRED, and LSD explicitly leads to more improvement in the SEE context. -RQ3. Is it possible to create models that emphasize particular performance measures should we wish to do so? For example, if there is a certain measure that we believe to be more appropriate than the others, can we create a model that particularly improves performance considering this measure? This is useful not only for the case where the software manager has sufficient domain knowledge to chose a certain measure, but also (and mainly) if there are future developments of the SEE research showing that a certain measure is better than others for a certain purpose.. In order to answer these questions, we formulate the problem of creating SEE models as a multiobjective learning problem that considers different performance measures explicitly as objectives to be optimised. This formulation is key to answer the research questions because it allows us to use a Multi-objective Evolutionary Algorithm (MOEA) [Wang et al. 2010] to generate SEE models that are generally good considering all the predefined performance measures. This feature allows us to use plots of the performances of these models to understand the relationship among the performance measures and how differently they behave. Once these models are obtained, the models that perform best for each different performance measure can be determined. These models behave differently from each other and can be used to form a diverse ensemble that provides an ideal tradeoff among these measures. Choosing a performance measure is not an easy task. By using our ensemble, the software manager does not need to chose a certain performance measure, as an ideal tradeoff among different measures is provided. As an additional benefit of this approach, each of the models that compose the ensemble can also be used separately to emphasize a specific performance measure if desired. The performance measures used by the MOEA in this work are MMRE, PRED(25) and LSD. Section 3 explains MOEAs and Section 4 explains our proposed approach to use MOEAs for creating SEE models.. Our analysis shows that the different performance measures behave very differently when analysed at their individual best level and sometimes present even opposite behaviour (RQ1). For example, when considering nondominated (Section 3) solutions, as MMRE is improved, LSD tends to get worse. This is an indicator that these measures can be used to create diverse ensembles for SEE. We then show that MOEA is successful in generating SEE ensemble models by optimising different performance measures explicitly at the same time (RQ2). The ensembles are composed of nondominated solutions selected from the last generation of the MOEA, as explained in Section 4. These ensembles present performance similar or better than a model that does not optimize the three measures concurrently. Furthermore, the similar or better performance is achieved considering all the measures used to create the ensemble, showing that these ensembles not only provide a better tradeoff among different measures, but also improve the performance considering these measures. We also show that MOEA is flexible, allowing us to chose solutions which emphasize certain measures, if desired (RQ3).. The base learners generated by the MOEA in this work are Multi-Layer Perceptrons (MLPs) [Bishop 2005 ], which have been showing success in the SEE literature for not being restricted to linear project data [Tronto et al. 2007] . Even though we generate MLPs in this work, MOEAs could also be used to generate other types of base learners, such as RBFs, RTs and linear regression equations. An additional comparison of MOEA to evolve MLPs was performed against nine other types of models. The comparison shows that the MOEA-evolved MLPs were ranked first more often in terms of five different performance measures, but performed less well in terms of LSD. They were also ranked first more often for the data sets likely to be more heterogeneous. It is important to note, though, that this additional comparison is not only evaluating the MOEA and the multiobjective formulation of the problem, but also the type of model being evolved (MLP). Other types of models could also be evolved by the MOEA and could possibly provide better results in terms of LSD. As the key point of this article is to analyse the multiobjective formulation of the creation of SEE models, and not the comparison among different types of models, the experimentation with different types of MOEA and different types of base models is left as future work.. This article is organised as follows. Section 2 presents related work on machine learning and ensembles for SEE. Section 3 explains MOEAs and the type of MOEA used in this work. Section 4 explains our approach for creating SEE models (including ensembles) through MOEAs. In particular, Section 4.1 explains the multiobjective formulation of the problem of creating SEE models. Section 5 explains the experimental setup for answering the research questions and evaluating our approach. Section 6 explains the data sets used in the experiment. Section 7 provides an analysis of the relationship among different performance measures (RQ1). Section 8 provides an evaluation of the MOEA's ability of creating ensembles by optimising several different measures at the same time (RQ2). Section 9 shows that MOEAs are flexible, allowing us to create models that emphasize particular performance measures, if desired (RQ3). Section 10 shows that there is still room for improvement in the choice of MOEA models to be used for SEE. Section 11 complements the evaluation by checking how well the Pareto ensemble of MLPs performs in comparison to other types of models. Section 12 presents the conclusions and future work. This article proposes and demonstrates how to view the problem of creating SEE models through a different perspective, by tackling it as a multiobjective problem that considers different performance measures explicitly and simultaneously. Using a MOEA to solve this problem allows us to better understand different performance measures and to produce SEE models with good overall performance in comparison to a model that does not consider these measures explicitly.. As an answer to RQ1, we show that LSD and MMRE are performance measures with somewhat opposite behaviour. Moreover, models with similar LSD/MMRE are likely to present different PRED(25) and vice-versa. So, one may wish to choose a model that does not behave particularly bad for any intended performance measures. This is one of the motivations for our approach to use MOEAs, as MOEAs consider all these performance measures at the same time. This difference in behaviour also provides the second motivation for our approach: it can produce diverse ensembles, which are more likely to have increased performance. The results above indicate that considering different performance measures explicitly when creating a model may be used to produce good ensembles. As an answer to RQ2, we show that indeed a MOEA can be used to create models by explicitly considering different performance measures at the same time. Pareto ensemble of MLPs produced by a MOEA generally obtained similar or better performance than Backpropagation MLPs considering both LSD, MMRE, and PRED(25). The average performances were generally better for the data sets with 60 or more projects.. As an answer to RQ3, we show that MOEAs are also flexible, allowing the software manager to choose models that emphasize certain performance measures over the others, if s/he desires to do so. A MOEA can be used to produce and choose models that emphasize a particular measure without completely ignoring the performance using other measures, whereas the Pareto ensemble can be used as a good tradeoff among measures, if the software manager does not wish to emphasize any particular measure.. As shown in our comparison of the Pareto ensemble against Backpropagation MLPs, which produce the same type of model as the MOEA, the Pareto ensemble has shown to be useful for both single and multicompany data sets. Our comparison against models of a different type than the base models evolved by the MOEA further shows that MOEAs may be particularly useful for more heterogeneous data sets. They can make types of models that would usually not be ranked first in terms of performance become ranked first through the Pareto ensemble.. Our proposed methodology still has room for improvements in terms of the choice of models to compose the Pareto ensemble. The investigation of new strategies is left as future work, as well as the experimentation of our approach using different MOEAs and creating different types of SEE models, and investigation on how much our approach would be affected by outliers in the training set used to build the models.

B I I O O O O O B I I I O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O