Motion adaptation based on character shape With the increasing performances of the graphics hardware, virtual reality applications are reaching a new level of realism. It is now possible to animate entire crowds of virtual avatars, or to display in real-time highly realistic deformable characters. This new type of applications requires that characters are deformed, either to generate a large number of individuals for creating a crowd, or to match a particular shape. To make the animation fit to the animated character and to prevent self-collisions, adapting the motion is mandatory. Unlike previous approaches which relied on skeletal models, this paper presents a method considering the actual shape of the character's body to perform the adaptation. Our approach uses spacetime optimization to remove the selfpenetration, and finally re-establishes the balance of the motion. We also introduce an interpolation scheme based on radial basis functions that can blend pre-calculated adaptations, and thus achieve real-time performances. Deformable characters allow to customize the shape of a body to reproduce the specific characteristics of a real person. Various approaches were proposed [1] and several companies from the clothing industry developed their own in-house solution [2] . Until now, none of these commercially available packages were able to provide body animation along with the deformations. The reason for this is twofold. First, the cloth simulations provided by these software packages are of high quality and thus would require a big amount of computation time. Second, a given body animation only matches the body for which it was created. Therefore, when a body is deformed the animation should be adapted accordingly.. Moreover, crowd applications [3] are demanding in terms of avatars: to be realistic, a crowd should feature a wide variety of characters and animations. This variety involves that a motion clip is modified for each character being part of the crowd, otherwise all of them would move the same way.. Deformable characters introduce a new kind of self-penetration which we call shape penetration. Unlike other types of self-penetrations, the shape penetrations last for a longer period of time, often during the entire animation. Even though self-penetrations created by the movements of the end-effectors were already addressed by previous works [4, 5] , these methods based on per-frame IK are unusable here. Indeed, they would introduce discontinuities in the final animations, as explained on figure 1.. Our adaptation process has two goals: remove the self penetrations and keep the phys-ical properties of the original motion. Our approach adapts the movements of each limb separately, and then corrects the balance of the character. This strategy has the advantage to divide the problem, making the adaptation easier to calculate.. The remainder of this paper is organized as follows. We review the previous works in next section. We then detail the creation of skeletal models from the actual body shape. Next we will explain the adaptation process and the real-time interpolation method. Eventually, we present the results we obtained and conclude this paper. In this paper we presented a method that adapts a given animation clip so that the motion fits to any kind of body. The proposed approach uses spacetime optimization to prevent any self penetration to occur, while maintaining the balance of the character. We also proposed a variant of the NRBF interpolation to make the corrections computable in real-time.. We did not apply the forearm orientation correction to the hand because it generated to much correction, producing unrealistic postures. When the legs motion is adapted, the hip rotation makes the entire upper body rotate as well. In case one would like to keep the original head orientation (for controlling the gaze for instance), then the head joint can be rotated with the inverse of the hip rotation. Similarly, when adapt the posture of a character, the forearm orientation is also changed. This can be adjusted by applying the opposite rotation value from the spine to the shoulder. However we recommend being careful with this as it may introduce new penetrations which did not exist before.. In the future we would like to extend our approach to incorporate more physics in the adaptation to make the actual strength of the character influence the final result. We would also like to introduce physical deformations of the body in order to produce a more accurate adaptation and skin deformation. Eventually, we would like to implement the footskating removal method from [23] because the one from [24] sometimes introduced small artifacts in the final animations. On the left is the original trajectory: The elbow joint is moving from the rear of the body towards the front, the circle representing a collision volume. On the middle a per-frame approach would apply a correction only when needed, thus discarding the natural movement.. On the right is the effect we intend to produce: the motion was changed globally so that no more penetration remains, while keeping the resulting movement close to the original. The adaptation data was precalculated and interpolated at runtime to adapt the animation according to the growth of the body. The cloth was animated using the method proposed in [28] . We implemented the algorithms described above in C++ using the RFSQP optimizer from AEM Design [21] . The motion adaptation itself took between 0.5 and 1 second per frame on a pentium IV 3.0 Ghz processor.. We tried our approach on various body shapes and animations and we successfully eliminated the self penetrations while maintaining a high level of realism. Figure 6A shows a the landing of a jump sequence. On figure 6B the posture was adapted to maintain the balance. The run-time system was tested by making each of the limbs successively grow and shrink in diameter while keeping the size of the other limbs fixed. We kept five dimensions for the interpolation space (legs, arms and trunk) and thus the number of optimizations to perform was 3 5 = 243, which took approximately one days for an height hundred frames animation. Snapshots of inputs to the system can be seen on figure 6E, while figure 7 shows an interpolated animation.

O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B


Program analysis for event-based distributed systems Designing distributed applications around the idiom of events has several benefits including extensibility and scalability. To improve conciseness, safety, and efficiency of corresponding programs, several authors have recently proposed programming languages or language extensions with support for event-based programming.The presence of a dedicated programming language and compilation process offers avenues for program analyses to further improve simplicity, safety, and expressiveness of distributed event-based software. This paper presents three program analyses specifically designed for event-based programs: immutability analysis avoids costly cloning of events in the presence of co-located handlers for same events; guard analysis allows for simple yet expressive subscriptions which can be further simplified and handled efficiently; causality analysis determines causal dependencies among events which are related, allowing unrelated events to be transferred independently for efficiency. We convey the benefits of our approach by empirically evaluating their performance benefits. Event-based design decouples system components thus improving extensibility and scalability of the software. More precisely, decoupling is achieved by using a dedicated runtime system that delivers events from sources (producers, publishers) to sinks (consumers, subscribers) which are not a priori aware of each other, yielding the following benefits:. Simplicity. Decoupling avoids name binding of components, yielding modules which are largely independent. Components can thus be developed, extended, or tested independently from each other. By the same token, new components can be deployed into a running application.. Efficiency. The use of a dedicated middleware system allows for communication to be performed efficiently and in a scalable manner by aggregating and sharing traffic among components where possible.. Event-based middleware systems include message queues (an event is consumed by one of many sinks) and publish/subscribe systems (an event is delivered to all sinks interested in it). Systems include research engines (e.g., Siena [10] , JEDI [12] , and PADRES [28] ) or industrial solutions (e.g., Amazon's Simple Queue Service, ActiveMQ, or FioranoMQ), several of which provide both types of communication.. Programming solutions for event-based programming range from simple design patterns (observer design pattern) to language extensions capturing concurrency through asynchronous events (e.g., Ptolemy [35] , Responders [11] , Scala [20] following an Actor model) to even higher-level coordination and event correlation abstractions (e.g., SCHOOL [32] , Cω [7] based on Join Calculus [16] ). Language-based approaches have several well-known benefits [9] . First, by providing specific constructs and abstractions these approaches further improve the simplicity of programs. Second -and among the prime motivations for language design research in general -choosing the right abstraction can improve the efficiency of language implementations. Third, based on language constructs, the compiler can ensure static or dynamic safety properties, such as type conformance.. Program analysis offers many intriguing avenues to increase the efficiency and other aspects of distributed eventbased applications, but remains underexploited. This is precisely the motivation for this paper. We present three program analyses yielding benefits in terms of simplicity, efficiency, as well as safety:. Immutability analysis: When an event is consumed multiple times by different handlers in a same process then the event and its attributes (representation) must be cloned in most cases to not hamper the decoupling nature of event-based interaction and pass-by-value semantics that are used to support it. Immutability analysis infers cases where attributes are not modified in any way (e.g., by assignment) by handlers or further down the line, such that expensive cloning can be avoided. This may have a cumulative effect as co-locating handlers of a same event allows for limiting the times an event is sent over the wire.. Guard analysis: Guards are the natural way to express subscriptions in event-based languages. Besides allowing for statically verifying compliance of subscriptions with the events they are expressed on, guards naturally capture advanced features such as parametric subscriptions [22, 25] which have been shown to benefit efficiency. In contrast to an API-based support for the subscriber-local variables underlying such subscriptions, programming language support allows for complex subscriptions, e.g., those involving arithmetic and multiple variables to be expressed, and for the compiler to automatically simplify these and infer the relevant parameters.. Causality analysis: Causal order between events, especially when these are broadcast, has traditionally been achieved by manually mapping events to "broadcast groups", implemented by means of appropriate protocols [37] , to capture dependencies. This is not only tedious for programmers (groups may overlap) but unsafe as dependencies are easily overlooked. Pessimistically funneling all events -even those requiring no ordering with respect to others and each other -through a same group strongly hampers efficiency. Causality analysis infers all possible dependencies (for safety) from programs, and with hints from the programmer on independencies (for efficiency) allows for groups to be be created adequately and manipulated automatically by the runtime.. These analyses can implemented on top of event processing middleware API as well with some minor additional effort. For the sake of presentation simplicity, we present them however in the context of our EventJava language [15] , which has been designed with emphasis on genericity, extensibility and flexibility to thwart the potential limitations of domain-specific programming languages. EventJava is an extension of the mainstream Java programming language, implemented with an extensible compiler. EventJava's runtime is implemented as a framework of substitutable components, accessible directly through API, which allows existing middleware systems to be plugged in.. In summary, this paper makes the following contributions:. 1. We introduce a subset of (mostly) static program analyses tailored to event-based programming which we have devised for EventJava, namely immutability analysis, guard analysis, and causality analysis.. 2. We illustrate the benefits of our approach through empirical evaluation of performance gains enabled by our analyses.. Roadmap. Section 2 introduces the EventJava framework. Sections 3-5 present our analyses. Section 6 demonstrates the performance benefits of our approach. Section 7 summarizes related work. Section 8 concludes with final remarks. The full potential of the event idiom for programming complex distributed applications has yet to be discovered and exploited. In this paper we have presented a programming language framework guiding developers in writing correct and efficient complex event-based distributed applications without hampering flexibility in the use of existing middleware systems or adaptability of resulting software. We have presented several program (mostly) static program analyses implemented in EventJava, yielding benefits in simplicity, efficiency, and safety. It is straightforward to see that guard analysis eliminates the need for a programmer to manually (1)introduce variables to capture expressions on guard fields, (2)examine application code to identify any point at which a guard field's value changes. Causality analysis eliminates the need to manually identify direct and indirect produce-consume/consume-consume relationships between events, and handle dynamically loaded classes. Guard and causality analyses respectively guarantee that no update to a guard field or causal dependency between events is missed, because the compiler generates code to track updates, manage broadcast groups and handle dynamic class loading. We are currently refining the core abstractions of our language framework (e.g., reaction synchronization and threading) and extending their options (e.g., correlation with multiple return values). We are also in the process of designing further analyses. We assess the benefits of dynamic guards, which are facilitated by our guard analysis. Figure 4 (a) shows the difference in throughput between EventJava (with expressive dynamic guards) and EventJava with CPS-Traditional. Figure 4(a) shows that the different in throughput between the two versions of EventJava becomes more pronounced as the number of subscribers increases. Figure 4 (a) also shows that the throughput is slightly higher when the number of hops decreases. The average throughput at subscribers located 1 hop away from the publishers is about 1%-4% higher than the average throughput at subscribers located 3 hops away. Similarly, the throughput at subscribers located 3 hops away from the publishers is about 9%-12% higher than average throughput at subscribers located 6 hops away. Figure 4 (c) shows the difference in throughput between EventJava (with expressive dynamic guards) and EventJava with CPS-Traditional. Figure 4 (c) shows that the difference in latency between the two versions of EventJava increases as the number of subscribers increases. As expected, the event dissemination latency increases as the number of hops increases. The average latency at 3 hops is approximately 28% higher than that at 1 hop, and the average latency at 6 hops is approximately 42% higher that that at 1 hop. A similar trend can be seen in update delay. Figure 4(c) shows that the average delay at 3 hops is approximately 32% higher than the delay at 1 hop, while the delay at 6 hops is approximately 49% higher than the delay at 1 hop.. Figure 4(d) shows that EventJava with resubscriptions receives spurious events at a higher frequency than EventJava with expressive dynamic guards, and this increases the number of subscribers increases. The differences between the average frequencies of spurious events when the distance between the producers and subscribers increases is statistically insignificant -this is because the edge broker immediately applies updates and unsubscriptions to its local data structures. Although the number of spurious events between brokers may increase when the number of hops between the producer and subscriber increases, they are filtered away at the edge broker, and hence the frequency of spurious events received by the subscriber doesn't depend on its distance from the producer. Figure 5 shows the difference in throughput, number of spurious events and delay between two versions of EventJava -(a) with support in the matcher for boolean variables, and (b) without such support but by (un-)subscribing upon changes in the boolean value. We summarize related work on support for engineering distributed event-based applications.

B I O O B O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O B O O O O O O O O O O O O O O B I O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O


A Simple Generalisation of the Area Under the ROC Curve for Multiple Class Classification Problems Abstract. The area under the ROC curve, or the equivalent Gini index, is a widely used measure of performance of supervised classification rules. It has the attractive property that it side-steps the need to specify the costs of the different kinds of misclassification. However, the simple form is only applicable to the case of two classes. We extend the definition to the case of more than two classes by averaging pairwise comparisons. This measure reduces to the standard form in the two class case. We compare its properties with the standard measure of proportion correct and an alternative definition of proportion correct based on pairwise comparison of classes for a simple artificial case and illustrate its application on eight data sets. On the data sets we examined, the measures produced similar, but not identical results, reflecting the different aspects of performance that they were measuring. Like the area under the ROC curve, the measure we propose is useful in those many situations where it is impossible to give costs for the different kinds of misclassification. This paper is concerned with supervised classification problems, in which the aim is to devise a method or construct a rule for assigning objects to one of a finite set of classes on the basis of a vector of variables measured on the objects. The information on which the rule is to be based is a design or training set of objects with known vectors of measurements and known classifications. In particular, this paper discusses problems involving more than two classes, and is concerned with measures of performance of such rules. We restrict ourselves to situations in which the rules yield estimates of the probability that a point belongs to each class, or yield scores indicating strength of class membership.. For the special case in which there are only two classes, there are many distinct criteria for comparing the performance of classification rules (e.g., see the detailed discussions in Hand, 1997 Hand, , 2000 . Amongst the most popular are misclassification (or error) rate, and the criterion with which this paper is concerned, the area under the Receiver Operating Characteristic (ROC) curve (e.g., see Hanley & McNeil, 1982; Zweig & Campbell, 1993; Bradley, 1997) .. Misclassification rate is simply the expected proportion of future cases which the rule will classify incorrectly. More generally, if the cost of misclassifying a class i point is c i , i = 0, 1, then the overall expected loss is L = π 0 p 0 c 0 + π 1 p 1 c 1. where p i is the probability of misclassifying a class i object, and π i is the probability that an object comes from class i (we are assuming no costs associated with correct classifications). Misclassification or error rate is the special case in which c 0 = c 1 (and both are then taken to be equal to 1, without loss of generality). Ifp(x) is the estimated probability that an object with measurement vector x belongs to class 0, then a standard result shows that minimum loss is achieved by choosing the classification threshold such that points are classified into class 0 ifp > t = c 1 /(c 0 + c 1 ). This is all very well if one knows the values of the costs c 0 and c 1 . Typically, however, these costs are difficult to determine (e.g., see Bradley, 1997; Provost, Fawcett, & Kohavi, 1998; Adams & Hand, 1999 , 2000 and the references in Turney (1996) .. If the costs cannot be determined, an alternative strategy is simply to compare the overall distributions ofp(x) for class 0 points and class 1 points. The classification rule will generally be better the more these two distributions differ. As we show in detail below, the area under the ROC curve (AUC) is such a measure of the difference between these two distributions. By focusing on a comparison of the distributions of thep(x), the AUC ignores the costs and also (a consequence of the way costs and priors appear together in Eq. (1) the class priors: it concentrates attention on how well the rule differentiates between the distributions of the two classes, and is not influenced by external factors which depend on the use to which the classification is to be put.. All of the above refers to the two class case. Often, however, we are faced with problems in which there are more than two classes. In principle, such problems may be tackled readily enough: one simply chooses a set of costs and defines a classification rule which minimises a multiple class extension of Eq. (1). (For convenience, we shall use the expression 'multiple class' to signify problems with more than two classes.) In practice, however, this is almost never feasible. The major problems of choosing realistic costs in the two class problem, described in Adams and Hand (1999) , are compounded in the multiple class problem. The default (and, indeed, popular) choice of equal costs for the various different kinds of misclassification, leading to overall misclassification rate, is in fact very rarely really suitable. What is needed is a multiple class extension of the AUC approach, which side-steps the problem of choosing costs altogether. This paper describes such a method.. In Section 2, we formally define the AUC measure and show how to estimate it from a sample of data. Common methods are based on explicit integration of areas under the ROC curve, and these risk introducing unnecessary error into the estimate. An alternative, based on the relationship to the Mann-Whitney-Wilcoxon test statistic is described. This relationship is well-known (e.g., see Hanley & McNeil, 1982) , but both it and the implications for estimation seem not always to be appreciated, so we describe it here for completeness.. In Section 3, we describe a straightforward multiple class extensions of the AUC measure. In Section 4 we illustrate some of the properties of the measure, and in Section 5 we present some examples. Misclassification rate is often a poor criterion by which to assess the performance of classification rules. It makes the implicit assumption that the costs of the different kinds of misclassification are equal, an assumption which is rarely justified in practice. For the two class case it is sometimes possible to provide more appropriate cost values, though this is typically difficult. When there are more than two classes, it is extremely difficult, and usually impossible, to provide realistic assessments of the relative severity of the different kinds of misclassification which can arise.. Because of these problems, one frequently adopted measure of performance for the two class case is the area under the ROC curve. This is equivalent to the probability that a randomly chosen member of one class has a smaller estimated probability of belonging to the other class than has a randomly chosen member of the other class, and so is a natural measure of separability between the two estimated probability distributions. This measure has the singular merit that it is independent of costs, priors, or (consequently) any classification threshold. There is a huge literature associated with ROC curves, spread across medicine, statistics, chemistry, psychology, and, most recently, machine learning and data mining.. When expressed in the form of the area under the ROC curve, the measure has no obvious generalisation to multiple classes. However, when expressed in its equivalent probabilistic form, it has a straightforward generalisation obtained by aggregation over all pairs of classes. This reduces to the standard form in the case of two classes, and so is a natural generalisation.. We showed, in an artificial situation, that the generalised form can behave in a way similar to standard measures such as error rate, but the fact that it is independent of costs and priors means that it is tapping different aspects of performance, and so will not always behave in the same way. To illustrate this, we evaluated the performance of three classification methods on eight real data sets, using the generalised measure developed above, and also two versions of error rate. In half of the cases that we investigated (which were chosen purely on the basis of the number of classes), the generalised measure produced a different rank order for the three classification methods.

O O O O O O O O B I O O O O O O O O O O B I O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O B I I B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Passengers

information

in

public

transport

and

privacy:

Can

anonymous

tickets

prevent

tracking?

Access

to

the

full

text

of

the

published

version

may

require

a

subscription.©

Elsevier

Ltd.

This

manuscript

version

is

made

available

under

the

CC-BY-NC-ND

license

Item

downloaded

from

public

transportation

companies

often

record

large

amounts

of

information.

Privacy

can

be

safeguarded

by

discarding

nominal

tickets,

or

introducing

anonymization

techniques.

But

is

anonymity

at

all

possible

when

everything

is

recorded?

In

this

paper

we

discuss

travel

information

management

in

the

public

transport

scenario

and

we

present

a

revealing

case

study

(relative

to

the

city

of

Cesena,

Italy),

showing

that

even

anonymous

bus

tickets

may

betray

a

user's

privacy

expectations.

We

also

propose

a

number

of

recommendations

for

the

design

and

management

of

public

transport

information

systems,

aimed

at

preserving

the

users'

privacy,

while

retaining

the

useful

analysis

features

enabled

by

the

e-ticketing

technology.

In

East

Japan

Railway

(JR

East),

the

largest

rail

company

in

the

country,

announced

the

intention

to

sell

to

Hitachi

corporation

a

large

dataset

of

its

passengers'

travel

histories

(Geuss,

This

information

has

been

gathered

by

JR

East

through

its

proprietary

e-ticketing

system,

Suica.

The

company

plans

to

anonymize

these

data

by

replacing

sensitive

information,

such

as

names

and

addresses

of

card

owners,

with

anonymous

ID's.

But

is

this

enough

to

protect

users'

identities,

and

therefore

their

privacy?

Historically,

public

releases

of

anonymized

personal

information

have

often

proved

to

be

dangerous

for

the

privacy

of

the

people

that

information

concerned.

In

America

On

Line

(AOL)

released

anonymized

data

regarding

the

search

queries

of

millions

of

users

of

its

web

search

engine.

Even

if

the

IP

adresses

of

the

users

were

replaced

by

anonymous

identifiers,

researchers

and

even

journalists

had

little

trouble

finding

the

real

names

of

people

corresponding

to

the

anonymous

ID's,

as

proved

by

the

famous

case

of

user

presented

in

a

New

York

Times

article

(Barbaro

and

Zeller,

In

Corresponding

author.

E-mail

address:

luca.calderoni@unibo.it

this

paper

we

show

how

the

disclosure

of

travel

histories

can

be

equally

dangerous,

as

travel

data

contain

a

great

deal

of

information

about

the

user,

even

when

her

real

identity

is

concealed

by

means

of

anonymization..

The

use

of

electronic

tickets,

usually

smart

cards,

in

public

transportation

networks

has

a

number

of

potential

benefits,

both

for

the

users

and

the

provider

of

the

transportation

service.

Often,

their

introduction

also

coincides

with

a

more

general

modernization

of

the

transportation

infrastructure.

Modern

networks

usually

integrate

a

positioning

system

(GPS)

for

monitoring

the

movements

of

buses

and

trams,

backed

by

a

constant

Internet

connection

to

a

central

control

infrastructure.

Enabling

location-awareness

allows,

for

instance,

to

display

real

time

information

and

waiting

times

for

each

line

on

the

provider's

website

or

on

information

screens

at

bus

stops

and

represent

a

value-added

city-to-citizen

service

in

the

smart

urban

ecosystem

(Calderoni

et

al.,

Internet

communication

between

vehicles

and

a

central

server

can

also

be

used

to

signal

traffic

congestion

or

unexpected

issues

efficiently,

in

both

directions.

These

innovations

help

in

making

our

cities

smarter

and

greener,

by

improving

the

quality

and

reliability

of

the

public

transport

service..

However,

the

technology

enabling

these

features

also

generates

an

unprecedented

amount

of

information

regarding

user

movements.

And

after

such

information

is

generated,

the

tendency

among

public

transportation

companies

is

to

record

it,

rather

than

discard

it

when

it

exhausted

its

original

goal.

In

this

paper

we

discuss

the

privacy

implications

of

such

a

large

amount

of

data,

and

we

analyze

the

potential

consequences

of

its

disclosure.

As

electronic

tickets

are

generally

characterized

by

a

unique

ID,

and

all

trips

are

recorded,

the

information

stored

in

the

information

system

of

a

public

transport

company

is

nothing

less

than

a

detailed

log

of

each

user's

movements

and

therefore

should

be

treated

as

sensitive

information.

However,

in

this

paper

we

show

that

even

when

personal

information

of

the

users

are

not

stored

in

the

system

-or

are

anonymized

-the

threat

to

privacy

remains.

In

fact,

combining

the

data

in

the

transportation

company

database

with

other

publicly

available

source

of

information

can

ultimately

be

enough

to

identify

a

specific

user,

even

in

the

case

of

anonymous

tickets.

The

amount

of

intelligent

technology

that

we

use

every

day,

knowingly

or

not,

has

seen

in

recent

years

an

exponential

increase.

Sensors,

smart

cards

and

many

other

intelligent

devices

are

now

deployed

all

around

us,

and

take

an

active

part

in

shaping

our

lifestyles.

The

amount

of

information

generated

or

collected

by

these

devices

is

enormous,

and

the

fall

in

prices

and

great

increase

in

capability

of

storage

devices,

made

possible

what

would

have

been

unimaginable

before:

keeping

this

information

indefinitely.

While

these

technological

developments

greatly

benefit

our

life,

we

are

still

only

beginning

to

grasp

their

side

effects.

The

most

common

and

obvious

concern

is

about

privacy:

what

control

do

we

have

on

our

personal

information,

once

it

has

entered

the

system?

Can

we

really

remain

anonymous

if

we

want

to?.

In

this

paper

we

analyze

the

risks

of

collecting

and

storing

passengers

information

in

the

context

of

public

transport.

We

not

only

show

that

real

anonymity

is

almost

impossible

to

achieve

even

when

using

anonymous

passes,

but

we

also

show

how

the

actual

amount

of

data

needed

to

identify

(with

some

statistical

error

margin)

and

track

a

user

is

surprisingly

low.

In

order

to

show

the

realism

of

this

threat,

we

perform

a

detailed

hardware

analysis

of

the

smart

card

tickets

used

by

the

transportation

company

Start

Romagna

(active

in

the

city

of

Cesena,

Italy),

and

we

identify

the

information

stored

in

the

chip

after

each

use

of

anonymous,

disposable

tickets.

This

allows

us

to

learn

what

information

is

collected

at

each

validation

and,

since

these

data

are

then

sent

to

the

company

databases

and

stored,

to

have

a

clear

picture

of

what

information

is

available

to

the

public

transportation

company..

In

light

of

this

knowledge,

we

analyze

the

possible

uses

of

this

information:

we

discuss

both

the

legitimate

goals

of

monitoring

traffic

in

order

to

improve

the

service

and

collect

statistics

by

the

company

and

the

malicious

tracking

capabilities

of

an

evil

actor.

We

also

analyze

the

current

implementation

of

the

information

system

that

collects,

stores

and

processes

this

information,

and

its

privacy

implications.

Since

the

aim

of

this

study

is

to

improve

the

state

of

the

art

and

build

privacy-preserving

public

transport

information

systems,

we

propose

management

recommendations

and

system

models

that

can

achieve

the

same

legitimate

goals

in

a

more

privacy-friendly

way,

without

any

need

of

modifying

the

current

infrastructure..

While

the

threat

we

discuss

in

this

work

is

specific

to

the

infrastructure

of

public

transport

in

Cesena,

this

experiment

brings

to

light

problems

that

are

common

to

all

e-ticketing

systems

that

store

travel

histories.

By

showing

how

to

de-anonymize

travel

histories

of

the

passengers,

we

stress

how

important

it

is

to

get

prior

informed

consent

of

the

users

before

disclosing

or

selling

even

anonymized

extracts

of

such

data.

Having

been

able

to

recognize

users

by

comparing

only

to

limited,

publicly

accessible

information,

we

also

pose

the

problem

of

what

could

be

actually

achieved

in

case

the

same

analysis

was

to

be

performed

by

companies

or

other

entities

that

have

also

access

to

data

generated

by

social

media

(social

networks,

location-aware

applications,

etc.).

We

can

only

speculate

how

big

a

threat

to

users'

privacy

the

unchecked

sale

of

travel

histories

might

become.

As

discussed

by

Diaz

and

Gürses

it

is

often

difficult

for

individuals

to

know

how

their

personal

data

are

used

by

companies

that

hold

them.

While

Diaz

and

Gurses

mostly

focus

on

sensitive

data

as

defined

by

European

Union

regulations,

their

reasoning

is

equally

valid

when

applied

to

companies

collecting

location

data

such

as

travel

histories,

as

in

the

case

of

public

transportation

companies.

In

fact,

users

are

often

unaware

of

the

risks

of

malicious

surveillance,

profiling

or

manipulation

they

are

exposed

to

(Avoine

et

al.,

The

security

of

this

information

is

therefore

best

assured

adopting

the

Privacy

by

Design

paradigm,

i.e.

providing

data

anonymity

by

designing

the

appropriate

protocols

and

procedures

as

hard-coded

in

the

system

itself

(Diaz

and

Gürses,

The

privacy-friendliness

of

the

infrastructure,

if

correctly

implemented,

does

not

necessarily

hinder

the

business

model

(Liu

et

al.,

In

the

case

of

public

transportation,

electronic

tickets

raise

privacy

concerns

for

their

ability

to

track

users.

Recent

studies

on

the

subject

discuss

this

issue

from

the

point

of

view

of

security

against

external

attackers

(Asadpour

and

Dashti,

Avoine

et

al.,

Heydt-Benjamin

et

al.,

Sadeghi

et

al.,

A

typical

attacker

is

therefore

some

unauthorized

person

trying

to

monitor

the

movements

of

a

victim,

for

instance

Pricing

information

should

be

stored

for

billing

purposes

only

and

thus

they

should

be

deleted

after

each

invoice

issuance..

For

billing

purposes

personal

information

of

the

user

might

be

needed.

However,

the

short-lived

time

required

to

perform

these

operations

reduces

the

privacy

implications.

Moreover

the

user

is

generally

well-aware

of

the

company

disposal

of

his

personal

information

required

for

billing

(address,

credit

card

data,

as

he

provided

them

himself..

Discounts

according

to

well-defined

categories

(students,

elderly,

people

with

disabilities

O O O B I O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B I O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Using multiple offspring sampling to guide genetic algorithms to solve permutation problems The correct choice of an evolutionary algorithm, a genetic representation for the problem being solved (as well as their associated variation operators) and the appropriate values for the parameters of the algorithm is a hard task and it is often considered as an optimization problem itself.In this contribution, we propose a new theoretical formalism, called Multiple Offspring Sampling (MOS). This new technique combines different evolutionary approaches taking advantage of the benefits provided by each of them. MOS dynamically balances the participation of different mechanisms to spawn the new offspring population, according to the benefits provided by each of them in previous generations. This approach evaluates multiple offspring generation methods (for example different coding strategies), and configures appropriate sampling sizes.This formalism has been applied to a well-known permutation problem, the traveling salesman problem (TSP). The results on several instances of this problem show that most of the combined techniques outperform the results obtained by single ones. Categories and Subject DescriptorsMOS could be defined as a mechanism to create new individuals, i.e., (a) a particular evolutionary algorithm model, (b) with an appropriate coding, (c) using specific operators (if required), and (d) configured with its necessary parameters.In our algorithm we have redefined the way the offspring and recombination of populations phases of a traditional Evolutionary Algorithm are performed. MOS proposes the definition of multiple mechanisms to generate new individuals, and makes them compete Copyright is held by the author/owner(s). GECCO'08, July 12-16, 2008, Atlanta, Georgia, USA. ACM 978-1-60558-130-9/08/07. during the evolutionary process. Each mechanism creates its own offspring O (j) i (i is the generation and j is the mechanism). The fitness of the individuals generated is used to evaluate the quality of each recombination technique. The most obvious measure that could be used for this purpose is the average fitness of the population, but more sophisticated measures can be proposed to take into account not only the current performance of the technique but its potentiality. This quality measure dynamically determines the participation of each of the available techniques.We have tested our approach with three standard datasets for the TSP that can be downloaded from the TSPLIB repository.Equation 1 The experimentation carried out in this study tries to confirm that MOS approach of combining different genetic codings and variation operators, under some circumstances, can improve the performance of these codings and operators when used separately.To test our approach, we have considered two different genetic codings: path and ordinal representation which are used simultaneously through all the execution of the algorithm. Both codings allow different variation (crossover and mutation) operators. For ordinal representation, traditional 1-point crossover and uniform mutation described by [3] have been used. For path representation we have considered two crossover operators: Order Crossover (OX) by [2] and Cycle Crossover (CX) [4] ; as well as two mutation operators: Repeated Exchange Mutation (REM) by [1] and Simple Inversion Mutation (SIM) by [3] .With all these alternatives we have constructed the following set of techniques: (a) t0: Integer Coding + OX + REM (b) t1: Integer Coding + CX + SIM (c) t2: Real Coding + 1-point Crossover + Uniform Mutation (d) t3: Integer Coding + OX + SIM (e) t4: Integer Coding + CX + REM.The results of this experimentation are too extensive to be reported in this paper. Instead of providing a large table with the results of each combination of techniques we propose a mechanism to rank the results obtained for each combination. The procedure to deal with this analysis, for each of the problems, is the following:1. all the combinations of techniques are considered, 1119 The general schema of an evolutionary technique carries out an iterative loop of (i) creation of new individuals (offspring spawning), (ii) evaluation of these new individuals, and (iii) selection of the new population from the individuals in the previous and offspring populations. In this contribution we have explored the possibility of using more than one mechanism to create new individuals. This possibility allows us to introduce Multiple Offspring Sampling, a theoretical framework that provides the tools to deal with multiple individual creation methods, even if different coding formats or representations are involved.. MOS is used in this paper to solve one of the most representative permutation problems, the TSP, using a parallel asynchronous genetic algorithm as the evolutionary model. From the results discussed in section 2 we can see that the combination of several offspring techniques tend to improve the results of the genetic algorithms. We can also observe that the combination of techniques reporting the best performance in different TSP instances is not always the same: different instances present different exploratory characteristics that can be exploited by different subsets of techniques. In general, the most suitable individual techniques usually appear in the best combination but, in some cases, the inclusion of an average technique can remarkably improve the overall performance.. Further work will include a deeper analysis of the influence of local search mechanisms in the performance of single techniques and combination of techniques. It would be also interesting to experiment with combinations of techniques where some of them only include one or none variation operators. Finally, we could think of a scenario where the number of available techniques is bigger than the number of techniques being used simultaneously and the algorithm could exchange the worse performing ones by some of the waiting techniques or, even, generate new techniques from the available ones.

O B I I O O B I O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O


Empirical study on the maintainability of Web applications: Model-driven Engineering vs Code-centric Abstract Model-driven Engineering (MDE) approaches are often acknowledged to improve the maintainability of the resulting applications. However, there is a scarcity of empirical evidence that backs their claimed benefits and limitations with respect to code-centric approaches. The purpose of this paper is to compare the performance and satisfaction of junior software maintainers while executing maintainability tasks on Web applications with two different development approaches, one being OOH4RIA, a model-driven approach, and the other being a code-centric approach based on Visual Studio .NET and the Agile Unified Process. We have conducted a quasi-experiment with 27 graduated students from the University of Alicante. They were randomly divided into two groups, and each group was assigned to a different Web application on which they performed a set of maintainability tasks. The results show that maintaining Web applications with OOH4RIA clearly improves the performance of subjects. It also tips the satisfaction balance in favor of OOH4RIA, although not significantly. Model-driven development methods seem to improve both the developers' objective performance and subjective opinions on ease of use of the method. This notwithstanding, further experimentation is needed to be able to generalize the results to different populations, methods, languages and tools, different domains and different application sizes. Model-driven Engineering (MDE) is a software development paradigm that heavily relies on models and model transformations for the development, maintenance and evolution of software (Mellor et al. 2003; Muñoz and Pelechano 2005; Mens and Van Gorp 2006) . The MDE community claims several advantages over traditional development (code-centric) approaches, among which (a) short and long term productivity gains, (b) improved project communication and (c) improved quality of the resulting application are specially relevant (López et al. 2006; Gustavsson et al. 2007; Heijstek and Chaudron 2009) . The reason for such claims is that current MDE approaches offer high-level abstractions that capture some of the most salient characteristics of modern applications. Such abstractions speed up the definition of models that can be used to generate applications for different implementation environments.. In particular, software maintenance is one of the areas in which MDE claims to be able to make a greater impact in terms of both reliability and efficiency (Ameller et al. 2010) . Software maintenance is defined in the IEEE 1219 Standard (IEEE Std.1219 -1998 1998 as "the modification of a software product after being delivered to correct faults, improve performance or other attributes, or to adapt the product to a modified environment". More concisely, the ISO/IEC 9126 standard (ISO/IEC 9126-1 2001) states that maintainability is "the capability of the software product to be modified". 1 Software maintenance is important because: (a) it consumes between 45 and 60 % of the overall life cycle costs and, (b) the inability to change software quickly and reliably means that business opportunities may be lost (Mens et al. 2010) .. However, practitioners still lack a body of practical evidence that soundly backs the maintainability advantages claimed by the MDE community (Heijstek and Chaudron 2009; Mohagheghi and Conradi 2004) . This contrasts with other disciplines and even other areas of Software Engineering (SE) in which empirical evidence is common (Zelkowitz 2009). Without empirical evidence, there is a danger for resources to be wasted and for software tools to fail to develop appropriately (Hutchinson et al. 2011a) . Such evidence can be provided in the shape of empirical studies, which are crucial to the evaluation of processes and human-based activities. The empirical studies can be classified into surveys, experiments (be them true experiments or quasi-experiments), case studies and postmortem analyses (Wohlin et al. 2000) . Experimentation, specifically, provides a systematic, disciplined, quantifiable and controlled way of evaluating human-based activities.. In an effort to increase the empirical evidence regarding the impact of MDE approaches on maintainability, this paper presents a quasi-experiment in which two groups of junior developers have been asked to perform a set of maintainability tasks on two Web applications with two different approaches: one code-centric and one model-driven.. The paper is structured as follows: Section 2 characterizes the maintainability concept. Both the maintainability characterization and the empirical evidence found in literature set the context for the definition of the experimental design (context, planning, operation and data collection mechanisms) in Section 3. Section 4 presents the data analysis and an interpretation of results that takes into account the identified threats to validity. Section 5 presents the existing empirical evidence regarding how maintainability is impacted by the adoption of MDE methods, and discusses the relationship and results of this experiment with respect to a previous one by the same authors. Last, Section 6 concludes the paper and presents some further lines of research. This study compares a model-driven approach, OOH4RIA, and a code-driven approach, based on Visual Studio and .NET. The study concludes that the use of OOH4RIA-as compared to .NET-greatly improves the actual performance (AEffv and, even more, AEffc) of subjects carrying out maintainability tasks. The study of the satisfaction-related variables (PEU and PU), however, throws mixed results: while PU-PEffv, PU-PCert, PEU-PEffc and PEU-PCompl do reveal significant differences among methods (all in favor of OOH4RIA), PU-PStab, PEU-PLearn, PEU-PSatisf and global PEU don't reveal significant differences between the two methods.. The significant differences all agree with previous results (see Section 5). These results already suggested that, in an experimental setting such as ours, the MDE approach would outrank the code-centric approach both in terms of efficiency (actual and perceived), effectiveness (actual and perceived). Also, in these four variables the Meth variable shows the greater effect size. The significant differences for PCert are also consistent with our a-priori thoughts. Models provide higher-level abstractions that are assumed to improve the understandability of the domain, including the understandability of the maintainability changes.. However, we also expected significant differences in the variables that actually did not attain significance. Users found OOH4RIA solution to be slightly less stable (meaning that they fear that more unnoticed errors may have been introduced), although the difference was not significant. This happens even though OOH4RIA includes a checking mechanism that is aimed at helping developers to avoid the introduction of unnoticed errors (a mechanism that was not available for the .NET treatment). We believe that this may be due to the perceived loss of control with MDE approaches that subjects have reported in a related study (Martínez et al. 2012b) . Also, subjects believe that OOH4RIA is slightly more learnable and less complex than .NET (although, again, not significantly). We expected that, having all of them a much higher experience with .NET than with OOH4RIA, they would find .NET to be more learnable and less complex. We can partly explain this fact by the subjects' knowledge of UML, which apparently was strong enough so as not to feel discomfort with the OOH4RIA approach. Also, the use of models -whose higher level of abstraction is supposed to simplify the maintenance tasks-seems to counterbalance the additional complexity that is likely to be perceived when the developer is faced with a new method. Also, the fact that the subjects were junior developers attending a master where they were being exposed to many different tools and techniques, may have had an impact on such perception of complexity. Last but not least, the subjects reported a PSatisf level slightly lower for OOH4RIA than for .NET (although, again, not significantly). We expected subjects to be significantly more satisfied with OOH4RIA, given the expected gains in efficiency and effectiveness. Again, based on self-reported opinions disclosed in a previous experiment (Martínez et al. 2012b) , we believe that this lower level of satisfaction (despite the increased objective and subjective productivity gains) may be due to code-centric practices being more consistent with the existing values, needs and past experiences of subjects, which suggests the convenience to introduce this variable in future replications of the experiment.. Since it is well known that it is subjective opinions (and not objective behavior) what better determines the intention to adopt new technologies and methods (Moore and Benbasat 1991) , it is not far-fetched to think that the discrete satisfaction results reflected in our data (with four variables being significant and three variables not being significant) may be at the root of the low adoption rates of the MDE paradigm beyond university courses (Walderhaug et al. 2008) . Actually checking whether this hypothesis is true requires from further experimentation.. Another important contribution of this article is the validation, through a PCA, of the structure of the theoretical model. Our analysis shows that the number of components and the assignment of measures to model variables is consistent with the data gathered in our context, and therefore increases the confidence in the goodness of fit of the chosen theoretical model to the context of method adoption.. Our results augment the repository of empirical data comparing maintainability performance and satisfaction of MDE methods with respect to traditional codecentric approaches. Further experimentation is needed to be able to generalize the results to a different population, different methods and languages, different application types, different maintainability tasks or different application sizes. In this sense, we have prepared a replica, and we plan to run a meta-analysis that includes the results from the WebML experiment, the experiment reported in this paper and the results from that replica. Also, extensive work needs to be done in the matter of defining reliable measures. Last but not least, more data is needed to perform a much more extensive validation of the theoretical model. In this paper we have just checked the soundness of the groupings of the measures under the PEU and PU variables. However, we have left out some constructs of interest (namely, the intention to use the method and how it relates to the actual use of the method once the subjects go back to their daily activities), and we have not checked the causal relationships inferred from that model. (OOPSLA, WISE, ER, EC-Web, ICWE, CADUI, etc.) . He regularly serves in the PC of several international conferences (WWW, ICWE, JISBD) and he has co-organized during three years the international workshop MDWE (2011, 2012 and 2013) . More detailed information could be found at: http://www.dlsi.ua.es/∼santi/ The importance and impact of maintainability on software development is reflected in the vast amount of empirical work found in literature. Regarding code-centric approaches, there are empirical studies which provide evidences about the impact of the quality of object-oriented implementation over the software maintainability. Dagpinar and Jahnke (2003) demonstrates that object-oriented metrics such as size, coupling, cohesion, inheritance, etc. can effectively be used to predict the maintainability of software systems. The results indicate that size and direct/import coupling metrics are significant predictors for measuring maintainability of classes while inheritance, cohesion and indirect/export coupling measures are not.. Other studies are focused on how software development problems and good practices may affect software maintainability. On the one hand, Chen and Huang (2009) classifies a set of problem factors related with software development that may affect the maintainability. These include documentation quality problems, programming quality problems, system requirements problems, personnel resources problems and process management problems. Results suggest that improvements in the software development process can help to reduce the level of severity of the problems related with documentation quality and process management, which in turn may enhance software maintainability. On the other side, Banker et al. (1998) estimates the impact of the development activities with a more practical perspective on the software maintainability. This research indicates that the software complexity is a key intermediate variable that links design and development decisions to their downstream effects on software maintenance. Interestingly, the experiment results indicate that the use of an MDE method in development is associated with increased software complexity and reduced software development effort. These results suggest an important link between software development practices and maintenance performance.. Also, according to a systematic mapping of the empirical evidence of the impact of MDE on the different ISO quality characteristics (Martinez et al. 2011b) , the empirical assessment of maintainability activities while using MDE methods has been gaining momentum during the last years. However, the empirical evidence regarding this topic is still scarce: out of the 599 publications included in the study, only 14 (that is, barely a 2.34 %) focused on maintainability. The available evidence states that, in academic settings, it takes around 37 % less time to evaluate the impact of a change with an MDE method than directly over the code (Mellegå rd and Staron 2010). In industry, evidence has been gathered on how the use of MDE causes defect reductions, reduced need for code inspections, and produces maintenance gains, although such evidence is anecdotal (Mohagheghi and Dehlen 2008) . In Hutchinson et al. (2011b) , based on the results of an online survey, the authors report how the majority of respondents (between 58 and 66 %) considered the use of MDE on their projects to be beneficial in terms of personal and team productivity, maintainability and portability. Organizationally, the benefits of MDE have been defined in terms of communication and control: according to the same survey, almost 75 % of respondents agreed that using MDE made them faster at implementing new requirements, while nearly two thirds of respondents reported that their use of MDE helped in the understanding and communication of organizational knowledge among stakeholders.. Last but not least, a recent empirical study (Martinez et al. 2011a ) compared an MDE method, WebML (Ceri et al. 2002) , and a code-centric method (based on PHP), with respect to the performance and satisfaction of junior software developers while executing analyzability, corrective and perfective maintainability tasks on Web applications. The experiment presented in this paper is a follow-up of the WebML empirical study. Both studies provide interesting results on the impact of MDE approaches on the maintainability of Web applications. Also, in the experiment presented in this paper we have maintained the paradigms (model-driven vs codecentric). However, we have changed the methods (now OOH4RIA and .NET). Also, in this study we have centered on changeability -leaving out analyzability-, and we have designed more tasks that provide a finer-grained measure for the maintainability sub-characteristic. Besides, we have measured much more extensively the developer's perceptions, based on an underlying theoretical model. Last but not least, whereas the WebML experiment is mainly focused on the use of the navigational model for obtaining the client-side of a Web application, this work is centered on the definition of the domain model to generate the server-side, that is, the business logic and the persistence layer of a Web application. Comparing the results regarding the productivity of the changeability and the satisfaction of each MDE approach versus the code-centric approach, we can observe how both WebML and OOH4RIA significantly improve the actual efficiency of the changeability tasks (3.21 and 317 times, respectively) with respect to the code-centric approach. However, while OOH4RIA also improves the effectiveness of the changeability by up to 27 %, the impact of WebML over effectiveness was not found significant. With regard to the results on the satisfaction of use of these approaches, the WebML study showed a slight preference of subjects towards tackling maintainability tasks directly over the source code. This result is consistent with the results of the OOH4RIA experiment.

O O O O B O O O O O O O O O O B O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O


Comparing the early research performance of PhD graduates in labor economics in Europe and the USA Abstract This paper analyzes the early research performance of PhD graduates in labor economics, addressing the following questions: Are there major productivity differences between graduates from American and European institutions? If so, how relevant is the quality of the training received (i.e. ranking of institution and supervisor) and the research environment in the subsequent job placement institution? The population under study consists of labor economics PhD graduates who received their degree in the years 2000-2005 in Europe or the USA. Research productivity is evaluated alternatively as the number of publications or the quality-adjusted number of publications of an individual. When restricting the analysis to the number of publications, results suggest a higher productivity by graduates from European universities than from USA universities, but this difference vanishes when accounting for the quality of the publication. The results also indicate that graduates placed at American institutions, in particular top ones, are likely to publish more quality-adjusted articles than their European counterparts. This may be because, when hired, they already have several good acceptances or because of more focused research efforts and clearer career incentives. Predicting the early academic success of PhD graduates by looking at their research productivity or the quality of their job placement has been an issue of interest in economics. Studies on this subject are most often based on the idea that the research performance of the young PhD graduate is determined by the quality of the academic training received: namely, the quality of the institution awarding the PhD or the research productivity of the supervisor. Examples of such studies include Ruane and Tol (2009) , Athey et al. (2007) , Grove and Wu (2007) , Hilmer and Hilmer (2007) and Krueger and Wu (2000) .. A different line of literature has been concerned with the identification of policies that can promote research excellence. Part of this literature is motivated by the low European research performance in economics when compared to the USA. Such studies invariably emphasize the relevance of incentives, both at individual and department levels, and the need to promote profound institutional reforms in most European countries (see for example Drèze and Estevan 2007) .. This study considers these two lines of literature, assessing the role of two different types of determinants of research performance of recent PhD graduates: the quality of the academic training received versus the institutional setting of the job placement institution upon completion of the PhD. Analyzing the productivity of PhD graduates by taking into consideration only their academic background seems an incomplete view, given the literature on the relevance of the institutional setting and career incentives for the promotion of research quality. Anecdotal evidence can illustrate this point: two equally talented students receiving their PhD degree from the same institution and at the same time, often perform subsequently quite differently in terms of research output, depending on the country or institution where they are placed and the conditions inducing or hindering quality research. Smeets et al. (2006) is among the few studies we could find that consider the interaction of these two factors when studying the research productivity of recent economics PhD graduates. Long et al. (2009) concentrate on a different scientific fieldinformation systems-and aim precisely at evaluating the relative importance of academic origin versus current affiliation on researcher productivity. Their results cast serious doubt on the relevance of the academic origin as an indicator of research potential and the authors advise academic departments to place less emphasis on the reputation of the PhD institution when making recruitment decisions. Our work enables testing whether such a strong result also holds for other scientific fields. Moreover, our methodology refines Long et al.'s approach, in the sense that we control for a wider set of factors that may have an impact on researcher productivity and we consider a wider set of journals.. We concentrate on PhD graduates in labor economics who received their degree in 2000-2005, from a European or USA university. Identification of the population under study relied on two sources: Dissertations Abstracts, a database covering every dissertation defended at an American accredited institution, at fifty British and a few other European institutions; the files of the IZA European Summer in Labor Economics, containing detailed information on both successful and unsuccessful applicants from all over Europe. Research productivity is evaluated alternatively as the raw number of publications or the quality-adjusted number of publications of an individual.. Section ''European versus North-American research productivity'' overviews the literature that has dealt with the comparison of research productivity in Europe and the USA. Section ''Labor versus other fields in economics'' places labor economics in the framework of the overall economics field. Section ''Dataset'' provides a detailed description of the dataset and section ''Descriptive statistics'' presents descriptive statistics. The model under estimation is presented in section ''Research performance of American and European graduates'' and its results are discussed. The final section concludes. We have analyzed differences in research productivity of recent PhD graduates from European and USA universities. The received wisdom is that research based in the USA dominates in quantity and quality. We find that this is no longer true at this simplistic level. At first sight it even seems that European graduates are the more productive. However, once we account for the quality of scientific journals, we find that differences between graduates from American and European institutions vanish. In terms of output quality, results indicate that graduates placed at American institutions, in particular top ones, are likely to publish more quality-adjusted articles than their European counterparts.. This may be because, when hired by a top-USA institution, graduates already have several good acceptances or because of more focused research efforts and clearer career incentives. This could also be because the American-based institutions still provide a better academic environment to reach the standard leading journals, which are still closer attached to the USA. Another explanation is that the rising quantity and the increased visibility the European researchers have provided over recent years needs more time to breed higher quality research and lead to high quality journal publications. Part of this is related to the fact that the very many new European-based journals where Europeans publish more than proportionally need many more years to move up the quality ladder in international competition.. Publication strategies also seem to differ across both continents. In the USA, PhD graduates aim at a decent number of articles in ''A-journals'' before the tenure decision; the probability of acceptance in those journals is low, and hence delays to acceptance and publication of papers are longer. In Europe, young researchers often publish before defending their PhD; journals are not necessarily A-ranked; and the rank of the journals (although of rising importance) is not yet so crucial for the tenure decision, which often follows other rules.. In terms of policy conclusions there is still a ''clear need to develop a coherent strategy to improve doctoral training in economics Europe'' that would ''counteract the strong tendency for the best European economists and students to gravitate towards that country [the USA]'' (Kirman and Dahl 1994, p. 520) . Drèze and Estevan (2007) had similarly mentioned the ''brain drain'', as the USA manages to retain its brains and attract foreign ones, stressing the role played in this process by its PhD programs. While there seems still much to do to improve the academic incentives in European PhD programs, the success in the catching-up process in the past decades is encouraging.

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O