# A Choreography-Based and Collaborative Road Mobility System for L'Aquila City
## 4 Case Study
### Experimental Evaluation  
Our approach has been evaluated by conducting an experiment for the CRMS use case introduced above. It is worth noticing that the work described in this paper is in the domain of software engineering for service-oriented distributed systems. Thus, the experiment that we carried out follows widely established guidelines for experimental software engineering that are described in [58]. The experiment that we describe in the following serves to evaluate the goodness of the CHOReVOLUTION approach with respect to using a traditional development method. This evaluation is based on comparing the development effort required by using the CHOReVOLUTION IDE with the development effort required by manually implementing the entire CRMS choreography from scratch. The development effort is measured in terms of person hours (ph) of development activities. Thus, the main goal of the experiment is to assess the benefit of the proposed approach against a set of Key Performance Indicators (KPIs) and hence to measure the time saving for realizing the use case with our approach when compared to the development approaches our Master Degree students have been learning during software engineering courses. The considered KPIs are:  
* **Effort for implementing the (possibly) distributed workflow (coordination logic)**: effort to develop the distributed coordination logic that guarantees the collaboration among the involved concrete services;
* **Effort for implementing the missing logic (business logic)**: effort to implement the missing business logic beyond the one implemented by the reused services;  
Figure 8: Smart Data Collector architecture. * **Effort for integrating third-party or legacy services**: effort required to interconnect heterogeneous third-party or legacy services to fit the (possibly) distributed workflow specification. The KPIs have been measured during the implementation phase (**Hypothesis**), which consists of the development of the choreography-based system from scratch. The time saving is measured in terms of person-hour (ph). In particular, regarding the CRMS use case, we employed the following two experimental units. **Experimental unit 1:**_CHOReVOLUTION approach_--full usage of the CHOReVOLUTION IDRE except for the development of the mobile application, which is out of the scope. **Experimental unit 2:**_General-purpose enterprise-oriented technology_--full usage of the technologies daily adopted by the students, i.e., JAX-WS and JAX-RS frameworks together with other supporting standards and libraries such as Java Persistence API, Jackson, etc., and Eclipse as IDE. Regarding the efficient usage of the CHOReVOLUTION technologies within the experimental unit 1, the involved students spent six hours of training during dedicated lectures in a master degree course on advanced service-oriented software engineering. The technologies of the experimental units 2 for the CRMS use case were selected considering that the students were already familiar and skilled with these technologies. It is clear that there exist many other equivalently powerful alternatives. However, opting for an alternative would have required a training effort that could not be afforded by the students because of exam time constraints. In any case, apart from time constraints and assuming the possibility to choose for an alternative, we can argue that, however (reasonably) long the training could last, it would be not easy to reach the same level of expertise, hence possibly compromising the validity of the experiment. The considered KPIs are mapped to the following experimental tasks. In particular, the effort for implementing the distributed workflow is mapped to the **coordination logic** implementation, whereas the effort for implementing the missing logic is mapped to the **prosumer services** implementation. Finally, the effort for integrating third-party or legacy services is mapped to the **adaptation logic** implementation. Both experiments were conducted by two different development teams.