## 5 Evaluation
### Multi-Case Evaluation of DSML4 PT
#### 5.2.3 Analysis  
Quantitative Analysis  
To find answers for RQ1 and RQ2, this analysis has two parts: measuring the generation of the public transportation applications (called generation performance), and the time saved during the entire development process (development time performance), respectively. _Generation Performance_: Since we aim to determine whether DSM4 PT is feasible to create public transportation applications considering RQ1, comparisons were made between the automatically generated code and the delta code added to complete the applications for all case studies. Performance evaluation was performed by comparing the percentage of automatically generated and manually developed software code. Figure 8 shows the bar chart of the Lines of Code (LoC) from the case studies perspective. The LoCs listed for each case study are the average of all developers in the evaluator group. For example, all evaluators developed 79.11% of the final code for the Validator use case by using DSM4 PT. After this, evaluators added manual code, 20.89% of the final code. All measurement results for each developer and for each case study can be found in [98]. Figure 8: DSM4 PT’s LoC generation performance. As can be seen from Figure 8, the overall average LoC is 79.56%, i.e., 4 out of every 5 LoCs are automatically generated with DSML4 PT. It allows us to say that this rate of code generation performance facilitates software development, and a significant part of the code is produced just by using DSML4 PT. The rate of generated code for the case studies ranges from approximately 72% to 84%. This variation primarily arises from the differences in both the complexity and the representability of the public transportation systems inside the DSML4 PT IDE for each case study. System components, including IoT elements and their relations, required to construct the system differ for each case study. For instance, most of the software required for the passenger count system was automatically generated from all developers' models since design with DSML4 PT succeeded in creating rich models for IoT, public transport, and cloud service elements and their relations. On the other hand, generated LoCs for the advertising management (case study #2) and passenger information (case study #5) applications were relatively lower since they were dependent on using external libraries and a set of hardware configuration files that cannot be modeled with DSML4 PT. Moreover, variety in both the domain knowledge and experience of the developers could have affected the quality of the models they created as well as the output of the code generation process. However, to keep this effect to a minimum, we conducted this multi-case evaluation with varying public transportation development complexities instead of only one use case. _Development Time Performance_: Time measurements and analyses were made to see the development time effects of using DSML4 PT. To measure the times recorded during development efforts, MDE of the applications using the DSML4 PT and its IDE (Experiment A) and development of the same applications without using MDE and DSML4 PT (Experiment B) were considered. The experimental results were compared and analyzed. As previously indicated, during Experiment B, the developers applied the current software development methodologies with which they were familiar. Development times were recorded for all developers, as were all steps of software development processes (discussed in Section 5.2.2) required for each case study of IoT-based public transportation systems development. The mean time for these steps was calculated separately for both experiments. The results from Experiments A and B are presented in Figure 9 for the comparison. Moreover, average times of system development for each specific use case elapsed for these experiments are also listed in Table 3. Times measured for each developer's software development processes can be found in [98].