## 5 Discussion and Conclusions  
Performance metrics have been identified as having a promising role in better assessing the quality of solutions provided by evolutionary algorithms and perform better in comparison with them, thus becoming a key ingredient to support the preferences of decision-makers. In this paper, the aim was to show the current practices or how the SBSE practitioners used these metrics. It is believed such discoveries will eventually highlight the possible sets of metrics, objective functions, or new areas in SE to explore in the future. To achieve this, we carried out a systematic review with a guided protocol to carefully (systematically) plan, collect, and present the dominant results, in detail. We technically defined the relevant research questions to answer and also conducted a manual search from a set of digital libraries to select the candidate papers. Inclusion and exclusion criteria were applied, and finally, the desired extracted data were stored in Excel worksheets. We then discussed the outcomes using tables and graphs to better digest the data. As a result, the final 105 relevant publications revealed that there are (based on the groupings) several studies that employed zero metrics (solver metrics). In the SBSE community, it is preferable to use more metrics, and it is worth noting that only four studies employed six metrics, and four others used five metrics. In addition, the analysis also discovered the number of sets of metrics used in the studies and their ranks over the study groups. To this effect, HV was the most widely used individual performance metric, while for groups, HV, IGD, GD, and the hypercube-based diversity metric are top-ranked, respectively (they had frequency scores of more 10). On the other hand, there is increasing interest in the community in re-formulating MaOPs with more than three objectives, additionally, software testing was the most applied area in software engineering. Furthermore, we addressed some of the open issues found in our study, and they are mainly related to these three main areas: Performance metrics, number of objectives, and SE application areas. All the issues related to these should be addressed in the future. Performance metric: We found that there are remarkable studies that did not employ performance metrics, while those that used two metrics increased in number, and the remaining studies, specifically in sets M3, M4, M5, and M6, used more diverse metrics. However, in the literature, most researchers did not agree to evaluate their algorithms based on a high number of performance metrics, but they agreed that no single performance metric alone can assess all the qualities of the solution sets, since each metric can only be targeted to evaluate a single or two desired properties. Figure 14: SE applied areas over the study groups. Another issue is metrics preference among researchers. We observed that some of the studies justify the reason they employ these metrics as either based on a metric's popularity (i.e., usage or related work in which it was used) or if it best fits their choice of algorithm (i.e., References [6, 86, 87]), while some other studies adopted some of these metrics because they are hybrids. For example, HV can cover both convergence and diversity [87], while some others avoided using more metrics because that might have led to different conclusions or threatened the validity of their results [122]. Thus, such gaps will remain in their future work [122]. Some avoid these metrics, which would be a visible gap in their future work [75]. With regard to such practice, it is also clear that the use of performance metrics has received little attention. Since the current practice is dominated using 0 to 2 metrics, hence such comparison might be unfair. Thus, we recommend employing more diverse sets of metrics, since they have been found to be low in quantity in current practice.