## 3 Research Methodology
### Research Questions  
To define what we are trying to answer, it is essential to design our research questions (RQs) at the studies that quantitatively evaluate the Pareto-based methods using performance metrics, number of objectives, and applied SE areas, thus, we consider the following RQs:  
RQ1: What are the studies that applied none or one or more performance metrics?  
To answer this RQ, we investigate the number of performance metrics that the SBSE community employed over the years, we aim to check the number of performance metrics each study employed by adopting a grouping strategy. This discovery will help the practitioners to understand how the existed studies measuring the quality of the solution sets obtained by the solvers.  
RQ2: What metrics are most or least used in the studies?  
In this question, we aim to identify the metrics that reported mostly or least used. From this point of discovery, tallying, and grouping (adopting previous grouping strategy), the different set of metrics in each group and their frequencies are discussed.  
RQ3: What is the rank order of the metrics most or least used in the studies?  
To see how the overall metrics and their ranks, we calculate their total frequencies. In RQ2, the overall rank of these metrics were not discussed. However, in this RQ, we intended to identify the metric frequencies and group them using their frequencies and calculate their percentages. In this case, we also avoid using the previous grouping strategy.  
RQ4: How do the top popular metrics (>5%) increase or decrease in the studies?  
It is beneficial to see how a set of metrics distributed in the study group (adopting previous grouping strategy), especially those gained more than five percent. This investigation will help us to increase our knowledge about how a set of metrics become increasingly popular or decreased in the study groups.  
RQ5: How well do the current studies in SBSE use performance metrics?  
In this RQ, we also identify and further investigate the study groups by showing the total number of studies in each study group, their total number of unique metrics, and their total frequency metric.  
RQ6: What are the number of objectives used in the studies?  
In this aspect, we show the number of objectives the community employing in practice by grouping the studies based on the objective count. This will discover the current practice of SBSE practitioners and the future direction of the research.  
RQ7: What are the applied areas in SE of the studies?  
In this RQ, we investigate the most and least common investigated software engineering (SE) areas by showing the studies' distribution among these. Previously software testing was nominated, but recently, many areas in SE were investigated in the SBSE community. In this case, we also adopt the grouping made by the previous studies [19; 20]. This might help and lead the current practitioners to further investigate these areas.  
RQ8: What are the performance metrics distribution in each SE applied area?  
In this RQ, we also identify how the previously investigated performance metrics (based on the grouping strategy) are distributed on the applied SE areas. This will help the SBSE practitioners to understand which SE areas are employing more or fewer metrics.  
Answering the above RQs will help the practitioners to understand how studies have measured the quality of the solution sets obtained by solvers. There may be philosophical ideas among members of SE communities that might be revealed through answering these RQs, thus giving them meaning. To address the scope of the RQs, we limit the research papers published over years, which ranges from 2000-2020. While, we selected the publications only in relation to SBSE, especially those involving the multi/many-objective methods and their employed performance metrics, and number of objectives utilized in their evaluation setups and in applied areas of SE.