## 6 Evaluation Experiment of the Proposed Approach
### Qualitative Analysis of the Crowdsourced Requirements  
The crowdsourced requirements are qualitatively analyzed by experts using a Likert scale. The Likert scale is a commonly and widely utilized instrument for gathering subjective data such as opinions, perceptions and motivation and can be conducted with relative ease [89]. It is capable of analyzing values written in qualitative form such as requirements. It can transform qualitative values into their quantitative counterparts to facilitate sound statistical analysis. In addition, it measures to what level a participant agrees or disagrees with a certain statement [90]. To build the survey using a Likert scale, we first identify the constructs that will be measured based on analysis of the crowdsourced requirements. We consider five critical quality factors of requirements. These factors were selected since they fit the context of this study, where the requirements are crowdsourced. The five quality factors/constructs are clarity, diversity, relatedness, feasibility and creativity, each of which is defined as follows:  
**Clarity. ** Clarity means the requirements should be clear, understandable, comprehensible and involve no ambiguity [91]. Since requirements are written in natural language, it is inevitable that they are prone to unclearness and ambiguity [92]. In addition, in the context of our study, requirements elicitation is conducted through crowdsourcing, and the crowd have not undergone any training nor are they overseen during the elicitation process. Hence, there is a chance that requirements might be ambiguous. Due to that, we need to take the clarity of the written requirements and features into consideration. **Diversity. ** The diversity of requirements is an important factor to be considered when crowdsourcing requirements. [PERSON] and [PERSON] in [2] have mentioned that diversity in the crowd and requirements is crucial to consider when crowdsourcing RE. Therefore, by considering this factor we ensure that the elicited requirements and features are comprehensive and can meet the expectations of a wide audience. In addition, diversity can yield satisfying, new and unprecedented requirements and features. Hence, measuring such factors is important to find out to what extent crowdsourcing can boost diversity in the elicited requirements. **Relatedness. ** Relatedness is another critical factor that should be added to the set of quality factors. The relatedness and relevance of the contributions to the scope of the crowdsourced task have been measured by [PERSON] et al. [93] when crowdsourcing ideas to investigate the quality of crowd contributions. Moreover, this factor can also be considered in crowdsourcing requirements. Indeed, measuring how related the crowdsourced requirements are to fitness mobile apps is a strong indicator of the suitability of the crowd involved in crowdsourcing requirements elicitation. Receiving out of scope requirements and features implies that the selected crowd does not possess the required knowledge. **Creativity. ** Creativity has recently attracted considerable attention from the RE research community. Creative requirements can be defined as requirements that are novel and useful. Producing creative ideas that are not realistic provides no value. Moreover, creative requirements can either be generated by exploring and discovering other software areas, combining existing familiar requirements to come up with novel requirements or by changing current rules in a certain area to come up with novel requirements [94]. We are interested in evaluating the creativity of the crowdsourced requirements to gain insights about to what extent our proposed approach can help in capturing creative and novel mobile app requirements. **Feasibility. ** This factor is important to consider when assessing the quality of the requirements. Feasibility means the ability of a requirement to be transformed into a workable implementation with a reasonable budget, time and effort [95].