# Crowdsourced Evaluation of Robot Programming Environments: Methodology and Application
## 4 Crowdsourced Evaluation Methodology  
The two most important techniques for evaluating a UI are heuristic and empirical evaluation [7]. The former represents a theoretical basis built upon a predetermined set of rules aimed at improving the usability of UIs. The latter is more practical and implies testing with real users. This paper reports on user studies conducted via crowdsourcing for the purpose of evaluating the usability of the _Assembly_ programming environment. This first evaluation explores the potential and limits of the proposed methodology by showing how usability evaluations can effectively be performed using this method. [PERSON] [7] further differentiates between summative and formative usability evaluations. Summative evaluations produce quantitative, predominantly statistical data. Besides backing our findings via comparative analysis or performance measurement tests, quantitative data are useful, e.g., for identifying and rejecting outliers. In the case of crowdsourcing, this is likely caused by scammers. The results that are more than two standard deviations from the mean are usually excluded in human-computer interaction research [17]. On the other hand, the results of formative usability evaluation contribute via qualitative answers on how and why interface users experience a certain design or the entirety or chunks of the process flow. For this paper, both summative and formative usability evaluations are considered equally relevant and will be carried out. For the purpose of evaluating _Assembly_, six experimental tasks have been designed and published on mTurk. Drawing on [17], the tasks progress from elementary mechanical ones, such as moving the robot in any direction, which demands a small amount of mental effort and concentration, to more cognitively and time-demanding ones, such as creating an algorithm. Methodological Framework  
The basis for our framework is the iterative user-centered design (UCD) process for interactive systems described by the ISO standard 9241-210 [44]. As is visible from Figure 5, the following steps specified by the standard can be adhered to and re-iterated if necessary [44]:  
1. Firstly, the need for a specific UCD task is identified through input and requirements from various stakeholders. In parallel, the work on the UX is planned considering time and other resources. This paper had a clear UCD task based on _Assembly_. 2. Secondly, the context of use is specified based on user background research or other input. Since the main goal of the CoMeMak project is ensuring all interested stakeholders access to cobots in makerspaces and additionally addressing other issues regarding robots, the context of the use of _Assembly_ here is defined rather broadly, with user profiles being diverse and unspecified. The projected users of the _Assembly_ tool should have basic computer skills and technical understanding. These prerequisites are also required of mTurk workers in order for them to be able to navigate the platform. 3. The third step handles user requirement specification. It is suggested to define and document relevant use cases in order to model the specifications. The use cases should be initially simple to allow the user to get to know and use the program and then gradually upgrading the complexity. The business relevance should not be disregarded here. One strategy is to use simple tasks as a part of the assessment test to encourage only qualified users to participate in the entire crowdsourcing project and later raise the difficulty bar. If every participant can solve all the tasks, then the use cases should be made more difficult; otherwise, the results will not be within the scale. 4. The fourth step delivers design solutions, such as UX concepts, tutorials, interaction evaluations, and prototyping. 5. Finally, the UX design is evaluated by testers or real users. This paper will focus only on crowdsourced results. Normally, if the users' design or functional needs are met, the entire UCD process is done.