# Crowdsourced Evaluation of Robot Programming Environments: Methodology and Application
## 2 Related Work: Crowdsourcing in the Context of User Studies  
The term crowdsourcing was coined quite far back, in 2006, with [PERSON] official definition of it being \"the act of a company or institution taking a function once performed by employees and outsourcing it to an undefined (and generally large) network of people in the form of an open call [10]\". Such a network of people is quite distinct from the actual internal employees of a given company since it can be composed of potentially any individual with access to the Internet. The members of the \"crowd\" may need to fulfil a set of requirements but are largely free to work under their own conditions. Usually, crowd members work alone and do not know the other members of the same crowd. Workers typically receive small remunerations for their work, if the results are acceptable. This way, anyone can outsource any task for the completion of which they do not have internal resources in terms of time and/or expertise. Based on this emerging phenomenon, many web-based platforms and applications were created to host the solution. In the robotics field, crowdsourcing has received relatively little attention as a technology evaluation method. In this section, we review some of the approaches that use crowdsourcing in the context of user studies and technology evaluation, which reach beyond the field of robotics. In terms of user experience (UX) evaluation, one can distinguish the following types. * _Traditional user studies_: These studies are being conducted in the field or in the lab, requiring both study participants and evaluators to come to a certain location. This obviously takes more time to perform and means higher costs, since the required effort is higher, having a lower number of participants as a consequence. Applying this in the context of evaluation of robotic programming interfaces, additional limitations in terms of finding knowledgeable or voluntary study participants, robot access, safety, and platform availability emerge. As the authors of [11] define it, the typical research cycle passes through four main stages, starting with the method formulation, method implementation, conducting the user study, and, finally, presentation of the results. The last step is where the next delay due to lack of time mostly happens, because neither the presented results nor the lessons learned are significant if not implemented into the final product [11]. * _Crowdsourced user studies_: Following [11], in order to progress in robotics, it is indispensable to build professional-grade robots and create user evaluation research appropriate for and available to the masses. Conducting web evaluations seems an obvious answer to this issue, benefiting even more from the use of crowdsourcing platforms. The authors in [12] conducted an experimental investigation of two distinct crowdsourced design evaluation approaches, one being the usual free evaluation in the crowdsourcing environment and the other using specially developed crowdsourced design evaluation criteria (CDEC). The outcomes of both approaches were finally benchmarked against a standard expert evaluation with a board of experienced designers. It is interesting that the outcome suggests that their CDEC approach produces design rankings with a strong correlation with those of the expert board. Their paper [12] assessment methodology demonstrates how crowdsourcing can be effectively used to evaluate, as well as generate, new design solutions. This paper followed their methodology, which needed to be adapted to the UI evaluation instead of design. [PERSON] et al. [13] have conducted a crowdsourced study aimed at exploring whether non-experts can successfully use off-the-shelf robot programming tools. These authors found that, with minimal instruction, the study participants were able to create simple programs using an off-the-shelf visual programming interface, a conclusion that is in line with other approaches to democratizing cobot programming [14; 15; 16].