# Crowdsourced Evaluation of Robot Programming Environments: Methodology and Application
## 1 Introduction
### Problem Statement  
Currently, in the field of industrial robotics, the design and development of the robotic hardware has resulted in flexible machines with a high number of capabilities. As a result, relatively little attention has been granted to the underlying software components [4] thatanimate robots. Aspects such as a visual representation of programming and interaction tools have recently started to become \"first-class citizens\" in robotics research. Previous research [5] has shown that older industrial robots can almost exclusively be introduced and controlled via proprietary, text-based, and predominantly low-level programming languages, derived from Pascal and BASIC and created back in the early 1990s. These programming languages have historically been designed by engineers for their own peers, which inevitably implies that writing or even reading these programs requires corresponding schooling or years of education and training. This furthermore means that only a minority of small- and medium-sized enterprises will be able to benefit from robotic automation since they cannot afford such highly educated workers [6]. As programming becomes ubiquitous and programming skills are increasingly required of employees with low or no technical background [4], developing simple and user-friendly robot programming interfaces to ease the programming task for non-experts has become an essential branch of research. There are already many robot programming environments available, but none have yet been established as the indispensable standard. One of the particularities of UIs is that their validation cannot be carried out quantitatively and objectively--as opposed to other kinds of software, which can be validated through testing. Since such interfaces are designed to be used by humans, their evaluation must be carried out by their projected users. Graphical robot programming environments make no exception to this rule. Determining whether one programming environment is better than another is a matter of asking a group of users to try out the respective tools and then answer some questions. This methodology is part of the broader user studies field, which is the de facto standard in evaluating user interfaces. A major drawback of this approach is that established usability evaluation methods are both resource- and time-consuming [7] because they imply a series of qualitative process steps. First, user interface (UI) designers need to determine a relevant user group and approach a sufficient number of users from this group for the study to be relevant. The candidates must then be persuaded to participate in the study and supported throughout the process. Typically, user evaluations are performed in one location over 1-2 days, during which the researchers must accommodate and provide support to a high number of users (e.g., over 100 in the case of [4]). This can, for example, be achieved by inviting users to a designated location or by going to a specific location, where a pool of relevant users is available, such as a company [4]. One significant drawback of this approach is that the diversity within the user pool is inherently limited (i.e., the employees of a specific company, university students, workers of a specific factory, etc.). In response to these issues, this work explores a more cost-effective and faster solution to the problem of evaluating the usability of simplified robot programming environments. The proposed methodological framework draws on crowdsourcing the evaluation of web-based robotic programming environments. Crowdsourcing comprises a number of methods for dividing and distributing tasks to a dynamic and diverse group of participants. The distribution and monitoring of task execution is usually performed on the Internet with the help of specialized platforms, such as Amazon Mechanical Turk (mTurk). The participants are paid for carrying out the specified tasks.