# Crowdsourced Evaluation of Robot Programming Environments: Methodology and Application
## 5 Application of the Framework
### Task Specification: Evaluation of Web-Based Robotic Programming Interfaces
#### 5.2.2 Use Case Definition  
It was necessary to keep in mind the low payment, small amount of time on disposal, low attention span, no moderating possibilities, and general complexity of the topic. When defining the use cases, the requester must keep a series of constraints in mind concerning the relatively small payment and short time that workers dedicate to a hit as well as the short attention span of the majority of worker and the limited possibilities of moderation. These constraints must be placed in balance with the complexity of the task so as to obtain the most valuable output from as few use cases as possible. To achieve this goal, a gradual increase in use case difficulty is necessary. In doing so, the solvability rate (i.e., how many participants can solve the most complicated task) must be monitored. It is to be assumed that not all workers will be able to solve all use cases. However, the tasks and post-task surveys must be designed in such a way that the workers will be compelled to provide reasons for not doing so. Rejected HITs are just as important as the approved ones because they can provide useful insights, which can be used to adjust the difficulty of the tasks. According to the International Federation of Robotics, _pick and place_ is the most common application scenario for industrial robots. _Pick and place_ is used in tasks such as loading and unloading machines, placing parts in a grid, or picking bulk material from a box with or without a camera support. The aim of the use cases was to start from basic tasks, such as moving a robotic arm in a certain direction, to more complex tasks, such as _pick_ and place. Based on these principles, six use cases have been designed in order to evaluate the efficiency and usability of _Assembly_, which were included in the HIT instructions. HIT Instructions  
The step-by-step textual instructions, which did not contain any images, explained how to use the robot simulator and how manipulate objects. This text could be reviewed at any point during the user study. During task execution, no means of support were offered. If a technical issue with mTurk occurs, the Amazon team is responsible for its resolution. Any issues with _Assembly_ encountered by the workers were likely to provide useful insights about the program being evaluated and were documented in the qualitative analysis part. The _Assembly_ tool did not provide any indications on which workers were supposed to deliver in each HIT. Moreover, explicit success or failure messages were not implemented in the interface. This deferred the decision concerning whether a certain HIT was completed or whether they should continue to the next use case to the worker. Although efforts were made to keep the instructions concise and informative, some workers did not follow them. As visible in Figure 7, there were 3 tabs to be completed by the requester, namely _Summary_, _Detailed instructions_, and _Examples_. Workers were provided with the following instructions in the second tab:  
Make sure you have a computer that supports the Google Chrome browser, and please use it for this survey. Before starting to solve each and every use case, make sure you click on the dashboard icon on the lower right part of the screen to show automatically generated JS code. Please read each use case carefully and remember to take a screenshot accordingly. (The following sentence was set as a reminder after every use case description: Make a screenshot of the current screen (with the visible code on the dashboard) and upload it. Use Case Instructions  
The program is drag-and-drop-based. The units used for the (x,y,z) coordinates are millimeters. The units used for the rotation angles along the spatial axes (RX, ry, rz) are degrees. By the workflow, the task line on the top starting with \"start\" and \"stop\" actors is meant. Useful hint: note that when you put your mouse over any component, a small \"Help\" window will appear. Use Case 1: Robot Motion  
* Create a program that performs a robot motion between two points.