# Crowdsourced Evaluation of Robot Programming Environments: Methodology and Application
## 5 Application of the Framework
### Task Specification: Evaluation of Web-Based Robotic Programming Interfaces
#### 5.2.1 Test Run  
Before publishing the HIT, we worked with the mTurk developer requester sandbox, which is a simulated environment, where requesters can test their HITs prior to official publication in the marketplace. This is a free and useful feature provided by mTurk, allowing all requesters to test, practice, and redesign their surveys so as to make sure that all the tasks function correctly in advance. This saves time and money in preparation for the actual evaluation. In the case of evaluating _Assembly_ without a fixed framework, we needed exactly 5 test runs, all of which yielded important findings and lessons learned. The test runs were executed in the period between 8 December 2020 and 30 January 2021, offering fees in the range of USD 0.80 to USD 3.00.  
The importance of creating simple yet relevant use cases in order to obtain the user feedback was recognized from the first test run. Another immediate takeaway was the importance of creating required fields, since most of the workers left every optional field empty. Additionally, the significance of setting both overall and individual, clearly addressed and specific, evaluation criteria was recognized, reducing a Likert scale from 10 to 5 since it takes users a long time to fill out while it is not even the most important or reliable measure.  
Reading user feedback on this test run pointed to some required UI adaptations so that the additional explanations about how to use the tool or the necessary time for solvingthe HIT could be reduced, which in turn reduces the likelihood of workers giving up on the HIT with the argument that the program does not react as expected.  
For the 3 rd run, the same use cases were used as the ones in the TU Wien course \"_Montage II: Advanced manufacturing_\" [46]. As expected, these use cases were too complicated for the mTurk audience, which additionally had the disadvantage of no template solutions shown and a lack of expertise. Nevertheless, one worker managed to solve all the tasks, including the open-ended question, which asked the user to think of a way to solve a particular task rather than providing step-by-step instructions. Although this unexpected success is statistically insignificant, it demonstrated that it is possible to encounter skilled workers on mTurk. This run made clear the need to reduce the amount of data needed to provide a proof of work considering workers' comments and the time required to check the quality of the results.  
In the following test run, the _pick_ task was decoupled from the _place_ task to check which of them was easier for the mTurk workers. Moreover, the need to introduce standardized usability measurements became apparent in order to analyze the results more easily. The first choice was the Post-Study System Usability Questionnaire (PSSUQ). With 16 questions that can be answered with a 7-point Likert scale (N/A option) and three sub-scales evaluating system usefulness, information, and interface quality, this fine-grained usability test turned out to be too long for mTurk workers to fill out correctly and honestly.  
The next run implemented the System Usability Scale (SUS) usability survey instead of the PSSUQ. Concerning the use cases, the _pick_ task turned out to be easy to solve, so it was immediately expanded with the _place_ task (supposing the same difficulty level) and seamlessly extended with the _reuse_ task.