# Crowdsourced Evaluation of Robot Programming Environments: Methodology and Application
## 6 Study Results
### Statistical Findings across All Groups  
This section focuses on general statistical findings across all mTurk worker groups, which we did not find essential to observe separately per group. Furthermore, direct comparison between them is what makes these findings valuable. Note that we did not have such data for the pilot study.  
Figure 8 depicts overall SUS scores for the student results, the accepted mTurk worker results, and the rejected mTurk worker results. The graphic shows a difference of almost 20% between the results of the rejected HIT workers and those of the pilot study. At the same time, the difference between accepted and rejected HIT workers was comparatively small, amounting to only 3.13%.  
This point raises at least two interesting questions:  
1. Could the SUS evaluation results be considered relevant even if they originate from mTurk rejected worker's input?
2. Is there an observable pattern sample in rejected workers' SUS evaluation results that can lead to early scammer recognition?  
The first question is beyond the scope of this paper. The answer to the second question, which will be discussed later in the paper, also provides a partial answer to the first one.  
Figure 8: SUS scores across all groups.  
The average HIT approval rate is rather low, at 24.79%. This is something that needs to be considered when opting for crowdsourcing. The approval rate also implies a certain amount of time that needs to be invested in making the acceptance or rejection decision for each HIT. An individual analysis of each HIT is unavoidable if the researcher wants to make sure that all the input is valid.  
Figure 9 focuses on the comparative difficulty level rating for accepted and rejected HITs. In terms of difficulty, the rejected workers rated all of the use cases more highly than the approved workers. N/A answers were considered in this rating as well. In our opinion, this measure is directly linked with the participants' previous experience with programming in general and, more specifically, with robotics. However, analyzing worker inputs on these two criteria turned out to be contradictory, since the rejected workers claimed to have considerably more experience in both of these fields, as is visible from Figure 10. In our understanding, this is likely to be the result of workers' dishonesty.  
Another measurement that, in comparison, turned out as expected was the average duration for completing the assignment. The workers were restricted to a maximum of 2 h for providing their results. However, since the data here were not scaled (there were 59 workers more in the rejected group than in the accepted), the trend clearly shows that the approved workers on average took more than twice the time that the rejected workers did, which implies that they dealt with the tasks in greater depth.  
When observing individual solving times in more detail, data scattering among rejected workers stands out. In the left part of Figure 11, completion times vary from a couple of minutes to the full 2 h. Similar to the conclusion in [32], we assume that the disparity in overall task completion times and results is a consequence of some participants skimming over the instructions or not reading them at all.  
Figure 10: Previous knowledge in programming and familiarity with robotics among approved vs. rejected workers.  
Figure 9: Use case difficulty evaluation among approved vs. rejected workers.  
We consider it rather inconvenient that the individual task resolution time cannot be deduced from the automatic mTurk output.