# Crowdsourced Evaluation of Robot Programming Environments: Methodology and Application
## 3 Materials and Methods
### Amazon mTurk  
Owned by Amazon and operated under Amazon Web Services (AWS), mTurk [42] was launched in 2008 and is currently one of the largest operating micro-task sites [43]. It consists of numerous listed custom webpages nested within the platform, each of them consisting of a unique task. Some tasks may require workers to interact with web pages outside of the mTurk platform. Tasks are called \"human intelligence tasks\" (HITs) and are issued by unique users registered as so-called requesters. A requester can issue any number of different or the same tasks and can set certain filters to only allow suitable workers access to them. In order to become an official requester, typical account registration steps need to be carried out. The first one needs to be authenticated by providing the full name and additional institution and how often one plans on using the platform. After this, a credit card needs to be added and subsequently verified by sending and receiving a certain small amount of money. The entire workflow is depicted in Figure 4.  
1. Starting from the lower left circle, marked with \"R\" for \"Requester\", the work is published on the crowdsourcing platform in the form of a HIT. Requesters are usually companies, research institutions, or any other individual offering the digital task. 2. mTurk [42] (marked with \"MT\") is the crowdsourcing platform of choice, which hosts and shows all available assignments to all potential mTurk workers. mTurk grants the \"R\" high-level control of who can see or accept their tasks by keeping a set of metrics for each registered worker. There are three system qualifications available free of cost: 1. Number of HITs approved This is the number of HITs that one worker has successfully completed since their registration on mTurk. This is the base rate that requesters use to select workers with existing experience. According to [22], higher than the 90% mark provides a good rule of thumb. 2. HIT approval rate This is the rate of accepted and completed HITs. For example, if a worker has completed 1000 HITs, out of which 150 were rejected, the worker's approval rate is 85%. Experienced requesters report that the combination of these two qualifications usually delivers results of significantly higher quality. 3. Location This qualification is important if a requester needs to filter workers by geographical location, which the workers usually specify in their profile. The location filter can be used to include or exclude workers from a specific continent, country, county, or even city. Figure 4: mTurk process workflow based on [12]. Additionally, the possibility of setting up two premium or entirely customized qualifications exists. These kinds of qualifications induce additional charges and include age, gender, frequency of participation in the platform, or the type of smartphone owned by workers (important for smartphone software usability testing) [42]. 3. To complete the first cycle, the mTurk workers (marked with \"W\"), on their own initiative, browse and decide to accept the offered assignment. \"R\" is notified about every assignment acceptance or rejection. 4. \"W\" starts solving the assignment immediately or according to the given time frame. 5. Subsequently, \"W\" uploads its results to the platform. 6. \"R\" is notified about the uploaded solution and has a limited, previously arranged, timeframe for revision. 7. The requester then has two options: 1. In case of approved assignment, \"R\" sends the agreed payment to \"W\", with the non-obligatory possibility of offering and later paying the bonus, and also pays the fee to \"MT\" for providing the platform. 2. The other possibility is rejecting the assignment, which can only be authorized by \"MT\" if a valid argument for doing so is offered. Next on the list is HIT creation. For this, numerous mTurk templates already exist. However, they are adapted to different use case scenarios (predominantly surveys, image or language recognition, or data collection templates).