It's getting crowded!: improving the effectiveness of microtask crowdsourcing

Microtask crowdsouring has emerged as an excellent means to acquire human input on demand, and has found widespread application in solving a variety of problems. Popular examples include surveys, content creation, acquisition of image annotations, etc. However, there are a number of challenges that need to be overcome to realize the true potential of this paradigm. With an aim to improve the effectiveness of microtask crowdsourcing, we identify three main challenges to address within the scope of this thesis. The first challenge is the limited understanding of crowdsourced tasks and crowd worker characteristics. Understanding the dynamics of tasks that are crowdsourced and the behavior of workers contributing to tasks, can play a vital role in effective task design. Secondly, current worker pre-selection mechanisms are simplistic and inadequate. It is challenging to recruit workers with desirable skills in the absence of historical knowledge regarding the quality of work produced. There is a need for stronger indicators of worker competence and more effective pre-selection mechanisms. Finally, there has been an incomplete consideration of factors that influence and shape the quality of work produced in crowdsourced microtasks. It is important to fully understand different aspects that influence crowd work. In this dissertation, we tackle the aforementioned challenges and propose novel methods to overcome existing problems in each case. Our main contributions are described below. • Advancing the Current Understanding of Task Types, Worker Behavior and Quality Control — Our findings from an extensive study of 1,000 workers on CrowdFlower advance the current understanding of crowdsourced microtasks and corresponding worker behavior. We propose a two-level categorization scheme for microtasks and revealed insights into the task affinity of workers, effort exerted to complete tasks of various types, and worker satisfaction with the monetary incentives. We analyze the prevalent malicious activity on crowdsourcing platforms and study the behavior exhibited by trustworthy and untrustworthy workers, particularly on crowdsourced surveys. To improve the overall quality of results, we propose behavioral metrics that can be used to measure and counter undesirable or potentially malicious activity in crowdsourced tasks. Considering these aspects, we prescribe guidelines for the effective design of crowdsourced tasks. • Novel Mechanisms for Worker Pre-selection — We propose two distinct and novel methods for worker pre-selection that outperform state-ofthe-art approaches across different types of tasks. First, we define a data-driven worker typology. By relying on low-level behavioral traces of workers, we propose behavioral features for worker modeling and preselection. Our findings bear important implications for crowdsourcing systems where a worker’s behavioral type is unknown prior to participation. Next, we make a case for competence-based pre-selection in crowdsourcing marketplaces. We show the implications of flawed selfassessments on real-world microtasks, and propose a novel worker preselection method that considers accuracy of worker self-assessments. Our results confirm that requesters in crowdsourcing platforms can greatly benefit by additionally considering the accuracy of worker selfassessments in the pre-screening phase. • Revealing Hidden Factors that Affect Crowd Work: The Cases of Task Clarity and Work Environments — Workers of microtask crowdsourcing marketplaces strive to find a balance between the need for monetary income and the need for high reputation. Such balance is often threatened by poorly formulated tasks, as workers attempt their execution despite a sub-optimal understanding of the work to be done. We unearth the role of task clarity as a characterising property of tasks in crowdsourcing, and propose a novel model for task clarity based on the goal and role clarity constructs. We reveal that task clarity is coherently perceived by crowd workers, and is affected by the type of the task. We then propose a set of features to capture task clarity, and used the acquired labels to train and validate a supervised machine learning model for task clarity prediction. Another aspect that has remained largely invisible in microtask crowdsourcing is that of work environments ; defined as the hardware and software affordances at the disposal of crowd workers which are used to complete microtasks on crowdsourcing platforms. Through multiple studies, we reveal the significant role of work environments in the shaping of crowd work. Our findings indicate that crowd workers are embedded in a variety of work environments which influence the quality of work produced. Depending on the design of UI elements in microtasks, we found that some work environments support crowd workers more than others.