The Role of Information Theory and Queuing Theory in Human Computation Systems

Although human computation has a long history [1], [2], crowd-based collective intelligence systems have flourished of late and now form the basis for many large-scale businesses, social enterprises, and citizen science consortia [3]–[5]. Microtask platforms with small tasks such as image labeling are perhaps most familiar, but there has been a proliferation of multifarious human computation models used in large scale by different platforms [6], [7]. Upwork operates as a freelance market and had 2.5 million workers and nearly 0.5 million clients in 2013 (then oDesk) [7]; an internal IBM crowdsourcing contest platform for software engineering drew on the company’s nearly 0.5 million workers in 2013 [8]; and the microtask platform Amazon Mechanical Turk has had transaction value in the hundreds of millions of dollars per year. These varying, oft-heuristic system design models suggest myriad possible tradeoffs among basic resources like time, task accuracy, human attentional cost, communication bandwidth, and human enjoyment. To engineering systems theorists like us, this zoo of design possibilities raises the natural question: are there fundamental limits and optimal designs for human computation? After all, the attention and cognitive energy of people are valuable resources that should be used as efficiently and effectively as possible. Theoretical foundations to understand what is possible and what is impossible, together with constructive approaches that achieve or approach these fundamental limits, are needed to push systems beyond heuristic and empirically-motivated designs. Previous studies on the mathematical foundations of human computation have drawn on game theory and mechanism design to understand strategic human behavior in human computation systems, e.g. [9]; machine learning theory to understand how to control quality when the outcomes of crowdsourcing are used for training algorithms, e.g. [10]; and theory of algorithms to improve efficiency of complicated crowd work, e.g. [11]. In this short position paper, we argue that information theory and queuing theory are also natural tools for studying certain fundamental limits of human computation systems. Such approaches can establish the capacity of human-based systems to perform intelligent work, in a form which may be computed from the properties of the people participating in them. To demonstrate our approach, we discuss establishing limits of two types of crowdsourcing frameworks: • skill-agnostic microtasking, where simple jobs are performed by crowd workers for a small payment, and • skill-based crowdsourcing, where jobs can only be performed by workers with requisite skills. A main challenge for skill-agnostic microtasking is the unreliability of random crowd workers, whereas for skillbased platforms it is the dynamic, time-varying, and random (un)availability of skilled workers. Information theory is well-suited to understand the limits of reliable computation in a system constructed from unreliable elements. On the other hand, queuing theory is naturally suited for studying limits of systems with dynamic and random resource availability. We also briefly discuss how certain important crowdsourcing scenarios necessitate a joint informationand queuing-theoretic treatment, leading to a union of these two mathematical fields in a completely novel way.