} Efficiency Speed Fun Satisfaction Social Reward Utility Beauty Clarity Learnability Reliability

Crowdsourcing, along with much of the Internet, only works when humans find tasks fun, enjoyable, or valuable enough to outweigh the time and effort they require to complete. The more utility that humans find in a task and interface, the more “work” they will do. However, we do not yet know how to objectively measure the fun, enjoyment, or value of a user interface applied to a particular task. My research empirically measures the economic utility, or “user preference,” of a user interface for a task by putting multiple versions of a user interface together with a task on Mechanical Turk and measuring the amount of money required to convince humans to use them. INTRODUCTION The ESP game [6] makes image labeling fun. Wikipedia works by making it easy to edit pages and contribute to a body of knowledge. Facebook and Twitter work by making it easy and rewarding to upload your life online for others to see. In general, Crowdsourcing and other humancomputer interactions only work when people like, prefer, and ultimately choose to use our interfaces and tasks out of the myriad options available to them. My research economically measures and quantifies the amount an interface motivates (or demotivates) a user to use it for a task, by putting the interface and task in a crowdsourced labor market and measuring the amount of money you must pay people to use it. Thus, this work relates to crowdsourcing in two ways: (1) it evaluates the most critical quality of a crowdsourcing user interface—its ability to convince a human to use it, and (2) it uses a crowdsourced method to infer this metric. More specifically, this work operationalizes the economic definition of utility, and applies it to Human-Computer Interactions. In Economics, utility is the degree to which a person prefers a particular choice amongst options [7]. We can infer it from user behavior: when a user chooses to use system A instead of B, it is said that Utility(A) > Utility(B). Utility encompasses all factors of function and usability that affect preference and use, and measures their net impact on user behavior (Figure 1). Economic utility quantifies user preference. To quantify the utility difference of two interfaces, we vary the wage we offer, and find the amount that compensates for a difference in use between the two interfaces. If a user has no preference between being paid 25¢ for using system A over being paid 50¢ to use system B, then we can describe the difference Utility(A)–Utility(B) = Utility(25¢). That is, system A provides a measure of 25¢ more utility than system B. This is a money-metric of utility, a number representing the value of an interface change that can be compared across interfaces and situations, providing a lingua franca for communicating results. CASE STUDIES Figures 2 and 3 illustrate a study measuring the utility of a standard HCI phenomenon: Fitts‘ Law [2]. Fitts’ Law models the difficulty of target acquisition as a function of target size and distance. To test our methodology, we ran a study online to determine the utility of Fitts’ Law—how target size and distance affect user preference. We posted the standard Fitts’ law task to Mechanical Turk, asking workers to click back and forth between a rectangle that switched sides on the screen. Our experiment manipulated the task’s index of difficulty, by changing the size of the rectangle and its distance from the user’s cursor (see Figure 2). We expected users to prefer easy tasks to difficult tasks, and in fact the data displays this trend (see Figure 3). That is, the degree a Mechanical Turk user prefers a Fitts’ law task is inversely proportional to the time it takes to use it. With further experiments, one could learn if this preference result generalizes, and perhaps produce a general law of user preference with respect to task-completion time. Such a law would be useful knowledge for designers. This shows that we can vary an interface and quantify its effect on user preference. Labor Supply Curves of Interaction Utility However, if we also vary the price we pay workers on Mechanical Turk, we can view preference economically, and express it in terms of dollars and cents. This is illustrated in Figure 4. In our Fitts’ Law study, we varied the amount of money we paid workers along with the index of difficulty. With six prices and three indices of difficulty, we created a total of eighteen experimental conditions. This gives us enough data to induce labor supply curves for the interface variations: graphs that show how much work you can expect the average worker on Mechanical Turk to produce with the interface variations, given varying levels of pay. With these curves, we can apply economic analyses to our interface. For instance, we can infer a money-metric of utility—quantifying the value or cost of an interface variation in terms of dollars and cents. To do so, we simply fix the number of jobs (the Y axis) to a single number, and measure the distance (in pay, on the X axis) between the two curves for two interface variations. On the other hand, if we fix instead a single value of pay on X axis, we can deduce the amount of work that a change in interface produces. 1 Figure 1. Utility is a summative descriptor of users ̓ decisions of use. Many lower-level factors affect a userʼs choices of use. Utility aggregates the effects of all lower-level factors into a single