bluuebunny/crossref_metadata_2025 huggingface dataset:
PAStE: A Platform for Adaptive Storytelling with Events
Authors: Shoulson Alexander , Kapadia Mubbasir , Badler Norman
Published: October 2022

We present ongoing work on PAStE, a system that supports a narrative director by describing a virtual world and managing the behavior of its characters. Rather than operating in the action domain of each individual actor, PAStE works at the scope of events -- pre-authored multi-actor sequences describing interactions between groups of actors and props. A director can select which event to perform, and populate that event with characters in the virtual world. Once an event is chosen, PAStE handles the event's execution with support for high-fidelity character animation on fully-featured 3D virtual humans. Events are an accessible tool for assembling narrative arcs, invoking contextual activities, and conceptualizing user intervention. We explore the possibilities presented by an event-centric behavior environment for conducting stories, and address some of the potential limitations.
Musical Meta-Creation: The Discourse of Process
Authors: Sigler Andie
Published: October 2022

Musical meta-creation is an artistic and technical fieldin which the results (i.e. music) often cannot be empirically tested. The processes by which music is generated,however, afford a rational level of discourse. This paper suggests ways in which we might orient our discussion and our system-building in order to explore musical possibility and use computational modeling to growinto a new kind of creative musical practice.
The Generative Electronic Dance Music Algorithmic System (GEDMAS)
Authors: Anderson Christopher , Eigenfeldt Arne , Pasquier Philippe
Published: October 2022

The Generative Electronic Dance Music Algorithmic System (GEDMAS) is a generative music system that composes full Electronic Dance Music (EDM) compositions.  The compositions are based on a corpus of transcribed musical data collected through a process of detailed human transcription.  This corpus data is used to analyze genre-specific characteristics associated with EDM styles. GEDMAS uses probabilistic and first order Markov chain models to generate song form structures, chord progressions, melodies and rhythms. The system is integrated with Ableton Live, and allows its user to select one or several songs from the corpus, and generate a 16 tracks/parts composition in a few clicks.
Towards Dynamically Crafted Affective Experiences Through Emotional Response Player Modelling
Authors: Nogueira Pedro
Published: October 2022

Designing adaptive games for individual emotional experiences is an elaborate task due to the necessary dedicated hardware, complex signal processing and unique emotional bonds each player may forge with the game. The research work I propose focuses on developing a novel affective interaction scheme where players’ emotional reactions are learned by a computational model that leverages this information to influence their future emotional states. To this end I aim at interpreting the causality relations between physiologically measured emotional response variations and their eliciting events. I then plan using these reactions to build each player’s individual emotional reaction model, which will guide the game engine’s logic on how to plan future game events, in order to elicit a target affective experience. Ultimately, I expect this technique to allow game designers to focus on defining high-level rules for generating their envisioned affective gameplay experience, rather than in manually tuning it in a repetitive and time-consuming fashion.
Narrative Extraction, Processing and Generation for Interactive Fiction and Computer Games
Authors: Valls-Vargas Josep
Published: October 2022

Often, computer games require meaningful stories and complex worlds in order to successfully engage players. Developing a high-quality story and rich characters can be one of the hardest tasks in the game development process. Narrative is a key element in building game worlds for interactive digital entertainment. I am particularly interested in computational narrative algorithms that can analyze stories, model narrative, and generate plots to be used in various forms and game domains.
Player Knowledge Modeling in Game Design Feedback and Automation
Authors: Butler Eric
Published: October 2022

Models that capture the knowledge of players of digital games could be used to great effect in AI-assisted tools that automate or provide feedback for game design. There are several important tasks knowledge models should perform: predicting player performance on a particular task to adjust difficulty, knowing in which order to give particular concepts for maximum learning, or understanding how the pacing of a concept impacts player engagement. While all of these have been explored individual both in games and related fields like intelligent tutoring systems, there have been no models that capture all of these effects together in a way that allows their use in design tools. We propose to expand on previous work in game authoring tools to create tools in which the designer can leverage information about how players learn their game's concepts to create better designs. We will survey the existing player modeling work to find the best representation for this task, deploy these models in adaptive games to learn from data, and then apply these models to create novel game design tools.
Ontological Knowledge for Goal-Driven Autonomy Agents in Starcraft
Authors: Dannenhauer Dustin
Published: October 2022

Starcraft, a commercial Real-Time Strategy (RTS) game that has enjoyed world-wide popularity (including televised professional matches), is a challenging domain for automated computer agents. Evidence of this difficulty comes not only from characteristics of the game (massive state space, stochastic actions, partial visibility, etc.) but also from three years of competitive entries in tournaments (i.e. AIIDE Annual Starcraft Competition) in which the best automated entry performs poorly against a human expert. We are interested in taking a new research direction: using semantic knowledge, such as description logic, to represent the game state with abstract concepts in order to perform high level actions.
Strategyproofing Peer Assessment via Partitioning: The Price in Terms of Evaluators’ Expertise
Authors: Dhull Komal , Jecmen Steven , Kothari Pravesh , Shah Nihar B.
Published: October 2022

Strategic behavior is a fundamental problem in a variety of real-world applications that require some form of peer assessment, such as peer grading of homeworks, grant proposal review, conference peer review of scientific papers, and peer assessment of employees in organizations. Since an individual's own work is in competition with the submissions they are evaluating, they may provide dishonest evaluations to increase the relative standing of their own submission. This issue is typically addressed by partitioning the individuals and assigning them to evaluate the work of only those from different subsets. Although this method ensures strategyproofness, each submission may require a different type of expertise for effective evaluation. In this paper, we focus on finding an assignment of evaluators to submissions that maximizes assigned evaluators' expertise subject to the constraint of strategyproofness. We analyze the price of strategyproofness: that is, the amount of compromise on the assigned evaluators' expertise required in order to get strategyproofness. We establish several polynomial-time algorithms for strategyproof assignment along with assignment-quality guarantees. Finally, we evaluate the methods on a dataset from conference peer review.
Taking Advice from (Dis)Similar Machines: The Impact of Human-Machine Similarity on Machine-Assisted Decision-Making
Authors: Grgić-Hlača Nina , Castelluccia Claude , Gummadi Krishna P.
Published: October 2022

Machine learning algorithms are increasingly used to assist human decision-making. When the goal of machine assistance is to improve the accuracy of human decisions, it might seem appealing to design ML algorithms that complement human knowledge. While neither the algorithm nor the human are perfectly accurate, one could expect that their complementary expertise might lead to improved outcomes. In this study, we demonstrate that in practice decision aids that are not complementary, but make errors similar to human ones may have their own benefits.

In a series of human-subject experiments with a total of 901 participants, we study how the similarity of human and machine errors influences human perceptions of and interactions with algorithmic decision aids. We find that (i) people perceive more similar decision aids as more useful, accurate, and predictable, and that (ii) people are more likely to take opposing advice from more similar decision aids, while (iii) decision aids that are less similar to humans have more opportunities to provide opposing advice, resulting in a higher influence on people’s decisions overall.
It Is like Finding a Polar Bear in the Savannah! Concept-Level AI Explanations with Analogical Inference from Commonsense Knowledge
Authors: He Gaole , Balayn Agathe , Buijsman Stefan , Yang Jie , Gadiraju Ujwal
Published: October 2022

With recent advances in explainable artificial intelligence (XAI), researchers have started to pay attention to concept-level explanations, which explain model predictions with a high level of abstraction. However, such explanations may be difficult to digest for laypeople due to the potential knowledge gap and the concomitant cognitive load. Inspired by recent work, we argue that analogy-based explanations composed of commonsense knowledge may be a potential solution to tackle this issue. In this paper, we propose analogical inference as a bridge to help end-users leverage their commonsense knowledge to better understand the concept-level explanations. Specifically, we design an effective analogy-based explanation generation method and collect 600 analogy-based explanations from 100 crowd workers. Furthermore, we propose a set of structured dimensions for the qualitative assessment of analogy-based explanations and conduct an empirical evaluation of the generated analogies with experts. Our findings reveal significant positive correlations between the qualitative dimensions of analogies and the perceived helpfulness of analogy-based explanations. These insights can inform the design of future methods for the generation of effective analogy-based explanations. We also find that the understanding of commonsense explanations varies with the experience of the recipient user, which points out the need for further work on personalization when leveraging commonsense explanations.
Crowdsourcing Perceptions of Gerrymandering
Authors: Kelly Benjamin , Kang Inwon , Xia Lirong
Published: October 2022

Gerrymandering is the manipulation of redistricting to influence the results of a set of elections for local representatives. Gerrymandering has the potential to drastically swing power in legislative bodies even with no change in a population’s political views. Identifying gerrymandering and measuring fairness using metrics of proposed district plans is a topic of current research, but there is less work on how such plans will be perceived by voters. Gathering data on such perceptions presents several challenges such as the ambiguous definitions of ‘fair’ and the complexity of real world geography and district plans. We present a dataset collected from an online crowdsourcing platform on a survey asking respondents to mark which of two maps of equal population distribution but different districts appear more ‘fair’ and the reasoning for their decision. We performed preliminary analysis on this data and identified which of several commonly suggested metrics are most predictive of the responses. We found that the maximum perimeter of any district was the most predictive metric, especially with participants who reported that they made their decision based on the shape of the districts.
When More Data Lead Us Astray: Active Data Acquisition in the Presence of Label Bias
Authors: Li Yunyi , De-Arteaga Maria , Saar-Tsechansky Maytal
Published: October 2022

An increased awareness concerning risks of algorithmic bias has driven a surge of efforts around bias mitigation strategies. A vast majority of the proposed approaches fall under one of two categories: (1) imposing algorithmic fairness constraints on predictive models, and (2) collecting additional training samples. Most recently and at the intersection of these two categories, methods that propose active learning under fairness constraints have been developed. However, proposed bias mitigation strategies typically overlook the bias presented in the observed labels. In this work, we study fairness considerations of active data collection strategies in the presence of label bias. We first present an overview of different types of label bias in the context of supervised learning systems. We then empirically show that, when overlooking label bias, collecting more data can aggravate bias, and imposing fairness constraints that rely on the observed labels in the data collection process may not address the problem. Our results illustrate the unintended consequences of deploying a model that attempts to mitigate a single type of bias while neglecting others, emphasizing the importance of explicitly differentiating between the types of bias that fairness-aware algorithms aim to address, and highlighting the risks of neglecting label bias during data collection.
Comparing Experts and Novices for AI Data Work: Insights on Allocating Human Intelligence to Design a Conversational Agent
Authors: Sun Lu , Liu Yuhan , Joseph Grace , Yu Zhou , Zhu Haiyi , Dow Steven P.
Published: October 2022

Many AI system designers grapple with how best to collect human input for different types of training data. Online crowds provide a cheap on-demand source of intelligence, but they often lack the expertise required in many domains. Experts offer tacit knowledge and more nuanced input, but they are harder to recruit. To explore this trade off, we compared novices and experts in terms of performance and perceptions on human intelligence tasks in the context of designing a text-based conversational agent. We developed a preliminary chatbot that simulates conversations with someone seeking mental health advice to help educate volunteer listeners at 7cups.com. We then recruited experienced listeners (domain experts) and MTurk novice workers (crowd workers) to conduct tasks to improve the chatbot with different levels of complexity. Novice crowds perform comparably to experts on tasks that only require natural language understanding, such as correcting how the system classifies a user statement. For more generative tasks, like creating new lines of chatbot dialogue, the experts demonstrated higher quality, novelty, and emotion. We also uncovered a motivational gap: crowd workers enjoyed the interactive tasks, while experts found the work to be tedious and repetitive. We offer design considerations for allocating crowd workers and experts on input tasks for AI systems, and for better motivating experts to participate in low-level data work for AI.
Allocation Schemes in Analytic Evaluation: Applicant-Centric Holistic or Attribute-Centric Segmented?
Authors: Wang Jingyan , Baharav Carmel , Shah Nihar B. , Woolley Anita Williams , Ravi R
Published: October 2022

Many applications such as hiring and university admissions involve evaluation and selection of applicants. These tasks are fundamentally difficult, and require combining evidence from multiple different aspects (what we term "attributes"). In these applications, the number of applicants is often large, and a common practice is to assign the task to multiple evaluators in a distributed fashion. Specifically, in the often-used holistic allocation, each evaluator is assigned a subset of the applicants, and is asked to assess all relevant information for their assigned applicants. However, such an evaluation process is subject to issues such as miscalibration (evaluators see only a small fraction of the applicants and may not get a good sense of relative quality), and discrimination (evaluators are influenced by irrelevant information about the applicants). We identify that such attribute-based evaluation allows alternative allocation schemes. Specifically, we consider assigning each evaluator more applicants but fewer attributes per applicant, termed segmented allocation. We compare segmented allocation to holistic allocation on several dimensions via theoretical and experimental methods. We establish various tradeoffs between these two approaches, and identify conditions under which one approach results in more accurate evaluation than the other.
In-HIT Example-Guided Annotation Aid for Crowdsourcing UI Components
Authors: Huang Yi-Ching , Wang Chun-I , Yu Shih-Yuan , Hsu Yung-jen
Published: October 2022

This paper presents an approach to crowdsourcing annotations of UI components from images. Using the “Find-Draw-Verify” task design, an in-HIT example-guided annotation aid is proposed to facilitate workers thereby improving the result quality.
Gameful Markets for Collaboration and Learning
Authors: Leutenmayr Stephan , Kneissl Fabian , Ziemer Sven , Bry François
Published: October 2022

The market metaphor offers a promising form of human computation for gathering and aggregating individual opinions into a collective point of view for groups of people. In this article, we identify different kinds of market forms, their specific goals, and their participants' behavior. Building on this, we first determine appropriate reward designs for a good functioning of the markets as opinion aggregators. Furthermore, we stress potential possibilities for facilitating participants' learning. To illustrate and validate the approach proposed, we investigate two different market applications with different kinds of rewards and goals.
Leveraging Collaboration: A Methodology for the Design of Social Problem-Solving Systems
Authors: Tabajara Lucas , Prates Marcelo , Noble Diego , Lamb Luis
Published: October 2022

Social collaboration has been shown to facilitate problemsolving activity in diverse sets of environments. Nevertheless, if not well designed, social and human computation systems may achieve results only similar to those of a single human subject performing a task. This scenario reflects a need for better understanding of the performance issues of human problem-solving social networks. Firstly, we propose a model for simulating social problem-solving. We then carry out several simulations with artificial agents supported by results of experiments carried out with human subjects, in order to analyse which parameters influence the performance of collaborative problem-solving social networks. We analyse the strategies humans follow when solving a problem, comparing them with alternative ones, and identify the consequences of the employed strategies in the collective performance of the social network. Our results also indicate that copying and guessing are beneficial to the performance of the social networks. We then propose mechanisms that can improve collaborative problem-solving. Finally, we show that our results lead to a methodology for the design of efficient problem-solving systems that can be applied to several kinds of collaborative social systems.
Why Stop Now? Predicting Worker Engagement in Online Crowdsourcing
Authors: Mao Andrew , Kamar Ece , Horvitz Eric
Published: October 2022

We present studies of the attention and time, or engagement, invested by crowd workers on tasks. Consideration of worker engagement is especially important in volunteer settings such as online citizen science. Using data from Galaxy Zoo, a prominent citizen science project, we design and construct statistical models that provide predictions about the forthcoming engagement of volunteers. We characterize the accuracy of predictions with respect to different sets of features that describe user behavior and study the sensitivity of predictions to variations in the amount of data and retraining. We design our model for guiding system actions in real-time settings, and discuss the prospect for harnessing predictive models of engagement to enhance user attention and effort on volunteer tasks.
Depth-Workload Tradeoffs for Workforce Organization
Authors: Heidari Hoda , Kearns Michael
Published: October 2022

We introduce and consider the problem of effectively organizing a population of workers of varying abilities. We assume that arriving tasks for the workforce are homogeneous, and that each is characterized by an unknown and one- dimensional difficulty value x ∈ [0, 1]. Each worker i is characterized by their ability wi ∈ [0, 1], and can solve the task if and only if x ≤ wi. If a worker is unable to solve a given task it must be forwarded to a worker of greater ability. For a given set of worker abilities W and a distribution P over task difficulty, we are interested in the problem of designing efficient forwarding structures for W and P. We give efficient algorithms and structures that simultaneously (approximately) minimize both the maximum workload of any worker, and the number of workers that need to attempt a task. We identify broad conditions under which workloads diminish rapidly with the workforce size, yet only a constant number of workers attempt each task.
Conversations in the Crowd: Collecting Data for Task-Oriented Dialog Learning
Authors: Lasecki Walter , Kamar Ece , Bohus Dan
Published: October 2022

A major challenge in developing dialog systems is obtaining realistic data to train the systems for specific domains. We study the opportunity for using crowdsourcing methods to collect dialog datasets. Specifically, we introduce ChatCollect, a system that allows researchers to collect conversations focused around definable tasks from pairs of workers in the crowd. We demonstrate that varied and in-depth dialogs can be collected using this system, then discuss ongoing work on creating a crowd-powered system for parsing semantic frames. We then discuss research opportunities in using this approach to train and improve automated dialog systems in the future.
Crowdsourcing Transcription Beyond Mechanical Turk
Authors: Zhou Haofeng , Baskov Denys , Lease Matthew
Published: October 2022

While much work has studied crowdsourced transcription via Amazon’s Mechanical Turk, we are not familiar with any prior cross-platform analysis of crowdsourcing service providers for transcription. We present a qualitative and quantitative analysis of eight such providers: 1-888-Type-It-Up, 3Play Media, Transcription Hub, CastingWords, Rev, TranscribeMe, Quicktate, and SpeakerText. We also provide comparative evaluation vs. three transcribers from oDesk. Spontanteous speech used in our experiments is drawn from USC-SFI MALACH collection of oral history interviews. After informally evaluating pilot transcripts from all providers, our formal evaluation measures word error rate (WER) over 10-minute segments from six interviews transcribed by three service providers and the three oDesk transcribers. We report the WER obtained in each case, and more generally assess tradeoffs among the quality, cost, risk and effort of alternative crowd-based transcription options.
Using Human and Machine Processing in Recommendation Systems
Authors: Colson Eric
Published: October 2022

This paper presents a case study of an online apparel retailer called Stitch Fix, which uses both machine and human processing in its recommendation system.
Statistical Quality Estimation for General Crowdsourcing Tasks
Authors: Baba Yukino , Kashima Hisashi
Published: October 2022

One of the biggest challenges for requesters and platform providers of crowdsourcing is quality control,which is to expect high-quality results from crowd workers.We propose an unsupervised statistical quality estimation method for general crowdsourcing tasks with unstructured response formats such as article writing, program coding, and logo designing, which occupy the majority on most crowdsourcing marketplaces.
OnDroad Planner: Building Tourist Plans Using Traveling Social Network Information
Authors: Cenamor Isabel , De la Rosa Tomás , Borrajo Daniel
Published: October 2022

One of the key challenges in automated planning is to define the sources of information that will feed the initial state and goals of each planning task. In many domains, the information comes from company's databases. In other applications, the information is harder to obtain and it is usually partial. In this paper, we will describe an application on travel planning, where the initial state and goals will be obtained by crowdsourcing. Travel planning requires the use of plenty Internet-based resources; some of them are related to human generated opinions on all kinds of matters (e.g. hotels, places to visit, restaurants, ...). We present the OnDroad planner, a system that creates personalized tourist plans using the human generated information gathered from the minube traveling social network. OnDroad proposes an initial tourist guide according to the recommendation of the users profiles and their contacts. In addition, this guide can be continuously updated with newly generated data.
Using Crowdsourcing to Generate an Evaluation Dataset for Name Matching Technologies
Authors: Asarina Alya , Simek Olga
Published: October 2022

Crowdsourcing can be a fast, flexible and cost-effective approach to obtaining data for training and evaluating machine learning algorithms. In this paper, we discuss a novel crowdsourcing application: creating a dataset for evaluating name matchers. Name matching is the challenging and subjective task of identifying which names refer to the same person; it is crucial for effective entity disambiguation and search. We have developed an effective question interface and work quality analysis algorithm for our task, which can be applied to other ranking tasks (e.g. search result ranking, recommendation system evaluation, etc.). We have demonstrated that our crowdsourced dataset can successfully be used to evaluate automatic name-matching algorithms.
Boosting OCR Accuracy Using Crowdsourcing
Authors: Wang Shuo-Yang , Wang Ming-Hung , Chen Kuan-Ta
Published: October 2022

Book digitizing is an important work in preserving ancient heritages. However, digitizing books contains a series of labor-intensive works, and one of them is to verify optical character recognition (OCR) outcomes. In this paper, we propose a crowdsourceable OCR verification method. Using our method, content holders are able to leverage the power of crowds to complete verification tasks and avoid content leakage. From the experiment results, our method is more efficient and reliable than the traditional method.
Lottery-Based Payment Mechanism for Microtasks
Authors: Celis L. , Roy Shourya , Mishra Vivek
Published: October 2022

We propose using lotteries as an alternative payment mechanism through which to incentivize and recruit workers to microtasking. We present initial findings gained via experiments on Amazon Mechanical Turk and focus on discussing the benefits and potential pitfalls in employing a lottery-based payment mechanism for microtasking.
Instance-Privacy Preserving Crowdsourcing
Authors: Kajino Hiroshi , Baba Yukino , Kashima Hisashi
Published: October 2022

Crowdsourcing is a technique to outsource tasks to a number of workers. Although crowdsourcing has many advantages, it gives rise to the risk that sensitive information may be leaked, which has limited the spread of its popularity. Task instances (data workers receive to process tasks) often contain sensitive information, which can be extracted by workers. For example, in an audio transcription task, an audio file corresponds to an instance, and the content of the audio (e.g., the abstract of a meeting) can be sensitive information. In this paper, we propose a quantitative analysis framework for the instance privacy problem. The proposed framework supplies us performance measures of instance privacy preserving protocols. As a case study, we apply the proposed framework to an instance clipping protocol and analyze the properties of the protocol. The protocol preserves privacy by clipping instances to limit the amount of information workers obtain. The results show that the protocol can balance task performance and instance privacy preservation. They also show that the proposed measure is consistent with standard measures, which validates the proposed measure.
Referral Incentives in Crowdfunding
Authors: Naroditskiy Victor , Stein Sebastian , Tonin Mirco , Tran-Thanh Long , Vlassopoulos Michael , Jennings Nicholas
Published: October 2022

Word-of-mouth, referral, or viral marketing is a highly sought-after way of advertising. In this paper, we investigate whether such marketing can be encouraged through incentive mechanisms, thus allowing an organisation to effectively crowdsource their marketing. Specifically, we undertake a field experiment that compares several mechanisms for incentivising social media shares in support of a charitable cause. Our experiment takes place on a website promoting a fundraising drive by a large cancer research charity. Site visitors who sign up to support the cause are asked to spread the word about it on Facebook, Twitter or other channels. They are randomly assigned to one of four treatments that differ in the way social sharing activities are incentivised. Under the control treatment, no extra incentive is provided. Under two of the other mechanisms, the sharers are offered a fixed number of points that help take the campaign further. We compare low and high levels of such incentives for direct referrals. In the final treatment, we adopt a multi-level incentive mechanism that rewards direct as well as indirect referrals (where referred contacts refer others). We find that providing a high level of incentives results in a statistically significant increase in sharing behaviour and resulting signups. Our data does not indicate a statistically significant increase for the low and multi-level incentive mechanisms.
Mechanism Design for Crowdsourcing Markets with Heterogeneous Tasks
Authors: Goel Gagan , Nikzad Afshin , Singla Adish
Published: October 2022

Designing optimal pricing policies and mechanisms for allocating tasks to workers is central to online crowdsourcing markets. In this paper, we consider the following realistic setting of online crowdsourcing markets -- we are given a set of heterogeneous tasks requiring certain skills; each worker has certain expertise and interests which define the set of tasks she is interested in and willing to do. Given this bipartite graph between workers and tasks, we design our mechanism \truthuniform which does the allocation of tasks to workers, while ensuring budget feasibility, incentive-compatibility and achieves near-optimal utility. We further extend our results by exploiting a link with online Adwords allocation problem and present a randomized mechanism \truthfractional with improved approximation guarantees. Apart from strong theoretical guarantees, we carry out extensive experimentation using simulations as well as on a realistic case study of Wikipedia translation project with Mechanical Turk workers. Our results demonstrate the practical applicability of our mechanisms for realistic crowdsourcing markets on the web.
Monetary Interventions in Crowdsourcing Task Switching
Authors: Yin Ming , Chen Yiling , Sun Yu-An
Published: October 2022

With a large amount of tasks of various types, requesters in crowdsourcing platforms often bundle tasks of different types into a single working session. This creates a task switching setting, where workers need to shift between different cognitive tasks. We design and conduct an experiment on Amazon Mechanical Turk to study how occasionally presented performance-contingent monetary rewards, referred as monetary interventions, affect worker performance in the task switching setting. We use two competing metrics to evaluate worker performance. When monetary interventions are placed on some tasks in a working session, our results show that worker performance on these tasks can be improved in both metrics. Moreover, worker performance on other tasks where monetary interventions are not placed is also affected: workers perform better according to one metric, but worse according to the other metric. This suggests that in addition to providing extrinsic monetary incentives for some tasks, monetary interventions implicitly set performance goals for all tasks. Furthermore, monetary interventions are most effective in improving worker performance when used at switch tasks, tasks that follow a task of a different type, in working sessions with a low task switching frequency.
Saving Money While Polling with InterPoll Using Power Analysis
Authors: Livshits Benjamin , Mytkowicz Todd
Published: October 2022

Crowd-sourcing is increasingly being used for providing responses to polls and surveys on a large scale. Companies such as SurveyMonkey and Instant.ly are attempting to make crowd-sourced surveys commonplace, by making it easy to pose survey questions using an easy-to-use UI and retrieve results with a relatively low latency by having dedicated crowds at their disposal. In this paper we argue that the ease with which polls can be created conceals an inherent difficulty: the survey maker does not know how many workers to hire for their survey. Asking too few may lead to samples sizes that `"do not look impressive enough." Asking too many clearly involves spending extra money, which can quickly become costly. Existing crowd-sourcing platforms do not provide help with this, neither, one can argue, do they have any incentive to do so. We present a systematic approach to determining how many samples (i.e. workers) are required to achieve a certain level of statistical significance by showing how to automatically perform power analysis on questions of interest. Using a range of queries we demonstrate that power analysis can save significant amounts of money and time by concluding that frequently, only a handful of results is required to arrive at a certain decision. We have implemented our approach within InterPoll, aprogrammable developer-driven polling system that uses a generic crowd (Mechanical Turk) as a back-end. Power analysis is automatically performed given both the structure of the query and the data that is being polled from the crowd. In all of our studies we are able to obtain statistically significant answers for under $30, with most costing less than $10. Our approach saves both time and money for the survey maker.
Glance Privacy: Obfuscating Personal Identity While Coding Behavioral Video
Authors: Gordon Mitchell , Lasecki Walter , Leung Winnie , Lim Ellen , Dow Steven , Bigham Jeffery
Published: October 2022

Behavioral researchers code video to extract systematic meaning from subtle human actions and emotions. While this has traditionally been done by analysts within a research group, recent methods have leveraged online crowds to massively parallelize this task and reduce the time required from days to seconds. However, using the crowd to code video increases the risk that private information will be disclosed because workers who have not been vetted will view the video data in order to code it. In this Work-in-Progress, we discuss techniques for maintaining privacy when using Glance to code video and present initial experimental evidence to support them.
Persistent and Pervasive Real-World Sensing Using Games
Authors: Frazier Spencer , Riedl Mark
Published: October 2022

Games With a Purpose can enable an intelligent agent to persistently and pervasively sense the real world by using game players as reconfigurable sensors. We propose a technique whereby an intelligent agent incentivizes players to collect data by translating data collection tasks into a series of quests played on a mobile device. In this paper, we define the concept of Proactive Sensing and provide a framework for Game-Based Proactive Sensing that can adapt games and narrative that optimizes for data collection and long-term player engagement.
Computing Through Movement
Authors: McAdam Rohan
Published: October 2022

One human capability that seems to have received little attention from researchers in human computation so far is our ability to understand and manipulate the complex dynamics of physical movement. This research investigates the use of human motor learning as a mechanism for exploration and problem solving for nonlinear dynamical systems. This paper illustrates the approach with an example from the study of sustainable economic growth.
Identifying Relevant Text Fragments to Help Crowdsource Privacy Policy Annotations
Authors: Ramanath Rohan , Schaub Florian , Wilson Shomir , Liu Fei , Sadeh Norman , Smith Noah
Published: October 2022

In today's age of big data, websites are collecting an increasingly wide variety of information about their users. The texts of websites' privacy policies, which serve as legal agreements between service providers and users, are often long and difficult to understand. Automated analysis of those texts has the potential to help users better understand the implications of agreeing to such policies. In this work, we present a technique that combines machine learning and crowdsourcing to semi-automatically extract key aspects of website privacy policies that is scalable, fast, and cost-effective.
A GWAP Approach for Collecting Qualitative Product Attributes and Perceptual Mapping
Authors: Miyashita Eiji , Nonaka Tomomi , Mizuyama Hajime
Published: October 2022

An ESP-like game is proposed for collecting qualitative product attributes relevant to the preferences of consumers efficiently. It is also discussed how to analyze the qualita-tive attributes and the supplemental data such as matching counts and input times acquired through the game and to yield a perceptual map.
Learning Pronunciation and Accent from The Crowd
Authors: Liu Frederick , Yang Jeremy , Hsu Jane
Published: October 2022

Learning a second language is becoming a more popular trend around the world. But the act of learning another language in a place removed from native speakers is difficult as there is often no one to correct mistakes nor examples to imitate. With the idea of crowd sourcing, we would like to propose an efficient way to learn a second language better.
Volunteer-Based Crowdsourcing with Crowd4U
Authors: Morishima Atsuyuki
Published: October 2022

We are operating a crowdsourcing platform named Crowd4U with the help of researchers from more than 24 universities and conducting several projects for academic and public purposes. In this paper, we first explain Crowd4U and some of the projects running on it. Then, we discuss challenges and lessons learned from our experience of crowdsourcing projects with volunteer workers.
Social Microvolunteering: Donating Access to Your Friends for Charitable Microwork
Authors: Brady Erin , Morris Meredith , Bigham Jeffrey
Published: October 2022

We propose social microvolunteering, in which people can do charitable microwork themselves for free, but also grant access to their Facebook friends as additional volunteers to magnify their effort. Social microvolunteering lets people volunteer despite temporal, financial, or physical limitations.
Combining Crowd and Expert Labels Using Decision Theoretic Active Learning
Authors: Nguyen An , Wallace Byron , Lease Matthew
Published: October 2022

We consider a finite-pool data categorization scenario which requires exhaustively classifying a given set of examples with a limited budget. We adopt a hybrid human-machine approach which blends automatic machine learning with human labeling across a tiered workforce composed of domain experts and crowd workers. To effectively achieve high-accuracy labels over the instances in the pool at minimal cost, we develop a novel approach based on decision-theoretic active learning. On the important task of biomedical citation screening for systematic reviews, results on real data show that our method achieves consistent improvements over baseline strategies. To foster further research by others, we have made our data available online.
To Play or Not to Play: Interactions between Response Quality and Task Complexity in Games and Paid Crowdsourcing
Authors: Krause Markus , Kizilcec René
Published: October 2022

Digital games are a viable alternative to accomplish crowdsourcing tasks that would traditionally require paid online labor. This study compares the quality of crowdsourcing with games and paid crowdsourcing for simple and complex annotation tasks in a controlled exper-iment. While no difference in quality was found for the simple task, paid contributors’ response quality was sub-stantially lower than players’ quality for the complex task (92% vs. 78% average accuracy). Results suggest that crowdsourcing with games provides similar and potentially even higher response quality relative to paid crowdsourcing.
A Game with a Purpose for Recommender Systems
Authors: Smyth Barry , Rafter Rachael , Banks Sam
Published: October 2022

Recommender systems learn about our preferences to make targeted suggestions. In this paper we outline a novel game-with-a-purpose designed to infer preferences at scale as a side-effect of gameplay. We evaluate the utility of this data in a recommendation context as part of a small live-user trial.
Crowdsourcing Based on GPS
Authors: Vazirabadkar Ketan , Gadre Siddhant , Sebastian Rajeev , Dwivedi Nikhil
Published: October 2022

Machines cannot answer all the queries. There are some queries which need human input for providing information that is missing from the database, to perform computationally difficult procedures, and for cross checking, ordering, or aggregating results based on fuzzy criteria. Crowd sourcing can be used to solve such queries. Smartphones can unfold the full potential of crowdsourcing, allowing users to transparently contribute to complex and novel problem solving. We present the intrinsic characteristics of smartphones, a taxonomy that gives classification of the emerging field of mobile crowdsourcing and application that optimize location-based queries and similarity services over data generated by a crowd. This paper will focus on our application which uses Internet as a source of getting information which is so relevant to us that it is about what we want to do right where we are right now in the best possible way. We have implemented two ideas over the course of the project — first, a Location-Based crowd sourced forum and secondly, a platform where one can get location based relevant infor-mation. The forum will enable a user to ask relevant and location based question which will be answered by the people who are familiar with that area and the platform will monitor user’s location at regular intervals and pop up relevant information as soon as s/he enters a particular radius of an area of his/her preferred choices.
On the Role of Task Design in Crowdsourcing Campaigns
Authors: Bernaschina Carlo , Catallo Ilio , Fraternali Piero , Martinenghi Davide
Published: October 2022

Despite the success of crowdsourcing marketplaces, fully harnessing their massive workforce remains challenging. In this work we study the effect on crowdsourcing campaigns of different feedback and payment strategies. Our results reveal the joint effect of feedback and payment on the quality and quantity of the outcome.
Flexible Reward Plans to Elicit Truthful Predictions in Crowdsourcing
Authors: Sakurai Yuko , Oyama Satoshi , Shinoda Masato , Yokoo Makoto
Published: October 2022

We develop a flexible reward plan to elicit truthful predictive probability distribution over a set of uncertain events from workers.  In our reward plan, the principal can assign rewards for incorrect predictions according to her similarity between events.  In the spherical proper scoring rule, a worker's expected utility is represented as the inner product of her truthful predictive probability and her declared probability. We generalize the inner product by introducing a reward matrix that defines a reward for each prediction-outcome pair. We show that if the reward matrix is symmetric and positive definite, the spherical proper scoring rule guarantees the maximization of a worker's expected utility when she truthfully declares her prediction.
Quality Estimation of Workers in Collaborative Crowdsourcing Using Group Testing
Authors: Ojha Prakhar , Talukdar Partha
Published: October 2022

Crowdsourcing is increasingly being used to solve complex tasks that require contributions from groups of individuals. In this paper, we consider the problem of distinguishing workers from idlers (who do not contribute positively) in group-based tasks. We consider a group as our smallest observable unit that can be evaluated and assume no knowledge of individual participant’s contribution. We propose the use of group testing based methods for estimating quality of an individual, based on the performance of teams they have been part of. We further extend these algorithms to identify subsets of workers and give theoretical analysis on size of these subsets. We account for several real-world constraints in our model and present empirical support to our theoretical guarantees by an array of simulation experiments.
Learning to Scale Payments in Crowdsourcing with PropeRBoost
Authors: Radanovic Goran , Faltings Boi
Published: October 2022

Motivating workers to provide significant effort has been recognized as an important issue in crowdsourcing. It is important not only to compensate worker effort, but also to discourage low-quality workers from participating. Several proper incentive schemes have been proposed for this purpose; they are either based on gold tasks or on peer consistency in individual tasks. As the rewards cannot become negative, these schemes have difficulty in achieving zero expected reward for random answers. We describe a novel boosting scheme, ProperRBoost, that improves the efficiency of existing incentive schemes by making a better separation between incentives for high and low quality work, and effectively discourages random answers by assigning them near minimal average rewards. We show the actual performance of the boosting scheme through simulations of various worker strategies.
Validating the Quality of Crowdsourced Psychometric Personality Test Items
Authors: Loe Bao , Smart Francis , Firtova Lenka , Brauner Corinna , Lueneborg Laura , Stillwell David
Published: October 2022

Personality questionnaire items are normally written by experts in personality theory or psychometrics. This makes personality test development expensive, especially since as many as 40% of the items are discarded after quantitative analysis. Could the crowd create personality items that are as good as the ones developed by professionals? We analyze the psychometric quality of 164 Big Five personality items generated by the crowd (n=82) and show that non-expert item writers are capable of designing high quality personality items. The analysis revealed that 59% of the crowdsourced items had adequate psychometric properties, and thus could be used as an alternative instrument to measure Big Five personality traits. We conclude that crowdsourcing personality items is a possible strategy to generate new items rapidly. We also discuss some of the challenges of crowdsourcing general psychological items.