bluuebunny/crossref_metadata_2025 huggingface dataset:
Model-Driven AI for Games: Research Plan
Authors: Dragert Christopher
Published: October 2022

The field of game AI is largely industry driven, lacking an agreed upon formalism for AI representation. Ad-hoc scripting languages, simple finite state machines, behaviour trees, and planners are employed, but not in a fashion adhering to any standard. As a result, reuse is sparse between games and formal analysis techniques are undeveloped. As research for a Ph.D. thesis, we propose to show that a layered Statechart-based AI is a suitable formalism for Game AI, enabling the use of model-driven development techniques such as reuse and high-level analysis including model-checking. The fundamentally modular nature of this approach leads naturally to reuse as a fundamental component of the design process. Supported by a clearly defined formalism, useful behavioural analyses become possible, such as testing reactions to various inputs at design time. We also explore transformations at the modelling level to enable procedural generation, allowing rapid deployment of varying AIs. Additionally, such a model allows for the generation of efficient code that can be directly inserted into games. Tool support for reuse, generation, and analysis will be developed, then employed in creating an industrial scale AI, proving that this formalism is appropriate for industrial use.
Narrative Intelligence Without (Domain) Boundaries
Authors: Li Boyang
Published: October 2022

Narrative Intelligence (NI) can help computational systems interact with users, such as through story generation, interactive narratives, and believable virtual characters. However, existing NI techniques generally require manually coded domain knowledge, restricting their scalability. An approach that intelligently, automatically and economically acquires script-like knowledge in any domain with strategic crowdsourcing will ease this bottleneck and broaden the application territory of narrative intelligence. This doctoral consortium paper defines the research problem, describes its significance, proposes a feasible research plan towards a Ph.D. dissertation, and reports on its current progress.
Organizers
Authors: Jhala Arnav
Published: October 2022

List of organizers of the 2012 Doctoral Consortium held at the AAAI AIIDE Conference.
AIIDE-13 Organization
Authors: Suthankar Gita
Published: October 2022

List of organizers of the Ninth Artificial Intelligence and Interactive Digital Entertainment Conference.
AIIDE 2013 StarCraft Competition
Authors: Buro Michael , Churchill David
Published: October 2022

In 2013, AIIDE will host the Fourth Annual Star-Craft AI Competition. Participants are given the task of building the best performing AI system for the popular real-time strategy game StarCraft Brood War (Blizzard Entertainment). Thee goals of the competition are to provide a testbed for real-time AI systems and to promote game AI research by exhibiting AI techniques such as scripting, planning, optimization, spatial reasoning, and opponent modeling in a fast-paced popular video game.
Viewpoints AI
Authors: Jacob Mikhail , Coisne Gaëtan , Gupta Akshay , Sysoev Ivan , Verma Gaurav , Magerko Brian
Published: October 2022

This article describes a technical approach to movement-based interactions between a human interactor and an intelligent agent based on the theatrical Viewpoints movement framework. The Viewpoints AI system features procedural gesture interpretation using shallow semantics and deep aesthetics analysis from the Viewpoints framework. The installation creates a liminal virtual / real space for the human and AI to interact by the use of digital projection for the AI visualization and shadow play to represent the human. Observations from a recent public demonstration of the system and future directions of work are also discussed.
Mimicking Humanlike Movement in Open World Games with Path-Relative Recursive Splines
Authors: Tomai Emmett , Salazar Rosendo , Flores Roberto
Published: October 2022

In this paper we explore the use of recursive cubic Hermite splines to mimic human movement in open world games. Human-like movement in an open world environment has many characteristics that are not optimal or directed towards clear, discrete goals. Using data collected from a simple MMORPG-like game, we use our spline representation to model human player movements relative to corresponding optimal paths. Using this representation, we show that simple distributions can be used to estimate control parameters to generate human-like movement across a population of agents in a novel environment.
Open Problem: Reusable Gameplay Trace Samplers
Authors: Smith Adam
Published: October 2022

We identify an open problem in game design assistance and automation: the development of reusable gameplay trace samplers. Inside many sophisticated content generators and design tools is a component that samples interesting and plausible sequences of player actions. Details and summary properties of these samples are used to assess generated content and to inform designers. As the development of this component is technically involved (sometimes comparable to making a second implementation of a game's mechanics), design tools often either make use of entirely custom, game-specific samplers or make due without the ability to sample interesting traces at all. This severely limits the population who could benefit from automation to those who are motivated to develop it for themselves. We propose the development of reusable samplers to ease the development of future design automation tools. This paper reviews several systems that demonstrate the availability of technology required by these samplers and the range of applications they may serve. It also sketches how future samplers might be architected. This proposal identifies one way for technical research to make progress on design automation challenges without making problematic assumptions about the nature of player behavior or designer intent. Filling in this missing infrastructure, we claim, will make the use of artificial intelligence in the design process more accessible and thus accelerate game design projects.
Using Expressive Language Generation to Increase Authorial Leverage
Authors: Walker Marilyn , Sawyer Jennifer , Jimenez Carolynn , Rishes Elena , Lin Grace , Hu Zhichao , Pinckard Jane , Wardrip-Fruin Noah
Published: October 2022

It is widely agreed that the interaction possibilities in interactive narrative are limited by the current approach to dialog creation, which typically relies on teams of scriptwriters. This paper reports on experiments testing whether authorial leverage can be increased by methods for natural language generation of dialog in a story world called Heart of Shadows. Our experiments show that (1) expert writers spend less time authoring dialogue variations by editing automatically generated content than creating it from scratch and (2) expert writers are more critical of automatically generated dialogue than amateur writers.
A Belief-Desire-Intention Model for Narrative Generation
Authors: Wadsley Theo , Ryan Malcolm
Published: October 2022

Narrative AI needs to model more than just action. In this paper, we investigate the Belief-Desire-Intention (BDI) agent architecture to allow plots to be modelled in terms of character motivation. This allows authors to focus on elements of the character model which are highly relevant to plot. We describe an extended implementation of the ConGolog agent programming language which includes BDI syntax and semantics. Using this language, we provide an example of how plot could be advantageously modelled in terms of character motivation.
Evaluation, Orientation, and Action in Interactive StoryTelling
Authors: Rahimtoroghi Elahe , Swanson Reid , Walker Marilyn , Corcoran Thomas
Published: October 2022

Interactive storytelling encompasses a wide variety of applications and systems that aim to allow players in a virtual world to affect the narrative progression prescribed by the author. The story world is often modeled by a set of hand-crafted causal and temporal relationships that define the possible narrative sequences based on the user's actions. In our view, a more promising approach involves applying methods from statistical language processing to create data-driven stories. Here, we explore integrating a typology of narrative clauses from Labov & Waletzky's theory of oral narrative(henceforth L&W) into the Say Anything data-driven storytelling approach. We show that we can automatically distinguish evaluation clauses from orientation and action clauses with 89% accuracy in fables, suggesting that it will be possible to develop new types of data-driven stories using L&W's typology.
Performing with Technology: Lessons Learned from the GenJam Project
Authors: Biles John
Published: October 2022

The author has been performing with GenJam, the Genetic Jammer, for nearly 20 years and has accumulated a wealth of experiences in performing live jazz with technology. This paper presents a discussion of the use of technology in jazz, both from the performer’s and from the audience’s perspective, and it proposes a classification scheme for live performance that is geared to mainstream performing situations.
Automatic Harmonization Using a Hidden Semi-Markov Model
Authors: Groves Ryan
Published: October 2022

Hidden Markov Models have been used frequently in the audio domain to identify underlying musical structure. Much less work has been done in the purely symbolic realm. Recently, a substantial amount of expert-labelled symbolic musical data has been injected into the research community. The new availability of data allows for the application of machine learning models to purely symbolic tasks. Similarly, the continued expansion of the field of machine learning provides new perspectives and implementations of machine learning methods, which are powerful tools when approaching complex musical challenges. This research explores the use of an extended probabilistic model such as the Hidden Semi-Markov Model (HSMM) to approach the task of automatic harmonization. One distinct advantage of the HSMM is that it is able to automatically differentiate harmonic boundaries, through its inclusion of an extra parameter: duration. In this way, a melody can be harmonized automatically in the style of a particular corpus. In the case of this research, the corpus was in the style of Rock 'n' Roll.
Using the Creative Process for Sound Design Based on Generic Sound Form
Authors: Mazzola Guerino , Thalmann Florian
Published: October 2022

Building on recent research in musical creativity and the composition process, this paper presents a specific practical application of our theory and software to sound design. The BigBang rubette module that brings gestural music composition methods to the Rubato Composer software was recently generalized in order to work with any kinds of musical and non-musical objects. Here, we focus on time-independent sound objects to illustrate several levels of metacreativity. On the one hand, we show a sample process of designing the sound objects themselves by defining appropriate datatypes, which can be done at runtime. On the other hand, we demonstrate how the creative process itself, recorded by the software once the composer starts working with these sound objects, can be used for both improvisation with and automation of any defined operations and transformations.
Improving Plan-Based Interactive Narrative Generation
Authors: Robertson Justus
Published: October 2022

I am interested in algorithms that generate well structured interactive narratives that allow human participants to effect change in a story world. Narrative can be structured as a sequence of actions to be carried out by a set of characters in a particular environment. This narrative structure can be generated by AI planning. A process called accommodative mediation may be used on a single narrative plan to create a participatory experience that branches based on a human's interactions as a character within the story. However, accommodation is limited by search time and the number of plans the algorithm can possibly reach. In this paper, I discuss an initial modification of accommodative mediation that expands the space of plans the algorithm may search. I also identify several avenues by which the process may continue to be improved.
Reader-Model-Based Story Generation
Authors: Mawhorter Peter
Published: October 2022

Several existing systems have used reader models for story generation, but they have focused on either interactive contexts or pure discourse-level manipulation. I intend to buid a reader-model story generator that not only applies reader modelling to full plot generation, but which also draws on theories about intentionality and emotions put forward by Lisa Zunshine, Keith Oatley, and Raymond Mar. To evaluate the contributions of the reader model, I'll compare it with human-authored stories using measures of reader engagement. I'll also run the model on human-authored stories and compare the results to a human gold-standard analysis.
What Comes Next? An Experiment in Interactive Narrative
Authors: Samuel Benjamin
Published: October 2022

Most interactive narratives fail to live up to their names. Static interactive narratives (e.g., fixed story graphs) tend to lack a large enough possibility space to allow for a sense of creation and ownership on the part of the player. Dynamic narratives can employ techniques that undercut, or remove entirely, the significance of player interaction. This paper introduces a proposal for a new system that lets the user continuously select from system generated sets of story events to include in an evolving narrative. Though this work is still in its preliminary stages, it could be an advance in interactive storytelling, allowing for a large possibility space of quality stories that truly depend on user input and interaction for their creation.
Goal-Setting Behavior of Workers on Crowdsourcing Platforms: An Exploratory Study on MTurk and Prolific
Authors: Abbas Tahir , Gadiraju Ujwal
Published: October 2022

A wealth of evidence across several domains indicates that goal setting improves performance and learning by enabling individuals to commit their thoughts and actions to goal achievement. Recently, researchers have begun studying the effects of goal setting in paid crowdsourcing to improve the quality and quantity of contributions, increase learning gains, and hold participants accountable for contributing more effectively. However, there is a lack of research addressing crowd workers' goal-setting practices, how they are currently pursuing them, and the challenges that they face. This information is essential for researchers and developers to create tools that assist crowd workers in pursuing their goals more effectively, thereby improving the quality of their contributions. This paper addresses these gaps by conducting mixed-method research in which we surveyed 205 workers from two crowdsourcing platforms -- Amazon Mechanical Turk (MTurk) and Prolific -- about their goal-setting practices. Through a 14-item survey, we asked workers regarding the types of goals they create, their goal achievement strategies, potential barriers that impede goal attainment, and their use of software tools for effective goal management. We discovered that (a) workers actively create intrinsic and extrinsic goals; (b) use a combination of tools for goal management; (c) medical issues and a busy lifestyle are some obstacles to their goal achievement; and (d) we gathered novel features for future goal management tools. Our findings shed light on the broader implications of developing goal management tools to improve workers' well-being.
Near-Optimal Reviewer Splitting in Two-Phase Paper Reviewing and Conference Experiment Design
Authors: Jecmen Steven , Zhang Hanrui , Liu Ryan , Fang Fei , Conitzer Vincent , Shah Nihar B.
Published: October 2022

Many scientific conferences employ a two-phase paper review process, where some papers are assigned additional reviewers after the initial reviews are submitted. Many conferences also design and run experiments on their paper review process, where some papers are assigned reviewers who provide reviews under an experimental condition. In this paper, we consider the question: how should reviewers be divided between phases or conditions in order to maximize total assignment similarity? We make several contributions towards answering this question. First, we prove that when the set of papers requiring additional review is unknown, a simplified variant of this problem is NP-hard. Second, we empirically show that across several datasets pertaining to real conference data, dividing reviewers between phases/conditions uniformly at random allows an assignment that is nearly as good as the oracle optimal assignment. This uniformly random choice is practical for both the two-phase and conference experiment design settings. Third, we provide explanations of this phenomenon by providing theoretical bounds on the suboptimality of this random strategy under certain natural conditions. From these easily-interpretable conditions, we provide actionable insights to conference program chairs about whether a random reviewer split is suitable for their conference.
Crowdsourcing Objective Answers to Subjective Questions Online
Authors: Iyer Ravi
Published: October 2022

In this demonstration, we show how Ranker’s algorithms use diverse sampling, measurement, and algorithmic techniques to crowdsource answers to subjective questions in a real-world online environment where user behavior is difficult to control. Ranker receives approximately 8 million visitors each month, as of September 2013, and collects over 1.5 million monthly user opinions. Tradeoffs between computational complexity, projected user engagement, and accuracy are required in such an environment, and aggregating across diverse techniques allows us to mitigate the sizable errors specific to individual imperfect crowdsourcing methods. We will specifically show how relatively unstructured crowdsourcing can yield surprisingly accurate predictions of movie box-office revenue, celebrity mortality, and retail pizza topping sales.
Curio: A Platform for Supporting Mixed-Expertise Crowdsourcing
Authors: Law Edith , Dalton Conner , Merrill Nick , Young Albert , Gajos Krzysztof
Published: October 2022

Curio is a platform that enables researchers in the sciences and humanities, who are domain experts but not necessarily technically savvy, to create, monitor and control complex crowdsourcing projects with minimal effort. With Curio, we aim to contribute to the practice of citizen science and to make fundamental contributions to the study of human computation by developing new interfaces and algorithms for supporting mixed-expertise crowdsourcing, task decomposition, incentive design and quality control.
Automating Crowdsourcing Tasks in an Industrial Environment
Authors: Kandylas Vasilis , Alonso Omar , Choksey Shiroy , Rudre Kedar , Jaiswal Prashant
Published: October 2022

Crowdsourcing based applications are starting to gain traction in industrial environments. Crowdsourcing research has proven that it is possible to get good quality labels at the fraction of the cost and time. However, implementing such applications at large scale requires new infrastructure. In this demo we present a system that allows the automation of crowdsourcing tasks for information retrieval experiments.
HCOMP-13 Organization
Authors: Hartmann Björn , Horvitz Eric
Published: October 2022

List of organizers of the First AAAI Conference on Human Computation and Crowdsourcing.
Incentives for Privacy Tradeoff in Community Sensing
Authors: Singla Adish , Krause Andreas
Published: October 2022

Community sensing, fusing information from populations of privately-held sensors, presents a great opportunity to create efficient and cost-effective sensing applications. Yet, reasonable privacy concerns often limit the access to such data streams. How should systems valuate and negotiate access to private information, for example in return for monetary incentives? How should they optimally choose the participants from a large population of strategic users with privacy concerns, and compensate them for information shared? In this paper, we address these questions and present a novel mechanism, SeqTGreedy, for budgeted recruitment of participants in community sensing. We first show that privacy tradeoffs in community sensing can be cast as an adaptive submodular optimization problem. We then design a budget feasible, incentive compatible (truthful) mechanism for adaptive submodular maximization, which achieves near-optimal utility for a large class of sensing applications. This mechanism is general, and of independent interest. We demonstrate the effectiveness of our approach in a case study of air quality monitoring, using data collected from the Mechanical Turk platform. Compared to the state of the art, our approach achieves up to 30% reduction in cost in order to achieve a desired level of utility.
Improving Your Chances: Boosting Citizen Science Discovery
Authors: Xue Yexiang , Dilkina Bistra , Damoulas Theodoros , Fink Daniel , Gomes Carla , Kelling Steve
Published: October 2022

Citizen scientists are playing an increasing role in helping collect, process, and/or analyze data used to study a variety of scientific phenomena. We address the problem of identifying tasks that are rewarding to the citizen scientists, which results in greater participation, leading to more data and better models. We apply our methodology to eBird, whose participants are avid birders interested in observing different species while contributing to science. In order to improve the birders' chances of meeting their goals, we consider the following probabilistic maximum coverage problem: Given a set of locations, select a subset of size k, such that the birders maximize the expected number of observed species by visiting such locations. We also consider a secondary objective that gives preference to birding sites not previously visited. We consider two variants of the probabilistic maximum coverage problem, provide a theoretical analysis, describe several algorithms with provable approximation guarantees, as well as heuristic approaches, and provide empirical results using eBird data. Our algorithms are fast and provide high quality recommendations.
What Will Others Choose? How a Majority Vote Reward Scheme Can Improve Human Computation in a Spatial Location Identification Task
Authors: Rao Huaming , Huang Shih-Wen , Fu Wai-Tat
Published: October 2022

We created a spatial location identification task (SpLIT) in which workers recruited from Amazon Mechanical Turk were presented with a camera view of a location, and were asked to identify the location on a two-dimensional map. In cases where these cues were ambiguous or did not provide enough information to pinpoint the exact location, workers had to make a best guess. We tested the effects of two reward schemes. In the “ground truth” scheme, workers were rewarded if their answers were close enough to the correct locations. In the “majority vote” scheme, workers were told that they would be rewarded if their answers were similar to the majority of other workers. Results showed that the majority vote reward scheme led to consistently more accurate answers. Cluster analysis further showed that the majority vote reward scheme led to answers with higher reliability (a higher percentage of answers in the correct clusters) and precision (a smaller average distance to the cluster centers). Possible reasons for why the majority voting reward scheme was better were discussed.
Frequency and Duration of Self-Initiated Task-Switching in an Online Investigation of Interrupted Performance
Authors: Gould Sandy , Cox Anna , Brumby Duncan
Published: October 2022

Online experimentation has been shown to be a reliable method for collecting data quickly and easily. Nevertheless, this method of data collection can affect some researchers’ confidence in results because of the loss of experimental control that comes with moving experiments online. To relieve these concerns, researchers have developed a number of tools and techniques for evaluating the performance of online participants and crowdworkers. In this paper we investigate a measure of participants’ attentiveness during an online data-entry experiment. We explored online participants’ propensity to switch away from the experiment to other tasks, and the duration of time that they spent away from the experiment during each switch. Our results show that switching is prevalent, even when participants are asked not to switch to other tasks.
Transcribing and Annotating Speech Corpora for Speech Recognition: A Three-Step Crowdsourcing Approach with Quality Control
Authors: Hämäläinen Annika , Pinto Moreira Fernando , Avelar Jairo , Braga Daniela , Sales Dias Miguel
Published: October 2022

Large speech corpora with word-level transcriptions annotated for noises and disfluent speech are necessary for training automatic speech recognisers. Crowdsourcing is a lower-cost, faster-turnaround, highly scalable alternative for expert transcription and annotation. In this paper, we showcase our three-step crowdsourcing approach motivated by the importance of accurate transcriptions and annotations.
Effect of Task Presentation on the Performance of Crowd Workers — A Cognitive Study
Authors: Sampath Harini , Rajeshuni Rajeev , Indurkhya Bipin , Karanam Saraschandra , Dasgupta Koustuv
Published: October 2022

We present our ongoing research on improving the task presentation using cognitively inspired features to optimize the performance of crowd workers. For the task of extracting text from scanned images, we generated three task-presentation designs by modifying two features of the task - visual saliency of target ﬁelds and working memory requirements. Our experiments conducted on Amazon Mechanical Turk (AMT) indicate that modifying visual saliency results in better performance.
Task Sequence Design: Evidence on Price and Difficulty
Authors: Yin Ming , Chen Yiling , Sun Yu-An
Published: October 2022

We experimentally study the effects of two variables in task sequence design, price and difficulty of tasks, on the quality of work produced in a task sequence. We find that in a three-task sequence, the work quality of a task is influenced by prices of all previous tasks, potentially at various degrees. For the effects of task difficulty, however, we observe similar work quality of a task regardless whether the task follows an easier or more difficult task. This observation is inconsistent with the prediction of either the task switching effect or the sequential difficulty effect in the literature. Interestingly, according to a post-task survey, workers seem to adjust their perception of appropriate payment for a task according to the change of difficulty levels over tasks in a sequence, suggesting a potential "anchoring" effect on task difficulty. The insensitivity of work quality to the change of task difficulty levels is possibly a result of a mixture of different effects.
Aggregating Human-Expert Opinions for Multi-Label Classification
Authors: Smirnov Evgueni , Zhang Hua , Peeters Ralf , Nikolaev Nikolay , Imkamp Maike
Published: October 2022

This paper introduces a multi-label classification problem to the field of human computation. The problem involves training data such that each instance belongs to a set of classes. The true class sets of all the instances are provided together with their estimations presented by m human experts. Given the training data and the class-set estimates of the m experts for a new instance, the multi-label classification problem is to estimate the true class set of that instance. To solve the problem we propose an ensemble approach. Experiments show that the approach can outperform the best expert and the majority vote of the experts.
Phylo and Open-Phylo: A Human-Computing Platform for Comparative Genomics
Authors: Waldispuhl Jerome , Blanchette Mathieu
Published: October 2022

Comparative genomics is a field of research that aims to provide us accurate mappings between the genetic material of multiple species. These techniques are useful for biomedical research and evolutionary studies. We present Phylo and Open-Phylo, an open citizen science platform and human computing-game for comparative genomic studies that reached more than 300,000 users.
Keynote Speakers
Authors: Grauman Kristen , Kraut Robert
Published: October 2022

Biographical details and talk summaries for the keynote talks presented at the AAAI Conference on Human Computation and Crowdsourcing.
Wish: Amplifying Creative Ability with Expert Crowds
Authors: Kulkarni Anand , Narula Prayag , Rolnitzky David , Kontny Nathan
Published: October 2022

We present Wish, a system that uses expert members of an online crowd to amplify a user’s ability to carry out complex creative tasks.  When presented with an incomplete or rough draft of creative content (for example, writing, programming, or art), Wish finds and recruits an expert to suggest a possible realization of the user’s vision. Wish contributes a new approach to crowdsourcing complex work. Rather than combining pools of unskilled workers to producing complex output, Wish uses the crowd to identify and recruit a pool of experts, then assigns a single high-quality expert to carry out the task.  This approach retains the benefits of using a crowd — scalability, speed, correctness, and responsiveness — while simplifying the process of crowdsourcing complex work. We demonstrate Wish in the context of three prototype tools.  Draft enables users to consult a crowd of authors for suggestions on writing.  Hack lets developers convert pseudocode into working code by pulling results on demand from a crowd of programmers. Sketch lets individuals convert rough sketches into fully-refined art by consulting artists.  We illustrate how novice creators can use Wish to amplify their ability, how expert designers can use Wish to explore design spaces and improve the speed of creation, and how new users can use Wish to gain feedback from experts.
Tranzzl!n9o: A Human Computation Approach to English Translation of Internet Lingo
Authors: Hong Ming-Tung , Hsu Yung-Jen
Published: October 2022

Lingo is an emerging language on the Internet. Providing a standardized definition remains difficult due to continuous changes made to its nature. We proposed Tranzzl!n9o, a crossword puzzle game for engaging crowds to translate Internet lingo. Players provide explanations for lingo in parallel and iteratively verify the explanations from other players. Crowd-sourced translations are very informative containing explanations as well as lingo usage.
Using Crowdsourcing to Generate Surrogate Training Data for Robotic Grasp Prediction
Authors: Unrath Matt , Zhang Zhifei , Goins Alex , Carpenter Ryan , Wong Weng-Keen , Balasubramanian Ravi
Published: October 2022

As an alternative to the laborious process of collecting training data from physical robotic platforms for learning robotic grasp quality prediction, we explore the use of surrogate training data from crowd-sourced evaluations of images of robotic grasps. We show that in certain regions of the grasp feature space, grasp predictors trained with this surrogate data were almost as accurate as predictors built using data from physical testing with robots.
Combining Non-Expert and Expert Crowd Work to Convert Web APIs to Dialog Systems
Authors: Huang Ting-Hao , Lasecki Walter , Ritter Alan , Bigham Jeffrey
Published: October 2022

Thousands of web APIs expose data and services that would be useful to access with natural dialog, from weather and sports to Twitter and movies. The process of adapting each API to a robust dialog system is difficult and time-consuming, as it requires not only programming but also anticipating what is mostly likely to be asked and how it is likely to be asked. We present a crowd-powered system able to generate a natural languageinterface for arbitrary web APIs from scratch without domain-dependent training data or knowledge.Our approach combines two types of crowd workers: non-expert Mechanical Turk workers interpret the functions of the API and elicit information from the user, and expert oDesk workers provide a minimal sufficient scaffolding around the API to allow us to make general queries.We describe our multi-stage process and present results for each stage.
Using Worker Quality Scores to Improve Stopping Rules
Authors: Abraham Ittai , Alonso Omar , Kandylas Vasilis , Patel Rajesh , Shelford Steven , Slivkins Alex
Published: October 2022

We consider the crowdsourcing task of learning the answer to simple multiple-choice microtasks. In order to provide statistically significant results, one often needs to ask multiple workers to answer the same microtask. A stopping rule is an algorithm that for a given microtask decides for any given set of worker answers if the system should stop and output an answer or iterate and ask one more worker. A quality score for a worker is a score that reflects the historic performance of that worker. In this paper we investigate how to devise better stopping rules given such quality scores. We conduct a data analysis on a large-scale industrial crowdsourcing platform, and use the observations from this analysis to design new stopping rules that use the workers’ quality scores in a non-trivial manner. We then conduct a simulation based on a real-world workload, showing that our algorithm performs better than the more naive approaches.
Data-Based Civic Participation
Authors: Le Dantec Christopher
Published: October 2022

Within the past five years, a new form of technology-mediated public participation that experiments with crowdsourced data production in place of community discourse has emerged. Examples of this class of system include SeeClickFix, PublicStuff, and Street Bump, each of which mediate feedback about local neighborhood issues and help communities mobilize resources to address those issues. The experiments being playing out by this new class of services are derived from a form of public participation built on the ideas of smart cities where residents and physical environments are instrumented to provide data to improve operational efficiency and sustainability (Caragliu, Del Bo, and Nijkamp 2011). Ultimately, smart cities is the application to local government all the efficiencies that computing has always promised—efficiencies of scale, of productivity, of data—minus the messiness and contention of citizenship that play out through more traditional modes of public engagement and political discourse. The question then, is what might it look like to incorporate more active forms of civic participation and issue advocacy in an app- and data-driven world?
Conducting Massively Open Online Social Experiments with Volunteer Science
Authors: Keegan Brian , Ognyanova Katherine , Foucault Welles Brooke , Riedl Christoph , Karbeyaz Ceyhun , Meleis Waleed , Lazer David , Radford Jason , Hoye Jefferson
Published: October 2022

Volunteer Science is an online platform enabling anyone to participate in social science research. The goal of Volunteer Science is to build a thriving community of research participants and social science researchers for Massively Open Online Social Experiments (“MOOSEs”). The architecture of Volunteer Science has been built to be open to researchers, transparent to participants, and to facilitate the levels of concurrency needed for large scale social experiments. Since then, 14 experiments and 12 survey-based interventions have been developed and deployed, with subjects largely being recruited through paid advertising, word of mouth, social media, search, and Mechanical Turk. We are currently replicating several forms of social research to validate the platform, working with new collaborators, and developing new experiments. Moving forward our priorities are continuing to grow our user base, developing quality control processes and collaborators, diversifying our funding models, and creating novel research.
Testing Pre-Annotation to Help Non-Experts Identify Drug-Drug Interactions Mentioned in Drug Product Labeling
Authors: Hernandez Andres , Hochheiser Harry , Horn John , Crowley Rebecca , Boyce Richard
Published: October 2022

In this study, a system for allowing combination of textmining and crowdsourcing of annotation approaches for detection of DDIs from drug package inserts is presented. An annotation study was designed to evaluate expert versus non-expert curation performance, and the impact of NLP pre-annotation on precision and recall on both groups. The design and development of the system and annotation study, consisted of three stages. First, our existing NLP pipeline for DDI extraction was improved, and it was used to preannotate 208 drug product labels with drug mentions and DDIs. Secondly, a DDI machine readable representation scheme was created using the Annotation Ontolgy. This model allowed us to load the NLP preannotated drug label sections into our plugin for human curation created using the Annotation tool DOMEO. Finally, the annotation study was performed along with usability questionnaires for collecting qualitative feedback. To our knowledge, this is the first study in comparing experts and non-experts for pharmacokinetic DDI annotation. Results showed lower performance on non-experts compared with expert annotation without the use of NLP,and an improvement of non-expert annotation performance using the NER module of the NLP assistance. Simplification of the workflow for NLP assisted annotation is necessary for scaling ourapproach.
Mapjack: A Mobile Wiki for Collaborative Map Making
Authors: Ntabathia Jude
Published: October 2022

Volunteered geographic information has introduced new forms of user generated content. Sadly, the generation of such content is not inclusive. There is need to improve not only tools but also social processes to bring about more inclusivity in participation.
A Sensor Network Approach to Managing Data Quality in Citizen Science
Authors: Wiggins Andrea , Lagoze Carl , Wong Weng-Keen , Kelling Steve
Published: October 2022

For most citizen science projects in which volunteers act as intelligent sensors, data quality cannot be determined through comparison to an objective ground truth nor through consensus. In this paper we discuss the approach implemented by eBird, based on strategies used in autonomous sensor networks, to address the challenge of establishing the accuracy of humans at the tasks of detection and identification.
CrowdAR: Augmenting Live Video with a Real-Time Crowd
Authors: Salisbury Elliot , Stein Sebastian , Ramchurn Sarvapali
Published: October 2022

Finding and tracking targets and events in a live video feed is important for many commercial applications, from CCTV surveillance used by police and security firms, to the rapid mapping of events from aerial imagery. However, descriptions of targets are typically provided in natural language by the end users, and interpreting these in the context of a live video stream is a complex task. Due to current limitations in artificial intelligence, especially vision, this task cannot be automated and instead requires human supervision. Hence, in this paper, we consider the use of real-time crowdsourcing to identify and track targets given by a natural language description. In particular we present a novel method for augmenting live video with a real-time crowd.
Reliable Aggregation of Boolean Crowdsourced Tasks
Authors: De Alfaro Luca , Polychronopoulos Vassilis , Shavlovsky Michael
Published: October 2022

We propose novel algorithms for the problem of crowdsourcing binary labels. Such binary labeling tasks are very common in crowdsourcing platforms, for instance, to judge the appropriateness of web content or to flag vandalism. We propose two unsupervised algorithms: one simple to implement albeit derived heuristically, and one based on iterated bayesian parameter estimation of user reputation models. We provide mathematical insight into the benefits of the proposed algorithms over existing approaches, and we confirm these insights by showing that both algorithms offer improved performance on many occasions across both synthetic and real-world datasets obtained via Amazon Mechanical Turk.
Moral Reminder as a Way to Improve Worker Performance on Amazon Mechanical Turk
Authors: Hwang Heeju
Published: October 2022

The present study explores a method to reduce abusive worker behavior on Amazon Mechanical Turk (AMT), namely reminding workers of moral standards. We manipulated workers’ awareness of moral standards via the presence or the absence of an honesty statement in a survey. The results showed that the honesty statement significantly improved workers’ performance during the first half of the survey. This suggests that a moral reminder is a simple and efficient way to reduce abusive worker behavior in a relatively short survey on AMT.
Learning to Hire Teams
Authors: Singla Adish , Horvitz Eric , Kohli Pushmeet , Krause Andreas
Published: October 2022

Crowdsourcing and human computation are being employed in sophisticated projects that require the solution of a heterogeneous set of tasks. We explore the challenge of composing or hiring an effective team from an available pool of applicants for performing tasks required for such projects on an ongoing basis. How can one optimally spend budget to learn the expertise of workers as part of recruiting a team? How can one exploit the similarities among tasks as well as underlying social ties or commonalities among the workers for faster learning? We tackle these decision-theoretic challenges by casting them as an instance of online learning for best action selection with side-observations. We present algorithms with PAC bounds on the required budget to hire a near-optimal team with high confidence. We evaluate our methodology on simulated problem instances using crowdsourcing data collected from the Upwork platform.
LoRUS: A Mobile Crowdsourcing System for Efficiently Retrieving the Top-k Relevant Users in a Spatial Window
Authors: Mondal Anirban , Raravi Gurulingesh , Chugh Amandeep , Mukherjee Tridib
Published: October 2022

The prevalence of mobile devices and applications strongly motivate mobile crowdsourcing for facilitating location-dependent services. We propose LoRUS, a Location-based Relevant User determination System for efficiently retrieving the top-k relevant mobile users in a given spatial window.
Extracting Structured Information via Automatic + Human Computation
Authors: Pavlick Ellie , Callison-Burch Chris
Published: October 2022

We present a system for extracting structured information from unstructured text using a combination of information retrieval, natural language processing, machine learning, and crowdsourcing. We test our pipeline by building a structured database of gun violence incidents in the United States. The results of our pilot study demonstrate that the proposed methodology is a viable way of collecting large-scale, up-to-date data for public health, public policy, and social science research.
Assigning Tasks to Workers by Referring to Their Schedules in Mobile Crowdsourcing
Authors: Hadano Mayumi , Nakatsuji Makoto , Toda Hiroyuki , Koike Yoshimasa
Published: October 2022

This paper focuses on task assignments to workers in mobile crowdsoucing systems. The current method does not work so well since it considers only workers who are ready to work at the time of optimization. Our method handles workers' day-long schedules, creates a `time-extended' worker-task graph that expresses the relationships between workers and tasks over a time period and finds the best set of worker-task-time triples. Our evaluation using real world visiting logs shows it increases the rate of assigned tasks by more than 8.2% compared with a state-of-the-art assignment method.
A Cross-Cultural Study of Motivations to Participate in a Crowdsourcing Project to Support People with Disabilities
Authors: Layas Fatma , Petrie Helen , Power Christopher
Published: October 2022

Crowdsourcing has now been used in many areas, including to support people with disabilities. We explored the perceptions of students from two countries with very different cultures (Libya and the UK) of what would motivate them to participate in a crowdsourcing project (DescribeIT) to support their blind and partially sighted peers by describing images in digital learning resources. Students reported that they were interested to participate and particularly motivated by their sense of altruism and wanting to help others. The results also showed that British students reported a mix of intrinsic and extrinsic motivations, whereas for Libyan participants intrinsic motivations dominated over extrinsic. The results of this study will help us design the DescribeIT system to motivate students most effectively and should make developers of international crowdsourcing projects aware that motivations may differ between participants from different cultures.
Acquiring Reliable Ratings from the Crowd
Authors: Valeri Beatrice , Elbassuoni Shady , Amer-Yahia Sihem
Published: October 2022

We address the problem of acquiring reliable ratings of items such as restaurants or movies from the crowd. We propose a crowdsourcing platform that takes into consideration the workers’ skills with respect to the items being rated and assigns workers the best items to rate. Our platform focuses on acquiring ratings from skilled workers and for items that only have a few ratings. We evaluate the effectiveness of our system using a real-world dataset about restaurants.
Job Complexity and User Attention in Crowdsourcing Microtasks
Authors: Rothwell Spencer , Carter Steele , Elshenawy Ahmad , Braga Daniela
Published: October 2022

This paper examines the importance of presenting simple, intuitive tasks when conducting microtasking on crowdsourcing platforms. Most crowdsourcing platforms allow the maker of a task to present any length of instructions to crowd workers who participate in their tasks. Our experiments show, however, most workers who participate in crowdsourcing microtasks do not read the instructions, even when they are very brief. To facilitate success in microtask design, we highlight the importance of making simple, easy to grasp tasks that do not rely on instructions for explanation.
Rigorously Collecting Commonsense Judgments for Complex Question-Answer Content
Authors: Sameki Mehrnoosh , Barua Aditya , Paritosh Praveen
Published: October 2022

Community Question Answering (CQA) websites are a popular tool for internet users to fulfill diverse information needs. Posted questions can be multiple sentences long and span diverse domains. They go beyond factoid questions and can be conversational, opinion-seeking and experiential questions, that might have multiple, potentially conflicting, useful answers from different users. In this paper, we describe a large-scale formative study to collect commonsense properties of questions and answers from 18 diverse communities from stackexchange.com. We collected 50,000 human judgments on 500 question-answer pairs. Commonsense properties are features that humans can extract and characterize reliably by using their commonsense knowledge and native language skills, and no special domain expertise is assumed. We report results and suggestions for designing human computation tasks for collecting commonsense semantic judgments.