bluuebunny/crossref_metadata_2025 huggingface dataset:
Learning Director Agent Strategies: An Inductive Framework for Modeling Director Agents
Authors: Lee Seung , Mott Bradford , Lester James
Published: October 2022

Interactive narrative environments offer significant potential for creating engaging narrative experiences that are tailored to individual users. Increasingly, applications in education, training, and entertainment are leveraging narrative to create rich interactive experiences in virtual storyworlds. A key challenge posed by these environments is devising accurate models of director agents’ strategies that determine the most appropriate director action to perform for crafting customized story experiences. A promising approach is developing an empirically informed model of director agents’ decision-making strategies. In this paper, we propose a framework for learning models of director agent decision-making strategies by observing human-human interactions in an interactive narrative-centered learning environment. The results are encouraging and suggest that creating empirically driven models of director agent decision-making is a promising approach to interactive narrative.
Towards a Computational Model of Narrative Visualization
Authors: Baikadi Alok , Goth Julius , Mitchell Christopher , Ha Eun , Mott Bradford , Lester James
Published: October 2022

The task of narrative visualization has been the subject of increasing interest in recent years. Much like data visualization, narrative visualization offers users an informative and aesthetically pleasing perspective on “storydata.” Automatically creating visual representations ofnarratives poses significant computational challenges due to the complex affective and causal elements, among other things, that must be realized in visualizations. In addition, narratives that are composed by novice writers pose additional challenges due to the disfluencies stemming from ungrammatical text. In this paper, we introduce the NARRATIVE THEATRE, a narrative visualization system under development in our laboratory that generates narrative visualizations from middle school writers’ text. The NARRATIVE THEATRE consists of a rich writing interface, a robust natural language processor, a narrative reasoner, and a storyboard generator. We discuss design issues bearing on narrative visualization, introduce the NARRATIVE THEATRE, and describe narrative corpora that have been collected to study narrative visualization. We conclude with a discussion of a narrative visualization research agenda.
Causality in Hundreds of Narratives of the Same Events
Authors: Tomai Emmett , Thapa Laxman , Gordon Andrew , Kang Sin-Hwa
Published: October 2022

Empirical research supporting computational models of narrative is often constrained by the lack of large-scale corpora with deep annotation. In this paper, we report on our annotation and analysis of a dataset of 283 individual narrations of the events in two short video clips. The utterances in the narrative transcripts were annotated to align with known events in the source videos, offering a unique opportunity to study the regularities and variations in the way that different people describe the exact same set of events. We identified the causal relationships between events in the two video clips, and investigated the role that causality plays in determining whether subjects will mention a particular story event and the likelihood that these events will be told in the order that they occurred in the original videos.
Seven Design Challenges for Fully-realized Experience Management
Authors: Roberts David
Published: October 2022

Drama Managers, a specific type of the more general Experience Manager, have become a common subject of study in the interactive narrative literature. With a range of representational and computational approaches, authors have repeatedly developed techniques that enable computers to generate, reason about, and adapt narratives in an interactive virtual setting. In order to fully realize an experience manager, seven representational and computational problems need to be solved, generally on a case-by-case basis. In other words, the choice to use an Experience Manager is the choice to model the design as, and implement solutions to, seven inter-dependent design problems. We explicitly articulate those design problems and provide a number of examples of methods that both motivate the design problems as well as illustrate a range of approaches to solving them.
A Rule-Based Framework for Modular Development of In-Game Interactive Dialogue Simulation
Authors: Tomai Emmett , Salinas David , Salazar Rosendo
Published: October 2022

In this paper, we discuss approaches to dialogue in interactive video games and interactive narrative research. We propose that situating interactive dialogue in the simplified expectations of video games is a profitable way to investigate computational dialogue simulation. Taking cues from existing physical simulations such as combat, we propose a hypothetical game environment and design goals for an embedded interactive dialogue system. We present a modular framework targeted at that environment, which is designed to enable incremental development and exploration of dialogue concepts. We describe this framework together with a work-in-progress system for simulating simple in-game negotiation dialogues.
Computational Caricatures: Probing the Game Design Process with AI
Authors: Smith Adam , Mateas Michael
Published: October 2022

We propose the creation of computational caricatures as a design research practice that aims to advance understanding of the game design process and to develop the reusable technology for design automation. Computational caricatures capture and exaggerate statements about the game design process in the form of computational systems (i.e. software and hardware). In comparison with empirical interviews of game designers, arguments from established design theory, and the creation of neutral simulations of the design process, computational caricatures provide more direct access to inquiry and insight about design. Further, they tangibly demonstrate architectures and subsystems for a new generation of human-assisting design support systems and adaptive games that embed aspects of automated design in their runtime processes. In this paper, we frame the idea of computational caricature, review several existing design automation prototypes through the lens of caricature, and call for more design research to be done following this practice.
Game Metrics Without Players: Strategies for Understanding Game Artifacts
Authors: Nelson Mark
Published: October 2022

Game metrics are an approach to understanding games and gameplay by analyzing and visualizing information collected from players in playtests. This paper proposes that another source of metrics is the game itself, and that not all information needs to (or ought to) come from empirical playtests. I discuss seven strategies for extracting information from games, and discuss how the information retrieved in this manner relates to empirical playtest metrics---which it differs from but can often complement.
Preface
Authors: Riedl Mark , Sukthankar Gita
Published: October 2022

Welcome to the Eighth AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment (AIIDE’12). Computer games are a staple of today’s popular culture, and have been called the preemi- nent form of entertainment of the 21st century. The computer game industry has become a multibillion- dollar commercial enterprise. Game system require- ments are an important driver of hardware and soft- ware innovation in the computer industry.
Adapting AI Behaviors To Players in Driver San Francisco: Hinted-Execution Behavior Trees
Authors: Ocio Sergio
Published: October 2022

The creative nature of games makes trying new ideas desirable, but these changes are sometimes very risky. We need to find ways to minimize risks while we build innovative experiences. Driver San Francisco did this by using Hinted-execution Behavior Trees; this technique allows developers to modify existing AI behaviors dynamically with very low risk, and was used to adapt Driver’s getaway AI to players’ skills.
Evolving Personalized Content for Super Mario Bros Using Grammatical Evolution
Authors: Shaker Noor , N. Yannakakis Georgios , Togelius Julian , Nicolau Miguel , O'Neill Michael
Published: October 2022

Adapting game content to a particular player's needs and expertise constitutes an important aspect in game design. Most research in this direction has focused on adapting game difficultyto keep the player engaged in the game. Dynamic difficulty adjustment, however, focuses on one aspect of the gameplay experience by adjusting the content to increase ordecrease perceived challenge. In this paper, we introduce a method for automatic level generation for the platform game Super Mario Bros using grammatical evolution. The grammatical evolution-based level generator is used to generate player-adapted content by employing an adaptation mechanism as a fitness function in grammatical evolution to optimizethe player experience of three emotional states: engagement, frustration and challenge. The fitness functions used are models of player experience constructed in our previous work from crowd-sourced gameplay data collected from over 1500 game sessions.
If Not Now, Where? Time and Space Equivalency in Strategy Games
Authors: Wetzel Baylor , Anderson Kyle , Koutstaal Wilma , Gini Maria
Published: October 2022

Spatiotemporal reasoning is a fundamental contributor to effective problem solving. In an effort to design better problem-solving agents, we examined and evaluated the strategies that humans use to solve Tower Defense puzzles, a complex and popular class of real-time strategy games. A consistent and unexpected finding was that humans frequently treated time and space as equivalent. Players stated temporal goals but solved spatial problems. An analysis of human data and computer simulations showed that re-representing temporal problems as spatial problems was beneficial, but treating the two separately can lead to higher scores. The work presented here holds several possibilities for level designers and others who design and analyze maps and spatial arrangements for domains requiring strategic reasoning.
Enhancing the Believability of Character Behaviors Using Non-Verbal Cues
Authors: Desai Neesha , Szafron Duane
Published: October 2022

Characters are vital to large video game worlds as they bring a sense of life to the world. However, background characters are known to rarely exhibit any sign of motivated behavior or emotional state. We want to change this by assigning these characters emotions that can be identified through their non-verbal behavior. We feel the addition of emotion will allow players to feel more connected to the game world and make the game world more believable. This paper presents the results of an experiment to test two ways of conveying emotion: 1) through a character's gait and 2) through a character's interactions with the game world. Results from the experiment suggest that a combination of gait and interactions is the most effective method to convey emotion.
Spatial Game Signatures for Bot Detection in Social Games
Authors: Barik Titus , Harrison Brent , Roberts David , Jiang Xuxian
Published: October 2022

Bot detection is an emerging problem in social games that requires different approaches from those used in massively multi-player online games (MMOGs). We focus on mouse selections as a key element of bot detection. We hypothesize that certain interface elements result in predictable differences in mouse selections, which we call spatial game signatures, and that those signatures can be used to model player interactions that are specific to the game mechanics and game interface. We performed a study in which users played a game representative of social games. We collected in-game actions, from which we empirically identified these signatures, and show that these signatures result in a viable approach to bot detection. We make three contributions. First, we introduce the idea of spatial game signatures. Second, we show that the assumption that mouse clicks are normally distributed about the center of buttons is not true for every interface element. Finally, we provide methodologies for using spatial game signatures for bot detection.
Punch and Judy AI Playset: A Generative Farce Manifesto, Or, The Tragical Comedy or Comical Tragedy of Predicate Calculus
Authors: Horswill Ian
Published: October 2022

Building complete interactive narrative systems is hard. Building systems that are satisfying for naïve users is especially hard since small deficiencies in component technologies can easily destroy the experience for a user. In this paper I argue that we can ameliorate some of these technical limitations through careful choice of genre and style, and discuss a number of properties of farce that make it a particularly attractive choice. Then I will describe work in progress on Punch and Judy AI Playset, a system that allows users to explore possible narratives in the Punch and Judy story world.
Generating Narrative Action Schemas for Suspense
Authors: Giannatos Spyridon , Cheong Yun-Gyung , Nelson Mark , Yannakakis Georgios N.
Published: October 2022

A bottleneck in interactive storytelling is the authorial burden of writing narrative units, and connecting them to the interactive narrative structure. To address this problem, we present a hybrid approach that combines AI planning and evolutionary optimization in order to generated new plan operators representing possible story actions, within the framework of a planning-based interactive narrative system. We focus our work on inventing plan operators that are useful for contributing to suspenseful interactive stories, using suspense metrics that have been proposed in the literature. We devise an encoding scheme for converting a plan operator into a genetic-algorithm chromosome and vice versa, respecting constraints that are needed for an operator to be well-formed. We discuss the performance of the system, and several examples from preliminary experiments carried out to evaluate the evolved operators.
Gestural Interactions for Interactive Narrative Co-Creation
Authors: Piplica Andreya , Deleon Chris , Magerko Brian
Published: October 2022

This paper describes a gestural approach to interacting with interactive narrative characters that supports co-creativity. It describes our approach using a Microsoft Kinect to created a short scene with an intelligent avatar and an AI-controlled actor. It describes our preliminary user studies and a recommendation for future evaluation.
Preface
Authors: Buro Michael
Published: October 2022

The covered topics in the Adversarial Real-Time Games 2012 AIIDE Workshop range from RTS game unit micro- management (kiting, building unit formations, minimax search for combat), over machine learning approaches to army clustering and strategy improvement, to multi-agent pursuit-evasion and policy switching in RTS games
Emergent Remix Culture in an Anonymous Collaborative Art System
Authors: Tuite Kathleen , Smith Adam
Published: October 2022

Many crowdsourcing systems have a contribution model that is shallow but massively parallel, with contributors rarely processing or iterating upon the work of others. Few systems, even those crowdsourcing creativity or artistic talent, are designed to allow deep chains where the ideas of one individual feed into and directly inspire another individual. To explore the ways in which creative ideas arise and evolve under the influence of specific artifacts created by others, we examine patterns from over 50,000 sketches created and uploaded with Sketch-a-bit, a collaborative mobile drawing application in which each sketch is directly prompted by a previous sketch. In this paper, we report results from two analyses of content created in the system's first two years of deployment. First, we apply qualitative coding to survey the range of effort and creativity in user actions (including actions ranging from unintentioned scribbles to subtly inspired reimaginations of source material through the unexpected preparation of blank canvases for others). Second, we perform an exploratory analysis of large-scale behaviors manifest in chains or trees of sketches (such as open-ended conversations and structured gameplay). The intent of this work is to describe an iterative model of collaborative creativity and to demonstrate a range of remixing behaviors that can be expected to arise in unrestricted, anonymous collaborative creativity applications.
Supporting STEM Learning With Gaming Technologies: Principles For Effective Design
Authors: Borge Marcela , White Barbara
Published: October 2022

In this paper, methods and models for the design of educational interventions and usable systems are presented and synthesized. The purpose is to suplliment the design process with educational considerations and discern design principles for the development of serious STEM games. This synthesis can contribute to the design of the next generation of technologically enhanced learning environments.
Invited Talks
Authors: Evans Richard , Harrell D. , Laidacker Aleissia , Abercrombie John
Published: October 2022

Abstracts of the invited talks presented at the Ninth Artificial Intelligence and Interactive Digital Entertainment Conference.
Endless Web
Authors: Smith Gillian , Othenin-Girard Alexei , Whitehead Jim , Wardrip-Fruin Noah
Published: October 2022

Endless Web is a game where a procedural content generator is deeply integrated into the mechanics, dynamics, and aesthetics. The game involves players exploring the generative space of its built-in content generator, by making decisions that influence the parameters to the generator. Players must build strategies around the choices they are making to ensure that levels are at an appropriate level of challenge for them, while simultaneously exploring to discover goals hidden in layers of the generative space.
AI Authoring for Virtual Characters in Conflict
Authors: Gomes Paulo , Jhala Arnav
Published: October 2022

Game developers strive to have engaging believable characters in their work. One of the elements that has been pointed out as contributing to believability is social behavior. A category of social behavior is interpersonal conflict. In our current research we compare two AI approaches to model NPC conflict resolution strategies: one using the reactive planning language ABL and another using the AI framework FAtiMA. We identify the following metrics to evaluate social behavior modeling: mapping theory, emotion, model checking, variability, policy change. In our analysis we found it was easier to map conflict concepts in ABL and the model checking process was faster. FAtiMA had better support for emotion and other emergent attributes.
Player Skill Modeling in Starcraft II
Authors: Avontuur Tetske , Spronck Pieter , Van Zaanen Menno
Published: October 2022

Starcraft II is a popular real-time strategy (RTS) game, in which players compete with each other online. Based on their performance, the players are ranked in one of seven leagues. In our research, we aim at constructing a player model that is capable of predicting the league in which a player competes, using observations of their in-game behavior. Based on cognitive research and our knowledge of the game, we extracted from 1297 game replays a number of features that describe skill. After a preliminary test, we selected the SMO classifier to construct a player model, which achieved a weighted accuracy of 47.3% (SD = 2.2). This constitutes a significant improvement over the weighted baseline of 25.5% (SD = 1.1). We tested from what moment in the game it is possible to predict a player’s skill, which we found is after about 2.5 minutes of gameplay, i.e., even before the players have confronted each other within the game. We conclude that our model can predict a player’s skill early in the game.
Evaluating Planning-Based Experience Managers for Agency and Fun in Text-Based Interactive Narrative
Authors: Ramirez Alejandro , Bulitko Vadim , Spetch Marcia
Published: October 2022

Artificial intelligence (AI) techniques have been applied to video games to make the overall experience more enjoyable. In games with interactive storytelling (IS), player actions can substantially affect plot events and plot characters. Therefore, AI planning techniques have been used to shape the plot inresponse to player actions that conflict with authorial goals. While such methods are poised to increase player fun andagency, two recent implementations (ASD and PAST) have not been formally evaluated to date. In this paper we do so via a series of user studies for the first time. We show that ASD significantly enhances fun and agency, whereas PAST gets mixed results with an interaction between effects of the experience manager and player prior gaming experience in one user study, and marginally significant results for increased agency in a study with a constrained story domain.
Empirical Evaluation of the Automatic Generation of a Component-Based Software Architecture for Games
Authors: Llansó David , Gómez-Martín Pedro , Gómez-Martín Marco , González-Calero Pedro
Published: October 2022

Component-based architecture is the software design of choice for most modern games and game engines, allowing for extensible and reusable software that easily adapts to changes in requirements. Nevertheless, designing a component-based architecture is not an easy achievement and some techniques have been previously proposed for the automatic generation of a component distribution to implement a given declarative model of the entities in the game. The goal of the work presented here is to empirically compare the quality of the component distributions designed by programmers, with those others automatically obtained with such techniques.
Evolving Playable Content for Cut the Rope through a Simulation-Based Approach
Authors: Shaker Noor , Shaker Mohammad , Togelius Julian
Published: October 2022

In order to automatically generate high-quality game levels, one needs to be able to automatically verify that the levels are playable. The simulation-based approach to playability testing uses an artificial agent to play through the level, but building such an agent is not always an easy task and such an agent is not always readily available. We discuss this prob- lem in the context of the physics-based puzzle game Cut the Rope, which features continuous time and state space, mak- ing several approaches such as exhaustive search and reactive agents inefficient. We show that a deliberative Prolog-based agent can be used to suggest all sensible moves at each state, which allows us to restrict the search space so that depth-first search for solutions become viable. This agent is successfully used to test playability in Ropossum, a level generator based on grammatical evolution. The method proposed in this paper is likely to be useful for a large variety of games with similar characteristics.
Generating Maps Using Markov Chains
Authors: Snodgrass Sam , Ontanon Santiago
Published: October 2022

In this paper we outline a method of procedurally generating maps using Markov Chains. Our method attempts to learn what makes a "good" map from a set of given human-authored maps, and then uses those learned patterns to generate new maps. We present an empirical evaluation using the game "Super Mario Bros.," showing encouraging results.
An Exploration Tool for Predicting Stealthy Behaviour
Authors: Tremblay Jonathan , Torres Pedro , Rikovitch Nir , Verbrugge Clark
Published: October 2022

Stealthy movement is an important part of many games in the First Person Shooter (FPS) and Role Playing Games (RPG) genres.  Structuring a game level to match stealth goals, however, is difficult, and can depend on subtle and fragile interactions between the game space, enemy motion, and other factors. Here we apply a probabilistic path-finding approach to efficiently analyze a 2D space and find stealthy paths. This approach naturally accommodates variation in the level design, numbers and movements of enemies, fields of view, and player start and goal placement. Our design is integrated directly into the Unity 3D game development framework, allowing for interactive and highly dynamic exploration of how different virtual spaces and enemy configurations affect the potential for stealthly movement by players, or other NPCs.
A Generic Method for Classification of Player Behavior
Authors: Etheredge Marlon , Lopes Ricardo , Bidarra Rafael
Published: October 2022

Player classification allows for considerable improvements on both game analytics and game adaptivity. With this paper we aim at reversing the ad-hoc tendency in player classification methods, by proposing an approach to player classification that can be integrated across different games and genres and is particularly suited to be used by game designers. This paper describes our generic method of interaction-based player classification, which consists of three components: (i) intercepting player interactions, (ii) finding player types through fuzzy cluster analysis and (iii) classification using Hidden Markov Models (HMM). To showcase our method we developed Blindmaze, a simple web-based hidden maze game publicly available, featuring a bounded set of interactions. All data collected from a game is interaction-based, requiring minimal implementation effort from the game developers. It is concluded that our method makes player classification even more available by making it generic and re-usable across different games.
An Argument for Large-Scale Breadth-First Search for Game Design and Content Generation via a Case Study of Fling!
Authors: Sturtevant Nathan
Published: October 2022

Search is a recognized technique for procedural content generation and game design, and it has been used successfully as part of commercial and academic games. In this context, search has almost always referred to selective search, as opposed to larger brute-force searches. The argument against brute-force search is that the state spaces of the games are almost always too large to be amenable for brute-force search. We believe, however, that brute-force search should not be too quickly dismissed. State spaces with trillions or tens of trillions states can now be exhaustively searched with relative ease, and growth in parallelism and computational power is expected to continue to scale this trend. We believe that this, combined with appropriate abstraction, will allow exhaustive search to be applied to many problems once thought to be prohibitively large. We explore this argument in the context of a game called "Fling!," available for mobile devices, showing a system for interactively designing and analyzing puzzles.
A Conceptual Blending Approach to the Generation of Cognitive Scripts for Interactive Narrative
Authors: Permar Justin , Magerko Brian
Published: October 2022

This paper presents a computational approach to the generation of cognitive scripts employed in freeform activities such as pretend play. Pretend play activities involve a high degree of improvisational narrative construction using cognitive scripts acquired from everyday experience, cultural experiences, and previous play experiences. Our computational model of cognitive script generation, based upon conceptual integration theory, applies operations to familiar scripts to generate new blended scripts.
Toward Character Role Assignment for Natural Language Stories
Authors: Valls-Vargas Josep , Ontañón Santiago , Zhu Jichen
Published: October 2022

In this paper we propose a method for automatically assigning narrative roles to characters in stories. To achieve this goal our proposal is to combine natural language processing techniques with domain knowledge extracted from Propp's morphology of the folktale. We use a matrix that encodes the narrative domain knowledge representing the interactions between character roles.
Science Considered Harmful
Authors: Horswill Ian
Published: October 2022

As the INT community matures, one hears increasing calls for the field to become more scientific, and even prior restraint in which researchers avoid a project entirely out fear it wouldn’t be sufficiently scientific. In this position paper, I’ll argue that science, and more generally rigor, are not well served by a discourse that labels things as “science” and “not science.” In particular, I will argue that we need to open ourselves to the adoption of methods from the arts, including case studies and the practice of studio critique.
Improving Goal Recognition in Interactive Narratives with Models of Narrative Discovery Events
Authors: Baikadi Alok , Rowe Jonathan , Rowe Jonathan , Mott Bradford , Mott Bradford , Lester James , Lester James
Published: October 2022

Computational models of goal recognition hold considerable promise for enhancing the capabilities of drama managers and director agents for interactive narratives. The problem of goal recognition, and its more general form plan recognition, has been the subject of extensive investigation in the AI community. However, there have been relatively few empirical investigations of goal recognition models in the intelligent narrative technologies community to date, and little is known about how computational models of interactive narrative can inform goal recognition. In this paper, we investigate a novel goal recognition model based on Markov Logic Networks (MLNs) that leverages narrative discovery events to enrich its representation of narrative state. An empirical evaluation shows that the enriched model outperforms a prior state-of-the-art MLN model in terms of accuracy, convergence rate, and the point of convergence.
Interactive Musical Partner: A Demonstration of Musical Personality Settings for Influencing the Behavior of an Interactive Musical Generation System
Authors: Albert Jeffrey
Published: October 2022

The Interactive Musical Partner (IMP) is software designed for use in duo improvisations, with one human improviser and one instance of IMP, focusing on a freely improvised duo aesthetic. IMP has Musical Personality Settings (MPS) that can be set prior to performance, and these MPS guide the way IMP responds to musical input. The MPS also govern the probability of particular outcomes from IMP's creative algorithms. The IMP uses audio data feature extraction methods to listen to the human partner, and react to, or ignore, the human‚Äôs musical input, based on the current MPS. This demonstration shows how the MPS interface with IMP's generative algorithm.
Jack 1: Pattern and Shape in Music Generation
Authors: Handelman Eliot , Sigler Andie
Published: October 2022

Jack 1 is an original meta-music piece which randomly builds a multilevel data structure to generate, extend andsequence musical shapes. Jack 1 is presented as an experiment in generating high-level, engaging music using simple ideas about pattern, shape and supershape,growth, and formal zoning.
Organizers
Authors: Pasquier Philippe
Published: October 2022

List of organizers of the Musical Metacreation workshop held at the Ninth AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment.
Improving Behaviour and Decision Making for Companions in Modern Digital Games
Authors: Tremblay Jonathan
Published: October 2022

Non-player character companions in video games should cooperate with players, understand them, and follow their lead during gameplay. In current games, however, companions tend to exhibit mainly static behaviours, and rarely live up to player expectations. In general, our work is aimed at improving this situation, developing both techniques and tools which allow companion NPCs to behave more appropriately, respecting player preferences and offering a more immersive gameplay for players.
Gesticulate for Health’s Sake! Understanding the Use of Gestures as an Input Modality for Microtask Crowdsourcing
Authors: Allen Garrett , Hu Andrea , Gadiraju Ujwal
Published: October 2022

Human input is pivotal in building reliable and robust artificial intelligence systems. By providing a means to gather diverse, high-quality, representative, and cost-effective human input on demand, microtask crowdsourcing marketplaces have thrived. Despite the unmistakable benefits available from online crowd work, the lack of health provisions and safeguards, along with existing work practices threatens the sustainability of this paradigm. Prior work has investigated worker engagement and mental health, yet no such investigations into the effects of crowd work on the physical health of workers have been undertaken. Crowd workers complete their work in various sub-optimal work environments, often using a conventional input modality of a mouse and keyboard. The repetitive nature of microtask crowdsourcing can lead to stress-related injuries, such as the well-documented carpal tunnel syndrome. It is known that stretching exercises can help reduce injuries and discomfort in office workers. Gestures, the act of using the body intentionally to affect the behavior of an intelligent system, can serve as both stretches and an alternative form of input for microtasks. To better understand the usefulness of the dual-purpose input modality of ergonomically-informed gestures across different crowdsourced microtasks, we carried out a controlled 2 x 3 between-subjects study (N=294). Considering the potential benefits of gestures as an input modality, our results suggest a real trade-off between worker accuracy in exchange for potential short to long-term health benefits.
Identifying Possible Winners in Ranked Choice Voting Elections with Outstanding Ballots
Authors: Jelvani Alborz , Marian Amelie
Published: October 2022

Several election districts in the US have recently moved to ranked-choice voting (RCV) to decide the results of local elections. RCV allows voters to rank their choices, and the results are computed in rounds, eliminating one candidate at a time. RCV ensures fairer elections and has been shown to increase elected representation of women and people of color.
A main drawback of RCV is that the round-by-round process requires all the ballots to be tallied before the results of an election can be calculated. With increasingly large portions of ballots coming from absentee voters, RCV election outcomes are not always apparent on election night, and can take several weeks to be published, leading to a loss of trust in the electoral process from the public.
In this paper, we present an algorithm for efficiently computing possible winners of RCV elections from partially known ballots and evaluate it on data from the recent New York City Primary elections. We show that our techniques allow to significantly narrow down the field of possible election winners, and in some case identify the winner as soon as election night despite a number of yet-unaccounted absentee ballots, providing more transparency in the electoral process.
An Introduction to the Zooniverse
Authors: Smith Arfon , Lynn Stuart , Lintott Chris
Published: October 2022

The Zooniverse (zooniverse.org) began in 2007 with the launch of Galaxy Zoo, a project in which more than 175,000 people provided shape analyses of more than 1 million galaxy images sourced from the Sloan Digital Sky Survey. In this session we will outline the current architecture of the Zooniverse platform and introduce new functionality being developed that should be of interest to the HCOMP community.
Real-Time Drawing Assistance through Crowdsourcing
Authors: Limpaecher Alex , Feltman Nicolas , Treuille Adrien , Cohen Michael
Published: October 2022

We propose a new method for the large-scale collection and analysis of drawings by using a mobile game specifically designed to collect such data. Analyzing this crowdsourced drawing database, we build a spatially varying model of artistic consensus at the stroke level. We then present a surprisingly simple stroke- correction method which uses our artistic consensus model to improve strokes in real-time. Importantly, our auto-corrections run interactively and appear nearly in- visible to the user while seamlessly preserving artistic intent. Closing the loop, the game itself serves as a plat- form for large-scale evaluation of the effectiveness of our stroke correction algorithm.
ARTigo: Building an Artwork Search Engine With Games and Higher-Order Latent Semantic Analysis
Authors: Wieser Christoph , Bry François , Bérard Alexandre , Lagrange Richard
Published: October 2022

This article describes how a semantic search engine has been build from, and still is continuously improved by, a semantic analysis of the “footprints” left by players on the gaming Web platform ARTigo. The Web platform offers several Games With a Purpose (GWAPs) some of which have been specifically designed to collect the data needed for building the artwork search engine. ARTigo is a “tagging ecosystem” of games that cooperate so as to gather a wide range of information on artworks. The ARTigo ecosystem generates a folksonomy saved as 3rd-order tensor, that is a generalization of a matrix, the three orders or dimensions of which represent (1) who (2) tagged an (3) an artwork. The semantic search engine is build using a non-trivial generalization of the well-known, matrix-based, Latent Semantic Analysis (LSA) methods and algorithms. ARTigo is in service for five years and is subject to an active research constantly resulting in new developments, some of which are reported about for the first time in this article.
Sponsors
Authors: Hartmann Björn , Horvitz Eric
Published: October 2022

List of sponsors of the First AAAI Conference on Human Computation and Crowdsourcing.
Dwelling on the Negative: Incentivizing Effort in Peer Prediction
Authors: Witkowski Jens , Bachrach Yoram , Key Peter , Parkes David
Published: October 2022

Agents are asked to rank two objects in a setting where effort is costly and agents differ in quality (which is the probability that they can identify the correct, ground truth, ranking). We study simple output-agreement mechanisms that pay an agent in the case she agrees with the report of another, and potentially penalizes for disagreement through a negative payment. Assuming access to a quality oracle, able to determine whether an agent's quality is above a given threshold, we design a payment scheme that aligns incentives so that agents whose quality is above this threshold participate and invest effort. Precluding negative payments leads the expected cost of this quality-oracle mechanism to increase by a factor of 2 to 5 relative to allowing both positive and negative payments. Dropping the assumption about access to a quality oracle, we further show that negative payments can be used to make agents with quality lower than the quality threshold choose to not to participate, while those above continue to participate and invest effort. Through the appropriate choice of payments, any design threshold can be achieved. This self-selection mechanism has the same expected cost as the cost-minimal quality-oracle mechanism, and thus when using the self-selection mechanism, perfect screening comes for free.
Winner-Take-All Crowdsourcing Contests with Stochastic Production
Authors: Cavallo Ruggiero , Jain Shaili
Published: October 2022

We study winner-take-all contests for crowdsourcing procurement in a model of costly effort and stochastic production. The principal announces a prize value P, agents simultaneously select a level of costly effort to exert towards production, yielding stochastic quality results, and then the agent who produces the highest quality good is paid P by the principal. We derive conditions on the probabilistic mapping from effort to quality under which this contest paradigm yields efﬁcient equilibrium outcomes, and demonstrate that the conditions are satisﬁed in a range of canonical settings.
Crowdsourcing Multi-Label Classification for Taxonomy Creation
Authors: Bragg Jonathan , Mausam - , Weld Daniel
Published: October 2022

Recent work has introduced CASCADE, an algorithm for creating a globally-consistent taxonomy by crowdsourcing microwork from many individuals, each of whom may see only a tiny fraction of the data (Chilton et al. 2013). While CASCADE needs only unskilled labor and produces taxonomies whose quality approaches that of human experts, it uses significantly more labor than experts. This paper presents DELUGE, an improved workflow that produces taxonomies with comparable quality using significantly less crowd labor. Specifically, our method for crowdsourcing multi-label classification optimizes CASCADE’s most costly step (categorization) using less than 10% of the labor required by the original approach. DELUGE’s savings come from the use of decision theory and machine learning, which allow it to pose microtasks that aim to maximize information gain.
Using Visibility to Control Collective Attention in Crowdsourcing
Authors: Lerman Kristina , Hogg Tad
Published: October 2022

Online crowdsourcing provides new opportunities for ordinary people to create original content. This has led to a rapidly growing volume of user-generated content, and consequently a challenge to readily identify high quality items. Due to people’s limited attention, the presentation of content strongly affects how people al- locate effort to the available content. We evaluate this effect experimentally using Amazon Mechanical Turk and show that it is possible to manipulate attention to accomplish desired goals.
Making Crowdwork Work: Issues in Crowdsourcing for Organizations
Authors: Anya Obinna , Cefkin Melissa , Dill Steve , Moore Robert , Stucky Susan , Omokaro Osarieme
Published: October 2022

Existing approaches to crowdwork center around the unique ways in which work is sourced from the crowd, often emphasizing the kind of work characterized by hyperspe­cialized, microtask labor, such as that found in Amazon’s Mechanical Turk. However, real work in organizations is complex and rich, and as crowdsourcing is increasingly used alongside mainstream organizational work, social, technological, human-factors and work practice-related challenges arise. This paper presents the preliminary results of a research study designed to investigate models and methods for effective organizational uses of the crowd. The results indicate that despite the growing trend in organiza­tional crowdsourcing, its implications on the organisational work performance and human requirements are yet to be fully understood.
Crowdsourcing Quality Control for Item Ordering Tasks
Authors: Matsui Toshiko , Baba Yukino , Kamishima Toshihiro , Kashima Hisashi
Published: October 2022

One of the biggest challenges in crowdsourcing is quality control which is to expect high quality results from crowd workers who are not necessarily very capable nor motivated.In this paper, we consider item ordering questions, where workersare asked to arrange multiple items in the correct order. We propose a probabilistic generative model of crowd answers by extending a distance-based order model to incorporate worker ability, and give an efficient estimation algorithm.
Task Redundancy Strategy Based on Volunteers’ Credibility for Volunteer Thinking Projects
Authors: Ponciano Lesandro , Brasileiro Francisco , Gadelha Guilherme
Published: October 2022

In this paper, we propose a task redundancy strategy based on measures of accuracy of volunteers. Simulation results show that our strategy reduces the number of generated task replicas compared to the pessimistic and moderate strategies. It also generates similar or less task replicas compared to the optimistic strategy.
TrailView: Combining Gamification and Social Network Voting Mechanisms for Useful Data Collection
Authors: Weingert Michael , Larson Kate
Published: October 2022

There is a dearth of structured, organized photographical information of hiking trails and nature in general. Although people frequently photograph these locations and some efforts have been conducted to create virtual walk-throughs of select locations using specialized equipment~\cite{Stanger13:Take,Olanoff13:Google}, the information is largely scattered across different social networks and other websites. With TrailView we aim to utilize the efforts and photographs of everyday hikers to create a structured view of hiking trails and nature. TrailView boasts a gamified system to encourage user-interaction and focuses on social interactions, feedback, and competition to drive data collection. TrailView also contains a point/incentive scheme designed to motivate users. This incentive scheme relies on a social voting model to drive useful data collection, and global leaderboards to encourage competition.
Ontology Quality Assurance with the Crowd
Authors: Mortensen Jonathan , Musen Mark , Noy Natalya
Published: October 2022

The Semantic Web has the potential to change the Web as we know it. However, the community faces a significant challenge in managing, aggregating, and curating the massive amount of data and knowledge. Human computation is only beginning to serve an essential role in the curation of these Web-based data. Ontologies, which facilitate data integration and search, serve as a central component of the Semantic Web, but they are large, complex, and typically require extensive expert curation. Furthermore, ontology-engineering tasks require more knowledge than is required  in a typical crowdsourcing-task. We have developed ontology-engineering methods that leverage the crowd. In this work, we describe our general crowdsourcing workflow. We then highlight  our work on applying this workflow to ontology verification and quality assurance. In a pilot study, this method approaches expert ability, finding the same errors that experts identified with 86% accuracy in a faster and more scalable fashion. The work provides a general framework with which to develop crowdsourcing methods for the Semantic Web. In addition, it highlights opportunities for future research in human computation and crowdsourcing.
Crowdsourcing Translation by Leveraging Tournament Selection and Lattice-Based String Alignment
Authors: Bourdaillet Julien , Roy Shourya , Jung Gueyoung , Sun Yu-An
Published: October 2022

Crowdsourcing translation tasks typically face issues due to poor quality and spam translations. We propose a novel method for generating large multilingual text corpora leveraging Tournament Selection and Lattice-Based String Alignment without requiring expert involvement or Gold data. We use crowdsourcing for gathering a set of candidate translations of a given source sentence. A crowd sourced Tournament Selection allows to prune this set and keep an n-best list. Finally, a lattice-based string alignment allows to compose parts of these n-best translations to generate a consensus translation. As a work-in-progress, the evaluation is still ongoing, but we are confident in the potential for this method to generate translations of good quality.
Manipulating Social Roles in a Tagging Environment
Authors: Leyssen Mieke , Van Ossenbruggen Jacco , De Vries Arjen , Hardman Lynda
Published: October 2022

Social roles structure daily life because people adjust their behavior according to the role that they have in a specific situation. Online social roles are not necessarily the same as those in daily life and, because they are not so strictly assigned, the question arises whether they can be manipulated. We conducted a structured experiment to investigate whether the manipulation of online social roles can affect user behavior in a tagging task.
Inserting Micro-Breaks into Crowdsourcing Workflows
Authors: Rzeszotarski Jeffrey , Chi Ed , Paritosh Praveen , Dai Peng
Published: October 2022

Participants in human computation workflows may become fatigued or get bored over long, interminable working hours. This leads to a slump of motivation and morale, which in the long run causes reductions in both productivity and work quality. In this paper we propose an initial investigation into possible ways to alleviate worker fatigue and boredom by employing micro-breaks that provide timely relax to workers during long sequences of tasks. We experimentally test micro-breaks on Amazon’s Mechanical Turk, showing that micro-breaks can significantly improve worker retention rate as task batches reach hours in length, and appear to increase overall worker engagement and commitment to their work.