bluuebunny/crossref_metadata_2025 huggingface dataset:
TRANSIT Routing on Video Game Maps
Authors: Antsfeld Leonid , Harabor Daniel , Kilby Philip , Walsh Toby
Published: October 2022

TRANSIT is a fast and optimal technique for computing shortest path costs in road networks. It is attractive for its usually modest memory requirements and impressive running times. In this paper we give a first analysis of TRANSIT routing on a set of popular grid-based video-game benchmarks taken from the AI pathfinding literature. We show that in the presence of path symmetries, which are inherent to most grids but normally not road networks, TRANSIT is strongly and negatively impacted, both in terms of performance and memory requirements. We address this problem by developing a new general symmetry breaking technique which adds small random epsilon-values to edges in the search graph, reducing the size of the TRANSIT network by up to 4 times while preserving optimality. Using our enhancements TRANSIT achieves up to four orders of magnitude speed improvement vs. A* search and uses in many cases only a small (<=10MB) or modest (<= 50MB) amount of memory. We also compare TRANSIT with CPDs, a recent and very fast database-driven pathfinding approach. We find the algorithms have complementary strengths but also identify a class of problems for which TRANSIT is up to two orders of magnitude faster than CPDs using a comparable amount of memory.
POMCoP: Belief Space Planning for Sidekicks in Cooperative Games
Authors: Macindoe Owen , Pack Kaelbling Leslie , Lozano-Pérez Tomás
Published: October 2022

We present POMCoP, a system for online planning in collaborative domains that reasons about how its actions will affect its understanding of human intentions, and demonstrate its use in building sidekicks for cooperative games. POMCoP plans in belief space. It explicitly represents its uncertainty about the intentions of its human ally, and plans actions which reveal those intentions or hedge against its uncertainty. This allows POMCoP to reason about the usefulness of incorporating information gathering actions into its plans, such as asking questions, or simply waiting to let humans reveal their intentions. We demonstrate POMCoP by constructing a sidekick for a cooperative pursuit game, and evaluate its effectiveness relative to MDP-based techniques that plan in state space, rather than belief space.
RRT-Based Game Level Analysis, Visualization, and Visual Refinement
Authors: Bauer Aaron , Popović Zoran
Published: October 2022

Automating parts of game creation benefits both professional and amateur game designers and much previous work has already made progress on this front. In this paper we tackle automating level design. We describe a general graph-based representation for game levels and present a preliminary system that leverages this representation. Our system automatically explores existing levels of a 2D platform game using the rapidly-exploring random tree (RRT) algorithm and constructs a compact graph representation from this exploration. Our system can also modify a graph representation on-the-fly to reflect user-directed changes to the existing level structure. This work constitutes an initial step toward the larger goal of automating level design in a general way.
A Dataset for StarCraft AI and an Example of Armies Clustering
Authors: Synnaeve Gabriel , Bessière Pierre
Published: October 2022

This paper advocates the exploration of the full state of recorded real-time strategy (RTS) games, by human or robotic players, to discover how to reason about tactics and strategy. We present a dataset of StarCraft games encompassing the most of the games' state (not only player’s orders). We explain one of the possible usages of this dataset by clustering armies on their compositions. This reduction of armies compositions to mixtures of Gaussian allow for strate- gic reasoning at the level of the components. We evaluated this clustering method by predicting the outcomes of battles based on armies compositions' mixtures components.
Maxine’s Turing Test – A Player-Program as Co-Ethnographer of Socio-Aesthetic Interaction in Improvised Music
Authors: Banerji Ritwik
Published: October 2022

Beyond the goal of refining system design to the needs and tastes of users, user evaluation of interactive music systems offers a method of examining the nature of musical creativity as understood by its human practitioners. In the case of improvising music systems, user study and evaluation of a system’s ability to improvise may be useful in the ethnomusicological study of musical interaction in contemporary improvised music. A survey of preliminary findings based on the interactions of an improvising system, Maxine, with several improvisers is discussed, with results suggesting methodological reconfigurations of the purpose and goals of evaluating of interactive musical metacreations.
Adaptive Storytelling Through User Understanding
Authors: Garber-Barron Michael , Si Mei
Published: October 2022

Storytelling, when happening face to face, is a highly interactive process. A good storyteller attracts the audience by continuously observing their responses and adjusting the storytelling accordingly. The goal of this project is to simulate this process in digital storytellers. We created an automatic storytelling system that periodically estimates the user’s preferences and adapts the content in the subsequent storytelling by balancing novelty and topic consistency. We have performed an empirical evaluation on the effectiveness of our approach. The results indicate that gender and age play important roles in affecting one’s subjective experience. For younger subjects, stories with mixed amount of novelty and topic consistency are more preferred while for older subjects, larger amounts of variation are preferred. Additionally, in general, women enjoyed the stories more than men.
Evaluating Analogy-Based Story Generation: An Empirical Study
Authors: Zhu Jichen , Ontanon Santiago
Published: October 2022

Evaluation is one of the major open problems in computational narrative. In this paper, we present an empirical study of SAM, an analogy-based story generation (ASG) algorithm, that was created as part of our Riu interactive narrative system. Specifically, our study focuses on SAM's capability to retrieve and generate short non-interactive stories. Combining qualitative and quantitative methods from different disciplines, the methodology in this study can be extended to evaluating other computational narrative systems.
Designing an Intelligent Virtual Agent for Social Communication in Autism
Authors: Bernardini Sara , Porayska-Pomsta Kaska , Sampath Harini
Published: October 2022

This paper describes the Intelligent Engine (IE) of ECHOES, a serious game built for helping young children with Autism Spectrum Conditions acquire social communication skills. ECHOES IE's main component is an autonomous virtual agent that acts as a credible social partner for children with autism by engaging them in interactive learning activities. The other IE components are a user model, a drama manager and a social communication engine. We discuss how AI technology allows us to satisfy the requirements for the design of the agent and the learning activities that we identified through consultations with children and carers and a review of best practices for autism intervention. We present experimental results pertaining to the agent's effectiveness, which show encouraging improvements for a number of children.
An Authoring Tool to Derive Valid Interactive Scenarios
Authors: Dang Kim , Champagnat Ronan
Published: October 2022

In recent years, many tools have been proposed to design interactive scenarios. The aim of these tools is to provide users with a framework in order to write a story with many branches which will be unfolded by an artificial intelligence application. However, the consistency and quality of the generated narratives are not guaranteed (deadlocks, flaws, etc.). Previous work has defined the properties of a valid interactive scenario and proposed an approach as well as a tool based on a formal model, Linear Logic, to validate these properties. Nevertheless, the application of this tool requires many special skills and so is not really suitable for normal users. In this paper, we present an authoring tool which allows modeling interactive scenarios and analysing them at the structural level using the deduction rules in Linear Logic. Thanks to our tool, normal users are able to create and validate interactive scenarios in comparison with a rich set of predefined properties/criteria of quality.
Straight-Ahead Jazz with GenJam: A Quick Demonstration
Authors: Biles John
Published: October 2022

The author’s GenJam project has spanned nearly 20 years of development and performance. This demonstration provides a brief overview of GenJam’s current configuration illustrated with a visually annotated performance of a tune. After a brief explanation of GenJam’s real-time interactive capabilities, we’ll play more tunes, as time permits.
CorpusDB: Software for Analysis, Storage, and Manipulation of Sound Corpora
Authors: Stoll Thomas
Published: October 2022

CorpusDB is a system for representing sound files and associated analysis metadata in a structured format. The formats and conventions used in conjunction with the database allow for representation of sound files and their processed variants; multiple, overlapping, hierarchical relationships between sound files and segments thereof; and connections between sounds, their transformations, and analysis metadata. The software described in this paper is a parallel implementation consisting of SuperCollider classes, Python classes, and a common data representation of corpora that allows for seamless sharing of data between the two complementary environments. Code examples and listings of multi-step algorithms are included that demonstrate the kinds of operations possible within this system.
Connecting Algorithmic Research and Usage Contexts: A Perspective of Contextualized Evaluation for Explainable AI
Authors: Liao Q. Vera , Zhang Yunfeng , Luss Ronny , Doshi-Velez Finale , Dhurandhar Amit
Published: October 2022

Recent years have seen a surge of interest in the field of explainable AI (XAI), with a plethora of algorithms proposed in the literature. However, a lack of consensus on how to evaluate XAI hinders the advancement of the field. We highlight that XAI is not a monolithic set of technologies---researchers and practitioners have begun to leverage XAI algorithms to build XAI systems that serve different usage contexts, such as model debugging and decision-support. Algorithmic research of XAI, however, often does not account for these diverse downstream usage contexts, resulting in limited effectiveness or even unintended consequences for actual users, as well as difficulties for practitioners to make technical choices. We argue that one way to close the gap is to develop evaluation methods that account for different user requirements in these usage contexts. Towards this goal, we introduce a perspective of contextualized XAI evaluation by considering the relative importance of XAI evaluation criteria for prototypical usage contexts of XAI. To explore the context dependency of XAI evaluation criteria, we conduct two survey studies, one with XAI topical experts and another with crowd workers. Our results urge for responsible AI research with usage-informed evaluation practices, and provide a nuanced understanding of user requirements for XAI in different usage contexts.
A Human-Centric Perspective on Model Monitoring
Authors: Shergadwala Murtuza N , Lakkaraju Himabindu , Kenthapadi Krishnaram
Published: October 2022

Predictive models are increasingly used to make various consequential decisions in high-stakes domains such as healthcare, finance, and policy. It becomes critical to ensure that these models make accurate predictions, are robust to shifts in the data, do not rely on spurious features, and do not unduly discriminate against minority groups. To this end, several approaches spanning various areas such as explainability, fairness, and robustness have been proposed in recent literature. Such approaches need to be human-centered as they cater to the understanding of the models to their users. However, there is little to no research on understanding the needs and challenges in monitoring deployed machine learning (ML) models from a human-centric perspective. To address this gap, we conducted semi-structured interviews with 13 practitioners who are experienced with deploying ML models and engaging with customers spanning domains such as financial services, healthcare, hiring, online retail, computational advertising, and conversational assistants. We identified various human-centric challenges and requirements for model monitoring in real-world applications. Specifically, we found that relevant stakeholders would want model monitoring systems to provide clear, unambiguous, and easy-to-understand insights that are readily actionable. Furthermore, our study also revealed that stakeholders desire customization of model monitoring systems to cater to domain-specific use cases.
On the Verification Complexity of Group Decision-Making Tasks
Authors: Amir Ofra , Shahar Yuval , Gal Ya'akov , Ilani Litan
Published: October 2022

A popular use of crowdsourcing is to collect and aggregate individual worker responses to problems to reach a correct answer. This paper studies the relationship between the computation complexity class of problems, and the ability of a group to agree on a correct solution. We hypothesized that for NP-Complete (NPC) problems, groups would be able to reach a majority-based correct solution once it was suggested by a group member and presented to the other members, due to the "easy to verify" (i.e., verification in polynomial time) characteristic of this complexity class. In contrast, when posed with PSPACE-Complete (PSC)  "hard to verify" problems (i.e., verification in exponential time), groups will not necessarily be able to choose a correct solution even if such a solution has been presented. Consequently, increasing the size of the group is expected to facilitate the ability of the group to converge on a correct solution when solving NPC problems, but not when solving PSC problems. To test this hypothesis we conducted preliminary experiments in which we evaluated people's ability to solve an analytical problem and their ability to recognize a correct solution. In our experiments, participants were significantly more likely to recognize correct and incorrect solutions for NPC problems than for PSC problems, even for problems of similar difficulties (as measured by the percentage of participants who solved the problem). This is a first step towards formalizing a relationship between the computationally complexity of a problem and the crowd's ability to converge to a correct solution to the problem.
Volunteering Versus Work for Pay: Incentives and Tradeoffs in Crowdsourcing
Authors: Mao Andrew , Kamar Ece , Chen Yiling , Horvitz Eric , Schwamb Megan , Lintott Chris , Smith Arfon
Published: October 2022

Paid and volunteer crowd work have emerged as a means for harnessing human intelligence for performing diverse tasks. However, little is known about the relative performance of volunteer versus paid crowd work, and how financial incentives influence the quality and efficiency of output. We study the performance of volunteers as well as workers paid with different monetary schemes on a difficult real-world crowdsourcing task. We observe that performance by unpaid and paid workers can be compared in carefully designed tasks, that financial incentives can be used to trade quality for speed, and that the compensation system on Amazon Mechanical Turk creates particular indirect incentives for workers. Our methodology and results have implications for the ideal choice of financial incentives and motivates further study on how monetary incentives influence worker behavior in crowdsourcing.
Community Clustering: Leveraging an Academic Crowd to Form Coherent Conference Sessions
Authors: André Paul , Zhang Haoqi , Kim Juho , Chilton Lydia , Dow Steven , Miller Robert
Published: October 2022

Creating sessions of related papers for a large conference is a complex and time-consuming task. Traditionally, a few conference organizers group papers into sessions manually. Organizers often fail to capture the affinities between papers beyond created sessions, making incoherent sessions difficult to fix and alternative groupings hard to discover. This paper proposes committeesourcing and authorsourcing approaches to session creation (a specific instance of clustering and constraint satisfaction) that tap into the expertise and interest of committee members and authors for identifying paper affinities. During the planning of ACM CHI'13, a large conference on human-computer interaction, we recruited committee members to group papers using two online distributed clustering methods. To refine these paper affinities — and to evaluate the committeesourcing methods against existing manual and automated approaches — we recruited authors to identify papers that fit well in a session with their own. Results show that authors found papers grouped by the distributed clustering methods to be as relevant as, or more relevant than, papers suggested through the existing in-person meeting. Results also demonstrate that communitysourced results capture affinities beyond sessions and provide flexibility during scheduling.
Inferring Users’ Preferences from Crowdsourced Pairwise Comparisons: A Matrix Completion Approach
Authors: Yi Jinfeng , Jin Rong , Jain Shaili , Jain Anil
Published: October 2022

Inferring user preferences over a set of items is an important problem that has found numerous applications. This work focuses on the scenario where the explicit feature representation of items is unavailable, a setup that is similar to collaborative filtering. In order to learn a user's preferences from his/her response to only a small number of pairwise comparisons, we propose to leverage the pairwise comparisons made by many crowd users, a problem we refer to as crowdranking. The proposed crowdranking framework is based on the theory of matrix completion, and we present efficient algorithms for solving the related optimization problem. Our theoretical analysis shows that, on average, only O(r log m) pairwise queries are needed to accurately recover the ranking list of m items for the target user, where r is the rank of the unknown rating matrix, r << m. Our empirical study with two real-world benchmark datasets for collaborative filtering and one crowdranking dataset we collected via Amazon Mechanical Turk shows the promising performance of the proposed algorithm compared to the state-of-the-art approaches.
Assessing the Viability of Online Interruption Studies
Authors: Gould Sandy , Cox Anna , Brumby Duncan , Wiseman Sarah
Published: October 2022

Researchers have been collecting data online since the early days of the Internet and as technology improves, increasing numbers of traditional experiments are being run online. However, there are still questions about the kinds of experiments that work online, particularly over experiments with time-sensitive performance measures. We are interested in one time-sensitive measure specifically, the time taken to resume a task following an interruption. We ran participants through an archetypal interruption study online and in the lab. Statistical comparisons showed no significant differences in the time it took to resume following an interruption. However, there were issues with data quality that stemmed from participant confusion about the task. Our findings have implications for experiments that assess time-sensitive performance measures in tasks that require continuous attention.
Crowd, the Teaching Assistant: Educational Assessment Crowdsourcing
Authors: Manohar Pallavi , Roy Shourya
Published: October 2022

We propose a novel educational assessment method to enhance teachers' productivity using crowdsourcing and clustering technique.
Two Methods for Measuring Question Difficulty and Discrimination in Incomplete Crowdsourced Data
Authors: Luger Sarah , Bowles Jeff
Published: October 2022

Assistance in creating high-quality exams would be welcomed by educators who do not have direct access to the proprietary data and methods used by educational testing companies. The current approach for measuring question difficulty relies on models of how good pupils will perform and contrasts that with their lower-performing peers. Inverting this process and allowing educators to test their questions before students answer them will speed up question development and utility. We cover two methods for automatically judging the difficulty and discriminating power of MCQs and how best to build sufficient exams from good questions.
CrowdUtility: A Recommendation System for Crowdsourcing Platforms
Authors: Chander Deepthi , Bhattacharya Sakyajit , Celis Elisa , Dasgupta Koustuv , Karanam Saraschandra , Rajan Vaibhav , Gupta Avantika
Published: October 2022

Crowd workers exhibit varying work patterns, expertise, and quality leading to wide variability in the performance of crowdsourcing platforms. The onus of choosing a suitable platform to post tasks is mostly with the requester, often leading to poor guarantees and unmet requirements due to the dynamism in performance of crowd platforms. Towards this end, we demonstrate CrowdUtility, a statistical modelling based tool for evaluating multiple crowdsourcing platforms and recommending a platform that best suits the requirements of the requester. CrowdUtility uses an online Multi-Armed Bandit framework, to schedule tasks while optimizing platform performance. We demonstrate an end-to end system starting from requirements specification, to platform recommendation, to real-time monitoring.
Crowdsourcing in Language Classes Can Help Natural Language Processing
Authors: Hladká Barbora , Hana Jirka , Lukšová Ivana
Published: October 2022

One way of teaching grammar, namely morphology and syntax, is to visualize sentences as diagrams capturing relationships between words. Similarly, such relationships are captured in a more complex way in treebanks serving as key building stones in modern natural language processing. However, building them is very time consuming, thus we have been seeking for an alternative cheaper and faster way, like crowdsourcing. The purpose of our work is to explore possibility to get sentence diagrams produced by students and teachers. In our pilot study, the object language is Czech, where sentence diagrams are part of elementary school curriculum.
Detecting Non-Adversarial Collusion in Crowdsourcing
Authors: KhudaBukhsh Ashiqur , Carbonell Jaime , Jansen Peter
Published: October 2022

A group of agents are said to collude if they share information or make joint decisions in a manner contrary to explicit or implicit social rules that results in an unfair advantage over non-colluding agents or other interested parties. For instance, collusion manifests as sharing answers in exams, as colluding bidders in auctions, or as colluding participants (e.g., Turkers) in crowd sourcing. This paper studies the latter, where the goal of the colluding participants is to "earn" money without doing the actual work, for instance by copying product ratings of another colluding participant, adding limited noise as attempted obfuscation. Such collusion not only yields fewer independent ratings, but may also introduce strong biases in aggregate results if undetected. Our proposed unsupervised collusion detection algorithm identifies colluding groups in crowd sourcing with fairly high accuracy both in synthetic and real data, and results in significant bias reduction, such as minimizing shifts from the true mean in rating tasks and recovering the true variance among raters.
To Re(label), or Not To Re(label)
Authors: Lin Christopher , Mausam . , Weld Daniel
Published: October 2022

One of the most popular uses of crowdsourcing is to provide training data for supervised machine learning algorithms. Since human annotators often make errors, requesters commonly ask multiple workers to label each example.  But is this strategy always the most cost effective use of crowdsourced workers? We argue "No" --- often classifiers can achieve higher accuracies when trained with noisy "unilabeled" data. However, in some cases relabeling is extremely important.  We discuss three factors that may make relabeling an effective strategy: classifier expressiveness, worker accuracy, and budget.
Crowd-Workers: Aggregating Information Across Turkers to Help Them Find Higher Paying Work
Authors: Callison-Burch Chris
Published: October 2022

The Mechanical Turk crowdsourcing platform currently fails to provide the most basic piece of information to enable workers to make informed decisions about which tasks to undertake: what is the expected hourly pay?  Mechanical Turk advertises a reward amount per assignment, but does not give any indication of how long each assignment will take.  We have developed a browser plugin that tracks the length of time it takes to complete a task, and a web service that aggregates the information across many workers. Our web service, crowd-workers.com, allows workers to discovery higher paying work by sorting tasks by estimated hourly rate.
Speech Synthesis Data Collection for Visually Impaired Person
Authors: Ashikawa Masayuki , Kawamura Takahiro , Ohsuga Akihiko
Published: October 2022

Crowdsourcing platforms provide attractive solutions for collecting speech synthesis data for visually impaired person. However, quality control problems remain because of low-quality volunteer workers. In this paper, we propose the design of a crowdsourcing system that allows us to devise quality control methods. We introduce four worker selection methods; preprocessing filtering, real-time filtering, post-processing filtering, and guess-processing filtering. These methods include a novel approach that utilizes a collaborative filtering technique in addition to a basic approach involving initial training or use of gold-standard data. These quality control methods improved the quality of collected speech synthesis data. Moreover, we have already collected 140,000 Japanese words from 500 million web data for speech synthesis data.
Guardian: A Crowd-Powered Spoken Dialog System for Web APIs
Authors: Huang Ting-Hao , Lasecki Walter , Bigham Jeffrey
Published: October 2022

Natural language dialog is an important and intuitive way for people to access information and services. However, current dialog systems are limited in scope, brittle to the richness of natural language, and expensive to produce. This paper introduces Guardian, a crowd-powered framework that wraps existing Web APIs into immediately usable spoken dialog systems. Guardian takes as input the Web API and desired task, and the crowd determines the parameters necessary to complete it, how to ask for them, and interprets the responses from the API. The system is structured so that, over time, it can learn to take over for the crowd. This hybrid systems approach will help make dialog systems both more general and more robust going forward.
Learning Privacy Expectations by Crowdsourcing Contextual Informational Norms
Authors: Shvartzshnaider Yan , Tong Schrasing , Wies Thomas , Kift Paula , Nissenbaum Helen , Subramanian Lakshminarayanan , Mittal Prateek
Published: October 2022

Designing programmable privacy logic frameworks that correspond to social, ethical, and legal norms has been a fundamentally hard problem. Contextual integrity (CI) (Nissenbaum, 2010) offers a model for conceptualizing privacy that is able to bridge technical design with ethical, legal, and policy approaches. While CI is capable of capturing the various components of contextual privacy in theory, it is challenging to discover and formally express these norms in operational terms. In the following, we propose a crowdsourcing method for the automated discovery of contextual norms. To evaluate the effectiveness and scalability of our approach, we conducted an extensive survey on Amazon's Mechanical Turk (AMT) with more than 450 participants and 1400 questions. The paper has three main takeaways: First, we demonstrate the ability to generate survey questions corresponding to privacy norms within any context. Second, we show that crowdsourcing enables the discovery of norms from these questions with strong majoritarian consensus among users. Finally, we demonstrate how the norms thus discovered can be encoded into a formal logic to automatically verify their consistency.
Understanding Crowdsourcing Workflow: Modeling and Optimizing Iterative and Parallel Processes
Authors: Goto Shinsuke , Ishida Toru , Lin Donghui
Published: October 2022

The advantages and disadvantages of different crowdsourcing workflow structures have been analyzed. Existing studies on crowdsourcing workflow mainly focused on the quality control of the tasks using iterative and parallel processes. On the other hand, the characteristics of workflow considering the various task and crowdsourcing environments are not yet fully analyzed. Therefore, we face two difficulties in making use of workflow: the workflow optimization and the prior quality estimation. This research proposes the crowdsourcing workflow model for the set of improvement tasks, considering the ability distribution of crowdsourcing workers, improvement difficulty of the task, and the preference of the requester. In addition, we show the optimal workflow can be found by the search algorithm on the proposed model. The result of optimization can be used for both constructing the best workflow and estimating the quality of task performance. Experimental results under various conditions indicate that the degree of parallelism of the optimal workflow increases with the variance of worker ability. Also, iterative processes should be used when the average ability of the workers trends away from the middle level. These results include the existing research and therefore the model presented is useful in understanding crowdsourcing workflows.
Crowdsourcing Paper Screening in Systematic Literature Reviews
Authors: Krivosheev Evgeny , Casati Fabio , Caforio Valentina , Benatallah Boualem
Published: October 2022

Literature reviews allow scientists to stand on the shoulders of giants, showing promising directions, summarizing progress, and pointing out existing challenges in research. At the same time conducting a systematic literature review is a laborious and consequently expensive process. In the last decade, there have been several studies on crowdsourcing in literature reviews. This paper explores the feasibility of crowdsourcing for facilitating the literature review process in terms of results, time and effort, and identifies which crowdsourcing strategies provide the best results based on the budget available. In particular we focus on the screening phase of the literature review process and we contribute and assess strategies for running crowdsourcing tasks that are efficient in terms of budget and classification error. Finally, we present our findings based on experiments run on Crowdflower.
CrowdMask: Using Crowds to Preserve Privacy in Crowd-Powered Systems via Progressive Filtering
Authors: Kaur Harmanpreet , Gordon Mitchell , Yang Yiwei , Bigham Jeffrey , Teevan Jaime , Kamar Ece , Lasecki Walter
Published: October 2022

Crowd-powered systems leverage human intelligence to go beyond the capabilities of automated systems, but also introduce privacy and security concerns because unknown people must view the data that the system processes. While automated approaches cannot robustly filter private information from these datasets, people have the ability to do so if the risk from them viewing the data can be mitigated. We present a crowd-powered approach to masking private content in data by segmenting and distributing smaller segments to crowd workers so that individual workers can identify potentially private content without being able to fully view it themselves. We introduce a novel pyramid workflow for segmentation that uses segments at multiple levels of granularity to overcome problems with fixed-sized approaches. We implement our approach in CrowdMask, a system that allows images with potentially sensitive content to be masked by appearing in progressively larger, more identifiable segments, and masking portions of the image as soon as a risk is identified. Our experiments with 4134 Mechanical Turk workers show that CrowdMask can effectively mask private content from images without revealing sensitive content to constituent workers, while still enabling future systems to use the filtered result.
Crowdsourcing Real-Time Viral Disease and Pest Information: A Case of Nation-Wide Cassava Disease Surveillance in a Developing Country
Authors: Mutembesa Daniel , Omongo Christopher , Mwebaze Ernest
Published: October 2022

In most developing countries, a huge proportion of the national food basket is supported by small subsistence agricultural systems. A major challenge to these systems is disease and pest attacks which have a devastating effect on the small-holder farmers that depend on these systems for their livelihoods. A key component of any proposed solution is a good disease surveillance network. However, current surveillance efforts are unable to provide sufficient data for monitoring such phenomena over a vast geographic area efficiently and effectively due to limited resources, both human and financial. Crowdsourcing with farmer crowds that have access to mobile phones offers a viable option to provide all year round real-time surveillance data on viral disease and pest incidence and severity. This work presents a mobile ad hoc surveillance system for monitoring viral diseases and pests in cassava. We present results from a pilot in Uganda where this system was deployed for 76 weeks. We discuss the participation behaviours of the crowds with mobile smartphones as well as the effects of several incentives applied.
Verifying Conceptual Domain Models with Human Computation: A Case Study in Software Engineering
Authors: Sabou Marta , Winkler Dietmar , Penzerstadler Peter , Biffl Stefan
Published: October 2022

Conceptual domain models, such as taxonomies, knowledge graphs or Extended Entity Relationship (EER) diagrams are core to all information systems. The task of verifying the correctness of these models is of high interest to the knowledge and software engineering communities and attracted the first solution approaches using human computation. Yet, since these solutions are published within the boundaries of their communities, there is a lack of concerted work on this topic. As a first step to alleviate this status quo, we formalize the problem of verifying conceptual models and propose a generic approach (VeriCoM) to solve it with human computation techniques. We show how VeriCoM was applied in a software engineering use case focusing on verifying the correctness of an EER diagram against a system specification document. An evaluation of VeriCoM in a series of four workshops within one controlled experiment performed with a crowd of semi-experts lead to the identification of a set of defects with precision of 73% and a recall from a Gold Standard defect set of 63%.
"A Game Without Competition Is Hardly a Game": The Impact of Competitions on Player Activity in a Human Computation Game"
Authors: Reeves Neal , West Peter , Simperl Elena
Published: October 2022

Virtual citizen science (VCS) projects enable new forms of scientific research using crowdsourcing and human computation to gather and analyse large-scale datasets. To attract and sustain the number of participants and levels of participation necessary to achieve research aims, some VCS projects have introduced game elements such as competitions to tasks. However, we still know very little about how some game elements, particularly competitions, influence participation rates. To investigate the impact of game elements on player engagement, we conducted a two-part mixed-methods study of EyeWire, a VCS game. First, we interviewed EyeWire designers to understand their rationale for introducing competitions. Guided by their answers, we analysed two datasets of EyeWire user task contributions and chat logs to assess the effectiveness of competitions in achieving designers' goals. Our findings contribute to the growing understanding of how competitions influence participant activity in human computation initiatives and socio-technical systems such as VCS.
The Effects of Meaningful and Meaningless Explanations on Trust and Perceived System Accuracy in Intelligent Systems
Authors: Nourani Mahsan , Kabir Samia , Mohseni Sina , Ragan Eric D.
Published: October 2022

Machine learning and artificial intelligence algorithms can assist human decision making and analysis tasks. While such technology shows promise, willingness to use and rely on intelligent systems may depend on whether people can trust and understand them. To address this issue, researchers have explored the use of explainable interfaces that attempt to help explain why or how a system produced the output for a given input. However, the effects of meaningful and meaningless explanations (determined by their alignment with human logic) are not properly understood, especially with users who are non-experts in data science. Additionally, we wanted to explore how explanation inclusion and level of meaningfulness would affect the user’s perception of accuracy. We designed a controlled experiment using an image classification scenario with local explanations to evaluate and better understand these issues. Our results show that whether explanations are human-meaningful can significantly affect perception of a system’s accuracy independent of the actual accuracy observed from system usage. Participants significantly underestimated the system’s accuracy when it provided weak, less human-meaningful explanations. Therefore, for intelligent systems with explainable interfaces, this research demonstrates that users are less likely to accurately judge the accuracy of algorithms that do not operate based on human-understandable rationale.
Learning to Predict Population-Level Label Distributions
Authors: Liu Tong , Venkatachalam Akash , Bongale Pratik Sanjay , Homan Christopher M.
Published: October 2022

As machine learning (ML) plays an ever increasing role in commerce, government, and daily life, reports of bias in ML systems against groups traditionally underrepresented in computing technologies have also increased. The problem appears to be extensive, yet it remains challenging even to fully assess the scope, let alone fix it. A fundamental reason is that ML systems are typically trained to predict one correct answer or set of answers; disagreements between the annotators who provide the training labels are resolved by either discarding minority opinions (which may correspond to demographic minorities or not) or presenting all opinions flatly, with no attempt to quantify how different answers might be distributed in society. Label distribution learning associates for each data item a probability distribution over the labels for that item. While such distributions may be representative of minority beliefs or not, they at least preserve diversities of opinion that conventional learning hides or ignores and represent a fundamental first step toward ML systems that can model diversity. We introduce a strategy for learning label distributions with only five-to-ten labels per item—a range that is typical of supervised learning datasets—by aggregating human-annotated labels over multiple, similarly rated data items. Our results suggest that specific label aggregation methods can help provide reliable, representative predictions at the population level.
Fast, Accurate, and Healthier: Interactive Blurring Helps Moderators Reduce Exposure to Harmful Content
Authors: Das Anubrata , Dang Brandon , Lease Matthew
Published: October 2022

While most user content posted on social media is benign, other content, such as violent or adult imagery, must be detected and blocked. Unfortunately, such detection is difficult to automate, due to high accuracy requirements, costs of errors, and nuanced rules for acceptable content. Consequently, social media platforms today rely on a vast workforce of human moderators. However, mounting evidence suggests that exposure to disturbing content can cause lasting psychological and emotional damage to some moderators. To mitigate such harm, we investigate a set of blur-based moderation interfaces for reducing exposure to disturbing content whilst preserving moderator ability to quickly and accurately flag it. We report experiments with Mechanical Turk workers to measure moderator accuracy, speed, and emotional well-being across six alternative designs. Our key findings show interactive blurring designs can reduce emotional impact without sacrificing moderation accuracy and speed.
The Role of Domain Expertise in User Trust and the Impact of First Impressions with Intelligent Systems
Authors: Nourani Mahsan , King Joanie , Ragan Eric
Published: October 2022

Domain-specific intelligent systems are meant to help system users in their decision-making process. Many systems aim to simultaneously support different users with varying levels of domain expertise, but prior domain knowledge can affect user trust and confidence in detecting system errors. While it is also known that user trust can be influenced by first impressions with intelligent systems, our research explores the relationship between ordering bias and domain expertise when encountering errors in intelligent systems. In this paper, we present a controlled user study to explore the role of domain knowledge in establishing trust and susceptibility to the influence of first impressions on user trust. Participants reviewed an explainable image classifier with a constant accuracy and two different orders of observing system errors (observing errors in the beginning of usage vs. in the end). Our findings indicate that encountering errors early-on can cause negative first impressions for domain experts, negatively impacting their trust over the course of interactions. However, encountering correct outputs early helps more knowledgable users to dynamically adjust their trust based on their observations of system performance. In contrast, novice users suffer from over-reliance due to their lack of proper knowledge to detect errors.
Batch Prioritization of Data Labeling Tasks for Training Classifiers
Authors: Kimura Masanari , Wakabayashi Kei , Morishima Atsuyuki
Published: October 2022

In a data labeling process for building machine learning, the choice of labeling data instances is known to have a significant impact on the performance of classifiers. So far, the study of active learning has addressed the issue of how to choose the subset by prioritizing the data instances based on the state of the current classifier. However, the active learning approach has two drawbacks that (i) require a training loop to update the priorities of labeling tasks and (ii) require us to choose a specific active learner while we do not know the optimal classification model. In this paper, we propose a new framework of priority-aware labeling system that allows a parallel task assignment to crowd workers without assuming a particular classifier, which is based on novel methods called “batch prioritization” and “label expansion”. We conducted experiments with multiple datasets to examine the effectiveness of the approach and found that the proposed method improves the performance of the final classifiers more quickly than the active learning approach despite that the labeling tasks can be processed in a fully parallel manner.
Forecast Aggregation via Peer Prediction
Authors: Wang Juntao , Liu Yang , Chen Yiling
Published: October 2022

Crowdsourcing enables the solicitation of forecasts on a variety of prediction tasks from distributed groups of people. How to aggregate the solicited forecasts, which may vary in quality, into an accurate final prediction remains a challenging yet critical question. Studies have found that weighing expert forecasts more in aggregation can improve the accuracy of the aggregated prediction. However, this approach usually requires access to the historical performance data of the forecasters, which are often not available. In this paper, we study the problem of aggregating forecasts without having historical performance data. We propose using peer prediction methods, a family of mechanisms initially designed to truthfully elicit private information in the absence of ground truth verification, to assess the expertise of forecasters, and then using this assessment to improve forecast aggregation. We evaluate our peer-prediction-aided aggregators on a diverse collection of 14 human forecast datasets. Compared with a variety of existing aggregators, our aggregators achieve a significant and consistent improvement on aggregation accuracy measured by the Brier score and the log score. Our results reveal the effectiveness of identifying experts to improve aggregation even without historical data.
Planning with Perspectives -- Decomposing Epistemic Planning using Functional STRIPS
Authors: Hu Guang , Miller Tim , Lipovetzky Nir
Published: October 2022

In this paper, we present a novel approach to epistemic planning called planning with perspectives (PWP) that is both more expressive and computationally more efficient than existing state-of-the-art epistemic planning tools. Epistemic planning — planning with knowledge and belief — is essential in many multi-agent and human-agent interaction domains. Most state-of-the-art epistemic planners solve epistemic planning problems by either compiling to propositional classical planning (for example, generating all possible knowledge atoms or compiling epistemic formulae to normal forms); or explicitly encoding Kripke-based semantics. However, these methods become computationally infeasible as problem sizes grow. In this paper, we decompose epistemic planning by delegating reasoning about epistemic formulae to an external solver. We do this by modelling the problem using Functional STRIPS, which is more expressive than standard STRIPS and supports the use of external, black-box functions within action models. Building on recent work that demonstrates the relationship between what an agent ‘sees’ and what it knows, we define the perspective of each agent using an external function, and build a solver for epistemic logic around this. Modellers can customise the perspective function of agents, allowing new epistemic logics to be defined without changing the planner. We ran evaluations on well-known epistemic planning benchmarks to compare an existing state-of-the-art planner, and on new scenarios that demonstrate the expressiveness of the PWP approach. The results show that our PWP planner scales significantly better than the state-of-the-art planner that we compared against, and can express problems more succinctly.