bluuebunny/crossref_metadata_2025 huggingface dataset:
EM-Based Inference of True Labels Using Confidence Judgments
Authors: Oyama Satoshi , Baba Yukino , Sakurai Yuko , Kashima Hisashi
Published: October 2022

We have developed a method for accurately inferring true labels from labels provided by crowdsourcing workers, with the aid of self-reported confidence judgments in their labels. Although confidence judgments can be useful information for estimating the quality of the provided labels, some workers are overconfident about the quality of their labels while others are underconfident. To address this problem, we extended the Dawid-Skene model and created a probabilistic model that considers the differences among workers in their accuracy of confidence judgments. Results of experiments using actual crowdsourced data showed that incorporating workers' confidence judgments can improve the accuracy of inferred labels.
Towards a Language for Non-Expert Specification of POMDPs for Crowdsourcing
Authors: Lin Christopher , Mausam - , Weld Daniel
Published: October 2022

Crowdsourcing requesters are trapped between a rock and a hard place. Typically they specify their crowdsourcing workflows procedurally, but current languages commit them to overly strict and static policies that waste human effort. While optimizing workflows with more sophisticated tools like POMDPs can significantly reduce labor costs, such advanced AI techniques are hard to use and understand. We report on our progress in developing Clowder, a system that provides the user with an adaptive programming language that looks and feels like Lisp, yet abstracts over POMDPs so that non-experts can write POMDPs without knowing anything about them. Such a system frees requesters from needing to resort to suboptimal techniques that use approximate heuristics or hire a planning expert to formally define and solve their problems.
GameLab: A Tool Suit to Support Designers of Systems with Homo Ludens in the Loop
Authors: Krause Markus
Published: October 2022

Digital games are an interesting method to motivate contributors to part take in a human computation process. Howeverthis approach poses its own challenges. Especially quality management or immediate and diverse feedback top layers are recurrent challenges. This paper introduces a tool suit to support designers with these challenges.
Herding the Crowd: Automated Planning for Crowdsourced Planning
Authors: Talamadupula Kartik , Kambhampati Subbarao , Hu Yuheng , Nguyen Tuan , Zhuo Hankz
Published: October 2022

An important application of human computation is crowdsourced planning and scheduling. In this paper, we present an architecture for an automated system that can significantly improve the effectiveness of the crowd in collaborating and coming up with effective plans by herding it. We define two main problems that have to be solved when designing such automated crowd-herding systems: interpretation, and steering; and discuss how automated planning techniques can be used to solve these problems.
A Framework for Adaptive Crowd Query Processing
Authors: Trushkowsky Beth , Kraska Tim , Franklin Michael
Published: October 2022

Search engines can yield poor results for information retrieval tasks when they cannot interpret query predicates. Such predicates are better left for humans to evaluate. We propose an adaptive processing framework for deciding (a) which parts of a query should be processed by machines and (b) the order the crowd should process the remaining parts, optimizing for result quality and processing cost. We describe an algorithm and experimental results for the first framework component.
A Ground Truth Inference Model for Ordinal Crowd-Sourced Labels Using Hard Assignment Expectation Maximization
Authors: Faridani Siamak , Buscher Georg , Xu Ya
Published: October 2022

We propose an iterative approach for inferring a ground truth value of an item from judgments collected form on-line workers. The method is specifically designed for cases in which the collected labels are ordinal. Our algorithm works by iteratively solving a hard-assignment EM model and later calculating one final expected value after the convergence of the EM procedure. This algorithm does not require any parameter tuning and can serve as turnkey algorithm for aggregating categorical and ordinal judgments.
LabelBoost: An Ensemble Model for Ground Truth Inference Using Boosted Trees
Authors: Faridani Siamak , Buscher Georg
Published: October 2022

We introduce LabelBoost, an ensemble model that utilizes various label aggregation algorithms to build a higher precision algorithm. We compare this algorithm with majority vote, GLAD and an Expectation Maximization model on a publicly available dataset. The results suggest that by building an ensemble model, one can achieve higher precision value for aggregating crowd-sourced labels for an item. These higher values are shown to be statistically significant.
Community Poll: Externalizing Public Sentiments in Social Media in a Local Community Context
Authors: Shih Patrick , Han Kyungsik , Carroll John
Published: October 2022

Social media platforms such as Twitter and Facebook are commonly used to disseminate up-to-date news information, but they also contain a lot of noise and irrelevant content. The contents of social media platforms are typically filtered by followship or friendship oriented relationships, and is almost always driven by trending news topics at the national scale, making it difficult for users to gather useful information that is most pertinent to a local community context. Research has utilized content analysis techniques to gain insights on the sentiment expressed about political topics on social media sites. However, there has been little attempt to understand how users would perceive this information if opinions and sentiments about news topics were externalized and made aware to them. We designed Community Poll, a smartphone application that aggregates local news feeds with relevant tweets about the local news topics. A Public Attitude Meter is calculated based on the sentiment score of the tweets for each of the local news topic presented in the system. We conducted a 2-week deployment with 16 users about their perception of the system. The users reported that Community Poll helps them digest locally relevant news topics, and quickly gather public opinions associated with the topics. They reported that being aware of public sentiment encouraged them to more actively participate in discussions on social media. Curiosity about a score-based representation is an important element that drove them to consume local news topics that they wouldn’t otherwise be exposed to. Interestingly, although being aware of public sentiment served to reaffirm people’s positions on the local topics, users expressed concerns about how sentiment awareness might bias other people’s judgments regarding news topics.
Output Agreement Mechanisms and Common Knowledge
Authors: Waggoner Bo , Chen Yiling
Published: October 2022

The recent advent of human computation -- employing non-experts to solve problems -- has inspired theoretical work in mechanism design for eliciting information when responses cannot be verified. We study a popular practical method, output agreement, from a theoretical perspective. In output agreement, two agents are given the same inputs and asked to produce some output; they are scored based on how closely their responses agree. Although simple, output agreement raises new conceptual questions. Primary is the fundamental importance of common knowledge: We show that, rather than being truthful, output agreement mechanisms elicit common knowledge from participants. We show that common knowledge is essentially the best that can be hoped for in any mechanism without verification unless there are restrictions on the information structure. This involves generalizing truthfulness to include responding to a query rather than simply reporting a private signal, along with a notion of common-knowledge equilibria. A final important issue raised by output agreement is focal equilibria and player computation of equilibria. We show that, for eliciting the mean of a random variable, a natural player inference process converges to the common-knowledge equilibrium; but this convergence may not occur for other types of queries.
Cost-Effective HITs for Relative Similarity Comparisons
Authors: Wilber Michael , Kwak Iljung , Belongie Serge
Published: October 2022

Similarity comparisons of the form "Is object a more similar to b than to c?" form a useful foundation in several computer vision and machine learning applications. Unfortunately, an embedding of n points is only uniquely specified by n3 triplets, making collecting every triplet an expensive task. In noticing this difficulty, other researchers investigated more intelligent triplet sampling techniques, but they do not study their effectiveness or their potential drawbacks. Although it is important to reduce the number of collected triplets to generate a good embedding, it is also important to understand how best to display a triplet collection task to the user to better respect the worker's human constraints. In this work, we explore an alternative method for collecting triplets and analyze its financial cost, collection speed, and worker happiness as a function of the final embedding quality. We propose best practices for creating cost effective human intelligence tasks for collecting triplets. We show that rather than changing the sampling algorithm, simple changes to the crowdsourcing UI can drastically decrease the cost of collecting similarity comparisons. Finally, we provide a food similarity dataset as well as the labels collected from crowd workers.
Scaling-Up the Crowd: Micro-Task Pricing Schemes for Worker Retention and Latency Improvement
Authors: Difallah Djellel , Catasta Michele , Demartini Gianluca , Cudré-Mauroux Philippe
Published: October 2022

Retaining workers on micro-task crowdsourcing platforms is essential in order to guarantee the timely completion of batches of Human Intelligence Tasks (HITs). Worker retention is also a necessary condition for the introduction of SLAs on crowdsourcing platforms. In this paper, we introduce novel pricing schemes aimed at improving the retention rate of workers working on long batches of similar tasks. We show how increasing or decreasing the monetary reward over time influences the number of tasks a worker is willing to complete in a batch, as well as how it influences the overall latency. We compare our new pricing schemes against traditional pricing methods (e.g., constant reward for all the HITs in a batch) and empirically show how certain schemes effectively function as an incentive for workers to keep working longer on a given batch of HITs. Our experimental results show that the best pricing scheme in terms of worker retention is based on punctual bonuses paid whenever the workers reach predefined milestones.
A Human Computation Framework for Boosting Combinatorial Solvers
Authors: Le Bras Ronan , Xue Yexiang , Bernstein Richard , Gomes Carla , Selman Bart
Published: October 2022

We propose a general framework for boosting combinatorial solvers through human computation. Our framework combines insights from human workers with the power of combinatorial optimization. The combinatorial solver is also used to guide requests for the workers, and thereby obtain the most useful human feedback quickly. Our approach also incorporates a problem decomposition approach with a general strategy for discarding incorrect human input. We apply this framework in the domain of materials discovery, and demonstrate a speedup of over an order of magnitude.
STEP: A Scalable Testing and Evaluation Platform
Authors: Christoforaki Maria , Ipeirotis Panagiotis
Published: October 2022

The emergence of online crowdsourcing sites, online work platforms, and evenMassive Open Online Courses (MOOCs), has created an increasing need for reliably evaluating the skills of the participating users in a scalable way.Many platforms already allow users to take online tests and verify their skills, but the existing approaches face many problems. First of all, cheating is very common in online testing without supervision, as the test questions often "leak" and become easily available online together with the answers.Second, technical skills, such as programming, require the tests to be frequently updated in order to reflect the current state-of-the-art. Third,there is very limited evaluation of the tests themselves, and how effectively they measure the skill that the users are tested for. In this paper, we present a Scalable Testing and Evaluation Platform (STEP),that allows continuous generation and evaluation of test questions. STEP leverages already available content, on Question Answering sites such as StackOverflow and re-purposes these questions to generate tests. The system utilizes a crowdsourcing component for the editing of the questions, while it uses automated techniques for identifying promising QA threads that can be successfully re-purposed for testing. This continuous question generation decreases the impact of cheating and also creates questions that are closer to the real problems that the skill holder is expected to solve in real life.STEP also leverages the use of Item Response Theory to evaluate the quality of the questions. We also use external signals about the quality of the workers.These identify the questions that have the strongest predictive ability in distinguishing workers that have the potential to succeed in the online job marketplaces. Existing approaches contrast in using only internal consistency metrics to evaluate the questions. Finally, our system employs an automatic "leakage detector" that queries the Internet to identify leaked versions of our questions. We then mark these questions as "practice only," effectively removing them from the pool of questions used for evaluation. Our experimental evaluation shows that our system generates questions of comparable or higher quality compared to existing tests, with a cost of approximately 3-5 dollars per question, which is lower than the cost of licensing questions from existing test banks.
Attendee-Sourcing: Exploring The Design Space of Community-Informed Conference Scheduling
Authors: Bhardwaj Anant , Kim Juho , Dow Steven , Karger David , Madden Sam , Miller Rob , Zhang Haoqi
Published: October 2022

Constructing a good conference schedule for a large multi-track conference needs to take into account the preferences and constraints of organizers, authors, and attendees. Creating a schedule which has fewer conflicts for authors and attendees, and thematically coherent sessions is a challenging task. Cobi introduced an alternative approach to conference scheduling by engaging the community to play an active role in the planning process. The current Cobi pipeline consists of committee-sourcing and author-sourcing to plan a conference schedule. We further explore the design space of community-sourcing by introducing attendee-sourcing -- a process that collects input from conference attendees and encodes them as preferences and constraints for creating sessions and schedule. For CHI 2014, a large multi-track conference in human-computer interaction with more than 3,000 attendees and 1,000 authors, we collected attendees’ preferences by making available all the accepted papers at the conference on a paper recommendation tool we built called Confer, for a period of 45 days before announcing the conference program (sessions and schedule). We compare the preferences marked on Confer with the preferences collected from Cobi’s author-sourcing approach. We show that attendee-sourcing can provide insights beyond what can be discovered by author-sourcing. For CHI 2014, the results show value in the method and attendees’ participation. It produces data that provides more alternatives in scheduling and complements data collected from other methods for creating coherent sessions and reducing conflicts.
Predicting Next Label Quality: A Time-Series Model of Crowdwork
Authors: Jung Hyun , Park Yubin , Lease Matthew
Published: October 2022

While temporal behavioral patterns can be discerned to underlie real crowd work, prior studies have typically modeled worker performance under a simplified i.i.d. assumption. To better model such temporal worker behavior, we propose a time-series label prediction model for crowd work. This latent variable model captures and summarizes past worker behavior, enabling us to better predict the quality of each worker's next label. Given inherent uncertainty in prediction, we also investigate a decision reject option to balance the tradeoff between prediction accuracy vs. coverage. Results show our model improves accuracy of both label prediction on real crowd worker data, as well as data quality overall.
Parallel Task Routing for Crowdsourcing
Authors: Bragg Jonathan , Kolobov Andrey , Mausam Mausam , Weld Daniel
Published: October 2022

An ideal crowdsourcing or citizen-science system would route tasks to the most appropriate workers, but the best assignment is unclear because workers have varying skill, tasks have varying difficulty, and assigning several workers to a single task may significantly improve output quality. This paper defines a space of task routing problems, proves that even the simplest is NP-hard, and develops several approximation algorithms for parallel routing problems. We show that an intuitive class of requesters' utility functions is submodular, which lets us provide iterative methods for dynamically allocating batches of tasks that make near-optimal use of available workers in each round. Experiments with live oDesk workers show that our task routing algorithm uses only 48% of the human labor compared to the commonly used round-robin strategy. Further, we provide versions of our task routing algorithm which enable it to scale to large numbers of workers and questions and to handle workers with variable response times while still providing significant benefit over common baselines.
Friendsourcing for the Greater Good: Perceptions of Social Microvolunteering
Authors: Brady Erin , Morris Meredith , Bigham Jeffrey
Published: October 2022

People with disabilities can be reluctant to friendsource help from their own friends for fear of appearing dependent or annoying.  Our social microvolunteering approach has volunteers post friendsourcing tasks on behalf of people with disabilities. We demonstrate this approach via a Facebook application that answers visual questions on behalf of blind users.
Proposal and Initial Study for Animal Crowdsourcing
Authors: Makiguchi Motohiro , Namikawa Daichi , Nakamura Satoshi , Yoshida Taiga , Yokoyama Masanori , Takano Yuji
Published: October 2022

We focus on animals as a resource of processing capability in crowdsourcing and propose an Animal Crowdsourcing (we call it "Animal Cloud”, too) that resolves problems with cooperation between computers and human or animals. This paper gives an overview of Animal Crowdsourcing and reports on the interim results of our learning experiments using rats (Long-Evans rats) to verify the feasibility of Animal Crowdsourcing.
Post It or Not: Viewership Based Posting of Crowdsourced Tasks
Authors: Manohar Pallavi , Chander Deepthi , Celis Elisa , Dasgupta Koustuv , Bhattacharya Sakyajit
Published: October 2022

We propose an online scheduling algorithm for posting crowdsourcing tasks which maximizes a novel metric called task viewership. This metric is computed using stochastic model based on coverage process and it measures the likelihood that a task is viewed by multiple crowd workers, which is correlated to the likelihood that it will be selected and completed.
Crowdsourced Data Analytics: A Case Study of a Predictive Modeling Competition
Authors: Baba Yukino , Nori Nozomi , Saito Shigeru , Kashima Hisashi
Published: October 2022

Predictive modeling competitions provide a new data mining approach that leverages crowds of data scientists to examine a wide variety of predictive models and build the best performance model. In this paper, we report the results of a study conducted on CrowdSolving, a platform for predictive modeling competitions in Japan. We hosted a competition on a link prediction task and observed that (i) the prediction performance of the winner significantly outperformed that of a state-of-the-art method, (ii) the aggregated model constructed from all submitted models further improved the final performance, and (iii) the performance of the aggregated model built only from early submissions nevertheless overtook the final performance of the winner.
The Air Sensor Citizen Science Toolbox: A Collaboration in Community Air Quality Monitoring and Mapping
Authors: Kaufman Amanda , Williams Ron , Barzyk Timothy , Hoang Anhthu , Sheridan Patricia
Published: October 2022

The Air Sensor Citizen Science Toolbox is being developed by the US EPA with resources and other tools that can be used by citizens to learn more about air sensor technology. The toolbox has an overarching goal of providing guidance and instructions to citizens to allow them to effectively collect, analyze, interpret, and communicate air quality data. EPA’s Office of Research and Development is collaborating with select EPA Regions and local community organizations to develop sensor packages for community-based monitoring to initiate the long-range goal. A pilot community involved in rolling out specific components of the toolbox is bordered by highways, rail yards, major port activities, international airports, and industrial facilities, amongst others. It is believed that residents might be exposed to a variety of airborne pollutants from these sources. To achieve the project’s goal to raise awareness of local air pollution levels through citizen science-based monitoring, scientists involved in the pilot project are: Working with community members to develop a study design for monitoring air quality in the community based on sound science and local community concerns; Assisting the community in selecting appropriate pollutants for citizen scientists to monitor; Providing necessary environmental sensors, one-on-one training, and written directions for sensor use; Guiding community members on data analysis, data viewing options, and communication. A future goal of this pilot project is to translate these methods across other EPA Regional offices so they can collaborate with their local community action groups and address similar issues. The Citizen Science Toolbox has the potential to be a valuable resource for such collaborations.
Finding a City’s Activity Bubbles in Geotagged Social Media
Authors: Tasse Dan , Hong Jason
Published: October 2022

Bill Bishop popularized the idea that we all live in a self-curated world of bubbles in his book, “The Big Sort” (Bishop 2009). People often spend most of their time in completely different places from their neighbors, and this fragmentation of society leads to reduced empathy, policy quarrels, and even violence. People may not even be aware that they are segregating themselves so dramatically. Fortunately, thanks to the abundance of available geotagged social media data, we can easily and cheaply detect these social bubbles and give people a greater insight into their activity patterns. Understanding how we spend our time can be the first step in changing from isolated citizens into a connected community.
Eight Guidelines for Designing Virtual Citizen Science Projects
Authors: Jennett Charlene , Cox Anna
Published: October 2022

Is it unknown how to best design virtual citizen science projects so as to meet the needs of both scientists and volunteers. We present 8 design guidelines that we developed as a result of common problems we identified in our usability evaluations of 3 Citizen Cyberlab prototypes.
Crowd-Training Machine Learning Systems for Human Rights Abuse Documentation
Authors: Aronson Jay
Published: October 2022

In this talk I will describe efforts being undertaken in a collaboration between human rights advocates and computer scientists at Carnegie Mellon University to develop tools, methods and algorithms that will make it easier to semi-automate the process of video forensics for human rights purposes.
PISCES: Participatory Incentive Strategies for Effective Community Engagement in Smart Cities
Authors: Biswas Arpita , Chander Deepthi , Dasgupta Koustuv , Mukherjee Koyel , Singh Mridula , Mukherjee Tridib
Published: October 2022

A key challenge in participatory sensing systems has been the design of incentive mechanisms that motivate individuals to contribute data to consuming applications. Emerging trends in urban development and smart city planning indicate the use of citizen reports to gather insights and identify areas for transformation. Consumers of these reports (e.g. city agencies) typically associate non-uniform utility (or values) to different reports based on the spatio-temporal context of the reports. For example, a report indicating traffic congestion near an airport, in early morning hours, would tend to have much higher utility than a similar report from a sparse residential area. In such cases, the design of an incentive mechanism must motivate participants, via appropriate rewards (or payments), to provide higher utility reports when compared to less valued ones. The main challenge in designing such an incentive scheme is two-fold: (i) lack of prior knowledge of participants in terms of their availability (i.e. who are in the vicinity) and reporting behaviour (i.e. what are the rewards expected); and (ii) minimizing payments to the reporters while ensuring that the desired number of reports are collected. In this paper, we propose STOC-PISCES, an algorithm that guarantees a stochastic optimal solution in the generalized setting of an unknown set of participants, with non-deterministic availabilities and stochastically rational reporting behaviour. The superior performance of STOC-PISCES in experimental settings, based on real-world data, endorses its adoption as an incentive strategy in participatory sensing applications like smart city management.
From "In" to "Over": Behavioral Experiments on Whole-Network Computation
Authors: Dworkin Lili , Kearns Michael
Published: October 2022

We report on a series of behavioral experiments in human computation on three different tasks over networks: graph coloring, community detection (or graph clustering), and competitive contagion. While these tasks share similar action spaces and interfaces, they capture a diversity of computational challenges: graph coloring is a search problem, clustering is an optimization problem, and competitive contagion is a game-theoretic problem. In contrast with most of the prior literature on human-subject experiments in networks, in which collectives of subjects are embedded "in" the network, and have only local information and interactions, here individual subjects have a global (or "over") view and must solve "whole network" problems alone. Our primary findings are that subject performance is impressive across all three problem types; that subjects find diverse and novel strategies for solving each task; and that collective performance can often be strongly correlated with known algorithms.
Tropel: Crowdsourcing Detectors with Minimal Training
Authors: Patterson Genevieve , Van Horn Grant , Belongie Serge , Perona Pietro , Hays James
Published: October 2022

This paper introduces the Tropel system which enables non-technical users to create arbitrary visual detectors without first annotating a training set. Our primary contribution is a crowd active learning pipeline that is seeded with only a single positive example and an unlabeled set of training images. We examine the crowd's ability to train visual detectors given severely limited training themselves. This paper presents a series of experiments that reveal the relationship between worker training, worker consensus and the average precision of detectors trained by crowd-in-the-loop active learning. In order to verify the efficacy of our system, we train detectors for bird species that work nearly as well as those trained on the exhaustively labeled CUB 200 dataset at significantly lower cost and with little effort from the end user. To further illustrate the usefulness of our pipeline, we demonstrate qualitative results on unlabeled datasets containing fashion images and street-level photographs of Paris.
Crowdsourcing Feature Discovery via Adaptively Chosen Comparisons
Authors: Zou James , Chaudhuri Kamalika , Kalai Adam
Published: October 2022

We introduce an unsupervised approach to efficiently discover the underlying features in a data set via crowdsourcing. Our queries ask crowd members to articulate a feature common to two out of three displayed examples. In addition, we ask the crowd to provide binary labels for these discovered features on the remaining examples. The triples are chosen adaptively based on the labels of the previously discovered features on the data set. This approach is motivated by a formal framework of feature elicitation that we introduce and analyze in this paper. In two natural models of features, hierarchical and independent, we show that a simple adaptive algorithm recovers all features with less labor than any nonadaptive algorithm. The savings are as a result of automatically avoiding the elicitation of redundant features or synonyms. Experimental results validate the theoretical findings and the usefulness of this approach.
Crowdsourcing in the Field: A Case Study Using Local Crowds for Event Reporting
Authors: Agapie Elena , Teevan Jaime , Monroy-Hernández Andrés
Published: October 2022

While crowd work typically involves tasks that performed at any time and anywhere, some tasks inherently require the physical presence of workers at a specific time and location. This paper presents a case study of a hybrid crowdsourcing process that involves the collaborative production of event reports using a combination of local and remote workers. The process extends human computation into the physical world by using local workers to collect information in person at events and remote workers to curate the collected information and generate event reports. We deployed the process at 11 events, employing 84 workers, and identified the challenges local workers face as constraints in mobility, time available to perform tasks, unpredictability of events, and interaction with others. We discuss issues related to collaboration with remote workers and bias in field reporting, and conduct a qualitative analysis to make design recommendations for extending human computation into the physical environment.
Crowdlines: Supporting Synthesis of Diverse Information Sources through Crowdsourced Outlines
Authors: Luther Kurt , Hahn Nathan , Dow Steven , Kittur Aniket
Published: October 2022

Learning about a new area of knowledge is challenging for novices partly because they are not yet aware of which topics are most important. The Internet contains a wealth of information for learning the underlying structure of a domain, but relevant sources often have diverse structures and emphases, making it hard to discern what is widely considered essential knowledge vs. what is idiosyncratic. Crowdsourcing offers a potential solution because humans are skilled at evaluating high-level structure, but most crowd micro-tasks provide limited context and time. To address these challenges, we present Crowdlines, a system that uses crowdsourcing to help people synthesize diverse online information. Crowdworkers make connections across sources to produce a rich outline that surfaces diverse perspectives within important topics. We evaluate Crowdlines with two experiments. The first experiment shows that a high context, low structure interface helps crowdworkers perform faster, higher quality synthesis, while the second experiment shows that a tournament-style (parallelized) crowd workflow produces faster, higher quality, more diverse outlines than a linear (serial/iterative) workflow.
Cheaper and Better: Selecting Good Workers for Crowdsourcing
Authors: Li Hongwei , Liu Qiang
Published: October 2022

Crowdsourcing provides a popular paradigm for data collection at scale. We study the problem of selecting subsets of workers from a given worker pool to maximize the accuracy under a budget constraint. One natural question is whether we should hire as many workers as the budget allows, or restrict on a small number of top-quality workers. By theoretically analyzing the error rate of a typical setting in crowdsourcing, we frame the worker selection problem into a combinatorial optimization problem and propose an algorithm to solve it efficiently. Empirical results on both simulated and real-world datasets show that our algorithm is able to select a small number of high-quality workers, and performs as good as, sometimes even better than, the much larger crowds as the budget allows.
Predicting Quality of Crowdsourced Image Segmentations from Crowd Behavior
Authors: Sameki Mehrnoosh , Gurari Danna , Betke Margrit
Published: October 2022

Quality control (QC) is an integral part of many crowd- sourcing systems. However, popular QC methods, such as aggregating multiple annotations, filtering workers, or verifying the quality of crowd work, introduce additional costs and delays. We propose a complementary paradigm to these QC methods based on predicting the quality of submitted crowd work. In particular, we pro- pose to predict the quality of a given crowd drawing directly from a crowd worker’s drawing time, number of user clicks, and average time per user click. We focus on the task of drawing the boundary of a single object in an image. To train and test our prediction models, we collected a total of 2,025 crowd-drawn segmentations for 405 familiar everyday images and unfamiliar biomedical images from 90 unique crowd workers. We first evaluated five prediction models learned using different combinations of the three worker behavior cues for all images. Experiments revealed that time per number of user clicks was the most effective cue for predicting segmentation quality. We next inspected the predictive power of models learned using crowd annotations collected for familiar and unfamiliar data independently. Prediction models were significantly more effective for estimating the segmentation quality from crowd worker behavior for familiar image content than unfamiliar image content.
Crowdclass: Designing Classification-Based Citizen Science Learning Modules
Authors: Lee Doris Jung-Lin , Lo Joanne , Kim Moonhyok , Paulos Eric
Published: October 2022

In this paper, we introduce Crowdclass, a novel framework that integrates the learning of advanced scientific concepts with the crowdsourcing microtask of image classification. In Crowdclass, we design questions to serve as both a learning experience and a scientific classification. This is different from conventional citizen science platforms which decompose high level questions into a series of simple microtasks that require no scientific background knowledge to complete. We facilitate learning within the microtask by providing content that is appropriate for the participant’s level of knowledge through scaffolding learning. We conduct a between-group study of 93 participants on Amazon Mechanical Turk comparing Crowdclass to the popular citizen science project Galaxy Zoo. We find that the scaffolding presentation of content enables learning of more challenging concepts. By understanding the relationship between user motivation, learning, and performance, we draw general design principles for learning-as-an-incentive interventions applicable to other crowdsourcing applications.
Modeling Task Complexity in Crowdsourcing
Authors: Yang Jie , Redi Judith , Demartini Gianluca , Bozzon Alessandro
Published: October 2022

Complexity is crucial to characterize tasks performed by humans through computer systems. Yet, the theory and practice of crowdsourcing currently lacks a clear understanding of task complexity, hindering the design of effective and efficient execution interfaces or fair monetary rewards.  To understand how complexity is perceived and distributed over crowdsourcing tasks, we instrumented an experiment where we asked workers to evaluate the complexity of 61 real-world re-instantiated crowdsourcing tasks. We show that task complexity, while being subjective, is coherently perceived across workers; on the other hand, it is significantly influenced by task type. Next, we develop a high-dimensional regression model, to assess the influence of three classes of structural features (metadata, content, and visual) on task complexity, and ultimately use them to measure task complexity. Results show that both the appearance and the language used in task description can accurately predict task complexity. Finally, we apply the same feature set to predict task performance, based on a set of 5 years-worth tasks in Amazon MTurk. Results show that features related to task complexity can improve the quality of task performance prediction, thus demonstrating the utility of complexity as a task modeling property.
Crowdsourcing Relevance Assessments: The Unexpected Benefits of Limiting the Time to Judge
Authors: Maddalena Eddy , Basaldella Marco , De Nart Dario , Degl'Innocenti Dante , Mizzaro Stefano , Demartini Gianluca
Published: October 2022

Crowdsourcing has become an alternative approach to collect relevance judgments at scale thanks to the availability of crowdsourcing platforms and quality control techniques that allow to obtain reliable results. Previous work has used crowdsourcing to ask multiple crowd workers to judge the relevance of a document with respect to a query and studied how to best aggregate multiple judgments of the same topic-document pair. This paper addresses an aspect that has been rather overlooked so far: we study how the time available to express a relevance judgment affects its quality. We also discuss the quality loss of making crowdsourced relevance judgments more efficient in terms of time taken to judge the relevance of a document. We use standard test collections to run a battery of experiments on the crowdsourcing platform CrowdFlower, studying how much time crowd workers need to judge the relevance of a document and at what is the effect of reducing the available time to judge on the overall quality of the judgments. Our extensive experiments compare judgments obtained under different types of time constraints with judgments obtained when no time constraints were put on the task. We measure judgment quality by different metrics of agreement with editorial judgments. Experimental results show that it is possible to reduce the cost of crowdsourced evaluation collection creation by reducing the time available to perform the judgments with no loss in quality. Most importantly, we observed that the introduction of limits on the time available to perform the judgments improves the overall judgment quality. Top judgment quality is obtained with 25-30 seconds to judge a topic-document pair.
Supporting Image Geolocation with Diagramming and Crowdsourcing
Authors: Kohler Rachel , Purviance John , Luther Kurt
Published: October 2022

Geolocation, the process of identifying the precise location in the world where a photo or video was taken, is central to many types of investigative work, from debunking fake news posted on social media to locating terrorist training camps. Professional geolocation is often a manual, time-consuming process that involves searching large areas of satellite imagery for potential matches. In this paper, we explore how crowdsourcing can be used to support expert image geolocation. We adapt an expert diagramming technique to overcome spatial reasoning limitations of novice crowds, allowing them to support an expert’s search. In two experiments (n=1080), we found that diagrams work significantly better than ground-level photos and allow crowds to reduce a search area by half before any expert intervention. We also discuss hybrid approaches to complex image analysis combining crowds, experts, and computer vision.
Let's Agree to Disagree: Fixing Agreement Measures for Crowdsourcing
Authors: Checco Alessandro , Roitero Kevin , Maddalena Eddy , Mizzaro Stefano , Demartini Gianluca
Published: October 2022

In the context of micro-task crowdsourcing, each task is usually performed by several workers. This allows researchers to leverage measures of the agreement among workers on the same task, to estimate the reliability of collected data and to better understand answering behaviors of the participants. While many measures of agreement between annotators have been proposed, they are known for suffering from many problems and abnormalities. In this paper, we identify the main limits of the existing agreement measures in the crowdsourcing context, both by means of toy examples as well as with real-world crowdsourcing data, and propose a novel agreement measure based on probabilistic parameter estimation which overcomes such limits. We validate our new agreement measure and show its flexibility as compared to the existing agreement measures.
Deja Vu: Characterizing Worker Reliability Using Task Consistency
Authors: Williams Alex , Goh Joslin , Willis Charlie , Ellison Aaron , Brusuelas James , Davis Charles , Law Edith
Published: October 2022

Consistency is a practical metric that evaluates an instrument's reliability based on its ability to yield the same output when repeatedly given a particular input. Despite its broad usage, little is understood about the feasibility of using consistency as a measure of worker reliability in crowdwork. In this paper, we explore the viability of measuring a worker's reliability by their ability to conform to themselves. We introduce and describe Deja Vu, a mechanism for dynamically generating task queues with consistency probes to measure the consistency of workers who repeat the same task twice. We present a study that utilizes Deja Vu to examine how generic characteristics of the duplicate task - such as placement, difficulty, and transformation - affect a worker’s task consistency in the context of two unique object detection tasks. Our findings provide insight into the design and use of consistency-based reliability metrics.
Dynamic Filter: Adaptive Query Processing with the Crowd
Authors: Lan Doren , Reed Katherine , Shin Austin , Trushkowsky Beth
Published: October 2022

Hybrid human-machine query processing systems, such as crowd-powered database systems, aim to broaden the scope of questions users can ask about their data by incorporating human computation to support queries that may be subjective and/or require visual or semantic interpretation. A common type of query involves filtering data by several criteria, some of which need human computation to be evaluated. For example, filtering a set of hotels for those that both (1) have great views from the rooms, and (2) have a fitness center. Criteria can differ in the amount of human effort required to decide if data satisfy them, due to criterion's subjectivity and difficulty. There is potential to reduce crowdsourcing costs by ordering the evaluation of each of the criteria such that criteria needing more human computation are not processed for data that have not satisfied the less costly criteria. Unfortunately, for queries specified on-the-fly, the information about subjectivity and difficulty is unknown a priori. To overcome this challenge, we present Dynamic Filter, an adaptive query processing algorithm that dynamically changes the order in which criteria are evaluated based on observations while the query is running. Using crowdsourced data from a popular crowdsourcing platform, we show that Dynamic Filter can effectively adapt the processing order and approach the performance of a "clairvoyant" algorithm.
Evaluating Visual Conversational Agents via Cooperative Human-AI Games
Authors: Chattopadhyay Prithvijit , Yadav Deshraj , Prabhu Viraj , Chandrasekaran Arjun , Das Abhishek , Lee Stefan , Batra Dhruv , Parikh Devi
Published: October 2022

As AI continues to advance, human-AI teams are inevitable. However, progress in AI is routinely measured in isolation, without a human in the loop. It is crucial to benchmark progress in AI, not just in isolation, but also in terms of how it translates to helping humans perform certain tasks, i.e., the performance of human-AI teams. In this work, we design a cooperative game — GuessWhich — to measure human-AI team performance in the specific context of the AI being a visual conversational agent. GuessWhich involves live interaction between the human and the AI. The AI, which we call ALICE, is provided an image which is unseen by the human. Following a brief description of the image, the human questions ALICE about this secret image to identify it from a fixed pool of images. We measure performance of the human-ALICE team by the number of guesses it takes the human to correctly identify the secret image after a fixed number of dialog rounds with ALICE. We compare performance of the human-ALICE teams for two versions of ALICE. Our human studies suggest a counterintuitive trend – that while AI literature shows that one version outperforms the other when paired with an AI questioner bot, we find that this improvement in AI-AI performance does not translate to improved human-AI performance. This suggests a mismatch between benchmarking of AI in isolation and in the context of human-AI teams.
Supporting ESL Writing by Prompting Crowdsourced Structural Feedback
Authors: Huang Yi-Ching , Huang Jiunn-Chia , Wang Hao-Chuan , Hsu Jane
Published: October 2022

Writing is challenging, especially for non-native speakers. To support English as a Second Language (ESL) writing, we propose StructFeed, which allows native speakers to annotate topic sentence and relevant keywords in texts and generate writing hints based on the principle of paragraph unity. First, we compared our crowd-based method with three naive machine learning (ML) methods and got the best performance on the identification of topic sentence and irrelevant sentence in the article. Next, we evaluated the StructFeed system with two feedback-generation mechanisms including feedback generated by one expert and by one crowd worker. The results showed that people who received feedback by StructFeed got the highest improvement after revision.
Leveraging Side Information to Improve Label Quality Control in Crowd-Sourcing
Authors: Jin Yuan , Carman Mark , Kim Dongwoo , Xie Lexing
Published: October 2022

We investigate the possibility of leveraging side information for improving quality control over crowd-sourced data. We extend the GLAD model, which governs the probability of correct labeling through a logistic function in which worker expertise counteracts item difficulty, by systematically encod- ing different types of side information, including worker in- formation drawn from demographics and personality traits, item information drawn from item genres and content, and contextual information drawn from worker responses and la- beling sessions. Modeling side information allows for better estimation of worker expertise and item difficulty in sparse data situations and accounts for worker biases, leading to bet- ter prediction of posterior true label probabilities. We demon- strate the efficacy of the proposed framework with overall improvements in both the true label prediction and the un- seen worker response prediction based on different combina- tions of the various types of side information across three new crowd-sourcing datasets. In addition, we show the framework exhibits potential of identifying salient side information fea- tures for predicting the correctness of responses without the need of knowing any true label information.