bluuebunny/crossref_metadata_2025 huggingface dataset:
HiveMind: Tuning Crowd Response with a Single Value
Authors: Singh Preetjot , Lasecki Walter , Barelli Paulo , Bigham Jeffrey
Published: October 2022

One common problem plaguing crowdsourcing tasks is tuning the set of worker responses: Depending on task requirements, requesters may want a large set of rich and varied worker responses (typically in subjective evaluation tasks) or a more convergent response-set (typically for more objective tasks such as fact-checking). This problem is especially salient in tasks that combine workers’ responses to present a single output: Divergence in these settings could either add richness and complexity to the unified answer, or noise. In this paper we present HiveMind, a system of methods that allow requesters to tune different levels of convergence in worker participation for different tasks simply by adjusting the value of one variable.
Personalized Human Computation
Authors: Organisciak Peter , Teevan Jaime , Dumais Susan , Miller Robert , Kalai Adam
Published: October 2022

Significant effort in machine learning and information retrieval has been devoted to identifying personalized content such as recommendations and search results. Personalized human computation has the potential to go beyond existing techniques like collaborative filtering to provide personal­ized results on demand, over personal data, and for complex tasks. This work-in-progress compares two approaches to personal­ized human computation. In both, users annotate a small set of training examples which are then used by the crowd to annotate unseen items. In the first approach, which we call taste-matching, crowd members are asked to annotate the same set of training examples, and the ratings of similar users on other items are then used to infer personal­ized ratings. In the second approach, taste-grokking, the crowd is presented with the training examples and asked to use them predict the ratings of the target user on other items.
Groupsourcing: Problem Solving, Social Learning and Knowledge Discovery on Social Networks
Authors: Chamberlain Jon
Published: October 2022

Increasingly social networks are being used for citizen science, where members of the public contribute knowledge to scientific endeavours. Tasks can be presented and solved using human computation, termed groupsourcing, with users benefiting from community tuition and experts gaining knowledge from the crowd. This paper gives details of a prototype that utilises groupsourcing to solve image classification tasks, to support social learning and to facilitate knowledge discovery in the domain of marine biology.
IdeaGens: A Social Ideation System for Guided Crowd Brainstorming
Authors: Chan Joel , Dang Steven , Kremer Peter , Guo Lucy , Dow Steven
Published: October 2022

Many crowd ideation systems seek to gather scores of ideas from people online. However, this often leads to many bad ideas and duplication. A dedicated facilitator who guides exploration of the solution space is a common and effective strategy for optimizing ideation in face-to-face brainstorming, but has not yet been explored in computer-supported crowd ideation. We introduce IdeaGens, a social ideation system for guided crowd brainstorming. IdeaGens divides the crowd into ideation and synthesis tasks, and enables efficient data-driven facilitation of the crowd’s ideation. This work can inform general strategies for shepherding the crowd to produce better results for complex collaborative tasks.
HCOMP-14 Organization
Authors: Bigham Jeffrey , Parkes David
Published: October 2022

List of organizers of the AAAI Conference on Human Computation and Crowdsourcing.
Sponsors
Authors: Bigham Jeffrey , Parkes David
Published: October 2022

List of sponsors of the 2014 AAAI Conference on Human Computation and Crowdsourcing.
Context Trees: Crowdsourcing Global Understanding from Local Views
Authors: Verroios Vasilis , Bernstein Michael
Published: October 2022

Crowdsourcing struggles when workers must see all of the pieces of input to make an accurate judgment. For example, to find the most important scenes in a novel or movie, each worker must spend hours consuming the entire plot to acquire a global understanding and then apply that understanding to each local scene. To enable the crowdsourcing of large-scale goals with only local views, we introduce context trees, a crowdsourcing workflow for creating global summaries of a large input. Context trees recursively combine elements through written summaries to form a tree. Workers can then ground their local decisions by applying those summaries back down to the leaf nodes. In the case of scale ratings such as scene importance, we introduce a weighting process that percolates ratings downwards through the tree so that important nodes in unimportant branches are not overweighted. When using context trees to rate the importance of scenes in a 4000-word story and a 100-minute movie, workers’ ratings are nearly as accurate as those who saw the entire input, and much improved over the traditional approach of splitting the input into independent segments. To explore whether context trees enable crowdsourcing to undertake new classes of goals, we also crowdsource the solution to a large hierarchical puzzle of 462,000 interlocking pieces.
Crowdsourcing for Participatory Democracies: Efficient Elicitation of Social Choice Functions
Authors: Lee David , Goel Ashish , Aitamurto Tanja , Landemore Helene
Published: October 2022

We present theoretical and empirical results demonstrating the usefulness of social choice functions in crowdsourcing for participatory democracies. First, we demonstrate the scalability of social choice functions by defining a natural notion of epsilon-approximation, and giving algorithms which efficiently elicit such approximations for two prominent social choice functions: the Borda rule and the Condorcet winner. This result circumvents previous prohibitive lower bounds and is surprisingly strong: even if the number of ideas is as large as the number of participants, each participant will only have to make a logarithmic number of comparisons, an exponential improvement over the linear number of comparisons previously needed. Second, we apply these ideas to Finland's recent off-road traffic law reform, an experiment on participatory democracy in real life. This allows us to verify the scaling predicted in our theory and show that the constant involved is also not large. In addition, by collecting data on the time that users take to complete rankings of varying sizes, we observe that eliciting partial rankings can further decrease elicitation time as compared to the common method of eliciting pairwise comparisons.
A Crowd of Your Own: Crowdsourcing for On-Demand Personalization
Authors: Organisciak Peter , Teevan Jaime , Dumais Susan , Miller Robert , Kalai Adam
Published: October 2022

Personalization is a way for computers to support people’s diverse interests and needs by providing content tailored to the individual. While strides have been made in algorithmic approaches to personalization, most require access to a significant amount of data. However, even when data is limited online crowds can be used to infer an individual’s personal preferences. Aided by the diversity of tastes among online crowds and their ability to understand others, we show that crowdsourcing is an effective on-demand tool for personalization. Unlike typical crowdsourcing approaches that seek a ground truth, we present and evaluate two crowdsourcing approaches designed to capture personal preferences. The first, taste-matching, identifies workers with similar taste to the requester and uses their taste to infer the requester’s taste. The second, taste-grokking, asks workers to explicitly predict the requester’s taste based on training examples. These techniques are evaluated on two subjective tasks, personalized image recommendation and tailored textual summaries. Taste-matching and taste-grokking both show improvement over the use of generic workers, and have different benefits and drawbacks depending on the complexity of the task and the variability of the taste space.
Robot Programming by Demonstration with Crowdsourced Action Fixes
Authors: Forbes Maxwell , Chung Michael , Cakmak Maya , Rao Rajesh
Published: October 2022

Programming by Demonstration (PbD) can allow end-users to teach robots new actions simply by demonstrating them. However, learning generalizable actions requires a large number of demonstrations that is unreasonable to expect from end-users. In this paper, we explore the idea of using crowdsourcing to collect action demonstrations from the crowd. We propose a PbD framework in which the end-user provides an initial seed demonstration, and then the robot searches for scenarios in which the action will not work and requests the crowd to fix the action for these scenarios. We use instance-based learning with a simple yet powerful action representation that allows an intuitive visualization of the action. Crowd workers directly interact with these visualizations to fix them. We demonstrate the utility of our approach with a user study involving local crowd workers (N=31) and analyze the collected data and the impact of alternative design parameters so as to inform a real-world deployment of our system.
Crowdsourced Explanations for Humorous Internet Memes Based on Linguistic Theories
Authors: Lin Chi-Chin , Huang Yi-Ching , Hsu Jane Yung-jen
Published: October 2022

Humorous images can be seen in many social media websites. However, newcomers to these websites often have trouble fitting in because the community subculture is usually implicit. Among all the types of humorous images, Internet memes are relatively hard for newcomers to understand. In this work, we develop a system that leverages crowdsourcing techniques to generate explanations for memes. We claim that people who are not familiar with Internet meme subculture can still quickly pick up the gist of the memes by reading the explanations. Our template-based explanations illustrate the incongruity between normal situations and the punchlines in jokes. The explanations can be produced by completing the two proposed human task processes. Experimental results suggest that the explanations produced by our system greatly help newcomers to understand unfamiliar memes. For further research, it is possible to employ our explanation generation system to improve computational humanities.
Behavior-Based Quality Assurance in Crowdsourcing Markets
Authors: Feldman Michael , Bernstein Abraham
Published: October 2022

Quality assurance in crowdsourcing markets has appeared to be an acute problem over the last years. We propose a quality control method inspired by Statistical Process Control (SPC), commonly used to control output quality in production processes and characterized by relying on time-series data. Behavioral traces of users may play a key role in evaluating the performance of work done on crowdsourcing platforms. Therefore, in our experiment we explore fifteen behavioral traces for their ability to recognize the drop in work quality. Preliminary results indicate that our method has a high potential for real-time detection and signaling a drop in work quality.
Product Concept Evaluation System Applying Preference Market
Authors: Imai Miku , Mizuyama Hajime
Published: October 2022

A product concept evaluation system combining conjoint analysis with prediction markets is developed. It is also proposed how to determine the payoff for each prediction security corresponding to a product concept, so as to have participants to behave truthfully in the market. Further, how the proposed system works is investigated by evolutionary game simulation.
Crowdsourcing the Extraction of Data Practices from Privacy Policies
Authors: Schaub Florian , Breaux Travis , Sadeh Norman
Published: October 2022

Website and mobile application privacy policies are intended to describe the system’s data practices. However, they are often written in non-standard formats and contain ambiguities that make it difficult for users to read and comprehend these documents. We propose a crowdsourcing approach to extract data practices from privacy policies to provide more concise and useable privacy notices to users and support the analysis of stated data practices. To that end, we designed a hierarchical task workflow for crowdsourcing the extraction of data practices from privacy policies. We discuss our workflow design and report preliminary results.
Low Effort Crowdsourcing: Leveraging Peripheral Attention for Crowd Work
Authors: Vaish Rajan , Organisciak Peter , Hara Kotaro , Bigham Jeffrey , Zhang Haoqi
Published: October 2022

Crowdsourcing systems leverage short bursts of focused attention from many contributors to achieve a goal. By requiring people’s full attention, existing crowdsourcing systems fail to leverage people’s cognitive surplus in the many settings for which they may be distracted, performing or waiting to perform another task, or barely paying attention. In this paper, we study opportunities for low-effort crowdsourcing that enable people to contribute to problem solving in such settings. We discuss the design space for low-effort crowdsourcing, and through a series of prototypes, demonstrate interaction techniques, mechanisms, and emerging principles for enabling low-effort crowdsourcing.
Contextual Procurement in Online Crowdsourcing Markets
Authors: Singla Adish , Lienert Ian , Bartók Gábor , Krause Andreas
Published: October 2022

Designing pricing mechanisms for recruitment of workers is a central challenge in online crowdsourcing markets. We consider a novel and realistic setting for such markets, where the private cost of the workers and utility derived from them is unknown to the mechanism; however, a set of the workers' features can be observed before making the price offer. How should the offered price be adapted to maximize the utility from recruited workers, while minimizing the cost of payments and the idling cost of failure to recruit? In this paper, we address these questions by formulating the problem as a {\em contextual partial monitoring game}, a generic framework for online learning problems that allows to deal with complex feedback structure. We present simulation results comparing our approach to the classical contextual bandit approach, demonstrating the complexity of the problem and the need for the partial monitoring framework.
AURORASAURUS: Citizen Science, Early Warning Systems and Space Weather
Authors: Tapia Andrea , LaLone Nicolas , MacDonald Elizabeth , Hall Michelle , Case Nathan , Heavner Matt
Published: October 2022

We have created Aurorasaurus, a website, a mobile application and a scientific tool that allows a community of users to better predict sightings of the aurora borealis. Aurorasuarus combines limited space weather science data, participant sightings and the analysis of social media data into a better prediction engine. This then serves to alert interested parties as to nearby sightings.
Learning Supervised Topic Models from Crowds
Authors: Rodrigues Filipe , Ribeiro Bernardete , Lourenço Mariana , Pereira Francisco
Published: October 2022

The growing need to analyze large collections of documents has led to great developments in topic modeling. Since documents are frequently associated with other related variables, such as labels or ratings, much interest has been placed on supervised topic models. However, the nature of most annotation tasks, prone to ambiguity and noise, often with high volumes of documents, deem learning under a single-annotator assumption unrealistic or unpractical for most real-world applications. In this paper, we propose a supervised topic model that accounts for the heterogeneity and biases among different annotators that are encountered in practice when learning from crowds. We develop an efficient stochastic variational inference algorithm that is able to scale to very large datasets, and we empirically demonstrate the advantages of the proposed model over state of the art approaches.
Modeling Temporal Crowd Work Quality with Limited Supervision
Authors: Jung Hyun Joon , Lease Matthew
Published: October 2022

While recent work has shown that a worker’s performance can be more accurately modeled by temporal correlation in task performance, a fundamental challenge remains in the need for expert gold labels to evaluate a worker’s performance. To solve this problem, we explore two methods of utilizing limited gold labels, initial training and periodic updating. Furthermore, we present a novel way of learning a prediction model in the absence of gold labels with uncertaintyaware learning and soft-label updating. Our experiment with a real crowdsourcing dataset demonstrates that periodic updating tends to show better performance than initial training when the number of gold labels are very limited (< 25).
Crowdsourced Nonparametric Density Estimation Using Relative Distances
Authors: Ukkonen Antti , Derakhshan Behrouz , Heikinheimo Hannes
Published: October 2022

In this paper we address the following density estimation problem: given a number of relative similarity judgements over a set of items D, assign a density value p(x) to each item x in D. Our work is motivated by human computing applications where density can be interpreted e.g. as a measure of the rarity of an item. While humans are excellent at solving a range of different visual tasks, assessing absolute similarity (or distance) of two items (e.g. photographs) is difficult. Relative judgements of similarity, such as A is more similar to B than to C, on the other hand, are substantially easier to elicit from people. We provide two novel methods for density estimation that only use relative expressions of similarity. We give both theoretical justifications, as well as empirical evidence that the proposed methods produce good estimates.
Online Assignment of Heterogeneous Tasks in Crowdsourcing Markets
Authors: Assadi Sepehr , Hsu Justin , Jabbari Shahin
Published: October 2022

We investigate the problem of heterogeneous task assignment in crowdsourcing markets from the point of view of the requester, who has a collection of tasks. Workers arrive online one by one, and each declare a set of feasible tasks they can solve, and desired payment for each feasible task. The requester must decide on the fly which task (if any) to assign to the worker, while assigning workers only to feasible tasks. The goal is to maximize the number of assigned tasks with a fixed overall budget. We provide an online algorithm for this problem and prove an upper bound on the competitive ratio of this algorithm against an arbitrary (possibly worst-case) sequence of workers who want small payments relative to the requester’s total budget. We further show an almost matching lower bound on the competitive ratio of any algorithm in this setting. Finally, we propose a different algorithm that achieves an improved competitive ratio in the random permutation model, where the order of arrival of the workers is chosen uniformly at random. Apart from these strong theoretical guarantees, we carry out experiments on simulated data which demonstrates the practical applicability of our algorithms.
Identifying and Accounting for Task-Dependent Bias in Crowdsourcing
Authors: Kamar Ece , Kapoor Ashish , Horvitz Eric
Published: October 2022

Models for aggregating contributions by crowd workers have been shown to be challenged by the rise of task-specific biases or errors. Task-dependent errors in assessment may shift the majority opinion of even large numbers of workers to an incorrect answer. We introduce and evaluate probabilistic models that can detect and correct task-dependent bias automatically. First, we show how to build and use probabilistic graphical models for jointly modeling task features, workers' biases, worker contributions and ground truth answers of tasks so that task-dependent bias can be corrected. Second, we show how the approach can perform a type of transfer learning among workers to address the issue of annotation sparsity. We evaluate the models with varying complexity on a large data set collected from a citizen science project and show that the models are effective at correcting the task-dependent worker bias. Finally, we investigate the use of active learning to guide the acquisition of expert assessments to enable automatic detection and correction of worker bias.
Proposal of Grade Training Method in Private Crowdsourcing System
Authors: Ashikawa Masayuki , Kawamura Takahiro , Ohsuga Akihiko
Published: October 2022

Current crowdsourcing platforms such as Amazon Mechanical Turk provide an attractive solution for processing of high-volume tasks at low cost. However, problems of quality control remain a major concern. We developed a private crowdsourcing system (PCSS) running in a intranetwork, that allow us to devise for quality control methods. In the present work, we designed a novel task allocation method to improve accuracy of task results in PCSS. PCSS analyzed relations between tasks from workers' behavior using Bayesian network, then created learning tasks according to analyzed relations. PCSS increased quality of task results by allocating learning tasks to workers before processing difficult tasks. PCSS created 8 learning tasks automatically for 2 target task categories and increased accuracy of task results by 10.77 point on average. We found that creating learning tasks according to analyzed relations is a practical method to improve the quality of workers.
Understanding Socially Constructed Concepts Using Blogs Data
Authors: Gill Alastair , Iacobelli Francisco
Published: October 2022

In this paper we propose a methodology to understand complex concepts, and which captures aspects of the contextual —and collaboratively constructed — meaning of these concepts with considerably less effort than manual coding. We use the word "quality" as one such concept to exemplify our methodology. By using unsupervised topic models along with a small corpus of human labeled data we explore the different uses of the concept "quality" in a large number of blogs. Our methodology is validated, qualitatively, by comparing our results to previous research. Finally, we note limitations and future directions of this work.
Enabling Physical Crowdsourcing On-the-Go with Context-Sensitive Notifications
Authors: Kim Yongsung , Harburg Emily , Azria Shana , Gerber Elizabeth , Gergle Darren , Zhang Haoqi
Published: October 2022

This paper introduces the idea of presenting physical tasks into people’s existing routine so that people can contribute small tasks on-the-go with minimal effort. We present two on-the-go crowdsourcing systems: Libero for package delivery, and CrowdFound for finding lost items. To encourage contributions, we introduce notification techniques that present task opportunities when potential helpers are likely to accept. To direct people to regions where help is most needed, we introduce techniques for tracking a person’s location within a task region and directing their attention based on task history. Evaluation studies demonstrate the feasibility of on-the-go crowdsourcing and investigate questions over the likelihood of task completion, the perceived cost of disruption, and the effectiveness of tracking and coordination.
How Effective an Odd Message Can Be: Appropriate and Inappropriate Topics in Speech-Based Vehicle Interfaces
Authors: Sirkin David , Fischer Kerstin , Jensen Lars , Ju Wendy
Published: October 2022

Dialog between drivers and speech-based vehicle interfaces can be used as an instrument to find out what drivers might be concerned, confused or curious about in driving simulator studies. Eliciting on-going conversation with drivers about topics that go beyond navigation, control of entertainment systems, or other traditional driving related tasks is important to getting drivers to engage with the activity in an open-ended fashion. In a structured improvisational Wizard of Oz study that took place in a highly immersive driving simulator, we engaged participant drivers (N=6) in an autonomous driving course where the vehicle spoke to drivers using computer-generated natural language speech. Using microanalyses of the drivers’ responses to the car’s utter- ances, we identify a set of topics that are expected and treated as appropriate by the participants in our study, as well as a set of topics and conversational strategies that are treated as inappropriate. We also show that it is just these unexpected, inappropriate utterances that eventually increase users’ trust in the system, make them more at ease, and raise the system’s acceptability as a communication partner.
Acquiring Planning Knowledge via Crowdsourcing
Authors: Gao Jie , Zhuo Hankz Hankui , Kambhampati Subbarao , Li Lei
Published: October 2022

Plan synthesis often requires complete domain models and initial states as input. In many real world applications, it is difficult to build domain models and provide complete initial state beforehand. In this paper we propose to turn to the crowd for help before planning. We assume there are annotators available to provide information needed for building domain models and initial states. However, there might be a substantial amount of discrepancy within the inputs from the crowd. It is thus challenging to address the planning problem with possibly noisy information provided by the crowd. We address the problem by two phases. We first build a set of Human Intelligence Tasks (HITs), and collect values from the crowd. We then estimate the actual values of variables and feed the values to a planner to solve the problem.
It’s Not Just What You Say, But How You Say It: Muiltimodal Sentiment Analysis Via Crowdsourcing
Authors: Elshenawy Ahmad , Carter Steele , Braga Daniela
Published: October 2022

This paper examines the effect of various modalities of expression on the reliability of crowdsourced sentiment polarity judgments. A novel corpus of YouTube video reviews was created, and sentiment judgments were obtained via Amazon Mechanical Turk. We created a system for isolating text, video, and audio modalities from YouTube videos to ensure that annotators could only see the particular modality or modalities being evaluated. Reliability of judgments was assessed using Fleiss Kappa inter-annotator agreement values. We found that the audio only modality produced the most reliable judgments for video fragments and that across modalities video fragments are less ambiguous than full videos.
Linguistic Wisdom from the Crowd
Authors: Chang Nancy , Lee-Goldman Russell , Tseng Michael
Published: October 2022

Crowdsourcing for linguistic data typically aims to replicate expert annotations using simplified tasks. But an alternative goal — one that is especially relevant for research in the domains of language meaning and use — is to tap into people's rich experience as everyday users of language. Research in these areas has the potential to tell us a great deal about how language works, but designing annotation frameworks for crowdsourcing of this kind poses special challenges. In this paper we define and exemplify two approaches to linguistic data collection corresponding to these differing goals (model-driven and user-driven) and discuss some hybrid cases in which they overlap. We also describe some design principles and resolution techniques helpful for eliciting linguistic wisdom from the crowd.
Studying the Effects of Task Notification Policies on Participation and Outcomes in On-the-go Crowdsourcing
Authors: Kim Yongsung , Harburg Emily , Azria Shana , Shaw Aaron , Gerber Elizabeth , Gergle Darren , Zhang Haoqi
Published: October 2022

Recent years have seen the growth of physical crowdsourcing systems (e.g., Uber; TaskRabbit) that motivate large numbers of people to provide new and improved physical tasking and delivery services on-demand. In these systems, opportunistically relying on people to make convenient contributions may lead to incomplete solutions, while directing people to do inconvenient tasks requires high incentives. To increase people's willingness to participate and reduce the need to incentivize participation, we study on-the-go crowdsourcing as an alternative approach that suggests tasks along people’s existing routes that are conveniently on their way. We explore as a first step in this paper the design of task notification policies that decide when, where, and to whom to suggest tasks. Situating our work in the context of practical problems such as package delivery and lost-and-found searches, we conducted controlled experiments that show how small changes in task notification policy can influence individual participation and actions in significant ways that in turn affect system outcomes. We discuss the implications of our findings on the design of future on-the-go crowdsourcing technologies and applications.
Extending Workers' Attention Span Through Dummy Events
Authors: Elmalech Avshalom , Sarne David , David Esther , Hajaj Chen
Published: October 2022

This paper studies a new paradigm for improving the attention span of workers in tasks that heavily rely on user's attention to the occurrence of rare events. Such tasks are highly common, ranging from crime monitoring to controlling autonomous complex machines, and many of them are ideal for crowdsourcing.  The underlying idea in our approach is to dynamically augment the task with some dummy (artificial) events at different times throughout the task, rewarding the worker upon identifying and reporting them.  This, as an alternative to the traditional approach of exclusively relying on rewarding the worker for successfully identifying the event of interest itself.  We propose three methods for timing the dummy events throughout the task. Two of these methods are static and determine the timing of the dummy events at random or uniformly throughout the task. The third method is dynamic and uses the identification (or misidentification) of dummy events as a signal for the worker's attention to the task, adjusting the rate of dummy events generation accordingly. We use extensive experimentation to compare the methods with the traditional approach of inducing attention through rewarding the identification of the event of interest and within the three. The analysis of the results indicates that with the use of dummy events a substantially more favorable tradeoff between the detection (of the event of interest) probability and the expected expense can be achieved, and that among the three proposed method the one that decides on dummy events on the fly is (by far) the best.
Learning and Feature Selection under Budget Constraints in Crowdsourcing
Authors: Nushi Besmira , Singla Adish , Krause Andreas , Kossmann Donald
Published: October 2022

The cost of data acquisition limits the amount of labeled data available for machine learning algorithms, both at the training and the testing phase. This problem is further exacerbated in real-world crowdsourcing applications where labels are aggregated from multiple noisy answers. We tackle classification problems where the underlying feature labels are unknown to the algorithm and a (noisy) label of the desired feature can be acquired at a fixed cost. This problem has two types of budget constraints - the total cost of feature labels available for learning at the training phase, and the cost of features to use during the testing phase for classification. We propose a novel budgeted learning and feature selection algorithm, B-LEAFS, for jointly tackling this problem in the presence of noise. Experimental evaluation on synthetic and real-world crowdsourcing data demonstrate the practical applicability of our approach.
Efficient Techniques for Crowdsourced Top-k Lists
Authors: De Alfaro Luca , Polychronopoulos Vassilis , Polyzotis Neoklis
Published: October 2022

We propose techniques that obtain top-k lists of items out of larger itemsets, using human workers to perform comparisons among items. An example application is to short-list a large set of college applications using advanced students as workers. A method that obtains crowdsourced top-k lists has to address several challenges of crowdsourcing: there are constraints in the total number of tasks due to monetary or practical reasons; tasks posted to workers have an inherent limitation on their size; obtaining results from human workers has high latency; workers may disagree on their judgments for the same items or provide wrong results on purpose; and, there can be varying difficulty among tasks of the same size. We describe novel efficient techniques and explore their tolerance to adversarial behavior and the tradeoffs among different measures of performance (latency, expense and quality of results). We empirically evaluate the proposed techniques using simulations as well as real crowds in Amazon Mechanical Turk. A randomized variant of the proposed algorithms achieves significant budget saves, especially for very large itemsets and large top-k lists, with negligible risk of lowering the quality of the output.
Practical Peer Prediction for Peer Assessment
Authors: Shnayder Victor , Parkes David
Published: October 2022

We provide an empirical analysis of peer prediction mechanisms, which reward participants for information in settings when there is no ground truth against which to score reports. We simulate the mechanisms on a dataset of three million peer assessments from the edX MOOC platform. We evaluate different mechanisms on score variability, which is connected to fairness, risk aversion, and participant learning. We also assess the magnitude of the incentives to invest effort, and study the effect of participant coordination on low-information signals. We find that the correlated agreement mechanism has lower variation in reward than other mechanisms. A concern is that the gain from exerting effort is relatively low across all mechanisms, due to frequent disagreement between peers. Our conclusions are relevant for crowdsourcing in education as well as other domains.
A Trust-Based Coordination System for Participatory Sensing Applications
Authors: Zenonos Alexandros , Stein Sebastian , Jennings Nicholas
Published: October 2022

Participatory sensing (PS) has gained significant attention as a crowdsourcing methodology that allows ordinary citizens (non-expert contributors) to collect data using low-cost mobile devices. In particular, it has been useful in the collection of environmental data. However, current PS applications suffer from two problems. First, they do not coordinate the measurements taken by their users, which is required to maximise system efficiency. Second, they are vulnerable to malicious behaviour. In this context, we propose a novel algorithm that simultaneously addresses both of these problems. Specifically, we use heteroskedastic Gaussian Processes to incorporate users' trustworthiness into a Bayesian spatio-temporal regression model. The model is trained with measurements taken by participants, thus it is able to estimate the value of the phenomenon at any spatio-temporal location of interest and also learn the level of trustworthiness of each user. Given this model, the coordination system is able to make informed decisions concerning when, where and who should take measurements over a period of time. We empirically evaluate our algorithm on a real-world human mobility and air quality dataset, where malicious behaviour is synthetically produced, and show that our algorithm outperforms the current state of the art by up to 60.4% in terms of RMSE while having a reasonable runtime.
A Lightweight and Real-Time Worldwide Earthquake Detection and Monitoring System Based on Citizen Sensors
Authors: Maldonado Flores Jazmine , Guzman Jheser , Poblete Barbara
Published: October 2022

We propose an algorithm and system that detects earthquakes worldwide in real time based on reports of social media users, or "citizen-sensors." Earthquake detections are based on user postings in any language and from any region. This approach is unsupervised, adapting automatically to changes in the input data stream, and only requires a general list of keywords for each language. Our method is noise tolerant and simple, providing good results both in terms of precision and recall. This complements prior work that mostly consists of supervised approaches that focus on performing detections in a specific geographical area and are difficult to generalize to a global scope. We demonstrate the effectiveness of this approach by using it within a real-time on-line system, which is publicly available and currently in use at National Seismology Center in Chile and Oceanographic and Hydrological Office of the Chilean Army. The quantitative evaluation of our system, performed during a 9-month period, shows that our solution is competitive to the best state-of-the-art methods. Overall, our findings indicate that our approach is an effective low-cost alternative for earthquake monitoring at a global scale.
Effective Prize Structure for Simple Crowdsourcing Contests with Participation Costs
Authors: Sarne David , Lepioshkin Michael
Published: October 2022

This paper studies the use of a multi-prize compensation scheme for "simple" contests where participation is costly and the quality of participants' contributions is a priori uncertain at the time they make their decision related to participating in the contest. The equilibrium analysis provided enables demonstrating not only that a multi-prize structure is often beneficial but also that in some cases the principal's expected profit is maximized when offering a second prize greater than the first prize. This may seem somehow counter-intuitive especially given that the principal's profit is only influenced by the quality of the best submission rather than the aggregate of submissions. Special emphasis is placed on the case where the contestants are a priori homogeneous which is often the case in real-life, whenever the contestants are basically a priori alike and the quality of their submissions is determined subjectively by some referee. Here, we manage to prove that a multi-prize structure is dominated by a winner-takes-all scheme, suggesting that the benefit in the multi-prize contest scheme fully derives from the heterogeneity between prospective contestants. Finally, we show that there is a class of settings where the use of the multi-prize crowdsourcing contest model enables achieving the performance of the fully cooperative model (which is an upper bound for the performance in any type of contest), and that for settings of this class the optimal prize allocation can be extracted through a set of linear equations.
"But You Promised": Methods to Improve Crowd Engagement In Non-Ground Truth Tasks
Authors: Elmalech Avshalom , Grosz Barbara
Published: October 2022

Crowdsourcing platforms were initially designed to recruit people to perform tasks that were simple cognitively but difficult for computers. One challenge in these settings is to identify an incentive mechanism for motivating workers to complete tasks and do high-quality work. Previous research has studied the use of financial incentive mechanisms and social comparison as motivators. These mechanisms can only be applied to ground truth tasks, tasks for which there is an objective performance scale. In this paper, we define and compare three innovative methods for improving worker engagement on non-ground truth tasks drawing on a psychological theory of commitment. The three methods are similar in asking participants to promise they will complete a task, but they differ in terms of how the commitment is made. In the first method, participants commit by signing a contract; in the second, by listening to a recording; in the third, by recording a personal commitment. The last two methods significantly improved the task completion rate when compared to two baseline conditions. The methods we propose can be implemented simply, can be used for any task, and do not affect participants' behavior other than by improving their engagement.
Crowd-O-Meter: Predicting if a Person Is Vulnerable to Believe Political Claims
Authors: Sameki Mehrnoosh , Zhang Tianyi , Ding Linli , Betke Margrit , Gurari Danna
Published: October 2022

Social media platforms have been criticized for promoting false information during the 2016 U.S. presidential election campaign. Our work is motivated by the idea that a platform could reduce the circulation of false information if it could estimate whether its users are vulnerable to believing political claims. We here explore whether such a vulnerability could be measured in a crowdsourcing setting. We propose Crowd-O-Meter, a framework that automatically predicts if a crowd worker will be consistent in his/her beliefs about political claims; i.e., consistently believes the claims are true or consistently believes the claims are not true. Crowd-O-Meter is a user-centered approach which interprets a combination of cues characterizing the user's implicit and explicit opinion bias. Experiments on 580 quotes from PolitiFact's fact checking corpus of 2016 U.S. presidential candidates show that Crowd-O-Meter is precise and accurate for two news modalities: text and video. Our analysis also reveals which are the most informative cues of a person's vulnerability.
Social Cues, Social Biases: Stereotypes in Annotations on People Images
Authors: Otterbacher Jahna
Published: October 2022

Human computation is often subject to systematic biases. We consider the case of linguistic biases and their consequences for the words that crowdworkers use to describe people images in an annotation task. Social psychologists explain that when describing oth- ers, the subconscious perpetuation of stereotypes is in- evitable, as we describe stereotype-congruent people and/or in-group members more abstractly than others. In an MTurk experiment we show evidence of these bi- ases, which are exacerbated when an image’s “popular tags” are displayed, a common feature used to provide social information to workers. Underscoring recent calls for a deeper examination of the role of training data quality in algorithmic biases, results suggest that it is rather easy to sway human judgment.
Utilizing Crowdsourced Asynchronous Chat for Efficient Collection of Dialogue Dataset
Authors: Ikeda Kazushi , Hoashi Keiichiro
Published: October 2022

In this paper, we design a crowd-powered system to efficiently collect data for training dialogue systems. Conventional systems assign dialogue roles to a pair of crowd workers, and record their interaction on an online chat. In this framework, the pair is required to work simultaneously, and one worker must wait for the other when he/she is writing a message, which decreases work efficiency. Our proposed system allows multiple workers to create dialogues in an asynchronous manner, which relieves workers from time restrictions. We have conducted an experiment using our system on a crowdsourcing platform to evaluate the efficiency and the quality of dialogue collection. Results show that our system can reduce the necessary time to input a message by 68% while maintaining quality.
Permutation-Invariant Consensus over Crowdsourced Labels
Authors: Giancola Michael , Paffenroth Randy , Whitehill Jacob
Published: October 2022

This paper introduces a novel crowdsourcing consensus model and inference algorithm — which we call PICA (Permutation-Invariant Crowdsourcing Aggregation) — that is designed to recover the ground-truth labels of a dataset while being invariant to the class permutations enacted by the different annotators. This is particularly useful for settings in which annotators may have systematic confusions about the meanings of different classes, as well as clustering problems (e.g., dense pixel-wise image segmentation) in which the names/numbers assigned to each cluster have no inherent meaning.The PICA model is constructed by endowing each annotator with a doubly-stochastic matrix (DSM), which models the probabilities that an annotator will perceive one class and transcribe it into another. We conduct simulations and experiments to show the advantage of PICA compared to two baselines (Majority Vote, and an "unpermutation" heuristic) for three different clustering/labeling tasks. We also explore the conditions under which PICA provides better inference accuracy compared to a simpler but related model based on right-stochastic matrices. Finally, we show that PICA can be used to crowdsource responses for dense image segmentation tasks, and provide a proof-of-concept that aggregating responses in this way could improve the accuracy of this labor-intensive task.
How Do Crowdworker Communities and Microtask Markets Influence Each Other? A Data-Driven Study on Amazon Mechanical Turk
Authors: Yang Jie , Van der Valk Carlo , Hoßfeld Tobias , Redi Judith , Bozzon Alessandro
Published: October 2022

Crowdworker online communities — operating in fora like mTurkForum and TurkerNation — are an important actor in microwork markets. Albeit central to market dynamics, how the behavior of crowdworker communities and the dynamics of online marketplaces influence each other is yet to be understood. To provide quantitative evidence of such influence, we performed an analysis on 6-years worth of mTurk market activities and community discussions in six fora. We investigated the nature of the relationships that exist between activities in fora, tasks published in mTurk, requesters for such tasks, and task completion speed. We validate -- and expand upon — results from previous work by showing that (i) there are differences between market demand and community activities that are specific to fora and task types; (ii) the temporal progression of HIT availability in the market is predictive of the upcoming amount of crowdworker discussions, with significant differences across fora and discussion categories; (iii) activities in fora can have a significant positive impact on the completion speed of tasks available in the market.
Visual Question Answer Diversity
Authors: Yang Chun-Ju , Grauman Kristen , Gurari Danna
Published: October 2022

Visual questions (VQs) can lead multiple people to respond with different answers rather than a single, agreed upon response. Moreover, the answers from a crowd can include different numbers of unique answers that arise with different relative frequencies. Such answer diversity arises for a variety of reasons including that VQs are subjective, difficult, or ambiguous. We propose a new problem of predicting the answer distribution that would be observed from a crowd for any given VQ; i.e., the number of unique answers and their relative frequencies. Our experiments confirm that the answer distribution can be predicted accurately for VQs asked by both blind and sighted people. We then propose a novel crowd-powered VQA system that uses the answer distribution predictions to reason about how many answers are needed to capture the diversity of possible human responses. Experiments demonstrate this proposed system accelerates capturing the diversity of answers with considerably less human effort than is required with a state-of-art system.