bluuebunny/crossref_metadata_2025 huggingface dataset:
Ability Grouping of Crowd Workers via Reward Discrimination
Authors: Sakurai Yuko , Okimoto Tenda , Oka Masaaki , Shinoda Masato , Yokoo Makoto
Published: October 2022

We develop a mechanism for setting discriminated reward prices in order to group crowd workers according to their abilities. Generally, a worker has a certain level of confidence in the correctness of her answers, and asking about it is useful for estimating the probability of correctness. However, we need to overcome two main obstacles to utilize confidence for inferring correct answers. One is that a worker is not always well-calibrated. Since she is sometimes over/underconfident, her confidence does not always coincide with the probability of correctness. The other is that she does not always truthfully report her confidence. Thus, we design an indirect mechanism that enables a worker to declare her confidence by choosing a desirable reward plan from the set of plans that correspond to different confidence intervals. Our mechanism ensures that choosing a plan including true confidence maximizes the worker's expected utility. We also propose a method that composes a set of plans that can achieve requester-specified accuracy in estimating the correct answer using a small number of workers. We show our experimental results using Amazon Mechanical Turk.
Automated Support for Collective Memory of Conversational Interactions
Authors: Lasecki Walter , Bigham Jeffrey
Published: October 2022

Maintaining consistency is a difficult challenge in crowd-powered systems in which constituent crowd workers may change over time. We discuss an initial outline for Chorus:Mnemonic, a system that augments the crowd's collective memory of a conversation by automatically recovering past knowledge based on topic, allowing the system to support consistent multi-session interactions. We present the design of the system itself, and discuss methods for testing its effectiveness. Our goal is to provide consistency between long interactions with crowd-powered conversational assistants by using AI to augment crowd workers.
Human Stigmergy in Augmented Environments
Authors: Greene Kshanti , Young Thomas
Published: October 2022

The potential of human computation may be stymied by a curious phenomenon that as the size of a group of collaborators increases, the efficiency of the group typically decreases. The study described in this paper tests our hypothesis that communication mechanisms are key to unlocking collaborative potential. In particular, we sought to evaluate whether a group taking on a complex task can increase its efficacy by communicating through a shared environment.
An Initial Study of Automatic Curb Ramp Detection with Crowdsourced Verification Using Google Street View Images
Authors: Hara Kotaro , Sun Jin , Chazan Jonah , Jacobs David , Froehlich Jon
Published: October 2022

In our previous research, we examined whether minimally trained crowd workers could find, categorize, and assess sidewalk accessibility problems using Google Street View (GSV) images. This poster paper presents a first step towards combining automated methods (e.g., machine vision-based curb ramp detectors) in concert with human computation to improve the overall scalability of our approach.
Joint Crowdsourcing of Multiple Tasks
Authors: Kolobov Andrey , Mausam - , Weld Daniel
Published: October 2022

Allocating tasks to workers so as to get the greatest amount of high-quality output for as little resources as possible is an overarching theme in crowdsourcing research. Among the factors that complicate this problem is the lack of information about the available workers’ skill, along with unknown difficulty of the tasks to be solved. Moreover, if a crowdsourcing platform customer is limited to a fixed-size worker pool to complete a large batch of jobs such as identifying a particular object in a collection of images or comparing the quality of many pairs of artifacts in crowdsourcing workflows, she inevitably faces the tradeoff between getting a few of these tasks done well or getting many done poorly. In this paper, we propose a framework called JOCR (Joint Crowdsourcing, pronounced as “Joker”) for analyzing joint allocations of many tasks to a pool of workers. JOCR encompasses a broad class of common crowdsourcing scenarios, and we pose the challenge of developing efficient algorithms for it.
Understanding Potential MicrotaskWorkers for Paid Crowdsourcing
Authors: Wang Ming-Hung , Chen Kuan-Ta , Wang Shuo-Yang , Lei Chin-Laung
Published: October 2022

More and more people leverage the power of crowds to obtain solutions of their problems, and the number of microtask workers also increases rapidly on paid crowdsourcing marketplaces. However, there is an order of magnitude discrepancy between the population of Internet users (approximately 2 billions) and that of microtask workers (approximately 0.5 millions); we believe that a large number of potential workers are interested in microtasks but have not involved in paid crowdsourcing activities yet. In this paper, we conduct an Internet survey to investigate potential workers regarding their demographics, experiences in microtasks, preferences on tasks and rewards. We hope that our findings from this research are helpful for establishing a prosperous paid crowdsourcing environment.
English to Hindi Translation Protocols for an Enterprise Crowd
Authors: Iyengar Srinivasan , Karande Shirish , Lodha Sachin
Published: October 2022

We present early results on crowd sourcing translations in an enterprise setting. We show that several weak translators can together converge to translation quality higher than individually plausible. We share learning about post editing of translations and the effort perceived by the crowd. A key observation is that a protocol “machine-human-human” that utilizes two-hop post editing can provide an almost expert quality translation. Finally, we discuss methods that can choose the best translation among several candidates.
Preface
Authors: Bigham Jeffrey , Parkes David
Published: October 2022

This conference is an opportunity to build on the success of the First AAAI Human Computation and Crowd- sourcing conference, and to promote the best scholarship in this vibrant and fast emerging, multidisciplinary area. The conference also comes on the heels of four HCOMP workshops, including two workshops hosted at the annual AAAI conference.
Predicting Own Action: Self-Fulfilling Prophecy Induced by Proper Scoring Rules
Authors: Oka Masaaki , Todo Taiki , Sakurai Yuko , Yokoo Makoto
Published: October 2022

This paper studies a mechanism to incentivize agents who predict their own future actions and truthfully declare their predictions. In a crowdsouring setting (e.g., participatory sensing), obtaining an accurate prediction of the actions of workers/agents is valuable for a requester who is collecting real-world information from the crowd. If an agent predicts an external event that she cannot control herself (e.g., tomorrow's weather), any proper scoring rule can give an accurate incentive. In our problem setting, an agent needs to predict her own action (e.g., what time tomorrow she will take a photo of a specific place) that she can control to maximize her utility. Also, her (gross) utility can vary based on an eternal event. We first prove that a mechanism can satisfy our goal if and only if it utilizes a strictly proper scoring rule, assuming that an agent can find an optimal declaration that maximizes her expected utility. This declaration is self-fulfilling; if she acts to maximize her utility, the probabilistic distribution of her action matches her declaration, assuming her prediction about the external event is correct. Furthermore, we develop a heuristic algorithm that efficiently finds a semi-optimal declaration, and show that this declaration is still self-fulfilling. We also examine our heuristic algorithm's performance and describe how an agent acts when she faces an unexpected scenario.
TRACCS: A Framework for Trajectory-Aware Coordinated Urban Crowd-Sourcing
Authors: Chen Cen , Cheng Shih-Fen , Gunawan Aldy , Misra Archan , Dasgupta Koustuv , Chander Deepthi
Published: October 2022

We investigate the problem of large-scale mobile crowd-tasking, where a large pool of citizen crowd-workers are used to perform a variety of location-specific urban logistics tasks. Current approaches to such mobile crowd-tasking are very decentralized: a crowd-tasking platform usually provides each worker a set of available tasks close to the worker's current location; each worker then independently chooses which tasks she wants to accept and perform. In contrast, we propose TRACCS, a more coordinated task assignment approach, where the crowd-tasking platform assigns a sequence of tasks to each worker, taking into account their expected location trajectory over a wider time horizon, as opposed to just instantaneous location. We formulate such task assignment as an optimization problem, that seeks to maximize the total payoff from all assigned tasks, subject to a maximum bound on the detour (from the expected path) that a worker will experience to complete her assigned tasks. We develop credible computationally-efficient heuristics to address this optimization problem (whose exact solution requires solving a complex integer linear program), and show, via simulations with realistic topologies and commuting patterns, that a specific heuristic (called Greedy-ILS) increases the fraction of assigned tasks by more than 20%, and reduces the average detour overhead by more than 60%, compared to the current decentralized approach.
Tracking Human Process Using Crowd Collaboration to Enrich Data
Authors: Pellow David , Eskenazi Maxine
Published: October 2022

A rich source of data that has been largely ignored in crowdsourcing is the processes that humans use to accomplish a task. If we can capture this information, we could model it for automatic processing and use it to better understand the phenomena being modeled. Using crowd collaboration to trace workers’ process produces a rich dataset that can be mined for new insights. We tested this approach on the task of sentence simplification and show the sources and types of additional information that we have obtained.
Adapting Collaborative Filtering to Personalized Audio Production
Authors: Kim Bongjun , Pardo Bryan
Published: October 2022

Recommending media objects to users typically requires users to rate existing media objects so as to understand their preferences. The number of ratings required to produce good suggestions can be reduced through collaborative filtering. Collaborative filtering is more difficult when prior users have not rated the same set of media objects as the current user or each other. In this work, we describe an approach to applying prior user data in a way that does not require users to rate the same media objects and that does not require imputation (estimation) of prior user ratings of objects they have not rated. This approach is applied to the problem of finding good equalizer settings for music audio and is shown to greatly reduce the number of ratings the current user must make to find a good equalization setting.
Quality Control for Crowdsourced Enumeration Tasks
Authors: Kajimura Shunsuke , Baba Yukino , Kajino Hiroshi , Kashima Hisashi
Published: October 2022

Quality control is one of the central issues in crowdsourcing research. In this paper, we consider a quality control problem of crowdsourced enumeration tasks that request workers to enumerate possible answers as many as possible. Since workers neither necessarily provide correct answers nor provide exactly the same answers even if the answers indicate the same idea, we propose a two-stage quality control method consisting of the answer clustering stage and the reliability estimation stage.
A Markov Decision Process Framework for Predictable Job Completion Times on Crowdsourcing Platforms
Authors: Lakshminarayanan Chandrashekar , Dubey Ayush , Bhatnagar Shalabh , Balamurugan Chithralekha
Published: October 2022

Task starvation leads to huge variation in the completion times of the tasks posted on to the crowd. The price offered to a given task together with the dynamics of the crowd at the time of posting affect its completion time. Large organizations/requesters who frequent the crowd at regular intervals in order to get their tasks done desire predictability in completion times of the tasks. Thus, such requesters have to take into account the crowd dynamics at the time of posting the tasks and price them accordingly. In this work, we study an instance of the pricing problem and propose a solution based on the framework of Markov Decision Processes (MDPs).
Adaptive Performance Optimization over Crowd Labor Channels
Authors: Karanam Saraschandra , Chander Deepthi , Celis Elisa , Dasgupta Koustuv , Rajan Vaibhav
Published: October 2022

We describe a system which monitors the performance of labor channels within a crowdsourcing platform in an online manner. This allows us to automatically determine if and when to switch between labor channels in order to improve overall performance of crowd tasks.
Crowd-Aware Space Monitoring by Crowdsourcing a Micro QA Task
Authors: Huang Yi-Ching , Hsu Jane
Published: October 2022

Effective space monitoring helps improve utilization of seminar rooms in the work place. This paper presents CrowdButtons as a simple solution to real-time space monitoring by crowdsourcing. CrowdButtons are tangible reporting devices designed to collect room status data from passersby. Anyone may participate in the micro QA task instantly by clicking a button, while the dashboard displays real-time room status as well as visualization of space usage patterns over time.
Optimal Worker Quality and Answer Estimates in Crowd-Powered Filtering and Rating
Authors: Das Sarma Akash , Parameswaran Aditya , Widom Jennifer
Published: October 2022

We consider the problem of optimally filtering (or rating) a set of items based on predicates (or scoring) requiring human evaluation. Filtering and rating are ubiquitous problems across crowdsourcing applications. We consider the setting where we are given a set of items and a set of worker responses for each item: yes/no in the case of filtering and an integer value in the case of rating. We assume that items have a true inherent value that is unknown, and workers draw their responses from a common, but hidden, error distribution. Our goal is to simultaneously assign a ground truth to the item-set and estimate the worker error distribution. Previous work in this area has focused on heuristics such as Expectation Maximization (EM), providing only a local optima guarantee, while we have developed a general framework that finds a maximum likelihood solution. Our approach extends to a number of variations on the filtering and rating problems.
Poetry of the Crowd: A Human Computation Algorithm to Convert Prose into Rhyming Verse
Authors: Chen Quanze , Lei Chenyang , Xu Wei , Pavlick Ellie , Callison-Burch Chris
Published: October 2022

Poetry composition is a very complex task that requires a poet to satisfy multiple constraints concurrently. We believe that the task can be augmented by combining the creative abilities of humans with computational algorithms that efficiently constrain and permute available choices. We present a hybrid method for generating poetry from prose that combines crowdsourcing with natural language processing (NLP) machinery. We test the ability of crowd workers to accomplish the technically challenging and creative task of composing poems.
AI-MIX: Using Automated Planning to Steer Human Workers Towards Better Crowdsourced Plans
Authors: Manikonda Lydia , Chakraborti Tathagata , De Sushovan , Talamadupula Kartik , Kambhampati Subbarao
Published: October 2022

Human computation applications that involve planning and scheduling are gaining popularity, and the existing literature on such systems shows that any automated oversight on human contributors improves the effectiveness of the crowd. In this paper, we present our ongoing work on the AI-MIX system, which is a first step towards using an automated planning and scheduling system in a crowdsourced planning application. In order to address the mismatch between the capabilities of the crowd and the automated planner, we identify two major challenges -- interpretation, and steering. We also present preliminary empirical results over the tour planning domain, and show how using an automated planner can help improve the quality of plans.
Crowd4U: An Initiative for Constructing an Open Academic Crowdsourcing Network
Authors: Morishima Atsuyuki , Amer-Yahia Sihem , Roy Senjuti
Published: October 2022

We describe the Crowd4U initiative, which aims at constructing an all-academic open and generic platform for microvolunteering and crowdsourcing worldwide. Crowd4U provides a microtask-based platform in which most workers are volunteers at universities and other research institutions. Crowd4U is open in the sense that the platform can interact with other platforms, researchers can register their tasks, and the underlying code is not a black box. It is generic as it allows to register virtually any task. Crowd4U has already been used by several projects for public and academic purposes.
Viability of Crowd-Volunteered Open Research Reviews
Authors: Bhattacharyya Malay
Published: October 2022

In this study, we examine the feasibility of reviewing research publications through crowdsourcing. We propose a post-publication review model where the research materials are posted online for the crowd volunteers to review. One such platform, named PuGLi, is prepared and the data collected from it is analyzed. The analysis raises some interesting questions related to the challenges of attracting crowd volunteers, determining truthfulness of opinions, and obtaining the final judgment from multiple opinions received from the crowd.
Measuring Impact of Local Community Initiatives: A Crowdsourcing Approach
Authors: López Moncada Claudia , Farzan Rosta , Clift Steven
Published: October 2022

Over the last decade, a number of initiatives have been investigating the application of information technologies to increase civic engagement in local communities. An important challenge faced by many of these initiatives is discovering ways to reliably measure their impact. We propose a panel during the Citizen+X workshop to promote discussion on engaging the crowd as sensors of impact for these local initiatives.
Naturalistic Distributed Experimentation as a Source of New Insight
Authors: Gould Sandy , Cox Anna , Brumby Duncan
Published: October 2022

Human performance experiments are often conducted online with the help of paid crowdworkers and citizen scientists. This approach produces reliable data, but there are concerns that the inevitable loss of control that accompanies online experimentation might confound results. Researchers have therefore spent time considering how to regain control and mitigate the effects of confounds. In this abstract we argue that confounding factors in online work can be put to novel use, giving us insight into research questions we might otherwise be unable to answer.
Connecting Diverse Communities Through Citizen Science Data Infrastructure
Authors: Hammock Jen , Wiggins Andrea
Published: October 2022

The BioCubes project is a citizen science biodiversity observation and reporting protocol designed for group participation in classroom and non-formal education settings. A two-year study will develop a proof-of-concept information technology infrastructure to support distribution and use of data from BioCubes across the US. The goal of this work is promoting novel STEM learning opportunities for diverse student populations while demonstrating the feasibility of exchanging and integrating citizen science data with ongoing biodiversity research, a key step in advancing citizen science. In this abstract, we briefly introduce key features of our project and identify several considerations for the design of the sociotechnical systems to support both data contributors and data users.
Defining, Designing, and Evaluating Civic Values in Human Computationand Collective Action Systems
Authors: Matias J. , Geiger Stuart
Published: October 2022

Collective action is often described in terms of the relationships, learning, principled processes, and community capacities it fosters. Despite this, human computation and collective action systems are often designed and evaluated with system outputs in mind: the quality of answers, the number of votes, the accuracy of content created. In this proposal, we review literature on the design values of "citizen-x” systems, put forward a series of models for describing the civic values in “citizen-x," and classify systems by those models. We conclude by urging greater attentiveness to civic values when designing and evaluating human computation and collective action systems.
A Crowdsourcing Method for Obtaining Rephrased Questions
Authors: Shimizu Nobuyuki , Morishima Atsuyuki , Hayashi Ryota
Published: October 2022

We propose a method for obtaining and ranking paraphrased questions from crowds to be used as a part of instructions in microtask-based crowdsourcing. With our method, we are able to obtain questions that differ in expression yet have the same semantics with respect to the crowdsourcing task. This is done by generating tasks that give hints and elicit instructions from workers. We conducted experiments with data used for a real set of gold standard questions submitted to a commercial crowdsourcing platform and compared the results with those from a direct-rewrite method.
The ActiveCrowdToolkit: An Open-Source Tool for Benchmarking Active Learning Algorithms for Crowdsourcing Research
Authors: Venanzi Matteo , Parson Oliver , Rogers Alex , Jennings Nick
Published: October 2022

We present an open-source toolkit that allows the easy comparison of the performance of active learning methods over a series of datasets. The toolkit allows such strategies to be constructed by combining a judgement aggregation model, task selection method and worker selection method.The toolkit also provides a user interface which allows researchers to gain insight into worker performance and task classification at runtime.
Modeling and Exploration of Crowdsourcing Micro-Tasks Execution
Authors: Kucherbaev Pavel , Daniel Florian , Tranquillini Stefano , Marchese Maurizio
Published: October 2022

There are two key issues which are critical for a person crowdsourcing work: quality control and timeliness. While the first topic is extensively covered in the literature, the timeliness is still not very well analyzed. In our past experiments on CrowdFlower we noticed that 90% of task instances were completed in only 10 minutes, the rest 10% could take hours to be completed. There were also extra peaks in performance on the 30th and 60th minutes since the task publication. From the observation of these patterns we wanted to understand better how the crowd workers execute tasks, to try to describe this behavior mathematically and to identify ways to speed up the tasks execution.
Using Expertise for Crowd-Sourcing
Authors: Merritt David , Ackerman Mark , Newman Mark , Hung Pei-Yao , Mandel Jacob , Ackerman Erica
Published: October 2022

In this paper, we examine whether the use of expertise ratings can help crowd-sourcing systems. We show, using simulations, that a crowd-sourcing system based in social navigation works better when users’ expertise levels are taken into account.
MicroTalk: Using Argumentation to Improve Crowdsourcing Accuracy
Authors: Drapeau Ryan , Chilton Lydia , Bragg Jonathan , Weld Daniel
Published: October 2022

Crowd workers are human and thus sometimes make mistakes. In order to ensure the highest quality output, requesters often issue redundant jobs with gold test questions and sophisticated aggregation mechanisms based on expectation maximization (EM). While these methods yield accurate results in many cases, they fail on extremely difficult problems with local minima, such as situations where the majority of workers get the answer wrong. Indeed, this has caused some researchers to conclude that on some tasks crowdsourcing can never achieve high accuracies, no matter how many workers are involved. This paper presents a new quality-control workflow, called MicroTalk, that requires some workers to Justify their reasoning and asks others to Reconsider their decisions after reading counter-arguments from workers with opposing views. Experiments on a challenging NLP annotation task with workers from Amazon Mechanical Turk show that (1) argumentation improves the accuracy of individual workers by 20%, (2) restricting consideration to workers with complex explanations improves accuracy even more, and (3) our complete MicroTalk aggregation workflow produces much higher accuracy than simpler voting approaches for a range of budgets.
Leveraging the Contributions of the Casual Majority to Identify Appealing Web Content
Authors: Hogg Tad , Lerman Kristina
Published: October 2022

Users of peer production web sites differ greatly in their activity levels.A small minority are engaged contributors, while the vast majority are only casual surfers. The casual users devote little effort to evaluating the site's content and many of them visit the site only once. This churn poses a challenge for sites attempting to gauge user interest in their content. The challenge is especially severe for sites focusing on content with subjective quality, including movies, music, restaurants and items in other cultural markets. A key question is whether content evaluation should use opinions of all users or only the minority who devote significant effort to reviewing content? Using Amazon Mechanical Turk, we experimentally address this question by comparing outcomes for these two approaches. We find that the larger numbers of less informed users more than offset their noisy signals on content quality to provide rapid evaluation. However, such users are systematically biased, and the speed of their assessments comes at the expense of limited collective accuracy.
Predicting Crowd Work Quality under Monetary Interventions
Authors: Yin Ming , Chen Yiling
Published: October 2022

Work quality in crowdsourcing task sessions can change over time due to both internal factors, such as learning and boredom, and external factors like the provision of monetary interventions. Prior studies on crowd work quality have focused on characterizing the temporal behavior pattern as a result of the internal factors. In this paper, we propose to explicitly take the impact of external factors into consideration for modeling crowd work quality. We present a series of seven models from three categories (supervised learning models, autoregressive models and Markov models) and conduct an empirical comparison on how well these models can predict crowd work quality under monetary interventions on three datasets that are collected from Amazon Mechanical Turk. Our results show that all these models outperform the baseline models that don’t consider the impact of monetary interventions. Our empirical comparison further identifies the random forests model as an excellent model to use in practice as it consistently provides accurate predictions with high confidence across different datasets, and it also demonstrates robustness against limited training data and limited access to the ground truth.
Evaluating Task-Dependent Taxonomies for Navigation
Authors: Sun Yuyin , Singla Adish , Yan Tori , Krause Andreas , Fox Dieter
Published: October 2022

Taxonomies of concepts are important across many application domains, for instance, online shopping portals use catalogs to help users navigate and search for products. Task-dependent taxonomies, e.g., adapting the taxonomy to a specific cohort of users, can greatly improve the effectiveness of navigation and search. However, taxonomies are usually created by domain experts and hence designing task-dependent taxonomies can be an expensive process: this often limits the applications to deploy generic taxonomies. Crowdsourcing-based techniques have the potential to provide a cost-efficient solution to building task-dependent taxonomies. In this paper, we present the first quantitative study to evaluate the effectiveness of these crowdsourcing based techniques. Our experimental study compares different task-dependent taxonomies built via crowdsourcing and generic taxonomies built by experts. We design randomized behavioral experiments on the Amazon Mechanical Turk platform for navigation tasks using these taxonomies resembling real-world applications such as product search. We record various metrics such as the time of navigation, the number of clicks performed, and the search path taken by a participant to navigate the taxonomy to locate a desired object. Our findings show that task-dependent taxonomies built by crowdsourcing techniques can reduce the navigation time up to $20\%$. Our results, in turn,demonstrate the power of crowdsourcing for learning complex structures such as semantic taxonomies.
Much Ado About Time: Exhaustive Annotation of Temporal Data
Authors: Sigurdsson Gunnar , Russakovsky Olga , Farhadi Ali , Laptev Ivan , Gupta Abhinav
Published: October 2022

Large-scale annotated datasets allow AI systems to learn from and build upon the knowledge of the crowd. Many crowdsourcing techniques have been developed for collecting image annotations. These techniques often implicitly rely on the fact that a new input image takes a negligible amount of time to perceive. In contrast, we investigate and determine the most cost-effective way of obtaining high-quality multi-label annotations for temporal data such as videos. Watching even a short 30-second video clip requires a significant time investment from a crowd worker; thus, requesting multiple annotations following a single viewing is an important cost-saving strategy. But how many questions should we ask per video? We conclude that the optimal strategy is to ask as many questions as possible in a HIT (up to 52 binary questions after watching a 30-second video clip in our experiments).We demonstrate that while workers may not correctly answer all questions, the cost-benefit analysis nevertheless favors consensus from multiple such cheap-yet-imperfect iterations over more complex alternatives. When compared with a one-question-per-video baseline, our method is able to achieve a 10% improvement in recall (76.7% ours versus 66.7% baseline) at comparable precision (83.8% ours versus 83.0% baseline) in about half the annotation time (3.8 minutes ours compared to 7.1 minutes baseline). We demonstrate the effectiveness of our method by collecting multi-label annotations of 157 human activities on 1,815 videos.
Cioppino: Multi-Tenant Crowd Management
Authors: Haas Daniel , Franklin Michael
Published: October 2022

Embedding human computation in systems for data analysis improves the quality of the analysis, but can significantly impact the end-to-end cost and performance of the system. Recent work in crowdsourcing systems attempts to optimize for performance, but focuses on single applications running homogeneous tasks. In this work, we introduce Cioppino, a system that accounts for human factors that can affect performance when running multiple applications in parallel. Cioppino uses a queueing model to represent the pool of workers, and leverages techniques for autoscaling used in cloud computing to adaptively adjust the pool size. Its model also accounts for worker abandonment, and automatically shifts workers between applications to improve performance and match workers with tasks they enjoy most. Our evaluation of Cioppino in simulation on traces extracted from a realtime crowd system running on Amazon’s Mechanical Turk demonstrates a 19X reduction in cost, a 20% increase in throughput, and a 2X increase in worker preference for assigned tasks as compared to state-of-the-art crowd management strategies.
Towards Quantifying Behaviour in Social Crowdsourcing Communities
Authors: Zaamout Khobaib , Barker Ken
Published: October 2022

We analyze crowdsourcing communities by creating a detailed process for quantifying individual behaviour in online environments. The key feature of our communities is their social interactions so we call these social crowdsourcing communities (SCC). First, we derive factors based on actions captured about textual contributions. We interpret and name these factors. Then we demonstrate their utility in predicting the quality of team contributions. We capture the actions of members using measurable variables and perform factor analysis on these to produce factors of behaviour in SCCs (i.e. dimensions of behaviour). We derive factor scores for each member. An abstract notion of teams is used that is based on the social interactions. Team scores are then determined by the aggregation of the individual factor scores. The relationship between the team-level factor scores and the quality of contributions made by each team are then used as a proxy for team effectiveness. We found that member behaviour has three dimensions/factors: Impact, Activity, Policing/Rowdiness and there is a linear relationship between a team's contribution quality and their Impact scores. We also found a moderate negative linear relationship between the smallest Activity scores in each team with the quality of their individual contributions. This shows that teams that produce higher quality contributions tend to have higher total and maximum Impact score with lower levels of Activity. Thus, we demonstrate that properly aggregated behavioural factors can predict the quality of team-level contributions.
Your Behavior Signals Your Reliability: Modeling Crowd Behavioral Traces to Ensure Quality Relevance Annotations
Authors: Goyal Tanya , McDonnell Tyler , Kutlu Mucahid , Elsayed Tamer , Lease Matthew
Published: October 2022

While peer-agreement and gold checks are well-established methods for ensuring quality in crowdsourced data collection, we explore a relatively new direction for quality control: estimating work quality directly from workers’ behavioral traces collected during annotation. We propose three behavior-based models to predict label correctness and worker accuracy, then further apply model predictions to label aggregation and optimization of label collection. As part of this work, we collect and share a new Mechanical Turk dataset of behavioral signals judging the relevance of search results. Results show that behavioral data can be effectively used to predict work quality, which could be especially useful with single labeling or in a cold start scenario in which individuals’ prior work history is unavailable. We further show improvement in label aggregation and reducing labeling cost while ensuring data quality.
WingIt: Efficient Refinement of Unclear Task Instructions
Authors: Manam V. K. , Quinn Alexander
Published: October 2022

Crowdsourcing has the inherent potential to shift tedious work from requesters with limited time (or patience) to an on-demand workforce. However, the time saved by requesters is offset by the time they must spend preparing instructions and refining them to address the ambiguities that typically arise. Hastily written instructions should be welcomed and supported. This paper presents a set of methods that enable workers to cope with unclear or ambiguous instructions and produce high quality results with minimal reliance on the requester. Workers’ intuition about the requester’s needs is leveraged to move onus for answering questions and revising instructions to workers. Our system, WingIt, implements these methods and demonstrates the relative tradeoffs between resolving by questions versus editing instructions directly, and between waiting for immediate response from the requester versus allowing workers to proceed based on their best guess of the requester’s intent.
Understanding the Impact of Text Highlighting in Crowdsourcing Tasks
Authors: Ramírez Jorge , Baez Marcos , Casati Fabio , Benatallah Boualem
Published: October 2022

Text classification is one of the most common goals of machine learning (ML) projects, and also one of the most frequent human intelligence tasks in crowdsourcing platforms. ML has mixed success in such tasks depending on the nature of the problem, while crowd-based classification has proven to be surprisingly effective, but can be expensive. Recently, hybrid text classification algorithms, combining human computation and machine learning, have been proposed to improve accuracy and reduce costs. One way to do so is to have ML highlight or emphasize portions of text that it believes to be more relevant to the decision. Humans can then rely only on this text or read the entire text if the highlighted information is insufficient. In this paper, we investigate if and under what conditions highlighting selected parts of the text can (or cannot) improve classification cost and/or accuracy, and in general how it affects the process and outcome of the human intelligence tasks. We study this through a series of crowdsourcing experiments running over different datasets and with task designs imposing different cognitive demands. Our findings suggest that highlighting is effective in reducing classification effort but does not improve accuracy - and in fact, low-quality highlighting can decrease it.
Second Opinion: Supporting Last-Mile Person Identification with Crowdsourcing and Face Recognition
Authors: Mohanty Vikram , Abdol-Hamid Kareem , Ebersohl Courtney , Luther Kurt
Published: October 2022

As AI-based face recognition technologies are increasingly adopted for high-stakes applications like locating suspected criminals, public concerns about the accuracy of these technologies have grown as well. These technologies often present a human expert with a shortlist of high-confidence candidate faces from which the expert must select correct match(es) while avoiding false positives, which we term the “last-mile problem.” We propose Second Opinion, a web-based software tool that employs a novel crowdsourcing workflow inspired by cognitive psychology, seed-gather-analyze, to assist experts in solving the last-mile problem. We evaluated Second Opinion with a mixed-methods lab study involving 10 experts and 300 crowd workers who collaborate to identify people in historical photos. We found that crowds can eliminate 75% of false positives from the highest-confidence candidates suggested by face recognition, and that experts were enthusiastic about using Second Opinion in their work. We also discuss broader implications for crowd–AI interaction and crowdsourced person identification.
Understanding the Effects of Explanation Types and User Motivations on Recommender System Use
Authors: Li Qing , Chu Sharon , Rao Nanjie , Nourani Mahsan
Published: October 2022

It is becoming increasingly common for intelligent systems, such as recommender systems, to provide explanations for their generated recommendations to the users. However, we still do not have a good understanding of what types of explanations work and what factors affect the effectiveness of different types of explanations. Our work focuses on explanations for movie recommender systems. This paper presents a mixed study where we hypothesize that the type of explanation, as well as user motivation for watching movies, will affect how users respond to recommendation system explanations. Our study compares three types of explanations: i) neighbor-ratings, ii) profile-based, and iii) event-based, as well as three types of user movie-watching motivations: i) hedonic (fun and relaxation), ii) eudaimonic (inspiration and meaningfulness), and iii) educational (learning new content). We discuss the implications of the study results for the design of explanations for movie recommender systems, and future novel research directions that the study results uncover.
Motivating Novice Crowd Workers through Goal Setting: An Investigation into the Effects on Complex Crowdsourcing Task Training
Authors: Rechkemmer Amy , Yin Ming
Published: October 2022

Training workers within a task is one way of enabling novice workers, who may lack domain knowledge or experience, to work on complex crowdsourcing tasks. Based on goal setting theory in psychology, we conduct a randomized experiment to study whether and how setting different goals—including performance goal, learning goal, and behavioral goal—when training workers for a complex crowdsourcing task affects workers’ learning perception, learning gain, and post-training performance. We find that setting different goals during training significantly affects workers’ learning perception, but overall does not have an effect on learning gain or post-training performance. However, higher levels of learning gain can be obtained when setting learning goals for workers who are highly learning-oriented. Additionally, giving workers a challenging behavioral goal can nudge them to adopt desirable behavior meant to improve learning and performance, though the adoption of such behavior does not lead to as much improvement as when the worker decides to take part in the behavior themselves. We conclude by discussing the lessons we’ve learned on how to effectively utilize goals in complex crowdsourcing task training.
The Challenges of Crowd Workers in Rural and Urban America
Authors: Flores-Saviaga Claudia , Li Yuwen , Hanrahan Benjamin , Bigham Jeffrey , Savage Saiph
Published: October 2022

Crowd work has the potential of helping the financial recovery of regions traditionally plagued by a lack of economic opportunities, e.g., rural areas. However, we currently have limited information about the challenges facing crowd workers from rural and super rural areas as they struggle to make a living through crowd work sites. This paper examines the challenges and advantages of rural and super rural Amazon Mechanical Turk (MTurk) crowd workers and contrasts them with those of workers from urban areas. Based on a survey of 421 crowd workers from differing geographic regions in the U.S., we identified how across regions, people struggled with being onboarded into crowd work. We uncovered that despite the inequalities and barriers, rural workers tended to be striving more in micro-tasking than their urban counterparts. We also identified cultural traits, relating to time dimension and individualism, that offer us an insight into crowd workers and the necessary qualities for them to succeed on gig platforms. We finish by providing design implications based on our findings to create more inclusive crowd work platforms and tools.
How Useful Are the Machine-Generated Interpretations to General Users? A Human Evaluation on Guessing the Incorrectly Predicted Labels
Authors: Shen Hua , Huang Ting-Hao
Published: October 2022

Explaining to users why automated systems make certain mistakes is important and challenging. Researchers have proposed ways to automatically produce interpretations for deep neural network models. However, it is unclear how useful these interpretations are in helping users figure out why they are getting an error. If an interpretation effectively explains to users how the underlying deep neural network model works, people who were presented with the interpretation should be better at predicting the model’s outputs than those who were not. This paper presents an investigation on whether or not showing machine-generated visual interpretations helps users understand the incorrectly predicted labels produced by image classifiers. We showed the images and the correct labels to 150 online crowd workers and asked them to select the incorrectly predicted labels with or without showing them the machine-generated visual interpretations. The results demonstrated that displaying the visual interpretations did not increase, but rather decreased, the average guessing accuracy by roughly 10%.
Making Time Fly: Using Fillers to Improve Perceived Latency in Crowd-Powered Conversational Systems
Authors: Abbas Tahir , Gadiraju Ujwal , Khan Vassilis-Javed , Markopoulos Panos
Published: October 2022

Crowd-Powered Conversational Systems (CPCS) are gaining traction due to their potential utility in a range of application fields where automated conversational interfaces are still inadequate. Currently, long response times negatively impact CPCSs, limiting their potential application as conversational partners. Related research has focused on developing algorithms for swiftly hiring workers and synchronous crowd coordination techniques to ensure high-quality work. Evaluation studies typically concern system reaction times and performance measurements, but have so far not examined the effects of extended wait times on users. The goal of this study, based on time perception models, is to explore how effective different time fillers
are at reducing the negative impacts of waiting in CPCSs. To this end, we conducted a rigorous simulation-based between-subjects (N = 930) study on the Prolific crowdsourcing platform to assess the influence of different filler types across three levels of delay (8, 16 & 32s) for Information Retrieval (IR) and stress management tasks. Our results show that asking users to perform secondary tasks (e.g., microtasks or breathing exercises) while waiting for longer periods of time helped divert their attention away from timekeeping, increased their engagement, and resulted in shorter perceived waiting times. For shorter delays, conversational fillers generated more intense immersion and contributed to shorten the perception of time.
Modeling Simultaneous Preferences for Age, Gender, Race, and Professional Profiles in Government-Expense Spending: A Conjoint Analysis
Authors: Ibrahim Lujain , Ghassemi Mohammad M. , Alhanai Tuka
Published: October 2022

Bias can have devastating outcomes on everyday life, and may manifest in subtle preferences for particular attributes (age, gender, ethnicity, profession). Understanding bias is complex, but first requires identifying the variety and interplay of individual preferences. In this study, we deployed a sociotechnical, web-based human-subject experiment to quantify individual preferences in the context of selecting an advisor to successfully pitch a government-expense. We utilized conjoint analysis to rank the preferences of 722 U.S. based subjects, and observed that their ideal advisor was White, middle-aged, and of either a government or STEM-related profession (0.68 AUROC, p < 0.05). The results motivate the simultaneous measurement of preferences as a strategy to offset preferences that may yield negative consequences (e.g. prejudice, disenfranchisement) in contexts where social interests are being represented.
Beauty Both Ubiquitous and Exclusive: Emmeline Stuart-Wortley and the Great Exhibition
Authors: Brown Daniel
Published: October 2022

This brief article focuses upon Lady Emmeline Stuart-Wortley’s response to the Great Exhibition of 1851, both her celebration of luxury goods in her poetry and her poetry as such a consumer good.
The Material and the Divine: Entwined Aesthetics in Alice Meynell’s ‘The Two Poets’
Authors: Scholl Lesa
Published: October 2022

Alice Meynell’s adherence to a reserved lyrical mode could be read as a desire to maintain a sense of teleological order, in which the ironic timelessness of the lyrical moment reinforces the divine purpose of the material. However, the silences and reserve for which Meynell’s work is known instead reinforce paradoxical moments of trauma that are held up and unspoken as impossible to understand within the human experience. The intensity of the lyric depends on the visceral yet disturbingly silent recognition of an aesthetic predicated on violence. From this perspective, the poem becomes a theological and aesthetic space in which a transformative encounter takes place, one that is predicated on human contact that confronts each subject’s understanding of the world and their position within it. Importantly, Meynell’s poetry, like her theology, rests on paradoxes — of theology and aesthetics; of the material and the divine; of the temporal and the eternal; of violence and beauty — in such a way that creates mutual, dynamic spaces rather than a binary structure that defines and divides.