Designing for Coordinated Action : Challenges and Opportunities of Working with Experts

Expert investigators bring advanced skills and deep experience to analyze visual evidence, but they face limits on their time and attention. In contrast, crowds of novices can be highly scalable and parallelizable, but lack expertise. Here, we discuss a real-world deployment of a system we built, GroundTruth, that enables experts to direct crowds in determining the exact geographic location where an image was taken, a key step in verifying visual evidence. We discuss the challenges working with these experts and the methods we have employed to study our software deployment. ACM Reference Format: Sukrit Venkatagiri, Jacob Thebault-Spieker, and Kurt Luther. 2019. Designing for Coordinated Action: Challenges and Opportunities of Working with Experts. CSCW 2019 Workshop on Mapping the “How” of Collaborative Action. Austin, TX, USA, 5 pages. CSCW 2019 Workshop on Mapping the “How” of Collaborative Action, 2019 © Copyright held by the owner/author(s). Designing for Coordinated Action CSCW 2019 Workshop on Mapping the “How” of Collaborative Action, 2019 Figure 1: Expert interface where expert specifies locations for the crowd to search, view crowd feedback, and conduct their own parallel search. Figure 2: Crowd worker interface where a crowd worker can provide feedback on whether the aerial diagram matches the cell of satellite imagery or not. INTRODUCTION Expert investigators, such as journalists and human rights advocates, increasingly leverage social media in their investigations as sources of visual evidence as well as volunteers. For example, Bellingcat [2], a community of citizen journalists, uses imagery from social media to investigate claims made by and about governmental or terrorist activity. They often ask their followers on social media for help. A key step in investigations is image geolocation [1, 4], which involves finding the exact location where photo or video imagery was taken. Initially, an expert inspects the image for clues, and then uses inference and experience to conduct a manual, brute-force search through large swathes of satellite imagery, looking for the location in the image. However, this process is time-consuming and does not scale easily. Although experts leverage online crowds in their investigations, these crowds lack expertise and their efforts may be uncoordinated, leading to wasted effort. Undirected, crowdsourced investigations can also lead to vigilantism [7] and misidentification [5]. PRIORWORK: SYSTEM DEVELOPMENT AND EVALUATION Over 18 months, we employed a user-centered design process to build GroundTruth [8], a software tool that allows experts to coordinate the efforts of novice crowds to geolocate images. We first developed a deep understanding of expert investigative work practice by studying existing literature and interviewing 12 experts, followed by sketching, prototyping, and empirical evaluations. GroundTruth enables experts to efficiently coordinate the efforts of crowds in three ways. First, it leverages the existing expert practice of creating a diagram to represent the image under investigation, highlighting for the crowd what to search for. This improves crowd performance by augmenting their spatial reasoning skills. Second, the expert uses deep domain knowledge to narrow down a set of possible locations for the crowd to search through. This better directs crowd worker effort. The system then generates microtasks for individual crowdworkers to search satellite imagery in parallel and at scale, reducing the time an expert must spend. Finally, the system aggregates and displays crowd feedback as a heatmap for the expert to review. It both visualizes the crowd’s progress and allows an expert to keep track of where they have searched. In a lab-based evaluation, we provided 11 experts with a diverse set of images to choose from and geolocate with crowds in real-time. We displayed images in a gallery to cater to experts’ varied backgrounds and preferences. We found that GroundTruth effectively combined the deep knowledge of experts with the speed and scale of crowds, narrowing a search area by up to 67%. Experts also said that GroundTruth facilitated the coordination of crowd feedback, allowing experts to better allocate their own time and effort. We also observed new collaboration dynamics where experts would choose to review crowd feedback, instead of conducting an independent search, or divert their search to locations where the crowd had not yet provided feedback. Designing for Coordinated Action CSCW 2019 Workshop on Mapping the “How” of Collaborative Action, 2019 Although we have learned much from our lab study, the controlled setting presented some limitations. The images we provided were not representative of what they typically encounter. We also recruited paid crowds on AmazonMechanical Turk, but some experts said that they would prefer to use GroundTruth with colleagues or social media followers acting as the crowd, or on their own. Experts also faced time constraints, meaning that we could not observe how they would use GroundTruth within a larger investigation. These limitations led us to where we are now: a real-world deployment. Table 1: Expert participants. CURRENTWORK: REAL-WORLD DEPLOYMENT Our motivations for conducting a real-world deployment are three-fold and involve developing a deep understanding of in-situ processes and contexts of use [6]. First, we wish to understand how experts with diverse skills and expertise will use GroundTruth to geolocate real-world images as part of their work practice. Second, we seek to understand how experts will work with volunteer crowds. Third, we see GroundTruth as a technology probe [3] to help experts to consider other parts of their investigative process that could benefit from crowdsourced support, eliciting needs and desires for the design of future tools. To date, we have deployed GroundTruth with three experts in three different organizations—Christiaan, Dakota, and Nick—who are currently integrating it into their workflows and using it for real-world investigations (Table 1).