National Library of MedicineArchive-It Partner Since: Mar, 2009Organization Type: National InstitutionsOrganization URL: http://www.nlm.nih.gov The National Library of Medicine (NLM), the world's largest medical library and a component of the National Institutes of Health, collects, preserves, and makes available to the public information about health, medicine, and the biomedical sciences. To continue fulfilling this mission, NLM is collecting and archiving related Web content, which also serves to document the histories of health and medicine. ||||| Abstract Massively parallel collaboration and emergent knowledge generation is described through a large scale survey for archaeological anomalies within ultra-high resolution earth-sensing satellite imagery. Over 10K online volunteers contributed 30K hours (3.4 years), examined 6,000 km2, and generated 2.3 million feature categorizations. Motivated by the search for Genghis Khan's tomb, participants were tasked with finding an archaeological enigma that lacks any historical description of its potential visual appearance. Without a pre-existing reference for validation we turn towards consensus, defined by kernel density estimation, to pool human perception for “out of the ordinary” features across a vast landscape. This consensus served as the training mechanism within a self-evolving feedback loop between a participant and the crowd, essential driving a collective reasoning engine for anomaly detection. The resulting map led a National Geographic expedition to confirm 55 archaeological sites across a vast landscape. A increased ground-truthed accuracy was observed in those participants exposed to the peer feedback loop over those whom worked in isolation, suggesting collective reasoning can emerge within networked groups to outperform the aggregate independent ability of individuals to define the unknown. 
 
 Citation: Lin AY-M, Huynh A, Lanckriet G, Barrington L (2014) Crowdsourcing the Unknown: The Satellite Search for Genghis Khan. PLoS ONE 9(12): e114046. doi:10.1371/journal.pone.0114046 Editor: Michael D. Petraglia, University of Oxford, United Kingdom Received: April 22, 2014; Accepted: September 16, 2014; Published: December 30, 2014 Copyright: © 2014 Lin et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Data Availability: The authors confirm that all data underlying the findings are fully available without restriction. All relevant data have been uploaded to the Harvard Dataverse and can be accessed at https://thedata.harvard.edu/dvn/dv/votk. Funding: This work was funded by National Science Foundation (EAGER ISS-1145291 and HCC IIS-1219138). The funder had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. Competing interests: The authors have declared that no competing interests exist. 
 
 Introduction Ultra-high resolution satellite imaging enables a new paradigm in global exploration. This study surveys sub-meter resolution satellite imagery of the Mongolian steppe to identify largely undocumented cultural heritage sites across a sparsely populated and undeveloped landscape. With continued advances in sensor technologies, the capabilities and limitations of remote sensing is being determined less by data resolution and more by the methods that analyze the increasingly massive datasets. Overwhelming data volumes have often led to automated analytical approaches. However, in visual analytics automated approaches lack the flexibility and sensitivity of human perception when seeking singular, undefined anomalies. This study therefore utilizes scalable, loosely guided, online volunteer participation to generate human identifications of unknown anomalies within massive volumes of geospatial remote sensing data. The emergence of statistical trends from a large sample of independent inputs highlights the collective human perception of the images' content. Similar to volunteer geographic information networks such as OpenStreetMaps [1], inputs from multiple contributors generate a collective map of local knowledge [2], only in this case the term “local” represents the global base of human visual perception. Turning to the crowd as the “partner of choice” for scalable problem solving is becoming increasingly attractive across broad domains both in science and industry [3], [4]. While many crowdsourcing approaches rely upon gameplay dynamics [5], [6], monetary incentives (e.g., Amazon's Mechanical Turk [7], [8] or [9]) to motivate participation in individual tasks, others utilize the social recognition of charismatic challenges such as space exploration [10] to encourage participation. It has been shown in several of these cases that complex problems, if strategically structured, can be solved by pools of volunteer participants with little to no pre-existing domain knowledge of their analytical challenge [11]. Here we leverage the charismatic challenge of directing a field expedition in search for the tomb of Genghis Khan (Chinggis Khaan) to engage tens of thousands of public volunteers and generate millions of micro-contributions towards an archaeological satellite imagery survey. These contributions aggregate into a collective geospatial map of analytical cross-verifications. In this particular case, since there exists no historical or archaeological record describing the physical appearance of this tomb, we turn towards the crowd not only to tackle the data size challenge of large scale satellite remote sensing, but more importantly to pool human perception and intuition when sifting through the data for anything that looks “out of the ordinary”. This loosely defined search criterium presents a challenge, where a participant's desire for validation or guidance cannot be met with an administrative answer. Thus, we introduce a framework to enable a crowd directed evolution of feature search criteria that mimics crowd reasoning processes in nature. The collective behavior of flocking animals, for example, is enabled through simple decision making and feedback criteria adhered to by the independent agents of that collective [12]. Similar to the vector and spatial referencing data required by individual birds within a flock, our study constructs multi-directional channels of visual reasoning feedback between the individual participant and the crowd. We hope that in this scheme, the collective will guide itself towards an agreement regarding what to look for, essentially facilitating a “collective intelligence” beyond that of individuals whom comprise the collective [13], [14]. We explore this in the case of perceptual reasoning. 
 
 Discussion We have described a crowd based solution to a satellite imagery remote sensing challenge of both data volume and search target ambiguity. Specifically, we charged an online crowd of volunteer participants with the challenge of finding the tomb of Genghis Khan, an archaeological enigma of unknown characteristics widely believed to be hidden somewhere within the range of our satellite imagery. This is a needle in a haystack problem where the appearance of the needle is unknown. To address this constraint we designed a system where participants actively evolve the collective training base of user feedback with their own inputs. Furthermore, the framework of the system created a resilient and self-validating data source through massively paralleled and constrained user inputs. Thus, we rely upon the emergence of agreement regions from independent tags to guide both the online volunteer community as well as the field archaeological expedition that surveyed anomalies on the ground. The entire data set is distributed in randomly chosen image tile subsets to participants, thus a “global” kernel density estimation approach is introduced to normalize saliency across all image tiles and to create an overall agreement region ranking. This concept could be applied as a distributed voting framework, where overlapping subsets allow for large data sets to be subdivided and parsed among many voters and then recombined into a single collective vote. Of the top 100 accessible locations identified by the crowd, 55 potential archaeological anomalies were verified by the field team, ranging from bronze age to Mongol period in origin. Yet, the question remains: could these results have been obtained just as effectively or more effectively without crowdsourcing? Or more specifically, could a small team of trained archaeologist have found the anomalies quickly by visually scanning the images on their own? After all we did expend 30K hours (3.4 years) of collective human survey effort. Looking first at the data size challenge, we have surveyed a historically significant area of roughly 6,000 km2. This is twice the size of Yosemite National Park, with equally diverse geologies and significantly greater in-accessibility. A ground survey of this detail for the entire range would have been prohibitive. Yet, at 0.5 meter/pixel resolution, a satellite imagery survey of the same area is in itself exhausting. A single archaeologist would have had to scroll through nearly 20,000 screens (assuming 1280×1024 screen resolution) before covering the whole area. But putting the data size challenge aside for a moment, we can observe the crowd's ability to be sensitive yet flexible. We continue to emphasize that very little is known about the likely visual appearance of the search target. Thus, we cannot limit our search criteria to what is traditionally “expected” from the known literature. Here is where the authors believe the power of crowdsourcing lies not only in harnessing parallel networks for scalable analytics, but in forming the collaborative frameworks necessary to cultivate collective reasoning. We depend on the crowd to process and identify the unexpected. Within our framework we observed that when participants were not provided the incremental peer based feedback loop they were statistically less likely to positively identify these anomalies, suggesting a form of collective reasoning has emerged within our participant pool that is variant and potentially more effective than the accumulated independent reasoning of individuals within that pool. Furthermore, while we acknowledge that there may be anomalies that remain undetected, this statistical variance suggests that largely parallel analytics (crowdsourcing) can provide better outcomes than an individual survey. While this study focused on an archaeological survey, there exists a broad range of challenges where scalable human perception networks could be effectively applied. These concepts have been further explored in applications ranging from humanitarian response to search & rescue [28]–[30]. The activities have not only tapped into our connectivity to scale human analytics, but also for the social mobilization of human attention [31], [32]. A recent direct derivative of the effort described here can be seen in Digital Globe Inc.'s “Tomnod” (Mongolian word meaning “big eye”) survey for the missing Malaysia Airlines flight MH370, where over 8 million participants surveyed over 1 million km2 of ultra-high resolution satellite imagery for anomalies. The shear mass of participation in this example provides a glimpse of the potential of our networked society. These crowdsourcing activities help us dive into the unknown and extract the unexpected. However, beyond that they present a fundamentally new construct for how we, as a digitally connected society, interact with information. The ability to focus and route networks of human attention at such massive scales, coupled with the functional ability for meaningful micro-contributions at individual scales, presents yet another evolutionary step in our collective ability to reason. 
 
 Acknowledgments We thank N. Ricklin, S. Har-Noy, K. Stamatiou, and the entire Valley of the Khans (VOTK) project team; S. Bira, and T. Ishdorj of the International Association for Mongol Studies and F. Hiebert of the National Geographic Society for co-leadership in field expeditions; D. Vanoni, K. Ponto, D. Lomas, J. Lewis, V. deSa, F. Kuester, S. Belongie, R. Rao, and E.A. Spencer for critical discussions and contributions; S. Poulton and A. Bucci with National Geographic Digital Media; and the entire Digitaria team. We sincerely thank the National Geographic Society, the Waitt Institute for Discovery, the Digital Globe Foundation (formerly the GeoEye Foundation), and the National Science Foundation.
– If we ultimately uncover Genghis Khan's tomb, we may have outer space and about 10,000 volunteers to thank. As Smithsonian recounts, legend has it that the location of the Mongolian warrior's tomb was safeguarded by soldiers who murdered the tomb builders and were then killed themselves. University of California-San Diego scientist Albert Yu-Min Lin has been on the hunt for this "needle in a haystack," and since 2010, he's had some help: Under his Valley of the Khans Project, volunteers were able to visit a National Geographic website, sift through high-res images of Mongolia taken by orbital satellites, and flag any features they spotted. Except no similar burials have been uncovered in Mongolia, so "the appearance of the needle is undefined." In a paper published Dec. 30 in PLOS One, Lin reports on what was accomplished in the first six months. Some 10,000 volunteers spent 30,000 hours (that's 3.4 years) examining roughly 2,300 square miles (two times Yosemite's size) that had been tiled into 84,183 images. They were asked to tag any feature they spotted as a road, river, modern structure, ancient structure, or other. Participants tagged 2.3 million sites, and the researchers used "consensus, defined by kernel density estimation, to pool human perception" for features with potentially ancient origins. The field team explored 100 of those and positively identified 55 sites with archaeological significance (no Genghis Khan tomb, sadly), including Bronze Age burial mounds. The researchers ask: Could archaeologists have found these themselves? Their answer: A ground search would have been "prohibitive," and a scan of the images would have required an archaeologist to scan through 20,000 screens. (A remarkable tomb was found in Egypt recently.)