Don’t Follow the Crowd: Incentives for Directed Spatial Sampling

The rapid spread of mobile phones globally provides new opportunities to collect data directly from citizens to inform decision making. However such “crowdsourcing” can suffer from a number of sampling problems. We address the issue of sub-optimal sampling of citizen reports through a new method that uses small incentives to induce respondents’ willingness to supply information of a given quality, enabling a social planner to induce a more optimal sampling distribution. We report the results of a field experiment that sought to induce pastoralists that live in the drought-prone rangelands of central Kenya to provide reports on localized vegetation conditions. We begin by estimating the reward elasticity of response to spatiallyvarying incentives, which is positive and shows that individuals are willing to shift their submission patterns in response to small spatial and temporal variation in rewards for their submissions. We then show that our dynamic rewards successfully increase submissions from regions that were initially under sampled. Finally, we use a value of information model to determine that the dynamic rewards led to increased net value of information for cost, as the net effect of our dynamic reward structure was to reduce costly redundant submissions and increase more informative submissions from never or rarely visited locations. Our results have implications for a range of applications of “big data” sources in the developing and developed world. *** PRELIMINARY VERSION: PLEASE DO NOT CITE OR CIRCULATE *** 1 First version: December 2015.We thank seminar audiences at the Sydney Experimental Brownbag Seminar (University of New South Wales), University of California at San Diego (BDES), the Helmholtz-Centre for Environmental Research (Leipzig, Germany), the International Livestock Research Institute, and Cornell University for helpful comments. We acknowledge research collaboration and fieldwork support from colleagues at the International Livestock Research Institute (Nairobi, Kenya), Cornell University and UC San Diego. We acknowledge the Atkinson Centre for a Sustainable Future’s Academic Venture Fund and Australian Aid. This research was also supported by National Science Foundation (0832782, 1059284, 1522054), ARO grant W911-NF-14-1-0498. The authors thank the International Livestock Research Institute for generous hospitality. The usual disclaimer applies. 2 Mara House, International Livestock Research Institute, 30709 Naivasha Rd., Nairobi, Kenya. n.jensen@cgiar.org 3 Room 363, Merewether H04, The University of Sydney, NSW 2006 Australia. russell.toth@sydney.edu. au. 4 344 Gates Hall, Cornell University, Ithaca, NY, USA. yexiang@cs.cornell.edu 5 344 Gates Hall, Cornell University, Ithaca, NY, USA. rab38@cornell.edu 6 9500 Gilman Drive, University of California San Diego, La Jolla, CA, USA. echebelyon@ucsd.edu 7 Mara House, International Livestock Research Institute, 30709 Naivasha Rd., Nairobi, Kenya. a.mude@cgiar.org 8 301G Warren Hall, Cornell University, NY, USA. cbb2@cornell.edu. 9 353 Gates Hall, Cornell University, Ithaca, NY, USA gomes@cs.cornell.edu Introduction The rapid and global spread of mobile phones provides new opportunities for low-cost data collection. There are close to 4 billion unique mobile phone users in the world, with about 83% of those connections in the developing world (GSM Assocation, 2015), with access now expanding rapidly even in rural and remote areas. This provides opportunities both for the passive collection of administrative call data records, and directly tapping the local knowledge and information of users, who are spatially dispersed and hence well-placed to generate reports at places and times that are otherwise costly to sample. The latter could include anything from reporting on local traffic conditions and accidents, to online ratings of restaurants and hotels, to providing information about conditions on the ground during the onset of disasters. Such information could be particularly valuable for improving access to information and decision making in the developing world, where gaps in infrastructure and government capacity particularly hinder the collection of useful data for decision making. Mobile phones raise promising opportunities to collect new data that could inform policy and business decision making. However efforts that rely on contributions from volunteers, commonly known as ‘crowdsourcing’, generally face a crucial trade-off: reduced cost of collecting information in return for reduced control over data collection. For many of the applications that utilize such data, we are not concerned with the goal of representative sampling, rather to have sufficient data on target events. For example, in an application of crowdsourcing to traffic accidents we would not be seeking a representative set of reports from all drivers in the city. Rather we would simply want a sufficient density of reports on each accident, which is an idiosyncratic set of localized events at any given time. The crowdsourcing problem, then, is often less about uniformity of sampling than simply ensuring that we receive sufficient reports on each event of interest. This may take on a temporal dimension, if we want to track the persistence of the event over time. A similar intuition would hold for events from crimes, to restaurant quality, to disasters. In this paper we present the results of a field experiment that demonstrates a method to quantify in monetary terms the extent to which respondents are willing to provide desired reports. These estimates could be used to design an incentive system that generates a closer-to-optimal sampling distribution for a given budget, by nudging users to provide more useful reports through monetary rewards. Specifically, we focus on a spatial-sampling problem in a droughtprone region of central Kenya, where it would be desirable for a variety of policy applications to have better micro-data on vegetation conditions ‘on the ground,’ particularly to identify “bad events” (locations with particularly poor conditions). These conditions are a leading indicator for localized risks of welfare and economic losses due to drought, the major source of humanitarian risk in this setting. Existing approaches to collect such data face drawbacks in terms of cost (i.e., data collection by scientists) or accuracy (i.e., remote sensing data from satellites, which are often aggregated at the level of tiles that can be thousands of square meters in size). We consider the possibility of collecting data on localized vegetation conditions at high spatial and temporal resolution from local citizens through smartphones. We ask participants to characterize the forage conditions throughout their day using a short, visually-oriented and geolocated survey, and then take a picture of the location they are characterizing. The crowdsourcing challenge in this setting is that participants are likely to provide highly clustered observations while landscape-scale forage modeling requires more data from locations where participants visit under typical conditions and from locations that are less frequently visited, which might be used drought periods. The project was implemented among pastoralists in a drought-prone region in northern Kenya. We use a baseline treatment, which offers participants a uniform payment (“piece rate”) per report. Such an intervention should not incentivize respondents to deviate from their natural movement patterns, identifying locations that are frequently and rarely visited at baseline. We then employ a series of randomized incentive treatments over the course of 12 approximately 9-day periods. We randomize 113 participants into one of six treatment groups across five training sites (villages). In each weekly period, two of the treatment groups face the same uniform piece rate incentives as in baseline. This allows us to control for temporal variation in movement patterns and participation. The other four treatment groups face a version of spatially-varying incentives, which are implemented and reported to participants through digital maps that attach different rewards to reports from different locations. Through this approach we receive about 95,000 valid survey submissions over the treatment period. First, we estimate an individual level ‘reward elasticity of supply’ – the percent increase in the propensity of each individual to increase submissions in a region due to an increase rewards attached to that region. The estimates are identified off the extent to which participants change the location from which they make submissions when under spatially-varying incentives than under uniform incentives. Nearly all participants express positive elasticities, reflecting a willingness to change their location in response to small changes in rewards. We then look at the sample average elasticity, as an indicator of the elasticity of the aggregate distribution of submissions, finding that at the median reward level, a 50% increase in rewards leads to approximately a 13.6% increase in (aggregate) submissions from that region. We then estimate the aggregate ‘reward elasticity of supply’, capturing the extent to which the aggregate spatial distribution of submissions responds to relatively small changes to incentives. Such a parameter estimate is a crucial input to designing an optimal spatial incentive scheme for the key application we have in mind, and for a much broader class of applications. To do so, we identify those regions that are under sampled at baseline. We find that adjusting the rewards to favor under-sampled regions increases the likelihood that a submission is from an under sampled region by a factor of at least 1.5 and that the number of reports per day from under sampled regions more than doubles under spatially-varying incentives. While these results are based on the starkest definition of under sampled – seeing zero submissions at basel