# Who is in Control in Crowdsourcing Initiatives? An Examination of the Case of Crowdmapping

Abdul Rehman Shahid, Amany Elbanna

<figure><img src="image_1.png" /></figure>

## To cite this version:

Abdul Rehman Shahid, Amany Elbanna. Who is in Control in Crowdsourcing Initiatives? An Examination of the Case of Crowdmapping. 7th Scandinavian Conference on Information Systems (SCIS), Aug 2016, Ljungskile, Sweden. pp.135-148, 10.1007/978-3-319-43597-8_10. hal-01771706

HAL Id: hal-01771706

https://inria.hal.science/hal-01771706

Submitted on 19 Apr 2018

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Who is in Control in Crowdsourcing Initiatives?
## An Examination of the Case of Crowdmapping

Abdul Rehman Shahid and Amany Elbanna

Royal Holloway, University of London, Egham, Surrey, TW20 0EX
{abdul.shahid.2012@live.rhul.ac.uk}
{amany.elbanna@rhul.ac.uk}

**Abstract.** The crowdsourcing literature is dominated by the view that the crowd can be controlled and that owners should adopt different technologies to control it and its output. This paper questions the agency and role of the crowd. Specifically, it questions how and to what extent can control over the crowd be exercised. To this end, the paper adopts an interpretive approach to the enquiry. It examines a case of crowdsourcing in the understudied area of humanitarian response. Concepts from structuration theory are adopted to interpret the data. The analysis reveals the paradox of crowd interaction and owner control in crowdsourcing. It shows the crowd to be made up of knowledgeable and reflexive groups that effectively tackle methods aimed at controlling them. The implications of the study are then discussed.

**Keywords:** Crowdsourcing, crowdmapping, structuration theory, social theory, OpenStreetMap (OSM).

## 1 Introduction

Crowdsourcing is “a type of participative online activity in which an individual, an institution, a non-profit organization, or company proposes to a group of individuals of varying knowledge, heterogeneity, and number, via a flexible open call, the voluntary undertaking of a task” [1]. It is an emerging phenomenon, the wide applications of which are witnessed in practice and “is yet to receive intense attention from the scholars” [2]. In practice, crowdsourcing initiatives have shown exponential growth, with one popular platform (www.crowdsourcing.org), reporting a 100% increase in the number of entities offering crowdsourcing services over a time frame of just two years [3]. It is surrounded by stories of crowd success in participating in and solving problems in many walks of life, varying from solving long-standing scientific problems (in a remarkably short time), to participating in the design of
apparels and in urban planning [4, 5]. This is in addition to the crowd getting involved
in innovation, generating commercial ideas, and swiftly providing disaster response
[6-9]. Scholars find crowdsourcing to be a ‘fascinating phenomenon’, and have urged
researchers to examine it both empirically and theoretically [10-12].

Crowdsourcing initiatives typically begin with an organisation or group issuing an
open call for unidentified individuals to participate in a particular task with a desired
outcome. To differentiate them from the crowdsourcing participants—or, more simply
put, ‘the crowd’—we call these initiators the crowdsourcing owners. The
crowdsourcing literature tends to side with the owners and largely adopts a
managerial perspective to examine and prescribe how to manage the crowd, control
the quality of its output, and keep it motivated [13-16]. Although a strand of literature
exists that examines the crowd side of crowdsourcing, it centres on investigating the
crowd’s motivations. This is mainly done in order to harness the crowd for the benefit
of the crowdsourcing owners and, hence, actually belongs to the managerial
perspective of crowdsourcing. This research portrays the crowd as a potentially
controllable multitude of individuals that owners only need to manage in order to
harness its creativity and innovation [17]. This view of the crowd contradicts the few
descriptive studies that show the crowd as being made up of individuals who are
capable of taking surprising initiatives to respond and find solutions to problems [8,
18, 19]. It is not clear how the crowd could take action while the owners are exerting
managerial control over it to achieve planned results, as described in the literature.
This study aims at exploring this issue and specifically answer the following
questions:

* Who is in control in crowdsourcing initiatives?

* How and to what extent can control over the crowd be exercised?

To do so, it examines a case study of crowdmapping during a natural disaster. As a
theoretical lens, it adopts concepts from structuration theory, which takes a balanced
relational view of control and hence provides an open lens for the enquiry. In this
regard, structuration theory ‘offers a valuable approach to theory development’ [20].

In the following section, we review the crowdsourcing literature. Section three
presents the concepts from structuration theory adopted by this study. Section four
presents the research methods, including the description of the case study. Section
five presents the analysis of the study. Sections six and seven respectively provide the
discussion and the conclusion and implications of the study.

## 2 Literature Review

Crowdsourcing literature is dominated by research on controlling the crowd; it tends
to adopt a descriptive and prescriptive managerial approach, and can be classified into
two streams. The first highlights the importance of quality control and describes ways
to effectively manage it. The second stream designs, tests or describes specific
methods to control the quality of the crowd's output.

Among this existing research, Kittur et al. [15] developed the ‘CrowdForge’
framework. The authors highlighted the increasing role played by micro-tasking, in
which organisations utilise a large number of workers for different purposes.
Importantly, their study articulated the importance of quality control to avoid a bad
contribution adversely impacting the task as a whole, when a complex task is broken
down into a number of individual contributions. The authors proposed quality control
methods which utilise human intelligence—namely, the map-reduce approach. Yung
et al. [16] built on the work of Kittur et al. [15] by proposing a crowdsourcing system
architecture that enables a new quality control approach that utilises evolutionary
computing and slow intelligence. Hansen et al. [14] explored the FamilySearch
crowdsourcing initiative, in which contributors transcribe/index ancestral records,
building up a comprehensive collection of genealogical records. The authors explored
the effectiveness (accuracy) and efficiency (time) of two quality control mechanisms
put in place by the crowdsourcing system—namely, arbitration and peer review.
Hansen et al. [14] found that peer review is significantly more efficient than
arbitration in terms of time, but is not as effective in terms of accuracy in certain
fields. Interestingly, the authors found that arbitration of peer-reviewed contributions
does not necessarily increase quality.

Hiltunen [21] highlighted training as an important component of an effective crowdsourcing system. The author explored the case of Finpro, an organisation that utilises crowdsourcing for its foresight activities, and argued that no foresight thinking can take place unless continuous and adequate training is provided to contributors. She asserted that “there can be no successful results in crowdsourcing without training” [21]. In their study, Le et al. [22] made an attempt at understanding how training crowdsourcing contributors could impact the quality of crowdsourced data and argued that those crowdsourcing systems that train contributors on relevance categorisation tasks achieve improvements in their overall data quality.

Other scholars explicitly connected effective training with quality control and went
into more detail in regard to training methods. For example, Poesio et al. [23] argued
for the importance of training and evaluation of contributors in ensuring quality data
for Phrase Detectives, an online crowdsourcing game that utilises the crowd to create
anaphorically annotated resources.

3 Theoretical Foundation

This study adopts concepts from Anthony Giddens’s structuration theory [24] as the
theoretical lens through which to trace and understand crowdmapping. Structuration
theory is particularly suited because of its balanced view of structure and action
(structuration theory refers to action as ‘agency’). The sections that follow briefly
present the concepts of agency, dialectic of control, knowledgeability, and reflexivity.
reflexivity, but *reflexivity in turn* is possible only because of the continuity of practices
that makes them distinctively 'the same' across space and time" [24]. The reflexive
monitoring of conduct mainly occurs in a continuous manner rather than in selective
moments [24].

4 Research Methods

This study adopts an interpretive approach, which enables the in-depth exploration of social and cultural phenomena [30]. Interpretive research attempts to understand phenomena through the eyes of the participants. This study adopts a case study approach in order to develop rich insights [31, 32]. The adopted case study is that of the development of crowdmapping in the context of humanitarian response, and will be detailed in the next section. The data collection is comprised of voice and video interviews, and of a document, newspaper, and media review. Moreover, online data were collected from a mailing list/forum and through Skype instant messaging. Forty-three voice and video interviews were conducted in addition to email communication with nine other participants. Documents and online resources—including agency reports, news items, television interviews, and video recordings—were reviewed. The 43 voice interviews were conducted with various actors, including the Humanitarian OpenStreetMap (HOT) community and various humanitarian organisations, including the American Red Cross, United Nations Office for the Coordination of Humanitarian Affairs (UN OCHA), MapAction, specialists from the Philippines Government—the National Economic and Development Authority (NEDA)—and various crowdmapping contributors. Further interviews were conducted with relevant actors from DigitalGlobe and Mapbox. The voice interviews were conducted between August 02, 2014, and July 20, 2015, and ranged in duration between 16m40s and 96m36s, with an average of 47m31s; they were recorded and transcribed verbatim. The interviews were semi-structured, as this provided greater flexibility. Guidelines were followed to ensure thick description whilst increasing the credibility and richness of the gathered data [31, 33, 34]. In terms of data analysis, this study followed the methodological guidance of Pozzebon and Pinsonneault [35]—namely, grounded, narrative, and temporal bracketing analysis—which the authors had presented having drawn and built upon Langley’s [36] nine strategies for analysing process data. According to Pozzebon and Pinsonneault [35], data-grounded analysis provides a basis for inductive theorising that benefits from the structurational lens. Narrative analysis is where the empirical data are structured in a systematic manner in terms of detailing a chronology of all phases, and temporal bracketing analysis is the breaking down of events into the effects of action [35].

4.1 Case Description

The case study looks at crowdmapping during the powerful tropical cyclone, Typhoon Haiyan, which struck the Philippines between the 6th and 9th of November 2013. Haiyan resulted in 6,201 fatalities, 28,626 injuries and 1,785 missing persons in 591 municipalities and 57 cities [37]. The humanitarian organisation detailed in the case description and analysis will be kept anonymous and will therefore be simply named
Humanitarian Organisation One. It is a large international humanitarian organisation that provides education, emergency assistance, and disaster relief.

OSM was a prominent crowdmapping platform that was engaged with in the response to Typhoon Haiyan through its humanitarian team—namely Humanitarian OpenStreetMap Team (HOT). The OSM Haiyan crowdmapping system was activated by HOT on the 7th of November 2013, around 24 hours before Haiyan made landfall in the Philippines. The system was activated to initially focus on mapping Tacloban City, which an early analysis had predicted would sustain the greatest damage. An email was then sent out to the HOT mailing list, calling the OSM crowdmappers to participate in the mapping of Tacloban City. By the end of the activation, the mapping efforts went beyond Tacloban City, to cover the majority of the affected areas. Figure 1 shows the ‘before and after’ maps of Tacloban City to demonstrate the work done by the OSM crowdmappers in developing a richly detailed crowdmap. By the end of the OSM Haiyan effort, around 1,679 crowdmappers from 82 countries had been involved, and 4.7 million changes had been made to the relevant maps.

HOT played an intermediary role between the OSM crowdmappers and Humanitarian Organisation One. Humanitarian Organisation One was working on the ground in the Philippines and was continually advising HOT of what areas needed to be mapped. The activated crowdmapping system detailed a map of Tacloban City and other affected regions, broken down into small areas. The tasks were coordinated by HOT based upon the mapping requirements passed on by Humanitarian Organisation One; HOT would then list these online to be actioned by the OSM contributor community. The tasking manager allowed HOT to define the areas that needed to be mapped by the crowdmappers. The crowdmappers would select a task or randomly be given one and then pick the desired tool to edit using either of three available electronic editors—namely: JOSM, iD or Potlatch—and using satellite imagery that was provided by HOT. The tasking manager also allowed more experienced crowdmappers to undertake validation tasks in which they would validate or invalidate the contributions made by others. The three editors suited the different mapping abilities of the crowdmappers; whereas iD and Potlatch are browser based editors more suitable for beginners, JOSM is a desktop application that requires the contributor to download and configure various plugins. For the most part, Bing imagery was used to develop pre-disaster maps, whilst satellite imagery provided by DigitalGlobe was used to develop post-disaster maps [38]. Instructions on how to successfully complete the task were included in the tasking manager. Examples of tasks from Haiyan included tracing isolated buildings and road networks, with the mapping of entire cities helping to build up detailed basemaps.
<figure><img src="image_3.png" /><figcaption>Figure 1: OSM before and after the crowdsourcing initiative [39].</figcaption></figure>

# 5 Analysis

This section provides an interpretation of the findings obtained through the theoretical lens of structuration theory, and in particular the concepts of dialectic of control, knowledgeability, and reflexivity. It argues that, although the social structures of crowdsourcing owners dominated resources and exerted their control over the crowd through the tasking manager technology, this domination and control was contested. Shortly after the start of the Haiyan crowdmapping response, crowdmappers emerged as knowledgeable and reflexive agents, seeking the improvement of their participation. They contested the domination over resources and actively explored options to control them. The following sections present the detail of how the crowd, as a knowledgeable and reflexive agent, exerted power and negotiated control over resources.

## 5.1 Crowd Owners

During an activation, the HOT hierarchy carries out various tasks—e.g., imagery requests—while the platform activators drive the activation and accomplishment of the required crowdmapping tasks. The HOT community consists of these senior members, and of OSM crowdmappers.

As a beneficiary of HOT crowdmaps, Humanitarian Organisation One was involved in the redevelopment of the HOT tasking manager and also in the organisation of mapping events to train OSM crowdmappers. Therefore, the HOT hierarchy includes both platform owners and Humanitarian Organisation One's owners of outcomes; both will be referred to as 'owners' in the following section of the analysis.
## 5.2 Protesting Domination and Control

Satellite imagery is a crucial component in the undertaking of crowdmapping by OSM crowdmappers. Humanitarian Organisation One typically liaises with imagery providers—namely; DigitalGlobe—to obtain imagery and then make it available to the HOT platform. The latter, in turn, provides the imagery to the crowdmappers. Many initiatives have been developed to streamline the procurement process and obtain increasingly effective and swift access to imagery; this includes Imagery to the Crowd, where imagery would be purchased by the United States Department of State under specific licences that would allow its use for humanitarian purposes. However, DigitalGlobe remained the foremost organisation providing ‘allocative resources’— namely, satellite imagery—to the OSM crowdmappers. During the Haiyan activation, OSM crowdmappers experienced delays of around seven to ten days in obtaining satellite imagery. As DigitalGlobe and Humanitarian Organisation One possessed the resources crucially needed by other actors, both controlled the imagery procurement process and held ‘relational power’ over the OSM crowdmappers. The latter were perceived to have little influence on any aspect of control of the activation, including the release of imagery, and to play a role of exclusively participating in mapping whatever tasks the owners provided.

The delays with which DigitalGlobe released its imagery to OSM crowdmappers, coupled with an unclear release timetable, caused concern amongst the latter. In response, an online petition was initiated by a OSM crowdmapping provider who “was frustrated with the pace at which imagery was being released, because it ended up taking a bit of time” (OSM crowdmapping provider). The online petition specifically advocated the ‘quicker release of imagery’ for a ‘longer period of time’. The announcement of the petition, which was made in the OSM forum, is shown in figure 2 below:

<figure><img src="image_4.png" /><figcaption>Figure 2 - The call for petition in the OSM forum (OpenStreetMap, 2013a)</figcaption></figure>

Once the petition had been launched, differences of opinion in regard to the form by which the protest had been made emerged. Senior members, both at HOT and Humanitarian Organisation One, found the petition to be less than courteous to the
people working at the institutions involved in imagery procurement. A senior member of HOT remarked;

> "It wasn't so much that it bothered them personally, but what happens is that you have institutions that donate imagery, and then this petition gets circulated calling for more imagery and it looks like the institution has screwed up or that the institution is attracting bad press and so, for the people that already kind of put their jobs on the line to help us out, it was kind of a slap on the face."

However, others found the petition to have had a positive impact, despite its inevitable unintended negative consequences;

> "It caused some issues, it kind of ruffled some feathers behind the scenes, because there are a lot of people who have stuck their necks out to get us imagery and to help us and those people kind of took it the wrong way and they did not like this... There was just a lot of bad blood between us and some of the groups, and it wasn't meant to be that way, but it just was the nature of these things; sometimes they don't go the way you want them to" (HOT senior member).

Many crowdmappers thought that it helped push the agencies and imagery providers into re-thinking their approach and also push the community into reconsidering how they obtained their resources. This view was expressed by a crowdmapping

> "I think it somehow ruffled a lot of people and concerned some imagery providers, because some of the opinions were asking them to provide us with this kind of imagery".

On the other hand, the HOT hierarchy found the community to be unrealistic in its expectations of and requests to the providers. One of the senior members commented;

> "I think this petition—sort of—was a slap; like, there is no way in hell that the imagery providers are just going to put all their data online, open access for free. It costs two billion dollars to launch a satellite and that money doesn't recoup itself; like, it's not happening, the economics don't work. So, like, being kind of unrealistic like that just made the OSM community look dumb, instead of sort of being, like, good partners".

This view was also shared by Humanitarian Organisation One. A Geospatial Architect and Developer at Humanitarian Organisation One who was involved in the discussions argued that satellite imagery is not;
"A human right and that, as imagery providers are aware of their social responsibilities, the commercial model within which they operate should be respected".

He further remarked that;

"For me, it is a little disheartening; I wish people had focussed less on the petitions and more on the work itself".

Despite not being approved and being disliked by both the owners of the crowdmapping platform—namely, HOT and Humanitarian Organisation One—the petition was signed by over 400 active crowdmappers.

Importantly, the petition was successful in promoting the discussion about the overall role of the imagery providers and about how the HOT community could source imagery. It also positioned the crowdmapping contributors both as active agents, who reflect on the ways in which they participate in the crowdmapping platform, and as agents who have power to obtain and control their own resources.

## 5.3 The Diversification of Imagery Procurement

Some voices appeared to question the possibility of breaking away from being locked in the relationship with imagery providers altogether, and find other sources from which to obtain the imagery. These active discussions led the HOT community to consider their imagery sourcing, not only in terms of the time it took to obtain it, but also of the source from which they obtained it and how they could possibly control those sources. Possessing ownership of the sources was regarded as the ultimate control over the imagery. Hence, crowdmappers were motivated to think of alternative imagery sources that could be owned by the community members, such as Unmanned Aerial Vehicles (UAVs), commonly known as drones.

Whilst the effects of the online petition were unfolding, the OSM crowdmappers undertook a complementary move aimed at bringing about change in the imagery procurement process, tackling its dependency on imagery providers and increasing its control over resources. During the last few days of the Haiyan crowdmapping response, some of the OSM crowdmappers, who were affiliated to technology organisations and research centres, tested imagery obtained through UAVs to explore its suitability for the HOT platform. The positive results of the testing phase provided the crowdmappers with a complementary and inexpensive source of imagery that could be collected and owned by them, rather than having to rely solely on satellite imagery providers.

This change in the crowdmappers thinking of their own resources, in turn, led the owners to rethink the resources made available to the crowd. The HOT hierarchy started formally looking at different ways in which it could gain better access to UAV
imagery. In addition, Humanitarian Organisation One began to consider the
possibility of acquiring drones and using UAV images.

6 Discussion

The analysis shows that, under crisis conditions, the crowd were engaged in their practices and tried to find ways to improve them and make them more efficient, to the ultimate benefit of those affected by Typhoon Haiyan. They wanted to ‘make a difference’ in the production of detailed crowdmaps to help the agencies working on the ground in the Philippines. They questioned the resources offered to them by the owners, and their dependency on these resources. Moreover, they contested the *modus operandi* of the owners and loudly expressed the need for change. Moreover, while negotiating the times involved in obtaining existing resources, they took active steps to find alternative ones. In doing so, they depicted themselves as powerful actors who have the ability—to some degree—to control their means of production. The following sections discuss the dialectic of control in crowdmapping, and the knowledgeability and reflexivity of the crowd.

6.1 Dialectic of Control in Crowdsourcing

The study shows that, in a high-pressure situation, the crowd was able to reconsider
the control on resources exercised by the owners. In an effort to challenge this
control, the online petition was launched to protest against the delay in providing the
lifeline resources to the crowd—namely, satellite imagery. This pushed the HOT and
Humanitarian Organisation One hierarchies to find ways in which to obtain satellite
imagery more quickly. It also initiated a wider discussion within the crowd regarding
the control of resources, and led to initiatives aimed at obtaining alternative sources of
imagery that went beyond satellite ones. Although the HOT hierarchy and various
technological components—i.e., the tasking manager—attempted to maintain quality
by controlling crowdmapping practices, the petition was still able to bypass the
perceived control and have an impact on the community. The petition and the
discussion, and the action it initiated, highlight the limitations in the owners’ control,
and the ability of the crowd to exercise control beyond the former’s approval.

This finding shows that control is a double-edged sword that could be wielded by
both the crowd owners and the crowd itself. It offers an alternative view to the
existing crowdsourcing literature that depicts the crowd merely as entities to be
controlled by the owners—e.g. Hansen et al. [14], Kittur et al. [15], and Yung et al.
[16]. By contrast, this study shows that control in crowdsourcing initiatives in not a
one-way force exerted only by the owners over the crowd, but it is a negotiation in
which the crowd plays an active role in the end result.

6.2 The Knowledgeability and Reflexivity of the Crowd

This study shows that the crowd is aware of its practices, and knowledgeable and
reflexive regarding their improvement. It sees itself as an active participant who is
able to improve its social world. It is able to contest the resources offered to it, and
obtain new ones.

This finding extends the literature beyond the current organisational perspective, to show that the crowd is not just some ‘unidentified’ workforce that can be controlled by means of codified methods [40, 41]. It is a knowledgeable actor who actively reflects on its circumstances and finds solutions to its own problems. It could have the desire to develop and advance further beyond the owners’ plan, and might not be necessarily content with being controlled. It could aspire to gain power and control over its resources and contest the owners’ domination. This shows that the crowd is not only an active participant but also a key composer of the social world [24]. This understanding of the crowd as a knowledgeable actor that could produce, reproduce, and change its social structure provides a fresh view of the crowd that goes beyond the current literature.

In essence, this study goes beyond a somewhat simplistic understanding of control by arguing that it cannot always be fully achieved or implemented despite the best efforts directed at achieving or implementing it. This is argued based upon the dynamics of the HOT and Humanitarian Organisation One hierarchies and their attempts at controlling crowdmappers, which were ultimately challenged by the online petition and its effects. The contrast between what is portrayed by the existing literature and the findings of this study raises important questions in regard to the type of control that can be achieved in crowdsourcing communities; in particular, this study doubts whether full control is ever even possible.

# 7 Conclusion and Implications

In conclusion, this study explored the following research questions:

* Who is in control in crowdsourcing initiatives?

* How and to what extent can control over the crowd be exercised?

To answer them, we examined a crowdmapping initiative in the humanitarian response context. Through the extensive collection of data from various actors and the application of concepts from structuration theory, the analysis revealed changes in structures of domination. Specifically, the paper articulated the knowledgeability, reflexivity and dialectic of control possessed by the crowd in effectively tackling methods aimed at controlling it. This analysis showed that it is not sufficient to simply state the ways in which the crowd can be controlled without giving due recognition to the agency that it itself possesses.

This study primarily makes a contribution to the IS crowdsourcing literature. Through its identification of the paradox of action and control, this study adds to the understanding of control afforded by the existing crowdsourcing literature. By
identifying the knowledgeability, reflexivity, and dialectic of control possessed by the crowd in effectively tackling methods aimed at controlling it, this study provides a more thorough understanding of crowdsourcing and quality control, thus adding to the existing literature on crowdsourcing and quality control. This study calls for more appreciation and acknowledgement of the crowd and of the role it plays as a partner, as a possessor of its own skills, and as a knowledgeable actor, and not as a simple provider of solutions, ideas and innovation, as it is depicted in the literature.

Moreover, in terms of the wider IS literature, Walsham [42] argued that the IS literature needs to expand its agenda to understand whether ICTs are making the world a better place; this paper attempts to answer these calls through the exploration of the impact of crowdsourcing in a situation in which it ‘made a difference’ by enabling a humanitarian organisation to more efficiently respond in assessing damaged areas and allocating resources to relief efforts.

In practice, crowdsourcing owners need to be active communicators with the crowd and to be willing to listen to its voice regarding any improvement of its practices. This study presents a new perspective of crowdsourcing that can be further examined through additional research.
References

1. Estellés-Arolas, E. and F. González-Ladrón-de-Guevara, *Towards an integrated crowdsourcing definition*. Journal of Information science, 2012. 38(2): p. 189-200.

2. Zhao, Y. and Q. Zhu, *Evaluation on crowdsourcing research: Current status and future direction*. Information Systems Frontiers, 2014. 16(3): p. 417-434.

3. Tarrell, A., et al., *Crowdsourcing: A Snapshot of Published Research*, in *AMCIS 2013 Proceedings*. 2013.

4. Brabham, D.C., *The myth of amateur crowds: A critical discourse analysis of crowdsourcing coverage*. Information, Communication & Society, 2012. 15(3): p. 394-410.

5. Savage, N., *Gaining wisdom from crowds*. Communications of the ACM, 2012. 55(3): p. 13-15.

6. Bergvall-Kåreborn, B. and D. Howcroft. *The Apple business model: Crowdsourcing mobile applications*. in *Accounting Forum*. 2013. Elsevier.

7. Djelassi, S. and I. Decoopman, *Customers' participation in product development through crowdsourcing: Issues and implications*. Industrial Marketing Management, 2013. 42(5): p. 683-692.

8. Majchrzak, A. and P.H.B. More, *Emergency! Web 2.0 to the rescue! Communications of the ACM*, 2011. 54(4): p. 125-132.

9. Goodchild, M.F. and J.A. Glennon, *Crowdsourcing geographic information for disaster response: a research frontier*. International Journal of Digital Earth, 2010. 3(3): p. 231-241.

10. Afuah, A. and C.L. Tucci, *Crowdsourcing as a solution to distant search*. Academy of Management Review, 2012, 37(3): p. 355-375.

11. Majchrzak, A. and A. Malhotra, *Towards an information systems perspective and research agenda on crowdsourcing for innovation*. The Journal of Strategic Information Systems, 2013. 22(4): p. 257-268.

12. Zogaj, S., U. Bretschneider, and J.M. Leimeister, *Managing crowdsourced software testing: a case study based insight on the challenges of a crowdsourcing intermediary*. Journal of Business Economics, 2014. 84(3): p. 375-405.

13. Allahbakhsh, M., et al., *Quality Control in Crowdsourcing Systems*. IEEE Internet Computing, 2013. 17(2): p. 76-81.

14. Hansen, D.L., et al. *Quality control mechanisms for crowdsourcing: peer review, arbitration, & expertise at familysearch indexing*. in *Proceedings of the 2013 conference on Computer supported cooperative work*. 2013. ACM.

15. Kittur, A., et al. *Crowdforge: Crowdsourcing complex work*. in *Proceedings of the 24th annual ACM symposium on User interface software and technology*. 2011. ACM.

16. Yung, D., M.-L. Li, and S. Chang, *Evolutionary approach for crowdsourcing quality control*. Journal of Visual Languages & Computing, 2014. 25(6): p. 879-890.
17. Majchrzak, A., L. Cherbakov, and B. Ives, Harnessing the power of the crowds with corporate social networking tools: How IBM does it. MIS Quarterly Executive, 2009. 8(2): p. 103-108.

18. Palen, L., S.R. Hiltz, and S.B. Liu, Online forums supporting grassroots participation in emergency preparedness and response. Communications of the ACM, 2007. 50(3): p. 54-58.

19. Palen, L. and S.B. Liu, Citizen communications in crisis: anticipating a future of ICT-supported public participation. in Proceedings of the SIGCHI conference on Human factors in computing systems. 2007. ACM.

20. Walsham, G. and C.-K. Han, Structuration theory and information systems research, in International Conference on Information Systems (ICIS). 1990.

21. Hiltunen, E., Crowdsourcing the future: the foresight process at Finpro. Journal of Futures Studies, 2011. 16(1): p. 189-196.

22. Le, J., et al. Ensuring quality in crowdsourced search relevance evaluation: The effects of training question distribution. in SIGIR 2010 workshop on crowdsourcing for search evaluation. 2010.

23. Poesio, M., et al., Phrase detectives: Utilizing collective intelligence for internet-scale language resource creation. ACM Transactions on Interactive Intelligent Systems (TiiS), 2013. 3(1): p. 3:1-3:44.

24. Giddens, A., The Constitution of Society: Outline of the Theory of Structuration. 1984, Cambridge: Polity.

25. Giddens, A., New rules of sociological method: A positive critique of interpretative sociologies. 1993: Stanford University Press.

26. Cohen, I.J., Structuration theory: Anthony Giddens and the constitution of social life. 1989: Macmillan London.

27. Tucker, K., Anthony Giddens and modern social theory. 1998: Sage.

28. Giddens, A., Central problems in social theory: Action, structure, and contradiction in social analysis. 1979: Univ of California Press.

29. Giddens, A., Profiles and critiques in social theory. 1982: Univ of California Press.

30. Myers, M.D. Qualitative Research in Information Systems. http://www.qual.auckland.ac.nz/ 2012 [cited 2012 9th November].

31. Walsham, G., Interpretive case studies in IS research: nature and method. European Journal of information systems, 1995. 4(2): p. 74-81.

32. Walsham, G. and C.K. Han, Information systems strategy formation and implementation: the case of a central government agency. Accounting, Management and Information Technologies, 1993. 3(3): p. 191-209.

33. Schultze, U. and M. Avital, Designing interviews to generate rich data for information systems research. Information and Organization, 2011. 21(1): p. 1-16.

34. Myers, M.D. and M. Newman, The qualitative interview in IS research: Examining the craft. Information and organization, 2007. 17(1): p. 2-26.

35. Pozzebon, M. and A. Pinsonneault, Challenges in conducting empirical work using structuration theory: Learning from IT research. Organization studies, 2005. 26(9): p. 1353-1376.

36. Langley, A., Strategies for theorizing from process data. Academy of Management review, 1999. 24(4): p. 691-710.
37. NDRRMC. *SitRep No. 108 Effects of Typhoon "Yolanda" (Haiyan)*. 2014 [cited 2015 21th September]; Available from: http://www.ndrrmc.gov.ph/attachments/article/1329/Effects_of_Typhoon_YOLANDA_(HAIYAN)_SitRep_No_108_03APR2014.pdf.

38. OpenStreetMap. *Typhoon Haiyan - Tacloban Post-Disaster Imagery Mapping*. 2013 [cited 2014 27th November]; Available from: http://tasks.hotosm.org/project/350.

39. Hern, A. *Online volunteers map Philippines after typhoon Haiyan*. 2013 [cited 2014 7th April]; Available from: http://www.theguardian.com/technology/2013/nov/15/online-volunteers-map-philippines-after-typhoon-haiyan.

40. Saxton, G.D., O. Oh, and R. Kishore, *Rules of crowdsourcing: Models, issues, and systems of control*. Information Systems Management, 2013. 30(1): p. 2-20.

41. Oleson, D., et al., *Programmatic Gold: Targeted and Scalable Quality Assurance in Crowdsourcing*. Human computation, 2011. 11(11).

42. Walsham, G., *Are we making a better world with ICTs? Reflections on a future agenda for the IS field*. Journal of Information Technology, 2012. 27(2): p. 87-93.