# Improving the Involvement of Digital Volunteers in Disaster Management

Roberto Dos Santos Rocha, Adam Widera, Roelof Berg, João Albuquerque,
Bernd Helingrath

► To cite this version:

Roberto Dos Santos Rocha, Adam Widera, Roelof Berg, João Albuquerque, Bernd Helingrath. Improving the Involvement of Digital Volunteers in Disaster Management. 1st International Conference on Information Technology in Disaster Risk Reduction (ITDRR), Nov 2016, Sofia, Bulgaria. pp.214-224, 10.1007/978-3-319-68486-4_17. hal-03213126

HAL Id: hal-03213126

https://inria.hal.science/hal-03213126

Submitted on 30 Apr 2021

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Improving the Involvement of Digital Volunteers in Disaster Management

Roberto dos Santos Rocha<fnref n="1" />,<fnref n="2" />, Adam Widera<fnref n="2" />, Roelof P. van den Berg<fnref n="2" />, João Porto de Albuquerque<fnref n="3" />,<fnref n="1" />, and Bernd Helingrath<fnref n="2" />

**Abstract.** Volunteered geographic information (VGI) has been seen as useful information in times of disasters. Several authors have shown that VGI is useful for coping with preparedness and response phases of disaster management. However, because it is still a young technology, the use of VGI remains uncertain, due to its lack of strong reliability and validity. It is our assumption that to improve reliability and validity the promotion of citizen engagement (CE) is needed. CE is not new topic, but in the digital humanitarian context, it involves important factors that are not yet considered by disaster managers, such as communication processes, motivation of volunteers, different media for production of information, etc. To fill this gap, we identified a set of preliminary factors which should be considered to promote the involvement of volunteers in disaster management. These factors were derived from critical review of CE literature and from an analysis of lessons learned from an experiment on interaction with citizens carried out in context of the EU-project "DRIVER – Driving Innovation in Crisis Management for European Resilience".

**Keywords:** Citizen Engagement, Volunteered Geographic Information, Motivation, Crowd Sensing, Disaster Management.

## 1 Introduction

Citizen Engagement (CE) refers to actions designed to identify and address issues of public concern [1]. Community participation can augment officials' abilities to govern in a crisis, improve application of communally held resources in a disaster or epidemic, and mitigate community wide losses [2].

Participatory community approaches in research and governance are not new [3, 4]. However, Web 2.0 platforms, mobile internet, and social networking access through smartphones have made a significant difference by encouraging the social responsibility.

<footnote n="1">¹ICMC, University of São Paulo, São Carlos, Brazil
rsrocha@usp.br</footnote>

<footnote n="2">²ERCIS, University of Münster, Münster, Germany
{adam.widera, roelof.vandenberg,
bernd.hellingrath}@ercis.uni-muenster.de</footnote>

<footnote n="3">³CIM, University of Warwick, Coventry, United Kingdom
j.porto@warwick.ac.uk</footnote>
ity and active engagement of citizens [3]. These technologies enable the public to con-
tribute and participate on an unprecedented scale and have led to many diverse initia-
tives using information from citizens [3, 4]. Examples include, among others, partici-
pation of the public in event reporting, environmental monitoring, and providing
information on natural disasters. This phenomenon is called volunteered geographic
information (VGI).

VGI is the harnessing of tools to create, assemble, and disseminate geographic da-ta
provided voluntarily by individuals [5]. VGI has been increasingly recognized by re-
searchers as an important resource to support disaster management [23-25]. The
production of geographic information is predominantly made through social media (e.g.
Twitter, https://twitter.com/), crowd sensing (e.g.citizens equipped with smartphones
can report about local conditions using dedicated applications) and online mapping
tools (e.g. OpenStreetMap, https://www.openstreetmap.org; Wikimapia
http://wikimapia.org/; Google Map Maker, https://www.google.com.br/mapmaker) [5-
7].

Whilst those platforms can be potentially used to provide useful information for
dealing with disaster management, there are still many challenges to be addressed, for
instance: (i) how can people be encouraged to provide valuable information; (ii) how
can information from volunteers be validated; and (iii) how can this information be
integrated with other sources of data [6, 8].

Many governments and agencies recognize the opportunities and challenges posed
by informal volunteers, and many have developed strategies and resources for en-gag-
ing and managing them. However, organizational culture, risks and liabilities im-pose
significant barriers to greater involvement of informal volunteers in emergency and
disaster management [2].

Different VGI categories – social media, crowd sensing, and collaborative mapping activities – require different strategies for promoting citizen engagement. It is our assumption that knowledge of the VGI categories is relevant for disaster managers to recruit and motivate users to utilize VGI-systems.

Additionally, works related to VGI in disaster management focus on production by
volunteers and the use of this information by disaster managers. They disregard the fact
that the production and consumption of VGI should be seen within a communication
process, i.e., the communication among the stakeholders should be multidirectional.

To help fill this gap, we present in this paper a preliminary set of key factors to help
promote the involvement of volunteers in the disaster management domain. These fac-
tors were derived from review of CE literature and from an analysis of les-sons learned
from a simulation exercise carried out in context of the EU-project "DRIVER – Driving
Innovation in Crisis Management for European Resilience". The DRIVER project was
launched in May 2014. This project, gathers the expertise of 37 organizations, and will
jointly develop solutions for improved crisis management. Representatives from the
security & defense industry, research and academia, SMEs, end-users and several Eu-
ropean institutions, from 13 EU member states and 2 associated countries participate in
this innovative venture.

With this work we aim to answer the following research question:
RQ. What factors should be considered by disaster managers to improve the involve-
ment of digital volunteers?

The remainder of this paper is organized as follows. First, in order to set a ground
on the different VGI approaches an overview is presented in Section 2. In Section 3 we
present a review on the motivation and engagement of digital volunteers. In Section 4
we present factors to improve citizen engagement, based on lessons learned from an
experiment on interaction with citizens carried out in context of the EU-project.
Finally, in Section 5, we conclude with final remarks and give potential directions
for future works.

2 Volunteered Geographic Information

## 2.1 Volunteered Geographic Information VGI Source

In general, VGI in the context of disaster risk management can be collected through
different collaborative sources [6]: (i) social media; (ii) crowd sensing; and (iii) collab-
orative mapping activities.

The first category of geo-information (i) involves the use of existing social media platforms to exchange information in an unstructured way. These platforms enable citizens to share self-produced content within a network of contacts or for the general public [6]. Common social media platforms include Twitter; Facebook, https://www.facebook.com/; Flickr, https://www.flickr.com/; YouTube, https://www.youtube.com/.

The second category of geo-information (ii) relies on citizens on the Web or
equipped with smartphones to act as sensors and share observations [6, 10]. The term
‘crowd sensing’ is used to describe approaches that make use of specific software ap-
plications to provide more precise structured data [6]. Ushahidi-based platforms and
mobile applications are the most commonly used in this category for data collection.
GDACSmobile, for instance, is a tool that facilitates the self-organization of volunteers
and improves the situational awareness of citizens by sharing an easy-to-understand
overview of the state of affairs. At the same time, GDACSmobile also provides a feed-
back mechanism to the crisis manager/control center [11].

The third category of geo-information (iii) consists of a specific type of information
and collaboration platform: the collaborative editing of geographic features to fulfill
internet-based interactive maps. Well-known platforms like OpenStreetMap (OSM),
Wikimapia and Google Map-Maker fall into this category [6].

Collaborative mapping activities are essential for disaster management, because they
collect a very specific type of data – namely, georeferenced data about features like
streets and roads, buildings etc. – and structures this information in the form of a map
[23]. The OpenStreetMap (OSM) project has great potential in disaster scenarios, which
was shown when a large number of volunteers provided their support in mapping events
after the 2015 Haiti earthquake [26, 27] and the 2015 Nepal earthquake [23]. Collabo-
rative mapping in OSM has emerged as a key mechanism through which individuals
can provide information about affected areas, thus making a tangible difference to aid
agencies and relief work without actually being physically present on-site [23].
## 2.2 VGI Types

Senaratne et al. [12], categorize the main types of VGI as (i) text-based VGI, (ii) image-based VGI, and (iii) map-based VGI.

Text-based VGI is generally produced implicitly on portals, such as Twitter or various blogs, where people contribute geographic information in the form of text by using smartphones, tablets etc. [12]. Twitter, for example is used as an information foraging source [12, 13].

Image-based VGI is generally produced implicitly within portals such as Flickr, Instagram, etc., where contributors take pictures of a particular geographic object or surrounding with cameras, smart phones, or any hand held device, and attach a geo-spatial reference to it [12].

Map-based VGI covers all sources that include geometries as points, lines, and polygons, which are the basic elements to design a map. Among others, OSM, Wikimapia, Google Map Maker, and Map Insight are examples of map-based VGI projects [12].

Table 1 presents the relationship between sources and the different types of VGI.

<table><caption>**Table 1.** A summary of the source and types of VGI</caption><thead><tr><th rowspan="2">Source</th><th colspan="3">Type</th></tr><tr><th>Text</th><th>Image</th><th>Map</th></tr></thead><tbody><tr><td>Social Media</td><td>X</td><td>X</td><td></td></tr><tr><td>Crowd Sensing</td><td>X</td><td>X</td><td></td></tr><tr><td>Collaborative Mapping</td><td></td><td></td><td>X</td></tr></tbody></table>

## 2.3 Typology of VGI

Craglia et al. [3] introduced the concept of typology in VGI. According to these authors, there are two modes through which individuals or communities contribute such information: first, the way the information was made available and second, the way geographic information forms a part of it.

Each of these two dimensions can be ‘explicit’ or ‘implicit’, with explicit denoting that the dimension is of primary concern to the piece of VGI, while implicit denotes that the dimension was not originally an integral part, and is of secondary concern [3]. Thus the topology of VGI proposed by Craglia et al. [3] is a matrix of four types of VGI as shown in Table 2.

Geographic location is essential in disaster analysis [14]. Thus, only the explicit geographic category – explicitly- or implicitly-volunteered information – of the VGI typology is used in this paper.

Nevertheless, the typology of VGI proposed does not take in account the fact that different VGI types have different translation needs, which this may imply excess noise (e.g., many useless messages before a useful message to be found). For example, a picture of a flooded area is more effective (i.e., it has less translation needs) for disaster managers than a short message (tweet) describing the same flooded area.
<table><caption>**Table 2.** Typology of VGI [14] (adapted from Craglia et al. [3])</caption><thead><tr><th rowspan="2"></th><th colspan="2">Geographic Information</th></tr><tr><th>Explicit</th><th>Implicit</th></tr></thead><tbody><tr><th>Explicitly Volunteered</th><td>“True” VGI, e.g., Open-StreetMap</td><td>Volunteered (geo)spatial. Information, such as Wikipedia articles about non-geographic topics containing place names</td></tr><tr><th>Implicitly Volunteered</th><td>Citizen-generated geographic content (CGGC), e.g., Tweets referring to the properties of an identifiable place</td><td>User-Generated (geo)Spatial Content (UGSC), such as Tweets only mentioning a place in the context of another (non-geographic) topic</td></tr></tbody></table>

Considering this aspect, we propose a new typology of VGI, which considers different levels of uncertainty – noise and translation needs. As can be seen, the explicit VGI sources – crowd sensing, and collaborative mapping activities – have fewer translation needs than social media data.

<figure><img src="image_2.png" /><figcaption>**Fig. 1.** Typology of VGI considering different levels of uncertainty - noise and translation needs</figcaption></figure>
# 3 Motivation and Engagement of Digital Volunteers

A Digital volunteer or a digital humanitarian is an individual that applies and leverages their technical skills in collecting, processing and managing data in support of response efforts for disasters [30]. In most cases, he/she is not physically present at the place where the disaster has occurred. The Digital Humanitarian Network (DHNetwork) grew out of this ecosystem of emerging technical volunteer involvement based throughout the globe [29]. Since the 2010 Haiti earthquake, these com-munities have provided support to formal humanitarian operations [27], and more recently have provided a crucial compliment to operational organizations and governments active in the field [29].

However, it is still a challenge to keep digital volunteers motivated and engaged for longer periods, especially considering that they do not have strong connections to events due to their digital presence instead of physical presence in affected areas. This requires different ways to motivate and engage them to provide high quality contributions in future crisis/disaster situations.

To deal with this critical issue, we propose in this section a first attempt to understand how motivation of digital volunteers can be understood using, for instance, the Valence, Instrumentality, Expectancy (VIE) approach [17].

In general, volunteers are motivated by many incentives. Examples include, ideology, personal satisfaction, community, and humanitarian values. Particularly, in the context of a digital humanitarian, there is another factor that should be considered: the desire to apply and improve technical knowledge. Considering that many of these volunteers come from open source communities [28], why is it so important to understand the incentives of citizens in order for them to share their observations from the field? Considering the 90-9-1-Rule [31], there is a so called “participation inequality”. According to this rule, in a collaborative online environment, 90% of the participants of a community only view content, 9% of the participants edit content, and only 1% of the participants actively create new content [31].

In addition to such factors, it is important to understand two specific questions: (i) why does a citizen report an observation from the field, and (ii) does the information reported supports the decision making of disaster managers.

The first question is related to behavioral aspects, i.e., what are the incentives of citizens in order for them to want to share their observations from the field? Literature commonly understands incentives as instruments influencing the behavior of members of an organization or community in order to adapt to the organization wide system of objectives [18]. By creating incentives, certain desired modes of behavior from individuals are promoted, enabling specific situational conditions which in turn result in the activation of individual motives. In this case, a motive denotes a time-invariant psychological disposition, i.e. an isolated, not yet activated incitement for the behavior. Capelo et al. [28] highlight some important incentives:

- Encourage volunteers by giving feedback, recognition, appreciation and gratitude;

- Cultivate a sense of ownership and accountability. Team members have to know that they matter, and that they are making a difference in the humanitarian operation;
- Generate a feeling of inclusivity based on a system of collaboration, partnership and sharing with multiple stakeholders;

- Provide training and capacity-building opportunities for volunteers.

The second question is related to the contribution of effort input. For instance, the reporting of a flooded road is higher, if the reporter (or citizen) is affected or not. According to the VIE approach [17], the effort input of an individual is high when it expects that the contribution will yield results, which are first important to the organization or community, secondly, due to the expected instrumentality, show close relationships to individually aspired results from extrinsic incentives, and thirdly exhibit valences as high as possible.

# 4 Lessons Learned to Improve the Involvement of Volunteers

In this section, we summarize the lessons learned to improve the involvement of volunteers in the production of high-quality relevant information for disaster management. The elements presented here were derived from the VGI and citizen engagement literature reviews, as well as the analysis of lessons learned from a field experiment carried out in the EU-funded demonstration project DRIVER.

The selection process of the literature was based on the experience of the authors of this paper. Regarding the VGI literature, four main works were selected as an input to identify the main characteristics that may affect the digital volunteers' engagement: Albuquerque et al. [6], Craglia et al. [3], Klonner et al. [14], Senaratne et al. [12]. Regarding motivation and engagement of volunteers, we selected Nielsen [31], Lawler [17], and Capelo et al. [28] to understand the main aspects that should be considered in the digital volunteers' engagement.

The field experiment was based on a storyline designed by practitioners. They defined a fictitious disaster event based on past experience, which resulted in a more realistic and relevant scenario compared to a more tool-friendly situation designed by the tool providers. The experiments conducted in the DRIVER "Interaction with Citizens" campaign concentrate on the following functions (a more detailed description of this experiment can be founded in Havlik et al. [33], Middelhoff et al. [32] and van den Berg et al. [22]):

- Provision of context-aware and timely information tailored to the specific needs of different societal groups over various channels, in order to improve their understanding of the crisis situation and to minimize adverse impacts.

- Context-aware (micro-)tasking of non-affiliated volunteers to perform real and virtual tasks.

- Efficient gathering of situational information about an incident from volunteers.

- Efficient usage of received information from volunteers to improve the situational awareness of crisis managers and consequently their handling of the crisis.

In the following subsections, we will summarize the findings into two categories: (i) characteristics of VGI, and (ii) communication processes.
## 4.1 Characteristics of VGI

The characteristics of VGI – origin, type and typology – shown in Section 2 have an impact on citizen engagement.

In regard to the origin of VGI, crowd sensing and collaborative mapping activities have the potential to promote citizen engagement, due the ways in which the information is created.

Thus, we propose a typology of engagement based on five levels of involvement from volunteers in scientific work [20] and a typology of participation proposed by Pretty and Hine [21]. At the first level, citizens provide resources, while having only a minimal cognitive engagement. This level is called ‘basic’. In second level, ‘distributed intelligence’ relies on the cognitive ability of the participants. After some training, the participants collect data or engage in minor interpretation activities. At this level, quality evaluation by the volunteers is crucial. The third level represents ‘participatory engagement’, where users take part actively in the problem definition and data collection. On the last level, ‘self-mobilization’, non-professionals collaborate with professionals, and together, decide on a problem they want to focus on and the methods for data collection. This allows for both the consideration of interests and motivation of the volunteers. On this level, volunteers are not only experts, but also have the role of facilitators [14, 20].

<table><caption>**Table 3.** Typology of engagement</caption><thead><tr><th rowspan="2">Source</th><th colspan="2">Typology of VGI</th><th colspan="4">Levels of Engagement</th></tr><tr><th>Explicitly</th><th>Implicitly</th><th>I</th><th>II</th><th>III</th><th>IV</th></tr></thead><tbody><tr><td>Social Media</td><td></td><td>X</td><td>X</td><td></td><td></td><td></td></tr><tr><td>Crowd Sensing</td><td>X</td><td></td><td>X</td><td>X</td><td>X</td><td>X</td></tr><tr><td>Collaborative Mapping</td><td>X</td><td></td><td>X</td><td>X</td><td>X</td><td>X</td></tr></tbody></table>

Legend: I-Basic. II-Distributed intelligence. III-Participatory engagement. IV-Self-mobilization

As shown in Table 3, social media presents a ‘basic’ level of engagement. This is because of the nature of its contribution, i.e., social media is provided implicitly. All other VGI sources, provided explicitly, require different strategies for citizen engagement, since many have volunteers with different levels of knowledge and motivation.

## 4.2 Communication Process

The elements in the communication process determine the quality of communication. A problem in any one of these elements can reduce communication effectiveness [15]. For instance, different perceptions of the message, language barriers, interruptions, emotions, and attitudes can all reduce communication effectiveness. There-after, a feedback mechanism should be considered to promote the involvement of volunteers.

The existing literature on VGI focuses on the production of geographic information, and the use of this information by disaster managers. It disregards the fact that the production of VGI should be seen within a more effective communication process, i.e., the
communication among the stakeholders should have to include a mechanism for con-
tinuous feedback.

In the context of the DRIVER Project, tools were proposed that should address both objectives for the benefit of the community, and for individual members of the population according to the VIE approach. In a recent field exercise, the software tool GDACSmobile was used to communicate observations to crisis mangers (com-munity-objective). The personal objectives of members of the population were ad-dressed by sharing reports with the community as an information layer on a map of the environment around the user. In this way, the users could assure themselves of their safety in the situation and strengthen their situational awareness using map representation.

Consequently, by acknowledging the perceptions of community members, a common language visualizing interactions becomes a basic requirement for an appropriate crisis communication environment. One way to establish a common language is to use VGI systems in combination with commonly used information categories and according pictograms [22]. In the meantime, many different VGI tools are available, having different pros and cons regarding particular tasks. However, as mentioned above, the main challenge here is less a technical problem, but rather an organizational one. According to the discussion on incentives, we identify a trustful and open solution as most appropriate. However, it must be able to visualize benefits for the community, i.e. an easy- and fast-to-understand situation overview including a connection to responding authorities. In order to do so, the information should be structured and visualized respecting the communities' attributes (like age distribution, language, technical affinity, etc.).

5 Conclusion

Further research is still necessary for engaging volunteers in the production of high-quality relevant information for disaster management. For instance, how can we ensure that local communities are involved at a meaningful level in different phases of a disaster? This could be achieved, for instance, by initiating a community group or by providing training to volunteers to produce high-quality VGI. Moreover, how to improve collaboration between formal humanitarian organizations and volunteer technical communities (VTCs) should also be explored in future works.

Additionally, the different VGI categories require different strategies for promoting citizen engagement, given that the knowledge of the VGI categories is relevant for disaster managers to recruit and motivate users to utilize VGI-systems. Therefore, one of the expected outcomes will be the development of a new framework for promoting engagement of digital volunteers in the disaster management context.

Acknowledgments

The research that led to this work was funded by the European Community's Seventh
Framework Programme: Marie Curie Actions/Initial Training Networks under
grant agreement n° 317382 and FP7 / 2007-2013 under grant agreement n° 607798.
João Porto de Albuquerque is grateful for the financial support from the CAPES Pró-Alertas (grant nº 88887.091744/2014-01 and nº 88887.091743/2014-01). The authors would like to thank Robin Mays and Lívia Degrossi for their helpful comments and suggestions, and the DRIVER “Interaction with Citizens” experiment team that has been working together for several months in order to prepare, conduct and finally assess the field exercise in The Hague.

## References

1. American Psychological Association, http://www.apa.org/education/undergrad/civic-engagement.aspx

2. Schoch-Spana, M., Franco, C. Nuzzo, J. B., Usenza, C.: Community Engagement: Leadership Tool for Catastrophic Health Events Biosecur. Bioterror 5(1), 8-25 (2007).

3. Craglia, M, Ostermann, F., Spinsanti L.: Digital Earth from vision to practice: making sense of citizen-generated content. Int. J. Digital Earth 5(5), 398-416 (2012).

4. Weiner, D. and Harris, T.M.: Participatory geographic information systems. In: Wilson, J.P. and Fotheringham, A.S. (eds.) The handbook of geographic information science, 466-480. Blackwell Publishing, Malden (2008).

5. Goodchild, M.F.: Citizens as sensors: the world of volunteered geography. GeoJournal 69(4), 211-221 (2007).

6. Albuquerque, J.P., Eckle, M., Herfort, B, Zipf, A.: Crowdsourcing geographic information for disaster risk management and improving urban resilience: an overview of the lessons learned. In: Capineri, C. et al. (eds) European Handbook of Crowdsourced Geographic Information, Ubiquity Press. (2016).

7. Haworth, B.: Emergency management perspectives on volunteered geographic information: Opportunities, challenges, and change. Comput Environ Urban Syst. 57, 189-198 (2016).

8. Horita, F.E., Albuquerque, J.P., Degrossi, L.C., Mendiondo, E.M., Ueyama, J.: Development of a spatial decision support system for flood risk management in Brazil that combines volunteered geographic information with wireless sensor networks. Computers & Geosciences 80, 84-94 (2015).

9. Poser, K., Dransch, D: Volunteered geographic information for disaster management with application to rapid flood damage estimation. Geomatica 64, 89-98 (2010).

10. Ma, H., Zhao, D., and Yuan, P.: Opportunities in mobile crowd sensing. IEEE Communications Magazine 52(8), 29-35 (2014).

11. Link, D., Widera, A., Hellingrath, B., Limbu, M.K.: GDACSmobile - An IT Tool Supporting Assessments for Humanitarian Logistics. In: The International Emergency Management Society 2015 Annual Conference (2015).

12. Senaratne, H. Mobasheri, A., Ali, A.L., Capineri, C., Haklay, M.: A review of volunteered geographic information quality assessment methods. Int. J. Geogr. Inf. Sci. 1-29 (2016).

13. MacEachren, A.M., Jaiswal, A., Robinson, A.C., Pezanowski, S., Savelyev, A., Mitra, P., Zhang, X., Blanford, J.: Senseplace2: geotwitter analytics support for situational awareness. In: IEEE conference on Visual Analytics Science and Technology, Providence, RI. Piscataway, NJ: IEEE, 181-190 (2011)

14. Klonner, C., Marx, S., Uson, T., Albuquerque, J.P., Höfle, B.: Volunteered Geographic Information in Natural Hazard Analysis: A Systematic Literature Review of Current Approaches with a Focus on Preparedness and Mitigation. ISPRS Int. J. Geo-Inf 5 (103), 1-20 (2016).
15. Keyton, J.: Communication and organizational culture: A key to understanding work experience. Thousand Oaks, CA: Sage (2011).

16. Ulmer, R.R.: Effective Crisis Management through Established Stakeholder Relationships: Malden Mills as a Case Study Manag. Commun. Q. 14, 590-615 (2011).

17. Lawler, E.E.: Pay and organization development. Reading, Mass (1981).

18. Grob, H.L., Buddendick, C., Albrecht, N., Widera, A.: Towards A Holistic Approach of E-learning Incentive Management. In: International Conference E-Activity and Leading Technologies, pp. 280-94 (2007).

19. Horita, F.E.A, Degrossi, L.C., Assis, L.F.F.G, Zipf, A., Albuquerque, J.P.: The use of Volunteered Geographic Information and Crowdsourcing in Disaster Management: A Systematic Literature Review. In: 19th Americas Conference on Information Systems (2013).

20. Haklay, M.: Citizen Science and Volunteered Geographic Information: Overview and Typology of Participation. Crowdsourcing Geographic Knowledge, 105-122 (2013).

21. Pretty, J., Hine, R.: Participatory appraisal for community assessment. Centre for Environment and Society, University of Essex (1999).

22. van der Berg, R., Widera, A., Lechtenberg, S., Middelhoff, M. Hellingrath, B.: Pictograms and Assessment Categories as Crisis Communication Language: Lessons from a field exercise with GDACSmobile. In: The 3rd International Conference on Information Communication Technologies for Disaster Management (submitted).

23. Poiani, T.H., Rocha, R.S., Degrossi, L.C., Albuquerque, J.P.: Potential of Collaborative Mapping for Disaster Relief: A Case Study of OpenStreetMap in the Nepal Earthquake 2015. In 49th Hawaii International Conference on System Sciences (HICSS) (2016).

24. Horita, F.E.A, Albuquerque, J.P.: An approach to support decision-making in disaster management based on volunteer geographic information (VGI) and spatial decision support systems (SDSS). In: 10th International Conference on Information Systems for Crisis Response and Management, Baden-Baden, Germany (2013).

25. Erskine, M., Gregg, D.: Utilizing Volunteered Geographic Information to Develop a RealTime Disaster Mapping Tool: A Prototype and Research Framework. In: CONFIRM 2012 (2012).

26. Eckle, M. and Albuquerque, J.P.: Quality Assessment of Remote Mapping in Open-StreetMap for Disaster Management Purposes. In Proceedings of the ISCRAM 2015 Conference - Kristiansand, May 24-27 (2015).

27. Zook, M., Graham, M., Shelton, T. and Gorman, S.: Volunteered Geographic Information and Crowdsourcing Disaster Relief: A Case Study of the Haitian Earthquake, World Medical & Health Policy 2, 7-33 (2010).

28. Capelo, L., Chang, N. and Verity, A.:Guidance for collaborating with volunteer and technical communities.

29. Humanitarian Response, https://www.humanitarianresponse.info/en/applications/tools/category/digital-volunteers

30. Liu, S. "Crisis Crowdsourcing Framework: Designing Strategic Configurations of Crowdsourcing for the Emergency Management Domain," Computer Supported Cooperative Work (23:4), pp. 389-443 (2014).

31. Jakob Nielsen, The 90-9-1 Rule for Participation Inequality in Social Media and Online Communities. https://www.nngroup.com/articles/participation-inequality/ (2006).

32. Middelhoff, M., Widera, A., van den Berg, R., Hellingrath, B., Auferbauer, D., Havlik, D., Pielorz, J.: Crowdsourcing and Crowdtasking in Crisis Management Lessons Learned from a Field Experiment Simulating a Flooding in City of The Hague. In: The 3rd Int. Conference on Information Communication Technologies for Disaster Management (submitted).