# The Citizen Science Paradox

Benjamin Bergerot

► To cite this version:

Benjamin Bergerot. The Citizen Science Paradox. Land, 2022, 11 (8), pp.1151.
10.3390/land11081151. hal-03770465

HAL Id: hal-03770465

https://hal.science/hal-03770465v1

Submitted on 6 Sep 2022

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
Review

# The Citizen Science Paradox

Benjamin Bergerot ID

ECOBIO—UMR CNRS 6553, Univ. Rennes 1, F-35042 Rennes, France; benjamin.bergerot@univ-rennes1.fr; Tel.: +33-2-23-23-56-16

**Abstract:** Citizen science (CS) is now very popular in ecology. The number of scientific publications referencing CS has increased steadily over the past 15 years, with more than 1150 publications today. However, the multiplicity of research involved suggests that this number is highly underestimated. Based on this paradox, a literature review on CS shows that while its formalization in 2009 facilitated its referencing, about 70% of the publications are not referenced using CS as keyword. To understand this under-representation, an analysis of 149 publications related to the famous Christmas Bird Count program shows that this underestimation is not mainly related to the diversity of keywords used to describe CS, but rather to the fact that CS is mainly considered as a method (four publications out of five). The results also show that taking into account the whole text of a publication would represent a substantial improvement for the analysis of scientific databases, whatever the field of research.

**Keywords:** citizen science; amateur; volunteer; observer; monitoring program; Christmas Bird Count

## 1. Introduction

The production of scientific knowledge based on data collected by volunteers (amateurs or not) is not recent. Many collections in natural history museums owe much to amateur naturalists of past centuries. Research areas such as astrology and biology have strongly involved the effort of volunteers since the end of the 19th century [1]. In 1874, the British government founded the project "the transit of Venus" to measure the distance from the earth to the sun with the help of sailors and astronomers (Ratcliff, 2008). In ecology, historically, sciences involving volunteers have played an important role, particularly in ornithology [2]. A first mention dating from 1749 is referenced for the counting of migratory birds by gathering amateurs in Finland [2]. In 1879, the United States Geological Survey requested the involvement of volunteers, resulting in the well-known Breeding Bird Survey [3], officially launched in 1966 in North America. In the UK, the British Trust for Ornithology was founded in 1932 with the aim of using ornithologists' observations for science and species conservation [4]. One of the most documented examples is the Christmas Bird Count in the United States, initiated in 1900 [5]. This project brings together tens of thousands of observers for a total of 63 million birds counted [6]. Currently, on a global scale, the majority of CS projects in ecology are animal-focused (83%) and 20% are multi-taxonomic. Of the projects focusing on a single taxonomic group, 24% are intersects (non-lepidopterans), 19% birds, 17% plants, 12% lepidopterans and 9% mammals [7]. Experiments carried out with the help of non-scientific volunteers contribute to 55% of the registrations in the Global Biodiversity Information Facility database [7], which includes 2.1 billion of occurrence data (February, 2022). Approximately 1.3 million volunteers participate in scientific programs around the world, an estimated in-kind contribution of $2.5 billion [8].

In science, this recognition of the involvement of volunteers in research programs was formalized by the term "citizen science" (CS) in the 90s [9,10]. The first conceptualization carried out by Irwin (1995), a science policy analyst, differs from Bonney (1996), a member of the Cornell Ornithology Laboratory [11]. In his book, Irwin (1995) explains that CS must take into account the bidirectionality of the relationship between scientists and citizens. CS

<figure><img src="image_2.png" /></figure>

### check for updates

**Citation:** Bergerot, B. The Citizen Science Paradox. Land 2022, 11, 1151.
https://doi.org/10.3390/land11081151

Academic Editors: María Fe Schmitz, Cristina Herrero-Jáuregui and Cecilia Arnaiz Schmitz

Received: 4 July 2022

Accepted: 22 July 2022

Published: 26 July 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

<figure><img src="image_3.png" /></figure>

**Copyright:** © 2022 by the author.
Licensee MDPI, Basel, Switzerland.
This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
is a science that must serve the interests of citizens (“Science for people”, [12]) but in which citizens must be involved (“Science by the people”). What Irwin (1995) said reflects more a discussion of the ideals and limits of voluntary participation during the 70s than on current scientific knowledge production practices. Bonney (1996) defines CS as a scientific project in which volunteers participate (by providing observational data) but also as a tool to make volunteers aware of science (“a two-way street”). However, these two paradigms converge towards three formalized trends [13], which are the production of new scientific knowledge, the disclosure of science and the transformation of the relationship between science and society. A new and relatively consensual paradigm therefore makes it possible to define CS as a scientific project involving a partnership with volunteers, both novices and experts, in the generation of new knowledge [14–19]. In the literature, these volunteers are considered unpaid contributors who are not professional scientists and do not claim to have a level of scientific expertise. In order to avoid confusion related to the different terms used implying the participation of volunteers in the scientific literature (such as “public participation in scientific research”, “volunteer-based monitoring”, “citizen science”, “participatory science”), the use of the term CS was recommended [20]. The formalization of CS in scientific projects has also made it possible to better define the role of volunteers [18,20], and thus to encourage their commitment [21].

The growing success of CS in ecology [22] can be explained because CS makes it possible to investigate research questions on spatial and temporal scales unattainable with scientists only [1,23,24]. Programs involving volunteers such as the Breeding Bird Survey [3], the Christmas Bird Count [5] or Vigie Nature in France [25,26] have made it possible to acquire biotic and abiotic data at national and continental scales [7]. These scales make it possible to highlight ecological phenomena linked in particular to global changes such as global warming, invasive species, species conservation, environmental restoration, population or community ecology, and many other environmental concerns [7, 16, 17, 22, 27]. For scientists, one of the major interests of CS is therefore the access to many types of data on ecosystems, allowing the quantification of its ecological status (e.g., through population monitoring [29]), impact studies (e.g., pollution [30]) or the analysis of adaptive management (e.g., [31]). More specifically, CS allows for the characterization of the composition of an ecosystem (e.g., monitoring of indicator species, species with a special conservation status, invasive species, etc. [32–34]), its structure (e.g., monitoring of communities, keystone species, prey–predator relationships, etc. [35–37]), its conservation status (e.g., monitoring of species with special conservation status, invasive species, etc., [38]) and its management. CS also allows for the study of specific ecological processes (e.g., species–environment relationships, nutrient cycling, etc. [29,39,40]). Another interest of CS is the sampling of diverse environments. Their democratization includes agricultural environments (e.g., [41]), alpine environments (e.g., [42]), polar environments (e.g., [30]), desert environments (e.g., [43]), freshwater environments (e.g., [44]), marine environments (e.g., [45]), aerial environments (e.g., [46]), or the ground (e.g., [47]). Some environments are even inaccessible without the help of volunteers, such as private spaces in urban areas (e.g., [26]). Indeed, urban ecology has been developing for several years [48], and many green spaces supporting biodiversity are privately owned, especially in residential areas [1]. CS is therefore an interesting alternative for scientists who want to access a large amount of data on areas that are not usually accessible [49,50]. Although CS allows for the study of phenomena on large spatial and temporal scales, in some cases, the time series are short. One of the strengths of CS is that it can rely on the principle of replacing time with space [51–53]. Indeed, it is possible to identify a number of sites that collectively represent the range of variability in environmental conditions that can be found over a long period of time for a single site [54,55]. This principle allows for the study of long ecological processes on a short time scale [51].

CS also presents a pedagogical and decisional interest. CS allows for the construction of a common knowledge through the involvement of volunteers in a process of learning new knowledge. It also promotes the acquisition of new skills and improves the expertise
capacities of participants [16]. This learning process is universal and applies to different kinds of people (e.g., novices [56], stakeholders, enlightened or expert volunteers [57,58]) and in different institutional structures (e.g., associations, schools, etc. [59–62]). However, as in all research, the knowledge and information generated by CS is constructed in particular socio-economic and political contexts. It therefore depends on the nature of the questions asked, the methods used, as well as how and why the scientists developed the protocols [63,64]. The success of CS therefore lies in informing and sensitizing volunteers, but also in integrating them as CS actors to encourage them to think about the approaches used and the results [65]. CS programs engaged in monitoring biodiversity at different taxonomic levels and/or abiotic variables at different spatiotemporal scales, such as essential biodiversity variables [66], are considered essential to facilitate the transcription of academic results to field actors [67,68].

The involvement of volunteers and the amount of scientific data generated by CS are nevertheless facing a paradox. When searching for the term CS in international databases such as the ISI Web of Science database, until 2010, very few publications are referenced compared to the large number of existing CS programs [7]. This result is counter-intuitive, as several hundred scientific publications are derived from data collected through programs such as the Christmas Bird Counts or the Breeding Bird Survey prior to 2010 [3,16,27]. Evidently, the proportion of CS data in published research results is much higher than is recognized [69] (Figure 1).

<figure><img src="image_4.png" /></figure>

**Figure 1.** Figure showing the main steps of citizen science formalization according to the literature and following the evolution of the number of scientific publications referenced in the Web of Science core collection database (histogram on the left). The histogram represents the number of scientific publications published (extracted in March 2022) per year including “citizen science” (red bars, CS only, Table 1) in their title, abstract or keywords according to the Web of Science core collection database. The black bars represent the number of publications found using 30 keywords (All, Table 1). The grey bars represent the number of publications found using the 29 keywords (All without CS, Table 1). H1 and H2 represent the two main hypotheses allowing understanding of the citizen science paradox. CS: citizen sciences.
<table><caption>**Table 1.** Number of publications associated with the searches performed in the Web of Science Core collection database on 24 March 2022 (period: 1900–2022).</caption><thead><tr><th>Search Terms on 24 March 2022 (Period: 1900–2022)</th><th>Number of Publications (without/with the Condition: And “Ecology”)</th><th>Percentage of Publications Published after 2009</th><th>Denomination Used in the Histogram Legend in Figure 1</th></tr></thead><tbody><tr><td>"public * engagement" OR "amateur * engagement" OR "citizen * engagement" OR "observer * engagement" OR "volunteer * engagement" OR "public* involve *" OR "amateur* involve *" OR "citizen* involve *" OR "observer * involve *" OR "volunteer * involve *" OR "public * participation" OR "amateur * participation" OR "citizen * participation" OR "observer* participation" OR "volunteer * participation" OR "public * monitoring" OR "amateur * monitoring" OR "citizen* monitoring" OR "observer * monitoring" OR "volunteer * monitoring" OR "volunteer * based monitoring" OR "participatory monitoring" OR "community monitoring" OR "monitoring scheme *" OR "monitoring program *" OR "participat * action research *" OR "participatory science*" OR "community science *" OR "citizen science *"</td><td>65,428/4408</td><td>84.14% of 4408 publications</td><td>All (4408)</td></tr><tr><td>"public * engagement" OR "amateur* engagement" OR "citizen * engagement" OR "observer * engagement" OR "volunteer * engagement" OR "public* involv*" OR "amateur* involv*" OR "citizen * involve *" OR "observer * involve *" OR "volunteer * involve *" OR "public * participation" OR "amateur* participation" OR "citizen * participation" OR "observer * participation" OR "volunteer * participation" OR "public * monitoring" OR "amateur * monitoring" OR "citizen* monitoring" OR "observer * monitoring" OR "volunteer * monitoring" OR "volunteer * based monitoring" OR "participatory monitoring" OR "community monitoring" OR "monitoring scheme *" OR "monitoring program *" OR "participat * action research *" OR "participatory science *" OR "community science *"</td><td>60,030/3461</td><td>80.26% of 3461 publications</td><td>All without CS (3461)</td></tr><tr><td>"citizen science *"</td><td>6624/1161</td><td>98.45% of 1161 publications</td><td>CS only (1161)</td></tr></tbody></table>

Asterisk (*) has been used to truncate and find the spelling variation of a specific word (e.g., plural, noun, verb). Quotation marks ("”) have been used to find exact phrases. The use of the operator "OR" retrieves publications that contain at least one of the terms in the title, keywords or abstract. CS: citizen science.

To elucidate this paradox and based on a literature review, an analysis of the representativeness of CS was conducted. In the literature, CS is not seen as a science in itself but rather as a new way of approaching science. It is therefore useful to analyze whether the term CS is used in publications (especially since 2009), and if not, which keywords have been used instead. Secondly, to explain why CS is underrepresented in the scientific literature, two non-exclusive hypotheses were tested (Figure 1) by reviewing 149 scientific publications related to the famous Christmas Bird Count CS program. The first hypothesis (H1) lies in the terminology used before the formalization of the CS made by [17]. Indeed,
there is an abundance of keywords that does not make it easy to identify the publications related to CS. The second hypothesis (H2) lies in the fact that CS could be seen more as a method or a tool [7,69]. Based on these two hypotheses, it was checked (1) whether or not the authors used specific keywords related to CS and (2) in which sections of the publication these keywords were used, if they were used.

## 2. Materials and Methods

A first approach based on CS-related keywords was carried out in a literature search to analyze the number of citations in the ISI Web of Science database, before and after 2009, and using different combinations of CS-related keywords. This first part of the analysis allows for quantifying the number of citations on the subject in the scientific literature. The second approach focuses on a bibliographic search for CS-related keywords based on Audubon's Christmas Bird Count program to understand why CS is underrepresented in the scientific literature.

### 2.1. *Literature Review Based on CS Keywords*

Before the formalization of the CS by [17,20], it is therefore necessary to list a set of terms that can be used to account for the inclusion of publications incorporating CS. Based on scientific publications [13,70] and personal expertise, fifteen keywords were used to identify publications related to CS in the Web of Science Core collection database. These keywords are: "public", "amateur", "citizen", "observer", "volunteer", "engagement", "involve", "participation", "participatory", "monitoring", "community", "scheme", "program", "action research" and "science". Many words used alone do not refer specifically to CS. This is the case for "science" or "citizen", for example. After several tests in the Web of Science Core collection database, combinations of these 15 keywords were used to develop a relevant literature search based on 29 combinations (Table 1). Finally, an association of 29 keyword combinations with the term "ecology" was made in order to focus the results. The search was completed on 24 March 2022.

### 2.2. *Audubon's Christmas Bird Count Review*

Audubon's Christmas Bird Count (CBC) program provides an exceptional dataset for unravelling the spatiotemporal drivers of bird occurrence because data have been systematically collected by volunteers on hundreds of species throughout North America since 1900. As the CBC is a CS program, by definition, all scientific publications mentioning this program should be related, at least partly, to one or more CS keywords. Analyzing publications based on CBC allows us to understand how CS is perceived scientifically in this emblematic subsample.

A search in the Web of Science core collection database was made with Christmas Bird Count keywords. The search yielded 171 results on 24 March 2022. Books (3) and chapters (2) were excluded. The relevance of the 166 remaining items was assessed by taking into consideration whether CBC data were used for analyses, and 149 publications were kept (Supplementary Table S1). For each publication, it was checked whether the 15 predefined keywords and the 29 associated combinations had a direct and unequivocal relationship with CS. The word "people" was also checked. This word was not previously included because it is a generic term leading to a high number of selected publications (1,283,534 results when "people" was added to the 29 keyword combinations; 19,710 when added to the 29 keyword combinations and "ecology").

Then, the section of each keyword was referenced (e.g., title, abstract, keyword, introduction, materials and methods, results, discussion, including conclusion where appropriate, and acknowledgements). To check whether CS was seen more as a method or a tool, the additional keywords "tool", "database", "dataset" and "data set" were also screened.
### 3. Results

The use of the 29 keyword combinations associated with the term “ecology” in the Web of Science core database returns 3461 publications (Table 1). The addition of the “citizen science” keyword increases the number of publications to 4408 (84.14% are published after 2009). Of the 947 additional publications, 214 publications have at least one of the 29 keyword combinations and “citizen science”. The use of the keyword “citizen science” alone and associated with the term “ecology” in the Web of Science core database returns 1161 publications. A total of 98.45% of these publications were published after 2009 (Table 1).

Focusing on the titles, abstracts and keywords of the 149 publications using CBC data, 101 publications had none of the 15 predefined keywords and the additional keywords “people”, “dataset, data set and database”. This represents 67.78% of the publications. If all parts of the publications are taken into consideration (i.e., title, keywords, abstract, introduction, materials and methods, results, discussion and acknowledgements), 93.23% have at least one of the keywords directly related to CS. A total of 10 publications do not present any of these keywords (6.71%).

More specifically, the three main keywords are “observer”, “database, dataset and data set” and “volunteer”, which are found in 59.73%, 53.02% and 49.66% of the publications, respectively (Table 2). More than one paper out of two contains at least one of these keywords, directly related to the sampling method (“observer”/“volunteer”) or as a dataset (“dataset”/“data set”/“dataset”).

<table><caption>**Table 2.** Occurrences of the 15 keywords related to citizen science and the 5 additional keywords (people, dataset, data set, database and program) in the 149 publications using Christmas Bird Count data. Numbers indicate the number of publications concerned by each of the keywords according to the section (T: Title, A: Abstract, Kw: Keyword, I: Introduction, M: Materials and Methods, R: Results, D: Discussion/Conclusion, Ak: Acknowledgements). The numbers in bold indicate the number of publications concerned for a given keyword (last column) or a given section (last row).</caption><thead><tr><th>Keywords</th><th>T</th><th>A</th><th>Kw</th><th>I</th><th>M</th><th>R</th><th>D</th><th>Ak</th><th>Number of Publications</th></tr></thead><tbody><tr><td>Observer</td><td>1</td><td>10</td><td></td><td>15</td><td>71</td><td>24</td><td>30</td><td>7</td><td>89</td></tr><tr><td>Database/dataset/data set</td><td></td><td>9</td><td></td><td>26</td><td>66</td><td>22</td><td>28</td><td>6</td><td>79</td></tr><tr><td>Volunteer</td><td>1</td><td>7</td><td>3</td><td>11</td><td>35</td><td>5</td><td>10</td><td>48</td><td>74</td></tr><tr><td>Monitoring/monitor</td><td>4</td><td>9</td><td>5</td><td>22</td><td>22</td><td>7</td><td>22</td><td>2</td><td>49</td></tr><tr><td>Participant</td><td></td><td>4</td><td></td><td>8</td><td>27</td><td>9</td><td>14</td><td>13</td><td>48</td></tr><tr><td>Citizen</td><td>7</td><td>16</td><td>15</td><td>19</td><td>16</td><td>6</td><td>14</td><td>10</td><td>36</td></tr><tr><td>People</td><td></td><td>1</td><td></td><td>6</td><td>12</td><td>5</td><td>14</td><td>4</td><td>36</td></tr><tr><td>Program</td><td>1</td><td>8</td><td></td><td>16</td><td>16</td><td>7</td><td>10</td><td>3</td><td>30</td></tr><tr><td>Participation/participate/participatory</td><td></td><td>1</td><td></td><td>7</td><td>14</td><td>4</td><td>12</td><td>4</td><td>27</td></tr><tr><td>Involvement/involve/involving</td><td></td><td>1</td><td></td><td>6</td><td>9</td><td></td><td>5</td><td>3</td><td>19</td></tr><tr><td>Public</td><td></td><td>3</td><td></td><td>9</td><td>7</td><td>1</td><td>5</td><td>1</td><td>15</td></tr><tr><td>Engagement/engage</td><td></td><td>2</td><td></td><td>7</td><td>2</td><td>2</td><td>5</td><td></td><td>10</td></tr><tr><td>Amateur</td><td></td><td>1</td><td></td><td>3</td><td>4</td><td>1</td><td>3</td><td></td><td>9</td></tr><tr><td>Tool</td><td></td><td>2</td><td></td><td>2</td><td>2</td><td></td><td>5</td><td></td><td>7</td></tr><tr><td>Scheme</td><td></td><td>2</td><td></td><td>2</td><td>3</td><td>2</td><td>3</td><td></td><td>5</td></tr><tr><td>Community science</td><td></td><td>1</td><td>2</td><td>1</td><td>1</td><td></td><td>2</td><td>1</td><td>3</td></tr><tr><td>Action research</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td>0</td></tr><tr><td>Number of publications</td><td>11</td><td>41</td><td>17</td><td>64</td><td>116</td><td>59</td><td>81</td><td>72</td><td></td></tr></tbody></table>

The sections with the most keywords are the materials and methods, the discussion and the acknowledgements (Table 2). By searching the 15 predefined keywords and the additional keywords “people, dataset, data set and database”, 116 (78.85%), 81 (54.36%) and 72 (48.32%) publications were found through an analysis of the materials and methods, the discussion and the acknowledgements, respectively (Figure 2).
<figure><img src="image_5.png" /><figcaption>**Figure 2.** Histogram representing the number of keywords related to CS found in the 149 CBC publications according to their citation section.</figcaption></figure>

**4. Discussion**

Today, CS in the scientific community faces a paradox. On the one hand, programs involving volunteers such as the Breeding Bird Survey [3], the Christmas Bird Count [5] or Vigie Nature in France [25,26] have made it possible to acquire biotic and abiotic data on national and continental scales to highlight ecological phenomena related to global changes (e.g., global warming, invasive species, species conservation, environmental restoration and many other environmental concerns [7,16,17,27,28]). One of the main interests of CS is therefore the access to many types of data on ecosystems, allowing for quantification of their ecological status (e.g., through population monitoring [29]), impact studies (e.g., pollution [30]) or analyses of adaptive management [31]. More specifically, CS makes it possible to characterize the composition of an ecosystem (e.g., monitoring indicator species, species with special conservation status, invasive species, etc., [32–34,38,71]), its structure (e.g., community monitoring, keystone species, prey–predator relationships, etc., [26,37]) or ecological processes (e.g., species–environment relationships, nutrient cycling, etc., [29,39,40]). On the other hand, there are not many publications on CS in ecology (4408) compared to the research areas involved. More specifically, the term CS is rarely cited before 2009 (98.45% after 2009), and only 19.74% of papers related to CS using other keywords are found before 2009. The formalization of CS [17] seems to be effective and has facilitated the indexing of scientific citations since 2010. However, by using the term CS only, just over a quarter (26.33%) of the publications are found. The keyword CS alone is thus not sufficient to report on scientific publications using CS. While it is logical to find more publications after 2009 because of the formalization of the CS, more paradoxical is the fact that few publications referring to CS programs (but using CS-related keywords) emerged from literature searches before 2009 [72].

In order to understand why CS seems to be under-cited in the scientific literature, the flagship of the Audubon's Christmas Bird Count (CBC) CS program was used. As the CBC is a CS program, by definition, all scientific publications mentioning this program should be
related, at least partly, to one or more CS keyword. Using 15 keywords and 29 predefined combinations, 67.78% of the publications (101/149) do not refer to keywords related to CS. It is therefore not by multiplying the number of keywords or keyword combinations in a bibliographic search that the majority of publications related to CBC can be obtained. In other words, based only on keywords related to CS, approximately 70% of the publications related to CBC will not be found. The hypothesis that the multitude of keywords would explain the under-representation in terms of citations of CS-related publications [69] is therefore not valid.

The main explanation of such a result comes from the method of referencing publica-
tions in bibliographic databases such as the Web of Science core collection itself. Indeed,
this database refers to keywords belonging to titles, abstracts and keywords. A rigorous
analysis of the whole text (including title, abstract, keywords, introduction, materials and
methods, results, discussion and acknowledgements) shows that when considering all
parts, 93.29% (139/149) of the publications refer to one or more CS-related keyword. More
precisely, 77.85% of the publications refer to a keyword relating to CS in the materials and
methods section (i.e., almost four publications out of five). Following our second hypothesis
and as suggested by some authors [7,69], CS is thus mainly considered a tool for scientists.
It should be noted that 10 publications do not refer to any CS-related keywords (not even
in the acknowledgements section). For these publications, without knowledge of the CBC,
an uninformed reader would not know that the data used came from a CS program.

A potentially explanatory but untestable factor may also explain the limited number
of scientific publications on CS. Beyond scientific referencing, the evolution of the roles of
participants in CS [18] allows for the integration of Community Based Monitoring (CBM)
in CS. CBM is described as “community science” or “participatory action research” [73].
CBM is supported directly by stakeholders and/or volunteers, not scientists [74]. They
use their own resources to monitor, record and respond to specific environmental issues
(e.g., [74–76]). For stakeholders, CS through the CBM has made it possible to initiate projects
that scientists would not have initiated by themselves. This is mainly due to the type of
question asked, but also due to the geographical location and the scale concerned. Some
stakeholders (e.g., watershed managers, park and reserve managers, green spaces, etc.)
need to have a holistic approach for understanding their own ecosystems and associated
ecological functions in order to provide adequate expertise and adaptive management
actions [21,77,78]. However, CS developed on large scales are often unsuitable for these
stakeholders [79]. Many local-scale projects have therefore been initiated by volunteers,
both professionals and non-professionals, to address their environmental concerns [79].
These projects can focus on the cause of an environmental problem, such as pollution,
“suspicious” animal deaths, the spread of invasive species or the conservation of species of
interest for their heritage [80]. As an example, the “Save our streams” program, initiated
in 1969 to monitor, protect and restore streams in the state of Maryland (USA), served as
a model for launching the national scientific program Izaak Walton League of America.
This new type of approach no longer reflects the traditional perception of scientists using
volunteers to collect data, but rather refers to a two-way process that goes beyond the
mere production of information and data [14,81]. Many CBM results are available through
reports (“grey” literature) or dedicated websites and are thus not published in scientific
journals [79,82].

In this review, the ISI Web of Science database was used. There is another widely
used scientific database, Scopus. ISI Web of Science and Scopus are the most widespread
databases on different scientific fields which are frequently used for searching the literature.
There are still debates about the use of Web of Science or Scopus, depending on the field
and the research objectives [83]. Scopus includes scientific journals, books and conference
proceedings, covering research topics across all scientific and technical disciplines, ranging
from medicine and social sciences to arts and humanities. Scopus is more inclusive and
contains more types of material, and therefore, it is less selective. ISI Web of Science is the
oldest citation database, having strong coverage with citation data and bibliographic data
which goes back to 1900. As the CBC appeared in 1900, that is why this review used ISI Web of Science rather than Scopus. It is nevertheless important to consider that the use of Scopus could have led to different results, mainly for the last few years, with potentially complementary publications related to CS.

**5. Conclusions**

The CS paradox highlights the limits of formalizing a recent scientific trend by keywords and referencing them in scientific databases in order to conduct an exhaustive bibliographic search (Question 1). Even with multiple keywords (Hypothesis 1), it is the referencing of specific parts of publications themselves in search engines (e.g., title, abstract and keywords) that generates a potential under-representation of publications. This review shows that while formalizing CS improves its referencing, CS is mainly considered as a method (Hypothesis 2). There are, however, two major elements that may limit the conclusions of this study. The first is the choice of bibliographic database (ISI Web of Science), which, although justified by its historical aspect, can be complemented by other databases (e.g., Scopus). The second is the question of the integration of the “grey” bibliography (CBM) and the way in which this existing literature considers CS. Despite the limitations of this study, the results show that basing searches only on titles, abstracts and keywords greatly underestimates the number of scientific publications related to CS. One way of avoiding a bias such as the CS paradox would be to improve the consideration of the whole text in bibliographic searches, in particular by allowing for the selection of specific parts (e.g., materials and methods or discussion). By taking this aspect into account, bibliographic database managers would prevent the under-representation of specific research areas solely because of the referencing system. It should be taken into account that many CS results are also published in the grey literature or are not published, as is the case for many CBM results. The referencing of CS should therefore also be completed through more general databases, covering both academic and non-academic databases.

**Supplementary Materials:** The following supporting information can be downloaded at: https://www.mdpi.com/article/10.3390/land11081151/s1, Table S1: Table with the list of scientific publications referring to the CBC's citizen science program (171 publications). Of these 171 publications, 149 (in green) were selected for this review.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** The data supporting the results are directly accessible in the two tables of the article. The list of the 149 publications used is available in the supplementary material.

**Acknowledgments:** I would like to thank the three reviewers who helped to improve this work. I would like to thank the thousands of volunteers who make it possible every day for scientists to answer questions on spatial and temporal scales that they would never have had access to without citizen science.

**Conflicts of Interest:** The author declares no conflict of interest.

**References**

1. Dickinson, J.L.; Zuckerberg, B.; Bonter, D.N. Citizen Science as an Ecological Research Tool: Challenges and Benefits. *Annu. Rev. Ecol. Evol. Syst.* **2010**, *41*, 149–172. [CrossRef]

2. Greenwood, J.J.D. Citizens, Science and Bird Conservation. *J. Ornithol.* **2007**, *148*, 77–124. [CrossRef]

3. Dunn, E.; Johnson, D.; Jones, S.; Petit, D.; Pollock, K.; Smith, C.; Trapp, J.; Welling, E. *A Programmatic Review of the North American Breeding Bird Survey: Report of a Peer Review Panel to USGS Patuxent*; USGS: Reston, VA, USA, 2000.

4. Pearce-Higgins, J.W.; Crick, H.Q.P. One-Third of English Breeding Bird Species Show Evidence of Population Responses to Climatic Variables over 50 Years. *Bird Study* **2019**, *66*, 159–172. [CrossRef]

5. Butcher, G.S.; Fuller, M.R.; McAllister, L.S.; Geissler, P.H. An Evaluation of the Christmas Bird Count for Monitoring Population Trends of Selected Species. *Wildl. Soc. Bull.* **1990**, *18*, 129–134.
6. Butcher, G.S.; Niven, D.K. *Combining Data from the Christmas Bird Count and the Breeding Bird Survey to Determine the Continental Status and Trends of North America Birds*; National Audubon Society: New York, NY, USA, 2007.

7. Chandler, M.; See, L.; Copas, K.; Bonde, A.M.Z.; López, B.C.; Danielsen, F.; Legind, J.K.; Masinde, S.; Miller-Rushing, A.J.; Newman, G.; et al. Contribution of Citizen Science towards International Biodiversity Monitoring. *Biol. Conserv.* **2017**, *213*, 280–294. [CrossRef]

8. Theobald, E.J.; Ettinger, A.K.; Burgess, H.K.; DeBey, L.B.; Schmidt, N.R.; Froehlich, H.E.; Wagner, C.; HilleRisLambers, J.; Tewksbury, J.; Harsch, M.A.; et al. Global Change and Local Solutions: Tapping the Unrealized Potential of Citizen Science for Biodiversity Research. *Biol. Conserv.* **2015**, *181*, 236–244. [CrossRef]

9. Irwin, A. *Citizen Science: A Study of People, Expertise and Sustainable Development*; Psychology Press: London, UK, 1995; ISBN 978-0-415-11548-3.

10. Bonney, R. *Citizen Science: A Lab Tradition*. *Living Bird-Study Conserv. Birds* **1996**, *15*, 7–15.

11. Riesch, H.; Potter, C. Citizen Science as Seen by Scientists: Methodological, Epistemological and Ethical Dimensions. *Public Underst. Sci.* **2014**, *23*, 107–120. [CrossRef]

12. Hogben, L. *Science for the Citizen*, 1st ed.; Alfred A. Knopf: New York, NY, USA, 1938.

13. Strasser, B.; Baudry, J.; Mahr, D.; Sanchez, G.; Tancoigne, E. "Citizen Science"? Rethinking Science and Public Participation. *Sci. Technol. Stud.* **2018**, *32*, 52–76. [CrossRef]

14. Cornwall, A.; Jewkes, R. What Is Participatory Research? *Soc. Sci. Med.* **1995**, *41*, 1667–1676. [CrossRef]

15. Kruger, L.; Shannon, M. Getting to Know Ourselves and Our Places through Participation in Civic Social Assessment. *Soc. Nat. Resour.* **2000**, *13*, 461–478. [CrossRef]

16. Silvertown, J. A New Dawn for Citizen Science. *Trends Ecol. Evol.* **2009**, *24*, 467–471. [CrossRef]

17. Bonney, R.; Cooper, C.B.; Dickinson, J.; Kelling, S.; Phillips, T.; Rosenberg, K.V.; Shirk, J. Citizen Science: A Developing Tool for Expanding Science Knowledge and Scientific Literacy. *BioScience* **2009**, *59*, 977–984. [CrossRef]

18. Miller-Rushing, A.; Primack, R.; Bonney, R. The History of Public Participation in Ecological Research. *Front. Ecol. Environ.* **2012**, *10*, 285–290. [CrossRef]

19. Buytaert, W.; Zulkafli, Z.; Grainger, S.; Acosta, L.; Alemie, T.C.; Bastiaensen, J.; De Bièvre, B.; Bhusal, J.; Clark, J.; Dewulf, A.; et al. Citizen Science in Hydrology and Water Resources: Opportunities for Knowledge Generation, Ecosystem Service Management, and Sustainable Development. *Front. Earth Sci.* **2014**, *2*, 26. [CrossRef]

20. Bonney, R.; Ballard, H.; Jordan, R.; McCallie, E.; Phillips, T.; Shirk, J.; Wilderman, C.C. *Public Participation in Scientific Research: Defining the Field and Assessing Its Potential for Informal Science Education*; A CAISE Inquiry Group Report; ERIC: Washington, DC, USA, 2009.

21. Harvey, G.K.A.; Nelson, T.A.; Paquet, P.C.; Ferster, C.J.; Fox, C.H. Comparing Citizen Science Reports and Systematic Surveys of Marine Mammal Distributions and Densities. *Biol. Conserv.* **2018**, *226*, 92–100. [CrossRef]

22. Burgess, H.K.; DeBey, L.B.; Froehlich, H.E.; Schmidt, N.; Theobald, E.J.; Ettinger, A.K.; HilleRisLambers, J.; Tewksbury, J.; Parrish, J.K. The Science of Citizen Science: Exploring Barriers to Use as a Primary Research Tool. *Biol. Conserv.* **2017**, *208*, 113–120. [CrossRef]

23. Dickinson, J.L.; Shirk, J.; Bonter, D.; Bonney, R.; Crain, R.L.; Martin, J.; Phillips, T.; Purcell, K. The Current State of Citizen Science as a Tool for Ecological Research and Public Engagement. *Front. Ecol. Environ.* **2012**, *10*, 291–297. [CrossRef]

24. Hochachka, W.M.; Fink, D.; Hutchinson, R.A.; Sheldon, D.; Wong, W.-K.; Kelling, S. Data-Intensive Science Applied to Broad-Scale Citizen Science. *Trends Ecol. Evol.* **2012**, *27*, 130–137. [CrossRef] [PubMed]

25. Devictor, V.; van Swaay, C.; Brereton, T.; Brotons, L.; Chamberlain, D.; Heliölä, J.; Herrando, S.; Julliard, R.; Kuussaari, M.; Lindström, Å.; et al. Differences in the Climatic Debts of Birds and Butterflies at a Continental Scale. *Nat. Clim. Chang.* **2012**, *2*, 121–124. [CrossRef]

26. Fontaine, B.; Bergerot, B.; Le Viol, I.; Julliard, R. Impact of Urbanization and Gardening Practices on Common Butterfly Communities in France. *Ecol. Evol.* **2016**, *6*, 8174–8180. [CrossRef] [PubMed]

27. Silvertown, J.; Cook, L.; Cameron, R.; Dodd, M.; McConway, K.; Worthington, J.; Skelton, P.; Anton, C.; Bossdorf, O.; Baur, B.; et al. Citizen Science Reveals Unexpected Continental-Scale Evolutionary Change in a Model Organism. *PLoS ONE* **2011**, *6*, e18927. [CrossRef]

28. McKinley, D.C.; Miller-Rushing, A.J.; Ballard, H.L.; Bonney, R.; Brown, H.; Cook-Patton, S.C.; Evans, D.M.; French, R.A.; Parrish, J.K.; Phillips, T.B.; et al. Citizen Science Can Improve Conservation Science, Natural Resource Management, and Environmental Protection. *Biol. Conserv.* **2017**, *208*, 15–28. [CrossRef]

29. Devictor, V.; Julliard, R.; Couvet, D.; Jiguet, F. Birds Are Tracking Climate Warming, but Not Fast Enough. *Proc. R. Soc. B Biol. Sci.* **2008**, *275*, 2743–2748. [CrossRef]

30. Bergmann, M.; Lutz, B.; Tekman, M.B.; Gutow, L. Citizen Scientists Reveal: Marine Litter Pollutes Arctic Beaches and Affects Wild Life. *Mar. Pollut. Bull.* **2017**, *125*, 535–540. [CrossRef]

31. Stem, C.; Margoluis, R.; Salafsky, N.; Brown, M. Monitoring and Evaluation in Conservation: A Review of Trends and Approaches. *Conserv. Biol.* **2005**, *19*, 295–309. [CrossRef]

32. Delaney, D.G.; Sperling, C.D.; Adams, C.S.; Leung, B. Marine Invasive Species: Validation of Citizen Science and Implications for National Monitoring Networks. *Biol. Invasions* **2008**, *10*, 117–128. [CrossRef]
33. Crall, A.W.; Newman, G.J.; Jarnevich, C.S.; Stohlgren, T.J.; Waller, D.M.; Graham, J. Improving and Integrating Data on Invasive Species Collected by Citizen Scientists. *Biol. Invasions* **2010**, *12*, 3419–3428. [CrossRef]

34. Azzurro, E.; Broglio, E.; Maynou, F.; Bariche, M. Citizen Science Detects the Undetected: The Case of *Abudefduf Saxatilis* from the Mediterranean Sea. *Manag. Biol. Invasions* **2013**, *4*, 167–170. [CrossRef]

35. Nolan, V.; Reader, T.; Gilbert, F.; Atkinson, N. The Ancient Tree Inventory: A Summary of the Results of a 15 Year Citizen Science Project Recording Ancient, Veteran and Notable Trees across the UK. *Biodivers. Conserv.* **2020**, *29*, 3103–3129. [CrossRef]

36. Gizzi, F.; Jiménez, J.; Schäfer, S.; Castro, N.; Costa, S.; Lourenço, S.; José, R.; Canning-Clode, J.; Monteiro, J. Before and after a Disease Outbreak: Tracking a Keystone Species Recovery from a Mass Mortality Event. *Mar. Environ. Res.* **2020**, *156*, 104905. [CrossRef] [PubMed]

37. Bergerot, B.; Julliard, R.; Baguette, M. Metacommunity Dynamics: Decline of Functional Relationship along a Habitat Fragmentation Gradient. *PLoS ONE* **2010**, *5*, e11294. [CrossRef] [PubMed]

38. Marion, L.; Bergerot, B. Northern Range Shift May Be Due to Increased Competition Induced by Protection of Species Rather than to Climate Change Alone. *Ecol. Evol.* **2018**, *8*, 8364–8379. [CrossRef] [PubMed]

39. Milne, R.; Rosolen, S.; Whitelaw, G.; Bennett, L. Multi-Party Monitoring in Ontario: Challenges and Emerging Solutions. *Environments* **2006**, *34*, 11–24.

40. Devictor, V.; Whittaker, R.J.; Beltrame, C. Beyond Scarcity: Citizen Science Programs as Useful Tools for Conservation Biogeography. *Divers. Distrib.* **2010**, *16*, 354–362. [CrossRef]

41. Ryan, S.F.; Adamson, N.L.; Aktipis, A.; Andersen, L.K.; Austin, R.; Barnes, L.; Beasley, M.R.; Bedell, K.D.; Briggs, S.; Chapman, B.; et al. The Role of Citizen Science in Addressing Grand Challenges in Food and Agriculture Research. *Proc. R. Soc. B Biol. Sci.* **2018**, *285*, 1977–2018. [CrossRef] [PubMed]

42. McDonough MacKenzie, C.; Murray, G.; Primack, R.; Weihrauch, D. Lessons from Citizen Science: Assessing Volunteer-Collected Plant Phenology Data with Mountain Watch. *Biol. Conserv.* **2017**, *208*, 121–126. [CrossRef]

43. Barrows, C.W.; Hoines, J.; Vamstad, M.S.; Murphy-Mariscal, M.; Lalumiere, K.; Heintz, J. Using Citizen Scientists to Assess Climate Change Shifts in Desert Reptile Communities. *Biol. Conserv.* **2016**, *195*, 82–88. [CrossRef]

44. Abbott, B.W.; Moatar, F.; Gauthier, O.; Fovet, O.; Antoine, V.; Ragueneau, O. Trends and Seasonality of River Nutrients in Agricultural Catchments: 18years of Weekly Citizen Science in France. *Sci. Total Environ.* **2018**, *624*, 845–858. [CrossRef] [PubMed]

45. Carcia-Soto, C.; van der Meeren, G.I. *Advancing Citizen Science for Coastal and Ocean Research*; European Marine Board: Ostend, Belgium, 2017; ISBN 978-94-92043-30-6.

46. Laughlin, A.J.; Taylor, C.M.; Bradley, D.W.; Leclair, D.; Clark, R.C.; Dawson, R.D.; Dunn, P.O.; Horn, A.; Leonard, M.; Sheldon, D.R.; et al. Integrating Information from Geolocators, Weather Radar, and Citizen Science to Uncover a Key Stopover Area of an Aerial Insectivore. *Auk* **2013**, *130*, 230–239. [CrossRef]

47. Iannone, B.V.; Umek, L.G.; Wise, D.H.; Heneghan, L. A Simple, Safe, and Effective Sampling Technique for Investigating Earthworm Communities in Woodland Soils: Implications for Citizen Science. *Nat. Areas J.* **2012**, *32*, 283–292. [CrossRef]

48. McDonnell, M.J.; MacGregor-Fors, I. The Ecological Future of Cities. *Science* **2016**, *352*, 936–938. [CrossRef] [PubMed]

49. Mccaffrey, R. Using Citizen Science in Urban Bird Studies. *Urban Habitats* **2005**, *3*, 70–86.

50. Wei, J.W.; Lee, B.P.Y.-H.; Wen, L.B. Citizen Science and the Urban Ecology of Birds and Butterflies—A Systematic Review. *PLoS ONE* **2016**, *11*, e0156425. [CrossRef]

51. Hargrove, W.W.; Pickering, J. Pseudoreplication: A Sine qua Non for Regional Ecology. *Landsc. Ecol* **1992**, *6*, 251–258. [CrossRef]

52. Wiens, J.A. Spatial Scaling in Ecology. *Funct. Ecol.* **1989**, *3*, 385–397. [CrossRef]

53. With, K.A. *Essentials of Landscape Ecology*; Oxford University Press: Oxford, UK, 2019; ISBN 978-0-19-883838-8.

54. Ferrier, S.; Guisan, A. Spatial Modelling of Biodiversity at the Community Level. *J. Appl. Ecol.* **2006**, *43*, 393–404. [CrossRef]

55. Elith, J.; Leathwick, J. Predicting Species Distributions from Museum and Herbarium Records Using Multiresponse Models Fitted with Multivariate Adaptive Regression Splines. *Divers. Distrib.* **2007**, *13*, 265–275. [CrossRef]

56. Bergerot, B.; Fontaine, B.; Renard, M.; Cadi, A.; Julliard, R. Preferences for Exotic Flowers Do Not Promote Urban Life in Butterflies. *Landsc. Urban Plan.* **2010**, *96*, 98–107. [CrossRef]

57. Julliard, R.; Clavel, J.; Devictor, V.; Jiguet, F.; Couvet, D. Spatial Segregation of Specialists and Generalists in Bird Communities. *Ecol. Lett.* **2006**, *9*, 1237–1244. [CrossRef] [PubMed]

58. Bergerot, B.; Fontaine, B.; Julliard, R.; Baguette, M. Landscape Variables Impact the Structure and Composition of Butterfly Assemblages along an Urbanization Gradient. *Landsc. Ecol.* **2011**, *26*, 83–94. [CrossRef]

59. Cohn, J.P. Citizen Science: Can Volunteers Do Real Research? *BioScience* **2008**, *58*, 192–197. [CrossRef]

60. Lepczyk, C.; Boyle, O.; Vargo, T.; Gould, P.; Jordan, R.; Liebenberg, L.; Masi, S.; Mueller, W.; Prysby, M.; Vaughan, H. Citizen Science in Ecology: The Intersection of Research and Education. *Bull. Ecol. Soc. Am.* **2009**, *90*, 308–317. [CrossRef]

61. Roger, E.; Klistorner, S. BioBlitzes Help Science Communicators Engage Local Communities in Environmental Research. *JCOM-J. Sci. Commun.* **2016**, *15*, A06. [CrossRef]

62. Storksdieck, M.; Shirk, J.L.; Cappadonna, J.L.; Domroese, M.; Göbel, C.; Haklay, M.; Miller-Rushing, A.J.; Roetman, P.; Sbrocchi, C.; Vohland, K. Associations for Citizen Science: Regional Knowledge, Global Collaboration. *Citiz. Sci. Theory Pract.* **2016**, *1*, 10. [CrossRef]

63. McAllister, K. *Understanding Participation: Monitoring and Evaluation Process, Outputs and Outcomes*; Research Paper; International Development Research Center: Ottawa, ON, Canada, 1999.
64. Evans, C.; Abrams, E.; Reitsma, R.; Roux, K.; Salmonson, L.; Marra, P.P. The Neighborhood Nestwatch Program: Participant Outcomes of a Citizen-Science Ecological Research Project. *Conserv. Biol.* **2005**, *19*, 589–594. [CrossRef]

65. Clayton, S.; Myers, G. *Conservation Psychology: Understanding and Promoting Human Care for Nature*; Wiley-Blackwell: Chichester, UK; Hoboken, NJ, USA, 2009; ISBN 978-1-4051-7678-1.

66. Pereira, H.M.; Ferrier, S.; Walters, M.; Geller, G.N.; Jongman, R.H.G.; Scholes, R.J.; Bruford, M.W.; Brummitt, N.; Butchart, S.H.M.; Cardoso, A.C.; et al. Essential Biodiversity Variables. *Science* **2013**, *339*, 277–278. [CrossRef]

67. Díaz, S.; Demissew, S.; Carabias, J.; Joly, C.; Lonsdale, M.; Ash, N.; Larigauderie, A.; Adhikari, J.R.; Arico, S.; Báldi, A.; et al. The IPBES Conceptual Framework—Connecting Nature and People. *Curr. Opin. Environ. Sustain.* **2015**, *14*, 1–16. [CrossRef]

68. Dunkley, R.; Baker, S.; Constant, N.; Sanderson-Bellamy, A. Enabling the IPBES Conceptual Framework to Work across Knowledge Boundaries. *Int. Env. Agreem.* **2018**, *18*, 779–799. [CrossRef]

69. Cooper, C.B.; Shirk, J.; Zuckerberg, B. The Invisible Prevalence of Citizen Science in Global Research: Migratory Birds and Climate Change. *PLoS ONE* **2014**, *9*, e106508. [CrossRef]

70. Shirk, J.; Ballard, H.; Wilderman, C.; Phillips, T.; Wiggins, A.; Jordan, R.; McCallie, E.; Minarchek, M.; Lewenstein, B.; Krasny, M.; et al. Public Participation in Scientific Research: A Framework for Deliberate Design. *Ecol. Soc.* **2012**, *17*, 29. [CrossRef]

71. Crall, A.W.; Jarnevich, C.S.; Young, N.E.; Panke, B.J.; Renz, M.; Stohlgren, T.J. Citizen Science Contributes to Our Knowledge of Invasive Plant Species Distributions. *Biol. Invasions* **2015**, *17*, 2415–2427. [CrossRef]

72. Bonney, R.; Shirk, J.L.; Phillips, T.B.; Wiggins, A.; Ballard, H.L.; Miller-Rushing, A.J.; Parrish, J.K. Next Steps for Citizen Science. *Science* **2014**, *343*, 1436–1437. [CrossRef] [PubMed]

73. Pretty, J.N.; Guijt, I.; Thompson, J.; Scoones, I. *Participatory Learning and Action: A Trainer's Guide*; IIED Participatory Methodology Series; IIED: London, UK, 1995.

74. Whitelaw, G.; Vaughan, H.; Craig, B.; Atkinson, D. Establishing the Canadian Community Monitoring Network. *Environ. Monit. Assess.* **2003**, *88*, 409–418. [CrossRef] [PubMed]

75. Keough, H.L.; Blahna, D.J. Achieving Integrative, Collaborative Ecosystem Management. *Conserv. Biol.* **2006**, *20*, 1373–1382. [CrossRef]

76. Danielsen, F.; Pirhofer-Walzl, K.; Adrian, T.P.; Kapijimpanga, D.R.; Burgess, N.D.; Jensen, P.M.; Bonney, R.; Funder, M.; Landa, A.; Levermann, N.; et al. Linking Public Participation in Scientific Research to the Indicators and Needs of International Environmental Agreements. *Conserv. Lett.* **2014**, *7*, 12–24. [CrossRef]

77. Bergerot, B.; Lasne, E.; Vigneron, T.; Laffaille, P. Prioritization of Fish Assemblages with a View to Conservation and Restoration on a Large European Basin, the Loire (France). *Biodivers. Conserv.* **2008**, *17*, 2247–2262. [CrossRef]

78. Garrido, P.; Elbakidze, M.; Angelstam, P. Stakeholders' Perceptions on Ecosystem Services in Östergötland's (Sweden) Threatened Oak Wood-Pasture Landscapes. *Landsc. Urban Plan.* **2017**, *158*, 96–104. [CrossRef]

79. Conrad, C.C.; Hilchey, K.G. A Review of Citizen Science and Community-Based Environmental Monitoring: Issues and Opportunities. *Environ. Monit. Assess.* **2011**, *176*, 273–291. [CrossRef]

80. Firehock, K.; West, J. A Brief History of Volunteer Biological Water Monitoring Using Macroinvertebrates. *J. N. Am. Benthol. Soc.* **1995**, *14*, 197–202. [CrossRef]

81. Lakshminarayanan, S. Using Citizens to Do Science versus Citizens as Scientists. *Ecol. Soc.* **2007**, *12*, r2. [CrossRef]

82. Conrad, C.T.; Daoust, T. Community-Based Monitoring Frameworks: Increasing the Effectiveness of Environmental Stewardship. *Environ. Manag.* **2008**, *41*, 358–366. [CrossRef] [PubMed]

83. Chadegani, A.A.; Salehi, H.; Yunus, M.M.; Farhadi, H.; Fooladi, M.; Farhadi, M.; Ebrahim, N.A. A Comparison between Two Main Academic Literature Collections: Web of Science and Scopus Databases. *Asian Soc. Sci.* **2013**, *9*, p18. [CrossRef]