dataesr/scientific-paragraphs-categorization huggingface dataset:
2012 (Deguines et al. , 2016;; Desaegher et al. 2018; Levé, Baudry, and Bessa-Gomes 2019). Specifically, we computed linear regressions between the number of sessions and the population size in the 22 and 16 administrative regions of France and South Korea respectively, after both variables were ln(variable + 1) transformed. We used the rho coefficients retrieved from Spearman correlation tests and the coefficient of determination R² to compare the two regressions. As a second spatial metric, we assessed the data spatial dispersion for each participant who did at least three sessions. We assumed that the more "exploratory" participants are, the less the data have the risk to be spatially auto-correlated. For each participant, we calculated the median distance of each session to the centroid of his/her sessions.
We then performed a Wilcoxon test to evaluate whether participants from South Korea and France differed in their spatial dispersion of participation. For this test, sample size was the number of participants from both programs who did three sessions or more (i. e. , n = 165, with 95 and 70 participants for the Spipoll and the K-Spipoll respectively). The metric "Sample size" was assessed by the total number of sessions done for each program. The size of this metric determines the statistical power for data analyses. Moderately large datasets (e. g. , 1000-3000 sessions distributed across a broad geographical areas) allowed investigating macro-ecological dynamics of pollinator communities (e. g. ,  Deguines et al. 2016, Levé, Baudry, and Bessa-Gomes 2019). However, species distribution modelling for a given species of interests could be done with as little as ca.
These two metrics are critical to increase the sample size and potentially the spatial representation. Wilcoxon tests were used to test whether these two metrics differed between participants in the Spipoll and the K-Spipoll. We further considered four additional metrics of participant's engagement: The number of participants, the proportion of participants contributing a single session, the average number of sessions per participant, and the distribution of numbers of contributions per participant in each program (allowing determination of the proportion of participants contributing to 50% of the data). These last metrics are also linked to the sample size. More particularly, the number of single sessions is further linked to temporal issues. Indeed, single participations are not ideal for assessing temporal trends.
Data quality In general, the Spipoll program had better results than the K-Spipoll program (Table  2 ). In regard to accuracy in data collection, we found that 57% of the Spipoll sessions followed the protocol in contrast to only 26% of K-Spipoll sessions. The strict violations of the protocol (non-georeferenced picture, pictures taken on different plant species or on leaves, no insect on the picture) were the most common source of non-respect of the protocol (29% of the sessions for Spipoll and 39% for the K-Spipoll). The proportion of single-species sessions was greater for the K-Spipoll (35%) than for the Spipoll (14%). This highlights the fact that these participants either misunderstood the protocol (which states to upload one picture per insect) or did not observe during the 15 or 20 minute period.
The percentage of sessions containing only one species decreased over time for the participants of the Spipoll. Participants having uploaded more than 10 sessions had only 15. 7% of single-species sessions, compared to 31. 6% for participants having uploaded only one session (Figure  1 ). In contrast, the amount of single-species sessions with K-Spipoll did not decrease with the volume of sessions contributed by an individual. The spatial distribution of sessions was more highly correlated to the spatial distribution of population in the Spipoll program (R² = 0. 82; p-value < 0. 0001) than in the K-Spipoll program (R² = 0. 49; p-value = 0. 0025) (Figure  2 ). Conversely, observations by K-Spipoll participants were more spread out in the landscape (median distance = 15. 3 km) than observations by Spipoll observers (median distance = 1.
Participant engagement K-Spipoll participants demonstrated greater engagement across metrics than Spipoll participants (Table  2 ). The number of connected days and the number of active days were greater for the participants of the K-Spipoll program with 70. 8 (±66. 5) connected days in average (n = 87) against 17. 2(±24. 5) for the Spipoll program (n = 417) (Figure  4 ; Wilcoxon tests associated p-values < 2. 2e-16, Z = -8. 15 and Z = -8. 31 respectively). With 75% fewer participants, the K-Spipoll reached almost the same number of sessions as the Spipoll, as    K-Spipoll participants each uploaded 13 sessions in average, compared to 2. 8 sessions in average for participants of the Spipoll. Additionally, the proportion of single participation is lower in the K-Spipoll (17. 1%) than the Spipoll (60.
8%). The number of observations per participant and their contribution to the entire dataset is represented in Figure  5 , which shows that for K-Spipoll, 13% of participants (i. e. , 13 observers) are contributing to 50% of the dataset. These participants did 26 sessions each or more. For the Spipoll, 10% of participants (i. e. , 42 observers) collected 50% of the data, each doing 5 sessions or more.
Discussion The comparison of Spipoll and K-Spipoll showed that both methodologies had strengths and weaknesses regarding the different metrics we used to assess data quality and participant engagement. The Spipoll program is providing data of high quality regarding specifically the accuracy of data collection and the sample size usable to conduct analyses, with most participants abiding by the standard protocol and providing identification for the photographed insects. As a result, this program successfully published analyses about contrasted affinities of pollinators with different land-use  (Deguines et al. 2012) , urbanization effects on community composition  (Deguines et al. 2016) , and more recently works about floral morphology as the main driver of flower-feeding insect occurrences in the Paris region  (Desaegher et al.
Some analysis using presence-only data to conduct Species Distribution Modeling and modeling of ecological networks are nevertheless possible where the sample size is large enough. We showed that consistency in protocol between individuals and over time was progressing for the Spipoll, as the number of single-species sessions decreased after several participations, showing that the participants were understanding the protocol better after several participations, but this was not the case for the K-Spipoll. The K-Spipoll program showed more efficiency for the participants' engagement, a full participation demanding less effort in terms of data input. Participants were "connected" to the In each panel, dotted lines represent the median value for Spipoll and K-Spipoll (in blue and red respectively).
project for a longer period, were participating more often (number of "active days"), and uploaded more sessions that were also spatially more widespread in K-Spipoll than the Spipoll. A "main contributor" (defined as being among the most active participants contributing to 50% of the dataset) for the K-Spipoll sent at least 26 sessions, whereas such contributors in the Spipoll did five sessions or more. Thus, the proportion of main contributors was slightly highest for the K-Spipoll (13% of participants) than for the Spipoll. The strong commitment of the participants of the K-Spipoll is encouraging in terms of long-term participation and to address the temporal monitoring aims of the program. We discuss below how these differences could have emerged, and provide suggestions for the implementation of future citizen science programs.
The social network vs. on-field activities We suggest that the community management of the social network dedicated to the Spipoll program drove participant respect of the protocol. Indeed, a community manager provided online personalized and constructive feedback (which could be seen by everyone) on each observation with a misidentified insect or that appeared to violate the protocol. These comments aimed to give participants some tips to better identify insects and to better follow the protocol. As a result, participants soon started to critically assess newly uploaded contributions, leaving comments to remind authors of "suspicious" contributions about the standardized protocol and to explain the importance of abiding with it. This eventually led to a self-managed community that likely contributed to participants quickly learning the importance of following the standardized protocol for the sake of scientific research.
The development of digital applications on smartphones could thus be considered as a way to develop future citizen science programs. Smartphones can easily be used to collect data, thanks to all the tools integrated such as digital camera or microphones, which have been used to monitor treefrog habitat preferences in South Korea  (Roh, Borzée, and Jang 2014). External devices can be used to improve the quality of the recording, such as ultrasonic microphones, used by the program iBat  (Gibb, Mac, and Jones 2016). However, to control the sending of accurate data, some features could be directly implemented on the application, such as protocol reminder questions ("Have you completed the required time of observation?" K-Spipoll has now been updated to ask this question in an attempt to improve the quality of the data); tick boxes to choose the best picture(s); and automatic identification allowing a first classification (i.
Conclusion If the development of new technologies and digital applications can be seen as a convenient way to collect a large amount of data, implementation of controls at the stage of data collection is critical to ensure data quality and, therefore, the possibility to use these data to address ecological research questions. This paper showed that the process and methodology of the Spipoll program ensured that data collection was optimal for their analysis. This has been proven by the research papers published thanks to these data  (Deguines et al. 2012; Deguines et al. 2016; Desaegher et al. 2018; Levé, Baudry, and Bessa-Gomes 2019). The K-Spipoll process and methodology were more efficient to engage people to participate. The strong commitment of the observers is promising for the future of this program, for which data collection has been enhanced by adding controls into the application.
Acknowledgements We would like to thank our partner  Donga Science  and all the participants from France and South Korea who participated in the two programs. We also want to warmly thank  Dr. Lee Heung-Sik  from  Plant Quarantine Technology Center and Wan-Hyeok Park  and  Soo-Jeong Cho  for their contributions to pollinator identification and the several hours they passed in the lab. We thank  Ariel Jacobs  for accurate review of the language. We are grateful to the editor and the three anonymous reviewers whose constructive comments improved our manuscript.
We illustrate the methods using three examples: a network of trials that compare outcomes of various diagnostic strategies in patients with suspected acute coronary syndrome  (11) , a network of trials comparing the effectiveness of 18 antidepressants for major depression  (12) , and a network comparing adverse events of statins  (13). The three examples are introduced in Error! Reference source not found. All analyses were done in R software using the netmeta package and the CINeMA web application (Box 2)  (6, 14).
THE CINEMA APPROACH While it is straightforward to gauge the impact of within-study biases on the summary relative treatment effect in a pairwise meta-analysis  (17) , in network meta-analysis studies contribute data to the estimation of each summary effect in a complex manner. In the first example discussed below we show the complexity underpinning the flow of information in the network of diagnostic modalities used to detect coronary artery disease. A treatment comparison directly evaluated in studies with low risk of bias might also be estimated indirectly (via a common comparator) using studies at high risk of bias, and vice versa. While studies at low risk of bias are expected to provide more credible results, it is often impractical to restrict the analysis to such studies. The treatment comparison of interest might not have been tested directly in any trial, or tested in only a few small trials with high risk of bias.
Now, it is much easier to judge study limitations for each odds ratio; the larger the contribution from studies at high or moderate risk of bias, the more concerned we are about study limitations. Using this graph, we can infer that the total evidence from the network metaanalysis for the comparison of exercise ECG with SPECT-MPI involves low, moderate and high risk of bias studies with percentages 44%, 32% and 24%, respectively. The CINeMA software offers the option to automate production of judgments, based on the data presented in these bar graphs combined with specific rules. One possible rule is to compute a weighted average level of risk of bias, assigning scores of -1, 0 and 1 to low, moderate and high risk of bias. For the comparison exercise ECG vs SPECT-MPI, this would produce a weighted score of 0.
ACROSS-STUDIES BIAS BACKGROUND AND DEFINITIONS Across-studies bias occurs when the studies included in the systematic review are not a representative sample of the studies undertaken. This phenomenon can be the result of the suppression of statistically significant (or "negative") findings (publication bias), their delayed publication (time-lag bias) or omission of unfavorable study results (outcome reporting bias). The presence and the impact of such biases has been well documented  (20) (21) (22) (23) (24) (25) (26). Across-studies bias is a missing data problem, and hence it is impossible to conclude with certainty for or against its presence in a given dataset. Consequently, and in agreement with the GRADE system, CINeMA assumes two possible descriptions for across-studies bias: suspected and undetected.
SUMMARIZING JUDGMENTS ACROSS THE SIX DOMAINS The output of the CINeMA framework is a table with the level of concern for each of the six domains. Some of the domains are interconnected: factors that may reduce the confidence in a treatment effect may affect more than one domain. Indirectness includes considerations on intransitivity, which manifest itself in the data as statistical incoherence. Heterogeneity may be related to most of the other domains. Pronounced heterogeneity will increase imprecision in treatment effects and may be related to variability in within-study biases or the presence of publication bias. Finally, in the presence of heterogeneity the ability to detect important incoherence will decrease  (49). Although the final output of CINeMA is a table with the level of concern for each of the six domains, reviewers may choose to summarize judgements across domains.
With the use of open-source free software (see Box 2), our approach can be routinely applied to any network meta-analysis (  6 ) and offers a step forward in transparency and reproducibility. The suggested framework operationalizes, simplifies and accelerates the process of evaluation of results from large and complex networks without compromising in statistical and methodological rigor. The CINeMA framework is a transparent, rigorous and comprehensive system for evaluating the confidence of treatment effect estimates from network meta-analysis. Box 1. Description of three network meta-analyses used to illustrate the CINeMA approach to assess confidence in network meta-analysis.
Diagnostic strategies for patients with low risk of acute coronary syndrome Siontis et al performed a network meta-analysis to of randomized trials to evaluate the differences between the non-invasive diagnostic modalities used to detect coronary artery disease in patients presenting with symptoms suggestive of acute coronary syndrome  (11). Differences between the diagnostic modalities were evaluated with respect to the number of downstream referrals for invasive coronary angiography and other clinical outcomes. For outcome referrals, 18 studies were included. The network is presented in Figure  1A  and the data in Table  S1. The results from the network meta-analysis are presented in Table  1.
Antidepressants for moderate and major depression Cipriani et al compared 18 commonly prescribed antidepressants, which were studied in 179 head-to-head randomized trials involving patients diagnosed with major/moderate depression  (12). The primary efficacy outcome was response measured as 50% reduction in the symptoms scale between baseline and 8 weeks of follow-up. According to the inclusion criteria specified in the protocol only studies at low or moderate risk of bias were included  (58). The methodological and statistical details presented in the published article and its appendix. Here, we will focus on how judgements about credibility of the results were derived. The network is presented in Figure  1B  and the data is available in Mendeley Data (DOI:10. 17632/83rthbp8ys. 2).
Comparative tolerability and harms of statins The aim of the systematic review by Naci et al. (37)  was to determine the comparative tolerability and harms of eight statins. The outcome considered here is the number of patients who discontinued therapy due to adverse effects, measured as an odds ratio. This outcome was evaluated in 101 studies. The network is presented in Figure  1C  and the outcome data are given in Table  S4. The results of the network meta-analysis are presented in Table  S5  and the results from SIDE splitting in Table  S6.
Box 2. Description of the CINeMA web-application. THE CINeMA WEB APPLICATION CINeMA framework has been implemented in a freely available, user-friendly webapplication aiming to facilitate the evaluation of confidence on the results from network meta-analysis (http://cinema. ispm. ch/ (6)). The web application is programmed in javascript, uses docker and is linked with R; in particular, packages meta and netmeta are used  (59). Knowledge of the aforementioned languages and technologies is however not required from the users.
Loading the data In 'My Projects' tab, CINeMA users are able to upload a. csv file with the by-treatment outcome study data and study-level risk of bias (RoB) and indirectness judgments. CINeMA web-application can handle all the formats used in network meta-analysis (long or wide format, binary or continuous, arm level or study level data) and provides flexibility in labelling variables as desired by the user. A demo dataset is available in 'My Projects' tab.
Acknowledgements The development of the software and part of the presented work was supported by the  Campbell Collaboration. ME was supported by special project funding (Grant No. 174281 ) from the  Swiss National Science Foundation. GS, AN, TP were supported by project funding (Grant No. 179158 ) from the  Swiss National Science Foundation. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
The Sentinel-2 satellite mission offers high resolution multispectral time-series image data, enabling the production of detailed land cover maps globally. When mapping large territories, the trade-off between processing time and result quality is a central design decision. Currently, this machine learning task is usually performed using pixel-wise classification methods. However, the radical shift of the computer vision field away from hand-engineered image features and towards more automation by representation learning comes with many promises, including higher quality results and less engineering effort. In particular, convolutional neural networks learn features which take into account the context of the pixels and, therefore, a better representation of the data can be obtained. In this paper, we assess fully convolutional neural network architectures as replacements for a Random Forest classifier in an operational context for the production of high resolution land cover maps with Sentinel-2 time-series at the country scale.
Our contributions include a framework for working with Sentinel-2 L2A time-series image data, an adaptation of the U-Net model (a fully convolutional neural network) for dealing with sparse annotation data while maintaining high resolution output, and an analysis of those results in the context of operational production of land cover maps. We conclude that fully convolutional neural networks can yield improved results with respect to pixel-wise Random Forest classifiers for classes where texture and context are pertinent. However, this new approach shows higher variability in quality across different landscapes and comes with a computational cost which could be to high for operational systems.
The Need for Land Cover Mapping Land cover information is a crucial asset for many scientific and operational applications, such as vegetation monitoring, urban management, etc. Many of these applications need high resolution up to date land cover information. Land cover databases are freely available over wide areas, for example, Corine Land Cover (CLC) over the European Union  [1]  or ESA's CCI land cover products globally available. However, they often fail to meet the needs of many users: CLC has a minimum mapping unit of 25 ha and is only updated every 6 years, and ESA's Globcover cover maps have a resolution of 300 m  [2]. Until recently, automatic land cover mapping using remote sensing imagery was only performed at coarse resolution (CCI, for instance). Higher resolution imagery was only used through photo-interpretation (as for CLC) leading to high costs and long production delays.
However, a post-processing step using CRF increases complexity and is rather sensitive to hyper-parameter selection  [16]. Volpi et al. [17]  compare a simple fully convolutional patch based method to a downsample-upsample network similar to U-Net but without skip connections. They obtain state-of-the-art results on 10 cm resolution imagery with a densely annotated dataset. In the case of operational land cover mapping of large geographical areas, compared with semantic segmentation techniques patch based approaches have an important disadvantage because of their lower computational efficiency at prediction time, since every pixel of the patch has to be classified. Segmentation network architectures were initially applied to dense annotations (see  [12]  and references therein), in other words, all the pixels of the training image patches need to be annotated.
In  [20] , a modified U-Net with residual connections and a guided filter were used in post-processing to straighten building prediction mask edges. Simpler methods have also been proposed, where CNN predictions are refined with object-based image analysis approaches  [21]. Although these approaches improve the geometric accuracy of segmentation CNNs, they rely on the availability of dense annotations in order to learn how to predict edges or to regularize the segmentation. Practical large-scale reference data for land cover prediction, outside of academic datasets, are hard to obtain. Maggiori et al. [22]  run an initial training pass on roughly annotated data automatically generated from OpenStreetMap and then fine-tune the network on precise manual annotations. In this fashion, they train a multi-scale patch classifier on 10 3000 × 3000 pixels from 0.