dataesr/scientific-paragraphs-categorization huggingface dataset:
We calculate specific ∆rNBR results for the four VHR test sites 1-4, exactly matching the periods covered by the particular reference image pairs. The reference scenes have a spatial resolution of 50 cm (GeoEye-1, Pleiades, WorldView-2; Table  1 ) and the investigation periods of the corresponding Landsat scenes are given in Table  2. Minimal shifts between the acquisition times of the reference scenes and those of the corresponding Landsat data have to be tolerated (see Section 2. 2. 4). For validation, we stratified the pixels of the ∆rNBR datasets for the test sites into two strata, namely 'crown cover disturbance' and 'no disturbance, including noise', using a threshold of 0. 02, empirically-derived and supported by field observation. We then randomly selected 50 points per stratum within each of the test sites.
Finally, all 400 points were visually checked with the reference datasets, allowing the calculation for each test site of the overall accuracy as well as user and producer accuracies after applying weighting factors  [60, 61] , reflecting the different portions and therefore the difference in sampling probability for disturbed and un-disturbed pixels. Furthermore, we derived F1 score values  [62]  as a harmonic mean between user and producer accuracy, which conservatively describe the results, derived from unbalanced user and producer accuracies. Additional information from the time stamp datasets showing the acquisition dates per single rNBR period helped us to understand the temporal context of the ∆rNBR value per sample point. As the reference scenes show distortions up to 15 m among themselves as well as compared to the Landsat data, a 'rigid' (type i) and a more 'flexible' (type ii) assessment approach were applied in parallel (Figure  3 ).
For the 'rigid' assessment (type i) we strictly focused on the extent of the Landsat pixel (marked with an X in Figure  3 ) selected by each random sample point and checked it against the corresponding area in the reference scenes, neglecting any shift between the different scenes and thus providing a 'worst-case scenario'. For the 'flexible' assessment (type ii) we analyzed a 3 × 3 Landsat pixel window around each sampling point, which permitted us to account for possible distortions between the datasets, thereby allowing a possible shift of maximum one Landsat pixel and accepting bearing deviations of up to ±45 degree if the pattern was confirmed by the reference scenes. Finally, a total of 400 reference points were visually interpreted two times, using ArcGIS. Eventually, confusion matrices were calculated between the ∆rNBR values and the visually-derived results.
Analysis of the Possible Effects of Topography Similar to the VHR sites, specific ∆rNBR results for the two HR test sites 5-6 (based on 5 m RapidEye reference data; Tables  1  and 2 ) were calculated to match the periods covered by the reference image pairs. In comparison to the VHR reference data, the HR data had a lower spatial resolution that restricted the visual identification of changes in forest canopy closure. To account for the latter, we raised the threshold for discriminating between signal (crown cover disturbance) and noise (no crown cover disturbance) from 0. 02 to 0. 03. As both test sites were considerably larger than the VHR sites, the number of sample points per stratum was set to 100, which were weighted to account for the unbalanced strata populations. Similar to the VHR sites, a total of 400 reference points were visually interpreted two times.
In such extreme cases, even the analysis with VHR reference data does not allow the reliable separation of the above-described commission errors from 'real' forest disturbance events. Clouds not detected and masked during pre-processing can result both in errors of omission and commission. For example, undetected clouds in period 2 may have resulted in commission errors as wrongly considered 'canopy openings'. However, undetected clouds in period 1 may have caused omission errors only in the case of actual canopy change at the same location in period 2. Even though the applied automated cloud-masking processes perform very well, the removal of still undetected clouds due to invasive cloud buffering further considerably improves this pre-processing step. The correlation between the ∆rNBR values and the intensity of the disturbance events is influenced not only by number or by size of trees extracted but also by the amount of exposed soil or non-photosynthetic vegetation being detected.
Accuracy Assessment and Performance The overall accuracy values derived for the four VHR test sites point towards the satisfying performance of the ∆rNBR approach in reliably monitoring forest disturbance within evergreen forests. The VHR reference data further confirm that even Landsat sub-pixel-scale disturbance events as small as 0. 005 ha can be detected, which are almost undetectable to a human interpreter of Landsat imagery, even when applying intense histogram stretching. However, drawing direct conclusions for the country products based on the test site results is hampered for two reasons: First, the test sites could not be selected randomly, but depend in terms of number and spatial distribution on the availability of VHR reference imagery. A possible adjustment approach for reducing potential bias caused by the non-probabilistic sampling scheme  [64]  was not applicable to our study.
The script allows the analysis of any desired (also historic) period as long as sufficient imagery is available. Using Google cloud computing, the script can quickly process a full series of satellite images over a defined observation period, thus retaining canopy disturbances even if only detected once. This is essential for tracing forest degradation processes in the tropics, as canopy disturbances can become quickly invisible due to fast re-growing vegetation. A clear advantage of our monitoring approach lies in its capacity to deal with large amounts of satellite imagery in an automated way, where any visual interpretation would probably already fail in the initial phase, while only screening such a number of images. Due to its transparent methodology, this automatic monitoring approach will be of interest for MRV (monitoring, reporting and verification) in the context of REDD+ (reducing emissions from deforestation and forest degradation) or under FLEGT (forest law enforcement, governance and trade) for identifying logging activities.
Integrating Sentinel 2 data will further increase the potential for operational real-time monitoring of forest degradation processes. For development versions of the ∆rNBR script (Google Earth Engine account necessary) please see https://github. com/Andi1974/Forest-degradation-monitoring. For the latest archived version of the library, use DOI https://doi. org/10. 5281/zenodo. 1187063. Already processed ∆rNBR maps covering several Southeast Asian countries (further areas from other continents will be added) can be accessed at http://forobs. jrc. ec. europa. eu/recaredd/.
Acknowledgments: We would like to thank the  Department of Inspectorate and Law Enforcement/General Department of Administration for Nature Conservation and Protection (GDANCP) of Cambodia , the  Forest Inventory and Planning Division (FIPD)  and the  Department of Forestry (DoF) of Laos  in cooperation with the  Scaling-Up Participatory Sustainable Forest Management (SUFORD-SU ) project for their support in planning and implementing the field studies. In addition, we would like to thank  SUFORD-SU  for providing the VHR reference datasets for accuracy assessment in Laos.
When matching with the same image, there's a risk for a patch to find itself (which is obviously the best candidate possible). Because that case is not interesting, matches at a distance smaller than τ m in the image are rejected. A full region is rejected instead of just the patch itself because the overlapping patches are often really good candidates as well (though also not interesting for the forgery detection). The pseudocode of the algorithm is presented in Algorithm 2. The extension of PatchMatch from image patches (with which the algorithm was originally defined) to dense descriptors is natural and changes nothing in the different steps of the algorithm. Later on in this review, PatchMatch (and several modified versions) will be used with descriptors instead of patches, in particular the Zernike moments presented in Section 2.
Using Zernike Moments The parameters that will be used for the experiments are those suggested in  [5]. They are reminded in Table  1. The different images used in this section come from the FAU database 3  presented by Christlein et al. in  [4]. The first example, shown in Figure  5 , is a straightforward example illustrating the quality of the detections by the method in the simplest case. We can see that every step performs as expected and provides an accurate estimation of the forged regions. It is also important to test the method's resilience to false positives. This is why we applied the exact same method to an intact image containing several instances of a similar object, shown in Figure  6. The result confirms the absence of detection in this case even though some regions could have appeared forged due to their similarity.
g. structural validity, internal consistency) for scales with binary and ordinal response options aiming to measure differences in degree or quantity, the most common in applied research. The protocol includes investigations of (1) item distributions and summary statistics, item properties via (2) non-parametric and (3) parametric item response theory, (4) scale structure using factor analysis, (5) reliability via classical test theory, and (6) calculation and description of global scores. I illustrate the procedure on a measure of self-reported disability, the 24-item Sickness Impact Profile Roland Scale (RM-SIP), administered in a survey of 222 chronic pain sufferers. An R Markdown script is provided that generates reproducible reports. Results: In this sample, 15 of 24 RM-SIP items formed a unidimensional ordinal scale with good homogeneity (H = 0.
43) and reliability (α =. 86[. 84-. 89]; ω =. 87 [. 85-. 88]). The two versions were highly correlated (r =. 96), and regression models predicting RM-SIP disability produced comparable results. Conclusions: The example analysis illustrates how psychometric properties may be assessed in substantive studies and identify avenues for measure improvement. Applied researchers can adapt this script to perform and communicate these analyses as part of questionnaire validation and substantive studies.
, 2000) , and consequently on the effectiveness of clinical and policy decisions relying on them. Until recently, the limited availability of data analysis software and training represented a considerable barrier against a more widespread use of psychometric methods in applied research. More advanced methods such as item response theory were available in dedicated software programmes, some proprietary, and only high-resource research groups could afford the time and financial resources to access this expertise. Recent developments in the R environment  (R Core Team, 2013)  and increased access to online information and instruction have largely overcome this barrier. Numerous R packages on psychometrics methods are now available in the open-source CRAN repository (https://cran. r-project. org).
Tutorials describing their practical applications with worked examples with R code are also published with the packages or as methodological articles  (Ark, 2007; Dunn, Baguley, & Brunsden, 2014; Mair & Hatzinger, 2007; Rizopoulos, 2007; Stochl, Jones, & Croudace, 2012). Moreover, automatic report generation is possible in R via tools such as Sweave  (Leisch, 2002)  and R markdown  (Allaire, Horner, Marti, & Porte, 2015). Applied researchers have now easier access to using comprehensive and advanced methods for measure validation within substantive studies and reporting the results in a transparent and reproducible way. A key remaining barrier now is time efficiency: even if analyses which used to be time-consuming decades ago now take seconds, a beginning researcher may not afford the time of combining these methods into a streamlined data analysis protocol given the constraints of a typical research project.
The corresponding R Markdown script is available as supplementary material, and at https://github. com/alexadima/6-steps-protocol, together with the report generated using this script and the general template researchers may adapt for their own needs (Supplementary Materials 1, 2 and 3). Finally, I discuss its possible uses in applied research in the broader context of generating high-quality evidence for health care. Suggestions for further reading are provided along the way, and summarized in Supplementary Material 4. Short journey into measurement theoryor why factor analysis and Cronbach's α are not enough Introductory statistics courses usually describe the process of scale validation as starting with an exploratory factor analysis (EFA), then selecting and interpreting subscales based on the preferred factor solution, and calculating internal consistency per subscale using Cronbach's α; based on these results, subscale scores are computed, eventually correlated to other relevant variables to test hypothesized relationships.
, 2010). For the applied researcher, psychometrics needs to strike a balance between parsimony and comprehensiveness. Ideally, we would need a standardized yet flexible data analysis protocol to test a core set of measurement properties relevant to most research designs, which can be reproduced for several data and item sets, and modified for study-specific purposes. The 'EFA, then Cronbach's α' approach in introductory statistics books rests on the proposal that structural validity and internal consistency represent the core measurement properties requiring proof for most purposes and datasets. The practical advantage of this analysis is that is does not require more data collection (as for other properties such as face validity or criterion validity); the responses to the scale items at one time-point are sufficient to evaluate these properties.
Among the NIRT models, Mokken Scaling Analysis (MSA) represents the ordinal-level version of Rasch and Rating Scale analyses  (Schuur, 2003) , is available in R in the mokken package  (Ark, 2007) , and has been increasingly applied to health research  (Stochl et al. , 2012; Watson et al. , 2012). According to MSA, a set of items can be used to measure differences in degree between persons/objects if they have three properties: (1) unidimensionality, (2) local independence, and (3) monotonicity. In essence, these mean that (1) endorsing more 'difficult' items is related to a higher probability of endorsing 'easier' items, while the opposite does not apply, (2) items should be related with each other only via the latent variable they measure, and (3) the probability of endorsing an item should not decrease at higher levels of the latent variable.
A fourth property, invariant item ordering, is necessary to meet the ordinal equivalent of the Rasch standard: items should keep the same order of difficulty at all levels of the latent variable; this allows comparisons between groups or datasets  (Schuur, 2003; Sijtsma & Hemker, 1998). Idiosyncratic response patterns can be also identified as cases with high number of Guttman errors, i. e. instances when a respondent chooses an answer not consistent with the expected overall pattern, for example not endorsing an easier item while endorsing a more difficult one  (Meijer, Niessen, & Tendeiro, 2016 ). An Automatic Item Selection Procedure (aisp) can be used to partition the item set into unidimensional scales  (Ark, 2007) ; items that group with each other into scales are described as 'scalable'.
Therefore, the 6-step protocol proposes to use MSA as the first psychometric method for item examination, after data preparation and descriptive statistics at item level. If MSA criteria are met, parametric IRT tests can further investigate whether both item and person fit are maintained when the latent variable is viewed as a continuum while items are modelled as binary or ordinal level. Factor analyses can be used next to explore or confirm the structure, particularly under assumptions of interval or quasi-interval level and multivariate normality of items. For item sets that prove unidimensional and with adequate properties, CTT analyses can then be performed, particularly for evaluating reliability. Finally, scale or subscale scores can be calculated and summarized. Each step answers specific questions about the properties required from an item set to ensure accurate measurement, as described below.
Dataset and script The use of the 6-step protocol is illustrated with an analysis of a 24-item measure of selfrated disability, the Sickness Impact Profile -Roland Scale (RM-SIP), commonly used in pain conditions  (Jensen, Strom, Turner, & Romano, 1992; Roland & Morris, 1983; Stroud, McKnight, & Jensen, 2004). The items refer to various difficulties experienced in daily activities due to pain. The respondents are asked to report whether the items describe their condition over the past few days by ticking a box if they experienced those difficulties. Thus, the items have a binary (yes/no) response format. The scale is hypothesized as unidimensional and the total score sums up the affirmative answers. The dataset includes responses from 222 adults participating in a survey on living with chronic pain conducted in 2007-2008 in the Lothian region, United Kingdom (ethical approval from the NHS Lothian Research Ethics Committee).
The dataset and the survey section with the RM-SIP questions are available as Supplementary Materials 5 and 6. To illustrate sensitivity analyses, several other relevant variables were included: background characteristics, the Short-Form McGill Pain Questionnaire (SF-MPQ;  Melzack, 1987) , and the Brief Illness Perceptions Questionnaire (BIPQ;  Broadbent, Petrie, Main, & Weinman, 2006). The 6-step protocol template and its adaptation to this example analysis (SM1 and SM2) have the same structure, which alternates sections of Markdown text with sections of R code. They produce data analysis reports with an initial summary text, some preparatory code, then progressing through all 6 steps following the same structure: outlining the aims, performing the analysis, displaying results in tables and figures, interpretation and decisions.
Commented text signposts the different sections, explains the relevant code, gives advice on analysis choices and criteria for interpretation, and suggests other methods that can be added to the script. For this tutorial, I will assume the readers are familiar with R and R studio and have already conducted statistical analysis in this environment. Many resources are available for familiarization with R and R studio, which include preparing the dataset and setting up your R studio session  (Gandrud, 2013; Torfs & Brauer, 2014). To reproduce the analysis I summarize below, the Scale_validation_-_RM-SIP_analy-sis_-_SM2. Rmd file (SM2) and the dataset (ChronicPainSurvey. csv, SM5) need to be in the same folder on your computer. Open the Rmd file in RStudio, set your working directory to the source file location and click the 'Knit to HTML' option on the upper band of the.
Data preparation The first section of R code installs and imports the R packages necessary for the analysis, sets up several additional options for markdown, and defines some functions for the analyses performed in this script. The second and third section import the dataset, select the item set for the analysis, check response frequencies, and make any necessary adjustments. In our dataset, it was necessary to transform the response options 'yes'/'no' into numeric (1/0). When adapting this script for a new dataset, here you would have the chance to spot any out-of-range or missing values and correct any data entry errors. The next analyses require a dataset of numeric variables with no missing values. Step 1: descriptive statistics Several basic outputs in this step allow familiarization with the data.
These differences in endorsement frequencies may reflect a latent construct of disability, if they also meet the criteria tested in the next step. Step 2: non-parametric item response theory (IRT) The two item response theory models are applied in steps 2 and 3 to examine item properties for optimal ordinal and interval-level measurement. For non-parametric IRT, the mokken package  (Ark, 2007)  examines coefficients of homogeneity (H) for items, item pairs and scale, and person fit (Guttman errors), and explores possible dimensionality solutions via the aisp algorithm performed at increasing thresholds  (Hemker, Sijtsma, & Molenaar, 1995). Unscalable items may need to be excluded from further analyses. High numbers of Guttman errors in individual cases may point to data collection errors that require exclusion of cases, and are therefore useful to check.
Sensitivity analyses The 6-step analysis identified several items with suboptimal performance. Excluding these items produced a short version that was highly correlated with the original version. To verify the effect that these items had on a substantive question, a sensitivity analysis is included in the script which includes bivariate correlations of the two RM-SIP versions with other variables available in the dataset: 8 items of the BIPQ measuring illness perceptions, and the visual analogue scale of the MPQ measuring pain intensity. Two multiple regression models examined the contribution of illness perceptions, pain intensity, and demographic variables (gender, age, and education) to predicting variance in the two RM-SIP versions. Results were consistent regarding bivariate correlations (Table  3 ), and the primary role of pain intensity and perceptions of illness consequences in explaining variations in perceived disability (Table  4 ).
Adapting the template The R Markdown file described above (SM2) has been created by adapting the general template (SM1) to the RM-SIP analysis. A comparison of these documents will help identify the changes required, therefore I will not discuss them individually, but rather highlight the main elements to consider. First, the characteristics of the dataset imported require some obvious adaptations: name and format, location of the items in the dataset, number of items, variable format and labels, scale structure. The template includes advice on how to adapt the code at each step (as commented text), and several options or examples for most common situations. For example, several options for importing data with different formats are given in the data preparation section; while the CFA model in Step 4 would need to be re-written (following the example given in the template for a 3-factor model with 10 items per factor).
Yet, while model testing becomes increasingly sophisticated in substantive studies, progress in measurement methods is comparatively less considered  (Hamilton, Marques, & Johnson, 2017). This protocol continues previous efforts to re-balance this state of affairs, such as recent consensus-building initiatives on measurement-related issues  (Mokkink et al. , 2010; Reeve et al. , 2013)  and proposals for raising analysis and reporting standards  (Crutzen & Peters, 2017). It should be seen as complementary to the ongoing development of PRO databases and item banks, which bring the unquestionable benefit of standardized tools applicable across studies and populations to facilitate evidence synthesis. It can also be taken as a 'base camp' for exploring more sophisticated methods, and is intended as a proposal for further debate, and perhaps future consensus on essential criteria for measure validation.