Product Validation Report(D12-PVR)
Document Ref:
Version:Contract No:
Creation Date:
Last Modified:WorldCover_PVR_v2.0
2.04000128231/19/I-LG
2022-03-21
2022-10-25

WorldCover

Product Validation Report | V2.0

Page | 2

DOCUMENT RELEASE SHEET
Name

Signature

Book Captain

Nandika Tsendbazar (WU)

Contributing Authors

Panpan Xu (WU)
Martin Herold (GFZ)
Myroslava Lesiv (IIASA)
Martina Duerauer(IIASA)
Ruben Van De Kerchove (VITO)
Olivier Arino
PublicApproval (internal)
Approval (ESA)
DistributionDate
25-10-2022

25-10-2022

DOCUMENT CHANGE LOG
Issue

Issue datePages affected

Relevant information

0.1
1.0

2021-08-31
2021-10-15

All
All

2.0

2022-10-25

All

First draft on statistical validation
Updates on statistical validation and othersections are added.
Updated for the 2021 product and version 2.0of the algorithm

WorldCover

Product Validation Report | V2.0

Page | 3

Table of Content
TABLE OF CONTENT.............................................................................................................................. 3
LIST OF FIGURES................................................................................................................................... 4
LIST OF TABLES .................................................................................................................................... 5
ACRONYMS AND ABBREVIATIONS ........................................................................................................ 6
1

2

INTRODUCTION ............................................................................................................................ 7
1.1

Project objectives and approach ......................................................................................................... 7

1.2

Purpose of the document .................................................................................................................... 7

1.3

Content of the document .................................................................................................................... 8

METHODS..................................................................................................................................... 9
2.1

2.1.1

The CGLS-LC validation dataset ................................................................................................. 10

2.1.2

Adopting the CGLS Validation dataset for validating WorldCover 2021 .................................. 12

2.1.3

Accuracy estimation of the WorldCover product ...................................................................... 14

2.2
3

Statistical Accuracy Assessment .......................................................................................................... 9

Qualitative Accuracy Assessment ...................................................................................................... 14

RESULTS ..................................................................................................................................... 15
3.1

Statistical Accuracy of the WorldCover 2021 v200 product.............................................................. 15

3.2

Qualitative comparison of the WorldCover 2020 v100 and 2021 v200 product .............................. 17

4

LIMITATIONS .............................................................................................................................. 21

5

CONCLUSIONS ............................................................................................................................ 23

6

REFERENCES ............................................................................................................................... 24

ANNEX 1. ........................................................................................................................................... 26
ANNEX 2. ........................................................................................................................................... 27

WorldCover

Product Validation Report | V2.0

Page | 4

List of Figures
Figure 1: Spatial distribution of all validation sample sites for different (sub) continents ............................... 11
Figure 2: Screenshot of the Geo-Wiki based interface for validation data collection ...................................... 12
Figure 3: A screenshot of an example sample interpretation (green – trees, orange – shrubs, yellow –grassland) .......................................................................................................................................................... 12
Figure 4: Snippets of validation data locations depicting shifts between the images sourced from Google map
and Bing map (Tsendbazar et al. 2021b). .......................................................................................................... 13
Figure 5: An example of primary and alternative validation labels .................................................................. 14
Figure 6: The legend of the WorldCover product.............................................................................................. 17
Figure 7: Comparison of WorldCover 2020 v100 (A) and WorldCover 2021 v200 (B) and VHR images available
from the Google Earth (C). 1. Midland, Texas, the USA (Coordinates: 31.9510°N, 102.0457°W); 2. Andalusia,
Spain (Coordinates:37.6877°N, 3.6986° W); 3. Near Fariman, Iran (Coordinates: 35.6369°N, 60.0684°E); 4.
Parana River, Buenos Aires, Argentina (Coordinates: 33.0958°S, 60.3706°W); 5. Khangai mountains in
Mongolia (Coordinates: 48.1886°N, 99.5896°E); see Figure 6 for legend......................................................... 18
Figure 8: Comparison of the WorldCover 2021 v200 (A) and Sentinel 2 data (B). 1. Mato Grosso, Brazil(Coordinates: 9.9469°S, 59.8814°W); 2. Chad (Coordinates: 16.3122°N, 21.0947°E); 3. Columbia(Coordinates: 9.0344°N, 74.6195°W ); 4. Sua-Pan, Botswana (Coordinates: 20.6461°S, 25.8411°E); 1, 3 & 4
show the false colour image of Sentinel 2 data of 2021 based on median composite; 2 shows a false colour
image of Sentinel 2 data acquired on August 17 2021; see Figure 6 for legend............................................... 20
Figure 9: Examples of artefacts in the WorldCover 2021 v200 product. (A) Nigeria (Coordinates: 13.0691°N,
8.6240°E); (B) Russia (Coordinates: 50.6518°N, 94.0797°E); see Figure 6 for legend....................................... 21
Figure 10: Examples of misclassification in the WorldCover 2021 v200 product. 1. Misclassification of trees(Coordinates: 68.1514°N, 132.1499°E); 2. Underestimation of wetlands caused by the mis-assigning ofcroplands (Coordinates: 48.1329°N, 134.1937°E); see Figure 6 for legend. ..................................................... 22

WorldCover

Product Validation Report | V2.0

Page | 5

List of Tables
Table 1: CEOS WGCV stage 3 and 4 validation requirements http://lpvs.gsfc.nasa.gov/ ................................. 10
Table 2: Confusion matrix (%) for the WorldCover 2021 v200 product at a global scale, corrected by sample
inclusion probabilities. ...................................................................................................................................... 15
Table 3: Overall accuracy for the WorldCover 2021 v200 product at the continental level and sample sites
used (SSU- secondary sample units, PSU – primary sample units) ................................................................... 16

WorldCover

Product Validation Report | V2.0

Acronyms and Abbreviations
AD
CCI
CEOS
CGLS
ESA
GLC
GOFC-GOLD
IIASA
LCCS
LPV
MMU
PA
PSU
PVR
RB
RD
SDGs
SoW
SSU
UA
URD
UTM
VHR
WGCV

Applicable Document
Climate Change Initiative
Committee on Earth Observation Satellites
Copernicus Global Land Service
European Space Agency
Global land cover
Global Observation of Forest Cover and Land Dynamics
International Institute for Applied Systems Analysis
Land Cover Classification System
Land Product Validation
Minimum Mapping Unit
Producer Accuracy
Primary Sampling Unit
Product Validation Report
Requirement Baseline
Reference Document
Sustainable Development Goals
Statement of Work
Secondary Sampling Unit
User Accuracy
User Requirement Document
Universal Transverse Mercator
Very High Resolution
Working Group on Calibration and validation

Page | 6

WorldCover

Product Validation Report | V2.0

Page | 7

1 Introduction
1.1 Project objectives and approach
In an ever-changing environment, the need for accurate, timely and high-resolution information onland use/land cover and its changes has increased tremendously. However, until now, regional orcontinental land cover maps either used low-resolution images (>100 m) as input or were based
exclusively on high-resolution optical Earth observation data, such as Sentinel-2 or Landsat. The useof Synthetic Aperture Radar (SAR) data such as Sentinel-1 to produce large area land cover maps isstill in its infancy.
For this purpose and inspired by the 2017 WorldCover conference (attended by more than 400
participants) the European Space Agency (ESA) initiated the WorldCover project. The key outcome ofthis project was the release in October 2021 of a freely accessible global land cover product at 10 m
resolution for 2020 based on both Sentinel-1 and Sentinel-2 data, containing 11 land cover classes andindependently validated with a global overall accuracy of 74.4%.
Following the positive feedback from the users, ESA decided to extend the WorldCover project andrequested the WorldCover consortium to also produce a 2021 version of the product with even higher
quality. This new WorldCover map for 2021 was released on 28 October 2022 and resulted in a global
overall accuracy of 76.7%.
Since the WorldCover maps for 2020 and 2021 were generated with different algorithm versions(v100 and v200, respectively), changes between the maps should be treated with caution, as they
include both changes in real land cover and changes due to the used algorithms.

1.2 Purpose of the document
The Product Validation Report (PVR) is a deliverable of contract N°4000128231/19/I-LG. The objectiveof this document is to describe the validation results of the WorldCover 2021 v200 product. The
validation results include statistical accuracy assessment of the WorldCover 2021 v200 product andimprovements as compared to the WorldCover 2020 v100 product.

WorldCover

Product Validation Report | V2.0

Page | 8

1.3 Content of the document
This document is structured as follows:• Section 2 details the methods for validating the WorldCover product. Here, the methods usedfor validation data update and statistical accuracy assessments are detailed.
• Section 3 details the main results on validation of the WorldCover 2021 v200 product, withdetails on qualitative comparisons with the previous release (WorldCover 2020 v100).
• The observed limitations of the WorldCover v200 product are included in Section 4.
• The final conclusions of the validation results are included in Section 5.

WorldCover

Product Validation Report | V2.0

Page | 9

2 Methods
Validation is an intrinsic part of satellite-derived land product generation as it informs on the productquality, consistency and builds the product users' confidence in using the land product. In addition,
validation should inform on the “fitness for use” to a variety of end-users that use land products.
Information on these aspects of a map can be achieved through map validation assessments.
Here we performed the following steps to provide information on map quality, consistency, andfitness for use of the WorldCover 2021 v200 product.
••

Statistical accuracy assessment following Committee on Earth Observation Satellites (CEOS)
Working Group on Calibration and Validation (WGCV) Land Product Validation (LPV)
requirements.
Qualitative comparison with the WorldCover 2020 v100 product.

The statistical accuracy assessment aimed to provide information on the product accuracy using astatistically rigorous method which is based on independent validation data selected using probability
sampling. Here we made use of the world’s most advanced (CEOS-WGCV Stage 4) and multi-purpose
independent GLC validation systems that have been developed, scientifically published (Tsendbazar
et al. 2018), and operationally used and regularly updated as part of the Copernicus Global Land
Service (CGLS) (Tsendbazar et al. 2021a).
For comparison, we assessed the WorldCover 2020 v100 visually. In the following subsection, we detailthe methods used.

2.1 Statistical Accuracy Assessment
The statistical accuracy assessment of the WorldCover product follows closely the best practices
guidelines for GLC validation supported by the international communities such as the Global
Observation of Forest Cover and Land Dynamics (GOFC-GOLD) and the Committee on Earth
Observation Satellites (CEOS).
The “validation” is defined according to the definition of the CEOS Working Group on Calibration andValidation (CEOS-WGCV):“The process of assessing, by independent means, the quality of the data products derived from thesystem outputs”
The statistical accuracy assessment of the WorldCover map meets the requirements of the CEOSWGCV Validation Stage 3 and Stage 4 (Table 1).

WorldCover

Product Validation Report | V2.0

Page | 10

Table 1: CEOS WGCV stage 3 and 4 validation requirements http://lpvs.gsfc.nasa.gov/
Stage 3

Uncertainties in the product and its associated structure are well-quantified fromcomparison with reference in situ and higher resolution airborne and satellite data.
Uncertainties are characterized in a statistically robust way over multiple locations andtime periods representing global conditions. The spatial and temporal consistency of the
product and consistency with similar products has been evaluated over globally
representative locations and periods. Results are published in the peer-reviewed literature.

Stage 4

Validation results for stage 4 are systematically and operationally updated by independent
actors for comparative assessment of existing products when new products are released
and as the time-series expands.

The multi-purpose Global Land Cover Validation dataset developed for the Copernicus Global Land
Service- Land Cover product (henceforth called CGLS-LC validation dataset) was used. The following
characteristics of the CGLS-LC validation dataset made it suitable to assess statistically the WorldCoverproduct meeting the requirements mentioned above:
1. A global stratification that is independent of any land cover maps and uses the Sentinel 2
Universal Transverse Mercator (UTM) grid as a geographic base
2. Globally more than 21 000 primary sampling units (PSUs) with each containing one hundred
10x10 m reference pixels, are suitable for assessing accuracy per continent (minimum of 3000
PSUs per continent).
3. Stage 4 validation dataset: the PSUs are updated every year by focusing on areas that went
under change since 2015 as part of the CGLS.
4. A multi-purpose validation data suitable for validating 10 m land cover maps, which allows
the validation for WorldCover product.
5. A dataset containing high-quality land cover elements information at 10 m resolution
contributed and reviewed by more than 30 international and regional experts from aroundthe world.
6. The 10 m land cover element information fully corresponds to Sentinel-2 10 m resolution
pixels and is collected on the Geo-Wiki reference data collection platform
The following subsections include a description of the CGLS-Validation data including a wellestablished method to estimate map accuracies with statistical rigorousness for validating theWorldCover product.

2.1.1 The CGLS-LC validation dataset
The CGLS-LC validation dataset is based on probability sampling to allow a design-based inference ofmap accuracies. The criterion of statistical probability sampling with known and non-zero inclusion
probabilities was followed. The validation dataset is based on stratified random sampling, employing
a global stratification (Olofsson et al. 2012). Globally there are 149 strata divided over seven(sub)continents. The validation dataset consists of 21 752 PSUs. The available PSUs per continent(minimum 3000 PSUs per continent) allow to carry out a statistical accuracy assessment for eachcontinent with a high level of precision (Tsendbazar et al. 2021a). The spatial distribution of thevalidation sites and (sub)continent divisions are shown in Figure 1.

WorldCover

Product Validation Report | V2.0

Page | 11

Figure 1: Spatial distribution of all validation sample sites for different (sub) continents
The CGLS-LC validation dataset is multi-purpose data compatible for validating maps with different
resolutions and legends. Each PSU (covering an area of 100 m × 100 m) was divided into 10×10 small
blocks (henceforth called SSU: secondary sampling unit). Since the reference land cover elements
were collected at 10 m × 10 m SSU level, the dataset is compatible for assessing land cover maps with
10-100 m resolutions. For the thematic representation, the generic land cover elements recorded ateach SSU include trees (phenology and leaf types), shrubs, grass, crops, built-up areas, bare area,lichens/mosses, open water, snow & ice, and regularly flooded areas. The land cover elements were
defined according to the United Nations Land Cover Classification System (LCCS) (Di Gregorio 2005).
The validation data was collected using a dedicated web-interface through the Geo-Wiki platform(Fritz et al. 2011). The interface provided access to different remote sensing data and allowed labelling
land cover (Figure 2). An example of labelling the land cover of SSUs within a PSU is provided in Figure3.
The reference land cover was visually interpreted by 30 regional experts remotely (Table 1, Annex 1).
All experts have experience in satellite-based land cover analysis and image interpretation. For quality
assurance, all the interpretations were reviewed and improved where necessary (Tarko et al. 2021).
The validation dataset contains reference land cover information for the years 2015-2019. Moredetailed information on the validation data can be found in Tsendbazar et al. (2021a).

WorldCover

Product Validation Report | V2.0

Page | 12

Figure 2: Screenshot of the Geo-Wiki based interface for validation data collection

Figure 3: A screenshot of an example sample interpretation (green – trees, orange – shrubs, yellow –grassland)

2.1.2 Adopting the CGLS Validation dataset for validating WorldCover 2021
To validate the WorldCover2020 v100 product, the CGLS-LC validation data for the reference year
2019 was updated with validation data for 2020 by revisiting 2860 validation points with a highpossibility of change as well as a random subset (Tsendbazar et al. 2021b). For the reference year
2021, we further updated the validation dataset by revisiting a total of 4410 validation sites. Next torevisiting a random subset of 2200 validation sites (~10%), we have revisited about 2210 validationsites, aiming to target sites with high possibility of land cover change between 2020 and 2021. Highpossibility of land cover change sites were identified based on the BFAST Monitor change detection
algorithm using Landsat 8 data (2015-2022) as well as change based on the WorldCover 2020 and 2021
(a preliminary version) maps. In addition, Sentinel-2 based NDVI and NDWI indices were calculated atthe validation sites to flag validation sites to revisit for possible land cover change. Validation sites
with inconsistent indices values with respect to their reference land cover labels were shortlisted. For
example, validation sites with grass label having very low NDVI value for 2021 were included in therevision. Revision was done by an independent regional expert who contributed to the initialWorldCover

Product Validation Report | V2.0

Page | 13

generation of the CGLS-LC validation data. In case of land cover change, the land cover information
was updated.
Note that the CGLS-LC validation data is used as an independent validation dataset for the CGLS-LC
mapping effort and to continue to support independent validation of maps, the dataset will not bereleased to the public.
The CGLS-LC validation dataset was collected by visual interpretation of VHR images. VHR images canhave some level of geolocation errors which can result in shifts between VHR base maps. Figure 4
illustrates examples of shifts between the images sourced from Google map and Bing maps
(Tsendbazar et al. 2021b). In case of such a shift, Google map locations were preferred for the CGLSLC validation data collection. This was due to the larger availability of VHR maps in the Google map forrecent years. However, in cases where no Google VHR images were available, Bing map and ESRI VHR
images were used, supported by Sentinel-2 imagery. As the WorldCover products come at a highresolution (10 m), geolocation errors of these VHR images (e.g., Google map and Bing map) used forvisual interpretation can have an impact on the obtained accuracy numbers.
To validate the WorldCover 2020 product, SSUs isolated in terms of land cover were excluded fromthe validation data to account for the possible impact of validation data ambiguity due to geolocation
errors of the VHR images and labelling uncertainty in heterogeneous areas at high resolution(Tsendbazar et al. 2021b). This was done with the expectation that the impact of ambiguity would beless in a more homogeneous area (Gu and Congalton 2020). Therefore, SSUs having the same land
cover type with at least two direct neighbours were included and the rest were not included in thevalidation. As a result, 1 935 650 SSUs across the world were used for the statistical accuracy
estimation of the WorldCover 2020 product, while ~10% of the SSUs were excluded for this validation(Tsendbazar et al. 2021b). This approach however has a limitation in heterogeneous areas, being morebiased towards homogeneous areas.

Figure 4: Snippets of validation data locations depicting shifts between the images sourced from
Google map and Bing map (Tsendbazar et al. 2021b).
To validate the WorldCover 2021 product, we revised therefore the approach to deal with validation
data ambiguity. We utilized information from the neighbouring SSUs as an alternative land cover label
in addition to the reference label for the SSU itself. Making use of alternative or secondary labels fromWorldCover

Product Validation Report | V2.0

Page | 14

the surrounding area of the sample unit is a widely used method (Olofsson et al. 2014; Stehman et al.
2003; Wickham et al. 2021). This approach addresses possible reference label ambiguity andgeolocation error, but all sample units are included in the validation; no sampling units are discarded.
Similar to Stehman et al. (2003) and Wickham et al. (2021), next to the primary land cover label of a
SSU, we used majority/modal land cover type from the surrounding neighbourhood (3x3) of the SSU.
Therefore, the reference land cover is represented both by the primary and alternative labels in thevalidation dataset. Both the labels are compared with the mapped label. An example for identifying
primary and alternative labels is included in Figure 5.

Figure 5: An example of primary and alternative validation labels

2.1.3 Accuracy estimation of the WorldCover productFor the accuracy estimation, we used a stratified one-stage cluster approach (Pengra et al. 2015). This
cluster approach is suitable since multiple SSUs within the PSUs are used for the assessment.
Calculation of inclusion probabilities for the PSUs and SSUs follows the methods described in Pengra
et al. (2015) and Tsendbazar et al. (2018). Based on the estimation weight (the inverse of inclusion
probability) per sampling unit, mapped and reference land cover types, a confusion matrix wasconstructed accounting for unequal sample inclusion probabilities (Stehman et al. 2003; Wickham etal. 2010). The estimation weights for each sample unit were available for this validation dataset.
Accuracy estimates, namely overall accuracies, class-specific accuracies, and their confidence intervals(at 95% confidence level) were calculated following the stratified one-stage cluster approach (Pengra
et al. 2015; Stehman et al. 2003; Tsendbazar et al. 2018). Due to the limited number of validation sites,
samples for the lichen/moss class were merged with the grass class in South America and Asia.
Next to global level accuracy estimates, the accuracy estimation was also done per continent (7 subcontinents) following the initial design of the validation dataset (Figure 1).

2.2 Qualitative Accuracy Assessment
For qualitative comparison, we compared the WorldCover 2021 v200 and 2020 v100 products visuallywith Google Earth VHR images as well as Sentinel-2 median composites accessed from the WorldCoverviewer.

WorldCover

Product Validation Report | V2.0

Page | 15

3 Results
3.1 Statistical Accuracy of the WorldCover 2021 v200 productOverall accuracy estimates of the WorldCover 2021 v200 product can be found in Table 2. On a global
scale, using 2.16 million SSUs at 21 624 PSU locations, the overall map accuracy was 76.7±0.5%.
In terms of class-specific accuracies, tree cover, snow/ice, cropland, water body, and bare/sparsevegetation classes had high accuracies. Grassland and built-up classes had moderate accuracies, while
shrubs, wetlands, and moss/lichen classes had relatively lower accuracies globally.
In general, at the global scale, there was a slight underestimation of moss and lichen and bare classwhile a slight overestimation of trees was observed when comparing against the validation dataset.
Shrubs, grassland, cropland, built-up and permanent water bodies classes were mapped with balanced
user’s and producer’s accuracies.

0.0

0.2

0.7

0.1

0.0

Snow andice

0.0

Confidence
interval ±Bare /
sparse veg

User'saccuracy

0.1
0.5
1.7
0.1
0.1

Total

0.1
0.0
0.1
0.0
0.6

Correct

Bare / sparse
veg

0.4
0.2
1.2
7.4
0.0

Moss and
LichenBuilt-up

3.1
2.2
16.9
1.2
0.1

Mangroves

Cropland

2.4
3.9
1.5
0.1
0.0

Herbaceous
wetlandGrassland

26.2
1.1
0.9
0.2
0.0

Permanent
water bodies

ShrublandTree cover
Shrubland
Grassland
Cropland
Built-up

Snow and ice

Tree cover

Table 2: Confusion matrix (%) for the WorldCover 2021 v200 product at a global scale, corrected bysample inclusion probabilities.

0.0

0.1
0.0
0.1
0.0
0.0

0.1
0.0
0.4
0.1
0.0

0.1
0.0
0.0
0.0

0.1
0.0
0.8

26.2
3.9
16.9
7.4
0.6

32.7
7.9
23.4
9.2
0.8

80.0
49.1
71.9
80.6
65.9

0.7
2.1
1.0
1.5
3.3

15.2

0.0

0.0

0.0

0.0

0.3

15.2

16.5

92.1

0.9

0.2

2.3

0.0

0.0

2.3

2.4

93.0

2.4

Permanentwaterbodies

0.0

0.0

0.0

0.0

0.0

0.1

0.0

2.3

0.1

0.0

0.1

2.3

2.6

89.4

1.8

Herbaceous
wetland

0.0

0.1

0.5

0.0

0.0

0.0

0.0

0.1

0.6

0.0

0.5

0.6

1.8

30.5

4.3

Mangroves

0.0

0.0

0.0

0.0

0.0

0.0

0.1

0.1

0.1

74.1

13.2

Moss and
Lichen

0.0

0.1

0.6

1.4

2.5

57.5

3.8

Correct
Total

26.2
28.5

3.9
8.3

16.9
25.3

7.4
9.3

Producer's
accuracy

91.9

46.9

66.7

Confidence
interval ±

0.5

2.3

1.1

0.0

1.4

0.4

0.0

0.0

0.0

0.6
0.8

15.2
18.4

2.3
2.3

2.3
2.7

0.6
1.2

0.1
0.2

1.4
3.1

79.3

73.2

82.5

99.1

86.4

44.6

46.2

46.4

1.5

2.6

1.2

0.4

1.7

5.4

16.0

3.5

76.7
0.5

WorldCover

Product Validation Report | V2.0

Page | 16

The accuracy of the WorldCover 2021 v200 product at the continental level is listed in Table 3. All
continents had an overall accuracy above 72%. The overall accuracy was highest for Asia (82.1%)
followed by South America, Europe, and Africa. The lower accuracy for Oceania could be due to highshrubland vs grassland and trees vs grassland confusions in the open woodlands of Australia. Similarly,
high shrubland vs trees and grassland vs trees confusions in Siberian temperate tundra regions could
explain the lower overall accuracy in Eurasia.
Table 3: Overall accuracy for the WorldCover 2021 v200 product at the continental level and sample
sites used (SSU- secondary sample units, PSU – primary sample units)
Overall accuracy Total SSU
GlobalAfricaEuropeEurasia
Asia
Oceania andAustralia
North America
South America

Total PSU

76.7 +/- 0.5

2,162,366

21,62476.5 +/- 1.3

359,900

3,599

77.9 +/- 1.0

311,800

3,118

72.5 +/- 1.3

300,271

3,003

82.1 +/- 1.0

306,400

3,064

72.5 +/- 1.3

296,100

2,961

74.6 +/- 1.2

287,995

2,880

77.9 +/- 1.1

299,900

2,999

The class-specific accuracies and the confusion matrices for each continent are listed in Tables A2.1A2.7, Annex 2. Similar to the global scale accuracy, tree cover was mapped with high accuracy for mostcontinents. For this class, the commission error (100-user’s accuracy) tended to be higher than theomission error (100-producer’s accuracy) for most continents except Africa. In Africa, the tree coverclass was balanced in terms of user’s and producer’s accuracy.
Cropland class was mapped with higher accuracies for all the continents except Africa where this classtended to be under-represented. Mangroves class was mapped with moderate accuracies forcontinents in which this class has a presence.
Similar to the global level accuracy, permanent water bodies and snow/ice classes were mapped withhigher accuracy, except for South America, where the snow/ice class had a higher confusion withbare/sparse vegetation class. Similarly, shrubs, herbaceous wetland, and moss and lichen classes
showed lower class accuracies for all the continents consistent with the global level accuracy.
Bare /sparse vegetation class had varied accuracies among the continents. This can be mainly due tothe differences in area coverage of this class in the continents. For example, this class had highaccuracy in Africa and Asia which have significant coverage of bare/sparse vegetation. In contrast, in
Europe where bare/sparse vegetation does not cover large areas, this class was mapped with lower
accuracies.

WorldCover

Product Validation Report | V2.0

Page | 17

3.2 Qualitative comparison of the WorldCover 2020 v100 and 2021 v200
product
The WorldCover 2021 v200 product has been visually compared with WorldCover 2020 v100. The
legend of the WorldCover product is included in Figure 6.

Figure 6: The legend of the WorldCover productFigure 7 depicts the visual comparison of five regions comparing the WorldCover 2020 v100 andWorldCover 2021 v200 maps. Figure 7-1 depicts the WorldCover product versions in the outskirts ofMidland, Texas of the USA. WorldCover v100 previously overestimated cropland in the suburban areas
significantly. In v200, the separation between cropland and grassland has been improved, with thebackyards of houses correctly classified as grassland, based on the comparison with VHR images
available in Google Earth. In this area, although slightly better compared to v100, v200 underestimates
shrubland areas in the natural parks and open spaces. Improvements in characterizing shrublands inarid regions can further be demonstrated in Europe (Figure 7-2). In Andalusia, Spain, arid shrublands
were previously mapped mostly as bare/sparse vegetation and grassland in WorldCover 2020 v100.
In WorldCover 2021 v200, the shrubland are correctly mapped in better balance with bare/sparsevegetation class. In addition, forest borders match with what can be observed from the VHR image.

WorldCover

Product Validation Report | V2.0

Page | 18

Figure 7: Comparison of WorldCover 2020 v100 (A) and WorldCover 2021 v200 (B) and VHR images
available from the Google Earth (C). 1. Midland, Texas, the USA (Coordinates: 31.9510°N,
102.0457°W); 2. Andalusia, Spain (Coordinates:37.6877°N, 3.6986° W); 3. Near Fariman, Iran
(Coordinates: 35.6369°N, 60.0684°E); 4. Parana River, Buenos Aires, Argentina (Coordinates:
33.0958°S, 60.3706°W); 5. Khangai mountains in Mongolia (Coordinates: 48.1886°N, 99.5896°E); see
Figure 6 for legend.
Figure 7-3 compares the WorldCover product versions near Fariman, Iran. This area is mostly bare andsparsely vegetated but also has significant cropland areas. The WorldCover 2020 v100 overestimatedcropland regions by including abandoned fields as croplands. However, cropland overestimation has
been reduced in the WorldCover 2021 v200 version with mapping the non-cultivated areas asbare/sparse vegetation class. Improvements made in wetland areas can also be seen in Figure 7-4
which depicts the wetlands of Parana River of Argentina. Here, WorldCover 2020 v100 underestimatedwetland class, but this has been improved in the WorldCover 2021 v200. Figure 7-5 depicts the Khangai

WorldCover

Product Validation Report | V2.0

Page | 19

mountains of Mongolia. This high altitude region (> 2000m above sea level) had an overestimation oflichen/moss class in the WorldCover 2020 v100. With the improvements in the WorldCover 2021 v200,
the grasslands in the mountain steppes have been correctly characterized.
Since the WorldCover v200 characterizes global land cover for the reference year of 2021, thedynamics of land cover in that year are depicted in this product. Figure 8 shows four examples of land
cover for the year 2021. In Figure 8-1, forest clearings in Mato Grosso, Brazil are correctly depicted;
The forest clearings occurred in the second half of 2020 around the centre west of area are includedin the image snippet. Figure 8-2 depicts a relatively green summer in the transition zones of the Sahara
Desert in Chad for the year 2021; Accordingly, increased grassland areas are captured in theWorldCover 2021 v200. Figure 8-3 shows the swampy marshlands of Ciénaga Grande de Santa Marta,
Columbia; In 2021, this area has seen increased influence of flooding and accordingly, this is depicted
as wetlands in the WorldCover 2021 v200 product. The seasonally flooded salt lakes, Sua-Pan of
Botswana are shown in Figure 8-4; This area has also seen increased water dynamics in 2021 andaccordingly, areas that were continuously flooded in 2021 were mapped correctly as water in theWorldCover 2021 v200, while the rest was mapped as bare/sparse vegetation since the flooding was
temporary.
These examples showcase the up-to-date characterization of land cover by the WorldCover v200
product for the year 2021. Note that the WorldCover v100 for 2020 and v200 for 2021 cannot becompared to identify changes in land cover. The difference in the versions are both due to theimprovements made in the classification algorithm as well as the dynamics between the two years.

WorldCover

Product Validation Report | V2.0

Page | 20

Figure 8: Comparison of the WorldCover 2021 v200 (A) and Sentinel 2 data (B). 1. Mato Grosso, Brazil(Coordinates: 9.9469°S, 59.8814°W); 2. Chad (Coordinates: 16.3122°N, 21.0947°E); 3. Columbia(Coordinates: 9.0344°N, 74.6195°W ); 4. Sua-Pan, Botswana (Coordinates: 20.6461°S, 25.8411°E); 1, 3
& 4 show the false colour image of Sentinel 2 data of 2021 based on median composite; 2 shows afalse colour image of Sentinel 2 data acquired on August 17 2021; see Figure 6 for legend.

WorldCover

Product Validation Report | V2.0

Page | 21

4 Limitations
The statistical validation of the WorldCover 2021 v200 product and the qualitative comparison withthe previous version v100 revealed improved quality of characterizing land cover at a global scale at
10 m resolution. However, there exist some limitations within this product which can be tackled infuture mapping efforts.
As can be seen in Figure 9, there remain some anomalies in the WorldCover 2021 v200 product,
although compared to WorldCover 2020 v100, considerably less. For example, a sharp divide between
cropland and grassland was observed in an agricultural area of Nigeria (Figure 9A). Due to anomaliesof the remote sensing data, sharp divides between grassland and lichen/moss class as well as artefacts
not matching with the landscapes are visible near Russia and Mongolian border (Figure 9B). This area
is generally wetlands, however, the wetland, grassland and lichen/moss transitions do not match withthe landscapes that can be observed in VHR images.

Figure 9: Examples of artefacts in the WorldCover 2021 v200 product. (A) Nigeria (Coordinates:
13.0691°N, 8.6240°E); (B) Russia (Coordinates: 50.6518°N, 94.0797°E); see Figure 6 for legend.
Figure 10 demonstrates examples of areas that can be further improved in future mapping efforts.
The WorldCover 2021 v200 tended to overestimate trees particularly in temperate regions such as
Eurasia and Europe. This is illustrated in Figure 10-1 which shows in Sakha republic of Russia where atundra landscape dominated by lichen/moss class is mostly mapped as trees. Although a closer lookof the VHR image shows the presence of some trees, they are not dominant in these landscapes.
Herbaceous wetlands is a challenging class to characterize accurately due to its high dynamics offlooding level. Figure 10-2 illustrates an example of wetland areas misclassified as cropland in North
Eastern China. Similar misclassification of wetlands as cropland is also visible for example in theestuaries of Licungo river, Mozambique.

WorldCover

Product Validation Report | V2.0

Page | 22

Figure 10: Examples of misclassification in the WorldCover 2021 v200 product. 1. Misclassification oftrees (Coordinates: 68.1514°N, 132.1499°E); 2. Underestimation of wetlands caused by the misassigning of croplands (Coordinates: 48.1329°N, 134.1937°E); see Figure 6 for legend.
The validation of the WorldCover products was done at 10 m resolution. At this high resolution, thegeolocation errors of VHR images used for validation data collection and the labelling uncertainties
can influence the accuracy estimation (Figure 4). We used primary and secondary labels to deal withpotential uncertainties in the validation data. In future efforts, the uncertainties of the validation data
can be quantified and taken into consideration in the accuracy estimation (Foody 2013). The
WorldCover 2021 v200 was validated using the CGLS-LC validation data by updating it to 2021. With
this, the validation fulfils the requirement of Stage 4 validation by the CEOS LPV. Efficient way ofkeeping the validation dataset up-to-date can be further investigated to allow timely assessments of
map products.

WorldCover

Product Validation Report | V2.0

Page | 23

5 Conclusions
This document reports the validation process of the WorldCover 2021 v200 product by ESA. The
validation was based on the CGLS-100 validation dataset, a statistical validation dataset developed aspart of the Copernicus Global Land Service- Dynamic Land cover product. To validate the WorldCover2021 v200 product, this validation dataset was updated to the year 2021 by revisiting a random subsetof the dataset and the sample sites which have higher possibility of land cover change. This allowed
to provide an independent and statistically-robust global and continental accuracy assessment at thetime of the release of the WorldCover 2021 v200 map in October 2022.
Our 10 m resolution validation showed that the overall accuracy of the WorldCover 2021 v200 productis 76.7±0.5%. In terms of land cover types, tree cover, snow/ice, cropland, permanent water body, andbare/sparse vegetation classes had high accuracies, while shrubs, herbaceous wetland, andmoss/lichen classes were mapped with relatively lower accuracies. Overall accuracies at thecontinental level are above 72%, with the highest accuracy of 82.1% for Asia.
Our visual comparison between the WorldCover 2021 v200 and WorldCover 2020 v100 further
confirmed the improved depiction of arid areas and the delineation of cropland, bare/sparsevegetation, grassland and lichen moss classes. The WorldCover 2021 v200 map also depicted the
dynamics that occurred in the year 2021 such as increase in vegetation, water/flooding influence andforest clearings. However, it is worth to note that the WorldCover v100 for 2020 and v200 for 2021
cannot be compared to identify changes in land cover. The difference in the versions are both due tothe improvements made in the classification algorithm as well as the dynamics between the two years.
Overall, the WorldCover 2021 v200 product shows promising improvements in characterizing theWorld’s land cover at 10 m resolution making use of Sentinel 1 and Sentinel 2 data for the year 2021.
Users of the map are encouraged to make use of the statistical accuracy analysis at the global andcontinental level to best apply the WorldCover product for their purposes.

WorldCover

Product Validation Report | V2.0

Page | 24

6 References
Di Gregorio, A. (2005). Land cover classification system: classification concepts and user manual:
LCCS. Rome, Italy: Food and Agriculture Organization of the United Nations
Foody, G.M. (2013). Ground reference data error and the mis-estimation of the area of land coverchange as a function of its abundance. Remote Sensing Letters, 4, 783-792
Fritz, S., McCallum, I., Schill, C., Perger, C., See, L., Schepaschenko, D., van der Velde, M., Kraxner, F.,& Obersteiner, M. (2011). Geo-Wiki: An online platform for improving global land cover.
Environmental Modelling & Software, 31, 13
Gu, J., & Congalton, R.G. (2020). Analysis of the Impact of Positional Accuracy When Using a Single
Pixel for Thematic Accuracy Assessment. Remote Sensing, 12, 4093
Olofsson, P., Foody, G.M., Herold, M., Stehman, S.V., Woodcock, C.E., & Wulder, M.A. (2014). Good
practices for estimating area and assessing accuracy of land change. Remote Sensing of Environment,
148, 42-57
Olofsson, P., Stehman, S.V., Woodcock, C.E., Sulla-Menashe, D., Sibley, A.M., Newell, J.D., Friedl,
M.A., & Herold, M. (2012). A global land-cover validation data set, part I: fundamental designprinciples. International Journal of Remote Sensing, 33, 5768-5788
Pengra, B., Long, J., Dahal, D., Stehman, S.V., & Loveland, T.R. (2015). A global reference databasefrom very high resolution commercial satellite data and methodology for application to Landsat
derived 30m continuous field tree cover data. Remote Sensing of Environment, 165, 234-248
Stehman, S.V., Wickham, J., Smith, J., & Yang, L. (2003). Thematic accuracy of the 1992 National
Land-Cover Data for the eastern United States: Statistical methodology and regional results. Remote
Sensing of Environment, 86, 500-516
Tarko, A., Tsendbazar, N.-E., de Bruin, S., & Bregt, A.K. (2021). Producing consistent visuallyinterpreted land cover reference data: learning from feedback. International Journal of Digital Earth,
14, 52-70
Tsendbazar, N., Herold, M., Li, L., Tarko, A., de Bruin, S., Masiliunas, D., Lesiv, M., Fritz, S., Buchhorn,
M., Smets, B., Van De Kerchove, R., & Duerauer, M. (2021a). Towards operational validation ofannual global land cover maps. Remote Sensing of Environment, 266, 112686
Tsendbazar, N., Linlin, L., Koopman, M., Carter, S., Herold, M., Georgieva, I., & Lesiv, M. (2021b).
WorldCover 2020: Product Validation Report.
https://worldcover2020.esa.int/data/docs/WorldCover_PVR_V1.1.pdf
Tsendbazar, N.E., Herold, M., de Bruin, S., Lesiv, M., Fritz, S., Van De Kerchove, R., Buchhorn, M.,
Duerauer, M., Szantoi, Z., & Pekel, J.F. (2018). Developing and applying a multi-purpose land covervalidation dataset for Africa. Remote Sensing of Environment, 219, 298-309
Wickham, J., Stehman, S., Fry, J., Smith, J., & Homer, C. (2010). Thematic accuracy of the NLCD 2001
land cover for the conterminous United States. Remote Sensing of Environment, 114, 1286-1296

WorldCover

Product Validation Report | V2.0

Page | 25

Wickham, J., Stehman, S.V., Sorenson, D.G., Gass, L., & Dewitz, J.A. (2021). Thematic accuracy
assessment of the NLCD 2016 land cover for the conterminous United States. Remote Sensing ofEnvironment, 257, 112357

WorldCover

Product Validation Report | V2.0

Page | 26

Annex 1.
Table A1.1 List of regional experts that contributed to the validation data collection
Continent Regions
Experts and their degree Affiliation
Greenland and
1 NorthCanadaAlemu Gonsamo (PhD) University of Toronto, Canada
2 America
MexicoLuis Valderrama (PhD)
CONABIO, Mexico
3
USA
Yinan He (PhD student) University of North Carolina, Charlotte, the USA
Konstantin Ivushkin (PhD
4Centrestudent)
Wageningen University, The Netherlands
5
East
Hongrun Ju (PhD student) IRSDE, Chinese Academy of Science, China
Asia
6
South
Hammad Gilani (PhD)
ICIMOD, Nepal and University of Illinois, the USA
7
South-East
Tony Widianto (MSc)
Regional Planning, R&D, Jawa-Timur, Indonesia
8
West
Ali Abkar (PhD)
Agriwatch, The Netherlands
Eastern Russia Eduard
Batotsyrenov Baikal Institute of Nature Management, Russian
9(Siberia)(MSc)
Academy of Science, Russia
10
Kazakhstan
Tom Bewernick (MSc)
GIS consultant, The Netherlands
Northern
Institute of Geography, Mongolian Academy ofEurasia11
Mongolia
Sainbuyan (MSc)
Science, Mongolia
Western
Maria Shchepashchenko Russian Institute of Continuous Education in
12
Russia(PhD)
Forestry, Russia
Australia
Nirmala Liyanage (PhD) University of Sydney, Australia
13
Australia
Bethany Melville (PhD) University of Tasmania, Australia
Oceania
New Zealand14
and Oceania Vega Xu (PhD)
University of Canterbury, New Zealand
Caribbean and
15
North
Kenneth Jimenez (MSc) University of Costa Rica, Costa Rica
East
George Longhitano (MSc) G Drones, Brazil
South
Gustavo Alckmin (PhD
16 America
East
student)
Wageningen University, The Netherlands
RedLaTIF / Agencia de Extensión Rural Cruz del Eje,
17
South
Nicolas Mari (MSc)
INTA, Argentina
18
West
Roberto Chavez (PhD)
Pontifical Catholic University of Valparaíso, Chile
Agnieszka Tarko (PhD19
Eaststudent)
Wageningen University, The Netherlands
20 Europe
North
Paola Codipietro (MSc) GIS consultant, Rome, Italy
21
South
Nadine Drigo (MSc)
Wageningen University, The Netherlands
22
West
Tim Jak (MSc)
Wageningen University, The Netherlands
HoLiN Training and Consultancy Services PLC,
23
East
Elias Buzayane (MSc)
Ethiopia
Institute of Experimental Ecology, University of
24
West
Matt Herkt (PhD student) Ulm, Germany
Universidade Eduardo Mondlane, MIOMBO and25 Africa
South
Natasha Ribeiro (PhD)
GOFC-GOLD network
26
CentreAndre Mazinga (BSc)
OSFAC, DRC
MarienNgouabi
University,
Brazzaville,
CentreIfo Suspence (PhD)
27
République du Congo.
Emmanuel Amoah BoakyeWest
WASCAL, Accra, Ghana
28
(MSc)

WorldCover

Product Validation Report | V2.0

Page | 27

Annex 2.

Correct

Total

User's accuracy

Confidence
interval ±

8.9

18.4

48.5

3.5

0.0

15.2

21.5

70.7

3.0

0.0

4.5

6.3

71.3

4.7

0.4

0.9

47.1

9.7

29.8

30.9

96.6

1.4

0.0

1.2

1.3

91.0

2.8

0.6

0.0

0.6

0.9

63.3

14.4

0.0

0.0

0.1

0.1

0.2

74.5

13.8

29.8

1.2

0.6

0.1

32.9

1.3

1.2

0.2

84.0

90.6

91.5

45.7

56.0

5.3

2.2

4.6

12.9

27.1

1.5

1.8

0.4

0.0

0.0

0.0

0.1

8.9

5.1

0.6

0.0

0.8

0.0

0.1

1.3

15.2

1.9

0.0

1.9

0.0

0.4

0.2

0.3

1.1

4.5

0.0

0.2

0.0

0.1

Built-up

0.1

0.0

0.2

0.0

0.4

0.2

0.0

Bare /
sparse veg

0.0

0.2

0.7

0.1

0.0

29.8

0.0

0.0

Permanentwaterbodies

0.0

0.0

0.0

0.0

0.0

0.1

1.2

0.0

Herbaceous
wetland

0.0

0.0

0.2

0.1

0.0

0.0

Mangroves

0.0

0.0

0.0

0.0

Correct

15.8

8.9

15.2

4.5

0.4

Total

19.8

12.2

24.2

7.6

0.5

Producer's
accuracy

80.1

73.1

62.7

58.9

Confidence
interval ±

2.0

3.6

3.1

4.9

Tree cover

Herbaceous
wetland0.0

Permanent
water bodies

2.0

Bare / sparse
veg

80.3

Built-up

19.7

Cropland

15.8

Grassland

0.1

Shrubland

Mangroves

Table A2.1 Confusion matrix (%) for the WorldCover 2021 v200 product for Africa, corrected by sample
inclusion probabilities.

Tree cover

15.8

Shrubland

2.9

Grassland

0.7

Cropland

76.6
1.3

Shrubland

Grassland

Cropland

Built-up

Bare / sparseveg

Permanent
water bodies

Herbaceous
wetlandMoss and
LichenCorrect

Total

User'saccuracy

Confidence
interval ±Tree cover

32.4

2.2

4.7

0.8

0.5

0.3

0.1

0.1

0.0

32.4

41.0

79.1

1.4

Shrubland

0.2

0.5

0.5

0.0

0.0

0.1

0.0

0.0

0.0

0.5

1.3

36.4

8.0

Grassland

1.3

1.1

17.0

3.6

0.2

0.6

0.1

0.3

0.5

17.0

24.5

69.2

2.5

Cropland

0.2

0.2

1.8

21.4

0.1

0.1

0.0

0.0

21.4

23.8

89.9

1.7

Built-up

0.1

0.0

0.2

0.1

2.0

0.1

0.0

0.0

2.0

2.4

81.6

2.9

Bare /
sparse veg

0.0

0.0

0.1

0.1

0.0

0.7

0.0

0.0

0.0

0.1

0.7

1.1

67.7

9.2

0.1

0.7

0.0

0.0

0.7

0.8

81.0

7.0

Snow andice

Snow and ice

Tree cover

Table A2.2 Confusion matrix (%) for the WorldCover 2021 v200 product for Europe, corrected by sample
inclusion probabilities.

WorldCover

Product Validation Report | V2.0

Permanentwaterbodies

0.0

0.0

0.0

Herbaceous
wetland

0.0

0.0

0.3

Moss and
Lichen

0.0

0.0

0.6

Correct

32.4

0.5

17.0

21.4

Total

34.2

4.1

25.2

Producer's
accuracy

94.9

11.5

Confidence
interval ±

0.7

3.6

Page | 28

0.0

0.1

0.0

2.4

0.0

0.0

2.4

2.5

95.1

1.8

0.0

0.0

0.0

0.0

0.3

0.0

0.3

0.7

39.1

12.5

0.5

0.0

0.0

0.0

0.6

0.6

1.8

33.9

10.1

2.0

0.7

0.7

2.4

0.3

0.6

26.0

2.8

2.6

0.7

2.7

0.6

1.2

67.5

82.3

71.2

28.0

93.4

89.5

41.7

51.6

2.2

2.1

3.2

4.9

3.9

3.1

11.5

12.1

0.1

78.0
1.0

Bare / sparseveg

Permanent
water bodies

Herbaceous
wetlandMoss and
LichenCorrect

Total

User's accuracy

0.1

0.2

0.1

0.4

33.6

43.9

76.6

1.9

0.0

0.1

0.0

0.0

0.0

0.6

1.2

52.2

11.2

0.6

0.0

1.6

0.1

0.5

1.8

21.4

27.7

77.2

2.2

1.2

4.7

0.0

0.1

0.0

0.0

4.7

5.9

79.0

5.6

0.0

0.1

0.0

0.1

0.0

0.0

0.1

0.2

57.4

7.7

0.0

0.4

0.0

0.0

4.0

0.0

0.0

0.2

4.0

4.6

86.1

3.5

0.1

0.2

0.0

0.0

0.2

0.3

78.6

24.0

Grassland

33.6

4.2

5.2

Shrubland

0.1

0.6

0.3

Grassland

0.6

1.1

21.4

Cropland

0.0

0.0

Built-up

0.0

Bare /
sparse veg

0.0

Snow andice

0.0

Permanentwaterbodies

0.0

0.0

0.0

Herbaceous
wetland

0.1

0.3

2.1

Moss and
Lichen

0.2

0.3

2.1

Correct

33.6

0.6

21.4

4.7

Total

34.7

6.7

32.7

Producer's
accuracy

97.0

9.7

Confidence
interval ±

0.7

3.2

0.0
0.0

0.0

Confidence
interval ±Built-up
0.0

Tree cover

Snow and ice

Cropland
0.1

ShrublandTree cover

Table A2.3 Confusion matrix (%) for the WorldCover 2021 v200 product for Eurasia, corrected bysample inclusion probabilities.

0.1

3.4

0.1

0.1

3.4

3.7

93.0

2.9

0.0

0.2

1.4

1.7

1.4

6.0

23.7

5.7

3.0

6.5

46.3

6.1

0.9

0.0

0.0

0.1

3.0

0.1

4.0

0.2

3.4

1.4

3.0

5.4

0.2

6.9

0.2

3.9

2.2

7.2

65.4

86.6

72.0

57.9

98.7

86.6

64.3

41.9

2.3

4.9

8.2

5.0

1.6

3.5

9.4

5.9

72.5
1.3

WorldCover

Product Validation Report | V2.0

Page | 29

Bare /
sparse veg

0.0

0.3

1.5

0.2

0.1

Snow andice

0.0

Permanentwaterbodies

0.0

0.0

0.0

0.0

Herbaceous
wetland

0.0

0.0

0.1

0.1

Mangroves
Correct
Total

0.0
24.6
26.3

0.0
0.4
3.4

0.0
11.1
17.1

12.1
14.7

Producer's
accuracy

93.4

10.8

64.8

Confidence
interval ±

1.0

4.1

3.0

0.0

0.1
0.0
0.1
0.1
0.0

0.1
0.0
0.2
0.3
0.0

0.1
0.0
0.0
0.0

30.3

0.0

0.0

0.0

0.0

0.4

0.2

0.0

Confidence
interval ±

0.2
0.2
1.8
0.3
0.2

User'saccuracy

0.2
0.0
0.1
0.1
1.1

Total

Bare / sparse
veg

1.0
0.1
1.0
12.1
0.1

Correct

Built-up

2.4
0.6
11.1
1.4
0.1

Mangroves

Cropland

1.6
0.4
0.9
0.2
0.0

Herbaceous
wetlandGrassland

24.6
0.2
0.8
0.6
0.1

Permanent
water bodies

ShrublandTree cover
Shrubland
Grassland
Cropland
Built-up

Snow and ice

Tree cover

Table A2.4 Confusion matrix (%) for the WorldCover 2021 v200 product for Asia, corrected by sample
inclusion probabilities.

24.6
0.4
11.1
12.1
1.1

30.3
1.5
16.0
15.0
1.6

81.3
24.4
69.4
80.6
68.0

1.8
7.7
2.9
2.8
4.0

30.3

32.4

93.5

1.1

0.4

0.4

90.0

4.2

1.9

0.1

0.0

1.9

2.3

83.4

6.2

0.0

0.0

0.1

0.0

0.1

0.3

35.2

22.0

0.0
0.1
0.8

0.1
0.1
0.2

0.2

76.7

28.1

0.4
0.4

0.0
1.9
2.3

0.1

1.1
1.5

0.0
30.3
33.3

82.7

71.6

91.0

93.9

82.1

12.9

63.4

2.5

5.0

1.2

4.9

5.2

9.5

35.8

0.0

82.1
1.0

Herbaceous
wetland

Mangroves

Correct

Total

User'saccuracy

Confidence
interval ±

64.1

2.5

Tree cover

14.0

2.1

5.4

0.0

0.0

0.2

Shrubland

1.1

7.5

6.2

0.0

0.0

1.1

0.0

0.0

Grassland

0.8

2.0

43.2

0.6

0.0

5.1

0.0

0.2

Cropland

0.0

0.0

0.8

3.2

0.0

0.0

Built-up

0.0

0.0

0.0

0.1

Bare /
sparse veg

0.0

0.1

1.3

0.0

Snow andice
Permanentwaterbodies

0.0

0.0

0.0

Snow andice

Bare /
sparse veg

21.9

Built-up

14.0

Cropland

0.0

Grassland

0.0

Shrubland

0.0Tree cover

Permanent
water

Table A2.5 Confusion matrix (%) for the WorldCover 2021 v200 product for Oceania & Australia,
corrected by sample inclusion probabilities.

7.5

15.9

47.1

3.6

0.0

43.2

52.0

83.2

1.5

0.0

3.2

4.1

79.5

6.7

0.0

0.0

0.1

0.1

82.3

10.0

4.1

0.0

4.1

5.5

74.3

4.8

0.0

0.0

0.0

0.0

0.0

0.1
0.3

0.0

0.0

0.3

0.3

0.0
86.9

6.7

WorldCover

Product Validation Report | V2.0

Herbaceous
wetland

0.0

0.0

0.1

0.0

0.0

Page | 30

0.1

0.0

0.01

0.0

0.0

0.2

5.1

5.8

0.1

0.1

77.5

21.6

Mangroves

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.1

Correct

14.0

7.5

43.2

3.2

0.1

4.1

0.3

0.0

0.1

Total

16.0

11.7

56.9

3.8

0.2

10.7

0.3

0.2

0.1

Producer's
accuracy

87.6

63.8

75.9

84.6

65.8

38.2

77.0

3.6

81.9

Confidence
interval ±

1.9

4.1

1.8

5.2

14.1

4.6

13.4

3.4

19.4

72.5
1.3

0.0

0.3

0.4

0.0

0.0

4.6

0.0

0.7

12.0

Snow andice
Permanentwaterbodies

0.0

0.0

0.1

Herbaceous
wetland

0.0

0.1

0.4

0.0

Confidence
interval ±Bare /
sparse veg

User's accuracy

0.2
0.5
0.5
0.1
0.0

Total

0.1
0.0
0.1
0.0
0.5

Correct

Bare / sparse
veg

0.1
0.0
0.8
6.8
0.0

Moss and
LichenBuilt-up

2.8
1.5
12.7
0.7
0.1

Mangroves

Cropland

2.2
3.2
2.2
0.0
0.0

Herbaceous
wetlandGrassland

24.4
1.0
1.1
0.0
0.0

Permanent
water bodies

ShrublandTree cover
Shrubland
Grassland
Cropland
Built-up

Snow and ice

Tree cover

Table A2.6 Confusion matrix (%) for the WorldCover 2021 v200 product for North America, correctedby sample inclusion probabilities.

0.1
0.0
0.2
0.0
0.0

0.1
0.0
0.3
0.0
0.0

0.0
0.0
0.0

0.3
0.1
2.5

24.4
3.2
12.7
6.8
0.5

30.3
6.2
20.3
7.6
0.7

80.6
51.0
62.4
88.8
72.9

2.0
4.5
3.0
3.1
5.1

0.1

1.3

4.6

6.7

68.4

4.6

0.0

0.1

12.0

12.7

93.8

2.5

0.1

4.6

0.1

0.2

4.6

5.1

90.0

3.1

0.0

0.2

0.5

1.1

0.5

2.3

23.2

7.7

0.0

0.0

86.1

25.3

5.3

8.0

67.0

4.9

Mangroves

0.01

0.0

Moss and
Lichen

0.1

0.1

1.2

Correct
Total

24.4
26.7

3.2
8.1

12.7
19.8

6.8
7.8

Producer's
accuracy

91.6

39.3

64.0

Confidence
interval ±

1.4

4.6

2.9

5.3

1.2

0.0

0.1

0.0

0.5
0.7

4.6
7.8

12.0
12.0

4.6
5.2

0.5
1.0

0.0
0.0

5.3
10.9

87.5

70.1

59.0

99.8

87.9

51.5

54.3

48.9

3.1

5.7

4.8

0.2

2.7

13.9

42.4

4.5

74.6
1.2

WorldCover

Product Validation Report | V2.0

Page | 31

Shrubland

Grassland

Cropland

Built-up

Bare / sparse veg

Permanent water
bodiesHerbaceous
wetland

Mangroves

Correct

Total

User's accuracy

Confidence
interval ±Tree cover

42.4

3.8

2.6

0.4

0.0

0.2

0.1

0.1

0.1

42.4

49.7

85.2

1.4

Shrubland

1.1

5.2

2.0

0.1

0.0

0.7

0.0

0.0

0.0

5.2

9.2

57.1

4.1

Grassland

0.9

2.1

16.7

1.1

0.0

2.0

0.1

0.9

0.1

16.7

23.9

69.9

2.5

Cropland

0.1

0.1

1.7

6.3

0.0

0.1

0.0

0.1

6.3

8.3

75.6

4.6

Built-up

0.0

0.0

0.0

0.0

0.3

0.0

0.0

0.3

0.4

75.5

6.9

0.0

0.1

0.2

0.0

5.0

0.1

5.0

5.4

92.5

2.5

0.0

0.2

0.2

0.2

85.1

8.1

0.1

0.0

Bare /
sparse veg
Snow andice
Permanentwaterbodies0.0

Snow and ice

Tree cover

Table A2.7 Confusion matrix (%) for the WorldCover 2021 v200 product for South America, correctedby sample inclusion probabilities.

0.0

0.0

0.0

0.0

1.3

0.1

0.0

1.3

1.5

85.8

5.5

0.0

0.0

0.6

0.1

0.6

1.4

43.0

12.1

0.0

0.0

0.0

0.1

0.1

0.1

66.2

33.2

0.2

1.3

0.6

0.1

8.1

0.2

1.6

1.8

0.4

74.6

61.3

69.7

81.4

33.5

21.9

8.4

4.8

20.5

4.7

9.8

18.6

0.0

0.0

0.0

0.0

Herbaceous
wetland

0.1

0.1

0.5

0.0

Mangroves

0.0

0.0

0.0

Correct

42.4

5.2

16.7

6.3

0.3

5.0

Total

44.7

11.4

23.7

7.7

0.4

Producer's
accuracy

94.9

46.0

70.3

81.1

Confidence
interval ±

0.8

4.0

2.5

4.3

0.0

77.9
1.1