2013 12th International Conference on Document Analysis and Recognition

ICDAR 2013 Robust Reading Competition
Dimosthenis Karatzas∗ , Faisal Shafait† , Seiichi Uchida‡ , Masakazu Iwamura§ , Lluis Gomez i Bigorda∗ ,Sergi Robles Mestre∗ , Joan Mas∗ , David Fernandez Mota∗ , Jon Almazàn Almazàn∗ and Lluis Pere de las Heras∗∗ Computer Vision Centre, Universitat Autònoma de Barcelona; dimos@cvc.uab.es† The University of Western Australia, Australia; faisal.shafait@uwa.edu.au‡ Kyushu University, Japan; uchida@ait.kyushu-u.ac.jp§ Osaka Prefecture University, Japan; masa@cs.osakafu-u.ac.jp

lines of the ICDAR 2013 competition), on-line performance evaluation tools and enhanced visualisation of
results.

Abstract—This report presents the final results of the ICDAR2013 Robust Reading Competition. The competition is structured
in three Challenges addressing text extraction in different application domains, namely born-digital images, real scene images
and real-scene videos. The Challenges are organised around
specific tasks covering text localisation, text segmentation andword recognition. The competition took place in the first quarterof 2013, and received a total of 42 submissions over the different
tasks offered. This report describes the datasets and ground truth
specification, details the performance evaluation protocols usedand presents the final results along with a brief summary of theparticipating methods.

I.

The competition consists of three Challenges: Reading Text
in Born-digital Images (Challenge 1), Reading Text in Scene
Images (Challenge 2), and Reading Text in Videos (Challenge3). Each Challenge is based on a series of specific tasks.
Challenges 1 and 2 offer tasks on Text Localisation, where
the objective is to detect the existence of text and return abounding box location of it in the image; Text Segmentation,
where the objective is to obtain a pixel-level separation of text
versus background and Word Recognition, where the objective
is to automatically provide a transcription for a list of prelocalised word images. Challenge 3 at this time offers a single
task on Text Localisation where the objective is to detect andtrack text objects in a video sequence.

I NTRODUCTION

Text extraction from versatile text containers like borndigital images, real scenes and videos has been a continuousinterest in the field for more than a decade. The series of Robust Reading Competitions addresses the need to quantify andtrack progress in this domain. The competition was initiatedin 2003 by Simon Lucas, focusing initially on text localisationand text recognition in real scene images [1] [2]. The firstedition was met with great success and was repeated in 2005[3], creating a reference framework for the evaluation of text
detection methods. In 2011, two challenges were organised
under the ICDAR Robust Reading Competition, one dealing
with text extraction from born-digital images [4], and the otherfrom real scene images [5]. The 2013 edition of the Robust
Reading Competition [6] marks a new milestone in the series.
The two challenges on real scenes and born-digital images
have been integrated further, unifying performance evaluation
metrics, ground truth specification and the list of offered tasks.
In parallel, a new challenge is established on text extraction
from video sequences, introducing new datasets, tools andevaluation frameworks.

This report presents the final results after analysing the submissions received. The presentation is following the structureof the competition and results are grouped by Challenge. For
each Challenge the datasets used and ground truth specification
are briefly described, the performance evaluation protocol isdetailed and finally the results are presented and analysed.
First, the competition protocol is briefly described in Section II.
Section III is dedicated to Challenge 1, Section IV to Challenge
2 and Section V to Challenge 3. Overall conclusions arepresented in Section VI. Finally, in the appendix the technicaldetails of all participating methods are summarised.
II.

The competition was run in open mode, meaning that
results over the test set were requested by the authors, andnot executables of their systems. At all times we relied on
the scientific integrity of the authors to follow the rules of thecompetition. The authors were free to participate in as many
Challenges and Tasks as they wished. They were allowed tomake multiple submissions to the same task as well. In the caseof submission of different methods from the same authors,each method is separately described and ranked in the final
results presented here. In the case where the results of different
variants of the same base method were submitted, only the best
performing variant is shown in the ranking tables of this report.
A full ranking including all different variants submitted will
be available in the competition Web after ICDAR 2013.

The 2013 Robust Reading Competition [6] brings to theDocument Analysis community:
•
•
•
•
•
•
•

An enhanced dataset on born-digital images
New ground truth at the pixel level for real-scene
images
Improved and intuitive performance evaluation protocols
A new dataset of video sequences obtained under various activities and with diverse capturing equipment
Video based ground truth for text detection in video
A single point of entry for all submissions to all
challenges
A comprehensive Web site allowing the continuous
submission of new results (over and above the dead-

1520-5363/13 $26.00 © 2013 IEEE
DOI 10.1109/ICDAR.2013.221

C OMPETITION P ROTOCOL

In total, 52 submissions were made (42 after excluding
variants of the same method) to the different Challenges andTasks of the competition from 15 individual participants.
1484

III.

C HALLENGE 1: R EADING T EXT IN B ORN -D IGITALI MAGES (W EB AND E MAIL )1) Task 1 - Text Localization: For the evaluation of text
localisation results we make use of the framework proposedby Wolf and Jolion [12]. The key principle of the scheme is that
evaluation is done at the object (word bounding box) level over
the whole collection, taking into account the quality of eachmatch between detected and ground truth text boxes. Matches
are first determined based on area overlapping, given certain
minimum quality thresholds. Then different weights for oneto-one, one-to-many and many-to-one matches are used when
pooling together the results.

Challenge 1 focuses on the extraction of textual content
from born-digital images, such as the ones used in Web
pages and email messages. Embedding text in images, rather
than encoding it explicitly in the electronic document, is afrequent choice of content authors. Text embedded in images
is practically invisible to any automatic processes, hindering
applications such as indexing and retrieval, content filteringetc.

Two thresholds on the area precision (tp ) and area recall(tr )control the way matches between ground truth rectangles
are determined. For this challenge, we use the default values
suggested in [12] for these thresholds, namely tr = 0.8
and tp = 0.4. For calculating the overall object Precision
and Recall the method considers all matches over the wholecollection. One-to-many and many-to-one matches can be
assigned different weights allowing the uneven penalisationof different behaviours of the method under question. In
this implementation, we give a lower weight to one-to-many
matches than the rest. The rationale behind this decision isthat we make use of the word level of our ground truth forevaluation; therefore, although we want to penalise methods
that produce multiple rectangles for a single ground truth word,
we do not want to penalise methods designed to produce results
at the text-line level, detecting many ground truth words ofthe same line with a single rectangle. Hence, for many-toone matches we do not inflict any penalty, while for oneto-many matches we use the suggested fixed weight of 0.8.
We encourage the interested reader to review [12] for further
details.

Challenge 1 of this competition aims to quantify the stateof the art in text extraction from born-digital images, and tohighlight progress since the last edition of the competition [4].
Notably, since the past edition of this challenge in 2011, a
new body of work has been published on the topic [7], [8],[9], [10]. Comparing to the last edition of the challenge, thisedition introduces an updated dataset as well as small changes
in the evaluation protocol that make it more comprehensive
and intuitive.
In the rest of this section we present the results overthree independent tasks related to text extraction from borndigital images: text localization, text segmentation and word
recognition. Section III-A describes the dataset and groundtruth used for the competition, while Section III-B details
the performance evaluation methodology followed. In Section
III-C we present the results and make some key observations.
A. Dataset
The dataset used for this competition comprises images
extracted from Web pages and email messages. We selected
a representative sample of Web pages of different categories(news, personal, commercial, social, government, etc) andemails of different type (spam, newsletters, etc) in proportions
that reflect their real world usage.

2) Task 2 - Text Segmentation: For the evaluation of text
segmentation results we make use of the framework proposedby Clavelli and Karatzas [11]. The units for the evaluation are
the atoms defined in the ground truth. An atom is the smallest
combination of text parts (connected components) that can be
assigned a transcription; therefore an atom may comprise one
or multiple text parts. See [11] for an extensive discussion
on atoms and their definition. The quality of a segmentation
method is measured by the degree to which it is able toproduce regions that preserve the morphological properties of
the ground-truth atoms, as opposed to simply counting thenumber of mislabelled pixels. In simple terms, the framework
is more permissive to segmentation methods that produce text
parts that are slightly more dilated or more eroded versions of
the ground truth ones, while it penalises methods that produce
text parts that have distinctly different shapes from the groundtruth ones, even if the same number of mislabelled pixels is
involved in both cases.

Overall, we analysed 315 Web pages, 22 spam and 75
ham emails, and extracted all the images that contained text.
We selected a subset of 561 images with a minimum size of
100x100 pixels. The collection was split into a training set of
420 images (same as the 2011 edition) and a test set of 141
images (including new images compared to the 2011 edition).
For the word recognition task only words with a length of 3
characters or more were considered. The dataset contains 5003
such words, out of which 3564 comprise the training set and
1439 the test set.
Ground truth information was created in an hierarchicalway spanning all levels from pixel level labelling to text parts,
atoms, words and text lines, as described in [11]. In cases
where pixel level labelling is not possible (e.g. extreme antialiasing or very low resolution) word and text line boundingboxes are directly defined. All bounding boxes are defined asaxis-aligned isothetic rectangles, a reasonable design choice as
in born-digital images most text is horizontal.

In order to establish a match between ground truth anddetected text parts, two conditions are to be satisfied. The
Minimal Coverage condition is satisfied if the detected text
part covers a minimum set of pixels of the ground truth text
part. In the implementation used here this is defined as apercentage of the area of the text part and controlled by aparameter Tmin . The maximal coverage criterion is satisfied if
no pixel of the detected text part lies outside a maximal area,
defined as a dilated version of the ground truth text part. In thecurrent implementation, a parameter Tmax controls the amountof dilation considered as a function of the stroke width of theB. Performance Evaluation
We use standard metrics for the evaluation of the different
tasks. These are largely shared between Challenges 1 and 2,
facilitating comparison between the two. A brief explanationof the evaluation protocols is given in this section.

1485

TABLE I.

component. For this evaluation, Tmin was set to 0.5 and Tmax
to 0.9. Each of the atoms in the ground truth is classified
as Well Segmented, Merged, Broken, Broken and Merged or
Lost, while False Positive responses are also counted. Atom
Precision, Recall and F-score metrics are calculated over thewhole collection summarising the performance of the method.
Word boxes that are directly defined at the word level inthe ground truth, and do not contain atoms or text parts,
are treated as “don’t care” regions and discounted from theevaluation. The interested reader is encouraged to read [11]for more details. Apart from atom level metrics, we also report
Precision, Recall and F-score results at the pixel level forcompleteness.

R ANKING OF SUBMITTED METHODS TO TASK 1.1

Method NameUSTB TexStar
TH-TextLoc
I2R NUS FAR
Baseline
Text Detection [15], [16]I2R NUSBDTD CASIAOTCYMIST [7]Inkam

TABLE III.
Method3) Task 3 - Word Recognition: For the evaluation of Word
Recognition results we implemented a standard edit distancemetric, with equal costs for additions, deletions and substitutions. For each word we calculate the normalized edit distance
between the ground truth and the submitted transcription and
we report the sum of normalised distances over all words ofthe test set. The normalisation is done by the length of theground truth transcriptions. The comparison we performed iscase sensitive. To assist qualitative analysis, we also provide
statistics on the number of correctly recognised words.

USTB FuStar
I2R NUSOTCYMISTI2R NUS FAR
Text Detection

Recall (%)
82.38
75.85
71.42
69.21
73.18
67.52
67.05
74.85
52.21

Precision (%)
93.83
86.82
84.17
84.94
78.62
85.19
78.98
67.69
58.12

F-score
87.74
80.96
77.27
76.27
75.81
75.34
72.53
71.09
55.00

A NALYSIS OF TASK 1.2 RESULTS .

Well
Segmented
6258
5051
5143
4619
3883

MergedBroken

Lost

920
1584
1420
1474
2716

56
30
34
12
36

587
1151
1223
1716
1187

False
Positives
370
685
1083
156
210

As it can be easily observed, most participating methods
rank below the baseline method, indicating that existing commercial solutions are doing reasonably well with these images.
Notably, at least 3 methods rank better than the baseline
one, with 2 of them (USTB TexStar and TH-TexLoc) doing
significantly better than the commercial solution.

C. Results and Discussion
Overall, 17 methods from 8 different participants (excluding variants of the same method) were submitted in the various
tasks of this challenge. In the sections below we provide thefinal ranking tables as well as a short analysis of the results foreach task. The participating methods are referred to by namein the ranking tables and in the text, please see Appendix VI
for technical details on the participating methods.

Using the performance of the baseline method as a yardstick (largely the same between the two competitions), an
indirect comparison to the results of 2011 is possible. During
the past edition [4] no method performed substantially better
than the baseline method, with the highest ranking methods(including the baseline) yielding results within 1% of eachother. Since the past edition many new methods were submitted online in the Challenge Web site [14], indicating an
improvement of up to 10% over the 2011 results had beenachieved. This is confirmed with this edition, with the top
ranking method performing 11.5% better than the baseline
method.

Similarly to the past edition of the competition, we haveincluded the performance of an out-of-the-box commercial
OCR software package as a baseline for text localisation (task1) and word recognition (task 3). For this purpose we haveused the ABBYY OCR SDK (version 10) [13]. Factory default
parameters were used for pre-processing with the following
exceptions. First, we enabled the option to look for text in low
resolution images, since the resolution of born-digital images
is typically below 100DP I. This is not exclusive, meaningthat text in high resolution images is also looked for withthis parameter enabled. Second, we set the OCR parameters
FlexiForms and FullTextIndex to true, to force detection of
text inside images. For text localization (task 1) we haveused the reported location of individual words, while for word
recognition (task 3) the returned transcription.

2) Task 2 - Text Segmentation: The final results for the
Text Segmentation task are shown in Table II. The ranking
metric used for the Text Segmentation task is the atom F-Score(column in grey) calculated as explained in section III-B2. All
metrics are calculated cumulatively over the whole test set. For
completeness, apart from the atom based metrics, table II also
shows the pixel Precision, Recall and F-score.

Please note that a direct comparison of the results reported
here to the 2011 edition of the challenge (and the additional
submissions received online between 2011 and 2013 [14]) is
not straightforward as the test dataset has been updated to include more images, while small changes have been introducedin the evaluation protocol.

Table III summarises the classification of the atoms aswell as the number of false positives produced by each of
the methods. It can be easily observed that the USTB FuStar
produces significantly more Well Segmented atoms than therest of the methods. At the same time it produces less Merged
atoms and quite a lot more Broken ones. Compared to thisperformance, all the rest of the methods seem to have atendency to over-merge.

1) Task 1 - Text Localization: The final results for the Text
Localisation task are shown in Table I. The ranking metric used
for the Text Localisation task is the F-score (column in grey)
calculated according to the methodology described in section
III-B1. All metrics are calculated cumulatively over the wholetest set (all detections over all 141 images pooled together).

In terms of Lost atoms (atoms that were not detected at all,or were only partially detected), again USTB FuStar seems tobe performing the best. In most of the cases, the participating
methods fail to detect the dots of the letters “i” and “j”, hence
missing most of the atoms corresponding to such characters.
Similarly, punctuation points seem to create problems to most1486

TABLE II.
MethodUSTB FuStar
I2R NUSOTCYMISTI2R NUS FAR
Text Detection

TABLE IV.

R ANKING OF SUBMITTED METHODS TO TASK 1.2

Recall (%)
87.21
87.95
81.82
82.56
78.68

Pixel Level
Precision (%)
79.98
74.40
71.00
74.31
68.63

R ANKING OF SUBMITTED METHODS TO TASK 1.3

MethodPhotoOCRMAPS [17]
PLT [18]
NESP [19]Baseline

Total Edit
Distance
105.5
196.2
200.4
214.5
409.4

Recall
80.01
64.57
65.75
59.05
49.64

Atom Level
PrecisionF-score
86.20
82.99
73.44
68.72
71.65
68.57
80.04
67.96
69.46
57.90

a unique color. Consequently, the pixel-level ground-truth canbe used for ground-truth of individuial character segmentation
as well as ground-truth of character detection. The operators
have made a double-check for more reliable ground-truth.

Correctly Recognised
Words (%)
82.21
80.4
80.26
79.29
60.95

One differece from Challenge 1 on attaching pixel-levelground-truth is that a single connected component showing
multiple characters (e.g., the connected component showing“oca” in the Coca-Cola logo) is segmented into individual
characters and then painted mutiple colors. (Thus, the “oca”
will be colored by three different colors.) This segmentation
is done rather intuitively.

of the methods. Finally, in terms of False Positives, the method
I2R NUS FAR is the best performing one, as is also indicatedby the high precision it yields in table II.

The other difference is that “don’t care” regions are introduced in the ground-truth. Scene images often contain illegible
tiny text regions. In ICDAR2011 dataset, each legible text
region was marked by a bounding box, whereas illegible text
regions were left unmarked. We have introduced the conceptof “don’t care” regions in ICDAR2013 dataset, where tiny text
regions are marked with a special colored rectangle. To keep
consistensy with ICDAR2011 competition as much as possible,
only those text regions are marked as don’t care that were notmarked as text in the previous dataset.

3) Task 3 - Word Recognition: The results for the Word
Recognition task are shown in Table IV. Two metrics areshown, namely the total edit distance, which is also used for the
final ranking (column in grey) and the percentage of correctly
recognised words as explained in section III-B3. Overall, we
can note a significant advancement of the state of the art sincethe 2011 edition. In the previous edition of the competition
there was a single participating method with performance very
close to the baseline, recognising correctly a 61.54% of thewords. The best performing methods in this edition show animportant improvement and an increment of 20% in terms ofcorrectly recognised words.
IV.

F-score
83.44
80.61
76.03
78.22
73.32

B. Performance Evaluation
The performance evaluation schemes for all tasks in Challenge 2 follow the same protocols as those in Challenge 1.
The only difference is in handling of “don’t care” regions.
Don’t care regions are treated during a pre-processing phase.
Before the core evaluation starts overlapping with don’t careregions is checked and any detected regions that fall inside
don’t cares are removed from the results list. This makes surethat no false alarms are reported if an algorithm marks a don’tcare region as text. On the other hand, another algorithm that
does not consider a don’t care region as text also does not get
penalized.

C HALLENGE 2: R EADING T EXT IN S CENE I MAGESIn this edition, a new task of Text Segmentation (Task
2.2) is introduced in Challenge 2. For this task, a pixel-levelground-truth has been created for scene images in the dataset.
The participating methods are asked to provide a pixel-levelsegmentation of the given real scene images such that all pixels
belonging to text are marked as fore-ground and other pixels
as background.
A. Dataset

C. Results and Discussion

The scene image dataset used for this competition isalmost the same as the dataset at ICDAR2011 Robust Reading
Competition. The difference from the ICDAR2001 dataset isrevision of ground-truth texts at several images. In addition, asmall number of images duplicated over training and test sets
were excluded. Accordingly, ICDAR2013 dataset is a subset
of ICDAR2011 dataset. The number of images of ICDAR2013
dataset is 462, which is comprised of 229 images for the
training set and 233 images for the test set.

Challenge 2 of the competition received 22 entries from 13
different participants (excluding variants of the same method).
In this edition of the competition, we have included theperformance of an out-of-the-box commercial OCR software
package as a baseline to compare against for text localisation(task 1) and word recognition (task 3) inline with Challenge
1 of the competition. As in Challenge 1, ABBYY OCR SDK(version 10) [13] was used as the baseline. Although the samedataset was used in this edition of the competition as that in2011 [5], some corrections were made in the ground-truth aswell as a few duplicated images were removed. Hence, a directcomparison with the results of previous competition is notstraightforward. However, since the corrections were minor, weA new feature of ICDAR2013 dataset is pixel-level groundtruth. Human operators carefully observed each scene image
and painted character pixels manually. (Precisely speaking,they used a semi-automatic segmentation software and thenmade further manual revision.) Every character is painted by

1487

TABLE V.

TABLE VII.

R ANKING OF SUBMITTED METHODS TO TASK 2.1

Method NameUSTB TexStar
Text Spotter [20], [21], [22]CASIA NLPR [23], [24]Text Detector CASIA [25], [26]I2R NUS FARI2R NUS
TH-TextLoc
Text Detection [15], [16]Baseline
Inkam

Recall (%)
66.45
64.84
68.24
62.85
69.00
66.17
65.19
53.42
34.74
35.27

Precision (%)
88.47
87.51
78.89
84.70
75.08
72.54
69.96
74.15
60.76
31.20

F-score
75.89
74.49
73.18
72.16
71.91
69.21
67.49
62.10
44.21
33.11

A NALYSIS OF TASK 2.2 RESULTS .

I2R NUS FAR
NSTextractor
USTB FuStar
I2R NUSNSTsegmentator
Text Detection
OTCYMIST

Well
Segmented
4028
3719
3992
3540
3990
3640
2452

TABLE VIII.

R ANKING OF SUBMITTED METHODS TO TASK 2.3

MethodMethodcan still safely comment on the overall trends in performance
between the two latest editions of the competition.

PhotoOCR
PicRead [27]NESP [19]
PLT [18]
MAPS [17]Feild’s Method
PIONEER [28], [29]Baseline
TextSpotter [20], [21], [22]1) Task 1 - Text Localization: The final results for the Text
Localization task are shown in Table V. The ranking metricused for the Text Localisation task is the F-score (column
in grey) calculated according to the methodology describedin Section III-B1. In contrast to Challenge 1, most of theparticipating methods rank above the baseline method. One
possible reason for this effect might be that the commercial
OCR software does not expect, and hence is not trained for the
distortions introduced by camera-captured images (backgroundclutter, perspective distortions, . . . ).

MergedBroken

Lost

297
106
279
637
148
314
276

11
10
12
7
25
27
23

1531
2033
1585
1684
1705
1884
3117

Total Edit
Distance
122.7
332.4
360.1
392.1
421.8
422.1
479.8
539.0
606.3

False
Positives
355
345
966
357
2792
1885
4549

Correctly Recognised
Words (%)
82.83
57.99
64.20
62.37
62.74
47.95
53.70
45.30
26.85

correctly recognize over 82% of the words, which is double
than the performance achieved by the winner in 2011 (41%).
Hence, we can safely conclude that the state of research inscene text recognition has now advanced to the level where
it can be used in practical applications – though room of
improvement still remains.

One can see from Table V that there are a number of methods achieving an F-score of more than 70%, whereas only one
method was able to achieve this level in the last contest (refer
to Table I in [5]). The two method that participated both times,
TH-TextLoc and Text Spotter (named as Neumann’s method
previously) have shown significant imporvements to their previous results. This indicates a promising overall advancementin the state-of-the-art. The winning method USTB TexStar
achieved an F-score of 75.9%, implying that the competition
dataset still poses significant challenges to state-of-the-art
methods today.

V.

C HALLENGE 3: R EADING T EXT IN V IDEOS

Challenge 3 focuses on text localization in videos. The
amount of videos available is rapidly increasing. This has been
led on one hand by the pervasive use of camera phones andhand-held digital cameras which allow easy video capture, andon the other hand by video-sharing websites such as YouTube1
and Nico Nico Douga2 . In addition to the videos actively
captured by people, a new kind of video feed, passively
captured by wearable cameras (e.g. Memoto3 and Google
Glass4 ), are used to record and observe the user’s daily life
(e.g. [30]). For both actively and passively captured videos,analyzing occurences of text in videos is highly demanded.

2) Task 2 - Text Segmentation: This was a newly introduced
task for Challenge 2 in this edition of the competition. The final
results for this task are given in Table VI. The ranking metricused is the atom F-Score (column in grey) calculated in a similar fashion as that for Challenge 1 (see Section III-B2). Besides
atom based metrics, Table VI also shows the pixel Precision,
Recall and F-score values. The results show that the winning
method I2R NUS FAR achieved the best performance bothwith respect to the atom-level as well as pixel-level metrics.

The aim of this challenge is to provide the impetus forthe development of methods that take advantage of video
sequences to localize text in the depicted scene. The objectiveof the text localisation task is to obtain the locations of words
in the video sequence in terms of their affine bounding boxes.
The task requires that words are both localised correctly inevery frame and tracked correctly over the video sequence.

Further analysis of the results in Table VII shows that thewinning method I2R NUS FAR not only obtained the highest
number of well segmentated atoms, but also the least numberof lost atoms. This explains the overall best performance of thismethod both in pixel level metrics and in atom level metrics.

Since a video is a collection of static images, one may thinkthat this challenge is not essentially different from Challenge
2. Although an existing method for static images could in
principle be used in this challenge, videos are of different
nature compared to static images. Consecutive frames present
in general a small differences, while images from videos are3) Task 3 - Word Recognition: It is for the first time in
the history of Robust Reading competitions that this challenge
has received more than three entries, clearly indicating renewedinterest on the topic. The results for the Word Recognition task
based on the total edit distance (the ranking metric) and thepercentage of correctly recognized words are shown in Table
VIII. Like in Challenge 1, we can note a huge improvement in
the performance of the state of the art since the 2011 edition.
The winning method PhotoOCR by Google Inc. was able to1 https://www.youtube.com/
2 http://www.nicovideo.jp/
3 http://memoto.com/
4 http://www.google.com/glass/start/

1488

TABLE VI.
MethodI2R NUS FAR
NSTextractor
USTB FuStar
I2R NUSNSTsegmentator
Text Detection
OTCYMISTR ANKING OF SUBMITTED METHODS TO TASK 2.2

Recall (%)
74.73
60.71
69.58
73.57
68.41
64.74
46.11

Pixel Level
Precision (%)
81.70
76.28
74.45
79.04
63.95
76.20
58.53

F-score
78.06
67.61
71.93
76.21
66.10
70.01
51.58

Recall
68.64
63.38
68.03
60.33
68.00
62.03
41.79

TABLE IX.

typically worse than static images due to motion blur andout of focus issues, and video compression introduces further
artifacts.

t
r
a
i
n
g

Applying text detection in a frame-by-frame fashion makes
little sense for video sequences as it ignores any temporalcues. Combining detection with tracking would be a reasonableapproach here for a few reasons. First, most text detection
methods in real scenes are far from working in a real-time
fashion, hence it is necessary to propagate hypotheses until
a new observation is made. Another important reason for a
tracking approach is the presence of occlusions, or equivalently
the variable quality of the video sequence that might makedetection in certain frames impossible. This also enables asophisticated rejection, which can cope with excessive amountsof false positives (see for example the performance of thebaseline method we employ in this competition in section
V-C).

A

s
e
t

B

t
e
s
t
s
e
t

A. Dataset
The challenge is based on various short sequences (around
10 seconds to 1 minute long) selected so that they represent awide range of real-life situations (high-level cognitive tasks),
using different types of cameras. The dataset was collected by
the organisers in different countries, in order to include text invarious languages (Spanish, French and English5 ). The video
sequences correspond to certain tasks that we asked users to
perform, like searching for a shop in the street or finding theirway inside a building. The tasks were selected so that they
represent typical real-life applications, and cover indoors andoutdoors scenarios. We used different cameras for different
sequences, so that we also cover a variety of possible hardware
used; these include mobile phones, hand-held cameras andhead-mounted cameras.

Atom Level
PrecisionF-score
80.59
74.14
83.57
72.09
72.46
70.18
76.62
67.51
54.35
60.41
57.43
59.64
31.60
35.99

S UMMARY OF VIDEOS IN THE DATASETS IN C HALLENGE 3.
Video ID

Task

Camera typeNo. of Frames

Duration

7
8
37
42
51
52
54
13
19
26
36
40
41

(5)
(5)
(2)
(2)
(7)
(7)
(7)
(4)
(5)
(5)
(2)
(2)
(2)

(A)
(A)
(C)
(C)
(D)
(D)
(D)
(A)
(A)
(B)
(C)
(C)
(C)

264
240
456
504
450
450
480
576
408
387
336
408
528

00:11
00:10
00:19
00:21
00:15
00:15
00:16
00:24
00:17
00:16
00:14
00:17
00:22

1
5
6
11
17
20
23
24
32
35
39
44
48
49
53

(1)
(3)
(3)
(4)
(3)
(5)
(5)
(5)
(2)
(2)
(2)
(6)
(6)
(6)
(7)

(B)
(B)
(B)
(A)
(A)
(A)
(B)
(B)
(C)
(C)
(C)
(D)
(D)
(D)
(D)

602
542
162
312
264
912
1020
1050
312
336
438
1980
510
900
450

00:20
00:18
00:05
00:13
00:11
00:38
00:34
00:35
00:13
00:14
00:20
01:06
00:17
00:30
00:15

B. Performance Evaluation
There are numerous evaluation frameworks proposed for
multiple object tracking systems [31], [32], an elaborated
review, including their use for text tracking in videos, can befound in Kasturi et al. [31]. In this competition, we selected touse the CLEAR-MOT [32] and VACE [31] metrics adapted tothe specificities of text detection and tracking, extending the
CLEAR-MOT code6 provided by Bagdanov et al. [33].
The CLEAR-MOT [32] evaluation framework provides two
overall performance measures: the Multiple Object Tracking
Precision (MOTP), which expresses how well locations ofwords are estimated, and the Multiple Object Tracking Accuracy (MOTA), which shows how many mistakes the tracker
system made in terms of false negatives, false positives, andID mismatches. On the other hand, the Average Tracking
Accuracy (ATA) VACE metric [31] provides a spatio-temporalmeasure that penalizes fragmentations while accounting forthe number of words correctly detected and tracked, false
negatives, and false positives.

We provided 28 videos in total; 13 videos comprised thetraining set and 15 the test set. The training set was divided into
two parts (A and B). The only difference between them was
the time of release. The videos were captured with 4 kindsof cameras ((A).Head-mounted camera, (B).Mobile Phone,(C).Hand-held camcorder and (D).HD camera) and categorized
into 7 tasks ((1).Follow wayfinding panels walking outdoors,(2).Search for a shop in a shopping street, (3).Browse products
in a super market, (4).Search for a location in a building,(5).Driving, (6).Highway watch, (7).Traing watch). See thesummary of the videos in Table IX.

Given a video sequence an ideal text tracking method
6 http://www.micc.unifi.it/masi/code/clear-mot/

5 Japanese data are also planned to be released in the near future.

1489

should be able to detect all text words present at every frameand estimate their bounding boxes precisely; additionally it
should also keep consistent track of each word over time, by
assigning a unique ID which stays constant throughout thesequence (even after temporary occlusion, etc).
For every time frame t a text tracking system outputs
a set of hypotheses {ht1 , ..., htn } for a set of words in thetground-truth {w1t , ..., wm}. Those frame level objects can begrouped by their unique identifiers into sequence level hypotheses {H1 , ..., Hp } and ground-truth words {W1 , ..., Wq }(that typically span more than one frame). For a distinctive
notation we refer as Hit and Wit to the frame level objects atframe t in Hi and Wi respectively.
The evaluation procedure is based on a mapping list ofword-hypothesis correspondences. At the frame level we havea mapping Mt for each frame t in the video sequence madeup with the set of pairs (wit , htj ) for which the sum of
overlap(wit , htj ) is maximized, where overlap(·) is a function

Fig. 1. CLEAR-MOT and ATA metrics for (a) the baseline algorithm and(b) TextSpotter [20], [21], [22] method. Diamond markers indicate averagevalues.

a(wt ∩ht )overlap(wit , htj ) = a(wit ∪hjt ) of the intersection area (a(·))
ijof their bounding boxes. Additionally, a pair (wit , htj ) is
considered a valid correspondence iff overlap(wit , htj ) > 0.5.
At the sequence level we have a unique mapping M of wordhypothesis correspondences (Wi , Hj ), but in this case maximizing the spatio-temporal overlap of all possible (Wi , Hj )combinations.

C. Results and Discussion
Figure 1 shows the box-plots of the detection and tracking
scores for comparison of the baseline algorithm and the only
participant in this challenge.
The baseline algorithm consists in a detection stage performed using the ABBYY OCR SDK (with same set-up as
in the other challenges of the competition - see section III),and a tracking stage where each detected word is assigned
the identifier of the previously detected word with the best
overlapping ratio (at least 0.5) searching backwards in a bufferof the 7 prior frames. In order to prevent an excessive numberof false positives the algorithm takes a conservative strategy
and words are not reported in the output unless there is amatching word detected in the immediate previous frame.

The two CLEAR-MOT metrics are calculated using theframe level mappings as:
Pii,t otM OT P = P(1)
t ct
where oit refers to the overlapping ratio of the ith correspondence in the mapping Mt and ct is the number ofcorrespondences in Mt ; and:P
(f nt + f pt + id swt )P
M OT A = 1 − t
(2)
t gt

Figure 1(a) shows the baseline algorithm performance interms of the metrics described in the previous section. The
mean tracking precision (MOTP) was 0.63, and the mean
tracking accuracy (MOTA) was -0.09, much lower due to thehigh number of false positives. Notice that in equation (2)negative values of MOTA are obtained when the evaluatedmethod counts more false positives and/or ID-switches perframe than the actual number of words in the ground-truth. The
ATA score obtained by the baseline algorithm remained close
to zero in all video sequences of the test set, as a consequenceof the high fragmentation in temporal overlapping producedby such a simplistic tracking strategy.

where f nt , f pt , id swt , and gt refer respectively to thenumber of false negatives, false positives, ID switches, andground-truth words at frame t.
The Sequence Track Detection Accuracy (STDA) is calculated by means of the sequence level mapping M as:
NM P
t
t
X
t m(Wi , Hi )(3)
ST DA =
NWi ∪Hi 6=∅
i=1

The TextSpotter [20], [21], [22] method (see Figure 1(b))
significantly outperforms the baseline algorithm in the MOTA
and ATA scores, while achieving a slightly better MOTP. The
mean for MOTP and MOTA metrics where 0.67 and 0.27
respectively, and the mean ATA was 0.12. In this comparison
it is clear that the TextSpotter method is able to detect more
words, while generating less false positives and ID-switches
than the baseline algorithm.

where NM is the number of correspondences in M , NWi ∪Hi 6=∅
is the number of frames where either Wi or Hi exist, andm(Wit , Hit ) takes a value of 1 iff overlap(Wit , Hit ) > 0.5 or
0 otherwise.
The STDA is a measure of the tracking performance over
all of the objects in the sequence and thus can take a maximum
value of NW , which is the number of ground-truth words inthe sequence. The Average Tracking Accuracy (ATA), whichis the normalized STDA per object, is defined as:
ST DA
AT A =  NW +NH 

The obtained results show coherency with the ones reported
in [31] for text detection and tracking in video while being
lower in general, however is worth to notice that the datasetsare very different in nature and also the evaluated tasks differ:
here the evaluation is done at the word level while in [31] it

(4)

2

1490

was done at the text block level.
VI.

[13]
[14]

C ONCLUSIONS

An overview of organization and the results of ICDAR2013 Robust Reading Competition was given in this paper.
Besides the two challenges on extracting text from still images,
a new challenge of text extraction from scene videos wasintroduced. Although limited participation was made in the
newly introduced challenge, we hope this competition will
trigger research on this exciting topic. It is planned to present
the detailed results of the competition in a journal paper
due to space restrictions here. Further results, as well as avisualisation of the results of each method on an image byimage basis will be given in the Web site of the competitionat [6]. Also, continuous submission in all three challenges is
encouraged and open.

[15]

[16]
[17]

[18]

[19]

[20]

ACKNOWLEDGMENT
This work was supported by the Spanish research projects
TIN2011-24631, RYC-2009-05031 and JST CREST project.

[21]

R EFERENCES

[22]

[1]

[2]

[3]

[4]

[5]

[6]
[7]

[8]

[9]

[10]

[11]

[12]

S. M. Lucas, A. Panaretos, L. Sosa, A. Tang, S. Wong, and R. Young,“ICDAR 2003 robust reading competitions,” in Proceedings of the Seventh International Conference on Document Analysis and Recognition,
vol. 2, 2003, pp. 682–687.
S. M. Lucas, A. Panaretos, L. Sosa, A. Tang, S. Wong, R. Young,
K. Ashida, H. Nagai, M. Okamoto, H. Yamamoto et al., “ICDAR
2003 robust reading competitions: entries, results, and future directions,”
International Journal of Document Analysis and Recognition (IJDAR),
vol. 7, no. 2-3, pp. 105–122, 2005.
S. M. Lucas, “ICDAR 2005 text locating competition results,” in Document Analysis and Recognition, 2005. Proceedings. Eighth International
Conference on. IEEE, 2005, pp. 80–84.
D. Karatzas, S. R. Mestre, J. Mas, F. Nourbakhsh, and P. P. Roy,“ICDAR 2011 robust reading competition-challenge 1: reading text
in born-digital images (web and email),” in Document Analysis andRecognition (ICDAR), 2011 International Conference on. IEEE, 2011,
pp. 1485–1490.
A. Shahab, F. Shafait, and A. Dengel, “ICDAR 2011 robust reading
competition challenge 2: Reading text in scene images,” in Document
Analysis and Recognition (ICDAR), 2011 International Conference on.
IEEE, 2011, pp. 1491–1496.
ICDAR 2013 Robust Reading Competition. [Online]. Available:
http://dag.cvc.uab.es/icdar2013competition/
D. Kumar and A. Ramakrishnan, “OTCYMIST: Otsu-Canny minimal
spanning tree for born-digital images,” in Proc. Int. Workshop on
Document Analysis Systems, 2012.
B. Su, S. Lu, T. Q. Phan, and C. L. Tan, “Character extraction in web
image for text recognition,” in Pattern Recognition (ICPR), 2012 21stInternational Conference on. IEEE, 2012, pp. 3042–3045.
H. Yang, B. Quehl, and H. Sack, “A framework for improved video
text detection and recognition,” Multimedia Tools and Applications, pp.
1–29, 2012.
A. González, L. M. Bergasa, J. J. Yebes, and S. Bronte, “Text locationin complex images,” in Pattern Recognition (ICPR), 2012 21st International Conference on. IEEE, 2012, pp. 617–620.
A. Clavelli, D. Karatzas, and J. Lladós, “A framework for the assessmentof text extraction algorithms on complex colour images,” in Proceedingsof the 9th IAPR International Workshop on Document Analysis Systems.
ACM, 2010, pp. 19–26.
C. Wolf and J.-M. Jolion, “Object count/area graphs for the evaluationof object detection and segmentation algorithms,” International Journalof Document Analysis and Recognition (IJDAR), vol. 8, no. 4, pp. 280–
296, 2006.

[23]

[24]

[25]

[26]

[27]

[28]

[29]

[30]

[31]

[32]

[33]

1491

Abbyy FineReader. [Online]. Available: finereader.abbyy.com/
ICDAR 2011 Robust Reading Competition - Challenge 1: ”Reading
Text in Born-Digital Images (Web and Email)”. [Online]. Available:
http://www.cvc.uab.es/icdar2011competition/
J. Fabrizio, B. Marcotegui, and M. Cord, “Text segmentation in naturalscenes using toggle-mapping,” in Proc. Int. Conf. on Image Processing,2009.
——, “Text detection in street level images,” Pattern Analysis andApplications, 2013, (Accepted for publication).
D. Kumar, M. Anil, and A. Ramakrishnan, “MAPS: midline analysis
and propagation of segmentation,” in Proc. Indian Conf. on Vision,
Graphics and Image Processing, 2012.
D. Kumar and A. Ramakrishnan, “Power-law transformation for enhanced recognition of born-digital word images,” in Proc. 9th Int. Conf.
on SPCOM, 2012.
D. Kumar, M. Anil, and A. Ramakrishnan, “NESP: nonlinear enhancement and selection of plane for optimal segmentation and recognition ofscene word images,” in Proc. SPIE Document Recognition and Retrieval
XX, 2013.
L. Neumann and J. Matas, “A method for text localization and recognition in real-world images,” in Proc. Asian Conf. on Computer Vision,
2010, pp. 2067–2078.
——, “Real-time scene text localization and recognition,” in Proc. Int.
Conf. on Computer Vision and Pattern Recognition, 2012, pp. 3538–
3545.
——, “On combining multiple segmentations in scene text recognition,”in Proc. Int. Conf. on Document Analysis and Recognition, 2013.
Y.-M. Zhang, K.-Z. Huang, and C.-L. Liu, “Fast and robust graph-basedtransductive learning via minimum tree cut,” in Proc. Int. Conf. on Data
Mining, 2011.
C.-L. L. B. Bai, F. Yin, “Scene text localization using gradient local
correlation,” in Proc. Int. Conf. on Document Analysis and Recognition,
2013.
C. Shi, C. Wang, B. Xiao, Y. Zhang, S. Gao, and Z. Zhang, “Scene
text recognition using part-based tree-structured character detections,”in Proc. Int. Conf. on Computer Vision and Pattern Recognition, 2013.
C. Shi, C. Wang, B. Xiao, Y. Zhang, and S. Gao, “Scene text detection
using graph model built upon maximally stable extremal regions,”
Pattern Recognition Letters, vol. 34, no. 2, pp. 107–116, 2013.
N. Tatiana, O. Barinova, P. Kohli, and V. Lempitsky, “Large-lexicon
attribute-consistent text recognition in natural images,” in Proc. European Conf. on Computer Vision, 2012, pp. 752–765.
J. J. Weinman, E. Learned-Miller, and A. Hanson, “A discriminative
semi-Markov model for robust scene text recognition,” in Proc. Intl.
Conf. on Pattern Recognition, 2008.
J. Weinman, Z. Butler, D. Knoll, and J. Feild, “Toward integrated scene
text reading,” IEEE Trans. on Pattern Anal. Mach. Intell., 2013, to be
published.
J. Gemmell, L. Williams, K. Wood, R. Lueder, and G. Bell, “Passive
capture and ensuing issues for a personal lifetime store,” in Proc.
1st ACM workshop on Continuous archival and retrieval of personal
experiences (CARPE’04), 2004, pp. 48–55.
R. Kasturi, D. Goldgof, P. Soundararajan, V. Manohar, J. Garofolo,
R. Bowers, M. Boonstra, V. Korzhova, and J. Zhang, “Framework for
performance evaluation of face, text, and vehicle detection and tracking
in video: Data, metrics, and protocol,” IEEE Transactions on PatternAnalysis and Machine Intelligence, vol. 31, no. 2, pp. 319–336, 2009.
K. Bernardin and R. Stiefelhagen, “Evaluating multiple object tracking
performance: The CLEAR MOT metrics,” EURASIP Journal on Imageand Video Processing, vol. 2008, May 2008.
A. D. Bagdanov, A. Del Bimbo, F. Dini, G. Lisanti, and I. Masi,“Compact and efficient posterity logging of face imagery for video
surveillance,” IEEE Multimedia, vol. 19, no. 4, pp. 48–59, 2012.

234 

8241124',-./0%+)/12+30-4+5/0-42+ -=!1!,!+%>+(,!%?0!1%-2+
!"#!$!%&" '/,-/-4+60-42+701/,0+ *'/0#?@+%>+'()*(+
810%2+)%-4+90%2+:1-;%-4+ )$11--$==+($
2+71A1-42+'/1-0+<,+

B1&=!2+C)DE=+0&+#!$!#+0=+!/+F0=1$+$%--$!#+$%?G%--!=H+
/-2+0+&41%-+F0=#+$;0==1>1&+1=+,=#+!%+$;,#+=%?+-%-I!!+C)DE=+0-#+!/+;>!+%-=+0&+4&%,G#+1-!%+$0-#1#0!+!!+&41%-=+0$$%&#1-4+!%+!/+G%=1!1%-2+=1.+0-#+$%;%&+%>+0$/+C)DEH+
B1-0;;@2+!&I=!&,$!,&#+$/0&0$!&+?%#;=+J)CK+0&+0GG;1#+%-+!/+!!+&41%-=+!%+=0&$/+!/+?1==1-4+$/0&0$!&=+0-#+;1?1-0!+!/+>0;=+G%=1!1L=H+<1&0&$/1$0;+=!&,$!,&+%>+C)DE=+1=+,=#+!%+!&0$!+$/0&0$!&+$0-#1#0!=H+'/0&0$!&+$0-#1#0!=+0&+?&4#+!%+>%&?+!!+P+N-1
N)7")!0&+
L&=1!@+%>+)$1-$+0-#+ &41%-=+,=1-4+0+=1-4;I;1-R+$;,=!&1-4+0;4%&1!/?H+B1-0;;@2+-%-I!!+$/-%;%4@2+71A1-42+'/1-0+ 0-#+!!+$;0==1>1&=+0&+=S,-!10;;@+0GG;1#+!%+4!+>1-0;+!!+8,O0-4+31-P2+8,I'/-4+ Q
+*-=!1!,!+%>+(,!%?0!1%-2++ &41%-=H+
31-P+0-#+<%-4I61+<0%Q+ '/1-=+($0#?@+%>+
!+&41%-=+%F!01-#+,=1-4+N)7" )!0&+0&+!/&=/%;##+)$1-$=2+71A1-42+'/1-0+ F0=#+%-+C)DE=H+T%=!IG&%$==1-4+1=+G&>%&?#+!%+&$%L&+N)7"B,)!0&+=?0;;+!!+$%?G%--!=+!/0!+?14/!+F+?1==#+F@+!/+G&L1%,=+=!04H+(+&4&==1%-+F0=#+!!+=4?-!0!1%-+!$/-1S,+1=+,=#+!%+
1L&=1!@+%>+
+B1;#+0-#+D&1R+ N=4?-!+!!+>&%?+F0$R4&%,-#H+6%&#+&$%4-1!1%-+1=+G&>%&?#+B1;#U=+C!/%#+ :V0$0S&-,#;1-IC1
C0
=
=
0
$
/
,
=

!
!
=
2
+
(?/

&
=
!
2
+
;;&+
,=1-4+0+'%-#1!1%-0;+E0-#%?+B1;#+>%;;%O#+F@+0+;0-4,04+?%#;+N)(+!&01-#+>&%?+%L&+PW+?1;;1%-+O%&#=H+';0==1$0;+%L&I=4?-!0!1%-+0-#+F0?+=0&$/+0&$/1!$!,&+1=+(;==0-#&%+71==0$$%2+C0&R+,=#H+(+#G+-,&0;+-!O%&R+=&L=+0=+!/+$/0&0$!&+$;0==1>1&2+
T/%!%X'E+ ',??1-=2+3,L0;+Y!.&2+ 9%%4;+*-$H2+N)(+
O/&0=+$/0&0$!&I-4&0?=+0&+,=#+>%&+;0-4,04+?%#;;1-4H+<0&!?,!+YL-+
%G+/@G%!/=1=+>&%?+!/+F0?+=0&$/+0&+&I&0-R#+,=1-4+0+
O%&#I-4&0?+?%#;H++;%&+4&0#1-!=+0&+,=#+!%+F1-0&1.+!/+1?04H+!+&41%-=+0&+
30-4+5/0-42',-./0%+)/12+ *-=!1!,!+%>+(,!%?0!1%-2+ '%!&0$!#+,=1-4+=L&0;+/,&1=!1$+&,;=+1-$%&G%&0!1-4+?%&G/%;%4@2+-4+90%2+:1-;%-4+<,2+ '/1-=+($0#?@+%>+
7ZZ"'()*(+ )%#+>1;!&1-42+0-#+=1.2+0;14-?-!2+0-#+$%;%&+'/,-/-4+60-42+701/,0+ )$1-$=2+71A1-42+'/1-0+ $%-==11=!@!IF-0$=@H+#B1-0;;@+)[C=+0&+,=#+!%+>,&!/&+&>1-+!/+&=,;!=+
810%+
F
@
+
&

?%
L
1
4
+
>
0
;
=+0;0&?=H+
0!10-0+Y%
L1R%L0P2+X;40+
6%&#+&$%4-1!1%-+1=+G&>%&?#+1-+0+,-1>1#+G&%F0F1;1=!1$+
P>&0?O%&R+,=1-4+?01?,?+0+G%=!&1%&1+JC(TK+1->&-$+F0=#+
70&1-%LP02+)&4@+C1;@0L2+[;P0#1?1&+%-+O14/!#+>1-1!+=!0!+!&0-=#,$&=H+/+?%#;+->%&$=+F%!/+
T1$E0#+P+\1&1$/-RP%L2+(;0-#&+ V%?%-%=%L+C%=$%O+ !/+;0-4,04+$%-=1=!-$@2+0-#+!/+$%-=1=!-$@+%>+!/+0!!&1F,!=+)0G0Q!%L2+T,=/?!+ W )!0!+N-1L&=1!@2+C%=$%O2+ %>+;!!&=+!/0!+$%-=!1!,!+0+O%&#H+\%/;12+[1$!%&+V?G1!=R@+ QE,==10+++C1$&%=%>!+E=0&$/2+ *?04+F1-0&1.0!1%-+1-$%&G%&0!=+;%$0;+$,=2+=,$/+0=+;%$0;+'0?F&1#42+N\+
F1-0&1.0!1%-+?0G+0-#+1?04+V0G;0$10-2+1-!%+0+4;%F0;+%G!1?1.0!1%-+W+PY)=4?-!%&+ )&4@+C1
)R%;R%L%+*-=!1!,!+%>+ >&0?O%&RH+!;1-I$0-#1#0!=+0&+4-&0!#+F0=#+%-+;@0L2+X;40+P)$1-$+0-#+$/-%;%4@2+ =1?1;0&1!@+%>+!/+=1.+0-#+$%;%&+%>+>1;!&#+$%--$!#+$%?G%--!=H+
70&1-%L02+0!10-0+
!I;1-+$0-#1#0!=+O1!/+;%O&+$;0==1>1&+=$%&=+0&+>1;!&#+%,!H+
Y%L1RQ%L0P2+T,=/?!+W C%=$%O2+E,==10+\%/;12+[1$!%&+V?G1!=R@+
/1=+?!/%#+1=+=1?1;0&+!%+Y)=4?-!0!%&+$G!+!/0!+>1;!&1-4+%>+Y)!&0$!%&+!!I;1-+$0-#1#0!=+0;=%+!0R1-4+1-!%+0$$%,-!+!/+-,?F&+%>+
F0$R4&%,-#+=4?-!=H+
N=1-4+4&0#1-!+;%$0;+$%&&;0!1%-2+!/+#-=1!@+%>+G01&O1=+#4=+%>+%GG%=1!+#1&$!1%-=2+0-#+!/+=!&%R+O1#!/+$%-=1=!-$@+0&+F%!/+*-=!1!,!+%>+(,!%?0!1%-2+ $/0&0$!&1.#+!%+$%?G,!+0+!!+$%->1#-$+?0GH+!+'()*("YVTE+ 7%+701+'/1-=+($0#?@+%>+$0-#1#0!=+0&+%F!01-#+,=1-4+0+>0=!+=?1I=,G&L1=#+;0&-1-4+)$1-$=2+71A1-42+'/1-0+ ?!/%#+]^_+0-#+0&+>,&!/&+G&,-#+,=1-4+0-+)[C+$;0==1>1&H+
B0!,&=+=,$/+0=+$%;%&+0-#+=!&%R+O1#!/+0&+,=#+!%+%F!01-+!!I;1-=+0-#+=,F=S,-!;@+O%&#=H+012345
7891

1492

M
M
M
M

M
M
M
M

M

M

M

M

M

M

M

M M

M

M

M

M

M M

M M

M

M

M

M

M

M

M

M

M

M

M

M

M M

M

M

M

M

M M

M M

M

M M

M M

M

M M

M

M

M

M

M

82824
82824
3811
3811
888888

A PPENDIX : D ESCRIPTION OF THE PARTICIPATING M ETHODS234 

8241124 !"#$%&'()!*+',-*+.'()-*+/%*+' /0*+)1-'5*06!7/0#8.'$01.'20-%30*+'40*+'
9!0:0*+.'()0*-')0/'0/'-*'0;<7%6!='6!7/0%*'%>'#)!';!#)%='#)-#'#%%?'<-7#'0*'@(4AB'CDEE'B%F1/#'B!-=0*+'&%;<!#0#0%*G')!'?!8'!*)-*&!;!*#'0/'#)!'1/!'%>'(%*=0#0%*-H'B-*=%;'I0!H='>%7';%7!'<7!&0/!'#!"#'&-*=0=-#!'/!H!&#0%*G'@*=060=1-H'&%H%7'&)-**!H/'-7!'F0*-70P!='/!<-7-#!H8'1/0*+'K#/1Q/'#)7!/)%H=G'L0*0;1;'/<-**0*+'#7!!'F-/!='+7%1<0*+'0/'<!7>%7;!='K(,L@M'%*'=!#!&#!='&%;<%*!*#/'#%'>0H#!7'F-&?+7%1*='&%;<%*!*#/'-*='+7%1<'#!"#'&-*=0=-#!/'0*#%'R%7=/G')!'+7-8'/&-H!'0/'!*)-*&!='F8'-<<H80*+'<%R!7H-R'#7-*/>%7;G'T*)-*&!='+7-8'/&-H!'0;-+!'0/'/!+;!*#!='1/0*+'K#/1Q/'
S$'
4G'N1;-7'-*='AG'OG'@*=0-*'@*/#0#1#!'%>'M&0!*&!' #)7!/)%H=GK;*0<-+!'K(B'0/'1/!='>%7'R%7='7!&%+*0#0%*'>7%;'#)!'
F0*-70P!='0;-+!G'
B-;-?70/)*-*'
9-*+-H%7!.'@*=0-'I0/&)!7'=0/&70;0*-#0%*'>-&#%7'0/'&-H&1H-#!='>%7'6-70%1/'&%H%7'
UTMS'<H-*!/'R0#)'=0>>!7!*#'<%R!7H-R'6-H1!/G'')!'<H-*!'R0#)';-"0;1;'=0/&70;0*-#0%*'6-H1!'0/'/!H!&#!='>%7'F0*-70P-#0%*G'
A'H%&-H'R0*=%R'F-/!=';0*;-"'#)7!/)%H=0*+'&70#!70-'
LASM'
0*&%7<%7-#0*+';!-*/'-*='6-70-*&!/'%>'#)1/'+!*!7-#!='>%7!+7%1*=VF-&?+7%1*='7!+0%*/'-7!'1/!=G'E'TS@0/';!#)%=')-/'#R%';-0*'/#!</]'-')8<%#)!/0/'+!*!7-#0%*'/#!<'
A'B!/!-7&)'-*=' )E"!/'-*='-')8<%#)!/0/'6-H0=-#0%*'/#!<'#%'
B-<)-WH'9%0//!H.'A*E-' 4!6!H%<;!*#'$-F%7-#%78' #>%0H#'!+7!'>#-'<H/%!#!'=*!#0#-!H&'##!0%"*#'/FG%')!')8<%#)!/0/'+!*!7-#0%*'/#!<'<7%&!//'!>-*0-'(-H-7-/-*E1.' CZ$'B4TTS@A[.'I7-*&!' 7!H0!/'%*'#)!'LLM'/!+;!
!"#'4!#!&#0%*' XM#';!#)%=.'R)0H!'#)!'6-H0=-#0%*'
5*06!7/0#Y'S0!77!'!#'L-70!' /#!<'0/'F-/!='%*'-'#!"#17!F*-#-/#!0%=*'M^L'%*-#)-*'I-F70P0%.'C&H-//0>0&-#0%*'-<<7%-&)G'
MY6!70*!'41F10//%*'(170!'Z$@S\5SL([.'
I7-*&!'
!"#'M<%##!7'0/'-*'1*&%*/#7-0*!='7!-H#0;!'!*=#%!*='#!"#'
H%&-H0P-#0%*'-*='7!&%+*0#0%*';!#)%=G')!'7!-H#0;!'<!7>%7;-*&!'
0/'-&)0!6!='F8'<%/0*+'#)!'&)-7-&#!7'=!#!&#0%*'<7%FH!;'-/'-*'
1?-/'U!1;-**.'X070'(P!&)'!&)*0&-H'!>>0&0!*#'/!31!*#0-H'/!H!&#0%*'>7%;'#)!'/!#'%>'T"#7!;-H'B!+0%*/'
!"#'M<%##!7' $L#-/.'L0&)-H'91/#-'
5*06!7/0#8.'(P!&)'B!<1FH0&' ZTB/[G'TB/'-7!'+7%1<!='0*#%'R%7='7!+0%*/'R)0&)'-7!'7!&%+*0P!='
1/0*+'-*'-<<7%"0;-#!'*!-7!/#*!0+)F%7'&H-//0>0!7'%<!7-#0*+'%*'-'&%-7/!'O-1//0-*'/&-H!/<-&!'<87-;0=G'A'=!;%'%>'#)!'/%>#R-7!'
0/'-6-0H-FH!'%*H0*!])##<]VVRRRG#!"#/<%##!7G%7+V'T-&)'R%7='0;-+!'0/'%6!7/!+;!*#!='-*='H%+0/#0&'7!+7!//0%*'0/'
1/!='#%'&H-//0>8'#!"#'&%;<%*!*#/G'S%H8*%;0-H/'>0#'#%'#)!'#%</'-*='
F%##%;/'%>'F0*-70P!='&)-7-&#!7/'-7!'*%7;-H0P!='#%'-')%70P%*#-H.'
O7
0
*
*
!
H
H
'
(%
H
H
!
+
!
.
'
@
%
R.
'
S@KUTTB' X!7%='_!0*;-*'
H0*!-7'%70!*#-#0%*'R0#)'-'#)0*<H-#!'/<H0*!G'A'=0/&70;0*-#06!'/!;05MA'
L-7?%6';%=!H':%0*#H8'/!+;!*#/'-*='7!&%+*0P!/'&)-7-&#!7/'0*'#)!'*%7;-H0P!='0;-+!'1/0*+'/#!!7-FH!'<87-;0='>!-#17!/.'&)-7-&#!7'
F0+7-;/.'-*='-'H-7+!'H!"0&%*G')0/';!#)%='F10H=/'1<%*'@CB`U5M'F8'>17#)!7'7!=1&0*+'>-H/!'@CB`U5M`IAB'-H-7;/'#)7%1+)'-';-&)0*!'H!-7*0*+'-<<7%-&)G'E'@*/#0#1#!'>%7'@*>%&%;;' )0/';!#)%=';-?!/'1/!'%>'<!7&!<#1-H'/-H0!*&8'%>'#!"#/'0*'
B!/!-7&).'AaMAB.'/&!*!/VF%7*'=0+0#-H'0;-+!/G'@*'<-7#0&1H-7.'>%17'#!"#/<!&0>0&'$1'M)0:0-*E.'C''0-*'*+-<%7!'/-H0!*&8'>!-#17!/'-7!'=!/0+*!='#)-#'-7!'&%*/0/#!*#H8'
M)-*+E"1-*.'$0;'X%%'C CM0'M&>'(%;<1#0*+.' -&&%;<-*0!='R0#)'#!"#/'0*'/&!*!VR!F'0;-+!/G'(%;F0*!='R0#)'
R!!.''-*'()!R'$0; '' U-#0)%%*%-HH''%5*@CB`U5M'
06!7/0#8'%>' #)!'#!"#'H-8%1#'0*>%7;-#0%*.'#)!'/-H0!*&8'>!-#17!/'-7!'0*#!+7-#!='
M0*+-<%7!'#%'&H-//0>8'#!"#'-*='*%*#!"#'%F:!&#/'-&&17-#!H8G'A>#!7'#!"#'
H%&-H0P-#0%*.'#!"#'F%1*=-78'0/'!"#7-&#!='#%'+!#'#)!'F0*-78'#!"#'
0;-+!/G'@*?-;'
U%*!'/<!&0>0!='U%*!'/<!&0>0!='U%*!'/<!&0>0!=b'$0*?'#%'/%>#R-7!]')##<]VV0*?-;%&7G&%;V!*+V'
9-/!H0*!'
K7+-*0P!7/'
UVA'
A99,,'K(B'M4N'6ED''

012345
7891

J

J

1493

J

J

J J
J J

J

J

J

J

J

J

J

J

J

J J

J

J J

J

J

J J

J J

J

J

J

J

J

J

J

J

J

J

J J
J J

J

J

J J

J

J

J

J

J J

J

J J

J J

J J

J

J

J

J

J

J

J

82824
82824
3811
3811
888888

A PPENDIX C ONTD .: D ESCRIPTION OF THE PARTICIPATING M ETHODS