International Conference on Artificial Reality and TelexistenceEurographics Symposium on Virtual Environments (2017)
R. Lindeman, G. Bruder, and D. Iwai (Editors)

Reference Framework on vSRT-method Benchmarking for MAR
R. Ichikari1 , T. Kurata1 , K. Makita2 , T. Taketomi3 , H. Uchiyama4 , T. Kondo5 , S. Mori6 and F. Shibata7
1 National Institute of Advanced Industrial Science and Technology (AIST), Japan
2 CANON INC., Japan
3 Nara Institute of Science and Technology, Japan
4 Kyushu University, Japan
5 The Open University of Japan, Japan
6 Keio University, Japan
7 Ritsumeikan University, Japan

Abstract
This paper presents a reference framework on benchmarking of vision-based spatial registration and tracking (vSRT) methods
for Mixed and Augmented Reality (MAR). This framework can provide typical benchmarking processes, benchmark indicators,
and trial set elements that are necessary to successfully identify, define, design, select, and apply benchmarking of vSRT methods
for MAR. In addition, we summarize findings from benchmarking activities for sharing how to organize and conduct on-site andoff-site competition.
CCS Concepts•Human-centered computing → Mixed / augmented reality;

1. Introduction
In the development of mixed and augmented reality (MAR) applications, spatial registration and tracking methods, especially visionbased methods, represent some of the most important technologies. In this context, research and development of vision-based spatial registration and tracking (vSRT) is flourishing and many algorithms have been proposed every year. Therefore, standardized
frameworks on benchmarking of vSRT methods have become necessary in order to foster objective evaluation and comparison of diverse vSRT methods. Such frameworks would also facilitate muchfairer competition among small and major companies and institutes
involved in MAR technologies, applications, and services.
Based on this background, the TrakMark working group (WG)
was established in 2009 as a subsidiary organization of the Special
Interest Group on Mixed Reality (SIG-MR) in the Virtual Reality
Society of Japan (VRSJ) to standardize benchmark schemes. Table
1 shows a history of TrakMark WG activities [TK*09] [SIKU*10][TMK11] [MOK14] [TMK12] [TMK13] [TMP14] [TMC15][TMW]. In accordance with such grass-roots activities including
on- and off-site comparisons of vSRT methods and MAR systems[LBM*09] [KMP*13] [WGB13], which are often held as contests[PITC], three core components as shown in Fig. 1 are identified anddefined for creating a reference framework.
• Benchmarking Processes, which include how to produce
benchmarking outcomes using benchmark indicators and trial
sets, as well as how to share benchmarking outcomes.
c 2017 The Author(s)Eurographics Proceedings c 2017 The Eurographics Association.

• Benchmark Indicators, which quantify the performance ofvSRT methods in MAR by considering not only the characteristics of vSRT methods in MAR such as reliability and temporalcharacteristics, but also fair comparison.
• Trial Set Elements, which consist of datasets and physical object instances for providing each benchmarking attempt with thesame condition.
On-site benchmarking methods are used to conduct benchmarking on the spot while capturing images of physical objects withworking MAR systems. Because human factors such as a contestant’s limited preparation time inevitably affect on-site benchmarking result, stakeholders often lose focus in terms of what they aresupposed to evaluate. In addition, time and cost constraints exist forcompetition organizers as benchmarking service providers. Therefore, simplifying the benchmarking framework is often necessary
for practical operation of on-site tracking competitions. However,the pros and cons shall be considered with the findings given in
Section 5.
By contrast, off-site benchmarking methods are used to conduct benchmarking with target images in datasets prepared in advance. Compared to on-site benchmarking methods, both contestants and competition organizers have more time to prepare andconduct benchmarking. However, organizers must make additional
effort to alleviate issues related to fine-tuning the benchmarking
process/methods.
Typical processes of on- and off-site benchmarking are extracted

R. Ichikari, T. Kurata, K. Makita, T. Taketomi, H. Uchiyama, T. Kondo, S. Mori & F. Shibata / Reference Framework on vSRT-method Benchmarking for MAR

from grass-roots activities, and they are schematically described in
Section 2 by referring to ISO/IEC 29155 series, especially ISO/IEC
29155-1 [I291].
Each benchmark indicator and trial set element is also extracted
from outcomes and discussions in the grass-roots activities. Section
3 describes three major types of benchmark indicators: reliability,
temporality, and variety. Section 4 describes reference elements ina trial set, which contains dataset elements and physical object instances.
2QVLWH%HQFKPDUNLQJ%HQFKPDUNLQJ%HQFKPDUN
3URFHVVHV 䠇 ,QGLFDWRUV 䠇

7ULDO6HW'DWDVHW

2IIVLWH%HQFKPDUNLQJ

Figure 1: Core components of on-site and off-site benchmarking
framework.

• Benchmark provider, who surveys benchmarking results, creates and gathers datasets, maintains benchmarking repositories,
and provides benchmark surveys;• Benchmarking service provider, who develops and provides
benchmarking instruments, conducts benchmarking at the request of technology users, and submits benchmarking results toa benchmarking repository;• Technology developer, who develops vSRT methods, MAR systems, or MAR services;• Technology supplier, who supplies vSRT methods, MAR systems, or MAR services that technology developers have developed;• Technology user, who utilizes the outcomes of benchmarking.
Of course, various role-sharing schemes can be used in practice. Any person or organization may fulfill one or more roles. For
example, benchmark providers can also be benchmarking service
providers. By contrast, one role may be fulfilled by several persons or organizations. For example, the competition organizers together with contestants often fulfill the role of benchmarking service provider in conducting benchmarking.
([WHUQDO%HQFKPDUNLQJ 5HSRVLWRU\([WUDFWGDWDVHWV

2. Benchmarking processes&UHDWHGDWDVHWVThis section outlines benchmarking processes and related components necessary to produce and share benchmarking outcomes.
Fig. 2 illustrates the basic benchmarking process flow. Although
the flow may differ for each specific benchmarking process, it generally consists of process, target, input, output, outcome, and organized storage, described in detail as follows:

0DLQWDLQUHSRVLWRU\%HQFKPDUNLQJ
5HSRVLWRU\

6XUYH\EHQFKPDUNLQJUHVXOWV6XEPLW
EHQFKPDUNLQJUHVXOWV

3K\VLFDOREMHFWV([WUDFWGDWDVHWV

7ULDOVHW,QSXW7DUJHW'DWDVHWV

2XWSXW2XWFRPH%HQFKPDUNV,QWHUPHGLDWHUHVXOWV%HQFKPDUNLQJVXUYH\%HQFKPDUNLQJLQVWUXPHQWV&RQGXFW
EHQFKPDUNLQJ%HQFKPDUNLQGLFDWRUV
7RROV0HWKRGV*XLGHV

0$5V\VWHP'HYHORSY657PHWKRGV

3URFHVV

3K\VLFDOREMHFWLQVWDQFHV

5HSRUWV

Y657PHWKRG'HYHORS0$5V\VWHPV

3URYLGHEHQFKPDUNLQJLQVWUXPHQWV*DWKHUY657PHWKRGV
9HULI\EHQFKPDUNLQJTXDOLW\*DWKHU0$5V\VWHPVFigure 2: Basic benchmarking process flow.

• Target, which is a vSRT method or MAR system used as abenchmarking target.
• Input, which consists of MAR systems that use vSRT methods,
trial sets, and physical objects.
• Output/Outcome, which includes:◦ benchmarking instruments such as benchmark indicators andtools, methods, or guides,◦ benchmarking results such as benchmarks, intermediate results, and reports,◦ and benchmarking surveys.

2UJDQL]HGVWRUDJH%HQFKPDUNLQJUHVXOWV• Process, which consists of one or more micro processes:◦ to develop and gather vSRT methods and MAR systems,◦ to prepare and conduct benchmarking,◦ to provide and maintain benchmarking instruments andrepositories,◦ and to share benchmarking results.

3URFHVV3URYLVLRQIORZ
3URYLGHWDUJHW
REMHFWLQVWDQFHV%HQFKPDUNLQJVHUYLFH SURYLGHU%HQFKPDUNSURYLGHU• Organized storage, which may be a benchmarking repository
or other external repositories.
Note that trial sets are classified in the Input process as previously described. However, they are also regarded as important
outcomes of benchmarking activities. Various stakeholders are involved in benchmarking vSRT methods for MAR. Fig. 3 illustrates
a typical example of the correspondence between stakeholders andtheir roles and activities. Based on roles, stakeholders can logicallybe classified into the following groups:

7HFKQRORJ\XVHU

7HFKQRORJ\GHYHORSHU

7HFKQRORJ\VXSSOLHU

4XDOLW\9HULILHU

Figure 3: Example of stakeholders and their roles.

c 2017 The Author(s)Eurographics Proceedings c 2017 The Eurographics Association.

R. Ichikari, T. Kurata, K. Makita, T. Taketomi, H. Uchiyama, T. Kondo, S. Mori & F. Shibata / Reference Framework on vSRT-method Benchmarking for MAR

Table 1: History of TrakMark WG activities: Breakthroughs and technical findings.
zĞĂƌ

ǀĞŶƚ

dƌĂŬDĂƌŬt'ĂĐƚŝǀŝƚŝĞƐ

䕔ƌĞĂŬƚŚƌŽƵŐŚ
䖃dĞĐŚŶŝĐĂůĨŝŶĚŝŶŐƐ

ϮϬϬϵ

/^DZϮϬϬϵ

dƌĂŬDĂƌŬ WƌŽƉŽƐĂůǁĂƐĚŝƐĐůŽƐĞĚ͘䕔dŚĞƉƌŽĐĞƐƐŽĨĞǀĂůƵĂƚŝŽŶĂƚƚŚĞϭƐƚƉŚĂƐĞ ǁĂƐƉƌŽƉŽƐĞĚ͘䖃ĂƚĞŐŽƌŝǌĂƚŝŽŶŽĨƚƌĂĐŬŝŶŐŵĞƚŚŽĚƐŝƐŶĞĞĚĞĚ͘䖃ƌŝƚĞƌŝĂ ĨŽƌĞǀĂůƵĂƚŝŽŶŝƐŶĞĞĚĞĚ͘䖃ĂƚĂƐĞƚŝƐŶĞĞĚĞĚ͘ϮϬϭϬ

/^DZϮϬϭϬ

dŚĞĨŝƌƐƚ/ŶƚĞƌŶĂƚŝŽŶĂůdƌĂŬDĂƌŬǁŽƌŬƐŚŽƉǁĂƐĐŽŶĚƵĐƚĞĚ͘䕔&ŝƌƐƚǀĞƌƐŝŽŶŽĨĞǀĂůƵĂƚŝŽŶĐƌŝƚĞƌŝĂŝŶdƌĂŬDĂƌŬǁĂƐƉƌŽƉŽƐĞĚ͘䖃ĂƚĂƐĞƚĂŶĚĞǀĂůƵĂƚŝŽŶŵĞƚŚŽĚƐĚĞƐŝŐŶĞĚďǇDĞƚĂŝŽ ǁĂƐƉƌŽƉŽƐĞĚ ;ĐůŽƐĞĚŐƌŽƵŶĚƚƌƵƚŚ͕ ŚŽǁƚŽƐĞƚƚŚƌĞƐŚŽůĚǀĂůƵĞƐͿ䖃,/d >Ăď͛ƐWŽƚĞŶƚŝĂů ;ĐĂƚĞŐŽƌŝǌĂƚŝŽŶŽĨĚĂƚĂƐĞƚ͕ ƐŝŵƵůĂƚŝŽŶƉůĂƚĨŽƌŵ͕ŚŽǁƚŽĐŽŶĚƵĐƚĂƚĞƐƚͿϮϬϭϭ

/^DZϮϬϭϭ

dŚĞϮŶĚ/ŶƚĞƌŶĂƚŝŽŶĂůtŽƌŬƐŚŽƉǁĂƐĐŽŶĚƵĐƚĞĚ͘䕔ĞŶĐŚŵĂƌŬŝŶŐƌĞƐƵůƚƐǁŝƚŚdƌĂŬDĂƌŬ ĚĂƚĂƐĞƚǁĞƌĞƉƌĞƐĞŶƚĞĚ;ďǇ,ĂǇĂƐŚŝ Ğƚ Ăů͕ ĂŶĚŶƚŽŝŶĞWĞƚŝƚĞƚĂůͿ䖃ĞƉƚŚĚĂƚĂ ŝƐŶĞĞĚĞĚĨŽƌďĞŶĐŚŵĂƌŬŝŶŐƚƌĂĐŬŝŶŐŵĞƚŚŽĚƐƵƐŝŶŐĚĞƉƚŚƐĞŶƐŽƌ͘

ϮϬϭϮ

/WZϮϬϭϮ

dŚĞϯƌĚ/ŶƚĞƌŶĂƚŝŽŶĂůtŽƌŬƐŚŽƉ;dƌĂŬDĂƌŬϮϬϭϮͿ
ǁĂƐĐŽŶĚƵĐƚĞĚ͘䕔/ŵĂŐĞƐĞƋƵĞŶĐĞƐĐĂƉƚƵƌĞĚďǇĂZ'ͲƐĞŶƐŽƌǁĞƌĞƉƌĞƐĞŶƚĞĚ ;ďǇĂŶŝĞůƌĞŵĞƌƐ
ĨƌŽŵdĞĐŚŶŝƐĐŚĞ hŶŝǀĞƌƐŝƚĂƚ DƵŶĐŚĞŶ ;dhDͿͿ䕔DĞƚŚŽĚƐ ŽĨĐƌĞĂƚŝŶŐŐƌŽƵŶĚ ƚƌƵƚŚŽƌƌĞĨĞƌĞŶĐĞĚĂƚĂŽĨĐĂŵĞƌĂƉŽƐŝƚŝŽŶĂŶĚŽƌŝĞŶƚĂƚŝŽŶǁĞƌĞƉƌŽƉŽƐĞĚ͘ͲŵĞƚŚŽĚƵƐŝŶŐŵŽƚŝŽŶĐĂƉƚƵƌĞƐǇƐƚĞŵ;ďǇĂŶŝĞůƌĞŵĞƌƐ ĞƚĂůͿͲŵĞƚŚŽĚƵƐŝŶŐǀŝƐƵĂůŵĂƌŬĞƌƐ;ďǇǀŐĞŶŝǇDŝŶĂĞǀ ĞƚĂůͿͲ ŵĞƚŚŽĚƵƐŝŶŐ'ŵŽĚĞůƐ;ďǇ<Žũŝ DĂŬŝƚĂĞƚ Ăů͕^ĂƌĂDĂƌƚƵůů ĞƚĂůͿ䖃DĞƚĂͲĚĂƚĂǁŝƚŚĂƚĂƐĞƚŵĂǇďĞĞĨĨĞĐƚŝǀĞ͘Ͳ&ŽƌǁĂƌĚͲďĂĐŬǁĂƌĚĚĂƚĂͬDŽǀŝŶŐŽďũĞĐƚ
ͲZĞŐŝŽŶŽĨ/ŶƚĞƌĞƐƚ
䖃ĂƚĂƐĞƚͲĂĚũƵƐƚŵĞŶƚĨŽƌƐƉĞĐŝĨŝĐƚƌĂĐŬŝŶŐŵĞƚŚŽĚƐ;Ğ͘Ő͘ ŵĞƚŚŽĚƐƵƐŝŶŐůŝŶĞĚĞƚĞĐƚŝŽŶ͕ŵĞƚŚŽĚƐƵƐŝŶŐǀŝƐƵĂůŵĂƌŬĞƌƐͿ͘;ďǇ<ĞŝƐƵŬĞ,ŝƌŽƐĞ ĞƚĂů͕zƵŬŝŬŽ<ƵďŽĞƚĂůͿ䖃ZĞůĂƚŝŽŶďĞƚǁĞĞŶƐƉĞĐŝĨŝĐĂƚŝŽŶƐ ŽĨŚĂƌĚǁĂƌĞĚĞǀŝĐĞƐĂŶĚƚƌĂĐŬŝŶŐƌĞƐƵůƚƐ͘䕔ĞŶĐŚŵĂƌŬŝŶŐŵĞƚŚŽĚŝŶĚĞƉĞŶĚĞŶƚŽĨƐƉĞĐŝĨŝĐĂƚŝŽŶƐ ŽĨŚĂƌĚǁĂƌĞĚĞǀŝĐĞƐǁŝƚŚďŽŽƚĂďůĞh^ŽĨ>ŝŶƵǆƐǇƐƚĞŵ;ďǇzŽƐŚŝŶĂƌŝ <DͿ͘ϮϬϭϯ/^DZϮϬϭϯ

:ŽŝŶƚtŽƌŬƐŚŽƉŽŶdƌĂĐŬŝŶŐDĞƚŚŽĚƐΘƉƉůŝĐĂƚŝŽŶƐĂŶĚdƌĂŬDĂƌŬǁĂƐĐŽŶĚƵĐƚĞĚ͘䖃ŽƐƚ ŽĨ ƐĞƚ ƵƉ ŶĞĞĚĞĚďĞĨŽƌĞƵƐŝŶŐZͬDZĂƉƉůŝĐĂƚŝŽŶ͘䕔ǀĂůƵĂƚŝŽŶ ƵƐŝŶŐ WsK;WƌŽũĞĐƚŝŽŶƌƌŽƌƐŽĨsŝƌƚƵĂůKďũĞĐƚƐͿ͘;ďǇDĂƐĂǇƵŬŝ ,ĂǇĂƐŚŝĞƚĂůͿ

ϮϬϭϰ

/^DZϮϬϭϰ

dƌĂŬDĂƌŬƉĂŶĞůƐĞƐƐŝŽŶǁĂƐĐŽŶĚƵĐƚĞĚ͘䕔ƵƌƌĞŶƚƐƚĂƚƵƐŽĨdƌĂŬDĂƌŬĚĂƚĂƐĞƚǁĂƐ ƉƌĞƐĞŶƚĞĚ͘;ϮϱĚĂƚĂƐĞƚƐƚŽƉƵďůŝĐ䚸ϯϴϬůƵͲƌĂǇĚŝƐŬƐ ŚĂǀĞďĞĞŶĚŝƐƚƌŝďƵƚĞĚͿϮϬϭϱ

/^DZϮϬϭϱ

dƌĂĐŬŝŶŐ ĐŽŵƉĞƚŝƚŝŽŶ ǁĂƐ ĐŽŶĚƵĐƚĞĚ͘;dƌĂŬDĂƌŬ ǁĂƐ ŽŶĞŽĨƐƵƉƉŽƌƚĞƌƐŽĨƚŚĞĐŽŵƉĞƚŝƚŝŽŶͿ

䕔>ĞǀĞůϯƐƵďŵŝƐƐŝŽŶƐ;ŵŽƐƚĚŝĨĨŝĐƵůƚ ůĞǀĞůͿĨŽƌŽĨĨͲƐŝƚĞĐŽŵƉĞƚŝƚŝŽŶǁĞƌĞĞǀĂůƵĂƚĞĚďǇƚŚĞ/^DZϮϬϭϱdƌĂĐŬŝŶŐŽŵƉĞƚŝƚŝŽŶĐŽŵŵŝƚƚĞĞ͘ǀĂůƵĂƚŝŽŶĐƌŝƚĞƌŝĂŝŶdƌĂŬDĂƌŬ
ďĂƐĞĚ ŽŶWsK;ΎͿǁĂƐĂƉƉůŝĞĚĨŽƌƚŚĞĞǀĂůƵĂƚŝŽŶ͘䖃EŽƚŽŶůǇWsK;ΎͿďƵƚĂůƐŽĐŽŵƉůĞƚĞŶĞƐƐŽĨĂƚƌŝĂůŝƐŝŵƉŽƌƚĂŶƚ͘;ΎWsK͗WƌŽũĞĐƚŝŽŶƌƌŽƌŽĨsŝƌƚƵĂůKďũĞĐƚƐͿ

Table 2: Benchmark indicators for off-site and on-site benchmarking.

5HOLDELOLW\

7HPSRUDOLW\

9DULHW\

2IIVLWH

2QVLWH '(92
 3(92
 5HSURMHFWLRQHUURURILPDJHIHDWXUHV 3RVLWLRQDQGSRVWXUHHUURUVRIDFDPHUD '(92
 3(92
 5HSURMHFWLRQHUURURILPDJHIHDWXUHV 3RVLWLRQDQGSRVWXUHHUURUVRIDFDPHUD
 &RPSOHWHQHVVRIDWULDO 7KURXJKSXW /DWHQF\

 7KURXJKSXW /DWHQF\
 7LPHIRUWULDOFRPSOHWLRQ

 1XPEHURIGDWDVHWV
 9DULHW\RQSURSHUWLHVRIGDWDVHWV

 1XPEHURIWULDOV 9DULHW\RQSURSHUWLHVRI
WULDOVc 2017 The Author(s)Eurographics Proceedings c 2017 The Eurographics Association.

3. Benchmarking indicators
This section outlines three major types of benchmark indicators (reliability, temporality, and variety), which should be considered forfair comparison of vSRT methods in MAR. Table 2 shows representative benchmark indicators for off- and on-site benchmarking.
3.1. Reliability indicators
The following four indicators on reliability are for both off- andon-site benchmarking.
• 3D error of a virtual object (3DEVO), which is the difference
between the estimated position of a virtual object and the groundtruth. 3DEVO is one of the most direct and intuitive indicator forvSRT methods for MAR, as one of the most principal functionsof MAR systems is to allign virtual objects in the 3D space basedon the results obtained by the target vSRT method.
• Projection error of a virtual object (PEVO), which is also oneof the most direct and intuitive indicators for vSRT methods forR. Ichikari, T. Kurata, K. Makita, T. Taketomi, H. Uchiyama, T. Kondo, S. Mori & F. Shibata / Reference Framework on vSRT-method Benchmarking for MAR

MAR, as one of the most important functions of MAR systems is
to render virtual objects based on the results obtained by the target vSRT method. Assuming the simplest case in which a virtual
point is projected as a virtual object to an estimated image plane,
the distance between the projected and correct point is calculated
as a PEVO value (see Fig. 4). The PEVO value may be measured
in degrees or pixels. The angular distance measure can provide
a uniform measure in a screen space, whereas the pixel number
varies depending on the position in a screen space.
• Re-projection error of an image feature, which is the distance
between a detected image feature in an image plane and the reprojection to the image plane with the 3D coordinates of the image feature that are recovered based on the target vSRT method.
Assuming the simplest case in which the image feature is a feature point, the re-projection error may be the distance between
the detected feature point and the re-projected point, and may bemeasured in degrees or pixels as with PEVO.
• Position and posture errors of a camera, which is the difference between the estimated position and posture of a camera andthe ground truth.
In addition to the aforementioned reliability indicators, completeness of a trial should be employed, especially for on-sitebenchmarking. This is because in many on-site competitions,many MAR systems cannot help but to stop performing spatial
registration and tracking in the middle of trial.
• Completeness of a trial, which involves evaluating the extent ofa trial completion. It is regarded as the robustness of the targetvSRT method.
sŝƌƚƵĂůƉůĂŶĞ
;ǌсĂͿ

ǌ

‫ݔ‬ൌെ


/ŵĂŐĞ
ƉůĂŶĞ
;'ƌŽƵŶĚƚƌƵƚŚͿ






'

&
,
/

;Ƶ͕ǀͿ

‫ݕ‬ൌെ

ܽ ή ‫ܥ‬௫
ʹ‫ܨ‬௫

‫ݔ‬ൌ

ܽ ή ‫ܥ‬௫ʹ‫ܨ‬௫ܽ ή ‫ܥ‬௬ʹ‫ܨ‬௬

Table 3: Trial set for benchmarking.
2IIVLWH,PDJHVHTXHQFHV,QWULQVLFH[WULQVLFFDPHUDSDUDPHWHUV&KDOOHQJHSRLQWV
2SWLRQDOFRQWHQWV 'PRGHOVIRU WKHWDUJHWREMHFWVDQGIRU YLUWXDOREMHFWV ,PDJHIHDWXUHFRUUHVSRQGHQFHV 'HSWKLPDJHVHTXHQFHV 6HOIFRQWDLQHGVHQVRUGDWDHWF

0HWDGDWD

6FHQDULR&DPHUDPRWLRQW\SH&DPHUDFRQILJXUDWLRQ,PDJHTXDOLW\&RQWHQWV

3K\VLFDOREMHFWV

0HWDGDWD,QIRUPDWLRQ RQKRZWRILQGWKHSK\VLFDOREMHFWV&RQWHQWV'DWDVHW

3K\VLFDO
REMHFW
LQVWDQFHV

6FHQDULR

For on-site benchmarking, the time for trial completion may be
used as a temporality indicator because it is easy to measure.
• Time for trial completion, which is the length of time fromstarting a trial to finishing it.
Actually, the time for trial completion is often used in on-sitecompetitions and can represent the overall performance of a target
MAR system. However, that it may include other aspects of the
MAR system such as image capturing, virtual object rendering, andhuman factors regarding the operator should be considered.

ǆ
,QSXW

ܽ ή ‫ܥ‬௬
‫ݕ‬ൌ
ʹ‫ܨ‬௬

,PDJH
W

,PDJH
W

Ǉ

,PDJH
W

,PDJH
W

,PDJH
W

3RVLWLRQ
3RVWXUHW

3RVLWLRQ
3RVWXUHW

3RVLWLRQ
3RVWXUH
W/ŵĂŐĞƉůĂŶĞ
;ƐƚŝŵĂƚĞĚĐĂŵĞƌĂƉĂƌĂŵĞƚĞƌƐͿ

Figure 4: PEVO with 3D coordinates of virtual objects.

3RVLWLRQ
3RVWXUHW

3RVLWLRQ
3RVWXUHW

7KURXJKSXWsŝƌƚƵĂůƉůĂŶĞ;ĨƌŽŶƚǀŝĞǁͿ;Ƶ͕͛ǀ͛Ϳ&KDOOHQJHSRLQWV'PRGHOVIRUWKHWDUJHW
REMHFWVDQGIRUYLUWXDO
REMHFWV

7DUJHW,PDJH

ǆ
Ǉ




with the system to when rendering a virtual object based on
the estimated position and posture of a camera with which
the target image was captured.
◦ For vSRT-method benchmarking, the latency may be thelength of time from when starting to load a target image into
the target method for input to when returning the estimatedposition and posture of a camera with which the target image
was captured.

2XWSXWDVD
Y657PHWKRGĂŵĞƌĂƉŽƐŝƚŝŽŶ

2QVLWH

/DWHQF\DVD
Y657PHWKRG/DWHQF\DVD
0$5V\VWHP
2XWSXWDVD
0$5V\VWHP

5HQGHULQJW

5HQGHULQJW

5HQGHULQJ
W

5HQGHULQJW

5HQGHULQJ
W

7LPH

3.2. Temporality indicators

Figure 5: Temporality indicators: latency and throughput.

The following two indicators on temporality (see Fig. 5) are basically suitable for off-site benchmarking.
• Throughput, which is the rate at which a target image is processed through a target vSRT method or target MAR system during a specific period. It is often called frame rate.
• Latency, which is the time delay inevitably produced by a targetvSRT method or target MAR system.
◦ For MAR-system benchmarking, the latency may be thelength of time from when starting to capture a target image

3.3. Variety indicators
This subsection presents two variety indicators required to prevent
fine tuning and cheating with some specific datasets for benchmarking. The following two indicators are for off-site benchmarking.
• Number of trial sets, which is the number of datasets used toobtain a benchmark indicator.

c 2017 The Author(s)Eurographics Proceedings c 2017 The Eurographics Association.

R. Ichikari, T. Kurata, K. Makita, T. Taketomi, H. Uchiyama, T. Kondo, S. Mori & F. Shibata / Reference Framework on vSRT-method Benchmarking for MAR• Variety on properties of trial sets, which is variety on properties of trial sets used to obtain a benchmark indicator. Typicalexamples of properties are the following:◦ Camera motion types, camera configurations, image quality,
and lighting conditions.
In general, performing fine tuning or cheating with many trial
sets is difficult. However, this difficulty may diminish if propertiesof the datasets used for benchmarking are homogeneous.
The following two indicators are for on-site benchmarking.
• Number of trials, which is the number of trials attempted in an
on-site benchmarking.
• Variety on properties of trials, which is variety on propertiesof trials in an on-site benchmarking.
4. Trial set for benchmark
This section identifies the reference elements in a trial set forbenchmarking. Table 3 shows representative elements in a trial set
for on- and off-site benchmarking.
4.1. Dataset for off-site benchmarking• Contents◦ Image sequences, which are target image sequences.
◦ Intrinsic/extrinsic camera parameters, which are the groundtruth of intrinsic and extrinsic parameters for one or more
video cameras to capture the image sequences.
◦ Optional contents◦ 3D models for both target and virtual objects, which are
3D model data for the target objects in image sequences
and for virtual objects overlaid in benchmarking.
◦ Image feature correspondences, which is the ground truthof image feature correspondences through the image sequences.
◦ Depth image sequences, which are image sequences thateach pixel value is measured with depth sensors using, forexample, an active stereo method.
◦ Self-contained sensor data, which are sensor data measured by self-contained sensors such as an accelerometer,gyro sensor, magnetometer, and barometer.
• Metadata◦ Scenario, which describes an application scenario of MAR,such as an indoor/outdoor navigation, tabletop MAR interface, or industrial application.
◦ Camera motion type, which describes camera motion suchas translation only, rotation only, walking motion, handheld
motion, or vehicle motion.
◦ Camera configuration, which describes the camera configuration such as white balance, shutter type (global or rolling),and shutter speed.
◦ Image quality, which describes the target-image quality suchas image resolution, defocusing, and motion blur.
The metadata can be used as a source of variety indicators andcan also correspond to properties of each dataset for ease of dataset
retrieval.
c 2017 The Author(s)Eurographics Proceedings c 2017 The Eurographics Association.

4.2. Dataset for on-site benchmarking• Contents◦ Challenge points, which are points for rating each MAR system with a 3DEVO or a PEVO measurement. The positionsof each challenge point are estimated and visualized by thetarget MAR system for rating.
4.3. Physical objects for off-site benchmarking• Contents◦ Physical objects, which must be easily available or deliverable physical objects that can be captured as images by contestants. Such physical objects are necessary for contestants
in developing and adjusting their vSRT method or MAR system before conducting off-site benchmarking with target image sequences in which the physical objects are observed.
• Metadata◦ Information on how to find the physical objects
In actual benchmarking activities, paper models and toy bricks
can be employed as easily available or deliverable physical objects,
as in [TMC15] [COS].
4.4. Physical objects for on-site benchmarking• Contents◦ Physical objects, which are physical objects for benchmarking preparation and actual benchmarking. For preparationuse, physical objects must be easily available or deliverable as in off-site benchmarking. However, for actual benchmarking, physical objects may be observable only during trials, as in [TMC15] [VTC14]
5. Findings from the ISMAR 2015 tracking competition
5.1. Off-site competition
Benchmark indicator
PEVO is one of the most direct and intuitive indicators forvision-based geometric registration and tracking methods forMAR. PEVO worked well because it is sensitive to errors of position and orientation of the camera. Actually, some cases existed
in which PEVO was large even if re-projection errors of image features was small. The organizers had asked each contestant to submit
projective camera matrices for data with lens distortions. However,some contestants submitted the normalized camera matrices thatconsisted of position and orientation and that did not contain the
intrinsic parameters of the camera. In addition, some other contestants submitted the projective camera matrices. This made it difficult for organizers to fairly and equally evaluate each contestant.
Scalability
Dealing with various submissions is becoming problematic withthe tremendous increase in the number of contestants. Ensuring thateach contestant thoroughly understand the rules and regulations is

R. Ichikari, T. Kurata, K. Makita, T. Taketomi, H. Uchiyama, T. Kondo, S. Mori & F. Shibata / Reference Framework on vSRT-method Benchmarking for MAR

also difficult. That the organizers provide software for evaluation as
well as datasets and that each contestant submits only certain parts,
such as final trajectories, are appropriate actions to take.
Cheating and fine tuning
Judges should evaluate each method by themselves, for instance,by having each contestant submit a binary (executable) program.
Otherwise, checking for cheating (such as using future information to conduct global optimization using the entire data) would be
difficult. To alleviate fine-tuning issues, providing contestants withvarious types of datasets should be effective.
5.2. On-site competition
Effects of Simplification
Simplifying the benchmarking process is often necessary forpractical operation of tracking competitions. However, the following pros and cons should be considered.
[Equality and Simplification]
In the competition, the condition for each contestant was notstrictly equal. Each contestant was supposed to indicate (with amark on a textured paper attached on a partition wall) the specified
challenge point, which was a virtual point determined as the simplest virtual object for accuracy evaluation and estimated by each
MAR system (see Fig. 6). To save time for competition operation,
the organizers did not change the paper for subsequent contestants
after being marked by previous contestants. Therefore, a paper having a mark became a part of the environment for subsequent contestants. This might not have been fair, as the appearance of the paper
had changed due to the mark, even if the mark was decidedly small.
[Measurement and Simplification]
Ideally, the measurement for accuracy evaluation should bebased on the distance between the 3D coordinates of the challenge
point given by the organizers and those of the challenge point estimated by the contestant’s MAR system in the 3D space. However, in the aforementioned competition, it was based on the 2D
coordinates on the textured paper. This was because measuring the
distance of arbitrary points in the air in the 3D space was quite difficult. More than three challenge points on a plane are necessary if
the organizers want to evaluate strictly the 3D position and orientation of the camera with these 2D coordinates.
Benchmark Indicator
In the on-site competition, the organizers employed the following strategy for evaluation:(1) Compare the number of challenge points each contestantfound.

As a result, only (1) was determined to be important because
only one contestant completed the trial. In terms of a comprehensive evaluation, utilizing other benchmark indicators such as
temporality indicators including frame rate and latency would be
worthwhile.
Measurement process
From the organizer’s point of view, on-site competitions shouldbe held at various locales. Therefore, using the same or a standardized tool in the preparation phase for measuring environments andground truth and in order to produce correspondence between thereal-world coordinate system and the local one is preferable. In thetrial phase of this competition, the 3DEVO was measured manually using a ruler. The measurement was easily realized becausea challenge point should be located on a plane. However, if it is
not on a plane, a better means of measuring the 3DEVO should beconsidered.
From the contestant’s point of view, two choices exist for acquiring and constructing environmental 3D models in each MAR
system by each contestant: before or during a trial. In this competition, online acquisition (or visual SLAM) was required. Considering various scenarios for MAR system usage, encompassing
modeling before a trial in competition is preferable.
Human factor
Because in the aforementioned competition the user of the MAR
system was supposed produce marks on the paper attached to thepartition wall, when the camera panned closer to the paper, image
registration and tracking often became unstable (see Fig. 6). As
expected, one of the major techniques used to obtain high scores in
the competition was to devise and master a method of moving thecamera.
Even though detailed rules and regulations were documented,
they were not well understood by the contestants because of thevery busy schedule of the competition and the rules and regulations became problematic during the tracking competition. Visual
aids such as presentation slides and handouts would help to reacquaint the rules and regulations. Step-by-step contestant acknowledgement and agreement of rules and regulations during each test
would also be helpful.
Difficulty level design
The difficulty of each trial of online registration and tracking
strongly depends on the combinations and positions of objectsaligned in the competitive environment. As a result, adjusting thedifficulty to a more moderate level is difficult. However, multiple
challenge points alleviated the problem to some extent. In this competition, millimetric accuracy had some meaning for the first challenge point. As previously mentioned, only one contestant completed the trial. For subsequent challenge points involving 10 to
20 m movement, millimetric evaluation was not important. Instead,
how to maintain stable tracking was the chief concern.

If (1) is the same for multiple contestants,

Dissemination(2) Compare the mean distance, which corresponds to 3DEVO

By showing a screen capture of each MAR system during eachtrial on a large screen, the organizers were able to ensure the audience enjoyed the competition. This is also effective at make thecompetition open and public.

If (2) is also the same for multiple contestants,(3) Compare the time for trial completion.

c 2017 The Author(s)Eurographics Proceedings c 2017 The Eurographics Association.

R. Ichikari, T. Kurata, K. Makita, T. Taketomi, H. Uchiyama, T. Kondo, S. Mori & F. Shibata / Reference Framework on vSRT-method Benchmarking for MAR

ization [PBS] [PCW17] These efforts will make this framework
not only more general but also more adaptive.
&KHFN

3URFHVV,WHP>@ 'HYHORSY657PHWKRGVDQGRU0$5V\VWHPV>@ *DWKHUY657PHWKRGVDQGRU0$5V\VWHPV>@ 3UHSDUHDQGFRQGXFWEHQFKPDUNLQJ

>@ 3URYLGHDQGPDLQWDLQEHQFKPDUNLQJLQVWUXPHQWV

>5HPDUNV@ 3URYLGHDQGPDLQWDLQEHQFKPDUNLQJUHSRVLWRULHV

>@ 6KDUHEHQFKPDUNLQJUHVXOWV
7,26

3URFHVVIORZ
7DUJHW 7 ,QSXW , 
2XWSXW 2 
2UJDQL]HGVWRUDJH
6>@ Y657PHWKRG

>@ 0$5V\VWHP>@ 7ULDOVHWVDQGSK\VLFDOREMHFWV

>@ %HQFKPDUNLQJLQVWUXPHQWV

>@ %HQFKPDUNLQJUHVXOWV>@ %HQFKPDUNLQJVXUYH\V>@ %HQFKPDUNLQJUHSRVLWRU\>@ ([WHUQDOUHSRVLWRULHV&KHFN

5HOLDELOLW\

Figure 6: One of the contestants marking on the textured paper.
,QGLFDWRUIRUPXUD

7HPSRUDOLW\
9DULHW\

6. Conclusions
This study presented a reference framework on benchmarking ofvSRT methods for MAR and summarized findings from benchmarking activities. Although we did not identify or define specificbenchmarking frameworks, specific processes, specific formulas ofbenchmarking indicators, or a specific format of trial sets for benchmarking, our framework can be used by the following individuals:• A benchmarking service provider, a benchmark provider, or abenchmarking competition organization who wants to align theirbenchmarking activities including open/closed competitions andself-benchmarking to be consistent with this document;• A technology developer or supplier who wants to estimate andevaluate performance of a vSRT method for MAR appropriately
with a benchmarking service provider, a benchmark provider,or a benchmarking competition organization who aligns theirbenchmarking activities to be consistent with this document inorder;• A technology user who wants to obtain the benchmarking results based on a benchmarking activity, which is consistent withthis document, to compare vSRT methods for MAR in terms of
performance.

>@ '(92

>@ 3(92

>@ 5HSURMHFWLRQHUURURILPDJHIHDWXUHV>@ 3RVLWLRQDQGSRVWXUHHUURUVRIDFDPHUD

>@ &RPSOHWHQHVVRIDWULDO>@ 7KURXJKSXW>@ /DWHQF\>@ 7LPHIRUWULDOFRPSOHWLRQ

>@ 1XPEHURIGDWDVHWVWULDOV>@ 9DULHW\RQSURSHUWLHVRIGDWDVHWVWULDOV&KHFN&RQWHQWV'DWDVHW7ULDOVHW
IRUPDW

0HWDGDWD>@ ,PDJHVHTXHQFHV>@ ,QWULQVLFH[WULQVLFFDPHUDSDUDPHWHUV

>@ &KDOOHQJHSRLQWV

>@ 2SWLRQDOFRQWHQWV>@ 6FHQDULR

>@ &DPHUDPRWLRQW\SH

>>

3K\VLFDO &RQWHQWV >REMHFW
LQVWDQFHV 0HWDGDWD >@ &DPHUDFRQILJXUDWLRQ@ ,PDJHTXDOLW\@ 3K\VLFDOREMHFWV@ +RZWRILQGWKHSK\VLFDOREMHFWVTable 4: Check sheet for summarization of benchmarking activities.

References[COS] The City of Sights: An Augmented Reality Stage Set. http://
studierstube.icg.tugraz.at/handheld_ar/cityofsights.php.
5[I291] ISO/IEC 29155-1, Systems and software engineering Information technology project performance benchmarking framework - Part 1: Concepts and definitions. . 2[IW185] ISO/IEC CD 18520, Information technology-Computer
graphics, image processing and environmental representationBenchmarking framework of vision-based spatial registration andtracking methods for MAR. 7

As in Table 4, we have been making a check sheet to systematically summarize how each benchmarking activity such as an on-siteor off-site competition is designed. The check sheet is also assumed
to facilitate our confirming the structure of target benchmarking
activities. One of future works is to evaluate how the check sheet
works in terms of summarization, confirmation, quality control, andso on.

[KMP*13] KURZ D., MEIER P., PLOPSKI A., KLINKER G.: An
outdoor ground truth evaluation dataset for sensor-aided visual handheld
camera localization. In Proc. IEEE and ACM International Symposiumon Mixed and Augmented Reality (ISMAR2013) (2013), pp. 263-264. 1

Currently, this framework has been standardized in ISO/IEC JTC
1/SC 24/WG 9 [IW185]. We hope those who are interested in it
will contribute to standardization activities.

[MOK14] MAKITA K., OKUMA T., KURATA T.: Benchmarking
indicators for AR/MR camera tracking. In Proc. The 7th Korea-Japan
Workshop on Mixed Reality (KJMR 2014) (2014). 1

Moreover, we believe this framework should become the baseline to standardize spatial registration and tracking methods thatutilize not only a video camera but combine a video camera withother sensors such as stereo cameras, depth cameras, inertial sensors, infrastructure-based positioning technologies with Internet ofThings, or global navigation satellite systems.
Finally, by analogically utilizing this framework, the benchmarking of pedestrian dead-reckoning is also discussed for the standardc 2017 The Author(s)Eurographics Proceedings c 2017 The Eurographics Association.

[LBM*09] LIEBERKNECHT S., BENHIMANE S., MEIER P.,
NAVAB N.: A dataset and evaluation methodology for templatebased
tracking algorithms. In Proc. IEEE and ACM International Symposiumon Mixed and Augmented Reality (ISMAR2009) (2009), pp. 145-151. 1[PBS] PDR Benchmark Standardization Committee:.
facebook.com/pdr.bms/. 7

http://www.

[PCW17] PDR Challenge in Warehouse Picking in IPIN 2017:.
http:/www.ipin2017.org/. 7[PITC] Previous ISMAR Tracking Competitions:. http://ypcex.
naist.jp/trakmark/tracking-competition/?page_id=538. 1[SIKU*10] SHIBATA F., IKEDA S., KURATA T., UCHIYAMA H.:
An intermediate report of TrakMark WG. In Proc. IEEE and ACM International Sympo-sium on Mixed and Augmented Reality (ISMAR 2010)(2010), pp. 233–236. 1

R. Ichikari, T. Kurata, K. Makita, T. Taketomi, H. Uchiyama, T. Kondo, S. Mori & F. Shibata / Reference Framework on vSRT-method Benchmarking for MAR[TK*09] TAMURA H., K ATO H.: Proposal of International Voluntary
Activities on Establishing Benchmark Test Schemes for AR/MR
Geometric registration and Tracking Methods. In Proc. IEEE andACM International Sympo-sium on Mixed and Augmented Reality (ISMAR2009) (2009), pp. 298–302. 1[TMC15] Results of ISMAR 2015 Tracking Competition:.
http://ypcex.naist.jp/trakmark/tracking-competition/?page_id=1042. 1, 5
[TMK11] Publications in the 2nd international TrakMark workshop (TrakMark 2011):. http://www.trakmark.net/workshop11/index.
html. 1
[TMK12] Publications in the 3rd international trakmark workshop (trakmark 2012):.
http://studierstube.icg.tugraz.at/
ISMARTrackingWorkshop/l. 1[TMK13] Publications in IEEE ISMAR 2013 Joint Workshop on
Tracking Methods & Applications and TrakMark:. http://www.
trakmark.net/workshop12/index.html. 1[TMP14] Publications in TrakMark Panel Session (How to Benchmark
AR/MR Geometric Registration and Tracking Methods) in IEEE ISMAR 2014:. http://www.trakmark.net/ismar14/. 1[TMW]

Trakmark web site:. http://www.trakmark.net/. 1[VTC14] Volkswagen Tracking Challenge in ISMAR 2014:.
http://www.tracking-challenge.de/content/tc/content/
en/history/challenge_2014/scenarios.html. 5[WGB13] WILLIAMS S., GREEN R., BILLINGHURST M.: Transform flow: A mobile augmented reality visualization and evaluation
toolkit. In Proc. IVCNZ 2013 (2013). 1

c 2017 The Author(s)Eurographics Proceedings c 2017 The Eurographics Association.