KJMR 2009

Annotation View Managementf WearableforW
bl AugmentedAt d Reality
R lit
Considering Movement of Target Objects

Koji Makita, Masayuki Kanbara and Naokazu Yokoya
Nara Institute of Science and Technology
Technology, Japan（NAIST）

Annotation overlay using wearable AR
Wearable AR shows virtual object as an annotation
using 2D images or 3D models[S.Feiner et al. 1997][Tenmoku et al. 2007]

The readability and the intelligibilityof annotations are important2

Our previous work
K. Makita,, M. Kanbara,, and N. Yokoya:y"Personal information annotation on wearable computer users with hybrid peer-to-peer
communication", Proc. 16th Int. Conf. on Artificial Reality and Telexistence (ICAT 2006),
pp. 217 - 227, 2006.

Motivation:
The readability and the intelligibility should be improved3 !

Related works
View management (VM) of annotations•Avoiding annotation-overlaps
The readability and the•Reducing the amount of hiding
intelligibility ofannotations are improved･･･

Without VM

With VM

A view management method formobile mixed reality systems[F.Shibata et al. EGVE2008]

Without VM

With VM

Evaluating label placement for augmented
reality view management[R.Azuma et al. ISMAR2003]

The positions
positions, orientations,orientations and shapes of annotated objects
4should be known for view management

Goal and approach
GoalView Management
for improving the readability and intelligibility ofannotations for moving target objects
ACD

B

Without view management

AC

B

D
With view management

Approach
Step 1:
Estimation of target object regions

Step 2:
Annotation overlay
CBA
D
5

Step1： Estimation of target object regions～Objective～
Estimating target’s region(with the individual recognition)～Assumption～
Each user has wearable PC
to estimate positionApproachObtaining the target’s position
data via wireless network

Estimating target’s region
by using primitive models
6

Estimation of existing region of the targets
Obtaining the position data via
wireless network（*）
Obtaining the position data with identifier
via hybrid P2P network(*)Personall information(*)
fannotation on wearable
blcomputer users with hybrid peer-to-peer
communication (Makita et al, ICAT2006)

Head

Generating a “probability map” byrendering primitive models

Upper
bodyLower
body

Example of a probability map

7LowgExisting
probabilittyp

Each part has an existing
probability as a parameter

User’sposition
High

Estimating target’s region byusing primitive models

Step2： Annotation overlay～Objective～
Decision off theh arrangementof annotations

AB～Assumption～C•Targets’ regions are known•Individual recognition is achieved

DA
ApproachhMaking a list offfactors
relatedl d tothe readability andintelligibility ofannotations

Introducing
penaltyfunction foreach factor

Overlaying
annotations byminimizing the sumof penalties
8Factors related to readability and intelligibility
Annotation

Annotation

Annotation

Annotation
Annotation

AnnotationPenalty
related tooverlapslAnnotation

Po

Annotation

Penalty relatedto indication
line

Penaltyrelated todistance

Annotation

PlP
Penaltylt ffunctionti（ w : weightedi ht d parameters）t ）Annotation

PdMinimizing P 9

P = w o Po + w l Pl + w m Pm

Calculation of penalty related to overlapping
Panno

Annotation

Annotation

Annotation

Annotation

Penaltylrelated tooverlaps

Annotation
Annotation

At ti
Annotation

Po = Panno + Pmap

Annotation

Pmap
Penalty related to overlapping between
annotations： Panno

Panno ( x, y ) =| A( x , y ) |A( x , y ) ：A set of points that are occluded when an
annotation is placed at ( x , y )| A( x , y ) |：The number of points in a set A( x , y )10Calculation of penalty related to overlapping
Panno

Annotation

Annotation

Annotation

Annotation
Annotation

Annotation

Penaltyrelated topoverlaps

Annotation

Po = Panno + Pmap

Annotation

Pmapp
Penalty related to overlapping between theannotation and users： Pmap

( x, y )A( x , y )Pmap ( x, y ) = ∑ Cii∈ A( x , y )i ：a point on the probability map
gvalue of the ppoint i
C i ：the brightness

A( x , y ) ( x, y )A( x , y ) ：a set of points that are occluded if anannotation is placed at ( x , y )11Calculation of penalty related to distance
Penaltyrelated toindication
PllineAnnotation

Penaltyrelated todistance

PdA
Annotationt ti

Penalty related to the line between the annotation
Plandad the user：
uPl ∝ ll ：The length of the line between
the annotation and the user

In this study, Pl is empirically defined as

Pl = l212

Calculation of penalty related to distance
Penaltyrelated toindication
PllineAnnotation

Penaltyrelated todistanceA
Annotation
t ti

Pd

Penalty related to the distance between annotation
Pdpositions
poo ono the view planep a in sequential
qua frames：
aPd = Bt − AtAt

Annotation

Pd

At −1

Annotation
Time： t − 1

Bt

Time： t13

Decision of the arrangement of annotations
Annotation

Annotation

Annotation

Annotation

Sequential
minimization

Annotation

Annotation

The position of each annotation is decided in order
AnnotationGlobalminimization

Annotation

Annotation

Annotation

Annotation

・・・

Annotation
Annotation

Annotation

Annotation

All the patterns of arrangements of annotations are checkedComparison

PenaltyAmount of calculation

Sequentialminimization△（Local minimum）

O ((n)

Globalminimization○（Global minimum）

O (e n )14

Experiment～Settings～• Targets：Wearable
Targets Wearable computer users• The number of targets：3• Estimating
E ti ti position
iti andd orientation：i t tiA method using invisible visual markers (*)～Experiment description～• Pilot study：Estimation of the region of the targetpoverlayy• Experiment：Annotation

15

Estimating position and orientation(*)
Ceiling with invisible visual markers
Infrared light
I f
Infraredd camera

Wearable PC
Estimation ofposition and orientation

Captured image

User

Position error：9mm
Orientation error：0.35deg
Frequency：29times/sec(*)Y. Nakazato, M. Kanbara and N. Yokoya:“A localization system using invisible retro-reflective markers,”
Proc. IAPR Conf. on Machine Vision Applications (MVA2005), pp. 140–143, 2005.

16

Experimental environment
Invisible visual markers（on the ceiling）
Wireless LAN antennaNetwork
shared DBServer[CPU:PentiumD,3.0GHz]]LANIbuki

Takahashi

KumeWearableW
bl computert users（Position and orientation are estimated in real time）（In our laboratory. Distance to the ceiling is about 2.8m）
17Configuration of wearable AR system
Infrared
cameraOriginal(Resolution：1024×768)(Frame rate：29fps)
Capturing invisible
markersVideo
cameraOriginal((Resolution：640×480))(Frame rate：10fps)
Capturinguser’s viewPC
ThinkPad x60(lenovo)
Estimating user’s
position and orientation
Generating annotation
overlay images

HeadHdworn
displaySV-6(MicroOptical)
Showing annotation
overlay images

18

Pilot study：Estimation of the region of thetarget
HeadUpper body
（25 solids）

3D object annotation（fi d in（fixedi reall world）
ld）（81 solids）

Lower body（169 solids）

Wearable computer
users
Experimental result(Estimated regions are overlaid)

Annotation overlaying

19

3D object annotation（fixed in real world）

・Setting of penalty function

P = w o Po + w l Pl + w m Pm

( w o = 0 . 1, w l = 0 . 5 , w m = 0 . 2 )・Method for deciding the arrangement ofannotations
Sequential optimization(The order of annotations is fixed)

Annotations of users（overlaid with view management）

g ((for faster pypenalty
・Limited search rangecalculation)：Relative position of the annotationin the previous frame
g：Search range（21×21 pixels）：Search positions（16 points around the user’s position）

20

Frame rate：8～12 [fps]Experimental result (video)

21

Comparative result
Without proposed view management（Relative positions of annotations are fixed）

With proposed view management

22

Penalty distribution andannotation overlay image
HighPenalty
Low：Target annotation to calculate the penalty
23Conclusion
We have proposed annotation view management method
for wearable AR considering movement of target objects
Estimating existing regions of target objectsb usingbyi position
iti data
d t andd primitive
i iti modelsd lMinimizing penalty to decide the arrangement of annotations

Future work
Considering different settings of weighted parameters ofpenalty
ImprovingIi theth orderd off deciding
d idi annotationt ti positionsitiFiltering of annotations
24