Patent Publication Number: US-10311833-B1

Title: Head-mounted display device and method of operating a display apparatus tracking an object

Description:
BACKGROUND 
     1. Technical Field 
     Embodiments of the present invention relate to a technique of an information processing device which processes information regarding a target object using a camera and a motion sensor (e.g., an inertial sensor). 
     2. Related Art 
     Method of estimating a pose of an object imaged by a camera are disclosed in JP-A-2013-50947, which discloses a technique in which a binary mask of an input image including an image of an object is created, singlets as points in inner and outer contours of the object are extracted from the binary mask, and sets of the singlets are connected to each other so as to form a mesh represented as a duplex matrix so that a pose of the object is estimated. 
     However, the prior art, such as JP-A-2013-50947, only use a camera to estimate the pose. However, a camera has latency, which increases as the motion or velocity of the user&#39;s head increases. These systems also conduct pose estimation continuously, which consumes power. 
     SUMMARY 
     Embodiments of the present application disclose systems with a camera, and systems with a camera and a motion sensor. The embodiment of the combined camera and motion sensor reduces latency and increases efficiency of the system. Certain embodiments achieve a reduction in power consumption by discontinuing object tracking, thus reducing processor load. This power saving is especially advantageous for head-mounted display devices which may rely battery power. 
     An advantage of some aspects of the invention is to solve at least a part of the problems described above, and aspects of the invention can be implemented as the following aspects. 
     According to aspects of the invention, methods, systems and computer readable mediums are provided for operating a display apparatus having a display, a camera, and an inertial sensor. The methods, systems and computer readable mediums are configured for: (a) tracking a pose of an object in a field of view of the camera, the tracking including repeatedly: (1) deriving, using a processor, the pose of the object relative to the display apparatus using at least one of: image data acquired by the camera and sensor data acquired by the inertial sensor, and (2) displaying, using the display, an image based on the derived pose of the object; (b) deriving, using the processor, movement of the display apparatus or movement in the field of view using at least one of: the sensor data and the image data; (c) determining, using the processor, whether or not the derived movement exceeds a first threshold; (d) if the derived movement exceeds the first threshold, continuing tracking the pose of the object in step (a); and (e) if the movement does not exceed the first threshold: (1) stopping tracking the pose of the object in step (a); and (2) displaying, using the display, the image based on a previously derived pose of the object. 
     The invention may be implemented in forms aspects other than the information processing device. For example, the invention may be implemented in forms such as a head mounted display, a display device, a control method for the information processing device and the display device, an information processing system, a computer program for realizing functions of the information processing device, a recording medium recording the computer program thereon, and data signals which include the computer program and are embodied in carrier waves. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       Aspects of the invention will be described with reference to the accompanying drawings, wherein like numbers reference like elements. 
         FIG. 1  is a block diagram illustrating a functional configuration of a personal computer as an information processing device in the present embodiment. 
         FIG. 2  is a flowchart illustrating a template creation process performed by a template creator. 
         FIG. 3  is a diagram for explaining a set of N points in two dimensions representing a target object for a three-dimensional model, calculated by using Equation (1). 
         FIGS. 4A-4C  are a schematic diagram illustrating a relationship among 3D CAD, a 2D model, and a 3D model created on the basis of the 2D model. 
         FIG. 5  is a diagram illustrating an exterior configuration of a head mounted display (HMD) which optimizes a pose of an imaged target object by using a template. 
         FIG. 6  is a block diagram functionally illustrating a configuration of the HMD in the present embodiment. 
         FIG. 7  is a flowchart illustrating a process of estimating a pose of a target object. 
         FIG. 8  is a diagram illustrating that a single model point can be combined with a plurality of image points. 
         FIG. 9  is a diagram illustrating an example in which a model point is combined with wrong image points. 
         FIG. 10  is a diagram illustrating an example of computation of CF similarity. 
         FIG. 11  is a diagram illustrating an example of computation of CF similarity. 
         FIG. 12  is a diagram illustrating an example of computation of CF similarity. 
         FIG. 13  is a diagram illustrating an example of computation of CF similarity in a second embodiment. 
         FIG. 14  is a diagram illustrating an example of computation of CF similarity in the second embodiment. 
         FIG. 15  is a diagram illustrating an example of computation of CF similarity in the second embodiment. 
         FIG. 16A  is a diagram illustrating high latency between a pose and an object in an AR application, according to an embodiment. 
         FIG. 16B  is a diagram illustrating low or no latency between a pose and an object in an AR application, according to an embodiment. 
         FIG. 17  is a diagram illustrating a schematic configuration of an HMD with an inertial sensor and a camera sensor, according to an embodiment. 
         FIG. 18  is a block diagram illustrating a functional configuration of the HMD of  FIG. 17 , according to an embodiment. 
         FIG. 19  is a flowchart showing fusion of fused sensor data with image data to predict object location, according to an embodiment. 
         FIG. 20  is a flowchart showing fusion of fused sensor data with image data, according to an embodiment. 
         FIG. 21  is a flowchart showing object pose prediction, according to an embodiment. 
         FIG. 22  is a flowchart showing initializing sensor fusion, according to an embodiment. 
         FIG. 23  is a flowchart showing inertial sensor object tracker, according to an embodiment. 
         FIG. 24  is a flowchart showing reinitializing IMU fusion, according to an embodiment. 
         FIG. 25  is a flowchart illustrating handling vision loss, according to an embodiment. 
         FIG. 26  is a flowchart illustrating jitter reduction for a generated pose, according to an embodiment. 
         FIG. 27  is a diagram illustrating feature matching, according to an embodiment. 
         FIG. 28  is a diagram showing a power saving module according to one embodiment. 
         FIG. 29  is a diagram showing an embodiment of motion detection using sensor data according to one embodiment. 
         FIG. 30  is a flowchart showing a power saving method according to one embodiment. 
         FIG. 31  is a diagram showing an embodiment of motion detection using image data according to one embodiment. 
         FIG. 32  is a flowchart showing a power saving method according to one embodiment. 
         FIG. 33  is a flowchart showing a power saving method according to one embodiment. 
         FIG. 34  is a flowchart showing a power saving method according to one embodiment. 
     
    
    
     DESCRIPTION OF EXEMPLARY EMBODIMENTS 
     In the present specification, description will be made in order according to the following items.
         A. First Embodiment   A-1. Configuration of information processing device   A-2. Creation of template (training)   A-2-1. Selection of 2D model point   A-2-2. Determination of 3D model point and creation of template   A-2-3. In-plane rotation optimization for training   A-2-4. Super-template   A-3. Configuration of head mounted display (HMD)   A-4. Execution of estimation of target object pose   A-4-1. Edge detection   A-4-2. Selection of template   A-4-3. 2D model point correspondences   A-4-4. Optimization of pose   A-4-5. Subpixel correspondences   B. Second Embodiment   C. Third Embodiment   D. Modification Examples   D-1. Modification Example 1   E. 3D Tracking Objects Using Fusing of Inertial Motion Sensor (IMU) and Camera Sensor   E-1. Introduction   E-2-1. Overview   E-2-1-1. System   E-2-1-2. Overview of Method of Sensor Fusion   E-2-2. Tracker Fusing IMU   E-2-2-1. Method of Fusing IMU with 3D Object Tracker   E-2-2-2. Initialize IMU Fusion   E-2-2-2-1 Detect Static Motion   E-2-2-3. IMU Pose Predication   E-2-2-4. IMU Divergence Determination   E-2-2-5. IMU Object Tracker   E-2-2-5-1. Detect static motion   E-2-2-5-2. Predict feature point location using IMU predicted pose/KLT matching using predicted feature location   E-2-2-5-3. Pose estimation using KLT matching results &amp; IMU predicted pose   E-2-2-5-4. Pose refinement by edge alignment &amp; IMU predicted pose   E-2-2-5-5. Outlier removal   E-2-2-6. Reinitialize IMU Fusion   E-2-2-7. Fuse IMU and Vision   E-2-2-8. Handle Vision Loss   E-2-3. Get Jitter Reduced IMU Pose   E-2-3-1. Detect static motion   E-2-3-2. Detect Motion Jitter   E-2-4. Power Saving   E-2-4-1. Movement Detection   E-2-4-1. Power Saving Modes   E-3. Experiment Results   E-3-1. Latency improvement   E-3-2. Reduce Tracking Drift   E-3-3. Tolerate faster user motion       

     A. First Embodiment 
     A-1. Configuration of Information Processing Device 
       FIG. 1  is a block diagram illustrating a functional configuration of a personal computer PC as an information processing device in the present embodiment. The personal computer PC includes a CPU  1 , a display unit  2 , a power source  3 , an operation unit  4 , a storage unit  5 , a ROM, and a RAM. The power source  3  supplies power to each unit of the personal computer PC. As the power source  3 , for example, a secondary battery may be used. The operation unit  4  is a user interface (UI) for receiving an operation from a user. The operation unit  4  is constituted of a keyboard and a mouse. 
     The storage unit  5  stores various items of data, and is constituted of a hard disk drive and the like. The storage unit  5  includes a 3D model storage portion  7  and a template storage portion  8 . The 3D model storage portion  7  stores a three-dimensional model of a target object, created by using computer-aided design (CAD). The template storage portion  8  stores a template created by a template creator  6 . Details of the template created by the template creator  6  will be described later. 
     The CPU  1  reads various programs from the ROM and develops the programs in the RAM, so as to execute the various programs. The CPU  1  includes the template creator  6  which executes a program for creating a template. The template is defined as data in which, with respect to a single three-dimensional model (3D CAD in the present embodiment) stored in the 3D model storage portion  7 , coordinate values of points (2D model points) included in a contour line (hereinafter, also simply referred to as a “contour”) representing an exterior of a 2D model obtained by projecting the 3D model onto a virtual plane on the basis of a virtual specific viewpoint (hereinafter, also simply referred to as a “view”), 3D model points obtained by converting the 2D model points into points in an object coordinate system on the basis of the specific view, and the specific view are correlated with each other. The virtual viewpoint of the present embodiment is represented by a rigid body transformation matrix used for transformation from the object coordinate system into a virtual camera coordinate system and represented in the camera coordinate system, and a perspective projection transformation matrix for projecting three-dimensional coordinates onto coordinates on a virtual plane. The rigid body transformation matrix is expressed by a rotation matrix representing rotations around three axes which are orthogonal to each other, and a translation vector representing translations along the three axes. The perspective projection transformation matrix is appropriately adjusted so that the virtual plane corresponds to a display surface of a display device or an imaging surface of the camera. A CAD model may be used as the 3D model as described later. Hereinafter, performing rigid body transformation and perspective projection transformation on the basis of a view will be simply referred to as “projecting”. 
     A-2. Creation of Template (Training) 
       FIG. 2  is a flowchart illustrating a template creation process performed by the template creator  6 . The template creator  6  creates T templates obtained when a three-dimensional model for a target object stored in the 3D model storage portion  7  is viewed from T views. In the present embodiment, creation of a template will also be referred to as “training”. 
     In the template creation process, first, the template creator  6  prepares a three-dimensional model stored in the 3D model storage portion  7  (step S 11 ). Next, the template creator  6  renders CAD models by using all possible in-plane rotations (1, . . . , and P) for each of different t views, so as to obtain respective 2D models thereof. Each of the views is an example of a specific viewpoint in the SUMMARY. The template creator  6  performs edge detection on the respective 2D models so as to acquire edge features (step S 13 ). 
     The template creator  6  computes contour features (CF) indicating a contour of the 2D model on the basis of the edge features for each of T (P×t) views (step S 15 ). If a set of views which are sufficiently densely sampled is provided, a view having contour features that match image points which will be described later can be obtained. The 2D model points are points representing a contour of the 2D model on the virtual plane or points included in the contour. The template creator  6  selects representative 2D model points from among the 2D model points in the 2D contour with respect to each sample view as will be described in the next section, and computes descriptors of the selected features. The contour feature or the edge feature may also be referred to as a feature descriptor, and is an example of feature information in the SUMMARY. 
     If computation of the contour features in the two dimensions is completed, the template creator  6  selects 2D contour features (step S 17 ). Next, the template creator  6  computes 3D points having 3D coordinates in the object coordinate system corresponding to respective descriptors of the features (step S 19 ). 
     A-2-1. Selection of 2D Model Points (Step S 17 ) 
     The template creator  6  selects N points which are located at locations where the points have high luminance gradient values (hereinafter, also referred to as “the magnitude of gradient”) in a scalar field and which are sufficiently separated from each other from among points disposed in the contour with respect to each sample view. Specifically, the template creator  6  selects a plurality of points which maximize a score expressed by the following Equation (1) from among all points having sufficient large magnitudes of gradient. 
     
       
         
           
             
               
                 
                   
                     ∑ 
                     
                       i 
                       = 
                       1 
                     
                     N 
                   
                   ⁢ 
                   
                     [ 
                     
                       
                         E 
                         i 
                       
                       ⁢ 
                       
                         
                           min 
                           
                             j 
                             ≠ 
                             i 
                           
                         
                         ⁢ 
                         
                           { 
                           
                             D 
                             ij 
                             2 
                           
                           } 
                         
                       
                     
                     ] 
                   
                 
               
               
                 
                   ( 
                   1 
                   ) 
                 
               
             
           
         
       
     
     In Equation (1), E 1  indicates a magnitude of gradient of a point i, and D ij  indicates a distance between the point i and a point j. In the present embodiment, in order to maximize a score shown in Equation (1), first, the template creator  6  selects a point having the maximum magnitude of gradient as a first point. Next, the template creator  6  selects a second point which maximizes E 2 D 21   2 . Next, the template creator  6  selects a third point which maximizes the following Equation (2). Then, the template creator  6  selects a fourth point, a fifth point, . . . , and an N-th point. 
     
       
         
           
             
               
                 
                   
                     E 
                     3 
                   
                   ⁢ 
                   
                     
                       min 
                       
                         J 
                         = 
                         
                           { 
                           
                             1 
                             , 
                             2 
                           
                           } 
                         
                       
                     
                     ⁢ 
                     
                       { 
                       
                         D 
                         
                           3 
                           ⁢ 
                           j 
                         
                         2 
                       
                       } 
                     
                   
                 
               
               
                 
                   ( 
                   2 
                   ) 
                 
               
             
           
         
       
     
       FIG. 3  is a diagram illustrating a set PMn of N 2D model points calculated by using Equation (1). In  FIG. 3 , the set PMn of 2D model points is displayed to overlap a captured image of a target object OBm. In order to differentiate the captured image of the target object OBm from the 2D model set PMn, a position of the target object OBm is deviated relative to the set PMn. As illustrated in  FIG. 3 , the set PMn of 2D model points which is a set of dots calculated by using Equation (1) is distributed so as to substantially match a contour of the captured image of the target object OBm. If the set PMn of 2D model points is calculated, the template creator  6  correlates a position, or location, of the 2D model point with gradient (vector) of luminance at the position, and stores the correlation result as a contour feature at the position. 
     A-2-2. Determination of 3D Model Point and Creation of Template (Steps S 19  and S 20 ) 
     The template creator  6  calculates 3D model points corresponding to the calculated set PMn of 2D model points. The combination of the 3D model points and contour features depends on views. 
     If a 2D model point and a view V are provided, the template creator  6  computes a 3D model point P OBJ  by the following three steps. 
     1. A depth map of a 3D CAD model in the view V is drawn (rendered) on the virtual plane. 
     2. If a depth value of a 2D model point p is obtained, 3D model coordinates P CAM  represented in the camera coordinate system are computed. 
     3. Inverse 3D transformation is performed on the view V, and coordinates P OBJ  of a 3D model point in the object coordinate system (a coordinate system whose origin is fixed to the 3D model) are computed. 
     As a result of executing the above three steps, the template creator  6  creates, into a single template, a view matrix V t  for each view t expressed by the following Expression (3), 3D model points in the object coordinate system associated with respective views expressed by the following Expression (4), and descriptors of 2D features (hereinafter, also referred to as contour features) corresponding to the 3D model points in the object coordinate system and associated with the respective views, expressed by the following Expression (5).
 
 tϵ{ 1, . . . , T}   (3)
 
{ P   1   , . . . ,P   N } t   (4)
 
{ CF   1   , . . . ,CF   N } t   (5)
 
       FIGS. 4A-4C  are a schematic diagram illustrating a relationship among 3D CAD, a 2D model obtained by projecting the 3D CAD, and a 3D model created on the basis of the 2D model. As illustrated in  FIGS. 4A-4C  as an image diagram illustrating the template creation process described above, the template creator  6  renders the 2D model on the virtual plane on the basis of a view V n  of the 3D CAD as a 3D model. The template creator  6  detects edges of an image obtained through the rendering, further extracts a contour, and selects a plurality of 2D model points included in the contour on the basis of the method described with reference to Equations (1) and (2). Hereinafter, a position of a selected 2D model point and gradient (a gradient vector of luminance) at the position of the 2D model point are represented by a contour feature CF. The template creator  6  performs inverse transformation on a 2D model point p i  represented by a contour feature CF i  in the two dimensional space so as to obtain a 3D model point P i  in the three dimensional space corresponding to the contour feature CF i . Here, the 3D model point P i  is represented in the object coordinate system. The template in the view V n  includes elements expressed by the following Expression (6).
 
( CF   1n   ,CF   2n , . . . ,3 DP   1n ,3 DP   2n   , . . . ,V   n )  (6)
 
     In Expression (6), a contour feature and a 3D model point (for example, CF 1n  and 3DP 1n ) with the same suffix are correlated with each other. A 3D model point which is not detected in the view V n  may be detected in a view V m  or the like which is different from the view V n . 
     In the present embodiment, if a 2D model point p is provided, the template creator  6  treats the coordinates of the 2D model point p as integers representing a corner of a pixel. Therefore, a depth value of the 2D model point p corresponds to coordinates of (p+0.5). As a result, the template creator  6  uses the coordinates of (p+0.5) for inversely projecting the 2D point p. When a recovered 3D model point is projected, the template creator  6  truncates floating-point coordinates so as to obtain integer coordinates. 
     A-2-3. In-Plane Rotation Optimization for Training 
     If a single view is provided, substantially the same features can be visually recognized from the single view, and thus the template creator  6  creates a plurality of templates by performing in-plane rotation on the single view. The template creator  6  can create a plurality of templates with less processing by creating the templates having undergone the in-plane rotation. Specifically, the template creator  6  defines 3D points and CF descriptors for in-plane rotation of 0 degrees in the view t according to the following Expressions (7) and (8), respectively, on the basis of Expressions (4) and (5).
 
{ P   1   , . . . ,P   N } t,0   (7)
 
{ CF   1   , . . . ,CF   N } t,0   (8)
 
     The template creator  6  computes 3D model points and contour feature descriptors with respect to a template at in-plane rotation of α degrees by using Expressions (7) and (8). The visibility does not change regardless of in-plane rotation, and the 3D model points in Expression (7) are represented in the object coordinate system. From this fact, the 3D model points at in-plane rotation of α degrees are obtained by only copying point coordinates of the 3D model points at in-plane rotation of 0 degrees, and are thus expressed as in the following Equation (9).
 
{ P   1   , . . . ,P   N } t,α   ={P   1   , . . . ,P   N } t,0   (9)
 
     The contour features at in-plane rotation of α degrees are stored in the 2D coordinate system, and thus rotating the contour features at in-plane rotation of 0 degrees by α degrees is sufficient. This rotation is performed by applying a rotation matrix of 2×2 to each vector CF i , and is expressed as in the following Equation (10). 
     
       
         
           
             
               
                 
                   
                     CF 
                     i 
                     
                       t 
                       , 
                       α 
                     
                   
                   = 
                   
                     
                       [ 
                       
                         
                           
                             
                               cos 
                               ⁢ 
                               
                                   
                               
                               ⁢ 
                               α 
                             
                           
                           
                             
                               sin 
                               ⁢ 
                               
                                   
                               
                               ⁢ 
                               α 
                             
                           
                         
                         
                           
                             
                               
                                 - 
                                 sin 
                               
                               ⁢ 
                               
                                   
                               
                               ⁢ 
                               α 
                             
                           
                           
                             
                               cos 
                               ⁢ 
                               
                                   
                               
                               ⁢ 
                               α 
                             
                           
                         
                       
                       ] 
                     
                     ⁢ 
                     
                       CF 
                       i 
                       
                         t 
                         , 
                         0 
                       
                     
                   
                 
               
               
                 
                   ( 
                   10 
                   ) 
                 
               
             
           
         
       
     
     The rotation in Equation (10) is clockwise rotation, and corresponds to the present view sampling method for training. The view t corresponds to a specific viewpoint in the SUMMARY. The set PMn of 2D model points corresponds to positions of a plurality of feature points in the two dimensions, and the 3D model points correspond to the positions of a plurality of feature points in the three dimensions, represented in the object coordinate system. 
     A-2-4. Super-Template 
     The template creator  6  selects K (for example, four) templates in different views t, and merges the selected K templates into a single super-template. The template creator  6  selects templates whose views t are closest to each other as the K templates. Thus, there is a high probability that the super-template may include all edges of a target object which can be visually recognized on an object. Consequently, in a case where a detected pose of the target object is optimized, there is a high probability of convergence on an accurate pose. 
     As described above, in the personal computer PC of the present embodiment, the template creator  6  detects a plurality of edges in the two dimensions in a case where a three-dimensional CAD model representing a target object is viewed from a specific view. The template creator  6  computes 3D model points obtained by transforming contour features of the plurality of edges. The template creator  6  creates a template in which the plurality of edges in the two dimensions, the 3D model points obtained through transformation, and the specific view are correlated with each other. Thus, in the present embodiment, due to the templates created by, for example, the personal computer PC, the pose of the imaged target object is estimated with high accuracy and/or within a short period of time, when the target object is imaged by a camera or the like and a template representing a pose closest to the pose of the target object in the captured image is selected. 
     A-3. Configuration of Head Mounted Display (HMD) 
       FIG. 5  is a diagram illustrating an exterior configuration of a head mounted display  100  (HMD  100 ) which optimizes a pose of an imaged target object by using a template. If a camera  60  which will be described later captures an image of a target object, the HMD  100  optimizes and/or estimates a position and a pose of the imaged target object by using preferably a super-template and the captured image of the target object. 
     The HMD  100  is a display device mounted on the head, and is also referred to as a head mounted display (HMD). The HMD  100  of the present embodiment is an optical transmission, or optical see-through, type head mounted display which allows a user to visually recognize a virtual image and also to directly visually recognize external scenery. In the present specification, for convenience, a virtual image which the HMD  100  allows the user to visually recognize is also referred to as a “display image”. 
     The HMD  100  includes the image display section  20  which enables a user to visually recognize a virtual image in a state of being mounted on the head of the user, and a control section  10  (a controller  10 ) which controls the image display section  20 . 
     The image display section  20  is a mounting body which is to be mounted on the head of the user, and has a spectacle shape in the present embodiment. The image display section  20  includes a right holding unit  21 , a right display driving unit  22 , a left holding unit  23 , a left display driving unit  24 , a right optical image display unit  26 , a left optical image display unit  28 , and the camera  60 . The right optical image display unit  26  and the left optical image display unit  28  are disposed so as to be located in front of the right and left eyes of the user when the user wears the image display section  20 . One end of the right optical image display unit  26  and one end of the left optical image display unit  28  are connected to each other at the position corresponding to the glabella of the user when the user wears the image display section  20 . 
     The right holding unit  21  is a member which is provided so as to extend over a position corresponding to the temporal region of the user from an end part ER which is the other end of the right optical image display unit  26  when the user wears the image display section  20 . Similarly, the left holding unit  23  is a member which is provided so as to extend over a position corresponding to the temporal region of the user from an end part EL which is the other end of the left optical image display unit  28  when the user wears the image display section  20 . The right holding unit  21  and the left holding unit  23  hold the image display section  20  on the head of the user in the same manner as temples of spectacles. 
     The right display driving unit  22  and the left display driving unit  24  are disposed on a side opposing the head of the user when the user wears the image display section  20 . Hereinafter, the right holding unit  21  and the left holding unit  23  are collectively simply referred to as “holding units”, the right display driving unit  22  and the left display driving unit  24  are collectively simply referred to as “display driving units”, and the right optical image display unit  26  and the left optical image display unit  28  are collectively simply referred to as “optical image display units”. 
     The display driving units  22  and  24  respectively include liquid crystal displays  241  and  242  (hereinafter, referred to as an “LCDs  241  and  242 ”), projection optical systems  251  and  252 , and the like (refer to  FIG. 6 ). Details of configurations of the display driving units  22  and  24  will be described later. The optical image display units  26  and  28  as optical members include light guide plates  261  and  262  (refer to  FIG. 6 ) and dimming plates. The light guide plates  261  and  262  are made of light transmissive resin material or the like and guide image light which is output from the display driving units  22  and  24  to the eyes of the user. The dimming plate is a thin plate-shaped optical element, and is disposed to cover a surface side of the image display section  20  which is an opposite side to the user&#39;s eye side. The dimming plate protects the light guide plates  261  and  262  so as to prevent the light guide plates  261  and  262  from being damaged, polluted, or the like. In addition, light transmittance of the dimming plates is adjusted so as to adjust an amount of external light entering the eyes of the user, thereby controlling an extent of visually recognizing a virtual image. The dimming plate may be omitted. 
     The camera  60  images external scenery. The camera  60  is disposed at a position where one end of the right optical image display unit  26  and one end of the left optical image display unit  28  are connected to each other. As will be described later in detail, a pose of a target object included in the external scenery is estimated by using an image of the target object included in the external scenery imaged by the camera  60  and preferably a super-template stored in a storage unit  120 . The camera  60  corresponds to an imaging section in the SUMMARY. 
     The image display section  20  further includes a connection unit  40  which connects the image display section  20  to the control section  10 . The connection unit  40  includes a main body cord  48  connected to the control section  10 , a right cord  42 , a left cord  44 , and a connection member  46 . The right cord  42  and the left cord  44  are two cords into which the main body cord  48  branches out. The right cord  42  is inserted into a casing of the right holding unit  21  from an apex AP in the extending direction of the right holding unit  21 , and is connected to the right display driving unit  22 . Similarly, the left cord  44  is inserted into a casing of the left holding unit  23  from an apex AP in the extending direction of the left holding unit  23 , and is connected to the left display driving unit  24 . The connection member  46  is provided at a branch point of the main body cord  48 , the right cord  42 , and the left cord  44 , and has a jack for connection of an earphone plug  30 . A right earphone  32  and a left earphone  34  extend from the earphone plug  30 . 
     The image display section  20  and the control section  10  transmit various signals via the connection unit  40 . An end part of the main body cord  48  on an opposite side to the connection member  46 , and the control section  10  are respectively provided with connectors (not illustrated) fitted to each other. The connector of the main body cord  48  and the connector of the control section  10  are fitted into or released from each other, and thus the control section  10  is connected to or disconnected from the image display section  20 . For example, a metal cable or an optical fiber may be used as the right cord  42 , the left cord  44 , and the main body cord  48 . 
     The control section  10  is a device used to control the HMD  100 . The control section  10  includes a determination key  11 , a lighting unit  12 , a display changing key  13 , a track pad  14 , a luminance changing key  15 , a direction key  16 , a menu key  17 , and a power switch  18 . The determination key  11  detects a pushing operation, so as to output a signal for determining content operated in the control section  10 . The lighting unit  12  indicates an operation state of the HMD  100  by using a light emitting state thereof. The operation state of the HMD  100  includes, for example, ON and OFF of power, or the like. For example, an LED is used as the lighting unit  12 . The display changing key  13  detects a pushing operation so as to output a signal for changing a content moving image display mode between 3D and 2D. The track pad  14  detects an operation of the finger of the user on an operation surface of the track pad  14  so as to output a signal based on detected content. Various track pads of a capacitance type, a pressure detection type, and an optical type may be employed as the track pad  14 . The luminance changing key  15  detects a pushing operation so as to output a signal for increasing or decreasing a luminance of the image display section  20 . The direction key  16  detects a pushing operation on keys corresponding to vertical and horizontal directions so as to output a signal based on detected content. The power switch  18  detects a sliding operation of the switch so as to change a power supply state of the HMD  100 . 
       FIG. 6  is a functional block diagram illustrating a configuration of the HMD  100  of the present embodiment. As illustrated in  FIG. 6 , the control section  10  includes the storage unit  120 , a power supply  130 , an operation unit  135 , a CPU  140 , an interface  180 , a transmission unit  51  (Tx  51 ), and a transmission unit  52  (Tx  52 ). The operation unit  135  is constituted of the determination key  11 , the display changing key  13 , the track pad  14 , the luminance changing key  15 , the direction key  16 , and the menu key  17 , and the power switch  18 , which receive operations from the user. The power supply  130  supplies power to the respective units of the HMD  100 . For example, a secondary battery may be used as the power supply  130 . 
     The storage unit  120  includes a ROM storing a computer program, a RAM which is used for the CPU  140  to perform writing and reading of various computer programs, and a template storage portion  121 . The template storage portion  121  stores a super-template created by the template creator  6  of the personal computer PC. The template storage portion  121  acquires the super-template via a USB memory connected to the interface  180 . The template storage portion  121  corresponds to a template acquisition section in the appended claims. 
     The CPU  140  reads the computer programs stored in the ROM of the storage unit  120 , and writes and reads the computer programs to and from the RAM of the storage unit  120 , so as to function as an operating system.  150  (OS  150 ), a display control unit  190 , a sound processing unit  170 , an image processing unit  160 , an image setting unit  165 , a location-correspondence determination unit  168 , and an optimization unit  166 . 
     The display control unit  190  generates control signals for control of the right display driving unit  22  and the left display driving unit  24 . Specifically, the display control unit  190  individually controls the right LCD control portion  211  to turn on and off driving of the right LCD  241 , controls the right backlight control portion  201  to turn on and off driving of the right backlight  221 , controls the left LCD control portion  212  to turn on and off driving of the left LCD  242 , and controls the left backlight control portion  202  to turn on and off driving of the left backlight  222 , by using the control signals. Consequently, the display control unit  190  controls each of the right display driving unit  22  and the left display driving unit  24  to generate and emit image light. For example, the display control unit  190  causes both of the right display driving unit  22  and the left display driving unit  24  to generate image light, causes either of the two units to generate image light, or causes neither of the two units to generate image light. Generating image light is also referred to as “displaying an image”. 
     The display control unit  190  transmits the control signals for the right LCD control portion  211  and the left LCD control portion  212  thereto via the transmission units  51  and  52 . The display control unit  190  transmits control signals for the right backlight control portion  201  and the left backlight control portion  202  thereto. 
     The image processing unit  160  acquires an image signal included in content. The image processing unit  160  separates synchronization signals such as a vertical synchronization signal VSync and a horizontal synchronization signal HSync from the acquired image signal. The image processing unit  160  generates a clock signal PCLK by using a phase locked loop (PLL) circuit or the like (not illustrated) on the basis of a cycle of the separated vertical synchronization signal VSync or horizontal synchronization signal HSync. The image processing unit  160  converts an analog image signal from which the synchronization signals are separated into a digital image signal by using an A/D conversion circuit or the like (not illustrated). Next, the image processing unit  160  stores the converted digital image signal in a DRAM of the storage unit  120  for each frame as image dat (RGB data) of a target image. The image processing unit  160  may perform, on the image data, image processes including a resolution conversion process, various color tone correction processes such as adjustment of luminance and color saturation, a keystone correction process, and the like, as necessary. 
     The image processing unit  160  transmits each of the generated clock signal PCLK, vertical synchronization signal VSync and horizontal synchronization signal HSync, and the image data stored in the DRAM of the storage unit  120 , via the transmission units  51  and  52 . Here, the image data which is transmitted via the transmission unit  51  is referred to as “right eye image data”, and the image data Data which is transmitted via the transmission unit  52  is referred to as “left eye image data”. The transmission units  51  and  52  function as a transceiver for serial transmission between the control section  10  and the image display section  20 . 
     The sound processing unit  170  acquires an audio signal included in the content so as to amplify the acquired audio signal, and supplies the amplified audio signal to a speaker (not illustrated) of the right earphone  32  connected to the connection member  46  and a speaker (not illustrated) of the left earphone  34  connected thereto. In addition, for example, in a case where a Dolby (registered trademark) system is employed, the audio signal is processed, and thus different sounds of which frequencies are changed are respectively output from the right earphone  32  and the left earphone  34 . 
     In a case where an image of external scenery including a target object is captured by the camera  60 , the location-correspondence determination unit  168  detects edges of the target object in the captured image. Then, the location-correspondence determination unit  168  determines correspondences between the edges (edge feature elements) of the target object and the contour feature elements of the 2D model stored in the template storage portion  121 . In the present embodiment, a plurality of templates are created and stored in advance with a specific target object (for example, a specific part) as a preset target object. Therefore, if a preset target object is included in a captured image, the location-correspondence determination unit  168  determines correspondences between 2D locations of edges of the target object and 2D locations of 2D model points of the target object included in a template selected among from a plurality of the templates in different views. A specific process of determining or establishing the correspondences between the edge feature elements of the target object in the captured image and the contour feature elements of the 2D model in the template will be described later. 
     The optimization unit  166  outputs 3D model points, which include respective 3D locations, corresponding to 2D model points having the correspondences to the image points from the template of the target object, and minimizes a cost function in Equation (14) on the basis of the image points, the 3D model points, and the view represented by at least one transformation matrix, so as to estimate a location and a pose in the three dimensions of the target object included in the external scenery imaged by the camera  60 . Estimation and/or optimization of a position and a pose of the imaged target object will be described later. 
     The image setting unit  165  performs various settings on an image (display image) displayed on the image display section  20 . For example, the image setting unit  165  sets a display position of the display image, a size of the display image, luminance of the display image, and the like, or sets right eye image data and left eye image data so that binocular parallax (hereinafter, also referred to as “parallax”) is formed in order for a user to stereoscopically (3D) visually recognize the display image as a three-dimensional image. The image setting unit  165  detects a determination target image set in advance from a captured image by applying pattern matching or the like to the captured image. 
     The image setting unit  165  displays (renders) a 3D model corresponding to the target object on the optical image display units  26  and  28  in a pose of target object which is derived and/or optimized by the optimization unit  166  in a case where the location-correspondence determination unit  168  and the optimization unit  166  are performing various processes and have performed the processes. The operation unit  135  receives an operation from the user, and the user can determine whether or not the estimated pose of the target object matches a pose of the target object included in the external scenery transmitted through the optical image display units  26  and  28 . 
     The interface  180  is an interface which connects the control section  10  to various external apparatuses OA which are content supply sources. As the external apparatuses OA, for example, a personal computer (PC), a mobile phone terminal, and a gaming terminal may be used. As the interface  180 , for example, a USB interface, a microUSB interface, and a memory card interface may be used. 
     The image display section  20  includes the right display driving unit  22 , the left display driving unit  24 , the right light guide plate  261  as the right optical image display unit  26 , the left light guide plate  262  as the left optical image display unit  28 , and the camera  60 . 
     The right display driving unit  22  includes a reception portion  53  (Rx  53 ), the right backlight control portion  201  (right BL control portion  201 ) and the right backlight  221  (right BL  221 ) functioning as a light source, the right LCD control portion  211  and the right LCD  241  functioning as a display element, and a right projection optical system  251 . As mentioned above, the right backlight control portion  201  and the right backlight  221  function as a light source. As mentioned above, the right LCD control portion  211  and the right LCD  241  function as a display element. The right backlight control portion  201 , the right LCD control portion  211 , the right backlight  221 , and the right LCD  241  are collectively referred to as an “image light generation unit”. 
     The reception portion  53  functions as a receiver for serial transmission between the control section  10  and the image display section  20 . The right backlight control portion  201  drives the right backlight  221  on the basis of an input control signal. The right backlight  221  is a light emitting body such as an LED or an electroluminescent element (EL). The right LCD control portion  211  drives the right LCD  241  on the basis of the clock signal PCLK, the vertical synchronization signal VSync, the horizontal synchronization signal HSync, and the right eye image data which are input via the reception portion  53 . The right LCD  241  is a transmissive liquid crystal panel in which a plurality of pixels are disposed in a matrix. 
     The right projection optical system  251  is constituted of a collimator lens which converts image light emitted from the right LCD  241  into parallel beams of light flux. The right light guide plate  261  as the right optical image display unit  26  reflects image light output from the right projection optical system  251  along a predetermined light path, so as to guide the image light to the right eye RE of the user. The right projection optical system  251  and the right light guide plate  261  are collectively referred to as a “light guide portion”. 
     The left display driving unit  24  has the same configuration as that of the right display driving unit  22 . The left display driving unit  24  includes a reception portion  54  (Rx  54 ), the left backlight control portion  202  (left BL control portion  202 ) and the left backlight  222  (left BL  222 ) functioning as a light source, the left LCD control portion  212  and the left LCD  242  functioning as a display element, and a left projection optical system  252 . As mentioned above, the left backlight control portion  202  and the left backlight  222  function as a light source. As mentioned above, the left LCD control portion  212  and the left LCD  242  function as a display element. In addition, the left backlight control portion  202 , the left LCD control portion  212 , the left backlight  222 , and the left LCD  242  are collectively referred to as an “image light generation unit”. The left projection optical system  252  is constituted of a collimator lens which converts image light emitted from the left LCD  242  into parallel beams of light flux. The left light guide plate  262  as the left optical image display unit  28  reflects image light output from the left projection optical system  252  along a predetermined light path, so as to guide the image light to the left eye LE of the user. The left projection optical system  252  and the left light guide plate  262  are collectively referred to as a “light guide portion”. 
     A-4. Execution (Run-Time) of Estimation of Target Object Pose 
       FIG. 7  is a flowchart illustrating a target object pose estimation process. In the pose estimation process, first, the location-correspondence determination unit  168  images external scenery including a target object with the camera  60  (step S 21 ). The location-correspondence determination unit  168  performs edge detection described below on a captured image of the target object (step S 23 ). 
     A-4-1. Edge Detection (Step S 23 ) 
     The location-correspondence determination unit  168  detects an edge of the image of the target object in order to correlate the imaged target object with a template corresponding to the target object. The location-correspondence determination unit  168  computes features serving as the edge on the basis of pixels of the captured image. In the present embodiment, the location-correspondence determination unit  168  computes gradient of luminance of the pixels of the captured image of the target object so as to determine the features. When the edge is detected from the captured image, objects other than the target object in the external scenery, different shadows, different illumination, and different materials of objects included in the external scenery may influence the detected edge. Thus, it may be relatively difficult to detect the edge from the captured image may than to detect an edge from a 3D CAD model. In the present embodiment, in order to more easily detect an edge, the location-correspondence determination unit  168  only compares an edge with a threshold value and suppresses non-maxima, in the same manner as in procedures performed in a simple edge detection method. 
     A-4-2. Selection of Template (Step S 25 ) 
     If the edge is detected from the image of the target object, the location-correspondence determination unit  168  selects a template having a view closest to the pose of the target object in a captured image thereof from among templates stored in the template storage portion  121  (step S 25 ). For this selection, an existing three-dimensional pose estimation algorithm for estimating a rough pose of a target object may be used separately. The location-correspondence determination unit  168  may find a new training view closer to the pose of the target object in the image than the selected training view when highly accurately deriving a 3D pose. In a case of finding a new training view, the location-correspondence determination unit  168  highly accurately derives a 3D pose in the new training view. In the present embodiment, if views are different from each other, contour features as a set of visually recognizable edges including the 2D outline of the 3D model are also different from each other, and thus a new training view may be found. The location-correspondence determination unit  168  uses a super-template for a problem that sets of visually recognizable edges are different from each other, and thus extracts as many visually recognizable edges as possible. In another embodiment, instead of using a template created in advance, the location-correspondence determination unit  168  may image a target object, and may create a template by using 3D CAD data while reflecting an imaging environment such as illumination in rendering on the fly and as necessary, so as to extract as many visually recognizable edges as possible. 
     A-4-3. 2D Point Correspondences (Step S 27 ) 
     If the process in step S 25  is completed, the location-correspondence determination unit  168  correlates the edge of the image of the target object with 2D model points included in the template (step S 27 ). 
       FIG. 8  is a diagram illustrating that a single 2D model point is combined with a plurality of image points included in a certain edge.  FIG. 9  is a diagram illustrating an example in which a 2D model point is combined with wrong image points.  FIGS. 8 and 9  illustrate a captured image IMG of the target object OBm, a partial enlarged view of the 2D model point set PMn, and a plurality of arrows CS in a case where the target object OBm corresponding to the 3D model illustrated in  FIG. 3  is imaged by the camera  60 . As illustrated in  FIG. 8 , a portion of an edge detected from the image IMG of the target object OBm which is correlated with a 2D model point PM 1  which is one of the 2D model points included in a template includes a plurality of options as in the arrows CS 1  to CS 5 .  FIG. 9  illustrates an example in which 2D model points PM 1  to PM 5  included in the template and arranged are wrongly combined with an edge (image points included therein) detected from the image IMG of the target object OBm. In this case, for example, in  FIG. 9 , despite the 2D model points PM 2 , PM 3 , PM 1 , PM 4  and PM 5  being arranged from the top, the arrows CS 7 , CS 6 , CS 8 , CS 10  and CS 9  are arranged in this order in the edge of the image IMG of the target object OBm. Thus, the arrow CS 8  and the arrow CS 6 , and the arrow CS 9  and the arrow CS 10  are changed. As described above, the location-correspondence determination unit  168  is required to accurately correlate 2D model points included in a template with image points included in an edge of the image IMG of the target object OBm to accurately estimate or derive a pose of the imaged target object OBm. 
     In the present embodiment, the location-correspondence determination unit  168  computes similarity scores by using the following Equation (11) with respect to all image points included in a local vicinity of each projected 2D model point. 
     
       
         
           
             
               
                 
                   
                     SIM 
                     ⁡ 
                     
                       ( 
                       
                         p 
                         , 
                         
                           p 
                           ′ 
                         
                       
                       ) 
                     
                   
                   = 
                   
                     
                        
                       
                         
                           → 
                           
                             E 
                             P 
                           
                         
                         ⁢ 
                         
                           · 
                           
                             
                               → 
                               ∇ 
                             
                             ⁢ 
                             
                               I 
                               
                                 p 
                                 ′ 
                               
                             
                           
                         
                       
                        
                     
                     / 
                     
                       
                         max 
                         
                           q 
                           ∈ 
                           
                             N 
                             ⁡ 
                             
                               ( 
                               p 
                               ) 
                             
                           
                         
                       
                       ⁢ 
                       
                          
                         
                           
                             → 
                             ∇ 
                           
                           ⁢ 
                           
                             I 
                             p 
                           
                         
                          
                       
                     
                   
                 
               
               
                 
                   ( 
                   11 
                   ) 
                 
               
             
           
         
       
     
     The measure of similarity scores indicated in Equation (11) is based on matching between a gradient vector (hereinafter, simply referred to as gradient) of luminance of a 2D model point included in a template and a gradient vector of an image point, but is based on an inner product of the two vectors in Equation (11) as an example. The vector of Ep in Equation (11) is a unit length gradient vector of a 2D model point (edge point) p. The location-correspondence determination unit  168  uses gradient ∇I of a test image (input image) in order to compute features of an image point p′ when obtaining the similarity scores. The normalization by the local maximum of the gradient magnitude in the denominator in Expression (11) ensures that the priority is reliably given to an edge with a locally high intensity. This normalization prevents an edge which is weak and thus becomes noise from being collated. The location-correspondence determination unit  168  enhances a size N(p) of a nearest neighborhood region in which a correspondence is searched for when the similarity scores are obtained. For example, in a case where an average of position displacement of a projected 2D model point is reduced in consecutive iterative computations, N(p) may be reduced. Hereinafter, a specific method for establishing correspondences using Equation (11) will be described. 
       FIGS. 10 to 12  are diagrams illustrating an example of computation of similarity scores.  FIG. 10  illustrates an image IMG OB  (solid line) of a target object captured by the camera  60 , a 2D model MD (dot chain line) based on a template similar to the image IMG OB  of the target object, and 2D model points as a plurality of contour features CFm in the 2D model MD.  FIG. 10  illustrates a plurality of pixels px arranged in a lattice form, and a region (for example, a region SA 1 ) formed of 3 pixels×3 pixels centering on each of the contour features CFm.  FIG. 10  illustrates the region SA 1  centering on the contour feature CF 1  which will be described later, a region SA 2  centering on a contour feature CF 2 , and a region SA 3  centering on a contour feature CF 3 . The contour feature CF 1  and the contour feature CF 2  are adjacent to each other, and the contour feature CF 1  and the contour feature CF 3  are also adjacent to each other. In other words, the contour features are arranged in order of the contour feature CF 2 , the contour feature CF 1 , and the contour feature CF 3  in  FIG. 10 . 
     As illustrated in  FIG. 10 , since the image IMG OB  of the target object does not match the 2D model MD, the location-correspondence determination unit  168  correlates image points included in an edge of the image IMG OB  of the target object with 2D model points represented by the plurality of contour features CFm of the 2D model MD, respectively, by using Equation (11). First, the location-correspondence determination unit  168  selects the contour feature CF 1  as one of the plurality of contour features CFm, and extracts the region SA 1  of 3 pixels×3 pixels centering on a pixel px including the contour feature CF 1 . Next, the location-correspondence determination unit  168  extracts the region SA 2  and the region SA 3  of 3 pixels×3 pixels respectively centering on the two contour features such as the contour feature CF 2  and the contour feature CF 3  which are adjacent to the contour feature CF 1 . The location-correspondence determination unit  168  calculates a score by using Equation (11) for each pixel px forming each of the regions SA 1 , SA 2  and SA 3 . In this stage, the regions SA 1 , SA 2  and SA 3  are matrices having the same shape and the same size. 
       FIG. 11  illustrates enlarged views of the respective regions SA 1 , SA 2  and SA 3 , and similarity scores calculated for the respective pixels forming the regions SA 1 , SA 2  and SA 3 . The location-correspondence determination unit  168  calculates similarity scores between the 2D model point as the contour feature and the nine image points. For example, in the region SA 3  illustrated on the lower part of  FIG. 11 , the location-correspondence determination unit  168  calculates, as scores, 0.8 for pixels px 33  and px 36 , 0.5 for a pixel px 39 , and 0 for the remaining six pixels. The reason why the score of 0.8 for the pixels px 33  and px 36  is different from the score of 0.5 for the pixel px 39  is that the image IMG OB  of the target object in the pixel px 39  is bent and thus gradient differs. As described above, the location-correspondence determination unit  168  calculates similarity scores of each pixel (image point) forming the extracted regions SA 1 , SA 2  and SA 3  in the same manner. 
     Hereinafter, a description will be made focusing on the contour feature CF 1 . The location-correspondence determination unit  168  calculates a corrected score of each pixel forming the region SA 1 . Specifically, the similarity scores are averaged with weighting factors by using pixels located at the same matrix positions of the regions SA 2  and SA 3  as the respective pixels forming the region SA 1 . The location-correspondence determination unit  168  performs this correction of the similarity scores not only on the contour feature CF 1  but also on the other contour features CF 2  and CF 3 . In the above-described way, it is possible to achieve an effect in which a correspondence between a 2D model point and an image point is smoothed. In the example illustrated in  FIG. 11 , the location-correspondence determination unit  168  calculates corrected scores by setting a weighting factor of a score of each pixel px of the region SA 1  to 0.5, setting a weighting factor of a score of each pixel px of the region SA 2  to 0.2, and setting a weighting factor of a score of each pixel px of the region SA 3  to 0.3. For example, 0.55 as a corrected score of the pixel px 19  illustrated in  FIG. 12  is a value obtained by adding together three values such as a value obtained by multiplying the score of 0.8 for the pixel px 19  of the region SA 1  by the weighting factor of 0.5, a value obtained by multiplying the score of 0 for the pixel px 29  of the region SA 2  by the weighting factor of 0.2, and a value obtained by multiplying the score of 0.5 for the pixel px 39  of the region SA 3  by the weighting factor of 0.3. The weighting factors are inversely proportional to distances between the processing target contour feature CF 1  and the other contour features CF 2  and CF 3 . The location-correspondence determination unit  168  determines an image point having the maximum score among the corrected scores of the pixels forming the region SA 1 , as an image point correlated with the contour feature CF 1 . In the example illustrated in  FIG. 12 , the maximum value of the corrected scores is 0.64 of the pixels px 13  and px 16 . In a case where a plurality of pixels have the same corrected score, the location-correspondence determination unit  168  selects the pixel px 16  whose distance from the contour feature CF 1  is shortest, and the location-correspondence determination unit  168  correlates the contour feature CF 1  with an image point of the pixel px 16 . The location-correspondence determination unit  168  compares edges detected in a plurality of images of the target object captured by the camera  60  with 2D model points in a template in a view close to the images of the target object, so as to determine image points of the target object corresponding to the 2D model points (contour features CF). 
     If the location-correspondence determination unit  168  completes the process in step S 27  in  FIG. 7 , the optimization unit  166  acquires 3D model points corresponding to the 2D model points correlated with the image points and information regarding the view which is used for creating the 2D model points, from the template of the target object stored in the template storage portion  121  (step S 29 ). The optimization unit  166  derives a pose of the target object imaged by the camera  60  on the basis of the extracted 3D model points and information regarding the view, and the image points (step S 33 ). Details of the derivation are as follows. 
     A-4-4. Optimization of Pose (Step S 33 ) 
     In the present embodiment, the optimization unit  166  highly accurately derives or refines a 3D pose of the target object by using contour features included in a template corresponding to a selected training view, and 3D model points corresponding to 2D model points included in the contour features. In the derivation, the optimization unit  166  derives a pose of the target object by performing optimization computation for minimizing Equation (14). 
     If the location-correspondence determination unit  168  completes establishing the correspondences between 2D model points and the image points in a predetermined view, the location-correspondence determination unit  168  reads 3D model points P i  corresponding to the 2D model points (or the contour features CF i ) from a template corresponding to the view. In the present embodiment, as described above, the 3D model points P i  corresponding to the 2D model points are stored in the template. However, the 3D model points P i  are not necessarily stored in the template, and the location-correspondence determination unit  168  may inversely convert the 2D model points whose correspondences to the image points is completed, every time on the basis of the view, so as to obtain the 3D model points P i . 
     The optimization unit  166  reprojects locations of the obtained 3D model points P i  onto a 2D virtual plane on the basis of Equation (12).
 
π( P   i )=( u   i   ,v   i ) T   (12)
 
     Here, π in Equation (12) includes a rigid body transformation matrix and a perspective projecting transformation matrix included in the view. In the present embodiment, three parameters indicating three rotations about three axes included in the rigid body transformation matrix and three parameters indicating three translations along the three axes are treated as variables for minimizing Equation (14). The rotation may be represented by a quaternion. The image points p i  corresponding to the 3D model points P i  are expressed as in Equation (13).
 
 p   i =( p   ix   ,p   iy ) T   (13)
 
     The optimization unit  166  derives a 3D pose by using the cost function expressed by the following Equation (14) in order to minimize errors between the 3D model points P i  and the image points p i . 
     
       
         
           
             
               
                 
                   
                     E 
                     match 
                   
                   = 
                   
                     
                       
                         ∑ 
                         
                           i 
                           = 
                           1 
                         
                         N 
                       
                       ⁢ 
                       
                         
                           w 
                           i 
                         
                         * 
                         
                            
                           
                             
                               π 
                               ⁡ 
                               
                                 ( 
                                 
                                   P 
                                   i 
                                 
                                 ) 
                               
                             
                             - 
                             
                               p 
                               i 
                             
                           
                            
                         
                       
                     
                     = 
                     
                       
                         ∑ 
                         
                           i 
                           = 
                           1 
                         
                         N 
                       
                       ⁢ 
                       
                         
                           w 
                           i 
                         
                         * 
                         
                           ( 
                           
                             
                               
                                 ( 
                                 
                                   
                                     u 
                                     i 
                                   
                                   - 
                                   
                                     p 
                                     ix 
                                   
                                 
                                 ) 
                               
                               2 
                             
                             + 
                             
                               
                                 ( 
                                 
                                   
                                     v 
                                     i 
                                   
                                   - 
                                   
                                     p 
                                     iy 
                                   
                                 
                                 ) 
                               
                               2 
                             
                           
                           ) 
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   14 
                   ) 
                 
               
             
           
         
       
     
     Here, w i  in Equation (14) is a weighting factor for controlling the contribution of each model point to the cost function. A point which is projected onto the outside of an image boundary or a point having low reliability of the correspondence is given a weighting factor of a small value. In the present embodiment, in order to present specific adjustment of a 3D pose, the optimization unit  166  determines minimization of the cost function expressed by Equation (14) as a function of 3D pose parameters using the Gauss-Newton method, if one of the following three items is reached: 
     1. An initial 3D pose diverges much more than a preset pose. In this case, it is determined that minimization of the cost function fails. 
     2. The number of times of approximation using the Gauss-Newton method exceeds a defined number of times set in advance. 
     3. A relative pose change in the Gauss-Newton method is equal to or less than a preset threshold value. In this case, it is determined that the cost function is minimized. 
     When a 3D pose is derived, the optimization unit  166  may attenuate refinement of a pose of the target object. Time required to process estimation of a pose of the target object directly depends on the number of iterative computations which are performed so as to achieve high accuracy (refinement) of the pose. From a viewpoint of enhancing the system speed, it may be beneficial to employ an approach that derives a pose through as small a number of iterative computations as possible without compromising the accuracy of the pose. According to the present embodiment, each iterative computation is performed independently from its previous iterative computation, and thus no constraint is imposed, the constraint ensuring that the correspondences of 2D model points are kept consistent, or that the same 2D model points are correlated with the same image structure or image points between two consecutive iterative computations. As a result, particularly, in a case where there is a noise edge structure caused by a messy state in which other objects which are different from a target object are mixed in an image captured by the camera  60  or a state in which shadows are present, correspondences of points are unstable. As a result, more iterative computations may be required for convergence. According to the method of the present embodiment, this problem can be handled by multiplying the similarity scores in Equation (11) by an attenuation weighting factor shown in the following Equation (15). 
     
       
         
           
             
               
                 
                   
                     w 
                     ( 
                     
                       → 
                       
                         Δ 
                         ⁢ 
                         
                             
                         
                         ⁢ 
                         p 
                       
                     
                     ) 
                   
                   = 
                   
                     e 
                     
                       
                         - 
                         
                           ( 
                           
                             → 
                             
                               
                                  
                                 
                                   Δ 
                                   ⁢ 
                                   
                                       
                                   
                                   ⁢ 
                                   p 
                                 
                                  
                               
                               2 
                             
                           
                           ) 
                         
                       
                       / 
                       
                         σ 
                         2 
                       
                     
                   
                 
               
               
                 
                   ( 
                   15 
                   ) 
                 
               
             
           
         
       
     
     Equation (15) expresses a Gaussian function, and σ has a function of controlling the strength (effect) of attenuation. In a case where a value of σ is great, attenuation does not greatly occur, but in a case where a value of σ is small, strong attenuation occurs, and thus it is possible to prevent a point from becoming distant from the present location. In order to ensure consistency in correspondences of points in different iterative computations, in the present embodiment, σ is a function of a reprojecting error obtained through the latest several iterative computations. In a case where a reprojecting error (which may be expressed by Equation (14)) is considerable, in the method of the present embodiment, convergence does not occur. In an algorithm according to the present embodiment, σ is set to a great value, and thus a correspondence with a distant point is ensured so that attenuation is not almost or greatly performed. In a case where a reprojecting error is slight, there is a high probability that a computation state using the algorithm according to the present embodiment may lead to an accurate solution. Therefore, the optimization unit  166  sets σ to a small value so as to increase attenuation, thereby stabilizing the correspondences of points. 
     A-4-5. Subpixel Correspondences 
     The correspondences of points of the present embodiment takes into consideration only an image point at a pixel location of an integer, and thus there is a probability that accuracy of a 3D pose may be deteriorated. A method according to the present embodiment includes two techniques in order to cope with this problem. First, an image point p′ whose similarity score is the maximum is found, and then the accuracy at this location is increased through interpolation. A final location is represented by a weighted linear combination of four connected adjacent image points p′. The weight here is a similarity score. Second, the method according to the present embodiment uses two threshold values for a reprojecting error in order to make a pose converge with high accuracy. In a case where great threshold values are achieved, a pose converges with high accuracy, and thus a slightly highly accurate solution has only to be obtained. Therefore, the length of vectors for the correspondences of points is artificially reduced to ½ through respective iterative computations after the threshold values are achieved. In this process, subsequent several computations are iteratively performed until the reprojecting error is less than a smaller second threshold value. 
     As a final step of deriving a pose with high accuracy, the location-correspondence determination unit  168  computes matching scores which is to be used to remove a wrong result. These scores have the same form as that of the cost function in Equation (14), and are expressed by the following Equation (16). 
     
       
         
           
             
               
                 
                   
                     S 
                     match 
                   
                   = 
                   
                     
                       1 
                       N 
                     
                     ⁢ 
                     
                       
                         ∑ 
                         
                           i 
                           = 
                           1 
                         
                         N 
                       
                       ⁢ 
                       
                         
                           SIM 
                           i 
                         
                         · 
                         
                           e 
                           
                             - 
                             
                                
                               
                                 
                                   π 
                                   ( 
                                   
                                     
                                       P 
                                       
                                         
                                           i 
                                           ) 
                                         
                                         - 
                                       
                                     
                                     ⁢ 
                                     
                                       p 
                                       i 
                                     
                                   
                                    
                                 
                                 / 
                                 
                                   σ 
                                   2 
                                 
                               
                             
                           
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   16 
                   ) 
                 
               
             
           
         
       
     
     In Equation (16), SIM i  indicates a similarity score between a contour feature i (a 2D model point) and an image point which most match the contour feature. The exponential part is a norm (the square of a distance between two points in the present embodiment) between the 2D model point reprojected by using the pose and the image point corresponding thereto, and N indicates the number of sets of the 2D model points and the image points. The optimization unit  166  may continuously perform optimization in a case where a value of Equation (16) is smaller than a threshold value without employing the pose, and employs the pose in a case where the value of Equation (16) is equal to or greater than the threshold value. As described above, if the optimization unit  166  completes the process in step S 33  in  FIG. 7 , the location-correspondence determination unit  168  and the optimization unit  166  finishes the pose estimation process. 
     As described above, in the HMD  100  of the present embodiment, the location-correspondence determination unit  168  detects an edge from an image of a target object captured by the camera  60 . The location-correspondence determination unit  168  establishes the correspondences between the image points included in an image and the 2D model points included in a template stored in the template storage portion  121 . The optimization unit  166  estimates or derives a pose of the imaged target object by using the 2D model points and 3D points obtained by converting the 2D model points included in the template. Specifically, the optimization unit  166  optimizes a pose of the imaged target object by using the cost function. Thus, in the HMD  100  of the present embodiment, if an edge representing a contour of the target object imaged by the camera  60  can be detected, a pose of the imaged target object can be estimated with high accuracy. Since the pose of the target object is estimated with high accuracy, the accuracy of overlapping display of an AR image on the target object is improved, and the accuracy of an operation performed by a robot is improved. 
     B. Second Embodiment 
     A second embodiment is the same as the first embodiment except for a computation method of similarity scores in establishing the correspondences of 2D points performed by the location-correspondence determination unit  168  of the HMD  100 . Therefore, in the second embodiment, computation of similarity scores, which is different from the first embodiment, will be described, and description of other processes will be omitted. 
       FIGS. 13 to 15  are diagrams illustrating an example of computation of CF similarity in the second embodiment.  FIG. 13  further illustrates perpendicular lines VLm which are perpendicular to a contour of a 2D model MD at respective contour features CFm compared with  FIG. 10 . For example, the perpendicular line VL 1  illustrated in  FIG. 13  is perpendicular to the contour of the 2D model MD at the contour feature CF 1 . The perpendicular line VL 2  is perpendicular to the contour of the 2D model MD at the contour feature CF 2 . The perpendicular line VL 3  is perpendicular to the contour of the 2D model MD at the contour feature CF 3 . 
     In the same manner as in the first embodiment, the location-correspondence determination unit  168  selects the contour feature CF 1  as one of the plurality of contour features CFm, and extracts the region SA 1  of 3 pixels×3 pixels centering on a pixel px including the contour feature CF 1 . Next, the location-correspondence determination unit  168  extracts the region SA 2  and the region SA 3  of 3 pixels×3 pixels respectively centering on the two contour features such as the contour feature CF 2  and the contour feature CF 3  which are adjacent to the contour feature CF 1 . The location-correspondence determination unit  168  allocates a score to each pixel px forming each of the regions SA 1 , SA 2  and SA 3 . In the second embodiment, as described above, a method of the location-correspondence determination unit  168  allocating scores to the regions SA 1 , SA 2  and SA 3  is different from the first embodiment. 
     Hereinafter, a description will be made focusing on a region SA 1 . The location-correspondence determination unit  168  assumes the perpendicular line VL 1  which is perpendicular to a model contour at a 2D model point through the 2D model point represented by the contour feature CF 1  in the region SA. The location-correspondence determination unit  168  sets a score of each pixel px (each image point) for the contour feature CF 1  by using a plurality of Gaussian functions each of which has the center on the perpendicular line VL 1  and which are distributed in a direction (also referred to as a main axis) perpendicular to the line segment VL 1 . Coordinates the pixel px are represented by integers (m,n), but, in the present embodiment, the center of the pixel px overlapping the perpendicular line VLm is represented by (m+0.5, n+0.5), and a second perpendicular line drawn from the center thereof to the perpendicular line VLm is used as the main axis. Similarity scores of a pixel px overlapping the perpendicular line VL 1  and a pixel px overlapping the main axis are computed as follows. First, with respect to the pixel px on the perpendicular line VL 1 , a value of the central portion of a Gaussian function obtained as a result of being multiplied by a weighting factor which is proportional to a similarity score of the pixel px is used as a new similarity score. Here, the variance of the Gaussian function is selected so as to be proportional to a distance from the contour feature CF 1 . On the other hand, with respect to the pixel px on the main axis of each Gaussian function, a value of each Gaussian function having a distance from an intersection (the center) between the perpendicular line VL 1  and the main axis as a variable, is used as a new similarity score. As a result, for example, the location-correspondence determination unit  168  allocates respective scores of 0.2, 0.7, and 0.3 to the pixels px 13 , px 16  and pixel  19  included in an image IMG OB  of the target object although the pixels have almost the same gradient, as illustrated in  FIG. 14 . This is because distances from the perpendicular line VL 1  to the respective pixels px are different from each other. 
     Next, the location-correspondence determination unit  168  locally smooths the similarity scores in the same manner as in the first embodiment. The regions SA 1 , SA 2  and SA 3  are multiplied by the same weighting factors as in the first embodiment, and thus a corrected score of each pixel forming the region SA 1  is calculated. The location-correspondence determination unit  168  determines the maximum score among corrected scores of the pixels forming the region SA 1 , obtained as a result of the calculation, as the score indicating the correspondence with the contour feature CF 1 . In the example illustrated in  FIG. 15 , the location-correspondence determination unit  168  determines 0.56 of the pixel px 16  as the score. 
     C. Third Embodiment 
     In the present embodiment, the location-correspondence determination unit  168  modifies Equation (11) regarding similarity scores into an equation for imposing a penalty on an image point separated from a perpendicular line which is perpendicular to a model contour. The location-correspondence determination unit  168  defines a model point p and an image point p′, a unit length vector which is perpendicular to an edge orientation (contour) of a 2D model as a vector E p , and defines the following Equation (17).
 
{right arrow over (Δ p )}= p′−p   (17)
 
     If the following Equation (18) is defined by using a weighting factor indicated by w, similarity scores between model points and image points may be expressed as in Equation (19).
 
 w ({right arrow over (Δ p )})= e   −({right arrow over (∥Δp∥)}     2     −{right arrow over (Δp)}·{right arrow over (Ep)})/σ     2     (18)
 
     
       
         
           
             
               
                 
                   
                     w 
                     ( 
                     
                       → 
                       
                         Δ 
                         ⁢ 
                         
                             
                         
                         ⁢ 
                         p 
                       
                     
                     ) 
                   
                   = 
                   
                     e 
                     
                       
                         - 
                         
                           ( 
                           
                             
                               → 
                               
                                 
                                    
                                   
                                     Δ 
                                     ⁢ 
                                     
                                         
                                     
                                     ⁢ 
                                     p 
                                   
                                    
                                 
                                 2 
                               
                             
                             ⁢ 
                             
                               - 
                               
                                 
                                   → 
                                   
                                     Δ 
                                     ⁢ 
                                     
                                         
                                     
                                     ⁢ 
                                     p 
                                   
                                 
                                 ⁢ 
                                 
                                   · 
                                   
                                     → 
                                     
                                       E 
                                       p 
                                     
                                   
                                 
                               
                             
                           
                           ) 
                         
                       
                       / 
                       
                         σ 
                         2 
                       
                     
                   
                 
               
               
                 
                   ( 
                   18 
                   ) 
                 
               
             
             
               
                 
                   
                     SIM 
                     ⁡ 
                     
                       ( 
                       
                         p 
                         , 
                         
                           p 
                           ′ 
                         
                       
                       ) 
                     
                   
                   = 
                   
                     
                       w 
                       ( 
                       
                         → 
                         
                           Δ 
                           ⁢ 
                           
                               
                           
                           ⁢ 
                           p 
                         
                       
                       ) 
                     
                     ⁢ 
                     
                       
                          
                         
                           
                             → 
                             
                               E 
                               P 
                             
                           
                           ⁢ 
                           
                             · 
                             
                               
                                 → 
                                 ∇ 
                               
                               ⁢ 
                               
                                 I 
                                 
                                   p 
                                   ′ 
                                 
                               
                             
                           
                         
                          
                       
                       / 
                       
                         
                           max 
                           
                             q 
                             ∈ 
                             
                               N 
                               ⁡ 
                               
                                 ( 
                                 p 
                                 ) 
                               
                             
                           
                         
                         ⁢ 
                         
                            
                           
                             
                               → 
                               ∇ 
                             
                             ⁢ 
                             
                               I 
                               p 
                             
                           
                            
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   19 
                   ) 
                 
               
             
           
         
       
     
     Next, the location-correspondence determination unit  168  locally smooths a similarity score of each pixel px in the regions SA 1 , SA 2  and SA 3 , obtained by using Equation (19), according to the same method as in the first embodiment, and then establishes correspondences between the image points and the contour features CF in each of the regions SA 1 , SA 2  and SA 3 . 
     D. Modification Examples 
     The invention is not limited to the above-described embodiments, and may be implemented in various aspects within the scope without departing from the spirit thereof. For example, the following modification examples may also occur. 
     D-1. Modification Example 1 
     In the above-described first and second embodiments, the location-correspondence determination unit  168  computes scores within a region of 3 pixels×3 pixels centering on the contour feature CFm so as to establish a correspondence to a 2D point, but various modifications may occur in a method of computing scores in establishing the correspondences. For example, the location-correspondence determination unit  168  may compute scores within a region of 4 pixels×4 pixels. The location-correspondence determination unit  168  may establish the correspondences between 2D points by using evaluation functions other than that in Equation (11). 
     In the above-described first embodiment, the location-correspondence determination unit  168  and the optimization unit  166  estimates a pose of an imaged target object by using the CF method, but may estimate a pose of the target object in combination of the CF method and the MA method of the comparative example. The MA method works in a case where the two-color base is established in a target object and a background. Therefore, the location-correspondence determination unit  168  and the optimization unit  166  may select either the CF method or the MA method in order to estimate a pose of a target object according to a captured image. In this case, for example, the location-correspondence determination unit  168  first estimates a pose of a target object according to the MA method. In a case where estimation of a pose of the target object using the MA method does not converge, the location-correspondence determination unit  168  may perform pose estimation again on the basis of an initial pose of the target object by using an algorithm of the CF method. The location-correspondence determination unit  168  can estimate a pose of a target object with higher accuracy by using the method in which the MA method and the CF method are combined, than in a case where an algorithm of only the MA method is used or a case where an algorithm of only the CF method is used. 
     In the above-described embodiments, one or more processors, such as the CPU  140 , may derive and/or track respective poses of two or more target objects within an image frame of a scene captured by the camera  60 , using templates (template data) created based on respective 3D models corresponding to the target objects. According to the embodiments, even when the target objects moves relative to each other in the scene, these poses may be derived and/or tracked at less than or equal to the frame rate of the camera  60  or the display frame rate of the right/left optical image display unit  26 / 28 . 
     The template may include information associated with the target object such as the name and/or geometrical specifications of the target object, so that the one or more processors display the information on the right/left optical display unit  26 / 28  or present to external apparatus OA through the interface  180  once the one or more processors have derived the pose of the target objet. 
     The invention is not limited to the above-described embodiments or modification examples, and may be implemented using various configurations within the scope without departing from the spirit thereof. For example, the embodiments corresponding to technical features of the respective aspects described in Summary of Invention and the technical features in the modification examples may be exchanged or combined as appropriate in order to solve some or all of the above-described problems, or in order to achieve some or all of the above-described effects. In addition, if the technical feature is not described as an essential feature in the present specification, the technical feature may be deleted as appropriate. 
     The entire disclosure of Japanese Patent Application No. 2016-065733, filed on Mar. 29, 2016, is expressly incorporated by reference herein. 
     E. 3D Tracking Objects Using Fusing of Inertial Motion Sensor (IMU) and Camera Sensor 
     The invention is not limited to the above-described embodiments, and the below-described embodiments are also within the scope without departing from the spirit thereof. 
     The embodiments discussed above via section headers A-E relate to  FIGS. 1-15  which relate to a HMD  100  including (but not limited to) a camera  60  for estimating a pose, and the below embodiments to  FIGS. 16-28  which relate to a HMD  100 ′ which includes (but is not limited to) a camera  60  and an inertial sensor  71  for estimating a pose in a new and improved manner. The below embodiments improve accuracy and speed of object tracking by using multiple sensors and fusing data of the sensors together. Various embodiments are discussed below. 
     E-1. Introduction 
     Augmented reality (AR) integrates digital information from live video and a user&#39;s environment in real time. The three requirements in AR applications has been summarized as follows: 
     (1) It is interactive in real time, 
     (2) It is three dimensional (“3D”), and 
     (3) It combines real elements with virtual elements. 
     A great number of technologies on computer vision and image processing have been studied in order to meet these requirements. Some AR applications do not require accurate 3D object pose so rendering augmented information in video overlay mode does not cause any issues, such as “hovering” augmented information over a book for educational applications. 
     However, many AR applications require low latency (e.g., less than 15 mm/s) and accurate 3D object pose, such as industrial, military and medical AR applications. Latency relates to the speed of the pose, when movement occurs of an object from an initial position to a second position, to move the pose from the initial location to the second location. 
     In the below-described embodiments, a 3D object tracking system is described which may be implemented in head-mounted displays for AR applications. However, the latency of the 3D object tracking may be too high for some AR applications as shown in  FIG. 16A , which shows that, when an object  302  has initially moved relative to the viewpoint of the user, the pose  300  is delayed from continuing to overlay on the object  302 . Instead the pose is temporarily out of place (i.e., not overlaying over the object  302 ). In addition, the 3D object tracking is easy to get lost when user moves faster or when user moves to the object views where less feature points exist. High latency, user moving speed limitations, and losing tracking frequently greatly impacts the user experience of using head-mounted displays for optical see-through AR applications. 
       FIG. 16 ( b )  shows when latency is reduced so that when the object  302  has initially moved relative to the viewpoint of the user, the pose  300  is not delayed and thus is perceived by the user&#39;s eyesight to be continually overlaying the object  302 . 
     According to various aspects of the present disclosure, to avoid the above-described issues and reduce latency, a head-mounted display may be equipped an inertial motion unit (“IMU”) sensor with a gyro sensor and an accelerometer. In order to reduce latency and therefore improve the 3D object tracking performance, the IMU sensors and camera sensor are “fused” or the data of the IMU sensors and camera sensor are combined together. An IMU sensor may be used because it operates at a much higher frequency (e.g., 125 Hz, 250 Hz, 1 MHz, etc.) than a camera, and the latency of a human eye is around 15 ms. As such, ideally, it is preferred to reduce the latency of the system to less than 15 ms. 
     It should be understood that the term “IMU sensor” and “inertial sensor” are used interchangeably throughout this disclosure. 
     In one embodiment, the head-mounted display is equipped with an IMU sensor (3-axis accelerometer and 3-axis gyroscope) as the motion sensor, as discussed above and throughout this application. However, in another embodiment, the head-mounted display may be equipped a 3-axis magnetic sensor besides or in addition to an the accelerometer and gyroscope. In this regard, the IMU sensor may include an accelerometer, a gyro sensor and/or a magnetic sensor. In one embodiment, only the accelerometer and gyro sensor are used in the algorithm. However, in another embodiment, a magnetic sensor can be used to further improve the fusion accuracy, and so the magnetic sensor could be added (and the algorithm is corresponding changed) to the accelerometer and gyro sensor. 
     It should be understood, however, that the present invention should not be limited to the above embodiments of using an IMU sensor including accelerometer, gyroscope and/or a magnetic sensor in detecting motion, and may employ any other type of motion sensor or motion sensing system(s) which is capable of detecting motion along 3-axis. 
     Also, one or more of the motion sensors or motion sensing systems may be used in a single HMD device to further reduce the latency. For example, these motion sensors or motion sensing systems could be staggered to output a pose at different times. In this regard, each sensor/system provides a pose at a time when the other sensors are not outputting a pose, to thereby reduce the overall latency. 
     The motion sensors or motion sensing systems herein may operate at 125 Hz according to one embodiment. However, in other embodiments, the motion sensors or motion sensing systems may operate at high frequencies (e.g., 250 Hz, 1 MHz) so that the latency is reduced below 15 ms. 
     E-2. IMU and 3D Object Tracking Fusion 
     In this disclosure, one or more non-linear filters, such as an Extended Kalman Filter (EKF), a particle filter, an unscented Kalman filter (UKF), a maximum likelihood nonlinear system estimation or the like, may be used to fuse data from an IMU sensor and an camera sensor in a single HMD system. An example of using EKF for sensor fusion is provided in Gabriele Ligorio and Angelo Maria Sabatini, “Extended Kalman Filter-Based Methods for Pose Estimation Using Visual, Inertial and Magnetic Sensors: Comparative Analysis and Performance Evaluation”, Sensor 2013, 13, 1919-1941, which is incorporated herein in its entirety. While Ligorio et al. discuss a basic concept of using EKF for sensor fusion, sensor quality varies between sensors, and thus, the methods for fusing data from different types of sensors used by vision with different sensors may be different, and fusing data of different sensors encounters different challenges and issues. It is disclosed herein how to fuse an IMU sensor and a camera sensor using the 3D vision tracking technology to improve 3D object tracking performance and efficiency. In this disclosure, the details of such a fusion framework are presented, problems and solutions are discussed, and finally performance evaluation results are shown. 
     It is noted that the terms “fuse” or “fusion” when used in relation to deriving a pose of an object, whether in motion or static, relates to a process of using data from at least one sensor and a camera (or other imaging device) in deriving the pose, such as deriving a pose by analysis and/or combining of data from an IMU sensor and a camera relates to fusion of the IMU sensor and the camera. 
     E-2-1. Overview 
     E-2-1-1. System 
       FIG. 17  is a diagram illustrating a schematic configuration of an HMD  100 ′ according to various embodiments. It is noted that  FIG. 17  is similar to  FIG. 5  but is a different embodiment and, while many of the features of  FIG. 5  may be similar or duplicative of  FIG. 17 , all of the features of  FIG. 17  are described below. 
     The HMD  100 ′ is a head mounted display according to the illustrative embodiments (but the embodiments of the invention should not be limited to a head mounted display and can be embodied in other devices, such as a mobile phone). Similar to HMD  100  of  FIG. 5 , the HMD  100 ′ is also an optical transmission type head mounted display which enables a user to view a virtual image and to simultaneously view outside scenery directly. The HMD  100 ′ includes a camera  60  that collects image data from the outside scenery so that the HMD  100 ′ can display relevant virtual images. 
     In this embodiment, the HMD  100 ′ includes a fitting band  90  that is fitted on the head of a user, a display portion  20  that displays images, and a controller  10  that controls the display portion  20 . The display portion  20  enables the user to view a virtual image when the display portion  20  is fitted on the head of the user. 
     The fitting band  90  includes a fitting base portion  91  formed, in this embodiment, of a resin, a fabric belt portion  92  connected to the fitting base portion  91 , a camera  60 , and an inertial sensor (Inertial Measurement Unit; IMU)  71 . The fitting base portion  91  has a curved shape matched to a person&#39;s forehead. The belt portion  92  is a belt that is fitted around the head of the user. In other embodiments, the camera  60  and IMU  71  are directly integrated with a frame of display portion  20 . 
     While  FIG. 17  illustrates the fitting band  90 , it should be understood that the present invention is not limited to requiring the fitting band  90 . Indeed, in such embodiments, the elements in the fitting band  90 , including camera  60  and IMU  71  may be integrated in or disposed onto a frame of display portion  20 , the display portion  20  itself, and/or any other portion of the HMD  100 ′. As such, in some embodiments, the fitting band  90  may not be included in HMD  100 ′, but for ease of illustration and discussion, the below embodiments describe the fitting band  90  as part of the HMD  100 ′. 
     The camera  60  can image outside scenery and is disposed in a middle portion of the fitting base portion  91  in the illustrated embodiment of  FIG. 17 . In other words, the camera  60  is disposed at a position corresponding to the middle of the forehead of the user in a state in which the fitting band  90  is fitted on the head of the user. Therefore, in the state in which the user fits the fitting band  90  on the head of the user, the camera  60  images outside scenery which is external scenery in a visual line direction of the user and acquires a captured image by imaging. 
     In this embodiment, the camera  60  includes a camera base portion  61  that is rotated with respect to the fitting base portion  91  and a lens portion  62  of which a relative position to the camera base portion  61  is fixed. When the fitting band  90  is fitted on the head of the user, the camera base portion  61  is disposed to be rotatable along an arrow CS 1  which is a predetermined range of an axis included in a plane including a central axis of the user. Therefore, the direction of an optical axis of the lens portion  62  which is an optical axis of the camera  60  can be changed within the range of the arrow CS 1 . The lens portion  62  images a range which is changed by zoom about the optical axis. 
     The IMU  71  is an inertial sensor that detects acceleration. In some embodiments, the IMU  71  can detect an angular velocity and geomagnetism using a gyro sensor and a magnetic sensor in addition to acceleration. In this embodiment, the IMU  71  is contained in the fitting base portion  91 , but, in other embodiments, as discussed above, the IMU  71  may be disposed at any other portion of the HMD  100 ′ such as the display portion frame. Therefore, in the embodiment where the IMU  71  is contained in the fitting base portion  91 , the IMU  71  detects acceleration, angular velocities, and geomagnetism of the fitting band  90  and the camera base portion  61 . Therefore, in the embodiment where the IMU  71  is disposed in a portion of the display portion frame of the HMD  100 ′, the IMU  71  detects acceleration, angular velocities, and geomagnetism of the display portion frame and the camera base portion  61 . In either event, the IMU  71  detects acceleration, angular velocities, and geomagnetism of the user&#39;s head. 
     Since a relative position of the IMU  71  to the fitting base portion  91  is fixed, the camera  60  is movable with respect to the IMU  71 . Thus, IMU  71  has an adjustably fixed spatial relationship with camera  60 . In another embodiment, IMU  71  may have a fixed spatial relationship with camera  60 . Further, since a relative position of the display portion  20  to the fitting base portion  91  is fixed, a relative position of the camera  60  to the display portion  20  is movable. 
     The display portion  20  is connected to the fitting base portion  91  of the fitting band  90  and has a glasses shape in this embodiment. The display portion  20  includes a right holder  21 , a right display driver  22 , a left holder  23 , a left display driver  24 , a right optical image display  26 , and a left optical image display  28 . The right optical image display  26  and the left optical image display  28  are located in front of the right and left eyes of the user when the display portion  20  is fitted on the user. One end of right optical image display  26  and one end of left optical image display  28  are connected to each other at a position corresponding to the middle of the forehead of the user when the display portion  20  is fitted on the user. 
     The right holder  21  has a shape which extends from an end portion ER which is the other end of the right optical image display  26  in a substantially horizontal direction and is inclined upward obliquely from the middle of the shape and connects the end portion ER to a right connector  93  of the fitting base portion  91 . Similarly, the left holder  23  has a shape which extends from an end portion EL which is the other end of the left optical image display  28  in a substantially horizontal direction and is inclined upward obliquely from the middle of the shape and connects the end portion EL to a left connector (not illustrated) of the fitting base portion  91 . The right holder  21  and the left holder  23  are connected to the fitting base portion  91  by the right and left connectors  93 , and thus the right optical image display  26  and the left optical image display  28  are located in front of the eyes of the user. The connectors  93  connect the right holder  21  and the left holder  23  to be rotatable and fixable at any rotation positions. As a result, the display portion  20  is installed to be rotatable with respect to the fitting base portion  91  in this embodiment. 
     The right holder  21  is a member installed to extend from the end portion ER which is the other end of the right optical image display  26  to a position corresponding to a temporal region of the user when the display portion  20  is fitted on the user. Similarly, the left holder  23  is a member installed to extend from the end portion EL which is the other end of the left optical image display  28  to a position corresponding to a temporal region of the user when the display portion  20  is fitted on the user. In this embodiment, the right display driver  22  and the left display driver  24  are disposed on sides facing the head of the user when the display portion  20  is fitted on the user. 
     The display drivers  22  and  24  include liquid crystal displays  241  and  242  (hereinafter also referred to as “LCDs  241  and  242 ”) and projection optical systems  251  and  252  to be described below with respect to  FIG. 18 . The details of the configurations of the display drivers  22  and  24  will be described below. 
     The optical image displays  26  and  28  include light-guiding plates  261  and  262  (see  FIG. 18 ) and light adjustment plates to be described below. The light-guiding plates  261  and  262  are formed of a light transmission resin material or the like and guide image light output from the display drivers  22  and  24  to the eyes of the user. In some embodiments, image displays  26  and  28  include prisms in addition to or substituting light guiding plates  261  and  262 . The light adjustment plates are optical elements with a thin plate shape and are disposed to cover the front side of the display portion  20  which is an opposite side to the side of the eyes of the user. By adjusting light transmittance of the light adjustment plates, it is possible to adjust the amount of external light entering the eyes of the user and adjust easiness of view of a virtual image. This may be useful to adjust for varying lighting conditions (e.g. indoor v. outdoor lighting levels) while maintaining visibility of the virtual image. 
     The display portion  20  further includes a connection portion  40  connecting the display portion  20  to the controller  10 . The connection portion  40  includes a body cord  48  connected to the controller  10 , a right cord  42 , a left cord  44 , and a connection member  46 . The right cord  42  and the left cord  44  are two branched cords of the body cord  48 . The display portion  20  and the controller  10  transmit various signals via the connection portion  40 . In the right cord  42 , the left cord  44 , and the body cord  48 , for example, a metal cable or an optical fiber can be used. 
       FIG. 18  is a block diagram illustrating a functional configuration of the HMD  100 ′. As illustrated in  FIG. 18 , the controller  10  includes a ROM  121 , a RAM  122 , a power source  130 , the operation section  135 , an identification target storage section  139 , a CPU  140 , an interface  180 , a transmission section  51  (Tx  51 ), and a transmission section  52  (Tx  52 ). 
     The power source  130  feeds power to each section of the HMD  100 ′. The ROM  121  stores various programs. The CPU  140  executes various programs by loading the various programs stored in the ROM  121  on the RAM  122 . 
     The interface  180  is an input and output interface that connects various external devices OA which are content supply sources to the controller  10 . Examples of the external devices OA include a storage device storing an AR scenario, a personal computer (PC), a mobile phone terminal, and a game terminal. Examples of the interface  180  include a USB interface, a micro USB interface, a memory card interface, and a video interface (e.g. DisplayPort, HDMI, etc. . . . ). 
     The CPU  140  loads programs stored in the ROM  121  on the RAM  122  to function as an operating system  150  (OS  150 ), a display controller  190 , an audio processor  170 , an image processor  160 , a marker identification section  165 , and a processor  167 . 
     The display controller  190  generates control signals to control the right display driver  22  and the left display driver  24 . The display controller  190  controls generation and emission of image light in accordance with each of the right display driver  22  and the left display driver  24 . The display controller  190  transmits control signals for the right LCD controller  211  and the left LCD controller  212  via the transmission sections  51  and  52 , respectively. The display controller  190  transmits control signals for a right backlight controller  201  and a left backlight controller  202 . 
     As illustrated in  FIG. 18 , the display portion  20  includes the right display driver  22 , the left display driver  24 , the right light-guiding plate  261  serving as the right optical image display  26 , and the left light-guiding plate  262  serving as the left optical image display  28 . 
     The right display driver  22  includes the reception section  53  (Rx  53 ), a right backlight controller  201 , a right backlight  221 , a right LCD controller  211 , the right LCD  241 , and the right projection optical system  251 . The right backlight controller  201  and the right backlight  221  function as a light source. The right LCD controller  211  and the right LCD  241  function as a display element. In another embodiment, instead of the foregoing configuration, the right display driver  22  may include a spontaneous emission type display element such as an organic EL display element or may include a scan type display element that scans an optical beam from a laser diode on a retina. The same also applies to the left display driver  24 . 
     The reception section  53  functions as a receiver that performs serial transmission between the controller  10  and the display portion  20 . The right backlight controller  201  drives the right backlight  221  based on an input control signal. The right backlight  221  is, for example, an emitter such as an LED or an electroluminescence (EL). The right LCD controller  211  drives the right LCD  241  based on control signals transmitted from the image processor  160  and the display controller  190 . The right LCD  241  is a transmission type liquid crystal panel in which a plurality of pixels is arrayed in a matrix form. 
     The right projection optical system  251  is configured to include a collimating lens that forms image light emitted from the right LCD  241  as a light flux in a parallel state. The right light-guiding plate  261  serving as the right optical image display  26  guides the image light output from the right projection optical system  251  to the right eye RE of the user while reflecting the image light along a predetermined light path. The left display driver  24  has the same configuration as the right display driver  22  and corresponds to the left eye LE of the user, and thus the description thereof will be omitted. 
     The image processor  160  acquires an image signal included in content and transmits the acquired image signal to reception sections  53  and  54  of the display portion  20  via the transmission sections  51  and  52 . The audio processor  170  acquires an audio signal included in the content, amplifies the acquired audio signal, and supplies the amplified audio signal to a speaker (not illustrated) inside the right earphone  32  and a speaker (not illustrated) inside the left earphone  34  connected to the connection member  46 . 
     The controller  10  is a device that controls the HMD  100 ′. In some embodiments, controller  10  is integrated into the display portion  20  and/or the fitting band  90 . In other embodiments, controller  10  is implemented on a separate computer. The controller  10  includes an operation section  135  that includes an electrostatic track pad or a plurality of buttons which can be pressed and can be used for calibration in imaging. The operation section  135  is disposed on the front surface of the controller  10 . In other embodiments, a portion of operation section  135 , or the entirety thereof, is disposed on a frame of display portion  20  and/or fitting band  90 . 
     After the calibration of each sensors included in the IMU  71  is performed, detected values (measured outputs) of the acceleration, the angular velocity, and the geomagnetism of the sensors in the IMU  71  are fused, and thus high precise IMU orientation can be obtained. This fusion means that measured movement values from the sensors are merged with predicted values in order to provide a smoother and more accurate final sensor output. 
     E-2-1-2. Overview of Method of Sensor Fusion 
     After fusion of predicted sensor data with presently collected sensor data as shown in  FIG. 19 , the fused sensor data is further fused with camera tracking data. According to the embodiment in  FIG. 20 , the CPU  140  operates two separate threads, one processing data from the IMU  71  and one processing data from the camera  60 . Information is exchanged between the two threads in order to fuse the camera data and the IMU data. The fusion of the camera data and the IMU data allows for more accurate object location tracking because the IMU data can be processed much more quickly (e.g. at 125 Hz) than the image data (e.g. 30 fps or 30 Hz). Thus, there will be less latency in the object tracking. 
     It should be understood that the term “camera data” may refer to data taken by a camera  60 , but should not be limited to camera and can be taken by any other imaging device. As such the term “camera data” should not be limited to data taken only by a camera. As such, the term “camera data” may be referred to herein as “image data.” 
     Similarly, it should be understood that the term “IMU data” may refer to data taken by an IMU  71 , but should not be limited to an IMU and can be obtained by other motion sensing devices. As such, the term “IMU data” should not be limited to data obtained only by an IMU. Accordingly, the term “IMU data” may be referred to herein as “sensor data.” 
     Referring still to  FIG. 20 , fusion is accomplished by placing timestamps on both the sensor data and the image data. This way, a precise timing of both the sensor data and the image data is known and they can be matched to each other, resulting in fusion of data from the same time. Thus, in steps S 300  and S 302 , IMU data and image data are acquired with timestamps. Subsequently image data and IMU data are fused in the vision thread in step S 306 . Step S 306  is discussed in more detail in  FIG. 21  later. 
     In the IMU thread, S 304  confirms that the HMD is in see-through mode. See-through mode is a mode in which the user simultaneously views the external environment and virtual image data. In some embodiments, the HMD  100 ′ is capable of operating in a non-see-through mode, in which the display area is covered by a virtual image and the user is intended to focus solely on the virtual image. Once see-through mode is confirmed, the fused data from S 306  is transmitted from the vision thread to the IMU thread in S 310 . 
     One downside of using the sensor data to track movement (as compared to image data) is that it can include jitter. This jitter may be caused by limited precision of the IMU  71  and normally occurring outliers in measured movement. For example, the IMU  71  may be coincidentally measuring acceleration during a fraction of a second when the user&#39;s head jerks. This could be logged as a sudden and extreme movement by the IMU  71 . This problem is solved by steps S 306 , S 310 , and S 308 . By fusing the sensor data with the image data and reintroducing the fused data in the IMU thread, these jitters are reduced in amplitude or eliminated, resulting in much smoother movement tracking. With this smoothed sensor data, the IMU thread finally outputs the IMU pose, or predicted object location, in step S 314 . In other words, if the HMD  100 ′ is tracking a moving real-world object with, e.g., an “information bubble,” the information bubble will be moved in the image display (and in the user&#39;s view) to follow the moving real-world object by an amount based on the prediction that is output in S 314 . 
     E-2-2. Tracker Fusing IMU 
     E-2-2-1. Method of Fusing IMU with 3D Object Tracker 
       FIG. 21  is a flow chart illustrating a method  400  of fusion of an IMU sensor and 3D Object tracker of step S 306  of  FIG. 20 , in accordance with an embodiment. 
       FIG. 21  is used as herein the foundational flowchart herein and will refer to each of  FIGS. 22-28  which are each referenced in  FIG. 21  using reference letters A-E (each surrounded by a circle) in the Figure. Accordingly, while  FIG. 21  will be discussed throughout the following points,  FIGS. 22-28  will be discussed throughout these portions as well and then revert back to the discussion of  FIG. 21 . 
     It will be noted that various terms may be used to refer to the camera data such as “3D object tracker.” 
     Starting first with step S 402  of  FIG. 21 , the CPU  140  determines whether the 3D object tracker using the camera  60  has accurately determined an initial 3D pose of an object, as previously discussed herein. If so, the CPU  140  may set a value of “tracker state” to be true which allows method  400  to proceed to step S 406 . 
     On the other hand, if, in step S 402 , the CPU  140  determines the 3D object tracker has not accurately determined an initial 3D pose of an object, an object pose estimation module is executed step S 404 , which is shown in  FIG. 22 . In  FIG. 22  (step S 502 ), before tracking is started (i.e., before step S 506 ), an initial object pose is detected and the location-correspondence determination unit  168  may perform object pose estimation on the basis of the initial pose of the target object by using an algorithm, for example, discussed in section A-4 above. If the initial pose is not detected or the object pose is not successfully estimated (step S 504 ), the method  500  may proceed to step S 506  where the object tracker is initialized to set the initial values of all tracking parameters. At step S 508 , the method  500  determines if the tracker is properly tracking the object and if so, a state parameter (e.g., “trackerInit”) is set to true and the method returns to  FIG. 21 ; otherwise, the method  500  may proceed to step S 510  where the state parameter (e.g., “trackerInit”) is set to false indicating a bad pose and/or the tracker is not initialized. 
     E-2-2-2. Initialize IMU Fusion 
     Returning to  FIG. 21 , the method  400  determines if the IMU fusion is initialized. In this regard, if the tracker is initialized (as discussed above in step S 506 ), the next step is to initialize IMU fusion module, which is discussed in  FIG. 23 . The first step in method  600  is to run the 3D object tracker in step S 602 , and if the CPU  140  determines that the 3D object tracker is providing accurate output (S 604 ), the CPU  140  executes instructions to determine whether the user is static or not (S 606 ) and when the user&#39;s head, for example, is determined to be static in step S 608  (which is discussed later), then the CPU  140  obtains a centered pose in step S 610 . In this regard, if tracking is successful, the fusion initialization function is then executed in step S 612 , which is described in depth below. 
     To initialize the IMU fusion, the main parameter to initialize is the transformation matrix between the object coordinate system and the global coordinate system, T o2G . The following is an automatic way to calculate the transformation matrix.
 
 T   O2G   =T   S2G   *T   C2S   *T   O2C  
 
     Where T C25  is the transformation matrix from camera to IMU, and is pre-known through calibration. An assumption is that the object is static, so T O2G  is fixed and needs to be calculated only once in the system initialization phase. T S2G  is the IMU pose in the global coordinate system. T O2C  is the object pose in the camera coordinate system, and output by the object tracker. 
     In the initialization phase, the user is recommended to keep his/her head static. When IMU is static, T S2G  can be calculated as follows (ax, ay, az are the accelerometer reading), 
     
       
         
           
             
               tan 
               ⁡ 
               
                 ( 
                 roll 
                 ) 
               
             
             = 
             
               tan 
               ⁡ 
               
                 ( 
                 
                   ay 
                   az 
                 
                 ) 
               
             
           
         
       
       
         
           
             
               tan 
               ⁡ 
               
                 ( 
                 pitch 
                 ) 
               
             
             = 
             
               tan 
               ⁡ 
               
                 ( 
                 
                   
                     - 
                     ax 
                   
                   
                     
                       ay 
                       * 
                       
                         sin 
                         ⁡ 
                         
                           ( 
                           roll 
                           ) 
                         
                       
                     
                     + 
                     
                       az 
                       * 
                       
                         cos 
                         ⁡ 
                         
                           ( 
                           roll 
                           ) 
                         
                       
                     
                   
                 
                 ) 
               
             
           
         
       
       
         
           
             yaw 
             = 
             0 
           
         
       
       
         
           
             
               
                 R 
                 x 
               
               ⁡ 
               
                 ( 
                 roll 
                 ) 
               
             
             = 
             
               [ 
               
                 
                   
                     1 
                   
                   
                     0 
                   
                   
                     0 
                   
                 
                 
                   
                     0 
                   
                   
                     
                       cos 
                       ⁡ 
                       
                         ( 
                         roll 
                         ) 
                       
                     
                   
                   
                     
                       sin 
                       ⁡ 
                       
                         ( 
                         roll 
                         ) 
                       
                     
                   
                 
                 
                   
                     0 
                   
                   
                     
                       - 
                       
                         sin 
                         ⁡ 
                         
                           ( 
                           roll 
                           ) 
                         
                       
                     
                   
                   
                     
                       cos 
                       ⁡ 
                       
                         ( 
                         roll 
                         ) 
                       
                     
                   
                 
               
               ] 
             
           
         
       
       
         
           
             
               
                 R 
                 y 
               
               ⁡ 
               
                 ( 
                 pitch 
                 ) 
               
             
             = 
             
               [ 
               
                 
                   
                     
                       cos 
                       ⁡ 
                       
                         ( 
                         pitch 
                         ) 
                       
                     
                   
                   
                     0 
                   
                   
                     
                       - 
                       
                         sin 
                         ⁡ 
                         
                           ( 
                           pitch 
                           ) 
                         
                       
                     
                   
                 
                 
                   
                     0 
                   
                   
                     1 
                   
                   
                     0 
                   
                 
                 
                   
                     
                       sin 
                       ⁡ 
                       
                         ( 
                         pitch 
                         ) 
                       
                     
                   
                   
                     0 
                   
                   
                     
                       cos 
                       ⁡ 
                       
                         ( 
                         pitch 
                         ) 
                       
                     
                   
                 
               
               ] 
             
           
         
       
       
         
           
             
               
                 R 
                 z 
               
               ⁡ 
               
                 ( 
                 yaw 
                 ) 
               
             
             = 
             
               [ 
               
                 
                   
                     
                       cos 
                       ⁡ 
                       
                         ( 
                         yaw 
                         ) 
                       
                     
                   
                   
                     
                       sin 
                       ⁡ 
                       
                         ( 
                         yaw 
                         ) 
                       
                     
                   
                   
                     0 
                   
                 
                 
                   
                     
                       - 
                       
                         sin 
                         ⁡ 
                         
                           ( 
                           yaw 
                           ) 
                         
                       
                     
                   
                   
                     
                       cos 
                       ⁡ 
                       
                         ( 
                         yaw 
                         ) 
                       
                     
                   
                   
                     0 
                   
                 
                 
                   
                     0 
                   
                   
                     0 
                   
                   
                     1 
                   
                 
               
               ] 
             
           
         
       
       
         
           
             
               R 
               
                 S 
                 ⁢ 
                 
                     
                 
                 ⁢ 
                 2 
                 ⁢ 
                 
                     
                 
                 ⁢ 
                 G 
               
             
             = 
             
               
                 ( 
                 
                   
                     R 
                     x 
                   
                   * 
                   
                     R 
                     y 
                   
                   * 
                   
                     R 
                     z 
                   
                 
                 ) 
               
               ′ 
             
           
         
       
       
         
           
             
               T 
               
                 S 
                 ⁢ 
                 
                     
                 
                 ⁢ 
                 2 
                 ⁢ 
                 
                     
                 
                 ⁢ 
                 G 
               
             
             = 
             
               [ 
               
                 
                   
                     
                       R 
                       
                         S 
                         ⁢ 
                         
                             
                         
                         ⁢ 
                         2 
                         ⁢ 
                         
                             
                         
                         ⁢ 
                         G 
                       
                     
                   
                   
                     
                       0 
                       3 
                     
                   
                 
                 
                   
                     
                       
                         
                           0 
                         
                         
                           0 
                         
                         
                           0 
                         
                       
                     
                   
                   
                     1 
                   
                 
               
               ] 
             
           
         
       
     
     Since T S2G  may not be accurate if user is not static, and this error will make the fusion accuracy become not reliable, before initializing IMU fusion, there is another function to check if user is static. Only when user is static, the IMU fusion initialization function will be called. Therefore, before initializing IMU fusion, the CPU executes instructions to detect that the user is static under step S 606 , as mentioned above. 
     E-2-2-2-1 Detect Static Motion 
     When user is static, IMU readings can be modeled as a Gaussian distribution, and the probability density function of each dimensional reading is: 
     
       
         
           
             
               
                 
                   
                     p 
                     ⁡ 
                     
                       ( 
                       x 
                       ) 
                     
                   
                   = 
                   
                     
                       1 
                       
                         σ 
                         ⁢ 
                         
                           
                             2 
                             ⁢ 
                             π 
                           
                         
                       
                     
                     ⁢ 
                     
                       e 
                       
                         - 
                         
                           
                             
                               ( 
                               
                                 x 
                                 - 
                                 μ 
                               
                               ) 
                             
                             2 
                           
                           
                             2 
                             ⁢ 
                             
                               σ 
                               2 
                             
                           
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   3 
                   ) 
                 
               
             
           
         
       
     
     Where μ represents the mean value and σ the standard deviation. μ and σ can be estimated from IMU data collected when user is static. For a real time IMU reading x, when p(x)&gt;th (where “th” is an experimental threshold), the user is determined to be static. 
     If the user is detected being static according to the above-discussed model, IMU fusion initialization is executed under step S 612 . After IMU fusion is initialized, a state flag is set to true to indicate that the fusion is initialized successfully; otherwise the flag is set to false. 
     E-2-2-3. IMU Pose Predication 
     Returning back to  FIG. 21 , in step S 408 , if IMU fusion is initialized (discussed above), the IMU pose is predicted before tracking, and the IMU predicted pose is then utilized in the tracking. Specifically, the CPU  140  predicts in sequence a sensor pose of the inertial sensor with respect to the global coordinate system by a non-linear estimation algorithm based at least on the sensor data sequence acquired from the IMU sensor. This is described in more detail below. 
     According to one embodiment of the present disclosure, the state vector x k ={p k , v k , q k } includes the IMU position p k , IMU velocity v k  and IMU orientation q k , all in the global coordinate system; and the control input u k  includes the accelerometer input and gyro input. The state transition and measurement models are,
 
 x   k =ƒ( x   k-1   ,u   k-1   ,w   k-1 ) and  z   k   =h ( x   k   ,v   k )
 
     Where w k  and v k  are the process and measurement noises which are assumed to be zero mean Gaussian noises with covariance Q k  and R k  respectively. 
     The equations to predict IMU pose are,
 
 {circumflex over (x)}   k =ƒ( x   k-1   ,u   k-1 )  (4)
 
 {circumflex over (P)}   k   =F   k-1   P   k-1   F   k-1   T   +L   k-1   Q   k-1   L   k-1   T   (5)
 
     where F is the Jacobian matrix of f with respect to x, and L is the Jacobian matrix of f with respect to the process noise w·{circumflex over (x)} k =ƒ(x k-1 , u k-1 ){circumflex over (P)} k =F k-1 P k-1 F k-1   T +L k-1 Q k-1 L k-1   T    
     IMU pose is derived from {circumflex over (x)} k  (translation from p k  and rotation from q k ). 
     E-2-2-4. IMU Divergence Determination 
     An IMU sensor, especially the accelerometer may diverge meaning that data is inaccurate for pose prediction. For example, when vision lost for a long period of time. 
     To best use IMU in the tracking, one needs to determine when the IMU diverges. The criteria to determine IMU divergence is defined below.
 
Position covariance: posCov=√{square root over (Σ i,j=0   2   P   i,j   *P   i,j )}
 
posCov=√{square root over (Σ i,j=0   2   P   i,j   *P   i,j )}  (6)
 
Velocity covariance: velCov=√{square root over (Σ i,j=3   5   P   i,j   *P   i,j )}  (7)
 
Orientation covariance: oriCov=√{square root over (Σ i,j=6   9   P   i,j   *P   i,j )}  (8)
 
     where P is the state covariance matrix. 
     The following formula determines if accelerometer readings are abnormal:
 
|(∥Accelerometer∥−∥calibrated greound truth of Accelerometer∥)|&gt; Th   Acc    (9)
 
     If posCov&gt;Th cov   pos  or velCov&gt;Th cov   vel  or oriCov&gt;Th cov   ori  or equation (9) is fulfilled, the IMU is diverged and the state flag “PredictPose” is set to false (step S 414 ), otherwise it is set to true (step S 416 ). The method  400  then may proceed to step S 420  discussed below for the IMU Object Tracker. 
     E-2-2-5. IMU Object Tracker 
     An example of the tracking an object using an IMU senor is illustrated in  FIG. 24  and discussed below. 
     E-2-2-5-1. Detect Static Motion (S 702 ) 
     In steps S 702  and S 704 , the CPU  140  executes a module to determine whether or not the user&#39;s head, and thus the inertial sensor, is held static or substantially static based on the sensor data sequence, as has been discussed above. If user or the inertial sensor is detected as being static or substantially static, no tracking may be performed. Instead, the sensor pose previously predicted by the non-linear estimation algorithm is used as the next predicted sensor pose in the case where the inertial sensor is determined to be held static or substantially static. In this regard, another pose is not predicted but a previous pose that has been stored in memory is simply retrieved from the memory and used for the next or current frame. Alternately, the object pose, or the second pose of the object, obtained previously by fusing the camera data and the sensor data may be used as the next fused object pose. That is, in some embodiments, it is not necessary to perform the vision tracking in the vision thread if user or the inertial sensor is detected as being static or substantially static. 
     This reduces computation time and processing power of the system since no other computations or steps need to be executed to determine the current pose. In this manner, tracking speed is improved. 
     However, if user is moving, the method  700  proceeds to performing the steps S 708 -S 722 . 
     It is noted that steps S 712 -S 716  may be optional (and thus are shown with a dotted box around these steps) when edge alignment or other feature is used or for low-feature objects which may not have many feature matching features (e.g., not many KLT features). Accordingly, one embodiment from S 708  is to proceed to S 709  or directly to S 718  and another embodiment is from S 708  is to proceed to either S 712  or S 710 . 
     E-2-2-5-2. Predict Feature Point Location Using IMU Predicted Pose/KLT Matching Using Predicted Feature Location 
     In step S 708 , if the IMU predicted pose state flag “PredictPose” is true (S 414  from  FIG. 21 ), the method  700  will behave very differently from its original tracker. The IMU predicted pose have multiple usage to improve the tracking performance. First, it is converted into the camera coordinate and becomes predicted object pose. Using the predicted object pose, all the feature points matched in previous frame are projected to the current frame so their locations in the current frame are predicted (step S 712 ), provided that the object has sufficient features for feature matching (but if not the method  700  may proceed directly to S 718 ). 
     Then, in steps S 714 -S 716 , the CPU  140  performs feature matching using the predicted feature location. The features matching, according to embodiments, may be performed using Kanade-Lucas-Tomasi (KLT) matching (S 714 ), removing any outliers (e.g., points with a difference greater than a predetermined threshold relative to the matching) (S 715 ), and then estimating the pose using inlier feature matching results and the IMU predicted pose (S 716 ). In KLT matching, with the predicted feature location, the matching becomes more accurate and faster. The reason is illustrated in  FIG. 28  where points indicated by squares in  FIG. 28  indicate the feature locations in the previous frame, and the points indicated by circles in  FIG. 28  indicate the true/predicted locations of those features in the current frame. From the length of the dotted line between these points, the inter-frame motion is large and the KLT feature matching may fail if the IMU is not used. With IMU predication, the feature locations in the current frame are predicted close to their real locations so the feature matching will be successful and the matching will be fast and accurate since the algorithm knows where to find those features. 
     In this regard, there is matching, between consecutive image frames in the image data sequence: 2D feature points of the object based at least on the predicted sensor pose, the second spatial relationship and 3D points on the 3D model, whereby the 3D points corresponding to the 2D feature points. 
     Referring back to step S 708  of  FIG. 24 , if the IMU predicted pose state flag “PredictPose” is false (S 416  from  FIG. 21 ), the CPU  140  feature matching is performed without using IMU data in S 710  (or the method  700  may proceed to S 709  where the CPU  140  determines that vision is lost). In this regard, only KLT matching, for example, may be used and then pose estimation using the feature matching results is outputted. In this regard, using step S 710  instead of steps S 712 -S 715  allows the system to avoid data from the IMU sensor because the IMU sensor has diverged and thus, such data is not useful. 
     E-2-2-5-3. Pose Estimation Using KLT Matching Results &amp; IMU Predicted Pose (S 716 ) 
     As mentioned above, KLT matching outputs the matched features. Using these features and their corresponding 3D coordinates, a pose can be estimated using a robust pose estimator by Gauss-Newton optimization, according to one embodiment. 
     The robust pose estimator using Gauss-Newton optimization requires an initial pose, and it may not converge if the initial pose is not close enough to the real pose, or it may converge to a wrong local minimum which causes the estimated pose may be inaccurate. In the original tracker, the object pose in a previous frame is the input to the robust pose estimator. If the inter-frame motion is obvious, the previous pose will not be close to the real pose of the current frame, so robust pose estimator may converge very slow, converge to a wrong minimum, or possibly not converged at all. Accordingly, in one embodiment, a predicted object pose converted from the IMU predicted pose is used as the initial pose to the robust estimator. Since the predicted pose as the initial pose is much more accurate than the object pose in the previous frame, the robust estimator converges faster and the pose converged to be more accurate. In this manner, the tracker performance is improved. 
     Accordingly, a second pose is derived based at least on (1) the matched 2D feature points, (2) the sensor pose or another sensor pose predicted in sequence, and (3) a second spatial relationship (i.e., the relationship between an object coordinate system defined on the object or the 3D model and a global coordinate system based at least on: (1) the first initial pose or a pose tracked from the first pose, (2) the sensor data sequence acquired from the inertial sensor, and (3) a first spatial relationship between the inertial sensor and the camera. 
     In some embodiments, the CPU  140 , or computer processor, displays an image such as a rendered image of a 3D AR model, using the display, using the thus derived second pose of the object, so that the user is allowed to visually perceive the position and pose of the AR object is substantially align with or anchored to those of the object through the HMD  100 . In this case, the processor derives the image position x dis   _   1 , of each 3D point consisting of the 3D AR object using the following equation.
 
 x   dis   _   1   =PT   Cam2Disp   T   Object2Cam   X  
 
where P is the projection matrix and T Cam2Disp  is the 3D transformation matrix from the camera coordinate system to the display coordinate system of the HMD  100 . T object2Cam  refers to the object pose represented in the camera coordinate system and in this embodiment the second pose of the object derived according to the present embodiment. X represents each 3D point included in the 3D AR model expressed in the 3D model coordinate system, which is stored in the memory of the HMD  100 .
 
E-2-2-5-4. Pose Refinement by Edge Alignment &amp; IMU Predicted Pose (S 718 )
 
     According to some embodiments as shown in step S 718 , an edge alignment method is used to refine the pose calculated from KLT features. The initial edge points are extracted based on the pose calculated from KLT features. When a view of object has less than 5 KLT feature points detected, no pose can be estimated from KLT matching, then edge alignment method will fail. In this work, when this case is encountered, the IMU predicted object pose will be the input to the edge alignment method. Since the predicted pose is relatively accurate, edge alignment method will most probably still work. 
     KLT matching can be even eliminated since IMU predicted pose can be used as the initial pose for edge alignment method to work. This is most useful for a low feature object since there may not be KLT features in a low feature object. 
     E-2-2-5-5. Outlier Removal (S 720 ) 
     General speaking, KLT features and edge features contain outliers and outlier removal is performed, as provided in step S 720 . 
     To remove outlier from KLT features, the difference between KLT matched results and IMU predicted locations will be compared. A histogram of the difference is calculated and features whose difference is bigger than a threshold will be removed as outliers. 
     To remove outliers edge features, besides using the histogram method as used for KLT features, an edge point that has too many possible matching points will be removed as outliers. 
     E-2-2-6. Reinitialize IMU Fusion 
     Referring back to  FIG. 21 , at step S 420 , the CPU  140  performs a “reinitialize IMU fusion” step (or module). This module checks if the pose detected from its previous step is accurate. If the pose is accurate and IMU is diverged, IMU fusion will be re-initialized. The IMU re-initialization as shown in  FIG. 25  is similar as the IMU fusion initialization module, which is provided in  FIG. 23  and previously discussed. The first step in method  800  is to run the 3D object tracker, and if the CPU  140  determines that the 3D object tracker is providing accurate output (S 802 ), the CPU  140  executes instructions to determine whether the IMU has experienced a diverged condition when the user&#39;s head, for example, is determined to have moved in step S 804 . If so, the CPU  140  obtains a centered pose in step S 806 , and the CPU reinitializes the IMU fusion function and the method  800  returns back to  FIG. 21 . 
     Referring back to step S 422  to  FIG. 21 , if the number of inliers (matched feature points left after outlier filter) is greater than a threshold, the next step will be “Fuse IMU and Vision” step S 424 . 
     E-2-2-7. Fuse IMU and Vision (S 424 ) 
     In step S 424 , IMU readings are updated, and the 2D and 3D information of features from the object tracker are available. 
     First, a new state is predicted using the latest gyroscope and accelerometer readings according to equations (3) and (4).
 
 y   k   =z   k   −h ( {circumflex over (x)}   k ) y   k   =z   k   −h ( {circumflex over (x)}   k )  (10)
 
 S   k   =H   k   {circumflex over (P)}   k   H   k   T   +M   k   R   k   M   k   T   (11)
 
 K   k   ={circumflex over (P)}   k   H   k   T   S   k   −1   (12)
 
 x   k   ={circumflex over (x)}   k   +K   k   y   k   (13)
 
 P   k =( I−K   k   H   k ) {circumflex over (P)}   k   (14)
 
     Where H is the Jacobian matrix of h with respect to x, and M is the Jacobian matrix with respect to v. The measurement z k  includes all the feature points matched by the vision object tracker. In order to improve the fusion speed, only d features are selected according to their re-projection errors. For example, d&lt;50 in one of our implementation. 
     E-2-2-8. Handle Vision Loss 
       FIG. 26  is a flowchart illustrating handling vision loss, according to an embodiment. The method  900  of  FIG. 26  handles the case when vision pose is not updated due to a particular reason (e.g., the pose estimation does not converge, etc.), but the IMU pose is fused correctly. 
     For example, if the IMU does not diverge (S 902 ), the fused vision pose is calculated in step S 904 , and the IMU fused pose is then set to the vision pose in step S 906 , as discussed below. 
     IMU pose T S2G  is derived from x k  (translation from p k  and rotation from q k ).
 
 T   S2C   =T   S2C   *inv ( T   S2G )* T   O2G   (15)
 
     The pose in the object tracker is then updated to T O2C . This module reduces tracking loss using IMU pose when vision tracker fails. 
     E-2-3. Get Jitter Reduced IMU Pose 
       FIG. 27  is for obtaining a pose with jitter reduction and runs throughout all steps of the IMU methods discussed above. 
     On HMD  100  or  100 ′, for optical see-through applications, IMU predicted pose is output for display. One problem observed on device is that user feels much more jittery with IMU fusion than the original tracker. Therefore, introduced is static jitter detection and motion jitter detection methods to detect jitter and apply corresponding jitter reduction methods to ensure user has good experience. 
     Pose smoothing is a traditional way to reduce pose jitter, but its drawback is it adds latency as well which makes the latency improvement less obvious than before. Therefore, we proposed the following new methods to reduce jitter without affecting latency improvement so much. In addition, any complicated calculation will increase latency, so we apply methods as fast and simple as possible. 
     E-2-3-1. Detect Static Motion (S 1002 ) 
     When a user is static, jitter is most obvious. Thus, static jitter is removed first. 
     If user is detected as being static (S 1002 ), the previous IMU pose is copied as the current IMU pose (S 1006 ). The current IMU pose will then be converted to pose in object coordinate using equation (15) for display (S 1008 ). 
     Using this function, static jitter is completely reduced. 
     Except for static jitter, many people observed jitter during moving, we call it motion jitter. The next few modules are for reducing the motion jitter. 
     On the other hand, if the user is not static, the IMU pose is then predicted (S 1010 ) and the IMU pose is then converted to an object pose (S 1012 ), as discussed herein. The method  1000  then may proceed to step S 1014 , discussed below. 
     E-2-3-2. Detect Motion Jitter (S 1014 ) 
     In step S 1014 , motion jitter is detected when the pose difference between the current IMU pose and the previous IMU pose is small (e.g., less than or equal to a predefine threshold). 
     When motion jitter is identified, it is processed in a similar way to when static motion is identified, i.e., copy the previous pose into the current pose (S 1006 ). If both static jitter and motion jitter are not detected, the CPU  140  can apply exponential smoothing (S 1018 ) to reduce the jitter and the smoothing factor can be set based on the pose difference. The simplest form of exponential smoothing is given by the formula,
 
 s   t   =α·x   t +(1−α)· s   t-1   (16)
 
     Where α is the smoothing factor, and 0&lt;α&lt;1. 
     The smoothed pose is then copied as the display pose under step S 1020 . 
     E-2-4. Power Saving 
     AR systems such as the HMD  100  generally have see-through displays and track real-world objects in a field of view of a user and camera. The derived location of these tracked objects is used to e.g. overlay images on top of the tracked objects. These real-world objects may be three-dimensional or planar, and the overlaid images may also be three-dimensional computer graphics or two-dimensional (e.g. text). Embodiments of the AR systems contain the following components:
         Camera,   IMU sensors,   AR algorithm,   Application rendering, and   Display.       

     In the normal use of AR systems, internal components are running at certain rate, for example, if the display refresh rate is 60 Hz, then application rendering also happens at 60 Hz. The camera provides images into AR system for object detection, tracking and scene construction, which can be running at its own frame rate. IMU sensors provide inertial measurement data which can be used to determine mobile device motion. Even though the underlying hardware may have its own power saving schemes, software can help reduce the power consumption if it can determine some components are idle at certain moments and shutting off those components. 
     Embodiments of the AR system render some virtual content with the motion of the device or images of the real world from camera. As a result of this operational configuration, the following phenomena can be exploited to save power:
         1. Motion determines virtual contents rendering: Without motion, new virtual content is not rendered. Thus, embodiments of the AR system can achieve power savings by stopping rendering in response to a lack of motion, which reduces CPU (central processor) and GPU (graphics processor) usage and therefore save power.   2. Motion determines AR algorithm calculation: Without motion, it is probably not necessary to constantly run the AR algorithm to keeping tracking objects or constructing 3D scene. Normally, the AR algorithm is running to determine object poses and the 3D scene based on the images from cameras and IMU data. Without motion, the objects in the camera images stay in the same location, and their poses will not change. Therefore, embodiments of the AR system can use previous poses by the application without any new calculations, which also results in reduced CPU and GPU usage to save power.   3. Motion determines physical display properties: Without motion, the physical display can be controlled to reduce power consumption under certain conditions. In the mobile devices, the physical display plays an important role in battery consumption, depending on the brightness settings and display refresh rate. In the case of no motion in AR system, some display properties can be changed to save power, and original settings can be restored once motion is restarted.       

       FIG. 28  shows a power saving module  2800  according to one embodiment. In some embodiments, power saving module  2800  is a software module that is configurable for some settings, and can be scheduled to run periodically to determine motions, and output power saving instructions  2830  to its controlled software and hardware components (outputs  2820 ). In other embodiments, power saving module is a processor (e.g. CPU  140 ) configured to perform a power saving method when executing software instructions. In still other embodiments, power saving module  2800  is an electronic circuit. 
     Power saving module  2800  can take inputs from cameras, sensors, AR algorithm and other hardware components, labeled as inputs  2810 . The images from cameras provide snapshot of real world at a time and the differences between consecutive images depict the motion of tracked objects. The IMU data from different kinds of sensors provide other ways to determine the motion of mobile device. 
     The output from AR algorithm can be fed into the power saving module  2800  so that power saving instructions can differ based on whether or not the object is actually being tracked. This is shown as the AR feedback  2840 , and allows the power saving module to perform different power saving schemes depending on the operational state of the AR algorithms. 
     Several kinds of motions calculated in the power saving module  2800  can be used to determine the final output power saving instructions, which will be further broadcasted to other software or hardware components to change their operation to save battery power. 
     The motions calculated in the power saving module  2800  can be classified into different levels, which can be used to generate power saving instructions  2830 . These events normally represent different levels of power saving, range from no power saving to more aggressive power saving. 
     The power saving module  2800  will send power saving instructions  2830  depending on the power saving mode to different software and hardware components  2820 , which are configured to handle power saving instructions  2830  in a variety of ways. 
     E-2-4-1. Movement Detection 
     In some embodiments, sensors (such as the IMU) on the display apparatus (e.g. HMD  100 ) provide data at higher rate than cameras. Device motion can be estimated using sensor data with less calculation and less power consumption than as with image data. Because sensor data may contain some jitters, a sensor data buffer can be used as sliding window, together with a low-pass filter, to output reliable data, which will be used to estimate device motion. This is shown in  FIG. 29 . The calculated motion value is then compared to a pre-defined threshold for this device to determine whether or not motion is detected. 
     In the motion detection shown in  FIG. 29 , a sensor data buffer  2900  with n elements contains real-time sensor data. When the power saving module  2800  is scheduled to run, the data in the buffer  2900  will first go through a low-pass filter  2910  to generate relatively reliable data, which can be used to perform motion calculation  2920 . A simple weighted-averaging filter can be used to reduce CPU load with less calculation. The output from this low-pass filter  2910  is fed into motion calculation block  2920  to generate estimated motion value. The norm of sensor data vector can be used as motion indicator and comparing with pre-defined threshold yields final motion events as shown in blocks  2930 ,  2932 , and  2934 . 
       FIG. 30  shows a method of saving power using motion detection based on sensor data. In this method, motion of a display apparatus is calculated based on data from a motion sensor such as an IMU  71 . The method of claim  30  would be implemented while the display apparatus is continuously tracking a pose of an object in a field of view of its camera. This tracking includes repeatedly deriving, using a processor (e.g. CPU  140 ) the pose of the object relative to the display apparatus using at least one of image data acquired by the camera and sensor data acquired by the sensor, and displaying using the display (i.e. image display section  20 ) an image based on the derived pose of the object (i.e. the target object). The image may be a 3D computer graphics model or 2D comment balloon, for example. 
     The power saving module  2800  may work in conjunction with a graphics processor (GPU), such as image processor  160 , or a discreet GPU that drives the display. This GPU renders the image (or AR object) being displayed on the display. This AR object is displayed based on the detected pose of the tracked object. For example, it is displayed to be overlaid on (and follow) the tracked object from the user&#39;s perspective. In another embodiment, the AR object is offset from the tracked object, and follows it. In another embodiment, the location of the AR object on the display does not correspond to the tracked object pose, but changes in response to a change in pose. The image or AR object is repeatedly rendered by the GPU and each new rendering is sequentially displayed on the display, so that the image can change over time in response to the object pose changes. 
     In step S 300  of  FIG. 30 , sensor data is obtained, e.g. from an IMU or a low-pass filter  2910 . The sensor data may be obtained from IMU  71  discussed above in  FIGS. 17-18 . As mentioned above, the sensor data from the IMU  71  may determine if the user is moving his/her head relative to an object or a scene. 
     Next, sensor motion is calculated in step S 301 . In this embodiment, the speed and/or amount of the motion of the display apparatus is derived in S 301  based on the sensor data obtained in step S 300 . This speed and/or amount of motion can be calculated by taking acceleration measured in the IMU and integrating it over a time period. In this regard, the speed and/or motion amount is determined on a continuous time basis to determine if the display apparatus (and thus, the user&#39;s head) has moved. 
     In step S 302 , the motion derived in step S 301  is compared to a predetermined motion threshold. This predetermined motion threshold can be set based on a number of factors, including optimizing the threshold for a desired sensitivity for entering a power saving mode. The predetermined motion threshold may be preset prior to using the HMD (e.g., by a user, manufacturer, etc.) and can be set to any desired value based on the sensitivity preferred. 
     In any event, if the motion derived in S 301  exceeds the predetermined motion threshold, power saving mode is exited and the method is restarted in S 303 . In other words, it is determined that there is sufficient motion to continue operation, and e.g. object tracking, and thus, full power is delivered to the HMD components to continue normal operation for determining and tracking pose. If the motion derived in S 301  does not exceed the threshold, the method proceeds to S 304  and enters one of two kinds of power saving mode, as discussed below with respect to steps S 304 -S 306 . 
     In step S 304 , the method determines whether the display apparatus is currently tracking an object or not (as previously discussed herein). If the display apparatus is in a tracking mode and tracking an object, the system recognizes that power is being consumed unnecessarily and thus, at step S 305 , on-tracking power saving mode is entered. On-tracking power saving mode may be simply stop pose tracking and instead use the previous pose determined (i.e., the pose determined before pose tracking is turned off). This may entail stopping the tracking algorithm. This is useful because there is no sensor motion and thus, the object is still. In this embodiment, the user may be able to use a previous pose because the sensor has not moved. 
     If the display apparatus is not in a tracking mode, at step S 306 , off-tracking power saving is entered, whereby pose tracking altogether can be stopped and/or the display can be turned off. Because tracking is not being performed, off-tracking power saving can be a more aggressive power saving mode than on-tracking power saving. For example, off-tracking power saving may disable more devices or software processes, or disable these components more quickly, than on-tracking power saving. 
       FIG. 31  shows how the power saving module  2800  can determine that there has been motion using camera image data of the display apparatus. Power saving module  2800  compares two frames of image data to determine if there has been motion of the display apparatus and/or motion of objects within a field of view of the camera. In order to do this, power saving module  2800  compares the same pixel, region, or group of pixels of two frames separated in time (in some embodiments, consecutive frames). If there is a difference in the image data for the compared portions of the frames, this is measured to determine an amount of motion (e.g. the speed of the motion). The speed of the motion can be calculated by determining a spatial difference between frames of a location of an object in the field of view, and dividing the time separating when the two frames were captured. 
     For example, image motion can be described as a function of differences between two consecutive camera images. Given two consecutive camera images F k  and F k-1 , the motion can be described as:
 
 M =ƒ( F   k   −F   k-1 )
 
Because calculating the motion on the full size image can consume much battery power with heavy CPU calculations, a block-based image motion detection is used in these embodiments to reduce overall CPU loads. According to such embodiments, the images are divided into non-overlapping blocks  3120 . Differences between each pair of blocks  3120  at same location on two frames are calculated.  FIG. 31  shows the image blocks  3120  and pairs of blocks  3130  on two frames ( 3100 ,  3110 ). To simplify the motion calculation, if there are n blocks  3120  on each image, then motion can be calculated as:
 
     
       
         
           
             M 
             = 
             
               
                 ∑ 
                 
                   i 
                   = 
                   0 
                 
                 n 
               
               ⁢ 
               
                 f 
                 ⁡ 
                 
                   ( 
                   
                     
                       B 
                       
                         ( 
                         
                           
                             k 
                             - 
                             1 
                           
                           , 
                           i 
                         
                         ) 
                       
                     
                     - 
                     
                       B 
                       
                         ( 
                         
                           k 
                           , 
                           i 
                         
                         ) 
                       
                     
                   
                   ) 
                 
               
             
           
         
       
     
     Where function ƒ( ) is defined as Euclidian distance between two blocks: 
               f   ⁡     (       B     (       k   -   1     ,   i     )       -     B     (     k   ,   i     )         )       =         ∑     p   =   0     m     ⁢       (       I     (       k   -   1     ,   i     )     p     -     I     (     k   ,   i     )     p       )     2               
Where I (k,i)   p  represents the intensity of pixel p in block B (k,i)  in image F k . After an amount of motion M is calculated, it is compared with pre-defined threshold for final output, as described further with respect to  FIG. 32  below.
 
       FIG. 32  shows a method of saving power using motion detection based on image data. In this method, movement of a display apparatus (or movement in a field of view of the camera) is calculated based on image data from the camera. 
     In step S 320 , image data is obtained (e.g. frames  3100 ,  3110 ) from the camera, such as camera  60  (as previously discussed herein). 
     Next, image movement is calculated in step S 321 . In this step, the speed or amount of movement of the display apparatus (or speed or amount of the movement of an object in the field of view) is detected in S 321  based on the image data obtained in step S 320 . The calculation of image movement is discussed above with regards to  FIG. 31 . 
     Next, in step S 322 , the movement derived in step S 321  is compared to a predetermined image movement threshold. This predetermined image movement threshold can be set based on a number of factors, including optimizing the predetermined image movement threshold for a desired sensitivity for entering power saving mode. If the movement derived in S 321  exceeds the predetermined image movement threshold, the method proceeds to S 303 , where power saving mode is exited and the method may then proceed back to step S 320 . In other words, the system determines that there is sufficient movement to continue operation, and e.g. object tracking. If the movement derived in S 321  does not exceed the predetermined image movement threshold, the method proceeds to S 304  and enters one of two kinds of power saving mode, as discussed below with respect to steps S 304 -S 306 . 
     In step S 304 , the method determines whether the display apparatus is in a tracking mode or not (“on tracking”). If the display apparatus is in a tracking mode, at step S 305  on-tracking power saving mode is entered. If the display apparatus is not in a tracking mode (“off tracking”), step S 306  is performed, and off-tracking power saving is entered. 
     In some embodiments, the determination of on-tracking or off-tracking can be made based on whether the object being tracked moves. For example, if the movement derived in step S 321  is movement of an object being tracked but is below the threshold in S 322 , the method enters on-tracking power saving (i.e. S 305 ). However, if there is no detected movement of the tracked object, movement is derived elsewhere in the field of view, and this movement is below the threshold, the method enters off-tracking power saving (i.e. S 306 ). 
     In some embodiments, both sensor data and image data are used to detect motion and determine whether a power saving mode should be entered. Such an embodiment is shown in  FIG. 33 . In this method, the steps from  FIGS. 30 and 32  are also performed as part of the method. First, the motion calculations based on the sensor data (S 302 ) is performed. If the sensor data motion exceeds the threshold, power saving is exited. 
     If the sensor data motion does not exceed the threshold, image movement is calculated in S 321 . In this way, the sensor data is used as an initial test to determine if the display apparatus us moving. If so, power saving is exited (S 303 ). If not, the method proceeds to analyzing image data for movement at S 321 . This step is undertaken so that the display apparatus does not enter a power saving mode due to a user being still while using the display apparatus, particularly while moving objects in the field of view are being tracked. 
     At S 322 , after image movement is calculated in step S 321 , it is determined whether the image movement exceeds the image movement threshold. If so, power saving is exited. If not, it is determined whether the display apparatus is on-tracking or off-tracking in step S 304 , to decide which power saving mode to enter (i.e., either “on-tracking power saving mode” and “off-tracking power saving mode”). “Power Saving (On tracking)” (also referred to herein as “on-tracking power saving mode”) and “Power Saving (Off tracking)” (also referred to herein as “off-tracking power saving mode”) are discussed later under Section E-2-4-2. 
       FIG. 34  shows another embodiment of the power saving method. In this embodiment, steps S 340  and S 341  are performed before step S 302  (determining whether sensor motion exceeds a sensor motion threshold). In step S 340 , it is determined whether the sensor motion exceeds a second predetermined motion threshold that is higher than the previously discussed first sensor predetermined motion threshold. If the sensor motion exceeds the second predetermined motion threshold, S 341  is performed, whereby a searching power saving mode is entered (the “searching power saving mode” is discussed later in more depth under Section E-2-4-2). If the sensor motion does not exceed the second predetermined motion threshold, S 302  and the remainder of the power saving method of  FIG. 33  is performed. 
     The purpose of determining whether the sensor motion exceeds a second predetermined motion threshold is to determine if the display apparatus has been moved quickly, such as by a rapid or heavy amount movement by the user. In this situation, it is unlikely that the AR algorithm and tracking software would be able to maintain tracking of the object being tracked. This may be due to the object leaving the field of view or motion blur of the captured images. Thus, the display apparatus can save power by disabling tracking when it detects these large movements. Also, the display apparatus can enter a searching mode where it attempts to locate the object that was previously tracked, or locate another object that is suitable to track. 
     E-2-4-2. Power Saving Modes 
     As discussed above, the display apparatus can enter different types of power saving modes according to the methods described herein. These modes include the on-tracking power saving mode (S 305 ), the off-tracking power saving mode (S 306 ), and the searching power saving mode (S 341 ). In each mode, different actions may be taken at different times. 
     The power-saving instructions generated in the power saving module  2800  can be sent to other software and hardware components, which can then be handled individually depending on the functionalities of those components. The following functions of the display apparatus can be disabled or altered in operation in response to low or absent movement: 
     Software Rendering
         In power-saving modes, the virtual contents rendering can be disabled in response to no sensor motion and/or image motion. Previously rendered graphics, stored for example in a frame memory area of RAM in the controller  10 , can be reused in place of newly rendered graphics in this scenario.       

     AR Algorithm
         In power-saving modes, the AR algorithm can be disabled or report a previously calculated pose, stored for example in a memory area of RAM in the controller  10 , to reduce CPU usage, which reduces power consumption.       

     Display
         In some power-saving modes, the display can be dimmed to reduce brightness or even completely turned off, which reduces power consumption.       

     Other Software Components:
         Other software components in the AR system can also receive power saving instructions and handle them accordingly to reduce CPU usage and power consumption.       

     In some embodiments, the power saving operations are:
         1. Stop tracking an object pose (1) while rendering and/or displaying AR object (e.g. 3D computer graphics model or 2D comment balloon) with a previously derived pose of the object stored for example in a memory area of RAM in the controller  10 , or (2) while displaying an AR object using an image data stored for example in a frame memory area of RAM in the controller  10 , obtained from previous rendering.   2. Stop rendering the AR object while displaying an AR object using an image data stored for example in a frame memory area of RAM in the controller  10 , obtained from previous rendering.   3. Stop displaying the AR object.       

     Operation 1 is freezing the location of the displayed AR object based on a previous pose. This allows for power savings by no longer actively tracking the pose of the tracked object. If the display apparatus is not moving for a set period, a user may not notice that tracking has been disabled, because the displayed AR object remains aligned with (or otherwise accurately represents) the pose of the object. 
     Operation 2 is the disabling of the GPU rendering of the AR object. This means that a previous rendering is used, and the object appearance is not updated based on object pose. Similarly as with operation 1, this may not be noticed by a user and can reduce power by disabling operations of the GPU. 
     Operation 3 is ceasing of the display of the AR object altogether. This operation would be noticed by a user, so it may be implemented at a time when it is assumed that the display apparatus is no longer in use for tracking, or in use at all. 
     Depending on the power saving mode, operations 1, 2, and 3 may or may not be implemented at various times. For example, in one embodiment of on-tracking power saving mode (S 305 ):
         If the movement (sensor and/or image) is below the threshold for time 1, operation 1 is performed; and   If the movement is below the threshold for time 2 (which is longer than time 1), operations 1 and 2 are performed.       

     In one embodiment of off-tracking power saving mode (S 306 ):
         If the movement is below the threshold for time 1, operation 1 is performed;   If the movement is below the threshold for time 2 (which is longer than time 1), operations 1 and 2 are performed; and   If the movement is below the threshold for time 3 (longer than time 2), operations 1, 2, and 3 are performed.       

     Accordingly, the aggressiveness of the power saving operations are determined by the power saving mode. In other embodiments, time delay before implementing operations (i.e. times 1, 2, and 3) can change depending on the power saving mode. Furthermore, the delay (times 1, 2, and 3) can be set based on a number of factors, including optimization to enter the desired power saving mode in a way that maximizes power savings while minimizing effects on user experience. 
     In the searching power saving mode, operations 1, 2, or 3 may be implemented, and additionally the CPU or power saving module  2800  may perform operations to try to find the object or another object suitable for tracking. 
     In another embodiment, the searching power saving mode simply disables the system from searching for an object that the system desires to track. 
     After entering a power saving mode and/or implementing the power saving operations 1, 2, or 3, the display apparatus can continue to monitor motion using sensor and image data. If it is determined that movement of the display apparatus (or of objects in the field of view) exceed a threshold, then object pose tracking can be restarted. 
     E-3. Experiment Results 
     The proposed IMU and 3D object tracking fusion method discussed above reduces the latency perceived by a user, allows for faster user head motion without compromising perception of the pose relative to the object, and handles vision loss. It improves the overall tracking performance and users&#39; experience. Some experimental results are shown in sections below. 
     E-3-1. Latency Improvement 
     Significant latency reduction is observed using the proposed method and the latency measurement results are given in table 1. The latency is about 32 ms with IMU, compared latency of (128 ms-140 ms) with the original tracker without using IMU. 
     
       
         
           
               
             
               
                 TABLE 1 
               
             
            
               
                   
               
               
                 Latency of 3D object tracking with/without IMU 
               
            
           
           
               
               
               
            
               
                   
                 Latency 
                 Motion Speed (mm/s) 
               
            
           
           
               
               
               
               
               
               
            
               
                   
                 (ms) 
                 350 
                 250 
                 150 
                 100 
               
               
                   
               
               
                   
                 No IMU 
                  140 ms 
                 139.2 ms 
                 135.4 ms 
                 128.7 ms 
               
               
                   
                 With IMU 
                 32.5 ms 
                  31.9 ms 
                  32.7 ms 
                  32.8 ms 
               
               
                   
               
            
           
         
       
     
     The latency improvement is observed obviously with the HMD with IMU. Devices prior to the present invention could get no lower than 100 ms, but the latency of an embodiment of the present disclosure is able to be around 32 ms. 
     E-3-2. Reduce Tracking Drift 
     With IMU fusion, no drift tracking is achieved. This is true for tracking with IMU for a rich feature object and for a low feature object. For both objects, the addition of the IMU, as discussed herein, improves tracking performance. 
     E-3-3. Tolerate Faster User Motion 
     As mentioned above, the IMU predicted pose is used to predict the feature location in the current frame before feature matching, which leads to faster and more accurate feature matching, and tolerate big inter-frame motion. 
     Also, as mentioned above, the IMU predicted pose is used as the initial pose for robust pose estimator, which leads to faster convergence and more accurate pose estimated. 
     Accordingly, the proposed method tolerates faster user motion. 
     E-3-3 Summary 
     The following features make the proposed approach unique and perform better than the prior 3D object tracking technology for AR applications on wearable devices. 
     The IMU and 3D object tracking fusion framework to fuse IMU sensor and our 3D object tracking technology is designed to maximize the 3D object tracking performance. 
     Two threads of IMU thread and vision thread runs on the HMD device, and the pose is updated at IMU output frequency, which reduces the optical see-through latency. 
     The IMU predicted pose is used as an initial pose for robust pose estimator, which improves the pose accuracy and pose estimation speed. 
     The IMU predicted pose is used as the input object pose for edge refinement module in case the previous pose estimation from feature matching fails, which improves the pose accuracy, pose estimation speed, and reduces the rich feature requirement so less feature of objects/views can be tracked. 
     The IMU pose is used to update vision tracker pose when tracker fails and IMU is not diverged, which reduces the tracking loss. 
     An automatic method to detect static motion and initialize the IMU fusion is disclosed. 
     A two step of jitter reduction method (static jitter reduction+motion jitter reduction) is disclosed to not only reduce jitter but also maintain the latency improvement. 
     The tracking speed is improved because when user is detected as static tracker just uses the pose from last frame without actually tracking the current frame. 
     With IMU prediction, less features can be used during tracking, the tracking speed is improved. 
     With IMU prediction, other complicated features can be considered since feature matching becomes faster and more accurate. 
     These features make the proposed technology improve the 3D object tracking speed, accuracy and latency, reduce tracking loss, reduce the limitation for user moving speed, and improve users&#39; experience for AR applications on wearable devices. 
     E-3-4 Spatial Relationships in Multiple Devices 
     In the above described embodiment, the IMU fusion initialization in step S 612  ( FIG. 23 ) provides data representing the spatial relationship of the object with respect to the scene. In another embodiment, such spatial relationships may be stored and used by multiple devices. According to such embodiment, the CPU  140  in different HMDs  100 ′ can obtain the position and pose of an object with respect to the same scene, even if the object is outside the field of view of the camera  60 . Then the CPU  140  displays to its user visual information on the position and/or pose of the object using the different HMD  100 &#39;s position and pose in the global coordinate system, provided that the global coordinate system is set common to these HMDs  100 ′. For that purpose, it is preferable that each of the HMD  100 ′ (e.g., first device) and different HMD  100 ′ (e.g., second device) includes a GPS (global positioning system) sensor and/or IMU  71  to define and share the global coordinate system with each other. If the IMU  71  includes a magnetic sensor in addition to the accelerometer and gyroscope, the CPU  140  can also define such global coordinate system common to the multiple devices without the GPS sensor, using, for example, the NED (North-East-Down) coordinate system. In this manner, the calculation power of the different HMD  100 ′ to obtain the object pose in the scene, or the global coordinate system, is saved. 
     For example, a camera acquires an image data sequence and an inertial sensor acquires a sensor data sequence. As discussed above, the inertial sensor is fixed or adjustably fixed in a first spatial relationship with respect to the camera. 
     The CPU  140  of a first device such as HMD  100 ′ derives a first pose, or a vision object pose, for each of the objects based at least on one of: image frames in the image data sequence and template data created based on 3D models corresponding respectively to the objects. The CPU  140  of the first device also derives respective second spatial relationships between object coordinate systems defined respectively on the objects or the 3D models and a global coordinate system based at least on the first poses or poses tracked from the first poses, the sensor data sequence and the first spatial relationship. 
     The second spatial relationships is stored in a storage medium so that the second spatial relationships are available to devices (i.e., devices, such as the second device, other than the first device) accessing a computer capable of communicating with the storage medium. For example, the second spatial relationships may be stored in a server which is accessible by all of the devices over a network. In this regard, the poses and spatial relationships can be used by these other devices to determine a spatial relationship to the other devices. 
     As will be appreciated by one skilled in the art, aspects of the present invention may be embodied as a system, method or computer program product. Accordingly, aspects of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a “circuit,” “module” or “system.” Furthermore, aspects of the present invention may take the form of a computer program product embodied in one or more computer readable medium(s) having computer readable program code embodied thereon. 
     Any combination of one or more computer readable medium(s) may be utilized. The computer readable medium may be a computer readable signal medium or a non-transitory computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the non-transitory computer readable storage medium would include the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a non-transitory computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. 
     A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may be any computer readable medium that is not a non-transitory computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. 
     Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing. Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the “C” programming language or similar programming languages. The program code may execute entirely on the user&#39;s computer, partly on the user&#39;s computer, as a stand-alone software package, partly on the user&#39;s computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user&#39;s computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider). 
     Aspects of the present invention are described above with reference to flowchart illustrations and block diagrams of methods, apparatuses (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. 
     These computer program instructions may also be stored in a computer readable medium that can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks. 
     The computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. 
     The flowcharts and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the Figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions. 
     The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of embodiments of the invention. As used herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “has,” “have,” “having,” “includes,” “including,” “comprises” and/or “comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. 
     The corresponding structures, materials, acts, and equivalents of all means or step plus function elements in the claims below are intended to include any structure, material, or act for performing the function in combination with other claimed elements as specifically claimed. The explicit description of the present invention has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to embodiments of the invention in the form explicitly disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of embodiments of the invention. The embodiment was chosen and described in order to best explain the principles of embodiments of the invention and the practical application, and to enable others of ordinary skill in the art to understand embodiments of the invention for various embodiments with various modifications as are suited to the particular use contemplated. 
     Although specific embodiments have been illustrated and described herein, those of ordinary skill in the art appreciate that any arrangement which is calculated to achieve the same purpose may be substituted for the specific embodiments shown and that embodiments of the invention have other applications in other environments. This application is intended to cover any adaptations or variations of the present invention. The following claims are in no way intended to limit the scope of embodiments of the invention to the specific embodiments described herein.