Patent Publication Number: US-2023133574-A1

Title: Augmented reality (ar) device and method of predicting pose therein

Description:
CROSS-REFERENCE TO RELATED APPLICATION 
     This application is a continuation of U.S. application Ser. No. 17/722,848, filed on Apr. 18, 2022, which is a continuation of U.S. application Ser. No. 17/037,102, filed on Sep. 29, 2020, now U.S. Pat. No. 11,334,747, issued on May 17, 2022, which claims priority to Korean Patent Application No. 10-2020-0046268, filed on Apr. 16, 2020, in the Korean Intellectual Property Office, the disclosures of which are incorporated herein in their entirety by reference. 
    
    
     BACKGROUND 
     1. Field 
     Example embodiments of the disclosure relate to an augmented reality (AR) device and a method of predicting a pose in the AR device. 
     2. Description of Related Art 
     With the recent technological developments, various types of wearable devices that can be worn on the human body have been developed. A glasses-type wearable device is worn on a head of a user and may provide an augmented reality service to the user by displaying visual information about a virtual object on a display of the device. 
     According to display technologies such as augmented reality (AR) technology, an image output on a device may be perceived by a user as being real. Accordingly, a user may experience situations that cannot be encountered in the real life as being real through AR technology display device or the like. In order to facilitate this experience, it is important that an output image is provided to a user through an AR device in real-time. Moreover, the output image has to be closely related to the pose of a wearable device worn on the user, and therefore accurate detection of a user&#39;s pose is required. When an output image is not displayed in real time, a user may feel uncomfortable due to an error caused by a difference between a detected pose and the output image. Accordingly, accurate pose prediction methods using various digital processing technologies such as computer vision or simultaneous localization and mapping (SLAM) are needed. 
     SUMMARY 
     Provided is an augmented reality (AR) device and a method of predicting a pose in the AR device. The technical objectives to be achieved by the disclosure are not limited to the above-described objectives, and other technical objectives may be inferred from the following embodiments. 
     Additional aspects will be set forth in part in the description which follows and, in part, will be apparent from the description, or may be learned by practice of the presented embodiments of the disclosure. 
     According to an aspect of the disclosure, there is provided a method of predicting a pose in an augmented reality (AR) device, the method comprising: obtaining, using an inertial measurement unit (IMU) sensor provided in the AR device, IMU values corresponding to a movement of the AR device at an IMU rate of a first frequency; estimating intermediate 6-degrees of freedom (6D) poses of the AR device based on the obtained IMU values and images surrounding the AR device, the images being obtained at a frame rate of a second frequency by a camera provided in the AR device; and generating, by a processor, a pose prediction model for predicting relative 6D poses of the AR device by performing learning based on the obtained IMU values and the intermediate 6D poses using a deep neural network. 
     According to another aspect of the disclosure, there is provided a non-transitory computer readable recording medium having recorded thereon a program for executing, in a computer, a method of predicting a pose in an augmented reality (AR) device, the method comprising: obtaining, using an inertial measurement unit (IMU) sensor provided in the AR device, IMU values corresponding to a movement of the AR device at an IMU rate of a first frequency; estimating intermediate 6-degrees of freedom (6D) poses of the AR device based on the obtained IMU values and images surrounding the AR device, the images being obtained at a frame rate of a second frequency by a camera provided in the AR device; and generating, by a processor, a pose prediction model for predicting relative 6D poses of the AR device by performing learning based on the obtained IMU values and the intermediate 6D poses using a deep neural network. 
     According to another aspect of the disclosure, there is provided an augmented reality (AR) device comprising: an inertial measurement unit (IMU) sensor configured to obtain IMU values corresponding to a movement of the AR device at an IMU rate of a first frequency; a camera configured to obtain images surrounding the AR device at a frame rate of a second frequency; a visual-inertial simultaneous localization and mapping (VI-SLAM) module configured to estimate intermediate 6-degrees of freedom (6D) poses of the AR device based on the obtained IMU values and the images surrounding the AR device; and a processor configured to generate a pose prediction model for predicting relative 6D poses of the AR device by performing learning based on the obtained IMU values and the intermediate 6D poses using a deep neural network. 
     According to another aspect of the disclosure, there is provided a method of predicting a pose in a processor, the method comprising: obtaining, by using an inertial measurement unit (IMU) sensor provided in an augmented reality (AR) device, IMU values corresponding to a movement of the AR device at an IMU rate of a first frequency; obtaining intermediate 6-degrees of freedom (6D) poses of the AR device estimated based on the obtained IMU values and images surrounding the AR device, the images being obtained at a frame rate of a second frequency using a camera provided in the AR device; and generating a pose prediction model for predicting relative 6D poses of the AR device by performing learning based on the obtained IMU values and the intermediate 6D poses using a deep neural network. 
     According to another aspect of the disclosure, there is provided a computer-implemented method of training a neural network for pose prediction comprising: collecting inertial measurement unit (IMU) values corresponding to a movement of an augmented reality (AR) device, the IMU values being collected at a first frequency; collecting images surrounding the AR device, the images collected at a second frequency; performing one or more operations on the collected IMU values and the collected images to create intermediate 6-degrees of freedom (6D) poses of the AR device; and training the neural network based on the collected IMU values and the intermediate 6D poses. 
     According to another aspect of the disclosure, there is provided an apparatus for training a neural network for pose prediction comprising: a memory storing one or more instructions; and a processor configured to execute the one or more instructions to: collect inertial measurement unit (IMU) values corresponding to a movement of an augmented reality (AR) device, the IMU values being collected at a first frequency; collect images surrounding the AR device, the images collected at a second frequency; perform one or more operations on the collected IMU values and the collected images to create intermediate 6-degrees of freedom (6D) poses of the AR device; and train the neural network based on the collected IMU values and the intermediate 6D poses. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The above and other aspects, features, and advantages of certain example embodiments of the disclosure will be more apparent from the following description taken in conjunction with the accompanying drawings, in which: 
         FIG.  1    illustrates an augmented reality (AR) device according to an example embodiment; 
         FIG.  2    is a block diagram of electronic hardware provided in the AR device according to an example embodiment; 
         FIG.  3    is a timing diagram of time points for obtaining inertial measurement unit (IMU) values via an IMU sensor according to an example embodiment; 
         FIG.  4    is a timing diagram of time points for obtaining images via a camera according to an example embodiment; 
         FIG.  5 A  is a timing diagram of time points for estimating an intermediate six degrees (6D) pose via a VI-SLAM module according to an example embodiment; 
         FIG.  5 B  is a timing diagram of time points for estimating an intermediate 6D pose via a visual-inertial simultaneous localization and mapping (VI-SLAM) module according to another example embodiment; 
         FIG.  6    is a timing diagram of a method of generating a pose prediction model according to an example embodiment; 
         FIG.  7 A  is a timing diagram of a method of performing pose prediction in a pose prediction model by a sliding window method according to an example embodiment; 
         FIG.  7 B  is a timing diagram of a method of performing pose prediction in a pose prediction model by a sliding window method according to another example embodiment; 
         FIG.  8    is a diagram of a pose prediction model based on a deep neural network (DNN) implemented by a recurrent neural network (RNN) using a long short term memory (LSTM) according to an example embodiment; 
         FIGS.  9  and  10    are block diagrams of performing pose prediction in a processor according to some example embodiments; 
         FIGS.  11 A and  11 B  are time diagrams of performing inference of pose prediction in a pose prediction model based on a DNN that has completed learning according to some example embodiments; and 
         FIG.  12    is a flowchart of a method of predicting a pose in an AR device according to an example embodiment. 
     
    
    
     DETAILED DESCRIPTION 
     Reference will now be made in detail to embodiments, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to like elements throughout. In this regard, the some example embodiments may have different forms and should not be construed as being limited to the descriptions set forth herein. Accordingly, the example embodiments are described below, by referring to the figures, to explain aspects. As used herein, the term “and/or” includes any and all combinations of one or more of the associated listed items. Expressions such as “at least one of,” when preceding a list of elements, modify the entire list of elements and do not modify the individual elements of the list. 
     The terms used in the example embodiments have been selected from currently widely used general terms in consideration of the functions in the disclosure. However, the terms may vary according to the intention of one of ordinary skill in the art, case precedents, and the advent of new technologies. Furthermore, for special cases, meanings of the terms selected by the applicant are described in detail in the description section. Accordingly, the terms used in the disclosure are defined based on their meanings in relation to the contents discussed throughout the specification, not by their simple meanings. 
     In the example embodiments, while such terms as “first,” “second,” etc., may be used to describe various components, such components must not be limited to the above terms. The above terms are used only to distinguish one component from another. 
     The terms used in some embodiments are merely used to describe example embodiments, and are not intended to limit the disclosure. An expression used in the singular encompasses the expression of the plural, unless it has a clearly different meaning in the context. Furthermore, in the example embodiments, when a constituent element “connects” or is “connected” to another constituent element, the constituent element contacts or is connected to the other constituent element not only directly, but also electrically through at least one of other constituent elements interposed therebetween. When a part may “include” a certain constituent element, unless specified otherwise, it may not be construed to exclude another constituent element but may be construed to further include other constituent elements. 
     The “above” and similar directives used in relation to the example embodiments, in particular, in claims, may refer to both singular and plural. Furthermore, the operations or steps of all methods according to the example embodiments described herein can be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by context. The example embodiments are not limited to the description order of the described operations or steps. 
     The disclosure may be described in terms of functional block components and various processing steps. Such functional blocks may be realized by any number of hardware and/or software components configured to perform the specified functions. For example, the functional blocks related to the example embodiments may be implemented by one or more microprocessors or by circuit elements for certain functions. Furthermore, the function blocks may be implemented with various programming or scripting languages. Furthermore, the functional blocks may be implemented in algorithms that are executed on one or more processors. The example embodiments could employ any number of related art techniques for electronics configuration, signal processing and/or control, data processing and the like. 
     Furthermore, the connecting lines, or connectors shown in the various figures presented are intended to represent functional relationships and/or physical or logical couplings between the various elements. It should be noted that many alternative or additional functional relationships, physical connections or logical connections may be present in a practical device. 
     The example embodiments of the disclosure are described below in detail with reference to the accompanying drawings. However, the example embodiments are not limited thereto and it will be understood that various changes in form and details may be made therein without departing from the spirit and scope of the following claims. 
       FIG.  1    illustrates an augmented reality (AR)  10  device according to an example embodiment. 
     Referring to  FIG.  1   , the AR device  10  may be a wearable electronic device that is worn by a user. For example, the AR device  10  may be a glasses type wearable device, but embodiments are not limited thereto. For example, according to another example embodiment, the AR device  10  may be a wearable device of a different type to be worn on other human body part of a user, for example, an AR helmet. 
     The AR device  10  may provide, through a display on the AR device  10 , an AR service that fuses real world information around a user and digital or virtual object information. For example, when an AR object (or AR scene) is displayed on the eyes of a user through the AR device  10 , the AR object may be displayed together with a scene of the real world seen at a current pose of the AR device  10 . According to an example embodiment, the AR object may be displayed, overlaid or superimposed on the scene of the real world environment perceived by a user at a current pose of the AR device  10 . While the user may see the AR object through the AR device  10 , the AR object does not exist in the real world. 
     The real world is a real scene that may be seen by an observer or a user through the AR device  10 , and may include a real world object. On the other hand, the AR object (or scene) is an image generated by graphics processing and may correspond to a static image or a dynamic image. For example, the AR object (or scene) may be an image that is overlaid on a real scene to provide information about a real object in the real scene, or information or a control menu regarding the operation of the AR device  10 . 
     The AR device  10  may correspond to a head-mounted wearable device, as illustrated in  FIG.  1   , and accordingly may move together when the head of a user moves. Thus, as the pose of the AR device  10  may be continuously changed depending on the movement of the head of a user, it is necessary that the pose of the AR device  10  is continuously updated considering a change in the pose of the head of a user. For example, when a user wearing the AR device  10  walks on the road, a real world scene seen through the AR device  10  is continuously changing. Accordingly, the AR object (or AR scene) provided by the AR device  10  may be re-rendered considering a current pose so that a seamless perspective that the AR object (or AR scene) is present on the real world scene is provided to a user. 
     According to an example embodiment, the AR device  10  may be provided with electronic hardware (HW)  100  including various types of sensing modules such as an inertial measurement unit (IMU) sensor, a simultaneous localization and mapping (SLAM) module, an accelerometer, or a compass, an image capture module such as a camera, a microphone, a GPS module, a communication interface, a processing unit, or a battery. According to an example embodiment, the electronic HW may be provided in a partial inner space of the AR device  10 . The AR device  10  may be further provided with an optical engine or an optical element to display the AR object. Although, for convenience of explanation,  FIG.  1    illustrates that the electronic HW  100  is provided in a frame at one side of the AR device  10 , the location of the electronic HW  100  in the AR device  10  is not limited thereto and the electronic HW  100  may be provided at various locations on the AR device  10  and each of hardware constituent elements may be provided in the same space or in different spaces. 
       FIG.  2    is a block diagram of electronic hardware provided in the AR device according to an embodiment. 
     Referring to  FIG.  2   , the electronic HW  100  may include an IMU sensor  110 , a camera  120 , a visual-inertial simultaneous localization and mapping (VI-SLAM) module  130 , a processor  140 , and a memory  150 . In the electronic HW  100  of  FIG.  2   , hardware constituent elements related to an embodiment of performing pose prediction are mainly illustrated for convenience of explanation. However, the electronic HW  100  may further include other hardware constituent elements such as a microphone, a global positioning system (GPS) module, a communication interface, or an optical engine, or other general-purpose constituent elements according to some example embodiments. 
     The IMU sensor  110  is a sensor for measuring speed, direction, gravity, or acceleration of the AR device  10 . The IMU sensor  110  may measure a free movement of the AR device  10  in a three-dimensional (3D) space by using a gyroscope, an accelerometer, or a geomagnetic field so that acceleration in a proceeding direction, a lateral direction, and a height direction, and rolling, pitching, and yaw angular velocity may be measured. Calculation of a speed, a relative location, or a pose angle of the AR device  10  may be possible by integrating the acceleration and angular velocity obtained from the IMU sensor  110 . 
     The IMU sensor  110  may obtain IMU values corresponding to the movement of the AR device  10  of  FIG.  1    at a frame rate (i.e., an IMU rate) of a first frequency. In other words, temporal sequence of the IMU values may be obtained from the IMU sensor  110 . 
       FIG.  3    is a timing diagram of time points for obtaining IMU values via an IMU sensor according to an example embodiment. Referring to  FIG.  3   , the IMU sensor  110  may measure, at the IMU rate of a first frequency, acceleration in a proceeding direction (X axis), a lateral direction (Y axis), and a height direction (Z axis) and roll, pitch, and yaw angular velocity, and output the measured IMU values. According to an example embodiment, the IMU rate may be 500 MHZ, but embodiments are not limited thereto. In a timing diagram  300  of  FIG.  3   , each of vertical bars denotes a time point when the IMU values are obtained. For example, an IMU value obtained at a time point I may be expressed as IMU i , . . . , an IMU value obtained at a time point L may be expressed as IMUS, an IMU value obtained at a time point L+1 may be expressed as IMU L+1 , and an IMU value obtained at a time point L+2 may be expressed as IMU L+2 , where i and L are natural numbers. 
     Referring back to  FIG.  2   , the camera  120  is an image capture device for photographing a real scene of an environment surrounding the AR device  10 . For instance, the camera  120  may be a front facing camera configured to capture the surrounding in front of the AR device  10 . The camera  120  obtains images around the AR device  10  at a frame rate of a second frequency. In other words, temporal sequence of images of the real scene around the AR device  10  may be obtained from the camera  120 . 
       FIG.  4    is a timing diagram of time points for obtaining images via a camera according to an example embodiment. Referring to  FIG.  4   , the camera  120  obtains surrounding images of the AR devices  10  at the frame rate of second frequency. According to an example embodiment, the frame rate may be 30 fps, but embodiments are not limited thereto. The second frequency at which images are obtained from the camera  120  is lower than the first frequency at which the IMU values are obtained from the IMU sensor  110 . In other words, the IMU rate of a first frequency is a relatively high frequency, and the frame rate of a second frequency is a relatively low frequency. In a timing diagram  400  of  FIG.  4   , each of vertical bars denotes a time point when an image is obtained. For example, an image obtained at a time point 0 may be expressed as I 0 , . . . , an image obtained at a time point n may be expressed as I n , . . . , an image obtained at a time point N−1 may be expressed as I N-1 , and an image obtained at a time point N may be expressed as IN, where n and N are natural numbers. 
     Referring back to  FIG.  2   , according to an example embodiment, the VI-SLAM module  130  is a hardware component (e.g., a hardware processor) that performs a calculation for estimating a relative position of a camera to a map by fusing visual data and inertia data, that is, visual-inertia fusion. 
     When a pose is estimated with only the IMU values at an IMU rate obtained from the IMU sensor  110 , the pose may not be accurate due to noise of the obtained IMU values and slowly varying bias. Furthermore, when a pose is estimated by performing SLAM processing with only the images obtained from the camera  120 , it may be difficult to perform a high speed pose calculation. Accordingly, the VI-SLAM module  130  may improve accuracy of the SLAM processing by performing visual-inertia fusion based pose estimating in a VI-SLAM method of compensating for error values by using the IMU values obtained from the IMU sensor  110  and the images obtained from the camera  120 . 
     For example, a good camera pose may not be calculated due to various factors such as occlusion due to flat areas, for example, walls, poor lighting, motion blur/focus problem, a high speed camera motion, or hand shake. However, in the example embodiment, as learning of camera poses by a visual-inertia fusion based deep neural network (DNN) is performed, the accuracy of the pose prediction of the AR device  10  may be increased even in a situation when an appropriate camera image is not obtained. 
     The VI-SLAM module  130  may estimate intermediate 6-degrees of freedom poses (intermediate 6D poses) of the AR device  10  based on the images (image sequence) obtained by the camera  120  at the frame rate of a second frequency and inputs of the IMU values (IMU sequence) obtained from the IMU sensor  110 . The “intermediate” 6D pose means a pose according to an intermediate calculation result before predicting a final 6D pose (that is, a relative 6D pose). 
     In one embodiment, the VI-SLAM module  130  may perform estimating of the intermediate 6D poses whenever images are obtained at the frame rate of a second frequency. In this case, the estimating of intermediate 6D poses may be performed at the frame rate of a second frequency. In other words, the intermediate 6D poses in this case may be first type pose data of a frame rate of a second frequency estimated by performing the SLAM technique for fusing the IMU values and the images obtained at the frame rate of the second frequency. 
     Furthermore, in another example embodiment, between the time points when images are obtained at the frame rate of a second frequency images, the VI-SLAM module  130  may perform estimating of intermediate 6D poses at the IMU rate of a first frequency by calculating an integral value of angular velocity and an integral value of acceleration of each axis through a pre-integration method from the IMU values between the time points when images are obtained at a frame rate. In other words, the intermediate 6D poses in this case may be second type pose data of the IMU rate including pose data corresponding to the integral values calculated by the pre-integration method from the IMU values and the image based pose data of a frame rate estimated by performing the above-described SLAM technique 
       FIG.  5 A  is a timing diagram of time points for estimating an intermediate 6D pose in a VI-SLAM module according to an example embodiment. Referring to  FIG.  5 A , the VI-SLAM module  130  may receive inputs of the IMU values  520  obtained at the IMU rate and the images obtained at a frame rate, and estimate intermediate 6D poses  510  at a frame rate. 
     In detail, the VI-SLAM module  130  may obtain an intermediate 6D pose P 0  by performing a calculation during a calculation time Δ TRK  with an image I 0  obtained at a time point 0 and the IMU values obtained between a time point A′ 0  and a time point A 0 . Although the calculation of the intermediate 6D pose P 0  is completed at the time point A 0 , the intermediate 6D pose P 0  may be regarded to correspond to a pose at the time point A′ 0 . In the same manner, the VI-SLAM module  130  may obtain each of an intermediate 6D pose P n , an intermediate 6D pose P N-1 , and an intermediate 6D pose P N  through a calculation during the calculation time Δ TRK , and the obtained intermediate 6D pose P n , intermediate 6D pose P N-1 , and intermediate 6D pose P N  may be regarded to correspond to a pose at a time point A′ n , a pose at a time point A′ N-1 , and a pose at a time point A′ N , respectively. As described below, whenever the VI-SLAM module  130  completes the estimation of an intermediate 6D pose, the estimated intermediate 6D pose is updated to a ground truth (GT) at an actual time point, for example, the time point A′ 0 , . . . , the time point A′ n , the time point A′ N-1 , the time point A′ N , when an image is photographed, and may be used for learning. 
       FIG.  5 B  is a timing diagram of time points for estimating an intermediate 6D pose via a VI-SLAM module according to another example embodiment. Referring to  FIG.  5 B , unlike  FIG.  5 A , the VI-SLAM module  130  estimates intermediate 6D poses  515  at the IMU rate. 
     In detail, the VI-SLAM module  130  obtains an integral value of acceleration of each axis and an integral value of angular velocity by performing integration twice on the IMU values  520  obtained at the IMU rate. The integral values may correspond to a 6D pose that is estimated singularly from the IMU values  520 . In other words, the VI-SLAM module  130  predicts intermediate 6D poses ( . . . , P′ n_1 , P′ n_2 , P′ n_3 , . . . P′ N_1 , P′ N_2 , P′ N_3 , . . . ) obtained at the IMU rate based on the IMU integration at time points between intermediate 6D poses (P 0 , P n , P N-1 , P N , . . . ) obtained at a frame rate, by performing pre-integration on the IMU values  520 . As such, as the IMU integration based intermediate 6D poses ( . . . , P′ n_1 , P′ n_2 , P′ n_3 , . . . P′ N_1 , P′ N_2 , P′ N_3 , . . . ) are estimated through pre-integration process {circle around (1)} in addition to the camera image based intermediate 6D poses (P 0 , P n , P N-1 , P N , . . . ) that are estimated through the SLAM process {circle around (2)}, even when a camera image may not be smoothly used due to a problem such as occlusion or blur, the VI-SLAM module  130  may continuously output an estimation result of the intermediate 6D poses  515 . 
     Referring back to  FIG.  2   , the processor  140  is a hardware component that may control the overall operation of the AR device  10  by running an operating system or an application program, and may perform various data processing and operations including pose prediction and AR image generation. The processor  140  may include, for example, central processing units (CPUs), microprocessors, graphic processing units (GPUs), application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), application processors (APs), neural processing units (NPUs), or tensor processing units (TPU), but embodiments are not limited thereto. 
     The processor  140  generates a pose prediction model that predicts a relative 6D poses of the AR device  10  by performing learning based on inputs of the intermediate 6D poses and the IMU values obtained by using the DNN. The “relative” 6D pose may mean a pose difference between an intermediately previously predicted 6D pose (original pose) and a currently predicted 6D pose (transformed pose). Also, the relative 6D pose may mean a final pose finally obtained by being calculated from an intermediate 6D pose. 
     According to an example embodiment, the pose prediction model may perform DNN learning with multi-rate inputs including the IMU inputs of an IMU rate and the intermediate 6D pose inputs of a frame rate. In other words, as the IMU rate is a frequency higher than the frame rate, the pose prediction model may perform learning for pose prediction and inference for pose prediction with inputs of different frequencies. 
     Furthermore, according to another example embodiment, the pose prediction model may perform DNN learning with multi-rate inputs of the intermediate 6D pose inputs including the IMU inputs of an IMU rate, the IMU integration based intermediate 6D pose inputs of an IMU rate, and the image based intermediate 6D pose inputs of a frame rate. 
     As the IMU inputs and the intermediate 6D pose inputs that are input to the pose prediction model correspond to a temporal data sequence, the DNN may be implemented as a recurrent neural network (RNN) using a long short term memory (LSTM) or a gated recurrent unit (GRU). The pose prediction model may be a model that performs DNN learning in a self-supervised method based on the update of a GT pose described above in  FIG.  5 A . 
     When IMU values that are newly obtained according to a movement of the AR device  10  and intermediate 6D poses that are newly estimated according to the movement of the AR device  10  are input to the learned DNN of a pose prediction model, the processor  140  may output a result of the inference of predicting a relative 6D poses in real time through the processing by the learned DNN of a pose prediction model. 
     The processor  140  may perform DNN learning by using the IMU values and camera images obtained during the use of the AR device  10 , or the 6D poses calculated by the VI-SLAM module  130  in a state when the AR device  10  is not used for a certain period, for example, in a charge state. When a user wears the AR device  10  again to use, the processor  140  may complete the DNN learning and perform inference of the DNN for prediction of the pose of the AR device  10 . 
     The memory  150  may include an internal memory such as a volatile memory or a non-volatile memory. The memory  150  may store, under the control of the processor  140 , various data, programs, or applications for driving and controlling the AR device  10 , and input/output signals or AR data. Furthermore, the memory  150  may store data related to the DNN. The memory  150  may correspond to a memory device such as random access memory (RAM), read-only memory (ROM), hard disk drive (HDD), solid state drive (SSD), compact flash (CF), secure digital (SD), micro secure digital (Micro-SD), mini secure digital (mini-SD), extreme digital (xD), Memory Stick, etc., and the type of the memory  150  is not limited thereto and to be various. 
       FIG.  6    is a timing diagram of a method of generating a pose prediction model according to an embodiment. 
     Referring to  FIG.  6   , the processor  140  receives the IMU values  520  obtained from the IMU sensor  110  of  FIG.  2    and the intermediate 6D poses  510  estimated by the VI-SLAM module  130  of  FIG.  2   , as inputs. 
     In detail, the processor  140  may perform learning for pose prediction on the DNN with inputs of a window moving in a sliding manner. The window moving in a sliding manner may include k IMU values, where k is a natural number, and m intermediate 6D poses, where m is a natural number. For example, WINDOW  1   610  may be set to include IMU values from IMU i  to IMUS and intermediate 6D poses from P n  to P N . In other words, the DNN performs learning with multi-rate inputs including the IMU inputs of an IMU rate and the intermediate 6D pose inputs of a frame rate. 
     The processor  140  may output relative 6D pose O 1  of the AR device  10  through processing of WINDOW  1   610  input to the DNN. 
       FIG.  7 A  is a timing diagram of a method of performing pose prediction in a pose prediction model by a sliding window method according to an example embodiment. 
     Referring to  FIG.  7 A , when a relative 6D pose O 1  is predicted through the processing on WINDOW  1   610 , the window may slide based on the IMU rate. In other words, the window may slide on a time axis in units of time points for obtaining one IMU value. 
     The processor  140  may output a relative 6D pose O S-1  of the AR device  10  through processing on WINDOW S- 1   620  sliding by using the DNN, and output a relative 6D pose O S  of the AR device  10  through processing on WINDOW S  630  sliding by using the DNN. In other words, the DNN based pose prediction model may receive inputs of a multi-rate (IMU rate and frame rate), but output a relative 6D pose corresponding to each window at an IMU rate. 
     The DNN based pose prediction model, when performing learning of predicting a relative 6D pose, may perform learning in the self-supervised learning method (or an unsupervised learning method) based on the update of a GT pose  710 . For example, during the DNN learning, an intermediate 6D pose at a timing matching a timing of outputting a window may be used as the GT pose  710  for outputting the window. 
     Whenever the estimation on an intermediate 6D pose is completed by the VI-SLAM module  130  of  FIG.  2   , the GT pose  710  may be set by updating the estimated intermediate 6D pose to GT pose(P 0 (GT), . . . , P N (GT), . . . , P M-1 (GT), P N (GT), . . . ) at a real time point when an image is photographed. 
     The DNN may perform a back propagation through time (BPTT) by reducing a loss function by using the updated GT pose, and accordingly weight parameters of the DNN are updated to appropriate values so that pose prediction accuracy of the DNN based pose prediction model may be improved. 
       FIG.  7 B  is a timing diagram of a method of performing pose prediction in a pose prediction model by a sliding window method according to another example embodiment. 
     Referring to  FIG.  7 B , unlike  FIG.  7 A , the processor  140  receives, as multi-rate inputs of the DNN, intermediate 6D pose inputs  515  including the IMU values  520 , the IMU integration based intermediate 6D pose inputs ( . . . , P′ N_1 , P′ N_2 , P′ N_3 , . . . ) of a IMU rate and the image based intermediate 6D pose inputs (P 0 , P n , P N-1 , P N , . . . ) of a frame rate. Accordingly, a GT pose  720  may be updated at a time interval shorter than the update of the GT pose  710  of  FIG.  7 A , and accordingly even at a timing when a camera image may not be smoothly used due to a problem such as occlusion or blur, the DNN may relatively accurately output a prediction result of relative 6D poses. 
       FIG.  8    is a diagram of a pose prediction model based on a DNN implemented by the RNN using the LSTM according to an example embodiment. 
     According to an example embodiment, since the inputs used for pose prediction in the AR device  10  of  FIG.  1    correspond to the temporal data sequence, the pose prediction model may be implemented by the RNN. However, even so, the pose prediction model is not necessarily limited to an RNN based neural network and may employ a different type of a neural network. 
     The RNN is a kind of a neural network that is efficient for predicting present information by using past information. However, as the RNN has a problem such as long-term dependency, an LSTM type RNN is introduced to solve the problem. The LSTM may separately include a cell state and a hidden layer. In the LSTM, the cell state manages information memory and erases unimportant information (forget gate), and the hidden layer updates only important information to a cell (input gate), so that the long-term dependency problem of the RNN may be solved. 
     Referring to  FIG.  8   , an unrolled multi-rate LSTM is illustrated. IMU values of an IMU rate of a first frequency, for example, 500 Hz, and intermediate 6D poses of a frame rate of second frequency, for example, 30 fps, are input to the LSTM. As such, as the LSTM performs learning of pose prediction with multi-rate inputs, the LSTM where an input (intermediate 6D pose) of a rate of a low frequency (second frequency) is stored may be changed, as indicated by a dashed line, to inactivated states in the middle of learning. The inputs of the past IMU values and past intermediate 6D poses stored in LSTM may be transmitted to cells of other LSTM to be related to prediction of a relative 6D pose at a different timing. 
     In a pose prediction model where learning for pose prediction is performed by an LSTM-type RNN, as loss function is reduced by performing the BPTT by using the updated GT poses (P 0 , . . . , P N , . . . ), pose prediction accuracy of the LSTM based pose prediction model may be improved. 
     According to another example embodiment, the DNN based pose prediction model may be implemented by the RNN using a gated recurrent unit (GRU), instead of the LSTM, and a person skilled in the art could understand that a GRU method may operate similarly to the method described above regarding the LSTM. 
       FIGS.  9  and  10    are block diagrams of performing pose prediction in a processor according to an example embodiment. 
     Referring first to  FIG.  9   , the IMU sensor  110  transmits temporal data of temporal data of an IMU sequence {circle around (1)} including the IMU values obtained at an IMU rate to the VI-SLAM module  130 , and the camera  120  transmits temporal data of an image sequence {circle around (1)} including images obtained at a frame rate to the VI-SLAM module  130 . The IMU sensor  110  transmits the IMU sequence {circle around (1)} to the processor  140  for pose prediction. 
     The VI-SLAM module  130  estimates an intermediate 6D pose based on the IMU sequence {circle around (1)} and the image sequence {circle around (1)}, and transmits an estimated intermediate 6D pose {circle around (1)} to the processor  140 . In this case, the VI-SLAM module  130  updates the estimated intermediate 6D pose {circle around (1)} to a GT pose {circle around (1)} at a real time point (advance Δ TRK ) when an image is photographed, and transmits an updated GT pose {circle around (1)} to the processor  140 . 
     The processor  140  calculates an output regarding the IMU sequence {circle around (1)} and the intermediate 6D pose {circle around (1)} by using the DNN based pose prediction model, and outputs a relative pose {circle around (1)} as a result thereof. 
     Next, referring to  FIG.  10   , the VI-SLAM module  130  estimates an intermediate 6D pose {circle around (2)} based on an IMU sequence {circle around (2)} obtained from the IMU sensor  110  and an image sequence {circle around (2)} obtained from the camera  120 , and transits an estimated intermediate 6D pose {circle around (2)} to the processor  140 . The processor  140  calculates an output regarding the IMU sequence {circle around (2)} and the intermediate 6D pose {circle around (2)} by using the DNN based pose prediction model, and outputs a relative pose {circle around (2)} as a result thereof. 
     When the relative pose {circle around (2)} is predicted, the processor  140  may update weight parameters through backpropagation, for example, the BPTT, of the DNN, based on the updated GT pose {circle around (1)} and the updated GT pose {circle around (2)}. According to an example embodiment, the processor  140  may repeatedly perform the above-described processes to predict the subsequent relative poses. 
     In  FIGS.  9  and  10   , signs of {circle around (1)} and {circle around (2)} denote that data of different time points are included. 
       FIGS.  11 A and  11 B  are time diagrams of performing inference of pose prediction in a pose prediction model based on a DNN that has completed learning according to some embodiments. 
     Referring to  FIGS.  11 A and  11 B , learning for pose prediction is performed based on the IMU values and the intermediate 6D poses input to the DNN, and when parameters of the DNN are updated by the updated GT poses, DNN learning is completed and thus a pose prediction model may be generated. When newly obtained IMU values and newly estimated intermediate 6D poses according to the movement of the AR device  10  of  FIG.  1    are input to the pose prediction model generated based on the learned DNN, the processor  140  of  FIG.  2    may output an inference result of predicting relative 6D poses in real time through processing by the learned DNN of a pose prediction model. 
     For example, as described above, IMU values and intermediate 6D poses may be input to the learned DNN in a sliding window method. When WINDOW  1   1110  is input, the processor  140  performs inference of a prediction result of a relative 6D pose by processing inputs in WINDOW  1   1110  during a calculation time Δ R , and outputs an inference result corresponding to WINDOW  1   1110 , that is, a predicted relative 6D pose. Likewise, the processor  140  performs inference regarding a case when WINDOW  2   1120  that has slid is input and a case when WINDOW  3   1130  that has slid is input, and outputs inference results, that is, predicted relative 6D poses. 
     As the DNN has completed learning for pose prediction based on the input IMU values and intermediate 6D poses, the processor  140  may not only infer present and future 6D poses of the AR device  10  based on a result of the DNN learning, but also output a prediction result of a future 6D pose of the AR device  10 . 
       FIG.  12    is a flowchart of a method of predicting a pose in an AR device according to an example embodiment. Referring to  FIG.  12   , as the pose prediction method in the AR device  10  of  FIG.  1    is related to the example embodiments in the above-described drawings, the contents described in the above drawings, even when omitted in the following description, may be applied to the method of  FIG.  12   . 
     In operation  1201 , the processor  140  may obtain, from the IMU sensor  110 , IMU values corresponding to the movement of the AR device  10  at an IMU rate of a first frequency. 
     In operation  1202 , the processor  140  may obtain, from the VI-SLAM module  130 , intermediate 6D poses of the AR device  10  which are estimated based on inputs of the IMU values and images of the environment surrounding the AR device  10  obtained by the camera  120  at a frame rate of a second frequency. 
     In operation  1203 , the processor  140  performs learning using the DNN based on the inputs of the IMU values and the intermediate 6D poses, thereby generating a pose prediction model that predicts relative 6D poses of the AR device  10 . 
     The above-described example embodiments may be written as a program that is executable in a computer, and may be implemented in a general purpose digital computer for operating the program by using a computer-readable recording medium. Furthermore, a structure of data used in the above-described example embodiments may be recorded through various means on a computer-readable recording medium. The computer-readable recording medium may include storage media such as magnetic storage media, for example, ROM, floppy disks, or hard disks, or optical reading media, for example, CD-ROM or DVD. 
     It should be understood that example embodiments described herein should be considered in a descriptive sense only and not for purposes of limitation. Descriptions of features or aspects within each example embodiment should typically be considered as available for other similar features or aspects in other example embodiments. While one or more example embodiments have been described with reference to the figures, it will be understood by those of ordinary skill in the art that various changes in form and details may be made therein without departing from the spirit and scope as defined by the following claims.