Patent ID: 12254041

DETAILED DESCRIPTION OF THE INVENTION

Hereinafter, example embodiments will be described with reference to the accompanying drawings.

A position recognition system according to the example embodiments may be implemented by at least one computer apparatus, and a position recognition method according to the example embodiments may be performed through at least one computer apparatus included in the position recognition system. Here, a computer program according to an example embodiment may be installed and run on the computer apparatus. The computer apparatus may perform the position recognition method according to the example embodiments under control of the executed computer program. The computer program may be stored in a computer-readable record medium to computer-implement the position recognition method in conjunction with the computer program.

FIG.1illustrates an example of a network environment according to at least one example embodiment. Referring toFIG.1, the network environment may include a plurality of electronic devices110,120,130, and140, a plurality of servers150and160, and a network170.FIG.1is provided as an example only. The number of electronic devices or a number of servers is not limited thereto. Also, the network environment ofFIG.1is provided as an example only among environments applicable to the example embodiments. The environments applicable to the example embodiments are not limited to the network environment ofFIG.1.

Each of the plurality of electronic devices110,120,130, and140may be a fixed terminal or a mobile terminal that is configured as a computer apparatus. For example, the plurality of electronic devices110,120,130, and140may be a smartphone, a mobile phone, a navigation device, a computer, a laptop computer, a digital broadcasting terminal, a personal digital assistant (PDA), a portable multimedia player (PMP), a tablet PC, and the like. For example, althoughFIG.1illustrates a shape of a smartphone as an example of the electronic device110, the electronic device110used herein may refer to one of various types of physical computer apparatuses capable of communicating with other electronic devices120,130, and140, and/or the servers150and160over the network170in a wireless or wired communication manner.

The communication scheme is not limited and may include a near field wireless communication scheme between devices, as well as a communication scheme using a communication network (e.g., a mobile communication network, wired Internet, wireless Internet, a broadcasting network, etc.) includable in the network170. For example, the network170may include at least one of a variety of network topologies that include a personal area network (PAN), a local area network (LAN), a campus area network (CAN), a metropolitan area network (MAN), a wide area network (WAN), a broadband network (BBN), and the Internet. Also, the network170may include at least one of a variety of network topologies that include a bus network, a star network, a ring network, a mesh network, a star-bus network, a tree or hierarchical network, and the like. However, these topologies are provided as examples only.

Each of the servers150and160may be configured as a computer apparatus or a plurality of computer apparatuses that provides an instruction, a code, a file, content, a service, etc., through communication with the plurality of electronic devices110,120,130, and140over the network170. For example, the server150may be a system that provides a service (e.g., a location-based service, a content providing service, a group call service (or an audio conference service), a messaging service, a mail service, a social network service, a map service, a translation service, a financial service, a payment service, and a search service) to the plurality of electronic devices110,120,130, and140connected over the network170.

FIG.2is a block diagram illustrating an example of a computer apparatus according to at least one example embodiment. Each of the plurality of electronic devices110,120,130, and140or each of the servers150and160may be implemented by the computer apparatus200ofFIG.2.

Referring toFIG.2, the computer apparatus200may include a memory210, a processor220, a communication interface230, and an input/output (I/O) interface240. The memory210may include a permanent mass storage device, such as a random access memory (RAM), a read only memory (ROM), and a disk drive, as a computer-readable record medium. The permanent mass storage device, such as ROM and a disk drive, may be included in the computer apparatus200as a permanent storage device separate from the memory210. Also, an Operating System (OS) and at least one program code may be stored in the memory210. Such software components may be loaded to the memory210from another computer-readable record medium separate from the memory210. The other computer-readable record medium may include a computer-readable record medium, for example, a floppy drive, a disk, a tape, a DVD/CD-ROM drive, a memory card, etc. According to other example embodiments, software components may be loaded to the memory210through the communication interface230, instead of the computer-readable record medium. For example, the software components may be loaded to the memory210of the computer apparatus200based on a computer program installed by files received over the network170.

The processor220may be configured to process instructions of a computer program by performing basic arithmetic operations, logic operations, and I/O operations. The instructions may be provided from the memory210or the communication interface230to the processor220. For example, the processor220may be configured to execute received instructions in response to the program code stored in the storage device, such as the memory210.

The communication interface230may provide a function for communication between the communication apparatus200and another apparatus (e.g., the aforementioned storage devices). For example, the processor220of the computer apparatus200may forward a request or an instruction created based on program code stored in the storage device, such as the memory210, data, and a file, to other apparatuses over the network170under control of the communication interface230. Inversely, a signal or an instruction, data, a file, etc., from another apparatus may be received at the computer apparatus200through the network170and the communication interface230of the computer apparatus200. A signal, an instruction, data, etc., received through the communication interface230may be forwarded to the processor220or the memory210, and a file, etc., may be stored in a storage medium (e.g., the permanent storage device) that can be included in the computer apparatus200.

The I/O interface240may be a device used for interfacing with an I/O apparatus250. For example, an input device may include a device, such as a microphone, a keyboard, a mouse, etc., and an output device may include a device, such as a display, a speaker, etc. As another example, the I/O interface240may be a device for interfacing with an apparatus in which an input function and an output function are integrated into a single function, such as a touchscreen. The I/O apparatus250may be configured as a single apparatus with the computer apparatus200.

Also, according to other example embodiments, the computer apparatus200may include a number of components greater than or less than the number of components shown inFIG.2. However, there is no need to clearly illustrate many components according to the related art. For example, the computer apparatus200may include at least a portion of the I/O apparatus250, or may further include other components, for example, a transceiver, a database.

FIG.3illustrates an example of a position recognition system according to an example embodiment. A position recognition system300according to the example embodiment may include a mobile device310, a repeater320, a global localization module330, and an object recognition and pose estimation module340.

The mobile device310may generate frame images through a camera and may transmit at least one frame image from among the generated frame images as a query image to the repeater320. For example, a computer program for an augmented reality (AR)-based navigation function may be installed and run on the mobile device310, and at least one frame image from among a plurality of frame images generated by capturing a surrounding space of the mobile device310through the camera may be transmitted as a query image to the repeater320under control of the running computer program. Here, since positions of the mobile device310over time need to be continuously recognized to provide a navigation function, the mobile device310may transmit a frame image to the repeater320at predetermined intervals. If the predetermined interval corresponds to an interval at which a frame image is generated, all the generated frame images may be transmitted as the query image to the repeater320. As another example, if the predetermined interval is less than the interval at which the frame image is generated, a portion of the frame images may be transmitted as the query image to the repeater320. An interval at which the query image is transmitted may be simply preset or may be dynamically set based on an amount of time used for a server end to process the query image and to calculate a position of the mobile device310and/or network traffic between the mobile device310and the server end.

The repeater320, the global localization module330, and the object recognition and pose estimation module340may be included on the server end. Here, the repeater320, the global localization module330, and the object recognition and pose estimation module340may be implemented in a single physical device or may be implemented in different physical devices. In certain example embodiments, the global localization module330may be implemented in a system of a third party such that the server end uses the global localization module330through the system of the third party.

The repeater320may transmit the query image received from the mobile device310to each of the global localization module330and the object recognition and pose estimation module340, and may request pose estimation of the mobile device310. Here, the term “pose” is defined as a concept that includes a position and an orientation. For example, a pose of the mobile device310may be understood as representing a position and an orientation of the mobile device310on a world coordinate system. The pose of the mobile device310may be represented by, for example, six degrees of freedom (6DOF) having 3DOF for the position and 3DOF for the orientation. The position of the mobile device310described below may correspond to the pose of the mobile device310depending on example embodiments.

Any module capable of performing global localization for the mobile device310, such as a global positioning system (GPS), simultaneous localization and mapping (SLAM), a Wi-Fi positioning system (WPS), visual localization (VL), and the like, may be employed for the global localization module330. In response to a request from the repeater320, the global localization module330may estimate the pose of the mobile device310and may transmit the same to the repeater320. In the example embodiment ofFIG.3, the pose of the mobile device310provided from the global localization module330is represented as ‘Global 6DOF1’.

The object recognition and pose estimation module340may estimate the pose of the mobile device310based on a target object recognized in an image and may transmit the estimated pose to the repeater320. The example embodiment ofFIG.3represents the pose of the mobile device310provided from the object recognition and pose estimation module340as ‘Global 6DOF2’. The object recognition and pose estimation module340may register the target object with the assistance of the global localization module330. For example, the object recognition and pose estimation module340may train a two-dimensional (2D) target object or a three-dimensional (3D) target object and may register the same to a database, and may further register a global pose of the target object to the database in association with the target object in an environment in which the global localization module330operates. Here, the object recognition and pose estimation module340may recognize the target object in the query image transmitted through the repeater320, and then calculate a relative position between the target object and the mobile device310(in detail, a relative position between the target object and the camera of the mobile device310). The object recognition and pose estimation module340stores the global pose of the target object in the database and thus, may estimate a global pose of the mobile device310(in detail, a global pose of the camera of the mobile device310) by transforming the global pose of the target object by the calculated relative position. A method of estimating, by the object recognition and pose estimation module340, the global pose of the mobile device310will be further described herein.

The repeater320may receive the pose of the mobile device310(e.g., ‘Global 6DOF1’) estimated by the global localization module330and the pose of the mobile device310(e.g., ‘Global 6DOF2’) estimated by the object recognition and pose estimation module340, and may transfer the same to the mobile device310. The example embodiment ofFIG.3represents that the mobile device310receives {Global 6DOF1, Global 6DOF2} from the repeater310.

In this case, the mobile device310may determine the global pose of the mobile device310based on the received poses {Global 6DOF1, Global 6DOF2}. For example, the mobile device310may determine the global pose of the mobile device310through confidence-based filtering by assigning a weight based on a confidence value provided from each of the global localization module330and the object recognition and pose estimation module340. In detail, for example, the global pose of the mobile terminal310may be calculated according to the following Equation 1.

Mrescam=w1w1+w2×MGLcam+w2w1+w2×MVOTcam[Equation⁢⁢1]

In Equation 1, Mrescamdenotes a final calculated global pose of the mobile device310, MGLcamdenotes a global pose of the mobile device310calculated through the global localization module330, and MVOTcamdenotes a global pose of the mobile device310calculated through the object recognition and pose estimation module340. Also, w1 denotes a confidence value of the global localization module330and w2 denotes a confidence value of the object recognition and pose estimation module340. Here, the confidence values w1 and w2 may be real numbers each having a value included in the range of 0 to 1. Such confidence values may be transmitted together when each of the global localization module330and the object recognition and pose estimation module340transmits the global pose of the mobile device310.

According to the example embodiment, the object recognition and pose estimation module340may recognize a relative pose of the mobile device310for a 2D or 3D object of a place registered to the database, and may provide a position of a user at the corresponding place. Also, in an environment in which an existing global pose estimation method such as GPS, WPS, VL, and the like, is available, it is possible to improve the accuracy of pose estimation using the position recognition method as an auxiliary method. In an environment in which the existing global pose estimation method is unavailable, it is possible to estimate a position of a user by using the position recognition method according to the example embodiment alone, and to provide a location-based service such as an AR-based navigation function. For example, according to Equation 1, if the confidence value w1 of the global localization module330=0, it can be known that a value of MVOTcamprovided from the object recognition and pose estimation module340is a value of the final calculated global pose Mrescamof the mobile device310.

FIG.4illustrates an example of a real-time pose estimation process in a global space according to an example embodiment.

In operation S410, the mobile device310receives a position (6DOF) through a server410. Operation S410is a process of acquiring, by the mobile device310, an initial position. For example, the server410includes the repeater320, the global localization module330, and the object recognition and pose estimation module340described above with reference toFIG.3, and provides an initial position of the mobile device310through the global localization module330in operation S410.

In operation S420, the mobile device310transmits a current position Mn and a query image In to the server410and requests a position. The current position Mn refers to the initial position received in operation S410, and the query image In refers to one of the frame images that are generated through a camera included in the mobile device310.

In operation S430, the mobile device310tracks a position according to a movement (Mn→Mn+1) of the mobile device310. For example, since the pose provided through the server410in the process ofFIG.3is calculated through a task that requires a large amount of arithmetic operations, it is difficult to use a result value in real time. To supplement this, the mobile device310tracks a movement from a first position Mn to a second position Mn+1 of the mobile device310until a response from the server410is received. For example, the mobile device310may track the second position Mn+1 of the mobile device310through visual inertial odometry or by accumulating a variance of the camera (e.g., a variance of a position and/or an orientation of the same object in frame images generated through the camera) to the first position Mn.

In operation S440, the mobile device310receives a response Mn′ to the query image In. The response Mn′ is a response to the request of operation S420. Depending on example embodiments, the mobile device310may receive the global poses MGLcamand MVOTcamfrom the global localization module330and the object recognition and pose estimation module340, respectively, and may calculate the global pose Mrescamaccording to Equation 1 and may use the same as the response Mn′.

In operation S450, the mobile device310calculates a new current position Mn+1′ by applying a position variance ΔM of the mobile device310to the response Mn′. Initially, the mobile device310calculates the position variance ΔM of the mobile device310based on a difference between the current position Mn and the second position Mn+1 estimated in operation S430. The mobile device310calculates the new current position Mn+1′ by applying the position variance ΔM of the mobile device310to the response Mn′.

In operation S460, the mobile device310determines a final position Mn+1″ by applying a weighted moving average to the second position Mn+1 and the new current position Mn+1′. When providing an AR-based navigation function, the mobile device310needs to continuously provide AR content through positions estimated between the first position Mn and the second position Mn+1. For example, in certain embodiments, it may be assumed that the mobile device310receives position information from the server410three times per second and the mobile device310processes30calculations per second to provide the AR content, but other timings are also contemplated. Here, the mobile device310sequentially processes an operation for providing the AR content through positions estimated by the mobile device310until subsequent position information is received from the server410and provides the AR content to the user. In this case, the positions estimated by the mobile device310gradually approaches the second position Mn+1. Here, as the new current position Mn+1′ is calculated in response to receiving the response Mn′ from the server410, a position close to the second position Mn+1 is immediately changed to the new current position Mn+1′. However, since an error may be present between the second position Mn+1 and the new current position Mn+1′, a sense of heterogeneity may occur in the AR content at a moment due to an error (difference) between the second position Mn+1 and the new current position Mn+1′. Therefore, the mobile device310may reduce a difference between positions applied to the AR content and thereby reduce the sense of heterogeneity that occurs in the AR content by applying the weighted moving average to the second position Mn+1 and the new current position Mn+1′.

The mobile device310may perform again operation S420. Here, the current position Mn calculated in operation S420may be the final position Mn+1″ calculated in operation S460.

FIG.5illustrates an example of capturing a target object through a camera according to an example embodiment, andFIG.6illustrates an example of the images captured through a camera according to an example embodiment.

FIG.5illustrates an example in which a camera510captures a target object while moving and rotating at first to third times (t1-t3).FIG.6illustrates a first image to a third image (Image1-Image3) of target objects (Ob1-Ob3) that are captured by the camera510at the first time to the third time (t1-t3).

A device (e.g., the mobile device310or the server) that includes the camera510or that is communicably connected to the camera510may generate an image by capturing a target object using the camera510. Hereinafter, an example embodiment in which the mobile device310with the camera510generates an image is described. Also, in the following, the server may refer to an entity that constitutes the server end described above withFIG.3.

Referring toFIGS.5and6, an image captured from the target object at a first point in time t1is referred to as the first image Image1, an image captured from the target object at a second point in time t2is referred to as the second image Image2, and an image captured from the target object at a third point in time t3is referred to as the third image Image3.

At least one of the images generated by the mobile device310may be transmitted to the server through the network170. For example, the first image Image1to the third image Image3may be transmitted to the server. As another example, a subset of the images included in the first image Image1to the third image Image3may be transmitted to the server. In the example embodiment, it is assumed that the first image Image1and the third image Image3are transmitted to the server and the second image Image2is not transmitted to the server. An amount of time is required until the mobile device310transmits an image to the server and then receives relative pose data from the server. Even in this time, the camera510or the mobile device310that includes the camera510may move, and relative pose data of the target object may vary according to movement of the mobile device310. In the case of providing an AR-based navigation function, a 3D image of a virtual object rendered by the mobile device310is generated based on previous relative pose data of the target object and thus, a sense of heterogeneity may occur in an augmented image. Even before receiving the relative pose data from the server, the mobile device310may estimate relative pose data of the target object in a new image based on relative pose data previously received from the server, and may render the 3D image of the virtual object based on the relative pose data of the target object. Therefore, the sense of heterogeneity may significantly decrease in the augmented image.

The second point in time t2may be after the first point in time t1. The third point in time t3may be after the second point in time t2. However, it is provided as an example only and the third point in time t3may be between the first point in time t1and the second point in time t2.

The camera510or the mobile device310that includes the camera510may move between the first to the third points in time t1, t2, and t3. For example, the camera510or the mobile device310that includes the camera510may perform at least one of a first magnitude of translational motion in a first direction and a second magnitude of rotational motion in a second direction between the first to the third points in time t1, t2, and t3.

The mobile device310may store and manage view data of the camera510at a point in time at which the camera510captures the target object. The mobile device310stores first view data at the first point in time t1, stores second view data at the second point in time t2, and stores third view data at the third point in time t3. The mobile device310may manage view data by deleting view data stored before a preset time.

The view data of the camera510refers to data that is directly related to the pose of the camera510. The view data of the camera510may be a view matrix that represents a movement and a rotation of the world, that is, a fixed subject on a camera coordinate system defined by the pose of the camera510. In other words, the view data of the camera510may refer to a matrix that converts a coordinate value on the world coordinate system to a camera coordinate system or element values of the matrix.

According to the movement of the camera510, the pose of the camera510and the camera coordinate system defined by the pose of the camera510varies. The view data of the camera510varies according to the movement of the camera510.

The view data of the camera510may have an inverse function relationship with the pose data of the camera510. The pose data of the camera510may be data that represents the pose of the camera510on the world coordinate system. That is, the pose data of the camera510may refer to a matrix that converts a coordinate value on the camera coordinate system to the world coordinate system or element values thereof.

The mobile device310may calculate the pose data of the camera510based on the view data of the camera510, or may calculate the view data of the camera510based on the pose data of the camera510.

Although the view data and the pose data of the camera510may be understood as provided with the aforementioned meanings herein, the pose data may be a view matrix and the view data may be an inverse matrix of the pose data, depending on example embodiments.

The mobile device310may detect a movement of the camera510and may change or newly generate view data according to the movement of the camera510.

According to an example embodiment, the mobile device310may detect a movement of the camera510based on a movement of feature points of images captured by the camera510. For example, the mobile device310may detect how feature points in the first image Image1and feature points in the second image Image2vary with respect to each other, and may estimate a direction and a magnitude of translational motion of the camera510and a direction and a magnitude of rotational motion of the camera510based on the detection. According to an example embodiment, the mobile device310may generate and manage view data of the camera510using visual odometry technology. According to another example embodiment, the mobile device310may generate and manage the view data of the camera510based on sensor values received from sensors. The sensors may be inertial sensors and may output sensor values showing where and how far the mobile device310has moved and rotated.

According to still another example embodiment, the mobile device310may generate and manage view data of the camera510based on a change in feature points in images captured by the camera510and sensor values received from the sensors. For example, the mobile device310may generate and manage view data of the camera510using visual inertial odometry technology.

The mobile device310may receive relative pose data of the target object from the server, and may estimate, store, and manage pose data of the target object based on the received relative pose data of the target object and view data of the camera at the same point in time as that.

For example, the mobile device310may calculate and store pose data of the target object based on first relative pose data of the target object of the first point in time t1received from the server and first view data of the camera510of the first point in time t1. The mobile device310may calculate pose data of the target object based on third relative pose data of the target object of the third point in time t3received from the server and third view data of the camera510of the third point in time t3.

Pose data (hereinafter, first pose data) of the target object calculated in association with the first point in time t1and pose data (hereinafter, second pose data) of the target object calculated in association with the third point in time t3need to theoretically have the same value. However, the first pose data and the second pose data may have a mutually intolerable error due to inaccuracy of a learning model of the server or inaccuracy of view data of the camera510.

When an error between the first pose data and the second pose data exceeds a reference value, the mobile device310may ignore the second pose data. On the other hand, when the error between the first pose data and the second pose data is within the reference value, the mobile device310may update pose data used by the mobile device310based on the first pose data and the second pose data. For example, the mobile device310may update pose data of the target object using an average or a weighted average for the first pose data and the second pose data.

The mobile device310may estimate relative pose data of the target object based on the pose data of the target object and the view data of the camera510.

According to another example, the mobile device310may also receive relative pose data of the target object from the server.

The mobile device310may track positions of target objects in the images at the first to the third point in times t1to t3. For example, the server that receives the first image Image1at the first point in time t1may estimate a type and a position of the target object in the first image Imgae1and may provide the estimated type and position of the target object to the mobile device310. The mobile device310may determine a position of the target object in the second image Image2by tracking the target object based on the position of the target object in the first image Image1. The mobile device310may track a position of the target object in the first image Image1from the second image Image2by extracting feature points in the first image Image1and the second image Image2and by comparing the feature points.

The mobile device310may estimate the pose data of the camera510based on the view data of the camera510. Here, the mobile device310may calculate the pose data of the camera510by calculating an inverse matrix of the view data of the camera510.

FIG.7is a flowchart illustrating an example of a position recognition method of a mobile device according to an example embodiment. The position recognition method according to the example embodiment may be performed by the computer apparatus200that implements the mobile device310. Here, the processor220of the computer apparatus200may be implemented to execute a control instruction according to a code of at least one computer program or a code of an OS included in the memory210. Here, the processor220may control the computer apparatus200to perform operations710to770included in the method ofFIG.7in response to the control instruction provided from the code stored in the computer apparatus200.

In operation710, the computer apparatus200generates a frame image through a camera. Here, the computer apparatus200preferably continuously generates frame images through the camera.

In operation720, the computer apparatus200transmits a first global pose of the camera and the generated frame image to a server. For example, an example in which the mobile device310transmits the current position Mn and the query image In to the server410and requests the position in operation S420ofFIG.4is described.

In operation730, the computer apparatus200tracks a change from the first global pose of the camera to a third global pose. For example, as described in operation S430ofFIG.4, the mobile device310tracks the position according to the movement (Mn→Mn+1) of the mobile device310.

In operation740, the computer apparatus200receives, from the server, a second global pose of the camera that is estimated based on a pose of an object included in the transmitted frame image. For example, as described in operation S440ofFIG.4, the mobile device310receives the response Mn′ to the query image In from the server410. In this case, the second global pose corresponds to the response Mn′. In an example embodiment, the second global pose is estimated by transforming a global pose of a target object stored in the server by a relative pose between the camera and the target object included in the frame image that is calculated by analyzing the frame image in the server.

Meanwhile, here, the second global pose corresponds to the global pose MVOTcamfrom the object recognition and pose estimation module340. Therefore, in addition to the second global pose, the computer apparatus200further receives, from the server, the first confidence value w1 corresponding to the second global pose, the global pose MGLcamof the computer apparatus200that is estimated based on global localization, and the second confidence value w2 corresponding to the global pose MGLcam. In this case, the computer apparatus200determines the global pose Mrescamby assigning different weights based on the first confidence value and the second confidence value to the second global pose MVOTcamand the global pose MGLcam. Here, the computer apparatus200may determine the global pose Mrescamsuch that a portion of the second global pose MVOTcamis relatively greater in the global pose Mrescamas the first confidence value is greater than the second confidence value. As a detailed example, an example embodiment of determining the global pose Mrescamthrough Equation 1 is described. As described above, the computer apparatus200may receive the global poses MGLcamand MVOTcamfrom the global localization module330and the object recognition and pose estimation module340, respectively, and may calculate the global pose Mrescamaccording to Equation 1 and may use the same as the response Mn′.

In operation750, the computer apparatus200calculates a pose variance based on the third global pose and the first global pose. For example, the computer apparatus200may calculate the pose variance based on a difference between the third global pose and the first global pose.

In operation760, the computer apparatus200determines a fourth global pose by applying the pose variance to the second global pose. For example, an example of calculating the new current position Mn+1′ by applying the position variance ΔM of the mobile device310to the response Mn′ through operation S450is described. Here, the second global pose may correspond to the response Mn′ and the fourth global pose may correspond to Mn+1′.

In operation770, the computer apparatus200determines a fifth global pose by applying a weighted moving average to the third global pose and the fourth global pose. For example, an example of determining the final position Mn+1″ by applying the weighted moving average to Mn+1 and Mn+1′ in operation S460ofFIG.4is described. Here, the third global pose may correspond to Mn+1, the fourth global pose may correspond to Mn+1′, and the fifth global pose may correspond to Mn+1″.

FIG.8is a flowchart illustrating an example of a position recognition method of a server according to an example embodiment. The position recognition method according to the example embodiment may be performed by the computer apparatus200that implements the object recognition and pose estimation module340. Here, the processor220of the computer apparatus200may be implemented to execute a control instruction according to a code of at least one computer program or a code of an OS included in the memory210. Here, the processor220may control the computer apparatus200to perform operations810to860included in the method ofFIG.8in response to the control instruction provided from the code stored in the computer apparatus200.

In operation810, the computer apparatus200registers objects to a database in association with a corresponding global pose. The computer apparatus200registers a plurality of objects to the database, and stores a global pose for each of the plurality of objects in the database in association with a corresponding object with the assistance of the global localization. For example, the global pose of the object may be determined through the global localization module330ofFIG.3.

In operation820, the computer apparatus200receives a query image from a mobile device. Here, the mobile device may correspond to the mobile device310, and the query image may be one of frame images generated through the camera included in the mobile device310.

In operation830, the computer apparatus200analyzes the query image, and calculates a relative pose between the camera of the mobile device and a target object included in the query image. A method of calculating the relative pose between the target object included in the image and the camera that captures the image using the image was described above with reference toFIGS.5and6.

In operation840, the computer apparatus200searches the database for a global pose of the target object. For example, the computer apparatus200may search the database for a target object recognized in the query image and may extract a global pose stored in the database in association with the corresponding object as the global pose of the target object.

In operation850, the computer apparatus200determines a global pose for the camera of the mobile device by transforming the found global pose by the calculated relative pose. The relative pose refers to a relative pose with the target object of the camera. When the global pose of the target object is known, the computer apparatus200may estimate the global pose for the camera by shifting the global pose of the target object.

In operation860, the computer apparatus200transmits the determined global pose to the mobile device. Depending on example embodiments, the computer apparatus200may transmit a first global pose as the determined global pose, a first confidence value corresponding to the first global pose, a second global pose of the mobile device that is estimated based on a global localization, and a second confidence value corresponding to the second global pose to the mobile device. In this case, a third global pose may be determined by assigning different weights based on the first confidence value and the second confidence value to the first global pose and the second global pose in the mobile device.

As described above, according to some example embodiments, it is possible to recognize a position of a user based on visual information processing about a POI.

The systems or apparatuses described herein may be implemented using hardware components a combination of hardware components and software components. For example, the apparatuses and the components described herein may be implemented using one or more general-purpose or special purpose computers, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of responding to and executing instructions in a defined manner. The processing device may run an operating system (OS) and one or more software applications that run on the OS. The processing device also may access, store, manipulate, process, and create data in response to execution of the software. For the purposes of simplicity, the description of a processing device is provided as singular; however, one skilled in the art will be appreciated that a processing device may include multiple processing elements and/or multiple types of processing elements. For example, a processing device may include multiple processors or a processor and a controller. In addition, different processing configurations are possible, such as parallel processors.

The software may include a computer program, a piece of code, an instruction, or some combinations thereof, for independently or collectively instructing or configuring the processing device to operate as desired. Software and/or data may be embodied in any type of machine, component, physical equipment, virtual equipment, computer storage medium or device, to be interpreted by the processing device or to provide an instruction or data to the processing device. The software also may be distributed over network coupled computer systems so that the software is stored and executed in a distributed fashion. The software and data may be stored by one or more computer readable storage media.

The methods according to the above-described example embodiments may be configured in the form of program instructions performed through various computer devices and recorded in computer-readable media. The media may include, alone or in combination with program instructions, a data file, a data structure, and the like. Here, the media may continuously store computer-executable programs or may in a transitory manner store the same for execution or download. Also, the media may be any of various types of recording devices or storage devices in a form in which one or a plurality of hardware components are combined. Without being limited to media directly connected to a computer system, the media may be distributed over the network. Examples of the media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical media such as CD-ROM and DVDs; magneto-optical media such as floptical disks; and hardware devices that are specially configured to store program instructions, such as read-only memory (ROM), random access memory (RAM), flash memory, and the like. Examples of other media include record media and storage media managed by an app store that distributes applications or a site that supplies and distributes other various types of software, a server, and the like. Examples of the program instruction include a machine code as produced by a compiler and include a high-language code executable by a computer using an interpreter and the like.

Although the example embodiments are described with reference to some specific example embodiments and accompanying drawings, it will be apparent to one of ordinary skill in the art that various alterations and modifications in form and details may be made in these example embodiments without departing from the spirit and scope of the claims and their equivalents. For example, suitable results may be achieved if the described techniques are performed in a different order, and/or if components in a described system, architecture, device, or circuit are combined in a different manner, or replaced or supplemented by other components or their equivalents.

Therefore, other implementations, other example embodiments, and equivalents of the claims are to be construed as being included in the claims.