Abstract:
The present invention relates to an object learning method that minimizes time required for learning an object, an object tracking method using the object learning method, and an object learning and tracking system. The object learning method includes: receiving an image to be learned through a camera to generate a front image by a terminal; generating m view points used for object learning and generating first images obtained when viewing the object from the m view points using the front image; generating second images by performing radial blur on the first images; separating an area used for learning from the second images to obtain reference patches; and storing pixel values of the reference patches.

Description:
BACKGROUND OF THE INVENTION 
       [0001]    1. Field of the Invention 
         [0002]    The present invention relates to an object learning method that minimizes time required for learning an object, an object tracking method using the object learning method, and an object learning and tracking system. 
         [0003]    2. Description of the Related Art 
         [0004]    A conventional object recognition/tracking algorithm previously learns and stores information on a target object (for example, feature points, color distributions, etc.) and compares the information with an image inputted through a camera in real time to perform object recognition/tracking. 
         [0005]    While exhibiting a robust performance on learned objects, the object recognition/tracking algorithm has a drawback in that learning steps are carried out offline because it takes a long time to learn an object. 
         [0006]    It is impossible to learn all of the objects in an environment and to store learned results. Thus, there is a need for developing a technology that may quickly perform learning of a target object not on-line but off-line so as to interoperate with an unlearned object. 
       SUMMARY OF THE INVENTION 
       [0007]    The present invention has been made in an effort to provide an object learning method that minimizes time required for learning an object, and an object tracking method and system using the object learning method. 
         [0008]    An exemplary embodiment of the present invention provides an object learning method including: receiving an image to be learned through a camera to generate a front image by a terminal; generating m view points used for object learning and generating first images obtained when viewing the object from the m view points using the front image; generating second images by performing radial blur on the first images; separating an area used for learning from the second images to obtain reference patches; and storing pixel values of the reference patches. 
         [0009]    Another exemplary embodiment of the present invention provides an object tracking method using an object learning method including: obtaining a front image of an object, a posture parameter of the camera corresponding to m view points for the object, and reference patches for the m view points using the object learning method by a terminal; receiving an input image for an object to be tracked through the camera and separating an area used for object tracking to obtain an input patch; determining whether the object is already tracked in a previous frame, if it is determined that the object is not tracked, obtaining a pseudo reference patch having the least difference from an input patch in a current frame and a posture parameter of the camera, performing optimization using the posture parameter of the camera in the current frame as an initial value to obtain an enhanced posture parameter of the camera, and if the object is already tracked, performing optimization using a posture parameter of the camera in a previous frame as the initial value to obtain the enhanced posture parameter of the camera; and generating a front patch viewed from the front using the enhanced posture parameter of the camera, measuring a similarity between the front patch and a front image obtained by the object learning method, and if the similarity is in excess of a threshold value, determining that the object is equal to a learned object. 
         [0010]    Yet another exemplary embodiment of the present invention provides an object learning and tracking system including an object learning unit that receives an image of an object to be learned through a camera to obtain a front image of the object, generates posture parameters of the camera for m view points used for learning, generating images viewed from the view points using the front image, performs radial blur, performs separation of an area used for learning, and obtain reference patches, an input patch obtaining unit that receives an input image of an object to be tracked through the camera, separates an area used for object tracking, and obtains an input patch, a tracking determination unit that determines whether the object is already tracked in a previous frame, a pseudo reference obtaining unit that if it is determined that the object is not tracked, obtains a pseudo reference patch having the least difference from an input patch in a current frame and a posture parameter of the camera, a camera posture enhancing unit that if it is determined that the object is not tracked, performs optimization using a posture parameter of the camera in a current frame as an initial value to obtain an enhanced posture parameter of the camera, and if it is determined that the object is already tracked, performs optimization using a posture parameter of the camera in a previous frame as the initial value to obtain the enhanced posture parameter of the camera, and a similarity determination unit that generates a front patch viewed from the front using the enhanced posture parameter of the camera, measures a similarity between the front patch and the front image of the object learning unit, and if the similarity is in excess of a threshold value, determines that the object is equal to the learned object. 
         [0011]    According to exemplary embodiments of the present invention may perform a quick learning process on an object by warping a front image obtained through a camera, generating images for learning viewed from m view points, and performing radial blur. 
         [0012]    Further, while using a terminal, the present invention may perform off-line learning on an object that is not previously learned and may perform tracking on the object. 
         [0013]    In addition, the present invention may generate an image viewed from the front using an accelerometer of a terminal even when a user (i.e., a camera) does not view the front while performing learning on an object, and may use the generated image for object learning. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0014]      FIG. 1  is a flowchart illustrating an object learning method according to an exemplary embodiment of the present invention; 
           [0015]      FIGS. 2A to 2D  are photographs illustrating exemplary results obtained by performing some steps of  FIG. 1 ; 
           [0016]      FIG. 3  is a view illustrating a step of obtaining a front image of the object shown in  FIG. 1 ; 
           [0017]      FIG. 4  is a flowchart illustrating an object tracking method according to an exemplary embodiment of the present invention; and 
           [0018]      FIG. 5  is a block diagram illustrating an object tracking system according to an exemplary embodiment of the present invention. 
       
    
    
     DETAILED DESCRIPTION OF THE EMBODIMENTS 
       [0019]    Hereinafter, an object learning method and an object tracking method and system according to exemplary embodiments of the present invention will be described with reference to the accompanying drawings. 
         [0020]    &lt;Object Learning Method&gt; 
         [0021]    Referring to  FIGS. 1 to 3 , an object learning method according to an exemplary embodiment of the present invention will be described. 
         [0022]    First, a terminal (for example, a mobile terminal) receives an image I* for an object to be learned through a camera and converts the received image into a front image If as shown in  FIG. 2A  (S 101 ). 
         [0023]    Specifically, the terminal receives the image I* through the camera and obtains an angle θ between a view point orientation vector of the camera and a front view point orientation vector (that is, a normal vector of a plane) as shown in  FIG. 3 , and then rotates the image I* by −θ, thus obtaining the front image I f . Although it has been illustrated in  FIG. 3  that the front image is obtained by rotating the image I* with respect to X axis, the present invention is not limited thereto. As necessary, the image I* may be rotated with respect to Y or Z axis to obtain the front image. 
         [0024]    The angle θ may be acquired in various methods. As an example, the angle θ may be acquired by using an accelerometer that is provided in the terminal and measures an angle between the camera and gravity direction. 
         [0025]    Next, the terminal generates m view points that are to be used for learning (S 102 ). 
         [0026]    Then, the terminal generates a posture parameter (RT) n  for each of the m view points. 
         [0027]    Next, the terminal performs warping on the front image I f  to generate first images In obtained when viewing the object from the m view points (S 103 ).  FIG. 2B  is a photograph illustrating an exemplary first image I n  obtained when viewing the object from an nth view point that is one of the m view points. 
         [0028]    In performing warping on the front image I f  to generate the first images In, warping using a virtual camera over a graphic processing unit (GPU) that may quickly generate the first images In is preferred to image-to-image warping using a transformation matrix. 
         [0029]    Next, the terminal performs radial blur on the first images I n  to generate second images I n ′ as shown in  FIG. 2C  (S 104 ). 
         [0030]    The radial blur rotates each pixel in an image with respect to a specific point. Performing radial blur on the first images I n  creates an effect as if k images rendered at k different view points close to current view points overlap one another. This eliminates the need of performing rendering k times for each view point, thus minimizing time required for learning and providing robustness against noise created during a subsequent object tracking process. The object tracking process will be described below in greater detail together with an object tracking method according to an exemplary embodiment of the present invention. 
         [0031]    Next, the terminal performs Gaussian blur on the second images I n ′ to generate third images I n ″ as shown in  FIG. 2D  (S 105 ). 
         [0032]    Such Gaussian blur on the second images I n ′ renders a subsequent tracking process more robust against noise created in the image I* received through the camera. 
         [0033]    And, the Gaussian blur on the second images I n ′ is performed by not a central processing unit (CPU) but a graphic processing unit (GPU) to offer enhanced processing speeds. 
         [0034]    Next, the terminal separates an area for object learning from the third images I n ″ to obtain a learning patch P n . A red rectangular area shown in  FIG. 3  represents an example of the learning patch P n . 
         [0035]    Then, the terminal performs an after-treatment, such as sampling and normalizing the learning patch P n , to obtain a reference patch p n ′ (S 106 ). 
         [0036]    That is, the terminal samples the learning patch P n  to a small size to obtain a sampling patch pn and normalizes the sampling patch pn to obtain a reference patch p n ′. 
         [0037]    Such sampling on the learning patch P n  may minimize time required for comparison between an input image and the reference patch p n ′ in the subsequent object tracking process. The object tracking process will be described below in greater detail together with an object tracking method according to an embodiment of the present invention. 
         [0038]    The normalization of the sampling patch pn allows a pixel value distribution of the reference patch p n ′ to have an average value of “0” and a standard deviation of “1”. After the normalization process, the subsequent object tracking process is less affected by a variation in pixel values due to a change in illumination. 
         [0039]    Thereafter, the terminal stores a pixel value of the reference patch p n ′ and a posture parameter (RT) n  of a view point corresponding to the pixel value. 
         [0040]    &lt;Object Tracking Method Using Object Learning Method&gt; 
         [0041]    Hereinafter, an object tracking method using the object learning method will be described with reference to  FIG. 4 . 
         [0042]    First, the terminal performs the object learning method including the above-described various steps to obtain a front image If of an object, a posture parameter (RT) n  of a camera corresponding to m view points with respect to the object, and a reference patch p n ′ for the m view points (S 201 ). 
         [0043]    Next, the terminal receives an input image for a target object and separates an area used for object tracking from the input image to obtain a patch F i . Then, the terminal samples the patch F i  to a small size, thereby obtaining a sampling patch f i . In this case, the sampling is performed with the same method as that performed in the after-treatment step of the object learning method. 
         [0044]    Next, the terminal normalizes the sampling patch f i  to obtain an input patch f n . In this case, the normalization is performed with the same method as that performed in the after-treatment step of the object learning method. 
         [0045]    Next, the terminal determines whether an object to be tracked in a current frame is already tracked. If it is determined that the object is not tracked, the terminal obtains a pseudo reference patch p n ′ having the least difference from the input patch in the current frame and a posture parameter (RT) n  of the camera and performs optimization with the posture parameter (RT) n  of the camera in the current frame as an initial value to obtain an enhanced posture parameter (RT) n ′ of the camera. If it is determined that the object is tracked, the terminal performs optimization with the posture parameter (RT) n  of the camera in a previous frame as the initial value to obtain an enhanced posture parameter (RT) n ′ of the camera (S 202 , S 203 , S 204 , and S 213 ). 
         [0046]    In this case, the enhanced posture parameter (RT) n ′ of the camera is obtained by using a template-based tracking algorithm. 
         [0047]    The template-based tracking algorithm obtains one or more images of a target object to be tracked and then continuously tracks the object from the images received through the camera in a frame-to-frame manner. 
         [0048]    The template-based tracking algorithm requires initialization of a relationship between the target object and the camera because of not having a function of recognizing the target object. While using the template-based tracking algorithm, the present invention performs optimization using the posture parameter (RT) n  of the camera as an initial value to obtain the enhanced posture parameter (RT) n ′ of the camera. 
         [0049]    Next, the terminal generates a front patch f f  viewed from the front by using the enhanced posture parameter (RT) n ′ of the camera and then measures a similarity between the front patch f f  and the front image If obtained by the object learning method. If the similarity is in excess of a threshold value, the terminal determines that the target object is equal to the learned object corresponding to the front image I f  (S 205 ). 
         [0050]    In the object tracking method using the object learning method according to the embodiment of the present invention, the step of obtaining the input patch for each frame of the input image inputted to the terminal through the camera in real time to the step of determining the similarity are performed to determine whether the target object is identical to the object learned during the object learning process. 
         [0051]    &lt;Object Learning And Tracking System&gt; 
         [0052]    An object learning and tracking system may be provided in a terminal (for example, a mobile terminal) to perform the object tracking method using the object learning method according to the embodiment of the present invention. 
         [0053]    An object learning and tracking system according to an embodiment of the present invention will now be described with reference to  FIG. 5 . 
         [0054]    As shown in  FIG. 5 , an object learning and tracking system according to an embodiment of the present invention includes an object learning unit  101 , an input patch obtaining unit  102 , a tracking determination unit  103 , a pseudo reference patch obtaining unit  104 , a camera posture enhancing unit  105 , and a similarity determination unit  106 . The object learning unit  101  receives an image of an object that is to be learned, obtains a front image If of the target object, generates a posture parameter (RT) n  of the camera with respect to m view points used for learning, generates images viewed from the view points using the front image, perform radial blur, and performs separation of an area used for learning to obtain a reference patch p n ′. The input patch obtaining unit  102  receives an input image for an object to be tracked, performs separation of an area used for tracking to obtain an input patch f n . The tracking determination unit  103  determines whether the object is already tracked in a previous frame. The pseudo reference patch obtaining unit  104  obtains a pseudo reference patch p n ′ having the least difference from the input patch fn in a current frame and a posture parameter (RT) n  of the camera if it is determined that the object is already tracked in the previous frame. The camera posture enhancing unit  105  performs optimization using the posture parameter (RT) n  of the camera in the current frame as an initial value to obtain an enhanced posture parameter (RT) n ′ of the camera if it is determined that the object is not tracked in the previous frame and performs optimization using the posture parameter (RT) n  of the camera in the previous frame as the initial value to obtain an enhanced posture parameter (RT) n ′ of the camera if it is determined that the object is already tracked. The similarity determination unit  106  generates a front patch f f  viewed from the front using the enhanced posture (RT) n ′ of the camera and measures a similarity between the front patch f f  and the front image I f  of the object learning unit  101  and, if the similarity is in excess of a threshold value, determines that the target object is equal to the learned object. 
         [0055]    Each component included in the object learning and tracking system according to the embodiment of the present invention will be described below in greater detail. 
         [0056]    Referring to  FIG. 5 , the object learning unit  101  receives an image I* of an object to be learned through a camera to obtain a front image If of the object, generates a posture parameter (RT) n  of the camera with respect to m view points used for learning, generate first images In viewed from the view points by using the front image If, perform radial blur and Gaussian blur, performs separation of an area used for object learning, and performs sampling and normalization to obtain a reference patch p n ′. The functions performed in the object learning unit  101  have been described above in connection with the object learning method, and thus, the detailed description will be omitted. 
         [0057]    The input patch obtaining unit  102  obtains a patch F i  for object tracking from an input image received through the camera, and samples and normalizes the patch F i  to obtain an input patch fn. 
         [0058]    The tracking determination unit  103  determines whether an object to be tracked in a current frame is already tracked. 
         [0059]    If the tracking determination unit  103  determines that the object is not tracked, the pseudo reference patch obtaining unit  104  obtains a pseudo reference patch p n ′ having the least difference from the input patch in the current frame and a posture parameter (RT) n  of the camera. 
         [0060]    If the tracking determination unit  103  determines that the object is not tracked, the camera posture enhancing unit  105  performs optimization using the posture parameter (RT) n  of the camera in the current frame as an initial value to obtain an enhanced posture parameter (RT) n ′ of the camera. If the tracking determination unit  103  determines that the object is already tracked, the camera posture enhancing unit  105  performs optimization using the posture parameter (RT) n  of the camera in the previous frame as the initial value to obtain the enhanced posture parameter (RT) n ′ of the camera. 
         [0061]    In this case, a template-based tracking algorithm may be used for the camera posture enhancing unit  105  to obtain the enhanced posture parameter of the camera. The template-based tracking algorithm has been described above in connection with the object tracking method, and thus, the detailed description will be omitted. 
         [0062]    The similarity determination unit  106  generates a front patch ff viewed from the front using the enhanced posture parameter (RT) n ′ of the camera and then measures a similarity between the front patch ff and the front image If. If the similarity is in excess of a threshold value, the similarity determination unit  106  determines that the target object is equal to the learned object. 
         [0063]    While this invention has been described in connection with what is presently considered to be practical exemplary embodiments, it is to be understood that the invention is not limited to the disclosed embodiments, but, on the contrary, is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims. Accordingly, the actual technical protection scope of the present invention must be determined by the spirit of the appended claims.