Patent Description:
Generally, to track an object, an object may be detected from an image acquired by a camera, representative feature points of the object may be extracted, and coordinates of the object may be extracted for each frame based on the extracted feature points. For more comfortable viewing of a three-dimensional (3D) image, 3D coordinates of both eyes may be required. To acquire the 3D coordinates, two-dimensional (2D) information relating to the eyes, rotation information relating to a face, and an interpupillary distance may be used. The 3D coordinates may be extracted for each frame, and thus positions of the eyes may be tracked and a 3D image may be generated based on the positions of the eyes. <NPL> relates to advanced surveillance systems, which combine video and thermal imagery for pedestrian detection. In such a surveillance system according to <NPL>, visible spectrum video cameras and thermal infrared (IR) cameras are located around premises of interest. In a first step, a database of known scenarios both indoor and outdoor with a few pedestrians is collected. These image sequences are synchronized, geometrically corrected and temperature calibrated. In a next step, the regions of interest corresponding to the pedestrians in the images are extracted. Subsequently, the regions of interest are grouped from image to image separately for both video and thermal IR sequences before a fusion algorithm proceeds to track and detect humans. Hereby, for each pair of objects, a best object detected in either the visible or the IR images is identified, which is called master, whereas the other one is called slave. Said identification of the master and the slave changes rapidly for an object when fast light illumination or temperature variation are present. Like this, a more robust performance of the surveillance system is insured. <CIT> refers to an eye-tracking system for determining the line of sight of a user. Said eye-tracking system comprises a camera to record the user's eyes, a source of light to illuminate the user's eyes with light of two different wavelengths, as well as a control unit. Said control unit is configured to recognize that the illumination of a user with light of a first wavelength is not sufficient to determine the line of sight of the user with high-enough quality based on extracting a signal-to-noise ratio (SNR) of the image. In response to said recognition, the control unit is further configured to illuminate the users with light of a second wavelength and thus to determine a line of sight of the user.

It is the object of the present application to provide systems and methods for tracking objects in an accurate and efficient way.

According to an aspect of an exemplary embodiment, there is provided an object tracking method including detecting a target object in a first-type input image that is based on light in a first wavelength band, tracking the target object in the first-type input image based on detection information of the target object when the target object is detected in the first-type input image, measuring a reliability of the first-type input image by comparing the first-type input image to an image in a first database (DB), and tracking the target object in a second-type input image based on the detection information when the reliability of the first-type input image is lower than a first threshold, the second-type input image being based on light in a second wavelength band, different than the first wavelength band.

The first wavelength band may include visible light and the second wavelength band may include infrared (IR) light. The first-type input image and the second-type input image may be acquired by a camera from which an IR-cut filter is removed for acquisition of the second-type input image. The object tracking method may further include controlling an IR light source configured to output IR light when the reliability of the first-type input image is lower than the first threshold. The first wavelength band may include visible light. The image stored in the first DB may have a reliability higher than a predetermined threshold, and may be a first reference image acquired based on light in the first wavelength band.

The object tracking method may further include measuring a reliability of the second-type input image by comparing the second-type input image to an image stored in a second DB, and detecting the target object from the first-type input image or the second-type input image when the reliability of the second-type input image is lower than a second threshold. The image stored in the second DB may have a reliability higher than a predetermined threshold, and may be a second reference image acquired based on light in the second wavelength band.

The object tracking method may further include detecting the target object in the second-type input image when the target object is not detected in the first-type input image. The object tracking method may further include tracking the target object in the second-type input image based on the detection information when the target object is detected in the second-type input image, measuring a reliability of the second-type input image by comparing the second-type input image to an image stored in a second DB, and tracking the target object in the first-type input image based on the detection information when the reliability of the second-type input image is lower than a second threshold.

The detecting of the target object on the first-type input image may be performed using a first detector trained in advance based on error data. The error data may include at least one of data obtained when a detection of the target object is not completed and data obtained when another object is incorrectly detected as the target object, from among training data.

The detection information may include a detection area corresponding to a location of the target object in a first frame of the first-type input image. The tracking of the target object in the first-type input image may include tracking the target object in a second frame of the first-type input image based on the detection area.

According to an aspect of another exemplary embodiment, there is provided an object tracking apparatus including a processor, and a memory including an instruction that is readable by the processor, wherein the instruction is executed by the processor, and the processor is thereby configured: to detect a target object in a first-type input image that is based on light in a first wavelength band, to track the target object in the first-type input image based on detection information of the target object when the target object is detected in the first-type input image, to measure a reliability of the first-type input image by comparing the first-type input image to an image stored in a first DB, and to track the target object in a second-type input image based on the detection information when the reliability of the first-type input image is lower than a first threshold, the second-type input image being based on light in a second wavelength band, different from the first wavelength band.

The above and/or other exemplary aspects will be more apparent by describing certain exemplary embodiments with reference to the accompanying drawings, in which:.

The following describes exemplary embodiments.

Hereinafter, exemplary embodiments will be described in detail below with reference to the accompanying drawings, and like reference numerals refer to the like elements throughout the present specification.

<FIG> is a block diagram illustrating an object tracking apparatus <NUM> according to an exemplary embodiment. Referring to <FIG>, the object tracking apparatus <NUM> includes an image processing apparatus <NUM>, a camera <NUM> and an infrared (IR) light source <NUM>. The object tracking apparatus <NUM> may detect a target object <NUM> from an input image acquired by the camera <NUM>, and may track the detected target object <NUM>. For example, the target object <NUM> may be the eyes of a user, and the object tracking apparatus <NUM> may track the eyes as the target object <NUM> in the input image, with a high accuracy. The target object <NUM> may alternately include, for example, an object such as a vehicle, a bicycle or a body part such as a face or a hand other than the eyes. In the following description, an example in which the target object <NUM> corresponds to a user's eyes will be described; however, the target object <NUM> may be an object other than the eyes.

For a glasses-free three-dimensional (3D) display, eye positions of a user may be required. A glasses-free 3D apparatus may track the eyes of a user using a camera and may output a 3D image corresponding to the positions of the eyes. A 3D heads-up display (HUD) may display, on a windshield, navigation information, information used to assist with driving in bad weather, and dangerous situations or hazards. Because an accurate representation of 3D information on a road is important in the 3D HUD, the eye positions may need to continue to be precisely detected. For example, eye positions may need to continue to be precisely detected even in a low illumination environment or an environment in which an obstacle such as glasses exists. When incorrect 3D information is provided to a user due to a crosstalk, a life-threatening situation such as a traffic accident may occur. Therefore, the object tracking apparatus <NUM> must be able to track the target object <NUM> in an input image captured in various environments, for example, a low illumination environment or an environment in which an obstacle such as glasses exists.

The camera <NUM> may capture the target object <NUM> and may provide the input image to the image processing apparatus <NUM>. The image processing apparatus <NUM> may track the target object <NUM> in the input image and may determine coordinates of the target object <NUM>. The camera <NUM> may be, for example, a single camera or a stereo camera. When the camera <NUM> is a single camera, the image processing apparatus <NUM> may extract 2D coordinates of the target object <NUM> from the input image, may combine the 2D coordinates with an interpupillary distance (IPD) of a user, and may determine 3D coordinates of the target object <NUM>. When the camera <NUM> is a stereo camera, the image processing apparatus <NUM> may extract 2D coordinates of the target object <NUM> from input images acquired in at least two positions, and may determine 3D coordinates of the target object <NUM> using a triangulation scheme.

The camera <NUM> may generate a first-type input image that is of a first type and that is based on a light of a first wavelength band, and may generate a second-type input image that is of a second type and that is based on a light of a second wavelength band. The camera <NUM> may function as a visual camera that uses visible rays and as an IR camera that uses IR rays. For example, the camera <NUM> may be a hybrid camera that may use both visible rays and IR rays. For example, an IR cut filter may be removed from the camera <NUM>, and the camera <NUM> may capture the target object <NUM> using visible rays in an environment in which visible rays are provided, and may capture the target object <NUM> using IR rays in an environment in which IR rays are provided. The camera <NUM> may be, for example, a hybrid type stereo camera.

The visual camera may be limited in its ability to track the target object <NUM> under a low illumination. For example, to track the target object <NUM> using the visual camera in a low illumination environment, a frame rate of the visual camera may be lowered or an aperture may be opened. In this example, due to a low frame rate, camera latency or image blurring may occur. The IR camera may also be used in the low illumination environment. However, when the IR camera is used, a problem of safety may occur due to continuous use of IR rays, white spots may appear around eyes due to glasses, or the accuracy of detecting the target object <NUM> may decrease in an environment in which strong external light exists.

Therefore, the camera <NUM> may properly operate as a visual camera or an IR camera depending on the circumstances. For example, when the image processing apparatus <NUM> fails to detect the target object <NUM> from an input image that is based on visible rays, the camera <NUM> may capture the target object <NUM> based on IR rays. Thus, when the camera <NUM> operates as an IR camera, the IR light source <NUM> may be activated for IR capturing, and the activated IR light source <NUM> may provide light of an IR wavelength band. Because the IR cut filter is removed from the camera <NUM>, the camera <NUM> may capture the target object <NUM> based on IR rays. Also, when the image processing apparatus <NUM> determines that a reliability of the input image based on the visible rays has decreased, the camera <NUM> may capture the target object <NUM> based on IR rays.

Use of a visible-ray image under a high illumination is not necessarily efficient for tracking. Also, using of an IR-ray image under a low illumination is not necessarily efficient for tracking. For example, even under a low illumination, use of a visible-ray image may be efficient for tracking. Thus, it may be difficult to guarantee an accuracy of tracking by determining a wavelength band that is to be used by camera <NUM> based on an illumination value alone.

For example, the reliability of an input image may be measured using a reliability measurer that is trained to output the reliability of the input image, and a modality may be switched based on the reliability of the input image. In this example, the modality may refer to an operation or a device associated with a predetermined wavelength band. A high reliability of the input image may include a high reliability of a training process using the input image. For example, when a low reliability is measured using the reliability measurer, a modality may be switched to enhance the reliability of a training process. When the reliability measurer is used, the tracking accuracy may be enhanced, in comparison to simply depending on an illumination.

The object tracking apparatus <NUM> may be used to track eye positions of a driver for a 3D HUD of a vehicle, or to track eye positions of a viewer for a 3D display of a display device such as a television (TV) or a mobile device. Also, the object tracking apparatus <NUM> may be used to monitor a driver's viewpoint and a driver's gaze tracking status.

The object tracking apparatus <NUM> may detect the target object <NUM> in a detection mode, and may track the target object <NUM> in a tracking mode based on area information of the detected target object <NUM>. For example, when an object is detected from a first frame, the object tracking apparatus <NUM> may generate detection information and may track the target object <NUM> in a second frame based on the detection information. In this example, the second frame may be a frame next to the first frame, and the detection information may include a detection area corresponding to the detected target object <NUM>. When the object tracking apparatus <NUM> enters a tracking mode, the target object <NUM> may be detected using only a limited number of areas of the input image rather than all of the areas of the input image. Thus, resources to detect the target object <NUM> may be saved.

The object tracking apparatus <NUM> may use a detector that has been trained based on error data in the detection mode. The error data may refer to training data corresponding to a relatively high level of object detection difficulty. The detection performance of the detector may be enhanced by training the detector based on the error data. For example, the error data may include at least one of data obtained when a detection of the target object <NUM> is not completed and data obtained when another object is incorrectly detected as the target object <NUM>, from among training data. The expression "detection of the target object <NUM> is not completed" may encompass any failure in the detection of the target object <NUM>.

The object tracking apparatus <NUM> may use a tracker trained based on quality in the tracking mode. A quality measurer may classify input images based the quality of the input images. For example, the quality measurer may classify qualities of input images as one of a high quality, a medium quality, and a low quality. The quality of an input image may include a level of tracking difficulty. The tracker may include a first tracker trained to track the target object <NUM> in a high-quality input image, a second tracker trained to track the target object <NUM> in a medium-quality input image, and a third tracker trained to track the target object <NUM> in a low-quality input image. When the quality measurer measures the quality of an input image, a tracker corresponding to the measured quality may track the target object <NUM>. When a tracker trained based on quality is used, the tracking accuracy may be enhanced.

<FIG> is a flowchart illustrating an object tracking method according to an exemplary embodiment. In the following description, a wavelength band used by a visible-ray camera, that is, a band including a visible wavelength band may be referred to as a "first wavelength band," and a wavelength band used by an IR camera, that is, a band including an IR wavelength band may be referred to as a "second wavelength band. " Also, an operation or a device associated with the first wavelength band may be referred to as a "first modality," and an operation or a device associated with the second wavelength band may be referred to as a "second modality.

The following description may be applied to both an example in which a camera configured to provide an input image is a single camera and an example in which the camera is a stereo camera. For example, when a single camera is used, the following description may be applicable to the single camera. When a stereo camera is used, the following description may be applicable to cameras of the stereo camera.

Referring to <FIG>, in operation <NUM>, an object tracking apparatus detects a target object from an input image based on a current modality. The current modality may be the first modality or the second modality. For example, when the current modality is the first modality, the object tracking apparatus may acquire a first-type input image that is based on light of the first wavelength band, and may detect a target object from the first-type input image. The current modality may be switched based on a predetermined condition in operation <NUM> or <NUM>. Although the current modality is presumed to be the first modality in the following description, the following description is equally applicable to an example in which the current modality is the second modality.

In operation <NUM>, the object tracking apparatus determines whether the target object is detected from the first-type input image. When the target object is detected from the first-type input image, an operating mode of the object tracking apparatus may be changed from a detection mode to a tracking mode, and operation <NUM> may be performed. When the target object is not detected from the first-type input image, operations <NUM> and <NUM> may be performed. Hereinafter, an example in which the target object is not detected from the first-type input image and an example in which the target object is detected from the first-type input image in operation <NUM> will be further described.

Example in which Target object is not detected from First-type input image.

When the target object is not detected from the first-type input image, the object tracking apparatus controls at least one of a camera and a light source in operation <NUM>, and switches a modality in operation <NUM>. For example, when the target object is not detected from the first-type input image, the object tracking apparatus may activate an IR light source, and may switch the current modality from the first modality to the second modality. Also, in operation <NUM>, one or more of an aperture, a shutter speed, and an ISO of the camera may be adjusted.

In the first modality, an operation based on the first-type input image that is based on the light of the first wavelength band may be performed. In the second modality, an operation based on a second-type input image that is based on a light of the second wavelength band may be performed. For example, when the current modality is switched from the first modality to the second modality in operation <NUM>, the object tracking apparatus may acquire the second-type input image and may detect the target object from the second-type input image in operation <NUM>. The object tracking apparatus may determine whether the target object is detected from the second-type input image in operation <NUM>. When the target object is detected from the second-type input image, the operating mode may be changed from the detection mode to the tracking mode, and operation <NUM> may be performed. When the target object is not detected from the second-type input image, the object tracking apparatus may repeat operations <NUM> and <NUM>.

When the target object is detected from the second-type input image, the object tracking apparatus may track the target object, in operation <NUM>, in the second-type input image based on detection information of the target objet. The detection information may be generated in response to the target object being detected from the second-type input image in operation <NUM>. The object tracking apparatus may measure the reliability of the second-type input image by comparing an image stored in a second database (DB) to the second-type input image. The reliability of the image is then compared to a second threshold. In operation <NUM>. When the reliability of the second-type input image is lower than the second threshold, operations <NUM>, <NUM> and <NUM> may be performed, as discussed in further detail below. A threshold compared to a reliability of the first-type input image may be referred to as a "first threshold," and a threshold compared to the reliability of the second-type input image may be referred to as a "second threshold.

When the current modality is switched from the second modality to the first modality based on a result indicating that the reliability of the second-type input image is lower than the second threshold in operation <NUM>, the object tracking apparatus may track the target object in the first-type input image in operation <NUM>. The images stored in the second DB may have a reliability higher than a predetermined threshold, and may include at least one second reference image acquired based on the light of the second wavelength band. Thus, the reliability of the second-type input image may be determined to increase as its similarity to the at least one second reference image increases.

The object tracking apparatus may track the target object based on the detection information. The detection information may be generated in response to the target object being detected from the second-type input image in operation <NUM>, as described above. For example, the detection information may be used regardless of the modality in which the detection information is generated. For example, when detection information is generated in the second modality, the generated detection information may also be used in the first modality. A detection area in the detection information may be a predetermined area, because the detection information may be used regardless of the current modality when input images have equal sizes.

Example in which Target object is detected from First-type input image.

When the target object is detected from the first-type input image, the object tracking apparatus may generate detection information including a detection area. In operation <NUM>, the object tracking apparatus may acquire a next frame of the first-type input image and may track the target object in the acquired frame. The object tracking apparatus may track the target object based on the detection information.

The object tracking apparatus may measure the reliability of the first-type input image by comparing an image stored in a first DB to the first-type input image. The reliability of the first-type input image may then be compared to the first threshold, in operation <NUM>. The image stored in the first DB may have a reliability higher than a predetermined threshold, and may include at least one first reference image acquired based on the light of the first wavelength band. Thus, the reliability of the first-type input image may be determined to increase as its similarity to the at least one first reference image increases.

When the reliability of the first-type input image is higher than the first threshold, operation <NUM> may be performed again. For example, when the reliability of the first-type input image is higher than the first threshold, the tracking mode may be maintained based on the first modality. The object tracking apparatus may acquire a next frame of the first-type input image and may track the target object in the acquired frame. When a high reliability of the first-type input image continues to be measured, the object tracking apparatus may continue to acquire consecutive frames of the first-type input image and may continue to track the target object in the first-type input image.

When the reliability of the first-type input image is lower than the first threshold, it is determined, in operation <NUM>, whether all modalities have been checked, and operations <NUM> and <NUM> may be performed, as discussed in further detail below. All the modalities may include the first modality and the second modality. For example, despite a low reliability based on the first modality, the tracking mode may be performed in the second modality instead of being immediately released. When low reliability is measured in both the first modality and the second modality, the tracking mode may be released and the detection mode may be performed again. In other words, when all the modalities are determined to be checked in operation <NUM>, the detection mode may be performed again in operation <NUM>. When fewer than all of the modalities have been checked, operations <NUM> and <NUM> may be performed.

In operation <NUM>, the object tracking apparatus may control at least one of the light source and the camera, in operation <NUM>, the object tracking apparatus may switch the modality in operation <NUM>. For example, when the target object is not detected from the first-type input image, the object tracking apparatus may activate an IR light source in operation <NUM> and may switch the current modality from the first modality to the second modality in operation <NUM>. The above description of operations <NUM> and <NUM> may also be applicable to operations <NUM> and <NUM>.

When the current modality is switched from the first modality to the second modality in operation <NUM>, the object tracking apparatus may acquire a next frame from the second-type input image and may track the target object in the acquired frame in operation <NUM>. The object tracking apparatus may track the target object based on the detection information. The detection information may be generated in response to the target object being detected in the first-type input image in operation <NUM>, as described above.

The object tracking apparatus may measure the reliability of the second-type input image by comparing an image stored in the second DB to the second-type input image. The reliability of the second-type input image may then be compared to the second threshold in operation <NUM>. When the reliability of the second-type input image is lower than the second threshold, operation <NUM> may be performed. When both the first modality and the second modality have been checked, the tracking mode may be released and the detection mode may be repeated in operation <NUM>. When the detection mode is performed, the current modality may be maintained or switched. In the above example, the object tracking apparatus may continue to operate in the second modality, or may operate in the first modality by switching the current modality from the second modality to the first modality. Thus, in operation <NUM>, the object tracking apparatus may detect the target object from the first-type input image or the second-type input image based on the current modality.

<FIG> is a diagram illustrating an operation of a reliability measurer <NUM> according to an exemplary embodiment. Referring to <FIG>, the reliability measurer <NUM> may compare an input image and an image stored in a DB <NUM> and may thereby determine and output a reliability of the input image. An object tracking apparatus may measure the reliability of the input image using the reliability measurer <NUM>, or may directly perform an operation of the reliability measurer <NUM> that will be described below.

The DB <NUM> may include a first DB <NUM> and a second DB <NUM>. When the input image corresponds to a first modality, the reliability measurer <NUM> may compare the input image and an image stored in the first DB <NUM>. When the input image corresponds to a second modality, the reliability measurer <NUM> may compare the input image and an image stored in the second DB <NUM>. The one or more images stored in the first DB <NUM> may have a reliability higher than a predetermined criterion and may include at least one first reference image acquired based on a light of a first wavelength band. The one or more images stored in the second DB <NUM> may have a reliability higher than a predetermined threshold and may include at least one second reference image acquired based on a light of a second wavelength band. For example, the DB <NUM> may include reference images classified with a high reliability by a tracker trained based on a quality.

<FIG> is a diagram illustrating a process of measuring a reliability of an input image according to an exemplary embodiment. A reference image <NUM> may be acquired from the first DB or the second DB based on the modality of an input image <NUM>. For example, when the input image <NUM> is of a second type, the reference image <NUM> may be acquired from the second DB. A number of reference images <NUM> stored in the first and second DBS may be determined based on, for example, a required tracking accuracy or a desired performance of the object tracking apparatus.

The object tracking apparatus may extract global features <NUM> and local features <NUM> from the reference images <NUM>, and may extract global features <NUM> and local features 423from the input image <NUM>. For example, it is assumed that a number of global features <NUM> and <NUM> is denoted by "l," that a number of local features <NUM> and <NUM> is denoted by "m" and that "l + m = n. " In this example, n denotes a total number of both global and local features extracted from the reference images <NUM> and a number of features extracted from the input image <NUM>. Also, l, m and n may be natural numbers.

The object tracking apparatus may calculate a mean value (mean) and a standard value (std) in association with each of the global features <NUM> and <NUM> and each of the local features <NUM> and <NUM>. For example, the object tracking apparatus may calculate mean values G_gf1_mean through G_gfl_mean and standard values G_gf1_std through G_gfl_std in association with the global features <NUM>, and may calculate mean values I_gf1_mean through I_gfl_mean and standard values I_gf1_std through I_gfl_std in association with the global features <NUM>. Also, the object tracking apparatus may calculate mean values G_lf1_mean through G_lfm_mean and standard values G_lf1_std through G_lfm_std in association with the local features <NUM>, and may calculate mean values I_lf1_mean through I_lfm_mean and standard values I_lf1_std through I_lfm_std in association with the local features <NUM>.

The object tracking apparatus may calculate a distance between features based on the calculated mean values and the calculated standard values. For example, the distance may be calculated using Equation <NUM>, shown below.

In Equation <NUM>, d_i denotes a distance between an i-th feature of a reference image and an i-th feature of an input image, I_gfi_mean denotes a mean value of an i-th global feature of the input image, I_gfi_std denotes a standard value of the i-th global feature of the input image, G_gfi_mean denotes a mean value of an i-th global feature of the reference image, and G_gfi _std denotes a standard value of the i-th global feature of the reference image. Also, I_lfi_mean denotes a mean value of an i-th local feature of the input image, I_lfi_std denotes a standard value of the i-th local feature of the input image, G_lfi_mean denotes a mean value of an i-th local feature of the reference image, and G_lfi_std denotes a standard value of the i-th local feature of the reference image.

The object tracking apparatus may calculate scores of features based on a distance between features. For example, a score of a feature may be calculated using Equation <NUM> shown below.

In Equation <NUM>, F_score_i denotes a score of an i-th feature, d_i denotes a distance between an i-th feature of a reference image and an i-th feature of an input image, d_i_max denotes a maximum value of the distance d_i, and d_i_min denotes a minimum value of the distance d_i.

The object tracking apparatus may calculate a reliability of an input image using a weighted average of scores of features. For example, the reliability of the input image may be calculated using Equation <NUM> shown below.

In Equation <NUM>, S denotes a reliability of an input image, F_score_k denotes a k-th feature score, wk denotes a k-th weight, n denotes a number of extracted features, and k denotes an operation index. The object tracking apparatus may measure the reliability of the input image using the process of <FIG>.

<FIG> is a diagram illustrating a process of tracking an object using a tracking area according to an exemplary embodiment. To track a target object, an object tracking apparatus may determine a detection area estimated to correspond to the target object in a frame F1. A location and a size of the detection area may be specified. For example, the size of the detection area may be set in advance, and the location of the detection area may be determined by the object tracking apparatus.

The object tracking apparatus may align the target object by extracting feature points of the target object from the detection area in the frame F1. For example, the object tracking apparatus may extract feature points representing a shape of the target object from a portion of an image that corresponds to the detection area from the frame F1, to identify a geometric structure of the target object. When the target object is aligned, the object tracking apparatus may determine a tracking area <NUM> used to track the target object, based on the extracted feature points. For example, the object tracking apparatus may determine, as a tracking area, an area that includes the feature points on a central portion of the area. The object tracking apparatus may track a target object in a frame F2 based on the tracking area <NUM>. When a reliability of the frame F1 is higher than a threshold, an object detection may be omitted in the frame F2.

A target object in the frame F2 may be located further upward and further rightward in comparison to a location of the target object in the frame F1. The object tracking apparatus may extract feature points of the target object from the tracking area <NUM> in the frame F2. The object tracking apparatus may determine a new tracking area <NUM> in the frame F2 based on the feature points extracted from the tracking area <NUM>. For example, the object tracking apparatus may determine, as the tracking area <NUM>, an area including the feature points extracted from the tracking area <NUM> in a central portion of the area. Similarly to the frame F2, in the frame F3, feature points of the target object may be extracted from the tracking area <NUM>, and a new tracking area <NUM> may be determined. As described above, when a reliability of an input image is higher than a threshold, the object tracking apparatus may continue to track the target object in a tracking mode.

The object tracking apparatus may minimize the use of a detector during tracking of a target object. Due to the consumption of computing resources by a detecting operation of the detector, the use of the detector may be minimized using the object tracking apparatus, as described. Since a detector scans all areas of an input image to detect a target object, a large amount of computing resources may be consumed for the detection operation of the detector.

The object tracking apparatus may output location information of a target object included in an input image while tracking the target object. The location information of the target object may include, for example, eye positions of a user. For example, the object tracking apparatus may track a target object in a plurality of frames included in the input image, and may output the eye positions of a user for each of the frames. Eye positions may be specified by 2D or 3D coordinates.

<FIG> is a block diagram illustrating a tracker <NUM> using a quality measurement according to an exemplary embodiment. Referring to <FIG>, the tracker <NUM> includes three sub trackers: a first tracker <NUM>, a second tracker <NUM> and a third tracker <NUM>. For convenience of description, the tracker <NUM> includes three sub-trackers in the following description, and as illustrated in <FIG>, however, the tracker <NUM> includes two sub-trackers or at least four sub-trackers. Sub-trackers are trained for use with images with different qualities. For example, the first tracker <NUM> is trained for use with a high-quality image, the second tracker <NUM> is trained for use with a medium-quality image, and the third tracker <NUM> is trained for use with a low-quality image.

An object tracking apparatus measures a quality of an input image, and transmits the input image and quality information of the input image to the tracker <NUM>. The object tracking apparatus measures the quality of the input image using a quality measurer <NUM>. The tracker <NUM> selects a sub-tracker corresponding to the quality information of the input image from among multiple sub-trackers, and provides the input image to the selected sub-tracker. For example, when an input image has a high quality, the tracker <NUM> provides the input image to the first tracker <NUM> trained for use with a high-quality image. When the input image is provided to the sub-tracker, the sub-tracker tracks a target object in the input image.

<FIG> is a block diagram illustrating a training apparatus <NUM> according to an exemplary embodiment. Referring to <FIG>, the training apparatus <NUM> includes a processor <NUM> and a memory <NUM>. The memory <NUM> can include a neural network <NUM>, and can store instructions readable by the processor <NUM>. The neural network <NUM> can correspond to a detector, a tracker, a reliability measurer, and a quality measurer. When the instructions are executed by the processor <NUM>, the processor <NUM> can train the neural network <NUM>. The training of the neural network <NUM> can include training of parameters of the neural network <NUM>, updating the neural network <NUM>, and/or updating parameters of the neural network <NUM>. The memory <NUM> may store data required for a training process and for a neural network <NUM> that is completely trained.

<FIG> is a diagram illustrating a training process of detectors according to an exemplary embodiment. Referring to <FIG>, a first sample detector <NUM> may detect a target object from first training data, and first error data <NUM> may be formed based on an output of the first sample detector <NUM>. The first training data may include images that are based on a first wavelength band (hereinafter, referred to as "first wavelength band-based images"), and the first sample detector <NUM> may be trained in advance to detect a target object from the first wavelength band-based images. A second sample detector <NUM> may detect a target object from second training data, and second error data <NUM> may be formed based on an output of the second sample detector <NUM>. The second training data may include images that are based on a second wavelength band (hereinafter, referred to as "second wavelength band-based images"), and the second sample detector <NUM> may be trained in advance to detect a target object from the second wavelength band-based images.

An error DB <NUM> may store the first error data <NUM> and the second error data <NUM>. Error data may refer to training data corresponding to a relatively high level of object detection difficulty, and the detection performance of a detector may be enhanced by training the detector based on the error data. For example, the error data may include at least one of data obtained when the detection of a target object is not completed and data obtained when another object is incorrectly detected as a target object, from among training data. The first error data <NUM> may include an image with a relatively high level of object detection difficulty from among the first wavelength band-based images, and the second error data <NUM> may include an image with a relatively high level of object detection difficulty from among the second wavelength band-based images.

A first detector <NUM> may be trained based on the first error data <NUM>, and a second detector <NUM> may be trained based on the second error data <NUM>. Thus, the first detector <NUM> may be trained to have the capability of detecting a target object from an image with a relatively high level of object detection difficulty from among the first wavelength band-based images, and the second detector <NUM> may be trained to have the capability of detecting a target object from an image with a relatively high level of object detection difficulty from among the second wavelength band-based images. The object tracking apparatus may use the first detector <NUM> to detect a target object from a first-type input image, and may use the second detector <NUM> to detect a target object from a second-type input image. Thus, the object tracking apparatus may detect a target object from an input image using detectors trained for each of a number of different modalities.

<FIG> is a diagram illustrating a process of training a tracker to track a target object in an input image that is based on a light of a first wavelength band according to an exemplary embodiment. Referring to <FIG>, training data <NUM> may be classified as high-quality data <NUM>, medium-quality data <NUM>, or low-quality data <NUM>. The training data <NUM> may include images captured using light of the first wavelength band. The training data <NUM> may be classified by a quality measurer. For example, an image that clearly represents eyes may be classified as high-quality data <NUM>, and an image that less clearly represents eyes may be classified as medium-quality data <NUM>. Also, an image captured under a low illumination, or an image that unclearly represents centers of eyes may be classified as low-quality data <NUM>.

A first tracker <NUM> may be trained based on the high-quality data <NUM>, a second tracker <NUM> may be trained based on the medium-quality data <NUM>, and a third tracker <NUM> may be trained based on the low-quality data <NUM>. Each of the first tracker <NUM> through the third tracker <NUM> may be referred to as a sub-tracker, and the first tracker <NUM> through the third tracker <NUM> that have been completely trained may respectively correspond to the first tracker <NUM> through the third tracker <NUM> of <FIG>.

For convenience of description, a tracker including sub trackers that have been trained based on particular qualities of data may be referred to as a "multi-model. " Such a multi-model may exhibit tracking performance that is higher than that of a single model. When the first DB <NUM> of <FIG> is formed based on a multi-model, the reliability of an input image may be measured with a high performance. For example, a multi-model may test a predetermined training data set and classify the training data set for each test error. When the training data set is divided into a first group with a small amount of error, a second group with a medium amount of error, or a third group with a large amount of error, the first DB <NUM> may store the data, included in the first group, having a small amount of error.

<FIG> is a diagram illustrating a process of training a tracker to track a target object in an input image that is based on a light of a second wavelength band according to an exemplary embodiment. Referring to <FIG>, training data <NUM> may be classified as high-quality data <NUM>, medium-quality data <NUM>, or low-quality data <NUM>. The training data <NUM> may include images captured using light of the second wavelength band. The training data <NUM> may be classified by a quality measurer. For example, an image that clearly represents eyes may be classified as high-quality data <NUM>, and an image that includes a weak reflection from glasses or that less clearly represents eyes may be classified as medium-quality data <NUM>. Also, an image that includes a strong reflection from glasses or that unclearly represents centers of eyes may be classified as low-quality data <NUM>.

A fourth tracker <NUM> may be trained based on the high-quality data <NUM>, a fifth tracker <NUM> may be trained based on the medium-quality data <NUM>, and a sixth tracker <NUM> may be trained based on the low-quality data <NUM>. The second DB <NUM> of <FIG> may be formed based on a multi-model that includes the fourth tracker <NUM> through the sixth tracker <NUM>. For example, the multi-model may test a predetermined training data set and divide the training data set into a fourth group with a small amount of error, a fifth group with a medium amount of error and a sixth group with a large amount of error, and the second DB <NUM> may store the data include in the fourth group.

<FIG> is a flowchart illustrating an object tracking method using a stereo camera according to an exemplary embodiment. The stereo camera may generate a first input image of a first type using a first camera, and may generate a second input image of the first type using a second camera. For example, the first camera and the second camera may generate a first input image of a first type and a second input image of the first type using light of a first wavelength band in a first modality. Operations <NUM>, <NUM>, <NUM>, <NUM> and <NUM> may be performed based on the first input image, and operations <NUM>, <NUM>, <NUM>, <NUM> and <NUM> may be performed based on the second input image. Operations <NUM> through <NUM> and operations <NUM> through <NUM> may be synchronized with each other. An example in which a current modality is the first modality will be described below.

In operation <NUM>, an object tracking apparatus acquires the first input image and detects a target object from the first input image. In operation <NUM>, the object tracking apparatus acquires the second input image and detects a target object from the second input image. In operations <NUM> and <NUM>, the object tracking apparatus determines whether the target object is detected. When the target object is detected, operations <NUM> and <NUM> may be performed. When the target object is not detected, operations <NUM> and <NUM> may be performed. When the target object is not detected from either the first input image or the second input image, operations <NUM> and <NUM> may be performed. In operations <NUM> and <NUM>, the object tracking apparatus controls at least one of a light source and a camera, and switches a modality.

In operation <NUM>, the object tracking apparatus acquires a next frame of the first input image, and tracks the target object in the next frame of the first input image. In operation <NUM>, the object tracking apparatus acquires a next frame of the second input image, and tracks the target object in the next frame of the second input image. The object tracking apparatus may track the target object based on detection information.

In operations <NUM> and <NUM>, the object tracking apparatus measures a reliability of each of the first input image and the second input image, and compares the measured reliability to a threshold. When the reliability is higher than the threshold, operations <NUM> and <NUM> may be performed. When the reliability is lower than the threshold, operations <NUM> and <NUM> may be performed. When either a reliability of the first input image or a reliability of the second input image is lower than the threshold, operations <NUM> and <NUM> may be performed. When the reliability lower than the threshold is measured and when all modalities are checked, operations <NUM> and <NUM> may be performed. The above description of <FIG> may also be applicable to the object tracking method of <FIG>.

<FIG> is a block diagram illustrating an image processing apparatus <NUM> according to an exemplary embodiment. Referring to <FIG>, the image processing apparatus <NUM> includes a processor <NUM> and a memory <NUM>. The memory <NUM> may include data for object tracking, and instructions readable and executable by the processor <NUM>. The memory <NUM> may include software for enabling the processor or implement a detector, a tracker, a reliability measurer and a quality measurer that are completely trained. When the instructions in the memory <NUM> are executed by the processor <NUM>, the processor <NUM> may perform the operations for object tracking. For example, the processor <NUM> may detect a target object from a first-type input image that is based on a light of a first wavelength band. When the target object is detected from the first-type input image, the processor <NUM> may track the target object in the first-type input image based on detection information of the target object. The processor <NUM> may measure a reliability of the first-type input image by comparing the first-type input image to a first DB. When the reliability of the first-type input image is lower than a first threshold, the processor <NUM> may track the target object in a second-type input image based on the detection information. The second-type input image may be based on a light of a second wavelength band. The above description of <FIG> is also applicable to the image processing apparatus <NUM>.

The exemplary embodiments described herein may be implemented using hardware components, software components, or a combination thereof. A processing device may be implemented using one or more general-purpose or special purpose computers, such as, for example, a processor, a controller and an arithmetic logic unit, a digital signal processor, a microcomputer, a field programmable array, a programmable logic unit, a microprocessor or any other device capable of responding to and executing instructions in a defined manner. The processing device may run an operating system (OS) and one or more software applications that run on the OS. The processing device also may access, store, manipulate, process, and create data in response to execution of the software. For purpose of simplicity, the description of a processing device is used as singular; however, one skilled in the art will appreciated that a processing device may include multiple processing elements and multiple types of processing elements. For example, a processing device may include multiple processors or a processor and a controller. In addition, different processing configurations are possible, such a parallel processors.

The software may include a computer program, a piece of code, an instruction, or some combination thereof, to independently or collectively instruct or configure the processing device to operate as desired. Software and data may be embodied permanently or temporarily in any type of machine, component, physical or virtual equipment, computer storage medium or device, capable of providing instructions or data to or being interpreted by the processing device.

Claim 1:
An object tracking method comprising:
detecting (<NUM>) a target object (<NUM>) in a first-type input image that is based on light in a first wavelength band;
tracking (<NUM>) the target object (<NUM>) in the first-type input image based on detection information of the target object (<NUM>) and by selecting a first tracker (<NUM>) corresponding to a measured quality of the first-type input image from among multiple trackers (<NUM>, <NUM>, <NUM>), wherein
the first tracker (<NUM>) is trained using images that are captured using light of the first wavelength band and that have a quality that is specific for the first tracker (<NUM>), and
providing the first-type input image to the first tracker (<NUM>);
measuring (<NUM>) a reliability of the first-type input image by comparing the first-type input image to at least one image stored in a first database, DB (<NUM>); and
when the reliability of the first-type input image is lower than a first threshold, controlling (<NUM>) at least one of a light source (<NUM>) and camera (<NUM>) and switching (<NUM>) a first modality referring to an operation associated with the first wavelength band to a second modality referring to an operation associated with a second wavelength band different from the first wavelength band to track the target object (<NUM>) in a second-type input image based on light in the second wavelength band using the detection information and by selecting a second tracker (<NUM>, <NUM>) corresponding to a measured quality of the second-type input image from among multiple trackers (<NUM>, <NUM>, <NUM>), wherein the second tracker (<NUM>, <NUM>) is trained using images that are captured using light of the second wavelength band and that have a quality that is specific for the second tracker (<NUM>, <NUM>), and providing the second-type input image to the second tracker (<NUM>, <NUM>).