Patent Description:
Technical automation of a recognition process may be implemented using, for example, a neural network model implemented by a processor as a special calculation structure, which may provide a computationally intuitive mapping between an input pattern and an output pattern after considerable training. The trained ability to generate such a mapping may be referred to as a learning ability of the neural network. Moreover, such a specialized and specially trained neural network, due to the specialized training, may have a generalization ability that allows the network to generate a relatively accurate output for an input pattern on which the network is not trained.

The present invention includes a processor-implemented method for object tracking as claimed in appended claim <NUM> is provided. The method includes: performing, using a first template, forward object tracking on first image frames in a first sequence group; determining a template candidate of a second template for second image frames in a second sequence group; performing backward object tracking on the first image frames using the template candidate; determining a confidence of the template candidate using a result of comparing a first tracking result determined by the forward object tracking performed on the first image frames and a second tracking result determined by the backward object tracking performed on the first image frames; determining the second template based on the confidence of the template candidate; and performing forward object tracking on the second image frames using the second template. The determining of the second template includes: determining that the template candidate is the second template in response to the confidence of the template candidate exceeding a preset threshold; and determining that the first template is the second template in response to the confidence of the template candidate being less than or equal to the preset threshold.

The forward object tracking may be performed on the first image frames in an order of an initial image frame of the first image frames to a last image frame of the first image frames, and the backward object tracking may be performed on the first image frames in an order of the last image frame of the first image frames to the initial image frame of the first image frames.

The forward object tracking and the backward object tracking may be performed on the first image frames using a same object tracking model.

The first tracking result may include first bounding boxes of the first image frames according to the forward object tracking performed on the first image frames, and the second tracking result may include second bounding boxes of the first image frames according to the backward object tracking performed on the first image frames.

The confidence of the template candidate may be determined based on any one or any combination of any two or more of a first score according to a degree of overlap between corresponding pairs of the first bounding boxes and the second bounding boxes for each corresponding image frame, a second score according to a degree of overlap between a corresponding pair of the first bounding boxes and the second bounding boxes for an initial image frame, and a third score according to a number of corresponding pairs of which a degree of overlap exceeds a preset level among the corresponding pairs of the first bounding boxes and the second bounding boxes for each image frame.

The confidence of the template candidate may be determined based on a degree of similarity between: a portion of the first tracking result corresponding to an image frame of the first image frames; and a portion of the second tracking result corresponding to the image frame.

The object tracking method further may include: determining a second template candidate of the second template; performing backward object tracking on the first image frames using the second template candidate; and determining a confidence of the second template candidate using a result of comparing the first tracking result determined by the forward object tracking performed on the first image frames and a third tracking result determined by the backward object tracking performed on the first image frames using the second template candidate, and the determining of the second template may include determining the second template based on the confidence of the template candidate and the confidence of the second template candidate.

The determining of the second template candidate, the backward object tracking performed on the first image frames using the second template candidate, and the determining of the confidence of the second template candidate may be performed in parallel with the determining of the template candidate, the backward object tracking performed on the first image frames using the template candidate, and the determining of the confidence of the template candidate.

The object tracking method further may include performing forward object tracking on intermediate image frames in an intermediate sequence group between the first sequence group and the second sequence group for a time used to determine the second template.

The determining of the template candidate may include determining the template candidate based on the first tracking result.

The present invention also includes a non-transitory computer-readable storage medium according to claim <NUM>.

The present invention also includes an apparatus for object tracking according to claim <NUM>. The apparatus includes: a processor configured to: perform, using a first template, forward object tracking on first image frames in a first sequence group; determine a template candidate of a second template for second image frames in a second sequence group; perform backward object tracking on the first image frames using the template candidate; determine a confidence of the template candidate using a result of comparing a first tracking result determined by the forward object tracking performed on the first image frames and a second tracking result determined by the backward object tracking performed on the first image frames; determine the second template based on the confidence of the template candidate; and perform forward object tracking on the second image frames using the second template. The determining of the second template includes: determining that the template candidate is the second template in response to the confidence of the template candidate exceeding a preset threshold; and determining that the first template is the second template in response to the confidence of the template candidate being less than or equal to the preset threshold.

The processor may be configured to: determine a second template candidate of the second template; perform backward object tracking on the first image frames using the second template candidate; determine a confidence of the second template candidate using a result of comparing the first tracking result determined by the forward object tracking performed on the first image frames and a third tracking result determined by the backward object tracking performed on the first image frames using the second template candidate; and, for the determining of the second template, determine the second template based on the confidence of the template candidate and the confidence of the second template candidate.

The processor may be configured to perform forward object tracking on intermediate image frames in an intermediate sequence group between the first sequence group and the second sequence group for a time used to determine the second template.

Another aspect of the present invention includes an apparatus according to claim <NUM>.

The following detailed description is provided to assist the reader in gaining a comprehensive understanding of the methods, devices, and/or systems described herein. However, various changes, modifications, and equivalents of the methods, devices, and/or systems described herein will be apparent after an understanding of the disclosure of this application. For example, the sequences of operations described herein are merely examples, and are not limited to those set forth herein, but may be changed with the exception of operations necessarily occurring in a certain order. Also, descriptions of features that are known after understanding of the disclosure of this application may be omitted for increased clarity and conciseness.

Although terms such as "first," "second," and "third" may be used herein to describe various members, components, regions, layers, or sections, these members, components, regions, layers, or sections are not to be limited by these terms (e.g., "first," "second," and "third").

In contrast, when an element is described as being "directly on," "directly connected to," or "directly coupled to" another element, there may be no other elements intervening therebetween.

It will be further understood that the terms "comprises/including" and/or "includes/including" when used herein, specify the presence of stated features, integers, steps, operations, elements, components, and/or combinations thereof, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and/or combinations thereof. The use of the term "may" herein with respect to an example or embodiment (for example, as to what an example or embodiment may include or implement) means that one or more examples or embodiments exists where such a feature is included or implemented, while all examples are not limited thereto.

Unless otherwise defined, all terms, including technical and scientific terms, used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains and based on an understanding of the disclosure of the present application. Terms, such as those defined in commonly used dictionaries, are to be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and the disclosure of the present application, and are not to be interpreted in an idealized or overly formal sense unless expressly so defined herein.

Hereinafter, examples will be described in detail with reference to the accompanying drawings. When describing the example embodiments with reference to the accompanying drawings, like reference numerals refer to like elements and any repeated description related thereto will be omitted.

<FIG> illustrates an example of a configuration and an operation of an apparatus with object tracking (e.g., an object tracking apparatus). Referring to <FIG>, an object tracking apparatus <NUM> may output a tracking result <NUM> based on a template image <NUM> and a search image <NUM>. The template image <NUM> may provide information on a target object to be tracked. The object tracking apparatus <NUM> may track the target object in the search image <NUM> using target object information in the template image <NUM>. The tracking result <NUM> may indicate a position of the target object in the search image <NUM>. For example, the tracking result <NUM> may be used for an automatic tracking operation, a zooming operation, and/or a focusing operation.

According to an example, the template image <NUM> and the search image <NUM> may correspond to a plurality of image frames of an input image. For example, the template image <NUM> may correspond to one of a plurality of image frames included in an input video file, and the search image <NUM> may correspond to one or more of the image frames after (e.g., temporally subsequent to) the image frame corresponding to the template image <NUM>. According to another example, the template image <NUM> and the search image <NUM> may correspond to files independent of each other. For example, the search image <NUM> may correspond to an input video file including a plurality of image frames, and the template image <NUM> may correspond to a still input image file irrelevant to the input video file. In any case, the template image <NUM> may include the target object, and the object tracking apparatus <NUM> may generate the tracking result <NUM> by tracking the target object in the search image <NUM>. The template image <NUM> and the search image <NUM> may correspond to an entire or a partial region of a corresponding image frame. For example, the object tracking apparatus <NUM> may set a search region in the search image <NUM> to track the target object in the search region.

The object tracking apparatus <NUM> may generate the tracking result <NUM> using an object tracking model <NUM>. The object tracking model <NUM> may be or include an artificial intelligence model trained or learned based on machine learning. For example, the object tracking model <NUM> may be or include a deep neural network (DNN) including a plurality of layers. The plurality of layers may include an input layer, one or more hidden layers, and an output layer.

The DNN may include any one or any combination of any two or more of a fully connected network (FCN), a convolutional neural network (CNN), and a recurrent neural network (RNN). For example, at least a portion of the layers included in the neural network may correspond to a CNN, and another portion of the layers may correspond to an FCN. In this case, the CNN may be referred to as convolutional layers, and the FCN may be referred to as fully connected layers.

In the case of the CNN, data input to each layer may be referred to as an input feature map, and data output from each layer may be referred to as an output feature map. The input feature map and the output feature map may also be referred to as activation data. For example, when the convolution layer corresponds to an input layer, an input feature map of the input layer may be an input image. In this example, an output feature map may be generated through a convolution operation between the input feature map and a weight kernel. The input feature map, the output feature map, and the weight kernel may be distinguished by a unit of a tensor.

After the neural network is trained based on deep learning, the neural network may perform an inference that is suitable for a purpose for the training by mapping input data and output data that are in a nonlinear relationship to each other. Deep learning is a machine learning technique for solving a problem such as recognizing image or speech data in a big data set. Deep learning may be construed as an optimization problem solving process of finding a point at which energy is minimized while training a neural network using prepared training data.

Through supervised or unsupervised deep learning, a structure of the neural network or a weight corresponding to a model may be obtained (e.g., generated or determined), and the input data and the output data may be mapped to each other through the weight. When the width and the depth of the neural network are sufficiently great, the neural network may have a capacity sufficient to implement a predetermined function. The neural network may achieve an optimized performance by learning a sufficiently large amount of training data through an appropriate training process.

In the following, the neural network may be represented as being trained "in advance". Here, being trained "in advance" means being trained before the neural network "starts". That the neural network "starts" means that the neural network is ready for inference. For example, that the neural network that "starts" may include that the neural network is loaded into a memory, or that input data for inference is input into the neural network after the neural network is loaded into the memory.

The object tracking apparatus <NUM> may input the template image <NUM> and the search image <NUM> to the object tracking model <NUM> and obtain the tracking result <NUM> from an output of the object tracking model <NUM>. The object tracking model <NUM> may be pre-trained to output the tracking result <NUM> based on the template image <NUM> and the search image <NUM> being input.

<FIG> illustrates an example of an operation of deriving a similarity score. Referring to <FIG>, an object tracking apparatus (the object tracking apparatus <NUM> of <FIG>, as a non-limiting example) may perform object tracking based on feature extracting <NUM>, similarity calculation (e.g., determination) <NUM>, and bounding box regression <NUM>. Any one or any combination of any two or more of the feature extracting <NUM>, the similarity calculation <NUM>, and the bounding box regression <NUM> may be performed through an object tracking model (the object tracking model <NUM> of <FIG>, as a non-limiting example). For example, the object tracking model may include any one or any combination of any two or more of a feature extracting network for the feature extracting <NUM>, a similarity calculation network for the similarity calculation <NUM>, and a bounding box regression network for the bounding box regression <NUM>. The feature extracting network, the similarity calculation network, and the bounding box regression network may each correspond to a neural network. As an example, the object tracking model may include a Siamese network.

The object tracking apparatus may extract a template feature map <NUM> from a template image <NUM> and extract a search feature map <NUM> from a search region <NUM>. The object tracking apparatus may extract the template feature map <NUM> and the search feature map <NUM> using the object tracking model and/or a feature extracting model that share a parameter. As shown by the illustration of the example of <FIG>, the template image <NUM> may correspond to a partial region of an initial image frame of an input image and the search region <NUM> may correspond to a partial region of an n-th image frame of the input image. "n" may be greater than "<NUM>". An operation of preparing the template feature map <NUM> of the template image <NUM> through the object tracking model may be an operation of initializing the object tracking model.

In response to a target object being determined in a first image frame, a target box <NUM> corresponding to the target object may be designated (e.g., generated or determined). For example, the target object may be determined according to a user input of selecting the target object. The target box <NUM> may be a type of a bounding box, and may be specified through box position information (e.g., an x-coordinate and a y-coordinate) and box size information (e.g., a width and a height). The box position information and the box size information may be collectively referred to as box information. The template image <NUM> may be determined based on a position and a size of the target box <NUM>. The search region <NUM> may be determined based on the template image <NUM>. A size of the search region <NUM> may be determined based on a size of the template image <NUM>. For example, the size of the search region <NUM> may be determined to be larger than the size of the template image <NUM>. A position of the search region <NUM> in the n-th image frame may be determined based on the position of a target box of a previous image frame. For example, in response to a target box being detected from an (n-<NUM>)-th image frame, the search region <NUM> of the n-th image frame may be determined based on a position of the target box.

The object tracking apparatus may calculate (e.g., determine) a similarity by comparing the template feature map <NUM> and the search feature map <NUM>. The similarity calculation <NUM> may be performed through the similarity calculation network. The similarity calculation network may derive a cross-correlation between the template feature map <NUM> and the search feature map <NUM> through a cross-correlation layer. A calculation result may indicate a position in the search region <NUM> corresponding to information on the target object and/or the template feature map <NUM>. For example, the calculation result may display a corresponding position <NUM> and/or a score of the corresponding position <NUM> on a search space <NUM> corresponding to the search region <NUM>.

The object tracking apparatus may perform a regression analysis using bounding boxes <NUM> of the corresponding position <NUM> in a search space <NUM> corresponding to the search region <NUM>. In a non-limiting example, the search space <NUM> may be the search space <NUM>. The object tracking apparatus may determine a target box <NUM> corresponding to the target object in the search region <NUM> through the regression analysis and generate a tracking result based on box information of the target box <NUM>.

<FIG> illustrates an example of an operation of updating a template. Referring to <FIG>, a plurality of image frames of a search image <NUM> may be divided into a plurality of sequence groups including a first sequence group <NUM> and a second sequence group <NUM>. Each of the sequence groups may include a plurality of image frames. A change in an appearance of a target object in the search image may cause a difference between an appearance of a target object in a template image and the appearance of the target object in the search image such that performance of object tracking of a typical object tracking apparatus may be reduced. However, when the difference between the appearance of the target object in the template image and the appearance of the target object in the search image occurs, the objection tracking apparatus of one or more embodiments may prevent a decrease in the performance of object tracking by applying the change in the appearance of the target object appearing in the search image to a template through a template update.

For example, a difference between an appearance of a target object appearing in first image frames in the first sequence group <NUM> and an appearance of a target object appearing in second image frames of the second sequence group <NUM> may occur, and a first template <NUM> for object tracking in the first sequence group <NUM> may be updated to a second template <NUM> for object tracking in the second sequence group <NUM>. For example, the first template <NUM> may be determined from an initial image frame of the first sequence group <NUM>, and the second template <NUM> may be determined from a last image frame of the first sequence group <NUM>.

The typical object tracking apparatus may significantly reduce the performance of object tracking by updating a template with an inappropriate image. For example, as a result of a template update, a new template image replacing an existing template image may indicate an incorrect object other than a target object, or the new template image may have an issue such as blurring and/or occlusion. However, the objection tracking apparatus of one or more embodiments may prevent deterioration in the performance of object tracking, due to an inappropriate template image, by verifying a new template through suitability evaluation and performing a template update according to the new template that passes the verification. For example, a template candidate <NUM> for the second template <NUM> may be selected, and the second template may be determined through a verification procedure to verify the template candidate <NUM>.

The performance of object tracking may depend on how suitability of a template is evaluated. The typical object tracking apparatus may not properly reflect a change in an appearance appearing in the new template and/or the existing template, and thus performance may be poor, by simply evaluating suitability of a new template based on a similarity between the new template and an existing template. For example, when a target object in a new template is captured with changes in illuminance, appearance, color, and the like, compared to an existing template, a confidence of the new template may be measured as low due to a difference between the new template and the existing template, even when the new template indicates a correct target object.

However, the object tracking apparatus of one or more embodiments may evaluate suitability of the template candidate <NUM> based on backward object tracking. Backward object tracking may indicate not only an appearance of an object in the template candidate <NUM> but also a history of the object in previous frames. The object tracking apparatus of one or more embodiments may improve tracking performance by basing object tracking on backward object tracking.

<FIG> illustrates an example of tracking results obtained by forward object tracking and backward object tracking. Referring to <FIG>, an object tracking apparatus (the object tracking apparatus <NUM> of <FIG>, as a non-limiting example) may perform forward object tracking on first image frames using a first template <NUM> for the first image frames in a first sequence group <NUM>. A first tracking result Fi may be determined according to the forward object tracking performed on the first image frames. The first tracking result Fi may include first bounding boxes of the first image frames generated according to the forward object tracking performed on the first image frames. "i" denotes a frame number. The object tracking apparatus may track a target object corresponding to a first template <NUM> in the first image frames using an object tracking model.

The object tracking apparatus may determine a template candidate <NUM> of a second template for second image frames in a second sequence group <NUM>. The object tracking apparatus may determine the template candidate <NUM> based on the first tracking result Fi. For example, the object tracking apparatus may determine the template candidate <NUM> from a bounding box of a last image frame among the first bounding boxes of the first tracking result Fi. The object tracking apparatus may perform backward object tracking on the first image frames in the first sequence group <NUM> using the template candidate <NUM>. A second tracking result Bi may be generated according to the backward object tracking performed on the first image frames. The object tracking apparatus may track a target object corresponding to the template candidate <NUM> in the first image frames using the object tracking model.

An order in which backward object tracking may process image frames may be opposite to an order in which forward object tracking may process image frames. For example, the forward object tracking may be performed on the first sequence group <NUM> in an order of an initial image frame of the first image frames to a last image frame of the first image frames, and the backward object tracking may be performed on the first sequence group <NUM> in an order of the last image frame of the first image frames to the initial image frame of the first image frames.

Forward object tracking and backward object tracking may be performed using a same object tracking model. For example, the object tracking apparatus may initialize an object tracking model with the first template <NUM> and generate the first tracking result Fi by inputting the first image frames to the object tracking model in an order of the initial image frame to the last image frame. The initialization of the object tracking model may include extraction of a template feature map from a template image through the object tracking model. The object tracking apparatus may initialize the object tracking model with the template candidate <NUM> and generate the second tracking result Bi by inputting the first image frames to the object tracking model in an order of the last image frame to the initial image frame. As such, while forward object tracking and backward object tracking are being performed, a parameter of the object tracking model may remain the same.

The object tracking apparatus may determine a confidence of the template candidate <NUM> using a result of comparing the first tracking result Fi and the second tracking result Bi, and may determine a second template based on the confidence of the template candidate <NUM>. For example, the object tracking apparatus may determine the confidence of the template candidate <NUM> based on a degree of overlap between at least some of corresponding pairs obtained from the first tracking result Fi and the second tracking result Bi. A corresponding pair may include bounding boxes having a same frame number, including a bounding box from the first tracking result Fi and another a bounding box from the second tracking result Bi. As examples, F<NUM> and B<NUM> may form a corresponding pair and F<NUM> and B<NUM> may form a corresponding pair. In a non-limiting example, a degree of overlap may be respectively determined for each corresponding pair, and the confidence of the template candidate <NUM> may be determined based on the degrees of overlap. The object tracking apparatus may determine that the template candidate <NUM> is the second template in response to the confidence of the template candidate <NUM> exceeding a preset threshold and may determine that the first template <NUM> is the second template in response to the confidence of the template candidate <NUM> being less than the preset threshold. When the first template <NUM> is determined as the second template, the first template <NUM> is continuously used to track an object in the second sequence group <NUM>.

<FIG> illustrates an example of a verification capability of backward object tracking. Referring to <FIG>, a first sequence group <NUM> may include image frames <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, and <NUM>. A first template <NUM> may be set from an initial image frame <NUM>, and an object tracking apparatus (the object tracking apparatus <NUM> of <FIG>, as a non-limiting example) may track an object corresponding to the first template <NUM> in the image frames <NUM> to <NUM> by performing forward object tracking. A template candidate <NUM> may be determined from a last image frame <NUM>. For example, an image of a bounding box corresponding to a tracking result of the last image frame <NUM> using the first template <NUM> may be determined as the template candidate <NUM>.

As illustrated in <FIG>, "A" indicates a target object, and "B" and "C" indicate a distractor. The distractor may be an object that is not the target object but may be mistaken for the target object. The template candidate <NUM> of the last image frame <NUM> may correspond to a distractor B not a target object A. The target object A may be obscured by a distractor C such that a bounding box may be formed to surround the distractor B, the distractor B being similar to the target object A. A typical object tracking apparatus may degrade object tracking performance by performing a template update with the template candidate <NUM>.

The object tracking apparatus of one or more embodiments may track an object corresponding to the template candidate <NUM> in the image frames <NUM> to <NUM> by performing backward object tracking. The backward object tracking may consider not only an appearance of an object (the distractor B) in the template candidate <NUM> but also a history of the object in the previous image frames <NUM> to <NUM> such that accuracy of suitability evaluation of the template candidate <NUM> may be improved over the typical object tracking apparatus. Referring to <FIG>, a tracking result obtained by the backward object tracking may indicate the distractor B not the target object A. For example, based on the tracking result obtained by the backward object tracking, the object tracking apparatus may determine that the template candidate <NUM> includes the distractor B and not the target object A. Accordingly, as a result of the backward object tracking, the template candidate <NUM> may be discarded, and the first template <NUM> may be continuously used to track an object in a next sequence group.

<FIG> illustrates an example of progress of forward object tracking and backward object tracking. Referring to <FIG>, an object tracking apparatus (the object tracking apparatus <NUM> of <FIG>, as a non-limiting example) may determine a current template in operation <NUM> and initialize an object tracking model with the current template in operation <NUM>. For example, it may be determined that the current template corresponds to a bounding box of a target object of an initial image template.

The object tracking apparatus may receive a t-th search frame in operation <NUM> and perform forward object tracking in operation <NUM>. A tracking result of the t-th search frame may be determined according to the forward object tracking. The tracking result may indicate a bounding box corresponding to the target object. In operation <NUM>, the object tracking apparatus may check whether a template update cycle has arrived. For example, the template update cycle may be set in advance to update a template for every predetermined number of frames. When the update cycle has not arrived (e.g., when "t" is less than the predetermined number of frames), the object tracking apparatus may increase "t" and then perform operations <NUM> and <NUM> again.

When the template update cycle has arrived (e.g., when "t" is greater than or equal to the predetermined number of frames), the object tracking apparatus may determine a template candidate of a next template in operation <NUM> and initialize an object tracking model with the template candidate in operation <NUM>. The object tracking model in operation <NUM> and the object tracking model in operation <NUM> may be initialized with different templates and use different template feature maps, but the models may share a network parameter. In other words, the object tracking model in operation <NUM> and the object tracking model in operation <NUM> may be a same model using different template feature maps. The object tracking apparatus may perform backward object tracking in operation <NUM>. The backward object tracking may be performed in a backward direction on the image frames on which the forward object tracking is performed.

The object tracking apparatus may determine a confidence of a template candidate (e.g., the template candidate determined in operation <NUM>) in operation <NUM>. The object tracking apparatus may determine a first tracking result obtained by the forward object tracking (e.g., of operation <NUM>) and a second tracking result obtained by the backward object tracking (e.g., of operation <NUM>). The object tracking apparatus may determine the confidence of the template candidate based on a degree of overlap between at least some of corresponding pairs obtained from the first tracking result and the second tracking result. A corresponding pair may include bounding boxes having a same frame number, one from the first tracking result and the other from the second tracking result.

A bounding box from the first tracking result may be a first bounding box, and a bounding box from the second tracking result may be a second bounding box. The object tracking apparatus may determine the confidence of the template candidate based on any one or any combination of any two or more of a first score according to a degree of overlap between corresponding pairs of first bounding boxes and second bounding boxes for each corresponding image frame, a second score according to a degree of overlap between a corresponding pair of the first bounding boxes and the second bounding boxes in the initial image frame, and a third score according to a number of corresponding pairs of which a degree of overlap exceeds a preset level among the corresponding pairs of the first bounding boxes and the second bounding boxes for each corresponding image frame. For example, the object tracking apparatus may determine confidence based on an average or a weighted average of at least some of the scores.

The first score may be expressed by Equation <NUM> below, for example.

In Equation <NUM>, score1 denotes the first score, N denotes a number of image frames (e.g., a number of image frames included in a sequence group), Bi denotes a tracking result obtained by backward object tracking, Fi denotes a tracking result obtained by forward object tracking, and IoU(Bi,Fi) denotes a degree of overlap between Bi and Fi.

The second score may be expressed by Equation <NUM> below, for example.

In Equation <NUM>, score2 denotes the second score, B<NUM> denotes a tracking result obtained by backward object tracking performed on an initial image frame, and F<NUM> denotes a tracking result obtained by forward object tracking performed on the initial image frame.

The third score may be expressed by Equation <NUM> below, for example.

In Equation <NUM>, score3 denotes the third score, and τ denotes a threshold for a degree of overlap.

The object detection apparatus may compare the confidence and a threshold k in operation <NUM>. In response to the confidence being greater than the threshold k, the object detection apparatus may determine that a template candidate (e.g., the template candidate determined in operation <NUM>) is reliable and may perform a template update with the template candidate. In response to the confidence being less than or equal to the threshold k, the object detection apparatus may determine that the template candidate is not reliable, may discard the template candidate, and may continuously use an existing template (e.g., the template determined in operation <NUM>) in a next sequence group.

<FIG> illustrates an example of an operation of updating a template based on multiple template candidates. Referring to <FIG>, an object tracking apparatus (the object tracking apparatus <NUM> of <FIG>, as a non-limiting example) may determine multiple template candidates in operation <NUM>. A plurality of objects having a high degree of similarity to a current template may be determined as multiple template candidates. For example, a predetermined number of objects that has a high degree of similarity to the current template in an image frame may be selected as the multiple template candidates in sequential order. <FIG> illustrates an example in which an object A, an object B, and an object C are multiple template candidates. A bounding box surrounding each of the objects selected as multiple template candidates may correspond to a template candidate.

The object tracking apparatus may perform backward object tracking in operation <NUM>. The object tracking apparatus may initialize an object tracking model with a template candidate of each of the multiple template candidates and perform backward object tracking on image frames using the object tracking model. As a result, a tracking result of the backward object tracking corresponding to each template candidate may be derived. The backward object tracking for each template candidate may be performed in parallel. In this case, a plurality of object tracking models sharing a parameter may be initialized with each template candidate.

The object tracking apparatus may select an optimal template candidate from among the multiple template candidates in operation <NUM>. The object tracking apparatus may determine a confidence corresponding to a tracking result of each template candidate of the multiple template candidates and select a template candidate having a highest confidence from among template candidates of the multiple template candidates. The object tracking apparatus may perform a template update with the selected optimal template candidate. Since histories of objects similar to a target object may be comprehensively considered through multiple template candidates, possibility of a template update being appropriate may be greater.

<FIG> and <FIG> illustrate implementation examples of an operation of updating a template. Referring to <FIG>, an object tracking apparatus (the object tracking apparatus <NUM> of <FIG>, as a non-limiting example) may perform forward object tracking on first image frames in a first sequence group <NUM> using a first template <NUM>. The object tracking apparatus may determine a template candidate <NUM> based on a tracking result of the forward object tracking. The object tracking apparatus may perform backward object tracking on the first image frames in the first sequence group <NUM> using the template candidate <NUM>.

The object tracking apparatus may evaluate a confidence of the template candidate <NUM> based on a tracking result of the forward object tracking and a tracking result of the backward object tracking and update the first template <NUM> to a second template <NUM> based on the confidence of the template candidate <NUM>. For example, in response to the confidence of the template candidate <NUM> being greater than a threshold, the object tracking apparatus determines that the template candidate <NUM> is the second template <NUM>, and in response to the confidence of the template candidate <NUM> being less than the threshold, the object tracking apparatus determines that the first template <NUM> is the second template <NUM>. The object tracking apparatus may perform forward object tracking on second image frames in a second sequence group <NUM> using the second template <NUM>.

In <FIG>, t1 represents a time at which backward object tracking starts, and t2 represents a time at which a template update is completed. Between t1 and t2, backward object tracking, confidence evaluation of the template candidate <NUM>, and a template update may be performed. The object tracking apparatus may wait for a new template (the second template <NUM>) without performing forward object tracking between t1 and t2 and perform forward object tracking on the second image frames after t2. For example, the object tracking apparatus may perform the forward object tracking on the second image frames in response to determining the second template <NUM>. A time delay corresponding to a time period between t1 and t2 may occur between when the forward object tracking is performed on the first image frames and when the forward object tracking is performed on the second image frames.

Referring to <FIG>, the object tracking apparatus may perform forward object tracking on first image frames in a first sequence group <NUM> using a first template <NUM>. The object tracking apparatus may determine a template candidate <NUM> based on a tracking result of the forward object tracking. The object tracking apparatus may perform backward object tracking on the first image frames in the first sequence group <NUM> using the template candidate <NUM>. The object tracking apparatus may evaluate a confidence of the template candidate <NUM> based on a tracking result of the forward object tracking and a tracking result of the backward object tracking and update the first template <NUM> to a second template <NUM> based on the confidence of the template candidate <NUM>.

In <FIG>, t1 represents a time at which backward object tracking starts, and t2 represents a time at which a template update is completed. Between t1 and t2, backward object tracking, confidence evaluation of the template candidate <NUM>, and a template update may be performed. The object tracking apparatus may perform forward object tracking on second image frames in a second sequence group <NUM> using an existing template (the first template <NUM>) instead of waiting for a new template (the second template <NUM>). In response to completion of a template update at t2, the object tracking apparatus may perform forward object tracking on third image frames in a third sequence group <NUM> after t2. Continuing to perform forward object tracking temporarily using the existing template (the first template <NUM>) while a template update is being performed may prevent a time delay corresponding to a time period between t1 and t2. A sequence group, such as the second sequence group <NUM>, to which the existing template (the first template <NUM>) is temporarily applied may be an intermediate sequence group, and an image frame in an intermediate sequence group may be an intermediate image frame.

<FIG> illustrates an example of a method for object tracking (e.g., an object tracking method). Operations <NUM> to <NUM> to be described hereinafter with reference to <FIG> may be performed sequentially or non-sequentially. For example, an order of operations <NUM> to <NUM> may be changed, and/or at least two operations <NUM> to <NUM> may be performed in parallel or simultaneously. Further, one or more of operations <NUM> to <NUM> may be omitted, without departing from the scope of the shown example. The operations <NUM> to <NUM> may be performed by one or more components of an object tracking apparatus (the object tracking apparatus <NUM> of <FIG>, as a non-limiting example).

Referring to <FIG>, an object tracking method <NUM> includes operation <NUM> of performing forward object tracking on first image frames using a first template for the first image frames in a first sequence group, operation <NUM> of determining a template candidate of a second template for second image frames in a second sequence group, operation <NUM> of performing backward object tracking on the first image frames using the template candidate, operation <NUM> of determining a confidence of the template candidate using a result of comparing a first tracking result obtained by the forward object tracking performed on the first image frames and a second tracking result obtained by the backward object tracking performed on the first image frames, operation <NUM> of determining the second template based on the confidence of the template candidate, and operation <NUM> of performing forward object tracking on second image frames using the second template.

The forward object tracking may be performed on the first image frames in order of an initial image frame of the first image frames to a last image frame of the first image frames. The backward object tracking may be performed on the first image frames in order of the last image frame of the first image frames to the initial image frame of the first image frames.

The confidence of the template candidate may be determined based on any one or any combination of any two or more of a first score according to a degree of overlap between corresponding pairs of the first bounding boxes and the second bounding boxes for each corresponding image frame, a second score according to a degree of overlap between a corresponding pair of the first bounding boxes and the second bounding boxes in the initial image frame, and a third score according to a number of corresponding pairs of which a degree of overlap exceeds a preset level among the corresponding pairs of the first bounding boxes and the second bounding boxes for each image frame.

Operation <NUM> includes an operation of determining that the template candidate is the second template in response to the confidence of the template candidate exceeding a preset threshold and an operation of determining that the first template is the second template in response to the confidence of the template candidate being less than the preset threshold.

The object tracking method <NUM> may further include an operation of determining the second template candidate of the second template, an operation of performing backward object tracking on the first image frames using the second template candidate, and an operation of determining a confidence of the second template candidate using a result of comparing the first tracking result obtained by the forward object tracking performed on the first image frames and a third tracking result obtained by the backward object tracking performed on the first image frames using the second template candidate. Operation <NUM> may include an operation of determining the second template based on the confidence of the template candidate and the confidence of the second template candidate. The determining of the second template candidate, the backward object tracking performed on the first image frames using the second template candidate, and the determining of the confidence of the second template candidate may be performed in parallel with the determining of the template candidate, the backward object tracking performed on the first image frames using the template candidate, and the determining of the confidence of the template candidate.

The object tracking method <NUM> may further include an operation of performing forward object tracking on intermediate image frames in an intermediate sequence group between the first sequence group and the second sequence group for a time required to determine the second template.

The provided description referring to <FIG>, <FIG> may apply to the object tracking method of <FIG>.

<FIG> illustrates an example of an apparatus with object tracking (e.g., an object tracking apparatus). Referring to <FIG>, an object tracking apparatus <NUM> includes a processor <NUM> (e.g., one or more processors) and a memory <NUM> (e.g., one or more memories). The object tracking apparatus <NUM> may be or include the object tracking apparatus <NUM> of <FIG>, as a non-limiting example, The memory <NUM> may be connected to the processor <NUM>, and store instructions executable by the processor <NUM>, data to be calculated by the processor <NUM>, or data processed by the processor <NUM>. The memory <NUM> includes a non-transitory computer readable medium, for example, a high-speed random access memory, and/or a non-volatile computer readable storage medium, for example, one or more disk storage devices, flash memory device, or other non-volatile solid state memory devices. The memory <NUM> may be or include a non-transitory computer-readable storage medium storing instructions that, when executed by the processor <NUM>, configure the processor <NUM> to perform any one, any combination, or all of the operations and methods described herein with references to <FIG> and <FIG>.

The processor <NUM> may execute the instructions to perform any one, any combination, or all of the operations and methods described herein with references to <FIG> and <FIG>. The processor <NUM> performs forward object tracking on first image frames using a first template for the first image frames in a first sequence group, determine a template candidate of a second template for second image frames in a second sequence group, perform backward object tracking on the first image frames using the template candidate, determine a confidence of the template candidate using a result of comparing a first tracking result obtained by the forward object tracking performed on the first image frames and a second tracking result obtained by the backward object tracking performed on the first image frames, determine the second template based on the confidence of the template candidate, and perform forward object tracking on the second image frames using the second template. The determining of the second template includes: determining that the template candidate is the second template in response to the confidence of the template candidate exceeding a preset threshold; and determining that the first template is the second template in response to the confidence of the template candidate being less than or equal to the preset threshold.

The provided description referring to <FIG>, <FIG> may apply to the object tracking apparatus <NUM>.

<FIG> illustrates an example of an electronic device. Referring to <FIG>, an electronic device <NUM> may include a processor <NUM> (e.g., one or more processor), a memory <NUM> (e.g., one or more memories), a camera <NUM> (e.g., one or more cameras), a storage device <NUM>, an input device <NUM>, an output device <NUM>, and a network interface <NUM>, and these components may communicate with one another through a communication bus <NUM>. For example, the electronic device <NUM> may be implemented as at least a part of a mobile device such as a mobile phone, a smart phone, a PDA, a netbook, a tablet computer or a laptop computer, a wearable device such as a smart watch, a smart band or smart glasses, a computing device such as a desktop or a server, a home appliance such as a television, a smart television or a refrigerator, a security device such as a door lock, or a vehicle such as an autonomous vehicle or a smart vehicle. The electronic device <NUM> may be or include either one or both of the object tracking apparatus <NUM> of <FIG> and the object tracking apparatus <NUM> of <FIG>.

The processor <NUM> executes instructions or functions to be executed in the electronic device <NUM>. For example, the processor <NUM> may process the instructions stored in the memory <NUM> or the storage device <NUM>. The processor <NUM> may perform any one, any combination, or all of the operations and methods described herein with references to <FIG>. The memory <NUM> may include a computer-readable storage medium or a computer-readable storage device. The memory <NUM> may store instructions to be executed by the processor <NUM> and store related information while software and/or an application is executed by the electronic device <NUM>. The memory <NUM> may be or include a non-transitory computer-readable storage medium storing instructions that, when executed by the processor <NUM>, configure the processor <NUM> to perform any one, any combination, or all of the operations and methods described herein with references to <FIG>.

The camera <NUM> may capture a photo and/or a video. For example, the camera <NUM> may generate an input image including a plurality of image frames. The plurality of image frames may include either one or both of a template image and a search image. The storage device <NUM> includes a computer-readable storage medium or computer-readable storage device. The storage device <NUM> may store a larger quantity of information than the memory <NUM> for a long time. For example, the storage device <NUM> may include a magnetic hard disk, an optical disc, a flash memory, a floppy disk, or other non-volatile memories known in the art.

The input device <NUM> may receive an input from the user in traditional ways such as through a keyboard and a mouse, and in new ways such as through touch, voice and an image. For example, the input device <NUM> may include a keyboard, a mouse, a touch screen, a microphone, or any other device that detects the input from the user and transmits the detected input to the electronic device <NUM>. The output device <NUM> may provide an output of the electronic device <NUM> to the user through a visual, auditory, or haptic channel. The output device <NUM> may include, for example, a display, a touch screen, a speaker, a vibration generator, or any other device that provides the output to the user. The network interface <NUM> may communicate with an external device through a wired or wireless network.

The provided description referring to <FIG> may apply to the electronic device <NUM>.

The object tracking apparatuses, processors, memories, electronic devices, cameras, storage devices, input devices, output devices, network interfaces, communication buses, object tracking apparatus <NUM>, object tracking apparatus <NUM>, processor <NUM>, memory <NUM>, electronic device <NUM>, processor <NUM>, memory <NUM>, camera <NUM>, storage device <NUM>, input device <NUM>, output device <NUM>, network interface <NUM>, communication bus <NUM>, and other apparatuses, units, modules, devices, and components described herein with respect to <FIG> are implemented by or representative of hardware components. Examples of hardware components that may be used to perform the operations described in this application where appropriate include controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more of the hardware components that perform the operations described in this application are implemented by computing hardware, for example, by one or more processors or computers. A processor or computer may be implemented by one or more processing elements, such as an array of logic gates, a controller and an arithmetic logic unit, a digital signal processor, a microcomputer, a programmable logic controller, a field-programmable gate array, a programmable logic array, a microprocessor, or any other device or combination of devices that is configured to respond to and execute instructions in a defined manner to achieve a desired result. In one example, a processor or computer includes, or is connected to, one or more memories storing instructions or software that are executed by the processor or computer. Hardware components implemented by a processor or computer may execute instructions or software, such as an operating system (OS) and one or more software applications that run on the OS, to perform the operations described in this application. The hardware components may also access, manipulate, process, create, and store data in response to execution of the instructions or software. For simplicity, the singular term "processor" or "computer" may be used in the description of the examples described in this application, but in other examples multiple processors or computers may be used, or a processor or computer may include multiple processing elements, or multiple types of processing elements, or both. For example, a single hardware component or two or more hardware components may be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components may be implemented by one or more processors, or a processor and a controller, and one or more other hardware components may be implemented by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller, may implement a single hardware component, or two or more hardware components. A hardware component may have any one or more of different processing configurations, examples of which include a single processor, independent processors, parallel processors, single-instruction single-data (SISD) multiprocessing, single-instruction multiple-data (SIMD) multiprocessing, multiple-instruction single-data (MISD) multiprocessing, and multiple-instruction multiple-data (MIMD) multiprocessing.

Claim 1:
A processor-implemented method for object tracking, the method comprising:
performing (<NUM>, <NUM>), using a first template of a target object to be tracked, forward object tracking on first image frames in a first sequence group;
determining (<NUM>, <NUM>) a template candidate of a second template of the target object to be tracked for second image frames in a second sequence group following the first sequence group;
performing (<NUM>, <NUM>) backward object tracking on the first image frames using the template candidate;
determining (<NUM>) a confidence of the template candidate using a result of comparing a first tracking result determined by the forward object tracking performed on the first image frames and a second tracking result determined by the backward object tracking performed on the first image frames;
determining (<NUM>, <NUM>) the second template based on the confidence of the template candidate;
characterized in that the method further comprises:
determining that the template candidate is the second template in response to the confidence of the template candidate exceeding a preset threshold; and
determining that the first template is the second template in response to the confidence of the template candidate being less than or equal to the preset threshold; and
performing forward object tracking on the second image frames using the second template.