Patent Description:
Visual object tracking technology is a technology being studied in the computer vision field. To perform visual object tracking, a system may collect image sequences through image collection equipment such as cameras. A user may mark a target object in a first frame (or an initial frame) of an image sequence, and an object tracking algorithm may continue to track an object in a subsequent frame of the image sequence and may provide position information of the object. Object tracking technology may be used to extract template information corresponding to a target of a first frame, calculate a matching degree between the template information and different candidate positions within a search region of a subsequent video frame, and select a position with the highest matching degree to determine a position of the target. Examples of tracking methods are e.g. known from IEEE article "<NPL> and from article "<NPL>.

In one general aspect, a method as claimed in annexed claim <NUM> is provided.

The first image frame and the second image frame may be collected by different image collectors of a same electronic device.

The first FoV and the second FoV may be selected from predetermined reference FoVs.

The determining of the first target tracking state may include: generating a first target tracking result by tracking the target from the first image frame; and determining the first target tracking state based on the first target tracking result.

The first target tracking result may include a prediction confidence. The determining of the first target tracking state based on the first target tracking result may include determining the first target tracking state according to a result of a comparison of the prediction confidence and a preset threshold.

The preset threshold may include a first threshold and a second threshold. The determining of the first target tracking state according to the result of the comparison may include: in response to the prediction confidence being greater than the first threshold, determining the first target tracking state as a first state; in response to the prediction confidence being less than the second threshold, determining the first target tracking state as a second state; and, in response to the prediction confidence being greater than the second threshold and less than the first threshold, determining the first target tracking state as a third state.

The first FoV and the second FoV may be selected from a first reference FoV, a second reference FoV, and a third reference FoV, among the predetermined reference FOVs. The second reference FoV may be greater than the first reference FoV. The third reference FoV may be less than the first reference FoV. The determining of the second FoV may include: when the first FoV is the first reference FoV, determining the second FoV as the second reference FoV in response to the first target tracking state being the second state, determining the second FoV as the third reference FoV in response to the first target tracking state being the first state, and determining the second FoV as the first reference FoV in response to the first target tracking state being the third state; when the first FoV is the second reference FoV, determining the second FoV as the second reference FoV in response to the first target tracking state being the second state, determining the second FoV as the first reference FoV in response to the first target tracking state being the first state, and determining the second FoV as the second reference FoV in response to the first target tracking state being the third state; and when the first FoV is the third reference FoV, determining the second FoV as the first reference FoV in response to the first target tracking state being the second state, determining the second FoV as the third reference FoV in response to the first target tracking state being the first state, and determining the second FoV as the third reference FoV in response to the first target tracking state being the third state.

The generating of the second target tracking result may include: setting a reference template feature based on an initial image frame of an image sequence to which the second image frame belongs; determining a search region of the second image frame based on the second FoV and a position of the target of the first image frame, and obtaining a search feature from the search region; and generating the second target tracking result based on the reference template feature and the search feature.

The generating of the second target tracking result based on the reference template feature and the search feature may include: in response to the second FoV being the second reference FoV or the third reference FoV, generating a scaled template feature by scaling the reference template feature; and generating the second target tracking result based on the scaled template feature and the search feature.

The generating of the scaled template feature may include: in response to the second FoV being the second reference FoV, generating the scaled template feature by scaling down the reference template feature; and, in response to the second FoV being the third reference FoV, generating the scaled template feature by scaling up the reference template feature.

The generating of the second target tracking result based on the scaled template feature and the search feature may include: generating a feature map by performing a convolution operation on the scaled template feature and the search feature using the scaled template feature as a convolution kernel; and obtaining the second target tracking result based on the feature map.

The first target tracking result may include a prediction confidence, and either one or both of a target position and a target size. The determining of the first target tracking state based on the first target tracking result may include: obtaining at least one of a target relative displacement corresponding to the first image frame and a ratio between a size of the target of the first image frame and a size of the first image frame, based on either one or both of the target position and the target size; and determining the first target tracking state based on a result of a comparison of the prediction confidence and a preset threshold, and either one or both of a result of a comparison of the target relative displacement and a reference displacement and a result of a comparison of the ratio and a reference ratio.

The determining of the first target tracking state based on the result of the comparison of the prediction confidence and the preset threshold, and either one or both of the result of the comparison of the target relative displacement and the reference displacement and the result of the comparison of the ratio and the reference ratio include: in response to the prediction confidence being greater than a first threshold and the target relative displacement being less than the reference displacement, determining the first target tracking state as a first state; in response to the prediction confidence being greater than a second threshold and the ratio being less than the reference ratio, determining the first target tracking state as a second state; and in response to the prediction confidence being greater than the second threshold and less than the first threshold, determining the first target tracking state as a third state.

In another general aspect, a non-transitory computer-readable storage medium stores instructions that, when executed by a processor, cause the processor to perform the method described above.

In another general aspect, an apparatus with object tracking includes: a processor; and a memory including instructions executable on the processor. The processor is configured to, in response to the instructions being executed: determine a first target tracking state by tracking a target from a first image frame with a first field of view (FoV); determine a second FoV based on the first FoV and the first target tracking state; and generate a second target tracking result by tracking a target from a second image frame with the second FoV.

The processor may be further configured to: generate a first target tracking result by tracking the target from the first image frame; and determine the first target tracking state based on the first target tracking result.

The first target tracking result may include a prediction confidence. The processor may be further configured to determine the first target tracking state according to a result of a comparison of the prediction confidence and a preset threshold.

The first target tracking result may include a prediction confidence, and at least one of a target position and a target size. The processor may be further configured to: obtain at least one of a target relative displacement corresponding to the first image frame and a ratio between a size of the target of the first image frame and a size of the first image frame, based on at least one of the target position and the target size; an determine the first target tracking state based on a result of a comparison of the prediction confidence and a preset threshold, and either one or both of a result of a comparison of the target relative displacement and a reference displacement and a result of a comparison of the ratio and a reference ratio.

In another general aspect, and electronic device includes: a camera configured to operate with a plurality of reference fields of view (FoVs); and a processor. The processor is configured to: generate a first target tracking state by tracking a target from a first image frame with a first FoV; determine a second FoV based on the first FoV and the first target tracking state; an generate a second target tracking result by tracking a target from a second image frame with the second FoV. The first FoV corresponds to a first reference FoV among the plurality of reference FoVs, and the second FoV corresponds to a second reference FoV among the plurality of reference FoVs.

Throughout the drawings and the detailed description, the same drawing reference numerals refer to the same elements, features, and structures.

However, various changes and modifications of the methods, apparatuses, and/or systems described herein will be apparent after an understanding of the disclosure, as long as those changes and modifications are convered by the appended claims. For example, the sequences of operations described herein are merely examples, and are not limited to those set forth herein, but may be changed as will be apparent after an understanding of the disclosure, if falling within the scope of the appended claims, with the exception of operations necessarily occurring in a certain order.

Rather, the examples described herein have been provided merely to illustrate some of the many possible ways of implementing the methods, apparatuses, and/or systems described herein that will be apparent after an understanding of the disclosure, if falling within the scope of the appended claims.

Herein, it is to be noted that use of the term "may" with respect to an embodiment or example, e.g., as to what an embodiment or example may include or implement, means that at least one embodiment or example exists in which such a feature is included or implemented while all examples and examples are not limited thereto.

Unless otherwise defined, all terms used herein including technical or scientific terms have the same meaning as commonly understood by in the art in view of the context of this disclosure. It will be further understood that terms, such as those defined in commonly-used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and the context of this disclosure, and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

In target tracking technology, a user may mark a target included in an initial video frame in a form of a bounding box. For example, the bounding box may be set with a high weight of an object and a low weight of a background. Subsequently, an algorithm may provide the bounding box and position information of a corresponding object in a subsequent video frame. Target tracking and marking may be provided in a form of a segmentation mask. The segmentation mask may finely mark the target in units of pixels by separating the target from a background. Thus, the algorithm may provide the segmentation mask and position information of a target object in a subsequent video frame. Mask tracking may have a large number of similarities to video object segmentation in comparison to bounding box tracking. In addition to bounding box tracking technology, the mask tracking may provide an additional segmentation branch, may output a central position of an object by integrating object segmentation technologies, and may output a segmentation mask map of the object. Although examples will be described below with reference to bounding box tracking technology, the examples may also apply to mask tracking technology.

In addition, target modeling may be converted by similarity training through a twin network-based target tracking algorithm. For example, when a similarity is calculated by comparing a template of an initial frame with a candidate region of a current frame, a target tracking result may be obtained based on a matching value. <FIG> illustrates an example of a process of deriving a similarity based on a twin network. Referring to <FIG>, a first feature vector <NUM> of a first image block <NUM> and a second feature vector <NUM> of a second image block <NUM> may be extracted through network operations (e.g., convolution, activation, normalization, and the like) through two networks <NUM> and <NUM>. A similarity score <NUM> of the first image block <NUM> and the second image block <NUM> may be derived by computing a dot product <NUM> of the first and second feature vectors <NUM> and <NUM>.

To obtain a similarity by calculating a convolution operation of a template feature kernel zf and a search region feature xf (hereinafter, referred to as a "search feature") using a twin network-based target tracking algorithm, a scale of a target of a template and a scale of a target of a search region may need to match each other. Referring to <FIG>, a target tracking algorithm may obtain a search region <NUM> by cropping a region corresponding to a fixed multiple (e.g., × <NUM>) of a size (e.g., a square root of a product of a width and a height corresponding to a target) of a target box <NUM> of a previous image frame based on a target position <NUM> predicted for the previous image frame from a current image frame <NUM>. In the following description, an image frame may simply be referred to as a frame. If a change in a target between two frames is within a normal physical movement range, a scale of a target of the search region obtained by the above scheme may match the scale of the target of the template.

A fixed size of the search region may be regarded as a fixed field of view (FoV). Here, due to the fixed size of the search region, the following problems may occur:.

In examples described herein, the above problems may be solved. However, the above problems merely correspond to an example, and aspects or effects of the examples are not limited by the problems.

<FIG> is a flowchart illustrating an example of an object tracking method. As shown in <FIG>, the object tracking method may include the following operations.

In operation <NUM>, a first target tracking state is determined by tracking a target from a first image frame with a first FoV. An n-th FoV may be an FoV used to track a target from an n-th image frame. An n-th target tracking state may be a target tracking state according to a target tracking result for the n-th image frame. The target tracking result for the n-th image frame may be referred to as an "n-th target tracking result.

In operation <NUM>, a second FoV is determined based on the first FoV and the first target tracking state. A target is tracked from a second image frame with the second FoV. The first image frame and the second image frame may belong to the same image sequence. Image frames of the same sequence may be sequentially arranged, and the first image frame may be a previous image frame of the second image frame. An example in which the second image frame corresponds to a current image frame and the first image frame corresponds to a previous image frame will be described below.

In a target tracking process, tracking of a target from a previous image frame may be performed first, and accordingly a target tracking result may be obtained. For example, when an FoV used to track a target from the previous image frame is obtained, the target may be tracked from the previous image frame using the FoV, and accordingly a target tracking state may be determined according to a result of tracking the target. Based on the FoV used to track the target from the previous image frame and the target tracking state, an FoV used to track a target from the current image frame may be determined.

In operation <NUM>, a second target tracking result is generated by tracking a target from the second image frame with the second FoV.

For example, the second target tracking result is obtained by tracking the target from the current image frame using the second FoV determined in operation <NUM>. Before the target is tracked from the current image frame, an FoV used for target tracking of the current image frame may be determined based on an FoV corresponding to the previous image frame and the target tracking state.

The FoV for the current image frame may be dynamically adjusted. A size of a search region of the current image frame may be determined based on a size of a search region of the previous image frame and the target tracking state. For example, when the target tracking state of the previous image frame is relatively stable, an amount of computation may be reduced by properly narrowing the FoV corresponding to the current image frame. When the target tracking state of the previous image frame corresponds to a target loss, a possibility of finding a target again and robustness of target tracking may be enhanced by widening the FoV corresponding to the current image frame. In addition, since the FoV for the current image frame is dynamically adjusted, the number of cumulative errors may be reduced and an accuracy of target tracking may be enhanced.

According to examples, in a target tracking process for the current image frame, the FoV used to track a target from the current image frame may be dynamically adjusted based on the target tracking state and the FoV for target tracking of the previous image frame, and thus it may be possible to prevent an accumulation of errors, a reduction in robustness and waste of computing power due to use of a fixed FoV.

According to the invention, a FoV for each image frame is selected from predetermined reference FoVs. For example, the reference FoVs may include at least two of a first reference FoV, a second reference FoV, and a third reference FoV. The first FoV and/or the second FoV may be one of at least two of the first reference FoV, the second reference FoV, and the third reference FoV. The second reference FoV may be obtained by magnifying the first reference FoV by a first factor, and the third reference FoV may be obtained by reducing the first reference FoV by a second factor.

In an example, the first reference FoV may correspond to a normal FoV, the second reference FoV may correspond to a wide FoV obtained by magnifying the first reference FoV, and the third reference FoV may correspond to a narrow FoV obtained by reducing the first reference FoV. In this example, the first factor used for magnification and the second factor used for reduction may be set according to actual needs. A target tracking state for each image frame may be divided into the same number of states as a number of reference FoVs. For example, when three reference FoVs are provided, the target tracking state may also be divided into three states.

As shown in <FIG>, a process of tracking a target from a video sequence using the object tracking method may include the following operations:.

Subsequently, the above operations may be repeatedly performed on subsequent frames.

For example, in the object tracking method, an FoV corresponding to a current image frame may be determined based on a target tracking state and an FoV corresponding to a previous image frame. In this example, the FoV corresponding to the previous image frame may be determined in advance during a target tracking process for the previous image frame. In determining of the FoV corresponding to the current image frame, it may be important to obtain the target tracking state corresponding to the previous image frame. A process of obtaining the target tracking state corresponding to the previous image frame will be described in detail below.

According to an example, determining of a target tracking state corresponding to a previous image frame of a current image frame may include obtaining a target tracking result corresponding to the previous image frame, determining the target tracking state corresponding to the previous image frame based on the target tracking result corresponding to the previous image frame.

The target tracking state corresponding to the previous image frame may be determined based on the target tracking result obtained in response to completion of tracking of a target of the previous image frame. For example, the target tracking result may include at least one of a target position (e.g., central coordinates of a target), a target size (e.g., a width and a height of a target), and a prediction confidence. A bounding box corresponding to a target may be determined based on the target position and the target size. The prediction confidence may indicate a confidence of the target position and the target size.

According to the invention, the target tracking result includes a corresponding prediction confidence. The determining of the target tracking state corresponding to the previous image frame based on the target tracking result corresponding to the previous image frame includes determining the target tracking state according to a result of a comparison of the prediction confidence and a preset threshold. The preset threshold value may include a first threshold and a second threshold. If the prediction confidence is greater than the first threshold, the target tracking state may be determined as a predicted steady state. If the prediction confidence is less than the second threshold, the target tracking state may be determined as a target loss state. If the prediction confidence is greater than the second threshold and less than the first threshold, the target tracking state may be determined as a state between the predicted steady state and the target loss state. The predicted steady state may be represented as a first state, the target loss state may be represented as a second state, and the state between the predicted steady state and the target loss may be represented as an intermediate state or a third state.

According to an example, the target tracking state corresponding to the previous image frame may be evaluated based on the prediction confidence. For example, the first threshold and the second threshold may be preset. In this example, the first threshold may be greater than the second threshold. The prediction confidence corresponding to the previous image frame may be compared to the above thresholds, and a target prediction state may be determined based on a result of the comparison.

If the prediction confidence is greater than the first threshold, the confidence of the target position and the target size in the target tracking result may be high. This may indicate that the target tracking result is extremely stable. Accordingly, the target tracking state may be determined as a predicted steady state (or a stable target tracking state). If the prediction confidence is less than the second threshold, the confidence of the target position and the target size in the target tracking result may be low. This may indicate that the target tracking result is unstable. Accordingly, the target tracking state may be determined as a target loss state. If the prediction confidence is greater than the second threshold and less than the first threshold, the confidence of the target position and the target size in the target tracking result may be acceptable. This may indicate that stability of the target tracking result is acceptable. In this example, the target tracking state may be determined as an intermediate state.

According to an example, a tracking result includes either one or both of a target position and a target size, and a prediction confidence. The determining of the target tracking state corresponding to the previous image frame based on the target tracking result corresponding to the previous image frame includes obtaining either one or both of a target relative displacement corresponding to the previous image frame and a ratio between a size of a target of the previous image frame and a size of the previous image frame, based on at least one of the target position and the target size; and determining the target tracking state based on a result of a comparison of the prediction confidence and a predetermined threshold, and either one or both of a result of a comparison of the target relative displacement and a predetermined reference displacement and a result of a comparison of the ratio and a predetermined reference ratio. If the prediction confidence is greater than the first threshold and the target relative displacement is less than the reference displacement, the target tracking state may be determined as a predicted steady state. If the prediction confidence is less than the second threshold and the ratio is less than the reference ratio, the target tracking status may be determined as a target loss state. If the prediction confidence is greater than the second threshold and less than the first threshold, the target tracking state may be determined as an intermediate state.

In this example, the target tracking state of the previous image frame may be evaluated by combining the target position and the target size based on the prediction confidence. For example, based on the first threshold and the second threshold, the reference displacement (to evaluate the target relative displacement) and the reference ratio (to evaluate the ratio between the size of the target and the total size) may be set. Subsequently, a result of a comparison of the prediction confidence corresponding to the previous image frame and thresholds, a result of a comparison of the target relative displacement corresponding to the previous image frame and the reference displacement, and a result of a comparison of the ratio between the target size and the size of the previous image frame and the reference ratio may be derived. The target prediction state may be determined based on the results of the comparisons.

The target relative displacement corresponding to the previous image frame may indicate a relative displacement corresponding to a change in a position of a target from an image frame preceding the previous image frame to the previous image frame. For example, the previous frame may be represented as a frame T-<NUM>, and a frame preceding the previous frame may be represented as a frame T-<NUM>. An absolute value of a difference between target positions respectively corresponding to the frames T-<NUM> and T-<NUM> may be calculated, and an average value of target sizes respectively corresponding to the frames T-<NUM> and T-<NUM> may be calculated. A ratio of the absolute value to the average value may be removed, and accordingly the target relative displacement may be obtained.

If the prediction confidence is greater than the first threshold and the target relative displacement is less than the reference displacement, the confidence of the target position and the target size in the target tracking result may be high. This may indicate that the target tracking result is extremely stable. Accordingly, the target tracking state may be determined as a predicted steady state (or a stable target tracking state).

If the prediction confidence is less than the second threshold and the ratio is less than the reference ratio, the confidence of the target position and the target size in the target tracking result may be low. This may indicate that the target tracking result is unstable. Accordingly, the target tracking state may be determined as a target loss state.

If the prediction confidence is greater than the second threshold and less than the first threshold, the confidence of the target position and the target size in the target tracking result may be acceptable. This may indicate that the stability of the target tracking result is acceptable. In this example, the target tracking state may be determined as an intermediate state. In a process of determining a predicted steady state and a target loss, a parallel determination condition may be added, and thus an accuracy of subsequent target tracking and the determined target tracking state may be enhanced.

When the target tracking state corresponding to the previous image frame is determined using the above scheme, an FoV corresponding to the current image frame may be determined by combining the target tracking state with an FoV corresponding to the previous image frame. Hereinafter, a corresponding process will be described in detail.

According to an example, an operation of determining an FoV used to track a target from the current image frame based on the FoV corresponding to the current image frame and a corresponding target tracking state may include the following operations:.

For example, the current image frame may be an image frame T (corresponding to a time T), the previous image frame may be an image frame T-<NUM> (corresponding to a time T-<NUM>), the first reference FoV may be a normal FoV, the second reference FoV may be a wide FoV, and the third reference FoV may be a narrow FoV. If an FoV corresponding to the image frame T-<NUM> is a normal FoV, an FoV corresponding to the image frame T may be determined, as shown in <FIG>. The FoV corresponding to the image frame T may be determined as a wide FoV, a normal FoV, or a narrow FoV depending on whether a target tracking state is a target loss state, an intermediate state, or a predicted steady state. If the FoV corresponding to the image frame T-<NUM> is a wide FoV, the FoV corresponding to the image frame T may be determined, as shown in <FIG>. If the FoV corresponding to frame T-<NUM> is a narrow FoV, the FoV corresponding to the image frame T may be determined, as shown in <FIG>.

As shown in <FIG>, a frame T-<NUM> may be acquired in operation <NUM>. An FoV corresponding to the frame T-<NUM> may be a normal FoV. In operation <NUM>, a target tracking result (e.g., a target position, a target size, and a prediction confidence) according to the frame T-<NUM> may be determined. In operation <NUM>, a prediction confidence may be checked - for example, the prediction confidence and a threshold may be compared. A target tracking state may be determined according to a result of a comparison of the prediction confidence and the threshold. For example, when a prediction confidence corresponding to the frame T-<NUM> is less than a threshold TH_2, the target tracking state may be determined as a target loss state in operation <NUM>. In this example, an FoV corresponding to a next frame (e.g., a frame T) acquired in operation <NUM> may be determined as a wide FoV. For example, when the prediction confidence corresponding to the frame T-<NUM> is greater than a threshold TH_1, the target tracking state may be determined as a predicted steady state in operation <NUM>. In this example, an FoV corresponding to a next frame (e.g., a frame T) acquired in operation <NUM> may be determined as a narrow FoV. Subsequently, operations <NUM> and <NUM> corresponding to operation <NUM> and operations <NUM> and <NUM> corresponding to operation <NUM> may be repeatedly performed. Hyperparameters TH_1 and TH_2 may correspond to preset thresholds. For example, TH_1 may be "<NUM>", and TH_2 may be "<NUM>". For example, when the prediction confidence is greater than the threshold TH_2 and less than the threshold TH_1, the target tracking state may be determined as an intermediate state, and the original FoV (e.g., a normal FoV) may be maintained in association with a next frame (e.g., a frame T) acquired in operation <NUM>.

As shown in <FIG>, a frame T-<NUM> may be acquired in operation <NUM>, and a target tracking result (e.g., a target position, a target size, and a prediction confidence) may be generated in operation <NUM>. An FoV corresponding to the frame T-<NUM> may be a normal FoV. In operation <NUM>, a prediction confidence, a ratio between a size of a target and a size of the frame T-<NUM>, and a target relative displacement according to a target tracking result may be checked. For example, when the prediction confidence corresponding to the frame T-<NUM> is less than a threshold THC_2 and when the ratio is less than a reference ratio THR, uncertainty of a target prediction may increase and the target may be regarded to be lost. Under an assumption that a target may be present within a region during magnification of an FoV, an FoV corresponding to a next frame (e.g., a frame T) may be determined as a wide FoV. For example, when the prediction confidence corresponding to the frame T-<NUM> is greater than a threshold THC_1 and the target relative displacement is less than a reference displacement THD, a target prediction may be regarded to be stable. Under an assumption that a target may be present within a region after an FoV is reduced, the FoV corresponding to the next frame (i.e., a frame T) may be determined as a narrow FoV. Hyperparameters THC_1, THC_2, THR, and THD may correspond to preset thresholds, a reference ratio, and a reference displacement. For example, THC_1, THC_2, THR, and THD may be "<NUM>", "<NUM>", "<NUM>", and "<NUM>", respectively. For example, when the target tracking result is in an intermediate state, for example, when the prediction confidence is greater than TH_2 and less than TH_1, the target relative displacement is greater than THD, and the ratio is greater than THR, the original FoV (e.g., a normal FoV) may be maintained. In addition, the description provided with reference to <FIG> may apply to an example of <FIG>.

According to an example, an operation (e.g., operation <NUM> of <FIG>) of generating a target tracking result corresponding to the current image frame based on an FoV corresponding to the current image frame may include setting a reference template feature based on an initial image frame of an image sequence to which the current image frame belongs, determining a search region of the current image frame (e.g., a second image frame) based on the FoV (e.g., a second FoV) corresponding to the current image frame and a target position (e.g., a target position according to a first target tracking result) of the previous image frame (e.g., a first image frame), obtaining a search feature from the search region, and generating the target tracking result (e.g., a second target tracking result) based on the reference template feature and the search feature.

For example, a target may be tracked using a twin fully convolutional network based on a twin network. For example, as shown in <FIG>, a process of obtaining a target tracking result based on a twin fully convolutional network may include the following operations:.

A twin region proposal network (RPN) may be derived by connecting an RPN to the twin fully convolutional network. When the RPN is combined with the twin fully convolutional network, "k" anchors may be generated each time a correlation corresponding to each position on the template feature zf and the search feature xf is computed. The "k" anchors may represent "k" candidate boxes, for example, a plurality of boxes in an RPN of <FIG>. Candidate boxes may be centered at the anchors, but may have different sizes (e.g. different widths and/or different heights). As shown in <FIG>, a convolution operation may be performed between a template feature zf of a first frame and a search feature xf of a current frame, to generate a similarity map <NUM>. For example, the template feature zf may have a size of <NUM> × <NUM>, the search feature xf may have a size of <NUM> × <NUM>, and the similarity map <NUM> may have a size of <NUM> × <NUM>. If "k" anchors are set to be generated at each position, "<NUM> × <NUM> × k" anchors in total may be generated in the similarity map <NUM>. A twin RPN may include two branches, that is, a classification branch and a regression branch. The classification branch may generate probability values of a positive sample and a negative sample of a prediction target, and the regression branch may generate central coordinates (x, y) and magnitudes (w, h) of a prediction box. A target tracking result <NUM> for the current frame may be generated through the classification branch and the regression branch.

According to an example, an operation of generating the target tracking result (e.g., a second target tracking result) corresponding to the current image frame based on the reference template feature and the search feature may include generating a scaled template feature by scaling the reference template feature when the FoV (e.g., a second FoV) corresponding to the current image frame is a second reference FoV or a third reference FoV, and generating the target tracking result corresponding to the current image frame based on the scaled template feature and the search feature.

In a target tracking process, a template feature may be extracted from an initial image frame (e.g., a first frame) of an image sequence, and an FoV corresponding to the initial image frame may be set as a normal FoV by default. If a dynamic FoV (e.g., a wide FoV or a narrow FoV) is used for a subsequent frame of the image sequence, a size of the template feature may be scaled to fuse feature scales extracted from different FoVs. For example, the following template feature kernel rescaling may be applied. For example, a wide FoV may be magnified by a factor of "r" compared to the normal FoV, the template feature may be scaled down to <NUM>/r times the original size of the template feature (i.e., a size of the template feature in the normal FoV). If a narrow FoV is reduced by a factor of "<NUM>/r" compared to the normal FoV, the template feature may be scaled up to r times the original size of the template feature. Although the first factor and the second factor are assumed to be the same as "r" in the above scheme, the first factor and the second factor may be set to be different from each other.

A target tracking method based on a twin network or a twin RPN according to examples may not change settings of a backbone network. Accordingly, a size of an input image of the backbone network may be maintained. Thus, (<NUM>) there is no need to retrain the backbone network, (<NUM>) an amount of computation of a fixed backbone network may assist subsequent development of algorithm-based software systems, and (<NUM>) quick replacement of other backbone network structures may be possible instead of having an influence on settings of other modules.

According to an example, an operation of generating the scaled template feature by scaling the reference template feature may include generating the scaled template feature by scaling down the reference template feature when an FoV (e.g., a second FoV) corresponding to the current image frame is a second reference FoV, and generating the scaled template feature by scaling up the reference template feature when the FoV corresponding to the current image frame is a third reference FoV.

For example, as shown in <FIG>, a template image <NUM> may be obtained through cropping/scaling <NUM> of a first image frame <NUM>. The template image <NUM> may be obtained by cropping the first image frame <NUM> to the same size as a size of a target and may be scaled to meet an input requirement (e.g., <NUM> × <NUM>) of a backbone network <NUM>. The backbone network <NUM> may generate a template feature <NUM> according to a scaling result. If a normal FoV is used for the first image frame <NUM> and an image frame T <NUM> (i.e., a current image frame), a search image <NUM> with a size that is "s" times a size of a target of a frame T-<NUM> (i.e., a previous image frame) may be extracted from the image frame T <NUM> based on a target position corresponding to the frame T-<NUM>. The search image <NUM> may be scaled according to an input requirement (e.g., <NUM> × <NUM>) of a backbone network <NUM>. The search image <NUM> may be extracted and scaled through cropping/scaling <NUM>. When a scaling result passes through the backbone network <NUM>, a search region feature <NUM> may be obtained, and a correlation between the search region feature <NUM> and the template feature <NUM> may be computed. For example, the template feature <NUM> may have a size of w × h × C, and the search feature <NUM> may have a size of W × H × C.

As shown in <FIG>, when a wide FoV is used for an image frame T <NUM>, for example, when the wide FoV is r times (r > <NUM>, for example, r = <NUM>) the normal FoV, a search image <NUM> with a size that is "s × r" times a size of a target of a frame T-<NUM> may be extracted from the image frame T <NUM> based on a target position corresponding to the frame T-<NUM>. The search image <NUM> may be scaled according to an input requirement (e.g., <NUM> × <NUM>) of a backbone network <NUM>. Through the scaling, the size of the target may be reduced by a factor of "<NUM>/r". To match a scale of a search feature and a scale of a template feature during computation of a correlation, a template feature <NUM> may also be scaled by a corresponding ratio. Accordingly, the template feature <NUM> may be reduced to <NUM>/r times its original size. The template feature <NUM> may correspond to the template feature <NUM> of <FIG>. The template feature <NUM> may be rescaled via a rescaling network <NUM>, and a correlation between a rescaling result <NUM> and a search feature <NUM> may be computed.

As shown in <FIG>, when a narrow FoV is used for an image frame T <NUM>, for example, when the narrow FoV is <NUM>/r times (r > <NUM>, for example, r = <NUM>, <NUM>/r = <NUM>/<NUM>) the normal FoV, a search image <NUM> with a size that is "s × <NUM>/r" times a size of a target of a frame T-<NUM> may be extracted from the image frame T <NUM>. The search image <NUM> may be scaled according to an input requirement (e.g., <NUM> × <NUM>) of a backbone network <NUM>. Accordingly, a size of a target may be magnified by a factor of "r". A template feature <NUM> may also be magnified at a corresponding ratio, that is, r times the original size, via a rescaling network <NUM>, and a correlation between a rescaling result <NUM> and a search feature <NUM> may be computed. In addition, the description provided with reference to <FIG> may apply to examples of <FIG> and <FIG>.

A rescaling network (e.g., the rescaling networks <NUM> and <NUM>, and rescaling networks <NUM>, <NUM>, <NUM>, and <NUM>) of a template feature kernel may be a neural network with a single layer, or a neural network with a small number of layers (e.g., two or three layers). A network according to examples may function to scale a template feature of w × h × C to (w/r) × (h/r) × C, or (rw) × (rh) × C. In this example, w/r, h/r, rw, and rh may all be integers. In an example, as shown in <FIG>, when w = h = <NUM>, r = <NUM> and C = <NUM> are satisfied, the rescaling network <NUM> may obtain a scaling result by scaling a feature vector of <NUM> × <NUM> × <NUM> to <NUM> × <NUM> × <NUM> using a max-pooling layer, or using a convolution layer with a convolution kernel size of <NUM> × <NUM> and a padding size of "<NUM>". In another example, as shown in <FIG>, the rescaling network <NUM> may obtain a scaling result by scaling a feature vector of <NUM> × <NUM> × <NUM> to <NUM> × <NUM> × <NUM> using an upsampling layer and a convolution layer with a convolution kernel size of <NUM> × <NUM>.

According to an example, an operation of obtaining the target tracking result (e.g., a second target tracking result) corresponding to the current image frame based on the scaled template feature and the search feature may include generating a corresponding feature map by performing a convolution operation on the scaled template feature and the search feature using the scaled template feature as a convolution kernel, and generating the target tracking result (e.g., a second target tracking result) based on the feature map. The generating of the target tracking result based on the feature map may be realized through a region proposal algorithm.

For example, as shown in <FIG> and <FIG>, dynamic FoV target tracking based on a twin network structure may be performed. A template <NUM> of a first frame may be converted to an image block through cropping and/or scaling and may be input to a backbone network <NUM>, and the backbone network <NUM> may output a template feature <NUM>. For example, the image block may have a size of <NUM> × <NUM>, and the template feature <NUM> may have a size of <NUM> × <NUM> × <NUM>. A search image <NUM> of a current frame (e.g., a <NUM>th frame) may be converted to an image block through cropping and/or scaling and may be input to a backbone network <NUM>, and the backbone network <NUM> may output a search region feature <NUM>. For example, the image block may have a size of <NUM> × <NUM>, and the search region feature <NUM> may have a size of <NUM> × <NUM> × <NUM>. If a normal FoV tracker is used, a correlation may be computed through a convolution operation between the template feature <NUM> with the size of <NUM> × <NUM> × <NUM> and the search region feature <NUM> with the size of <NUM> × <NUM> × <NUM>, and "<NUM> × <NUM> × K" anchors may be output via an RPN <NUM>. Here, K may indicate a number of anchors per unit. Each anchor may include a probability value of a positive sample according to a classification branch, and position information of a candidate frame according to a regression branch. A predicted target position <NUM> may be output based on the probability value and position information.

As shown in <FIG>, when a wide FoV tracker is used, a search image <NUM> of a search region scaled up by a factor of "r" (e.g., r = <NUM>) may be converted to an image block through cropping and/or scaling and may be input to a backbone network <NUM>, and the backbone network <NUM> may output a search region feature <NUM>. In a process of cropping and/or scaling the search image <NUM>, a size of an object may be reduced by a factor of "<NUM>/r". The template feature <NUM> may be scaled down by a factor of "<NUM>/r" via the template rescaling network <NUM>, and a correlation may be computed through a convolution operation between a rescaling result <NUM> and the search region feature <NUM>. An RPN <NUM> may output "<NUM> × <NUM> × K" anchors. A predicted target position <NUM> may be output based on a probability value and position information.

As shown in <FIG>, when a narrow FoV tracker is used, a search image <NUM> of a search region scaled down by a factor of "<NUM>/r" (e.g., r = <NUM>) may be converted to an image block through cropping and/or scaling and may be input to a backbone network <NUM>, and the backbone network <NUM> may output a search region feature <NUM>. In a process of cropping and/or scaling the search image <NUM>, a size of an object may be increased by a factor of "r". A template feature <NUM> may be scaled up by a factor of "r" via the template rescaling network <NUM>, and a correlation may be computed through a convolution operation between a rescaling result <NUM> and the search region feature <NUM>. An RPN <NUM> may output "<NUM> × <NUM> × K" anchors. In addition, the description provided with reference to <FIG> may apply to an example of <FIG>.

The object tracking method according to examples described herein may apply to a combination of cameras with different FoVs in a multi-camera system as well as different sizes of a search region in a single camera system. For example, the examples may apply to a mobile device including a plurality of cameras with different FoVs. Cameras may have different parameters such as an aperture range and an equivalent focal length. Hereinafter, operations according to examples will be further described based on a target tracking function of a mobile device with a dual camera. For example, a dual camera may be composed of a normal camera with an equivalent focal length of <NUM> and f/<NUM>, and a wide-angle camera with an equivalent focal length of <NUM> and f/<NUM> may be used as a dual camera. In the object tracking method according to the examples herein, a first image frame (e.g., a previous image frame) and a second image frame (e.g., a current image frame) may be collected by different image collectors (e.g., cameras) of the same mobile device (e.g., a smartphone). For example, the first image frame may be collected using a normal camera, and the second image frame may be collected using a wide-angle camera.

As shown in <FIG>, an image frame <NUM> may be acquired using a normal camera of a normal FoV. Since a portion of a target vessel <NUM> in the image frame <NUM> is outside the image frame <NUM>, a neighboring vessel <NUM> having a higher similarity to a template than that of the target vessel <NUM> may be incorrectly set as a target. If it is confirmed that a prediction confidence is low, an image frame <NUM> may be acquired using a wide-angle camera with a wide FoV according to an FoV determination scheme according to the examples. The image frame <NUM> may be the next frame of the image frame <NUM>. An image block <NUM> may be acquired from the image frame <NUM>, and a bounding box corresponding to the target vessel <NUM> may be derived.

<FIG> illustrates an example of a structure of an object tracking apparatus <NUM>. Referring to <FIG>, the object tracking apparatus <NUM> may include a processor <NUM> and a memory <NUM>. The memory <NUM> may be connected to the processor <NUM> and may store instructions executable by the processor <NUM>, data to be computed by the processor <NUM>, or data processed by the processor <NUM>. The memory <NUM> may include a non-transitory computer-readable medium, for example, a high-speed random-access memory (RAM), and/or a non-volatile computer-readable storage medium, for example, at least one disk storage device, a flash memory device, or another non-volatile solid-state memory devices.

The processor <NUM> may execute the instructions stored in the memory <NUM> to perform the operations described above with reference to <FIG> and operations that will be described below with reference to <FIG>. For example, the processor <NUM> may determine a first target tracking state by tracking a target from a first image frame with a first FoV, determine a second FoV based on the first FoV and the first target tracking state, and generate a second target tracking result by tracking a target from a second image frame with the second FoV. The processor <NUM> may generate a first target tracking result by tracking the target from the first image frame and may determine the first target tracking state based on the first target tracking result.

The first target tracking result may include a prediction confidence, and the processor <NUM> may determine the first target tracking state according to a result of a comparison of the prediction confidence and a preset threshold. The first target tracking result may include a prediction confidence, and either one or both of a target position and a target size. The processor <NUM> may obtain at least one of a target relative displacement corresponding to the first image frame and a ratio between a size of the target of the first image frame and a size of the first image frame, based on either one or both of the target position and the target size, and may determine the first target tracking state based on a result of a comparison of the prediction confidence and a preset threshold, and either one or both of a result of a comparison of the target relative displacement and a reference displacement and a result of a comparison of the ratio and a reference ratio.

In addition, the foregoing description provided with reference to <FIG> and the following description provided with reference to <FIG> may apply to the object tracking apparatus <NUM>.

<FIG> illustrates an example of an electronic device <NUM>. Referring to <FIG>, the electronic device <NUM> may include a processor <NUM>, a memory <NUM>, a camera <NUM>, a storage device <NUM>, an input device <NUM>, an output device <NUM>, and a network interface <NUM>, and these components may communicate with one another through a communication bus <NUM>. For example, the electronic device <NUM> may be embodied as at least a portion of a mobile device (e.g., a mobile phone, a smartphone, a personal digital assistant (PDA), a netbook, a tablet computer, a laptop computer, etc.), a wearable device (e.g., a smartwatch, a smart band, smart eyeglasses, etc.), a computing device (e.g., a desktop, a server, etc.), a home appliance (e.g., a television (TV), a smart TV, a refrigerator, etc.), a security device (e.g., a door lock, etc.), or a vehicle (e.g., an autonomous vehicle, a smart vehicle, etc.). The electronic device <NUM> may include, structurally and/or functionally, the object tracking apparatus <NUM> of <FIG>.

The processor <NUM> may execute instructions and functions in the electronic device <NUM>. For example, the processor <NUM> may process instructions stored in the memory <NUM> or the storage device <NUM>. The processor <NUM> may perform one or more of the operations described above with reference to <FIG>. The memory <NUM> may include a non-transitory computer-readable storage medium or a non-transitory computer-readable storage device. The memory <NUM> may store instructions that are to be executed by the processor <NUM>, and may also store information associated with software and/or applications when the software and/or applications are being executed by the electronic device <NUM>.

The camera <NUM> may capture a photo and/or a video. The camera <NUM> may operate with a plurality of reference FoVs, for example, a normal FoV, a wide FoV, and a narrow FoV. For example, the camera <NUM> may generate images of different FoVs, using a plurality of cameras with different FoVs or using lenses with different FoVs.

The storage device <NUM> may include a non-transitory computer-readable storage medium or a non-transitory computer-readable storage device. The storage device <NUM> may store a greater amount of information than the memory <NUM> and store the information for a long period of time. For example, the storage device <NUM> may include a magnetic hard disk, an optical disk, a flash memory, a floppy disk, or other known non-volatile memories.

The input device <NUM> may receive an input from a user through a traditional input scheme using a keyboard and a mouse, and through a newer input scheme such as a touch input, a voice input and an image input. The input device <NUM> may include, for example, a keyboard, a mouse, a touchscreen, a microphone, and/or other devices that may detect the input from the user and transmit the detected input to the electronic device <NUM>. The output device <NUM> may provide an output of the electronic device <NUM> to a user through a visual, auditory, or tactile channel. The output device <NUM> may include, for example, a display, a touch screen, a speaker, a vibration generator, or any other device that provides an output to a user. The network interface <NUM> may communicate with an external device through a wired or wireless network.

According to examples, an apparatus may implement at least one module among a plurality of modules through an artificial intelligence (AI) model. AI-related functions may be performed by a non-volatile memory, a volatile memory, and a processor.

The processor may include one or more processors. The one or more processors may be, for example, general-purpose processors such as a central processing unit (CPU) and an application processor (AP), dedicated graphics processors such as a graphics processing unit (GPU) and a vision processing unit (VPU), and/or dedicated AI processors such as a numeric processing unit (NPU).

The one or more processors may control processing of input data based on a predefined operation rule or AI model stored in the non-volatile memory and the volatile memory. The predefined operation rule or AI model may be provided through training or learning.

Herein, providing of the predefined operation rules or AI model through learning may indicate obtaining a predefined operation rule or AI model with desired characteristics by applying a learning algorithm to a plurality of pieces of training data. The training may be performed by a device having an AI function according to the disclosure, or by a separate server and/or system.

The AI model may include a plurality of neural network layers. Each of the neural network layers may have a plurality of weight values, and calculation of one layer may be performed through a calculation result of a previous layer and a plurality of weight values of a current layer. A neural network may include, for example, a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a generative adversarial network (GAN), and a deep Q network, but is not limited thereto.

The learning algorithm may be a method of training a predetermined target apparatus, for example, a robot, based on a plurality of pieces of training data and of enabling, allowing or controlling the target apparatus to perform determination or prediction. The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited thereto.

The RPNs <NUM>, <NUM>, and <NUM>, the backend networks <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, and <NUM>, the rescaling networks <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, and <NUM>, the processors <NUM> and <NUM>, the memories <NUM> and <NUM>, the storage device <NUM>, the input device <NUM>, the output device <NUM>, the network interface <NUM>, the twin networks, the twin fully convolutional neural networks, the twin RPNs, the neural networks, the processors, and the memories in <FIG> that perform the operations described in this application are implemented by hardware components configured to perform the operations described in this application that are performed by the hardware components. Examples of hardware components that may be used to perform the operations described in this application where appropriate include controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more of the hardware components that perform the operations described in this application are implemented by computing hardware, for example, by one or more processors or computers. A processor or computer may be implemented by one or more processing elements, such as an array of logic gates, a controller and an arithmetic logic unit, a digital signal processor, a microcomputer, a programmable logic controller, a field-programmable gate array, a programmable logic array, a microprocessor, or any other device or combination of devices that is configured to respond to and execute instructions in a defined manner to achieve a desired result. In one example, a processor or computer includes, or is connected to, one or more memories storing instructions or software that are executed by the processor or computer. Hardware components implemented by a processor or computer may execute instructions or software, such as an operating system (OS) and one or more software applications that run on the OS, to perform the operations described in this application. The hardware components may also access, manipulate, process, create, and store data in response to execution of the instructions or software. For simplicity, the singular term "processor" or "computer" may be used in the description of the examples described in this application, but in other examples multiple processors or computers may be used, or a processor or computer may include multiple processing elements, or multiple types of processing elements, or both. For example, a single hardware component or two or more hardware components may be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components may be implemented by one or more processors, or a processor and a controller, and one or more other hardware components may be implemented by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller, may implement a single hardware component, or two or more hardware components. A hardware component may have any one or more of different processing configurations, examples of which include a single processor, independent processors, parallel processors, single-instruction single-data (SISD) multiprocessing, single-instruction multiple-data (SIMD) multiprocessing, multiple-instruction single-data (MISD) multiprocessing, and multiple-instruction multiple-data (MIMD) multiprocessing.

Claim 1:
An object tracking method, the method comprising:
determining (<NUM>) a first target tracking state by tracking a target from a first image frame with a first field of view, FoV;
determining (<NUM>) a second FoV based on the first FoV and the first target tracking state; and
generating (<NUM>) a second target tracking result by tracking a target from a second image frame with the second FoV;
wherein the first FoV and the second FoV are selected from predetermined reference FoVs;
wherein the determining (<NUM>) of the first target tracking state comprises:
generating (<NUM>) a first target tracking result by tracking the target from the first image frame; and
determining (<NUM>) the first target tracking state based on the first target tracking result;
wherein the first target tracking result comprises a prediction confidence, and either one or both of a target position and a target size, and
wherein the determining (<NUM>) of the first target tracking state based on the first target tracking result comprises:
obtaining (<NUM>) at least one of a target relative displacement corresponding to the first image frame and a ratio between a size of the target of the first image frame and a size of the first image frame, based on either one or both of the target position and the target size; and
determining the first target tracking state based on a result of a comparison of the prediction confidence and a preset threshold, and either one or both of a result of a comparison of the target relative displacement and a reference displacement and a result of a comparison of the ratio and a reference ratio.