METHOD AND APPARATUS WITH OBJECT TRACKING

A method and apparatus for object tracking are provided, where the object tracking method includes determining box information of candidate boxes in a current image frame and similarity scores of the candidate boxes based on including a search region of the current image frame with a template image corresponding to a target object, adjusting the similarity scores of the candidate boxes using a distractor map including distractor information of a previous image frame, determining a target box corresponding to the target object and a distractor box corresponding to a distractor of the target object from the candidate boxes based on the adjusted similarity scores, and updating the distractor map based on distractor information of the current image frame according to the distractor box.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit under 35 USC § 119(a) of Korean Patent Application No. 10-2021-0188436, filed on Dec. 27, 2021, in the Korean Intellectual Property Office, the entire disclosure of which is incorporated herein by reference for all purposes.

BACKGROUND

The following description relates to a method and apparatus for object tracking.

2. Description of Related Art

Object recognition has been automated using, for example, a neural network model, which may provide a computationally intuitive mapping between an input pattern and an output pattern after considerable training. An ability to be trained to generate such mapping may be referred to as a learning ability of the neural network. Moreover, due to specialized training, such a specialized and trained neural network may have a generalization ability to generate a relatively accurate output for an input pattern that has not been trained.

SUMMARY

In one general aspect, there is provided a processor-implemented object tracking method, including determining box information of candidate boxes in a current image frame and similarity scores of the candidate boxes based on including a search region of the current image frame with a template image corresponding to a target object, adjusting the similarity scores of the candidate boxes using a distractor map including distractor information of a previous image frame, determining a target box corresponding to the target object and a distractor box corresponding to a distractor of the target object from the candidate boxes based on the adjusted similarity scores, and updating the distractor map based on distractor information of the current image frame according to the distractor box.

The adjusting of the similarity scores may include determining a mask according to the distractor information of the previous image frame, and adjusting the similarity scores based on an overlap between the candidate boxes and the mask.

The adjusting of the similarity scores based on the overlap may include reducing a similarity score of a candidate box from among the candidate boxes overlapping with the mask at a ratio greater than or equal to a threshold ratio.

The adjusting of the similarity scores based on the overlap state may include reducing the similarity scores of each of the candidate boxes in proportion to an overlap ratio of the respective candidate boxes with the mask.

The updating of the distractor map may include applying motion of the distractor to the distractor map.

The applying of the motion of the distractor to the distractor map may include inputting the distractor information of the previous image frame and the distractor information of the current image frame to a neural network-based motion estimation model, and estimating the motion of the distractor from an output of the motion estimation model.

The object tracking method further may include determining a tracking state of the current image frame based on the box information and the similarity scores, and setting an object tracking mode to any one of a precise tracking mode for performing object tracking with the distractor map or a normal tracking mode for performing object tracking without the distractor map based on the tracking state.

The object tracking may be performed based on the adjusting of the similarity scores, the determining of the target box and the distractor box, and the updating of the distractor map, in response to the object tracking mode being set to the precise tracking mode, and the target box may be determined based on the box information of the candidate boxes and the similarity scores of the candidate boxes without performing the adjusting of the similarity scores, the determining of the target box and the distractor box, and the updating of the distractor map, in response to the object tracking mode being set to the normal tracking mode.

The setting of the object tracking mode may include setting the object tracking mode to the normal tracking mode, in response to any one or any combination of non-existence of the distractor of the target object, occlusion of the target object, and detachment of the target object from a frame.

The determining of the box information of the candidate boxes and the similarity scores of the candidate boxes may include inputting the search region and the template image to a neural network-based image comparison model, and determining the box information of the candidate boxes and the similarity scores of the candidate boxes from an output of the image comparison model.

In another general aspect, there is provided an object tracking apparatus, including a memory configured to store instructions executable by the processor, and a processor configured to execute the instructions to configure the processor to determine box information of candidate boxes in a current image frame and similarity scores of the candidate boxes based on including a search region of the current image frame with a template image corresponding to a target object, adjust the similarity scores of the candidate boxes using a distractor map including distractor information of a previous image frame, determine a target box corresponding to the target object and a distractor box corresponding to a distractor of the target object from the candidate boxes based on the adjusted similarity scores, and update the distractor map based on distractor information of the current image frame according to the distractor box.

The processor may be configured to determine a mask according to the distractor information of the previous image frame, and adjust the similarity scores based on an overlap between the candidate boxes and the mask.

The processor may be configured to update the distractor map by applying motion of the distractor to the distractor map.

The processor may be configured to determine a tracking state of the current image frame based on the box information and the similarity scores, and set an object tracking mode to any one of a precise tracking mode for performing object tracking with the distractor map or a normal tracking mode for performing object tracking without the distractor map based on the tracking state.

The processor may be configured to input the search region and the template image to an image comparison model based on a neural network, and determine the box information of the candidate boxes and the similarity scores of the candidate boxes from an output of the image comparison model.

In another general aspect, there is provided an electronic apparatus, including a camera configured to generate an input image including image frames, and a processor configured to execute the instructions to configure the processor to determine box information of candidate boxes in a current image frame and similarity scores of the candidate boxes based on including a search region of the current image frame among the image frames with a template image corresponding to a target object, adjust the similarity scores of the candidate boxes using a distractor map including distractor information of a previous image frame among the image frames, determine a target box corresponding to the target object and a distractor box corresponding to a distractor of the target object from the candidate boxes based on the adjusted similarity scores, and update the distractor map based on distractor information of the current image frame according to the distractor box.

The processor may be configured to determine a mask according to the distractor information of the previous image frame, and adjust the similarity scores based on an overlap state between the candidate boxes and the mask.

The processor may be configured to update the distractor map by applying motion of the distractor to the distractor map.

The processor may be configured to determine a tracking state of the current image frame based on the box information and the similarity scores, and set an object tracking mode to any one of a precise tracking mode for performing object tracking with the distractor map or a normal tracking mode for performing object tracking without the distractor map based on the tracking state.

The template image may include an image frame from among the image frames, and the search region may include another image frame from among the image frames succeeding the image frame.

A processor-implemented object tracking method, including determining a template image corresponding to a target object from image frames received from a sensor; determining a search region from an image frame of the image frames subsequent to an image frame of the template image; extracting a template feature map from the template image and a search feature map from the search region; determining box information of candidate boxes and similarity scores of the candidate boxes based on a comparison of the template feature map and the search feature map using a neural network-based image comparison model; adjusting the similarity scores of the candidate boxes using a distractor map including distractor information of an image frame prior to the image frame of the search region; and generating a target box corresponding to the target object and a distractor box corresponding to a distractor of the target object from the candidate boxes based on the adjusted similarity scores.

The object tracking method may include outputting tracking result of the target object based on box information corresponding to the target box.

The object tracking method may include updating the distractor map based on box information corresponding to the distractor box.

A size of the search region may be greater than a size of the template image.

DETAILED DESCRIPTION

Throughout the specification, when a component is described as being “connected to,” or “coupled to” another component, it may be directly “connected to,” or “coupled to” the other component, or there may be one or more other components intervening therebetween. In contrast, when an element is described as being “directly connected to,” or “directly coupled to” another element, there can be no other elements intervening therebetween.

The use of the term “may” herein with respect to an example or embodiment (e.g., as to what an example or embodiment may include or implement) means that at least one example or embodiment exists where such a feature is included or implemented, while all examples are not limited thereto.

Also, in the description of example embodiments, detailed description of structures or functions that are thereby known after an understanding of the disclosure of the present application will be omitted when it is deemed that such description will cause ambiguous interpretation of the example embodiments. Hereinafter, examples will be described in detail with reference to the accompanying drawings, and like reference numerals in the drawings refer to like elements throughout.

FIG.1illustrates an example of a configuration and operation of an object tracking apparatus. Referring toFIG.1, an object tracking apparatus100may output a tracking result103based on a template image101and a search image102. The template image101may provide information of a target object to be tracked. The object tracking apparatus100may track the target object in the search image102using the information of the target object (or hereinafter the target object information) of the template image101. The tracking result103may represent a position of the target object in the target image102. In an example, the tracking result103may be used for automated tracking, zooming, and focusing.

In an example, the template image101and the search image102may correspond to a plurality of image frames of an input image. For example, the template image101may correspond to an image frame of an input video file including a plurality of image frames, and the search image102may correspond to at least one image frame after the image frame corresponding to the template image101. For another example, the template image101and the search image102may correspond to files independent of each other. In this example, the search image102may correspond to an input video file including a plurality of image frames, and the template image101may correspond to a still input image file that is not related to the input video file. In either case, the template image101may include the target object, and the object tracking apparatus100may generate the tracking result103by tracking the target object in the search image102. The template image101and the search image102may correspond to either an entire region of a corresponding image frame or a portion of the corresponding image frame. For example, the object tracking apparatus100may set a search region in the search image102and track the target object in the search region.

The neural network or DNN may generate mapping between input information and output information, and may have a generalization capability to infer a relatively correct output with respect to input information that has not been used for training. The neural network may refer to a general model that has an ability to solve a problem or perform tasks, as non-limiting examples, where nodes form the network through connections and other parameter adjustment through training

The DNN may include at least one of a fully connected network (FCN), a convolutional neural network (CNN), and a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent DNN (BRDNN), a deep Q-network, or a combination of two or more thereof, but examples thereof are not limited to the foregoing examples. For example, at least a portion of the layers in the neural network may correspond to the CNN, and another portion of the layers may correspond to the FCN. In this case, the CNN may be referred to as a convolutional layer, and the FCN may be referred to as a fully connected layer.

In the case of the CNN, data input to each layer may be referred to as an input feature map, and data output from each layer may be referred to as an output feature map. The input feature map and the output feature map may also be referred to as activation data. When the convolutional layer corresponds to an input layer, an input feature map of the input layer may be an input image. The output feature map may be generated through a convolution operation between the input feature map and a weight kernel. Each of the input feature map, the output feature map, and the weight kernel may be distinguished by a unit of a tensor.

In an example, training an artificial neural network may indicate determining and adjusting weights and biases between layers or weights and biases among a plurality of nodes belonging to different layers adjacent to one another, as only non-limiting examples of such parameters.

After trained based on deep learning, the neural network may perform inference that is suitable for a training purpose by mapping input data and output data that are in a nonlinear relationship to each other. The deep learning is a machine learning technique for solving a problem such as image or speech recognition from a big data set. The deep learning may be construed as an optimized problem solving process of finding a point at which energy is minimized while training the neural network using prepared training data.

Through supervised or unsupervised learning of the deep learning, a structure of the neural network or a weight corresponding to the model may be obtained, and the input data and the output data may be mapped to each other through the weight. When a width and a depth of the neural network are sufficiently large, the neural network may have a capacity sufficient to implement an arbitrary function. The neural network may achieve an optimized performance when learning a sufficiently large amount of training data through an appropriate training process.

The neural network will be expressed as being trained in advance or pre-trained. The expression “trained in advance” or “pre-trained” may indicate a time before the neural network “starts.” That the neural network “starts” means that the neural network is ready for inference. For example, that the neural network that “starts” may include that the neural network is loaded into a memory, or that input data for the inference is input to the neural network after the neural network is loaded into the memory.

The neural network may include a hardware structure that may be implemented through execution of instructions by a processor.

FIG.2illustrates an example of deriving a similarity score. Referring toFIG.2, an object tracking apparatus may perform object tracking through feature extraction210, similarity calculation220, and bounding box regression230. At least one of the feature extraction210, the similarity calculation220, and the bounding box regression230may be performed through an object tracking model. For example, the object tracking model may include at least one of a feature extraction network for the feature extraction210, a similarity calculation network for the similarity calculation220, and a bounding box regression network for the bounding box regression230. Each of the feature extraction network, the similarity calculation network, and the bounding box regression network may correspond to a neural network. The object tracking model may include a Siamese network, for example.

The object tracking apparatus may extract a template feature map211from a template image201and extract a search feature map212from a search region203. The object tracking apparatus may extract the template feature map211and the search feature map212using the object tracking model and/or a feature extraction model sharing parameters. In the example ofFIG.2, the template image201corresponds to a partial region of an initial image frame (which may be referred to as a first image frame) of an input image, and the search region203corresponds to a partial region of an nth image frame of the input image. Here, n may be a value greater than 1.

When a target object is determined in the first image frame, a target box202corresponding to the target object may be determined. For example, the target object may be determined according to a user input for selecting the target object. The target box202is a type of a bounding box and may be specified based on box position information (e.g., x-coordinate and y-coordinate) and box size information (e.g., a width and a height). The box position information and the box size information may be collectively referred to as box information. The template image201may be determined based on a position and a size of the target box202. The search region203may be determined based on the template image201. A size of the search region203may be determined based on a size of the template image201. For example, the size of the search region203may be determined to be greater than the size of the template image201. A position of the search region203may be determined based on a position of a target box in a previous image frame. For example, when a target box is detected from an n-1 th image frame, a search region of an nth image frame may be determined based on a position of the target box.

The object tracking apparatus may calculate similarity by comparing the template feature map211and the search feature map212. The similarity calculation220may be performed through the similarity calculation network. The similarity calculation network may derive a cross correlation between the template feature map211and the search feature map212through a cross correlation layer. A calculation result may represent information of the target object and/or a position in the search region203corresponding to the template feature map211. For example, the calculation result may display a corresponding position222in a search space221corresponding to the search region203and/or a score of the position222.

The object tracking apparatus may perform regression using bounding boxes232of the position222in the search space231corresponding to the search region203. The object tracking apparatus may determine a target box204corresponding to the target object in the search region203through the regression and generate a tracking result based on box information of the target box204.

In an example, a similarity score is based on objectness of the target object. Thus, when a distractor of the target object is in the search region203, accuracy of object tracking may be affected. The distractor may be an object that is not the target object but may be highly similar to the target object. For example, in an image where track and field athletes run on a track, other athletes around an athlete who is the target object may be distractors. When a scene has many distractors, there may be numerous activation peaks corresponding to high similarity scores.

For example, the distractors may be distinguished from the target object by giving a low weight to a similarity score of an object far from a center of the target object. This technique, however, may not be effective in object tracking, for example, when the target object is occluded, when the similarity score decreases due to deformation of the target object, when an object highly similar to the target object approaches the target object, and the like. For another example, to increase distinction between a distractor and a target, a technique of configuring a backbone and a loss function and a technique using multiple object tracking (MOT) may be employed. However, these techniques may need large-scale operations.

FIG.3illustrates an example of an object tracking operation using a distractor map. In an example, an object tracking apparatus may distinguish between a target object and a distractor using a distractor map including distractor information. The distractor map may represent the distractor information in a low-capacity representation and may be applicable to object tracking through a simple operation. For example, the distractor map may include representation bits of the same size as a size of an image frame, and each representation bit may indicate the presence or absence of the distractor in its position. An adjustment operation using the distractor map may be added to a typical object tracking operation in a manner of an add-on, adding only a substantially small amount of computation. Because such an adjustment operation uses candidate boxes, an additional operation cost may be substantially smaller than other techniques that need an additional neural network model such as an MOT for handling the distractor. Also, to reduce a computation amount, most deep learning-based object trackers mainly use a partial region, not an entire frame, for object tracking. However, the distractor map may provide information on the entire frame and may thereby compensate for a limitation that may be caused by using the partial region.

Referring toFIG.3, the object tracking apparatus may perform an object tracking operation310based on a search region311of a Tth image frame321at time T (t=T) and determine a target box312of the search region311. The object tracking apparatus may perform the object tracking operation310using distractor information of a T-1th image frame at time T-1 (t=T-1) and perform an update operation320on a distractor map based on distractor information of the Tth image frame321. The distractor map may have the same size as a size of the Tth image frame321and include distractor information of at least a partial region in the Tth image frame321. The distractor map including distractor information of a region of interest (ROI)322will be described below, but the distractor map may include distractor information of a region other than the ROI322. The distractor information may include a mask323corresponding to a distractor box.

The object tracking apparatus may perform an object tracking operation330based on a search region331of a T+1th image frame at time T+1 (t=T+1) and determine a target box332of the search region331. The object tracking apparatus may perform the object tracking operation330using the distractor information of the Tth image frame321. The ROI322may have the same size as a size of the search region331. The object tracking apparatus may distinguish between a target object and a distractor by applying the mask323of the ROI322to the search region331and determine the target box332of the search region331.

FIG.4illustrates an example of an operation of object tracking using a distractor map. Referring toFIG.4, an object tracking apparatus may determine box information xn, yn, wn, and hn of candidate boxes in a Tth image frame and similarity scores sn of the candidate boxes by comparing a search region401of the Tth image frame with a template image402corresponding to a target object403. Here, n may be a box identifier and have a value of 1 through N. N may be a total number of the candidate boxes. The object tracking apparatus may input the search region401and the template image402to an image comparison model410based on a neural network and determine the box information xn, yn, wn, and hn and the similarity scores sn from an output411of the image comparison model410.

The object tracking apparatus may adjust the similarity scores sn by performing a score adjustment operation420using a distractor map405including distractor information of a T-1th image frame. When the Tth image frame corresponds to a first image frame, the object tracking apparatus may uniformly adjust the similarity scores sn of all the candidate boxes under the assumption that all the candidate boxes of the output411are distractors. The object tracking apparatus may determine the target box421corresponding to a target object and a distractor box422corresponding to a distractor of the target object from the candidate boxes based on the adjusted similarity scores. The object tracking apparatus may update the distractor map405to a distractor map406based on distractor information of the Tth image frame according to the distractor box422.

FIG.5illustrates an example of selecting a target box and a distractor box from among candidate boxes. Referring toFIG.5, an object tracking apparatus may remove duplicate or overlapping candidate boxes through a candidate box removal operation510. For example, the object tracking apparatus may perform a non-maximum suppression (NMS).

The object tracking apparatus may adjust similarity scores sn through a score adjustment operation520. The object tracking apparatus may determine a mask according to distractor information of a previous image frame and adjust the similarity scores sn based on an overlap state between candidate boxes and the mask. When there is a candidate box overlapping with the mask, the candidate box may correspond to a distractor, and the object tracking apparatus may thus reduce a similarity score of the candidate box in such a way that the candidate box is not selected as a target box. The mask may display an inner region of a distractor box of the previous image frame. When there is a plurality of distractor boxes in the previous image frame, the mask may be set corresponding to all the distractor boxes.

In an example, the object tracking apparatus may reduce a similarity score of a candidate box overlapping with the mask at a ratio greater than or equal to a threshold ratio (50% or more, for example) among the candidate boxes. The object tracking apparatus may reduce the similarity scores of the candidate boxes in proportion to an overlap ratio of each of the candidate boxes with the mask. For example, the object tracking apparatus may reduce by 50% a similarity score of a first candidate box overlapping the mask 50% and reduce by 60% a similarity score of a second candidate box overlapping the mask 60%.

The object tracking apparatus may select the target box and the distractor box from among the candidate boxes through a box selection operation530. The object tracking apparatus may select K candidate boxes with a high reliability score from among the candidate boxes, determine one with the highest reliability score among the K candidate boxes as the target box, and determine the rest of the K candidate boxes as distractor boxes. In the example ofFIG.5, K may be 3. A similarity score s1 of a distractor may have the highest value before the score adjustment operation520. Through the score adjustment operation520, however, the similarity score s1 may be reduced to a similarity score s1′ and a similarity score s3′ of the target object may have the highest value.

FIG.6illustrates an example of updating a distractor map by applying distractor motion. Referring toFIG.6, an object tracking apparatus may determine a target box612and distractor boxes613in a search region611of a Tth image frame615and determine a distractor map614based on distractor information according to the distractor boxes613. The distractor map614may include a mask corresponding to the distractor boxes613.

The object tracking apparatus may determine a distractor map618by applying distractor motion617to the distractor map614and determine a target box622of a search region621by performing an object tracking operation620on the search region621of a T+1th image frame using the distractor map618. The object tracking apparatus may adjust the mask and/or an ROI616based on the distractor motion617. The object tracking apparatus may move the mask in the distractor map614in the same direction as the distractor motion617. In another example, the object tracking apparatus may move the ROI616in the opposite direction to the distractor motion617.

The distractor maps614and618are illustrated inFIG.6as corresponding to a partial region of the image frame615. However, the distractor maps614and618may correspond to an entire region of the image frame615and a partial region corresponding to the ROI616in the entire region may be extracted to be used for a score adjustment operation. Thus, when the object tracking apparatus moves the mask in the distractor map614in the same direction as the distractor motion617or moves the ROI616in the opposite direction to the direction of the distractor motion617, the mask in the distractor map614may be adjusted as with the mask in the distractor map618.

The object tracking apparatus may estimate the distractor motion617based on difference between distractor information of a previous image frame and distractor information of a current image frame. In an example, the object tracking apparatus may estimate the distractor motion617using a neural network -based motion estimation model. The object tracking apparatus may input at least one of the distractor information of the previous image frame and the distractor information of the current image frame to the motion estimation model and estimate motion of a distractor from output of the motion estimation model. The motion estimation model may be trained in advance to estimate the motion of the distractor from distractor information of a plurality of consecutive image frames.

FIG.7illustrates an example of object tracking motion in a current object tracking mode. The operations inFIG.7may be performed in the sequence and manner as shown, although the order of some operations may be changed or some of the operations omitted without departing from the spirit and scope of the illustrative examples described. Many of the operations shown inFIG.7may be performed in parallel or concurrently. One or more blocks ofFIG.7, and combinations of the blocks, can be implemented by special purpose hardware-based computer, such as a processor, that perform the specified functions, or combinations of special purpose hardware and computer instructions. For example, operations of the method may be performed by a computing apparatus (e.g., the object tracking apparatus800inFIG.8). In addition to the description ofFIG.7below, the descriptions ofFIGS.1-6are also applicable toFIG.7, and are incorporated herein by reference. Thus, the above description may not be repeated here.

Referring toFIG.7, in operation701, an object tracking apparatus may determine a template image. For example, the object tracking apparatus may determine the template image based on a user input selecting a target object in a first image frame of an input image. In operation702, the object tracking apparatus may extract a template feature from the template image. In operation703, the object tracking apparatus may determine a search region. For example, the object tracking apparatus may determine the search region in an nth image frame of the input image. Here, n may be a value greater than 1. The search region of the nth image frame may be determined based on box information of a target box of an n-1th image frame. In operation704, the object tracking apparatus may extract a search feature from a search image. In operation705, the object tracking apparatus may determine candidate boxes. The determining of the candidate boxes may include the determining of box information of the candidate boxes and similarity scores of the candidate boxes. The object tracking apparatus may determine the box information and the similarity scores by comparing the template feature and the search feature.

In operation706, the object tracking apparatus may determine whether a current object tracking mode is a precise tracking mode. An object tracking mode may include the precise tracking mode for performing object tracking with a distractor map and a normal tracking mode for performing the object tracking without the distractor map. The object tracking apparatus may determine a tracking state of a current image frame based on the box information and the similarity scores and set the object tracking mode to any one of the precise tracking mode and the normal tracking mode based on the tracking state. In the precise tracking mode, the object tracking may be performed based on adjusting of the similarity scores, determining of the target box and a distractor box, and updating of the distractor map. In the normal tracking mode, the target box may be determined based on the box information of the candidate boxes and the similarity scores of the candidate boxes without such adjusting of the similarity scores, determining of the target box and the distractor box, and updating of the distractor map.

For example, the object tracking apparatus may set the object tracking mode to the normal tracking mode in at least any one case of non-existence of the distractor of the target object, occlusion of the target object, and detachment of the target object from a frame. For example, when the search region has only one candidate box, the search region may only have the target object without the distractor. In this case, using the distractor map may only decrease operational efficiency rather than increasing tracking performance, and the object tracking apparatus may thus perform the object tracking in the normal tracking mode without the distractor map. The normal tracking mode may also be used when a degree of the occlusion of the target object is high, the target object is detached from the frame, or there is high probability of such cases.

In operation707, when the current object tracking mode is the precise tracking mode, the object tracking apparatus may adjust the similarity scores using the distractor map. In operation708, the object tracking apparatus may perform box selection based on the similarity scores. When the current object tracking mode is the precise tracking mode, the box selection may be performed based on the similarity scores adjusted in operation707, and the target box and the distractor box may be determined through the box selection. When the current object tracking mode is the normal tracking mode, the box selection may be performed without the score adjustment in operation707, and only the target box may be determined through the box selection.

In operation709, the object tracking apparatus may update the distractor map. Operation709may be performed in the precise tracking mode. When the distractor box is determined in the precise tracking mode, the distractor map may be updated based on the distractor information of the distractor box. In operation711, when the motion of the distractor is applied, the object tracking apparatus may identify the motion of the distractor. In operation709, the object tracking apparatus may update the distractor map by applying the motion of the distractor to the distractor map.

In operation710, the object tracking apparatus may output target box information. The target box information may correspond to a tracking result. The target box information may include position information and size information of the target box. The object tracking apparatus may repeatedly perform operations703through711based on next frames.

FIG.8illustrates an example of a configuration of an object tracking apparatus. Referring toFIG.8, an object tracking apparatus800may include a processor810and a memory820. The memory820may be connected to the processor810and store instructions executable by the processor810, data to be computed by the processor810, or data processed by the processor810.

The memory820may include a volatile memory and/or a non-volatile memory. The volatile memory device may be implemented as a dynamic random-access memory (DRAM), a static random-access memory (SRAM), a thyristor RAM (T-RAM), a zero capacitor RAM (Z-RAM), or a twin transistor RAM (TTRAM).

The non-volatile memory device may be implemented as an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic RAM (MRAM), a spin-transfer torque (STT)-MRAM, a conductive bridging RAM(CBRAM), a ferroelectric RAM (FeRAM), a phase change RAM (PRAM), a resistive RAM (RRAM), a nanotube RRAM, a polymer RAM (PoRAM), a nano floating gate Memory (NFGM), a holographic memory, a molecular electronic memory device), or an insulator resistance change memory. Further details regarding the memory820is provided below.

The processor810may execute instructions for performing the operations described with reference toFIGS.1through7,9, and10. For example, the processor810may determine box information of candidate boxes in a current image frame and similarity scores of the candidate boxes by comparing a search region of the current image frame with a template image corresponding to a target object, adjust the similarity scores of the candidate boxes using a distractor map including distractor information of a previous image frame, determine a target box corresponding to the target object and a distractor box corresponding to a distractor of the target object from the candidate boxes based on the adjusted similarity scores, and update the distractor map based on distractor information of the current image frame according to the distractor box.

The processor810may read/write neural network data, for example, image data, feature map data, kernel data, biases, weights, for example, connection weight data, hyperparameters, and other parameters etc., from/to the memory820and implement the neural network-based image comparison model and the neural network-based motion estimation model using the read/written data. When the neural network is implemented, the processor210may repeatedly perform operations between an input and parameters, in order to generate data with respect to an output. Here, in an example convolution layer, a number of convolution operations may be determined, depending on various factors, such as, for example, the number of channels of the input or input feature map, the number of channels of the kernel, a size of the input feature map, a size of the kernel, number of the kernels, and precision of values. Such a neural network may be implemented as a complicated architecture, where the processor810performs convolution operations with an operation count of up to hundreds of millions to tens of billions, and the frequency at which the processor810accesses the memory820for the convolution operations rapidly increases.

The processor810may be a processing device implemented by hardware including a circuit having a physical structure to perform operations. For example, the operations may be implemented by execution of computer-readable instructions that configure the processing device to perform any one, or any combination, of the operations described.

For example, the hardware-implemented data processing device may include a microprocessor, a central processing unit (CPU), a processor core, a multi-core processor, a multiprocessor, an application-specific integrated circuit (ASIC), and a field-programmable gate array (FPGA). Further details regarding the processor810is provided below.

In addition, the description provided with reference toFIGS.1through7,9, and10may apply to the object tracking apparatus800.

FIG.9illustrates an example of a configuration of an electronic apparatus. Referring toFIG.9, an electronic apparatus900may include a processor910, a memory920, a camera930, a storage device940, an input device950, an output device960, and a network interface970. The processor910, the memory920, the camera930, the storage device940, the input device950, the output device960, and the network interface970may communicate with each other via a communication bus. For example, the electronic apparatus900may be embodied as at least a portion of a mobile device (e.g., a mobile phone, a smartphone, a personal digital assistant (PDA), a netbook, a tablet computer, a laptop computer, etc.), a wearable device (e.g., a smartwatch, a smart band, smart eyeglasses, etc.), a computing device (e.g., a desktop, a server, etc.), a home appliance (e.g., a television (TV), a smart TV, a refrigerator, etc.), a security device (e.g., a door lock, etc.), or a vehicle (e.g., an autonomous vehicle, a smart vehicle, etc.). The electronic apparatus900may include, structurally and/or functionally, an object tracking apparatus (e.g., the object tracking apparatus100ofFIG.1and the object tracking apparatus800ofFIG.8).

The processor910may execute instructions and functions in the electronic apparatus900. For example, the processor910may process instructions stored in the memory920or the storage device940. The processor910may perform the operations described with reference toFIGS.1through8and10. In addition to the description of the processor910above, the description of the processor810ofFIG.8is also applicable to processor910, and are incorporated herein by reference. Thus, the above description may not be repeated here.

The memory920may include a non-transitory computer-readable storage medium or a non-transitory computer-readable storage device. The memory920may store instructions that are to be executed by the processor910, and store information associated with software and/or applications when the software and/or applications are being executed by the electronic apparatus900. In addition to the description of the memory920above, the description of the memory820ofFIG.8is also applicable to memory920, and are incorporated herein by reference. Thus, the above description may not be repeated here.

The camera930may capture a photo and/or an image. For example, the camera930may generate an input image including a plurality of image frames. In an example, the camera930may obtain, for example, a color image, a black and white image, a gray image, an infrared (IR) image, or a depth image. The image frames may include at least a portion of the template image and a search image.

The storage device940may include a non-transitory computer-readable storage medium or a non-transitory computer-readable storage device. The storage device940may store a greater amount of information than the memory920and store the information for a long period of time. For example, the storage device940may include magnetic hard disks, optical discs, flash memories, floppy disks, or other forms of non-volatile memories known in the art.

The input device950may receive an input from a user through a traditional input scheme using a keyboard and a mouse, and through a new input scheme such as a touch input, a voice input, and an image input. The input device950may include, for example, a keyboard, a mouse, a touchscreen, a microphone, or any other device that may detect an input from a user and transfer the detected input to the electronic apparatus900. The output device960may provide a user with output of the electronic apparatus900through a visual channel, an auditory channel, or a tactile channel. The output device960may include, for example, a display, a touchscreen, a speaker, a vibration generator, or any other device that may provide a user with the output. In an example, the output device960may also be configured to receive an input from the user, such as, a voice input, a gesture input, or a touch input. The network interface970may communicate with an external device via a wired or wireless network.

In addition, the description provided with reference toFIGS.1through8and10may apply to the electronic apparatus900.

FIG.10illustrates an example of an object tracking method. The operations inFIG.7may 10 be performed in the sequence and manner as shown, although the order of some operations may be changed or some of the operations omitted without departing from the spirit and scope of the illustrative examples described. Many of the operations shown inFIG.10may be performed in parallel or concurrently. One or more blocks ofFIG.10, and combinations of the blocks, can be implemented by special purpose hardware-based computer, such as a processor, that perform the specified functions, or combinations of special purpose hardware and computer instructions. For example, operations of the method may be performed by a computing apparatus (e.g., the object tracking apparatus800inFIG.8). In addition to the description ofFIG.10below, the descriptions ofFIGS.1-9are also applicable toFIG.10, and are incorporated herein by reference. Thus, the above description may not be repeated here.

Referring toFIG.10, in operation1010, an object tracking apparatus may determine box information of candidate boxes in a current image frame and similarity scores of the candidate boxes by comparing a search region of the current image frame with a template image corresponding to a target object. The object tracking apparatus may input the search region and the template image to a neural network-based image comparison model and determine the box information of the candidate boxes and the similarity scores of the candidate boxes from an output of the image comparison model.

In operation1020, the object tracking apparatus may adjust the similarity scores of the candidate boxes using a distractor map including distractor information of a previous image frame. The object tracking apparatus may determine a mask according to the distractor information of the previous image frame and adjust the similarity scores based on an overlap state between the candidate boxes and the mask. The object tracking apparatus may reduce a similarity score of a candidate box overlapping with the mask at a ratio greater than or equal to a threshold ratio among the candidate boxes. The object tracking apparatus may reduce the similarity scores of the candidate boxes in proportion to an overlap ratio of each of the candidate boxes with the mask.

In operation1030, the object tracking apparatus may determine a target box corresponding to the target object and a distractor box corresponding to a distractor of the target object from the candidate boxes based on the adjusted similarity scores.

In operation1040, the object tracking apparatus may update the distractor map based on distractor information of the current image frame according to the distractor box. The object tracking apparatus may update the distractor map by applying motion of distractor to the distractor map. The object tracking apparatus may input the distractor information of the previous image frame and the distractor information of the current image frame to the motion estimation model based on a neural network and estimate motion of a distractor from output of the motion estimation model.

The object tracking apparatus may determine a tracking state of the current image frame based on the box information and the similarity scores and set an object tracking mode to any one of a precise tracking mode for performing object tracking with the distractor map and a normal tracking mode for performing the object tracking without the distractor map based on the tracking state. When the object tracking mode is set to the precise tracking mode, the object tracking may be performed based on adjusting of the similarity scores, determining of the target box and the distractor box, and updating of the distractor map. When the object tracking mode is set to the normal tracking mode, the target box may be determined based on the box information of the candidate boxes and the similarity scores of the candidate boxes without performing the adjusting of the similarity scores, the determining of the target box and the distractor box, and the updating of the distractor map. The object tracking apparatus may set the object tracking mode to the normal tracking mode in at least any one case of non-existence of the distractor of the target object, occlusion of the target object, and detachment of the target object from a frame.

In addition, the description provided with reference toFIGS.1through9may apply to the object tracking method.

The instructions or software to control a processor or computer to implement the hardware components and perform the methods as described above, and any associated data, data files, and data structures, are recorded, stored, or fixed in or on one or more non-transitory computer-readable storage media. Examples of a non-transitory computer-readable storage medium include read-only memory (ROM), random-access programmable read only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random-access memory (RAM), magnetic RAM (MRAM), spin-transfer torque(STT)-MRAM, static random-access memory (SRAM), thyristor RAM (T-RAM), zero capacitor RAM (Z-RAM), twin transistor RAM (TTRAM), conductive bridging RAM(CBRAM), ferroelectric RAM (FeRAM), phase change RAM (PRAM), resistive RAM(RRAM), nanotube RRAM, polymer RAM (PoRAM), nano floating gate Memory(NFGM), holographic memory, molecular electronic memory device), insulator resistance change memory, dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROMs, CD-Rs, CD+Rs, CD-RWs, CD+RWs, DVD-ROMs, DVD-Rs, DVD+Rs, DVD-RWs, DVD+RWs, DVD-RAMs, BD-ROMs, BD-Rs, BD-R LTHs, BD-REs, blue-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), flash memory, a card type memory such as multimedia card micro or a card (for example, secure digital (SD) or extreme digital (XD)), magnetic tapes, floppy disks, magneto-optical data storage devices, optical data storage devices, hard disks, solid-state disks, and any other device that is configured to store the instructions or software and any associated data, data files, and data structures in a non-transitory manner and providing the instructions or software and any associated data, data files, and data structures to a processor or computer so that the processor or computer can execute the instructions. In an example, the instructions or software and any associated data, data files, and data structures are distributed over network-coupled computer systems so that the instructions and software and any associated data, data files, and data structures are stored, accessed, and executed in a distributed fashion by the one or more processors or computers.