Lingually constrained tracking of visual objects

A computer-implemented method for tracking with visual object constraints includes receiving a lingual constraint and a video. A word embedding is generated based on the lingual constraint. A set of features is extracted for one or more frames of the video. The word embedding is cross-correlated to the set of features for the one or more frames of the video. A prediction indicating whether the lingual constraint is in the one or more frames of the video is generated based on the cross-correlation.

FIELD OF INVENTION

Aspects of the present disclosure generally relate to tracking of objects in a video.

BACKGROUND

Artificial neural networks may comprise interconnected groups of artificial neurons (e.g., neuron models). The artificial neural network may be a computational device or may be represented as a method to be performed by a computational device.

Neural networks consist of operands that consume tensors and produce tensors. Neural networks can be used to solve complex problems, however, because the network size and the number of computations that may be performed to produce the solution may be voluminous, the time for the network to complete a task may be long. Furthermore, because these tasks may be performed on mobile devices, which may have limited computational power, the computational costs of deep neural networks may be problematic.

Convolutional neural networks are a type of feed-forward artificial neural network. Convolutional neural networks may include collections of neurons that each have a receptive field and that collectively tile an input space. Convolutional neural networks (CNNs), such as deep convolutional neural networks (DCNs), have numerous applications. In particular, these neural network architectures are used in various technologies, such as image recognition, pattern recognition, speech recognition, autonomous driving, and other classification tasks.

Neural networks also have numerous applications in image-based processing of videos or video streams such as object detection and tracking. Visual object tracking is the task of following a target object throughout a given video. Visual object tracking has many practical applications, including video surveillance and target-specific video summarization, where a target is monitored with respect to certain predefined constraints. Conventional tracking systems may provide motion trajectory information for an object. However, because there is an absensce of semantic information, tracking with visual object constraints is challenging.

SUMMARY

In an aspect of the present disclosure, a computer-implemented method for tracking visual objects is provided. The computer-implemented method includes receiving a lingual constraint and a video. The computer-implemented method also includes generating a word embedding based on the lingual constraint. Additionally, the computer-implemented method includes extracting a set of features for one or more frames of the video. The computer-implemented method also includes cross-correlating the word embedding and the set of features for the one or more frames of the video. Further, the computer-implemented method includes generating a prediction based on the cross-correlation.

In other aspects of the present disclosure, an apparatus for tracking visual objects is provided. The apparatus includes a memory and one or more processors coupled to the memory. The processor(s) are configured to receive a lingual constraint and a video. The processor(s) are also configured to generate a word embedding based on the lingual constraint. In addition, the processor(s) are configured to extract a set of features for one or more frames of the video. The processor(s) are also configured to cross-correlate the word embedding and the set of features for the one or more frames of the video. Further, the processor(s) are configured to generate a prediction based on the cross-correlation.

In other aspects of the present disclosure, an apparatus for tracking visual objects is provided. The apparatus includes means for receiving a lingual constraint and a video. The apparatus also includes means for generating a word embedding based on the lingual constraint. Additionally, the apparatus includes means for extracting a set of features for one or more frames of the video. The apparatus also includes means for cross-correlating the word embedding and the set of features for the one or more frames of the video. Further, the apparatus includes means for generating a prediction based on the cross-correlation.

In further aspects of the present disclosure, a non-transitory computer readable medium is provided. The computer readable medium has encoded thereon program code for tracking visual objects. The program code is executed by a processor and includes code to receive a lingual constraint and a video. The program code also includes code to generate a word embedding based on the lingual constraint. Additionally, the program code includes code to extract a set of features for one or more frames of the video. The program code also includes code to cross-correlate the word embedding and the set of features for the one or more frames of the video. Furthermore, the program code includes code to generate a prediction based on the cross-correlation.

DETAILED DESCRIPTION

The word “exemplary” is used to mean “serving as an example, instance, or illustration.” Any aspect described as “exemplary” is not necessarily to be construed as preferred or advantageous over other aspects.

Neural networks have numerous applications in image-based processing of videos or video streams such as object detection and tracking. Visual object tracking is the task of following a target object throughout a given video. Visual object tracking has many practical applications, including video surveillance and target-specific video summarization, where a target is monitored with respect to certain predefined constraints. Conventional tracking systems may provide motion trajectory information for an object. However, because there is an absence of semantic information, tracking with visual object constraints is challenging.

Accordingly, aspects of the present disclosure are directed to lingual constraints in the form of sentences in the tracking domain. The lingual constraints are incorporated within tracking. That is, additional constraints such as a natural language sentence may be imposed on the tracking process, rather than only performing similarity learning to match a target to the ground-truth frame. Unlike conventional tracking techniques, the present disclosure relieves the burden of a user having to review or watch all the images of a potentially very long track (e.g., the sequence of frames the target was tracked) to locate the target. The lingual constraint is a lingual specification that the set of frames are matched to, such that the tracking sequence may be filtered to these frames. For example, a lingual constraint such as the phrase “next to a yellow car” may be applied to a person to determine frames in a video (e.g., a video stream) in which a person is near a yellow car.

To determine whether or not the lingual constraint is satisfied, the described constraint (e.g., object) may have to be proximate or ‘close to’ the target object that is being tracked. In some aspects, the constraint may be satisfied when the target object and the constraint object are in close proximity. For example, if the lingual object constraint is a pencil, and the tracking target is a person, the constraint may be satisfied if the person is holding the pencil or otherwise near the pencil. To track (or find) the target when the constraint is satisfied, the target is located in those frames. In some aspects, the tracker may continuously track the target, even while the constraint is unsatisfied. If the constraint is satisfied, the user may also gain this information for these video frames.

FIG.1illustrates an example implementation of a system-on-a-chip (SoC)100, which may include a central processing unit (CPU)102or a multi-core CPU configured for visual object tracking with lingual constraints. Variables (e.g., neural signals and synaptic weights), system parameters associated with a computational device (e.g., neural network with weights), delays, frequency bin information, and task information may be stored in a memory block associated with a neural processing unit (NPU)108, in a memory block associated with a CPU102, in a memory block associated with a graphics processing unit (GPU)104, in a memory block associated with a digital signal processor (DSP)106, in a memory block118, or may be distributed across multiple blocks. Instructions executed at the CPU102may be loaded from a program memory associated with the CPU102or may be loaded from a memory block118.

The SoC100may also include additional processing blocks tailored to specific functions, such as a GPU104, a DSP106, a connectivity block110, which may include fifth generation (5G) connectivity, fourth generation long term evolution (4G LTE) connectivity, Wi-Fi connectivity, USB connectivity, Bluetooth connectivity, and the like, and a multimedia processor112that may, for example, detect and recognize gestures. In one implementation, the NPU108is implemented in the CPU102, DSP106, and/or GPU104. The SoC100may also include a sensor processor114, image signal processors (ISPs)116, and/or navigation module120, which may include a global positioning system.

The SoC100may be based on an ARM instruction set. In an aspect of the present disclosure, the instructions loaded into the general-purpose processor102may include code to receive a lingual constraint and a video. The general-purpose processor102may also include code to generate a word embedding based on the lingual constraint. The general-purpose processor102may further include code to extract a set of features for one or more frames of the video. The general-purpose processor102may also include code to cross-correlate the word embedding and the set of features for the one or more frames of the video. The general-purpose processor102may include code to generate a prediction based on the cross-correlation.

Deep learning architectures may perform an object recognition task by learning to represent inputs at successively higher levels of abstraction in each layer, thereby building up a useful feature representation of the input data. In this way, deep learning addresses a major bottleneck of traditional machine learning. Prior to the advent of deep learning, a machine learning approach to an object recognition problem may have relied heavily on human engineered features, perhaps in combination with a shallow classifier. A shallow classifier may be a two-class linear classifier, for example, in which a weighted sum of the feature vector components may be compared with a threshold to predict to which class the input belongs. Human engineered features may be templates or kernels tailored to a specific problem domain by engineers with domain expertise. Deep learning architectures, in contrast, may learn to represent features that are similar to what a human engineer might design, but through training. Furthermore, a deep network may learn to represent and recognize new types of features that a human might not have considered.

The connections between layers of a neural network may be fully connected or locally connected.FIG.2Aillustrates an example of a fully connected neural network202. In a fully connected neural network202, a neuron in a first layer may communicate its output to every neuron in a second layer, so that each neuron in the second layer will receive input from every neuron in the first layer.FIG.2Billustrates an example of a locally connected neural network204. In a locally connected neural network204, a neuron in a first layer may be connected to a limited number of neurons in the second layer. More generally, a locally connected layer of the locally connected neural network204may be configured so that each neuron in a layer will have the same or a similar connectivity pattern, but with connections strengths that may have different values (e.g.,210,212,214, and216). The locally connected connectivity pattern may give rise to spatially distinct receptive fields in a higher layer, because the higher layer neurons in a given region may receive inputs that are tuned through training to the properties of a restricted portion of the total input to the network.

One example of a locally connected neural network is a convolutional neural network.FIG.2Cillustrates an example of a convolutional neural network206. The convolutional neural network206may be configured such that the connection strengths associated with the inputs for each neuron in the second layer are shared (e.g.,208). Convolutional neural networks may be well suited to problems in which the spatial location of inputs is meaningful.

One type of convolutional neural network is a deep convolutional network (DCN).FIG.2Dillustrates a detailed example of a DCN200designed to recognize visual features from an image226input from an image capturing device230, such as a car-mounted camera. The DCN200of the current example may be trained to identify traffic signs and a number provided on the traffic sign. Of course, the DCN200may be trained for other tasks, such as identifying lane markings or identifying traffic lights.

The DCN200may be trained with supervised learning. During training, the DCN200may be presented with an image, such as the image226of a speed limit sign, and a forward pass may then be computed to produce an output222. The DCN200may include a feature extraction section and a classification section. Upon receiving the image226, a convolutional layer232may apply convolutional kernels (not shown) to the image226to generate a first set of feature maps218. As an example, the convolutional kernel for the convolutional layer232may be a 5×5 kernel that generates 28×28 feature maps. In the present example, because four different feature maps are generated in the first set of feature maps218, four different convolutional kernels were applied to the image226at the convolutional layer232. The convolutional kernels may also be referred to as filters or convolutional filters.

The first set of feature maps218may be subsampled by a max pooling layer (not shown) to generate a second set of feature maps220. The max pooling layer reduces the size of the first set of feature maps218. That is, a size of the second set of feature maps220, such as 14×14, is less than the size of the first set of feature maps218, such as 28×28. The reduced size provides similar information to a subsequent layer while reducing memory consumption. The second set of feature maps220may be further convolved via one or more subsequent convolutional layers (not shown) to generate one or more subsequent sets of feature maps (not shown).

In the example ofFIG.2D, the second set of feature maps220is convolved to generate a first feature vector224. Furthermore, the first feature vector224is further convolved to generate a second feature vector228. Each feature of the second feature vector228may include a number that corresponds to a possible feature of the image226, such as “sign,” “60,” and “100.” A softmax function (not shown) may convert the numbers in the second feature vector228to a probability. As such, an output222of the DCN200is a probability of the image226including one or more features.

In the present example, the probabilities in the output222for “sign” and “60” are higher than the probabilities of the others of the output222, such as “30,” “40,” “50,” “70,” “80,” “90,” and “100”. Before training, the output222produced by the DCN200is likely to be incorrect. Thus, an error may be calculated between the output222and a target output. The target output is the ground truth of the image226(e.g., “sign” and “60”). The weights of the DCN200may then be adjusted so the output222of the DCN200is more closely aligned with the target output.

In practice, the error gradient of weights may be calculated over a small number of examples, so that the calculated gradient approximates the true error gradient. This approximation method may be referred to as stochastic gradient descent. Stochastic gradient descent may be repeated until the achievable error rate of the entire system has stopped decreasing or until the error rate has reached a target level. After learning, the DCN may be presented with new images and a forward pass through the network may yield an output222that may be considered an inference or a prediction of the DCN.

The performance of deep learning architectures may increase as more labeled data points become available or as computational power increases. Modern deep neural networks are routinely trained with computing resources that are thousands of times greater than what was available to a typical researcher just fifteen years ago. New architectures and training paradigms may further boost the performance of deep learning. Rectified linear units may reduce a training issue known as vanishing gradients. New training techniques may reduce over-fitting and thus enable larger models to achieve better generalization. Encapsulation techniques may abstract data in a given receptive field and further boost overall performance.

FIG.3is a block diagram illustrating a deep convolutional network350. The deep convolutional network350may include multiple different types of layers based on connectivity and weight sharing. As shown inFIG.3, the deep convolutional network350includes the convolution blocks354A,354B. Each of the convolution blocks354A,354B may be configured with a convolution layer (CONV)356, a normalization layer (LNorm)358, and a max pooling layer (MAX POOL)360.

The convolution layers356may include one or more convolutional filters, which may be applied to the input data to generate a feature map. Although only two of the convolution blocks354A,354B are shown, the present disclosure is not so limiting, and instead, any number of the convolution blocks354A,354B may be included in the deep convolutional network350according to design preference. The normalization layer358may normalize the output of the convolution filters. For example, the normalization layer358may provide whitening or lateral inhibition. The max pooling layer360may provide down sampling aggregation over space for local invariance and dimensionality reduction.

The parallel filter banks, for example, of a deep convolutional network may be loaded on a CPU102or GPU104of an SoC100to achieve high performance and low power consumption. In alternative embodiments, the parallel filter banks may be loaded on the DSP106or an ISP116of an SoC100. In addition, the deep convolutional network350may access other processing blocks that may be present on the SoC100, such as sensor processor114and navigation module120, dedicated, respectively, to sensors and navigation.

The deep convolutional network350may also include one or more fully connected layers362(FC1 and FC2). The deep convolutional network350may further include a logistic regression (LR) layer364. Between each layer356,358,360,362,364of the deep convolutional network350are weights (not shown) that are to be updated. The output of each of the layers (e.g.,356,358,360,362,364) may serve as an input of a succeeding one of the layers (e.g.,356,358,360,362,364) in the deep convolutional network350to learn hierarchical feature representations from input data352(e.g., images, audio, video, sensor data and/or other input data) supplied at the first of the convolution blocks354A. The output of the deep convolutional network350is a classification score366for the input data352. The classification score366may be a set of probabilities, where each probability is the probability of the input data including a feature from a set of features.

FIG.4is a block diagram illustrating an exemplary software architecture400that may modularize artificial intelligence (AI) functions. Using the architecture, applications may be designed that may cause various processing blocks of a system-on-a-chip (SoC)420(for example a CPU422, a DSP424, a GPU426and/or an NPU428) to support adaptive rounding as disclosed for post-training quantization for an AI application402, according to aspects of the present disclosure.

The AI application402may be configured to call functions defined in a user space404that may, for example, provide for the detection and recognition of a scene indicative of the location in which the device currently operates. The AI application402may, for example, configure a microphone and a camera differently depending on whether the recognized scene is an office, a lecture hall, a restaurant, or an outdoor setting such as a lake. The AI application402may make a request to compiled program code associated with a library defined in an AI function application programming interface (API)406. This request may ultimately rely on the output of a deep neural network configured to provide an inference response based on video and positioning data, for example.

A run-time engine408, which may be compiled code of a runtime framework, may be further accessible to the AI application402. The AI application402may cause the run-time engine, for example, to request an inference at a particular time interval or triggered by an event detected by the user interface of the application. When caused to provide an inference response, the run-time engine may in turn send a signal to an operating system in an operating system (OS) space, such as a Linux Kernel412, running on the SoC420. The operating system, in turn, may cause a continuous relaxation of quantization to be performed on the CPU422, the DSP424, the GPU426, the NPU428, or some combination thereof. The CPU422may be accessed directly by the operating system, and other processing blocks may be accessed through a driver, such as a driver414,416, or418for, respectively, the DSP424, the GPU426, or the NPU428. In the exemplary example, the deep neural network may be configured to run on a combination of processing blocks, such as the CPU422, the DSP424, and the GPU426, or may be run on the NPU428.

The application402(e.g., an AI application) may be configured to call functions defined in a user space404that may, for example, provide for the detection and recognition of a scene indicative of the location in which the device currently operates. The application402may, for example, configure a microphone and a camera differently depending on whether the recognized scene is an office, a lecture hall, a restaurant, or an outdoor setting such as a lake. The application402may make a request to compiled program code associated with a library defined in a SceneDetect application programming interface (API)406to provide an estimate of the current scene. This request may ultimately rely on the output of a differential neural network configured to provide scene estimates based on video and positioning data, for example.

A run-time engine408, which may be compiled code of a Runtime Framework, may be further accessible to the application402. The application402may cause the run-time engine, for example, to request a scene estimate at a particular time interval or triggered by an event detected by the user interface of the application. When caused to estimate the scene, the run-time engine may in turn send a signal to an operating system410, such as a Linux Kernel412, running on the SoC420. The operating system410, in turn, may cause a computation to be performed on the CPU422, the DSP424, the GPU426, the NPU428, or some combination thereof. The CPU422may be accessed directly by the operating system, and other processing blocks may be accessed through a driver, such as a driver414-418for a DSP424, for a GPU426, or for an NPU428. In the exemplary example, the differential neural network may be configured to run on a combination of processing blocks, such as a CPU422and a GPU426, or may be run on an NPU428.

Aspects of the present disclosure are directed to tracking an object in a video using lingual constraints. A lingual constraint is a sentence or phrase that describes a certain state or environment of a target object. Lingual constraints may be provided by a user, for example, and may be in the form of a natural language sentence. A natural language sentence is a human language (e.g., English, Spanish, or French, rather than a computer programming language) as spoken or written, which may include contextual nuances. In some aspects, the lingual constraint is not a predefined entity and may be unrestricted in content. That is, the lingual constraint is not bound to any set of words or classes.

In accordance with aspects of the present disclosure, lingual constraints are incorporated into a tracking model architecture. To integrate the prediction of the constraint into the architecture, a natural language query is first processed with a word embedding model. The word embedding model may generate word embeddings, which provide a robust feature map for the lingual constraints, to reduce, or in some aspects avoid, training a language model for this purpose. In some aspects, each of the word embeddings may have 300 dimensions. The input sequence may be padded with zeros to a fixed length L, the maximum sentence length, for consistent representation for each of the word embeddings. For example, given a fixed maximum sentence size of L=20, the word embeddings produce a feature map S∈20×300where the input sentence is denoted as the word sequence (w1, . . . , wk). Given the lingual constraint processed into an embedding S∈20×300and a search image processed into a feature map by the backbone network φ, X=φ(x)∈(3×256)×31×31, the constraint may be predicted from the features of the feature map.

FIG.5is a high-level block diagram illustrating an example architecture500for lingually constrained tracking of visual objects, in accordance with aspects of the present disclosure. Referring toFIG.5, the architecture500includes a tracking component502and a constraint prediction component504. In the example ofFIG.5, the tracking component502is shown as a Siamese neural network. However, the present disclosure is not so limiting and other systems and network architectures may be used for object tracking. A Siamese neural network is a class of artificial neural network that includes two or more subnetworks that have the same parameters and weights. The subnetworks work in tandem on two different input vectors (e.g., an image of a person taken at two different angles or images of two signatures) and the outputs of each are compared to determine whether there is a match.

The example architecture500may receive as inputs, a lingual constraint (may also be referred to as a “sentence constraint”) and a video to be searched. The video may, for example, be a video stream or sequence of frames. Each frame in the sequence of frames may include an image that may be referred to as search image x. The lingual constraint, may for example, be a natural language sentence or phrase for which the video is to be searched. For example, the lingual constraint may be “a boy with a backpack,” “a bird on a yellow car,” or “a girl on a bicycle.” Thus, one task for the example architecture500is to search the video (e.g., search image xtof a frame at time t) to find frames that include images corresponding to the lingual constraint.

One challenge for the constraint prediction is the integration of the information in the lingual constraint features S and the search image features X. In other words, this challenge relates to detection and classification of the lingual constraint in the search image. The tracking component502receives as input, the xtof the target at time-step t (e.g., image of a frame of the video at time t), using the ground truth reference image zt=0to determine whether the xtincludes the reference image z. The search image xtand the reference image z are respectively processed via successive layers of convolutional filters506aand506b, which have the same parameters. The convolutional filters506agenerate a feature map Z corresponding to the reference image z and the convolutional filters506bgenerate a feature map X corresponding to the search image xt. The feature maps Z and X output via the convolutional filters506aand506b, respectively, are supplied to a region proposal network (RPN)508. The RPN508processes the feature maps Z and X to generate a set of bounding box proposals510of different sizes based on the feature maps and a corresponding set of classifications512. The set of classifications512may indicate whether the corresponding bounding box proposals510includes a match of the search image xtand the reference image z. In other words, the set of classifications512may indicate whether an object shown in reference image z is detected in the search image xt(e.g., a frame of the video).

On the other hand, the constraint prediction component504may receive the lingual constraint as an input. The lingual constraint may be supplied to an embedding block514. The embedding block514processes the lingual constraint to generate a word embedding. The embedding block514may include a neural network that learns word associations from a corpus of text. The embedding block may represent each distinct word with a list of vectors. Using a cosine similarity, for example, a semantic similarity between words may be indicated. The word embedding may, for example, be a feature map S. The word embedding is supplied to a constraint prediction block516along with the feature map output of convolutional filters506bcorresponding to the search image xt. In turn, the constraint prediction block516may generate a constraint prediction ŷ via an activation layer518. The constraint prediction ŷ indicates whether the search image xtmatches the lingual constraint.

In some aspects, the constraint prediction ŷimay be optimized using a binary cross-entropy loss:
Li(ŷi,yi)=−(yilog(ŷi)+(1−yi)log(1−ŷi)).  (1)

FIG.6is a diagram illustrating an example constraint prediction block516for tracking with lingual constraints, in accordance with aspects of the present disclosure. As shown inFIG.6, the constraint prediction block516may include a dynamic filter generation (DFG) block602and an attention block604. The DFG block602generates word embeddings attended by or included in the search frame x. Having generated the word embeddings, the DFG block602may generate dynamic convolutional filters from the lingual constraint (e.g., sentence). The dynamic convolutional filters may enable the generation of filters specific to the more important words in the lingual constraint, thus producing activation specific to the more important words present in the lingual constraint. In turn, the constraint prediction block516produces a cross-correlation between the dynamic filters f and features of the search frame x to generate the constraint prediction ŷi.

Rather than using a long short term memory (LSTM) for processing in conventional approaches, the word embedding may be processed via a network that is fully convolutional (e.g., deep convolutional network350). For example, as shown inFIG.6, the word embedding may be fed through a convolutional neural network (CNN)606. The CNN606may receive the word embedding (e.g., feature map S) as an input. The CNN606may include a one-dimensional convolution layer622to process the word embedding. In some aspects, padding (e.g., adding zeros) may be implemented to maintain the dimensions of the word embedding. The features are max pooled via the max pool layer624. A rectifier linear unit (ReLU) activation function is applied via ReLU626to generate a feature matrix H. In one example, a word embedding S∈20×300may be transformed to a feature matrix H=CNN(S) E10×150, for example.

The attention block604integrates information from the search frame (search image xt) into the word embedding S. In some aspects, the attention block604may be implemented as a multilayer perceptron (MLP). A MLP is a feedforward network that uses a mathematical function to map a set of inputs to a set of outputs. The attention block604generates attention weights for words in the lingual constraint that are likely (e.g., having the greatest probability) to be the most important. The attention block604generates the attention weights based on the search frame and each of the word embeddings themselves. One purpose for doing so is to attend or place emphasis on the constraint based on the words in the constraint that are also visible in the search frame. That is, the attention block604draws the focus to the more important words in the sentence (e.g., the constraint), which are represented by a word embedding. To do so, the attention block604may incorporate the search frame (search image xt) and the sentence (e.g., constraint) itself, where any matches between the words in the sentence (constraint) and the search frame should get attention by the attention block604.

The attention block604receives each word embedding and features of the search frame x as inputs. The word embedding S and the features of the search frame are respectively processed via linear layers632and634. Linear layers are layers that may learn a constant such as an average rate of correlation between an output and an input. In the example of theFIG.6, linear layers (e.g.,612,632,634,642) may modify the feature dimensions of its input feature. The bias636may represent a vector initialized to all zeros that may be tuned by the backpropagation mechanism. The bias636may be added such that the output may be offset by a scalar value. Accordingly, the word embedding S and the features of the search frame X may be linearized via linear layers632,634and combined via summing node638. The output of summing node638is supplied to ReLU640to produce the set of attention weights which based on the word embedding S and the features of the search frame X. The dimensions of the attention weights may be modified via the linear layer642. The attention weights may be normalized via softmax layer644and output to multiplier node648. A matrix multiplication operation is performed multiplying the feature matrix H by the attention weights to produce the feature vector h, where the resulting feature vector represents interpolated words as hi∈1×150for instance.

The DFG block602receives a feature vector hirepresenting each word embedding and generates dynamic (convolution) filters f from the sentence. The dynamic convolution filters enable the model to create filters specific to the provided lingual constraint. The DFG block602thus generates activations specific to the attended (e.g., emphasized or more important) words in the constraint. In some aspects, the DFG block602may include a linear layer and bias652with a tanh activation function654that produces the convolutional filters:
f=tanh(Wf{tilde over (v)}+bf),.  (2)
where, for example, f∈768×1×1, bfis a bias term and Wfis the attention weights and {tilde over (v)} is the feature vector representing the word embedding.

After computing the dynamic filters f, the dynamic filters f are convolved with the visual features of the search frame x, using a depth-wise cross-correlation layer610. The depth-wise cross-correlation layer610reduces the number of parameters compared to a normal cross-correlation. The depth-wise cross-correlation layer610produces an activation map, A=X*f, where * denotes a depth-wise cross correlation, convolving the search features and the dynamic filters. In some aspects, the activation map A may be supplied to a linear layer612, which flattens the activation map A to a single probability and project a scalar prediction ŷ.

FIG.7is a diagram illustrating another example constraint prediction block516shown inFIG.5for tracking with lingual constraints, in accordance with aspects of the present disclosure. As shown inFIG.7, the constraint prediction block516is similar to the example shown inFIG.6, but includes self-attention networks that operate as a deep co-attention encoder-decoder706. Features (e.g., each word embedding) S∈20×300of the lingual constraint are received along with the feature map of the search image X∈(3×256)×31×31. Although particular dimensions of the features and other aspects are provided, such dimensions are merely an example, provided for ease of understanding. As shown in the example ofFIG.7, the constraint prediction block516may include a fully connected layer (FC)702and a convolutional neural network (CNN)704in addition to the deep co-attention encoder-decoder706(may be referred to as co-attention network706). The features (e.g., each word embedding) S of the lingual constraint and the feature map search image features X are processed such that the dimensions are matched. The features (e.g., word embedding) S of the lingual constraint are supplied to the FC layer702. In some examples, the FC layer702may be a linear layer such as a fully connected rectified linear unit (ReLU)). The FC layer702includes a set of input nodes (not shown) connected in an all-to-all fashion to expand the features (e.g., each word embedding) S of the lingual constraint to a feature dimension d, which corresponds to the channel dimension of the image features X.

A CNN704receives the image features X. In some aspects, the CNN704may be configured similar to CNN606shown inFIG.6. The CNN704may process the image features X to reshape the image feature data to produce reshaped image features Xr∈(256×3)×75for instance. In some aspects, the width and height dimension of the image may be concatenated. Additionally, in some aspects, the number of image features X may be further reduced. For instance, the locations of the image features X may be reduced. In doing so, the computational complexity and size of the constraint prediction block516may be reduced. Accordingly, the features (e.g., each word embedding) S of the lingual constraint and image features X may be reshaped and matched.

The reshaped image features Xr, and the features (e.g., each word embedding) S of the lingual constraint, are supplied to the co-attention network706. The co-attention network706encodes the features (e.g., each word embedding) S of the lingual constraint and self-attends to image features X. Then, the self-attention network706correlates or co-attends the image features X based upon the features (e.g., each word embedding) S of the lingual constraint. Each self-attention (SA) block720a-720ztakes in the word embedding and attends or ‘puts emphasis’ on words of the embeddings that are the most important. For example, the SA blocks (e.g.,720a-720z) may attend characteristic words such as colors or objects, while ‘de-attending’ or de-emphasizing articles that have less semantic meaning.

The SA-gated attention (GA) pairs (e.g.722a,724a) in each block first self-attend the search image features X in a similar fashion, but now visually. After each SA block (e.g.,722a-722z), the GA block (e.g.,724a-724z) attends the image features X based on the encoded word embedding from the output of the SA block720zof block L. Accordingly, the deep co-attention encoder-decoder706may attend the visual objects that are also present in the word embedding S. The outputs of the encoder-decoder blocks are of the same shape and are the attended (encoded) word embedding and the attended visual features.

The image features and features representing the lingual constraint are fused to one representation via attention reduction blocks708a,708band classified. The attentional reduction blocks708a,708bmay reduce the number of highly dimensional features from the fused representation and may reduce a loss of important information. After the attentional reduction708,708b, the features are added together and projected to a single scalar (e.g., the constraint prediction ŷi) using a linear layer712. Once the constraint prediction ŷiis obtained, the constraint prediction block516may be optimized with the binary cross-entropy loss.

In some aspects, the constraint prediction block516may further include a pyramid pooling module (PPM) (not shown) to further improve the localization of the lingual constraint. The PPM may add global features at every location in the image, computed at different scales s. The global features provide additional channels, which may function as a prior for the image. Given the feature map for the image X, adaptive average pooling may be applied to reduce the feature maps to a scale of s×s, where s∈{1,2,3,6}, after which these features may be supplied to a convolution layer, and then concatenated onto the output features. In one example, a PPM layer may be included between the search image features X and the CNN704to further improve the feature embedding of the search image features X.

FIG.8is a flow diagram illustrating a computer-implemented method800for tracking a visual object with lingual constraints via an artificial neural network, in accordance with aspects of the present disclosure. As shown inFIG.8, at block802, a lingual constraint and a video are received. As described with reference toFIG.5, The example architecture500may receive as inputs, a lingual constraint (may also be referred to as a “sentence constraint”) and a video to be searched. The video may, for example, be a video stream or sequence of images and is referred to as search image xt. The lingual constraint, may for example, be a natural language sentence or phrase for which the video is to be searched. For example, the lingual constraint may be “a boy with a backpack,” “a bird on a yellow car,” or “a girl on a bicycle.”

At block804, a word embedding is generated based on the lingual constraint. For instance, as described with reference toFIG.5, the lingual constraint may be supplied to an embedding block514. The embedding block514processes the lingual constraint to generate a word embedding. The word embedding may, for example, be a feature map S.

At block806, a set of features is extracted for one or more frames of the video. As described with reference toFIG.5, the tracking component502receives as input, the search image xtof the target at time-step t (e.g., frame t of the video), using the ground truth reference image zt=0to determine whether the search image xtincludes the reference image z. The search image xtand the reference image z are respectively processed via successive layers of convolutional filters506aand506b, which have the same parameters. The convolutional filters506agenerate a feature map Z corresponding to the reference image z and the convolutional filters506bgenerate a feature map X corresponding to the search image xt.

At block808, the word embedding and the set of features for the one or more frames of the video are cross-correlated. For example, as described with reference toFIG.5, The word embedding is supplied to a constraint prediction block516along with the feature map output of convolutional filters506bcorresponding to the search image xt.

At block810, a prediction is generated based on the cross-correlation. For instance, as described with reference toFIG.5, the constraint prediction block516may generate a constraint prediction ŷ via an activation layer518. The constraint prediction ŷ indicates whether the search image xtmatches the lingual constraint.

Implementation examples are provided in the following numbered clauses:

1. A computer-implemented method, comprising:receiving a lingual constraint and a video;generating a word embedding based on the lingual constraint;extracting a set of features for one or more frames of the video;cross-correlating the word embedding and the set of features for the one or more frames of the video; andgenerating a prediction based on the cross-correlation.

2. The computer-implemented method of clause 1, in which one or more words of the lingual constraint are represented as vectors, the word embedding being determined based on a semantic similarity between the vectors.

3. The computer-implemented method of clause 1 or 2, in which the prediction provides an indication of whether the word embedding matches the one or more frames of the video.

4. The computer-implemented method of any of clauses 1-3, in which information from the one or more frames is integrated into the word embedding.

5. The computer-implemented method of any of clauses 1-4, in which the word embedding is attended based on words in the lingual constraint that are visible in the one or more frames of the video.

6. The computer-implemented method of any of clauses 1-5, in which the cross-correlation comprises a depth-wise cross-correlation.

7. The computer-implemented method of any of clauses 1-6, in which a set of convolutional filters extract the set of features.

8. The computer-implemented method of any of clauses 1-8, further comprising generating dynamic filters to produce activations specific to attended words in the lingual constraint.

9. The computer-implemented method of clause 1, in which the prediction provides an indication of whether the lingual constraint and the set of features for the one or more frames of the video are matched.

10. An apparatus, comprising:a memory; andat least one processor coupled to the memory, the at least one processor being configured:to receive a lingual constraint and a video;to generate a word embedding based on the lingual constraint;to extract a set of features for one or more frames of the video;to cross-correlate the word embedding and the set of features for the one or more frames of the video; andto generate a prediction based on the cross-correlation.

11. The apparatus of clause 10, in which the at least one processor is further configured to represent one or more words of the lingual constraint as vectors, the word embedding being determined based on a semantic similarity between the vectors.

12. The apparatus of clause 10 or 11, in which the prediction provides an indication of whether the word embedding matches the one or more frames of the video.

13. The apparatus of any of clauses 10-12, in which the at least one processor is further configured to integrate information from the one or more frames into the word embedding.

14. The apparatus of any of clauses 10-13, in which the word embedding is attended based on words in the lingual constraint that are visible in the one or more frames of the video.

15. The apparatus of any of clauses 10-14, in which the at least one processor is further configured to perform a depth-wise cross-correlation on the word embedding and the set of features for the one or more frames of the video.

16. The apparatus of any of clauses 10-15, in which the at least one processor is further configured to extract the set of features via a set of convolutional filters.

17. The apparatus of any of clauses 10-16, in which the at least one processor is further configured to generate dynamic filters to produce activations specific to attended words in the lingual constraint.

18. The apparatus of any of clauses 10-17, in which the prediction provides an indication of whether the lingual constraint and the set of features for the one or more frames of the video are matched.

19. An apparatus, comprising:means for receiving a lingual constraint and a video;means for generating a word embedding based on the lingual constraint;means for extracting a set of features for one or more frames of the video;means for cross-correlating the word embedding and the set of features for the one or more frames of the video; andmeans for generating a prediction based on the cross-correlation.

20. The apparatus of clause 19, further comprising means for representing one or more words of the lingual constraint as vectors, the word embedding being determined based on a semantic similarity between the vectors.

21. The apparatus of clause 19 or 20, in which the prediction provides an indication of whether the word embedding matches the one or more frames of the video.

22. The apparatus of any of clauses 19-21, further comprising means for integrating information from the one or more frames into the word embedding.

23. The apparatus of any of clauses 19-22, in which the word embedding is attended based on words in the lingual constraint that are visible in the one or more frames of the video.

24. The apparatus of any of clauses 19-23, further comprising means for performing a depth-wise cross-correlation on the word embedding and the set of features for the one or more frames of the video.

25. A non-transitory computer readable medium having encoded thereon, program code, the program code being executed by a processor and comprising:program code to receive a lingual constraint and a video;program code to generate a word embedding based on the lingual constraint;program code to extract a set of features for one or more frames of the video;program code to cross-correlate the word embedding and the set of features for the one or more frames of the video; andprogram code to generate a prediction based on the cross-correlation.

26. The non-transitory computer readable medium of clause 25, further comprising program code to represent one or more words of the lingual constraint as vectors, the word embedding being determined based on a semantic similarity between the vectors.

27. The non-transitory computer readable medium of clause 25 or 26, in which the prediction provides an indication of whether the word embedding matches the one or more frames of the video.

28. The non-transitory computer readable medium of any of clauses 25-27, further comprising program code to integrate information from the one or more frames into the word embedding.

29. The non-transitory computer readable medium of any of clauses 25-28, in which the word embedding is attended based on words in the lingual constraint that are visible in the one or more frames of the video.

30. The non-transitory computer readable medium of any of clauses 25-29, further comprising program code to perform a depth-wise cross-correlation on the word embedding and the set of features for the one or more frames of the video.

In one aspect, the receiving means, the receiving means, generating means, extracting means, cross-correlating means, and/or the predicting means may be the CPU102, program memory associated with the CPU102, the dedicated memory block118, fully connected layers362, NPU428, and/or the routing connection processing unit216configured to perform the functions recited. In another configuration, the aforementioned means may be any module or any apparatus configured to perform the functions recited by the aforementioned means.