Spatio-temporal interaction network for learning object interactions

Systems and methods for improving video understanding tasks based on higher-order object interactions (HOIs) between object features are provided. A plurality of frames of a video are obtained. A coarse-grained feature representation is generated by generating an image feature for each of for each of a plurality of timesteps respectively corresponding to each of the frames and performing attention based on the image features. A fine-grained feature representation is generated by generating an object feature for each of the plurality of timesteps and generating the HOIs between the object features. The coarse-grained and the fine-grained feature representations are concatenated to generate a concatenated feature representation.

BACKGROUND

Technical Field

The present invention relates to machine learning, and more particularly to learning of object interactions.

Description of the Related Art

Video understanding tasks, such as action recognition, video captioning, video question-answering, etc., can be useful for various applications, such as surveillance, video retrieval, human behavior understanding, etc. Actions or activity displayed within a video can involve complex interactions across several inter-related objects in a scene. Learning interactions across multiple objects from a large number of frames of a video for action recognition (e.g., human action recognition) can be computationally infeasible and performance can suffer due to a large combinatorial space that has to be modeled.

SUMMARY

According to an aspect of the present principles, a computer-implemented method is provided for improving video understanding tasks based on higher-order object interactions (HOIs) between object features. The method includes obtaining a plurality of frames of a video, generating a coarse-grained feature representation and generating a fine-grained feature representation. Generating the coarse-grained feature representation includes generating an image feature for each of a plurality of timesteps respectively corresponding to each of the frames and performing attention based on the image features. Generating the fine-grained feature representation includes generating an object feature for each of the plurality of timesteps and generating the HOIs between the object features. The method further includes concatenating the coarse-grained and the fine-grained feature representations to generate a concatenated feature representation.

According to another aspect of the present principles, a computer program product having program instructions embodied therewith is provided. The program instructions are executable by a computer to cause the computer to perform a method for improving video understanding tasks based on higher-order object interactions (HOIs) between object features. The method includes obtaining a plurality of frames of a video, generating a coarse-grained feature representation and generating a fine-grained feature representation. Generating the coarse-grained feature representation includes generating an image feature for a plurality of timesteps respectively corresponding to each of the frames and performing attention based on the image features. Generating the fine-grained feature representation includes generating an object feature for each of the plurality of timesteps and generating the HOIs between the object features. The method further includes concatenating the coarse-grained and the fine-grained feature representations to generate a concatenated feature representation.

According to another aspect of the present principles, a system is provided for improving video understanding tasks based on higher-order object interactions (HOIs) between object features. The system includes at least one processor operatively coupled to a memory. The at least one processor is configured to obtain a plurality of frames of a video, generate a coarse-grained feature representation by generating an image feature for a plurality of timesteps respectively corresponding to each of the frames and performing attention based on the image features, generate a fine-grained feature representation by generating an object feature for each of the plurality of timesteps and generating the HOIs between the object features, and concatenate the coarse-grained and the fine-grained feature representations to generate a concatenated feature representation.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

As used herein, the term “object” is defined to be a certain region within an image (e.g., scene or frame of a video) that can be used to determine visual relationships and/or interactions.

Learning relationships between objects in images, such as scenes or frames of videos, can be important for many applications, such as in surveillance, video retrieval, human-machine interaction, video captioning, etc. In the simplest setting, an interaction between objects in a scene can be represented via a summation operation of individual object information. One exemplary method is to add the learnable representations and project these representation into a high-dimensional space, where the object interactions can be exploited by summing up the object representations. Another exemplary approach is to pair all possible object candidates (or subject-object pairs).

When approaches rely on single object representation or pairwise/triplet object relationships, higher-order interactions (HOIs) between objects cannot be learned. HOIs look at an overall scene to get a context of an object in the scene, as well as look at how the scene changes

While approaches relying on single object representation or pairwise/triplet object relationships may be suitable for images, videos often contain hundreds or thousands of frames. Thus, learning such object relationships across all objects and all timesteps of a video can be very expensive. Also, instead of using relationships between the objects in their predictions, predictions can be made based on a detection of related objects or background information in the scenes. As a result, such approaches, apart from being non-interpretable, may not be suitable for tasks that require deeper video understanding such as action recognition, video caption generation, scene graph generation, visual question answering, etc. Accordingly, approaches relying on single object representation or pairwise/triplet object relationships are computationally intensive and can be infeasible for very large datasets or for real-time applications that require low feature extraction costs.

To overcome at least the above-noted issues, the embodiments described herein provide for an interaction network that can be used to detect and learn HOIs between objects depicted in videos. The goal is for the interaction network to analyze a video, understand objects in the video, and learn the HOIs between the objects to perform an operation, such as predict an action being depicted in the video (e.g., a person playing basketball), video caption generation, etc.

To do so, the interaction network in accordance with the embodiments described herein can generate two types of representations, referred to herein as a coarse-grained representation and a fine-grained representation. The coarse-grained representation can learn the overall scene, and the fine-grained representation can detect the objects and generate the HOIs.

To generate the coarse-grained representation, the interaction network can illustratively use Scale Dot Product (SDP) attention over image features generated by a convolutional neural network (CNN).

To generate the fine-grained representation, the interaction network can illustratively include a recurrent HOI component that can dynamically select groups of objects with inter-relationships via an attention mechanism, and encode the attended object features. That is, HOIs between arbitrary subgroups of objects can be learned, in which inter-object relationships in one group are detected, and objects with significant relations (e.g., those that serve to improve action recognition, captioning, etc.) can be attentively selected. The combinations of these objects can then be concatenated to model HOIs using, e.g., group to group or triplet groups of objects.

The coarse-grained and fine-grained representations can then be concatenated to generate a concatenated representation for an overall prediction. Accordingly, frame-level information can be discovered by using coarse-to-fine frame-level image features and object features.

The embodiments described herein can combine object features from object detectors that leverage neural networks, such as CNNs and/or fully-convolutional networks (FCNs) (e.g., R-FCN object detectors), that are gated over the image features. From this, the relationships between the objects can be learned by constructing object pairs, and temporal behavior can be learned using a temporal pooling model, such as long short-term memory (LSTM). The features can be extracted by processing a video at a frame rate of, e.g., about 1 frame per second (FPS).

The embodiments described herein can provide for improved accuracy and lower computational cost, as compared to conventional approaches. For example, modeling object interactions in accordance with the embodiments described herein can save over three times the amount of computation as compared to conventional pairwise relationship approaches.

Referring now in detail to the figures in which like numerals represent the same or similar elements,FIG. 1illustrates an overview of an interaction network100in accordance with one embodiment of the present principles.

An input plurality of scenes or frames110of a video are provided. The frames110can include T frames associated with T timesteps. The frames110can be processed by a coarse-grained analysis portion of the network100to generate a coarse-grained representation associated with overall image context and a fine-grained analysis portion of the network100to generate a fine-grained object representation associated with HOIs in the spatiotemporal domain for general video understanding tasks. As will be described in further detail below, the coarse-grained representations can be generated via attention, such as, e.g., SDP attention, and the fine-grained representations can be generated via a recurrent HOI component. The recurrent HOI component can select groups of objects with inter-relationships via an attention mechanism, and encode attended features of the objects with LSTM. The coarse-grained and fine-grained object representations generated by the coarse-grained analysis and fine-grained analysis portions, respectively, can then be concatenated to form a final prediction.

The coarse-grained analysis portion of the network100will now be described in further detail.

The coarse-grained analysis portion of the network100includes a CNN component120, a multilayer perceptron (MLP) component140and an attention component150.

The CNN component120receives the frames110to generate a representation (or a matrix) that describes what is important in a scene from a visual perspective. For example, as shown, the CNN component120generates a sequence of image features130corresponding to respective ones of the images, including vc,1through vc,T. Each image feature of the sequence130can include a feature vector encoded from a corresponding one of the frames110. For example, each feature vector can have a dimension of, e.g., m=2048.

The sequence of image features130is fed into the MLP component140. An MLP having a parameter ϕ, referred to herein as gϕ, is a type of feedforward neural network that includes at least three layers of nodes (e.g., an input layer, an output layer and at least one hidden layer).

Illustratively, the MLP component140can include two sets of fully-connected layers each with batch normalization and a rectified linear unit (ReLU) employing a rectifier. The MLP component140can maintain the same dimension of the input feature vectors130(e.g., 2048).

Using LSTM to aggregate a sequence of image representations can result in limited performance since image representations can be similar to each other and, thus, lack temporal variances. Therefore, the outputs of the MLP component140are received by the attention component150to attend to key image-level representations to summarize the entire video sequence. Generally, attention is performed by generating a linear combination of vectors using attention weights calculated using an attention function. The attention weights can be normalized so that they have a sum of 1 (e.g., applying a SoftMax function).

An example of an SDP attention function that can be utilized in accordance with the embodiments described herein is provided follows:

αc=softmax(XcT⁢Xcdϕ),Xc=gϕ⁡(Vc)(1)vc=αc⁢XcT_(2)
where Vcis a set including the sequence of image features130(e.g., Vc={vc,1, vc,2, . . . , vc,T}, where vc,tϵmis an image feature generated by the CNN component120at time t and 1≤t≤T for a given video length T), gϕis an MLP having a parameter ϕ, dϕis the dimension of the last fully-connected layer of gϕ, Xcϵ6ϕ×Tis the projected image feature matrix, √{square root over (dϕ)} is a scaling factor, and αcϵT×Tis an attention weight. Accordingly, equation (1) calculates the attention weight based on a projected image feature matrix and a scaling factor, and equation (2) generates attended context information (vc) by mean-pooling the weighted image representations. The attention component150outputs the attended context information152as a coarse-grained feature representation.

As described above, conventional pairwise object interactions only consider how each object interacts with another object. The embodiments described herein can model inter-relationships between arbitrary subgroups of objects using a fine-grained analysis portion of the network100, the members of which are determined by a learned attention mechanism.

The fine-grained analysis portion of the network100can use object detectors trained to detect regions of interest (ROIs) from the frames110, and extract information about objects depicted in each of the frames110from the ROIs. For example, the network100can use recurrent CNN (R-CNN) for object detection, such as e.g., faster R-CNN. In one embodiment, and as shown, the fine-grained analysis portion of the network100includes a region proposal network (RPN) component160and a recurrent HOI component180.

The RPN component160generates ROI proposals that can be used to detect objects within an image. In one embodiment, the RPN component160includes a fully convolutional network (FCN) to generate the ROI proposals. The ROI proposals generated by the RPN component160can then be used to obtain sets of object features from each of the frames110. For example, sets of object features including O1170-1, O2170-2and OT170-T corresponding to respective ones of the T frames110can be obtained by the RPN160. Set of object features OT170-T is illustratively shown having n object features at time T, including o1,T172-1through on,T172-n. Each object feature can include a feature vector encoded from a corresponding one of the frames110.

Each set of object features O1170-1through OT170-T is fed into the recurrent HOI component180. The recurrent HOI component180can select groups of objects with inter-relationships via an attention mechanism, and encode the attended object features with LSTM. The recurrent HOI component180can include one or more MLPs associated with one or more respective attentive selection components. Illustratively, the recurrent HOI component180can include n MLPs and corresponding attentive selection components, where gθkrepresents the k-th MLP. Illustratively, as will be described in further detail below with reference toFIG. 2, the MLP component can include 3 MLPs gϕ1, gϕ2and gϕ3and corresponding attentive selection components.

Further details regarding the recurrent HOI component180will now be described with reference toFIG. 2.

Referring now toFIG. 2, an exemplary recurrent HOI module200, such as recurrent HOI module128inFIG. 1, is illustratively depicted in accordance with one embodiment of the present principles.

Learnable parameters for the incoming object features are introduced via MLP projection, since the object features are pre-trained from another domain and may not necessarily present relevant interactions. For example, the object features can be fed into K MLPs to generate projected object features. The k-th MLP gθkhas a corresponding parameter θk, which is a learnable synaptic weight shared across all objects and through all timesteps. Illustratively, gθkcan include three sets of batch normalization layers, fully-connected layers, and ReLUs.

In the illustrative embodiment ofFIG. 2, object features o1,t202-1through on,t202-nincluded in a set of object features Otcorresponding to a given timestep t are shown being input into MLP (gθ1)210-1having parameter θ1, MLP (g742)210-2having parameter θ2and MLP (gθ3)210-3having parameter θ3, thereby generating projected object features respectively corresponding to the object features202-1through202-n. Accordingly, K=3 in this illustrative embodiment. However, any number of MLPs can be implemented in accordance with the embodiments described herein.

Attention weights can be computed using inputs from current (projected) object features, overall image visual representation, and previously discovered object interactions. For example, the projected object features can be combined with the corresponding (encoded) image feature, shown as vc,t204, and any previous object interaction(s) to generate k sets of weights to select k groups of objects via attentive selection components, including attentive selection components220-1through220-3. Objects with inter-relationships are selected from an attention weight, which generates a probability distribution over all object candidates. For example, the k-th attention weight αkcorresponding to gθkcan be calculated by the k-th attentive selection component as follows:
αk=Attention(gθk(Ot),νc,t,ht-1)
where gϕkis the k-th MLP having a parameter ϕk, Otis the set of objects corresponding to timestep t (e.g., Ot={o1,t, o2,t, . . . , oN,t}, where on,tϵmis the nth object feature representation at timestep t), vc,tis the (encoded) image feature corresponding to timestep t, and ht-1is the previous output of LSTM (memory) cell250, which represents the previous object interaction representation. Formally, given an input sequence, the LSTM cell250computes a hidden vector sequence h=(h1, h2, . . . , hT) for the T timesteps. Accordingly, an attention weight for a given timestep can be calculated based on a previously discovered object interaction corresponding to the previous timestep, the image feature corresponding to the current timestep, and the set of objects corresponding to the current timestep.

As shown, for a given timestep t, attentive selection component220-1outputs a corresponding attended object feature at time t (νo,t1)230-1, attentive selection component220-2outputs a corresponding attended object feature at time t (νo,t2)230-2, and attentive selection component220-3outputs a corresponding attended object feature at time t (νo,t3)230-3.

Possible attention mechanisms for implementing the attentive selection components220-1through220-3include dot product attention and α-attention. Dot product attention can be used to model HOIs, which models inter-object relationships in each group of selected objects, since the attention weights computed for each object are the combination of all objects. Unlike, dot product attention, the α-attention mechanism does not consider inter-relationships when selecting the objects. The α-attention mechanism can be used as a baseline to show how consideration the inter-relationships of objects (e.g., using dot product attention) can further improve the accuracy when ROIs are selected separately. Further details regarding implementing dot product attention and α-attention will now be provided with reference toFIG. 3.

Referring toFIG. 3, block/flow diagrams300are provided illustrating a dot product attention module310for implementing dot product attention and an α-attention module330for implementing α-attention.

As shown, the dot product attention module310can include a first matrix multiplication (MatMul) layer312, a scaling layer (Scale)314, a masking layer316, SoftMax layer318, a second MatMul layer320, and a mean-pooling layer (Mean-Pool)322.

Current image feature vc,tand previous object interaction representation ht-1can be projected to introduce learnable weights. The projected vc,tand ht-1are then repeated and expanded N times (e.g., the number of objects in Ot). This information is combined with projected objects via matrix addition and used as input to the dot product attention. The input to the MatMul layer312can be defined as, e.g.:
Xk=repeat(Wkkht-1+Whkνc,t)+gθk(Ot)
and the k-th attention weight, αk, can be defined using the dot product attention module310as, e.g.:

Wck∈ℝdθ×dvc,t
are learned weights for vc,tand ht-1, dθis the dimension of the last fully-connected layer of gθ, Xkϵdθ×Nis the input to the k-th attentive selection component, and √{square root over (dϕ)} is a scaling factor. The bias term is omitted in this equation for simplicity, although a bias term can be introduced in other embodiments. An attended object feature at time t, νo,tk, can then calculated for dot product attention as, e.g.:

vo,tk=αk⁡(gθk⁡(Ot))T_
where νo,tkis the k-th attend object feature that encodes the k-th object inter-relationships of a video frame at timestep t. Accordingly, the attended object feature at timestep t can be calculated via dot product attention as a mean-pooling on weighted objects. An additional input to the MatMul layer320can be, e.g., gϕk(Ot).

The α-attention module330can use the same input as the dot product attention module310(e.g., Xk), but, as shown, the input is sent to a hyperbolic tangent function (tanh) layer332. As further shown, the α-attention module330further includes a fully connected (FC) layer334, masking layer336, a SoftMax layer338, and a MatMul layer340.

The k-th attention weight, αk, can be defined using the α-attention module330as, e.g.:
αk=softmax(wkTtanh(Xk))
where wkϵdθis a learned weight. The k-th attended object feature at time t, νo,tk, can then defined using the α-attention module330as, e.g.:

An additional input to the MatMul layer340can be the same as in the additional input to the MatMul layer320(e.g., gϕk(Ot)).

Referring back toFIG. 2, the attended object features230-1through230-3are concatenated to generate concatenated representation240, and the concatenated representation240is used as an input into the LSTM cell250to determine the HOI representation at timestep t (voi,t). That is,
νoi,t=LSTMCell(νo,t1∥νo,t2∥ . . . ∥νo,tK)
where ∥ denotes concatenation. The last hidden state of the LSTM cell250(e.g., hT=voi,t) is selected as the representation of overall object interactions for the entire video sequence.

The dimension of the hidden layers in gθkcan be adjusted based on the number of MLPs (K). For example, the dimension of the hidden layers in gθkcan have an inverse relationship with K (e.g., the dimension of the hidden layers in gθkcan be reduced as K increases). In this way, the input to the LSTM cell250can have the same or similar feature dimension. In one embodiment, the hidden dimension of the LSTM cell250can be set to the dimension of the image feature vectors (e.g., 2048).

Note that by concatenating selected inter-object relationships into a single HOI representation, each selective attention component tends to select different groups of inter-relationships, since concatenating duplicate inter-relationships does not provide extra information and will be penalized.

Referring back toFIG. 1, the recurrent HOI component180outputs the object interactions discovered through the video sequences (voi,T)182as a fine-grained feature representation. The outputs vc152and voi,T182can then be concatenated to generate a concatenated feature representation190. In one embodiment, prior to generating the concatenated feature representation190, the feature vector can be separately re-normalized with a batch normalization layer. The concatenated feature representation190can then be used as input to the last fully-connected layer. Then, the model can be trained to make a final prediction p(y). For example, p(y) can be calculated as follows:
p(y)=softmax(Wp(νc∥νoi,T)+bp)
where

Wp⁢ϵ⁢⁢ℝdy×(dvc+dvoi,T)
are learned weights, bpϵdyare learned biases, and ∥ denotes concatenation. Accordingly, a combination of the coarse-grained feature representation and the fine-grained feature representation can be used to make a final prediction (e.g., regarding action recognition).

As mentioned, the interaction network described herein can be applied to perform (human) action recognition regarding scenes of a video. The interaction network described herein selectively attends to various regions with relationships and interactions across time. For example, a video frame or scene can have multiple ROIs corresponding to respective bounding box colors. ROIs with the same color can indicate the existence of inter-relationships, and interactions between groups of ROIs can be modeled across different colors. The color of each bounding box can be weighted by the attention generated in accordance with the embodiment described herein. Thus, if some ROIs are not important, they can have smaller weights and/or may not be shown on the corresponding image. The same weights can then be used to set the transparent ratio for each ROI. Accordingly, there is a direct relationship between ROI brightness and ROI importance.

The interaction network described herein can focus on the details of a scene and neglect visual content that may be irrelevant (e.g., background information). Furthermore, the interaction network described herein tends to explore an entire frame early in the video (e.g., the attentions tend to be distributed to the ROIs that cover a large portion of the frame), and the attentions become more focused after this exploration stage.

The interaction network described herein can be extended to video captioning applications. The goal in providing fine-grained information for video captioning is that, for each prediction of a word, the model is aware of the past generated word, previous output, and the summary of the video content. At each word generation, the model has the ability to selectively attend to various parts of the video content spatially and temporally, as well as to the detected object interactions.

As will be described in further detail with reference toFIG. 4, video captioning can be employed in accordance with the embodiments described herein using a two-layered LSTM integrated with the coarse-grained and fine-grained representations described above. The two LSTM layers are referred to herein as “Attention LSTM” and “Language LSTM.”

Referring toFIG. 4, a system/method400for employing video captioning is illustratively depicted in accordance with one embodiment of the present principles. In contrast to prior systems/methods, which applied attention directly over all image patches in an entire video (e.g., attended to objects individually), the system/method400attends to object interactions while considering their temporal order.

As shown, the system/method400includes an attention LSTM layer410, a temporal attention (“attend”) component420, a co-attention (“co-attend) component430, and a language LSTM layer440.

The attention LSTM layer410identifies which part of the video in spatio-temporal feature space is needed for the Language LSTM layer440. To do this, the attention LSTM layer410fuses the previous hidden state output of the Language LSTM component440(htw,12), the overall representation of the video, and the input word at timestep tw−1 to generate the hidden representation for the attend component420. For example, the input to the attention LSTM layer410(xtw1) can be defined as:
xtw1=htw-12∥gϕ(Vc)∥WeΠtw-1
where gϕis an MLP having parameter ϕ, gϕ(Vc) are the projected image features that are mean-pooled, Wcis a word embedding matrix for a vocabulary of size Σ, and Πtw−1is a one-hot encoding of the input word at timestep tw−1, where twis the timestep for each word generation.

In one embodiment, as shown inFIG. 4, the attend component420can adapt α-attention, similar to the α-attention module330ofFIG. 3, to attend over the projected image features gϕ(Vc). The attend component420uses the output of the Attention LSTM layer410(htw1) and the projected image features gϕ(Vc) as input. For example, as shown inFIG. 4, the input to the attend component420(Xa) can be defined as:
Xa=repeat(Whhtw1)+Wcgϕ(Vc)
where dϕis the dimension of the last FC layer of gϕand

Wh⁢ϵ⁢⁢ℝdϕ×dhtw1
and Wcϵdϕ×dϕare learned weights for htw1and gϕ(Vc), respectively. Details regarding the layers of the attend component420(e.g., Tanh, FC, Masking, SoftMax, MatMul) are described above with reference toFIG. 3regarding the layers332-340of the α-attention module330.

The co-attend component430can apply the temporal attention obtained from image features to object interaction representations h=(h1, h2, . . . , hT).

The language LSTM layer440receives, as input (xtw2), a concatenation of the output of the attention LSTM layer410(htw1), an attended image representation ({circumflex over (v)}c,tw) and co-attended object interactionsat timestep tw, e.g.:
xtw2=htw1∥{circumflex over (v)}c,tw∥ĥtw

The output of the language LSTM layer440(ht2) can be used to generate each word. In one embodiment, each word is generated by implementing a conditional probability distribution. For example, the conditional probability distribution can be represented by:
p(ytw|y1:tw−1)=softmax(Wphtw2)
where y1:tw−1is a sequence of outputs (y1, . . . , ytw−1) and

Wp⁢ϵ⁢⁢ℝΣ×dhtw2
is a learned weight for ht2. Bias terms have been omitted for simplicity in this illustrative example.

Referring toFIG. 5, a block/flow diagram500is provided illustratively depicting a system/method for implementing an interaction network, in accordance with one embodiment of the present principles.

At block510, frames of a video are obtained. The video can have frames spanning T timesteps.

At block520, an image feature for each timestep is generated. Each image representation can include a feature vector encoded from a corresponding one of the images. In one embodiment, the image features are generated by employing a CNN.

At block530, attention is performed based on the image features to generate a coarse-grained feature representation. In one embodiment, performing the attention includes performing SDP attention. The coarse-grained feature representation can include attended context information.

At block540, an object feature for each timestep is generated. In one embodiment, generating the object features includes generating region proposals, and obtaining the object features (e.g., ROIs) from each of the frames based on the region proposals. The region proposals can be generated using, e.g., an RPN.

At block550, HOIS between the object features are generated to generate a fine-grained feature representation. The fine-grained feature representation can include objects interactions discovered using recurrent HOI.

At block560, the coarse-grained and the fine-grained feature representations are concatenated to generate a concatenated feature representation. The concatenated feature representation serves as an input to the last fully-connected layer. In one embodiment, prior to generating the concatenated feature representation, the feature vector can be separately re-normalized with a batch normalization layer.

At block570, a prediction is made based on the concatenated feature representation to perform one or more operations. The one or more operations can include, for example, action recognition, video captioning, etc.

Further details regarding blocks510-570are described above with reference toFIGS. 1-4.

The embodiments described herein provide for more improved floating point operation per second (FLOP) efficiency as compared to conventional object interaction detection systems and methods, thereby improving object interaction detection system performance. Mean-pooling over the object features per frame and using LSTM for temporal reasoning can outperform single compact image representations, the latter of which are the trend for video classification methods. Combining image features with temporal SDP-Attention and object features over LSTM can reach, e.g., 73.1% top-1 accuracy, which can outperform Temporal Segment Networks (TSNs) using a deeper ConvNet with a higher video sampling rate. Beyond using mean-pooling as the simplest form of object interaction, the embodiments described herein for dynamically discovering and modeling HOIs can achieve, e.g., 74.2% top-1 and 91.7% top-5 accuracy.

For example, the following table compares the performance of the interaction network described herein (SINet) as compared to conventional interaction networks:

As shown in Table 1, the SINet can achieve pair-wise interaction (K=2) performance corresponding to around 5.3 gigaFLOPs (GLOPS) and third-order interaction (K=3) performance corresponding to around 8.0 GFLOPs. In contrast, conventional object interaction detection systems and methods can achieve pair-wise interaction performance corresponding to around 18.3 GFLOPs and can achieve third-order interaction performance corresponding to around 77 GFLOPs.

Embodiments described herein may be entirely hardware, entirely software or including both hardware and software elements. In a preferred embodiment, aspects of the present invention are implemented in software, which includes but is not limited to firmware, resident software, microcode, etc.

Referring now toFIG. 6, an exemplary computer system600is shown which may represent a server or a network device, in accordance with an embodiment of the present invention. The computer system600includes at least one processor (CPU)605operatively coupled to other components via a system bus602. A cache606, a Read Only Memory (ROM)608, a Random-Access Memory (RAM)610, an input/output (I/O) adapter620, a sound adapter630, a network adapter690, a user interface adapter650, and a display adapter660, are operatively coupled to the system bus602.

A first storage device622and a second storage device629are operatively coupled to system bus602by the I/O adapter620. The storage devices622and629can be any of a disk storage device (e.g., a magnetic or optical disk storage device), a solid state magnetic device, and so forth. The storage devices622and629can be the same type of storage device or different types of storage devices.

A speaker632may be operatively coupled to system bus602by the sound adapter630. A transceiver695is operatively coupled to system bus602by network adapter690. A display device662is operatively coupled to system bus602by display adapter660.

A first user input device652, a second user input device659, and a third user input device656are operatively coupled to system bus602by user interface adapter650. The user input devices652,659, and656can be any of a sensor, a keyboard, a mouse, a keypad, a joystick, an image capture device, a motion sensing device, a power measurement device, a microphone, a device incorporating the functionality of at least two of the preceding devices, and so forth. Of course, other types of input devices can also be used, while maintaining the spirit of the present invention. The user input devices652,659, and656can be the same type of user input device or different types of user input devices. The user input devices652,659, and656are used to input and output information to and from system600.

An interaction network640is illustratively shown operatively coupled to the system bus602. The interaction network640is configured to perform the operations described above with reference toFIGS. 1-5. The interaction network640can be implemented as a standalone special purpose hardware device, or may be implemented as software stored on a storage device. In the embodiment in which the interaction network640is software-implemented, although the interaction network640is shown as a separate component of the computer system600, the interaction network640can be stored on the first storage device622and/or the second storage device629. Alternatively, the interaction network640can be stored on a separate storage device (not shown).