Patent ID: 12192543

Corresponding reference characters indicate corresponding parts throughout the drawings.

DETAILED DESCRIPTION

The various examples will be described in detail with reference to the accompanying drawings. Wherever preferable, the same reference numbers will be used throughout the drawings to refer to the same or like parts. References made throughout this disclosure relating to specific examples and implementations are provided solely for illustrative purposes but, unless indicated to the contrary, are not meant to limit all examples.

Online action detection is the task of predicting the action as soon as it happens in streaming video, such as a golfer beginning to swing a club or a person beginning to speak. In contrast, with action detection in an offline setting, the entire untrimmed video is observable at any given moment, making detection of the particular frame at which action begins considerably easier. A major challenge for online action detection is that the predictions are solely based on observations of history (i.e., the video frames observed thus far) for context when making predictions, without access to video frames in the future.

The primary challenge in leveraging history of action detection is that for long untrimmed videos, the length becomes intractably long over time. One option is to limit the history to only on the most recent frames, for example a minute or less. Unfortunately, informative history may be discarded and provides no value (i.e., does not improve the probability of making a correct action/no-action prediction). Not every history frame is informative and useful, and some uninformative history may actually degrade accuracy, if used.

It is therefore valuable to accentuate portions the history that are more informative to the prediction of the current frame in online action detection with untrimmed streaming video. The disclosure presents a position-guided gated cross-attention mechanism to enhance or suppress portions of the history based on how informative they are for current frame prediction. Some examples further render history features more informative, by using subsequently observed frames when available.

The disclosure integrates a transformer's ability of long-range temporal modeling and a recurrent model's capacity to selectively encode relevant information. Some examples also introduce a background suppression objective to further mitigate false positive background frames that closely resemble action frames. Additionally, a flow-free version is able to achieve higher or close accuracy at a higher frame rate to prior solutions that require both color pixel information (“RGB”) and optical flow information for prediction.

Example solutions for video frame action detection (a.k.a. online action detection) use a gated history and include: receiving a video stream comprising a plurality of video frames; grouping the plurality of video frames into a set of present video frames and a set of historical video frames, the set of present video frames comprising a current video frame; determining a set of attention weights for the set of historical video frames, the set of attention weights indicating how informative a video frame is for predicting action in the current video frame; weighting the set of historical video frames with the set of attention weights to produce a set of weighted historical video frames; and based on at least the set of weighted historical video frames and the set of present video frames, generating an action prediction for the current video frame.

Aspects of the disclosure improve the operations of computing devices, for example, improving the accuracy and/or speed of video frame action detection at least by weighting a set of historical video frames with a set of attention weights that indicate how informative a video frame is for predicting action in the current video frame. Examples combine the benefits of selective encoding of a long-short term memory (LSTM) recurrent neural network (NN) with long-range modeling of a transformer to better leverage informative frames in a long-duration history. Practical applications include: safety, surveillance, content moderation, augmented reality (AR), self-driving cars, and autonomous vehicles.

FIG.1Aillustrates an example architecture100that advantageously provides for video frame action detection using gated history. A video stream102has a set of present video frames106that includes a current video frame102q, and a set of historical video frames104that together form a long history. For example, a current video frame is an immediately present frame while a set of present video frames includes the immediately present frame (or current frame) as well as one or more immediate past present frames. In other words, the set of present video frames include the most immediate previous frames to the current video frame for a given point in time. (SeeFIG.5Afor more detail.) Video stream102is provided to a featurizer500, which includes a future-augmented history (FAH)502component. A history encoder400, which includes a gated history unit (GHU)300, determines a set of attention weights and uses the set to weight set of historical video frames104(and also some or all of set of present video frames106). This produces a set of weighted historical video frames that is provided to a present decoder600.

In parallel, featurizer500also encodes set of present video frames106and provides that as an additional input to present decoder600. Present decoder600has a set of attention networks and a classifier620that outputs an action prediction610for at least current video frame102q. GHU300is described in further detail in relation toFIG.3. History encoder400is described in further detail in relation toFIG.4. Featurizer500and FAH are described in further detail in relation toFIG.5. Present decoder600and background suppression are described in further detail in relation toFIG.7. Video stream102, set of present video frames106, and set of historical video frames104are shown in further detail in relation toFIG.7. A more detailed operation of architecture100is provided after describing the various components in each ofFIGS.2-7.

FIGS.2A and2Billustrate two among many practical applications for using the architecture ofFIG.1. InFIG.2A, a video frame action detection scenario200adepicts a video camera216capturing a scene218and outputting video stream102, which is subject to a broadcast delay202. Video stream102is also provided to architecture100that outputs action prediction610. Action prediction610is provided to an action response204that responds to action prediction610based on the class of the predicted action and the specific application being employed for the video frame action detection (e.g., safety, surveillance, content moderation, AR, self-driving car, autonomous vehicle, or other). In the current scenario200a, the predicted action is represented as an annotation210superimposed on current video frame102q.

Architecture100operates in real-time, such that annotation210is ready during broadcast delay202, which may on the order of seconds (e.g., seven seconds, in some examples). A broadcast function206transmits current video frame102qto a display208, where current video frame102qis displayed with annotation210. In this illustrated example, annotation210comprises an outline of an object212involved with action prediction610, for example, a golfer swinging a golf club. In this scenario, action prediction610is not offensive, but is instead the type of action for which a viewer's attention is desired.

InFIG.2B, a video frame action detection scenario200bdepicts an AR engine222generating video stream102, for example by mixing a live scene capture by a camera (e.g., video camera216and scene218of scenario200a). Video stream102is provided to architecture100that outputs action prediction610. Action prediction610is provided to action response204that responds to action prediction610based on the specific application being employed for the video frame action detection. In the current scenario200a, the action is also an annotation210superimposed on current video frame102q.

Architecture100operates in real-time, such that annotation210is ready at approximately the same time as AR engine222is able to insert icons (e.g., AR object214) and/or virtual objects into current video frame102q. AR engine222provides current video frame102qto display208, where current video frame102qis displayed with annotation210. In this illustrated example, annotation210comprises a brightening of pixels in a region of current video frame102qin proximity to object212involved with action prediction610. Other annotation possibilities include an obscuration of the object involved with the action prediction and blanking the current video frame, for example if the predicted action is a class that indicates offensive material.

Multiple additional practical applications exist for architecture100. For example, architecture100, including training and testing, may be operated and deployed in customer premises such as internet of things (IoT) and edge devices. For example, architecture100may be deployed in a retail store where one or more surveillance cameras capture video feed for long durations. The video feed may be annotated and utilized to train architecture100for online action detection of actions/events including customer behavior, and suspicious or criminal activities. Upon training, architecture100model may be deployed in the on-premises IoT devices for the online detection of the actions and events.

Further deployments may leverage edge artificial intelligence (AI) scenarios such as on-premises devices and cloud services. Architecture100may perform action anticipation tasks by training the model such that, rather than predicting the action for the current observed frame, it predicts the action for a frame that will be observed sometime later, in the future. This is useful in scenarios for content moderation in live-video streams where the model may predict if some harmful or otherwise inappropriate activity is about to occur and allow for a time buffer to block the video feed in time.

This application may also be useful in autonomous driving where it may help predict the trajectory of vehicles and pedestrians on the road in advance, to ensure informed driving-related decisions. It may further benefit augmented reality (AR) scenarios in which the intent and the behavior of users may be anticipated to improve user experience.

Although architecture100performs online action detection where the future frames are unavailable to the model, architecture100may also perform per-frame action prediction in offline settings, in which access to all frames of the video stream is available. To do so, the current frame may be set as the middle frame in the present decoder of the model, with frames subsequently observed after the current frame being “future” frames. This way, the model may leverage the future frames available in the offline setting for per-frame action prediction. The ability to perform per-frame action prediction in an offline setting is beneficial in scenarios that are not time critical, and the primary objective is improving the accuracy of action prediction across the entire video stream.

Some examples of such scenarios include generating highlights of a sports event to telecast at a later time, or assisting coaches to assess a player's performance. Other scenarios include analyzing a video uploaded by a user for any assessing activity occurring at an unknown location within the video stream.

In some examples, architecture100may take input and process features from multiple modalities simultaneously to improve the performance of action detection. Modality may include RGB-based appearance, optical flow/motion, depth data from time-of-flight sensors, audio data, text and/or language data, data from sensors such as accelerometer, gyroscope, magnetometer etc. or the like. This may support several multimodal user scenarios such as audio-visual content moderation, autonomous driving involving multiple sensors and AR/metaverse involving both RGB and depth sensors. Architecture100may support multimodal scenarios for various tasks—online action detection, action anticipation and per-frame offline action detection.

FIG.3illustrates GHU300, a position-guided gated cross-attention component of architecture100that enhances or suppresses frames of video stream102(e.g., frames of set of historical video frames104) according to how informative each frame is for predicting action for current frame103q. An introduction to the components of GHU300are provided here, and further details regarding operation of GHU300are provides after the other various components of architecture100are similarly introduced inFIGS.4-7.

A query (Q302), key (K304), and value (V308) are provided by featurizer500. A gating score (G306) is a separate learned encoding that in some examples, ranges from negative infinity to e (2.78). Q302and K304are provided to a matrix multiplication312, which is then scaled by a scaler314. G306is provided to a sigmoid function316, the output is subjected to a log function318and added to itself by an addition320. That is gated by a gate322and added to the output of scaler314by a soft gating addition324. A softmax326is applied and the result is provided to a matrix multiplication328with V308. This produces set of attention weights310for at least set of historical video frames104(and set of present video frames106, in some examples).

FIG.4illustrates history encoder400. A set of encoded features (described below, for example in relation toFIG.5) is provided to GHU300, along with Q302. The output of GHU300(e.g., set of attention weights310) is provided to a self-attention network402that outputs a set of weighted historical video frames410.

FIG.5Aillustrates featurizer500and further detail regarding video stream102. Video stream102is illustrated as comprising a plurality of video frames, video frames102a-102q. Set of historical video frames104includes video frame102a, video frame102b, video frame102c, video frame102d, video frame102e, video frame102f, video frame102g, video frame102h, video frame102i, video frame102j, video frame102k, video frame102l, video frame102m, video frame102n, and video frame102o.

Set of present video frames106is the set of the most recent video frames, including the latest frame (or immediately present frame), current video frame102q. Set of present video frames106also includes video frame102oand video frame102p, which are the immediate past present frames relative to current video frame102q. Video frame102ois also in set of historical video frames104, providing overlap between set of historical video frames104and set of present video frames106. A future video frame102ris not yet available. It should be understood that these numbers of video frames are for illustrative purposes, and some examples of architecture100may use a significantly larger number of video frames.

Although current video frame102qdoes not have a future video frame (e.g., video frame102r) available for refining an action prediction, video frames in set of historical video frames104do have “future” video frames available. Turning briefly toFIG.5B, the idea of “future” video frames for historical video frames is illustrated.

For any given video frame within set of historical video frames104, and video frame may be designated as a history frame. For each individual history frame, there is a set of subsequently-observed video frames within video stream102that is more recent in time than that individual history frame. For example, for video frame102c, video frames102d,102e, and102fare more recent; for video frame102d, video frames102e,102f, and102gare more recent; and for video frame102e, video frames102f,102g, and102hare more recent. From the perspective of any given history frame, the set of subsequently-observed video frames represents “future” video frames relative to that given history frame, because those “future” video frames are later in time relative to the history frame.

This can be exploited to improve set of historical video frames104, providing for future-augmented encoding by FAH502. In other words, FAH502leverages hindsight to provide “future” frames for history frames to improve the encoding of history for current frame prediction. FAH502aggregates observed “future” information into the features of a history frame to make it aware of its so-far-observable future. At each new time step with one more new frame observed, FAH502will feed-forward through the feature extraction backbone twice to extract features for the new frame.

Returning toFIG.5A, video stream102, including set of historical video frames104and set of present video frames106, is weighted by attention weights508. Video frames of set of historical video frames104, and video frames of set of present video frames106have already been through architecture100as the then-current video frame. Thus, they have each been assessed for whether they had action or no action, and so attention weights508may be derived by remembering this activity for each of the video frames.

FAH502extracts features from only the most informative video frames, producing features504aand features504bfor set of historical video frames104, and features504sfor set of present video frames106. Features504a-504care encoded by an encoder506into encoded features510that is provided to history encoder400ofFIG.4. Features504care encoded by encoder506into encoded features512that is provided to present decoder600ofFIG.6.

As described below, some examples of featurizer500use optical flow for improved accuracy, but at the expense of slower execution time. Optical flow estimation identifies pixel-wise motion of objects (e.g., object212ofFIG.2) between consecutive video frames. Some examples of featurizer500do not use optical flow, but instead use only red, green, and blue (RGB) pixel information to improve computation time, for example cutting up to two-thirds of the execution time. Some examples of architecture100may operate in time-critical applications and/or on lower-performance computational platforms, and thus not use optical flow.

FIG.6illustrates present decoder600that correlates a small set of the most recent video frames (e.g., set of present video frames106) with history of video stream102(e.g., at least set of historical video frames104) to make the current frame prediction. Encoded features512and set of weighted historical video frames410are provided to a set of attention networks. For example, encoded features512is provided to a self-attention network602with a causal mask, and the output of that is provided to a cross-attention network604, along with set of weighted historical video frames410.

The output of cross-attention network604is provided to both another self-attention network606and, along with the output of self-attention network606, to another cross-attention network608. The output of cross-attention network608is provided to classifier620that outputs action prediction610. In some examples, action prediction610includes both an action class612(e.g., “no action” or one of a pre-defined set of action classes) and a confidence614, which is a measure of how confident classifier620is regarding action prediction610.

Some examples of present decoder600also use background suppression622to mitigate the false positive prediction of background frames that closely resemble action frames. Background suppression622uses a loss function624and is applied as a loss on classifier620. Background suppression622adds emphasis on low confidence predictions that occur between no action video frames and action video frames, to incentivize learning these frames correctly—so that classifier620is trained better. In other words, background suppression622modifies the confidence of the action prediction by weighting low confidence video frames more heavily, with separate emphasis on action and background classes, for classifier620that generates the action prediction.

FIG.7illustrates no action video frames, low confidence action video frames, and high confidence action video frames, as may be encountered when using examples of architecture100. Action prediction610mfor video frame102mshows an action class612mas “no action” and confidence612mas relatively high. Action prediction610nfor video frame102nshows an action class612nas “golf swing” and confidence612nas relatively low. Action prediction610ofor video frame102oshows an action class612oas “golf swing” and confidence612oas relatively high. Action prediction610pfor video frame102pshows an action class612pas “golf swing” and confidence612pas relatively high. Action prediction610qfor current video frame102qshows an action class612qas “golf swing” and confidence612qas relatively high.

Thus, video frame102mis a no action video frame, video frame102nis a low confidence action video frame, and video frames102o-102qare high confidence action video frames. Background suppression622modifies the confidence, for example by using loss function624, to place emphasis on video frame102nwhen training classifier620. In some examples, confidence values range from zero to one, [0,1], and denotes the probability of predicting the correct action.

Further detail is now provided for the operation of architecture100. Architecture100includes GHU300, FAH502, and background suppression622, which enable improving accuracy and/or speeding execution time by a factor of approximately 3×. GHU300provides position-guided, gated cross-attention that explicitly enhances or suppresses parts of video history as per how informative they are to predicting action for the current frame. FAH502extracts features from history frames using their subsequently observed frames, to enhance history encoding. Background suppression622mitigates false positive predictions of background frames that closely resemble action frames.

Given a streaming video sequence h=[ht]t=−T+10(video stream102), the task is to identify if and what action y0∈{0, 1, . . . C} occurs at the current frame h0(i.e., h0is current frame102q). There is a total of C action classes and a label “0” for background frames with no action (i.e., action class612=0 for “no action” video frames) are available for use by classifier620. Since future frames denoted as h1, h2, . . . , (e.g., future video frame102rand others) are not yet accessible, the network model makes a (C+1)-way prediction for the current frame (h0) based on the recent T frames, h=[ht]t=T+10, observed up until the current frame. While T may be large in an untrimmed video stream, all frames observed in past history h=[ht]t=T+10may not be equally informative to the prediction for the current frame.

To make the (C+1)-way prediction accurately for current frame h0based on T history frames, h=[ht]t=T+10, transformers encode the video sequence history (e.g., at least set of historical video frames106of video stream102) and then associate the current frame with the encoding for prediction. History encoder400uses cross-attention to project the variable length history to a fixed-length learned latent encoding. Using cross-attention may be more efficient than using self-attention because its computational complexity is quadratic with respect to latent encoding size, instead of the video sequence length which is typically orders of magnitude larger. In some scenarios, the resulting execution time difference is important for online video.

Specifically, given

h=[ht]t=-T+10
as the streaming sequence of T history frames ending at current frame h0, each frame h is encoded with a feature extraction backbone, u, followed by a linear encoding layer E. The output is subjected to a learnable position encoding, Eos, relative to the current frame, h0, to give zh=u(h)E+EPOS, where u(h)∈T×M, E∈M×D, zh∈T×Dand EPOS∈T×D. M and D denote the dimensions of extracted features and post-linear encoding features, respectively. A learnable latent query encoding, q∈L×D, is cross-attended with h. Following a multi-headed cross-attention setup, NHEADSis the number of heads in GHU300such that Qi=qWiq, Ki=zhWik, and Vi=zhWivare queries, keys and values, respectively (i.e., Q302, K304, and V308), for each head i∈{1, . . . , NHEADS} where projection matrices Wiq, Wik∈×dv. There is an assignment of dk=dv=D/NHEADS.

The position-guided gating scores G are obtained, for h by:

zg=σ⁡(zh⁢Wg)Eq.(1)

G=log⁡(zg)+zgEq.(2)
where Wg∈D×1is the matrix projecting each history frame to a scalar. Then, zg∈T×1is a sequence of scalars for the history frames h after applying sigmoid σ.

The gating score (G308) for history frames in GHU300is G∈T×1. By using zh, which already contains the position encoding, the gates are guided by the relative position of the history frame to the current frame h0. The gated cross-attention for each head, GHUi, is computed as:

GHUi=Softmax⁢(Qi⁢KiTdk+G)⁢ViEq.(3)
And multi-headed gated cross-attention defined as:

MultiHeadGHU⁡(Q,K,V,G)=Concat⁡([GHUi]i=0NHEADS)⁢W0Eq.(4)
where W0∈D×Dre-projects the attention output to D dimensions. It is possible to define G separately for each, however, in some examples, sharing G across all heads performs better.

From Eqs. (1) and (2), it can be observed that each scalar in zglies in [0, 1] due to sigmoid, which implies that each gating score in G lies in [−∞, 1]. This enables the softmax function in Eq. (3) to calibrate the attention weight for each history frame by a factor in [0, e] such that a factor in [0, 1) suppresses a given history frame and a factor in (1, e] enhances a given history frame. This provides an explicit ability for GHU300to learn to calibrate the attention weight of a history frame based on how informative the history frame is for prediction of current frame h0.

Thus, G is input-dependent and learns based on the history frame and its position with respect to the current frame. This enables GHU300to assess how informative each history frame is based on its feature representation and relative position from the current frame h0. The output of GHU300is fed to a series of N self-attention layers (self-attention network402) to obtain the final history encoding (i.e., set of weighted historical video frames410) output from history encoder400.

FAH502leverages hindsight to provide “future” frames for history frames to improve the encoding of history for current frame prediction. (SeeFIG.5B.) FAH502aggregates observed “future” information into the features of a history frame to make it aware of its so-far-observable future. For a history frame htand a feature extraction backbone u, when tf“future” history frames for htmay be observed, FAH502extracts features for htusing a set of frames

[hi]i=tt+tf
(i.e., the history frame itself and its subsequently observed tffuture frames). Otherwise, FAH502extracts features for htusing a set of frames

[hi]i=t-tpst
(i.e., the history frame itself and its past tpsframes):

u⁡(ht)={u⁢([hi]i=t-tpst),if⁢t>-tfu⁡([hi]i=tt+tf),if⁢t≤-tfEq.(5)

At each new time step with one more new frame observed, FAH502will feed-forward through u twice to extract features for the new frame using

[hi]i=-tps0
frames and, also h−tfthat is now eligible to aggregate future information using

[hi]i=-tf0
frames.

In order to correlate the present with the history to perform the current frame prediction, a subset of tprmost recent history frames

[hi]t=-tpr-1t
is sampled to model the present (i.e., the most immediate context) for h0using present decoder600. After extracting the features via FAH500, a learnable position encoding,

Epospr
is applied to each of the tprframe features, which are subjected to a multi-headed self-attention with a causal mask. The causal mask limits the influence of only the preceding frames on a given frame.

The output from self-attention is cross-attended with the history encoding from history encoder400. This is repeated, although the self-attention does not need a causal mask the second time. The output corresponding to each of tprframes is fed to the classifier layer (e.g., classifier620) for prediction.

Background (“no action”) video frames may be anything from completely blank at the beginning of video stream102to closely resembling action frames—but without actually being action frames (e.g., aiming before making a billiards shot). The latter scenario is a common cause for false positives (e.g., classifying a “no action” video frame as an “action” video frame). To reduce false positives, background suppression622applies emphasis to low-confident action and background predictions during training of classifier620to increase the margin between action and background (“no action”) video frames.

The objective (loss) function, Lt, (loss function524) for frame htis defined as:

ℒt={-yt0(1-pt0)γb⁢log⁡(pt0),if⁢t>-tf-∑i=1C⁢yti(1-pti)γa⁢log⁡(pti),if⁢t≤-tfEq.(6)
where γa, γb>0, enables low-confident samples to contribute more to the overall loss forcing the model to put more emphasis on correctly predicting these samples. Loss function524(t) applying separate γ to action classes and background. This separation distinguishes the action classes that have a more constrained distribution from the background class, whose distribution is more complex and unconstrained.

Some examples use optical flow in addition to RGB to capture fine-grained motion among frames. Computing optical flow, however, requires more computational time than feature extraction or model inference alone, and may be too slow in some scenarios for time-critical applications (e.g., autonomous vehicles and self-driving cars). Thus, some examples do not use optical flow.

To capture motion without optical flow using only RGB frames, multiple temporal resolutions using a spatio-temporal backbone is used, in some examples. In some examples, two feature vectors are extracted for a frame htby encoding a frame sequence sampled at a higher frame rate, spanning a smaller time duration, and another frame sequence sampled at a lower frame rate spanning, a longer time duration. The two feature vectors are concatenated.

In an example, video stream102is sampled at 24 frames per second (FPS), and frames are extracted at 4 FPS for training and evaluation. The size of set of historical video frames104is set to 1024, and the size of set of present video frames106is set to 8 most recently-observed frames. This spans 256 seconds and 2 seconds, respectively, at 4 FPS. A two-stream temporal segment network (TSN) is used to extract frame-level RGB and optical flow features. The RGB and optical flow features are concatenated the along channel dimension prior to feeding to a linear encoding layer. The time duration for FAH502is set for past tpsand future tfframes to be 1 second and 2 seconds, respectively.

For a version that does not use optical flow, the optical flow features are replaced with features obtained from an additional multi-frame input of RGB frames uniformly sampled over a duration of 2 seconds. Training is performed for 10 epochs, with a weight decay of 5E-5, and a batch size of 50. D is set to 1024, latent encoding size is 16, and two layers are used in the history decoder. Each attention layer has 16 heads (NHEADS=16), and γa=0.6, γb=0.2 for background suppression.

FIG.8shows a flowchart800illustrating exemplary operations that may be performed by architecture100. In some examples, operations described for flowchart800are performed by computing device1000ofFIG.10. Flowchart800commences with receiving video stream102comprising a plurality of video frames in operation802. Given a streaming video sequence, video stream102for example, the task is to identify if and what action occurs at the current frame. There may be any number of different action class labels as well as a label for background frames with no action (i.e., “no action” video frames) that are available for use by a classifier, such as classifier620.

Operation804groups plurality of video frames102a-106q(of video stream102) into set of present video frames106(comprising current video frame102q) and set of historical video frames104. In some examples, set of present video frames106and set of historical video frames104overlaps. Since future frames are not yet accessible, the process uses the recent frames observed up until the current frame to make a prediction for the current frame. While the recent frames may be large in an untrimmed video stream, all frames observed in past history may not be equally informative to the prediction for the current frame. Operation806weights the sets of subsequently-observed video frames with previously-determined attention weights.

Operation808extracts features504aand504bfrom set of historical video frames104, based on at least a set of history frames and their sets of subsequently-observed video frames. In some examples, extracting features uses optical flow. In some examples, extracting features does not use optical flow. Operation810encodes extracted features504aand504b. In some examples, this includes determining a latent encoding for each video frame in set of historical video frames104. Some examples of operation810include operation812that encodes extracted features504aand504busing cross-attention to project a variable length portion of the plurality of video frames to a fixed-length learned latent encoding.

In some examples, to make the prediction accurately for the current frame based on the history frames, transformers encode the video sequence history (e.g., at least a set of historical video frames106of video stream102) and then associate the current frame with the encoding for prediction. History encoder400uses cross-attention to project the variable length history to a fixed-length learned latent encoding. Specifically, each frame is encoded with a feature extraction backbone followed by a linear encoding layer. The output is subjected to a learnable position encoding relative to the current frame. A learnable latent query encoding is then cross-attended with each frame.

Operation814determines set of attention weights310for set of historical video frames104. Set of attention weights310indicates how informative a video frame is for predicting action in current video frame102qand enhances or suppresses video frames of set of historical video frames104based on at least how informative a video frame is for predicting action in current video frame102q. In some examples, each attention weight of set of attention weights310is positive-valued. In some examples, set of attention weights310is within a range of zero to a maximum positive value. A value between 0 and 1 suppresses attention to a video frame and a value between 1 and the maximum positive value enhances attention to a video frame. Some examples of operation814use operation816that determines, for each video frame of set of historical video frames104, a position-guided gating score, G306. In one example, the position-guided gating scores are obtained for each frame using a matrix projecting each history frame to a scalar and generating a sequence of scalars for the history frames after applying the sigmoid function. This enables the softmax function to calibrate the attention weight for each history frame to suppress or enhance a given history frame, and provides an ability for GHU300to learn to calibrate the attention weight of a history frame based on how informative the history frame is for prediction of a given current frame. This demonstrates that a position-guided gating score is input-dependent and learns based on the history frame and its position with respect to the current frame. This enables GHU300to assess how informative each history frame is based on its feature representation and relative position from the current frame.

Operation818weights set of historical video frames104with set of attention weights310to produce set of weighted historical video frames410. For example, the output of GHU300is fed to a series of self-attention layers, such as self-attention network402, to obtain the final history encoding, such as set of weighted historical video frames410, output from history encoder400. Operation820generates action prediction610for current video frame102qbased on at least set of weighted historical video frames410and set of present video frames106. Operation820uses operations822-826. Operation822generating action prediction610for current video frame102qby cross-attending an encoded history that is based on at least set of weighted historical video frames410with a self-attention network output that is based on at least encoded extracted features of set of present video frames106. In some examples, action prediction610comprises a no action prediction or an action class612prediction selected from a plurality of action classes, determined in operation824. In some examples, action prediction610comprises confidence614, determined in operation826.

Operation828performs background suppression, using operation830to modify confidence614, such as by generating loss function624for example, that weights low confidence video frames more heavily, with separate emphasis on action and background classes, for classifier620that generates action prediction610. In some examples, training of classifier620is ongoing, during operation.

Decision operation832determines whether current video frame102qis classified as an action frame. If so, then based on at least action prediction610for current video frame102q, operation834generates annotation210for current video frame102q. In some examples, annotation210for current video frame102qcomprises an annotation selected from the list consisting of: an outline of object212involved with action prediction610, a brightening of pixels in a region of current video frame102qin proximity to object212involved with action prediction610, an obscuration of object212involved with action prediction610, and blanking current video frame102q. Operation836displays current video frame102qsubject to annotation210for current video frame102q. Otherwise, operation838displays current video frame102qnormally.

FIG.9shows a flowchart900illustrating exemplary operations that may be performed by architecture100. In some examples, operations described for flowchart900are performed by computing device1000ofFIG.10. Flowchart900commences with operation902, which includes receiving a video stream comprising a plurality of video frames. Operation904includes grouping the plurality of video frames into a set of present video frames and a set of historical video frames, the set of present video frames comprising a current video frame.

Operation906includes determining a set of attention weights for the set of historical video frames, the set of attention weights indicating how informative a video frame is for predicting action in the current video frame. Operation908includes weighting the set of historical video frames with the set of attention weights to produce a set of weighted historical video frames. Operation910includes, based on at least the set of weighted historical video frames and the set of present video frames, generating an action prediction for the current video frame.

Additional Examples

An example system comprises: a processor; and a computer-readable medium storing instructions that are operative upon execution by the processor to: receive a video stream comprising a plurality of video frames; group the plurality of video frames into a set of present video frames and a set of historical video frames, the set of present video frames comprising a current video frame; determine a set of attention weights for the set of historical video frames, the set of attention weights indicating how informative a video frame is for predicting action in the current video frame; weight the set of historical video frames with the set of attention weights to produce a set of weighted historical video frames; and based on at least the set of weighted historical video frames and the set of present video frames, generate an action prediction for the current video frame.

An example computerized method comprises: receiving a video stream comprising a plurality of video frames; grouping the plurality of video frames into a set of present video frames and a set of historical video frames, the set of present video frames comprising a current video frame; determining a set of attention weights for the set of historical video frames, the set of attention weights indicating how informative a video frame is for predicting action in the current video frame; weighting the set of historical video frames with the set of attention weights to produce a set of weighted historical video frames; and based on at least the set of weighted historical video frames and the set of present video frames, generating an action prediction for the current video frame.

One or more example computer storage devices have computer-executable instructions stored thereon, which, on execution by a computer, cause the computer to perform operations comprising: receiving a video stream comprising a plurality of video frames; grouping the plurality of video frames into a set of present video frames and a set of historical video frames, the set of present video frames comprising a current video frame; determining a set of attention weights for the set of historical video frames, the set of attention weights indicating how informative a video frame is for predicting action in the current video frame; weighting the set of historical video frames with the set of attention weights to produce a set of weighted historical video frames; and based on at least the set of weighted historical video frames and the set of present video frames, generating an action prediction for the current video frame.

Alternatively, or in addition to the other examples described herein, examples include any combination of the following:based on at least the action prediction for the current video frame, generating an annotation for the current video frame;displaying the current video frame subject to the annotation for the current video frame;determining the set of attention weights comprises determining, for each video frame of the set of historical video frames, a position-guided gating score;the plurality of video frames comprises a set of history frames, and for each history frame in the set of history frames, a set of subsequently-observed video frames;the set of subsequently-observed video frames is more recent than the history frame;based on at least the set of history frames and their sets of subsequently-observed video frames, extracting features from the set of historical video frames;encoding the extracted features;extracting features does not use optical flow;performing background suppression;the action prediction comprises a confidence;performing the background suppression comprises weighting low confidence video frames more heavily, with separate emphasis on action and background classes, for a classifier that generates the action prediction;the action prediction comprises a no action prediction or an action class prediction selected from a plurality of action classes;the set of present video frames and the set of historical video frames overlaps;for each video frame in the set of historical video frames, determining a latent encoding;each attention weight of the set of attention weights is positive-valued;the set of attention weights is within a range of zero to a maximum positive value, wherein a value between zero and one suppresses attention to a video frame and a value between one and the maximum positive value enhances attention to a video frame;the set of attention weights enhances or suppresses video frames of the set of historical video frames based on at least how informative a video frame is for predicting action in the current video frame;the annotation for the current video frame comprises an annotation selected from the list consisting of an outline of an object involved with the action prediction, a brightening of pixels in a region of the current video frame in proximity to the object involved with the action prediction, an obscuration of the object involved with the action prediction, and blanking the current video frame;the sets of subsequently-observed video frames are weighted with previously-determined attention weights;weighting the sets of subsequently-observed video frames with previously-determined attention weights;encoding the extracted features comprises using cross-attention to project a variable length portion of the plurality of video frames to a fixed-length learned latent encoding;generating the action prediction for the current video frame comprises cross-attending an encoded history that is based on at least the set of weighted historical video frames with a self-attention network output that is based on at least encoded extracted features of the set of present video frames; andextracting features uses optical flow.

While the aspects of the disclosure have been described in terms of various examples with their associated operations, a person skilled in the art would appreciate that a combination of operations from any number of different examples is also within scope of the aspects of the disclosure.

Example Operating Environment

FIG.10is a block diagram of an example computing device1000(e.g., a computer storage device) for implementing aspects disclosed herein, and is designated generally as computing device1000. In some examples, one or more computing devices1000are provided for an on-premises computing solution. In some examples, one or more computing devices1000are provided as a cloud computing solution. In some examples, a combination of on-premises and cloud computing solutions are used. Computing device1000is but one example of a suitable computing environment and is not intended to suggest any limitation as to the scope of use or functionality of the examples disclosed herein, whether used singly or as part of a larger set.

Neither should computing device1000be interpreted as having any dependency or requirement relating to any one or combination of components/modules illustrated. The examples disclosed herein may be described in the general context of computer code or machine-useable instructions, including computer-executable instructions such as program components, being executed by a computer or other machine, such as a personal data assistant or other handheld device. Generally, program components including routines, programs, objects, components, data structures, and the like, refer to code that performs particular tasks, or implement particular abstract data types. The disclosed examples may be practiced in a variety of system configurations, including personal computers, laptops, smart phones, mobile tablets, hand-held devices, consumer electronics, specialty computing devices, etc. The disclosed examples may also be practiced in distributed computing environments when tasks are performed by remote-processing devices that are linked through a communications network.

Computing device1000includes a bus1010that directly or indirectly couples the following devices: computer storage memory1012, one or more processors1014, one or more presentation components1016, input/output (I/O) ports1018, I/O components1020, a power supply1022, and a network component1024. While computing device1000is depicted as a seemingly single device, multiple computing devices1000may work together and share the depicted device resources. For example, memory1012may be distributed across multiple devices, and processor(s)1014may be housed with different devices.

Bus1010represents what may be one or more busses (such as an address bus, data bus, or a combination thereof). Although the various blocks ofFIG.10are shown with lines for the sake of clarity, delineating various components may be accomplished with alternative representations. For example, a presentation component such as a display device is an I/O component in some examples, and some examples of processors have their own memory. Distinction is not made between such categories as “workstation,” “server,” “laptop,” “hand-held device,” etc., as all are contemplated within the scope ofFIG.10and the references herein to a “computing device.” Memory1012may take the form of the computer storage media referenced below and operatively provide storage of computer-readable instructions, data structures, program modules and other data for the computing device1000. In some examples, memory1012stores one or more of an operating system, a universal application platform, or other program modules and program data. Memory1012is thus able to store and access data1012aand instructions1012bthat are executable by processor1014and configured to carry out the various operations disclosed herein.

In some examples, memory1012includes computer storage media. Memory1012may include any quantity of memory associated with or accessible by the computing device1000. Memory1012may be internal to the computing device1000(as shown inFIG.10), external to the computing device1000(not shown), or both (not shown). Additionally, or alternatively, the memory1012may be distributed across multiple computing devices1000, for example, in a virtualized environment in which instruction processing is carried out on multiple computing devices1000. For the purposes of this disclosure, “computer storage media,” “computer-storage memory,” “memory,” and “memory devices” are synonymous terms for the computer-storage memory1012, and none of these terms include carrier waves or propagating signaling.

Processor(s)1014may include any quantity of processing units that read data from various entities, such as memory1012or I/O components1020. Specifically, processor(s)1014are programmed to execute computer-executable instructions for implementing aspects of the disclosure. The instructions may be performed by the processor, by multiple processors within the computing device1000, or by a processor external to the client computing device1000. In some examples, the processor(s)1014are programmed to execute instructions such as those illustrated in the flow charts discussed below and depicted in the accompanying drawings. Moreover, in some examples, the processor(s)1014represent an implementation of analog techniques to perform the operations described herein. For example, the operations may be performed by an analog client computing device1000and/or a digital client computing device1000. Presentation component(s)1016present data indications to a user or other device. Exemplary presentation components include a display device, speaker, printing component, vibrating component, etc. One skilled in the art will understand and appreciate that computer data may be presented in a number of ways, such as visually in a graphical user interface (GUI), audibly through speakers, wirelessly between computing devices1000, across a wired connection, or in other ways. I/O ports1018allow computing device1000to be logically coupled to other devices including I/O components1020, some of which may be built in. Example I/O components1020include, for example but without limitation, a microphone, joystick, game pad, satellite dish, scanner, printer, wireless device, etc.

Computing device1000may operate in a networked environment via the network component1024using logical connections to one or more remote computers. In some examples, the network component1024includes a network interface card and/or computer-executable instructions (e.g., a driver) for operating the network interface card. Communication between the computing device1000and other devices may occur using any protocol or mechanism over any wired or wireless connection. In some examples, network component1024is operable to communicate data over public, private, or hybrid (public and private) using a transfer protocol, between devices wirelessly using short range communication technologies (e.g., near-field communication (NFC), Bluetooth™ branded communications, or the like), or a combination thereof. Network component1024communicates over wireless communication link1026and/or a wired communication link1026ato a remote resource1028(e.g., a cloud resource) across network1030. Various different examples of communication links1026and1026ainclude a wireless connection, a wired connection, and/or a dedicated link, and in some examples, at least a portion is routed through the internet.

Although described in connection with an example computing device1000, examples of the disclosure are capable of implementation with numerous other general-purpose or special-purpose computing system environments, configurations, or devices. Examples of well-known computing systems, environments, and/or configurations that may be suitable for use with aspects of the disclosure include, but are not limited to, smart phones, mobile tablets, mobile computing devices, personal computers, server computers, hand-held or laptop devices, multiprocessor systems, gaming consoles, microprocessor-based systems, set top boxes, programmable consumer electronics, mobile telephones, mobile computing and/or communication devices in wearable or accessory form factors (e.g., watches, glasses, headsets, or earphones), network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, virtual reality (VR) devices, augmented reality (AR) devices, mixed reality devices, holographic device, and the like. Such systems or devices may accept input from the user in any way, including from input devices such as a keyboard or pointing device, via gesture input, proximity input (such as by hovering), and/or via voice input.

Examples of the disclosure may be described in the general context of computer-executable instructions, such as program modules, executed by one or more computers or other devices in software, firmware, hardware, or a combination thereof. The computer-executable instructions may be organized into one or more computer-executable components or modules. Generally, program modules include, but are not limited to, routines, programs, objects, components, and data structures that perform particular tasks or implement particular abstract data types. Aspects of the disclosure may be implemented with any number and organization of such components or modules. For example, aspects of the disclosure are not limited to the specific computer-executable instructions, or the specific components or modules illustrated in the figures and described herein. Other examples of the disclosure may include different computer-executable instructions or components having more or less functionality than illustrated and described herein. In examples involving a general-purpose computer, aspects of the disclosure transform the general-purpose computer into a special-purpose computing device when configured to execute the instructions described herein.

By way of example and not limitation, computer readable media comprise computer storage media and communication media. Computer storage media include volatile and nonvolatile, removable and non-removable memory implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules, or the like. Computer storage media are tangible and mutually exclusive to communication media. Computer storage media are implemented in hardware and exclude carrier waves and propagated signals. Computer storage media for purposes of this disclosure are not signals per se. Exemplary computer storage media include hard disks, flash drives, solid-state memory, phase change random-access memory (PRAM), static random-access memory (SRAM), dynamic random-access memory (DRAM), other types of random-access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that may be used to store information for access by a computing device. In contrast, communication media typically embody computer readable instructions, data structures, program modules, or the like in a modulated data signal such as a carrier wave or other transport mechanism and include any information delivery media.

The order of execution or performance of the operations in examples of the disclosure illustrated and described herein is not essential and may be performed in different sequential manners in various examples. For example, it is contemplated that executing or performing a particular operation before, contemporaneously with, or after another operation is within the scope of aspects of the disclosure. When introducing elements of aspects of the disclosure or the examples thereof, the articles “a,” “an,” “the,” and “said” are intended to mean that there are one or more of the elements. The terms “comprising,” “including,” and “having” are intended to be inclusive and mean that there may be additional elements other than the listed elements. The term “exemplary” is intended to mean “an example of.” The phrase “one or more of the following: A, B, and C” means “at least one of A and/or at least one of B and/or at least one of C.”

Having described aspects of the disclosure in detail, it will be apparent that modifications and variations are possible without departing from the scope of aspects of the disclosure as defined in the appended claims. As various changes could be made in the above constructions, products, and methods without departing from the scope of aspects of the disclosure, it is intended that all matter contained in the above description and shown in the accompanying drawings shall be interpreted as illustrative and not in a limiting sense.