Skip convolutions for efficient video processing

A method for video processing via an artificial neural network includes receiving a video stream as an input at the artificial neural network. A residual is computed based on a difference between a first feature of a current frame of the video stream and a second feature of a previous frame of the video stream. One or more portions of the current frame of the video stream are processed based on the residual. Additionally, processing is skipped for one or more portions of the current frame of the video based on the residual.

FIELD OF DISCLOSURE

Aspects of the present disclosure generally relate to energy efficient video processing via an artificial neural networks.

BACKGROUND

Artificial neural networks may comprise interconnected groups of artificial neurons (e.g., neuron models). The artificial neural network may be a computational device or represented as a method to be performed by a computational device.

Neural networks consist of operands that consume tensors and produce tensors. Neural networks can be used to solve complex problems, however, because the network size and the number of computations that may be performed to produce the solution may be voluminous, the time for the network to complete a task may be long. Furthermore, because these tasks may be performed on mobile devices, which may have limited computational power, the computational costs of deep neural networks may be problematic.

Convolutional neural networks are a type of feed-forward artificial neural network. Convolutional neural networks may include collections of neurons that each have a receptive field and that collectively tile an input space. Convolutional neural networks (CNNs), such as deep convolutional neural networks (DCNs), have numerous applications. In particular, these neural network architectures are used in various technologies, such as image recognition, pattern recognition, speech recognition, autonomous driving, and other classification tasks.

Neural networks also have numerous applications in image-based processing of videos or video streams such as human pose estimation, object detection, semantic segmentation, as well as video compression and denoising. Unfortunately, such video processing is computationally intensive which results in significant time and energy consumption.

SUMMARY

The present disclosure is set forth in the independent claims, respectively. Some aspects of the invention are described in the dependent claims.

In an aspect of the present disclosure, a method for video processing with an artificial neural network (ANN) is provided. The method includes receiving a video stream as an input at the artificial neural network. The method also includes computing a residual based on a difference between a first feature of a current frame of the video stream and a second feature of a previous frame of the video stream. Additionally, the method includes processing one or more portions of the current frame of the video stream based on the residual.

In another aspect of the present disclosure, an apparatus for video processing with an artificial neural network (ANN) is provided. The apparatus includes a memory and one or more processors coupled to the memory. The processor(s) are configured to receive a video stream as an input at the artificial neural network. The processor(s) are also configured to compute a residual based on a difference between a first feature of a current frame of the video stream and a second feature of a previous frame of the video stream. In addition, the processor(s) are configured to process one or more portions of the current frame of the video stream based on the residual.

In an aspect of the present disclosure, an apparatus for video processing with an artificial neural network (ANN) is provided. The apparatus includes means for receiving a video stream as an input at the artificial neural network. The apparatus also includes means for computing a residual based on a difference between a first feature of a current frame of the video stream and a second feature of a previous frame of the video stream. Additionally, the apparatus includes means for processing one or more portions of the current frame of the video stream based on the residual.

In a further aspect of the present disclosure, a non-transitory computer readable medium is provided. The computer readable medium has encoded thereon program code for video processing with an artificial neural network (ANN). The program code is executed by a processor and includes code to receive a video stream as an input at the artificial neural network. The program code also includes code to compute a residual based on a difference between a first feature of a current frame of the video stream and a second feature of a previous frame of the video stream. Furthermore, the program code includes code to process one or more portions of the current frame of the video stream based on the residual.

DETAILED DESCRIPTION

The word “exemplary” is used to mean “serving as an example, instance, or illustration.” Any aspect described as “exemplary” is not necessarily to be construed as preferred or advantageous over other aspects.

Neural networks have numerous applications in image-based processing of videos or video streams such as human pose estimation, object detection, semantic segmentation, as well as video compression and de-noising. Unfortunately, such video processing is computationally intensive which results in significant time and energy consumption.

Videos may have substantial similarity from one frame to the next. Because of such similarities, there may be a large amount of redundancy in the computations that are performed for consecutive frames. This may be particularly so when a neural network performs the video processing, because such networks may perform convolution operations for each pixel in each video frame. Aspects of the present disclosure are directed to skip convolutions to leverage the large amount of redundancies in video streams, in order to reduce computations and conserve energy. Skip convolutions adaptively adjust the computing with respect to the amount of information observed per frame. That is, convolutional kernels may be applied to salient regions of a video frame while skipping the regions that are substantially similar in consecutive frames. The saliency maps may be obtained based on the magnitude of changes between frames or by learning a gating function efficiently integrated into each convolution layer. In some aspects, skip convolution can incorporate various structures (e.g., a block-wise implementation) for efficient implementation of hardware platforms. Skip convolutions may also be incorporated in any image processing network to optimize the inference cost in stream settings.

FIG.1illustrates an example implementation of a system-on-a-chip (SoC)100, which may include a central processing unit (CPU)102or a multi-core CPU configured for processing a video with an artificial neural network (e.g., a neural end-to-end network). Variables (e.g., neural signals and synaptic weights), system parameters associated with a computational device (e.g., neural network with weights), delays, frequency bin information, and task information may be stored in a memory block associated with a neural processing unit (NPU)108, in a memory block associated with a CPU102, in a memory block associated with a graphics processing unit (GPU)104, in a memory block associated with a digital signal processor (DSP)106, in a memory block118, or may be distributed across multiple blocks. Instructions executed at the CPU102may be loaded from a program memory associated with the CPU102or may be loaded from a memory block118.

The SoC100may also include additional processing blocks tailored to specific functions, such as a GPU104, a DSP106, a connectivity block110, which may include fifth generation (5G) connectivity, fourth generation long term evolution (4G LTE) connectivity, Wi-Fi connectivity, USB connectivity, Bluetooth connectivity, and the like, and a multimedia processor112that may, for example, detect and recognize gestures. In one implementation, the NPU108is implemented in the CPU102, DSP106, and/or GPU104. The SoC100may also include a sensor processor114, image signal processors (ISPs)116, and/or navigation module120, which may include a global positioning system.

The SoC100may be based on an ARM instruction set. In an aspect of the present disclosure, the instructions loaded into the general-purpose processor102may include code to receive a video stream as an input at the artificial neural network. The general-purpose processor102may also include code to compute a residual based on a difference between a first feature of a current frame of the video stream and a second feature of a previous frame of the video stream. The general-purpose processor102may further include code to process of one or more portions of the current frame of the video stream based at least in part on the residual.

Deep learning architectures may perform an object recognition task by learning to represent inputs at successively higher levels of abstraction in each layer, thereby building up a useful feature representation of the input data. In this way, deep learning addresses a major bottleneck of traditional machine learning. Prior to the advent of deep learning, a machine learning approach to an object recognition problem may have relied heavily on human engineered features, perhaps in combination with a shallow classifier. A shallow classifier may be a two-class linear classifier, for example, in which a weighted sum of the feature vector components may be compared with a threshold to predict to which class the input belongs. Human engineered features may be templates or kernels tailored to a specific problem domain by engineers with domain expertise. Deep learning architectures, in contrast, may learn to represent features that are similar to what a human engineer might design, but through training. Furthermore, a deep network may learn to represent and recognize new types of features that a human might not have considered.

The connections between layers of a neural network may be fully connected or locally connected.FIG.2Aillustrates an example of a fully connected neural network202. In a fully connected neural network202, a neuron in a first layer may communicate its output to every neuron in a second layer, so that each neuron in the second layer will receive input from every neuron in the first layer.FIG.2Billustrates an example of a locally connected neural network204. In a locally connected neural network204, a neuron in a first layer may be connected to a limited number of neurons in the second layer. More generally, a locally connected layer of the locally connected neural network204may be configured so that each neuron in a layer will have the same or a similar connectivity pattern, but with connections strengths that may have different values (e.g.,210,212,214, and216). The locally connected connectivity pattern may give rise to spatially distinct receptive fields in a higher layer, because the higher layer neurons in a given region may receive inputs that are tuned through training to the properties of a restricted portion of the total input to the network.

One example of a locally connected neural network is a convolutional neural network.FIG.2Cillustrates an example of a convolutional neural network206. The convolutional neural network206may be configured such that the connection strengths associated with the inputs for each neuron in the second layer are shared (e.g.,208). Convolutional neural networks may be well suited to problems in which the spatial location of inputs is meaningful.

One type of convolutional neural network is a deep convolutional network (DCN).FIG.2Dillustrates a detailed example of a DCN200designed to recognize visual features from an image226input from an image capturing device230, such as a car-mounted camera. The DCN200of the current example may be trained to identify traffic signs and a number provided on the traffic sign. Of course, the DCN200may be trained for other tasks, such as identifying lane markings or identifying traffic lights.

The DCN200may be trained with supervised learning. During training, the DCN200may be presented with an image, such as the image226of a speed limit sign, and a forward pass may then be computed to produce an output222. The DCN200may include a feature extraction section and a classification section. Upon receiving the image226, a convolutional layer232may apply convolutional kernels (not shown) to the image226to generate a first set of feature maps218. As an example, the convolutional kernel for the convolutional layer232may be a 5×5 kernel that generates 28×28 feature maps. In the present example, because four different feature maps are generated in the first set of feature maps218, four different convolutional kernels were applied to the image226at the convolutional layer232. The convolutional kernels may also be referred to as filters or convolutional filters.

The first set of feature maps218may be subsampled by a max pooling layer (not shown) to generate a second set of feature maps220. The max pooling layer reduces the size of the first set of feature maps218. That is, a size of the second set of feature maps220, such as 14×14, is less than the size of the first set of feature maps218, such as 28×28. The reduced size provides similar information to a subsequent layer while reducing memory consumption. The second set of feature maps220may be further convolved via one or more subsequent convolutional layers (not shown) to generate one or more subsequent sets of feature maps (not shown).

In the example ofFIG.2D, the second set of feature maps220is convolved to generate a first feature vector224. Furthermore, the first feature vector224is further convolved to generate a second feature vector228. Each feature of the second feature vector228may include a number that corresponds to a possible feature of the image226, such as “sign,” “60,” and “100.” A softmax function (not shown) may convert the numbers in the second feature vector228to a probability. As such, an output222of the DCN200is a probability of the image226including one or more features.

In the present example, the probabilities in the output222for “sign” and “60” are higher than the probabilities of the others of the output222, such as “30,” “40,” “50,” “70,” “80,” “90,” and “100”. Before training, the output222produced by the DCN200is likely to be incorrect. Thus, an error may be calculated between the output222and a target output. The target output is the ground truth of the image226(e.g., “sign” and “60”). The weights of the DCN200may then be adjusted so the output222of the DCN200is more closely aligned with the target output.

In practice, the error gradient of weights may be calculated over a small number of examples, so that the calculated gradient approximates the true error gradient. This approximation method may be referred to as stochastic gradient descent. Stochastic gradient descent may be repeated until the achievable error rate of the entire system has stopped decreasing or until the error rate has reached a target level. After learning, the DCN may be presented with new images and a forward pass through the network may yield an output222that may be considered an inference or a prediction of the DCN.

The performance of deep learning architectures may increase as more labeled data points become available or as computational power increases. Modern deep neural networks are routinely trained with computing resources that are thousands of times greater than what was available to a typical researcher just fifteen years ago. New architectures and training paradigms may further boost the performance of deep learning. Rectified linear units may reduce a training issue known as vanishing gradients. New training techniques may reduce over-fitting and thus enable larger models to achieve better generalization. Encapsulation techniques may abstract data in a given receptive field and further boost overall performance.

FIG.3is a block diagram illustrating a deep convolutional network350. The deep convolutional network350may include multiple different types of layers based on connectivity and weight sharing. As shown inFIG.3, the deep convolutional network350includes the convolution blocks354A,354B. Each of the convolution blocks354A,354B may be configured with a convolution layer (CONV)356, a normalization layer (LNorm)358, and a max pooling layer (MAX POOL)360.

The convolution layers356may include one or more convolutional filters, which may be applied to the input data to generate a feature map. Although only two of the convolution blocks354A,354B are shown, the present disclosure is not so limiting, and instead, any number of the convolution blocks354A,354B may be included in the deep convolutional network350according to design preference. The normalization layer358may normalize the output of the convolution filters. For example, the normalization layer358may provide whitening or lateral inhibition. The max pooling layer360may provide down sampling aggregation over space for local invariance and dimensionality reduction.

The parallel filter banks, for example, of a deep convolutional network may be loaded on a CPU102or GPU104of an SoC100to achieve high performance and low power consumption. In alternative embodiments, the parallel filter banks may be loaded on the DSP106or an ISP116of an SoC100. In addition, the deep convolutional network350may access other processing blocks that may be present on the SoC100, such as sensor processor114and navigation module120, dedicated, respectively, to sensors and navigation.

The deep convolutional network350may also include one or more fully connected layers362(FC1and FC2). The deep convolutional network350may further include a logistic regression (LR) layer364. Between each layer356,358,360,362,364of the deep convolutional network350are weights (not shown) that are to be updated. The output of each of the layers (e.g.,356,358,360,362,364) may serve as an input of a succeeding one of the layers (e.g.,356,358,360,362,364) in the deep convolutional network350to learn hierarchical feature representations from input data352(e.g., images, audio, video, sensor data and/or other input data) supplied at the first of the convolution blocks354A. The output of the deep convolutional network350is a classification score366for the input data352. The classification score366may be a set of probabilities, where each probability is the probability of the input data including a feature from a set of features.

FIG.4is a block diagram illustrating an exemplary software architecture400that may modularize artificial intelligence (AI) functions. Using the architecture, applications may be designed that may cause various processing blocks of a system-on-a-chip (SoC)420(for example a CPU422, a DSP424, a GPU426and/or an NPU428) to support adaptive rounding as disclosed for post-training quantization for an AI application402, according to aspects of the present disclosure.

The AI application402may be configured to call functions defined in a user space404that may, for example, provide for the detection and recognition of a scene indicative of the location in which the device currently operates. The AI application402may, for example, configure a microphone and a camera differently depending on whether the recognized scene is an office, a lecture hall, a restaurant, or an outdoor setting such as a lake. The AI application402may make a request to compiled program code associated with a library defined in an AI function application programming interface (API)406. This request may ultimately rely on the output of a deep neural network configured to provide an inference response based on video and positioning data, for example.

A run-time engine408, which may be compiled code of a runtime framework, may be further accessible to the AI application402. The AI application402may cause the run-time engine, for example, to request an inference at a particular time interval or triggered by an event detected by the user interface of the application. When caused to provide an inference response, the run-time engine may in turn send a signal to an operating system in an operating system (OS) space, such as a Linux Kernel412, running on the SoC420. The operating system, in turn, may cause a continuous relaxation of quantization to be performed on the CPU422, the DSP424, the GPU426, the NPU428, or some combination thereof. The CPU422may be accessed directly by the operating system, and other processing blocks may be accessed through a driver, such as a driver414,416, or418for, respectively, the DSP424, the GPU426, or the NPU428. In the exemplary example, the deep neural network may be configured to run on a combination of processing blocks, such as the CPU422, the DSP424, and the GPU426, or may be run on the NPU428.

The application402(e.g., an AI application) may be configured to call functions defined in a user space404that may, for example, provide for the detection and recognition of a scene indicative of the location in which the device currently operates. The application402may, for example, configure a microphone and a camera differently depending on whether the recognized scene is an office, a lecture hall, a restaurant, or an outdoor setting such as a lake. The application402may make a request to compiled program code associated with a library defined in a SceneDetect application programming interface (API)406to provide an estimate of the current scene. This request may ultimately rely on the output of a differential neural network configured to provide scene estimates based on video and positioning data, for example.

A run-time engine408, which may be compiled code of a Runtime Framework, may be further accessible to the application402. The application402may cause the run-time engine, for example, to request a scene estimate at a particular time interval or triggered by an event detected by the user interface of the application. When caused to estimate the scene, the run-time engine may in turn send a signal to an operating system410, such as a Linux Kernel412, running on the SoC420. The operating system410, in turn, may cause a computation to be performed on the CPU422, the DSP424, the GPU426, the NPU428, or some combination thereof. The CPU422may be accessed directly by the operating system, and other processing blocks may be accessed through a driver, such as a driver414-418for a DSP424, for a GPU426, or for an NPU428. In the exemplary example, the differential neural network may be configured to run on a combination of processing blocks, such as a CPU422and a GPU426, or may be run on an NPU428.

As described, artificial neural networks (ANN) are useful for processing videos. However, videos may have large amounts of redundancy across video frames. As such, ANN-based video processing systems may perform the same convolution operation multiple times. Such processing is time consuming and results in significant energy consumption. Accordingly, aspects of the present disclosure are directed to avoiding processing or skipping performance of convolution operations of redundant portions of consecutive video frames. The skip convolution may be applied to video processing systems including convolutional neural networks. The neural network may learn to skip the processing of residual frames or a portion of frames within the video stream.

Rather than considering a video to be a sequence of still images, the video may be represented as a series of residual frames defined both for the input frames and for intermediate feature maps. A gating function may be employed to determine whether to process or skip each location in residual frames.

For example, a convolutional neural network (CNN) may be configured with L layers. At a given layer l, a kernel w∈co×ci×kh×kwis applied to a sequence of feature maps {xt}t=1T, xt∈ci×h×wto produce a sequence of feature maps {zt}t=1T, zt∈co×h×w, independently across time-steps, for each frame, such that:
zt=w*xt(1)
where h and w are the height and width of the input feature map, cois the output channel, ciis the input channel, and kh, kware the height and width of the kernel w.

As the convolution is a linear function, the output sequence may be obtained by convolving the kernel w and a difference between two consecutive feature maps:

zt=zt-1+w*(xt-xt-1)=zt-1+w*rt,(2)
where rtrepresents the residual frame as a temporal difference between the current feature map and the previous feature map, expressed as xt−xt−1. Although the decomposition in Equation 2 describes a single time step, it can, of course, be generalized to longer sequences. Because consecutive or sequential frames in a video may be highly correlated, the residual frame rtmay be very sparse, and carry a non-zero difference signal in fewer, and in some aspects, very few spatial locations. For every kernel support (e.g., pattern of zero elements) filled with zero values in rt, the corresponding output will also be zero. As such, a convolution operation for the subsequent feature maps may be skipped. For example, the convolution operation for the subsequent feature map xtmay be avoided or skipped by copying values from a previous convolution zt−1to the value of the current convolution zt.

FIG.5is a diagram illustrating an example residual determination500, in accordance with aspects of the present disclosure. Referring toFIG.5, a video sequence including frames (e.g., frames with sequential timings)502and504is shown. Frame502provides an image of a boy walking at a time t. Frame504provides an image of a boy walking at a previous time t−1. While the pose or position of the boy appears to change from frame502and frame504, much of the image in frame502(e.g., the background) appears to be the same or very similar to the image of frame504. Each of the frames502,504may be supplied in a sequence of inputs of a video stream. To improve processing (e.g., object detection or human pose estimation), a residual or difference between the frames502,504may be computed. The residual frame may put a strong prior on the informative regions to be processed versus non-informative regions that may be skipped. As shown inFIG.5, a residual frame506may be determined by computing the difference between the input features for frame502xtand frame504xt−1. A video processing system, including a CNN (e.g., deep convolutional network350), may be configured to skip processing of residual portions or portions of the frame502that are determined to be the same as the previous frame504. That is, based on the residual rt, convolution operations for one or more regions or portions of frame502may be avoided. Instead, the result of the convolution operation ztfor such regions of frame502may be equal to the result of the convolutions operations zt−1performed for a corresponding region of frame504. Thus, convolutions for processing frame504may be limited to regions with non-zero rt.

In some aspects, residuals that are sparse may still include many locations with small non-zero values that may be sufficient to prevent skipping processing for such locations. To further improve processing efficiency, a gating function may be included at each convolutional layer, for example, to impose skipping of residual frames. The gating function may be defined as g:ci×h×w→{0,1}h×w. The gating function may predict a binary mask indicating locations that may be processed and which may be skipped as based on the input residual rt, and in some aspects, based only on the input residual rt. As such, the gating function g may generate, based on the input residual rt, a binary decision for each output pixel. With this binary decision, the gating function output determines whether the convolution operation is to be performed or skipped for specific pixel locations of a video frame. Accordingly, the convolution output may be adapted as:
{tilde over (z)}t={tilde over (z)}t−1+g(rt)⊙(w*rt)  (3)
where ⊙ indicates a broadcasted Hadamard product and the ˜ symbol indicates that {tilde over (z)}tis an approximation of zt. The Hadamard product is a binary operation that takes two matrices of the same dimensions and produces another matrix of the same dimension as the operands, where each element i,j is the product of elements i,j of the original two matrices. Thus, Equation 3 characterizes a skip convolution model, where a convolution operation is performed only for the previous frame of the sequence, and a gating function g controls whether to perform convolution operations for one or more portions of a kernel w for a current frame, based on the residual rt.

FIG.6is a block diagram illustrating an example skip convolution layer600, in accordance with aspects of the present disclosure. Referring toFIG.6, the skip convolution layer600includes a gating function604. The gating function604may enforce the sparsity of the residual. The gating function604may receive the residual as input and may predict a binary variable (e.g., 0 or 1) to sparsify the residual. That is, the gating function learns to zero out or mask negligible residuals (e.g., residuals where the input does not change the output (e.g., a prediction)).

As shown inFIG.6, the skip convolution layer600receives as inputs a feature xtcorresponding to a previous frame and a feature corresponding to a previous frame xt−1. The difference between the feature xtand the previous feature xt−1is computed at a difference node606to determine a residual rt. The residual rtis supplied to the gating block604. The gating block604applies a gating function to determine whether to process or mask out each portion of the frame corresponding to the feature. For example, where the residual rtfor a portion of the feature xtis negligible (e.g., <0.5), the gating block604may output a zero to a convolution node608to skip the convolution operation with a weight w supplied via the convolution block602. As such, the convolution node608outputs a zero to a summing node610, which also receives the prior computed output zt−1. Thus, the summing node610outputs ztas equal to the value of the previous output zt−1.

On the other hand, where the residual is not negligible (e.g., >0.5), the summing node610outputs ztas equal to the sum of a convolution operation for the corresponding portion of the feature xtand the previous output zt−1to produce output zt.

In some aspects, the skip convolution layer600may replace a convolution layer of a CNN. For example, the skip convolution layer600may be used in place of one or more convolution layers356of the deep convolutional network350, shown inFIG.3.

In some aspects, the gating function g may be configured post-training (e.g., without training) or the gating function may be learned. For post-training gates, the gating function may determine whether to skip a residual based on its magnitude. For instance, the gating function may skip a residual based on a scalar threshold E. Such gates may be referred to as Norm gates. In one example, the gating decision may be determined by applying the scalar threshold E to the norm of each output pixel:
g(rt,w,∈)=round(σ(∥w*rt∥p−∈)),  (4)
where σ(•) indicates a sigmoid function and p represents the order of the norm, and the norm is computed over all channels for each position. However, such a gating strategy may result in the computation of the convolution at each pixel of the residual, which would reintroduce inefficiency. Accordingly, the post-training gating function of Equation 4 may be approximated by considering the norm of each kernel supported in the residual rt, as follows:
g(rt,∈)=round(θ(∥rt∥p−∈)).  (5)

The gating strategy may be referred to as input-norm gates. In Equation 5 (and below, in Equation 7), the norm of each local support in the residual rtis computed by applying an absolute value function to the residual rttensor, and taking the sum within the di×kh×kwneighborhood, where diis the dimension.

A more accurate approximation may be achieved without computing a full convolution, by involving the norm of the weight matrix w. Further, by considering Young's inequality, an upper bound on the norm of the convolution of two vectors f and g may be determined:

By following Equation 6, a more precise approximation may be defined based on the norms of both the input residual rtand the weight matrix w. This gating strategy may be referred to as output-norm gates:
gl(rt,w,∈)=round(σ(∥w∥p·∥rt∥p−∈)).  (7)
where the norm ∥w∥pis computed over all four dimensions.

In one implementation, the order p for both input-norm and output-norm gates may be set to 1 (e.g., llnorm), and the margin E is shared between all layers. More flexible strategies, such as layer-specific E, may also yield better results at the cost of more hyper-parameter adjustment. One advantage of post-training gates is that by employing post-training gates, skip convolution may be readily applied to a pre-trained CNN. In doing so, the efficiency of a pre-trained CNN may be improved without training or fine-tuning.

The post-training gates effectively suppress computation in static regions, but may remain sensitive to significant changes regardless of their content. For instance, because post-training gates are task-agnostic, a change in the background may trigger unnecessary computations that may not impact model performance (e.g., object recognition or tracking). To address this issue, a trainable gating module may learn when to skip residuals. Each layer l in the CNN may be paired with a lightweight gating module fl(rt; ϕl), parametrized by ϕl, composed by a convolution with the same kernel size, stride, and dilation as layer l, but featuring a single output channel. This additional model outputs unnormalized scores, which may be transformed to pixel-wise Bernoulli distributions by applying a sigmoid function. During training, the gating function gl(rt; ϕl) samples binary decisions from such distributions, whereas a 0.5 threshold, for example, may be applied during test:

During training, a Gumbel re-parametrization and a straight-through gradient estimator may enable back propagation through the sample procedure. In order to enforce sparsity, an auxiliary objective S(Φ) may be applied over all gating parameters Φ=[ϕ1, . . . , ϕL]:

S⁡(Φ)=β⁢1T⁢∑t=1T⁢∑l=1L⁢ml·μ⁡(fl⁡(rt;ϕl)),(9)
where β is a training hyperparameter, T is a temperature, μ(•) is the mean function, and mlrepresents layer specific coefficients, dependent on each layer's multiply-accumulate (MAC) counts. A sparsity objective may be added to the model's loss function (e.g., depending on the downstream task). While the sparsity objective seeks sparsity, the loss function instructs gates to be active in presence of meaningful residuals (and thus skip one or more convolution operations).

When training skip convolution layers with learned gates, the stream processing model may acquire a recurrent architecture, and thus may be trained over fixed-length frame sequences. During testing, the model can either output its predictions recursively to the end of the sequence or instantiate key frames at arbitrary temporal intervals, where a regular convolution is applied and the recurrent state resets.

In some aspects, structured gating of residual feature maps may also be applied. For instance, agreement of sparsity within regions may be suitable for optimized on-device implementations. Thus, an inductive bias in our learned gating modules may be implemented by down-sampling the gating decisions with a structuring element (e.g., max pooling or convolution) to provide block-wise gating.

FIG.7is a block diagram illustrating an example block-wise implementation700of skip convolution with learned gates, in accordance with aspects of the present disclosure. Referring toFIG.7, in the block-wise implementation700, a set of output filters Cout, are provided to generate convolution outputs704. An additional convolution filter706is added to map an input tensor702to generate a gate block708. The gate block708may, for example, be a single output channel binary. The gate block708may be applied to indicate which locations of the output704are to be masked. Black areas712of the gate block708may indicate that a mask is to be applied and white areas714may indicate an active region. Accordingly, the convolution operation may be performed on the active regions of the output704as shown in block710. Because convolutions for video processing may be performed block-wise (e.g., 8×8 block structures), rather than pixel-by-pixel, the block-wise implementation700beneficially predicts the gates on a block-wise basis to leverage hardware parallelization.

Additionally, block structures may be leveraged to reduce memory overhead involved in the gathering and scattering of input and output tensors. Furthermore, hardware platforms may perform convolutions distributed over patches (e.g., 8×8) and thus may not leverage any fine-grained spatial sparsity smaller than such block sizes. Accordingly, the aspects of the present disclosure may be extended to generate structured sparsity by adding a down-sampling and an up-sampling function on the predicted gates. For instance, a max pooling layer with the kernel size and stride of b may be added. The max pooling layer may be followed by a nearest neighbor up-sampling with the same scale factor of b. This may enforce the predicted gates to have a b×b structure.

FIG.8is a block diagram illustrating an example implementation of skip convolution on a graphics processing unit800, in accordance with aspects of the present disclosure. Referring toFIG.8, aspects of the present disclosure may further increase the image processing speed using the example implementation800by conducting the convolutions in a vectorized fashion. As shown, a set of feature tensors802are supplied. The feature tensors802may be separated into blocks810. For ease of illustration one block810is shown. However, it should be understood that feature tensors802includes multiple blocks810. Each of the blocks810may be transformed into a column vector to provide a feature matrix. A mask may be calculated, for example based on the difference between features of the current frame and features of the previous frame to produce a mask vector806. A mask vector806is applied to select locations of the feature matrix for which a convolution operation is to be performed.

A weight matrix808may be correspondingly transformed into row vectors. The selected locations of the feature matrix may be gathered or grouped and a matrix multiplication operation may be performed with the corresponding weight matrix808. The outputs may, in turn, be scattered to the corresponding columns of the output matrix. Accordingly, the output matrix may then be reformed to an output block804.

FIG.9illustrates a method900for video processing via an artificial neural network, in accordance with aspects of the present disclosure. As shown inFIG.9, at block902, a video stream is received as an input at the artificial neural network. For instance, as described with reference toFIG.6, the skip convolution layer600receives as inputs a feature xtcorresponding to a current frame and a feature corresponding to a previous frame xt−1.

At block904, a residual is computed based on a difference between a first feature of a current frame of the video stream and a second feature of a previous frame of the video stream. As described with reference toFIG.6, the difference between the feature xtof the current frame and the feature xt−1of the previous frame is computed at a difference node606to determine a residual rt.

At block906, one or more portions of the current frame of the video stream are processed based on the residual. For example as described with reference toFIG.6, the gating block604applies a gating function to determine whether to process or mask out each portion of the frame corresponding to the feature. In some examples, where the residual rtis not negligible (e.g., >0.5), a convolution operation for the corresponding portion of the feature xtof the current frame may be performed and summed with the previous output zt−1to produce output zt.

Additionally, at block908, the process optionally refrains from or skips processing one or more portions of the current frame of the video based on the residual. For example, as described with reference toFIG.6, where the residual rtfor a portion of the feature xtis negligible (e.g., <0.5), the gating block604may output a zero to a convolution node608to skip the convolution operation with a weight w. As such, the convolution node608outputs a zero to a summing node610. Thus, the summing node610outputs ztthat is equal to the value of the previous output zt−1.

Implementation Examples are provided in the following numbered clauses:1. A method for video processing with an artificial neural network (ANN), comprising:receiving a video stream as an input at the artificial neural network;computing a residual based on a difference between a first feature of a current frame of the video stream and a second feature of a previous frame of the video stream; and processing one or more portions of the current frame of the video stream based on the residual.2. The method of clause 1, in which the one or more portions of the current includes only salient regions of the current frame.3. The method of clause 1 or 2, further comprising applying a convolution kernel to only the salient regions of the current frame.4. The method of any of clauses 1-3, further comprising determining the salient regions based on whether the residual is greater than a predetermined threshold value.5. The method of any of clauses 1-4, further comprising refraining from processing at least one portion of the current frame of the video stream based on the residual.6. The method of any of clauses 1-5, in which a first output corresponding to the at least one portion of the current frame is set equal to a second output corresponding to at least one portion of the previous frame.7. The method of any of clauses 1-6, further comprising:comparing the residual to a predefined threshold value; and applying a mask to the corresponding second features based on the comparing.8. The method of any of clauses 1-7, further comprising learning a gating function to apply a mask to one or more portions of the current frame based on the residual.9. The method of any of clauses 1-8, further comprising generating a saliency map based on the gating function.10. The method of any of clauses 1-9, further comprising adaptively adjusting an amount of computation performed in processing the video stream based on an amount of information observed per frame.11. An apparatus for video processing with an artificial neural network (ANN), comprising:a memory; andat least one processor coupled to the memory, the at least one processor being configured:to receive a video stream as an input at the artificial neural network;to compute a residual based on a difference between a first feature of a current frame of the video stream and a second feature of a previous frame of the video stream; andto process one or more portions of the current frame of the video stream based on the residual.12. The apparatus of clause 11, in which the one or more portions of the current frame includes only salient regions of the current frame.13. The apparatus of clause 11 or 12, in which the at least one processor is further configured to apply a convolution kernel to only the salient regions of the current frame.14. The apparatus of any of clauses 11-13, in which the at least one processor is further configured to determine the salient regions based on whether the residual is greater than a predetermined threshold value.15. The apparatus of any of clauses 11-14, in which the at least one processor is further configured to refrain from processing at least one portion of the current frame of the video stream based on the residual.16. The apparatus of any of clauses 11-15, in which a first output corresponding to the at least one portion of the current frame is set equal to a second output corresponding to at least one portion of the previous frame.17. The apparatus of any of clauses 11-16, in which the at least one processor is further configured:to compare the residual to a predefined threshold value; andto apply a mask to the corresponding second features based on the comparing.18. The apparatus of any of clauses 11-17, in which the at least one processor is further configured to learn a gating function to apply a mask to one or more portions of the current frame based on the residual.19. The apparatus of any of clauses 11-18, in which the at least one processor is further configured to generate a saliency map based on the gating function.20. The apparatus of any of clauses 11-19, in which the at least one processor is further configured to adaptively adjust an amount of computation based on an amount of information observed per frame.21. An apparatus for video processing with an artificial neural network (ANN), comprising:means for receiving a video stream as an input at the artificial neural network;means for computing a residual based on a difference between a first feature of a current frame of the video stream and a second feature of a previous frame of the video stream; andmeans for processing one or more portions of the current frame of the video stream based on the residual.22. The apparatus of clause 21, in which the one or more portions of the current frame includes only salient regions of the current frame.23. The apparatus of clause 21 or 22, further comprising means for applying a convolution kernel to only the salient regions of the current frame.24. The apparatus of any of clauses 21-23, further comprising means for determining the salient regions based on whether the residual is greater than a predetermined threshold value.25. The apparatus of any of clauses 21-24, further comprising means for learning a gating function to apply a mask to one or more portions of the current frame based on the residual.26. A non-transitory computer readable medium having encoded thereon program code for video processing with an artificial neural network (ANN), the program code being executed by a processor and comprising:program code to receive a video stream as an input at the artificial neural network;program code to compute a residual based on a difference between a first feature of a current frame of the video stream and a second feature of a previous frame of the video stream; andprogram code to process one or more portions of the current frame of the video stream based on the residual.27. The non-transitory computer readable medium of clause 26, in which the one or more portions of the current frame includes only salient regions of the current frame.28. The non-transitory computer readable medium of clause 26 or 27, in which the at least one processor is further configured to apply a convolution kernel to only the salient regions of the current frame.29. The non-transitory computer readable medium of any of clauses 26-28, in which the at least one processor is further configured to determine the salient regions based on whether the residual is greater than a predetermined threshold value.30. The non-transitory computer readable medium of any of clauses 26-29, in which the at least one processor is further configured to learn a gating function to apply a mask to one or more portions of the current frame based on the residual.

In one aspect, the receiving means, the computing means, processing means and/or the refraining means may be the CPU102, program memory associated with the CPU102, the dedicated memory block118, fully connected layers362, NPU428, and/or the routing connection processing unit216configured to perform the functions recited. In another configuration, the aforementioned means may be any module or any apparatus configured to perform the functions recited by the aforementioned means.