ON-DEVICE INFERENCE METHOD FOR MULTI-FRAME PROCESSING IN A NEURAL NETWORK

A method for optimizing multi-frame processing model of a neural network includes: receiving a plurality of input frames by a processing engine that is configured to execute a multi frame processing model, the multi frame processing model including a plurality of convolution layers; selecting a pre-determined number of frames from the received plurality of frames for processing by the plurality of convolution layers; determining, as a sequence of frames, at least a preceding frame and a plurality of following frames amongst the selected pre-determined number of frames; removing the preceding frame by processing the sequence of frames using a plurality of filters in the multi frame processing model; and concatenating the plurality of following frames in an order, to the plurality of input frames for subsequent receiving by the multi frame processing model.

TECHNICAL FIELD

The disclosed embodiments generally relates to image processing, and more particularly to a method for optimizing multi-frame processing model of a neural network for providing fast on-device inference for a multi-frame architecture.

BACKGROUND

Multi-frame architecture refers to a design and a structure of computational models or algorithms that leverage information from multiple frames or a sequence of data. Recurrent Neural Networks (RNNs), Convolutional Neural Networks (CNNs), and multi-frame Deep Neural Networks (DNNs) inference architecture are some examples of the multi-frame architecture. The methodology implemented in the multi-frame architecture may be referred to as multi-frame model.

The multi-frame model is designed to handle and exploit temporal and spatial information across multiple frames for various tasks, such as video processing, image processing, and/or sequential data analysis. Further, the multi-frame model relies on the utilization of multiple frames to extract useful information, like temporal and spatial information, to solve complex problems. Due to the inherent dependency of the multiple frames and corresponding useful information obtained from the multiple frames, the multi-frame processing is more effective than a single-frame processing technique. Consequently, the multi-frame model has found widespread adoption in applications, such as High Dynamic Range (HDR), Noise Removal (NR), Super-Resolution (SR), and low light photography. Further, the multi-frame architecture is also implemented in various electronic devices, such as cameras, video-capturing devices, and/or monitoring devices.

Conventionally, the multi-frame architecture maintains a queue in a buffer based on a number of frames required for inference. Accordingly, the required number of frames are extracted from a sequence of input frames and stored in the buffer by maintaining the queue. The multi-frame architecture processes all the required number of frames in the queue to obtain a result. However, this approach necessitates the processing of all the frames in the queue in higher dimensions with repetitive computations. For example, the multi-frame Deep Neural Network (DNN) inference architecture deploys buffer queuing and a sub-optimal operation in graphs (i.e. models).FIG.1illustrates a queuing mechanism in a conventional multi-frame DNN inference architecture, according to a conventional technique. According to the multi-frame DNN inference architecture, frames from the sequence of inputs need to be buffered in a queue before they can be processed. This queuing allows for sequential extraction and processing of frames in a correct order. For example, in the exemplary queuing mechanism100depicted inFIG.1, consider that Frame1to Frame n is inserted into a queue101. Further, consider that during the first processing cycle, frame1to frame5are considered for processing in the queue101. Hence, the frame1to frame5are extracted from the queue101and given to a multi-frame DNN inference model103for processing. Thereafter, during a next processing cycle, consider that the frame2to frame6are considered for processing and given to the multi-frame DNN model103. However, during these two processing cycles, the frame2to frame5are common, hence the frame2to frame5are processed again. This leads to redundant processing of frames between two processing cycles. This redundant operation further introduces delays in the overall process. Accordingly, the buffering process and the extraction process consumes unnecessary computation and device cycles. As the buffering process and the extraction process involves data bus, it leads to a consumption of extra power cycles in the electronic device. Accordingly, in power and resource constraint electronic devices (e.g., the camera), an inference architecture, that is sub-optimal, can result in inferior performance and higher power usage. This further impacts the overall user experience of the electronic device. Thus, there is a need for a solution that helps to avoid redundant computations.

FIG.2illustrates a first layer processing in a higher dimension in the conventional multi-frame DNN inference architecture, according to a conventional technique. The multi-frame DNN inference often involves the processing of data in higher dimensions, such as handling spatio-temporal information. For example, in a first layer processing211, a concatenation (concat)201operation is performed in higher dimensions width (w) and height (h) that are large for high-resolution applications such as High Definition (HD), Full High definition (FHD), Ultra High Definition (4K), and application with resolution 2048×1080 (2K). The output is then processed by successive layers which also process the output in higher dimensions of width (w) and height (h). For example, as shown inFIG.2, output_1from the concat201is processed by the convolution (conv) layers203. The first operation in the multi-frame DNN inference model103is performed in the higher dimension (height/width). However, this is sub-optimal for on-device computational elements, like Digital Signal Processing (DSP)/Neural Processing Unit (NPU), which are primarily optimized to process in a higher channel than the higher dimension. Hence, such architecture is sub-optimal for on-device computational elements. This largely affects the performance of the on-device computational elements. Accordingly, there is a need to process the combined information in lower dimensions and higher channels for better performance of the on-device computational elements.

FIG.3illustrates pre-processing and post-processing in the multi-frame DNN inference architecture with pipeline execution, according to a conventional technique. The data used in multi-frame DNN inference may need to be pre-processed or post-processed to ensure that it is in the expected format for the neural network. The pre-processing and post-processing steps can involve data conversion, resizing, normalization, quantization, dequantization, or other transformations to prepare the frames or sequences for input into the multi-frame DNN inference model. These additional operations add computational overhead and may impact the overall efficiency of the inference process. For example, as can be seen inFIG.3, input frames, i.e., frame1to frame n are pre-processed by quantizing at step301. The quantized frames are then processed at step303by performing a series of operations involving concatenation and convolution operations. Thereafter, at step305, the processed output is further dequantized to convert the frames into the original form. Accordingly, these pre-processing and post-processing operations take extra computational cycles which result in high power and poor performance of the electronic device. Accordingly, there is a need to eliminate aforesaid pre-processing and post-processing steps.

Thus, it is desired to address at least the above-mentioned disadvantages or other shortcomings or at least provide a useful alternative for optimizing multi-frame processing in the multi-frame architecture.

SUMMARY

The disclosed embodiments are provided to introduce a selection of concepts, in a simplified format, which is further described in the detailed description. This summary is neither intended to identify key or essential concepts nor is it intended for determining the scope of the disclosed embodiments.

A method for optimizing multi-frame processing model of a neural network may include: receiving a plurality of input frames by a processing engine that is configured to execute a multi frame processing model, the multi frame processing model including a plurality of convolution layers; selecting a pre-determined number of frames from the received plurality of frames for processing by the plurality of convolution layers; determining, as a sequence of frames, at least a preceding frame and a plurality of following frames amongst the selected pre-determined number of frames; removing the preceding frame by processing the sequence of frames using a plurality of filters in the multi frame processing model; and concatenating the plurality of following frames in an order, to the plurality of input frames for subsequent receiving by the multi frame processing model.

The processing engine may be a component of the neural network that is at least one of a recurrent neural network (RNN), a convolutional neural network (CNN), or a deep neural networks (DNN).

A method for optimizing multi-frame processing model of a neural network may include: receiving a first set of concatenated frames and a plurality of previously processed input frames by a processing engine that is configured to execute a multi frame processing model, the first set of concatenated frames including a current input frame; generating a second set of concatenated frames by discarding an oldest previously processed input frame of the plurality of previously processed input frames that is concatenated within the first set of concatenated frames; and providing the second set of concatenated frames to the multi frame processing model for concatenating the second set of concatenated frames with the current input frame.

The multi frame processing model may include a plurality of predetermined filters. The generating the second set of concatenated frames may include: performing one or more convolution operations on the first set of concatenated frames using one or more filters of the plurality of predetermined filters; and discarding the oldest previously processed input frame based on a result of the performing the one or more convolution operations.

The one or more convolution operations may be performed using all of the plurality of predetermined filters.

A filter weight of each of the plurality of predetermined filters may correspond to a dummy weight. The dummy weight may be associated with each of the plurality of predetermined filters via a dummy weight interleaving method.

The method may further include: downsampling the current input frame and each of the second set of concatenated frames to a specific resolution; and generating a third set of concatenated frames by concatenating the down sampled current input frame and the down sampled second set of concatenated frames.

The downsampling the current input frame to the specific resolution may include performing a convolution on the current input frame using a filter weight of the one or more filters.

The multi frame processing model may further include a plurality of convolution layers. The method may further include adjusting a weight of a convolution layer of the plurality of convolution layers that receives the first set of concatenated frames based on a change in the specific resolution and a change in an input channel.

The multi frame processing model may include a clipping layer. The method may further include: determining a clipping range of the clipping layer based on a bit depth and an input range associated with one or more input frames within the second set of concatenated frames; and limiting one or more outputs of the processing engine to a pre-determined range based on the determined clipping range of the clipping layer.

The processing engine may be a component of the neural network that is at least one of a recurrent neural network (RNN), a convolutional neural network (CNN), or a deep neural networks (DNN).

An apparatus for optimizing multi-frame processing model of a neural network may include: a processing engine that is configured to execute a multi frame processing model. The processing engine being further configured to: receive a first set of concatenated frames and a plurality of previously processed input frames, the first set of concatenated frames including a current input frame; generate a second set of concatenated frames by discarding an oldest previously processed input frame of the plurality of previously processed input frames that is concatenated within the first set of concatenated frames; and provide the second set of concatenated frames to the multi frame processing model for concatenating the second set of concatenated frames with the current input frame.

The multi frame processing model may include a plurality of predetermined filters. The processing engine being configured to generate the second set of concatenated frames includes being configured to: perform one or more convolution operations on the first set of concatenated frames using one or more filters of the plurality of predetermined filters; and discard the oldest previously processed input frame based on a result of the performed one or more convolution operations.

The one or more convolution operations may be performed using all of the plurality of predetermined filters.

A filter weight of each of the plurality of predetermined filters may correspond to a dummy weight. The dummy weight may be associated with each of the plurality of predetermined filters via a dummy weight interleaving method.

The processing engine may be further configured to: downsample the current input frame and each of the second set of concatenated frames to a specific resolution; and generate a third set of concatenated frames by concatenating the down sampled current input frame and the down sampled second set of concatenated frames.

The processing engine being configured to downsample the current input frame to the specific resolution includes being configured to perform a convolution on the current input frame using a filter weight of the one or more filters.

The multi frame processing model may further include a plurality of convolution layers. The processing engine may be further configured to adjust a weight of a convolution layer of the plurality of convolution layers that receives the first set of concatenated frames based on a change in the specific resolution and a change in an input channel.

The multi frame processing model may include a clipping layer. The processing engine may be further configured to: determine a clipping range of the clipping layer based on a bit depth and an input range associated with one or more input frames within the second set of concatenated frames; and limit one or more outputs of the processing engine to a pre-determined range based on the determined clipping range of the clipping layer.

The processing engine may be a component of the neural network that is at least one of a recurrent neural network (RNN), a convolutional neural network (CNN), or a deep neural networks (DNN).

To further clarify the advantages and features of the disclosed embodiments, a more particular description will be rendered by reference to specific embodiments thereof, which are illustrated in the appended drawings. It is appreciated that these drawings depict only typical embodiments and are therefore not to be considered limiting of its scope. The disclosed embodiments will be described and explained with additional specificity and detail in the accompanying drawings.

Further, skilled artisans will appreciate those elements in the drawings are illustrated for simplicity and may not have necessarily been drawn to scale. For example, the flow charts illustrate the method in terms of the most prominent steps involved to help to improve understanding of aspects of the disclosed embodiments. Furthermore, in terms of the construction of the device, one or more components of the device may have been represented in the drawings by conventional symbols, and the drawings may show only those specific details that are pertinent to understanding the disclosed embodiments so as not to obscure the drawings with details that will be readily apparent to those of ordinary skill in the art having the benefit of the description herein.

DETAILED DESCRIPTION

For the purpose of promoting an understanding of the principles of the disclosed embodiments, reference will now be made to the embodiment illustrated in the drawings and specific language will be used to describe the same. It will nevertheless be understood that no limitation of the scope of the disclosed embodiments are thereby intended, such alterations and further modifications in the illustrated system, and such further applications of the principles of the disclosed embodiments as illustrated therein being contemplated as would normally occur to one skilled in the art to which the disclosed embodiments relate.

It will be understood by those skilled in the art that the foregoing general description and the following detailed description are explanatory of the disclosed embodiments and are not intended to be restrictive thereof.

Referring now to the drawings, and more particularly toFIGS.4to12, where similar reference characters denote corresponding features consistently throughout the figures, there are shown preferred embodiments.

FIG.4illustrates a block diagram of an electronic device401for optimizing a multi-frame processing model of a neural network, according to an embodiment as disclosed herein. Examples of the electronic device401may include but are not limited to, a smartphone, a tablet computer, a Personal Digital Assistance (PDA), a camera, a monitoring device, an image capturing device, a video recording device, or any other electronic device capable of processing images or video data.

The electronic device401may include an input unit403, a processor(s)405, an output unit409, and a memory415coupled with each other. The processor(s)405is coupled with a processing engine407. The processing engine407includes a multi-frame processing engine411. A detailed explanation of each of the components as mentioned above will be explained in the forthcoming paragraphs.

The input unit403may be configured to receive a plurality of frames. In a non-limiting example, the plurality of frames413may correspond to multiple frames from images or video data. The plurality of frames may be provided as an input to the muti-frame processing engine411of the processing engine407. The plurality of frames may be alternatively referred to as frames without deviating from the scope of the disclosed embodiments.

The processor405operates with the memory415, the processing engine407, and the output unit409. The processor405may be a single processing unit or a number of units, all of which could include multiple computing units. The processor405may be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, logical processors, virtual processors, state machines, logic circuitries, and/or any devices that manipulate signals based on operational instructions. Among other capabilities, the processor405is configured to fetch and execute computer-readable instructions and data stored in the memory415. The processor120may include one or a plurality of processors, maybe a general-purpose processor, such as a central processing unit (CPU), an application processor (AP), a graphics-only processing unit such as a graphics processing unit (GPU), a visual processing unit (VPU), and/or an Artificial intelligence (AI) dedicated processor such as a neural processing unit (NPU).

The processing engine407is implemented by processing circuitry such as logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, active electronic components, optical components, and/or hardwired circuits, and may optionally be driven by firmware. The circuits may, for example, be embodied in one or more semiconductor chips, or on substrate supports such as printed circuit boards.

The memory415may store instructions to be executed by the processor405for optimizing the multi-frame processing engine411. The memory415may include non-volatile storage elements. Examples of such non-volatile storage elements may include magnetic hard discs, optical discs, floppy discs, flash memories, or forms of electrically programmable memories (EPROM) or electrically erasable and programmable (EEPROM) memories. In addition, the memory415may, in some examples, be considered a non-transitory storage medium. The term “non-transitory” may indicate that the non-transitory storage medium is not embodied in a carrier wave or a propagated signal. However, the term “non-transitory” should not be interpreted that the memory415is non-movable. In some examples, the memory415can be configured to store larger amounts of information. In certain examples, the non-transitory storage medium may store data that can, over time, change (e.g., in Random Access Memory (RAM) or cache). The memory415can be an internal storage unit, or it can be an external storage unit of the electronic device401, a cloud storage, or any other type of external storage.

The multi-frame processing engine411may be implemented in a neural network such as CNN, DNN, and/or RNN.FIG.5illustrates a detailed architecture of the multi-frame processing engine411. The multi-frame processing engine411may include a plurality of predetermined filters501, a plurality of convolution layers505, a concatenation layer503, a custom weight convolution layer507, and a509.

Tor the plurality of predetermined filters501a filter weight of each of the plurality of predetermined filters501may correspond to a dummy weight. The dummy weights may be alternatively referred to as custom weights or filter weights. Further, the plurality of predetermined filters501may be alternatively referred to as predetermined filters or filters without deviating from the scope of the disclosed embodiments. The dummy weight may be assigned to each of the plurality of predetermined filters501using a dummy weight interleaving method for preserving features in any of the frames. For example, consider a case where a first feature in a 2×2 block matrix of the input frames is required to be preserved. Accordingly, the dummy weights of the predetermined filters501are assigned as “1,0,0,0”. That is to say, a corresponding filter weight of a corresponding filter whose feature is required to be preserved is assigned as “1” and the rest of the filters may be assigned as zero. Thus, when a value of the first feature is multiplied by the assigned dummy weight (i.e. '1), the corresponding first feature gets preserved. Further, for the rest of the features when multiplied with assigned dummy weights (i.e. 000), the corresponding features of the rest of the features in the 2×2 block matrix gets zero, hence not preserved.

The concatenation layer503may be configured to concatenate one or more frames. The frames may be either processed frames or unprocessed frames. For example, the frames such as raw frames, or original frames, that have not gone through any processing such as convolution or concatenation operations, may be referred to as unprocessed frames. Further, the frames that have gone through any processing like convolution, or any other operation, may be referred to as processed frames. The concatenation operation, performed by the concatenation layer503, may be alternatively referred to as ‘concat’ or concat operation.

The plurality of convolution layers505may be configured to perform a convolution operation for the received plurality of input frames. The convolution operation, performed by the convolution layer505, may be alternatively referred to as ‘conv’ or conv operation. As explained above, for preserving the features the convolution operation is performed using the filter weights. In a non-limiting example, the convolution operation may be performed by, for example, a stride convolution, a dilated convolution, a transposed convolution, and/or a grouped convolution. An explanation of the convolution operation is made with reference to the stride convolution. However, the disclosed methodology may be performed by any convolution operation. In general, the stride convolution is a type of convolution operation where a kernel is moved over an input feature map with a step size greater than one. This results in a downsampled output feature map compared to the input. The stride value determines how many pixels the kernel shifts in each direction during the convolution process. A stride convolution with the dummy weight interleaving method may be performed on the input frames. This operation preserves the width and the height while preserving an original information.

An additional convolution layer referred to herein as the custom weight convolution layer507may be implemented in the multi-frame processing engine411.

The custom weight convolution layer507may be configured to determine a custom weight of the predetermined filter501and also a number of filters required in the custom weight convolution layer507. Further, an additional convolution layer, referred to herein as the clip layer509, is implemented in the multi-frame processing engine411. The clip layer509may be implemented in order to eliminate the conventional preprocessing and post-processing step. Accordingly, the clip layer509is configured to limit one or more outputs of the processing engine407to a pre-determined range. The detailed operation of determination of the custom weight and operation of the clip layer is explained in the following paragraphs.

Referring back toFIG.4, the output unit409may be configured to provide output processed by the processing engine407.

A framework that converts the multi-frame processing model into a device-optimal multi-frame inference model through an auxiliary feedback loop may be provided. Accordingly, an output from the auxiliary feedback loop is provided as an input to an original multi-frame processing model which reduces computing time and power consumption. Detailed implementation and working will be explained in the forthcoming paragraphs.

FIG.6illustrates a flow chart depicting a method600for optimizing the multi-frame processing model, according to an embodiment. The method600may be implemented at the electronic device401. The method600is explained with reference toFIGS.4-11. The method600may be implemented in any multi-frame processing model during offline mode to obtain a device-optimal model for inference.FIG.7illustrates an operational flow for optimizing the multi-frame processing model, according to an embodiment. Referring toFIG.7, the auxiliary feedback loop700A is added in an original loop700of the multi-frame processing engine411.FIGS.6and7are described in conjunction with each other for the sake of brevity.

Initially, during the processing of images or video streams, the processing engine407may be configured to receive the plurality of the frames413as explained inFIGS.4and5. As explained above inFIG.5, generally, in the multi-frame inference model, for every processing cycle, a predetermined number of frames are selected for processing. Subsequently, after selecting the predetermined number of frames, the selected frames are processed in the original loop700of the multi-frame processing engine411. For example, the predetermined number of frames are considered as three, according to the example embodiment as shown inFIG.7. Accordingly, the selected frames undergo a first processing cycle through the original loop700.

At step601, the processing engine407may be configured to receive a first set of concatenated frames including a current input frame and a plurality of previously processed input frames. The plurality of previously processed input frames may be the frames that are modified and received from the previous execution or previous processing cycle or the first processing cycle after processing through the original loop700. As an example, the first set of concatenated frames may be the frames that are processed from the predetermined number of frames. Further, the current input frame may be an incoming frame that is received by the processing engine407during each processing cycle of images or video. The current input frame may be alternatively referred to as the current frame.

The step601may be explained by referring toFIG.7. As depicted inFIG.7, at block703, the concatenation layer503of the processing engine407, may be configured to receive the first set of concatenated frames including the current input frame. In an example scenario, frame1and frame2are depicted as F1and F2, respectively. The frames F1and F2may be the plurality of previously processed input frames717received by the concatenation layer503at the block703. Further, the frame3, depicted as F3, is the current frame at block705which is received by the concatenation layer503at the block703. According to an example embodiment, the frames, i.e., the previously processed input frames and the current input frames are concatenated to output concatenated frames719, i.e., F1, F2, and F3are concatenated at the block703. Thus, according to the given example, the first set of concatenated frames719are frames F1, F2, and F3that are arranged as a sequence.

After, generating the first set of concatenated frames, at step603, the multi-frame processing engine411of the processing engine407, may be configured to generate a second set of concatenated frames by discarding a previously processed input frame that is oldest among the plurality of previously processed input frames, and is concatenated within the first set of concatenated frames. Referring to the same example scenario as considered above, in the step603a second set of concatenated frames721is generated by discarding the frame F1which is the oldest among the plurality of previously processed input frames717and is concatenated within the first set of concatenated frames719. As the frames are arranged in a sequence, the oldest frame in the sequence may be identified and discarded. Accordingly, the second set of concatenated frames may include the frames F2and F3in the sequence. In the example embodiment, only a single frame is shown to be removed or discarded. However, according to an alternate embodiment, more than one frame may be removed.

The generation of the second set of concatenated frames may include steps603-1and603-2. Accordingly, for generating the second set of concatenated frames, the convolution operation may be performed by the plurality of convolution layers505. In a non-limiting example, the stride convolution with the dummy weight interleaving method may be utilized for generating the second set of concatenated frames. An explanation of the dummy weight interleaving method using predetermined filters and filter weight to preserve the feature is explained in the above paragraphs and with reference to theFIG.8. Therefore, for the sake of brevity, the detailed explanation is omitted here.

Accordingly, at step603-1, the plurality of convolution layers505, of the multi-frame processing engine411in the processing engine407, are configured to perform one or more convolution operations on the first set of concatenated frames using one or more filters among the plurality of predetermined filters501. Thereafter, at step603-2, the plurality of convolution layers505, of the multi-frame processing engine411in the processing engine407, are configured to discard the previously processed input frame that is oldest among the plurality of previously processed input frames based on the performed one or more convolution operations. The convolution operations using the plurality of predetermined filters501are explained with reference to the example scenario depicted inFIG.8.

FIG.8illustrates an example scenario depicting the convolution operations using the plurality of predetermined filters based on the dummy weight interleaving method, according to an embodiment. In an example, an output of the concatenation operation703of the input frames is represented as a matrix801. A stride convolution with the dummy weight interleaving method is performed on the matrix801. The convolution operation is a multiplication of the input with the filter weights. Therefore, for example, if the input is “F1, F2, F3” and the filter is “0,1,0” then the output for this filter process is given by equation 1.

Accordingly, if the filter is “0,0,1” then the output is given by equation 2.

From the above equations, the filter weight is required to be set in order to retain the feature corresponding to the respective frames. Therefore, in the example scenario ofFIG.7, for removing or discarding the feature of the corresponding frame F1, the respective filter weight of F1is set to zero and the respective filter weight of the frames F2and F3is set to 1. Accordingly, the filter weights for filter1and filter2correspond to (0,1,0) and (0,0,1).

FIG.9illustrates a flow diagram depicting a method900for setting the filter weight and the generation of the custom weight, according to an embodiment. The method900may be an operation flow for setting the filter weight for preserving the feature of the respective frames and generating the custom weight for a new output. The new output may refer to the output generated after discarding or removal of the oldest frames among the predetermined frames in the auxiliary feedback loop700A. The method900may be implemented in the electronic device401ofFIG.4. Initially, at step901, the processing engine407may be configured to determine the predetermined number of frames for processing by the plurality of convolution layers505. The predetermined number of frames may be received as an input for inference. Thereafter, at step903, the processing engine407is configured to determine the number of filters required in the custom weight convolution layer507. The determination of the number of filters required in the custom weight convolution layer507is given by equation 3.

a number of filters required in the custom weight convolution layer=a number of previous frames required for processing   (3)

At step905, the processing engine may be configured to create a matrix of size 1*1*number of frames needed−1. The matrix hence created is depicted in the example ofFIG.8at block801. Thereafter, at step907, the processing engine407may be configured to initialize the matrix with zeros. Then, at step909, the processing engine407may be configured to determine whether the number of filters remaining is more than zero. Based on the determination as ‘no’, i.e., the number of filters remaining is not more than zero, at step911, one or more weights for the custom weight convolution layer507for the new output are generated. The new output is the second set of concatenated frames which will be used for the next processing in conjunction with the next current input frame. The generated custom weights for the new output are then utilized for processing at block711ofFIG.7. Further, based on the determination as ‘yes’ i.e. the number of filters remaining is more than zero, at step915, the processing engine407is configured to set the respective filter to 1. Then at step913, the filter count is increased and the processing engine407again performs the step909. Accordingly, the custom weight is assigned to the plurality of predetermined filters using the dummy weight interleaving method.

The generated second set of concatenated frames and the current input frame may be downsampled.FIG.10illustrates a custom kernel generation method1000for downsampling the generated second set of concatenated frames and the current input frame, according to an embodiment. The method1000may be an operation flow for a custom kernel generation for downsampling the generated second set of concatenated frames and the current input frame. The downsampling may be a method to reduce a resolution of the frame to a lower resolution. The method1000is implemented in the electronic device401ofFIG.4. Initially, at step1001, the processing engine407may be configured to determine a required reduction in the dimension of the second set of concatenated frames and the current frame to a specific resolution. As an example, the requirement of the reduction is based on the model complexity. Based on the determination, at step1003, the processing engine407may be configured to compute a kernel shape. In a non-limiting example, the kernel shape, the number of filters, and a stride required for convolution are determined. The equations for the kernel shape, the number of filters, and the stride are provided below:

Number of filters=reduction scale×reduction scale   (5)

In a non-limiting example, consider that the required reduction in the dimension is two. Then based on the equations 4, 5, and 6 the kernel shape, the number of filters, and stride are found to be as below:

Number of filters=2×2=4

Thereafter, at step1005, the processing engine407may be configured to initialize the filter weights with zero. Since few positions will have values, the rest will be dummies interleaved with zeros, hence initial values are initialized to zeros. At step1007, the processing engine407may be configured to determine whether the number of remaining filters is greater than zero. The determination of whether the number of remaining filters is greater than zero is performed based on equation 3. Thus, based on the determination as ‘yes”, i.e., the number of remaining filters is greater than zero, at step1009the filter weights are set. Accordingly, the reshaped kernel weights at step1010are provided for the convolution operation to the custom weight convolution layer507when the number of remaining filters is not greater than zero. The reshaped kernel weight is given to block707ofFIG.7for further processing. Accordingly, at step1111, a filter count is increased.

The processing engine407may be configured to generate a third set of concatenated frames by concatenating the down-sampled current input frame and the down-sampled second set of concatenated frames at block703. The third set of concatenated frames is further processed through the original loop700to provide the output.

Referring back toFIG.6, after performing various operations at step603, the method600, at step605, further includes providing the second set of concatenated frames to the multi-frame processing model411for concatenating the second set of concatenated frames with a next receiving input frame. Accordingly, the new output that was generated as the second set of concatenated frames is provided as an input for the next processing cycle. Thus, the new output act as an auxiliary feedback output that preserves the data while dropping the dimension. This auxiliary feedback output eliminates redundant processing as present in the conventional solutions. Referring toFIG.7, the second set of concatenated frames F2and F3is provided to the block701for utilizing it for the next processing cycle. Further, a new input frame, for example, frame F4(not shown) then becomes the current frame. The rest of the procedure remains the same as explained above.

The processing engine407may be configured to adjust a weight of the convolution layer of the plurality of convolution layers505that receives the first set of concatenated frames based on a change in the specific resolution and an input channel. Referring toFIG.7, at the block709, the convolution layer with adjusted original weight is provided to a next convolution layer713. As the previous frame and the current frames at block705are downsampled, the weights of the plurality of convolution layers505are required to be adjusted in the subsequent operation. Therefore, based on a required specific resolution and the input channels, the weights are adjusted for the successive operation.

A clip layer509may be provided at an output stage in order to eliminate the need for pre-processing and post-processing steps. Referring toFIG.7, a clip layer 1715is provided at the output of the original loop700, and a clip layer 2501is provided at the auxiliary feedback loop700A. Accordingly, the processing engine407may be configured to determine a clipping range of the clipping layer based on a bit depth and an input range associated with one or more input frames within the second set of concatenated frames. In particular, while determining the clip layer range, a value of input or output and the bit depth is considered. In case the value range and bit depth are different, the custom quantization values can be used to adjust bias and scale. Further, the processing engine407may be further configured to limit one or more outputs of the processing engine to a pre-determined range based on the determined clipping range of the clipping layer. For example, if the input range is from −10 to 200 and the input frame bit depth is 8 bits, i.e., a max number of levels is 256, then the clip min and max range is 0 and 255 respectively.

FIG.11illustrates a flow chart for optimizing the multi-frame processing model, according to an embodiment.FIG.11depicts a method1111implemented at the electronic device401, according to some embodiments. The method1111is analogous to method600ofFIG.6, therefore for the sake of brevity, a detailed explanation of the same is omitted here. Further, the explanation of the methods as depicted inFIGS.5-10is also applicable in method1111. At step1101the processing engine404may be configured to receive the plurality of input frames. Thereafter, at step1103, the processing engine407may be configured to select the pre-determined number of frames from the received plurality of frames for processing by the plurality of convolution layers505. After that, at step1105, the processing engine407may be configured to determine, as the sequence of frames, at least a preceding frame and a plurality of following frames amongst the selected pre-determined number of frames. Here the preceding frame is the oldest frame. After that, at step1107, the processing engine407may be configured to remove or discard the preceding frame by processing the sequence of frames using the plurality of filters in the multi-frame processing model411. Thereafter, at step1109, the processing engine407may be configured to concatenate the plurality of following frames, other than the preceding frames, in a sequence, to the plurality of input frames for subsequent receiving into the multi-frame processing model411.

FIG.12illustrates a conversion of an original multi-frame processing model to device optimized model using the method ofFIG.6, according to an embodiment. Initially, an original multi-frame processing model which is pre-trained is considered as a starting point for the multi-frame processing. The frames may be concatenated at block1201and weights are adjusted based on the dummy weight interleaving method to preserve the output at block1203. The process at blocks1201and1203is performed at the original loop700as shown in FIG.7. This may be considered as the first processing cycle. The output quality of the original loop700is checked at1205with respect to a desired output quality. If the output is of the desired output quality, then the same weights and parameters are provided to the auxiliary feedback loop700A ofFIG.7for further processing. That is to say, at block1209, the multi-frame processing engine411adds the auxiliary feedback loop700A for custom output with the same weights and parameters that is to be utilized in the next processing cycle. If the output is not of the desired output quality, then the multi-frame processing engine is then trained with the new operation for further processing through the auxiliary feedback loop700A to generate a device-optimized model.

The disclosed methodology eliminates the redundant processing of input frames based on the application of discarding of the oldest frames. Further, by adding clip layers at the output further eliminates the computation of post-processing.

Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one ordinary skilled in the art. The system, methods, and examples provided herein are illustrative only and not intended to be limiting.

While specific language has been used to describe the present subject matter, any limitations arising on account thereto, are not intended. As would be apparent to a person in the art, various working modifications may be made to the method to implement the inventive concept as taught herein. The drawings and the forgoing description give examples of embodiments. Those skilled in the art will appreciate that one or more of the described elements may well be combined into a single functional element. Alternatively, certain elements may be split into multiple functional elements. Elements from one embodiment may be added to another embodiment.

The embodiments disclosed herein can be implemented using at least one hardware device and performing network management functions to control the elements.