Patent Description:
On <NPL>) described a system that uses an autoencoder architecture to learn a joint video/language embedding space for e.g. activity recognition. On <NPL>) described jointly modelling the visual and audio components of a video signal and in one application separates on- and off-stream audio streams using spectrograms. On <NPL>) described a two-stream inflated 3D convolutional neural network employing optical flow fields. On <NPL>) presented a slidedeck that illustrates residual learning to avoid needlessly touching uncorrupted pixels. In <NPL>) disclosed a cascaded residual convolutional autoencoder, estimating differences between corrupted/incomplete (input) data and complete (target) data.

Preferred embodiments are further defined in the dependent claims. The processed video data may ultimately be derived from a video camera or other video image generating device such as a LIDAR system, but may be input to the system from data storage or over a network.

The processing decomposes input video data into two or more sets of decomposed video data, one which represents reconstructed video and one or more others which represent unwanted features such as reflections, shadows, a color of illuminating light, and the like. The sets of decomposed video data may be referred to as layered representations of the input video data. The output video data may be output, for example, to a data store, communications network, and/or user interface.

Thus in implementations an input video including one or more undesired features such as reflections, shadows, occlusions and the like may be decomposed into two (or more) sequences of video image frames, one containing the undesired features the other containing video image frames in which the undesired features are suppressed or removed. Thus in some implementations only one of the sets of decomposed video data is needed. In other implementations both sequences of decomposed video image frames may contain useful information, and both may be retained.

In some implementations the system inputs a batch of video data representing the sequence of input video image frames and outputs one or more batches of video data derived from one or more of the sets of decomposed video data. Thus the system may operate offline. In other implementations the system may operate online, in a streaming mode. For example the system may process batches of video data sequentially, with an output delay corresponding to the batch duration. In another streaming mode the system may start by processing a first batch of video data for a first time segment, to provide corresponding output video data, and thereafter may output video frames one at a time. In this latter mode the system may retain some intermediate computation results, for example so as not to have to fully re-compute convolutions or optical flow (see later).

The encoder and decoder neural networks are termed predictor neural networks because they form part of a system which predicts the video decomposition e.g. the reconstructed video without the undesired features.

The neural network system further comprises a corrector 3D spatio-temporal encoder neural network, coupled to an output of the predictor 3D spatio-temporal decoder neural network. The corrector 3D spatio-temporal encoder neural network is configured to receive each of the sets of decomposed video data and to encode the respective sequences of decomposed video image frames into a second set of latent variables representing a compressed version of the sequences of decomposed video image frames. The neural network system further comprises a corrector 3D spatio-temporal decoder neural network to receive and process the second set of latent variables to generate two or more sets of correction video data, that is, a set of correction video data for each set of decomposed video data. The neural network system further comprises a combiner to combine each set of decomposed video data with a respective set of correction video data to provide two or more sets of combined video data. The video output data then comprises one or more of the sets of combined video data. The combiner may comprise a summer to add set of decomposed video data to a respective set of correction video data.

The combination of the corrector 3D spatio-temporal encoder neural network and corrector 3D spatio-temporal decoder neural network acts as a corrector neural network system. Thus the corrector neural network system takes the sets of decomposed video data from the predictor 3D spatio-temporal decoder neural network and apply a correction to improve the decomposition. The data from the different sets of decomposed video data from the predictor may be mixed and recombined in the corrector neural network system, to provide a correction to the sets of decomposed video data from the predictor to improve the decomposition in the output video data. The predictor neural networks and corrector neural network system are trained end-to-end as described later.

In some implementations the system includes multiple, i.e. chained, corrector neural network systems to apply successive corrections to the sets of decomposed video data from the predictor to improve the decomposition in the output video data. Thus each of the corrector neural network systems may be configured to provide a set of correction video data for each set of decomposed video data. The combiner may be configured to combine, e.g. sum, each set of decomposed video data with a respective set of correction video data from each of the corrector neural network systems.

The encoder and decoder neural networks are 3D spatio-temporal neural networks because they process 3D spatio-temporal data, that is data with two spatial dimensions and a time dimension, although in principle more (or fewer) spatial dimensions could be present. In this context different color channels, where present, are not considered additional dimensions as such.

The 3D spatio-temporal encoder neural network comprises a 3D convolutional neural network. This may be similar to a 2D convolutional neural network in which the 2D filters (kernels) have been inflated to 3D by endowing them with an additional temporal dimension. The receptive field in the time dimension may, for example, be chosen according to the task so as to approximately match growth in time (with increasing depth within the network) with motion in space of features within the images. That is, the "motion" of features in time as captured by a 3D convolution filter may approximately match the motion of features in the video. This may be achieved, for example, by adjusting the stride in time and/or pooling. For example there may be spatial pooling but no temporal pooling in one or more lower (closer to the input) layers of the 3D spatio-temporal encoder neural network.

In some implementations an optical flow image determination system is included to determine a sequence of optical flow image frames from the sequence of input video image frames, in some implementations an optical flow image frames for each input video image frame (apart, potentially from a first and/or last input video image frame). This may implement any suitable technique, e.g. a differential technique based on local and/or global first and optionally second derivatives of pixel values with respect to time and/or x- and y-displacement, or a TV-L1 algorithm (total variation flow field regularization, L<NUM> norm for data fidelity). An optical flow image may comprise, for example a set of x- and y- pixel displacement values for each pixel location. Alternatively, for example, it may comprise x- and y-components of a vector mapping motion of a pixel from one frame to one or more later frames. The optical flow may be unidirectional, either forwards or backwards, or bidirectional. Optionally a global, e.g. mean, motion estimation component may be subtracted to compensate for global motion. Then the predictor 3D spatio-temporal encoder neural network may comprise a pair of convolutional neural networks, a first convolutional neural network to process the sequence of input video image frames and a second convolutional neural network to process the sequence of optical flow image frames; and a combiner to combine outputs from the first and second convolutional neural networks to provide the first set of latent variables. Each of the first and second convolutional neural networks may comprise a 3D spatio-temporal convolutional neural network.

The predictor 3D spatio-temporal decoder neural network comprises a 3D de-convolutional neural network, configured to implement a series of transposed convolutions. The predictor 3D spatio-temporal decoder neural network may have a multichannel output such that the sets of decomposed video data are provided by different channels of the multichannel output. Thus the two or more sets of decomposed video data may be outputs are from same layer of the decoder neural network. This can help maintain internal consistency between the different sequences of decomposed video image frames.

In some implementations the predictor 3D spatio-temporal encoder neural network and the predictor 3D spatio-temporal decoder neural network each include a plurality of neural network layers and one or more skip (or residual) connections between neural network layers of corresponding resolution. The architecture of the combined predictor 3D spatio-temporal encoder and decoder neural networks may be similar to the U-Net neural network architecture, described by <NPL>(although there may be one or more fully-connected layers).

In some implementations the predictor 3D spatio-temporal decoder neural network is configured to generate three or more sets of decomposed video data. This can help to produce better decompositions if the best two are selected. Thus there may be included an output selection system to automatically select two of the sets of decomposed video data for the output video data.

In this context the "best" two may be, for example, the two sequences of decomposed video image frames which are most different from (distant to) one another. Thus the output selection system may be configured to determine a difference metric between the sets of decomposed video data, and to select for output the two sets of decomposed video data with the greatest mutual difference metric. For example the output selection system may determine the difference metric for each possible combination of the sets of decomposed video data. The difference metric may be any suitable metric, for example based upon the inverse of any of many video similarity metrics; optionally it may operate in a video embedding space in which may it may comprise, for example, a dot product metric, or a Euclidean distance, between the embeddings.

In another aspect there is provided a method of (unsupervised) training of the neural network system. The method comprises generating a training data set comprising a plurality of training videos by, for each of the training videos, combining different first and second videos e.g. with randomly selected relative weights. The training data set may be enhanced by random spatial and/or temporal cropping and/or flipping and the like. The method further comprises providing the training data set to the neural network system to generate output video data comprising two of the sets of decomposed video data. The method further comprises training the neural network system by adjusting parameters of the neural networks using a permutation invariant loss function.

The permutation invariant optimization function is dependent upon a reconstruction loss. The reconstruction loss defines, for each training video, a difference between the sets of decomposed video data and the first and second videos. The reconstruction loss is invariant as to which of the first and second videos is reconstructed by which set of decomposed video data.

The training may comprise backpropagating gradients of the loss function through the neural network system to update values (i.e. weights) of the neural network parameters. The training is performed end-to-end, including one or more of the corrector neural network systems where implemented.

The permutation invariant loss function may comprise a minimum, over the sets of decomposed video data (i.e. with respect to permutation of the first and second videos), of the sum of a first term and a second term. The first term may represent a difference between the first video and one of the sets of decomposed video data; the second term may represent a difference between the second video and a different one of the sets of decomposed video data. For example the training loss may be as described later; the particular reconstruction loss described later is not essential. This general form of loss may be applied where there are two, three or more sets of decomposed video data provided by the neural network system. In some implementations the reconstruction loss includes a term representing a difference between spatial gradients in corresponding video image frames of compared videos, i.e. between a frame of the decomposed video data and a frame of whichever one of the first and second videos it is compared with. This can help to distinguish videos by placing weight on edge-type features, which may be harder to distinguish than regional features.

There is also provided a computer-implemented method of processing input video data representing input video image frames to decompose the input video data into two or more sets of decomposed video data each representing a respective sequence of decomposed video image frames, the sequences of decomposed video image frames representing a decomposition of the input video image frames. The method comprises receiving a sequence of input video image frames. The method further comprises encoding the sequence of input video image frames using a predictor 3D spatio-temporal convolutional encoder neural network, into a first set of latent variables representing a compressed version of the input video image frames. The method further comprises processing the first set of latent variables using a predictor 3D spatio-temporal transposed convolutional decoder to generate two or more sets of decomposed video data representing respective sequences of decomposed video image frames. The method further comprises providing output video data derived from one or more of the sets of decomposed video data.

The subject matter described in this specification can be implemented in particular embodiments so as to realize one or more of the following advantages.

The neural network systems described herein improves the quality of, or restore, video data by removing undesired video features, for example reflections, shadows, lighting, occlusions and the like. They can also be used for color correction. This is because a video with a color tint decomposes into a first sequence of video frames comprising the color-corrected video and a second sequence of video frames comprising essentially just the removed color tint (and essentially no spatial features). Some implementations of the system can be trained unsupervised, using unlabeled training data (although this does not preclude the use of labelled training data). There are many applications of the systems and methods described to cleaning or editing video data, for example to remove artifacts from videos caused by colored windows, dirty mirrors, smoke, rain, shadows, clouds and so forth. The techniques may also be used to reconstruct portions of video frames temporarily occluded by another object. This may be used to "rescue" a video which might otherwise by difficult or expensive to capture again. Implementations of the system also have similar applications in cleaning video employed by robots, self-driving land, air or sea vehicles, or mechanical agents in general. Whilst in many instances the video may be captured from a camera, the systems and methods described may also be applied to video data from other image sensors such as LIDAR. The systems and methods described may also be applied to scientific, medical, and other video data.

The systems and methods described are more useful than other techniques because they are more generally applicable, that is unlike previous techniques they do not rely on very specific assumptions regarding the video data to be processed such as a degree or type of motion, and they may produce fewer artefacts in their output video. The systems and methods described are able to decompose, and hence improve, video in situations where other approaches perform less well, because the described techniques may use both motion within a video sequence and semantic content of the sequence to determine the decomposed video image frames. Thus the systems and methods described are able to restore video better and/or faster than previous systems, and hence may be able to restore video with reduced memory and/or processing requirements.

In some cases the systems and methods described succeed where other approaches fail, e.g. video which includes people or other objects moving around, and video which has been captured by a shaky video camera. Thus, for example, implementations of the systems and methods described may be used to improve the robustness of perceptual front-ends for safety critical systems operating in complex environments e.g. for robots or autonomous vehicles.

Like reference numbers and designations in the drawings indicate like elements.

This specification generally describes a neural network system for decomposing input video data into two or more sets of decomposed video data. The sets of decomposed video data may be considered to define layered representations of the input video data, for example one layer representing cleaned or reconstructed video and another layer representing factored out features such as unwanted shadows, reflections and the like.

The input video data may comprise a sequence of images derived from any source, real or artificial. In some applications the input video data may be derived from a visible light camera or LIDAR sensor; in others the images may represent non-visible images, such as infrared, ultraviolet, radio, or X-ray images or images derived from other than light e.g. neutron images; or synthetic images constructed from multiple different types of sensors. The images may be derived, for example, from a robot or autonomous or semi-autonomous vehicle, or from a scientific instrument or apparatus e.g. a microscope or telescope.

The video decomposition may be used to process the sequence of images to remove unwanted features or enhance the visibility of a useful part of the video. Such unwanted features may include reflections, shadows, effects resulting from imaging through a colored or dirty window, smoke or rain, and effects due to colored lighting. Sometimes, for example in the case of reflections, a scene may include multiple objects with independent motions. Alternatively the video decomposition may be used to discover features, information, or video layers the presence of which might be difficult to detect or unsuspected.

<FIG> shows an overview of such a computer-implemented neural network system <NUM> for decomposing input video data. The neural network system <NUM> comprises a 3D spatio-temporal encoder neural network <NUM> having an input <NUM> to receive a sequence of input video image frames, and one or more outputs <NUM> to provide a compressed i.e. encoded representation of the input video image frames. The neural network system <NUM> further comprises a 3D spatio-temporal decoder neural network <NUM> having an input <NUM> to receive the compressed representation of the input video image frames and an output <NUM> comprising a set of outputs 124a,b each to provide a respective set of decomposed video data. These neural networks are 3D in the sense that they process monochrome or color video data comprising two spatial dimensions and one time dimension.

The 3D spatio-temporal encoder neural network comprises a 3D convolutional neural network and the 3D spatio-temporal decoder neural network comprises a 3D transposed convolutional neural network. The neural network system <NUM> is referred to as a predictor neural network system and followed by a corrector neural network system, as described later.

The neural network system <NUM> may be trained using synthetically generated training videos. <FIG> shows an example of a synthetically generated training video V generated by combining a first video V<NUM> and a second video V<NUM>. Each of these videos may comprise a total of T frames with height H, width W and <NUM> RGB channels and the first and second videos may be combined additively in a ratio α to (<NUM> - α) where α is a blending value in the range [<NUM>,<NUM>], for example according to V = (<NUM> — α)V<NUM> + αV<NUM>.

In the visual domain reflections combine additively in this way but other features such as shadows and highly specular surfaces do not. Nonetheless it has been found that when the neural network system is trained with a large and diverse set of blended videos it is able to factor out features such as shadows, e.g. because a shadow in one video will tend to darken the blended video.

During training the neural network system <NUM> includes a training engine <NUM>. The training engine <NUM> trains the neural network system <NUM> so that the outputs 124a,b, denoted O<NUM>, O<NUM>, reconstruct the original video data V<NUM>, V<NUM>, subject to a permutation. That is, it is not defined which output reconstructs which original training video. Rather than try to enforce a particular output to reconstruct a particular video the neural network system <NUM> is trained with a training loss which is permutation invariant. For example the loss <IMG> may be the minimum of a reconstruction loss l<NUM> for O<NUM> reconstructing V<NUM> and O<NUM> reconstructing V<NUM> and a reconstruction loss l<NUM> for O<NUM> reconstructing V<NUM> and O<NUM> reconstructing V<NUM>. The reconstruction loss may comprise any distance metric between the videos, e.g. a per-pixel L1 or L2 loss optionally including a spatial gradient term as described later.

The neural network system <NUM> may use cues such as different motion fields and semantic content to reconstruct the original videos, as well as more subtle cues such as color and blurring. Surprisingly, therefore, a neural network system of this type, trained with additively blended videos, can perform video decomposition on input video to separate out undesired features and recover a clean video sequence, even though the undesired features may not themselves combine additively with the clean video sequence.

<FIG> shows a predictor-corrector neural network system <NUM> for decomposing input video data. In <FIG> a predictor neural network system <NUM> comprises first 3D spatio-temporal encoder and decoder neural networks <NUM>, <NUM> as previously described. The predictor-corrector neural network system <NUM> also includes a corrector neural network system <NUM> which comprises a second 3D spatio-temporal encoder neural network <NUM> and a second 3D spatio-temporal decoder neural network <NUM>, each of which are respectively similar to the first 3D spatio-temporal encoder and decoder neural networks <NUM>, <NUM>. The decomposed video data outputs <NUM> of the predictor neural network system <NUM> provide an input to the corrector neural network system <NUM>, which provides an output <NUM> comprising correction video data Δ to the output <NUM>, i.e. a correction for each of the set of outputs 124a,b. The correction is combined, e.g. summed by summer <NUM>, with the output of the predictor neural network system <NUM>, to provide combined i.e. corrected video data <NUM>. Denoting the output <NUM> of the by <NUM>, a final output, O, of the predictor-corrector neural network system <NUM> may be given by O = Õ + Δ.

In implementations the predictor neural network system <NUM> and the corrector neural network system <NUM> do not share weights. The predictor and corrector neural network systems <NUM>, <NUM> are trained end-to-end using the process described with reference to <FIG> and <FIG> later; a two-stage training process is not needed.

The predictor-corrector neural network system <NUM> of <FIG> may be extended as illustrated, by one or more further corrector neural network systems <NUM> and one or more further summers <NUM>, in a chain to provide a further corrected video data output.

The 3D spatio-temporal decoder neural network <NUM> of <FIG> and <FIG> may provide more than two outputs. This is helpful because of inherent uncertainty in the problem of video deconstruction; multiple different decompositions may satisfactorily reconstruct the input, even though only two videos may have been blended for training. Providing more than two outputs allows the 3D spatio-temporal decoder neural network <NUM> to generate multiple potential decompositions.

To generate n sets of decomposed video data 3D spatio-temporal decoder neural network <NUM> may be configured to provide n output channels e.g. from a last layer before the output. For example where the input video data has dimension T × H × W × <NUM>, the output <NUM> may have dimension T × H × W × 3n defining a set of n decomposed video data outputs, where n may be equal to or greater than <NUM>.

The example implementation of a predictor-corrector neural network system shown in <FIG> illustrates a 3D spatio-temporal decoder neural network <NUM> which provides n outputs; in such a system the 3D spatio-temporal decoder neural network <NUM> also provides n outputs.

The neural network systems <NUM>, <NUM> may include an output selection system <NUM> to automatically select a subset n' of the set of n output channels when decomposing an input video data after training, where n' = <NUM>, n' = <NUM>, or n' > <NUM>. For example the system may be configured to select the n', e.g. two, most dissimilar output channels (video layers) according to a dissimilarity measure. The dissimilarity measure may comprise any distance metric between the sets of decomposed video data, e.g. a per-pixel L1 or L2 loss, optionally including a spatial gradient term.

Experimentally it has been found that predicting more than two layers, by using more than two output channels, can provide substantially better decomposition. That is, two video layers selected from a set of n > <NUM> can provide more accurate un-mixing that when decomposing video into just two layers overall.

In one example implementation each of the 3D spatio-temporal encoder neural networks <NUM>, <NUM> may have an 13D architecture, as described in <NPL>. Thus <FIG> shows an example 3D spatio-temporal encoder neural network <NUM> based on an 13D architecture, in which the input video data <NUM> comprising a set (sequence) of T image frames provides an input to a first 3D convolutional neural network <NUM>. The input video data <NUM> is also provided to an optical flow determiner <NUM> configured to determine a set of optical flow image frames <NUM> e.g. from pairs of image frames using any of a range of techniques e.g. a TV-L1 algorithm. The set of optical flow frames <NUM> provides an input to a second 3D convolutional neural network <NUM>, an output of which is combined e.g. averaged or otherwise linearly combined, with an output of the first 3D convolutional neural network <NUM> by a combiner <NUM> to provide the output <NUM>.

Referring back to <FIG> and <FIG>, each 3D spatio-temporal encoder-decoder neural network pair may include one or more skip connections between intermediate neural network layers of the encoder and decoder neural networks, e.g. in a U-Net-type architecture (arXiv:<NUM>).

Merely by way of example <FIG> shows one possible implementation of the neural network system <NUM> which comprises 3D spatio-temporal encoder and decoder neural networks <NUM> with skip connections and a U-Net architecture. In <FIG> s = stride, k = kernel shape, c = channels, and n is the number of output channels. The I3D network block is described in the I3D paper (ibid); reference may also be made to the associated code on GitHub. <FIG> shows one possible implementation of the predictor-corrector neural network system <NUM> of <FIG>.

<FIG> shows a process for training either of the neural network systems <NUM>, <NUM>. The process begins by generating a training data set comprising blended training videos derived from first and second videos as previously described (step <NUM>). The blending value may be varied or fixed at e.g. α = <NUM>. The training data may be augmented by applying an augmentation process to the training videos e.g. by random left-right flipping and/or random spatio-temporal cropping e.g. by first resizing the training videos to slightly larger than the crop size.

Then each training video in the training data set is processed in turn by the neural network system <NUM>, <NUM> to be trained, to generate n sets of decomposed video data, where n ≥ <NUM> (step <NUM>). Each set of decomposed video data defines a respective output video Oi where i = <NUM>.

The process then determines a permutation invariant loss from the sets of decomposed video data and the ground truth videos V<NUM>, V<NUM>, i.e. the videos blended to generate the training video. The permutation invariant loss may be defined as: <MAT> where i, j = <NUM>. n (i.e. i,j run over all the outputs), and l is a video reconstruction loss as previously described. Thus the permutation invariant loss comprises a reconstruction loss for the two output videos which are most similar to the ground truth videos.

In some implementations the reconstruction loss may include a term representing a difference between spatial gradients in corresponding video image frames. For example the reconstruction loss for two videos U and V may be defined as: <MAT> where ∇(·) denotes a spatial gradient operator and ∥·∥<NUM> denotes the L1 norm; an L2 norm for either or both terms may be used instead. Differences due to edges in the videos can be harder to distinguish than differences due to approximately constant areas; including a spatial gradient term helps to address this.

Once the permutation invariant loss has been determined, the process backpropagates gradients of this loss to update parameters, e.g. weights of the neural network system (step <NUM>) using any of a range of backpropagation techniques, e.g. with momentum. The process then repeats with the next training video.

<FIG> shows a process for decomposing a video input using a trained neural network system as described above. The process inputs a sequence of video images (step <NUM>), and encodes the sequence into a set of latent variables using the 3D spatio-temporal encoder neural network <NUM> (step <NUM>). The set of latent variables is then processed using the 3D spatio-temporal decoder neural network <NUM> to generate two or more sets of decomposed video data (step <NUM>). The process then selects one or more of these for output (step <NUM>), e.g. as previously described. The output video data may be stored, communicated, and/or made available to a user interface.

<FIG> shows an example of an input video sequence decomposed in this way, using the neural network system <NUM>, into output video data for two decomposed video sequences, labelled "Layer <NUM>" and "Layer <NUM>". The input video sequence shows a person driving a car with trees reflecting from its windscreen. The two decomposed video layers separate out the interior of the car from the reflected trees.

Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non transitory program carrier for execution by, or to control the operation of, data processing apparatus. The computer storage medium is not, however, a propagated signal.

The typical elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data.

While this specification contains many specific implementation details, these should not be construed as limitations on the scope of the invention as defined by the claims, but rather as descriptions of features that may be specific to particular embodiments.

Claim 1:
A computer-implemented neural network system (<NUM>) for processing input video data representing input video image frames to decompose the input video data into two or more sets of decomposed video data (124a,b) each representing a respective sequence of decomposed video image frames, the sequences of decomposed video image frames representing a decomposition of the input video image frames, the neural network system comprising:
a video data input (<NUM>) to receive a sequence of input video image frames;
a predictor 3D spatio-temporal convolutional encoder neural network (<NUM>) to encode the sequence of input video image frames into a first set of latent variables representing a compressed version of the input video image frames;
a predictor 3D spatio-temporal transposed convolutional decoder neural network (<NUM>) to receive and process the first set of latent variables to generate two or more sets of decomposed video data representing respective sequences of decomposed video image frames; and
a video data output (<NUM>) to output video data derived from one or more of the sets of decomposed video data, characterized in that the neural network system further comprises:
a corrector 3D spatio-temporal convolutional encoder neural network (<NUM>) to receive each of the sets of decomposed video data and to encode the respective sequences of decomposed video image frames into a second set of latent variables representing a compressed version of the sequences of decomposed video image frames; and
a corrector 3D spatio-temporal transposed convolutional decoder neural network (<NUM>) to receive and process the second set of latent variables to generate two or more sets of correction video data, a set of correction video data for each set of decomposed video data; and
a first combiner (<NUM>) to combine each set of decomposed video data with a respective set of correction video data to provide two or more sets of combined video data, wherein the video output data comprises one or more of the sets of combined video data.