Patent Description:
Guided up-scaling, which is commonly used in many signal processing applications, including especially image up-scaling methods for image quality improvement, super-resolution and many others [<NPL>], is a process in which input data is being combined with additional input in form of up-scaling weights that control the influence of each input data value on the result to form the output data.

In deep-learning, a common approach recently used in many application fields is the utilization of convolutional neural networks (CNNs). Generally, a specific part of such convolutional neural networks is at least one convolution (or convolutional) layer which performs a convolution of input data values with a learned kernel K producing one output data value per convolution kernel for each output position [<NPL>]. For the two-dimensional case used, for instance, in image processing the convolution using the learned kernel K can be expressed mathematically as follows: <MAT> wherein out(x, y) denotes the array of output data values, in(x - i, y - j) denotes a sub-array of input data values and K(i, j) denotes the kernel comprising an array of kernel weights or kernel values of size (2r+<NUM>)×(2r+<NUM>). B denotes an optional learned bias term, which can be added for obtaining each output data value. The weights of the kernel K are the same for the whole array of input data values in(x, y) and are generally learned during a learning phase of the neural network which, in case of 1st order methods, consists of iteratively back-propagating the gradients of the neural network output back to the input layers and updating the weights of all the network layers by a partial derivative computed in this way. An extension of CNNs are deconvolutional neural networks (DNNs) with a specific element that extends their functionality relative to CNNs that is called deconvolution. Deconvolution can be interpreted as an "inversed" convolution known from classical CNNs.

<NPL>) introduces different blur kernels can be used for different images, and does not disclose that different blur kernels can be used for different spatial positions of the same image.

<NPL>) introduces filter weights are generated by the filter manifold network (FMN) from the auxiliary input z (i.e. side information), and does not disclose that different filter weights can be used for different spatial positions of the input map.

<NPL>) introduces convolutional kernels are applied on the image feature maps to generate question-guided attention maps, and does not disclose that different convolutional kernels can be used for different spatial positions of the image.

<NPL>) introduces bilateral filters depending on the image content, and does not disclose that different bilateral filters can be used for different spatial positions of the image. <NPL>) introduces dynamic filtering layer takes images as inputs (page <NUM>, section <NUM> of D1), also discloses a specific filter is associated with each position of the input.

<NPL>) introduces filter weights information), and does not disclose that different filter weights can be used for different spatial positions of the input map) introduces an alternative direction based on coupling the recognition capacity of DCNNs and the fine-grained localization accuracy of fully connected CRFs, which using contrast sensitive potentials in conjunction to local-range CRFs and fully connected CRF model, which employs the energy function.

It is an object of the invention to provide an improved data processing apparatus and method based on neural networks.

Generally, embodiments of the invention provide a new approach for deconvolution or upscaling of data for neural networks that is implemented into a neural network as a new type of neural network layer. The neural network layer can compute up-scaled data using individual up-scaling weights that are learned for each individual spatial position. Up-scaling weights can be computed as a function of position dependent weights or similarity features and position independent learned weight kernels, resulting in individual up-scaling weights for each input spatial position. In this way a variety of sophisticated position dependent or position adaptive kernels learned by the neural network can be utilized for better adaptation of the up-scaling weights to the input data.

More specifically, according to a first aspect the invention relates to a data processing apparatus comprising one or more processors configured to provide a neural network. The data to be processed by the data processing apparatus can be, for instance, two-dimensional image or video data or one-dimensional audio data.

The neural network provided by the one or more processors of the data processing apparatus comprises a neural network layer being configured to process an array of input data values, such as a two-dimensional array of input data values in(x, y), into an array of are generated by the filter manifold network (FMN) from the auxiliary input z (i.e. side information), and does not disclose that different filter weights can be used for different spatial positions of the input map) introduces an alternative direction based on coupling the recognition capacity of DCNNs and the fine-grained localization accuracy of fully connected CRFs, which using contrast sensitive potentials in conjunction to local-range CRFs and fully connected CRF model, which employs the energy function.

In particular, there is provided a data processing apparatus according to claim <NUM>, a computer-implemented data processing method according to claim <NUM>, and a computer program according to claim <NUM>.

The neural network provided by the one or more processors of the data processing apparatus comprises a neural network layer being configured to process an array of input data values, such as a two-dimensional array of input data values in(x, y), into an array of output data values, such as a two-dimensional array of output data values out(x, y). The neural network layer can be a first layer or an intermediate layer of the neural network.

The array of input data values can be one-dimensional (i.e. a vector, e.g. audio or other e.g. temporal sequence), two-dimensional (i.e. a matrix, e.g. an image or other temporal or spatial sequence), or N-dimensional (e.g. any kind of N-dimensional feature array, e.g. provided by a conventional pre-processing or feature extraction and/or by other layers of the neural network).

The array of input data values can have one or more channels, e.g. for an RGB image one R-channel, one G-channel and one B-channel, or for a black/white image only one grey-scale or intensity channel. The term "channel" can refer to any "feature", e.g. features obtained from conventional pre-processing or feature extraction or from other neural networks or neural network layers of the same neural network. The array of input data values can comprise, for instance, two-dimensional RGB or grey scale image or video data representing at least a part of an image, or a one-dimensional audio signal. In case the neural network layer is implemented as an intermediate layer of the neural network, the array of input data values can be, for instance, an array of similarity features generated by previous layers of the neural network on the basis of an initial, i.e. original array of input data values, e.g. by means of a feature extraction.

The neural network layer is configured to generate from the array of input data values the array of output data values on the basis of a plurality of position dependent, i.e. spatially variable kernels and a plurality of different input data values of the array of input data values. Each kernel comprises a plurality of kernel values (also referred to as kernel weights). For a respective position or element of the array of input data values a respective kernel is applied thereto for generating a respective sub-array of the array of output data values. In an implementation form, the plurality of kernel values of a respective position dependent kernel can be respectively multiplied with a respective input data value for generating a respective sub-array of the array of output data values having the same size as the position dependent kernel, i.e. the array of kernel values. Generally, the size of the array of input data values can be smaller than the size of the array of output data values.

A "position dependent kernel" as used herein means a kernel whose kernel values can depend on the respective position or element of the array of input data values. In other words, for a first kernel used for a first input data value of the array of input data values the kernel values can differ from the kernel values of a second kernel used for a second input data value of the array of input data values. In a two-dimensional array the position could be a spatial position defined, for instance, by two spatial coordinates x,y. In a one-dimensional array the position could be a temporal position defined, for instance, by a time coordinate t.

Thus, an improved data processing apparatus based on neural networks is provided. The data processing apparatus allows upscaling or deconvolving the input data in a way that can better reflect mutual data similarity. Moreover, the data processing apparatus allows adapting the kernel weights for different spatial positions of the array of input data values. This, in turn, allows, for instance, minimizing the influence of some of the input data values on the result, for instance the input data values that are associated with another part of the scene (as determined by semantic segmentation) or a different object that is being analysed.

In a further implementation form of the first aspect, the neural network comprises at least one additional network layer configured to generate the plurality of position dependent kernels on the basis of an original array of original input values of the neural network, wherein the original array of original input values of the neural network comprises the array of input values or another array of input values associated to the array of input values. The original array of original input values can be the array of input data values or a different array.

In a further implementation form of the first aspect, the neural network is configured to generate the plurality of position dependent kernels based on a plurality of learned position independent kernels and a plurality of position dependent weights (also referred to as similarity features). Generally, the position independent kernels can be learned by the neural network and the position dependent weights (i.e. similarity features) can be computed, for instance, by a further preceding layer of the neural network. This implementation form allows minimizing the amount of data being transferred to the neural network layer in order to obtain the kernel values. This is because the kernel values are not transferred directly, but computed from the plurality of position dependent weights (i.e. similarity features) substantially reducing the amount of data for each element of the array of output data values. This can minimize the amount of data being stored and transferred by the neural network between the different network layers, which is especially important during the learning process on the basis of the mini-batch approach as the memory of the data processing apparatus (GPU) is currently the main bottleneck. Moreover, this implementation form allows for a better adaption of the kernel values to the processed data and utilizing more sophisticated similarity features. For instance, information about object shapes or object segmentations can be utilized in order to better preserve better object boundaries or even increase the level of details in the higher-resolution output. In this way, information about some small details from the original array of original input values not present in the possibly low-resolution array of input data values can be combined with the array of input data values in order to create higher-resolution array of output data values.

In a further implementation form of the first aspect, the neural network is configured to generate a kernel of the plurality of position dependent kernels by adding the learned position independent kernels each weighted by the associated non-learned position dependent weights (i.e. similarity features). This implementation form provides a very efficient representation of the plurality of position dependent kernels using a linear combination of position independent "base kernels".

In a further implementation form of the first aspect, the plurality of position independent kernels are predetermined or learned, and wherein the neural network comprises at least one additional neural network layer or "conventional" pre-processing layer configured to generate the plurality of position dependent weights (i.e. similarity features) based on an original array of original input values of the neural network, wherein the original array of original input values of the neural network comprises the array of input values or another array of input values associated to the array of input values. The original array of original input values can be the array of input data values or a different array. In an implementation form the at least one additional neural network layer or "conventional" pre-processing layer can generate the plurality of position dependent weights (i.e. similarity features) using, for instance, bilateral filtering, semantic segmentation, per-instance object detection, and data importance indicators like ROI (region of interest).

In a further implementation form of the first aspect, the array of input data values and the array of output data values are two-dimensional arrays, and the convolutional neural network layer is configured to generate the plurality of position dependent kernels wL(x, y, i, j) on the basis of the following equation: <MAT> wherein Ff(x, y) denotes the plurality of Nf position dependent weights (i.e. similarity features) and Kf (i,j) denotes the plurality of position independent "base" kernels.

In a further implementation form of the first aspect, the neural network layer is a deconvolutional network layer or an upscaling network layer.

In a further implementation form of the first aspect, the array of input data values and the array of output data values are two-dimensional arrays, wherein the neural network layer is a deconvolution network layer configured to generate the array of output data values on the basis of the following equations: <MAT> <MAT> <MAT> wherein x, y, x', y', i, j denote array indices, out(x, y, co) denotes the multi-channel array of output data values, in(x', y', ci) denotes the array of input data values, r denotes a size of each kernel of the plurality of position dependent multi-channel kernels wL(x', y', co, ci, i, j) and WL'(x, y, c<NUM>) denotes a normalization factor. In an implementation form the normalization factor WL'(x, y, c<NUM>) can be set equal to <NUM>.

In a further implementation form of the first aspect, the array of input data values and the array of output data values are two-dimensional arrays, wherein the neural network layer is an upscaling network layer configured to generate the array of output data values on the basis of the following equations: <MAT> <MAT> <MAT> wherein x, y, x', y', i, j denote array indices, out(x, y) denotes the array of output data values, in(x', y') denotes the array of input data values, r denotes a size of each kernel of the plurality of position dependent kernels wL(x', y', i, j) and WL'(x, y) denotes a normalization factor. In an implementation form the normalization factor WL'(x, y) can be set equal to <NUM>. As will be appreciated, the sum in the equation above extends over every possible position (x',y') of the array of input data values, where x' and y' meet the conditions: x' - i = x and y' - j = y. In this way, overlapping positions of different position dependent kernels are obtained that are summed to generate the final output data value out(x, y).

In a further implementation form of the first aspect, the array of input data values and the array of output data values are two-dimensional arrays and the neural network layer is configured to generate the array of output data values on the basis of the following equations: <MAT> <MAT> <MAT> <MAT> wherein x, y, x', y', i,j, k, l denote array indices, out(x, y) denotes the array of output data values, in(x', y') denotes the array of input data values, r denotes a size of each kernel of the plurality of position dependent kernels wL(x, y, i, j), sel(x, y, i, j) denotes a selection function and WL'(x, y) denotes a normalization factor. In an implementation form the normalization factor WL'(x, y) can be set equal to <NUM>.

In a further implementation form of the first aspect, the array of input data values and the array of output data values are two-dimensional arrays and the neural network layer is configured to generate the array of output data values on the basis of the following equations: <MAT> <MAT> <MAT> <MAT> wherein x, y, x', y', x", y", i, j, k, l denote array indices, out(x, y) denotes the array of output data values, in(x', y') denotes the array of input data values, r denotes a size of each kernel of the plurality of position dependent kernels wL(x' , y' , i, j), sel(x, y, x', y', i, j) denotes a selection function and WL'(x, y) denotes a normalization factor. In an implementation form the normalization factor WL'(x, y) can be set equal to <NUM>.

According to a second aspect the invention relates to a corresponding data processing method comprising the step of generating by a neural network layer of a neural network from an array of input data values an array of output data values based on a plurality of position dependent kernels and a plurality of different input data values of the array of input data values.

In a further implementation form of the second aspect, the method comprises the further step of generating the plurality of position dependent kernels by an additional neural network layer of the neural network based on an original array of original input values of the neural network, wherein the original array of original input values of the neural network comprises the array of input values or another array of input values associated to the array of input values.

In a further implementation form of the second aspect, the step of generating the plurality of position dependent kernels comprises generating the plurality of position dependent kernels based on a plurality of position independent kernels and a plurality of position dependent weights.

In a further implementation form of the second aspect, the step of generating the plurality of position dependent kernels comprises the step of adding, i.e. summing the position independent kernels weighted by the associated position dependent weights.

In a further implementation form of the second aspect, the plurality of position independent kernels are predetermined or learned and the step of generating the plurality of position dependent weights comprises the step of generating the plurality of position dependent weights by an additional neural network layer or a processing layer of the neural network based on an original array of original input values of the neural network, wherein the original array of original input values of the neural network comprises the array of input values or another array of input values associated to the array of input values.

In a further implementation form of the second aspect, the array of input data values and the array of output data values are two-dimensional arrays, and the step of generating a kernel of the plurality of position dependent kernels wL(x, y, i, j) is based on the following equation: <MAT> wherein Ff(x, y) denotes the plurality of Nf position dependent weights (i.e. similarity features) and Kf (i, j) denotes the plurality of position independent kernels.

In a further implementation form of the second aspect, the neural network layer is a deconvolutional network layer or an upscaling network layer.

In a further implementation form of the second aspect, the array of input data values and the array of output data values are two-dimensional arrays, wherein the neural network layer is a deconvolution network layer and the step of generating the array of output data values comprises generating the array of output data values on the basis of the following equations: <MAT> <MAT> <MAT> wherein x, y, x', y', i,j denote array indices, out(x, y, co) denotes the multi-channel array of output data values, in(x', y', ci) denotes the array of input data values, r denotes a size of each kernel of the plurality of position dependent multi-channel kernels wL(x', y', co, ci, i, j) and WL'(x, y, co) denotes a normalization factor. In an implementation form the normalization factor WL'(x, y, c<NUM>) can be set equal to <NUM>.

In a further implementation form of the second aspect, the array of input data values and the array of output data values are two-dimensional arrays, wherein the neural network layer is an upscaling network layer and the step of generating the array of output data values comprises generating the array of output data values on the basis of the following equations: <MAT> <MAT> <MAT> wherein x, y, x', y', i, j denote array indices, out(x, y) denotes the array of output data values, in(x', y') denotes the array of input data values, r denotes a size of each kernel of the plurality of position dependent kernels wL(x', y', i, j) and WL'(x, y) denotes a normalization factor. In an implementation form the normalization factor WL'(x, y) can be set equal to <NUM>.

In a further implementation form of the second aspect, the array of input data values and the array of output data values are two-dimensional arrays and the step of generating the array of output data values comprises generating the array of output data values on the basis of the following equations: <MAT> <MAT> <MAT> <MAT> wherein x, y, x', y', i,j, k, l denote array indices, out(x, y) denotes the array of output data values, in(x', y') denotes the array of input data values, r denotes a size of each kernel of the plurality of position dependent kernels wL(x, y, i, j), sel(x, y, i, j) denotes a selection function and WL'(x, y) denotes a normalization factor. In an implementation form the normalization factor WL'(x, y) can be set equal to <NUM>.

In a further implementation form of the second aspect, the array of input data values and the array of output data values are two-dimensional arrays and the step of generating the array of output data values comprises generating the array of output data values on the basis of the following equations: <MAT> <MAT> <MAT> <MAT> wherein x, y, x', y', x", y", i, j, k, l denote array indices, out(x, y) denotes the array of output data values, in(x', y') denotes the array of input data values, r denotes a size of each kernel of the plurality of position dependent kernels wL(x' , y' , i, j), sel(x, y, x', y', i, j) denotes a selection function and WL'(x, y) denotes a normalization factor. In an implementation form the normalization factor WL'(x, y) can be set equal to <NUM>.

According to a third aspect the invention relates to a computer program comprising program code for performing the method according to the second aspect, when executed on a processor or a computer.

The invention can be implemented in hardware and/or software.

In the various figures, identical reference signs will be used for identical or at least functionally equivalent features.

In the following description, reference is made to the accompanying drawings, which form part of the disclosure, and in which are shown, by way of illustration, specific aspects in which the present invention may be placed. It is understood that other aspects may be utilized and structural or logical changes may be made. The following detailed description, therefore, is not to be taken in a limiting sense, as the scope of the present invention is defined by the appended claims.

<FIG> shows a schematic diagram illustrating a data processing apparatus <NUM> according to an embodiment configured to process data on the basis of a neural network. To this end, the data processing apparatus <NUM> shown in <FIG> comprises a processor <NUM>. In an embodiment, the data processing apparatus <NUM> can be implemented as a distributed data processing apparatus <NUM> comprising more than the one processor <NUM> shown in <FIG>.

The processor <NUM> of the data processing apparatus <NUM> is configured to provide a neural network <NUM>. As will be described in more detail further below, the neural network <NUM> comprises a neural network layer being configured to generate from an array of input data values an array of output data values based on a plurality of position dependent kernels and a plurality of different input data values of the array of input data values. As shown in <FIG>, the data processing apparatus <NUM> can further comprise a memory <NUM> for storing and/or retrieving the input data values, the output data values and/or the kernels.

Each kernel comprises a plurality of kernel values (also referred to as kernel weights). For a respective position or element of the array of input data values a respective kernel is applied thereto for generating a respective sub-array of the array of output data values. Generally, the size of the array of input data values is smaller than the size of the array of output data values. A "position dependent kernel" as used herein means a kernel whose kernel values depend on the respective position or element of the array of input data values. In other words, for a first kernel used for a first input data value of the array of input data values the kernel values can differ from the kernel values of a second kernel used for a second input data value of the array of input data values. In a two-dimensional array the position could be a spatial position defined, for instance, by two spatial coordinates x,y. In a one-dimensional array the position could be a temporal position defined, for instance, by a time coordinate t.

The array of input data values can be one-dimensional (i.e. a vector, e.g. audio or other e.g. temporal sequence), two-dimensional (i.e. a matrix, e.g. an image or other temporal or spatial sequence), or N-dimensional (e.g. any kind of N-dimensional feature array, e.g. provided by a conventional pre-processing or feature extraction and/or by other layers of the neural network <NUM>). The array of input data values can have one or more channels, e.g. for an RGB image one R-channel, one G-channel and one B-channel, or for a black/white image only one grey-scale or intensity channel. The term "channel" can refer to any "feature", e.g. features obtained from conventional pre-processing or feature extraction or from other neural networks or neural network layers of the neural network <NUM>. The array of input data values can comprise, for instance, two-dimensional RGB or grey scale image or video data representing at least a part of an image, or a one-dimensional audio signal. In case the neural network layer <NUM> is implemented as an intermediate layer of the neural network <NUM>, the array of input data values can be, for instance, an array of similarity features generated by previous layers of the neural network on the basis of an initial, i.e. original array of input data values, e.g. by means of a feature extraction, as will be described in more detail further below.

As will be described in more detail below, the neural network layer <NUM> can be implemented as an up-scaling layer <NUM> configured to process each channel of the array of input data values separately, e.g. for an input array of R-values one (scalar) R-output value is generated. The position dependent kernels may be channel-specific or common for all channels. Moreover, the neural network layer <NUM> can be implemented as a deconvolution (or deconvolutional) layer configured to "mix" all channels of the array of input data values. For instance, in case the generated array of output data values is an RGB image, i.e. a multi-channel array, every single channel of a multi-channel input data array is used to generate all three channels of the multi-channel array of output data values. The position dependent kernels may be channel-specific, i.e. multi-channel arrays, or common for all channels.

<FIG> shows a schematic diagram illustrating elements of the neural network <NUM> provided by the data processing apparatus <NUM> according to an example not encompassed by the claims but useful for understanding the invention. In the example shown in <FIG>, the neural network layer <NUM> is implemented as a up-scaling layer <NUM>. In a further example, the neural network layer <NUM> can be implemented as a deconvolution layer <NUM> (also referred to as deconvolutional layer <NUM>), as will be described in more detail further below. As indicated in <FIG>, in this example the up-scaling layer <NUM> is configured to generate a two-dimensional array of output data values out(x, y) <NUM> on the basis of the two-dimensional array of input data values in(x, y) <NUM> and the plurality of position dependent kernels <NUM> comprising a plurality of kernel values or kernel weights.

In an example, the up-scaling layer <NUM> of the neural network <NUM> shown in <FIG> is configured to generate the array of output data values out(x, y) <NUM> on the basis of the array of input data values in(x, y) <NUM> and the plurality of position dependent kernels <NUM> comprising the kernel values wL(x, y, i, j) using the following equations: <MAT> <MAT> <MAT> wherein x, y, x', y', i, j denote array indices, out(x, y) denotes the array of output data values <NUM>, in(x', y') denotes the array of input data values <NUM>, r denotes a size of each kernel of the plurality of position dependent kernels wL(x', y', i, j) <NUM> (in this example, each kernel has (2r+<NUM>)*(2r+<NUM>) kernel values) and WL'(x, y) denotes a normalization factor and can be optionally set to <NUM>. As will be appreciated, the sum in the equation above extends over every possible position (x', y') of the array of input data values <NUM>, where x' and y' meet the conditions: x' - i = x and y' - j = y. In this way, overlapping positions of different position dependent kernels <NUM> are obtained that are summed to generate the final output data value out(x, y).

In other examples, the normalization factor can be omitted, i.e. set to one. For instance, in case the neural network layer <NUM> is implemented as a deconvolutional network layer the normalization factor can be omitted. For upscaling the normalization factor allows to keep the DC component. This is usually not necessary in the case of the deconvolutional network layer <NUM>.

As will be appreciated, the above equations for a two-dimensional input array and a kernel having a quadratic shape can be easily adapted to the case of an array of input values <NUM> having one dimension or more than two dimensions and/or a kernel having a rectangular shape, i.e. different horizontal and vertical dimensions.

For an example, where the neural network layer <NUM> is implemented as a deconvolution layer and the array of input data values in(x, y, ci) <NUM> is a two-dimensional array of input data values the deconvolutional layer <NUM> is configured to generate the array of output data values <NUM> as a multi-channel array of output data values out(x, y, co) <NUM>, an array having more than one channel co. In this case, also the plurality of position dependent kernels <NUM> will have the corresponding number of channels, wherein each multi-channel position dependent kernel comprises the kernel values wL(x', y', co, ci, i, j). For instance, the deconvolutional layer <NUM> could be configued to deconvolve a monochromatic image into an RGB image with higher resolution using a plurality of position dependent kernels <NUM> having three channels.

In an example, the deconvolutional layer <NUM> is configured to generate the multi-channel array of output data values out(x, y, co) <NUM> on the basis of the array of input data values in(x, y, ci) <NUM> having one or more channels and the plurality of multi-channel position dependent kernels <NUM> comprising the kernel values wL(x', y', co, ci, i, j) using the following equations: <MAT> <MAT> <MAT> wherein x, y, x', y', i, j denote array indices, r denotes a size of each kernel of the plurality of positon dependent kernels <NUM> and WL'(x, y, co) denotes a normalization factor. In other examples, the normalization factor can be omitted, i.e. set to one.

In an example, the neural network layer <NUM> is configured to generate the array of output data values <NUM> with a larger size than the array of input data values <NUM>. In other words, in an example, the neural network <NUM> is configured to perform an up-step or upscaling operation of the array of input data values <NUM> on the basis of the plurality of position dependent kernels <NUM>. <FIG> illustrates an up-step or upscaling operation provided by a neural network <NUM> of the data processing apparatus <NUM> according to an embodiment. Using an up-step or upscaling operation allows increasing the receptive field, enables processing the data with a cascade of smaller filters as compared with a single layer with a kernel covering an equal receptive field, and also enables the neural network <NUM> to better analyse the data by finding more sophisticated relationships among the data.

In the up-step or upscaling operation illustrated in <FIG> the neural network layer <NUM> can up-scale the input data produced by a preceding cascade of down-layers for generating an array of output data values having an increased resolution. This upscaling operation can be performed by deconvolving every channel of each spatial position of the array of input data values with position dependent kernels with a stride S greater than <NUM>, producing a data volume of increased resolution. The stride S specifies the spacing between neighboring input spatial positions for which deconvolutions are computed. If the stride S is equal to <NUM>, the deconvolution is performed for each spatial position. If the stride S is an integer greater than <NUM>, deconvolution is performed for every S spatial position, increasing the output resolution by a factor of S for each spatial dimension.

In the exemplary embodiment shown in <FIG>, the neural network layer <NUM> up-scales every element of the array of input data values <NUM> into a respective sub-array of the array of output data values <NUM> with a size of (2r+<NUM>)×(2r+<NUM>) (defined by the size of the position dependent kernels <NUM>). In this way, the input data values <NUM> can be up-scaled to the higher resolution array of output data values <NUM>.

According to an embodiment, the upscaling operation performed by the neural network layer <NUM> for the exemplary case of two-dimensional input and output arrays <NUM>, <NUM> comprises multiplying a respective input data value of the array of input data values <NUM> with the plurality of kernel weights wL(x, y, i, j) of a respective position dependent kernel <NUM>. In case the respective position dependent kernel <NUM> has an exemplary size of (2r+<NUM>)×(2r+<NUM>) this operation will generate a sub-array of the array of output data values <NUM> (which can also be considered as an interpolation area) having also a size of (2r+<NUM>)×(2r+<NUM>). As will be appreciated, depending on the selected stride S, the interpolation areas of neighboring input data values may overlap. In order to handle such case, according to an embodiment, the values from all overlapping interpolation areas <NUM> located at the spatial position (x, y) (i.e. overlapping spatial position) can be aggregated and (optionally) normalized by a normalization factor producing the final output data value out(x, y). This operation is illustrated in <FIG> for the exemplary case of having R sub-arrays or interpolation areas at the spatial position (x, y).

In the example shown in <FIG>, the neural network <NUM> comprises one or more preceding layers <NUM> preceding the neural network layer <NUM> and one or more following layers <NUM> following the neural network layer <NUM>. In an embodiment, the neural network layer <NUM> could be the first and/or the last data processing layer of the neural network <NUM>, i.e. in an embodiment there could be no preceding layers <NUM> and/or no following layers <NUM>.

In an embodiment, the one or more preceding layers <NUM> can be further neural network layers, such as a convolutional network layer, and/or "conventional" pre-processing layers, such as a feature extraction layer. Likewise, in an embodiment, the one or more following layers <NUM> can be further neural network layers and/or "conventional" post-processing layers.

As shown in the example shown in <FIG>, one or more of the preceding layers <NUM> can be configured to provide, i.e. to generate the plurality of position dependent kernels <NUM>. In an embodiment, the one or more layers of the preceding layers <NUM> can generate the plurality of position dependent kernels <NUM> on the basis of an original array of original input data values. As indicated in <FIG>, in an example, the original array of original input data values can be an array of input data <NUM> being the original input of the neural network <NUM>. In another embodiment, the one or more preceding layers <NUM> could be configured to generate just the plurality of position dependent kernels <NUM> on the basis of the original input data <NUM> of the neural network <NUM> and to provide the original input data <NUM> of the neural network <NUM> as the array of input data values <NUM> to the neural network layer <NUM>.

As indicated in <FIG>, in a further examplethe one or more preceding layers <NUM> of the neural network <NUM> are configured to generate the plurality of position dependent kernels <NUM> on the basis of an array of guiding data <NUM>. A more detailed view of the processing steps of the neural network <NUM> of the data processing apparatus <NUM> according to such an embodiment is shown in <FIG> for the exemplary case of two-dimensional input and output arrays. The array of guiding data <NUM> is used by the one or more preceding layers <NUM> of the neural network <NUM> to generate the plurality of position dependent kernels wL(x, y) <NUM> on the basis of the array of guiding data g(x, y) <NUM>. As already described in the context of <FIG>, the neural network layer <NUM> is configured to generate the two-dimensional array of output data values out(x,y) <NUM> on the basis of the two-dimensional array of input data values in(x, y) <NUM> and the plurality of position dependent kernels wL(x, y) <NUM>, which, in turn, are based on the array of guiding data g(x, y) <NUM>.

In an embodiment, the one or more preceding layers <NUM> of the neural network <NUM> are neural network layers configured to learn the plurality of position dependent kernels wL(x, y) <NUM> on the basis of the array of guiding data g(x, y) <NUM>. In another embodiment, the one or more preceding layers <NUM> of the neural network <NUM> are pre-processing layers configured to generate the plurality of position dependent kernels wL(x, y) <NUM> on the basis of the array of guiding data <NUM> using one or more pre-processing schemes, such as feature extraction.

In an embodiment, the one or more preceding layers <NUM> of the neural network <NUM> are configured to generate the plurality of position dependent kernels wL(x, y) <NUM> on the basis of the array of guiding data g(x, y) <NUM> in a way analogous to up-scaling based on bilateral filters, as illustrated in <FIG>. In image processing, a common approach to perform data up-scaling is to use bilateral filter weights [<NPL>] as a sort of guiding information for interpolating the input data. The usage of bilateral filter weights has the advantage of decreasing the influence of input data values on some spatial positions of the interpolation results, while amplifying its influence for others. As illustrated in <FIG>, the weights <NUM> utilized for up-scaling the array of input data values <NUM> adapt to input data using the guiding image data g <NUM> which provides additional information to control the up-scaling process. In the up-scaling process, a single input data value of the array of input data values in(x, y) <NUM> is multiplied by the kernel w <NUM> of size (2r+<NUM>)×(2r+<NUM>) creating an interpolated area of output data out(x ± r, y ± r) <NUM> of size (2r+<NUM>)×(2r+<NUM>). As will be appreciated, however, the interpolation areas of neighbouring input positions may overlap. In order to handle such cases, values from different overlapping interpolation areas located at the spatial position x, y can be aggregated and normalized by a normalization factor W'(x, y) producing the final output value out(x, y). If the stride S is greater than <NUM>, the spatial resolution of the output data created by the interpolation areas will be increased. Mathematically, this can be expressed in the following way: <MAT> where: <MAT> i ∈ {-r,. ,r}, j ∈ {-r,.

In an embodiment, the bilateral filter weights <NUM> are defined by the following equation: <MAT> wherein d(. ) denotes a distance function. Thus, the bilateral filter weights <NUM> can take into account the distance of the value within the kernel from the center of the kernel and, additionally, the similarity of the data values with data in the center of the kernel.

<FIG> shows a schematic diagram highlighting the main processing stage <NUM> of the data processing apparatus <NUM> according to an embodiment, for instance, the data processing apparatus <NUM> providing the neural network <NUM> shown in <FIG>. As already described above, in a first processing step <NUM> the neural network <NUM> can generate the plurality of position dependent kernels wL(x, y) <NUM> on the basis of the array of guiding data g(x, y) <NUM>. In a second processing step <NUM> the neural network <NUM> can generate the array of output data values out(x, y) <NUM> on the basis of the array of input data values in(x, y) <NUM> and the plurality of position dependent kernels wL(x, y, i, j) <NUM>.

<FIG> shows a schematic diagram illustrating the neural network <NUM> provided by the data processing apparatus <NUM> according to a further embodiment. As will be described in more detail in the following, the main difference to the example shown in <FIG> is that in the embodiment shown in <FIG> the neural network <NUM> is configured to generate the plurality of position dependent kernels based on a plurality of position independent kernels 119b (shown in <FIG>) and a plurality of position dependent weights Ff(x, y) 119a (also referred to as similarity features 119a). In an embodiment, the similarity features 119a could indicate higher-level knowledge about the input data, including e.g. semantic segmentation, per-instance object detection, data importance indicators like ROI (Region of Interest) and many others - all learned by the neural network <NUM> itself or being an additional input to the neural network <NUM>. In an embodiment, the neural network <NUM> of <FIG> is configured to generate the plurality of position dependent kernels <NUM> by adding the position independent kernels 119b weighted by the associated position dependent weights Ff(x, y) 119a.

In an embodiment, the plurality of position independent kernels 119b can be predetermined or learned by the neural network <NUM>. As illustrated in <FIG>, also in this embodiment the neural network <NUM> can comprise one or more preceding layers <NUM>, which precede the neural network layer <NUM> and which can be implemented as an additional neural network layer or a pre-processing layer. In an embodiment, one or more layers of the preceding layers <NUM> are configured to generate the plurality of position dependent weights Ff(x, y) 119a on the basis of an original array of original input data values. The original array of original input data values of the neural network <NUM> can comprise the array of input data values <NUM> to be processed by the neural network layer <NUM> or another array of input data values <NUM> associated to the array of input data values <NUM>, for instance, the initial array of input data <NUM>.

In the exemplary embodiment shown in <FIG>, the array of input data values in(x, y) <NUM> and the array of output data values out(x, y) <NUM> are two-dimensional arrays and the neural network layer <NUM> is configured to generate a respective kernel of the plurality of position dependent kernels wL(x, y, i, j) <NUM> on the basis of the following equation: <MAT> wherein Ff(x, y) denotes the set of Nf position dependent weights (or similarity features) 119a and Kf (i, j) denotes the plurality of position independent kernels 119b, as also illustrated in <FIG>.

<FIG> shows a schematic diagram highlighting the main processing stage <NUM> implemented in the data processing apparatus <NUM> according to an embodiment, for instance, the data processing apparatus <NUM> providing the neural network <NUM> illustrated in <FIG> and <FIG>. As already described above, in a first processing step <NUM> the neural network <NUM> can generate the plurality of position dependent weights or similarity features Ff(x, y) 119a on the basis of the array of guiding data g(x, y) <NUM>. In a second processing step <NUM> the neural network <NUM> can generate the plurality of position dependent kernels wL(x, y, i, j) <NUM> on the basis of the plurality of position dependent weights or similarity features Ff(x, y) 119a and the plurality of position independent kernels Kf (i, j) 119b. In a further step (not shown in <FIG>, but similar to the processing step <NUM> shown in <FIG>) the neural network layer <NUM> can generate the array of output data values out(x, y) <NUM> on the basis of the array of input data values in(x, y) <NUM> and the plurality of position dependent kernels wL(x, y, i, j) <NUM>.

In a further embodiment, the neural network layer <NUM> is configured to process the array of input data values <NUM> on the basis of the plurality of position dependent kernels <NUM> using an "inverse" maximum or minimum pooling scheme. More specifically, according to such an embodiment, the array of input data values <NUM> and the array of output data values <NUM> are two-dimensional arrays and the neural network layer <NUM> is configured to generate the array of output data values <NUM> on the basis of the following equations: <MAT> <MAT> <MAT> <MAT> wherein x, y, x', y', i,j, k, l denote array indices, out(x, y) denotes the array of output data values <NUM>, in(x', y') denotes the array of input data values <NUM>, r denotes a size of each kernel of the plurality of position dependent kernels wL(x, y, i, j) <NUM>, sel(x, y, i, j) denotes a selection function and WL'(x, y) denotes a normalization factor. In an implementation form the normalization factor WL'(x, y) can be set equal to <NUM>.

In this embodiment, the neural network layer <NUM> can be considered to adaptively guide data from the array of input data values <NUM> to a spatial position of a sub-array of the array of output data values <NUM> (i.e. the interpolated area) based on the individual position dependent kernel values <NUM>. In this way a sort of more intelligent data un-pooling can be performed. In an embodiment, the input data value corresponding to the spatial position (x, y) is copied to the position (x - imax/min, y - jmax/min) of the sub-array of output data values (i.e. the interpolated area) of size (2r+<NUM>)×(2r+<NUM>), where (imax/min, jmax/min) are the indices of the individual kernel values with the largest (max) or slowest (min) value among all individual kernel values. As can be taken from the equations above, in this embodiment, other values can be set to zero or, in an alternative embodiment, remain unset. Additionally, an aggregation of overlapping sub-arrays, i.e. interpolated areas can be performed, as in the embodiments described above.

In another embodiment, the array of input data values <NUM> and the array of output data values <NUM> are two-dimensional arrays and the neural network layer <NUM> is configured to generate the array of output data values <NUM> on the basis of the following equations: <MAT> <MAT> <MAT> <MAT> wherein x, y, x', y', x", y", i, j, k, l denote array indices, out(x, y) denotes the array of output data values <NUM>, in(x', y') denotes the array of input data values <NUM>, r denotes a size of each kernel of the plurality of position dependent kernels wL(x' , y' , i, j) <NUM>, sel(x, y, x', y', i, j) denotes a selection function and WL'(x, y) denotes a normalization factor. In an implementation form the normalization factor WL'(x, y) can be set equal to <NUM>.

In this embodiment, the neural network layer <NUM> can be considered to adaptively select output data out(x, y) from input data guided into position (x, y) without performing a weighted average, but selecting as the output data value out(x, y) the input data value in(x', y') of the array of input data values <NUM> which corresponds to the maximum or minimum kernel value wL(x' , y' , i, j). As a result, the output is computed as the input data value which would originally contribute the most (or in the alternative embodiment the least) to the weighted average.

<FIG> shows a flow diagram illustrating a data processing method <NUM> based on a neural network <NUM> according to an embodiment. The data processing method <NUM> can be performed by the data processing apparatus <NUM> shown in <FIG> and its different embodiments described above. The data processing method <NUM> comprises the step <NUM> of generating by the neural network layer <NUM> of the neural network <NUM> from the array of input data values <NUM> the array of output data values <NUM> based on a plurality of position dependent kernels <NUM> and a plurality of input data values of the array of input data values <NUM>. As will be appreciated, further embodiments of the data processing method <NUM> result directly from the embodiments of the corresponding data processing apparatus <NUM> described above. Embodiments of the data processing methods may be implemented and/or performed by one or more processors as described above.

In the following some further details about various aspects and embodiments (aggregation network layer, convolution network layer, correlation network layer and normalization) are provided.

In embodiments the proposed guided aggregation can be applied for feature map up-scaling (spatial resolution increase). Input values which are features of the feature map are up-scaled one-by-one forming overlapping output sub-arrays of values which are than aggregated and optionally normalized to form output data array. Due to additional guiding information in form of position dependent kernels, the up-scaling process for each input value can be performed in a controlled way, enabling addition of higher resolution details, e.g. object or region borders, that was originally not present in the input low-resolution representation. Here, guiding data represents information about object or region borders in higher resolution, and can be obtained by e.g. color-based segmentation, semantic segmentation using preceding neural network layers or an edge map of a texture image corresponding to processed feature map.

In embodiments the proposed guided deconvolution can be applied for switchable feature extraction or mixing. Input values which are features of the feature map are deconvolved with adaptable filters which are formed from the input guiding data in form of position dependent kernels. This way, each selected area of the input feature map can be processed with filters especially adapted for that area producing and mixing only features desired for these regions. Here, guiding data in form of similarity features represents information about object/region borders, obtained by e.g. color-based segmentation, semantic segmentation using preceding neural network layers, an edge map of a texture image corresponding to processed feature map or a ROI (region of interest) binary map.

In general, normalization is advantageous if the output values obtained for different spatial positions are going to be compared to each other per-value, without any intermediate step. As a result, preservation of the mean (DC) component is beneficial. If such comparison is not performed, normalization is not necessary but increases complexity. Additionally, one can omit normalization in order to simplify the computations and compute only an approximate result.

While a particular feature or aspect of the disclosure may have been disclosed with respect to only one of several implementations or embodiments, such feature or aspect may be combined with one or more other features or aspects of the other implementations or embodiments as may be desired and advantageous for any given or particular application. Furthermore, to the extent that the terms "include", "have", "with", or other variants thereof are used in either the detailed description or the claims, such terms are intended to be inclusive in a manner similar to the term "comprise". Also, the terms "exemplary", "for example" and "e.g." are merely meant as an example, rather than the best or optimal. The terms "coupled" and "connected", along with derivatives may have been used. It should be understood that these terms may have been used to indicate that two elements cooperate or interact with each other regardless whether they are in direct physical or electrical contact, or they are not in direct contact with each other.

Although specific aspects have been illustrated and described herein, it will be appreciated by those of ordinary skill in the art that a variety of alternate implementations may be substituted for the specific aspects shown and described.

Claim 1:
A data processing apparatus (<NUM>) comprising:
a processor (<NUM>) configured to provide a neural network (<NUM>), wherein the neural network (<NUM>) comprises a neural network layer (<NUM>) being configured to generate from an array of input data values (<NUM>) an array of output data values (<NUM>) based on a plurality of position dependent kernels (<NUM>) and a plurality of input data values of the array of input data values (<NUM>), wherein a respective position dependent kernel is a kernel whose kernel values depend on a respective position of the array of input data values; and the array of input data values represents a part or a whole of an image,
wherein the neural network (<NUM>) comprises one or more preceding neural network layers (<NUM>) configured to generate the plurality of position dependent kernels (<NUM>) based on an array of guiding data (<NUM>) representative of information about object or region borders;
characterized in that:
the data processing apparatus (<NUM>) is for creating a higher-resolution image of an input image;
the neural network (<NUM>) is configured to generate the plurality of position dependent kernels (<NUM>) based on a plurality of position independent kernels (119b) and a plurality of position dependent weights (119a); and
the neural network (<NUM>) is configured to generate the plurality of position dependent weights (119a) based on the array of guiding data (<NUM>).