Patent Description:
The present invention relates to combining a generative model with existing high efficiency coding schemes for media signals. Specifically, the present invention relates to a method for predicting the transform coefficients of an adaptive block length media signal with a trained neural network.

In low-rate adaptive block length encoding and decoding the encoder is configured to optimize the trade-off between frequency and time resolution. This may be achieved by selecting, by the encoder, a transform length for each signal sample block. In general, the encoder will select a long block, with a higher number of transform coefficients, for signal sample blocks representing signals with slowly evolving temporal characteristics and will select a set of short blocks, each with a lower number of transform coefficients, for signal sample blocks representing signals with rapidly evolving temporal characteristics.

A problem with encoding and decoding adaptive block length signals lies in that the blocks to be decoded may comprise a varying number of transform coefficients representing the frequency content of the media signal over varying time durations of the media signal. Adaptive block lengths are thus incompatible with traditional decoding schemes developed for fixed block length signals. Also, it would be beneficial to obtain in the decoder a more accurate representation of the original media signal which has been sampled in the encoder to form the signal sample blocks and adaptively divided into blocks of varying numbers of transform coefficients. An example of an encoder in which frequency transform coefficients are obtained by a neural-based autoencoder to output a corresponding lower-dimension representation is provided in <CIT>.

Based on the above, it is therefore an object of the present invention to provide a method for predicting, with a neural network, transform coefficients of an adaptive block length media signal, and in particular an adaptive block length general audio signal.

According to a first aspect of the invention there is provided a method for predicting, with a computer implemented neural network system, at least one transform coefficient representing frequency content of an adaptive block length audio signal as defined in claim <NUM>.

As an alternative to quantized transform coefficients, the transform coefficients may be distorted or impaired. The transform coefficients outputted by the output stage (output neural network) are enhanced in the sense that they more closely resemble an original set of transform coefficients and/or that the enhanced transform coefficients inversely transformed into time domain describe a media signal which is perceived as a higher quality media signal compared to a time domain media signal described by the quantized transform coefficients. Further, a frame, as referred to herein, may include one or more blocks (e.g., a set of blocks).

The invention is at least partially based on the understanding that by converting the (short) first block into a (long) converted block with the first number of transform coefficients the generative properties of the trained main neural network may be introduced into variable block switching decoding. As neural networks have a fixed dimension in their output layers they are incompatible with adaptive length blocks. By converting the first block of the quantized transform coefficients into a converted block, and using a representation of the converted block and a representation of block length information to condition the main neural network, the neural network may predict the at least one (enhanced or non-quantized) transform coefficient in a dynamic manner based on block length. That is, as a representation of the block length information is comprised in the conditioning information (upon which the at least one conditioning variable is based), the main neural network will be trained to respond appropriately to a block having been converted to comprise the first number of transform coefficients.

Additionally, it may further be determined that a block of the frame comprises the first number of quantized transform coefficients. Such a (long) block may not be converted to a converted block and instead a representation of the block with the first number of quantized transform coefficients is comprised in the conditioning information. Besides not converting a long block, the long block may be treated analogously to a determined short block. The transform coefficients outputted by the output stage comprise the first number of transform coefficients representing either a quantized transform coefficient block with the first number of transform coefficients or converted block of the first number of quantized transform coefficients, which in turn represents at least one quantized transform coefficient block with the second number of transform coefficients.

As the main neural network may predict at least one transform coefficient for each of the variable length blocks in sequence, the main neural network takes temporal and/or frequency dependencies into consideration. The main neural network may have a memory function such that previous inputs affect the current processing and such that the prediction of a current (enhanced) at least one transform coefficient is influenced by earlier transform coefficients.

The adaptive length blocks represent a trade-off between frequency and time. A longer block comprises more transform coefficients and will represent a longer duration of the media signal, while a shorter block comprises fewer transform coefficients and will represent a shorter duration of the media signal.

According to a second aspect of the invention there is provided a computer implemented neural network system for predicting transform coefficients representing frequency content of an adaptive block length audio signal as defined in claim <NUM>.

By modifying the weights of the neural network system in response to the measure of the predicted blocks, the training will result in the neural network system learning to predict (generate) at least one transform coefficient from at least one quantized transform coefficient. The training will result in the neural network system learning to properly recognize the at least one conditioning variable representing a short block(s) and process it in a manner such that the resulting at least one predicted transform coefficient closely resembles the at least one transform coefficient of the media signal.

It is understood that, based on acquiring the measure, the neural network system may be trained, preferably iteratively, by modifying parameters (e.g. the weights) of each neural network until a satisfactory small measure is achieved.

The invention according to the second aspect features the same or equivalent embodiments and benefits as the invention according to the first aspect. Further, any functions described in relation to a method, may have corresponding structural features in a system or code for performing such functions in a computer program product.

Experiments have been performed for encoding and decoding a reference media signal with a fixed block length and an adaptive block length. In the case of a fixed block length, a fixed length neural network system was implemented in the decoder, and in the case of adaptive block length the neural network system according to an implementation of the current invention was implemented in the decoder. The fixed block length encoding used <NUM> MDCT coefficient blocks and the adaptive block length encoding used adaptive <NUM>/<NUM> MDCT coefficient blocks. When comparing the decoded signals, adaptive block length switching with the neural network system of the present invention in the decoder showed reduced preecho distortion compared to the fixed block length counterpart.

The present invention will be described in more detail with reference to the appended drawings, showing currently preferred embodiments of the invention.

<FIG> depicts an adaptive block length encoder/decoder system including an encoder <NUM> and a decoder <NUM>. A media signal is received at the input port, at a transient detector <NUM>. The media signal may be divided in a series of time domain frames and may further be divided into a plurality of time domain segments wherein each segment comprises a number of media signal samples. For example, a time domain frame comprises <NUM> signal samples and is divided into four segments of <NUM> samples. The number of signal samples in the time domain frame and the segments (thereby also the number of segments in the time domain frame) is merely exemplary, and may be any number. The transient detector <NUM> is configured to optimize, for each segment, the trade-off between frequency and time resolution by selecting a transform length. In general, the transient detector <NUM> selects a long transform length for segments containing signals with slowly-evolving or stationary temporal characteristics and selects shorter transform lengths for segments containing signals with rapidly-evolving temporal characteristics. By optimizing 'perceptual coding gain' for both short and long signal classes, this approach offers a fundamental advantage over coding with time-invariant transform lengths.

Depending on the temporal characteristics of a segment of the media signal the transient detector <NUM> may select to request that the segment should be represented by a transform domain block with a first number of transform coefficients (for slowly-evolving temporal signal segments) or a plurality of transform domain blocks each comprising a second number of transform coefficients (for rapidly-evolving temporal signal segments), where the first number is greater than the second number. For example, the transient detector <NUM> may request that a slowly-evolving segment is represented with <NUM> transform coefficients Xk while a rapidly-evolving segment is represented with two sets (transform domain blocks) of <NUM> transform coefficients Xk, or four sets of <NUM> transform coefficients Xk. The number of chosen transform coefficients are not limited to the included examples, and any number may be chosen. The transient detector <NUM> may request a number of transform coefficients among a set of block lengths, wherein the set of block lengths comprises at least two lengths such as <NUM>/<NUM>. In some implementations, the set of block lengths comprises at least three or more lengths such as <NUM>/<NUM>/<NUM> among which the transient detector <NUM> may select a suitable length for a block. For example, the transient detector <NUM> may request that a segment is represented by a combination of short blocks of varying lengths. For example, a slowly evolving segment is represented by <NUM> transform coefficients Xk, while a following rapidly evolving segment is represented by one block with <NUM> transform coefficients Xk and two blocks with <NUM> transform coefficients Xk. The transient detector <NUM> generates block length information which represents the requested number of transform domain blocks (and/or number transform coefficients Xk for each block) with which the time domain segments should be represented. The block length information is transmitted to the decoder <NUM>. The transient detector <NUM> passes the block length information to the transform unit <NUM>.

The transform unit <NUM> transforms the segments according to the block length information and outputs the adaptive length transform blocks comprising transform coefficients Xk to a quantizer <NUM>. For the example mentioned in the above, a <NUM> sample time frame having been divided into four <NUM> sample segments is transformed into a series of transform blocks with <NUM>, <NUM>, <NUM>, <NUM> and <NUM> transform coefficients Xk respectively. These transform blocks may then form a transform domain frame (frame) in the encoder <NUM> and/or decoder <NUM>. In other words, a frame may be referred to as a set of one or more transform blocks and/or one or more segments. In parts of the encoder <NUM> and in the decoder <NUM>, the frame to which a transform block belongs may not be explicitly indicated or considered as it suffices to treat the transform blocks in series without regard to their respective time or transform domain frame.

The received media signal is further received by a perceptual model <NUM> which computes a masking threshold. The masking threshold is passed to a bit allocation unit <NUM>.

In the bit allocation unit <NUM>, a bit allocation for the soon to be quantized transform coefficients is assigned based on the received perceptual masking threshold information received from the perceptual model <NUM>. The bit allocation unit <NUM> may allocate bits to reduce or minimize the quantization noise. The bit allocation unit <NUM> passes the bit allocation information to the quantizer <NUM>.

The quantizer <NUM> quantizes the transform coefficients Xk of each block among the adaptive block length blocks by allocating bits to each transform coefficient according to the received bit allocation information, to form quantized transform coefficient X̃k blocks. The quantizer <NUM> transmits the adaptive block length blocks comprising quantized transform coefficients (X̃k) to the decoder <NUM>.

In the decoder <NUM>, a neural network (NN) system <NUM> receives a frame, where each block of the frame comprises at least one quantized transform coefficient X̃k, from the quantizer <NUM> of the encoder <NUM>, and block length information, from the transient detector <NUM> of the encoder. The neural network system <NUM> comprises a main neural network and an output stage (e.g., an output neural network) trained to predict at least one transform coefficient (the at least one predicted transform coefficient Xk) from quantized transform coefficients X̃k. A conversion stage of the neural network system <NUM> converts blocks with the second number of quantized transform coefficients X̃k to converted blocks comprising the first number of quantized transform coefficients X̃k. In some implementations the conversion stage neural network system <NUM> merely passes on blocks with the first number of quantized transform coefficients X̃k. Accordingly, the output stage of the neural network system <NUM> may output a sequence of static length blocks (e.g. each comprising the first number of predicted transform coefficients Xk) wherein some blocks represent a quantized block of the same length and wherein some blocks represent at least one, and in some implementations more than one, short blocks of a different (shorter) length.

The at least one predicted transform coefficient Xk is received at an inversion transform unit <NUM> configured to transform the at least one predicted transform coefficient Xk of each transform domain block into time domain segments (i.e. predicted time domain segments). The inverse transform unit <NUM> may in some implementations receive block length information from the transient detector <NUM> of the encoder <NUM>.

As described in the above, the at least one predicted transform coefficient Xk that arrives as blocks to the inverse transform unit <NUM> may be of a static predetermined length despite some blocks representing one or more quantized blocks of an originally (pre-conversion) shorter length. As the inverse transform unit <NUM> receives information of this original transform domain block length in the form of block length information, the inverse transform unit <NUM> may take necessary pre-inverse transform processing steps. For instance, in response to a predicted long block being associated with an originally short block which was up-sampled to form a converted block in the conversion unit, the inverse transform unit <NUM> may downsample the predicted long block to a predicted short block prior to inverse transforming the short block to the time domain. In another example, at least two short blocks with quantized transform coefficients X̃k are converted into a single converted block in the conversion unit and are predicted by the neural network system as a single long block of at least one predicted transform coefficient Xk. In such a case, the inverse transform unit <NUM> may determine from the block length information that the predicted long block is in fact a prediction based on at least two short blocks (which have been combined) and in response perform pre-inverse transform processing steps, such as splitting or performing an inverse conversion procedure, i.e. the inverse of the conversion carried out in the neural network system <NUM>, to obtain predicted blocks of the same length as determined by the transient detector <NUM> in the encoder <NUM>. The pre-inverse transform processing steps may be carried out by a separate (not shown) unit preceding an inverse transforming unit for some pre-existing coding scheme for adaptive block length media signals. For instance, the neural network system (together with pre-inverse transform processing) may be implemented together with any existing codecs, e.g. to refine AC-<NUM> transform coefficients, or using it with a new codec designed for decoding with a neural network system <NUM>.

In yet a further implementation, the inverse transform unit <NUM> transforms each predicted block (being of a static length) into the time domain such as if the set of predicted blocks is from a static length media signal. In such implementations, the inverse transform unit does not need to consider the block length information and the neural network system effectively converts an adaptive block switching media signal to a static block length media signal. The neural network system <NUM> receives blocks of varying lengths and is trained to output fixed length blocks. The inverse transform unit <NUM> transforms the static length blocks to a time domain media signal.

The inverse transform unit <NUM> outputs a time domain media signal (or a sequence of time domain media signal blocks) suitable for playback by a playback device (not shown). The neural network system <NUM> is configured to receive at least one quantized transform coefficient in a block and predict at least one transform coefficient.

With reference to <FIG>, an embodiment of the computer implemented neural network system <NUM> in <FIG> is depicted in more detail. The neural network system <NUM> is configured to receive a set of adaptive length blocks <NUM> each comprising a set of quantized transform coefficients X̃k representing the frequency content of a partial time segment of a media signal and block length information <NUM> indicating a number of quantized transform coefficients for each block in frame <NUM>, the number of quantized transform coefficients being one of a first number or a second number. The computer implemented neural network system <NUM> further comprises a conversion stage <NUM> that is configured to determine that at least a first block has the second number of quantized transform coefficients, and convert at least the first block into a converted block having the first number of quantized transform coefficients. From frame <NUM> to the conversion stage <NUM>, where frame <NUM> has at least one block with the second number of quantized transform coefficients, the conversion stage generates an output frame <NUM>' wherein the output blocks in the output frame all have the first number of quantized transform coefficients.

The neural network system <NUM> further receives block length information <NUM> indicating a number of quantized transform coefficients for each block in frame <NUM>. The block length information <NUM> thereby indicates the sequence of blocks comprising the first or second number of transform coefficients. The block length information <NUM> may be a sequence of integers or symbols, each integer or symbol representing a block and the value of each integer (or the type of symbol) representing the number of quantized transform coefficients X̃k of that block.

The block length information <NUM> may comprise more than two alternative block lengths. In some implementations a block with the first number of transform coefficients Xk that precedes a block with the second number of transform coefficients Xk may be labelled as a bridge-in block and a block with the first number of transform coefficients Xk that succeeds a block with the second number of transform coefficients Xk may be labelled as a bridge-out block. Accordingly, the block length information <NUM> may be a sequence of four (or more) different integers, one for each of a long block (first number of transform coefficients Xk), a short block (with the second number of transform coefficients Xk), a bridge-in block and a bridge-out block.

The neural network system <NUM> forms at least one conditioning variable <NUM> based on conditioning information, wherein the conditioning information comprises at least two components, (i) information representing the converted block (or representing a block comprising the first number of quantized transform coefficients) and (ii) information representing the block length information <NUM>. In a simple case, information representing the converted block is the quantized transform coefficients X̃k per se, and the block length information representation is an integer. The at least one conditioning variable <NUM> and the main neural network <NUM> may feature a separate dimension for each piece of conditioning information or a single dimension onto which each piece of conditioning information is projected.

The at least one conditioning variable <NUM> is used to condition a main neural network <NUM>. The main neural network <NUM> is trained to predict at least one output variable given at least one conditioning variable <NUM>, and the at least one output variable is provided to an output neural network <NUM> trained to make a final prediction of at least one transform coefficient (i.e. outputting at least one predicted transform coefficient Xk) given at least one output variable from the main neural network <NUM>. The output neural network <NUM> may comprise one or more hidden layers.

The main neural network <NUM> may be any type of neural network, e.g. a deep neural network, a recurrent neural network or any neural network system. The main neural network <NUM> may be a regressive model. The media signal may be any of type of media signal including an audio or video signal. In case of the media signal being an audio signal, the main neural network <NUM> is in a preferred embodiment serving as a general audio generative model in the transform domain. The main neural network <NUM> is configured to operate in the transform domain and is trained to predict at least one output variable given at least one conditioning variable. The at least one output variable may be considered a hidden state and is provided to the output neural network <NUM>, wherein the output neural network <NUM> is configured (e.g. trained) to output at least one predicted transform coefficient given the at least one output variable. The output neural network <NUM> may be implemented together with the main neural network <NUM> as a single unit, e.g. as an output stage of the main neural network <NUM> or as a separate neural network. Regardless, the output neural network <NUM> and the main neural network <NUM> exchange hidden state information.

The at least one transform coefficient Xk is thus predicted from the at least one quantized transform coefficients X̃k. by the main neural network <NUM> and the output neural network <NUM> by capturing temporal and/or frequency dependencies of the representation of the quantized transform coefficients. That is, the main neural network <NUM> and the output neural network <NUM> may be trained such that previous representations of transform coefficients having been processed by the main neural network <NUM> may influence the prediction of the current at least one transform coefficient. Additionally or alternatively, the main neural network <NUM> and output neural network <NUM> are trained such that interdependencies between transform coefficients in a current block and past blocks are considered. As the transform coefficients represent frequency content, the main neural network <NUM> and the output neural network <NUM> may be trained to predict at least one transform coefficients by learning how the frequency content (which is represented in the transform coefficients) of a first frequency band affects the frequency content of a second frequency band.

In some implementations the neural network system <NUM> further comprises an additional neural network, such as a conditioning neural network <NUM> connected to receive output from the conversion unit <NUM> and receive block length information from block length information neural network <NUM>. The conditioning neural network <NUM> and the block length information neural network <NUM> are used to predict a respective piece of conditioning information and may be any type of neural network, e.g. a convolutional layer, and using one type does not necessitate the other type.

The conditioning neural network <NUM> and/or the block length information neural network <NUM> may be trained to predict a respective at least one output variable, where the at least one conditioning variable <NUM> is then obtained as the sum of the respective at least one predicted output variable. Further, the at least one conditioning variable <NUM> being passed to the main neural network <NUM> (being e.g. a sum of the respective at least one output variable from the conditioning neural network <NUM> and block length neural network <NUM>) may be regarded as a hidden neural network layer. Besides establishing an inner dimension (as a hyperparameter) for the hidden layer which matches the input dimension of the main neural network <NUM>, the neural network system <NUM> may be operated (and trained) without any constraint on the interpretability of the hidden layer. For example, the conditioning information representing the quantized transform coefficients and the representation of the block length information may each be at least one output variable in the shape of matrices of a dimension matching the inner dimension. The at least one condition variable <NUM> may then be the sum of the at least one matrix output variable. In a further example, the matrices are two-dimensional and comprise a single row or column (i.e. a vector).

The conditioning neural network <NUM> is trained to predict a representation of a block from output frame <NUM>' given the quantized transform coefficients X̃k of the block. By predicting the representation of the quantized transform coefficients of the converted block, with a conditioning neural network <NUM> trained to predict the representation, a representation which further facilitates prediction by the main neural network <NUM> may be achieved. As opposed to assigning a static translation function for the quantized transform coefficients X̃k that translates them into information representing the quantized transform coefficients X̃k, the conditioning neural network <NUM> may be trained to predict a representation which facilitates making the final prediction by the main neural network <NUM> and the output neural network <NUM>.

In a similar manner, the block length information neural network <NUM> is trained to predict a representation of the block length information given block length information <NUM>. By implementing a block length neural network <NUM> trained to predict a representation of the block length information given block length information <NUM> of at least the first block, the conditioning information used to condition the main neural network <NUM> will carry information indicating the number of quantized transform coefficients X̃k in the first block in a format that facilitates prediction of at least one transform coefficient Xk by the main neural network <NUM> and the output neural network <NUM>. In one example, the block length neural network <NUM> outputs a representation of the block length information which indicates a block with the first number of transform coefficients Xk. Accordingly, the main neural network <NUM> is conditioned differently, and will respond differently, when the represented quantized transform coefficients X̃k are from a converted block or from a quantized block with the first number of transform coefficients X̃k. As the main neural network <NUM> and output neural network <NUM> have been trained to predict at least one transform coefficient from information representing the quantized transform coefficients X̃k together with conversion unit <NUM>, the prediction of the at least one transform coefficient may be accomplished regardless of the manner in which the converted block was constructed from at least the first block.

As opposed to conditioning the block length neural network with e.g. an integer from the sequence of integers, some implementations of the neural network system <NUM> comprise a One-Hot encoder <NUM>, which converts the block length information <NUM> to One-Hot vectors which in turn are used to condition the block length neural network <NUM>. The block length information is categorical and indicates for each block a separate state (e.g. long, short, bridge-in or bridge-out). With One-Hot encoding, these categorises are separated into individual vector elements which facilitates the training and prediction of the block length neural network <NUM> by clearly distinguishing between the different possible states. For example, One-Hot encoding promotes a strong spatial dependence between the predicted at least one output variable and which input element of the input layer of the block length neural network that receives the one hot (on-state) vector element.

In some implementations the neural network system <NUM> further receives for each block perceptual model coefficients pEnvQ and/or a spectral envelope. The conditioning information may thus further include additional pieces of information that are a representation of perceptual model coefficient pEnvQ information and/or spectral envelope information. The perceptual model coefficients pEnvQ and/or spectral envelope may be processed in parallel with the block length information and the quantized transform coefficients and either combined with other information in the at least one conditioning variable <NUM> or provided as side information in a separate dimension, to the main neural network <NUM>.

The set of perceptual model coefficients pEnvQ may be derived from a perceptual model, such as those occurring in the encoder. The perceptual model coefficients pEnvQ are computed per frequency band and are preferably mapped onto the same resolution as the frequency coefficients of a block to facilitate processing.

In implementations where a single short block has been converted to a converted block, the pEnvQ coefficients are converted to an equivalent long block representation by an analogous conversion procedure and used as conditioning information. For example, if a short block is up-sampled, the pEnvQ coefficients are up-sampled in the same way.

It is noted that with a neural network system <NUM> that is 'trained' in implementations featuring more than one neural network, all the neural networks in the system are, during at least a portion of the training, trained together. For example, the block length neural network <NUM> may be trained together with the main neural network <NUM> wherein the inner parameters (e.g. weights) of each neural network <NUM>, <NUM> are modified to optimize some measure of the predicted at least one transform coefficient Xk compared to some target predicted at least one transform coefficient, such as the original non-quantized transform coefficients Xk. The block length neural network <NUM> is then trained to output at least one conditioning variable <NUM> which brings the predicted at least one transform coefficient of the main neural network <NUM> and the output neural network <NUM> to resemble the original transform coefficients as closely as possible. The main neural network <NUM> and output neural network <NUM> are simultaneously trained to predict at least one transform coefficient Xk that resemble the original transform coefficients Xk as closely as possible.

The conversion in the conversion unit <NUM> of blocks with the second number of transform coefficients may involve the up-sampling of a block with the first number of quantized transform coefficients X̃k to a converted block. Up-sampling may include linear or polynomial interpolation (and optionally extrapolation) of the second number of quantized transform coefficients to the first number of quantized transform coefficients. Alternatively, up-sampling to form a converted block may comprise one of: repeating each quantized transform coefficient a predetermined number of times, adding zero elements in between non-zero elements or interleaving the quantized transform coefficients X̃k. Alternatively, any other suitable up-sampling, expansion or interpolation technique is applicable. In some implementations the conversion unit <NUM> merely forwards the quantized transform coefficients X̃k of a block to the main neural network <NUM>, which is trained to predict at least one output parameter for the output neural network <NUM>. In this case the main neural network <NUM> will learn to recognize a block with the second number of quantized transform coefficients X̃k and absorb by training the functions of the converter.

As an alternative to converting in the conversion unit <NUM> a first block comprising the second number of quantized transform coefficients X̃k into at least two blocks, a first block and a second block, each comprising the second number of quantized transform coefficients X̃k, the first block and the second block may jointly be converted into a converted block comprising the first number of quantized transform coefficients X̃k. Accordingly, the main neural network <NUM> and output neural network <NUM> may be trained to predict at least one transform coefficient Xk given a representation of a converted block comprising a first number of quantized transform coefficients X̃k, where the quantized transform coefficients X̃k of the converted block originate from the quantized transform coefficients X̃k of at least the first and second block.

In general, the at least first and second blocks having the second number of quantized transform coefficients X̃k may be N consecutive blocks having the second number of quantized transform coefficients X̃k, where the first number is a multiple N of the second number. The N consecutive blocks may then be converted to a converted block with the first number of quantized transform coefficients X̃k. The adaptive block switching media signal may, for example, include a first number of quantized transform coefficients X̃k equal to <NUM> and a second number equal to <NUM>, i.e. for N = <NUM>. A first number equal to <NUM> and N=<NUM> would result in four short blocks, each comprising <NUM> quantized transform coefficients X̃k, being converted into one converted block. In yet a further example, N = <NUM>, when the first number of transform coefficients is <NUM>, then the second number of quantized transform coefficients X̃k is <NUM>.

Converting at least the first and second block into a converted block may comprise concatenating at least the first and the second block into a converted block. Concatenation is an efficient and easily implemented method of converting at least the first and second block into a converted block.

In some implementations the conversion unit <NUM> receives for each block a representation of a respective time domain window function, where the window function of the first and second block partially overlap.

The window functions may be received together with the quantized transform coefficients X̃k or with the block length information <NUM> (being passed onto the conversion unit <NUM>). Alternatively, the window functions may be constructed from the block length information <NUM> (being passed to the conversion unit <NUM>). Or, the window functions may be constructed by determining the number of quantized transform coefficients X̃k for a block in the conversion unit <NUM> by utilizing the correlation between number of quantized transform coefficients in a block and the sequence of the blocks with at least the first and second numbers of quantized transform coefficients in each block. For example, a block with the first number of quantized transform coefficients X̃k is associated with a long window function and a block with the second number of transform quantized coefficients X̃k is associated with a short window function. In a further example, a block with the first number of quantized transform coefficients X̃k may be associated with a bridge-in window function if this block precedes a block with the second number of quantized transform coefficients X̃k.

In <FIG>, all of the functions and units described as operating up-stream of the (optional) conditioning neural network <NUM> and the (optional) block length information neural network <NUM> may be referred to as a pre-processing unit or an adaptive block pre-processing unit. The pre-processing unit may thus be a multiple input multiple/single output unit, e.g. receiving block length information <NUM> and quantized transform coefficients X̃k and output information representing the quantized transform coefficients X̃k and representing the block length information <NUM> as separate pieces of information (at least one variable) or a combined piece of information (at least one variable).

With further reference to <FIG> there is depicted a flow chart illustrating a method for training the neural network system, for example the embodiment depicted in <FIG>. At S311 a set of adaptive length target prediction (true) blocks are provided. This occurs alongside providing a set of training blocks being an impaired representation of the target prediction blocks (e.g. a quantized representation) at S321. The target prediction blocks comprise a non-quantized set of transform coefficients Xk. The training blocks are provided to the neural network system <NUM> and processed such that a set of predicted blocks are obtained at S331. By comparing the outputted predicted blocks comprising the at least one predicted transform coefficient Xk with the target prediction blocks, a measure, e.g. of similarity, is obtained at S332. The measure may be an error measure, wherein a low error measure indicates a high level of similarity. The measure may be a negative likelihood, such as the negative log likelihood (NLL), wherein a low measure indicates a high level of similarity. The measure may be a Mean Absolute Error (MAE) or a Mean Square Error (MSE), where a high level of similarity will be indicated by a low MAE or MSE. At S333 the measure is used for modifying the weights of the neural network system <NUM> to reduce or minimize the measure.

In one example, the measure is referred to as a loss function or 'loss', as is directly computed as the NLL as <MAT>.

In calculating the NLL loss the predicted at least one transform coefficent Xk is repersented by at least one distribution parameter for the at least one predicted transform coefficent Xk. The NLL function is thus applied to the at least one distribution parameter which repersents the predicted at least one transfrom coefficent Xk. The at least one distribution parameter parametrizes a probability distribution for the at least one the at least one predicted transform coefficent Xk.

In other implementations the loss is calculated as the MSE according to: <MAT> or the loss may be calculated as the MAE according to: <MAT>.

In calculating the MSE and MAE loss the at least one predicted transform coefficient Xk is used as such. In some cases, a predicted block may represent more than one training block (and the associated target prediction block) with a single predicted converted block, in such cases the predicted blocks may be inversely converted into blocks individually corresponding to a training block (and the associated target prediction block) such that the measure may be computed.

With reference to <FIG> there is illustrated a sequence of time domain window functions <NUM>, 32a, 32b, <NUM>. <FIG> illustrates the window sequence for a typical <NUM>:<NUM> block length switch. The first long window <NUM> is followed by two short windows 32a, 32b, which in turn are followed by a second long window <NUM>. The short time domain window functions 32a, 32b may overlap by <NUM>%, where adding the squared short window functions results in a value of one for the overlapping portion. Additionally, the sum of the square of each window function <NUM>, 32a, 32b, <NUM> will result in a value of one for every overlap.

In some implementations, the long windows <NUM>, <NUM> may further be a bridge-in window <NUM> and a bridge-out window <NUM> respectively, especially adapted to respectively precede and succeed short windows 32a, 32b. The window functions <NUM>, 32a, 32b, <NUM> are at least partially overlapping in time. Each window function <NUM>, 32a, 32b, <NUM> is associated with a set of transform coefficient blocks, a long transform coefficient block with a long window function <NUM>, <NUM>, and a short transform coefficient block with a short window function 32a, 32b.

In some additional implementations, where the number of transform coefficients in each block is one out of more than two alternatives (e.g. one out of <NUM>, <NUM> and <NUM> coefficients as mentioned above) the bridge-in window <NUM> and a bridge-out window <NUM> functions may comprise more than two bridging window functions, e.g. one for each type of transition between the variable length blocks. If the blocks have a length of one out of <NUM>, <NUM> and <NUM> there may be defined an in and out bridging window function for each of: <NUM> to <NUM>, <NUM> to <NUM> and <NUM> to <NUM>.

With further reference to <FIG> there is illustrated a long converted window <NUM> (with an associated long converted block) that is the result of a conversion of two short window functions 32a, 32b (and two short transform coefficient blocks).

By inverse transforming the quantized transform coefficients of a first and second (short) block (their respective window function is shown in <FIG> as 32a and 32b) back into a windowed time domain representation, they may be merged into a long converted block. This may be achieved by overlap adding the windowed time domain representation of the first and second blocks and transforming the overlap added time domain representation of the first and second blocks into a converted block having the first number of quantized transform coefficients.

For example, if the transform coefficients are Modified Discrete Cosine Transform (MDCT) coefficients, the intervening short blocks (associated with window functions 32a, 32b) may be merged into a single long block by inverting the MDCT to short time domain segments and overlap adding the short time domain segments. A DCT type <NUM> may then be used to compute transform coefficients of the equivalent converted long block <NUM> with a flat-top window. The window sequence after this merging/conversion operation is shown in <FIG>. It is further noted that this procedure of conversion may be accomplished while preserving perfect reconstruction properties of the transform coefficents (in the absence of quantization).

With reference to <FIG>, there is depicted a flow chart illustrating a method for predicting at least one transform coefficient from quantized transform coefficients according to an embodiment of the invention. At S111, the neural network system receives a frame comprising quantized transform coefficients. The neural network system determines that at least one block of the frame comprises the second number of transform coefficients at S112 and proceeds by converting at least the block with the second number of transform coefficients into a converted block with the first number of transform coefficients at S113. Information representing the quantized transform coefficients of a converted block is one piece of information upon which at least one conditioning variable, used to condition the main neural network at S131, is based upon. Optionally, the method involves conditioning a conditioning neural network at S114 with information representing the quantized transform coefficients of a converted block and using the at least one output variable of the conditioning neural network to condition the main neural network at S131.

Further, the method involves receiving block length information at S121. A representation of the block length information is used as one piece of information for conditioning the main neural network at S131. Optionally, the block length information is used to first condition a block length neural network at S123 wherein the predicted at least one output variable of the block length neural network is used to condition the main neural network at S131. Also, some embodiments comprise One-Hot encoding of the block length information at S122, wherein the One-Hot encoded block length information is used to either condition the block length neural network at S123 or as information which is part of the information used to condition the main neural network at S131.

At S131, the main neural network predicts at least one output variable given the at least one conditioning variable and wherein the at least one output variable is provided to the output stage (e.g., an output neural network) at S132. The output stage at S132 predicts the at least one transform coefficient.

<FIG> depicts a flow chart illustrating a method for obtaining training blocks (training blocks for input and target predicted blocks for comparison with the output) for training a neural network system for predicting the transform coefficients of an adaptive block length media signal according to embodiments of the present invention. At S211, a set of transform blocks is obtained. For example, a batch of waveforms or a media signal has been divided into a set of time domain segments (e.g. forming a time domain frame) and each time domain segment has been transformed into a set of varying length transform blocks (e.g. a transform domain frame). Alternatively, a batch of waveforms or a media signal has been processed with a transient detector as described in the above to determine the length of each block. At S212 it is determined that a first block comprises the second number of transform coefficients and this block is converted at S213 to a converted block with a first number of transform coefficients. At S221, a target predicted block is obtained. The target predicted block obtained at S221 may be the converted block itself.

At S231 the converted block is quantized to form a quantized block. That is, the quantized block does not represent the complete information originally present in the determined first block, thus the quantized block may be referred to as an impaired block which the neural network should learn to use to predict a nonimpaired block. At S232 a training block is obtained from the quantized block obtained at S231. The training block may be quantized block as such. In some implementations, the further steps of using the target training block as input to the neural network during training and using the target predicted block as the training is included.

Blocks determined to comprise the first number of transform coefficients may be processed analogously to obtain training blocks and target predicted blocks, wherein the step S213 is omitted.

In some implementations, a media signal or a batch of waveforms is processed with a transient detector which determines the transform length as discussed in the above. Thus, the set of transform blocks will contain all different types of blocks and window functions.

In the above, possible methods of training and operating a deep-learning-based system for determining an indication of an audio quality of an input audio sample, as well as possible implementations of such system have been described. Additionally, the present disclosure also relates to an apparatus for carrying out these methods. An example of such apparatus may comprise a processor (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), one or more application specific integrated circuits (ASICs), one or more radio-frequency integrated circuits (RFICs), or any combination of these) and a memory coupled to the processor. The processor may be adapted to carry out some or all of the steps of the methods described throughout the disclosure.

The apparatus may be a server computer, a client computer, a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a cellular telephone, a smartphone, a web appliance, a network router, switch or bridge, or any machine capable of executing instructions (sequential or otherwise) that specify actions to be taken by that apparatus. Further, the present disclosure shall relate to any collection of apparatus that individually or jointly execute instructions to perform any one or more of the methodologies discussed herein.

The present disclosure further relates to a program (e.g., computer program) comprising instructions that, when executed by a processor, cause the processor to carry out some or all of the steps of the methods described herein.

Yet further, the present disclosure relates to a computer-readable (or machine-readable) storage medium storing the aforementioned program. Here, the term "computer-readable storage medium" includes, but is not be limited to, data repositories in the form of solid-state memories, optical media, and magnetic media, for example.

Unless specifically stated otherwise, as apparent from the following discussions, it is appreciated that throughout the disclosure discussions utilizing terms such as "processing", "computing", "calculating", "determining", "analyzing" or the like, refer to the action and/or processes of a computer or computing system, or similar electronic computing devices, that manipulate and/or transform data represented as physical, such as electronic, quantities into other data similarly represented as physical quantities.

The methodologies described herein are, in one example embodiment, performable by one or more processors that accept computer-readable (also called machine-readable) code containing a set of instructions that when executed by one or more of the processors carry out at least one of the methods described herein. Any processor capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken are included. Thus, one example is a typical processing system that includes one or more processors. Each processor may include one or more of a CPU, a graphics processing unit, and a programmable DSP unit. The processing system further may include a memory subsystem including main RAM and/or a static RAM, and/or ROM. A bus subsystem may be included for communicating between the components. The processing system further may be a distributed processing system with processors coupled by a network. If the processing system requires a display, such a display may be included, e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT) display. If manual data entry is required, the processing system also includes an input device such as one or more of an alphanumeric input unit such as a keyboard, a pointing control device such as a mouse, and so forth. The processing system may also encompass a storage system such as a disk drive unit. The processing system in some configurations may include a sound output device, and a network interface device. The memory subsystem thus includes a computer-readable carrier medium that carries computer-readable code (e.g., software) including a set of instructions to cause performing, when executed by one or more processors, one or more of the methods described herein. Note that when the method includes several elements, e.g., several steps, no ordering of such elements is implied, unless specifically stated. The software may reside in the hard disk, or may also reside, completely or at least partially, within the RAM and/or within the processor during execution thereof by the computer system. Thus, the memory and the processor also constitute computer-readable carrier medium carrying computer-readable code. Furthermore, a computer-readable carrier medium may form, or be included in a computer program product.

In alternative example embodiments, the one or more processors operate as a standalone device or may be connected, e.g., networked to other processor(s), in a networked deployment, the one or more processors may operate in the capacity of a server or a user machine in server-user network environment, or as a peer machine in a peer-to-peer or distributed network environment. The one or more processors may form a personal computer (PC), a tablet PC, a Personal Digital Assistant (PDA), a cellular telephone, a web appliance, a network router, switch or bridge, or any machine capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by that machine.

Note that the term "machine" shall also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein.

Thus, one example embodiment of each of the methods described herein is in the form of a computer-readable carrier medium carrying a set of instructions, e.g., a computer program that is for execution on one or more processors, e.g., one or more processors that are part of web server arrangement. Thus, as will be appreciated by those skilled in the art, example embodiments of the present disclosure may be embodied as a method, an apparatus such as a special purpose apparatus, an apparatus such as a data processing system, or a computer-readable carrier medium, e.g., a computer program product. The computer-readable carrier medium carries computer readable code including a set of instructions that when executed on one or more processors cause the processor or processors to implement a method. Accordingly, aspects of the present disclosure may take the form of a method, an entirely hardware example embodiment, an entirely software example embodiment or an example embodiment combining software and hardware aspects. Furthermore, the present disclosure may take the form of carrier medium (e.g., a computer program product on a computer-readable storage medium) carrying computer-readable program code embodied in the medium.

The software may further be transmitted or received over a network via a network interface device. While the carrier medium is in an example embodiment a single medium, the term "carrier medium" should be taken to include a single medium or multiple media (e.g., a centralized or distributed database, and/or associated caches and servers) that store the one or more sets of instructions. The term "carrier medium" shall also be taken to include any medium that is capable of storing, encoding or carrying a set of instructions for execution by one or more of the processors and that cause the one or more processors to perform any one or more of the methodologies of the present disclosure. A carrier medium may take many forms, including but not limited to, non-volatile media, volatile media, and transmission media. Non-volatile media includes, for example, optical, magnetic disks, and magneto-optical disks. Volatile media includes dynamic memory, such as main memory. Transmission media includes coaxial cables, copper wire and fiber optics, including the wires that comprise a bus subsystem. Transmission media may also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications. For example, the term "carrier medium" shall accordingly be taken to include, but not be limited to, solid-state memories, a computer product embodied in optical and magnetic media; a medium bearing a propagated signal detectable by at least one processor or one or more processors and representing a set of instructions that, when executed, implement a method; and a transmission medium in a network bearing a propagated signal detectable by at least one processor of the one or more processors and representing the set of instructions.

It will be understood that the steps of methods discussed are performed in one example embodiment by an appropriate processor (or processors) of a processing (e.g., computer) system executing instructions (computer-readable code) stored in storage. It will also be understood that the disclosure is not limited to any particular implementation or programming technique and that the disclosure may be implemented using any appropriate techniques for implementing the functionality described herein. The disclosure is not limited to any particular programming language or operating system.

Reference throughout this disclosure to "one example embodiment", "some example embodiments" or "an example embodiment" means that a particular feature, structure or characteristic described in connection with the example embodiment is included in at least one example embodiment of the present disclosure. Thus, appearances of the phrases "in one example embodiment", "in some example embodiments" or "in an example embodiment" in various places throughout this disclosure are not necessarily all referring to the same example embodiment. Furthermore, the particular features, structures or characteristics may be combined in any suitable manner, as would be apparent to one of ordinary skill in the art from this disclosure, in one or more example embodiments.

It should be appreciated that in the above description of example embodiments of the disclosure, various features of the disclosure are sometimes grouped together in a single example embodiment, Fig., or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more of the various inventive aspects. This method of disclosure, however, is not to be interpreted as reflecting an intention that the claims require more features than are expressly recited in each claim.

However, it is understood that example embodiments of the disclosure may be practiced without these specific details.

Claim 1:
A method for predicting, with a computer implemented neural network system (<NUM>), at least one transform coefficient representing frequency content of an adaptive block length audio signal, comprising the steps of:
receiving (S111) a frame including one or more blocks, each block of the frame comprising a set of quantized transform coefficients representing a partial time segment of said audio signal,
receiving (S121) block length information indicating a number of quantized transform coefficients for each block of the frame, the number of quantized transform coefficients being one of a first number or a second number, wherein said first number is greater than said second number,
determining (S112) that at least a first block of the frame has said second number of quantized transform coefficients,
converting (S113) at least said first block into a converted block having said first number of quantized transform coefficients,
conditioning (S131) a main neural network (<NUM>) trained to predict at least one output variable given at least one conditioning variable, the at least one conditioning variable being based on conditioning information, said conditioning information comprising a representation of said converted block and a representation of block length information for said first block,
providing (S132) said at least one output variable to an output stage (<NUM>) configured to provide at least one predicted transform coefficient from said at least one output variable.