Patent ID: 12190901

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Referring now to the drawings,FIG.1schematically illustrates a signal processor100which receives a source input signal90and/or metadata defining properties of the source input signal, along with a target input signal or metadata defining properties of the target input signal80. The use of metadata will be discussed further below. The signal processor applies a processing operation so as to generate an output signal70which contains the actual content of the source input signal but with the spectrum of its envelope modified to be the same as or at least closer to that of the target input signal.

This processing operation will be referred to below as so-called “dynamic matching”. Its aim is to impose on the source input signal a dynamic behaviour that is abstracted from properties of the target input signal. Such a dynamic matching process may involve for example at least partly conforming the dynamics of a source signal to that of a target audio signal.

Purely by way of example, a possible use of the dynamic matching process is in the generation of alternate language voice files to be used in, for example, a video game soundtrack. Typically in a video game situation, the game is authored with a single primary language voice track (often but not always in English) and, although the voice track may include a very large number of individual voice files, it is often the case that a lot of effort is applied to the process of mixing the voice track in the primary language. Then, however, if the game is to be sold with other language variants, it is necessary to generate corresponding voice tracks in each language. To apply the same effort in terms of audio mixing to those alternate language voice tracks would potentially be uneconomic, but by using the dynamic matching process summarised with reference toFIG.1and described in more detail below, a similar “feel” or loudness behaviour can be obtained for the listener in each of the languages by imposing the dynamic behaviour of the target input signal (in this example, a particular voice track representing a portion of dialogue in the primary language) to each individual source input signal (in this example, the corresponding portion of dialogue in one of the alternate languages).

Note that although the example situation just described relates to audio signals, in fact the target input signal, source input signal and output signal could be audio signals or could be signals representing other values and in general are simply defined as a sequence of values which may (for convenience of the processing) be normalised to a given range.

FIGS.2ato2dare schematic diagrams illustrating such a signal processing operation. InFIG.2a, the temporal envelope of the target input signal is shown, with a time access running from left to right as drawn.FIG.2bshows an example temporal envelope of the source input signal on the same time axis.

As part of the processing operations carried out by the signal processor100, a modification indication (FIG.2c) is generated which, when applied to the source input signal ofFIG.2bas a gain modification or the like, produces the output signal ofFIG.2dwhich contains the content of the source input signal but exhibits the dynamic behaviour of the target input signal.

By way of summary, an overview of signal processing methods will now be described, with a more detailed discussion being given below.

FIG.3is a schematic flowchart illustrating a summary signal processing method comprising:comparing (at a step300) a first frequency domain representation of a sequence of power values for respective windows of source input samples of a source input signal with a second frequency domain representation of a sequence of power values for respective windows of target input samples of a target input signal so as to generate a frequency domain difference representation;inverse-frequency-transforming (at a step310) the frequency domain difference representation to generate a modification indication; andapplying (at a step320) the modification indication to the source input samples to generate respective output samples of an output signal.

FIG.4is a schematic flowchart illustrating a signal processing method comprising:detecting (at a step400) a sequence of power values for respective windows of input samples of an input signal;generating (at a step410) a frequency domain representation of the sequence of power values; andproviding (at a step420) the frequency domain representation of the sequence of power values as a metadata signal to accompany the input signal.

Note that other data can also be included with the metadata, and examples will be given below.

An optional final step430can be employed to store the signal and metadata on a storage medium such as a non-transitory machine-readable storage medium (for example the medium500ofFIG.5).

FIGS.6and7are schematic flow charts illustrating the example dynamic matching process in more detail. In particular,FIG.6provides an intermediate level of detail andFIG.7expands on certain aspects ofFIG.6to provide a greater level of detail of an example embodiment. Reference is also made toFIGS.8to19which illustrate some of the stages in the process.

As mentioned, an aim of the process to be discussed is to make a source input signal sound more like (in the example of audio signals) a target input signal in terms of its dynamic properties, by generating an output signal as a modified version of the source input signal. The example to be discussed will be referenced to audio signals for clarity of the explanation but this is not an essential feature of the present disclosure.

The process as shown inFIG.6starts with a source audio signal101. This is represented as successive audio samples at a sampling rate (for example of 44.1 kHz), but as discussed above may be a different format of audio signal or indeed another numeric signal altogether. For convenience, the audio samples may be normalised (for example to a range between −1 and +1) but this is not essential to the process to be described.

At a step203, the root mean square (RMS) power is evaluated for respective windows of the source input samples of the signal101. These windows may be successive (adjacent and contiguous) or may be overlapping, or may be intermittent with respect to the time axis of the source audio signal. The windows may be all of the same length (for example, equivalent to 50 ms (milliseconds) of the source audio signal101, or at least a range within an order of magnitude of 50 ms) or may have different lengths. For convenience, a system of successive contiguous windows of the same length (at least a consistent length for an individual input signal) is used.

The RMS power is evaluated for each of the windows giving a set of power values205representing the windows RMS power values.

To assist in this discussion,FIGS.8to19are schematic diagrams illustrating various stages in the signal processing operations to be described.FIG.8represents the source audio signal101on a time axis running from left to right, andFIG.9schematically illustrates the windowed RMS power values205, again on a time axis running from left to right. Each window has a window length900and in the example shown, the windows all have the same window length and are contiguous. The RMS power is shown on a vertical axis inFIG.9.

At a step207, a time to frequency domain transform is performed on the windowed RMS power values205, such as a fast Fourier transform (FFT), to generate the spectrum209(FIG.10) of the windowed RMS power values.

A step311adjusts this spectrum to the “right spectral form”, which in this case is a spectrum221shown schematically inFIG.12. Referring to the more detailed discussion ofFIG.7, the step311involves evaluating the spectrum209, which as shown schematically inFIG.10is a symmetric spectrum formed of two halves, retaining the first half (FIG.11) except the very first (dc) element to give a power spectrum114. This is then interpolated to a log scale along the x (frequency band or bin) axis with fewer bands or bins then the spectrum114. For example, the number of frequency bands can be reduced to, for example, 20 bands, and the y (amplitude) scale is converted to a log representation, providing the spectrum221ofFIG.12. The result of the processing just described is that this represents a downsampled version of the spectrum114.

In examples, the generation of the spectra221,243can involve frequency-transforming (207) the sequence of power values for respective windows of source input samples to generate an intermediate frequency domain representation (209) according to a first plurality of frequency bands; and downsampling (311) the intermediate frequency domain representation to generate the first frequency domain representation having a second plurality of frequency bands than the intermediate frequency domain representation, the second plurality being lower than the first plurality; and in which the second frequency domain representation has the second plurality of frequency bands (such as 20 bands).

In the example shown inFIG.6, corresponding processing is applied to the target audio signal123using steps225,229,333to generate intermediate signals227(equivalent to205),231(equivalent to209) and243(equivalent to221).FIGS.13aand13brespectively show the spectral profiles221(for the source audio signal101) and243(for the target audio signal123).

At a step245, the difference is evaluated between the spectral profiles221and243so as to provide a difference representation247(FIG.14).

A step349involves setting the difference values247to the “right difference value form” which, as shown in more detail inFIG.7, involves converting the y axis back to a linear scale, interpolating the x axis back to a linear scale with the original number of elements (that is to say, the number of elements in the spectrum209) and rebuilding the spectrum at a step251by reflecting the spectrum in the x axis.

These processes are shown inFIGS.15to17, in thatFIG.15shows the difference247interpolated back to the original number of elements in the spectrum209.FIG.16shows the form of the reflection used (a reflection around a frequency position1600).

Returning toFIG.6, the spectrum252is multiplied at a step253, term by term, (which is to say [band or bin] by [band or bin]) by the spectrum209to produce the spectrum255(FIG.17) required of the output signal to be generated.

In examples, the method therefore comprises generating (245) an intermediate difference representation (247) in dependence upon a difference between the first frequency domain representation of the sequence of power values for respective windows of the source input samples and the second frequency domain representation of the sequence of power values for respective windows of the target input samples; upsampling (349) the intermediate difference representation to the first plurality of frequency bands to generate an upsampled intermediate difference representation (252); and combining (253) the upsampled intermediate difference representation with the intermediate frequency domain representation to generate the frequency domain difference representation.

A step257involves performing an inverse frequency transform (the inverse of the transform formed at the step207,229) on the spectrum255to generate an indication259representing the RMS power, window by window, required of the output signal.

At a step261, this indication259is divided by corresponding windows of the source window RMS power values205to generate a representation262referred to inFIG.6as the “RMS automation” but which represents a gain modification, window by window, to be applied to the source audio signal101.FIG.18schematically represents the RMS automation262.

The step261therefore provides an example of comparing the modification indication with the sequence of power values for respective windows of source input samples to generate a correction indication (262) for applying to the source input samples, for example after an operation of upsampling (263) the correction indication to the same number of samples as the number of source input samples.

At a step263, the RMS automation262is interpolated to the length of the source audio signal101to give a modification indication1900shown inFIG.19(similar in form to the signal shown inFIG.2cdescribed above). This is then multiplied, term by term (sample by sample) at a step265by the source audio signal samples to generate the output signal267(FIG.2d).

Comparing the summary processes ofFIGS.3and4withFIG.6, an example mapping is:first and second frequency domain representations:221,243step300: step245through to generation of the difference indication as252or255step310: step257through to generation of the modification indication as259or262step320: at least the step265but possibly including the steps261and/or263step400: step203step410: at least step207, possibly also including step311as well

As an example of apparatus to implement the methods discussed above,FIG.20schematically represents signal processing apparatus comprising:comparator circuitry2000configured to compare a first frequency domain representation of a sequence of power values for respective windows of source input samples of a source input signal with a second frequency domain representation of a sequence of power values for respective windows of target input samples of a target input signal so as to generate a frequency domain difference representation;transform circuitry2010configured to inverse-frequency-transform the frequency domain difference representation to generate a modification indication; andoutput circuitry2020to apply the modification indication to the source input samples to generate respective output samples of an output signal.

The input to the comparator input circuitry2000can be either the spectral profiles221,243generated by the processors ofFIG.6(which is to say, the steps203,207,311for the spectral profile221or the steps225,229,333for the spectral profile243) all can be provided as metadata2030(for the spectral profile221or2040for the spectral profile233).

This allows repeated applications of the process ofFIG.6without necessarily needing to recreate the profile information each time. For example, the same target profile may be applied to multiple source audio signals, or in alternative arrangements, the same source audio signal may be processed according to multiple target profiles. The metadata can be generated and potentially stored by the process ofFIG.4discussed above. Note that even if metadata2030is provided to give the spectral profile231of the source audio signal101, the source audio signal itself is still required by the output circuitry2020in order to perform the step265ofFIG.6to generate the output signal267.

Note that in at least some embodiments, the metadata may include not only the spectrum221(243) but also the power values205(227) and/or the spectrum209(231). Alternatively, since the spectrum221can be generated from the spectrum209which in turn can be generated from the values205, any one or more of the values/spectra205,209,221(or227,231,243for the target) may be provided as metadata so as either to reduce the need for processing leading up to the step245, or to eliminate the need for processing leading up to the step245in that branch ofFIG.6. If at least the values205(227) are provided then the others can readily be derived from that data.

In the case that metadata is not used, then (still referring toFIG.20) a detector2002is provided in the source signal path (and/or a detector2102in the target signal path) and a generator2004is provided in the source signal path (and/or a detector2104in the target signal path). For the respective signal the detector2002,2102is configured to detect the sequence of power values for the respective windows of the respective input samples. For the respective signal the generator2004,2104is configured to generate at least the frequency domain representation of the sequence of power values.

In a separate mode of operation, the circuitries2002,2004can be used in conjunction with metadata output circuitry2060, and the circuitries2102,2104can be used in conjunction with metadata output circuitry2050, to generate and provide metadata2062,2052for the source and target signals respectively. This can be performed as a separate process to the remainder of the processing carried out inFIG.20, which is to say that an apparatus could be provided which has the circuitries2002,2004,2060and/or the circuitries2102,2104,2050only, in order just to derive the metadata discussed above. In this way, the arrangement ofFIG.20provides an example of signal processing apparatus comprising: detector circuitry2002,2102configured to detect a sequence of power values for respective windows of input samples of an input signal; generator circuitry2004,2104configured to generate a frequency domain representation of the sequence of power values; and output circuitry2060,2050configured to provide the frequency domain representation of the sequence of power values as a metadata signal to accompany the input signal.

As an alternative implementationFIG.21schematically illustrates a data processing apparatus suitable to carry out the methods discussed above under the control of suitable programming instructions, comprising a central processing unit or CPU2100, a random access memory (RAM)2110, a non-transitory machine-readable memory or medium (NTMRM)2120such as a flash memory, a hard disc drive or the like, a user interface such as a display, keyboard, mouse, or the like2130, and an input/output interface2140. These components are linked together by a bus structure2150. The CPU2100can perform any of the above methods under the control of program instructions stored in the RAM2110and/or the NTMRM2120. The NTMRM2120therefore provides an example of a non-transitory machine-readable medium which stores computer software by which the CPU2100perform the method or methods discussed above.

FIGS.22and23provide alternative and additional arrangements for use in the case of input signals containing significant periods of silence (in the example of audio signals) or low magnitude samples (in the case of other types of signals). InFIG.22, the source audio signal101is represented as having a relatively short portion2200of non-silence, with the remainder being silence2210. Here, “silence” is taken to be represented by a window (of the type used in the step203ofFIG.6) having an RMS power value below a threshold power value.

For the purposes of processing to be discussed below, the periods2210of silence can be removed (“stripped” away from) the source audio signal to leave a stripped signal2309. Similarly, with respect to the target audio signal123, a version2220can be envisaged in which periods2230of silence have been stripped out. In actual fact, in the processing to be discussed below, no specific use is made of the stripped target audio signal2220, but it is drawn here for the sake of the present description.

A set of power values2310takes the place of the set205and contains power values for the stripped source signal2309. Similarly, a set of power values2306takes the place of the set227and contains power values of the stripped target signal2220. Both of these signals can be generated as discussed below either by stripping the silence portions from the audio signals and then taking window power values or, more conveniently, starting with all of the windowed power values and stripping away the power values representing windows classified as silence.

In terms of the spectral profile of the source (221) and target (223) RMS power, depending on the length of the stripped signal2309,2220, or alternatively depending on the number of power values in the respective sets2310,2306, it may be that one or more spectral bands do not have validly associated data in the spectral profiles221,243(for example, because very low frequency components of the power values cannot be represented when the set of power values is not long enough). In the example ofFIG.22, a set of bands2240in the spectral profile2241do not contain valid frequency domain power data because the length of the stripped portion2309is so short as to not allow the derivation of very low frequency power components such as those represented by the components2240.

In such cases, when such invalid components occur in any of (one or both of) the spectral profiles, the processing ofFIG.6is amended so as to set the corresponding values in the difference247to a difference which is neutral (that is to say, 1 in a multiplicative system (linear domain) or 0 in an additive system (log domain).

Note that the same consideration may apply to high frequency components in the instance that the individual window length (used to generate the windowed power values) is too long.

This handling of invalid components provides an example of generating a frequency domain difference representation indicative of no difference in respect of any frequency bands for which a valid frequency domain representation of power values is unavailable for one or both of the source input samples and the target input samples, given the length of the windows of the source input samples, the length of the windows of the target input samples, the number of source input samples and the number of target input samples.

Referring toFIG.23, potential modifications to the process ofFIG.6is described in more detail, relating to the use of stripped signals so as to remove silence portions.

A silence threshold2301is defined, relating to the RMS power values. After the evaluation of the windowed RMS power at the step203and at the step225, the RMS power values are compared at steps2302and2304with the silence threshold2301. RMS power values2310,2306above the threshold are retained and the dynamic matching process from207onwards is performed at a step2311.

The skilled person will appreciate that although this test includes the actual threshold value2301in the “silence” category, the test could instead be expressed as silence being represented by power values less than a threshold, and non-silence as greater than or equal to the threshold.

It will also be appreciated that the term “silent” refers to an analytical test such as comparison of RMS power with a threshold value, rather than necessarily to a subjective interpretation. It is also noted that the respective thresholds used at the steps2302and2304may be the same or may be different.

On the source signal side, using the results of the comparison at the step2302, the windowed RMS power values are partitioned into (i) windows of the source signal2309with the RMS power value above the threshold which are passed to the step265inFIG.6and (ii) windows2305with the RMS power value below the threshold. The windows2305of the source signal are provided to a step2313at which the non-silent processed signal2320from the modified step265has the silent windows reinserted at their original positions in the source signal101, so as to generate an output or result signal2315which once again includes the silent portions at their original positions within the audio file.

In other words, the silent portions are removed for the purposes of the majority of the processing ofFIG.6and are then reinserted to recreate the output signal. In order to avoid the introduction of artefacts at the reinsertion step2313, a short duration (for example, 20% of the length of one window) cross-fade can be introduced whenever a reinserted silent window is adjacent to a process non-silent window.

It will also be appreciated (and this is particularly shown in the example ofFIG.22) that the portions of silence may differ between the source and target signals. This would lead to sets of power values2310,2306of different lengths, implying that the respective spectra209,231are of different numbers of bands or bins. However, the downsampling steps217,239can still be to the same number of bands or bins (for example, 20 for source and target) so that a valid comparison can still be made at the step245.

The silence processing discussed above provides an example of disregarding power values indicative of windows having less than a threshold power prior to the generating step. In some examples, as shown by the reinsertion step2313, windows of source input samples having less than a threshold power can be removed for the processing and then reinserted into the output signal.

FIG.24concerns a variation of the process ofFIG.6, which could apply with or without the variation discussed above with reference toFIGS.22and23, in which the process ofFIG.6is carried out with respect to so-called loudness weighted audio signals, for example with the loudness weighting applied before the generating step. Here, both the source audio signal101and the target audio signal123are processed to generate so-called loudness representations2405,2407by steps2401and2403respectively.

Loudness relates to a measure of the human perception of sound intensity. The perception varies with audio frequency, so a loudness weighting is a frequency-dependent adjustment to account for the different contributions of various frequencies to the human perception of volume. Some systems of loudness weighting, if applied at the steps2401,2403, would in fact just cancel out at the comparison step245. However, a family of so-called multi-band loudness weightings do not cancel out and are therefore useful in this context.

The process of steps203to225through to263ofFIG.6is carried out as a summary step2410with respect to the weighted signals245,267. However, the steps265is carried out with respect to the original non-weighted source audio samples101to generate the output signal267.

Finally,FIG.25concerns a further alternate or additional modification of the processFIG.6, in which, instead of using an entire file of samples of the source audio signal101and/or the target audio signal123, a windowed portion2500,2510is processed by the steps203/225. . .265.

Depending on the length of the windows2500,2510, there may be values2520,2530in the spectral profiles221,243, for which valid spectral profiles of the RMS power values cannot be obtained. Once again, in these instances, the different247is set to neutral at those spectral positions.

As discussed, in at least some embodiments the source input signal and the target input signal are sampled audio signals; and the windows are time windows. For example, the time windows of the source input samples may be equal in length to one another; and the time windows of the target input samples may be equal in length to one another. In some examples the time windows each have a length of at least 50 milliseconds, this value corresponding to the limit of normal human pitch perception at 20 kHz.

In so far as embodiments of the disclosure have been described as being implemented, at least in part, by software-controlled data processing apparatus, it will be appreciated that a non-transitory machine-readable medium carrying such software, such as an optical disk, a magnetic disk, semiconductor memory or the like, is also considered to represent an embodiment of the present disclosure. Similarly, a data signal comprising coded data generated according to the methods discussed above (whether or not embodied on a non-transitory machine-readable medium) is also considered to represent an embodiment of the present disclosure.

It will be apparent that numerous modifications and variations of the present disclosure are possible in light of the above teachings. It is therefore to be understood that within the scope of the appended clauses, the technology may be practised otherwise than as specifically described herein.

Various respective aspects and features will be defined by the following numbered clauses:1. A signal processing method comprising:comparing a first frequency domain representation of a sequence of power values for respective windows of source input samples of a source input signal with a second frequency domain representation of a sequence of power values for respective windows of target input samples of a target input signal so as to generate a frequency domain difference representation;inverse-frequency-transforming the frequency domain difference representation to generate a modification indication; andapplying the modification indication to the source input samples to generate respective output samples of an output signal.2. A method according to clause 1, comprising the steps of:detecting the sequence of power values for the respective windows of the target input samples; andgenerating the second frequency domain representation of the sequence of power values.3. A method according to clause 2, comprising the step of:disregarding power values indicative of windows having less than a threshold power prior to the generating step.4. A method according to clause 2 or clause 3, in which the target input samples represent an audio signal, the method comprising:prior to the generating step, applying a loudness weighting to the target input samples.5. A method according to any one of the preceding clauses, comprising the steps of:detecting the sequence of power values for the respective windows of source input samples of the source input signal; andgenerating the first frequency domain representation of the sequence of power values.6. A method according to clause 5, comprising the step of:disregarding power values indicative of windows having less than a threshold power prior to the generating step.7. A method according to clause 5 or clause 6, in which the source input samples represent an audio signal, the method comprising:prior to the generating step, applying a loudness weighting to the source input samples.8. A method according to any one of the preceding clauses, comprising:removing windows of source input samples having less than a threshold power; andreinserting the removed windows into the output signal.9. A method according to any one of clauses 5 to 8, in which the generating step comprises:frequency-transforming the sequence of power values for respective windows of source input samples to generate an intermediate frequency domain representation according to a first plurality of frequency bands; anddownsampling the intermediate frequency domain representation to generate the first frequency domain representation having a second plurality of frequency bands than the intermediate frequency domain representation, the second plurality being lower than the first plurality;and in which the second frequency domain representation has the second plurality of frequency bands.10. A method according to clause 9, in which the comparing step comprises:generating an intermediate difference representation in dependence upon a difference between the first frequency domain representation of the sequence of power values for respective windows of the source input samples and the second frequency domain representation of the sequence of power values for respective windows of the target input samples;upsampling the intermediate difference representation to the first plurality of frequency bands to generate an upsampled intermediate difference representation; andcombining the upsampled intermediate difference representation with the intermediate frequency domain representation to generate the frequency domain difference representation.11. A method according to any one of the preceding clauses, in which the applying step comprises:comparing the modification indication with the sequence of power values for respective windows of source input samples to generate a correction indication; andapplying the correction indication to the source input samples.12. A method according to clause 11, comprising:upsampling the correction indication to the same number of samples as the number of source input samples.13. A method according to any one of the preceding clauses, in which:the source input signal and the target input signal are sampled audio signals; andthe windows are time windows.14. A method according to clause 13, in which:the time windows of the source input samples are equal in length to one another; andthe time windows of the target input samples are equal in length to one another.15. A method according to clause 14, in which the time windows each have a length of at least 50 milliseconds.16. A method according to any one of the preceding clauses, in which the comparing step comprises generating a frequency domain difference representation indicative of no difference in respect of any frequency bands for which a valid frequency domain representation of power values is unavailable for one or both of the source input samples and the target input samples, given the length of the windows of the source input samples, the length of the windows of the target input samples, the number of source input samples and the number of target input samples.17. Computer software comprising program instructions which, when executed by a computer, cause the computer to perform the method of any one of the preceding clauses.18. A non-transitory, machine-readable medium which stores computer software according to clause 17.19. A signal processing method comprising:detecting a sequence of power values for respective windows of input samples of an input signal;generating a frequency domain representation of the sequence of power values; andproviding the frequency domain representation of the sequence of power values as a metadata signal to accompany the input signal.20. Computer software comprising program instructions which, when executed by a computer, cause the computer to perform the method of clause 19.21. A non-transitory, machine-readable medium which stores computer software according to clause 20.22. A non-transitory machine-readable medium which stores the target input signal and the metadata signal provided by the providing step of clause 19.23. Signal processing apparatus comprising:comparator circuitry configured to compare a first frequency domain representation of a sequence of power values for respective windows of source input samples of a source input signal with a second frequency domain representation of a sequence of power values for respective windows of target input samples of a target input signal so as to generate a frequency domain difference representation;transform circuitry configured to inverse-frequency-transform the frequency domain difference representation to generate a modification indication; andoutput circuitry to apply the modification indication to the source input samples to generate respective output samples of an output signal.24. Signal processing apparatus comprising:detector circuitry configured to detect a sequence of power values for respective windows of input samples of an input signal;generator circuitry configured to generate a frequency domain representation of the sequence of power values; andoutput circuitry configured to provide the frequency domain representation of the sequence of power values as a metadata signal to accompany the input signal.