Patent Description:
In the present disclosure and the appended claims:.

A state-of-the-art conversational codec can represent with a very good quality a clean speech signal with a bitrate of around <NUM> kbps and approach transparency at a bitrate of <NUM> kbps. However, at bitrates below <NUM> kbps, low processing delay conversational codecs, most often coding an input speech signal in time-domain, are not suitable for generic audio signals, like music and reverberant speech. To overcome this drawback, switched codecs have been introduced, basically using a time-domain approach for coding speech-dominated input sound signals and a frequency-domain approach for coding generic audio signals. However, such switched solutions typically require longer processing delay, needed both for speech-music classification and for calculating a transform to frequency-domain.

To overcome the above drawback related to longer processing delay, a more unified time-domain and frequency-domain coding model has been proposed in <CIT> (See Reference [<NUM>]). This unified time-domain and frequency-domain coding model is part of the EVS (Enhanced Voice Services) sound codec standardized by 3GPP (<NUM>rd Generation Partnership Project) as described in Reference [<NUM>]. In recent years, 3GPP started working on developing a 3D (Three-Dimensional) sound codec for immersive services called IVAS (Immersive Voice and Audio Services), based on the EVS codec (See reference [<NUM>]).

To make the coding model even more efficient for a specific kind of signal, a coding mode has been added to efficiently allocate the available bits between time-domain and frequency-domain and between low and high frequency. The additional coding mode is triggered by a new speech/music classifier of which the output allows for an unclear category for signals that cannot be clearly classified as music nor speech (See Reference [<NUM>]).

The present disclosure relates to a unified time-domain/frequency-domain coding method for coding an input sound signal. The method comprises: classifying the input sound signal into one of a plurality of sound signal categories, wherein the sound signal categories comprise an unclear signal type category showing that the nature of the input sound signal is unclear; selecting one of a plurality of coding sub-modes for coding the input sound signal if the input sound signal is classified in the unclear signal type category; and mixed time-domain/frequency-domain coding the input sound signal using the selected coding sub-mode.

According to the present disclosure, there is further provided a unified time-domain/frequency-domain coding device for coding an input sound signal, comprising: a classifier of the input sound signal into one of a plurality of sound signal categories, wherein the sound signal categories comprise an unclear signal type category showing that the nature of the input sound signal is unclear; a selector of one of a plurality of coding sub-modes for coding the input sound signal if the input sound signal is classified in the unclear signal type category; and a mixed time-domain/frequency-domain encoder for coding the input sound signal using the selected coding sub-mode.

The present disclosure provides a sound signal decoding method comprising: receiving a bitstream conveying information usable to reconstruct a mixed time-domain/frequency-domain excitation representative of a sound signal classified in an unclear signal type category showing that the nature of the sound signal is unclear, wherein the information includes one of a plurality of coding sub-modes used for coding the sound signal classified in the unclear signal type category; reconstructing the mixed time-domain/frequency-domain excitation in response to the information conveyed in the bitstream, including the coding sub-mode used for coding the input sound signal; converting the mixed time-domain/frequency-domain excitation to time-domain; and filtering the mixed time-domain/frequency-domain excitation converted to time-domain through a synthesis filter to produce a synthesized version of the sound signal.

In accordance with the present disclosure, there is provided a sound signal decoder comprising: a receiver of a bitstream conveying information usable to reconstruct a mixed time-domain/frequency-domain excitation representative of a sound signal classified in an unclear signal type category showing that the nature of the sound signal is unclear, wherein the information includes one of a plurality of coding sub-modes used for coding the sound signal classified in the unclear signal type category; a re-constructor of the mixed time-domain/frequency-domain excitation in response to the information conveyed in the bitstream, including the coding sub-mode used for coding the input sound signal; a converter of the mixed time-domain/frequency-domain excitation to time-domain; and a synthesis filter for filtering the mixed time-domain/frequency-domain excitation converted to time-domain to produce a synthesized version of the sound signal.

The foregoing and other features will become more apparent upon reading of the following non-restrictive description of illustrative embodiments of the unified time-domain/frequency-domain coding method, the unified time-domain/frequency-domain coding device, the decoding method and decoder device, given by way of example only with reference to the accompanying drawings.

The present disclosure proposes a unified time-domain and frequency-domain coding model which improves synthesis quality for generic audio signals such as, for example, music and/or reverberant speech, without increasing the processing delay and the bitrate. This unified time-domain and frequency-domain coding model comprises:.

depending upon the characteristics of the input sound signal.

To achieve a low processing delay and low bitrate conversational sound codec that improves the synthesis quality of generic audio signals such as, for example, music and/or reverberant speech, the frequency-domain coding mode is integrated as close as possible to a CELP (Code-Excited Linear Prediction) time-domain coding mode. For that purpose, the frequency-domain coding mode uses a frequency transform performed in the LP (Linear Prediction) residual domain. This allows switching nearly without artifact from one frame, for example a <NUM> frame, to another. As well known in the art of sound codecs, the input sound signal is sampled at a given sampling rate and processed by groups of these samples called "frames", usually divided into a number of "sub-frames". Here, the integration of the two (<NUM>) time-domain and frequency-domain coding modes is sufficiently close to allow dynamic reallocation of the bit budget to another coding mode if it is determined that the current coding mode is not sufficiently efficient.

One feature of the proposed unified time-domain and frequency-domain coding model is a variable time support of the time-domain component, which varies from a quarter frame (sub-frame) to a complete frame on a frame-by-frame basis. As a non-limitative illustrative example, a frame may represent <NUM> of input sound signal. Such a frame corresponds to <NUM> samples of the input sound signal if the inner sampling rate of the sound codec is <NUM> or to <NUM> samples per frame if the inner sampling rate of the codec is <NUM>. Then a sub-frame (quarter of a frame in the present example) represents <NUM> or <NUM> samples depending on the inner sampling rate of the sound codec. In the present non-restrictive illustrative embodiment, the inner sampling rate of the sound codec is <NUM> giving a frame length of <NUM> samples and a sub-frame length of <NUM> samples of the input sound signal.

The variable time support makes it possible to capture major temporal events with a minimum bitrate to create a basic time-domain excitation contribution. At very low bitrate, the time support is usually the entire frame. In that case, the time-domain contribution of the excitation is composed only of the adaptive codebook; corresponding adaptive-codebook (pitch) information and gain are then transmitted once per frame. When more bitrate is available, it is possible to capture more temporal events by shortening the time support and increasing the bitrate allocated to the time-domain coding mode. Eventually, when the time support is sufficiently short (shorter than a quarter of a frame (sub-frame)), and the available bitrate is sufficiently high, the time-domain contribution of the excitation may include, for each sub-frame, the adaptive-codebook contribution with the corresponding adaptive-codebook gain, a fixed-codebook contribution with a corresponding fixed-codebook gain, or both the adaptive-codebook and fixed-codebook contributions with the corresponding gains. Alternatively, it is also possible to transport, for each half of a frame (sub-frame), an adaptive-codebook contribution with the corresponding adaptive-codebook gain and a fixed-codebook contribution with the corresponding fixed-codebook gain; this has the advantage of not consuming too much bitrate while still being able to code temporal events. Parameters describing codebook indices and gains are then transmitted for each sub-frame.

At low bitrate, conversational sound codecs are incapable of coding properly higher frequencies. This causes an important degradation of the synthesis quality when the input sound signal includes music and/or reverberant speech. To solve this issue, a feature is added to compute the efficiency of the time-domain excitation contribution. In some cases, whatever the input bitrate and the time frame support are, the time-domain excitation contribution is not valuable. In those cases, all the bits are reallocated to the next step of frequency-domain coding. But most of the time, the time-domain excitation contribution is valuable up only to a certain frequency (herein after the "cut-off frequency"). In these cases, the time-domain excitation contribution is filtered out above the cut-off frequency. The filtering operation permits to keep valuable information coded with the time-domain excitation contribution and remove the non-valuable information above the cut-off frequency. In a non-restrictive illustrative embodiment, the filtering is performed in frequency-domain by setting the frequency bins above a certain frequency (cut-off frequency) to zero.

The variable time support in combination with the variable cut-off frequency makes the bit allocation inside the unified time-domain and frequency-domain coding model very dynamic. The bitrate after the quantization of the LP filter can be allocated entirely to the time domain or entirely to the frequency domain or somewhere in between. The bitrate allocation between the time and frequency domains is conducted as a function of the number of sub-frames used for the time-domain excitation contribution, of the available bit budget, and of the cut-off frequency computed. To make the unified time-domain and frequency-domain coding model even more efficient for a specific kind of input sound signal, specific coding sub-modes are added to efficiently allocate the available bits between the time domain, the frequency domain and between low and high frequencies. These added specific coding sub-modes are determined using a new speech/music audio classifier producing an output allowing for an unclear signal category (signals that cannot be clearly classified as music nor speech).

To create a total excitation which will match more efficiently the input LP residual, the frequency-domain coding mode is applied. A feature is that frequency-domain coding is performed on a vector which contains a difference between a frequency representation (frequency transform) of the input LP residual and a frequency representation (frequency transform) of the filtered time-domain excitation contribution up to the cut-off frequency, and which contains a frequency representation (frequency transform) of the input LP residual itself above that cut-off frequency. A smooth spectrum transition is inserted between both segments just above the cut-off frequency. In other words, the high-frequency part of the frequency representation of the time-domain excitation contribution is first zeroed out above the cut-off frequency. A transition region between the unchanged part of the spectrum and the zeroed part of the spectrum of the time-domain excitation contribution is inserted just above the cut-off frequency to ensure a smooth transition between both parts of the spectrum. This modified spectrum of the time-domain excitation contribution is then subtracted from the frequency representation of the input LP residual. The resulting spectrum thus corresponds to the difference of both spectra below the cut-off frequency, and to the frequency representation of the LP residual above it, with some transition region. The cut-off frequency, as mentioned hereinabove, can vary from one frame to another.

Whatever the frequency quantization method (frequency-domain coding mode) chosen, there is always a possibility of pre-echo especially with long windows. In the herein disclosed technique, the used windows are square windows, so that the extra window length compared to the coded input sound signal is zero (<NUM>), i.e. no overlap-add is used. While this corresponds to the best window to reduce any potential pre-echo, some pre-echo may still be audible on temporal attacks. Many techniques exist to solve such pre-echo problem but the present disclosure proposes a simple feature for cancelling this pre-echo problem. This feature is based on a memory-less time-domain coding mode which is derived from the "Transition Mode" of ITU-T Recommendation G. <NUM>; Reference [<NUM>], sections <NUM>. <NUM> and <NUM>. The idea behind this feature is to take advantage of the fact that the proposed unified time-domain and frequency-domain coding model is integrated to the LP residual domain, which allows for switching without artifact almost at any time. When an input sound signal is considered as generic audio (music and/or reverberant speech) and when a temporal attack is detected in a frame, then this frame only is encoded with the memory-less time-domain coding mode. This memory-less time-domain coding mode will take care of the temporal attack thus avoiding the pre-echo that could be introduced when using frequency-domain coding of that frame.

In the proposed unified time-domain and frequency-domain coding model, the above mentioned adaptive codebook, one or more fixed codebooks (for example an algebraic codebook, a Gaussian codebook, etc.), i.e. the so called time-domain codebooks, and the frequency-domain quantization (frequency-domain coding mode) can be seen as a codebook library, and the bits can be distributed among all the available codebooks, or a subset thereof. This means for example that if the input sound signal is a clean speech, all the bits will be allocated to the time-domain coding mode, basically reducing the coding to the legacy CELP scheme. On the other hand, for some music segments, all the bits allocated to encode the input LP residual are sometimes best spent in the frequency-domain, for example in transform-domain. Furthermore, specific cases can be added in which (a) the time-domain uses a larger part of the total available bitrate to code more time-domain events while still maintaining bits to code some of the frequency information or (b) low frequency content is prioritized over high frequency content and vice versa.

As indicated in the foregoing description, temporal support for the time-domain and frequency-domain coding modes does not need to be the same. While the bits spent on the different time-domain coding operations (adaptive and algebraic codebook searches) are usually distributed on a sub-frame basis (typically a quarter of a frame, or <NUM> of time support), the bits allocated to the frequency-domain coding mode are distributed on a frame basis (typically <NUM> of time support) to improve frequency resolution.

The bit budget allocated to the time-domain CELP coding mode can be also dynamically controlled depending on the input sound signal. In some cases, the bit budget allocated to the time-domain CELP coding mode can be zero, effectively meaning that the entire bit budget is attributed to the frequency-domain coding mode. The choice of working in the LP residual domain both for the time-domain and the frequency-domain coding modes has two (<NUM>) main benefits. First, this is compatible with the time-domain CELP coding mode, proved efficient in speech signals coding. Consequently, no artifact is introduced due to the switching between the two types of coding modes (time-domain and frequency-domain coding modes). Second, lower dynamics of the LP residual with respect to the original input sound signal, and its relative flatness, make easier the use of a square window for the frequency transforms thus permitting use of a non-overlapping window.

In a non limitative example where the inner sampling rate of the codec is <NUM> (meaning <NUM> samples per frame), similarly as in the ITU-T recommendation G. <NUM> (Reference [<NUM>]), the length of the sub-frames used in the time-domain CELP coding mode can vary from a typical ¼ of the frame length (<NUM>) to a half frame (<NUM>) or a complete frame length (<NUM>). The sub-frame length decision is based on the available bitrate and on an analysis of the input sound signal, particularly the spectral dynamics of this input sound signal. The sub-frame length decision can be performed in a closed loop manner. To save on complexity, it is also possible to base the sub-frame length decision in an open loop manner. The sub-frame length decision can be also controlled by the nature of the input sound signal as detected by a signal classifier, for example a speech/music classifier. The sub-frame length can be changed from frame to frame.

Once the length of the sub-frames is chosen in a current frame, a standard closed-loop pitch analysis is performed and the first contribution to the excitation signal is selected from the adaptive codebook. Then, depending on the available bit budget and the characteristics of the input sound signal (for example in the case of an input speech signal), a second contribution from one or several fixed codebooks can be added before conversion in the transform domain. The resulting excitation contribution is the time-domain excitation contribution. On the other hand, at very low bitrates and in the case of a generic audio signal, it is often better to skip the fixed codebook stage and use all the remaining bits for the transform-domain coding. The transform-domain coding can be for example a frequency-domain coding mode. As described above, the sub-frame length can be one fourth of the frame, one half of the frame, or one frame long. The fixed-codebook contribution is used only if the sub-frame length is equal to <NUM>/<NUM> of the frame length. In case the sub-frame length is decided to be half a frame or the entire frame long, then only the adaptive-codebook contribution is used to represent the time-domain excitation contribution, and all remaining bits are allocated to the frequency-domain coding mode. Alternatively, an additional coding mode will be described where the fixed codebook can be used when the sub-frame length is equal to half the frame length. This addition has been made to improve the quality of particular kinds of input sound signals containing a temporal event while keeping an acceptable bit budget to code the frequency-domain excitation contribution.

Once the computation of the time-domain excitation contribution is completed, its efficiency needs to be assessed and quantized. If the gain of the coding in time-domain is very low, it is more efficient to remove the time-domain excitation contribution altogether and to use all the bits for the frequency-domain coding mode. On the other hand, for example in the case of a clean input speech signal, the frequency-domain coding mode is not needed, and all the bits are allocated to the time-domain coding mode. But often the coding in time-domain is efficient only up to a certain frequency. This frequency corresponds to the above mentioned cut-off frequency of the time-domain excitation contribution. Determination of such cut-off frequency ensures that the entire time-domain coding is helping to get a better final synthesis rather than working against the frequency-domain coding.

The cut-off frequency can be estimated in the frequency domain. To compute the cut-off frequency, the spectrums of both the LP residual and the time-domain excitation contribution are first split into a predefined number of frequency bands in each of which a number of frequency bins are defined. The number of frequency bands and the number of frequency bins covered by each frequency band can vary from one implementation to another. For each of the frequency bands, a normalized correlation is computed between the frequency representation of the time-domain excitation contribution and the frequency representation of the LP residual, and the correlation is smoothed between adjacent frequency bands. As a non-limitative example, the per-band correlations are lower limited to <NUM> and normalized between <NUM> and <NUM>, and an average correlation is then computed as the average of the correlations for all the frequency bands. For the purpose of a first estimation of the cut-off frequency, the average correlation is then scaled between <NUM> and half the internal sampling rate (half the internal sampling rate corresponding to the normalized correlation value of <NUM>). At very low bitrate or for the additional coding sub-modes as described herein below, the average correlation is doubled before finding the cut-off frequency. This is done for cases where it is known that the time-domain excitation contribution would be needed even if the correlation is not very high because of the low bitrate being used, or because the type of input sound signal would not allow for a high correlation. The first estimation of the cut-off frequency is then found as the upper bound of the frequency band being closest to the value of the scaled average correlation. In an example of implementation, sixteen (<NUM>) frequency bands at a <NUM> internal sampling rate are defined for correlation computation.

Taking advantage of the psychoacoustic property of the human ear, the reliability of the estimation of the cut-off frequency may be improved by comparing the estimated position of the <NUM>th harmonic frequency of the pitch to the cut-off frequency estimated by the correlation computation. If this position is higher than the cut-off frequency estimated by the correlation computation, the cut-off frequency is modified to correspond to the position of the <NUM>th harmonic frequency of the pitch. If one of the additional coding sub-modes is used, the cut-off frequency has a minimum value above or equal to, for example, <NUM> (<NUM>th band). The final value of the cut-off frequency is then quantized and transmitted to a distant decoder. In an example of implementation, <NUM> or <NUM> bits are used for such quantization, giving <NUM> or <NUM> possible cut-off frequencies depending on the bitrate.

Once the cut-off frequency is known, frequency quantization of the frequency-domain excitation contribution is performed. First the difference between the frequency representation (frequency transform) of the input LP residual and the frequency representation (frequency transform) of the time-domain excitation contribution is determined. Then a new vector is created, consisting of this difference up to the cut-off frequency, and a smooth transition to the frequency representation of the input LP residual for the remaining spectrum. A frequency quantization is then applied to the whole new vector. In an example of implementation, the quantization consists of coding the sign and the position of dominant (most energetic) spectral pulses. The number of pulses to be quantized per frequency band is related to the bitrate available for the frequency-domain coding mode. If the available bits are insufficient to cover all the frequency bands, the remaining bands are filled with noise only.

Frequency quantization of a frequency band using the quantization method described in the previous paragraph does not guarantee that all frequency bins within this band are quantized. This is especially true at low bitrates where the number of spectral pulses quantized per frequency band is relatively low. To prevent the apparition of audible artifacts due to these non-quantized bins, some noise is added to fill these gaps. As at low bitrates the quantized spectral pulses should dominate the spectrum rather than the inserted noise, the noise spectrum amplitude corresponds only to a fraction of the amplitude of the pulses. The amplitude of the added noise in the spectrum is higher when the bit budget available is low (allowing more noise) and lower when the bit budget available is high.

In the frequency-domain coding mode, gains are computed for each frequency band to match the energy of the non-quantized signal to the quantized signal. The gains are vector quantized and applied per band to the quantized signal. When, for example, the unified time-domain and frequency-domain coding model changes the bit allocation from a time-domain only coding mode to a mixed time-domain/frequency-domain coding mode, the per band excitation spectrum energy of the time-domain only coding mode does not match the per band excitation spectrum energy of the mixed time-domain/frequency-domain coding mode. This energy mismatch can create some switching artifacts especially at low bitrate. To reduce any audible degradation created by this bit reallocation, a long-term gain can be computed for each band and can be applied to correct the energy of each frequency band for a few frames after the switching from the time-domain only coding mode to the mixed time-domain/frequency-domain coding mode.

After the completion of the frequency-domain coding mode, the total excitation is found by adding the frequency-domain excitation contribution to the frequency representation (frequency transform) of the time-domain excitation contribution and then the sum of these two (<NUM>) excitation contributions is transformed back to time-domain to form a total excitation. Finally, the synthesized signal is computed by filtering the total excitation through a LP synthesis filter.

In one embodiment, while the CELP coding memories are updated on a sub-frame basis using only the time-domain excitation contribution, the total excitation is used to update those memories at frame boundaries.

In another possible implementation, the CELP coding memories are updated on a sub-frame basis and also at the frame boundaries using only the time-domain excitation contribution. This results in an embedded structure where the frequency-domain coded signal constitutes an upper quantization layer independent from the core CELP layer. In this particular case, the fixed codebook is always used in order to update the adaptive codebook content. However, the frequency-domain coding mode can apply to the whole frame. This embedded approach works for bit rates around <NUM> kbps and higher.

<FIG> is a schematic block diagram illustrating concurrently an overview of a unified time-domain/frequency-domain CELP coding method <NUM> and a corresponding unified time-domain/frequency-domain CELP coding device <NUM>, for example ACELP method and device. Of course, other types of CELP coding method and device can be implemented using the same concept.

<FIG> is a schematic block diagram of a more detailed structure of the unified time-domain/frequency-domain CELP coding method <NUM> and device <NUM> of <FIG>.

The unified time-domain/frequency-domain CELP coding device <NUM> comprises a pre-processor <NUM> (<FIG>) for performing an operation <NUM> of analyzing parameters of the input sound signal <NUM> (<FIG> and <FIG>). Referring to <FIG>, the pre-processor <NUM> comprises an LP analyzer <NUM> for performing an operation <NUM> of LP analysis of the input sound signal <NUM>, a spectral analyzer <NUM> for performing an operation <NUM> of spectral analysis, an open loop pitch analyzer <NUM> for performing an operation <NUM> of open loop pitch analysis, and a signal classifier <NUM> for performing an operation <NUM> of classification of the input sound signal. The analyzers <NUM> and <NUM> and the associated operations <NUM> and <NUM> perform the LP and spectral analyses usually carried out in CELP coding, as described for example in ITU-T recommendation G. <NUM>, Reference [<NUM>], sections <NUM> and <NUM>. <NUM>, and, therefore, will not be further described in the present disclosure.

The pre-processor <NUM> conducts a first level of analysis to classify the input sound signal <NUM> between speech and non-speech (generic audio (music or reverberant speech)), for example in a manner similar to that described in Reference [<NUM>], or with any other reliable speech/non-speech discrimination methods.

After this first level of analysis, the pre-processor <NUM> performs a second level of analysis of input signal parameters to allow the use of time-domain CELP coding (no frequency-domain coding) on some sound signals with strong non-speech characteristics, but that are still better encoded with a time-domain approach. When an important variation of energy occurs, this second level of analysis allows the unified time-domain/frequency-domain CELP coding device <NUM> to switch into a memory-less time-domain coding mode, generally called Transition Mode in Reference [<NUM>].

During this second level of analysis, the signal classifier <NUM> calculates and uses a variation σC of a smoothed version Cst of an open-loop pitch correlation from the open-loop pitch analyzer <NUM>, a current total frame energy Etot (total energy of the input sound signal in the current frame) and a difference between the current total frame energy and the previous total frame energy Ediff. First, the signal classifier <NUM> computes the variation of the smoothed open loop pitch correlation using, for example, the following relation:
<MAT>
where:.

When, during the first level of analysis, the signal classifier <NUM> classifies a frame as non-speech, the following verifications are performed by the signal classifier <NUM> to determine, in the second level of analysis, if it is really safe to use a mixed time-domain/frequency-domain coding mode. Sometimes, it is however better to encode the current frame with the time-domain coding mode only, using one of the time-domain approaches estimated by the pre-processing function of the time-domain coding mode. In particular, it might be better to use the memory-less time-domain coding mode to reduce at a minimum any possible pre-echo that can be introduced with a mixed time-domain/frequency-domain coding mode.

As a non-limitative implementation of a first verification whether the mixed time-domain/frequency-domain coding mode should be used, the signal classifier <NUM> calculates a difference between the current total frame energy and the previous frame total energy. When the difference Ediff between the current total frame energy Etot and the previous frame total energy is higher than, for example, <NUM> dB, this corresponds to a so-called "temporal attack" in the input sound signal <NUM>. In such a situation, the speech/non-speech decision and the selected coding mode are overwritten and a memory-less time-domain coding mode is forced. More specifically, the unified time-domain/frequency-domain CELP coding device <NUM> comprises a time/time-frequency coding selector <NUM> (<FIG>) for performing an operation <NUM> of selection between time-domain only coding and mixed time-domain/frequency-domain coding. For that purpose, the time/time-frequency coding selector <NUM> comprises a speech/generic audio selector <NUM> (<FIG>) for performing an operation <NUM> of selecting between speech and generic audio for the classification of the input sound signal <NUM>, a temporal attack detector <NUM> (<FIG>) for performing an operation <NUM> of detecting a temporal attack in the input sound signal <NUM>, and a selector <NUM> (<FIG>) for performing an operation <NUM> of selecting the memory-less time-domain coding mode. In other words:.

The closed-loop CELP encoder <NUM> forms part of the time-domain-only encoder <NUM> of <FIG>. A closed-loop CELP encoder is well known to those of ordinary skill in the art and will not be further described in the present description.

As a non-limitative implementation of second verification whether the mixed time-domain/frequency-domain coding mode should be used, when the difference Ediff between the current total frame energy Etot and the previous frame total energy is below or equal to <NUM> dB, but:.

and this is at least the second consecutive frame (cnt ≥ <NUM>) where the decision of the first level of the analysis is changed, then the speech/generic audio selector <NUM> determines that the current frame will be coded using a time-domain only coding mode using the closed-loop CELP encoder <NUM> (<FIG>).

Otherwise, the time/time-frequency coding selector <NUM> selects the mixed time-domain/frequency-domain coding mode as disclosed in the following description.

The second verification can be summarized, for example when the non-speech input sound signal is music, using the following pseudo code:
<IMG>
where Etot is the current total frame energy expressed as: <MAT> where x(i) represents the samples of the input sound signal in the current frame, N is the number of samples of the input sound signal by frame, and Ediff is the difference between the current total frame energy Etot and the last previous frame total energy.

<FIG> is a schematic block diagram illustrating concurrently an alternative implementation of the unified time-domain/frequency-domain CELP coding method <NUM> and corresponding unified time-domain/frequency-domain CELP coding device <NUM>, in which the pre-processor <NUM> also performs a first level of analysis to classify the input sound signal <NUM>.

Specifically, the unified time-domain/frequency-domain CELP coding method <NUM> comprises an operation <NUM> of pre-processing the input sound signal <NUM> as described in Reference [<NUM>] to obtain the parameters required to classify this input sound signal. To perform operation <NUM>, the mixed time-domain/frequency-domain CELP coding device <NUM> comprises the pre-processor <NUM>.

The unified time-domain/frequency-domain CELP coding method <NUM> comprises an operation <NUM> of classifying the input sound signal <NUM> into speech, music and unclear signal type categories using the parameters from pre-processor <NUM> in a manner similar to that also described in Reference [<NUM>], or using any other reliable speech/music and unclear signal type discrimination methods. The unclear signal type category shows that the nature of the input sound signal <NUM> is unclear and, in particular, that the input sound signal <NUM> is not classified as speech nor music. To perform operation <NUM>, the unified time-domain/frequency-domain CELP coding device <NUM> comprises a sound signal classifier <NUM>.

If the sound signal classifier <NUM> classifies the input sound signal <NUM> into the music category, a frequency-domain encoder <NUM> performs an operation <NUM> of coding the input sound signal <NUM> using frequency-domain coding as described, for example, in Reference [<NUM>]. The frequency-domain encoded music signal can then be synthesized in a music synthesis operation <NUM> performed by a synthesizer <NUM> to recover the music signal.

In the same manner, if the sound signal classifier <NUM> classifies the input sound signal <NUM> into the speech category, a time-domain encoder <NUM> performs an operation <NUM> of coding the input sound signal <NUM> using time-domain coding as described, for example, in Reference [<NUM>]. The time-domain encoded speech signal can then be synthesized in a synthesis filtering operation <NUM> performed by a synthesizer <NUM> including a synthesis filter to recover the speech signal.

Accordingly, the unified time-domain/frequency-domain coding device <NUM> and method <NUM> maximise the performances of time-domain coding only and frequency-domain coding only by respectively limiting their usage to input sound signals having clear speech characteristics and input sound signals having clear music characteristics. This increases the overall quality of all types of input sound signals at low to medium bitrates.

Coding sub-modes have been designed as part of the unified time-domain and frequency-domain coding model to efficiently code input sound signals that are not classified as speech nor music (unclear signal type category). Two (<NUM>) bits are used to signal three (<NUM>) coding sub-modes identified by corresponding sub-mode flags. A fourth sub-mode allows for a backward interoperability to the legacy unified time-domain and frequency-domain coding model (EVS).

As illustrated in <FIG>, the operation <NUM> of classifying the input sound signal <NUM> comprises an operation <NUM> of selecting one of the coding sub-modes in response to the bitrate available for coding the input sound signal <NUM> and characteristics of this input sound signal classified in the unclear signal type category. To perform operation <NUM>, the sound signal classifier <NUM> incorporates a sub-mode selector <NUM>.

The coding sub-modes are identified by a sub-mode flag Ftfsm. In the non-limitative implementation of <FIG>, the sub-mode selector <NUM> selects the coding sub-modes as follows:.

The probability of the input sound signal <NUM> of being speech or music or in between is described in Reference [<NUM>]. When the decision of speech or music classification is unclear, if the probability wdlp(n) is greater than <NUM>, it is considered that the signal has some music characteristic. The table below shows the threshold where the probability would be high enough to be considered as music or speech.

The selected coding sub-mode, for example the sub-mode flag Ftfsm, is transmitted into the bitstream to a distant decoder. The path chosen inside the decoder depends of signaling bits included in the bitstream. Once the decoder detects the presence of a frame coded using mixed time-domain/frequency-domain coding, the sub-mode flag Ftfsm is decoded from the bitstream. If the detected sub-mode flag Ftfsm is "<NUM>", then the EVS backward interoperable legacy unified time-domain and frequency-domain coding model will be used to decode the remaining part of the bitstream. On the other hand, if the sub-mode flag Ftfsm is different from "<NUM>", sub-mode decoding is followed. The decoder will replicate the procedure followed by the encoder, in particular the bit distribution between time-domain and frequency-domain and the bit allocation in the different frequency bands as described later in section <NUM>.

In typical CELP, input sound signal samples are processed in frames of <NUM>-<NUM> and these frames are divided into sub-frames for adaptive codebook and fixed codebook analysis. For example, a frame of <NUM> (<NUM> samples when the internal sampling rate is <NUM>) can be used and divided into <NUM> sub-frames of <NUM>. A variable sub-frame length is a feature used to integrate time-domain and frequency-domain into one coding mode. The sub-frame length can vary from a typical ¼ of the frame length to half of the frame length or a complete frame length. Of course, the use of another number of sub-frames (sub-frame length) can possibly be implemented.

The parameter analysis operation <NUM> of the unified time-domain/frequency-domain CELP coding method <NUM> comprises, as illustrated in <FIG>, an operation <NUM> of determining a high spectral dynamic of the input sound signal <NUM>, and an operation <NUM> of calculating a number of sub-frames by frame. To perform operations <NUM> and <NUM>, the pre-processor <NUM> of the unified time-domain/frequency-domain CELP coding device <NUM> respectively comprises a high spectral dynamic analyzer <NUM> and a calculator <NUM> of the number of sub-frames.

The decision as to the length of the sub-frames (the number of sub-frames), or the time support, is determined by the calculator <NUM> based on the available bitrate and on the input sound signal analysis, in particular the high spectral dynamic of the input sound signal <NUM> from the analyzer <NUM> and the open-loop pitch analysis including the smoothed open loop pitch correlation Cst from analyzer <NUM>. The high spectral dynamic analyzer <NUM> is responsive to the information from the spectral analyzer <NUM> to determine high spectral dynamic of the input sound signal <NUM>. The high spectral dynamic is computed, for example as described in ITU-T recommendation G. <NUM>, Reference [<NUM>], section <NUM>. <NUM>, as an input spectrum without noise floor giving a representation of the input spectrum dynamic. When the average spectral dynamic of the input sound signal <NUM> in the frequency band between <NUM> and <NUM> as determined by the analyzer <NUM> is below, for example, <NUM> dB and the last frame was considered as having a high spectral dynamic, the input sound signal <NUM> is no longer considered as having high spectral dynamic. In that case, more bits can be allocated to the frequencies below, for example, <NUM>, by adding more sub-frames to the time-domain coding mode or by forcing more pulses in the lower frequency part of the frequency-domain coding mode.

On the other hand, if an increase of the average spectral dynamic of the input sound signal <NUM> against the average spectral dynamic of the last frame that was not considered as having a high spectral dynamic as determined by the analyser <NUM> is greater than, for example, <NUM> dB, the input sound signal <NUM> is considered as having high spectral dynamic content above, for example, <NUM>. In that case, depending on the available bitrate, some additional bits are used for coding the high frequencies of the input sound signal <NUM> to allow one or more frequency pulses coding.

The sub-frame length as determined by the calculator <NUM> (<FIG>) is also dependent on the bit budget available for coding the input sound signal <NUM>. At very low bitrate, e.g. bit rates below <NUM> kbps, only one sub-frame is available for time-domain coding otherwise the number of available bits will be insufficient for the frequency-domain coding. At medium bitrates, e.g. bit rates between <NUM> kbps and <NUM> kbps, one sub-frame is used for the case where the high frequencies contain high spectral dynamic content and two sub-frames if not. For medium-high bitrates, e.g. bit rates around <NUM> kbps and higher, the four (<NUM>) sub-frames case becomes also available if the above defined smoothed open loop pitch correlation Cst is higher than, for example, <NUM>.

While the case with one or two sub-frames limits the time-domain coding to an adaptive codebook contribution only (with coded pitch lag and pitch gain), i.e. no fixed codebook is used in that case, the case with four (<NUM>) sub-frames allow for adaptive and fixed codebook contributions if the available bit budget is sufficient. The four (<NUM>) sub-frame case is allowed at bitrates starting from around <NUM> kbps up. Because of bit budget limitations, the time-domain excitation contribution consists only of the adaptive codebook contribution at lower bitrates. A fixed-codebook contribution can be added at higher bit rates, for example starting at <NUM> kbps. For all cases the time-domain coding efficiency will be evaluated afterward to decide up to which frequency (the above mentioned cut-off frequency) such time-domain coding is valuable.

The alternative implementation of <FIG> and <FIG> uses the above defined first, second or third coding sub-modes when the input sound signal <NUM> is classified by the classifier <NUM> into the unclear signal type category and the sub-mode flag Ftfsm is greater than zero "<NUM>".

The sound signal classifier <NUM> determines that the number of sub-frames is four (<NUM>) unless the sub-mode flag Ftfsm is set to "<NUM>" or "<NUM>" (selection of the first or second coding sub-mode), meaning that the content of the input sound signal <NUM> is closer to speech ("speech" like characteristics or likelihood of a temporal attack is/are detected in the input sound signal <NUM>) and the available bitrate is below <NUM> kbps. Specifically:.

In the unified time-domain/frequency-domain CELP coding device <NUM> and method <NUM> (<FIG>), a mixed time-domain/frequency-domain coding method <NUM> and a corresponding mixed time-domain/frequency domain encoder <NUM> are used when generic audio is selected by selector <NUM> as the classification of the input sound signal <NUM> and no temporal attack is detected in detector <NUM>. Alternatively, in the unified time-domain/frequency-domain CELP coding device <NUM> and method <NUM> (<FIG>), a mixed time-domain/frequency-domain coding method <NUM> and a corresponding mixed time-domain/frequency domain encoder <NUM> are used when the sound signal classifier <NUM> classifies the input sound signal <NUM> in the "unclear signal type" category and one of the above defined first, second and third coding sub-modes is selected (sub-mode flag Ftfsm set to "<NUM>", "<NUM>" or "<NUM>").

When the mixed time-domain/frequency-domain coding mode is used, a closed-loop pitch analysis followed, if needed, by a fixed algebraic codebook search are performed. For that purpose, the mixed time-domain/frequency domain coding method <NUM>/<NUM> comprises an operation <NUM> of calculating the time-domain excitation contribution. To perform operation <NUM>, the mixed time-domain/frequency domain encoder <NUM>/<NUM> comprises a calculator of time-domain excitation contribution <NUM>. The calculator <NUM> itself comprises an analyzer <NUM> (<FIG>) responsive to the open-loop pitch analysis conducted in the open-loop pitch analyzer <NUM> (or pre-processor <NUM>) and the sub-frame length (or the number of sub-frames in a frame) determined in calculator <NUM> or sound signal classifier <NUM> to perform an operation <NUM> of closed-loop pitch analysis. The closed-loop pitch analysis is well known to those of ordinary skill in the art and an example of implementation is described for example in ITU-T G. <NUM> recommendation, Reference [<NUM>]; Section <NUM>. The closed-loop pitch analysis results in computing the pitch parameters, also known as adaptive-codebook parameters, which mainly consist of a pitch lag (adaptive-codebook index T) and pitch gain (adaptive-codebook gain b). The adaptive-codebook contribution is usually the past excitation at delay T or an interpolated version thereof. The adaptive-codebook index T is encoded and transmitted to a distant decoder. The pitch gain b is also quantized and transmitted to the distant decoder.

When the closed-loop pitch analysis has been completed in operation <NUM> and a fixed-codebook contribution is used, the calculator of time-domain excitation contribution <NUM> comprises a fixed algebraic codebook <NUM> searched during an operation <NUM> of fixed codebook search to find the best fixed-codebook parameters usually comprising a fixed-codebook index and a fixed-codebook gain. The fixed-codebook index and gain form the fixed-codebook contribution. The fixed-codebook index is encoded and transmitted to the distant decoder. The fixed-codebook gain is also quantized and transmitted to the distant decoder. The fixed-algebraic codebook and searching thereof are believed to be well known to those of ordinary skill in the art of CELP coding and, therefore, will not be further described in the present disclosure.

The adaptive-codebook index and gain and, if used, the fixed-codebook index and gain form the time-domain CELP excitation contribution.

During the frequency-domain coding of the mixed time-domain/frequency-domain coding mode, two signals are represented in transform-domain, for example in frequency-domain. In one embodiment, the time-to-frequency transform can be achieved using a <NUM> points type II (or type IV) DCT (Discrete Cosine Transform) giving a resolution of <NUM> with an inner sampling rate of <NUM> but any other suitable transform could be used. In the case another transform is used, the frequency resolution (defined above), the number of frequency bands and the number of frequency bins per band (defined further below) might need to be revised accordingly.

As indicated in the foregoing description, in the unified time-domain/frequency-domain CELP coding device <NUM> and method <NUM> (<FIG> and <FIG>), the mixed time-domain/frequency-domain coding mode is used when generic audio is selected by selector <NUM> as the classification of the input sound signal <NUM> and no temporal attack is detected in detector <NUM>. Alternatively, in the unified time-domain/frequency-domain CELP coding device <NUM> and method <NUM> (<FIG>), the mixed time-domain/frequency-domain coding mode is used when the sound signal classifier <NUM> classifies the input sound signal <NUM> in the "unclear signal type" category. The mixed time-domain/frequency domain encoder <NUM>/<NUM> comprises a calculator <NUM> (<FIG> and <FIG>) of frequency-domain excitation contribution performing an operation <NUM> of calculating the frequency-domain excitation contribution in response to the input LP residual res(n) (Reference [<NUM>]) resulting from the operation <NUM> of LP analysis of the input sound signal <NUM> performed by the analyzer <NUM> (and pre-processor <NUM>). As illustrated in <FIG>, the calculator <NUM> may calculate a DCT <NUM>, for example a type II DCT of the input LP residual res(n). The mixed time-domain/frequency domain encoder <NUM>/<NUM> also comprises a calculator <NUM> (<FIG> and <FIG>) for performing an operation <NUM> of calculating a frequency transform of the time-domain excitation contribution. As illustrated in <FIG>, the calculator <NUM> may calculate a DCT <NUM>, for example a type II DCT of the time-domain excitation contribution. The frequency transforms of the input LP residual fres and the time-domain CELP excitation contribution fexc can be calculated using, for example, the following expressions:
<MAT>
and:
<MAT>
where res(n) is the input LP residual, etd(n) is the time-domain excitation contribution, and N is the frame length. In a possible implementation, the frame length is <NUM> samples for a corresponding inner sampling rate of <NUM>. The time-domain excitation contribution is given by the following relation: <MAT>
where v(n) is the adaptive-codebook contribution, b is the adaptive-codebook gain, c(n) is the fixed-codebook contribution, and g is the fixed-codebook gain. It should be noted that the time-domain excitation contribution may consist only of the adaptive codebook contribution as described in the foregoing description.

With sound signal samples classified as generic audio (<FIG>) or sound signal samples classified in the "unclear signal type" category (<FIG>), the time-domain excitation contribution does not always contribute much to the coding improvement compared to the frequency-domain coding. Often, it does improve coding of the lower part of the spectrum while the coding improvement in the higher part of the spectrum is minimal. The mixed time-domain/frequency domain encoder <NUM>/<NUM> comprises a cut-off frequency finder and filter <NUM> (<FIG> and <FIG>) for performing an operation <NUM> of determining a cut-off frequency above which coding improvement afforded by the time-domain excitation contribution becomes too low to be valuable. The cut-off frequency finder and filter <NUM> comprises, as illustrated in <FIG>, a calculator of cut-off frequency <NUM> and a filter <NUM>.

An operation <NUM> of estimating the cut-off frequency of the time-domain excitation contribution is first completed by the calculator <NUM> (<FIG>) using a computer <NUM> (<FIG> and <FIG>) performing an operation <NUM> of normalized cross-correlation for each frequency band between the frequency transform of the input LP residual <NUM> from calculator <NUM> and the frequency transform of the time-domain excitation contribution <NUM> from calculator <NUM>, respectively designated fres and fexc which are defined in the foregoing Section <NUM>. The last frequency Lf included in each of, for example, the sixteen (<NUM>) frequency bands are defined in Hz as: <MAT>.

For this illustrative example, the number of frequency bins j per band Bb, the cumulative frequency bins per band CBb, and the normalized cross-correlation CC(i) per frequency band i are defined, for example, as follows, for a <NUM> frame at <NUM> internal sampling rate: <MAT> <MAT> <MAT> Where <MAT> and <MAT>
where Bb is the number of frequency bins j per band Bb, CBb is the cumulative frequency bins per band, CC(i) is the normalized cross-correlation per frequency band i, <MAT> is the excitation energy for a band and similarly <MAT> is the residual energy per band.

The calculator of cut-off frequency <NUM> comprises a smoother <NUM> (<FIG> and <FIG>) of cross-correlation through the frequency bands performing some operations <NUM> to smooth the cross-correlation vector between the different frequency bands. More specifically, the smoother <NUM> of cross-correlation through the frequency bands computes a new cross-correlation vector Cc<NUM> using, for example, the following relation: <MAT> where, in an illustrative embodiment, <MAT>.

The calculator of cut-off frequency <NUM> further comprises a calculator <NUM> (<FIG> and <FIG>) performing an operation <NUM> of calculating an average of the new cross-correlation vector Cc<NUM> over the first Nb bands (for example Nb =<NUM> representing <NUM>).

The calculator <NUM> of cut-off frequency also comprises a cut-off frequency module <NUM> (<FIG>) including, as illustrated in <FIG>, a limiter <NUM> of the cross-correlation, a normaliser <NUM> of the cross-correlation and a finder <NUM> of the frequency band where the cross-correlation is the lowest. More specifically, the limiter <NUM> performs an operation <NUM> of limiting the average of the cross-correlation vector Cc<NUM> to a minimum value of <NUM> and the normaliser <NUM> performs an operation <NUM> of normalising the limited average of the cross-correlation vector Cc<NUM> between <NUM> and <NUM>. The finder <NUM> performs an operation <NUM> of obtaining a first estimate of the cut-off frequency by finding the last frequency Lf of a frequency band i which minimizes the difference between the said last frequency Lf of a frequency band i and the normalized average Cc<NUM> of the cross-correlation vector Cc<NUM> multiplied by half the internal sampling rate (Fs/<NUM>) of the input sound signal <NUM>: <MAT> where <MAT>.

In the above relations, ftc<NUM> represents the first estimate of the cut-off frequency.

At low bitrate, where the normalized average Cc<NUM> is never really high (in the case of the unified time-domain/frequency-domain coding device <NUM> and method <NUM> of <FIG>), or when the sub-mode flag Ftfsm is greater than "<NUM>", meaning that the input sound signal is categorized as "unclear signal type" (in the case of the unified time-domain/frequency-domain coding device <NUM> and method <NUM> of <FIG>), or to artificially increase the value of ftc<NUM> to give more weight to the time-domain excitation contribution, it is possible to upscale, using the normaliser <NUM>, the value of the normalized average Cc<NUM> with a fixed scaling factor. As a non-limitative example, at bitrate below <NUM> kbps, the first estimate of the cut-off frequency ftc<NUM> is multiplied by <NUM>.

The precision of the cut-off frequency may be improved by adding the following component to the computation. For that purpose, the cut-off frequency module <NUM> comprises an extrapolator <NUM> (<FIG>) of the <NUM>th harmonic computed, in a corresponding operation <NUM>, from the minimum or lowest pitch lag value of the time-domain excitation contribution of the sub-frames of the frame, using, for example, the following relation: <MAT> where Fs = <NUM> Hz is the internal sampling rate or frequency, Nsub is the number of sub-frames in a frame, and T(i) is the adaptive-codebook index or pitch lag for sub-frame i.

The cut-off frequency module <NUM> comprises a finder <NUM> (<FIG>) of the frequency band in which the <NUM>th harmonic h<NUM>th is located. More specifically, for the sub-frames i<Nsub, the finder <NUM> performs an operation <NUM> of searching for the highest frequency band for which, for example, the following inequality is still verified: <MAT> The index of that band will be called i<NUM>th and it indicates the band where the <NUM>th harmonic is likely located.

The cut-off frequency module <NUM> finally comprises a selector <NUM> (<FIG>) of the final cut-off frequency ftc. More specifically, the selector <NUM> performs an operation <NUM> of retaining the higher frequency between the first estimate ftc1 of the cut-off frequency from finder <NUM> and the last frequency of the frequency band in which the <NUM>th harmonic is located (Lf(i<NUM>th)) from finder <NUM>, using the following relation:
<MAT>.

When coding sub-modes are used, in the case of the unified time-domain/frequency-domain coding device <NUM> and method <NUM> of <FIG>, the cut-off frequency ftc is further thresholded using, for example, the following relation: <MAT>.

As a non-limitative, illustrative example, when the cut-off frequency ftc from the selector <NUM> is below or equal to <NUM>, the analyzer <NUM> considers that the cost of the time-domain excitation contribution is too high. The selector <NUM> then selects all the frequency bins of the frequency representation of the time-domain excitation contribution to be zeroed and the zeroer <NUM> forces to zero all the frequency bins and also force the cut-off frequency ftc to zero. All bits allocated to the time-domain excitation contribution are then reallocated to the frequency-domain coding mode. Otherwise, the analyzer <NUM> forces the selector <NUM> to choose the high-frequency bins above the cut-off frequency ftc for being zeroed by the filter (zeroer) <NUM>.

Finally, the calculator <NUM> of cut-off frequency comprises a quantizer <NUM> (<FIG> and <FIG>) for performing an operation <NUM> of quantizing the cut-off frequency ftc into a quantized version ftcQ of this cut-off frequency for transmission to a distant decoder. If, for example, three (<NUM>) bits are associated to the cut-off frequency parameter, a possible set of output values can be defined (in Hz) as follows:
<MAT>.

Many mechanisms could be used by the selector <NUM> to stabilize the choice of the final cut-off frequency ftc to prevent the quantized version ftcQ to switch between <NUM> and <NUM> in inappropriate signal segment. To achieve this, as a non-restrictive example, the analyzer <NUM> is responsive to the long-term average pitch gain Glt <NUM> from the closed loop pitch analyzer <NUM> (<FIG>), the open-loop pitch correlation Col <NUM> from the open-loop pitch analyzer <NUM> and the smoothed open-loop pitch correlation Cst <NUM>. To prevent switching to frequency-domain coding only, the analyzer <NUM> does not allow such frequency-domain coding only when, for example, the following conditions are met, i.e. ftcQ cannot be set to <NUM>: <MAT> or <MAT> or <MAT> or <MAT>
where Col is the open-loop pitch correlation <NUM> and Cst corresponds to the smoothed version of the open-loop pitch correlation <NUM> defined as Cst = <NUM> · Col - <NUM> · Cst. Further, Glt (item <NUM> of <FIG>) corresponds to the long-term average of the pitch gain obtained by the closed loop-pitch analyzer <NUM> within the time-domain excitation contribution. The long-term average of the pitch gain <NUM> is defined as Glt = <NUM> · Gp - <NUM> · Glt, where Gp is the average pitch gain over the current frame. To further reduce the rate of switching between frequency-domain coding only and mixed time-domain/frequency-domain coding, a hangover can be added.

Once the cut-off frequency ftc of the time-domain excitation contribution is determined, frequency-domain coding is performed. To perform such frequency-domain coding, the mixed time-domain/frequency domain coding method <NUM>/<NUM> comprises a subtracting operation <NUM>, a frequency quantizing operation <NUM> and an adding operation <NUM>. The mixed time-domain/frequency domain encoder <NUM>/<NUM> comprises a subtractor or calculator <NUM>, a frequency quantizer <NUM> and an adder <NUM> to perform the operations <NUM>, <NUM> and <NUM>, respectively.

<FIG> is a schematic block diagram illustrating concurrently an overview of a frequency quantizer <NUM> and corresponding frequency quantizing operation <NUM>. Also, <FIG> is a schematic block diagram of a more detailed structure of the frequency quantizer <NUM> and corresponding frequency quantizing operation <NUM>.

The subtractor or calculator <NUM> (<FIG>, <FIG>, <FIG> and <FIG>) forms a first portion of a difference vector fd with the difference between the frequency transform fres <NUM> (<FIG> and <FIG>) (or other frequency representation) of the input LP residual from DCT <NUM> (<FIG>) and the frequency transform fexc <NUM> (<FIG> and <FIG>) (or other frequency representation) of the time-domain excitation contribution from DCT <NUM> (<FIG>) from zero up to the cut-off frequency ftc of the time-domain excitation contribution. A downscale factor <NUM> (<FIG>) may be applied (see multiplier <NUM> and corresponding multiplying operation <NUM>) to the frequency transform fexc <NUM> for the next transition region of ftrans=<NUM> (<NUM> frequency bins in this example of implementation) before the respective spectral portion of the frequency transform fres <NUM> is subtracted therefrom. The result of the subtraction constitutes a second portion of the difference vector fd representing a frequency range from the cut-off frequency ftc up to ftc+ftrans. The frequency transform fres <NUM> of the input LP residual is used for the remaining third portion of the difference vector fd.

The downscaled part of the difference vector fd resulting from application of the downscale factor <NUM> can be performed with any type of fade out function, it can be shortened to only a few frequency bins, but it could also be omitted when the available bit budget is judged sufficient to prevent energy oscillation artifacts when the cut-off frequency ftc is changing. For example, with a <NUM> resolution, corresponding to <NUM> frequency bin fbin = <NUM> in <NUM> points DCT at <NUM> internal sampling rate, the difference vector can be built as: <MAT> where <NUM> ≤ k ≤ ftc / fbin <MAT>.

where fres, fexc and ftc have been defined in the foregoing description.

In the unified time-domain/frequency-domain CELP coding method <NUM> as illustrated in <FIG>, the mixed time-domain/frequency domain encoder <NUM> comprises a band selector and bit allocator <NUM> and the mixed time-domain/frequency domain coding method <NUM> comprises a corresponding operation of band selection and bit allocation detection <NUM>.

<FIG> is a schematic block diagram illustrating concurrently the band selector and bit allocator <NUM> and the corresponding operation <NUM> of band selection and bit allocation of <FIG> for distributing the available bit budget to frequency quantization of the difference vector fd when the input sound signal <NUM> is not categorized as speech nor as music in the alternative implementation of unified time-domain/frequency-domain CELP coding method <NUM>/<NUM> of <FIG> and <FIG>.

Specifically, <FIG> shows an innovative way how the band selector and bit allocator <NUM> may distribute the available bits to the frequency quantization when the input sound signal <NUM> is not categorized as speech nor as music, but in the "unclear signal type", depending on the previously chosen coding sub-modes. In <FIG>, the frequency quantization is performed on a per band manner. For matter of simplicity, the frequency bands have the same number of frequency bins, which is sixteen (<NUM>) frequency bins, at a <NUM> internal sampling rate in the current illustrative example. Frequency band "<NUM>" represents the lower part of the spectrum while frequency band "<NUM>" represents the higher part of that spectrum.

To make the best possible use of the bits available for the frequency quantization, the band selection and bit allocation operation <NUM> comprises a first operation <NUM> of pre-fixing a fraction of the available bit budget (see <NUM>) for quantizing the lower frequencies of the difference vector fd as a function of the quantized cut-off frequency ftcQ from the cut-off frequency finder and filter <NUM>. To perform operation <NUM>, an estimator <NUM> uses, for example, the following relation: <MAT> <MAT> where PBlf is the fraction of the available bits allocated to frequency quantizing of the lower frequencies of the difference vector fd. In this example, the lower frequencies refer to the first five (<NUM>) frequency bands, or the first two (<NUM>) kHz. The term Lf(ftcQ) refers to the number of frequency bins up to the quantized cut-off frequency ftcQ.

Then, the estimator <NUM> adjusts the fraction of the available bits allocated to frequency quantizing of the lower frequencies PBlf based on the coding sub-mode flag Ftfsm. If the coding sub-mode flag Ftfsm is set to "<NUM>" (<FIG>), meaning that the likelihood of a temporal attack is detected in the current frame of the input sound signal <NUM>, then the fraction of bits allocated to frequency quantizing of the lower frequencies PBlf is increased by <NUM>% of the available bits. If "music" like characteristics are detected in the content of the current frame, indicated by a sub-mode coding flag Ftfsm being set to "<NUM>", the fraction of bits allocated to frequency quantizing of the lower frequencies PBlf is decreased by <NUM>% of the available bits.

Another parameter that affects the overall number of bits per frequency band available for frequency quantizing the difference vector fd, is an estimated maximum number NBmx of frequency bands of this difference vector fd to quantize. In the presently described illustrative example, at an internal sampling rate of <NUM>, the maximum total number Ntt of frequency bands is sixteen (<NUM>).

When the coding sub-modes are used, the band selection and bit allocation operation <NUM> comprises an operation <NUM> of estimating the maximum number NBmx of frequency bands of the difference vector fd to quantize. To perform operation <NUM>, an estimator <NUM> sets, if the coding sub-mode flag Ftfsm is set to "<NUM>" (first coding sub-mode being selected), the maximum number NBmx of frequency bands to "<NUM>". If the coding sub-mode flag Ftfsm is set to "<NUM>" (second coding sub-mode being selected), then the estimator <NUM> sets the maximum number NBmx of frequency bands to "<NUM>". If the coding sub-mode flag Ftfsm is set to "<NUM>" (third coding sub-mode being selected), then the estimator <NUM> sets the maximum number NBmx of frequency bands to "<NUM>". The estimator <NUM> then readjusts the maximum number NBmx of frequency bands to quantize as a function of the bit budget available for the frequency quantization of the difference vector fd using, for example, the following relations: <MAT> <MAT> where BF represents the number of bits available for frequency quantization of the difference vector fd (see <NUM>), BT is the total bitrate available to code the channel under processing (see <NUM>), Ftfsm is the sub-mode flag (see <NUM>), and Ntt is the maximum total number of frequency bands.

The estimator <NUM> can further reduce the maximum number of frequency bands of the difference vector fd to quantize in relation to the number of bits allocated to quantizing of middle and higher frequency bands of the difference vector fd. For the purpose of such limitation, the last lower frequency band and the first frequency band thereafter are assumed to have a similar number of bits mb or roughly <NUM>% of the bits PBlf allocated to frequency quantizing of the lower frequencies. For the last frequency band to be quantized, a minimum number of <NUM> bits mp is used to quantize at least one (<NUM>) frequency pulse. If the available bitrate BT is greater than or equal to <NUM> kbps, then the minimum number of bits mp will be nine (<NUM>) to allow for the quantizing of more pulses per frequency band. However, if the total available bitrate BT is below <NUM> kbps but the sub-mode flag Ftfsm is set to "<NUM>", meaning content having similarities to music, then the number of bits mp of the last frequency band to be frequency quantized will be <NUM> to allow for a more precise quantization. Then, the estimator <NUM> computes a corrected maximum number of frequency bands <MAT> using, for example, the following relation: <MAT> where <MAT> corresponds to the corrected maximum number of frequency bands to quantize, NBmx is the estimated maximum number of frequency bands, the number "<NUM>" represents the minimum number of frequency bands, BF represents the number of bits available for frequency quantization of the difference vector fd, PBlf is the fraction of bits allocated to quantizing of the five (<NUM>) lower frequency bands, mp is the minimum number of bits allocated to frequency quantize a frequency band, and mb the number of bits allocated to quantizing the first frequency band after the five (<NUM>) lower frequency bands.

After the computation of the maximum number of frequency bands, the estimator <NUM> may perform an additional verification such that mp remains lower or equal to mb. While this additional verification is an optional step, at low bitrate, it helps to allocate the bits more efficiently between the frequency bands of the difference vector fd.

The band selection and bit allocation operation <NUM> comprises an operation <NUM> of calculating low frequency bits. To perform operation <NUM>, a calculator <NUM> is provided. If the computation of the maximum number of frequency bands <MAT> leads to a smaller number of frequency bands to quantize, the calculator <NUM> re-allocates the portion of bits previously allocated to the higher frequency bands such that is no longer relevant to quantizing of the lower frequency bands using, for example, the following relation: <MAT> where BLF corresponds to the bits allocated to the five (<NUM>) lower frequency bands, BF corresponds to the number of bits available for frequency quantizing the lower frequencies of the difference vector fd, PBlf is the above mentioned fraction of bits from estimator <NUM> allocated, for example, to frequency quantizing of the five (<NUM>) lower frequency bands, mp is the minimum number of bits allocated to quantize a frequency band, and mb the number of bits allocated to quantizing the first frequency band after the five (<NUM>) lower frequency bands.

The band selection and bit allocation operation <NUM> comprises an operation <NUM> of frequency band characterization. To perform operation <NUM>, the band selector and bit allocator <NUM> comprises a frequency band characterizer <NUM> which, once the bitrate is distributed between the lower frequency bands and the rest of the frequency bands, performs a dual sorting of the frequency bands, to decide the importance of each band. The first sorting comprises finding whether one or more bands have a lower energy compared to their neighbor frequency bands. When it happens, the characterizer <NUM> marks these bands such that only the pre-determined minimum number of bits mp can be allocated to frequency quantizing these low energy frequency bands, even if the available bit budget is high. The second sorting comprises performing a position sorting of the middle and higher energy frequency bands, for example in decreasing energy order. These first and second sorting (dual sorting) are not performed for the lower frequency bands but are performed up to the maximum number of frequency bands <MAT>. The operation <NUM> of frequency band characterization can be summarized as follows: <MAT> <MAT> <MAT> where Ppb(i) is set to "<NUM>" for frequency bands where only the minimum number of bits mp will be used, EPmax(i) contains the position of the middle and higher energy frequency bands in decreasing energy order, and E(i) corresponds to the energy of each band. CBb and Bb are defined herein above in Section <NUM>. The difference vector fd has been defined in Section <NUM>.

The energy E(i) of each frequency band of the difference vector fd is computed in a calculator <NUM> and corresponding operation <NUM> of <FIG> and <FIG>. Calculator <NUM> and operation <NUM> also compute a gain per frequency band as described with reference to calculator <NUM> and operation <NUM> of <FIG>. The energy E(i) of each frequency band of the difference vector fd and the gain for each frequency band are quantized for example as described in relation to quantizer <NUM> and operation <NUM> of <FIG>, and both transmitted to a distant decoder. In the case of the implementation of <FIG> for the unified time-domain/frequency-domain coding device <NUM> and method <NUM>, calculator <NUM> and operation <NUM> replaces calculator <NUM> and operation <NUM> as well as quantizer <NUM> and operation <NUM>.

The band selection and bit allocation operation <NUM> comprises an operation <NUM> of final distribution of bits per frequency band. To perform operation <NUM>, the band selector and bit allocator <NUM> comprises a bits per frequency band final distributor <NUM>.

Once the frequency bands have been characterized, the distributor <NUM> allocates the bitrate or number of bits BF available to frequency quantize the difference vector fd among selected frequency bands.

In the non-limitative example, for the first five (<NUM>) lower frequency bands, the distributor <NUM> linearly distributes the bits BLF allocated to frequency quantize the lower frequencies, with the first lowest frequency band receiving <NUM>% of the bits BLF and the fifth (<NUM>th) lower frequency band receiving the last <NUM>% of the bits BLF. In this manner, the lower frequencies of the spectrum of the difference vector fd can be quantized with sufficient accuracy to recover a better quality synthesis of the input sound signal <NUM>.

The distributor <NUM> distributes the remaining bits BF allocated to frequency quantize the difference vector fd over the other, middle and higher frequency bands as a linear function but again taking into consideration the previous frequency band energy characterization (operation <NUM>) such that more bits can be allocated to higher energy frequency bands and less bits to the frequency bands having a lower energy compared to the energy of its neighbor frequency bands and, thereby, making a more relevant use of the available bits by quantizing with more precision more important portions of the spectrum of the difference vector fd. As a non-limitative example, the following relation illustrates how the bit distribution (operation <NUM>) can be performed: <MAT> <MAT> where Bp(i) represents the number of bits allocated per frequency band i, BF represents the number of bits available to frequency quantize the difference vector fd, BLF corresponds to the bitrate or bits allocated to the five (<NUM>) lower frequency bands, mp is the minimum number of bits to quantize a frequency pulse in a frequency band, Ppb(i) contains the position where the minimum number mp of bits will be used, and <MAT> is the maximal number of frequency bands to be quantized.

If, after operation <NUM>, there are some bits not allocated, the distributor <NUM> will allocate them to the lower frequency bands. As a non-limitative example, the distributor <NUM> will allocate one remaining bit per frequency band starting from the fifth (<NUM>th) band and going back to the first band and repeating this procedure if needed to allocate all the remaining bits.

Later, the distributor <NUM> may have to floor, truncate or round the number of bits per frequency band depending on the algorithm being used to perform the quantizing of the frequency pulses and potential fixed-point implementation.

The mixed time-domain/frequency-domain CELP coding method <NUM>/<NUM> comprises an operation of frequency quantizing <NUM> (<FIG>, <FIG> and <FIG>) the difference vector fd. To perform operation <NUM>, the mixed time-domain/frequency-domain CELP encoder <NUM>/<NUM> comprises a frequency quantizer <NUM> (<NUM> in <FIG>).

The difference vector fd can be quantized using several methods. In every case, frequency pulses have to be searched for and quantized. In one possible implementation, the frequency quantizer <NUM> searches for the most energetic pulses of the difference vector fd across the spectrum. The method to search the pulses can be as simple as splitting the spectrum into frequency bands and allowing a certain number of pulses per frequency band. The number of pulses per frequency bands depends on the bit budget available and on the position of the frequency band inside the spectrum. Typically, more pulses are allocated to the lower frequencies.

Depending on the bitrate available, the quantization of the frequency pulses can be performed by the frequency quantizer <NUM> using different techniques. In one embodiment, at bitrate below <NUM> kbps, a simple search and quantization scheme can be used to code the position and sign of the pulses. This scheme is described herein below as a non-limitative example.

For frequencies lower than <NUM>, the simple search and quantization scheme uses an approach based on factorial pulse coding (FPC) which is described in the literature, for example in Reference [<NUM>].

More specifically, referring to <FIG> and <FIG>, the frequency quantizer <NUM> comprises a selector <NUM> to perform an operation <NUM> of determining whether all the spectrum is quantized using FPC. As illustrated in <FIG>, if the selector <NUM> determines that all the spectrum is not quantized using FPC, an operation <NUM> of FPC coding and pulse position and sign coding is performed in a coder <NUM>.

As illustrated in <FIG>, the operation <NUM> of FPC coding and pulse position and sign coding comprises a frequency pulse searching operation <NUM>, a FPC coding operation <NUM>, an operation <NUM> of finding most energetic pulses, and an operation <NUM> of quantizing the position and sign of frequency pulses. To perform operations <NUM>-<NUM>, the coder <NUM> respectively comprises a searcher <NUM> of frequency pulses, a FPC coder <NUM>, a finder <NUM> of most energetic pulses and a quantizer <NUM> of the position and sign of frequency pulses.

The searcher <NUM> searches frequency pulses through all the frequency bands for the frequencies lower than <NUM>. The FPC coder <NUM> then processes the frequency pulses. The finder <NUM> determines the most energetic pulses for frequencies equal to and larger than <NUM>, and the quantizer <NUM> codes the position and sign of the found, most energetic pulses. If more than one (<NUM>) pulse is allowed within a frequency band then the amplitude of the pulse previously found is divided by <NUM> and the search is again conducted over the entire frequency band. Each time a pulse is found, its position and sign are stored for quantization and the bit packing stage. The following pseudo code illustrates, as a non-limitative example, this simple search and quantization scheme:
<IMG>
where NBD is the number of frequency bands (NBD = <NUM> in the illustrative example), Np is the number of pulses i to be coded in a frequency band k, Bb is the number of frequency bins per frequency band, CBb is the cumulative frequency bins per band as defined previously in Section <NUM>), pp represents the vector containing the pulse position found, ps represents the vector containing the sign of the pulse found and pmax represents the energy of the pulse found.

At bitrates above <NUM> kbps, the selector <NUM> determines that all the spectrum is to be quantized using FPC (<FIG> and <FIG>). As illustrated in <FIG>, an operation <NUM> of FPC coding is then performed in a FPC coder <NUM>. Referring to <FIG>, the coder <NUM> comprises a searcher <NUM> of frequency pulses and the operation <NUM> comprises a corresponding operation <NUM> of searching the frequency pulses. The search for frequency pulses is conducted through the entire frequency bands. The operation <NUM> comprises an operation <NUM> of coding the found frequency pulses and the coder <NUM> comprises, for performing operation <NUM>, a FPC processor <NUM>.

Then, the FPC processor <NUM> or the quantizer of position and sign of pulses <NUM> obtains the quantized difference vector fdQ by adding the number of pulses nb_pulses with the pulse sign ps to each of the position pp found. For each frequency band the quantized difference vector fdQ can be written using, for example, the following pseudo code:
<IMG>.

The frequency bands are quantized with more or less precision; the quantization method described in the previous section does not guarantee that all frequency bins within the frequency bands are quantized. This is especially the case at low bitrates where the number of pulses quantized per frequency band is relatively low. To prevent the apparition of audible artifacts due to these unquantized frequency bins, the frequency quantizer <NUM> comprises a noise filler <NUM> (<FIG>) to perform a corresponding operation <NUM> of adding some noise in the unquantized frequency bins in order to fill these gaps. This noise addition may be made over all the spectrum at bitrate below <NUM> kbps, for example, but can be applied only above the cut-off frequency ftc of the time-domain excitation contribution for higher bitrates. For simplicity, the noise intensity varies only with the bitrate available. At high bitrates the noise level is low but the noise level is higher at low bitrates.

The noise filler <NUM> comprises an adder <NUM> (<FIG>) which performs an operation <NUM> of adding noise to the quantized difference vector fdQ after the intensity or energy level of such added noise has been determined. For that purpose, the frequency quantizing operation <NUM> comprises an operation <NUM> of estimating the intensity or energy level of the added noise and the frequency quantizer <NUM> comprises, to perform operation <NUM>, a corresponding estimator <NUM> of noise energy level. The operation <NUM> of estimating the intensity or energy level of the added noise is made by the estimator <NUM> and prior to an operation <NUM> of determining a gain per frequency band in a per band gain calculator <NUM> of the frequency quantizer <NUM>.

In the illustrative embodiment, in the estimator <NUM>, the noise level is directly related to the coding bitrate. For example, at <NUM> kbps the estimator <NUM> sets the noise level <MAT> to <NUM> times the amplitude of the frequency pulses coded in a specific frequency band and progressively down to a value of <NUM> times the amplitude of the frequency pulses coded in a frequency band at <NUM> kbps. The adder <NUM> injects the noise only to section(s) of the spectrum where a certain number of consecutives frequency bins has a very low energy, for example when the cumulative bins energy of half of a frequency band is below <NUM>. For a specific frequency band i, the noise is injected for example as follows:
<IMG>
Where <MAT> where, for a band i, CBb is the cumulative number of frequency bins per frequency band, Bb is the number of frequency bins in a specific band i, <MAT> is the level of the added noise, and rand is a random number generator which is limited between -<NUM> to <NUM>.

Referring to <FIG> and <FIG>, the frequency quantizing operation <NUM> of the unified time-domain/frequency-domain coding device <NUM> and method <NUM> comprises the operation <NUM> of determining a gain per frequency band followed by an operation <NUM> of quantizing the per band gain. The frequency quantizer <NUM> comprises, to perform operation <NUM> and <NUM>, a per band gain calculator <NUM> and a per band gain quantizer <NUM>.

Once the quantized difference vector fdQ, including the noise fill if needed, is found, calculator <NUM> computes the gain per band for each frequency band. The per band gain for a specific band Gb(i) is defined as the ratio between the energy of the unquantized difference vector fd to the energy of the quantized difference vector fdQ in the log domain using, for example, the following relations:
<MAT>
<MAT>
where CBb and Bb are defined hereinabove in Section <NUM>).

The per band gain quantizer <NUM> vector quantizes the per band frequency gains. Prior to vector quantization, at low bitrate, the last gain (corresponding to the last frequency band) is quantized separately, and the remaining fifteen (<NUM>) per band gains (when, for example, a number <NUM> of frequency bands is used) are divided by the quantized last gain. Then, the normalized fifteen (<NUM>) remaining gains are vector quantized by the quantizer <NUM>. At higher bitrate, the mean of the per band gains is quantized first and then removed from all per band gains of the, for example, sixteen (<NUM>) frequency bands prior the vector quantization of those per band gains. The vector quantization being used can be a standard minimization in the log domain of the distance between the vector containing the per band gains and the entries of a specific codebook.

In the frequency-domain coding mode, gains are computed in the calculator <NUM> for each frequency band to match the energy of the unquantized vector fd to the quantized vector fdQ. The gains are vector quantized in quantizer <NUM> and applied per frequency band (operation <NUM>) to the quantized vector fdQ through a multiplier <NUM> (<FIG> and <FIG>).

Alternatively, it is also possible to use the FPC coding scheme at rate below <NUM> kbps for the whole spectrum by selecting only some of the frequency bands to be quantized. Before performing the selection of the frequency bands, the energy Ed of the frequency bands of the unquantized difference vector fd, are quantized using quantizer <NUM>. The energy is computed using, for example, the following relation: <MAT> where <MAT> where CBb and Bb are defined hereinabove in Section <NUM>).

To perform the quantization of the frequency band energy Ed', first the average energy over the first <NUM> frequency bands out of the sixteen bands being used is quantized and subtracted from all the sixteen (<NUM>) band energies. Then all the frequency bands are vectors quantized per group of <NUM> or <NUM> bands. The vector quantization being used can be a standard minimization in the log domain of the distance between the vector containing the gains per band and the entries of a specific codebook. If not enough bits are available, it is possible to only quantize the first <NUM> frequency bands and to extrapolate the last four (<NUM>) frequency bands using an average of the previous three (<NUM>) frequency bands or by any other methods.

Once the energy of frequency bands of the unquantized difference vector are quantized, it becomes possible to sort the energy in decreasing order in such a way that it would be replicable on the decoder side. During the sorting, all the energy bands below <NUM> are always kept and then only the most energetic bands will be passed to the FPC scheme for coding frequency pulse amplitudes and signs. With this approach the FPC scheme codes a smaller vector but covering a wider frequency range. In others words, it takes less bits to cover important energy events over the entire spectrum.

In the particular case of implementation of the unified time-domain/frequency-domain coding device <NUM> and method <NUM> of <FIG>, the frequency band selection and bit distribution is performed instead as determined by the energy per band and gain per band calculator <NUM> and calculating operation <NUM> and the band selector and bits allocator <NUM> and band selecting and bits allocating operation <NUM> of <FIG> and <FIG> as described herein above.

After the pulse quantization process, a noise fill similar to what has been described earlier is performed. Then, a gain adjustment factor Ga is computed per frequency band to match the energy EdQ of the quantized difference vector fdQ to the quantized energy Ed' of the unquantized difference vector fd. Then this per band gain adjustment factor is applied to the quantized difference vector fdQ. This can be expressed as follows: <MAT> where <MAT> and Ed' is the quantized energy per band of the unquantized difference vector fd as defined earlier.

After the completion of the frequency-domain coding stage, the total time-domain/frequency domain excitation is found. For that purpose, the mixed time-domain/frequency-domain CELP coding method <NUM>/<NUM> comprises an operation <NUM> of adding, using an adder <NUM> (<FIG>, <FIG>, <FIG> and <FIG>) of the mixed time-domain/frequency-domain CELP encoder <NUM>/<NUM>, the frequency quantized difference vector fdQ from the frequency quantizer <NUM> to the filtered frequency-transformed time-domain excitation contribution fexcF. When the unified time-domain/frequency domain coding device <NUM>/<NUM> changes its bit allocation from a time-domain only coding mode to a mixed time-domain/frequency-domain coding mode, the excitation spectrum energy per frequency band of the time-domain only coding mode does not match the excitation spectrum energy per frequency band of the mixed time-domain/frequency domain coding mode. This energy mismatch can create switching artifacts that are more audible at low bitrate. To reduce any audible degradation created by this bit reallocation, a long-term gain can be computed for each band and can be applied to the summed excitation to correct the energy of each frequency band for a few frames after the reallocation. Then, the mixed time-domain/frequency-domain CELP coding method <NUM>/<NUM> comprises an operation <NUM> (<FIG>, <FIG> and <FIG>) to transform the sum of the frequency quantized difference vector fdQ and the frequency-transformed and filtered time-domain excitation contribution fexcF to time-domain using, for example, an IDCT (Inverse DCT) <NUM> (<FIG>).

The unified time-domain/frequency domain coding method <NUM>/<NUM> comprises an operation <NUM>/<NUM> of producing a synthesized signal by filtering the total time-domain/frequency domain excitation from the IDCT <NUM> through a LP synthesis filter <NUM>/<NUM> (<FIG>, <FIG> and <FIG>) of the coding device <NUM>/<NUM>.

The quantized positions and signs of the frequency pulses forming the quantized difference vector fdQ are transmitted to the distant decoder (not shown).

In one non-limitative embodiment, while the CELP coding memories are updated on a sub-frame basis using only the time-domain excitation contribution, the total time-domain/frequency-domain excitation is used to update those memories at frame boundaries. In another possible implementation, the CELP coding memories are updated on a sub-frame basis and also at the frame boundaries using only the time-domain excitation contribution. This results in an embedded structure where the frequency-domain quantized signal constitutes an upper quantization layer independent of the core CELP layer. This presents advantages in certain applications. In this particular case, the fixed codebook is always used to maintain good perceptual quality, and the number of sub-frames is always four (<NUM>) for the same reason. However, the frequency-domain analysis can apply to the whole frame. This embedded approach works for bit rates around <NUM> kbps and higher.

<FIG> is a schematic block diagram illustrating concurrently a decoder device <NUM> and corresponding decoding method <NUM> for decoding a bitstream <NUM> from the above described unified time-domain/frequency-domain coding device <NUM> and corresponding unified time-domain/frequency-domain coding method <NUM>.

The decoder device <NUM> comprises a receiver (not shown) for receiving the bitstream <NUM> from the unified time-domain/frequency-domain coding device <NUM>.

If the sound signal coded by the unified time-domain/frequency-domain coding device <NUM> has been classified as "music", this is indicated in the bitstream <NUM> by corresponding signaling bits and detected by the decoder device <NUM> (see <NUM>). The received bitstream <NUM> is then decoded by a "music" decoder <NUM>, for example a frequency-domain decoder.

If the sound signal coded by the unified time-domain/frequency-domain coding device <NUM> has been classified as "speech", this is indicated in the bitstream <NUM> by corresponding signaling bits and detected by the decoder device <NUM> (see <NUM>). The received bitstream <NUM> is then decoded by a "speech" decoder <NUM>, for example a time-domain decoder using ACELP (Algebraic Code-Excited Linear Prediction) or more generally CELP (Code-Excited Linear Prediction).

If the sound signal coded by the unified time-domain/frequency-domain coding device <NUM> has not been classified either as "music" or "speech" (see <NUM> and <NUM>) and the bitrate available for coding the sound signal was equal to or lower than <NUM> kbps (see <NUM>), this is indicated in the bitstream by the sub-mode flag Ftfsm set to "<NUM>". The received bitstream <NUM> is then decoded using the backward coding mode, i.e. the legacy unified time-domain and frequency-domain coding model of <FIG> and <FIG> (EVS) as shown at <NUM>.

Finally, if the sound signal coded by the unified time-domain/frequency-domain coding device <NUM> has not been classified either as "music" or "speech" (see <NUM> and <NUM>) and the bitrate available for coding the sound signal was higher than <NUM> kbps (see <NUM>), this is indicated in the bitstream <NUM> by a sub-mode flag Ftfsm set to "<NUM>", "<NUM>" or "<NUM>". The received bitstream <NUM> is then decoded using the sound signal decoder <NUM> and corresponding sound signal decoding method <NUM> of <FIG>.

<FIG> is a schematic block diagram illustrating concurrently a sound signal decoder <NUM> and corresponding sound signal decoding method <NUM> for decoding a bitstream from the above described unified time-domain/frequency-domain coding device <NUM> and corresponding unified time-domain/frequency-domain coding method <NUM> in the case of a sound signal classified in the unclear signal type category.

As mentioned in the foregoing description, the adaptive-codebook index T and the adaptive-codebook gain b are quantized and transmitted, and therefore received in the bitstream by the receiver (not shown). In the same manner, when used, the fixed-codebook index and the fixed-codebook gain are also quantized and transmitted to the decoder, and therefore received in the bitstream <NUM> by the receiver (not shown). The sound signal decoding method <NUM> comprises an operation <NUM> of calculating a decoded time-domain excitation contribution using the adaptive-codebook index and gain and, if used, the fixed-codebook index and gain as commonly made in the art of CELP coding. To perform operation <NUM>, the sound signal decoder <NUM> comprises a calculator <NUM> of the decoded time-domain excitation contribution.

The sound signal decoding method <NUM> also comprises an operation <NUM> of calculating a frequency transform of the decoded time-domain excitation contribution using the same procedure as in operation <NUM> using a DCT transform. To perform operation <NUM>, the sound signal decoder <NUM> comprises a calculator <NUM> of the frequency transform of the decoded time-domain excitation contribution.

As mentioned in the foregoing description, a quantized version ftcQ of the cut-off frequency is transmitted to the decoder, and therefore received in the bitstream <NUM> by the receiver (not shown). The sound signal decoding method <NUM> comprises an operation <NUM> of filtering the frequency transform of the time-domain excitation contribution from the calculator <NUM> using the decoded cut-off frequency ftcQ recovered from the bitstream <NUM> and a procedure which is the same or similar to previously described filtering operation <NUM>. For completing operation <NUM>, the sound signal decoder <NUM> comprises a filter <NUM> of the frequency transform of the time-domain excitation contribution using the recovered cut-off frequency ftcQ. Filter <NUM> has the same, or to the least a similar structure as filter <NUM> of <FIG>.

The filtered frequency transform of the time-domain excitation contribution from filter <NUM> is supplied to a positive input of an adder <NUM> performing a corresponding adding operation <NUM>.

The sound signal decoding method <NUM> comprises an operation <NUM> of calculating the decoded energy and gain per frequency band of the difference vector fd. To perform operation <NUM>, the sound signal decoder <NUM> comprises a calculator <NUM>. Specifically, the calculator <NUM> de-quantizes, using procedures inverse to those as described in the present disclosure for the quantization, the quantized energy per frequency band and quantized gain per frequency band received in the bitstream <NUM> by the receiver (not shown) from the unified time-domain/frequency-domain coding device <NUM>.

The sound signal decoding method <NUM> comprises an operation <NUM> of recovering the frequency quantized difference vector fdQ. To perform operation <NUM>, the sound signal decoder <NUM> comprises a calculator <NUM>. The calculator <NUM> extracts from the bitstream <NUM> the quantized positions and signs of the frequency pulses and replicates the selection of the frequency bands to be used for quantization and the bit allocation in the different frequency bands as determined by the operation <NUM> and allocator <NUM> and employed by the unified time-domain/frequency-domain coding device <NUM> for coding the input sound signal. The calculator <NUM> uses this replicated information to recover the frequency quantized difference vector fdQ from the extracted frequency pulse quantized positions and signs. Specifically, for that purpose, the sound signal decoder <NUM> replicates the procedure used in the unified time-domain/frequency-domain coding device <NUM> as illustrated in <FIG> in response to the number of bits (bitrate) available in the decoder <NUM> for the frequency quantized difference vector fdQ (see <NUM>), the total bitrate available to the channel under processing (see <NUM>), and the sub-mode flag (see <NUM>).

The sound signal decoding method <NUM> comprises an operation <NUM> of adding the recovered frequency quantized difference vector fdQ from calculator <NUM> and the frequency-transformed and filtered time-domain excitation contribution fexcF from the filter <NUM> to form the mixed time-domain/frequency-domain excitation.

As can be appreciated, the estimators <NUM> and <NUM>, calculator <NUM>, characterizer <NUM>, distributor <NUM>, calculators <NUM> and <NUM>, filter <NUM>, calculators <NUM> and <NUM>, and adder <NUM> form a re-constructor of the mixed time-domain/frequency-domain excitation using information conveyed in the bitstream <NUM>, including the sub-mode flag identifying of one of the coding sub-modes selected and used for coding the sound signal classified in the unclear signal type category.

In the same manner, the operations <NUM>-<NUM> form a method of reconstructing the mixed time-domain/frequency-domain excitation using the information conveyed in the bitstream <NUM>.

The sound signal decoder <NUM> comprises a converter <NUM> to perform an operation <NUM> of transforming the mixed time-domain/frequency-domain excitation back to time-domain using for example the IDCT (Inverse DCT) <NUM>.

Finally, the synthesized sound signal is computed in the decoder <NUM> by an operation <NUM> of filtering through a LP (Linear Prediction) synthesis filter <NUM> the total excitation from the converter <NUM>. Of course, LP parameters required by the decoder <NUM> to reconstruct the synthesis filter <NUM> are transmitted from the unified time-domain/frequency-domain coding device <NUM> and extracted from the bitstream <NUM> as well known in the art of CELP coding.

<FIG> is a simplified block diagram of an example configuration of hardware components forming the above described unified time-domain/frequency-domain coding device <NUM>/<NUM> and method <NUM>/<NUM>, decoder device <NUM> and decoding method <NUM>.

The unified time-domain/frequency-domain coding device <NUM>/<NUM> and the decoder device <NUM> may be implemented as a part of a mobile terminal, as a part of a portable media player, or in any similar device. The device <NUM>/<NUM> and decoder device <NUM> (identified as <NUM> in <FIG>) comprises an input <NUM>, an output <NUM>, a processor <NUM> and a memory <NUM>.

The input <NUM> is configured to receive the input sound signal <NUM>/bitstream <NUM> of <FIG> and <FIG>, in digital or analog form. The output <NUM> is configured to supply the output signal. The input <NUM> and the output <NUM> may be implemented in a common module, for example a serial input/output device.

The processor <NUM> is operatively connected to the input <NUM>, to the output <NUM>, and to the memory <NUM>. The processor <NUM> is realized as one or more processors for executing code instructions in support of the functions of the various components of the unified time-domain/frequency-domain coding device <NUM>/<NUM> for coding an input sound signal as illustrated in <FIG>, or of the decoder device <NUM> of <FIG>.

The memory <NUM> may comprise a non-transient memory for storing code instructions executable by the processor(s) <NUM>, specifically, a processor-readable memory comprising/storing non-transitory instructions that, when executed, cause a processor(s) to implement the operations and components of the unified time-domain/frequency-domain coding device <NUM>/<NUM> and method <NUM>/<NUM> and the decoder device <NUM> and decoding method <NUM> described in the present disclosure. The memory <NUM> may also comprise a random access memory or buffer(s) to store intermediate processing data from the various functions performed by the processor(s) <NUM>.

Those of ordinary skill in the art will realize that the description of the unified time-domain/frequency-domain coding device <NUM>/<NUM> and method <NUM>/<NUM> and the decoder device <NUM> and decoding method <NUM> is illustrative only and is not intended to be in any way limiting. Other embodiments will readily suggest themselves to such persons with ordinary skill in the art having the benefit of the present disclosure. Furthermore, the disclosed unified time-domain/frequency-domain coding device <NUM>/<NUM> and method <NUM>/<NUM>, decoder device <NUM> and decoding method <NUM> may be customized to offer valuable solutions to existing needs and problems of encoding and decoding sound.

In the interest of clarity, not all of the routine features of the implementations of the unified time-domain/frequency-domain coding device <NUM>/<NUM> and method <NUM>/<NUM> and the decoder device <NUM> and decoding method <NUM> are shown and described. It will, of course, be appreciated that in the development of any such actual implementation of the unified time-domain/frequency-domain coding device <NUM>/<NUM> and method <NUM>/<NUM> and the decoder device <NUM> and decoding method <NUM>, numerous implementation-specific decisions may need to be made in order to achieve the developer's specific goals, such as compliance with application-, system-, network- and business-related constraints, and that these specific goals will vary from one implementation to another and from one developer to another. Moreover, it will be appreciated that a development effort might be complex and time-consuming, but would nevertheless be a routine undertaking of engineering for those of ordinary skill in the field of sound processing having the benefit of the present disclosure.

In accordance with the present disclosure, the components/processors/modules, processing operations, and/or data structures described herein may be implemented using various types of operating systems, computing platforms, network devices, computer programs, and/or general purpose machines. In addition, those of ordinary skill in the art will recognize that devices of a less general purpose nature, such as hardwired devices, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), or the like, may also be used. Where a method comprising a series of operations and sub-operations is implemented by a processor, computer or a machine and those operations and sub-operations may be stored as a series of non-transitory code instructions readable by the processor, computer or machine, they may be stored on a tangible and/or non-transient medium.

The unified time-domain/frequency-domain coding device <NUM>/<NUM> and method <NUM>/<NUM> and the decoder device <NUM> and decoding method <NUM> as described herein may use software, firmware, hardware, or any combination(s) of software, firmware, or hardware suitable for the purposes described herein.

In the unified time-domain/frequency-domain coding device <NUM>/<NUM> and method <NUM>/<NUM> and the decoder device <NUM> and decoding method <NUM> as described herein, the various operations and sub-operations may be performed in various orders and some of the operations and sub-operations may be optional.

Although the present disclosure has been described hereinabove by way of non-restrictive, illustrative embodiments thereof, these embodiments may be modified at will within the scope of the appended claims.

Claim 1:
A unified time-domain/frequency-domain coding device for coding an input sound signal, characterised by comprising:
a classifier of the input sound signal into one of a plurality of sound signal categories, wherein the sound signal categories comprise an unclear signal type category showing that the nature of the input sound signal is unclear;
a selector of one of a plurality of coding sub-modes for coding the input sound signal if the input sound signal is classified in the unclear signal type category; and
a mixed time-domain/frequency-domain encoder for coding the input sound signal using the selected coding sub-mode.