Source: https://patents.justia.com/patent/10115402
Timestamp: 2020-08-09 06:12:55
Document Index: 794064036

Matched Legal Cases: ['§ 119', 'art1', 'art1', 'Application No. 11842953', 'Application No. 15184203', 'Application No. 15', 'Application No. 15', 'Application No. 201510317625']

US Patent for Audio encoding device, method and program, and audio decoding device, method and program Patent (Patent # 10,115,402 issued October 30, 2018) - Justia Patents Search
Justia Patents TimeUS Patent for Audio encoding device, method and program, and audio decoding device, method and program Patent (Patent # 10,115,402)
Oct 20, 2016 - NTT DOCOMO, INC.
This application is a continuation of U.S. patent application Ser. No. 13/899,233, filed May 21, 2013, which is a continuation of PCT/JP2011/075489, filed Nov. 4, 2011, and which claims the benefit of the filing date pursuant to 35 U.S.C. § 119(e) of JP2010-260447, filed Nov. 22, 2010, and JP2011-033915, filed Feb. 18, 2011, all of which are incorporated herein by reference.
In transmitting an audio or acoustic signal (which will be generally referred to as an “audio signal”) via an IP network or mobile communication, the audio signal is encoded to be expressed by a small bit count, the encoded data is divided into audio packets, and the audio packets are transmitted via the communication network. The audio packets received through the communication network are decoded by a receiver-side server, MCU, or terminal to obtain a decoded audio signal.
“Concealment technologies on the receiver side” and “concealment technologies on the transmitter side,” may be described as packet loss concealment technologies to interpolate the audio or acoustic signal in the lost portions due to the packet losses.
The “concealment technologies on the receiver side” can duplicate a decoded audio signal included in a packet normally received in the past, in pitch units, and multiply the duplication by a predetermined attenuation coefficient to generate an audio signal corresponding to a packet loss part. “Concealment technology on the receiver side” can be, for example, similar to the technology described in ITU-T G.711 Appendix I. However, the “concealment technologies on the receiver side” are based on the premise that the property of audio of the packet loss part resembles that of audio immediately before the packet loss, and therefore cannot demonstrate a sufficient concealment effect if the packet loss part has a property different from that of the audio immediately before the loss, or if the power, or the energy of the audio, changes suddenly.
Furthermore, the “concealment technologies on the receiver side” may also include a more advanced technology such as, for example, similar to that of PCT publication WO2007/000988. More advanced technology, such as that of PCT publication WO2007/000988, can be different from the aforementioned technology of ITU-T G.711. For example, while the concealment signal may be generated by duplicating the decoded audio contained in the packet normally received in the past, the duplication may be multiplied by an attenuation coefficient that varies depending upon the property of the duplication source audio (shape of a power spectrum thereof), so as to implement high-quality shaping of the concealment signal with little abnormal sound.
On the other hand, the “concealment technologies on the transmitter side” can, for example, include the technology of Japanese Patent Application Laid-open No. 2003-316670 and the technology of Japanese Patent Application Laid-open No. 2008-111991.
When a concealment signal is generated by prediction from a decoded signal normally received in the past, such as similar to Japanese Patent Application Laid-open No. 2003-316670, it is difficult to highly accurately generate the concealment signal with a power change of the audio signal that is significantly different than the prediction result, such as, like generation of “clacks” of castanets as the concealment signal, from a past audio signal that does not include such “clacks.”
If the amplitude information about the silent interval on the transmitter side is generated so as to prevent the concealment signal from being generated in the case of the packet loss part being the silent interval, such as similar to Japanese Patent Application Laid-open No. 2003-316670, but fails to demonstrate a satisfactory concealment effect on sound with a sudden power change like the “clacks” of castanets as discussed above.
As described by the above examples, a satisfactory error concealment effect is difficult to achieve on a signal with a temporally quick power change (which will be referred to hereinafter as “transient signal”) like hand claps and “clacks” of castanets. Namely, it is extremely difficult for the receiver side to accurately estimate at what timing the transient signal appears in the audio signal, based on the decoded signal obtained by decoding the audio packets normally received immediately before.
A packet separation unit 3 separates the audio packet received through the network, into the packet header information and the other part (the audio code and auxiliary information code, which will be referred to hereinafter as “bitstream”) and outputs the bitstream to a decoding unit 4.
The audio encoding unit 11 saves audio signal for a predetermined period of time and encodes a signal of an encoding target out of the saved audio signal (step S1101 in FIG. 3). The encoding may be performed, for example, using the audio encoding such as 3GPP enhanced aacPlus defined in Literature “3GPP TS26.401 ‘Enhanced aacPlus general audio codec General description’” and G.718 defined in Literature “Recommendation ITU-T G.718 ‘Frame error robust narrow-band and wideband embedded variable bit-rate coding of speech and audio from 8-32 kbit/s’”, or using any other encoding method.
The subframe power calculation unit 121 in the auxiliary information encoding unit 12 saves the audio signal for a predetermined period of time and later calculates a subframe power sequence for audio signals s(dT), s(1+dT), . . . , s((d+1)T−1) out of the saved audio signal. The calculation may occur later than encoding of target signals s(0), s(1), . . . , s(T−1) by a predetermined number of frames (d frames in the present embodiment) (step S1211 in FIG. 3). The number of samples contained in one frame is defined as T herein. When a prediction target signal is defined by the following formula:
v(K·l+k)+s(K·l+k+dT),
a power P(l) of a subframe l (0≤l≤L−1) is obtained by the formula below. The letter k represents an index of a sample in each subframe (0≤k≤K−1). It is assumed herein that the number of samples in a digital signal in each subframe is K.
P ⁡ ( l ) = 10 ⁢ log 10 ⁡ ( 1 K ⁢ ∑ k = 0 K - 1 ⁢ v 2 ⁡ ( K · l + k ) )
Although it is assumed in this first embodiment that the length of subframes is K, it is also possible to use different lengths determined in advance for the respective subframes. The subframe power sequence may be calculated according to the following formula, where kstart1 represents an index of a start of the lth subframe and kend1 represents an index of an end thereof.
P ⁡ ( l ) = 10 ⁢ log 10 ⁡ ( 1 k end l - k start l ⁢ ∑ k = k start l k end l ⁢ v 2 ⁡ ( k end l - 1 + k ) )
The attenuation coefficient estimation unit 122 acquires from the subframe power sequence a slope γopt of a straight line representing a temporal change of power for example, by the least square method or the like (step S1221 in FIG. 3). More simply, the slope may be calculated from P(0) and P(L−1). In this example, the letter L represents the number of subframes contained in one frame. In other examples, the letter L may represent the number of subframes in a part of a frame, such as two subframes in half of a frame. In addition to the slope γopt of the straight line, an intercept Popt may be calculated by a straight-line approximation of the subframe power sequence P(l).
The power of subframe m is expressed herein by the following formula.
{circumflex over (P)}(m)=γopt·m+Popt
At this time, the slope γopt and intercept Popt of the straight line are acquired in accordance with the following formulas (the least square method).
γ opt = L ⁢ ∑ m = 0 L - 1 ⁢ m · P ⁡ ( m ) - ∑ m = 0 L - 1 ⁢ m ⁢ ∑ m = 0 L - 1 ⁢ P ⁡ ( m ) L ⁢ ∑ m = 0 L - 1 ⁢ m 2 - ( ∑ m = 0 L - 1 ⁢ m ) 2 P opt = ∑ m = 0 L - 1 ⁢ m 2 ⁢ ∑ m = 0 L - 1 ⁢ P ⁡ ( m ) - ∑ m = 0 L - 1 ⁢ m · P ⁡ ( m ) ⁢ ∑ m = 0 L - 1 ⁢ P ⁡ ( m ) L ⁢ ∑ m = 0 L - 1 ⁢ m 2 - ( ∑ m = 0 L - 1 ⁢ m ) 2
The attenuation coefficient quantization unit 123 performs scalar quantization of the slope γopt of the straight line, then encodes the quantized data, and outputs the auxiliary information code (step S1231 in FIG. 3). It may use a scalar quantization codebook prepared in advance. In the case of the straight-line approximation of subframe powers P(l), the intercept Popt may also be encoded in addition to the slope γopt of the straight line.
The audio decoding unit 42 decodes the audio code to generate a decoded signal and outputs it as decoded audio. The decoding of audio code is performed using a decoding method corresponding to the aforementioned audio encoding unit 11. At this time, the audio decoding unit 42 also sends the decoded signal to the first concealment signal generation unit 43 (step S4311 in FIG. 7). At this time, the first concealment signal generation unit 43 stores the sent decoded signal into the decoding coefficient storage unit 431 shown in FIG. 11. The stored decoded signal in storage therein is denoted by b(k, l). The stored signal may be at least d or more past frames. The letter k herein represents an index of a sample in a subframe (provided that 0≤k≤K−1) and the letter l an index of a subframe stored in the decoding coefficient storage unit 431 (provided that 0≤l≤dL−1).
In above step S4202 the auxiliary information decoding unit 45 decodes the auxiliary information code output from the code separation unit 40, to generate an index, and obtains a slope γJ of a straight line corresponding to the index from a codebook. Here, P(−1) represents a power of the last subframe in a signal received normally immediately before a frame loss.
{circumflex over (P)}(m)=γJ·m+P(−1)
In the case where an intercept of the straight line is simultaneously encoded by a straight-line approximation of powers of subframes, the subframe power is obtained by the following formula using the intercept PJ.
{circumflex over (P)}(m)=γJ·m+PJ
The stored decoding coefficient repetition unit 432 in the first concealment signal generation unit 43 obtains a first concealment signal z(k) using a stored decoding signal stored in the decoding coefficient storage unit 431 (step S4321 in FIG. 7). Specifically, it calculates the first concealment signal by repetition of the last subframe, for example, as expressed by the following formula.
z(K·l+k)=b(k,dL−1)
(provided that 0≤l≤dL−1 and 0≤k≤K−1)
It should be noted herein that the unit of repetition does not have to be limited to the last subframe but instead any part of b(k, l) may be extracted and repeated. Generation of the first concealment signal is not limited to the repetition as described above, and instead the first concealment signal may be calculated by extracting and repeating a waveform in a pitch unit from the decoding coefficient storage unit 431 or the first concealment signal may be generated by a prediction, for example, using the linear prediction. Alternatively, the first concealment signal may be generated in accordance with a model determined in advance, for example, as shown below.
[z(K·(L−1)), . . . ,z(K·L−1)]=f(b(0,0),b(1,0) . . . ,b(K−1,dL−1))
The subframe power correction unit 442 corrects the first concealment signal for a value of power of the first concealment signal in each of the subframes in accordance with the formula below to acquire a concealment signal y(K·l+k). Specifically, it performs the correction according to the below formula (provided that 0≤l≤L−1 and 0≤k≤K−1). In the formula, P−d(m) represents a power about a subframe contained in the auxiliary information code transmitted in the d-th packet before the packet (packet as a first concealment signal generation target) (step S4421 in FIG. 7).
P ^ ⁡ ( m ) = P - d ⁡ ( m ) z ′ ⁡ ( K · l + k ) = z ⁡ ( K · l + k ) 1 K ⁢ ∑ k = 0 K - 1 ⁢ z 2 ⁡ ( K · l + k ) y ⁡ ( K · l + k ) = 10 P ^ ⁡ ( m ) / 20 · z ′ ⁡ ( K · l + k )
For example, the subframe power correction unit 442, as shown in FIG. 8, extracts the auxiliary information previously transmitted in the d-th packet, from the auxiliary information storage unit 441 (step S60 in FIG. 8), calculates a mean square amplitude value for each subframe as to the first concealment signal, and divides a value contained in each subframe, by the mean square amplitude value (step S61 in FIG. 8). This operation results in obtaining z′(K·l+k). Then it calculates a power of each subframe from the auxiliary information and multiplies the foregoing value of the subframe by a mean amplitude value obtained from the power (step S62 in FIG. 8). This multiplication results in obtaining the concealment signal y(K·l+k).
The auxiliary information may be auxiliary information obtained by encoding a subframe power sequence by vector quantization using preliminarily-learned or empirically-determined vectors ci(l). The second embodiment will describe an example of encoding or decoding, using as the auxiliary information, information about a vector obtained by vector quantization of powers of subframes, in the auxiliary information encoding unit 12 or in the auxiliary information decoding unit 45 in the first embodiment.
The subframe power vector quantization unit 124 performs vector quantization of powers P(l) of subframes 1 (provided that 0≤l≤L−1), encodes the result, and outputs the auxiliary information code. The letter I represents the number of entries of straight lines or vectors in a codebook and the letter J represents an index of a straight line or a vector selected. ci(l) represents the lth element of the ith code vector in the codebook.
J = argmin i = 0 , … ⁢ , I - 1 ⁢ ∑ l = 0 L - 1 ⁢ ( c i ⁡ ( l ) - P ⁡ ( l ) ) 2
Selected J is encoded by binary encoding to obtain the auxiliary information code.
On the other hand, the auxiliary information decoding unit 45 decodes the auxiliary information code output from the code separation unit 40, to generate the index J, obtains a vector cJ(l) corresponding to the index J from the codebook, and outputs it.
{circumflex over (P)}(m)=cJ(l)
The subframe power calculation unit 121 saves audio signal for a predetermined period of time and the subframe power sequence for audio signals s(−dT), s(1−dT), . . . , s(−1) is calculated earlier by a predetermined number of frames (d frames in the present embodiment) than the encoding of target signals s(0), s(1), . . . , s(T−1) out of the saved audio signal. It is assumed herein that the number of samples contained in one frame is T. When a prediction target signal is expressed by the following formula:
v(K·l+k)=s(K·l+k+dT),
the power P(l) of subframe l (0≤l≤L−1) is obtained by the formula below. The letter k represents an index of a sample in a subframe (0≤k≤K−1). It is assumed herein that the number of samples of digital signals contained in each subframe is K.
P ⁡ ( l ) = 10 ⁢ ⁢ log 10 ⁡ ( 1 K ⁢ ∑ k = 0 K - 1 ⁢ ⁢ v 2 ⁡ ( K · l + k ) )
On the other hand, the subframe power correction unit 442 corrects the first concealment signal for a value of power of the first concealment signal in each subframe in accordance with the formula below to obtain the concealment signal y(K·l+k). Specifically, it performs the correction in accordance with the below formula (provided that 0≤l≤L−1 and 0≤k≤K−1). Pd(m) represents the power about the subframe contained in the auxiliary information code transmitted in the d-th packet after the pertinent packet (packet of a first concealment signal generation target).
P ^ ⁡ ( m ) = P d ⁡ ( m ) z ′ ⁡ ( K · l + k ) = z ⁡ ( K · l + k ) 1 K ⁢ ∑ k = 0 K - 1 ⁢ ⁢ z 2 ⁡ ( K · l + k ) y ⁡ ( K · l + k ) = 10 P ^ ⁡ ( m ) / 20 · z ′ ⁡ ( K · l + k )
V ⁡ ( k , l ) = ∑ n = - E · K 2 ⁢ ⁢ K - 1 ⁢ ⁢ p A ⁡ ( n ) · x ⁡ ( n ) ⁢ ⁢ cos ⁡ [ π K ⁢ ( n + 1 2 - K 2 ) ⁢ ( k + 1 2 ) ]
In this formula, the letter E represents the number of subframes in the time direction and the letter K represents the number of frequency bins. The letter k represents an index of a frequency bin (provided that 0≤k≤K−1) and the letter l represents an index of a subframe (provided that 0≤l≤L−1). As an alternative to the analysis QMF, the time-frequency transform can also be executed by MDCT (Modified Discrete Cosine Transform) or the like.
P ⁡ ( l + d ) = 10 ⁢ ⁢ log 10 ⁡ ( 1 K ⁢ ∑ k = 0 K - 1 ⁢ ⁢ V 2 ⁡ ( k , l + d ) )
The code multiplexing unit 13 writes the audio code and the auxiliary information code in a predetermined order, in the same manner as in the first and second embodiments, and outputs the resulting bitstream.
As shown in FIG. 11, the first concealment signal generation unit 43 is provided with the decoding coefficient storage unit 431 and the stored decoding coefficient repetition unit 432. The decoding coefficient storage unit 431 stores the decoded signal fed from the audio decoding unit 42. The stored decoded signal in storage is denoted by B(k, l). The letter k herein represents an index of a sample in a subframe (provided that 0≤k≤K−1) and 1 represents an index of a subframe stored in the decoding coefficient storage unit 431 (provided that 0≤l≤L−1).
When the error flag is on (to indicate a packet abnormality), the stored decoding coefficient repetition unit 432 obtains the first concealment signal z(k, l) using the stored decoded signal stored in the decoding coefficient storage unit 431. Specifically, it calculates the first concealment signal, for example, by repetition of the last subframe in accordance with the following formula.
z(k,l)=B(k,L−1)
(provided that 0≤l≤L−1 and 0≤k≤K−1)
The unit of repetition does not have to be limited to the last subframe, and any part of B(k, l) may be extracted and repeated, or the first concealment signal may be generated, for example, by prediction using the linear prediction. Alternatively, the first concealment signal may be generated, for example, in accordance with a model determined in advance as described below.
[z(k,0) . . . ,z(k,L−1)]=f(B(0,0),B(1,0) . . . ,B(K−1,L−1))
The auxiliary information decoding unit 45 decodes the auxiliary information code output by the code separation unit 40 to generate an index, obtains a slope γJ of a straight line corresponding to the index from the codebook, and outputs it. Here, P(−1) represents the power of the last subframe in the signal received normally immediately before the frame loss.
In the case where the intercept of the straight line is simultaneously encoded based on the straight-line approximation of powers of subframes, the subframe powers are obtained by the following formula using the intercept PJ.
As shown in FIG. 12, the concealment signal correction unit 44 is provided with the auxiliary information storage unit 441 and the subframe power correction unit 442. The auxiliary information storage unit 441 stores the auxiliary information fed from the auxiliary information decoding unit 45 when the error flag is off (to indicate packet normality). The auxiliary information to be stored is preferably that of several past frames. The subframe power correction unit 442 corrects the first concealment signal for a value of power of the first concealment signal in each subframe in accordance with the formula below to obtain the concealment signal Y(k, l). Specifically, it performs the correction in accordance with the below formula (provided that 0≤l≤L−1 and 0≤k≤K−1). P−d(m) represents the power about the subframe contained in the auxiliary information code transmitted in the d-th packet before the pertinent packet (packet of a first concealment signal generation target).
P ^ ⁡ ( m ) = P - d ⁡ ( m ) z ′ ⁡ ( k , l ) = z ⁡ ( k , l ) 1 K ⁢ ∑ k = 0 K - 1 ⁢ ⁢ z 2 ⁡ ( k , l ) Y ⁡ ( k , l ) = 10 P ^ ⁡ ( m ) / 20 · z ′ ⁡ ( k , l )
y ⁡ ( k , l ) = 1 K ⁢ ∑ k = 0 K - 1 ⁢ ⁢ p s ⁡ ( n ) · Y ⁡ ( k , l ) ⁢ cos ⁡ [ π K ⁢ ( n + 1 2 - K 2 ) ⁢ ( k + 1 2 ) ]
In this formula, the letter l represents an index of a signal in the time domain, provided that 0≤l≤K(2+L).
The subframe power calculation unit 121 saves the audio signal for the predetermined period of time, and calculates the subframe power sequence for the audio signal v(k, l+d) that is later by the predetermined number of frames (d frames in the present embodiment) than the encoding of the target signal v(k, l) out of the saved audio signal. It is assumed herein that the number of samples contained in one frame is T. Supposing a prediction target signal is defined as v(k, l+d)=s(k, l+d), the power Pi(l) of the ith subband in the subframe l (0≤l≤L−1) is obtained by the following formula. The letter k represents an index of a sample in a subframe (provided that 0≤k≤K−1).
P i ⁡ ( l + d ) = 10 ⁢ ⁢ log 10 ⁡ ( 1 K max i - K min i ⁢ ∑ k = K min i K max i ⁢ ⁢ v 2 ⁡ ( k , l + d ) )
The subbands may be determined so that the widths of the subbands are unequal intervals, or they may be set to the width of the critical band, or the subband widths may be set to 1.
The attenuation coefficient estimation unit 122 obtains a slope γopti of a straight line indicative of a temporal change of power for each subframe from the subframe power sequence, for example, by the least square method or the like. More simply, the slope may be determined from Pi(0) and Pi(L−1). In addition to the slope γopti of the straight line, an intercept Popti obtained by a straight-line approximation of the subframe power sequence Pi(l) may be obtained. The power of subframe m is represented herein by the following formula.
{circumflex over (P)}i(m)=γopti·m+Popti
In this case, a slope γopt and an intercept PJ of a straight line are determined according to the following formulas (the least square method).
γ opt = L ⁢ ∑ m = 0 L - 1 ⁢ ⁢ m · P ⁡ ( m ) - ∑ m = 0 L - 1 ⁢ ⁢ m ⁢ ⁢ ∑ m = 0 L - 1 ⁢ ⁢ P ⁡ ( m ) L ⁢ ∑ m = 0 L - 1 ⁢ ⁢ m 2 - ( ∑ m = 0 L - 1 ⁢ ⁢ m ) 2 P opt = ∑ m = 0 L - 1 ⁢ ⁢ m 2 ⁢ ∑ m = 0 L - 1 ⁢ ⁢ P ⁡ ( m ) - ∑ m = 0 L - 1 ⁢ ⁢ m · P ⁡ ( m ) ⁢ ∑ m = 0 L - 1 ⁢ ⁢ P ⁡ ( m ) L ⁢ ∑ m = 0 L - 1 ⁢ ⁢ m 2 - ( ∑ m = 0 L - 1 ⁢ ⁢ m ) 2
The attenuation coefficient quantization unit 123 performs scalar quantization of slopes γopti of straight lines, encodes the result, and outputs the auxiliary information code. The scalar quantization may be performed using a scalar quantization codebook prepared in advance. In the case of the straight-line approximation of the subframe powers Pi(l), the intercept Popti may be encoded in addition to the slope γopti of the straight line. The vector quantization and subsequent encoding may be applied to a vector obtained by arranging γopti of all the subbands, or the vector quantization and subsequent encoding may be applied to a vector obtained by arranging γopti and Popti.
When the error flag is on (to indicate a packet abnormality), the stored decoding coefficient repetition unit 432 obtains the first concealment signal Z(k, l), using the stored decoded signal stored in the decoding coefficient storage unit 431. The stored decoded signal stored in the decoding coefficient storage unit 431 is denoted by B(k, l). The letter k herein represents an index of a sample in a subframe (0≤k≤K−1) and the letter l represents an index of a subframe stored in the decoding coefficient storage unit 431 (0≤l≤L−1).
Specifically, the stored decoding coefficient repetition unit 432 calculates the first concealment signal by repetition of the last subframe, as represented by the following formula.
Z(k,l)=B(k,dL−1)
The unit of repetition does not have to be limited to the last subframe, and any part of B(k, l) may be extracted and repeated. Without being limited to the generation of the first concealment signal by the repetition as described above, the first concealment signal may be generated, for example, by a prediction using the linear prediction. Alternatively, the first concealment signal may be generated, for example, in accordance with a model determined in advance as described below.
[Z(0,0), . . . ,Z(K−1,L−1)]=f(b(0,0),b(1,0) . . . ,b(K−1,dL−1))
The auxiliary information decoding unit 45 decodes the auxiliary information code output from the code separation unit 40, to generate indexes, and obtains a slope γJi of a straight line corresponding to each of the indexes from the codebook. Here, Pi(−1) represents the power of the last subframe in the signal received normally immediately before the packet loss.
{circumflex over (P)}i(m)=γJi·m+Pi(−1)
In the case where the intercepts of the straight lines are simultaneously encoded based on the straight-line approximation of subframe powers, the subframe powers are obtained by the following formula using the intercepts PJi.
{circumflex over (P)}i(m)=γJi·m+PJi
In the concealment signal correction unit 44 as described above, the subframe power correction unit 442 corrects the first concealment signal for a value of power of the first concealment signal in each subframe in accordance with the formula below to obtain the concealment signal Y(k, l). Specifically, it performs the correction according to the below formula (provided that 0≤l≤L−1 and 0≤k≤K−1). P−di(m) represents the power of the ith subband about the subframe contained in the auxiliary information code transmitted in the d-th packet before the pertinent packet (packet of a first concealment signal generation target).
P ^ i ⁡ ( m ) = P - d i ⁡ ( m ) Z ′ ⁡ ( k , l ) = Z ⁡ ( k , l ) 1 K max i - K min i ⁢ ∑ k = K min i K max i ⁢ ⁢ Z 2 ⁡ ( k , l ) , ( K min i ≤ k ≤ K max i , 0 ≤ i ≤ I - 1 ) Y ⁡ ( k , l ) = 10 P ^ i ⁡ ( m ) / 20 · Z ′ ⁡ ( k , l ) , ( K min i ≤ k ≤ K max i , 0 ≤ i ≤ I - 1 )
The above fifth embodiment showed the example in which the auxiliary information was calculated and encoded for the frame “later by d frames” than the encoding of the target signal, but the auxiliary information may be calculated and encoded for the frame “earlier by d frames” than the encoding of the target signal, as in the third embodiment.
The subframe power calculation unit 121 saves the audio signal for a predetermined period of time, and calculates a subframe power sequence P1(l) for audio signals s(dT), s(1+dT), . . . , s((d+1)T−1) that are later by a predetermined number of frames (d frames in the present embodiment) than the encoding of the target signals s(0), s(1), . . . , s(T−1) out of the saved audio signal.
Furthermore, the subframe power calculation unit 121 calculates a subframe power sequence P2(l) for audio signals s((d+1)T), s(1+(d+1)T), . . . , s((d+2)T−1) later by a predetermined number of frames ((d+1) frames in the present embodiment).
It is assumed herein that the number of samples contained in one frame is T. When a prediction target signal is expressed by the following formula:
the powers P1(l), P2(l) of subframe l (0≤l≤L−1) are obtained by the following formulas. The letter k represents an index of a sample in each subframe (0≤k≤K−1).
P 1 ⁡ ( l ) = 10 ⁢ ⁢ log 10 ⁡ ( 1 K ⁢ ∑ k = 0 K - 1 ⁢ ⁢ v 2 ⁡ ( K · l + k ) ) P 2 ⁡ ( l ) = 10 ⁢ ⁢ log 10 ⁡ ( 1 K ⁢ ∑ k = 0 K - 1 ⁢ ⁢ v 2 ⁡ ( K · l + k + T ) )
The present embodiment defines K as the length of each subframe, but different lengths may be used for the respective subframes, which are determined in advance for the respective subframes. The subframe power sequence may also be calculated in accordance with the following formula where kstart1 represents an index of a start of the lth subframe and kend1 represents an index of an end thereof.
P ⁡ ( l ) = 10 ⁢ ⁢ log 10 ⁡ ( 1 k end l - k start l ⁢ ∑ k = k start l k end l ⁢ ⁢ v 2 ⁡ ( k end l - 1 + k ) )
The attenuation coefficient estimation unit 122 calculates slopes γopt1, γopt2 of straight lines indicative of respective temporal changes of power from the subframe power sequences P1(l), P2(l), for example, by the least square method or the like. The calculation method is the same as that performed by the attenuation coefficient estimation unit 122 in the first embodiment.
The attenuation coefficient quantization unit 123 performs the scalar quantization of each of the slopes γopt1, γopt2 of the straight lines, encodes the results of the scalar quantization, and outputs auxiliary information codes C1, C2. It may use the scalar quantization codebook prepared in advance. In the case of the straight-line approximation of subframe power P(l), intercepts Popt1, Popt2 may also be encoded in addition to the slopes γopt1, γopt2 of the straight lines.
The code multiplexing unit 13 writes the audio code and the auxiliary information codes C1, C2 in a predetermined order and outputs a bitstream. FIG. 14 shows an example of temporal relationship between signals as audio encoding targets and signals as auxiliary information encoding targets, and a configuration of bitstreams. As shown in FIG. 14, for example, the auxiliary information code of frame (N+1) and the auxiliary information code of frame (N+2) are added to the audio code of frame N to obtain a bitstream, which is output from the code multiplexing unit 13. Furthermore, the packet configuration unit 2 in FIG. 1 adds the packet header information to the bitstream to obtain an audio packet to be transmitted as the N-th packet. Although the present embodiment shows the generation of the two pieces of auxiliary information, the auxiliary information to be generated may be three or more pieces of auxiliary information. The auxiliary information may be calculated for a target of an audio signal that is earlier by one or more frames than the audio signal encoded by the audio encoding unit.
The code separation unit 40 reads the audio code and auxiliary information codes C1, C2 from the bitstream, and sends the audio code to the audio decoding unit 42 and the auxiliary information codes C1, C2 to the auxiliary information decoding unit 45.
The auxiliary information decoding unit 45 decodes the auxiliary information codes C1, C2, calculates the auxiliary information, and sends the result to the concealment signal correction unit 44. For example, the auxiliary information decoding unit 45 decodes the auxiliary information codes C1, C2 output from the code separation unit 40, to generate indexes, and obtains slopes γJ of straight lines corresponding to the respective indexes from the codebook. Here, P(−1) represents the power of the last subframe in the signal received normally immediately before the frame loss.
When the intercepts of the straight lines are simultaneously encoded based on the straight-line approximation of subframe powers, the subframe powers are obtained according to the following formula using the intercepts PJ.
The subframe power correction unit 442 corrects the first concealment signal for a value of power of the first concealment signal in each subframe in accordance with the formula below to obtain the concealment signal Y(K·l+k). Specifically, it performs the correction according to the below formula (provided that 0≤l≤L−1 and 0≤k≤K−1). P−d(m) represents the power about the subframe contained in the auxiliary information code C1 transmitted in the d-th packet before the pertinent packet (packet of a first concealment signal generation target).
P ^ ⁡ ( m ) = P - d ⁡ ( m ) z ′ ⁡ ( K · l + k ) = z ⁡ ( K · l + k ) 1 K ⁢ ∑ k = 0 K - 1 ⁢ ⁢ z 2 ⁡ ( K · l + k ) Y ⁡ ( K · l + k ) = 10 P ^ ⁡ ( m ) / 20 · z ′ ⁡ ( K · l + k )
For example, the subframe power correction unit 442, as shown in FIG. 8, earlier extracts the auxiliary information transmitted in the d-th packet, from the auxiliary information storage unit 441 (step S60 in FIG. 8), calculates the mean square amplitude value for each subframe as to the first concealment signal, and divides the value contained in the subframe, by the mean square amplitude value (step S61). This calculation results in obtaining z′(K·l+k). Then powers of respective subframes are calculated from the auxiliary information and the value of the subframe is multiplied by a mean amplitude value obtained from the powers (step S62). This multiplication results in obtaining the concealment signal Y(K·l+k). The above processing of steps S4101 to S4421 (FIG. 7) is repeated to the end of the audio signal (step S4431).
When a consecutive packet loss further occurs, the packet loss can also be concealed in the case of occurrence of the consecutive packet loss by carrying out the same processing, using the power about the subframe contained in the auxiliary information code C2 transmitted in the d-th packet before the pertinent packet (packet of a first concealment signal generation target).
The seventh embodiment will describe an example in which the auxiliary information about a sudden change of power (which will be referred to hereinafter as “transient”) to be used herein is a position of the transient in a frame as an auxiliary information encoding target, and a power of a subframe at the position of the transient.
The operation of the auxiliary information encoding unit 12 of this configuration will be described based on FIG. 21. The transient detection unit 124A saves the audio signal for a predetermined period of time, and detects a transient using audio signals s(dT), s(1+dT), . . . , s((d+1)T−1) that is later by a predetermined number of frames (d frames in the present embodiment) than the encoding of the target signals s(0), s(1), . . . , s(T−1) out of the saved audio signal (step S7401 in FIG. 21). The auxiliary information encoding target frame may be a frame that is later by one or more frames than an audio encoding target frame or may be a frame that is earlier by one or more frames than an audio encoding target frame. The auxiliary information codes may be calculated from two or more frames selected from frames that are earlier or later by one or more frames than the audio encoding target frame.
A method for detection of the transient can be, for example, the method described in Section 7.2 in “ITU-T Recommendation G.719.” The transient may also be detected using one of other standard technologies and non-standard technologies. In the above method described in Section 7.2, the power is calculated in each subframe and then a temporal change of each subframe is compared with a threshold to determine whether or not there is a transient. Calculated as a result of the transient detection are: a transient flag Ftran indicative of whether a transient is contained in the auxiliary information encoding target frame, a position ltran of the transient, and a subframe power sequence P(l). When a power of a subframe at the position ltran of the transient is represented by P(ltran) as shown in FIG. 41, the transient detection unit 124A outputs the position ltran of the transient through line 1L45, outputs the power P(ltran) of the subframe at the position ltran of the transient through line 1L46, and outputs the transient flag Ftran through line 1L47. The transient detection unit 124A may be configured to output the position ltran of the transient and the subframe power sequence P(l) through line 1L46.
For example, when the transient detection is carried out by the method described in Section 7.2 in “ITU-T Recommendation G.719,” the transient detection unit 124A is supposed to calculate the same parameter as the subframe power sequence calculated by the subframe power calculation unit 121 in FIG. 4. When the transient detection is carried out by other methods, the transient detection unit 124A also calculates and outputs the same parameter as the subframe power sequence calculated by the subframe power calculation unit 121 in FIG. 4.
When the transient flag Ftran does not indicate a value for inclusion of a transient in a frame, a value indicative of a normal frame is entered in Ftran. In this case, the parameter encoding unit 127 encodes only the transient flag and outputs the encoded data as an auxiliary information code (step S7702 in FIG. 21).
On the other hand, when the transient flag Ftran indicates a value for inclusion of a transient in a frame, the transient position quantization unit 125 performs the scalar quantization of the position ltran of the transient by a predetermined bit count and outputs quantized position information (step S7501 in FIG. 21). The scalar quantization may be performed by a method of binary coding with ltran being regarded as a binary number, or by a method of providing predetermined positions with indexes, and performing binary encoding of an index at the closest position to ltran, or by entropy coding such as Huffman coding, or by any other quantization method. FIG. 42(a) shows a schematic diagram of an example of transient position information encoding by the binary coding, and FIG. 42(b) a schematic diagram of an example of transient position information encoding by the scalar quantization. As a modification example, another available method is as follows: two or more subframe indexes are selected as “information indicative of a change of power,” in addition to the position of the transient, and the two or more subframe indexes thus selected are encoded and transmitted. There are no particular restrictions on the method of encoding herein.
When the value for inclusion of a transient in a frame is set in the transient flag Ftran, the transient power scalar quantization unit 126 performs the scalar quantization of the power of the subframe corresponding to the position ltran of the transient and outputs the quantized transient power (step S7601 in FIG. 21). For example, in a case where the quantization is performed between 0 dB and 96 dB with use of a 6-bit linear encoder, the quantization is carried out according to the below formula. In this formula, C can be the value of 1.55 and c can be the value of 0.001 or the like, but these constants may be changed according to the quantization bit count or the like.
I E = ⌊ 10 ⁢ ⁢ log ⁡ ( P ⁡ ( l tran ) + ɛ ) C ⌋
According to the above formula, the power of the transient is quantized into an index ranging from 0 to 63. The quantization may be carried out using a codebook determined in advance by learning or the like, or any other quantization means may be applied. When the transient flag Ftran does not indicate the value for inclusion of a transient in a frame, the value indicative of a normal frame is entered in IE in the above formula.
The operation of the auxiliary information decoding unit 45 of this configuration will be described based on FIG. 23. The auxiliary information decoding unit 45 decodes the auxiliary information code and determines whether the obtained transient flag Ftran is on (indicative of a frame including a transient) or off (indicative of a frame including no transient) (step S7901 in FIG. 23).
When the transient flag Ftran indicates a frame containing no transient, only the value of the transient flag Ftran is output as auxiliary information (step S7142 in FIG. 23).
On the other hand, when the transient flag Ftran indicates a frame including a transient, the auxiliary information decoding unit reads the quantized position information ltran out of the auxiliary information code, decodes it, and outputs the quantized position information (step S7121 in FIG. 23). Furthermore, the unit reads and decodes the quantized transient power IE from the auxiliary information code and outputs the decoded transient power (step S7131 in FIG. 23). For example, where the linear quantization as described above is used, the decoded transient power is obtained from the quantized transient power in accordance with the following formula.
{circumflex over (P)}tran=10C·IE/20
Then the auxiliary information decoding unit 45 outputs the calculated transient flag Ftran, quantized position information, and decoded transient power as auxiliary information (step S7141 in FIG. 23).
On the other hand, when the error flag is on (to indicate a packet loss), the subframe power correction unit 442 reads the transient flag, quantized position information, and decoded transient power from the auxiliary information storage unit 441, and corrects the first concealment signal for a value of power of the first concealment signal z(K·l+k) in each subframe to obtain a concealment signal y(K·l+k) (provided that 0≤l≤L−1 and 0≤k≤K−1) (step S7901 in FIG. 25). Specifically, the subframe power correction unit 442 corrects the value of the power of the first concealment signal z(K·l+k) in accordance with the following procedure. First, the first concealment signal output from the first concealment signal generation unit 43 is fed through line 6L002 in FIG. 24 to the subframe power correction unit 442. Next, the subframe power correction unit 442 reads the transient flag Ftran, the transient position information ltran, and the decoded transient power represented by
{circumflex over (P)}tran,
from the auxiliary information storage unit 441.
Next, the subframe power correction unit 442 calculates a corrected power of each subframe from the transient position information ltran and the decoded transient power represented by
which are read from the auxiliary information storage unit 441 (step S7121 in FIG. 25). Specifically, the calculation is carried out according to the following procedure. First, the power of each subframe is calculated according to the following formula.
P ⁡ ( m ) = { 10 ⁢ ⁢ log ⁢ ⁢ 10 ⁢ ( 1 K ⁢ ∑ k = 0 K - 1 ⁢ ⁢ z 2 ⁡ ( K · m + k ) )
Next, the subframe power correction unit calculates a difference between the power of the first concealment signal at the position of the transient and the decoded transient power (differential transient power).
{dot over (P)}tran=P(ltran)−{circumflex over (P)}tran
Then the subframe power correction unit corrects the power of the first concealment signal corresponding to each subframe after the position of the transient, using the foregoing differential transient power, to obtain a corrected concealment signal subframe power.
P ^ ⁡ ( m ) = { P ⁡ ( m ) ⁢ ( 0 ≤ m < l tran ) P ⁡ ( m ) + P . tran ⁡ ( l tran ≤ m < L - 1 )
z ′ ⁡ ( K · l + k ) = z ⁡ ( K · l + k ) 1 K ⁢ ∑ k = 0 K - 1 ⁢ z 2 ⁡ ( K · l + k )
Finally, the subframe power correction unit multiplies the normalized first concealment signal by the corrected concealment signal subframe power to calculate a concealment signal (step S7131 in FIG. 25).
y(K·l+k)=10{circumflex over (P)}(m)/20·z′(K·l+k)
As a modification example of step S7121 in FIG. 25, the method of calculating from the subframe power P(m) and the decoded transient power:
the corrected concealment signal subframe power:
{circumflex over (P)}(m)
may be a method as represented by the following formula.
P . tran = P ⁡ ( l tran ) - P ^ tran P ′ ⁡ ( m ) = { P ⁡ ( m ) ⁢ ( 0 ≤ m < l tran ) P ⁡ ( m ) + P . tran ⁡ ( l tran ≤ m < L - 1 )
Finally, a corrected concealment signal power is calculated using a predetermined prediction coefficient ap. The prediction coefficient may be switched to another, depending upon properties of subframe power sequences.
{circumflex over (P)}(m)=Σp=0Pap·P′(m−p)
Alternatively, smoothing may be carried out using a model determined in advance.
{circumflex over (P)}(m)=f(P′(0), . . . ,P′(L−1))
P _ ⁡ ( m ) = P ⁡ ( m ) P ⁡ ( l tran )
The vector quantization is carried out according to the following formula.
J = arg ⁢ ⁢ min i = 0 , … ⁢ , I - 1 ⁢ ∑ l = 0 L - 1 ⁢ ( c i ⁡ ( l ) - P _ ⁡ ( l + l tran - L + 1 ) ) 2
The letter I represents the number of entries of straight lines or vectors in a codebook and the letter J represents an index of a selected straight line or vector (which will be referred to hereinafter as “code vector index”). ci(l) indicates the lth element of the ith code vector in the codebook.
J = arg ⁢ ⁢ min i = 0 , … ⁢ , I - 1 ⁢ ∑ l = 0 L - 1 ⁢ ( c i ⁡ ( l ) - P ⁡ ( l + l tran - L + 1 ) ) 2
The operation of the auxiliary information decoding unit 45 is shown in FIG. 31. The auxiliary information decoding unit 45 reads the transient flag Ftran, the quantized position information ltran, the quantized transient power IE, and the code vector index J from the auxiliary information code and determines the state of the transient flag Ftran (step S901 in FIG. 31). When the value of the transient flag Ftran indicates no transient, only the value of the transient flag Ftran is output as auxiliary information (step S906 in FIG. 31), as in the seventh embodiment.
On the other hand, when the value of the transient flag Ftran indicates a transient, the quantized position information ltran is decoded by the same method as in step S7121 in FIG. 23 in the seventh embodiment and the decoded position information is output (step S902 in FIG. 31).
A code vector cJ(m) corresponding to the code vector index J is output (step S904 in FIG. 31).
On the other hand, when the value of the error flag indicates a packet loss (on), the subframe power correction unit 442 corrects the first concealment signal z(K·l+k) for a value of power of the first concealment signal in each subframe in accordance with the below-described formula to obtain the concealment signal y(K·l+k) (provided that 0≤l≤L−1 and 0≤k≤K−1). Specifically, the value of power of the first concealment signal is corrected in each subframe in accordance with the following procedure.
P ⁡ ( m ) = { 10 ⁢ ⁢ log ⁢ ⁢ 10 ⁢ ( 1 K ⁢ ∑ k = 0 K - 1 ⁢ z 2 ⁡ ( K · m + k ) )
Next, the correction unit calculates the differential transient power which is the difference between the subframe power corresponding to the transient position and the decoded transient power.
{dot over (P)}tran=P(ltran)−Ptran
Next, the corrected concealment signal subframe power is calculated using the differential transient power and the code vector.
P ^ ⁡ ( m ) = { P tran · c J ⁡ ( L - l tran - 1 + m ) ⁢ ( 0 ≤ m < l tran ) P ⁡ ( m ) + P . tran ⁡ ( l tran ≤ m < L - 1 )
The present embodiment shows the example of the vector quantization after the normalization of the values of the subframe power sequence on the encoder side, but it is also possible to adopt a method in which the vector quantization of the subframe power sequence is carried out without execution of the normalization. In the case without execution of the normalization, the corrected concealment signal subframe power is calculated as follows.
P ^ ⁡ ( m ) = { c J ⁡ ( L - l tran - 1 + m ) ⁢ ( 0 ≤ m < l tran ) P ⁡ ( m ) + P . tran ⁡ ( l tran ≤ m < L - 1 )
Finally, the normalized first concealment signal is multiplied by the corrected subframe power and the concealment signal is output (step S1505 in FIG. 32).
The encoding unit 1 in the ninth embodiment has the same configuration as in FIG. 2 described in the first embodiment, and thus the detailed description of the entire unit will be omitted herein. The time-frequency transform is as described in the fourth embodiment and the signals after the transform into the frequency domain are denoted by V(k, l). The letter k herein is an index of a frequency bin (provided that 0≤k≤K−1) and l an index of a subframe (provided that 0≤l≤L−1).
The transient detection unit 124A detects a transient, using the signals obtained by the transform into the frequency domain. The detection of transient may be carried out using the means used in the seventh embodiment, or using TS26.404 or the like which is the standard technology of transient detection for signals in the frequency domain, or using another transient detection technology for frequency-domain signals. The subband power sequence is calculated herein about values in a range (Ks≤k<Ke) in the frequency domain preliminarily determined in the transient detection. The signals in the frequency band to be used in the detection of transient may be signals in the entire band or only at least one specific subband may be used.
P ⁡ ( l ) = 10 ⁢ ⁢ log 10 ⁡ ( 1 K e - K s ⁢ ∑ k = K s K e - 1 ⁢ V 2 ⁡ ( k , l ) )
When the error flag indicates a normal frame, the auxiliary information decoding unit 45 reads the transient flag Ftran, quantized position information ltran, and quantized transient power IE from the auxiliary information code. In the case of the transient flag, quantized position information, and quantized transient power being encoded, the auxiliary information decoding unit 45 decodes the auxiliary information code by corresponding decoding means to obtain these parameters. For example, in the case using the linear quantization as described above, the decoded transient power is obtained from the quantized transient power in accordance with the following formula.
Next, the operation of the concealment signal correction unit will be described. When the error flag indicates a packet loss, the subframe power correction unit 442 reads the auxiliary information from the auxiliary information storage unit 441 and corrects the first concealment signal Z(l, k) for a value of power of the first concealment signal in each subframe in accordance with the below formula to obtain the concealment signal Y(l, k). Specifically, it performs the correction in accordance with the below formula (provided that 0≤l≤L−1 and 0≤k≤K−1).
P ⁡ ( m ) = { 10 ⁢ ⁢ log ⁢ ⁢ 10 ⁢ ( 1 K e - K s ⁢ ∑ k = K s K e - 1 ⁢ Z 2 ⁡ ( m , k ) )
Furthermore, the correction unit calculates the difference between the power of the first concealment signal at the position of the transient and the decoded transient power (differential transient power).
Furthermore, it corrects the power of the first concealment signal corresponding to each subframe after the position of the transient, using the aforementioned differential transient power, to obtain the corrected concealment signal subframe power.
Z ′ ⁡ ( l , k ) = Z ⁡ ( l , k ) 1 K e - K s ⁢ ∑ k = K s K e - 1 ⁢ Z 2 ⁡ ( l , k ) , ( K s ≤ k < K e )
Finally, the normalized first concealment signal is multiplied by the corrected concealment signal subband power to calculate the concealment signal.
Y(l,k)=10{circumflex over (P)}(l)/20·Z′(l,k),(Ks≤k<Ke)
On the other hand, when the transient flag indicates that the auxiliary information code corresponds to a transient, the transient position decoding unit 1212 decodes the quantized transient position information and outputs the resulting transient position information (which will be referred to hereinafter as “decoded position information”) (step S1904 in FIG. 36), and the transient power decoding unit 1213 decodes the encoded quantized power and outputs the resulting decoded transient power (step S1905 in FIG. 36), thereby outputting the transient flag, the decoded position information, and the decoded transient power as auxiliary information (step S1906 in FIG. 36). The operations of the transient position decoding unit 1212 and the transient power decoding unit 1213 are the same as in the seventh embodiment.
When the transient flag Ftran indicates the value for inclusion of a transient in a frame, the code length selection unit 128A outputs a predetermined bit count larger than one bit (step S2204 in FIG. 39).
The transient position quantization unit 125 scalar-quantizes the position ltran of the transient by the predetermined bit count and outputs the quantized position information (step S2205 in FIG. 39). The operation of the transient position quantization unit 125 is the same as in the seventh embodiment.
Next, the transient power scalar quantization unit 126 performs the scalar quantization of the power of the subframe corresponding to the position ltran of the transient and outputs the quantized transient power (step S2206 in FIG. 39). The operation of the transient power scalar quantization unit 126 is the same as in the seventh embodiment.
On the other hand, when it is determined in step S2201 that the transient flag Ftran does not show the value for inclusion of a transient in a frame, the code length selection unit 128A determines the code length to be one bit (step S2202 in FIG. 39). Next, the parameter encoding unit 127 encodes only the transient flag by one bit and outputs it (step S2203 in FIG. 39).
The operation of the auxiliary information decoding unit 45 of this configuration will be described based on FIG. 40. The auxiliary information decoding unit 45 decodes the auxiliary information code and determines whether the resulting transient flag Ftran is on (to indicate a frame containing a transient) or off (to indicate a frame containing no transient) (step S2401 in FIG. 40).
When the transient flag Ftran shows a frame containing a transient, the transient flag decoding unit 129 further reads the quantized position information from the auxiliary information code and outputs the information to the transient position decoding unit 1212, and it further reads the quantized transient power IE from the auxiliary information code and outputs the power to the transient power decoding unit 1213 (step S2402 in FIG. 40).
Next, the transient position decoding unit 1212 decodes the quantized position information and outputs the resulting decoded position information ltran (step S2403 in FIG. 40). Furthermore, the transient power decoding unit 1213 decodes the quantized transient power IE and outputs the resulting decoded transient power P(ltran) (step S2404 in FIG. 40).
This operation results in outputting the transient flag Ftran, decoded position information ltran, and decoded transient power P(ltran) as auxiliary information (step S2405 in FIG. 40). The steps S2403 to S2405 in FIG. 40 are the same as in the seventh embodiment.
On the other hand, when the transient flag Ftran shows a frame containing no transient, only the transient flag Ftran is output as auxiliary information (step S2406 in FIG. 40).
The configuration of the auxiliary information decoding unit 45 in the present embodiment is as shown in FIG. 44. In the present embodiment, the auxiliary information code transmitted from the encoding unit 1 does not contain the transient flag and the quantized position information. Then, in the present embodiment the transient flag is always set to the value of on and a predetermined value lconst is always set as the transient position information. The transient power decoding unit 1213 decodes the auxiliary information code (quantized power code) containing only the quantized transient power by the same processing as in the seventh embodiment and outputs the decoded transient power.
The operation of the transient flag decoding unit 129 and the operation of the transient power decoding unit 1213 are the same as in the seventh embodiment. In the present embodiment, the predetermined value lconst is always set in the transient position information, as in the twelfth embodiment.
In the fourteenth embodiment, the subframe at the transient position is divided into subbands and a power of at least one subband is quantized as auxiliary information. In the quantization of the power of at least one subband, at least one subband among one or more subbands is defined as “core subband.” Next, for a subband except for the core subband, a difference between a power of the subband (the subband except for the core subband) and a power of the core subband is calculated and the power of the core subband and the foregoing difference are quantized as auxiliary information. The power of the core subband may be contained in the auxiliary information or, may not be contained in the auxiliary information while a value contained in the audio code itself may be used instead.
The encoding unit 1 in the present embodiment has the same configuration as in FIG. 10 described in the first embodiment, and the detailed description of the entire unit is omitted herein. The time-frequency transform is as described in the fourth embodiment. The signal after the transform into the frequency domain is denoted by V(k, l). The letter k herein represents an index of a frequency bin (provided that 0≤k≤K−1) and l an index of a subframe (provided that 0≤l≤L−1). The time-frequency transform unit 10 supplies both of the signal V(k, l) after the transform into the frequency domain and the audio signal before the time-frequency transform to the auxiliary information encoding unit 12.
The subband power calculation unit 128B calculates subband powers of the subframe corresponding to the transient position, in accordance with the formula below. P(i)(ltran) represents the power of the ith subband at the transient position. Furthermore, Ks(i) and Ke(i) represent an index of the first frequency bin of the ith subband and an index of the last frequency bin of the ith subband, respectively.
P ( i ) ⁡ ( l tran ) = 10 ⁢ ⁢ log 10 ⁡ ( 1 K e ( i ) - K s ( i ) ⁢ ∑ k = K s ( j ) K e ( i ) - 1 ⁢ V 2 ⁡ ( k , l tran ) )
The core subband power quantization unit 129A defines a predetermined icore-th subband as a core subband, quantizes the power of the core subband defined as follows:
P(icore)(ltran),
and outputs a core subband power code. The quantization may be quantization using a predetermined quantization codebook or quantization by entropy coding using the Huffman coding or the like. In another method, J subbands of not less than one subband preliminarily determined as follows:
(icore(1) . . . icoreJ)
are defined as core subbands, and an average of powers of the J subbands is defined as a power of the core subbands. It is also possible to adopt a maximum, a minimum, or the median of the J subbands as a power of the core subbands. Furthermore, the core subband power quantization unit 129A decodes the core subband power code and outputs the decoded core subband power denoted as follows.
{circumflex over (P)}(icore)(ltran)
The difference quantization unit 1210A calculates a differential subband power sequence expressed as follows:
{dot over (P)}(i)(ltran)
in accordance with the formula below, quantizes the sequence, and outputs the differential subband power code. The quantization may be quantization using a predetermined quantization codebook, quantization by entropy coding using the Huffman coding or the like, or quantization by the vector quantization if the differential subband power sequence has two or more subbands.
{dot over (P)}(i)(ltran)=P(i)(ltran)−{circumflex over (P)}(icore)(ltran)
The core subband power decoding unit 1214A decodes the quantized core subband power and outputs the decoded core subband power expressed as follows.
{circumflex over (P)}(icore) (ltran)
The difference decoding unit 1215 decodes the differential subband power code and outputs the decoded differential subband power sequence expressed as follows.
{tilde over (P)}(i)(ltran)
Furthermore, the difference decoding unit 1215 adds the decoded differential subband power sequence and the decoded core subband power in accordance with the formula
{circumflex over (P)}(i)(ltran)={tilde over (P)}(i)(ltran)+{circumflex over (P)}(icore)(ltran)
to calculate a transient power spectrum expressed as follows.
{circumflex over (P)}(i)(ltran)
Next, the operation of the subframe power correction unit 442 (FIG. 24) in the present embodiment will be described. The auxiliary information storage unit 441 stores the transient flag and the transient power spectrum obtained by the forgoing auxiliary information decoding unit 45, as auxiliary information, and the subframe power correction unit 442 reads the transient flag and the transient power spectrum from the auxiliary information storage unit 441, and corrects the first concealment signal z(K·l+k) for a value of power thereof in each subframe to obtain the concealment signal y(K·l+k). Specifically, it performs the correction in accordance with the following procedure (provided that 0≤l≤L−1 and 0≤k≤K−1).
Next, the subframe power correction unit 442 sets a predetermined value in the transient position information ltran.
P ^ ( i ) ⁡ ( l tran ) = 10 ⁢ ⁢ log 10 ⁡ ( 1 K e ( i ) - K s ( i ) ⁢ ∑ k = K s ( j ) K e ( i ) - 1 ⁢ Z 2 ⁡ ( k , l tran ) )
Next, the subframe power correction unit 442 calculates a difference between the subband power sequence of the first concealment signal at the position of the transient and the transient power spectrum (differential transient power) in accordance with the formula below.
P(i)(l)={circumflex over (P)}(i)(l)−{circumflex over (P)}(i)(ltran)
Finally, the subframe power correction unit 442 multiplies the first concealment signal by the corrected concealment signal subframe power in accordance with the formula below for all the subbands i, to calculate the concealment signal. However, Ks(i)≤k<Ke(i) and l≥ltran.
y(k,l)=10P(i)(l)/20·z(k,l)
The audio encoding unit 11 feeds the decoded core subband power Pcore obtained by decoding the code about the power included in the audio code, to the difference quantization unit 1210A.
The difference quantization unit 1210A calculates the differential subband power sequence expressed as follows:
in accordance with the formula below, quantizes the sequence, and outputs the resulting differential subband power code. The quantization may be quantization using a predetermined quantization codebook, quantization by entropy coding using the Huffman coding or the like, or quantization by vector quantization if the differential subband power sequence has two or more subbands.
{dot over (P)}(i)(ltran)=P(i)(ltran)−Pcore
The audio decoding unit 42 decodes the code about the power included in the audio code and feeds the resulting decoded core subband power Pcore to the difference decoding unit 1215. If Pcore is a value obtained in a domain different from the signal V(k, l) after the transform into the frequency domain, e.g., a value in the time domain, an offset is added to express Pcore is in the same unit, and then Pcore is fed to the difference decoding unit 1215.
Furthermore, the difference decoding unit 1215 adds the decoded differential subband power sequence and the decoded core subband power to calculate the transient power spectrum expressed as follows:
{circumflex over (P)}(i)(ltran),
in accordance with the formula below.
{circumflex over (P)}(i)(ltran)={tilde over (P)}(i)(ltran)+Pcore
1. An audio encoding device for encoding an audio signal consisting of a plurality of frames, the encoding device comprising:
an audio encoding unit executed by the processor to encode the audio signal;
an auxiliary information encoding unit executed by the processor to estimate and encode auxiliary information about a temporal change of power of the audio signal, the auxiliary information used in packet loss concealment in decoding of the audio signal,
wherein the auxiliary information encoding unit estimates and encodes quantized transient power and a flag of sudden change of power, as the auxiliary information about the temporal change of power of the audio signal; and
wherein the auxiliary information encoding unit is configured to operate in a first mode to encode the quantized transient power and the flag of sudden change of power in response to the flag being indicative of the presence of a transient in the audio signal, and the auxiliary information encoding unit is further configured to operate in a second mode to encode only the flag in response to the flag being indicative of the absence of a transient in the audio signal.
2. An audio encoding method for encoding an audio signal consisting of a plurality of frames, the audio encoding method comprising:
encoding the audio signal with an audio encoding device; and
estimating and encoding auxiliary information about a temporal change of power of the audio signal with the audio encoding device, the auxiliary information being used in packet loss concealment during subsequent decoding of the audio signal,
wherein the step of encoding auxiliary information comprises estimating and encoding quantized transient power and a flag of sudden change of power, as the auxiliary information; and
wherein encoding the quantized transient power and the flag of sudden change of power comprises encoding the quantized transient power and the flag of sudden change of power in response to a transient being present in the audio signal, and encoding only the flag of sudden change of power in response to absence of the transient in the audio signal.
8010353 August 30, 2011 Kawashima et al.
8150684 April 3, 2012 Kawashima et al.
8320391 November 27, 2012 Ohmuro et al.
8620644 December 31, 2013 Ryu et al.
20030002588 January 2, 2003 Faller
20040083110 April 29, 2004 Wang
20040138886 July 15, 2004 Absar et al.
20070094009 April 26, 2007 Ryu et al.
20080126904 May 29, 2008 Sung et al.
20080262845 October 23, 2008 Keohane et al.
20090177478 July 9, 2009 Jax et al.
20090288546 November 26, 2009 Takeda
20100036656 February 11, 2010 Kawashima
20100049509 February 25, 2010 Kawashima et al.
20100094642 April 15, 2010 Zhan et al.
20130253939 September 26, 2013 Tsutsumi et al.
1906663 January 2007 CN
07-336310 December 1995 JP
2007-336310 December 1995 JP
2003-316670 November 2003 JP
2008-261904 October 2008 JP
2010-511201 April 2010 JP
WO 2005/109401 November 2005 WO
WO 2007/000988 January 2007 WO
Digital Transmission Systems-Terminal Equipments-Coding of Analogue Signals by Pulse Code Modulation Series G. Pulse Code Modulation (PCM) of Voice Frequencies, Intl. Telecommunication Union (ITU-T) Recommendation G.711, Appendix 1 (Sep. 1999) 26 Pgs.
3rd Generation Partnership Project: Technical Specification Group Services and System Aspects; General audio codec audio processing functions; Enhanced aacPlus General Audio Codec; General Description (Release 6) 3GPP TS 26.401 V0.0.1 (May 2004) 12 Pgs. (GSM).
Digital Terminal Equipments-Coding of Voice and Audio Signals Series G. Frame error robust narrow-band and wideband embedded variable bit-rate coding of speech and audio from 8-32 kbits/s. Intl Telecommunication Union (ITU-T) Recommendation G.718 (Jun. 2008) 257 Pgs.
3rd Generation Partnership Project: Technical Specification Group Services and System Aspects; General audio codec audio processing functions; Enhanced aacPlus general audio codec; Enhanced aacPlus encoder SBR part (Release 8) 3GPP TS 26.404 V8.0.1 (Dec. 2008) 12 Pgs. (Lte).
PCT International Search Report dated Nov. 17, 2011, PCT/JP2011/075489, 2Pgs.
Noriko Komaki et al., “A Packet Loss Concealment Technique For VoIP Using Steganography,” dated Aug. 1, 2003, vol. E86-A No. 8, pp. 2069-2072, Institute of Electronics, Information and Communication Engineers Transactions on Fundamentals of Electronics, Communications and Computer Sciences, Engineering Sciences Society, Tokyo, Japan.
“G.729-based Embedded Variable Bit-rate Coder: An 8-32 kbit/s Scalable Wideband Coder Bitstream Interoperable with G.729,” dated May 29, 2006, pp. 1-100, ITU-T Standard, International Telecommunication Union, Geneva, Switzerland.
Angel M. Gómez et al., “A Multipulse-Based Forward Error Correction Technique for Robust CELP-Coded Speech Transmission Over Erasure Channels,” dated Aug. 1, 2010, vol. 18 No. 6, pp. 1258-1268, Institute of Electrical and Electronics Engineers Transactions on Audio, Speech and Language Processing, Institute of Electrical and Electronics Engineers Service Center, New York, New York.
Extended European Search Report, dated Apr. 7, 2014, pp. 1-11, Issued in International Application No. PCT/JP2011075489, European Patent Office, Munich Germany.
Tian, W., et al., “Low-Delay Subband CELP Coding for Wideband Speech,” Nov. 26, 1996, Proceedings, 1996 IEEE Tencon. Digital Signal Processing Applications, Perth, WA, Australia Nov. 26-29, 1996, New York, NY, USA, IEEE, US, vol. 1, pp. 198-194, DOI: 11.1109/TENCON.1996.608783, ISBN: 978-0-78803-3679-7.
European Office Action, dated Jun. 17, 2015, pp. 1-7, issued in European Patent Application No. 11842953.9, European Patent Office, Munich, Germany.
Japanese Office Action with English translation, dated Sep. 29, 2015, pp. 1-7, issued in Japanese Patent Application No. P2012-545668, Japanese Patent Office, Tokyo, Japan.
Geiser, B., et al., “Joint Pre-Echo Control and Frame Erasure Concealment for VOIP Audio Codecs,” dated Aug. 24, 2009, pp. 1259-1263, IEEE, 17th European Signal Processing Conference (EUSIPCO 2009), Glasgow, Scotland, Aug. 24-28, 2009, XP032758754.
Geiser, B., et al., “Candidate Proposal for ITU-T Super-Wideband Speech and Audio Coding,” dated Apr. 19, 2009, pp. 4121-4124, IEEE International Conference on Acoustics, Speech and Signal Processing, 2009 (ICASSP 2009), Piscataway, NJ, USA, XP031460181.
Chibani, M., et al., “Fast Recovery for a CELP-Like Speech Codec After a Frame Erasure,” Nov. 1, 2007, pp. 2485-2495, IEEE Transactions on Audio, Speech and Language Processing, vol. 15 No. 8, Nov. 2007, IEEE Service Center, New York, NY, USA, XP011192967, ISSN: 1558-7916, DOI: 10.1109/TASL.2007.907332.
Extended European Search Report, dated Dec. 4, 2015, pp. 1-16, issued in European Patent Application No. 15184203.6, European Patent Office, Munich, Germany.
Japanese Office Action with English translation, dated Apr. 12, 2016, pp. 1-6, issued in Japanese Patent Application No. P2012-545668, Japanese Patent Office, Tokyo, Japan.
European Office Action, dated Sep. 22, 2016, pp. 1-6, issued in European Pateht Application No. 15 184 203.6, European Patent Office, Munich, Germany.
European Office Action, dated Jun. 12, 2017, pp. 1-6, issued in European Patent Application No. 15 184 203.6, European Patent Office, Munich, Germany.
Chinese Office Action with translation, dated Dec. 5, 2017, pp. 1-12, issued in Chinese Patent Application No. 201510317625.5, The State Intellectual Property Office of People's Republic of China, Beijing, PRC.
Patent number: 10115402
Patent Publication Number: 20170076729
Inventors: Kimitaka Tsutsumi (Kanagawa), Kei Kikuiri (Yokosuka)
Application Number: 15/298,979
International Classification: G10L 19/00 (20130101); G10L 19/005 (20130101); G10L 25/21 (20130101); G10L 19/025 (20130101);