Audio signal processing device, audio signal processing method, and recording medium storing a program

An audio signal processing device that includes: a processor configured to execute a procedure, the procedure comprising: detecting a speech segment of an audio signal; suppressing noise in the audio signal; and adjusting an amount of suppression of noise such that the amount of suppression during a specific period, which starts from a position based on a terminal end of the detected speech segment and is a period shorter than a period spanning from the terminal end of the detected speech segment to a starting end of a next speech segment, becomes greater than in other segments, and a memory configured to store audio signals before and after noise suppression and the amount of suppression before and after adjustment.

CROSS-REFERENCE TO RELATED APPLICATION

This application is based upon and claims the benefit of priority of the prior Japanese Patent Application No. 2015-190254, filed on Sep. 28, 2015, the entire contents of which are incorporated herein by reference.

FIELD

The embodiments discussed herein are related to an audio signal processing device, an audio signal processing method, and a recording medium storing a program.

BACKGROUND

Audio has increasingly been used as a user interface for electronic devices. When using audio as a user interface for an electronic device, speech is generally recognized by an application that performs speech recognition. Noise suppression is performed on the audio input to the application in order to increase the speech recognition rate of the application that performs speech recognition. For example, technology exists to detect sound source directions in each band on the frequency axis, and to suppress noise in cases in which the sound source direction is in a noise suppression range.

Related Patent Documents

SUMMARY

According to an aspect of the embodiments, an audio signal processing device includes a processor configured to execute a procedure. The procedure includes detecting a speech segment of an audio signal, suppressing noise in the audio signal, and adjusting an amount of suppression of noise such that the amount of suppression during a specific period, which starts from a position based on a terminal end of the detected speech segment and is a period shorter than a period spanning from the terminal end of the detected speech segment to a starting end of the next speech segment, becomes greater than in other segments. The audio signal processing device further includes a memory configured to store audio signals before and after noise suppression and the amount of suppression before and after adjustment.

DESCRIPTION OF EMBODIMENTS

First Exemplary Embodiment

Detailed explanation follows regarding a first embodiment, which is an example of an embodiment, with reference to the drawings.

An audio signal processing device10illustrated inFIG. 1includes a speech segment detector11, a suppressor12, and an adjustment section13. The speech segment detector11detects speech segments of an audio signal. The suppressor12suppresses noise in the audio signal. The adjustment section13adjusts an amount of suppression by the suppressor12such that the amount of suppression by the suppressor12during a specific period, which starts from a position based on a terminal end of a speech segment detected by the speech segment detector11and is a period shorter than a period spanning from the terminal end of the speech segment detected by the speech segment detector11to a starting end of the next speech segment, becomes greater than in other segments.

As illustrated inFIG. 2, the audio signal processing device10, as an example, includes a central processing unit (CPU)31, which is an example of a processor, a primary storage section32, a secondary storage section33, and a microphone35. The CPU31, the primary storage section32, the secondary storage section33, and the microphone35are connected to one another through a bus36. Note that the microphone35may be an external microphone connected to the audio signal processing device10through a microphone terminal or the like.

The primary storage section32is volatile memory such as random access memory (RAM). The secondary storage section33is non-volatile memory such as a hard disk drive (HDD) or a solid state drive (SSD).

As an example, the secondary storage section33stores an audio signal processing program34. The CPU31reads the audio signal processing program34from the secondary storage section33and expands the audio signal processing program34into the primary storage section32. The CPU31operates as the speech segment detector11, the suppressor12, and the adjustment section13ofFIG. 1by executing the audio signal processing program34. Note that the audio signal processing program34may be stored on an external server and expanded into the primary storage section32via a network, or may be stored on a non-transient recording medium such as a DVD and expanded into the primary storage section32via a recording medium reading device.

The microphone35picks up audio, and converts the audio into an audio signal. Note that the audio signal processing device10may be, for example, a personal computer, a tablet, a smartphone, a cellular phone, a dedicated audio signal processing device, or an audio signal processing device for a vehicle installed electronic device.

Next, explanation follows regarding an outline of operation of the audio signal processing device10. In the present exemplary embodiment, as illustrated inFIG. 3, in block43, the CPU31suppresses noise in an audio signal x that corresponds to audio picked up by the microphone35. The noise suppression may, for example, employ existing noise suppression technology such as noise suppression by filtering or noise suppression by spectral restoration.

In block41, the CPU31detects speech segments of the audio signal picked up by the microphone35. The speech segment detection may employ existing audio segment detection technology. The solid lines of graph (a) inFIG. 4illustrate speech segments51spoken by a user, and the dashed lines illustrate non-speech segments52between one speech segment51and another speech segment51. The non-speech segments52are segments that include background noise. The horizontal axis (c) ofFIG. 4represents passage of time T.

In block42, the CPU31controls such that the amount of suppression of noise in the audio signal performed in block43during specific periods Ts starting from positions based on terminal ends of speech segments51, is greater than the amount of suppression performed in segments other than the specific periods Ts. Hereafter, the specific periods Ts are also referred to as increased-suppression-amount segments Ts. The increased-suppression-amount segments Ts are periods shorter than a period spanning from the terminal end of a speech segment51to the starting end of the next speech segment51.

The CPU31transmits an audio signal in which noise has been suppressed to block44(also referred to as a speech recognition block44hereafter) so that speech recognition processing is performed on an audio signal in which noise has been suppressed.

In more detail, as illustrated by the example inFIG. 5, at step61, the CPU31picks up, for example, one frame worth of the audio signal x corresponding to the audio picked up by the microphone35. At step62, the CPU31determines whether or not the picked up audio signal x is an audio signal of a speech segment51.

In this example, the power P of the audio signal x is calculated using Equation (1), and determination of a speech segment is made in cases in which the power of the audio signal x is a specific value or above.
P=Σx(t)2(1)
In Equation (1), x(t) denotes an audio signal at a time t (t=Tfi (i=1, . . . , n; and n denotes the number of signal frame partitions and Tf1to Tfn represents a time length of one signal frame)).

In cases in which affirmative determination is made at step62, at step65, the CPU31suppresses noise in the audio signal x. (Note that, as described below, in cases in which negative determination is made at step62, the CPU31still suppresses noise in the audio signal x at step65, but after having executed other steps.)

For example, for additive noise, audio signals x(t) including noise, audio signals s(t) that do not include noise, and noise signals n(t) have the relationship indicated by Equation (2).
x(t)=s(t)+n(t)  (2)
When Equation (2) is considered in the frequency domain, the audio signal spectrum X(ω) including noise, the audio signal spectrum S(ω) that does not include noise, and the noise signal spectrum N(ω) have the relationship indicated by Equation (3).
X(ω)=S(ω)+N(ω)  (3)
Where ω denotes frequency.

For example, in noise suppression by filtering, as indicated by the example of Equation (4), an audio signal spectrum S′(ω) in noise is suppressed (also referred to as a noise suppressed signal spectrum S′(ω) hereafter) can be acquired by multiplying the audio signal spectrum X(ω) including noise by a gain G(ω) that is a filter.
S′(ω)=G(ω)X(ω)  (4)

In cases in which negative determination has been made at step62, at step63, the CPU31determines whether or not the picked up audio signal x is an audio signal x of an increased-suppression-amount segment Ts. In more detail, determination is made as to whether or not the picked up audio signal x is an audio signal x from during a specific period Ts starting from a position based on the terminal end of a speech segment51. The position based on the terminal end of the speech segment51may, for example, be a position included within a range spanning from the terminal end of the speech segment51to from 0 seconds to several hundred milliseconds onward. Moreover, the increased-suppression-amount segment Ts may, for example, be a period of several hundred milliseconds.

In cases in which negative determination has been made at step63, namely, in cases in which it has been determined that the audio signal x is not an audio signal x of an increased-suppression-amount segment Ts, noise in the audio signal x is suppressed at step65.

In cases in which affirmative determination has been made at step63, namely, in cases in which it has been determined that the audio signal x is an audio signal x of an increased-suppression-amount segment Ts, at step64, the CPU31increases the amount of suppression of noise performed at step65.

In Equation (4), the closer the gain G(ω) is to 1, the lower the amount of suppression (X(ω)−G(ω) X(ω)), this being the difference between the noise suppressed signal spectrum S′(ω) and the audio signal spectrum X(ω) including noise. On the other hand, the closer the value of the gain G(ω) is to 0, the greater the amount of suppression (X(ω)−G(ω) X(ω)), this being the difference between the noise suppressed signal spectrum S′(ω) and the audio signal spectrum X(ω). Accordingly, for example, as indicated by the solid line54of graph (b) ofFIG. 4, the amount of suppression is increased here for the duration of the increased-suppression-amount segment Ts by multiplying the gain G(ω) by a suppression gain α (0<α<1).

At step65, the CPU31suppresses noise in the audio signal x using the amount of suppression that was increased at step64. Namely, the noise in the audio signal x is more strongly suppressed during the increased-suppression-amount segment Ts than in other segments.

At step66, the CPU31determines whether or not processing has completed for all of the audio signal x. The CPU31returns to step61in cases in which negative determination has been made at step66. The CPU31ends the audio signal processing in cases in which affirmative determination has been made at step66.

Although explanation has been given above using noise suppression by filtering, the present exemplary embodiment is not limited thereto. For example, existing noise suppression technology such as noise suppression by spectral restoration or model-based audio noise suppression may be employed. Similar applies to the other exemplary embodiments explained below.

In the present exemplary embodiment, speech segments51of the audio signals x are detected, and noise in the audio signals x is suppressed. In the present exemplary embodiment, the amount of suppression is adjusted such that the amount of suppression during the specific period Ts, which starts from the position based on the terminal end of the detected speech segment51and is a period shorter than a period spanning from the terminal end of the detected speech segment51to the starting end of the next speech segment51, is greater than in other segments.

As explained above, in the present exemplary embodiment, noise is suppressed over the entire audio signal x, but the amount of suppression is increased in the increased-suppression-amount segments Ts rather than over the entire audio signal x. This prevents distortion from arising in the audio signal x due to the amount of suppression being increased excessively in the present exemplary embodiment. Namely, the recognition rate of the speech recognition block44at a later stage can be prevented from being decreased by distortion arising in the audio signal x.

Moreover, in the present exemplary embodiment, the recognition rate of the speech recognition block44at a later stage can be prevented from being decreased due to the amount of suppression for suppressing noise not being high over the entire audio signal x. The power of speech by a user generally decreases as the terminal end of the speech is approached. Moreover, background noise is still present in the non-speech segments52, making it difficult to recognize terminal ends of speech, these being boundaries between speech segments51and non-speech segments52.

If the amount of suppression here for suppressing noise in the audio signal x is not high, namely, is insufficient, then the difference between audio signals x in the speech segments51and audio signals x that are background noise in non-speech segments52becomes unclear due to residual noise. This makes it more difficult for the speech recognition block44to recognize terminal ends of speech, and lowers the speech recognition rate of the speech recognition block44. According to the present exemplary embodiment, the recognition rate of the speech recognition block44can be prevented from being decreased since recognition of terminal ends of speech by the speech recognition block44is facilitated by increasing the amount of suppression for the increased-suppression-amount segments Ts.

There is an issue in that excessively suppressing noise gives rise to distortion in the audio, lowering the recognition rate of speech recognition at a later stage, and insufficient suppression of noise results in speech segments, which are segments in which a user speaks, not being appropriately detected, lowering the recognition rate of speech recognition at a later stage.

In consideration of these particulars, the present disclosure enables noise to be suppressed in audio such that terminal ends of speech segments of audio can be appropriately determined.

Second Exemplary Embodiment

Next, explanation follows regarding a second embodiment, which is an example of an embodiment. Explanation regarding configuration and operation similar to that of the first exemplary embodiment is omitted.

In the present exemplary embodiment, as illustrated by the example ofFIG. 6, before detection of the speech segments at step62, at step65A, the CPU31suppresses noise in the audio signal x picked up at step61. In cases in which affirmative determination has been made at step63, namely, cases in which it has been determined that the audio signal x is an increased-suppression-amount segment Ts, at step65B, the CPU31further suppresses noise in the audio signal x by multiplying an audio signal s″ that has undergone noise suppression at step65A (also referred to as a noise suppressed signal s″ hereafter) by the suppression gain α.

Namely, in the first exemplary embodiment, noise suppression is performed on the audio signal x using gain G outside of increased-suppression-amount segments Ts, and noise suppression is performed by multiplying the suppression gain α by the gain G in the increased-suppression-amount segments Ts. In contrast thereto, in the second exemplary embodiment, first, noise suppression is performed by first using gain G across the entire audio signal x, and then the noise suppressed signal s″ is multiplied by the suppression gain α in increased-suppression-amount segments Ts. The amount of suppression of noise is increased in the increased-suppression-amount segments Ts by multiplying the noise suppressed signal s″ by the suppression gain α.

In the present exemplary embodiment, noise is suppressed in the audio signal x. In the present exemplary embodiment, the amount of suppression is adjusted such that the amount of suppression during the specific period Ts, which starts from a position based on the terminal end of the detected speech segment51and is a period shorter than a period spanning from the terminal end of the detected speech segment51to the starting end of the next speech segment51, is greater than in other segments.

Moreover, in the present exemplary embodiment, the amount of suppression of noise is adjusted such that the amount of suppression is greater during the increased-suppression-amount segments Ts than in other segments due to further suppressing the noise in the noise suppressed signal s″ during increased-suppression-amount segments Ts.

As explained above, in the present exemplary embodiment, noise is suppressed over the entire audio signal x, but the amount of suppression is increased for the increased-suppression-amount segments Ts rather than over the entire audio signal x. This prevents distortion from arising in the audio signal x due to excessively increasing the amount of suppression in the present exemplary embodiment. Namely, the recognition rate of the speech recognition block44at a later stage is prevented from being decreased by distortion arising in the audio signal x.

Moreover, in the present exemplary embodiment, the recognition rate of the speech recognition block44at a later stage is prevented from being lowered due to the amount of suppression for suppressing the noise not being high in the audio signal x. The power of speech by a user generally decreases as the terminal end of the speech is approached. Moreover, background noise is still present in non-speech segments52, making it difficult to recognize terminal ends of speech, these being boundaries between speech segments51and non-speech segments52.

If the amount of suppression here for suppressing noise in the audio signal x is not high, namely, is insufficient, then the difference between audio signals x in the speech segments51and audio signals x that are background noise in non-speech segments52becomes unclear due to residual noise. This makes it more difficult for the speech recognition block44to recognize terminal ends of speech, and lowers the speech recognition rate of the speech recognition block44. According to the present exemplary embodiment, the recognition rate of the speech recognition block44can be prevented from being decreased since recognition of terminal ends of speech by the speech recognition block44is facilitated by increasing the amount of suppression for the increased-suppression-amount segments Ts.

Third Exemplary Embodiment

Next, explanation follows regarding a third embodiment, which is an example of an embodiment. Explanation regarding configuration and operation similar to those of the first exemplary embodiment is omitted. As illustrated by the example ofFIG. 7, the third exemplary embodiment differs from the first exemplary embodiment in that, at step71, the power of the audio signal x of a non-speech segment52is integrated, and at step72, the suppression gain α is acquired based on an average value of the power of the audio signal x integrated at step71.

In more detail, at step62, in cases in which it is determined that the audio signal x is not a speech segment51, namely, cases in which it is determined that the audio signal x is a non-speech segment52, at step71, the CPU31integrates one frame worth of the power of the audio signal x, Σx(t)2, with respect to time. The audio signal x is a background noise signal in non-speech segments52.

In cases in which affirmative determination has been made at step63, namely, cases in which it has been determined that the audio signal x is an increased-suppression-amount segment Ts, at step72, the CPU31acquires the suppression gain α. For example, at step71, an average value γ of the power of the audio signal x is found by dividing the total integrated power of the non-speech segment of the audio signal x by the total amount of time, and a value α corresponding to the average value γ in the graph illustrated in the example ofFIG. 8is acquired as the suppression gain α.

In the graph illustrated in the example ofFIG. 8, the vertical axis represents the value of the suppression gain, and the horizontal axis represents the average value of the power of the audio signal x of the non-speech segment52. Note that the graph ofFIG. 8is merely an example and the present exemplary embodiment is not limited thereto.

The total power of the audio signal x integrated at step71and total time may, for example, be periodically reset. The total of the power of the audio signal x integrated by the audio signal processing performed earlier and the total amount of time may be employed, respectively, as the initial value of the power of the audio signal x integrated by the audio signal processing the current time and the total amount of time.

Although explanation has been given above regarding an example in which the processing of step71and step72is added to the processing of the first exemplary embodiment, the present exemplary embodiment may also be applied to the second exemplary embodiment. In cases in which the present exemplary embodiment is applied to the second exemplary embodiment, for example, step71may be included after step62, and step72may be included after step63.

In the present exemplary embodiment, noise in the audio signal x is suppressed. In the present exemplary embodiment, the amount of suppression is adjusted such that the amount of suppression during the specific period Ts, which starts from the position based on the terminal end of the speech segment51and is a period shorter than a period spanning from the terminal end of the speech segment51to the starting end of the next speech segment51, is greater than in other segments.

As explained above, in the present exemplary embodiment, noise is suppressed over the entire audio signal x, but the amount of suppression is increased for the increased-suppression-amount segments Ts rather than over the entire audio signal x. This prevents distortion from arising in the audio signal x due to the amount of suppression being increased excessively in the present exemplary embodiment. Namely, the recognition rate of audio by the speech recognition block44at a later stage is prevented from being decreased by distortion arising in the audio signal x.

Moreover, the recognition rate of the speech recognition block44at a later stage is prevented from being lowered due to the amount of suppression for suppressing the noise in the audio signal x not being high in the present exemplary embodiment. The power of speech by a user generally decreases as the terminal end of the speech is approached. Moreover, background noise is still present in non-speech segments52, making it difficult to recognize terminal ends of speech, these being boundaries between speech segments51and non-speech segments52.

If the amount of suppression here for suppressing noise in the audio signal x is not high, namely, is insufficient, then the difference between audio signals x in the speech segments51and audio signals x that are background noise in non-speech segments52becomes unclear due to residual noise. This makes it more difficult for the speech recognition block44to recognize terminal ends of speech, and lowers the speech recognition rate of the speech recognition block44. According to the present exemplary embodiment, the recognition rate of the speech recognition block44can be prevented from being decreased since recognition of terminal ends of speech by the speech recognition block44is facilitated by increasing the amount of suppression for the increased-suppression-amount segments Ts.

Moreover, in the present exemplary embodiment, the amount of suppression is adjusted such that the amount of suppression is increased during the increased-suppression-amount segments Ts according to an amount acquired based on the audio signal x of the non-speech segment52. In the present exemplary embodiment, the amount of suppression is adjusted such that the amount of suppression during the increased-suppression-amount segments Ts is greater than in other segments by further suppressing the noise in the noise suppressed signal s″ during the increased-suppression-amount segment Ts by the amount acquired based on the audio signal x of the non-speech segment52. This enables the amount of suppression during the increased-suppression-amount segment Ts to be appropriately adjusted in the present exemplary embodiment.

Fourth Exemplary Embodiment

Next, explanation follows regarding a fourth embodiment, which is an example of an embodiment. Explanation regarding configuration and operation similar to those of the first exemplary embodiment are omitted. As illustrated by the example ofFIG. 9, the fourth exemplary embodiment differs from the first exemplary embodiment in that a first microphone35A and a second microphone35B are included instead of the microphone35. Note that the first microphone35A and the second microphone35B may be external microphones connected to the audio signal processing device10via a microphone terminal or the like.

Next, explanation follows regarding an outline of operation of the audio signal processing device10. In the present exemplary embodiment, as illustrated by the example ofFIG. 10, in block43, the CPU31suppresses noise in an audio signal x1corresponding to the audio picked up by the first microphone35A and an audio signal x2corresponding to the audio picked up by the second microphone35B. The noise suppression may, for example, employ existing noise suppression technology such as technology in which noise suppression by filtering, noise suppression by spectral restoration, or the like for a single microphone is applied to plural microphones.

In block41, the CPU31detects speech segments51based on relative values of the audio signal x1and the audio signal x2. Moreover, the CPU31controls such that the amount of suppression of noise for the audio signal performed by block43during the increased-suppression-amount segments Ts is greater than the amount of suppression in segments other than the increased-suppression-amount segments Ts.

In more detail, at step61ofFIG. 5, the CPU31, for example, picks up one frame worth of the audio signal x1corresponding to the audio picked up by the first microphone35A and the audio signal x2corresponding to the audio picked up by the second microphone35B. At step62, the CPU31determines whether or not the picked up audio signals x1and x2are audio signals of a speech segment51.

The CPU31, for example, calculates a relative value R between the audio signal x1and the audio signal x2using Equation (5).
R=Σx1(t)x2(t−d)/(Σx1(t)2Σx2(t−d)2)1/2(5)
For example, suppose that the distance between the first microphone35A and a sound source (for example, a driver in an automobile) is further than the distance between the second microphone35B and the sound source. d is then a delay time matching the direction of the sound source. The CPU31determines a speech segment51in cases in which the relative value R is greater than a specific value.

Although explanation has been given above regarding an example in which there are two microphones, the present exemplary embodiment is not limited thereto. For example, there may be three or more microphones.

Moreover, although explanation has been given in the first exemplary embodiment regarding an example in which the microphone35is replaced by the first microphone35A and the second microphone35B, the present exemplary embodiment may also be applied to the second exemplary embodiment and the third exemplary embodiment.

In the present exemplary embodiment, noise is suppressed in the audio signal x1and the audio signal x2. In the present exemplary embodiment, the amount of suppression is adjusted such that the amount of suppression during the specific period Ts, which starts from the position based on the terminal end of the speech segment51and is a period shorter than a period spanning from the terminal end of the speech segment51to the starting end of the next speech segment51, is greater than in other segments.

As explained above, in the present exemplary embodiment, noise suppression is performed on the entirety of the audio signal x1and the audio signal x2, but the amount of suppression is increased for the increased-suppression-amount segments Ts rather than over the entirety of the audio signal x1and the audio signal x2. This prevents distortion from arising in the audio signal after noise suppression due to the amount of suppression being increased excessively in the present exemplary embodiment. Namely, the recognition rate of audio by the speech recognition block44at a later stage is prevented from being decreased by distortion arising in the audio signal after noise suppression.

Moreover, the recognition rate of the audio by the speech recognition block44at a later stage is prevented from being lowered due to the amount of suppression for suppressing the noise in the audio signal x1and the audio signal x2not being high in the present exemplary embodiment. The power of speech by a user generally decreases as the terminal end of the speech is approached. Moreover, background noise is still present in non-speech segments52, making it difficult to recognize terminal ends of speech, these being boundaries between speech segments51and non-speech segments52.

If the amount of suppression here for suppressing noise in the audio signal x1and the audio signal x2is not high, namely, is insufficient, then the difference between audio signals x1and audio signals x2of speech segments51and audio signals x1and audio signal x2that are background noise in non-speech segments52becomes unclear due to residual noise. It accordingly becomes more difficult for the speech recognition block44to recognize terminal ends of speech. This lowers the speech recognition rate of the speech recognition block44. According to the present exemplary embodiment, the recognition rate for audio by the speech recognition block44can be prevented from being decreased since recognition of terminal ends of speech by the speech recognition block44is facilitated by increasing the amount of suppression for the increased-suppression-amount segments Ts.

Comparative Results Example

Detection rates for speech segments of an audio signal with the exemplary embodiments above applied and an audio signal with the exemplary embodiments above not applied were compared using 640 items of audio data recorded inside a travelling automobile. In the audio signal with the exemplary embodiments above applied, speech segments of the audio data were appropriately detected for all 640 audio signals out of the 640 items of audio data recorded in the travelling automobile. However, in the audio signal with the exemplary embodiments above not applied, appropriate detection of speech segment of the audio signal failed for 11 items of audio data out of the 640 items of audio data above.

For example, excessively suppressing noise gives rise to distortion in the audio, lowering the recognition rate of speech recognition at a later stage, and insufficient suppression of noise results in speech segments, these being segments in which a user speaks, not being appropriately detected, lowering the recognition rate of speech recognition at a later stage.

According to the present disclosure, noise in audio can be suppressed such that terminal ends of speech segments of audio can be appropriately determined.