VOICE ACTIVITY DETECTION APPARATUS, LEARNING APPARATUS, AND STORAGE MEDIUM

According to one embodiment, a voice activity detection apparatus includes a processing circuit. The processing circuit acquires an acoustic signal and a non-acoustic signal relating to a same source, calculates an acoustic feature based on the acoustic signal, calculates a non-acoustic feature based on the non-acoustic signal, calculates a reliability weight based on a difference between the acoustic signal and the non-acoustic signal, calculates an integrated feature of the acoustic feature and the non-acoustic feature based on the reliability weight, calculates a voice existence probability based on the integrated feature, and detects a voice section and/or a non-voice section based on comparison of the voice existence probability with a threshold.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application is based upon and claims the benefit of priority from prior Japanese Patent Application No. 2022-001940, filed, Jan. 7, 2022, the entire contents of which are incorporated herein by reference.

FIELD

Embodiments described herein relate generally to a voice activity detection apparatus, a learning apparatus, and a storage medium.

BACKGROUND

Voice activity detection (VAD) is a technique of detecting a voice section including the user's voice from an input signal. Voice activity detection is mainly used for improving recognition accuracy of voice recognition, and/or used for supporting data compression in a non-voice section in the field of voice encoding.

Voice activity detection requires processing of detecting a voice section including predetermined voice from a time section of an input signal. For example, for determining whether the processing target frame is a voice (for example, speech) section or not, the voice section is detected from the input acoustic signal using a model trained in advance.

To further enhance processing accuracy of voice activation detection in an environment in which processing is difficult, such as a noise environment, a voice activity detection apparatus has been presented. The voice activity detection apparatus detects a voice section using both an acoustic signal and a non-acoustic signal, such as a lip image signal, as an input signal. For example, a technique disclosed in Patent Literature 1 (Japanese Patent Application Publication No. 2011-191423) calculates an utterance probability and detecting a voice section, based on a Bayesian network using a feature quantity of acoustic information extracted from an acoustic voice collection apparatus and a visual feature quantity relating to the lengths in the horizontal direction and the vertical direction of the lips in a lip area extracted from an image pickup unit.

DETAILED DESCRIPTION

According to one embodiment, a voice activity detection apparatus includes a processing circuit. The processing circuit acquires an acoustic signal and a non-acoustic signal relating to a same voice generation source. The processing circuit calculates an acoustic feature based on the acoustic signal. The processing circuit calculates a non-acoustic feature based on the non-acoustic signal. The processing circuit calculates a reliability weight based on a difference between the acoustic signal and the non-acoustic signal. The processing circuit calculates an integrated feature of the acoustic feature and the non-acoustic feature based on the reliability weight. The processing circuit calculates a voice existence probability based on the integrated feature. The processing circuit detects a voice section and/or a non-voice section based on comparison of the voice existence probability with a threshold. The voice section is a time section in which voice is presence. The non-voice section is a time section in which voice is absence.

The technique of Patent Literature 1 is based on the premise that a normal lip image signal is an input signal, and has the problem that accuracy of voice activity detection largely decreases due to abnormal image input caused by, for example, the speaker's wearing a mask, the speaker's face turning sideways, failure in estimation of a mouth region, and/or an error in reading moving images in actual processing.

Specifically, the technique disclosed in Patent Literature 1 is calculation of the conditional probability using the acoustic feature quantity and the visual feature quantity with respect to presence/absence of voice, and the detection accuracy thereof decreases due to the abnormal visual feature quantity calculated from the abnormal image input.

The problem to be solved by the present embodiment is to provide a voice activity detection apparatus, a learning apparatus, and a storage medium capable of reducing decrease in detection accuracy caused by abnormal image input.

The following is an explanation of a voice activity detection apparatus, a learning apparatus, and a storage medium according to the present embodiment with reference to drawings.

Voice Activity Detection Apparatus

FIG.1is a diagram illustrating a configuration example of a voice activity detection apparatus100. The voice activity detection apparatus100is a computer detecting a voice section of an input signal. As illustrated inFIG.1, the voice activity detection apparatus100includes a processing circuit11, a storage device12, an input device13, a communication device14, a display device15, and an acoustic device16.

The processing circuit11includes a processor, such as a CPU (Central Processing Unit), and a memory, such as a RAM (Random Access Memory). The processing circuit11executes voice activity detection processing of detecting a voice section of the input signal, by executing a voice activity detection program stored in the storage device12. The voice activity detection program is recorded on a non-transitory computer-readable storage medium. The processing circuit11achieves an input signal acquisition unit111, an acoustic feature calculation unit112, a non-acoustic feature calculation unit113, a reliability weight calculation unit114, an integrated feature calculation unit115, a voice existence probability calculation unit116, a voice section detection unit117, and an output control unit118, by reading and executing the voice activity detection program from the storage medium. The voice activity detection program may include a plurality of modules implemented with functions of the units111to118in a divided manner.

Hardware implementation of the processing circuit11is not limited to only the mode described above. For example, the processing circuit11may be formed of a circuit, such as an application specific integrated circuit (ASIC), achieving the input signal acquisition unit111, the acoustic feature calculation unit112, the non-acoustic feature calculation unit113, the reliability weight calculation unit114, the integrated feature calculation unit115, the voice existence probability calculation unit116, the voice section detection unit117, and/or the output control unit118. The input signal acquisition unit111, the acoustic feature calculation unit112, the non-acoustic feature calculation unit113, the reliability weight calculation unit114, the integrated feature calculation unit115, the voice existence probability calculation unit116, the voice section detection unit117, and/or the output control unit118may be implemented in a single integrated circuit or individually implemented in a plurality of integrated circuits.

The input signal acquisition unit111acquires an acoustic signal and a non-acoustic signal relating to the same voice generation source. The acoustic signal and the non-acoustic signal are time-series signals, and are temporally synchronized in units of frame. The acoustic signal is a signal relating to voice uttered by a speaker serving as the voice generation source. The non-acoustic signal is a signal other than the acoustic signal relating to the speaker and collected substantially simultaneously with the acoustic signal. For example, the non-acoustic signal is an image signal relating to the uttering speaker, and/or a sensor signal relating to physiological response of lips and/or a face muscle and brain waves of the speaker generated by utterance.

The acoustic feature calculation unit112calculates a feature quantity (hereinafter referred to as “acoustic feature”) of the acoustic signal. The acoustic feature has a value based on the acoustic signal and a value correlated with voice uttered by the speaker. The acoustic feature is calculated for each of frames. As an example, the acoustic feature is calculated using a first trained model. The first trained model is a neural network trained to receive an acoustic signal and output an acoustic feature. The first trained model is stored in the storage device12or the like.

The non-acoustic feature calculation unit113calculates a feature quantity (hereinafter referred to as “non-acoustic feature”) of the non-acoustic signal. The non-acoustic feature has a value based on the non-acoustic signal and a feature value correlated with voice uttered by the speaker. The non-acoustic feature is calculated for each of frames. As an example, the non-acoustic feature is calculated using a second trained model. The second trained model is a neural network trained to receive a non-acoustic signal and output a non-acoustic feature. The second trained model is stored in the storage device12or the like.

Each of the first trained model and the second trained model is a neural network trained to provide a penalty for a difference between the acoustic feature and the non-acoustic feature relating to the same voice generation source.

The reliability weight calculation unit114calculates a reliability weight on the basis of a difference between the acoustic feature and the non-acoustic feature. The reliability weight is calculated for each of frames as a determination value to determine input of an abnormal non-acoustic signal. As an example, the reliability weight include a first reliability weight calculated as a value between an intermediate value and an upper limit value for the difference between the acoustic feature and the non-acoustic feature, and a second reliability weight calculated as a value acquired by subtracting the first reliability weight from the upper limit value. For example, in the case where each of the first reliability weight and the second reliability weight has a numerical range of 0 to 1, the intermediate value is preferably set to 0.5, and the upper limit value is preferably set to 1.

The integrated feature calculation unit115calculates an integrated feature of the acoustic feature and the non-acoustic feature on the basis of the reliability weight calculated with the reliability weight calculation unit114. The integrated feature is calculated for each of frames. As an example, the integrated feature calculation unit115calculates an integrated feature of the acoustic feature corrected with the first reliability weight and the non-acoustic feature corrected with the second reliability weight.

The voice existence probability calculation unit116calculates a voice existence probability on the basis of the integrated feature. The voice existence probability is used as a scale to distinguish a voice section from a non-voice section. The voice section is a time section in which voice is uttered in the time sections of the input signal, and a non-voice section is a time section in which no voice is uttered in the time sections of the input signal. The voice existence probability is calculated for each of frames. As an example, the voice existence probability is calculated using a third trained model. The third trained model is a neural network trained to receive an integrated feature and output a voice existence probability. The neural network is a neural network trained to provide a penalty for a difference between a correct label relating to the voice section and the non-voice section and the voice existence probability. The third trained model is stored in the storage device12or the like.

The voice section detection unit117detects a voice section and/or a non-voice section based on a comparison of the voice existence probability with a threshold. The voice section is a time section in which voice is present. The non-voice section is a time section in which voice is absent.

The output control unit118displays various types of information via the display device15and/or the acoustic device16. For example, the output control unit118displays an image signal on the display device15and/or outputs an acoustic signal via the acoustic device16.

The storage device12is formed of, for example, a ROM (Read Only Memory), a HDD (Hard Disk Drive), a SSD (Solid State Drive), and/or an integrated circuit storage device. The storage device12stores therein various arithmetic calculation results acquired with the processing circuit11and/or the voice section detection program executed with the processing circuit11and the like. The storage device12is an example of a computer-readable storage medium.

The input device13inputs various commands from the user. Applicable examples of the input device13include a keyboard, a mouse, various types of switches, a touch pad, and a touch panel display. The output signal from the input device13is supplied to the processing circuit11. The input device13may be a computer connected with the processing circuit11in a wired or wireless manner.

The communication device14is an interface to execute information communication with an external device connected with the voice activity detection apparatus100via a network. The communication device14receives an acoustic signal and a non-acoustic signal from, for example, a device collecting the acoustic signal and the non-acoustic signal, and/or receives the first trained model, the second trained model, and the third trained model from a learning apparatus described later.

The display device15displays various types of information. Applicable examples of the display device15include a CRT (Cathode-Ray Tube) display, a liquid crystal display, an organic EL (Electro Luminescence) display, a LED (Light-Emitting Diode) display, a plasma display, and any other displays known in the technical field. The display device15may be a projector.

The acoustic device16converts an electrical signal into voice and emits the voice. Applicable examples of the acoustic device16include a magnetic loudspeaker, a dynamic loudspeaker, a capacitor loudspeaker, and any other loudspeakers known in the technical field.

The following is an explanation of an example of voice activity detection processing executed with the processing circuit11of the voice activity detection apparatus100. To specifically execute the following explanation, suppose that the non-acoustic signal is an image signal.

FIG.2is a diagram illustrating an example of flow of voice activity detection processing executed with the processing circuit11.FIG.3is a diagram schematically illustrating the voice activity detection processing. The voice activity detection processing is executed with an operation of the processing circuit11in accordance with the voice activity detection program stored in the storage device12or the like.

As illustrated inFIG.2andFIG.3, the input signal acquisition unit111acquires an input signal including an acoustic signal and an image signal (Step SA1). The input signal is a video signal including an acoustic signal and an image signal relating to the same voice generation source.

FIG.4is a diagram illustrating an example of an input signal (video signal), an acoustic signal, and an image signal. As illustrated inFIG.4, a video signal is a time-series signal including a time-series acoustic signal and a time-series image signal temporally synchronized with each other. The length of the time sections of the video signal is not particularly limited, but is supposed to be a frame length of 10 seconds or around.

The video signal is collected with a video camera device including a microphone and an imaging device. The acoustic signal is collected with the microphone. The microphone collects voice relating to utterance of the speaker, converts the sound pressure of the collected voice into an analog electrical signal (acoustic signal), and subjects the acoustic signal to A/D conversion to convert the acoustic signal into a digital time region electrical signal (acoustic signal). The time region acoustic signal is acquired with the input signal acquisition unit111, and converted into a frequency region acoustic signal by short-time Fourier transform or the like. The image signal is collected almost simultaneously with the acoustic signal. The image signal is collected with the imaging device including a plurality of imaging elements, such as a CCD (Charge Coupled Device). The imaging device optically images the uttering speaker, and generates an image signal (image data) of a digital spatial region image signal (image data) relating to the speaker in units of frame. The image signal is required to be correlated with the speaker's utterance. As the imaging target, the image frame is required to include at least a lip area the form of which is changed in accordance with utterance. The image frame may include the whole face region of the speaker. The image signal is acquired in units of frame with the input signal acquisition unit111.

In this example, suppose that the time-series acoustic signal A and the time-series image signal V are defined in accordance with the following expression (1). The time-series acoustic signal A is an acoustic signal including a T dimension of the time region and a F dimension of the frequency region of the frame serving as the processing target. The image signal V is an image signal including dimensions of the time T, the height H, the width W, and a color channel C.

When Step SA1is executed, the acoustic feature calculation unit112calculates an acoustic feature EAfrom the acoustic signal A acquired at Step SA1using the first trained model (Step SA2). The acoustic feature EAis calculated on the basis of the acoustic signal A for each of the frames. The acoustic feature EAis time-series data. The first trained model is a neural network trained to receive the acoustic signal A and output the acoustic feature EA. For example, an encoder network trained to convert the acoustic signal A into the acoustic feature EAis used as the neural network. The first trained model is generated with a learning apparatus described later.

The relation between the acoustic signal and the acoustic feature herein is as follows. The acoustic signal is time-series waveform data of sound pressure values of the voice uttered by the speaker. The acoustic signal is correlated with the voice uttered by the speaker. For example, the peak value of the acoustic signal has a relatively high value when the speaker utters, and has a relatively low value when the speaker does not utter. The acoustic feature is designed such that the value has correlation with the peak value of the acoustic signal, in other words, to distinguish a voice component and a silent component included in the acoustic signal. For example, the value of the acoustic feature increases as the peak value of the acoustic signal increases, and the value of the acoustic feature decreases as the peak value of the acoustic signal decreases.

When Step SA2is executed, the non-acoustic feature calculation unit113calculates an image feature Evfrom the image signal V acquired at Step SA1using the second trained model (Step SA3). The image feature EVis calculated on the basis of the image signal V for each of frames. Specifically, the image feature EVis time-series data. The second trained model is a neural network trained to receive the image signal V and output the image feature EV. For example, an encoder network trained to convert the image signal V into the image feature EVis used as the neural network. The second trained model is generated with a learning apparatus described later.

The relation between the image signal and the image feature is as follows. The image signal is correlated with the form of the face part region at the time when the speaker is uttering voice. The image feature is designed to distinguish an utterance component from a non-utterance component included in the image signal. Specifically, the lip area of the speaker indicated with the image signal has different forms between the time when the speaker utters voice and the time when the speaker does not utter sound. The image feature is designed such that the value thereof is correlated with the form of the face part region of the speaker. For example, the image feature has a higher value as the speaker opens one's mouth wider, and the image feature has a lower value as the speaker closes one's mouth.

The order of Step SA2and Step SA3is not particularly limited. Step SA2may be executed after Step SA3, or Step SA2and Step SA3may be executed in parallel.

When Step SA3is executed, the reliability weight calculation unit114calculates a reliability weight on the basis of a difference between the acoustic feature calculated at Step SA2and the image feature calculated at Step SA3(Step SA4). At Step SA4, first, the reliability weight calculation unit114calculates a first reliability weight ΛAcalculated as a value between an intermediate value (for example, 0.5) and an upper limit value (for example, 1) for the difference between the acoustic feature EAand the non-acoustic feature EV, and a second reliability weight ΛVcalculated as a value acquired by subtracting the first reliability weight from the upper limit value. The first reliability weight ΛAand the second reliability weight ΛVare expressed with the following expressions (2) and (3). Specifically, the first reliability weight ΛAis based on Kullback-Leibler divergence serving as a scale to measure a difference between the acoustic feature EAand the non-acoustic feature EV. The first reliability weight ΛAis a scale acquired by non-linearly converting the Kullback-Leibler divergence based on the acoustic feature EAand the image feature EVin a range from the intermediate value (for example, 0.5) to the upper limit value (for example, 1). As expressed with the expression (3), the second reliability weight ΛVis a scale having a value acquired by subtracting the first reliability weight ΛAfrom the upper limit value.

In the expression, EA(t, d)is an acoustic feature vector at the frame time t∈{1, 2, . . . , T} and the coordinates d∈{1, 2, . . . , D} compressed to D dimension, and serves as an example of the acoustic feature. EV(t, d)is an image feature vector at the frame time t∈{1, 2, . . . , T} and coordinates d∈{1, 2, . . . , D} compressed to the D dimension, and an example of the image feature. The sign δ indicates a scale factor enlarging and/or reducing the magnitude of the difference between the acoustic feature EAand the image feature EV. Each of the first reliability weight ΛAand the second reliability weight ΛVis a weight depending on the frame time t.

As described later, the first trained model and the second trained model are trained to reduce a difference between the acoustic feature EAand the image feature EVwith respect to the normal input for the same voice generation source. Because of use of the first trained model and the second trained model trained like this, in the case where input of an abnormal image occurs, the difference between the acoustic feature EAand the image feature EVincreases. In this case, the first reliability weight ΛAhas a value close to the upper limit value, and the second reliability weight ΛVhas a value close to the lower limit value on the basis of the expressions (2) and (3).

When Step SA4is executed, the integrated feature calculation unit115calculates an integrated feature of the acoustic feature and the non-acoustic feature on the basis of the reliability weight calculated at Step SA4(Step SA5). At Step S5, the integrated feature calculation unit115calculates an integrated feature of the acoustic feature EAand the non-acoustic feature EVusing the first reliability weight ΛAand the second reliability weight ΛV. The integrated feature ZAVis calculated for each frame time t. Specifically, the integrated feature ZAVis calculated as a sum of an integrated value of the first reliability weight ΛAand the acoustic feature EAand the integrated value of the second reliability weight ΛVand the image feature EV, as expressed with the following expression (4).

When Step SA5is executed, the voice existence probability calculation unit116calculates a voice existence probability Y using the third trained model from the integrated feature ZAVacquired at Step SA5(Step SA6). The voice existence probability Y indicates a probability of existence of voice in the frame. The voice existence probability Y is calculated on the basis of the integrated feature ZAVfor each of the frames. The third trained model is a neural network trained to receive the integrated feature ZAVand provide a penalty for a difference between a correct label relating to the voice section and the voice existence probability. For example, an estimation network trained to estimate a voice existence probability Y from the integrated feature ZAVis used as the neural network. The third trained model is generated with a learning apparatus described later.

When Step SA6is executed, the voice section detection unit117detects a voice section on the basis of comparison of the voice existence probability Y calculated at Step SA6with a threshold n (Step SA7).

The value of the voice existence probability Y is compared with a threshold n for each frame time. The threshold n is set to a boundary between the value corresponding to utterance and the value not corresponding to utterance. For example, when the voice existence probability Y has the value ranging from “0” to “1”, the threshold q is set to the value “0.5”. In the case where the value of the voice existence probability Y is larger than the threshold n, the frame time is determined as a voice section. In the case where the value of the voice existence probability Y is smaller than the threshold n, the frame time is determined as a non-voice section. By executing the determination processing for each frame time, the voice sections and the non-voice sections are detected in the time sections corresponding to the input signal. A label of a voice section or a non-voice section is assigned to each frame time of the input signal.

When Step SA7is executed, the output control unit118outputs the voice sections and/or the non-voice sections detected at Step SA7(Step SA8). Various forms are possible as the output form. As an example, at Step SA8, the output control unit118displays the voice sections and/or the non-voice sections on the display device15. In this operation, the output control unit18preferably displays the voice sections and/or the non-voice sections in visually association with the acoustic signal and/or the image signal.

The voice activity detection processing with the processing circuit11is finished with the operation described above. The input signal after voice activity detection is subjected to processing, such as voice recognition and data compression.

As described above, the voice activity detection apparatus100according to the present embodiment includes the acoustic feature calculation unit112, the non-acoustic feature calculation unit113, the reliability weight calculation unit114, the integrated feature calculation unit115, the voice existence probability calculation unit116, and the voice section detection unit117. The acoustic feature calculation unit112calculates an acoustic feature on the basis of the acoustic signal. The acoustic feature has a value correlated with pronunciation. The non-acoustic feature calculation unit113calculates an acoustic feature on the basis of the non-acoustic signal. The non-acoustic feature has a value correlated with pronunciation. The reliability weight calculation unit114calculates a reliability weight on the basis of the acoustic feature and the non-acoustic feature. The integrated feature calculation unit115calculates an integrated feature of the acoustic feature and the non-acoustic feature on the basis of the reliability weight. The voice existence probability calculation unit116calculates a voice existence probability from the integrated feature. The voice section detection unit117detects a voice section serving as a time section in which voice is uttered and/or a non-voice section serving as a time section in which no voice is uttered, on the basis of comparison of the voice existence probability with the threshold.

According to the present embodiment, when a voice existence probability is calculated from the integrated feature of the acoustic feature based on the acoustic signal and the non-acoustic feature based on the non-acoustic signal, a reliability weight is calculated for each of the acoustic feature and the non-acoustic feature, and the integrated feature and the voice existence probability are calculated in consideration of the reliability weights. This structure enables detection of a voice section with high accuracy even in the case where a continuous or sudden abnormal non-acoustic signal occurs. In addition, preferably, the acoustic feature is calculated from the acoustic signal using the first trained model, and the non-acoustic feature is calculated from the non-acoustic signal using the second trained model. Each of the first trained model and the second trained model is generated by learning using a loss function defined to provide a penalty for a difference between the acoustic feature and the non-acoustic feature output for the same voice generation source. By acquiring the acoustic feature and the non-acoustic feature using the first trained model and the second trained model acquired by such learning, a difference between the acoustic feature and the non-acoustic feature decreases for input of a normal non-acoustic signal, and a difference between the acoustic feature and the non-acoustic feature increases for input of an abnormal non-acoustic signal. By using such a difference value between the acoustic feature and the non-acoustic feature as a reliability weight and acquiring an integrated feature of the weighted acoustic feature and the weighted non-acoustic feature, the accuracy of detection of the voice section and/or the non-voice section can be further improved even if input of an abnormal non-acoustic signal occurs.

Learning Apparatus

FIG.5is a diagram illustrating a configuration example of a learning apparatus200. The learning apparatus200is a computer generating a first trained model used for calculation of the acoustic feature, a second trained model used for calculation of the image feature, and a third trained model used for detection of a voice section and/or a non-voice section. As illustrated inFIG.5, the learning apparatus200includes a processing circuit21, a storage device22, an input device23, a communication device24, a display device25, and an acoustic device26.

The processing circuit21includes a processor, such as a CPU, and a memory, such as a RAM. The processing circuit21executes learning processing to generate the first trained model, the second trained model, and the third trained model, by executing a learning program stored in the storage device22. The learning program is recorded on a non-transitory computer-readable storage medium. The processing circuit21achieves an input signal acquisition unit211, an acoustic feature calculation unit212, a non-acoustic feature calculation unit213, a reliability weight calculation unit214, an integrated feature calculation unit215, a voice existence probability calculation unit216, an update unit217, an update completion determination unit218, and an output control unit219, by reading and executing the learning program from the storage medium. The learning program may include a plurality of modules implemented with functions of the units211to219in a divided manner.

Hardware implementation of the processing circuit21is not limited to only the mode described above. For example, the processing circuit21may be formed of a circuit, such as an application specific integrated circuit (ASIC), achieving the input signal acquisition unit211, the acoustic feature calculation unit212, the non-acoustic feature calculation unit213, the reliability weight calculation unit214, the integrated feature calculation unit215, the voice existence probability calculation unit216, the update unit217, the update completion determination unit218, and the output control unit219. The input signal acquisition unit211, the acoustic feature calculation unit212, the non-acoustic feature calculation unit213, the reliability weight calculation unit214, the integrated feature calculation unit215, the voice existence probability calculation unit216, the update unit217, the update completion determination unit218, and the output control unit219may be implemented in a single integrated circuit or individually implemented in a plurality of integrated circuits.

The input signal acquisition unit211acquires training data including a plurality of training samples. Each of the training samples is an input signal including a pair of an acoustic signal and a non-acoustic signal. The input signal is a time-series signal, and includes a time-series acoustic signal and a time-series non-acoustic signal. As described above, the non-acoustic signal is an image signal relating to the uttering speaker, and/or a sensor signal relating to physiological response of lips and/or a face muscle of the speaker generated by utterance.

The acoustic feature calculation unit212calculates an acoustic feature from the acoustic signal using a first neural network. The acoustic feature calculated with the acoustic feature calculation unit212is similar to the acoustic feature calculated with the acoustic feature calculation unit112. The first trained model is generated by training the first neural network.

The non-acoustic feature calculation unit213calculates a non-acoustic feature from the non-acoustic signal using a second neural network. The non-acoustic feature calculated with the non-acoustic feature calculation unit213is similar to the non-acoustic feature calculated with the non-acoustic feature calculation unit113. The second trained model is generated by training the second neural network.

The first neural network and the second neural network are trained to provide a penalty for a difference between the acoustic feature and the non-acoustic feature relating to the same voice generation source.

The reliability weight calculation unit214calculates a reliability weight on the basis of a difference between the acoustic feature and the non-acoustic feature. The reliability weight calculated with the reliability weight calculation unit214is similar to the reliability weight calculated with the reliability weight calculation unit114.

The integrated feature calculation unit215calculates an integrated feature of the acoustic feature and the non-acoustic feature on the basis of the reliability weight calculated with the reliability weight calculation unit214. The integrated feature calculated with the integrated feature calculation unit215is similar to the integrated feature calculated with the integrated feature calculation unit115.

The voice existence probability calculation unit216calculates a voice existence probability from the integrated feature using a third neural network. The voice existence probability calculated with the voice existence probability calculation unit216is similar to the voice existence probability calculated with the voice existence probability calculation unit116. The third neural network is trained to provide a penalty for a difference between a correct label relating to the voice section and the non-voice section and the voice existence probability. The third trained model is generated by training the third neural network.

The update unit217updates the first neural network, the second neural network, and the third neural network using a total loss function including a first loss function to provide a penalty for a difference between the acoustic feature and the non-acoustic feature relating to the same voice generation source and a second loss function to provide a penalty for a difference between a correct label relating to the voice section and the non-voice section and the voice existence probability.

The update completion determination unit218determines whether the condition for stopping the learning processing is satisfied. In the case where it is determined that the stop condition is not satisfied, the processing circuit21repeats calculation of the acoustic feature with the acoustic feature calculation unit212, calculation of the non-acoustic feature with the non-acoustic feature calculation unit213, calculation of the reliability weight with the reliability weight calculation unit214, calculation of the integrated feature with the integrated feature calculation unit215, calculation of the voice existence probability with the voice existence probability calculation unit216, and update of the first neural network, the second neural network, and the third neural network with the update unit217. In the case where it is determined that the stop condition is satisfied, the first neural network at this point in time is output as the first trained model, the second neural network at this point in time is output as the second trained model, and the third neural network at this point in time is output as the third trained model.

The output control unit219displays various types of information via the display device25and/or the acoustic device26. For example, the output control unit219displays an image signal on the display device25and/or outputs an acoustic signal via the acoustic device26.

The storage device22is formed of, for example, a ROM, a HDD, a SSD, and/or an integrated circuit storage device. The storage device22stores therein various arithmetic calculation results acquired with the processing circuit21and/or the learning program executed with the processing circuit21and the like. The storage device22is an example of a computer-readable storage medium.

The input device23inputs various commands from the user. Applicable examples of the input device23include a keyboard, a mouse, various types of switches, a touch pad, and a touch panel display. The output signal from the input device23is supplied to the processing circuit21. The input device23may be a computer connected with the processing circuit21in a wired or wireless manner.

The communication device24is an interface to execute information communication with an external device connected with the learning apparatus200via a network.

The display device25displays various types of information. Applicable examples of the display device25include a CRT display, a liquid crystal display, an organic EL display, a LED display, a plasma display, and any other displays known in the technical field. The display device25may be a projector.

The acoustic device26converts an electrical signal into voice and emits the voice. Applicable examples of the acoustic device26include a magnetic loudspeaker, a dynamic loudspeaker, a capacitor loudspeaker, and any other loudspeakers known in the technical field.

The following is an explanation of an example of learning processing executed with the processing circuit21of the learning apparatus200. To specifically execute the following explanation, suppose that the non-acoustic signal is an image signal.

FIG.6is a diagram illustrating an example of flow of learning processing executed with the processing circuit21. The learning processing is executed by operation of the processing circuit21in accordance with the learning program stored in the storage device22or the like. In the learning processing, the processing circuit21trains a first neural network NN1, a second neural network NN2, and a third neural network NN3in parallel. More specifically, the processing circuit21subjects an integrated neural network including the first neural network NN1, the second neural network NN2, and the third neural network NN3to supervised learning using training data.

FIG.7is a diagram illustrating a configuration example of an integrated neural network. As illustrated inFIG.7, the integrated neural network includes the first neural network NN1, the second neural network NN2, a reliability weight calculation module NM1, an integrated feature calculation module NM2, and the third neural network NN3. The first neural network NN1receives an acoustic signal A and outputs an acoustic feature EA. The second neural network NN2receives an image signal V and outputs an image feature EV. The reliability weight calculation module NM1receives the acoustic feature EAoutput from the first neural network NN1and the non-acoustic feature EVoutput from the second neural network NN2, and outputs a reliability weight A, that is, a first reliability weight ΛAand a second reliability weight ΛV. The reliability weight calculation module NM1is a module corresponding to the function of the reliability weight calculation unit214. The integrated feature calculation module NM2receives the acoustic feature EAoutput from the first neural network NN1, the non-acoustic feature EVoutput from the second neural network NN2, and the first reliability weight AA and the second reliability weight ΛVoutput from the reliability weight calculation module NM1, and outputs an integrated feature ZAV. The integrated feature calculation module NM2is a module corresponding to the function of the integrated feature calculation unit215. The third neural network NN3receives the integrated feature ZAVoutput from the integrated feature calculation module NM2, and outputs a voice existence probability Y{circumflex over ( )}. Suppose that initial values of learning parameters or the like are assigned to the first neural network NN1, the second neural network NN2, and the third neural network NN3. The learning parameters are weights and/or biases and the like. Desired hyperparameters may be included as the learning parameters.

The first neural network NN1has architecture of an encoder network capable of calculating the acoustic feature EAfrom the acoustic signal A. As an example, the first neural network NN1includes three one-dimensional convolutional layers and a L2 normalization layer. The second neural network NN2has architecture of an encoder network capable of calculating the image feature EVfrom the image signal V. As an example, the second neural network NN2includes three three-dimensional convolutional layers and a L2 normalization layer. At least one of the three three-dimensional convolutional layers is preferably connected to a maximum value pooling layer and/or a global mean value pooling layer. The third neural network NN3has architecture of a detection network capable of calculating a voice existence probability Y{circumflex over ( )} from the integrated feature ZAV. As an example, the third neural network NN3includes three dense (Dense) layers. Each of all the convolutional layers and the Dense layers of the first neural network NN1, the second neural network NN2, and the third neural network NN3are followed by a normalization linear function unit, and a normalization linear function is applied to the output of the layer. The final layer of each of the first neural network NN1, the second neural network NN2, and the third neural network NN3is followed by a sigmoid activation function unit, not a normalization linear function unit, and a sigmoid function is applied to the output of the final layer.

The integrated neural network receives the acoustic signal A and the image signal V, and outputs a voice existence probability Y{circumflex over ( )}. The learning parameters of the first neural network NN1, the second neural network NN2, and the third neural network NN3are trained to minimize the total loss function evaluating an error between the voice existence probability Y{circumflex over ( )} and the correct label (ground truth) Y. The following is an explanation of learning of the first neural network NN1, the second neural network NN2, and the third neural network NN3with reference toFIG.6andFIG.7.

As illustrated inFIG.6andFIG.7, the input signal acquisition unit211acquires an input signal including an acoustic signal A and an image signal V (Step SB1). At Step SB1, an input signal serving as a training sample is acquired. The frame length of the time sections of the input signal is not particularly limited, but is supposed to be, for example, 10 frames or around. The acoustic signal A and the image signal V included in the input signal are temporally synchronized with each other.

When Step SB1is executed, the acoustic feature calculation unit212calculates an acoustic feature EAfrom the acoustic signal A acquired at Step SB1using the first neural network (Step SB2). Learning of the first neural network at Step SB2is not finished. The acoustic signal A input to the first neural network NN1has been converted from the time region into the frequency region.

When Step SB2is executed, the non-acoustic feature calculation unit213calculates an image feature EVfrom the image signal V acquired at Step SB1using the second neural network (Step SB3). Learning of the second neural network at Step SB3is not finished.

The order of Step SB2and Step SB3is not particularly limited. Step SB2may be executed after Step SB3, or Step SB2and Step SB3may be executed in parallel.

When Step SB3is executed, the reliability weight calculation unit214calculates a reliability weight using the reliability weight calculation module NM1on the basis of a difference between the acoustic feature EAcalculated at Step SB2and the image feature EVcalculated at Step SB3(Step SB4). At Step SB4, first, the reliability weight calculation unit214calculates a first reliability weight AA calculated as a value between an intermediate value (for example, 0.5) and an upper limit value (for example, 1) for the difference between the acoustic feature EAand the non-acoustic feature EV, and a second reliability weight ΛVcalculated as a value acquired by subtracting the first reliability weight from the upper limit value.

When Step SB4is executed, the integrated feature calculation unit215calculates an integrated feature ZAVof the acoustic feature EAand the image feature EVusing the integrated feature calculation module NM2on the basis of the reliability weights ΛAand ΛVcalculated at Step SB4(Step SB5). At Step SB5, the integrated feature calculation unit215calculates the integrated feature of the acoustic feature EAand the image feature EVusing the first reliability weight ΛAand the second reliability weight ΛV. The integrated feature ZAVis calculated for each frame time t. In the case where the image signal includes no abnormal images but includes only normal images at the learning stage, the first reliability weight ΛAand the second reliability weight ΛVmay be set to the intermediate value 0.5.

When Step SB5is executed, the voice existence probability calculation unit216calculates the voice existence probability Y{circumflex over ( )} using the third neural network NN3from the integrated feature ZAVacquired at Step SB5(Step SB6).

When Step SB6is executed, the update unit217updates the first neural network NN1, the second neural network NN2, and the third neural network NN3using a total loss function LTOTALincluding a first loss function LKLDto provide a penalty for a difference between the acoustic feature and the non-acoustic feature relating to the same voice generation source and a second loss function LBCEto provide a penalty for a difference between a correct label relating to the voice section and the non-voice section and the voice existence probability (Step SB7). As expressed with the following expression (5), the total loss function LTOTALat the frame time t and the coordinates d is provided by a sum of the first loss function LKLDand the second loss function LBCE. Specifically, the first loss function LKLDis provided by the Kullback-Leibler divergence based on the acoustic feature EAand the image feature EV, as expressed with the following expression (6). The Kullback-Leibler divergence is used as a scale to evaluate a difference between the acoustic feature EAand the image feature EV. As expressed with the following expression (7), the second loss function LBCSis provided by binary cross entropy (BCE). Binary cross entropy is used as a scale to evaluate a difference between a correct label y relating to the voice section and the non-voice section and the voice existence probability y{circumflex over ( )}.

The update unit217updates learning parameters of the first neural network NN1, the second neural network NN2, and the third neural network NN3to minimize the total loss function LTOTAL, in accordance with a desired optimization method. In this manner, the update unit217updates the learning parameters of the first neural network NN1, the second neural network NN2, and the third neural network NN3to minimize a difference between the acoustic feature EAand the image feature EVand a difference between the correct label y and the voice existence probability y{circumflex over ( )} comprehensively. Any method, such as a stochastic gradient descent and adaptive moment estimation (Adam), may be used as the optimization method.

When Step SB7is executed, the update completion determination unit218determines whether the stop condition is satisfied (Step SB8). The stop condition may be, for example, a condition that the number of updates of the learning parameters has reached a predetermined number of times and/or a condition that the update quantity of the learning parameters is less than the threshold. When it is determined that the stop condition is not satisfied (Step SB8: NO), the input signal acquisition unit211acquires another acoustic signal and another image signal (Step SB1). Thereafter, the acoustic signal and the image signal is successively subjected to calculation of the acoustic feature with the acoustic feature calculation unit212(Step SB2), calculation of the non-acoustic feature with the non-acoustic feature calculation unit213(Step SB3), calculation of the reliability weight with the reliability weight calculation unit214(Step SB4), calculation of the integrated feature with the integrated feature calculation unit215(Step SB5), calculation of the voice existence probability with the voice existence probability calculation unit216(Step SB6), update of the first neural network, the second neural network, and the third neural network with the update unit217(Step SB7), and determination with the update completion determination unit218as to whether the stop condition is satisfied (Step SB8).

Steps SB2to SB7may be repeated for a training sample (batch learning), or Steps SB2to SB7may be repeated for a plurality of training samples (mini-batch learning).

When it is determined at Step SB8that the stop condition is satisfied (Step SB8: YES), the update completion determination unit218outputs the first neural network NN1at the time when the stop condition is satisfied as the first trained model, the second neural network NN2at the time when the stop condition is satisfied as the second trained model, and the third neural network NN3at the time when the stop condition is satisfied as the third trained model (Step SB9). The first trained model, the second trained model, and the third trained model are transmitted to the voice activity detection apparatus100via the communication device24or the like, and stored in the storage device12. The update completion determination unit218may output an integrated neural network including the first trained model, the second trained model, the third trained model, the reliability weight calculation module NM1, and the integrated feature calculation module NM2.

When the step SB8is executed, learning processing with the processing circuit21is finished. As described above, the voice activity detection apparatus100calculates a voice existence probability from the acoustic signal and the image signal using the first trained model, the second trained model, and the third trained model generated with the learning apparatus200. The voice activity detection apparatus100may calculate a voice existence probability by inputting the acoustic signal and the image signal to the integrated neural network.

As described above, the learning apparatus200according to the present embodiment includes the input signal acquisition unit211, the acoustic feature calculation unit212, the non-acoustic feature calculation unit213, the reliability weight calculation unit214, the integrated feature calculation unit215, the voice existence probability calculation unit216, and the update unit217. The input signal acquisition unit211acquires an acoustic signal and a non-acoustic signal relating to the same voice generation source. The acoustic feature calculation unit212calculates an acoustic feature from the acoustic signal using the first neural network. The non-acoustic feature calculation unit213calculates a non-acoustic feature from the non-acoustic signal using the second neural network. The reliability weight calculation unit214calculates a reliability weight on the basis of a difference between the acoustic feature and the non-acoustic feature. The integrated feature calculation unit215calculates an integrated feature of the acoustic feature and the non-acoustic feature on the basis of the reliability weight. The voice existence probability calculation unit216calculates a voice existence probability from the integrated feature using the third neural network. The update unit217updates the first neural network, the second neural network, and the third neural network using a total loss function including a first loss function providing a penalty for a difference between the acoustic feature and the non-acoustic feature and a second loss function providing a penalty for a difference between a correct label relating to the voice section and the non-voice section and the voice existence probability.

By acquiring the acoustic feature and the non-acoustic feature using the first trained model and the second trained model acquired by such learning, a difference between the acoustic feature and the non-acoustic feature decreases for input of a normal non-acoustic signal, and a difference between the acoustic feature and the non-acoustic feature increases for input of an abnormal non-acoustic signal. By using such a difference value between the acoustic feature and the non-acoustic feature as a reliability weight and acquiring an integrated feature of the weighted acoustic feature and the weighted non-acoustic feature, the accuracy of detection of the voice section and/or the non-voice section can be further improved even if input of an abnormal non-acoustic signal occurs.

Verification Example

Verification was executed for the performance of the integrated neural network (hereinafter referred to as “proposed model”) according to the present embodiment and including the first neural network, the second neural network, and the third neural network generated by the embodiment described above. An audio-only model, an AV-catenation model, a baseline model (AV-summation model), and the proposed model were trained and evaluated using a GRID-AV sentence corpus (hereinafter simply referred to as “GRID corpus”) formed of AV recording of 1000 sentences spoken by each of 34 speakers (18 men and 16 women).

For learning of the models in consideration of noise, background noises provided in the 4th CHiME challenge were selected at random, and mixed into all acoustic training data of the GRID corpus having a SNR falling within a range of −5 to +20 dB. The noises of the CHiME 4 were recorded in four places, that is, the bus, the cafeteria, the pedestrian area, and the street intersection. To evaluate the performance under matched noise conditions, the noises of the CHiME 4 were mixed into all test acoustic records of the GRID corpus at 5 dB intervals with the SNR in a range of −5 to +15 dB. In the same manner, to evaluate the performance under mismatched noise conditions, the background noises from an EastAnglia data set were mixed into all test acoustic records of the GRID corpus at 5 dB intervals with the SNR in a range of −5 to +15 dB. The EastAnglia data set provides environment noise collected in 10 places, that is, the bar, the beach, the bus, the car, the soccer game, the laundromat, the lecture, the office, the railway station, and on the road.

FIG.8is a diagram illustrating an example of an image signal (image data) according to the example. Because the GRID corpus includes no abnormal images, three types of abnormal images as illustrated inFIG.8were generated. The abnormal images were used only for evaluation. The first type is random pixel images in each of which random pixels supposing a video read error and/or a crash are uniformly distributed. The second type is non-lip area images extracted from the GRID corpus supposing a face landmark detection error. The third type is 10-type face mask images formed on the supposition that the speaker wearing a mask.

Records of each of the speakers of the GRID corpus were divided into a training data set, a verification data set, and a test data set at a ratio of 6:2:2. The verification data set was used to specify a hyperparameter suitable for each model and each experiment condition. With respect to the input voice, all the voice records were subjected to resampling at a sampling rate of 16 kHz, and 128-dimensional logarithmic compression mel-spectrogram was calculated using short-time Fourier transform having a window size of 1,280 samples (80 milliseconds) and a hop length of 640 samples (40 milliseconds). With respect to the input image, all the image records are cut into lip areas using a face landmark detector of 68 coordinates, and converted into 25 frames/seconds (40 milliseconds) with resolution of H×W=40×64 pixels, and the RGB channels were normalized between 0 to 1. All the models were trained by gradient clipping using Adam optimization. The learning rate was initialized to 0.0001, and the batch size was set to 1. The correct label of VAD of the expression (7) was generated by applying the VAD method to the corresponding good-quality voice data of the GRID corpus. The ground truth value of “k=1” was set to 1 in the case where the frame is an input acoustic frame including voice, and set to 0 in the case where the frame is a frame including no voice or including only noise. The scale factor6in the expression (2) was set to 100 in all the experiment conditions in the proposed model.

FIG.9is a table illustrating verification results under matched noise conditions using a data set of CHiME4.FIG.10is a table illustrating verification results under mismatched noise conditions using a data set of EastAnglia.FIG.9andFIG.10use an area under the receiver operating characteristic (AUROC) as an index of quantitative performance evaluation of AV-VAD.FIG.9andFIG.10illustrate evaluation results in the case where abnormal image input is present/absent for all the models. The bold-type characters indicate the best values, and the second best values are underlined.

In the case where no abnormal image is input, the AV model generally presented better results than the Audio-only (A-only) model, in particular, at a low SNR. The baseline (Baseline) model presented the best results on average in the AV models, except the proposed (Proposed) model. However, in the case where an abnormal image is input, the baseline model presented bad results and presented relative improvement rate of −13.75% at maximum in comparison with the Audio-only model.

FIG.11is a diagram illustrating mean ROC curves of an Audio-only model and baseline models under matched noise conditions. As illustrated inFIG.11, the false acceptance rate (FAR) tends to increase in the case of an abnormal input image in which a pixel value frequently varies between frames, and the false rejection rate (FRR) tends to increase in the case of abnormal image input (face mask). The results indicates that change of the pixel value relating to motion of the mouth is used as an important visual feature in AV-VAD.

The proposed model presented values markedly higher than those of the baseline models in all the experiment conditions, and presented relative improvement of 2.07$ at maximum for normal image input, and 15.97% at maximum for abnormal image input. In addition, the proposed model presented relative improvement of 2.37% at maximum for an abnormal input image in comparison with the Audio-only model. The proposed model acquired good results only for a normal input image in comparison with the baseline models by using the loss function expressed with the expression (5). The results present high similarity between the acoustic feature and the image feature in the latent space of AV-VAD. This structure improves the whole learning efficiency of neural networks relating to AV-VAD, and contributes to improvement of performance.

Additional Example

The following is an explanation of the voice activity detection apparatus100according to an additional example of the present embodiment. In the following explanation, constituent elements having substantially the same functions as those of the present embodiment will be denoted by the same reference numerals, and an overlapping explanation is made only when necessary.

FIG.12is a diagram illustrating a configuration example of a voice activity detection apparatus according to the additional example. As illustrated inFIG.12, the processing circuit11achieves a normal/abnormal determination unit119, in addition to an input signal acquisition unit111, an acoustic feature calculation unit112, a non-acoustic feature calculation unit113, a reliability weight calculation unit114, an integrated feature calculation unit115, a voice existence probability calculation unit116, a voice section detection unit117, and an output control unit118.

The normal/abnormal determination unit119determines whether the non-acoustic signal acquired with the input signal acquisition unit111is a normal signal or an abnormal signal on the basis of the reliability weight calculated with the reliability weight calculation unit114.

The output control unit118outputs a determination result as to whether the non-acoustic signal acquired with the input signal acquisition unit111is a normal signal or an abnormal signal to the display device15or the like.

The following is an explanation of an example of voice activity detection processing executed with the processing circuit11of the voice activity detection apparatus100according to the additional example. To specifically execute the following explanation, suppose that the non-acoustic signal is an image signal.

FIG.13is a diagram illustrating an example of flow of voice activity detection processing executed with the processing circuit11according to the additional example. The voice activity detection processing is executed with an operation of the processing circuit11in accordance with the voice activity detection program stored in the storage device12or the like. The details of Steps SC1to SC7illustrated inFIG.13are the same as those of Steps SA1to SA7illustrated inFIG.2, and an explanation thereof herein is omitted.

As illustrated inFIG.13, when Step SC4is executed, Steps SC5to SC7are executed, and simultaneously the normal/abnormal determination unit119determines whether the non-acoustic signal (image signal) acquired at Step SC1is a normal signal or an abnormal signal on the basis of the reliability weight calculated at Step SC4(Step SC8). At Step SC8, the normal/abnormal determination unit119determines whether the image signal is a normal signal or an abnormal signal on the basis of comparison of the reliability weight with a predetermined threshold. Specifically, the normal/abnormal determination unit119compares a first reliability weight ΛAwith a threshold θA, determines that the image signal is abnormal in the case of “θA<ΛA”, and determines that the image signal is normal in the case of “θA≥ΛA”. The threshold θAcan be experimentally or empirically set from any natural number in a range from the intermediate value (for example, 0.5) to the upper limit value (for example, 1.0). As an example, the threshold θAcan be set to 0.7 or around. The second reliability weight ΛVmay be compared with a threshold θV. In the case of “θV<ΛV”, the normal/abnormal determination unit119determines that the image signal is normal. In the case of “θV≥ΛV”, the normal/abnormal determination unit119determines that the image signal is abnormal. The threshold θVcan be experimentally or empirically set from any natural number in a range from the intermediate value (for example, 0.5) to the lower limit value (for example, 0.0). As an example, the threshold θVcan be set to 0.3 or around.

When Step SC7and Step SC8are executed, the output control unit118outputs voice section detection results acquired at Step SC7and normal/abnormal determination results acquired at Step SC8(Step SC9). At Step SC9, the output control unit118displays the voice section detection results and the normal/abnormal determination results on the display device15. As an example, the output control unit118displays a display screen including the voice section detection results and the normal/abnormal determination results on the display device15.

FIG.14is a diagram illustrating an example of a display screen I1including the voice section detection results and the normal/abnormal determination results. As illustrated inFIG.14, the display screen I1displays the input signal, the voice section detection results, and the normal/abnormal determination results. The input signal being a video signal is displayed at predetermined frame intervals in time series in the lateral direction. As the voice section detection results, labels each indicating that the frame is a voice section or a non-voice section are displayed in time series.FIG.14illustrates labels “voice” indicating a voice section and labels “non-voice” indicating a non-voice section are displayed. As the normal/abnormal image determination results, labels each indicating that the image signal in the input signal of the frame is a normal signal or an abnormal signal are displayed in time series.FIG.14illustrates labels “normal” indicating a normal signal and labels “abnormal” indicating an abnormal signal.

The operator is enabled to recognize whether the image signal is abnormal or normal by observing the display screen I1. The operator is enabled to estimate the reliability of the voice section detection result by recognizing whether the image signal is abnormal or normal.

The voice activity detection processing with the processing circuit11according to the additional example is finished as described above.

The flow of the voice activity detection processing according to the additional example described above is an example, and the flow is not limited to the flow illustrated inFIG.13. The determination processing with the normal/abnormal determination unit119at Step SC8may be executed at any timing, as long as it is executed after the reliability weight calculation processing with the reliability weight calculation unit114at Step SC4. For example, the determination processing may be executed before Steps SC5to SC7are executed after the reliability weight calculation processing with the reliability weight calculation unit114at Step SC4. In this case, Step SC5to SC7are executed in the case where it is determined that the image signal is normal, and no Steps SC5to SC7may be executed in the case where it is determined that the image signal is abnormal. As described above, the whole processing quantity can be reduced by stopping the subsequent processing in the case where it is determined that the image signal is abnormal. In addition, because voice section detection is executed only in the case where it is determined that the image signal is normal, the reliability of voice section detection results increases.

The embodiment described above is an example, and various modifications are possible. For example, the acoustic signal is a voice signal in the embodiment, but may be a vocal cord signal or a vocal tract signal acquired by decomposing the voice signal. In addition, the acoustic signal is waveform data of sound pressure values in the time region or the frequency region in the embodiment, but may be data acquired by converting the waveform data into any space.

Accordingly, the embodiment described above suppresses reduction in detection accuracy due to abnormal image input.