Patent Description:
Speech recognition systems enable communication between a user and a device or a system without separate button controls, or without having to contact a touch module allowing to control the device or the system more easily. In addition, speech recognition systems may form a part of a human-machine interface. The speech recognition systems are used, for example, in voice guided systems and telephone bots. Some functionalities of smartphones, for example, may also rely on speech recognition.

In speech recognition, silence recognition plays a crucial role. This is particularly important in real time speech recognition systems since the system has to be able to differentiate between natural pauses in the speech and the end of the speech. In the prior art, it is known from the patent application <CIT>, a method for recognizing speech using an endpoint state transition network for starting and stopping the speech recognition process. It is known according to the patent application <CIT>, techniques using a frontend speech endpointing with end-of-speech timeout before a speech recognizer. It is known according to the patent application <CIT>, techniques setting an end of speech duration period based on user habits or settings. It known from patent application <CIT>, techniques for adjusting a trailing silence period based on a trailing silence reference period.

A new approach for silence detection is provided.

According to a first example aspect there is provided a computer implemented method for speech recognition from an audio signal. The method comprises:.

In an embodiment, the method further comprises: adjusting the lead period based on the duration from the second time T2 to a first instance the amplitude of the audio signal exceeds the threshold amplitude.

In an embodiment, the method further comprises: adjusting the terminal period based on duration of periods wherein the amplitude of the audio signal is below the threshold amplitude.

In an embodiment, the method further comprises: adjusting the threshold amplitude based on background noise level and/or the detected amplitude of the speech.

In an embodiment, the method further comprises: repeating the method for multiple segments of the audio signal.

In an embodiment, the initial values of the lead period and/or the terminal period; and/or the adjusting the lead period and/or the terminal period is based on a speech context comprising: an expected speech type and/or an expected use-case type.

In an embodiment, the initial values of the silence detection parameters; and/or the adjusting the silence detection parameters is/are based on a statistical estimate of the silence detection parameters; and the method further comprises calculating the statistical estimate using any one or more of: average; moving average; squared average; Kalman filtering; or time-series analysis based methods comprising autoregressive moving average (ARMA) and autoregressive integrated moving average (ARIMA).

In an embodiment, the method further comprises: calculating the statistical estimates from audio signal samples comprising at least one of: audio signal samples of the current use-case, audio signal samples of the previous use-case(s) of the current user, and audio signal samples from multiple users.

In an embodiment, the method further comprises: providing an output comprising indication of the termination of the speech recognition.

In an embodiment, the method further comprises: terminating the speech recognition and the silence detection after a pre-set time limit in response to detecting that the speech recognition has not been terminated.

In an embodiment, the method further comprises: coding the audio signal using pulse code modulation; and detecting the amplitude of the audio signal based on the pulse code modulation.

According to a second example aspect of the present invention, there is provided an apparatus comprising a processor and a memory including computer program code; the memory and the computer program code configured to, with the processor, cause the apparatus to perform the method of the first aspect or any related embodiment.

In an embodiment, the apparatus is selected from a group consisting of: a telephone bot; a member of a voice guiding system; and a voice guided device.

According to a third example aspect there is provided a computer program comprising computer executable program code which when executed by at least one processor causes an apparatus at least to perform the method of the first aspect or any related embodiment.

According to a fourth example aspect there is provided a computer program product comprising a non-transitory computer readable medium having the computer program of the third example aspect stored thereon.

According to a fifth example aspect there is provided an apparatus comprising means for performing the method of any preceding aspect.

Any foregoing memory medium may comprise a digital data storage such as a data disc or diskette, optical storage, magnetic storage, holographic storage, opto-magnetic storage, phase-change memory, resistive random-access memory, magnetic random-access memory, solid-electrolyte memory, ferroelectric random access memory, organic memory or polymer memory. The memory medium may be formed into a device without other substantial functions than storing memory or it may be formed as part of a device with other functions, including but not limited to a memory of a computer, a chip set, and a sub assembly of an electronic device.

<FIG> shows an example scenario of speech recognition. The scenario shows a user <NUM> and a speech recognition system <NUM>. In an embodiment of the invention, the scenario of <FIG> operates as follows: In a first phase <NUM>, the speech recognition system <NUM> inputs an audio signal from the user <NUM>. The speech recognition system <NUM> recognizes speech from the audio signal. In a second phase <NUM>, the speech recognition system <NUM> outputs the recognized speech. In an embodiment, the recognized speech is in text format, e.g., a transcription. In an embodiment, the output comprises instructions or commands based on the recognized speech. In a third phase <NUM>, the speech recognition system <NUM> may provide a communication to the user <NUM>. The communication may comprise an audio part, such as a spoken prompt. In an embodiment, the communication comprises the recognized speech. In another embodiment, the communication comprises a question to the user <NUM>.

In another example, the scenario of <FIG> starts from the third phase <NUM>: The speech recognition system <NUM> provides the communication to the user <NUM>. In the first phase <NUM>, the user <NUM> provides input to the speech recognition system <NUM>. Then, in the second phase <NUM>, the speech recognition system <NUM> provides an output. Alternatively, or additionally, the third <NUM> and first <NUM> phases may be performed repeatedly before the second phase <NUM>.

The process may be manually or automatically triggered. The process may be triggered for example by a user calling to an automated telephone bot or a caller bot calling to a user. The telephone bot may be operating, e.g., in a customer service system. In an embodiment, the process may be triggered by a user giving voice instructions to a voice-guided system.

The voice-guided system may be comprised in, or configured to control, e.g., any one or more of: a smartphone; a tablet computer; or a car.

<FIG> shows a block diagram of an apparatus <NUM> according to an example embodiment. The apparatus <NUM> comprises a communication interface <NUM>; a processor <NUM>; a user interface <NUM>; and a memory <NUM>. The apparatus <NUM> is for example a general-purpose computer or server or some other electronic data processing apparatus. The apparatus <NUM> can be used for implementing embodiments of the invention. That is, with suitable configuration the apparatus <NUM> is suited for operating for example as the speech recognition system <NUM> of the foregoing disclosure.

The communication interface <NUM> comprises in an embodiment a wired and/or wireless communication circuitry, such as Ethernet; Wireless LAN; Bluetooth; GSM; CDMA; WCDMA; LTE; and/or <NUM> circuitry. The communication interface can be integrated in the apparatus <NUM> or provided as a part of an adapter, card or the like, that is attachable to the apparatus <NUM>. The communication interface <NUM> may support one or more different communication technologies. The apparatus <NUM> may also or alternatively comprise more than one of the communication interfaces <NUM>.

The processor <NUM> may be a central processing unit (CPU), a microprocessor, a digital signal processor (DSP), a graphics processing unit, an application specific integrated circuit (ASIC), a field programmable gate array, a microcontroller or a combination of such elements.

The user interface may comprise a circuitry for receiving input from a user of the apparatus <NUM>, e.g., via a keyboard, graphical user interface shown on the display of the apparatus <NUM>, speech recognition circuitry, or an accessory device, such as a headset, and for providing output to the user via, e.g., a graphical user interface or a loudspeaker.

The memory <NUM> comprises a work memory <NUM> and a persistent memory <NUM> configured to store computer program code <NUM> and data <NUM>. The memory <NUM> may comprise any one or more of: a read-only memory (ROM); a programmable read-only memory (PROM); an erasable programmable read-only memory (EPROM); a random-access memory (RAM); a flash memory; a data disk; an optical storage; a magnetic storage; a smart card; a solid state drive (SSD); or the like. The apparatus <NUM> may comprise a plurality of the memories <NUM>. The memory <NUM> may be constructed as a part of the apparatus <NUM> or as an attachment to be inserted into a slot, port, or the like of the apparatus <NUM> by a user or by another person or by a robot. The memory <NUM> may serve the sole purpose of storing data, or be constructed as a part of an apparatus <NUM> serving other purposes, such as processing data.

A skilled person appreciates that in addition to the elements shown in <FIG>, the apparatus <NUM> may comprise other elements, such as microphones, displays, as well as additional circuitry such as input/output (I/O) circuitry, memory chips, application-specific integrated circuits (ASIC), processing circuitry for specific purposes such as source coding/decoding circuitry, channel coding/decoding circuitry, ciphering/deciphering circuitry, and the like. Additionally, the apparatus <NUM> may comprise a disposable or rechargeable battery (not shown) for powering the apparatus <NUM> if external power supply is not available.

<FIG> shows a flow chart according to an example embodiment. <FIG> illustrates a computer implemented method for speech recognition from an audio signal comprising various possible process steps including some optional steps while also further steps can be included and/or some of the steps can be performed more than once:.

In some embodiments, the adjusting the lead period and/or the terminal period may be based on a speech context. The speech context may comprise: an expected speech type and/or an expected use-case type. For example, the lead period after a familiar question having a yes/no answer may be short. The terminal period for a speech expected to comprise multiple words may be longer than for a speech known to comprise single words or only a few words. The lead period and the terminal period for an answer to a more complicated question may be longer than that for an answer to a simple question.

In some embodiments, the adjusting the silence detection parameters may be based on a statistical estimate of the silence detection parameters, wherein the statistical estimate is calculated any one or more of: average; moving average; squared average; Kalman filtering; or time-series analysis based methods comprising autoregressive moving average (ARMA) and autoregressive integrated moving average (ARIMA). The statistical estimates may be calculated from audio signal samples comprising audio signal samples of the current use-case. The statistical estimates may be calculated from audio signal samples of the previous use-case(s) of the current user. The statistical estimates may be calculated from audio signal samples from multiple users. In an example embodiment, the statistical estimates are selected so that <NUM>% of the samples fall within the estimate.

In an embodiment, the silence detection parameters may be adjusted between pre-set minimum and maximum vales. In an embodiment, the silence detection parameters are adjusted with fixed or relative adjustments. In an example embodiment, the adjustment is selected from a group consisting of: <NUM> %; <NUM> %; <NUM> %; and <NUM> % of the current value.

In some embodiments, the initial values for the silence detection parameters obtained in step <NUM> are adjusted based on the same approaches as the adjusting in step <NUM>.

<NUM>: Optionally, the method is repeated for multiple segments of the audio signal. In an embodiment, the speech recognition system may provide multiple question to the user and the speech recognition system may recognize multiple answer from the audio signal. In an example embodiment, the speech recognition system may repeat the same question multiple times if the speech recognition system fails to recognize speech from the audio signal. In an example embodiment, a telephone bot communicates with the user, provides multiple questions, and recognizes multiple answers. In an example embodiment, a voice-guided device may require further instructions from a user in response to failing to recognize the speech of the user.

<NUM>: Optionally, terminating the speech recognition and the silence detection after a pre-set time limit in response to detecting that the speech recognition has not been terminated.

<NUM>: Optionally, providing an output. The output may indicate the termination of the speech recognition. The output may comprise the adjusted silence detection parameters. The output may indicate that the speech recognition failed. The output may comprise the recognized speech. The output may comprise a transcription of the recognized speech. The output may comprise commands or instructions based on the recognized speech. The output may comprise the audio signal. The output may comprise statistical estimates calculated based on the audio signal.

<FIG> illustrate example pulse code modulated audio signals and various periods of some example embodiments. Amplitude of a pulse code modulated audio signal is shown in the vertical axis and time from the beginning of the audio signal is shown in the horizontal axis. The pillars indicate the measured amplitudes. The lead period is indicated with an arrow <NUM>, a dashed line <NUM> indicates the threshold amplitude, and the terminal period is in indicated with an arrow <NUM>.

<FIG> shows an example of the terminal period. The amplitude exceeds the threshold value at t1. Since the duration between t0 and t1 is shorter than the terminal period, silence is not detected. Also, at t2 the measured amplitude <NUM> is below the threshold amplitude <NUM>, but the silence is not detected since the amplitude of the audio signal exceeds the threshold amplitude <NUM> before the terminal period has lapsed. Amplitude <NUM> below the threshold amplitude <NUM> may be measured due to, e.g., normal pause in the speech. The amplitude of the audio signal stays below the threshold amplitude <NUM> for a duration of the terminal period <NUM>. Thus, silence is detected at t4 and the speech recognition and the silence detection terminate.

<FIG> shows an example of the lead and the terminal periods. The silence detection starts after the lead period <NUM> at t1. The amplitude exceeds the threshold value at t2. Since the duration between t1 and t2 is shorter than the terminal period, silence is not detected. Since the amplitude of the audio signal stays below the threshold amplitude <NUM> for a period longer than the terminal period <NUM> between t0 and t2, a false silence would have been detected if the lead period <NUM> had not been used. In this example, the user starts speaking at t2 after a long delay. These may be, e.g., due to a telephone bot asking a complicated question. For example, the bot may be asking a customer number and the user may need to check it from a letter etc. Detected amplitudes <NUM> and <NUM> are below the threshold, but since the duration is shorter than the terminal period, silence is not detected. Silence is detected at t5, since the amplitude of the audio signal stays below the threshold amplitude a duration of the terminal period after t4. The lead period and/or the terminal period may be adjusted using the disclosed methods.

<FIG> shows an example of adjusting the threshold amplitude. The amplitude detection starts at t0. A telephone bot, for example, may give instruction to the user between t0 and t1. During this period, the user may be silent, but background noise level may be detected from the audio signal. The threshold amplitude may be adjusted according to the detected noise level such that the threshold is set, e.g., to the average of the detected noise level plus a safety-margin. Between t1 and t2, the user is still silent, due to e.g., thinking how to cope with the instruction given by the telephone bot, and thus only some background noise may be detected. After t2, amplitude exceeding the threshold amplitude is detected and the audio signal is interpreted as speech of the user.

<FIG> shows an example where a user communicates with a speech recognition system. In this example embodiment, the user calls to a telephone bot and the bot consecutively asks questions and recognizes the answers to the questions from the audio signal. During periods <NUM> and <NUM>, the telephone bot speaks giving instructions and asking questions from the user. The amplitude detection from the audio signal starts at t0. The speech recognition starts after telephone bot speeches <NUM> and <NUM> at t1 and t9. The lead periods <NUM> start after <NUM> and <NUM> at t1 and t9. The user stays silent during periods <NUM> and <NUM> after the telephone bot has asked questions during periods <NUM> and <NUM> due to, for example, the user thinking how to reply. The user speaks during periods <NUM> and <NUM> starting at t2 and t10. During period <NUM> between t5 and t6, the user has a pause in user's speech. In this example, the threshold amplitude is not adjusted, since no significant background noise is detected. The threshold amplitude may be adjusted also based on amplitude level of the speech of the user. The second lead period <NUM> staring at t9 has been adjusted shorter based on the silence periods <NUM> between t1 and t2, and <NUM> between t5 and t6. Furthermore, the terminal period <NUM> may be adjusted after t8 according to the detected behavior of the user. In this example, the speech recognition and the silence detection terminate at t8 since the silence have been detected. However, the amplitude detection from the audio signal still continues, since the telephone bot has multiple questions. After the second question, the speech recognition and silence detection start again according to the presented embodiments.

The embodiments of the invention provide automated speech recognition methods. An advantage of the invention is that adaptive silence detection may be provided. An advantage is that the silence detection may adapt to the user's speech. An advantage is that the silence detection may adapt to the environmental factors. An advantage is that the silence detection may adapts to the context of the speech.

Various embodiments have been presented. It should be appreciated that in this document, words comprise, include and contain are each used as open-ended expressions with no intended exclusivity.

Claim 1:
A computer implemented method for speech recognition from an audio signal, the method comprising:
obtaining initial values (<NUM>) for silence detection parameters comprising:
a lead period;
a threshold amplitude; and
a terminal period;
starting to detect an amplitude (<NUM>) of the audio signal at a first time T1 of the audio signal;
optionally, adjusting the threshold amplitude (<NUM>) based on the detected amplitude;
starting the speech recognition (<NUM>) from a second time T2 of the audio signal, wherein the second time T2 is at or after the first time T1;
starting silence detection (<NUM>) from the audio signal when lead period has elapsed after the second time T2 comprising:
responsive to detecting amplitude below the threshold amplitude for a duration of the terminal period, terminating the speech recognition and the silence detection at a third time T3 of the audio signal; and
adjusting the silence detection parameters (<NUM>) based on the detected amplitude changes of the audio signal between the first time T1 and the third time T3.