Patent Description:
<CIT> and <CIT> disclose methods for improving speech recognition of hand-held devices such as smartphones when a user whispers. An intention to whisper is recognised through proximity, light or touch sensors.

It is the object of the present invention to provide a microphone and a corresponding method for voice input for detecting a silent voice.

Although performance of voice input has been greatly improved, the voice input is still rarely used in public spaces, such as office or even homes. This is mainly because the voice leakage could disturb and even annoy surrounding people in quiet environment. On the other hand, there is still a risk of scattering private information to unintended audiences. These are not technical issues but social issues. Hence there is no easy fix even if voice recognition system performance is greatly improved.

Implementations of the subject matter described herein provide a silent voice input solution without being noticed by surroundings. Compared with conventional voice input solutions which are based on normal speech or whispering that use egressive (breathing-out) airflow while speaking, the proposed "silent" voice input method is performed by using opposite (ingressive or breathing-in) airflow while speaking. By placing the apparatus (e.g. microphone) of the apparatus very close to the user's mouth with an small gap formed between the mouth and the apparatus, the proposed silent voice input solution can capture stable utterance signal with a very small voice leakage, and thereby allowing the user to use ultra-low volume speech input in public and mobile situations, without disturbing surrounding people. Besides of air flow direction (ingressive and egressive), all other utterance manners are same as our whispering, so that proposed method can be used without special practice.

It is to be understood that the Summary is not intended to identify key or essential features of implementations of the subject matter described herein, nor is it intended to be used to limit the scope of the subject matter described herein. Other features of the subject matter described herein will become easily comprehensible through the description below.

The above and other objectives, features and advantages of the subject matter described herein will become more apparent through more detailed depiction of example implementations of the subject matter described herein in conjunction with the accompanying drawings, wherein in the example implementations of the subject matter described herein, same reference numerals usually represent same components.

The subject matter described herein will now be discussed with reference to several example implementations. It should be understood these implementations are discussed only for the purpose of enabling those skilled persons in the art to better understand and thus implement the subject matter described herein, rather than suggesting any limitations on the scope of the subject matter.

As used herein, the term "includes" and its variants are to be read as open terms that mean "includes, but is not limited to. " The term "based on" is to be read as "based at least in part on. " The term "one implementation" and "an implementation" are to be read as "at least one implementation. " The term "another implementation" is to be read as "at least one other implementation. " The terms "first," "second," and the like may refer to different or same objects. A definition of a term is consistent throughout the description unless the context clearly indicates otherwise.

<FIG> shows a schematic configuration of an example voice input system according to an implementation of the subject matter described herein. As shown, the system includes a silent voice input apparatus <NUM> which can be an individual component or a module integrated in the cellphone or in any other type of electronic device. Generally speaking, the silent voice input apparatus <NUM> is configured to receive and process the user's "silent" voice input. As used herein, the phrase "silent voice" indicates a voice that is generated during the user's utterance with ingressive airflow. Upon receiving the user's silent voice, the apparatus <NUM> converts the user's silent voice to a recognized signal and directs the recognized signal as an output over a network <NUM> to a remote entity, such as another user who is receiving a phone call from the user <NUM>. In this way, the user <NUM> in a public space can have a conversation with the other party at a remote location, without disturbing the surrounding people.

The remote entity may receive the user's voice via a terminal device <NUM>, such as a cell phone. Meanwhile, if the terminal device <NUM> that the other entity has is also equipped with a silent voice input apparatus <NUM> as shown in <FIG>, the other entity can input the silent voice in a similar manner and likewise direct his/her voice back to the user <NUM>. Of course, the other entity may only have a regular cell phone as illustrated in <FIG> that does not support silent input function as described above.

Those skilled in the art may understand that the communication system as illustrated in <FIG> is merely for illustration without suggesting any limitations as to the scope of the subject matter described herein. The silent input apparatus <NUM> can also be applied in other application scenarios.

In some implementations, the silent voice input apparatus <NUM> enables the user to interact with a local computer or any local device or any resources on the network (e.g. the Internet). For example, the user <NUM> speaks query words or sentences by using silent voice, then the voice recognition unit <NUM> converts the words or sentences to text information and sends it to the corresponding applications and obtains the results. Thus, the user <NUM> can confirm schedules or e-mails or search results triggered by silent voice. The voice recognition unit <NUM> may be placed or deployed on the network side. For example, the retrieved data is converted to voice with, for example, the Text-To-Speech (TTS) system, and fed back to the user <NUM> via, for example, an earphone <NUM> worn by the user <NUM>. Other feedback methods such as display may also be used. In one example implementation, the user <NUM> may query with short words like "Next Meeting?" by using silent voice. The TTS system converts the retrieved information, such as "<NUM> am", to sound, and then transfers the sound back to the user <NUM> via the earphone <NUM>.

Still referring to <FIG>, in order to generate the silent voice, the user <NUM> may place the apparatus <NUM> very close to the mouth, thereby defining a gap <NUM> between the user's lower lips and the apparatus <NUM>. It is effective to keep the gap <NUM> as small as possible (such as <NUM> in some examples) for generating stable silent voice with low amount of ingressive airflow.

As further illustrated in <FIG>, the apparatus <NUM> according to implementations of the subject matter described herein mainly includes a microphone <NUM>. It is effective to place the microphone <NUM> at the very close position of narrowly opened mouth (such as <NUM> in some examples) for capturing small silent voice and increasing signal-to-noise (S/N) ratio of the signal. The microphone <NUM> can be configured to detect sound generated by the user during the ingressive utterance as a silent voice and convert the silent voice to a signal representing the silent voice for further processing. The ingressive air flow <NUM> herein is defined as an air flow flowing from outside into the mouth through the formed gap <NUM>.

Acoustically, the ingressive air flow <NUM> is generated by the user's silent voice during the ingressive utterance. In this case, the gap <NUM> forms an artificial sound source which generates air turbulence that is very similar as generated at the narrow gap between the vocal cords of a human when performing a whisper speech. In both case (whisper speech and silent speech), the vocal cords are not vibrated.

<FIG> illustrates the air flow of the silent speech that uses ingressive air flow, and <FIG> illustrates the air flow of a normal speech (herein, normal speech sometimes means both of normal speech and whispering, both of which use "egressive" air flow). In the normal speech (and whispering speech, too) as shown in <FIG>, egressive (or exhaled) air flow <NUM> directly hits diaphragm (not shown) of the microphone <NUM>, which may make the captured voice heavily distorted, especially when the microphone <NUM> is placed very close to the mouth and the mouth is narrowly opened. Most significant case is so called "pop noise" that is caused when uttering /f/ or /p/ sound. In this event, the microphone <NUM> in not suitable to be placed at the very close position of the mouth even though it might be the best position for capturing small voice sound.

In contrast, the proposed ingressive speech as shown in <FIG> generate almost no "pop noise", because contrary to the egressive air flow <NUM>, the generated ingressive air flow <NUM> only passes by the microphone <NUM> without directly hitting the diaphragm of the microphone <NUM>. Therefore, the microphone <NUM> is suitable to be placed at a very close position of the narrowly-opened mouth, and ultra-small voice sound with a good S/N ratio thus can be captured.

In some implementations, the microphone <NUM> can be an omni-directional type microphone. In some other implementations, the microphone <NUM> can be a noise canceling (bi-directional) type microphone that is beneficial for eliminating surrounding noise. However, other types of microphones are also possible according to specific requirements.

As discussed above, it may be beneficial to keep a narrow air gap between the microphone <NUM> and mouth for generating large and substantial turbulence with a small amount of inspiratory air. Therefore, in some implementations, the size of the gap is adjusted to approximately <NUM>-<NUM> from tip to lower lip. In this case, the microphone <NUM> can capture the ingressive air flow corresponding to the user's ultra-small voice, and thus can convert the ingressive utterance sound to a signal for voice communication or voice instruction with a good enough S/N ratio.

Now referring back to <FIG>, the gap <NUM> and the position of the microphone can be adjusted by the user <NUM>. By feeding captured silent voice signal back to the user (indicated by line <NUM> in <FIG>) in real time by using the earphone <NUM>, the user can easily keep proper positioning of the apparatus relative to the mouth.

In some implementations, the microphone <NUM> can be arranged at a substrate <NUM>. <FIG> illustrates a schematic design of the apparatus <NUM> including a substrate <NUM> accommodating the microphone <NUM>. Such a plate- or stick-shaped shielding substrate <NUM> as shown in <FIG> may facilitate the formation of the narrow gap and further enable a stable gap size. Specifically, the substrate <NUM> may have an end portion <NUM> with contact surface <NUM> (as shown in <FIG>) for touching the user's upper lip <NUM>. Upon a contact of an end portion <NUM> of the substrate <NUM> to the user's upper lip <NUM>, the microphone <NUM> will be substantially placed at the very close position (e.g. <NUM>-<NUM>) of narrowly opened mouth, and the gap <NUM> between shielding substrate <NUM> and the lower lip <NUM> (as shown in <FIG>) will be substantially kept with narrow amount (e.g. <NUM>). The convex shape of the end portion <NUM> also prevents the collision between the lower lip <NUM> and the substrate <NUM> during the utterance, for reducing unexpected touch or friction noise. In some embodiments, if the microphone <NUM> is bi-directional (or noise-canceling) type, it may have two sound ports (that is, front-side and back-side), and in this case, the front-side sound port is faced towards the mouth, and the back-side sound port <NUM> is opened for opposite side of the substrate <NUM> for maximizing noise canceling effect. <FIG> illustrate the apparatus <NUM> as shown in <FIG> that is being used by a user.

Moreover, compared to the normal speech and whispering, the combination of use of silent voice with the closely placed shielding plate to the mouth as illustrated in <FIG> has been proven to be able to reduce the sound leakage to the outside.

Now referring to <FIG>, in some implementations, the end portion <NUM> may include a top surface <NUM> of a rounded shape in X-Y plane. The rounded top surface <NUM> is helpful for blocking nasal air flow during utterance. Alternatively, or in addition, the end portion <NUM> may include a stream-lined top profile <NUM> in Y-Z plane, which also blocks nasal air flow, and thereby eliminating the snort noise.

In overall, such design of the apparatus <NUM> as illustrated in <FIG> enables both good air flow performance and good portability (or wearability). For example, pendant and pen-shaped devices provide both easy-to-grip (when in use), and good wearability (when not in use). Moreover, the voice-based interaction system enables eyes-free operation, so that users can use the system quickly and safely in many situations even while walking or driving. However, it is to be understood, the design of the apparatus <NUM> may vary for meeting different situations/conditions or depending on the user's preferences. More possible designs and application scenarios will be discussed later.

Continuing to refer to <FIG>, the apparatus <NUM> includes a flow sensor <NUM>. The flow sensor <NUM> is configured to sense the ingressive air flow (as well as the egressive air flow) by detecting the direction of the air flow. It is to be understood that the flow sensor <NUM> in some embodiments is required to be placed as close to the microphone <NUM> as possible for enhancing sensitivity. In this way, the normal speech and the silent speech are easily separated by simply measuring the direction of the air flow. As such, the same apparatus <NUM> can be used as either a silent voice input device or a normal voice input device. Further, with such flow sensor <NUM>, no manual switches (such as a button) or activation words (such as "Hi Cortana", or "Hz Siri") are needed to switch between the two modes, which enables a hands-free and smooth switching between the two modes.

In some implementations, the apparatus <NUM> may further includes an amplifier <NUM> coupled to the microphone <NUM> and configured to amplify the signal output from the microphone <NUM>, such as a lox microphone amplifier. <FIG> shows an example block diagram of the apparatus <NUM> according to one implementation of the subject matter described herein. In this case, the captured signal is amplified by the amplifier <NUM> only when the inspiratory conditions are detected by the flow sensor <NUM>. In the example as illustrated in <FIG>, upon detecting an ingressive air flow, the flow sensor <NUM> will generate an enabling signal <NUM> for the amplifier <NUM> to activate the silent voice mode. Otherwise, the microphone <NUM> will be working under the normal mode for receiving the normal speech, in which the received signal is not amplified.

The circuit structure of apparatus <NUM> as illustrated in <FIG> is simple, and the control logic for the switching condition can also be achieved by using one or more logic gates (not shown). In this case, in some implementations, the microphone <NUM>, the amplifier <NUM>, the flow sensor <NUM>, and the logic gates can be integrated within one common circuit board with fabrication technologies now known or later developed. In some other implementations, the amplifier <NUM> may be included in the microphone <NUM> as a component of the microphone <NUM>.

In some implementations, a user may use ingressive speech to give commands or instructions to a local device, and those (usually short) commands/instructions may rarely occur in people's normal utterances. In this case, the user may freely switch from an ongoing normal conversation with the other entity to silent-voice-based commands/instructions to a local device, without being noticed by the other entity. This is very helpful especially for the users who work in call centers or some other telephone-intensive jobs, or for anybody wishing to have both hands free during a telephone conversation.

The apparatus <NUM> may additionally include a proximity sensor <NUM>. The proximity sensor <NUM> is configured to sense the touch of the user's mouth (or upper lip only) to the microphone <NUM> by detecting a size of the gap between the microphone <NUM> and the mouth. Such proximity sensors <NUM> usually are much smaller and cheaper than the flow sensors <NUM>, thus may, in examples not covered by the present claims, substitute the flow sensor <NUM> in many applications, such as watch-shaped apparatus or TV-remote controller based apparatus that does not support any long-term continuous operation such as dictation or telecommunication function. In those cases, the user <NUM> may raise his/her hand only when saying silent-voice-based query words or commands, therefore the much simpler proximity sensor <NUM> can also be used instead of the flow sensor <NUM>. Accordingly, the enabling signal <NUM> for activating the amplifier <NUM> (or the silent voice mode) in <FIG> may be a signal indicating that the user's mouth is getting close to the apparatus <NUM>.

The types of proximity sensors <NUM> include but not limited to, optical, magnetic, capacitive, and inductive and so on. It is to be understood that scope of the subject matter described herein is not limited in this aspect.

Now referring back to <FIG>, the apparatus <NUM> may further includes a voice recognition unit <NUM> that is coupled to the microphone <NUM> and configured to receive the signal corresponding to the silent voice of the user <NUM>, and generate a recognized signal based on an acoustic model exclusively for ingressive utterance.

Almost all general voice recognition systems are designed for normal utterance, and therefore may not properly recognize other types of utterance (such as whispering and silent voice) in initial settings. Therefore, in some implementations, the acoustic model is previously trained for the ingressive utterance, in order to enhance the recognition accuracy.

Alternatively, or in addition, in some implementations, the voice recognition unit <NUM> can be further configured to generate the recognized signal, based on one or more special pronunciation dictionaries for the ingressive voice to further enhance the recognition accuracy. This is because, for the ingressive voice, some "unvoiced" phonemes (e.g. /k/, /s/, and /f/) may be mixed with corresponding "voiced" phonemes (e.g. /g/, /z/, and /b/). In addition, signal level of nasal sounds (e.g. /m/, /n/) may decrease. For example, captured silent voice sound /zin/ may reflect /zin/ or /sin/, silent voice sound /am/ may reflect /nam/ or /mam/. Therefore, in some implementations, using a special pronunciation dictionary that reflects above mentioned phonemes substitution may be efficient. The voice recognition unit <NUM> as shown in <FIG> can be a client unit or a cloud-based apparatus, or it can be part of the server. It should be understood that scope of the subject matter described herein is not limited in this aspect.

As discussed above, the apparatus <NUM> for silent voice input according to implementations of the subject matter only requires simple components including a for example, microphone <NUM> and a flow or proximity sensor <NUM>, <NUM>. Therefore, it is easy to integrate such simple circuit structure into other type of objects that are commonly used in people's daily life. <FIG> illustrate some possible application scenarios based on various types of apparatus.

In some implementations as illustrated in <FIG>, the ring style device <NUM> that has a thin microphone <NUM> at the bottom of the ring and an optical proximity sensor at the side of the ring (not shown). When the ring device <NUM> is worn on the index finger of a user and the user may touch the bottom of the index finger to the upper lip with a lightly grasping hand, the proper shaped air gap and acoustic insulation is thus naturally obtained. Moreover, this posture (covering mouth with hand) is very natural and does not seem strange from surroundings.

In some implementation, especially in mobile situations, the user can quickly take a note (text memo or voice memo) by using a pen or smartphone-style apparatus <NUM> without disturbing nearby individuals.

<FIG> shows examples of smart watch <NUM> with a small screen <NUM>. Proposed apparatus <NUM> can be easily embedded in the watchband <NUM> with a proximity sensor <NUM>. Convex shape <NUM> can keep proper gap between the watchband <NUM> and the mouth, keep proper position of the microphone <NUM>, and also prevent nasal air flow. In some embodiments, the convex shape <NUM> may be combined with the edge of the screen <NUM> or chassis of the smart watch <NUM>. The retrieved information, upon a query by the user <NUM>, can be displayed on the smart watch's screen <NUM> as an answer to the query. As an example as illustrated in <FIG>, if the user is interested in the moon phase of the day, a moon phase <NUM> may be displayed on the screen <NUM> upon the user's query. Further, as discussed above, a voice feedback may be additionally provided to the user <NUM> as well, to facilitate the user <NUM> understanding the retrieved information. <FIG> illustrates that the ring style device <NUM> as shown in <FIG> is being used by a user, and <FIG> illustrates that the smart watch <NUM> as shown in <FIG> is being used by a user.

As illustrated in <FIG>, the apparatus <NUM> can also easily be embedded in a smart phone <NUM> (with the flow sensor <NUM>). The smart phone <NUM> serves as an effective shielding "plate" and can be easily fixed at upper lip with for example the upper edge of the chassis <NUM>. To achieve much securer positioning and blocking nasal air flow, convex shape <NUM> may be used. This form-factor is suitable for visual information retrieval in mobile situations such as map navigation, or photo-search. <FIG> illustrates that the smart phone <NUM> as shown in <FIG> is being used by a user.

<FIG> illustrates an example of a remote controller <NUM> with a proximity sensor <NUM> and the microphone <NUM>. Convex shape <NUM> likewise can keep proper gap between the chassis <NUM> and the mouth, keep proper position of the microphone <NUM>, and also prevents nasal air flow. Proposed silent voice input method enables unnoticeable voice control in a living room. For this purpose, a cost-effective proximity sensor <NUM> is good enough for short command operations. <FIG> illustrates that the remote controller <NUM> as shown in <FIG> is being used by a user.

<FIG> is an example of headset <NUM> for real-time voice communication by directly transferring captured voice signals. As shown in <FIG>, the apparatus <NUM> is combined with a head phone <NUM>. The user can make/receive telephone calls in public spaces without annoying surrounding people. A flow sensor <NUM> is needed for long-term operations. In addition, as discussed above, the microphone <NUM> can also be used as a normal microphone when deactivated. Another possible "special" case is using proposed method for face-to-face communication, enabling secure conversations in public spaces.

As further illustrated in <FIG>, the apparatus <NUM> is combined with a head phone <NUM> via a connectable cable or cord <NUM>. However, it is to be understood that the cable <NUM> is not limited to the hard-wired connections as illustrated in <FIG>. Rather, wireless connections such as Bluetooth, Wi-Fi, or optical communication are also possible. In some implementations, it is of course possible to have a cordless headset <NUM>, and in this case, the apparatus <NUM> could be even an integrated or embedded component in the head phone <NUM>.

<FIG> is an example of overhead-style headset <NUM> that is commonly used at offices and call centers. Convex shape <NUM> can keep proper gap between the sensor chassis <NUM> and the mouth, keep proper position of the microphone <NUM>, and also prevents nasal air flow. Hands-free switching with flow sensor <NUM> enables speedy document creation by combining proposed silent voice input method and conventional keyboard input. <FIG> illustrates that the overhead-style headset <NUM> as shown in <FIG> is being used by a user.

Claim 1:
An apparatus (<NUM>) for voice input, comprising:
a flow sensor (<NUM>) configured to detect an ingressive air flow (<NUM>) flowing into the mouth of a user (<NUM>) through a gap (<NUM>) formed between a microphone (<NUM>) and the mouth during an ingressive utterance by the user (<NUM>) by detecting a direction of the air flow and to activate a silent voice mode upon detection of the ingressive air flow; and
the microphone (<NUM>) configured to, in response to the detection of an ingressive air flow by the flow sensor, detect a silent voice generated by the ingressive utterance.