Bone conduction confirmation

According to an aspect, there is provided an apparatus for a wearable terminal device including circuitry configured for performing the following. The apparatus receives, via a bone conduction sensor, a bone conduction signal and, via at least one microphone over the air, an audio signal. The bone conduction signal and the audio signal are, at least in part, substantially concurrently recorded signals. The apparatus calculates a value of a similarity metric for evaluating an extent of similarity between the bone conduction and audio signals. In response to the value exceeding a pre-defined threshold, the apparatus causes performing one or more actions including executing, in response to detecting a voice command, the voice command and/or modifying, if the audio signal is received via a plurality of microphones over the air, one or more audio focusing parameters of the plurality of microphones for increasing the value of the similarity metric.

TECHNICAL FIELD

Various example embodiments relate to wireless communications.

BACKGROUND

Nowadays many user devices possess voice recognition capabilities: cell phones, smart speakers, computers, cameras, smart watches, HMDs, cars and so on. Speech recognition is especially useful in situations where user's hands are not free, the user needs to focus on other things like driving a car or the user is located at a distance from the user device and can thus only ask questions verbally and hear answers generated, e.g., using speech synthesis. Such user devices typically respond to all spoken questions/commands equally and thus may respond also to unintentional commands or commands from wrong persons (i.e., person who are not the actually or intended user of the user device). Also, user devices typically respond to the loudest source of speech in the vicinity, not to the speech of a specific desired person.

BRIEF DESCRIPTION

According to an aspect, there is provided the subject matter of the independent claims. Embodiments are defined in the dependent claims. The scope of protection sought for various embodiments is set out by the independent claims.

According to an aspect, there is provided an apparatus for a wearable terminal device, the apparatus comprising

at least one processor, and

at least one memory for storing instructions to be executed by the processor, wherein the at least one memory and the instructions are configured to, with the at least one processor, cause the apparatus at least to perform:

receiving, via a bone conduction sensor, a bone conduction signal;

receiving, via at least one microphone over the air, an audio signal, wherein the bone conduction signal and the audio signal are, at least in part, substantially concurrently recorded signals;

calculating a value of a similarity metric for evaluating an extent of similarity between the bone conduction signal and the audio signal; and

in response to the value of the similarity metric exceeding a pre-defined threshold indicating that the bone conduction signal and the audio signal relate, at least in part, to a same segment of sound originating from a mouth of a user of the wearable terminal device, causing performing one or more actions,

wherein, if the at least one microphone consists of a single microphone, the one or more actions comprise:executing, in response to detecting a voice command in at least one of the audio signal and the bone conduction signal, the voice command or

if the at least one microphone comprises a plurality of microphones, the one or more actions comprise at least one of:executing, in response to detecting a voice command in at least one of the audio signal and the bone conduction signal, the voice command; andmodifying one or more audio focusing parameters of the plurality of microphones for increasing the value of the similarity metric.

According to an aspect, there is provided an apparatus comprising

at least one processor, and

at least one memory for storing instructions to be executed by the processor, wherein the at least one memory and the instructions are configured to, with the at least one processor, cause the apparatus at least to perform:

receiving, via at least one microphone over the air or via said at least one communication link or network from the wearable terminal device, an audio signal;

receiving, via at least one communication link or network from a wearable terminal device comprising a bone conduction sensor, a bone conduction signal, wherein the bone conduction signal and the audio signal are, at least in part, substantially concurrently recorded signals;

calculating a value of a similarity metric for evaluating an extent of similarity between the bone conduction signal and the audio signal; and

in response to the value of the similarity metric exceeding a pre-defined threshold indicating that the bone conduction signal and the audio signal relate, at least in part, to a same segment of sound originating from a mouth of a user of the wearable terminal device, causing performing one or more actions,

wherein, if the at least one microphone consists of a single microphone, the one or more actions comprise:executing, in response to detecting a voice command in at least one of the audio signal and the bone conduction signal, the voice command or

if the at least one microphone comprises a plurality of microphones, the one or more actions comprise at least one of:executing, in response to detecting a voice command in at least one of the audio signal and the bone conduction signal, the voice command; andmodifying one or more audio focusing parameters of the plurality of microphones for increasing the value of the similarity metric.

The embodiments and features, if any, described in this specification that do not fall under the scope of the independent claims are to be interpreted as examples useful for understanding various embodiments.

DETAILED DESCRIPTION OF SOME EMBODIMENTS

The expression “communicatively connected” as used in the following may have the meaning of connected so as to enable communication (i.e., transmission and/or reception of signals) between the connected elements. Elements which are communicatively connected may be connected, for example, via one or more wired communication links, one or more wireless communication links, one or more wired communication networks and/or one or more wireless communication networks. The expression “communicatively connected” does not necessarily imply that the associated elements are electrically connected (i.e., connected via a conducting path) and/or physically connected.

The example ofFIG.1shows a part of an exemplifying radio access network.

FIG.1shows terminal devices100and102. The terminal devices100and102may, for example, be user devices. The terminal devices100and102are configured to be in a wireless connection on one or more communication channels with a node104. The node104is further connected to a core network110. In one example, the node104may be an access node such as (e/g)NodeB providing or serving devices in a cell. In one example, the node104may be a non-3GPP access node. The physical link from a terminal device to a (e/g)NodeB is called uplink or reverse link and the physical link from the (e/g) NodeB to the device is called downlink or forward link. It should be appreciated that (e/g)NodeBs or their functionalities may be implemented by using any node, host, server or access point etc. entity suitable for such a usage.

The terminal device (also called a user device, a UE, user equipment, a user terminal etc.) illustrates one type of an apparatus to which resources on the air interface are allocated and assigned, and thus any feature described herein with a terminal device may be implemented with a corresponding apparatus, such as a relay node. An example of such a relay node is a layer 3 relay (self-backhauling relay) towards the base station.

The terminal device typically refers to a device (e.g. a portable or nonportable computing device) that includes wireless mobile communication devices operating with or without a subscriber identification module (SIM), including, but not limited to, the following types of devices: a mobile station (mobile phone), smartphone, personal digital assistant (PDA), handset, device using a wireless modem (alarm or measurement device, etc.), laptop and/or touch screen computer, tablet, game console, notebook, and multimedia device. It should be appreciated that a terminal device may also be a nearly exclusive uplink only device, of which an example is a camera or video camera loading images or video clips to a network. A terminal device may also be a device having capability to operate in Internet of Things (IoT) network which is a scenario in which objects are provided with the ability to transfer data over a network without requiring human-to-human or human-to-computer interaction, e.g., to be used in smart power grids and connected vehicles. The terminal device may also utilise cloud. In some applications, the terminal device may comprise a user portable device with radio parts (such as a watch, earphones or eyeglasses) and the computation is carried out in the cloud. The terminal device (or in some embodiments a layer 3 relay node) is configured to perform one or more of user equipment functionalities. The terminal device may also be called a subscriber unit, mobile station, remote terminal, access terminal, user terminal or user equipment (UE) just to mention but a few names or apparatuses.

Nowadays many user devices such as cell phones, smart speakers, computers, cameras, smart watches, HMDs and cars possess voice recognition capabilities. Speech recognition is especially useful in situations where user's hands are not free, the user needs to focus on other things like driving a car or the user is located at a distance from the user device and can thus only ask questions verbally and hear answers generated, e.g., using speech synthesis. Such user devices typically respond to all spoken questions/commands equally and thus may respond also to unintentional commands or commands from wrong persons (i.e., person who are not the actually or intended user of the user device). Also, user devices typically respond to the loudest source of speech in the vicinity, not to the speech of a specific desired person.

Another architecture of a system to which embodiments of the invention may be applied is illustrated inFIG.2.FIG.2illustrates a simplified system architecture only showing some elements and functional entities, all being logical units whose implementation may differ from what is shown. The connections shown inFIG.2are logical connections; the actual physical connections may be different. It is apparent to a person skilled in the art that the systems also comprise other functions and structures.

FIG.2illustrates an exemplary environment200comprising a system230, a user211and a plurality of persons212,213sharing the space or environment200with the user211. Said system230comprises at least a wearable terminal device201worn by the user211and a (non-wearable) terminal device205.

The exemplary environment200may correspond, for example, to a particular room or a particular set of rooms of a house or an office or an interior of a vehicle. At least some of the plurality of persons212,213may be assumed be within a speaking distance from the user211. At least some or all of the plurality of persons212,213may be speaking simultaneously at a given time, as illustrated with the acoustic wavefronts223, resulting in a noise environment. The environment200may also comprise other sound sources (e.g., a television). The environment200may be affected by background or ambient noise (e.g., environmental noises such as water waves, traffic noise and/or alarms and/or bioacoustic noise from animals, and/or electrical noise from devices such as refrigerators, air conditioning, power supplies and/or motors). Such a noisy environment presents multiple challenges for any voice recognition functionalities of the wearable terminal device201and/or the terminal device205as any voice commands (i.e., audio signals221) provided by the user211may be lost in the noise.

The wearable terminal device201, worn in the example ofFIG.2by the user211, may be adapted to be worn around a wrist, around a finger, on a hand or on an arm. For example, the wearable terminal device201may be a smart watch, an activity tracker, a smart band, a smart bracelet or smart jewellery. The wearable terminal device201may correspond to one of the terminal devices100,102ofFIG.1.

The wearable terminal device201may comprise at least one microphone202. The at least one microphone202may be configured to receive audio signals221over the air when the user211is speaking. Thus, the at least one microphone202may be conventional non-contact microphones. Preferably, the audio signals221should be received irrespective of the arm position of the user211at the time, that is, the audio signals221should be received both when the wearable terminal device201worn by the user211is held close to the mouth of the user211and when the wearable terminal device201worn by the user211is held far from the mouth of the user211.

In some alternative embodiments, the wearable terminal device201may not comprise any microphones (measuring of audio signals being handled, instead, by the terminal device205).

Moreover, the wearable terminal device201comprises at least one bone conduction sensor203for receiving or measuring bone conduction (audio) signals. Bone conduction may be defined as conduction of sound (waves) primarily via the bones of a person. The at least one bone conduction sensor203may be configured at least to receive bone conduction signals222when the user211is speaking. It should be noted that as the bone conduction signals propagate within the body of the user, as opposed to over the air, the arm/hand position of the user211does not significantly affect the reception of the bone conduction signals222(i.e., the received signal strength). In other words, the bone conduction propagation distance from the mouth of the user211to the bone conduction sensor wearable terminal device201is not significantly changed by the movement of the arm of the user211(which is not true for the at least one microphone202). The bone conduction sensor203may be, for example, a contact microphone (also known as a piezo microphone) or an accelerometer.

In some embodiments, the wearable terminal device201may comprise at least one display.

The wearable terminal device201is communicatively connected to the terminal device205. Said communication connection204between the wearable terminal device201and the terminal device205may be provided, e.g., via at least one wireless or wired communication link (e.g., using Bluetooth) or via at least one wireless and/or wired communication network.

The terminal device205may be specifically a non-wearable terminal device (or user device or a computing device). For example, the terminal device205may be a smart speaker, a mobile phone, a desktop computing device, a laptop computing device, a touch-based computing device, a camera or a computing device for a vehicle (e.g., a so-called carputer). The terminal device205may correspond to one of the terminal devices100,102ofFIG.1.

The terminal device205comprises a set (or an array) of one or more microphones206. The set of one or more microphones206may be configured to receive or measure audio signals221at least when the user211is speaking within the environment200. In practice, the set of one or more microphones206measure also other audio signals from the environment200such as audio signals containing speech from the plurality of persons212,213.

In some embodiments (such as in the illustrated example), the terminal device205comprises a set (or an array)206of microphones. In such embodiments, the terminal device may be configured to perform audio focusing or more specifically audio beamforming using said set of microphones206. Specifically, audio focusing or audio beamforming may be employed in embodiments for implementing a (reception) beam which is focused on the wearable terminal device201.

Audio focusing may involve amplifying or attenuating sounds received from one or more directions with respect to sounds received from other directions. Audio focusing may be implemented, for example, using a spatial filtering technique. Beamforming is one example of such a spatial filtering technique which may be used in embodiments. In general, audio (or acoustic) beamforming may involve applying different phase shifts (and possibly also adjusting relative amplitudes) at different microphones in such a way that that signals received from particular angles experience constructive interference while others experience destructive interference when the signals from set of microphones206are combined.

Another example of a spatial filtering technique which may be employed in embodiments is direction analysis where individual phases and/or relative amplitudes at the plurality of microphones are analyzed but not modified (in contrast to beamforming). Instead, the analysis results are used to amplify or attenuate one or more parts of the received audio signal (e.g., certain frequencies or certain time segments). Here, the attenuation or amplification may, in some cases, even be global (i.e., the same for all signals received at the plurality of microphones).

In some alternative embodiments, no terminal device205may be provided, that is, all the functionalities may be carried out by the wearable terminal device201as will be discussed in detail in the following.

In some alternative embodiments not illustrated inFIG.2, the wearable terminal device201may be earphones (or equally headphones) or an earpiece (i.e., a singular earphone to be inserted into, over or onto a single ear which may or may not form a part of earphones). Any type of earphones or earpiece may be employed here. For example, the earphones may be in-ear, on-ear or over-the-ear earphones. The earphones or earpiece may be wired, wireless or true wireless. In other alternative embodiments not illustrated inFIG.2, the wearable terminal device201may be adapted to be worn around a head or around a neck. The discussion provided above applies, mutatis mutandis, for the embodiments discussed in this paragraph though it should be noted, in contrast to the wearable terminal device attached to a hand or arm of a user, the distance between the mouth of the user and the earphones or earpiece or a wearable terminal device worn around a head or a neck of the user remains typically substantially constant during use.

FIG.3illustrates a process according to embodiments for performing actions based on the detected voice of the user. The process ofFIG.3may be carried out by a wearable terminal device comprising or being electrically connected to a bone conduction sensor and at least one microphone, such as the wearable terminal device201ofFIG.2, or a part thereof. It should be noted that no further device (e.g., the terminal device205ofFIG.2) is required for carrying out the embodiment ofFIG.3. The wearable terminal device in question may correspond to either of terminal device100,102ofFIG.1. In the following discussion, the apparatus carrying out the process is called the wearable terminal device without loss of generality.

Referring toFIG.3, the wearable terminal device receives, in block301, a bone conduction signal via a bone conduction sensor.

The wearable terminal device receives, in block302, via at least one microphone, an audio signal (or specifically an over-the-air audio signal). The bone conduction signal and the audio signal are, at least in part, substantially concurrently or simultaneously recorded signals (i.e., blocks301,302may be carried out substantially simultaneously). In other words, the bone conduction signal comprises at least a segment substantially matching a segment of the audio signal in terms of time of recording. Here, the expression “substantially concurrently” may include at least the time difference in recording the signals resulting from different propagation speeds of the bone conduction signal and the audio signal (i.e., an acoustic wave) and different propagation paths for the two signals.

Assuming the bone conduction signal and the audio signal relate to speech of a user wearing the wearable terminal device, it should be noted that the bone conduction propagation time and speed of the bone conduction acoustic wave from the mouth of the user via the bones of the user to the bone conduction sensor differs, in general, from the propagation time of the (over-the-air) acoustic wave from the mouth of the user over the air to the at least one microphone. Thus, the exact time of the recording or measuring of a particular word spoken by the user by the bone conduction signal and the (over-the-air) audio signal may differ slightly. The term “substantially” as used in the previous paragraph encompasses such slight variations in the recording of the bone conduction signal and the (over-the-air) audio signal.

In some embodiments, the wearable terminal device may, between blocks302and303, synchronize the bone conduction signal and the audio signal (assuming a pre-defined expected maximum delay window) to overcome the issue mentioned in the previous paragraph. The synchronization may involve applying a positive or negative delay to at least one of the bone conduction signal and the audio signal. The following steps of the process ofFIG.3may, in such a case, be carried out for the modified pair of bone conduction and audio signals.

The wearable terminal device calculates, in block,303a value of a similarity metric for evaluating an extent of similarity between the bone conduction signal and the audio signal. The similarity metric (or equally a similarity parameter) is defined such that a high value of the similarity metric indicates that the bone conduction and audio signals correspond to the same sound (e.g., to the same voice command given by the user of the wearable terminal device). As the bone conduction signal may only relate to sounds made by the user of the wearable terminal device (i.e., to the voice of the user), a high value of the similarity metric can only be achieved if the voice of the user of the wearable terminal device is recorded in both of the bone conduction and audio signals. The similarity metric may be, for example, a cross correlation metric.

The wearable terminal device determines, in block304, whether the value of the similarity metric exceeds a pre-defined threshold indicating that the bone conduction signal and the audio signal relate, at least in part, to a same segment of sound originating from a mouth of a user of the wearable terminal device. Said sound may typically correspond to speech produced by the user (e.g., a spoken voice command given by the user) though it may, alternatively or additionally, correspond to another mouth-based sound such as whistling, a breathing sound, a dental click or a throat-clear sound. Thus, block304enables recognizing a desired speaker from other speakers in the vicinity with high accuracy and without requiring a complex speaker recognition algorithm.

In response to the value of the similarity metric exceeding the pre-defined threshold in block304, the wearable terminal device performs, in block305, one or more (pre-defined) actions. Said one or more actions may comprise actions which depend on voice commands captured in the bone conduction and audio signals and/or actions carried out irrespective of the cognitive or verbal content of the bone conduction and audio signals.

If the process ofFIG.3is performed by an apparatus forming a part (or a unit) of the wearable terminal device (as mentioned above as one alternative), said apparatus may specifically cause (or trigger) performing, in block305, one or more (pre-defined) actions (by the wearable terminal device). In other words, in such embodiments, the apparatus forming a part of the wearable terminal device may not fully carry out all of the one or more actions itself.

The one or more actions may comprise executing, in response to detecting a voice command in at least one or both of the audio signal and the bone conduction signal, said voice command. The wearable terminal device may maintain, in a memory, information on a set of recognized voice commands for use in the detection.

Alternatively or additionally, assuming that the wearable terminal device comprises a plurality of microphones using which the audio signal was received, the one or more actions may comprise modifying one or more audio focusing parameters of the plurality of microphones for increasing (or maximizing) the value of the similarity metric. In other words, the wearable terminal device may adjust the reception beam so that it is better focused on the user of the wearable terminal device and, as a consequence, the wearable terminal device is able to more clearly record any voice commands issued by the user (resulting in the improvement in the value of the similarity metric). The one or more audio focusing parameters may be any parameters which have an effect on audio focusing in reception as carried out using the plurality of microphones. Each audio focusing parameter may be associated with a single microphone of the plurality of microphones or two or more microphones of the plurality of microphones.

In some embodiments, the modifying of the one or more audio focusing parameters may be based at least on the (current) value of the similarity metric and one or more previously calculated values of the similarity metric (maintained in a memory). The one or more previously calculated values of the similarity metric may relate to at least partially different focusing parameters compared to the current value of the similarity metric.

Additionally or alternatively, the modifying of the one or more audio focusing parameters of the plurality of microphones may comprise causing performing one or more further measurements of bone conduction signals and audio signals using at least partially different focusing parameters and basing the adjustment of the one or more audio focusing parameters also on one or more values of the similarity metric calculated based on said further measurements.

As described in connection withFIG.2, the audio focusing may correspond, in some embodiments, specifically to audio beamforming. Thus, said one or more audio focusing parameters may comprise or consist of one or more audio beamforming parameters. Said one or more audio beamforming parameters may comprise, for example, one or more phase shifts and/or one or more gains applied to one or more signals to be inputted to one or more of the plurality of microphones, respectively.

The beamforming as used in embodiments may correspond to fixed or switched beam beamforming or adaptive beamforming. In some embodiments, machine learning-based adaptive beamforming (e.g., based on neural networks or deep learning neural networks) may be employed.

Alternatively or additionally, the one or more actions may comprise actions performance of which is dependent on the amplitude and/or delay of the recorded audio signal and/or of the recorded bone conduction signal, as will be discussed in more detail in connection withFIG.5.

In response to the value of the similarity metric failing to exceed the pre-defined threshold in block304, the wearable terminal device may ignore the bone conduction and audio signals (i.e., do nothing) as it is highly probable that the audio signal and the bone conduction signal do not contain any speech of the user.

FIGS.4A and4Billustrate processes according to embodiments for performing actions based on the detected voice of the user. The process ofFIG.4Amay be carried out by a terminal device comprising or being electrically connected to one or more microphones and being communicatively connected to a wearable terminal device comprising a bone conduction sensor. Said terminal device may correspond to the terminal device205ofFIG.2and/or to either of the terminal devices100,102ofFIG.1. The process ofFIG.4Bmay be carried out by a wearable terminal device comprising or being electrically connected to a bone conduction sensor, such as the wearable terminal device201ofFIG.2, and being communicatively connected to a terminal device configured to carry out the process ofFIG.4A. The wearable terminal device in question may correspond to either of terminal device100,102ofFIG.1. In some embodiments, the process ofFIGS.4A and4Bmay be carried out, respectively, by a part of said terminal device and a part of said wearable terminal device. In the following discussion, the apparatuses carrying out the process ofFIGS.4A and4Bare called the terminal device and the wearable terminal device, respectively.

It should be noted that many of actions discussed in connection withFIGS.4A and4Bcorrespond, to a large extent, to actions of the process ofFIG.3(though, e.g., the entity performing may differ). Thus, any of the definitions provided in connection withFIG.3may apply, mutatis mutandis, to corresponding features ofFIGS.4A and4B.

Referring toFIG.4A, it may be initially assumed the terminal device (e.g., a smart speaker) is paired with the wearable terminal device. The terminal device receives, in block401, an audio signal via at least one microphone of the terminal device over the air or via at least one communication link and/or at least one communication network from the wearable terminal device. In the latter case, the audio signal may have been originally measured or recorded by the wearable terminal device. Said at least one communication link and/or at least one communication network may comprise at least one wireless communication link and/or at least one wireless communication network.

Then, the terminal device receives, in block402, a bone conduction signal via said at least one communication link and/or at least one communication network from the wearable terminal device (which comprises a bone conduction sensor). The bone conduction signal and the audio signal are, at least in part, substantially concurrently recorded signals, similar to as discussed in connection withFIG.3. Here, the expression “substantially concurrently” may include at least the time difference in recording the signals resulting from differing distances between the sound source (i.e., the mouth of the user) and the terminal device and between the sound source and the wearable terminal device and different propagation speeds of the bone conduction signal and the audio signal (i.e., an acoustic wave).

In some embodiments, synchronization between the audio signal and the bone conduction signal may be carried out between blocks402,403, similar to as discussed in connection withFIG.3.

The terminal device calculates, in block403, a value of a similarity metric for evaluating an extent of similarity between the bone conduction signal and the audio signal. The terminal device determines, in block404, whether the value of the similarity metric exceeds a pre-defined threshold indicating that the bone conduction signal and the audio signal relate, at least in part, to a same segment of sound originating from a mouth of a user of the wearable terminal device. The actions relating to blocks403,404may correspond fully to actions discussed above in connection with blocks303,304ofFIG.3(though the actor is different).

In response to the value of the similarity metric exceeding a pre-defined threshold in block404, the terminal device performs, in block405, one or more actions. Said one or more actions may comprise actions which depend on voice commands captured in the bone conduction and/or audio signals and/or actions carried out irrespective of the cognitive or verbal content of the bone conduction and audio signals.

The one or more actions may comprise executing, in response to detecting a voice command in at least one or both of the audio signal and the bone conduction signal, said voice command. The terminal device may maintain, in a memory, information on a set of recognized (or recognizable) voice commands for use in the detection. Executing the voice command may comprise, for example, playing or stopping a particular sound or piece of music, playing or stopping playing a particular video, opening or closing an application, turning volume up or down, turning a display on or off, switching off the wearable terminal device, switching the wearable terminal device to a particular mode (e.g., a stand-by mode), making a web search using a web search engine based on the voice command, displaying information on a screen, providing a reply to a query defined by the voice command using a voice synthesizer and/or via the display, calling a particular number based on the voice command or any combination thereof.

Alternatively or additionally, assuming that the terminal device comprises a plurality of microphones using which the audio signal was received, the one or more actions may comprise modifying one or more audio focusing parameters (or specifically one or more audio beamforming parameters) of the plurality of microphones for increasing (or maximizing) the value of the similarity metric. In other words, the terminal device may adjust the reception beam so that it is better focused on the user of the wearable terminal device (being also the user or owner of the non-wearable terminal device). As a consequence, the voice of the user is amplified relative to other voices or noises in the environment and the terminal device is able to more clearly record any voice commands issued by the user (resulting in an improvement in the value of the similarity metric).

Alternatively or additionally, assuming that both the bone conduction signal and the audio signal were recorded by the wearable terminal device and subsequently communicated to the terminal device, the one or more actions may comprise actions performance of which is dependent on the (relative) amplitudes (or signal levels) and/or the delays of the recorded audio signal and the bone conduction signal (i.e., on the distance between the mouth of the user and the wearable terminal device), as will be discussed in more detail in connection withFIG.5.

In response to the value of the similarity metric failing to exceed the pre-defined threshold in block404, the terminal device may ignore the bone conduction and audio signals (i.e., do nothing) as it is highly probable that the audio signal and the bone conduction signal do not contain any speech of the user.

As mentioned above, the process ofFIG.4Bcorrespond to a process carried out by a wearable terminal device while the (non-wearable) terminal device (e.g., a smart speaker) performs the process ofFIG.4A. The process ofFIG.4Bis notably simpler compared to the process ofFIG.3as some of the functionalities have been “outsourced” to the (non-wearable) terminal device.

Referring toFIG.4B, the wearable terminal device receives, in block411, a bone conduction signal via the bone conduction sensor of the wearable terminal device.

Optionally, the wearable terminal device may also receive, in block412, an audio signal via at least one microphone of the wearable terminal device. The bone conduction signal and the audio signal are, at least in part, substantially concurrently recorded signals. As described above, the audio signal may be, alternatively or additionally, recorded by the terminal device using its own microphone(s).

The wearable terminal device transmits, in block413, at least the bone conduction signal to the terminal device via said at least one wireless or wired communication link or network. Optionally, also the audio signal recorded by the wearable terminal device may be transmitted in block413(assuming that such a signal was recorded in the first place).

FIG.5illustrates another process according to embodiments for performing actions based on the detected voice of the user. The process ofFIG.5may be carried out by a wearable terminal device comprising or being electrically connected to a bone conduction sensor and at least one microphone, such as the wearable terminal device201ofFIG.2, or a part thereof. The wearable terminal device may specifically be assumed to be worn by a user around a wrist, around a finger, on a hand or on an arm in the embodiments discussed in connection withFIG.5. It should be noted that no further device (e.g., the terminal device205ofFIG.2) is required for carrying out the embodiment ofFIG.5. The wearable terminal device in question may correspond to either of terminal device100,102ofFIG.1. In the following discussion, the apparatus carrying out the process is called the wearable terminal device without loss of generality.

The process ofFIG.5may be considered one more detailed implementation of the process ofFIG.3discussed above. The initial steps of the process ofFIG.5illustrated with blocks501to504may correspond fully to actions described above in connection with blocks301to304ofFIG.3and are thus not repeated for brevity.

Following the determination in block504that the value of the similarity metric exceeds the pre-defined threshold (indicating that the bone conduction and the audio signal comprise voice of the user), the wearable terminal device determines, in block505, whether a (known) voice command is comprised in the audio and/or bone conduction signal.

In response to detecting the voice command in the audio and/or bone conduction signal in block505, the wearable terminal device estimates, in block506, a distance of a mouth of the user from the wearable terminal device based on (relative) amplitudes (or signal levels) or delays of the audio signal and the bone conduction signal (or difference thereof). This estimation may be based on the known rate at which the amplitude of the acoustic wave attenuates as it propagates in air and/or bone and/or the known speed of the acoustic wave in air and/or bone.

The estimation in block506may be based on the following considerations. The bone conduction signal has close to the same amplitude irrespective of the hand and/or arm position of the user of the wearable terminal device (attached to the arm or hand) as the distance that the conduction signal has to travel through the body of the user is always (at least approximately) the same. However, the amplitude of the audio signal and the delay of the audio signal varies significantly, as acoustic waves travel over the air and the distance between the mouth of the user differ considerably if the user is holding the wearable terminal device next to his/her mouth and when the user is holding the wearable terminal device as far away from his/her mouth as possible. The relative amplitude (or signal level) and/or delay difference of the audio signal (from the mouth of the user to the at least one microphone of the wearable terminal device) and the bone conduction signal (from the mouth of the user to the bone conduction sensor of the wearable terminal device) may be used, in block506, for estimating how far the wearable terminal device is from the mouth of the user. If the delays for the audio and bone conduction signals are approximately the same, the user may be assumed to be holding his arm straight. If the audio signal arrives sooner than bone conduction signal, the user may be assumed to be holding his arm folded so that the wearable terminal device is close to his/her mouth. If the amplitude of the audio signal is higher than normal when compared to the amplitude of the bone conduction signal, the user may be assumed to be holding the wearable terminal device close to his/her mouth.

It should be noted that this way of estimating the distance provides the benefit of being capable of distinguishing between the case where the wearable terminal device is far from the mouth of the user and the case where the user is just talking quietly. The voice level of the user does not affect the result of determination in block506as only the difference between the bone conduction and audio signal levels or delays is used (not the raw signals as such).

In response to the estimated distance being within a first pre-defined distance range in block507, the wearable terminal device executes, in block508, the voice command based on the audio signal and/or the bone conduction signal by performing a first function. The first pre-defined distance range may be defined to comprise distances from a first pre-defined lower distance to a first pre-defined upper distance. The first lower distance may be zero. In other words, the first function may be performed if the wearable terminal device is positioned (or held) sufficiently near to the mouth of the user when the voice command is given (e.g., the user may be speaking directly to the wearable terminal device). The first function may, in general, correspond to any of the actions listed in connection with block305ofFIG.3.

The first pre-defined upper distance of the first pre-defined distance range may be defined so that it may be assumed that the mouth of the user is within a viewing distance of a display of the wearable terminal device when the estimated distance is within the first pre-defined distance range. The first function may, then, comprise displaying information on the display of the wearable terminal device. For example, if the user says “What's the weather like?”, the wearable terminal device may display a weather report on the display of the wearable terminal device (where the weather report may be based on an Internet query).

In response to the estimated distance not being within the first pre-defined distance range in block507, the process proceeds to block509. At least in some embodiments, the process may proceed to block509also following the execution of the voice command by performing the first function in block508. In some alternative embodiments (not shown inFIG.5), the process may proceed back to block501following the execution of the voice command by performing the first function in block508.

In response to the distance being within a second pre-defined distance range at least partially different from the first pre-defined distance range in block509, the wearable terminal device executes, in block510, the voice command based on the audio signal and/or the bone conduction signal by performing a second function different from the first function or ignoring the voice command. The second pre-defined distance range may be defined to comprise distances from a second pre-defined lower distance to a second pre-defined upper distance (a bounded range) or to infinity (a half-bounded range). The first and second pre-defined distance ranges may be defined to be overlapping or non-overlapping. In the latter case, the second pre-defined distance range may be defined to be above the first pre-defined upper distance.

To give a more specific example, the second pre-defined distance range may be a half-bounded distance range comprising all distances not within the first pre-defined distance range (which is defined to start at zero). The first and second pre-defined distance ranges may, thus, encompass together all (non-negative) distances. The second function may, then, be performed always when the wearable terminal device is positioned (or held) sufficiently far from the mouth of the user when the voice command is given.

For example, if the user says the aforementioned phrase “What's the weather like?” while the wearable terminal device is positioned far from the mouth of the user, the wearable terminal device may provide a voice response comprising a weather report using a voice synthesizer (where the weather report may be based on an Internet query), instead or in addition to displaying corresponding information on the display of the wearable terminal device. Thus, in general, if the voice command is of a first type (such as “What's the weather type?”), the first function may be defined as causing displaying information on a screen of the wearable terminal device and/or the second function may be defined as playing a sound or speech via at least one speaker comprised in or electrically and/or communicatively connected to the wearable terminal device, where said information or said sound or speech is dependent on the voice command. Said at least one speaker may comprise at least one internal speaker of the wearable terminal device and/or at least one external speaker such as a dedicated speaker paired with the wearable terminal device (e.g., using Bluetooth) or a (terminal) device electrically or communicatively connected to the wearable terminal device (being, e.g., a wearable terminal device adapted to be worn around a wrist, on a hand, around a finger or on an arm). To give an example of the latter alternative, said at least one speaker may comprise at least one speaker of wired, wireless or truly wireless headphones or a wired, wireless or truly wireless earpiece worn by the user.

It may be assumed at least with certain voice commands that if the voice command is given when the wearable terminal device is held close to the mouth of the user, the voice command is directed directly towards the wearable terminal device and thus immediate action should be taken. For example, if the user says: “How can I increase volume?” when the wearable terminal device is positioned close to the mouth of the user (block507, yes), the wearable terminal device may just increase the volume of the wearable terminal device in block508. On the other hand, if the user says: “How can I increase volume?” when the wearable terminal device is positioned far from the mouth of the user (block507, yes), the wearable terminal device may present search results for increasing volume in at least one device other than the wearable terminal device in block510. Thus, in general, if the voice command is of a second type (such as “How can I increase volume?”), the first function may be defined as directly executing the voice command (i.e., carrying out the task explicitly defined in the voice command) and/or the second function may be defined as causing displaying information on a screen of the wearable terminal device, where said displayed information is dependent on the voice command.

To give yet another example of this scenario, if the user says: “Stop that!” when the wearable terminal device is positioned close to the mouth of the user (block507, yes), the wearable terminal device may terminate or interrupt one or more of its present on-going actions (e.g., stop playing music or hang up a phone call). On the other hand, if the user says: “Stop that!” when the wearable terminal device is positioned far from the mouth of the user (block507, yes), the wearable terminal device may assume that the voice command was not addressed to the wearable terminal device and simply ignore the voice command in block510.

In some embodiments, the relative order between blocks507&508and blocks509&510may be different (i.e., the second pre-defined distance range may evaluated first).

In some embodiments, only one of the features discussed in connection with blocks507&508and with blocks509&510may be implemented. For example, the wearable terminal device may be configured, in some embodiments, only to react to voice commands which correspond to the first or second pre-defined distance range.

In some embodiments, blocks508and510may be swapped. In other words, the voice command may be executed by performing the second function or alternatively the voice command may be ignored following the positive evaluation in block507(using the first pre-defined distance range) and the voice command may be executed by performing the first function following the positive evaluation in block509(using the second pre-defined distance range).

FIG.6shows an apparatus601(e.g., a computing device) according to some embodiments. The apparatus601may be a wearable terminal device (e.g., the terminal device100,102ofFIG.1and/or the wearable terminal device201ofFIG.2) or a part thereof.FIG.6may illustrate an apparatus configured to carry out at least the functions described above in connection with the wearable terminal device. The apparatus601may comprise one or more control circuitry620, such as at least one processor, and at least one memory630, including one or more algorithms, such as a computer program code (software631) wherein the at least one memory and the computer program code (software) are configured, with the at least one processor, to cause, respectively, the apparatus to carry out any one of the exemplified functionalities of the wearable terminal device as described above.

Referring toFIG.6, the control circuitry620of the apparatus601comprises at least bone conduction measurement circuitry621. The bone conduction measurement circuitry621may be configured to carry out at least the functionalities described above by means of any of block301ofFIG.3, block402ofFIG.4Aand block501ofFIG.5using one or more individual circuitries. The control circuitry620of the apparatus601further comprises audio processing and control circuitry622. The audio processing and control circuitry622may be configured to carry out any of the functionalities described above in connection with any ofFIGS.3,4B and5, apart from said block301ofFIG.3, block402ofFIG.4Aand block501ofFIG.5, using one or more individual circuitries.

The at least one memory630may comprise at least one database632which may comprise, for example, information on the pre-defined threshold for the similarity metric. Each memory630may comprise software631and at least one database632. The memory630may also comprise other databases which may not be related to the functionalities of the apparatus according to any of presented embodiments. The at least one memory630may be implemented using any suitable data storage technology, such as semiconductor based memory devices, flash memory, magnetic memory devices and systems, optical memory devices and systems, fixed memory and removable memory.

Referring toFIG.6, the apparatus601may further comprise different interfaces610such as one or more communication interfaces (TX/RX) comprising hardware and/or software for realizing communication connectivity over one or more communications network according to one or more communication protocols. Specifically, the one or more communication interfaces610may provide the apparatus with communication capabilities to communicate in one or more mobile network and enable communication with one or more access nodes, one or more terminal devices (via said plurality of access nodes or directly), one or more other network nodes or elements and/or an apparatus701ofFIG.7. The one or more communication interfaces610may comprise standard well-known components such as an amplifier, filter, frequency-converter, analog-to-digital converts, (de)modulator, and encoder/decoder circuitries, controlled by the corresponding controlling units, and one or more antennas.

FIG.7shows an apparatus701according to some embodiments. The apparatus701may be a terminal device (e.g., the terminal device100,102ofFIG.1and/or the terminal device205ofFIG.2) or a part thereof. The apparatus701may be configured to carry out at least the functions described above in connection with the (non-wearable) terminal device communicatively connected to a wearable terminal device. The apparatus701may comprise one or more control circuitry720, such as at least one processor, and at least one memory730, including one or more algorithms731, such as a computer program code (software) wherein the at least one memory and the computer program code (software) are configured, with the at least one processor, to cause the apparatus to carry out any one of the exemplified functionalities of the terminal device described above.

Referring toFIG.7, the control circuitry720of the apparatus comprises at least audio processing & control circuitry721. The audio processing & control circuitry721may be configured to carry out at least some of the functionalities described above by means ofFIG.4Ausing one or more individual circuitries.

The at least one memory730may comprise at least one database732which may comprise, for example, information on a pre-defined threshold for the similarity metric. Each memory730may comprise software and at least one database. The at least one memory730may also comprise other databases which may not be related to the functionalities of the apparatus according to any of presented embodiments. The at least one memory730may be implemented using any suitable data storage technology, such as semiconductor based memory devices, flash memory, magnetic memory devices and systems, optical memory devices and systems, fixed memory and removable memory.

Referring toFIG.7, the apparatus may further comprise different interfaces710such as one or more communication interfaces (TX/RX) comprising hardware and/or software for realizing communication connectivity over one or more communications network according to one or more communication protocols. Specifically, the one or more communication interfaces710may provide the apparatus with communication capabilities to enable communication with one or more terminal devices, one or more wearable terminal devices, one or more access nodes and/or an apparatus601ofFIG.6. The one or more communication interfaces710may comprise standard well-known component(s) such as an amplifier, filter, frequency-converter, analog-to-digital converts, (de)modulator, and encoder/decoder circuitries, controlled by the corresponding controlling units, and/or one or more antennas.

As used in this application, the term ‘circuitry’ may refer to one or more or all of the following: (a) hardware-only circuit implementations, such as implementations in only analog and/or digital circuitry, and (b) combinations of hardware circuits and software (and/or firmware), such as (as applicable): (i) a combination of analog and/or digital hardware circuit(s) with software/firmware and (ii) any portions of hardware processor(s) with software, including digital signal processor(s), software, and memory(ies) that work together to cause an apparatus, such as a terminal device or an access node, to perform various functions, and (c) hardware circuit(s) and processor(s), such as a microprocessor(s) or a portion of a microprocessor(s), that requires software (e.g. firmware) for operation, but the software may not be present when it is not needed for operation. This definition of ‘circuitry’ applies to all uses of this term in this application, including any claims. As a further example, as used in this application, the term ‘circuitry’ also covers an implementation of merely a hardware circuit or processor (or multiple processors) or a portion of a hardware circuit or processor and its (or their) accompanying software and/or firmware. The term ‘circuitry’ also covers, for example and if applicable to the particular claim element, a baseband integrated circuit for an access node or a terminal device or other computing or network device.

In embodiments, the at least one processor, the memory, and the computer program code form processing means or comprises one or more computer program code portions for carrying out one or more operations according to any one of the embodiments ofFIGS.2,3,4A,4B and5or operations thereof.

In an embodiment, at least some of the processes described in connection with ofFIGS.2,3,4A,4B and5may be carried out by an apparatus comprising corresponding means for carrying out at least some of the described processes. Some example means for carrying out the processes may include at least one of the following: detector, processor (including dual-core and multiple-core processors), digital signal processor, controller, receiver, transmitter, encoder, decoder, memory, RAM, ROM, software, firmware, display, user interface, display circuitry, user interface circuitry, user interface software, display software, circuit, antenna, antenna circuitry, and circuitry. In an embodiment, the at least one processor, the memory, and the computer program code form processing means or comprises one or more computer program code portions for carrying out one or more operations according to any one of the embodiments ofFIGS.2,3,4A,4B and5or operations thereof.