Patent Description:
<CIT> is a prior art example of a mobile microphone assembly and to a method for capturing audio signals from sound by using such microphone assembly. <CIT> is a prior art example of a microphone assembly to be worn at a user's chest.

The purpose of the invention is to provide an audio streaming device for streaming an audio signal to a hearing assistive device and protecting the user against annoying audio caused by handling the audio streaming device.

This purpose is achieved according to the teaching of claim <NUM>. According to a second aspect of the invention, there is provided a method of managing an audio streaming device according to claim <NUM>. The dependent claims define various embodiments.

The invention will be described in further detail with reference to preferred aspects and the accompanying drawing, in which:.

<FIG> shows in perspective an audio streaming device <NUM> according to one embodiment of the invention. The audio streaming device <NUM> is in the illustrated embodiment disc-shaped but may in other embodiments assume other shapes serving the purpose. The audio streaming device <NUM> has a plurality of microphones <NUM>, and in the illustrated example, the number is three. In other examples, the number may be higher or lower. When the audio streaming device <NUM> performs a kind of beamforming, the processing relies on a predetermined geometry, which may be as illustrated in <FIG>. Here the center of the audio streaming device <NUM> is marked as the starting point for a vector <NUM>, and the center point is equidistant from the three microphones <NUM> as the distances between center point and each of the three microphones <NUM> are equal. Furthermore, an angle, α, defined by lines passing through the center point and respective microphones <NUM>, is the same, <NUM>°. The lines are equiangular as all the lines intersect at center point and makes the <NUM>° angle.

The audio streaming device <NUM> may however be formed as a bar with several microphones aligned, as a semi sphere or as a ball structure with multiple (+<NUM>) microphones integrated.

A vector <NUM> marks a direction for beamforming performed by the audio streaming device <NUM>. In some embodiments, the audio streaming device <NUM> aims to pick up audio originating from this direction and aims to remove interferences introduced by noise and reverberation originating from other directions. Beamforming can be considered as multidimensional filtering in space and time and is a method of signal processing involving spatially distributed sensors. By means of beamforming, the audio streaming device <NUM> aims to place a virtual microphone at various positions without physical movement. Those virtual microphones are useful for applications like conference telephony and for companion microphones picking up speech during a meeting and immediately streaming the audio signal to a set of hearing aids worn by one or more hearing impaired persons present in the meeting room. Several beamforming algorithms exist for combining the audio data, and these often rely on passing the audio signal through digital filters.

<FIG> illustrates a side view of the audio streaming device <NUM> shown in <FIG> tilted relatively to ground. For this purpose, the audio streaming device <NUM> has a rest <NUM> that can be attached when need or is pivotally secured to the audio streaming device <NUM>. The rest <NUM> may also serve the purpose as a clip for mounting the audio streaming device <NUM> to cloths, whereby the audio streaming device <NUM> can be used as a companion microphone for a hearing aid user. The rest tilts the audio streaming device <NUM> relatively to a base surface <NUM> which affects the angle between the beam direction <NUM> and the plane defined by the three microphones <NUM>.

<FIG> illustrates schematically one embodiment of the audio streaming device <NUM> and one hearing assistive device <NUM> according to one embodiment of the invention. The audio streaming device <NUM> has a plurality (e.g. <NUM>) of microphones <NUM>. The sound picked up is converted by the microphones <NUM> into an electric audio signal which by means of respective A/D converters <NUM> is converted into digital signal. The A/D converters <NUM> are in some embodiments provided by Delta-Sigma converters. The three digital audio signals originating from respective microphones are then fed to a beamforming digital signal processor <NUM> which processes the captured audio signals in a manner to create several acoustic beams having directions uniformly spread above the plane defined by the three microphones <NUM>. The microphones <NUM> are in one embodiment omnidirectional microphones.

In the beamforming digital signal processor <NUM>, uniformly spread beams are produced by delay-and-sum beamforming of the audio signals of pairs of the microphones <NUM>, by applying an appropriate phase difference.

The audio streaming device <NUM> further comprises an acceleration sensor <NUM>, e.g. a <NUM>-axis accelerometer, and a microprocessor <NUM>. The acceleration sensor <NUM> outputs a measure for the acceleration of the audio streaming device <NUM> along three orthogonal axes. <FIG> illustrates an example of such a measure, and it is seen that there is a linear relationship between acceleration and output voltage, when the acceleration in the direction measured is within the dynamic range of the accelerometer. This counts for accelerometer output for all three axes. The microprocessor <NUM> receives the accelerometer output for all three axes, and based on this, the microprocessor <NUM> calculates the overall acceleration acting on the audio streaming device <NUM> and the direction of the overall acceleration. When the audio streaming device <NUM> rests on e.g. a table, gravity will be the only contribution to the acceleration, and the orientation of the audio streaming device <NUM> can by means of the overall acceleration measure be determined relatively to the gravity, can be determined by combining the amount of acceleration measured along each axis. These data are forwarded to the beamforming digital signal processor <NUM> for use in the beam selection.

The beamforming digital signal processor <NUM> furthermore includes a functionality for estimating the quality of the speech present in available beams. The beamforming digital signal processor <NUM> furthermore comprises a beam selection functionality for selecting the one of the uniformly spread beams produced by delay-and-sum beamforming that best fulfils the criteria set for the desired beam based on the speech quality estimation and the direction of the audio source. Once the desired beam has been selected, the beamforming digital signal processor <NUM> is adapted to adaptively adjust the applied phase difference for the beamforming in order to maximize the speech quality present in the selected beam.

The beamforming digital signal processor <NUM> outputs a processed audio signal to an encoder and packetizing unit <NUM> in which the processed audio signal is compressed and encoded according to a predefined streaming media audio coding format in order to represent the audio signal with minimum number of bits while retaining quality. This effectively reduces the bandwidth required for transmission of the audio stream. The encoded audio signal is then placed as payload in data packets delivered a radio <NUM> modulating and amplifying the data packets for transmission. In one embodiment, the audio stream is transmitted according to the Bluetooth™ Core Specification Version <NUM>, where the audio streaming device <NUM> act as audio source for one or more persons having respective audio sink devices.

The microprocessor <NUM> receives the accelerometer output for all three axes, and calculates the overall acceleration acting on the audio streaming device <NUM>. From these values, the microprocessor <NUM> determines accelerometer measures defining a state in which the audio streaming device <NUM> has rested stabile on a surface for period, e.g. for more than <NUM> seconds. Then, the microprocessor <NUM> compares at least one of the detected accelerometer measures to the accelerometer measures defining the state in which the audio streaming device <NUM> is stationary. In case the comparison exceeds a first predetermined threshold, the microprocessor <NUM> assumes that the audio streaming device <NUM> is moved on the surface or dropped therefrom. This may create noise that will be streamed to the hearing aids.

In order to avoid transmitting annoying audio to one or more hearing aids, the microprocessor <NUM> is connected to a shock protection unit <NUM> for interfering the audio stream transmission when justified. In the embodiment illustrated in <FIG>, the encoder and packetizing unit <NUM> provides a copy of the data packets delivered to the radio <NUM> to the shock protection unit <NUM>, too. When the microprocessor <NUM>, based on the output from the acceleration sensor <NUM>, identifies an increased risk that the audio stream may contain an acoustic shock that shall be prevented from reaching the hearing aid user, the shock protection unit <NUM> may instruct the radio <NUM> to discard the next data packet from the encoder and packetizing unit <NUM> and use a replacement data packet from the shock protection unit <NUM> instead.

The replacement data packet from the shock protection unit <NUM> may in one embodiment be a copy of the previous data packet delivered to the radio <NUM> which have been buffered in the shock protection unit <NUM>. In one embodiment, the shock protection unit <NUM> has an audio classifier classifying the audio sample an delivers a pre-stored audio sample matching the audio classification of the previous data packet delivered from the encoder and packetizing unit <NUM>. In both embodiments, it is the shock protection unit <NUM> that compensates for the missing audio packets by means of packet loss concealment (PLC) on the transmitter side (in the audio streaming device <NUM>).

In yet another embodiment not forming part of the invention, the shock protection unit <NUM> simply disables the radio <NUM> until the risk of sending an acoustic shock is over. Then it is up to the controller <NUM> to compensate for the missing audio packets by means of packet loss concealment (PLC) on the receiver side (in the hearing aid <NUM>).

The hearing aid <NUM> has at least one input transducer or microphone <NUM> picking up an audio signal. The audio signal is digitized in an A/D converter <NUM>, e.g. a Delta-Sigma converter, and fed to digital signal processor <NUM> adapted for amplifying and conditioning of the audio signal intended to become presented for the hearing aid user. The amplification and conditioning are carried out according to a predetermined setting stored in the hearing aid <NUM> to alleviate a hearing loss by amplifying sound at frequencies in those parts of the audible frequency range where the user suffers a hearing deficit. The amplified and conditioned audio signal is reproduced for the user via a receiver or speaker <NUM>. The at least one microphone <NUM>, the A/D converter <NUM>, the digital signal processor <NUM>, and speaker <NUM> provides an audio signal path with hearing loss alleviation.

Furthermore, the hearing aid <NUM> includes a radio <NUM> adapted for receiving and demodulating the audio stream received as data packets. The radio <NUM> may be used for inter-ear communication, or for communication with another remote device, such as the smartphone. From the radio <NUM>, the audio stream passes a decoder and depacketizing unit <NUM> in which the compressed data stream becomes unpacked and decoded again. The received audio is hereafter loaded into the digital signal processor <NUM>.

A controller <NUM> controls the reception of data packets and is among other responsible for packet loss concealment (PLC) which is a technique to mask the effects of packet loss in audio over IP communication. Due to multipath propagation, individual packet may be subject to poor signal to noise ratios (SNR) and therefor be corrupted by the receiver. Packet loss concealment includes a method for accounting for and compensating for the loss of voice packets by replacing the lost packet with audio content corresponding to the recently received audio packet, either playing the latest received packet once more or synthetizing an audio stream segment based on the audio packets. The controller <NUM> also controls mixing of the received audio stream and the audio present in the audio signal path of the hearing aid <NUM>.

<FIG> illustrates the output from one of the channels in an Inertial Sensor System or an Inertial Measurement Unit (IMU), such as the acceleration sensor <NUM>, e.g. a <NUM>-axes accelerometer. The Inertial Measurement Unit is an electronic sensor device that provides an output from which orientation, velocity, and gravitational forces of the electronic sensor device can be calculated. By measuring the amount of acceleration due to gravity, an accelerometer can figure out the angle it is tilted at with respect to horizontal. Some accelerometers use piezoelectric effect in crystal structures that get stressed by accelerative forces generating a voltage across the structures. Alternatively, the accelerometer may operate by sensing changes in capacitance. For use as a tilt sensor and for detecting drop of the audio streaming device <NUM>, a dynamic range of ±<NUM>,<NUM> is sufficient.

When the audio streaming device <NUM> is positioned on a stable surface, the audio streaming device <NUM> and thereby the acceleration sensor <NUM> will not observe any acceleration and the acceleration sensor <NUM> will output a voltage, V<NUM>, corresponding to an acceleration equal to <NUM>. It is seen that there is a linear relationship between the observed acceleration and the output voltage from the acceleration sensor <NUM>, if the numerical value of the acceleration observed is below e.g. <NUM> (twice the gravity). The output voltage will vary between V<NUM> and V<NUM>.

From <FIG>, it is seen that a small deviation of the resulting acceleration equal to <NUM> will result in corresponding small deviation from the voltage, V<NUM>. According to the invention, small accelerations, e.g. caused by small vibrations and person walking in the room, will not affect the audio streaming. However, when observing larger accelerations, e.g. caused by sliding the audio streaming device <NUM> along the surface on which it rests, the microprocessor <NUM> will analyze the output voltage from the acceleration sensor <NUM>. When the output voltage differs more than ΔVTH from the voltage, V<NUM>, corresponding to an acceleration equal to <NUM>, the acceleration is deemed to potentially cause an annoying sound, the streaming by the transmitter <NUM> will be ceased by the microprocessor <NUM>.

In one embodiment, the first threshold value corresponds to an acceleration of <NUM>,<NUM>*g, where g is the gravity. In some embodiments, the first threshold value corresponds to an acceleration of <NUM>,<NUM>*g, and in other embodiments the first threshold value corresponds to an acceleration of <NUM>,<NUM>*g or <NUM>,<NUM>*g. In order to prevent transmission of an audio stream containing noise due to scratching or the like, it is important apply a first threshold value is below a value corresponding to <NUM>*g as this will represent a free fall.

<FIG> illustrates a flowchart for a decision process used in the microprocessor in the audio streaming device <NUM> according to the invention. When the audio streaming device <NUM> is placed stable to a surface <NUM>, it starts as indicated in step <NUM> by picking up audio signal from the environment by means of the microphones <NUM>. In parallel hereto, the acceleration sensor <NUM>, e.g. a <NUM>-axes accelerometer, picks up inertial sensor data in step <NUM>, and the microprocessor <NUM> compares sensor data to a first threshold in step <NUM>. The sensor data may as discussed above consist of sensor data for each of the three axes and the total acceleration calculated from sensor data for the three axes. Any combination of the sensor data available may be compared to appropriate thresholds for detecting scratching, movements, and drops of the audio streaming device <NUM>. In case the microprocessor <NUM> detects that the comparison that the sensor data exceeds the first threshold in step <NUM>, the microprocessor <NUM> disables the transmission of the audio picked up in step <NUM>. This is also the case if only one of the four parameters exceed the relevant predefined threshold.

At step <NUM>, the microprocessor <NUM> compares the sensor data to a second threshold. This comparison is to ensure that the audio streaming device <NUM> is not moved. In case the audio streaming device <NUM> moves in an unpredictive manner, it does not make sense to adapt the beamforming provided in the audio streaming device <NUM> during the move, why the microprocessor <NUM> disables the adaptive beamforming in step <NUM>. Then the audio picked up by the microphones <NUM> and processed by the beamforming digital signal processor <NUM> based on the parameters determined prior to the movement detected will be encoded and transmitted in step <NUM>. Once the audio streaming device <NUM> has stopped moving and the microprocessor <NUM> identifies the sensor data to be below the second threshold, the microprocessor <NUM> instructs the beamforming digital signal processor <NUM> to apply adaptive beamforming again in step <NUM>. Hereafter the audio signal will be encoded and transmitted in step <NUM>.

The audio streaming device <NUM> continues picking up audio and monitoring inertial sensor data and starts using the audio data for streaming once the microprocessor <NUM> in step <NUM> detects that the sensor data is below the first threshold.

<FIG> illustrates a state diagram for audio streaming device <NUM> according to one embodiment of the invention. The audio streaming device <NUM> has a stable state <NUM> in which it acts as remote microphone, e.g. placed on a table (horizontal and no movement for a period). When movement corresponding to touching the audio streaming device <NUM> is detected with the acceleration sensor <NUM>, and the microprocessor <NUM> observes that the sensor data exceeds a first threshold, the audio streaming device <NUM> enters an unknown state <NUM> in which audio streaming of the signal picked up by means of the microphones <NUM> is interrupted. The microprocessor <NUM> mutes the microphone <NUM> and/or filters out low frequencies for a short period. In some embodiments, the audio streaming device <NUM> applies packet loss concealment techniques on the transmitter side for masking the interrupted audio signal. The packet loss concealment techniques include Zero insertion, Waveform substitution (the missing gap is reconstructed by repeating a portion of already transmitted audio signal), or Model-based methods (an algorithm applies speech models for interpolating and extrapolating speech gaps).

In some embodiments, the transmission of the audio stream is interrupted when the audio streaming device <NUM> is in the unknown state <NUM>. Then it is up to the radio <NUM> in the hearing aid <NUM> to apply packet loss concealment techniques on the receiver side.

When using a wireless technology standard, such as Bluetooth™, for exchanging data, the transmitted microphone audio will have a latency greater than <NUM> due to the applied codec. By entering the unknown state <NUM>, the transmission of the audio stream is interrupted for e.g. <NUM>-<NUM> leaving package loss concealment on the receiver or the transmitter side to clean up the gap.

In one embodiment, a timer in the microprocessor <NUM> is used for setting a predefined period. If an event detected by the accelerometer <NUM> is over at the expiry of this predefined period, the audio streaming device <NUM> reverts to the stable state <NUM>.

High-pass filtering of the audio signal picked up by the microphones <NUM> is initiated once the event is detected and the unknown state <NUM> is entered, and once the high-pass filtered audio reaches a processed stage, e.g. after <NUM>-<NUM>, the transmission of the audio stream is resumed. The filtering of microphone audio continues until a timeout is reached for either accelerometer movement or total duration of the event. A timer for tabletop detection is reset to eliminate repeat events.

<FIG> also illustrates that the audio streaming device <NUM>, when being in the stable state <NUM> in which it acts as remote microphone, applies adaptive beamforming as the beamforming digital signal processor <NUM> is adapted to adaptively adjust the applied phase difference for the beamforming in order to maximize the speech quality present in the selected beam.

Claim 1:
An audio streaming device for streaming an audio signal to at least one hearing assistive device (<NUM>), and comprising
- at least one input transducer (<NUM>),
- an electronic sensor device (<NUM>) adapted for sensing gravitational forces acting on the audio streaming device (<NUM>), and
- a transmitter (<NUM>) adapted for streaming audio from the at least one input transducer (<NUM>) to the at least one hearing assistive device (<NUM>),
wherein the audio streaming device (<NUM>) further comprises a microprocessor (<NUM>) adapted for:
- comparing an output signal from the electronic sensor device (<NUM>) to a first threshold value, and
- ceasing streaming by the transmitter (<NUM>) of the audio stream from the at least one input transducer (<NUM>), when the output signal from the electronic sensor device (<NUM>) exceeds the first threshold value; and
wherein the transmitter (<NUM>) is adapted for streaming audio as data packets, and characterized in that
a shock protection unit (<NUM>) is adapted to instruct the transmitter (<NUM>) to discard the next data packet and use a replacement data packet from the shock protection unit (<NUM>) instead, when the output signal from the electronic sensor device exceeds the first threshold value.