Patent Description:
<CIT> discloses a talker prediction device that predicts a next talker among a plurality of people.

<CIT> discloses a speech pattern detector that can predict what talker is going to talk next among a plurality of talkers. The speech pattern detector may receive camera and microphone inputs such that the speech pattern detector can accurately detect participants' physical placement relative to each other and detect who is actively speaking.

It is desired to predict the next talker with higher accuracy while the current talker is talking.

In view of the foregoing, an object of an embodiment of the present disclosure is to provide a talker prediction method according to claim <NUM> and a talker prediction device according to claim <NUM> that are capable of predicting a next talker with high accuracy while the current talker is talking.

A talker prediction method obtains a voice from a plurality of talkers, and records a conversation history of the plurality of talkers. The talker prediction method identifies a talker of the obtained voice, and predicts a next talker among the plurality of talkers, based on the identified talker and the conversation history.

According to an embodiment of the present disclosure, while the current talker is talking, a next talker is able to be predicted with high accuracy.

<FIG> is a block diagram showing a configuration of an audio signal processing apparatus <NUM>. The audio signal processing apparatus <NUM> includes a camera <NUM>, a CPU <NUM>, a DSP <NUM>, a flash memory <NUM>, a RAM <NUM>, a user interface (I/F) <NUM>, a speaker <NUM>, six microphones 18A to 18F, and a communicator <NUM>. It is to be noted that, in the present embodiment, a signal means a digital signal.

The camera <NUM>, the speaker <NUM>, and the microphones 18A to 18F are disposed above or below a display (not shown), for example. The camera <NUM> obtains an image of a user present in front of the display (not shown). The microphones 18A to 18F obtain a voice of the user present in front of the display (not shown). The speaker <NUM> outputs a sound to the user present in front of the display (not shown). It is to be noted that the number of microphones is not limited to six. The number of microphones may be one. The number of microphones of the present embodiment is six, which configures an array microphone. The DSP <NUM> performs beamforming processing on an audio signal obtained by the microphones 18A to 18F.

The CPU <NUM> reads an operating program from the flash memory <NUM> to the RAM <NUM> and functions as a controller that collectively controls the operations of the audio signal processing apparatus <NUM>. It is to be noted that the program does not need to be stored in the flash memory <NUM> of the own device. The CPU <NUM> may download the program each time from a server or the like, for example, and may read out the program to the RAM <NUM>.

The DSP <NUM> is a signal processor that processes each of a video signal and an audio signal according to the control of the CPU <NUM>. The DSP <NUM> functions as an image processor that performs framing processing in which an image of a talker is extracted from a video signal, for example. In addition, the DSP <NUM> also functions as an audio signal processor that performs beamforming, for example.

The communicator <NUM> sends a video signal and an audio signal on a near-end side that have been processed by the DSP <NUM>, to a different apparatus. The different apparatus is an audio signal processing apparatus on a far-end side to be connected through the Internet or the like. In addition, the communicator <NUM> receives a video signal and an audio signal from the different apparatus. The communicator <NUM> outputs the received video signal to a display (not shown). The communicator <NUM> outputs the received audio signal to the speaker <NUM>. The display displays an image obtained by a camera of the different apparatus. The speaker <NUM> outputs the voice of a talker obtained by the microphone of the different apparatus. As a result, the audio signal processing apparatus <NUM> functions as a communication system to perform a voice conversation with a person at a remote location.

<FIG>, <FIG>, and <FIG> are functional block diagrams of the audio signal processing apparatus <NUM>. <FIG>, <FIG>, and <FIG> are flow charts showing an operation of the audio signal processing method. The functional configuration shown in <FIG>, <FIG>, and <FIG> is implemented by the CPU <NUM> and the DSP <NUM>.

The audio signal processing apparatus <NUM> functionally includes a voice obtainer <NUM>, an image obtainer <NUM>, a near-end talker identifier <NUM>, a conversation history recorder <NUM>, a far-end talker identifier <NUM>, a camera image controller <NUM>, and a predictor <NUM>.

The near-end talker identifier <NUM> and the far-end talker identifier <NUM> correspond to a talker identifier of the present disclosure. <FIG> shows a configuration in a case in which talker identification is performed based on a video signal and an audio signal on a near-end side and recorded as conversation history, and <FIG> shows the operation in such case.

The voice obtainer <NUM> receives an input of an audio signal from the microphones 18A to 18F as a near-end audio signal (S11). In addition, the image obtainer <NUM> obtains a video signal from the camera <NUM> as a near-end talker image (S12). The near-end talker identifier <NUM> identifies a talker on a near-end side, based on the near-end audio signal and the near-end talker image (S13). The conversation history recorder <NUM> records an identifier that indicates the talker identified by the near-end talker identifier <NUM> and the order of talkers, as conversation history (S14).

The near-end talker identifier <NUM> identifies a talker, based on a voice feature amount, for example. The voice feature amount is unique information different for each talker. The voice feature amount includes an average volume level (power), a highest pitch frequency, a lowest pitch frequency, or a speech speed, for example. The near-end talker identifier <NUM> extracts the voice feature amount from an audio signal and identifies a talker, by a predetermined algorithm using a neural network or the like, for example. In addition, the near-end talker identifier <NUM> may identify a talker, based on an arrival direction of a voice. For example, the near-end talker identifier <NUM> is able to determine a timing at which the voice of a talker has reached a microphone by determining a cross correlation of audio signals obtained by a plurality of microphones. The near-end talker identifier <NUM> is able to determine the arrival direction of the voice of a talker, based on the positional relationship of each microphone and the arrival timing of the voice.

In addition, the near-end talker identifier <NUM> may identify a talker by performing face recognition of the talker. The face recognition of a talker is processing of recognizing a face position of a talker from a near-end talker image, by using a predetermined algorithm using the neural network or the like, for example.

<FIG> is a view showing an example of an image captured by the camera <NUM>. In the example of <FIG>, the camera <NUM> captures a face image of each of a plurality of users in the longer direction (the depth direction) of a table T.

The table T has a rectangular shape in a plan view. The camera <NUM> captures four users on the left side and the right side of the table T in the shorter direction, and a user at a position far from the table T.

The near-end talker identifier <NUM> recognizes the face of a talker from such image captured by the camera <NUM>. In the example of <FIG>, a user A1 in the lower left of the image is talking. The near-end talker identifier <NUM> recognizes a face image with a change in the image of a mouth, for example, as the face of a talker, based on images of a plurality of frames. It is to be noted that, in the example of <FIG>, other users A2 to A5, although the faces have been recognized, are not talkers. Therefore, the near-end talker identifier <NUM> recognizes the face of the user A1 as the face of a talker.

In addition, the near-end talker identifier <NUM> may set a boundary box (Bounding Box) as shown by the square in <FIG>, at a position of the face of the recognized talker. The near-end talker identifier <NUM> is also able to obtain a distance from a talker, based on the size of the boundary box. The flash memory <NUM> previously stores a table, a function, or the like that shows the relationship between the size of the boundary box and the distance. The near-end talker identifier <NUM> is also able to obtain a distance from a talker by comparing the size of a set boundary box and the table stored in the flash memory <NUM>.

The near-end talker identifier <NUM> obtain two-dimensional coordinates (X, Y coordinates) of the set boundary box and the distance from a talker, as the position information of the talker. The conversation history recorder <NUM> also records the position information of a talker.

<FIG> shows a configuration in a case in which talker identification is performed based on a video signal and an audio signal on a far-end side and recorded as conversation history, and <FIG> shows the operation in such a case. The voice obtainer <NUM> receives a far-end audio signal through the communicator <NUM> (S21). In addition, the image obtainer <NUM> receives a far-end talker image through the communicator <NUM> (S22). The far-end talker identifier <NUM> identifies a talker on a far-end side, based on the far-end audio signal and the far-end talker image (S23). The conversation history recorder <NUM> records an identifier that indicates the talker identified by the far-end talker identifier <NUM> and the order of talkers, as conversation history (S24).

The method of identifying a talker of the far-end talker identifier <NUM> is the same as the method of identifying a talker of the near-end talker identifier <NUM>. The far-end talker identifier <NUM> identifies a talker, based on a voice feature amount, for example. Alternatively, the far-end talker identifier <NUM> may identify a talker by performing face recognition of the talker from a received far-end talker image.

It is to be noted that, in a case in which an apparatus on the far-end side is an apparatus capable of identifying a talker in the same manner as the own apparatus, the far-end talker identifier <NUM> may receive information for identifying a talker from the apparatus on a far-end side. In such a case, the far-end talker identifier <NUM> is able to identify a talker on a far-end side, based on received information.

In addition, the near-end talker identifier <NUM> and the far-end talker identifier <NUM> do not necessarily need to identify a single talker (utterance of the same talker) for a plurality of utterances by a certain talker. For example, the near-end talker identifier <NUM> and the far-end talker identifier <NUM> may identify talkers even having a similar voice feature amount, as different talkers. The near-end talker identifier <NUM> and the far-end talker identifier <NUM> may identify talkers having even slightly different voice feature amount or the like, as different talkers. Even when the near-end talker identifier <NUM> and the far-end talker identifier <NUM> identify the same talker as a different talker and record the conversation history, the predictor <NUM> is still able to predict the next talker. In other words, the near-end talker identifier <NUM> and the far-end talker identifier <NUM> may identify a talker based on an excessively segmented element to the extent that the next talker is able to be predicted or that one talker is able to be identified as a different talker. In addition, the near-end talker identifier <NUM> and the far-end talker identifier <NUM> may record the voice feature amount itself of a talker, the arrival direction itself of the voice of a talker, the face image itself of a talker, or the position information itself of a talker in the conversation history, as a result of the talker identification.

It is to be noted that, although the present embodiment shows an example in which the number of devices on a far-end side is one and the one device communicates one to one, the audio signal processing apparatus <NUM> may also perform communication with devices on a far-end side at multiple points and function as a multipoint communication system. In a case of performing communication with the devices on a far-end side at multiple points, the far-end talker identifier <NUM> performs processing of identifying a talker for each device on a far-end side and identifies all far-end talkers.

<FIG> shows an example of conversation history. The conversation history recorder <NUM> records conversation history in time series. For example, in the example of <FIG>, a talker A1 on a near-end side talks from time t1 to time t2, and a talker B1 on a far-end side talks from the time t2 to time t3. A talker A2 on the near-end side talks from the time t3 to time t4, and a talker B2 on the far-end side talks from the time t4 to time t5. The talker A1 on the near-end side talks from the time t5 to time t6, and the talker B1 on the far-end side talks from the time t6 to time t7. The talker A2 on the near-end side talks from the time t7 to time t8, and the talker B1 on the far-end side talks from the time t8 to time t9. Subsequently, a talker A4 on the near-end side talks from the time t9. It is to be noted that it is not necessary to record time information. The conversation history recorder <NUM> may only record an identifier that indicates who the talker is and an order (number) of the talker.

The audio signal processing apparatus <NUM> predicts a next talker based on the conversation history. The audio signal processing apparatus <NUM> functions as a talker prediction device. <FIG> shows a functional configuration of the audio signal processing apparatus <NUM> in a case of predicting, and <FIG> shows the operation in such a case. The example of <FIG> shows the operation (the operation of a talker prediction method) of prediction when a talker on a far-end side is currently talking.

The voice obtainer <NUM> receives a far-end audio signal through the communicator <NUM> (S31). In addition, the image obtainer <NUM> receives a far-end talker image through the communicator <NUM> (S32). The far-end talker identifier <NUM> identifies a talker on a far-end side based on the far-end audio signal and the far-end talker image (S33). It is to be noted that, in a case in which an apparatus on the far-end side is an apparatus capable of identifying a talker in the same manner as the own apparatus, the far-end talker identifier <NUM> may receive information for identifying a talker from the apparatus on the far-end side. In such case, the far-end talker identifier <NUM> is able to identify a talker on the far-end side, based on received information.

Subsequently, the predictor <NUM> refers to the conversation history that the conversation history recorder <NUM> has recorded (S34) and predicts a next talker (S35). Specifically, the predictor <NUM>, from the conversation history, detects a part that the far-end talker has talked, where the far-end talker is identified by the far-end talker identifier <NUM>, and predicts the next talker according to talk probability of the talker who is talking immediately after a detected part. For example, in a case in which the far-end talker identifier <NUM> identifies the talker B1 on the far-end side, the predictor <NUM> detects the times t2 to t3, t6 to t7, and t8 to t9 in the conversation history shown in <FIG>. The predictor <NUM> obtains the talk probability of each talker based on the history of a talker who talks immediately after such times. In the example of <FIG>, after the talker B1 talks, the talker A2 talks twice and the talker A4 has talked once. Therefore, the talk probability of the talker A2 is <NUM>%, and the talk probability of the talker A4 is <NUM>%. The predictor <NUM> predicts that the next talker will be the talker A2 according to the talk probability.

In addition, the predictor <NUM>, in a case in which the near-end talker identifier <NUM> or the far-end talker identifier <NUM> record the voice feature amount itself of a talker, the arrival direction itself of the voice of a talker, the face image itself of a talker, the position information itself of a talker, or the like, in the conversation history as talker identification results, searches a talker identification result similar to the current talker identification result, from the conversation history recorder <NUM>. Then, the predictor <NUM> may predict the next talker from the talker identification result that the conversation history recorder <NUM> has recorded, based on a plurality of talker identification results recorded immediately after the time when searched current talker identification results are recorded. In such a case, the predictor <NUM> may also use the average of the plurality of talker identification results recorded immediately after the time when the searched current talker identification results are recorded, for example, as a result of talker prediction. Alternatively, the predictor <NUM> may also use a talker identification result that has the smallest sum of differential amount with other talker prediction results as a representative value and as a result of talker prediction.

It is to be noted that the predictor <NUM> may further predict the next talker based on the image received by the communicator <NUM> or the image captured by the camera <NUM>. For example, a person who is raising the hand may be predicted as the next talker. Alternatively, a person whose eyes are wide open may be predicted as the next talker. In addition, the predictor <NUM> may predict the next talker based on the audio signal received by the communicator <NUM> or the audio signal obtained by the microphones 18A to 18F. For example, the predictor <NUM>, when recognizing a particular utterance such as "er" or "uh," predicts that a corresponding talker will be the next talker. However, the predictor <NUM> gives priority to the prediction based on the conversation history. The predictor <NUM>, in a case in which a plurality of talkers with the same talk probability or similar talk probability (within <NUM>%, for example) are present, preferably further predicts the next talker based on the received image.

Subsequently, the camera image controller <NUM> controls the image that the camera <NUM> has captured, according to the prediction result of the predictor <NUM> (S36). The control of an image includes framing processing, for example. The camera image controller <NUM> performs the framing processing in which the face image of the talker A4 is extracted from the image captured by the camera <NUM> and obtained by the image obtainer <NUM>. The position of each talker is recorded by the conversation history recorder <NUM>. The camera image controller <NUM> extracts an image at a position corresponding to a predicted talker. The camera image controller <NUM> outputs the image on which the framing processing has been performed, to the communicator <NUM>. The communicator <NUM> sends the image on which the framing processing has been performed, to the far-end side.

As a result, the face image of the talker A2 is displayed on a display (not shown) on the far-end side. Therefore, the talker B1 on the far-end side is able to talk while looking at the most likely next talker.

It is to be noted that, while the above shows that the camera image controller <NUM> performs the framing processing in which the image of the predicted talker is extracted from the image captured by the camera <NUM>, the camera image controller <NUM> may perform processing of blurring positions other than the position of the predicted talker, for example. In addition, the camera image controller <NUM> may control the shooting direction and zoom of the camera <NUM> to magnify and capture the face of the predicted talker.

Moreover, in a case in which the near-end talker identifier <NUM> identifies a talker on the near-end side, the camera image controller <NUM> performs the framing processing of extracting the image of the identified talker on the near-end side. Therefore, the image of the currently talking talker is sent to the apparatus on the far-end side.

However, the predictor <NUM> may predict the next talker even when the near-end talker identifier <NUM> identifies a talker on the near-end side. The camera image controller <NUM> may extract the image of the predicted next talker and send the image to the apparatus on the far-end side through the communicator <NUM>. In a case in which the next talker predicted by the predictor <NUM> is a talker on the near-end side, the camera image controller <NUM> performs the framing processing of extracting the face image of the talker on the near-end side from the image captured by the camera <NUM>. The camera image controller <NUM> sends the extracted image of the talker on the near-end side to the apparatus on the far-end side.

In a case in which the next talker predicted by the predictor <NUM> is a talker on the far-end side, the camera image controller <NUM> performs the framing processing of extracting the face image of the talker on the far-end side from the image received through the communicator <NUM>. The camera image controller <NUM> displays the extracted image of the talker on the far-end side, on the display (not shown).

However, the camera image controller <NUM>, in the case of sending the extracted image of the talker on the near-end side to the apparatus on the far-end side, sends the image to the apparatus on the far-end side, where the image is preferably one of the image of a current talker on the near-end side and the image of a predicted next talker on the near-end side periodically switched. For example, the camera image controller <NUM> sends the image of the predicted next talker on the near-end side about once every <NUM> seconds. Alternatively, the camera image controller <NUM> may combine a small image of the predicted next talker with the image of the current talker, and may send the combined image to the far-end side.

It is to be noted that the audio signal processing apparatus <NUM> may perform audio signal processing on the audio signal obtained by the microphone based on a prediction result of the predictor <NUM>. <FIG> is a block diagram showing a functional configuration of the audio signal processing apparatus <NUM> in a case of performing audio signal processing based on the prediction result. The audio signal processing apparatus <NUM> functionally includes a filter generator <NUM> and an audio signal processor <NUM>. In addition, the audio signal processor <NUM> functionally includes a beamforming processor <NUM>, a gain corrector <NUM>, and an equalizer <NUM>. Such functional configurations are implemented by the CPU <NUM> and the DSP <NUM>.

The filter generator <NUM> generates a correction filter based on the prediction result of the predictor <NUM>. The correction filter includes the filter processing at least in the beamforming processor <NUM>.

The beamforming processor <NUM> performs beamforming by performing the filter processing on each of the audio signals obtained by the microphones 18A to 18F and combining the audio signals. The signal processing according to the beamforming can be any processing such as the Delay Sum type, the Griffiths Jim type, the Sidelobe Canceller type, or the Frost Adaptive Beamformer.

The filter generator <NUM> generates a filter coefficient to form directivity toward the position of a predicted next talker and sets the filter coefficient to the beamforming processor <NUM>. As a result, the audio signal processing apparatus <NUM> is able to obtain the voice of the predicted next talker with a high SN ratio from the beginning of the talk.

In addition, the filter generator <NUM> sets the gain of the gain corrector <NUM>, based on the prediction result of the predictor <NUM>. The voice of a talker is attenuated as a distance from a microphone is increased. Therefore, the filter generator <NUM> generates a gain correction filter to cause the level of an audio signal to be increased as a distance from a predicted next talker is increased, and sets the gain correction filter to the gain corrector <NUM>. As a result, the audio signal processing apparatus <NUM> is able to obtain the voice of the predicted next talker at a stable level regardless of the distance from the talker.

In addition, the filter generator <NUM> may set the frequency characteristics of the equalizer <NUM> to cause the level of the high band to be increased as the distance from the predicted next talker is increased. The voice of a talker tends to be attenuated more greatly in the high band than in the low band, as the distance from a microphone is increased. Therefore, the audio signal processing apparatus <NUM> is able to obtain the voice of the predicted next talker with stable voice quality regardless of the distance from the talker.

Claim 1:
A talker prediction method comprising:
obtaining a voice from a plurality of talkers (S11, S21, S31);
recording a conversation history of the plurality of talkers (S14, S24);
identifying a talker of the obtained voice (S13, S23, S33);
detecting a part that the identified talker has talked from the conversation history; and
obtaining a talk probability of each of the plurality of talkers who is talking immediately after the detected part based on the identified talker and the conversation history (S34, S35), wherein the talk probability of each of the plurality of talkers who is talking immediately after the detected part is obtained based on a number of times each talker talked after the identified talker in the history;
predicting a next talker among the plurality of talkers according to the talk probability.