Patent Publication Number: US-2022224747-A1

Title: Signal processing method and signal processing apparatus

Description:
CROSS REFERENCE TO RELATED APPLICATIONS 
     This Nonprovisional application claims priority under 35 U.S.C. § 119(a) on Patent Application No. 2021-002750 filed in Japan on Jan. 12, 2021, the entire contents of which are hereby incorporated by reference. 
     BACKGROUND 
     Technical Field 
     An embodiment of the present disclosure relates to a signal processing method and a signal processing apparatus that process an audio signal or a video signal. 
     Background Information 
     Japanese Unexamined Patent Application Publication No. 2014-120830 discloses a configuration in which a delay time in wireless communication is measured, and, among a plurality of encoder parameters, an encoder parameter with the shortest delay time is set. 
     Delay in communication with a device at a remote place includes a delay time by signal processing, and a network delay time. A user feels uncomfortable when a sum of the delay time exceeds a predetermined period of time. 
     The configuration disclosed in Japanese Unexamined Patent Application Publication No. 2014-120830 sets the encoder parameter with the minimum delay time, so that the user feels less uncomfortable. However, the configuration of Japanese Unexamined Patent Application Publication No. 2014-120830, since setting the encoder parameter with the minimum delay time, may reduce accuracy of the signal processing. 
     SUMMARY 
     In view of the foregoing, an object of an embodiment of the present disclosure is to provide a signal processing method and a signal processing apparatus that increase accuracy of signal processing without making a user feel uncomfortable. 
     Solution to Problem 
     A signal processing method according to an embodiment of the present disclosure includes obtaining, by a signal processing apparatus, a network delay time with respect to a device connected to the signal processing apparatus via a network, obtaining an input signal, determining an allowable upper limit of a delay time for an output signal corresponding to the obtained input signal based on the obtained network delay time and a total allowable delay time, selecting a signal processing having a longest delay time that is less than or equal to the allowable upper limit of the delay time, performing the selected signal processing on the obtained input signal, and transmitting the obtained input signal on which the selected signal processing has been performed, as the output signal, to the device connected to the signal processing apparatus via the network. 
     According to an embodiment of the present disclosure, it is possible not to make a user feel uncomfortable and to improve accuracy of signal processing. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         FIG. 1  is a block diagram showing a configuration of a signal processing apparatus  1 . 
         FIG. 2A  is a block diagram showing a functional configuration of a processor  12 . 
         FIG. 2B  is a block diagram showing a functional configuration of the processor  12 . 
         FIG. 3  is a flow chart showing an operation of a signal processing program  141 . 
         FIG. 4  is a flow chart showing a detailed operation of measurement of a network delay time. 
         FIG. 5  is a flow chart showing a measurement operation of a network delay time according to a modification. 
         FIG. 6  is a block diagram showing a configuration of a signal processing apparatus  1 A according to a first modification. 
         FIG. 7  is a block diagram showing a functional configuration of a processor  12  in the signal processing apparatus  1 A. 
     
    
    
     DETAILED DESCRIPTION 
       FIG. 1  is a block diagram showing a configuration of a signal processing apparatus  1 . The signal processing apparatus  1  includes a communicator  11 , a processor  12 , a RAM  13 , a flash memory  14 , a microphone  15 , an amplifier  16 , and a speaker  17 . 
     The signal processing apparatus  1  configures a remote conversation device connecting to a device at a remote place and transmitting and receiving audio data, for example. The signal processing apparatus  1  performs predetermined signal processing on an audio signal obtained by the microphone  15 . The signal processing apparatus  1  transmits the audio signal on which the signal processing has been performed, as audio data, to a far-end side. In addition, the signal processing apparatus  1  outputs sound from the speaker  17 , based on the audio signal of the audio data received from the far-end side. 
     The communicator  11  connects to the remote conversation device on the far-end side through a network, and transmits and receives the audio data with the remote conversation device on the far-end side. 
     The processor  12  reads a program from the flash memory  14  being a storage medium, and temporarily stores the program in the RAM  13 , and thus performs various operations. The program includes a signal processing program  141 . The flash memory  14  further stores a program for operating the processor  12 , such as firmware. 
     The microphone  15  is an example of an input signal obtainer, and obtains various types of sound such as voice of a talker and noise, as an audio signal. The microphone  15  performs digital conversion on an obtained audio signal. The microphone  15  outputs a digitally converted audio signal to the processor  12 . 
     The processor  12  performs predetermined signal processing on the audio signal obtained by the microphone  15 . For example, the processor  12  performs noise reduction processing on the audio signal obtained by the microphone  15 . In addition, the processor  12  performs echo reduction processing on the audio signal obtained by the microphone  15 . The processor  12  transmits the audio signal on which the signal processing has been performed, as audio data, to the far-end side through the communicator  11 . In addition, the processor  12  outputs the audio data received through the communicator  11 , as an audio signal, to the amplifier  16 . 
     The amplifier  16  performs analog conversion on the audio signal received from the processor  12 , and amplifies the audio signal. The amplifier  16  outputs an amplified audio signal to the speaker  17 . The speaker  17  outputs sound based on the audio signal outputted from the amplifier  16 . 
     The processor  12  performs an audio signal processing method of the present disclosure.  FIG. 2A  is a block diagram showing a functional configuration of the processor  12 . The processor  12  functionally configures a buffer  121 , a noise reducer  122 , a transmitter  123 , a receiver  124 , a measurer  125 , and a delay time calculator  126 . The configurations are implemented by the signal processing program  141 . 
     The buffer  121  temporarily stores the audio signal obtained by the microphone  15  for a predetermined period of time. The noise reducer  122  is an example of a signal processor, and performs noise reduction processing, using the audio signal stored in the buffer  121 . The transmitter  123  transmits the audio signal of which the noise has been reduced by the noise reducer  122 , as audio data, to an apparatus of a connection destination. The receiver  124  receives the audio data from the apparatus of a connection destination, and outputs the audio data to the amplifier  16 , as an audio signal. The measurer  125  measures a network delay time. The delay time calculator  126 , based on the network delay time, calculates an allowable upper limit of the delay time that occurs in an output signal with respect to an input signal by performing signal processing in the signal processing program  141 . In addition, the delay time calculator  126  selects signal processing with the longest delay time less than or equal to the upper limit. 
       FIG. 3  is a flow chart showing an operation of the signal processing program  141 . The measurer  125  measures a network delay time (S 11 ).  FIG. 4  is a flow chart showing a detailed operation of measurement of the network delay time. The measurer  125  first transmits a first DTMF (Dual-Tone Multi-Frequency) signal as a signal for a test, to an apparatus of a connection destination through the sender  123 , and records a transmission time (S 101 ). The first DTMF signal is embedded, for example, in the payload of VoIP (Voice over Internet Protocol). 
     The apparatus of a connection destination receives the first DTMF signal (S 201 ). The apparatus of a connection destination transmits a second DTMF signal (S 202 ) as a response to the first DTMF signal. The second DTMF signal is also embedded, for example, in the payload of VoIP. The measurer  125  receives the second DTMF signal through the receiver  124 , and records a reception time (S 102 ). The measurer  125  measures the network delay time from a difference between the transmission time and the reception time that have been recorded (S 103 ). 
     The network delay time corresponds to a time difference between transmission of certain data and reception of the certain data by the apparatus of a connection destination. The difference between the transmission time and the reception time that have been recorded by the measurer  125  is a time difference between transmission of certain data and reception of a response. Therefore, the measurer  125  measures half the time difference between the transmission time and the reception time that have been recorded, as a network delay time. 
     It is to be noted that the measurement of the network delay time may be taken during conversation but is preferably taken immediately after connection between the apparatuses is established. As a result, the measurer  125  does not interfere with the conversation, by sound of the DTMF signals. 
     In a case in which the measurement of the network delay time is taken during conversation, the measurer  125  does not preferably affect the conversation between users, for example, by embedding the signal for a test in a high-frequency band (a band of about 20 kHz, for example). 
     In addition, the measurer  125  may measure the network delay time by adding specific frequency characteristics or phase characteristics to an audio signal of a conversation sound. The measurer  125  adds a dip to a specific frequency (1 kHz, for example) of an audio signal, for example. An apparatus of a transmission destination, in a case of detecting the dip in the frequency, transmits a reply. The reply may be the second DTMF signal or may add the specific frequency characteristics or the phase characteristics to the audio signal of a conversation sound. 
     It is to be noted that the measurer  125  may embed special information corresponding to the first DTMF signal in a header of an RTP (Real-time Transport Protocol) packet instead of the payload in VoIP, for example. The apparatus of a transmission destination, in a case of extracting the special information in the header of the RTP packet, transmits a reply. The reply may be the second DTMF signal or may embed reply information in the header of the RTP packet. 
     In addition, the measurer  125  may obtain the transmission time of packet data received from the apparatus of a connection destination, from a remote conversation program (a program to transmit and receive audio data).  FIG. 5  is a flow chart showing a measurement operation of a network delay time according to a modification. In the modification, the remote conversation program transmits audio data with a transmission time. 
     The apparatus of a transmission destination transmits audio data with a transmission time (S 301 ). The measurer  125  receives the audio data through the receiver  124 , and records a reception time (S 401 ). The measurer  125  extracts the transmission time from the received audio data (S 402 ). The measurer  125  calculates a network delay time from a difference between the extracted transmission time and the recorded reception time (S 403 ). 
     The remote conversation program transmits audio data with a transmission time in this example, so that the transmission and reception of a test signal such as a DTMF signal is unnecessary. In addition, in this example, the measurer  125 , since using time information added to the audio data of a conversation sound, does not affect conversation between users even when taking the measurement during the conversation. 
     Returning to  FIG. 3 , the delay time calculator  126  calculates the upper limit, based on the network delay time that the measurer  125  has measured (S 12 ). For example, the upper limit corresponds to a difference between the total allowable delay time (200 msec, for example) to the extent that a user does not feel uncomfortable and the network delay time. In a case in which the network delay time is large, the upper limit is reduced, and the upper limit is increased in a case in which the network delay time is small. 
     In addition, the delay time calculator  126  selects signal processing with the longest delay time less than or equal to the calculated upper limit (S 13 ). In the example of  FIG. 2A , the delay time calculator  126  changes a buffer amount of the buffer  121  without changing processing content of the noise reducer  122 . In other words, the delay time calculator  126  sets the buffer amount to the largest less than or equal to the upper limit. The noise reducer  122  performs noise reduction processing, using the audio signal temporarily stored with the set largest buffer amount (S 14 ). The transmitter  123  transmits the audio signal on which the noise reduction processing has been performed, to the apparatus of a connection destination (S 15 ). 
     The noise reduction processing is an example of processing to determine a target signal and cause the target signal to pass. The noise reduction processing causes the target sound (voice) to pass, and reduces other sound as noise. For example, the noise reduction processing is filter processing to convert a certain input signal into a certain output signal, using a predetermined algorithm such as a learned neural network (especially, the Convolutional Neural Network (CNN), the Recurrent Neural Network (RNN), or the Long-Short Term Model (LSTM)). The algorithm of the filter processing is constructed by machine learning. The noise reducer  122  previously repeats processing and learning to convert a certain inputted audio signal into an audio signal of which the noise has been reduced, and constructs a learned model. The noise reducer  122  performs the noise reduction processing, using the learned model. 
     The accuracy of the noise reduction processing using such a learned neural network depends on an amount of information of an input signal. The accuracy of the noise reduction processing is increased as the amount of information of an input signal is large. The delay time calculator  126  according to the present embodiment sets the buffer amount to the largest less than or equal to the upper limit. Therefore, the accuracy of the noise reducer  122  is set to be the highest less than or equal to the upper limit. 
     As described above, in a case in which the network delay time is large, the upper limit is reduced, and the upper limit is increased in a case in which the network delay time is small. In short, the signal processing apparatus  1  according to the present embodiment performs highly accurate noise reduction processing in a good communication environment, and performs the noise reduction processing without delay to the extent that a user does not feel uncomfortable even in a poor communication environment. Therefore, the signal processing apparatus  1  is able to perform optimal noise reduction processing according to the communication environment. 
     The above embodiment shows an example in which the processing content of the noise reducer  122  is not changed and the buffer amount of the buffer  121  is set to the largest, as an example of selecting signal processing with the longest delay time less than or equal to the upper limit. However, the delay time calculator  126  may change the content of the signal processing of the noise reducer  122 . For example, the delay time calculator  126  may change an algorithm according to the upper limit. 
     For example, as shown in  FIG. 2B , the processor  12  may not include the buffer  121 , and may directly input the audio signal obtained by the microphone  15  into the noise reducer  122 . In such a case, the delay time calculator  126  may change the content of the signal processing of the noise reducer  122 . For example, the delay time calculator  126  may select signal processing with the longest delay time less than or equal to the upper limit, such as the recurrent neural network or the LSTM. The recurrent neural network and the LSTM, since having an internal variable, is also able to be configured not to explicitly have a buffer that stores the audio signal obtained by the microphone  15 . 
     The above embodiment shows the noise reduction processing as an example of signal processing. However, the signal processing is not limited to the noise reduction processing. For example, echo reduction processing may be performed as signal processing. In the echo reduction processing, the delay time calculator  126  sets the buffer amount to the largest less than or equal to the upper limit. 
     In addition, the signal processing may be processing to perform speech recognition processing and convert speech into text data. Moreover, the signal processing may perform determination (speech recognition) of voice of a specific talker, and may perform processing to emphasize the voice of a specific talker or reduce the voice of a specific talker. 
     In addition, the signal processing is not limited to processing on an audio signal.  FIG. 6  is a block diagram showing a configuration of a signal processing apparatus  1 A according to a first modification. The same reference numerals are used to refer to components common to  FIG. 1 , and the description will be omitted. The signal processing apparatus  1 A further includes a display  18  and a camera  19  in addition to the components of the signal processing apparatus  1 . 
       FIG. 7  is a block diagram showing a functional configuration of a processor  12  in the signal processing apparatus  1 A. The same reference numerals are used to refer to components common to  FIG. 2A , and the description will be omitted. The processor  12  of the signal processing apparatus  1 A includes an auto framing processor  152  in place of the noise reducer  122 . Other configurations are the same as the configurations of the processor  12  in the signal processing apparatus  1 . 
     The buffer  121  stores a video signal captured by the camera  19  for a predetermined period of time. The auto framing processor  152  performs auto framing processing to extract and enlarge a face of a talker in the video signal stored in the buffer  121 . The auto framing processing is an example of processing to determine a target signal and cause the target signal to pass. 
     More specifically, the auto framing processing is processing to perform face recognition (image recognition) and extract a recognized face portion. The auto framing processing may be processing to extract a face image of a specific talker. In addition, the auto framing processing may be processing to extract only a face image of a talker during conversation. 
     As with the noise reduction processing, the auto framing processing is filter processing to convert a certain input signal into a certain output signal, using a predetermined algorithm such as a neural network, for example. The algorithm of the auto filter processing is also constructed by machine learning. 
     The accuracy of the auto framing processing using such a neural network also depends on an amount of information of an input signal. The delay time calculator  126  sets the buffer amount to the largest less than or equal to the upper limit. Therefore, the accuracy of the auto framing processor  152  is set to be the highest less than or equal to the upper limit. In addition, the delay time calculator  126  may change the algorithm of the auto framing processing according to the upper limit. In the same manner as described above, the processor  12  may not include the buffer and may directly input the video signal obtained by the camera  19  into the auto framing processor  152 . In such a case, the delay time calculator  126  may select signal processing with the longest delay time less than or equal to the upper limit, such as the recurrent neural network or the LSTM. 
     The signal processing apparatus  1 A performs highly accurate auto framing processing in a good communication environment, and performs the auto framing processing without delay to the extent that a user does not feel uncomfortable even in a poor communication environment. Therefore, the signal processing apparatus  1 A is able to perform optimal auto framing processing according to the communication environment. 
     The foregoing embodiments are illustrative in all points and should not be construed to limit the present disclosure. The scope of the present disclosure is defined not by the foregoing embodiments but by the following claims for patent. Further, the scope of the present disclosure includes the scopes of the claims for patent and the scopes of equivalents.