Patent ID: 12212939

DESCRIPTION OF EMBODIMENTS

Hereinafter, embodiments of the present disclosure will be described in detail. Components having the same function are given the same numeral, and duplicated description will be omitted.

Prior to describing each embodiment, the method of notation herein will be described.

A caret ({circumflex over ( )}) represents a superscript. For example, xy{circumflex over ( )}zindicates that yzis the superscript of x, and xy{circumflex over ( )}zindicates that yzis the subscript of x. An underscore (_) represents a subscript. For example, xy_zindicates that yzis the superscript of x, and xy_zindicates that yzis the subscript of x.

Superscripts of a certain character x such as “{circumflex over ( )}” in {circumflex over ( )}x and “{tilde over ( )}” in {tilde over ( )}x should normally be written directly above “x”, but {circumflex over ( )}x and {tilde over ( )}x are used due to limitations of the description notation herein.

Furthermore, a complex conjugate transpose of a matrix M or a vector v is represented by a superscriptH, such as in vHor MH. An inverse matrix of the matrix M is represented by a superscript−1, such as in M−1. A complex conjugate of a scalar s is represented by a superscript *, such as in s*.

Technical Background

In an embodiment of the present disclosure, a steering vector is generated by approximately determining an eigenvector corresponding to a maximum eigenvalue, by using only a matrix operation. This eliminates the need for solving an eigenvalue decomposition problem, enabling instability in the calculation to be prevented in training a neural network by using an error back propagation method to further reduce an estimation error of a beamformer.

The present method includes a predetermined iterative calculation. If the number of repetitions increases, it is possible to suppress an error of the approximation calculation for determining an eigenvector corresponding to the maximum eigenvalue and improve the estimation accuracy of the beamformer.

A signal is hereinafter regarded as a value in a time frequency domain after the signal is applied with a short-time Fourier transform (STFT). t denotes an index representing a time frame, and f denotes an index representing a frequency bin.

First Embodiment

A target sound signal generation apparatus100generates, from an observed signal vector xt,fcorresponding to an observed sound collected by using a plurality of microphones, a target sound signal yt,fcorresponding to a target sound included in the observed sound.

The target sound signal generation apparatus100will be described below with reference toFIGS.1and2.FIG.1is a block diagram illustrating a configuration of the target sound signal generation apparatus100.FIG.2is a flowchart illustrating an operation of the target sound signal generation apparatus100. As illustrated inFIG.1, the target sound signal generation apparatus100includes a mask generation unit110, a steering vector generation unit120, a beamformer vector generation unit130, a target sound signal generation unit140, and a recording unit190. The recording unit190is a constituent component configured to appropriately record information required for processing of the target sound signal generation apparatus100.

The operation of the target sound signal generation apparatus100will be described with reference toFIG.2.

In S110, the mask generation unit110receives the observed signal vector xt,fas an input to generate and output a mask γt,ffrom the observed signal vector xt,f. Here, the mask is used to calculate a spatial covariance matrix described later. Specifically, the mask is an index having a value from 0 to 1. For example, the mask γt,fmay indicate a probability that a target sound signal is included in each time frame t and each frequency bin f. In this case, γt,f=1 indicates that the target sound signal is included, and γt,f=0 indicates that the target sound signal is not included. Furthermore, γt,fhaving a value between 0 and 1 indicates an intermediate state between a state where the target sound signal is included and a state where the target sound signal is not included. Moreover, the mask γt,fmay indicate a probability that a target sound is included in each time frame t. In this case, the mask γt,fhas the same value at any frequency.

Furthermore, the mask generation unit110may be configured by using a neural network described in NPL 1 and NPL 2. That is, the mask generation unit110is configured as a neural network trained by using an error back propagation method.

In S120, the steering vector generation unit120receives the observed signal vector xt,fand the mask γt,fgenerated in S110as an input to generate and output a steering vector hffrom the observed signal vector xt,fand the mask γt,f. Here, the steering vector is used to calculate a beamformer vector described later.

The steering vector generation unit120may be configured to generate the steering vector hfby determining an eigenvector corresponding to a maximum eigenvalue of a predetermined matrix generated from the observed signal vector xt,fand the mask γt,fby using a power method. The steering vector generation unit120will be described below with reference toFIGS.3and4.FIG.3is a block diagram illustrating a configuration of the steering vector generation unit120.FIG.4is a flowchart illustrating an operation of the steering vector generation unit120. As illustrated inFIG.3, the steering vector generation unit120includes a spatial covariance matrix generation unit122and a steering vector calculation unit124.

An operation of the steering vector generation unit120will be described with reference toFIG.4.

In S122, the spatial covariance matrix generation unit122receives the observed signal vector xt,fand the mask γt,fgenerated in S110as an input to generate and output a target sound spatial covariance matrix Φsfand a noise spatial covariance matrix Φnffrom the observed signal vector xt,fand the mask γt,f. The spatial covariance matrix generation unit122generates, according to the following equations, the target sound spatial covariance matrix Φsfand the noise spatial covariance matrix Φnf.

Φfs=∑t⁢γt,f⁢xt,f⁢xt,fH∑t⁢γt,f⁢Φfn=∑t⁢(1-γt,f)⁢xt,f⁢xt,fH∑t⁢(1-γt,f)[Math.1]

In S124, the steering vector calculation unit124receives the target sound spatial covariance matrix Φsfand the noise spatial covariance matrix Φnfgenerated in S122as an input, and uses the target sound spatial covariance matrix Φsfand the noise spatial covariance matrix Φnfto calculate and output the steering vector hffrom an initial vector u. Here, the initial vector u may be any vector, and may be, for example, a vector in which an element corresponding to a reference microphone r is 1 and an element corresponding to another microphone is 0. The steering vector calculation unit124calculates the steering vector hfaccording to the following equation
[Math. 2]
hf=Φfn((Φfn)−1Φfs)mu(1)

where m is an integer of 1 or greater representing the number of repetitions. ((Φnf)−1Φsf)mu in Equation (1) corresponds to approximately calculating, by using the power method, an eigenvector corresponding to a maximum eigenvalue of the matrix (Φnf)−1Φsf. It is known that an eigenvector corresponding to the maximum eigenvalue can be accurately obtained for any initial vector u by selecting a sufficiently great positive integer for m representing the number of repetitions. It is also known that, even when m is a relatively small value, for example, m=1, the eigenvector mentioned above can be approximated with a certain accuracy. Consequently, instead of solving the eigenvalue decomposition problem, the steering vector can be estimated with a high accuracy from the calculation of Equation (1).

In S130, the beamformer vector generation unit130receives the observed signal vector xt,fand the steering vector hfgenerated in S120as an input to generate and output a beamformer vector wffrom the observed signal vector xt,fand the steering vector hf. The beamformer vector generation unit130generates the beamformer vector wfaccording to the following equation

wf=Rf-1⁢hfhfH⁢Rf-1⁢hf⁢hfr*[Math.3]

where hfris an element of the steering vector hfcorresponding to the reference microphone r. Furthermore, a matrix Rfis calculated according to the following equation
Rf=Σtxt,fxt,fH[Math. 4]

where the sum mentioned above is a sum for the time frame t included in a noise section.

In S140, the target sound signal generation unit140receives the observed signal vector xt,fand the beamformer vector wfgenerated in S130as an input to generate and output the target sound signal yt,ffrom the observed signal vector xt,fand the beamformer vector wf. The target sound signal generation unit140generates the target sound signal yt,faccording to the following equation.
yt,f=wfHxt,f[Math. 5]

As described above, in the present embodiment, the output (that is, the target sound signal) of a beamformer is determined depending on a mask estimated by using a neural network. Consequently, if the accuracy in the estimation of the mask by the neural network can be improved, further improvement in the accuracy of the output of the beamformer can also be expected. NPL 2 discloses the use of an error back propagation method, for example, as a method for achieving this improvement. In NPL 2, a gradient of weights for updating a neural network is determined so that a cost function E ({yt,f}) for measuring an estimation accuracy of all pieces of output {yt,f} of a beamformer is minimized. Here, {·} collectively represents a set of symbols (for example, y) having different values of subscripts. In general, the error back propagation method can be employed when processing from the input to the output is configured as a connection of processing blocks having differentiable input/output relationships. In the case of the beamformer processing according to the present embodiment, processing blocks including the estimation of the mask by the neural network, the estimation of the beamformer based on the mask, and the application of the beamformer can each be expressed as a differentiable function, as described below.

The estimation of the mask by the neural network can be expressed as a differentiable function M where an observed signal vector {xt,f} and a weighting factor {θi} (where θirepresents a weighting factor of an i-th neural network) are received as an input to output a mask {γt,f}.
γt,f=M({xt,f},{θi})  [Math. 6]

Similarly, the estimation of the beamformer based on the mask can be expressed as a differentiable function W where the mask {γt,f} and the observed signal vector {xt,f} are received as an input to output a beamformer vector {wf}.
wf=W({γt,f},{xt,f})  [Math. 7]

Similarly, the application of the beamformer can be expressed as a differentiable function G where the beamformer vector wfand the observed signal vector xt,fare received as an input to output the target sound signal yt,f.
yt,f=G(wf,xt,f)  [Math. 8]

In the error back propagation method, training of a neural network is achieved by transmitting information required for calculating a gradient ∂E/∂θiof weighting factors of the neural network, in a reverse order of the procedure of the estimation of the beamformer, that is, in the direction from the output to the input. In recent years, it is possible to easily perform calculations in the error back propagation method by using software provided for training neural networks (for example, PyTorch or TensorFlow). Unfortunately, including a portion for solving the eigenvalue decomposition problem in the above-described processing blocks causes the calculations in the error back propagation method to be unstable, and thus the neural network cannot be appropriately trained. In the present embodiment, the eigenvalue decomposition problem is not solved, and thus, it is possible to appropriately train a neural network by using the error back propagation method.

The embodiment of the present disclosure allows for preventing instability in the calculation when the neural network is trained by using the error back propagation method to reduce the estimation error of the beamformer. Furthermore, it is possible to estimate the beamformer by using the steering vector generated with a high accuracy by the power method, without solving the eigenvalue decomposition problem.

Second Embodiment

Here, as described in Referential non-patent literature 1, an aspect is described in which, instead of the observed signal vector xt,f, an intermediate signal vector {circumflex over ( )}xt,fbeing a predetermined vector obtained from the observed signal vector xt,fis used to generate the target sound signal yt,f. (Referential non-patent literature 1: T. Nakatani, K. Kinoshita, “Maximum-likelihood convolutional beamformer for simultaneous denoising and dereverberation,” 2019 27th European Signal Processing Conference (EUSIPCO), 2019.)
A target sound signal generation apparatus200generates, from an observed signal vector xt,fcorresponding to an observed sound collected by using a plurality of microphones, a target sound signal yt,fcorresponding to a target sound included in the observed sound.

The target sound signal generation apparatus200will be described below with reference toFIGS.5and6.FIG.5is a block diagram illustrating a configuration of the target sound signal generation apparatus200.FIG.6is a flowchart illustrating an operation of the target sound signal generation apparatus200. As illustrated inFIG.5, the target sound signal generation apparatus200includes the mask generation unit110, an intermediate signal vector generation unit210, a steering vector generation unit220, a beamformer vector generation unit230, a target sound signal generation unit240, and a recording unit290. The recording unit290is a constituent component configured to appropriately record information required for processing of the target sound signal generation apparatus200.

The operation of the target sound signal generation apparatus200will be described with reference toFIG.6.

In S110, the mask generation unit110receives the observed signal vector xt,fas an input to generate and output a mask yt,ffrom the observed signal vector xt,f.

In S210, the intermediate signal vector generation unit210receives the observed signal vector xt,fas an input to generate and output an intermediate signal vector {circumflex over ( )}xt,fbeing a predetermined vector obtained by using the observed signal vector xt,f. For example, the intermediate signal vector {circumflex over ( )}xt,fmay be a vector including the observed signal vector xt,fand several observed signal vectors having the same frequency bin as the observed signal vector xt,f, and a different time frame from that of the observed signal vector xt,f(that is, a vector obtained from a plurality of observed signal vectors including the observed signal vector xt,f) (see Referential non-patent literature 1). Furthermore, the intermediate signal vector {circumflex over ( )}xt,fmay be, for example, a vector being obtained by using a weighted prediction error (WPE) method and corresponding to a sound with suppressed reverberation effects included in an observed sound (that is, an output vector according to the WPE method).

In S220, the steering vector generation unit220receives the intermediate signal vector {circumflex over ( )}xt,fgenerated in S210and the mask γt,fgenerated in S110as an input to generate and output the steering vector hffrom the intermediate signal vector {circumflex over ( )}xt,fand the mask γt,f.

The steering vector generation unit220may be configured to generate the steering vector hfby determining an eigenvector corresponding to a maximum eigenvalue of a predetermined matrix generated from the intermediate signal vector {circumflex over ( )}xt,fand the mask γt,fby using a power method. The steering vector generation unit220will be described below with reference toFIGS.7and8.FIG.7is a block diagram illustrating a configuration of the steering vector generation unit220.FIG.8is a flowchart illustrating an operation of the steering vector generation unit220. As illustrated inFIG.7, the steering vector generation unit220includes a spatial covariance matrix generation unit222and a steering vector calculation unit224.

An operation of the steering vector generation unit220will be described with reference toFIG.8.

In S222, the spatial covariance matrix generation unit222receives the intermediate signal vector {circumflex over ( )}xt,fgenerated in S210and the mask γt,fgenerated in S110as an input to generate and output the target sound spatial covariance matrix Φnfand the noise spatial covariance matrix Φnffrom the intermediate signal vector {circumflex over ( )}xt,fand the mask γt,f. The spatial covariance matrix generation unit222generates the target sound spatial covariance matrix Φsfand the noise spatial covariance matrix Φnfaccording to the following equations.

Φfs=∑t⁢γt,f⁢x^t,f⁢x^t,fH∑t⁢γt,f⁢Φfn=∑t⁢(1-γt,f)⁢x^t,f⁢x^t,fH∑t⁢(1-γt,f)[Math.9]

In S224, the steering vector calculation unit224receives the target sound spatial covariance matrix Φsfand the noise spatial covariance matrix Φnfgenerated in S222as an input, and uses the target sound spatial covariance matrix Φsfand the noise spatial covariance matrix Φnfto calculate and output the steering vector hffrom the initial vector u. The steering vector calculation unit224calculates the steering vector hfaccording to the following equation
hf=Φfn((Φfn)−1Φjs)mu[Math. 10]where m is an integer of 1 or greater representing the number of repetitions. In S230, the beamformer vector generation unit230receives the intermediate signal vector {circumflex over ( )}xt,fgenerated in S210and the steering vector hfgenerated in S220as an input to generate and output the beamformer vector wffrom the intermediate signal vector {circumflex over ( )}xt,fand the steering vector hf. The beamformer vector generation unit230generates the beamformer vector wfaccording to the following equation

Wf=Rf-1⁢hfhfH⁢Rf-1⁢hf⁢hfr*[Math.11]where hfris an element of the steering vector hfcorresponding to the reference microphone r. Furthermore, a matrix Rfis calculated according to the following equation

Rf=∑t⁢x^t,f⁢x^t,fHλt,f[Math.12]where the sum mentioned above is a sum for the time frame t included in a noise section, and λtis the power calculated from the observed signal vector xt,f.

In S240, the target sound signal generation unit240receives the intermediate signal vector {circumflex over ( )}xt,fgenerated in S210and the beamformer vector wfgenerated in S230as an input to generate and output the target sound signal yt,ffrom the intermediate signal vector {circumflex over ( )}xt,fand the beamformer vector wf. The target sound signal generation unit240generates the target sound signal yt,faccording to the following equation.
yt,f=wfH{circumflex over (x)}t,f[Math. 13]

The embodiment of the present disclosure allows for preventing instability in the calculation when the neural network is trained by using the error back propagation method to reduce the estimation error of the beamformer. Furthermore, it is possible to estimate the beamformer by using the steering vector generated with a high accuracy by the power method, without solving the eigenvalue decomposition problem.

Supplement

FIG.9is a diagram illustrating an example of a functional configuration of a computer realizing each of the apparatuses described above. The processing in each of the above-described apparatuses can be performed by causing a recording unit2020to read a program for causing a computer to function as each of the above-described apparatuses, and operating the program in a control unit2010, an input unit2030, an output unit2040, and the like.

The apparatus according to the present disclosure includes, for example, as single hardware entities, an input unit to which a keyboard or the like can be connected, an output unit to which a liquid crystal display or the like can be connected, a communication unit to which a communication apparatus (for example, a communication cable) capable of communication with the outside of the hardware entity can be connected, a central processing unit (CPU, which may include a cache memory, a register, and the like), a RAM or a ROM that is a memory, an external storage apparatus that is a hard disk, and a bus connected for data exchange between the input unit, the output unit, the communication unit, the CPU, the RAM, the ROM, and the external storage apparatuses. Furthermore, in the apparatus of the present disclosure, an apparatus (drive) capable of reading and writing from and to a recording medium such as a CD-ROM may be provided in the hardware entity as necessary. An example of a physical entity including such hardware resources is a general-purpose computer.

A program necessary to implement the above-described functions, data necessary for processing of this program, and the like are stored in the external storage apparatus of the hardware entity (for example, the program may be stored not only in the external storage apparatus but in a ROM that is a read-only storage apparatus). For example, data obtained by the processing of the program is appropriately stored in a RAM, the external storage apparatus, or the like.

In the hardware entity, each program and data necessary for the processing of each program stored in the external storage apparatus (or a ROM, for example) are read into a memory as necessary and appropriately interpreted, executed, or processed by a CPU. As a result, the CPU achieves a predetermined function (each of the constituent components expressed as the above-described, unit, means, or the like).

The present disclosure is not limited to the above-described embodiments, and appropriate changes can be made without departing from the spirit of the present disclosure. The processing described in the embodiments is not only executed in the chronological order following the above-described order, but may also be executed in parallel or individually, according to a processing capability of an apparatus executing the processing, or as necessary.

As described above, when a processing function in the hardware entity (the apparatus of the present disclosure) described in the embodiments is implemented by a computer, a processing content of a function that the hardware entity should have is described by a program. By executing this program using a computer, the processing function in the hardware entity is implemented on the computer.

A program in which the processing content is described can be recorded on a computer-readable recording medium. The computer-readable recording medium may be, for example, a magnetic recording apparatus, an optical disc, a magneto-optical recording medium, or a semiconductor memory. Specifically, for example, a hard disk apparatus, a flexible disk, a magnetic tape, or the like can be used as the magnetic recording apparatus, a digital versatile disc (DVD), a DVD-random access memory (RAM), a compact disc read only memory (CD-ROM), a CD-recordable (R)/rewritable (RW), or the like can be used as the optical disc, a magneto-optical disc (MO) or the like can be used as the magneto-optical recording medium, and an electronically erasable and programmable-read only memory (EEP-ROM) or the like can be used as the semiconductor memory.

Furthermore, this program is distributed, for example, by selling, transferring, or renting a portable recording medium such as a DVD or CD-ROM on which the program has been recorded. The program may be stored in a storage apparatus of a server computer and transmitted from the server computer to another computer via a network, so that the program is distributed.

The computer executing such a program first temporarily stores, for example, the program recorded on the portable recording medium or the program transferred from the server computer in a storage apparatus of the computer. When executing the processing, the computer reads the program stored in the storage apparatus of the computer and executes the processing in accordance with the read program. As another execution mode of this program, a computer may directly read a program from a portable recording medium and execute processing according to the program. Furthermore, each time the program is transferred from the server computer to the computer, the computer may sequentially execute processing according to the received program. In addition, the above-described processing may also be executed by a so-called application service provider (ASP) type service in which a processing function is implemented simply by an instruction to execute the program and by acquiring a result without transferring the program from the server computer to the computer. Furthermore, the program having this aspect is assumed to include information that is provided for processing in an electronic calculator and is equivalent to a program (data or the like that has characteristics for defining a processing of a computer rather than being a direct instruction to the computer).

Although in the present aspect, the hardware entity is configured by causing a computer to execute a predetermined program, at least a part of the processing content may be implemented by hardware.

The foregoing description of the embodiments of the present disclosure has been presented for purposes of illustration and description. The foregoing description does not intend to be exhaustive and does not intend to limit the invention to the precise forms disclosed. Modifications and variations are possible from the teachings above. The embodiments have been chosen and expressed in order to provide the best demonstration of the principles of the present invention, and to enable those skilled in the art to utilize the present invention in numerous embodiments and with the addition of various modifications suitable for the actual use considered. All such modifications and variations are within the scope of the present invention defined by the appended claims that are interpreted according to the width provided justly, lawfully, and fairly.