Patent Description:
A sound source separation technology of extracting a target sound source signal from a mixed sound signal containing a plurality of sound source signals is known. For example, Patent Document <NUM> discloses a sound source separation technology using a deep neural network (DNN). The scientific publication "<NPL> describes music separation with DNNs.

Techniques using the DNN achieve high sound source separation performance, but a large amount of operations such as multiplication and addition need to be performed. Furthermore, in the DNN that achieves high sound source separation performance, a larger number of coefficients are used, so that the capacity of a memory for storing the coefficients also needs to be increased.

It is therefore an object of the present disclosure to provide a program, an information processing method, a recording medium, and an information processing device that minimize the amount of operations while achieving sound source separation performance equal to or higher than a certain level.

According to a first aspect, the invention provides a program in accordance with independent claim <NUM>. According to a second aspect, the invention provides an information processing method in accordance with independent claim <NUM>. According to a third aspect, the invention provides an information processing system in accordance with independent claim <NUM>. Further aspects are set forth in the dependent claims, the drawings, and the following description.

Embodiments and the like of the present disclosure will be described below with reference to the drawings. Note that the description will be given in the following order.

The embodiments and the like described below are preferred specific examples of the present disclosure, and the contents of the present disclosure are not limited to these embodiments and the like.

First, in order to facilitate understanding of the present disclosure, a technology related to the present disclosure will be described. <FIG> is a block diagram illustrating a configuration example of an information processing device (information processing device 1A) according to the technology related to the present disclosure. The information processing device 1A is a sound source separation device that separates a desired sound source signal from a mixed sound signal containing a plurality of sound source signals (for example, a vocal sound and each instrument sound constituting an accompaniment sound). Specifically, the information processing device 1A is incorporated into a smartphone, a personal computer, or an in-vehicle device. For example, the information processing device 1A is used to separate an accompaniment sound signal from a mixed sound signal stored in a medium such as a compact disc (CD) or a semiconductor memory or a mixed sound signal distributed over a network such as the Internet. The separated accompaniment sound signal is reproduced. A user sings along with the reproduction of the accompaniment sound signal. It is therefore possible for the user to easily perform karaoke without preparing the accompaniment sound signal itself. It goes without saying that the use of the information processing device 1A is not limited to karaoke. Text transcription processing or the like may be performed using the sound source separation result from the information processing device 1A. Note that the sound source separation processing performed by the information processing device 1A may be performed as online (real-time) processing or offline (batch) processing.

As illustrated in <FIG>, the information processing device 1A includes, roughly speaking, a feature extraction unit <NUM>, a DNN unit <NUM>, a multiplication unit <NUM> that is an example of an operation unit, and a separated sound source signal generation unit <NUM>. The mixed sound signal is input to the feature extraction unit <NUM>. Furthermore, the sound source signal (hereinafter, also referred to as separated sound source signal SA as appropriate) separated from the mixed sound signal is output from the separated sound source signal generation unit <NUM>. As described above, the mixed sound signal is a signal containing a mixture of a plurality of sound source signals, and is a signal digitized by pulse code modulation (PCM) or the like. A source of the mixed sound signal may be any source such as a recording medium or a server device on a network.

The feature extraction unit <NUM> performs a feature extraction process of extracting a feature of the mixed sound signal. For example, the feature extraction unit <NUM> equally splits data of the mixed sound signal into sections (frames) of a predetermined length, and performs frequency conversion (for example, a short-time Fourier transform) on each frame after the split. Such a frequency conversion process yields a time-series signal of a frequency spectrum. For example, in a case where the frame length is <NUM>, the frequency conversion length is also <NUM>, and conversion into <NUM> frequency spectra below the alias frequency is performed. That is, the process performed by the feature extraction unit <NUM> yields a frequency spectrum, specifically, a multidimensional vector (in this example, a vector having <NUM> dimensions). The process result from the feature extraction unit <NUM> is supplied to the following DNN unit <NUM>.

The DNN unit <NUM> generates sound source separation information for separating a predetermined sound source signal from the mixed sound signal. Specifically, the DNN <NUM> is an algorithm having a multi-layered structure based on a model of a human neural circuit (neural network) designed by machine learning to the generate sound source separation information.

The DNN unit <NUM> includes an encoder <NUM> that transforms the feature extracted from the mixed sound signal by the feature extraction unit <NUM>, a sub-neural network unit <NUM> to which the process result from the encoder <NUM> is input, and a decoder <NUM> to which the process result from the encoder <NUM> and the process result from each sub-neural network unit <NUM> are input.

The encoder <NUM> includes one of a plurality of affine transformation unit. The affine transformation unit performs a process represented by the following expression (<NUM>):<MAT> where, x denotes is an input vector, y denotes an output vector, W denotes a weighting coefficient to be obtained, b denotes a bias coefficient, and f denotes a nonlinear function.

The values of W and b are numerical values obtained as a result of learning performed in advance using a large data set.

As the nonlinear function f, for example, a rectified linear unit (ReLU) function, a sigmoid function, or the like can be used.

In this example, the encoder <NUM> includes a first affine transformation unit 31A and a second affine transformation unit 31B. The number of affine transformation units included in the encoder <NUM> is appropriately set so as to achieve sound source separation performance equal to or higher than a certain level. The encoder <NUM> transforms the feature by reducing the size of the feature, for example. More specifically, the encoder <NUM> reduces the number of dimensions of the multidimensional vector.

The sub-neural network unit <NUM> is a neural network present in the DNN unit <NUM>. As the sub-neural network unit <NUM>, a recurrent neural network (RNN) that uses at least one of a temporally past process result or a temporally future process result obtained for current input. The future process result can be used in a case of batch processing. As the recurrent neural network, a neural network using a gated recurrent Unit (GRU) or a long short term memory (LSTM) as an algorithm can be used.

The sub-neural network unit <NUM> includes a first RNN unit 32A, a second RNN unit 32B, and a third RNN unit 32C. The number of RNN units included in the sub-neural network unit <NUM> is appropriately set so as to achieve sound source separation performance equal to or higher than a certain level. Parameters used by each RNN unit are different, and the parameters are stored in a read only memory (ROM) or a random access memory (RAM) (not illustrated) of each RNN unit. In the following description, in a case where there is no particular need to distinguish between the ROM and the RAM, the ROM and the RAM are referred to as memory cell as appropriate. The first RNN unit 32A, the second RNN unit 32B, and the third RNN unit 32C sequentially perform a process on the process result from the encoder <NUM>.

The decoder <NUM> generates the sound source separation information on the basis of the process result from the encoder <NUM> and the process result from the sub-neural network unit <NUM>. The decoder <NUM> includes, for example, a third affine transformation unit 33A and a fourth affine transformation unit 33B. The third affine transformation unit 33A connects the process result from the encoder <NUM>, that is, the process result obtained by skipping the sub-neural network unit <NUM>, and the output from the sub-neural network unit <NUM> (also referred to as skip connection). The fourth affine transformation unit 33B performs affine transformation represented by the above-described expression (<NUM>) on the process result from the third affine transformation unit 33A. As a result of the processes performed by the third and fourth affine transformation units 33A and 33B, the feature size-reduced by the encoder <NUM> is restored, and a mask that is an example of the sound source separation information is obtained accordingly. The mask information is output from the DNN unit <NUM> and supplied to the multiplication unit <NUM>.

The multiplication unit <NUM> multiplies the feature extracted by the feature extraction unit <NUM> by the mask supplied from the DNN unit <NUM>. Multiplying the frequency spectrum by the mask allows a signal in the corresponding frequency band to be passed as it is (a predetermined numerical value in the mask = <NUM>) or to be blocked (a predetermined numerical value in the mask = <NUM>). That is, it can be said that the DNN unit <NUM> estimates a mask for passing only the frequency spectrum of the sound source that is to be separated and blocking the frequency spectrum of the sound source that is not to be separated.

The separated sound source signal generation unit <NUM> performs a process (for example, short-time inverse Fourier transform) of transforming the operation result from the multiplication unit <NUM> back to a time-axis signal. As a result, a desired sound source signal (sound source signal to be separated and time-axis signal) is generated. The separated sound source signal SA generated by the separated sound source signal generation unit <NUM> is used for application-specific purposes.

<FIG> illustrates examples of input/output sizes of each module constituting the DNN unit <NUM>. A <NUM>-dimensional frequency spectrum is input to the first affine transformation unit 31A, and the first affine transformation unit 31A performs affine transformation on the input to output a <NUM>-dimensional vector. The <NUM>-dimensional frequency spectrum (output from the first affine transformation unit 31A) is input to the second affine transformation unit 31B, and the second affine transformation unit 31B performs affine transformation on the input to output a <NUM>-dimensional vector. As described above, in the present embodiment, the size (number of dimensions) of the multidimensional vector input to the sub-neural network unit <NUM> is reduced by the first affine transformation unit 31A and the second affine transformation unit 31B. This allows an improvement in generalization ability of the DNN unit <NUM>.

The first RNN unit 32A, the second RNN unit 32B, and the third RNN unit 32C receive, as input, a multidimensional vector with <NUM> dimensions and output a multidimensional vector with the same number of dimensions.

The third affine transformation unit 33A receives, as input, a <NUM>-dimensional vector obtained by connecting the output from the second affine transformation unit 31B and the output from the third RNN unit 32C. Connecting the vector before the sub-neural network unit <NUM> performs the process allows an improvement in performance of the DNN unit <NUM>. The third affine transformation unit 33A receives the <NUM>-dimensional vector as input, and performs affine transformation on the input to output a <NUM>-dimensional vector. The fourth affine transformation unit 33B receives a <NUM>-dimensional vector as input, and performs affine transformation on the input to output a <NUM>-dimensional vector. The <NUM>-dimensional vector corresponds to the mask by which the multiplication unit <NUM> multiplies the frequency spectrum supplied from the feature extraction unit <NUM>. Note that the number of connected modules constituting the DNN unit <NUM> and the vector size of each input/output are examples, and the effective configuration differs in a manner that depends on each data set.

<FIG> is a block diagram illustrating a configuration example of another information processing device (information processing device 1B). The information processing device 1A is configured to separate one sound source signal from the mixed sound signal, but the information processing device 1B separates two sound source signals from the mixed sound signal. For example, the information processing device 1B separates the separated sound source signal SA and a separated sound source signal SB from the mixed sound signal.

As illustrated in <FIG>, the information processing device 1B includes a DNN unit <NUM>, a multiplication unit <NUM>, and a separated sound source signal generation unit <NUM> in addition to the configuration of the information processing device 1A. The DNN unit <NUM> includes an encoder <NUM>, a sub-neural network unit <NUM>, and a decoder <NUM>. The encoder <NUM> includes a first affine transformation unit 61A and a second affine transformation unit 61B. The sub-neural network unit <NUM> includes a first RNN unit 62A, a second RNN unit 62B, and a third RNN unit 62C. The decoder <NUM> includes a third affine transformation unit 63A and a fourth affine transformation unit 63B.

Roughly speaking, the flow of operation of the DNN unit <NUM> is substantially the same as of the DNN unit <NUM>. That is, the DNN unit <NUM> performs a process similar to the process performed by the DNN unit <NUM> on the feature of the mixed sound signal extracted by the feature extraction unit <NUM>. As a result, a mask for obtaining the separated sound source signal SB is generated. The multiplication unit <NUM> multiplies the feature of the mixed sound signal by the mask. The multiplication result is transformed into a time-axis signal by the separated sound source signal generation unit <NUM> to generate the separated sound source signal SB.

Note that the DNN <NUM> and the DNN unit <NUM> are individually trained. That is, even if the arrangement of the modules in each DNN unit is similar, the values of weighting coefficients and bias coefficients in the affine transformation units and the values of coefficients used in the RNN units are different, and such values are optimized for the sound source signal to be separated. As described above, when the number of sound source signals to be separated increases N-fold, the number of multiply-accumulate operations and the memory cell usage required for the DNN unit increase N-fold. Details of the present disclosure made in view of the above-described points will be described in more detail with reference to the embodiments.

<FIG> is a block diagram illustrating a configuration example of an information processing device (information processing device <NUM>) according to a first embodiment. Note that, among components included in the information processing device <NUM>, components similar to those of the information processing device 1A or the information processing device 1B are denoted by the same reference numerals, and redundant description will be omitted as appropriate. Furthermore, the matters described for the information processing devices 1A and 1B are applicable to each embodiment unless otherwise specified.

The information processing device <NUM> includes a DNN unit <NUM> instead of the DNN unit <NUM>. The DNN unit <NUM> generates a mask for separating a predetermined sound source signal (for example, the separated sound source signal SA) from the mixed sound signal and outputting the predetermined sound source signal.

The DNN unit <NUM> includes the encoder <NUM> and the decoder <NUM> described above. The DNN unit <NUM> further includes a plurality of sub-neural network units, specifically, two sub-neural network units (sub-neural network units <NUM> and <NUM>) arranged in parallel with each other. The sub-neural network unit <NUM> includes a first RNN unit 12A, a second RNN unit 12B, and a third RNN unit 12C. Furthermore, the sub-neural network unit <NUM> includes a first RNN unit 13A, a second RNN unit 13B, and a third RNN unit 13C. Each sub-neural network unit performs an RNN-based process on input given thereto.

The output from the encoder <NUM> is divided. In a case where a <NUM>-dimensional vector is output from the encoder <NUM> (see <FIG>), the number of dimensions of the vector is divided into two to generate a first vector with <NUM>-dimensions and a second vector with <NUM>-dimensions. Such a process is performed by the encoder <NUM>, for example. The first vector is input to, for example, the sub-neural network unit <NUM>, and the second vector is input to, for example, the sub-neural network unit <NUM>. The sub-neural network unit <NUM> performs a process using the RNN on the first vector to output a <NUM>-dimensional vector. Furthermore, the sub-neural network unit <NUM> performs a process using the RNN on the second vector to output a <NUM>-dimensional vector.

Next, the third affine transformation unit 33A of the decoder <NUM> connects the <NUM>-dimensional vector output from the sub-neural network unit <NUM>, the <NUM>-dimensional vector output from the sub-neural network unit <NUM>, and the <NUM>-dimensional vector output from the encoder <NUM>, and performs affine transformation on the connected vectors. The other processing is similar to the processing performed by the information processing device 1A, so that redundant description will be omitted.

A flow of processing performed by the information processing device <NUM> will be described with reference to the flowchart illustrated in <FIG>.

When the processing is started, each module constituting the DNN unit <NUM> reads coefficients stored in the ROM or the like (not illustrated) in step ST1. Then, the processing proceeds to step ST2.

In step ST2, the mixed sound signal is input to the information processing device <NUM>. Then, the processing proceeds to step ST3.

In step ST3, the feature extraction unit <NUM> extracts a feature vector from the mixed sound signal. For example, a <NUM>-dimensional feature vector is input to the encoder <NUM> of the DNN unit <NUM>. Then, the processing proceeds to step ST4.

In step ST4, the encoder <NUM>, specifically, the first affine transformation unit 31A and the second affine transformation unit 31B, performs an encoding process. As a result of the process, for example, a <NUM>-dimensional vector is output from the second affine transformation unit 31B. Then, the processing proceeds to step ST5.

In step ST5, the <NUM>-dimensional vector is equally divided into two <NUM>-dimensional vectors (first and second vectors). The first vector is input to the sub-neural network unit <NUM>, and the second vector is input to the sub-neural network unit <NUM>. Note that the process related to step ST5 may be included in the encoding process of step ST4. Then, the processing proceeds to step ST6 and step ST7.

In step ST6, the sub-neural network unit <NUM> performs a process using the first vector. Furthermore, in step ST7, the sub-neural network unit <NUM> performs a process using the second vector. Note that the processes related to steps ST6 and ST7 may be performed in parallel or sequentially. Then, the processing proceeds to step ST8.

In step ST8, a process of connecting vectors is performed. This process is performed by the decoder <NUM>, for example. The third affine transformation unit 33A generates a <NUM>-dimensional vector by connecting the <NUM>-dimensional vector output from the second affine transformation unit 31B, the <NUM>-dimensional vector output from the sub-neural network unit <NUM>, and the <NUM>-dimensional vector output from the sub-neural network unit <NUM>. Then, the processing proceeds to step ST9.

In step ST9, the third affine transformation unit 33A and the fourth affine transformation unit 33B of the decoder <NUM> perform a decoding process. As a result of the decoding process, a mask represented by a <NUM>-dimensional vector is output from the fourth affine transformation unit 33B. Note that the process of step ST8 described above may be included in the decoding process of step ST9. Then, the processing proceeds to step ST10.

In step ST10, a multiplication process is performed. Specifically, the multiplication unit <NUM> multiplies the vector output from the feature extraction unit <NUM> by the mask obtained by the DNN unit <NUM>. Then, the processing proceeds to step ST11.

In step ST11, a separated sound source signal generation process is performed. Specifically, the separated sound source signal generation unit <NUM> transforms a frequency spectrum obtained as a result of the operation performed by the multiplication unit <NUM> into a time-axis signal. Then, the processing proceeds to step ST12.

In step ST12, it is determined whether or not the input of the mixed sound signal is continuing. Such determination is performed, for example, by a central processing unit (CPU) (not illustrated) that centrally controls how the information processing device <NUM> operates. In a case where there is no input of the mixed sound signal (in a case of No), the processing is brought to an end. In a case where the input of the mixed sound signal is continuing (in a case of Yes), the processing returns to step ST2, and the above-described processes are repeated.

An example of the effect obtained by the present embodiment described above will be described.

Since the total size of the divided vectors is <NUM> + <NUM> = <NUM> dimensions, it is apparently the same as before the division. It is, however, possible to reduce the number of coefficients stored in the DNN <NUM> and the number of multiply-accumulate operations. A specific example will be described below.

Consider, for example, vector-to-vector multiplication (matrix operation) performed by the sub-neural network unit <NUM> (the same applies to the sub-neural network unit <NUM>). In a matrix operation on <NUM>-dimensional vector input and <NUM>-dimensional vector output, multiplication is performed <NUM> × <NUM> = <NUM> times. On the other hand, in a case of division into two with <NUM> dimensions, multiplication of the <NUM>-dimensional matrix only needs to be performed twice, so that the number of times of multiplication is (<NUM> × <NUM>) × <NUM> = <NUM>, which is smaller than in the case of no division. As described above, it can be seen that the use of a plurality of small matrices has merit in terms of the amount of operations as compared with the use of a large matrix. There is a plurality of matrix operations depending on the input/output vector size in the modules of the RNN unit such as the GRU or the LSTM, the configuration according to the present embodiment can effectively reduce the number of operations.

On the other hand, even if the number of operations can be reduced, it is not preferable that the accuracy of sound source separation be thereby reduced. In the present embodiment, it is, however, possible to minimize a reduction in the accuracy of sound source separation. This point will be described in detail with reference to <FIG>.

<FIG> is a graph showing a relation between the number of coefficients held by the DNN unit and the sound source separation performance. The horizontal axis (number of weights) of the graph represents the number of coefficients present in the DNN unit (affine transformation unit or sub-neural network unit), and is a value roughly proportional to the number of operations and the capacity of the memory cell required for the process performed by the DNN unit. Furthermore, the vertical axis of the graph represents a signal to distortion ratio (SDR) [dB]. The SDR is an index indicating the accuracy with which the target sound source is separated, and is an index indicating that the larger the value, the higher the separation performance. Therefore, in the graph shown in <FIG>, the closer data is plotted to the upper-left corner, the smaller the amount of used computation resources, and the higher the sound source separation performance.

Consider how the number of coefficients and the SDR change in a case where the configuration of the DNN unit is changed. As a result, as shown in <FIG>, four plots (hereinafter, referred to as patterns PA, PB, PC, and PD as appropriate) were obtained. In this example, an example where the GRU is used as the algorithm of the RNN unit will be described, but a similar result can be obtained even in a case where another algorithm is used.

The pattern PA in <FIG> corresponds to a case where a typical configuration (configuration illustrated in <FIG>) is used, and the input/output vector size for the sub-neural network unit is <NUM> dimensions (<NUM> Grouped-GRU [<NUM>]). The pattern PB in <FIG> corresponds to a case where the typical configuration (configuration illustrated in <FIG>) is used, and the input/output vector size for the sub-neural network unit is <NUM> dimensions (<NUM> Grouped-GRU [<NUM>]). The pattern PC in <FIG> corresponds to a case where two sub-neural network units are used as in the configuration according to the present embodiment, and the input/output vector size for the sub-neural network units is equally divided (into <NUM> dimensions) (<NUM> Grouped-GRU [<NUM>, <NUM>]). The pattern PD in <FIG> corresponds to a case where four sub-neural network units are used, and the input/output vector size for the sub-neural network units are unequally divided (into <NUM> dimensions, <NUM> dimensions, <NUM> dimensions, and <NUM> dimensions) divided (<NUM> Grouped-GRU [<NUM>, <NUM>, <NUM>, <NUM>]).

In a case where the configuration and the vector size correspond to the pattern PA, the number of coefficients was approximately <NUM> million, and the SDR was approximately <NUM>. Although the sound source separation performance is high, the number of operations increases due to the larger number of coefficients. On the other hand, in a case where the configuration and the vector size correspond to the pattern PB, that is, in a case where the vector size is reduced with the configuration of the DNN unit the same as in the case of the pattern PA, the number of coefficients was slightly less than about <NUM>,<NUM>, thereby allowing a reduction in the number of operations. The SDR in the case of the pattern PB, however, was approximately <NUM>, and the sound source separation performance deteriorated as compared with the case of the pattern PA. Therefore, the sound source separation performance deteriorates only by a simple reduction in the number of coefficients.

In a case where the configuration and the vector size correspond to the pattern PC, the number of coefficients was slightly greater than about <NUM> million. The number of coefficients was able to be reduced as compared with the pattern PA, thereby allowing a reduction in the number of operations. Moreover, the SDR in the case where the configuration and the vector size correspond to the pattern PC was slightly greater than approximately <NUM>, and high sound source separation performance as compared with the pattern PA according to the typical configuration was achieved. Furthermore, in a case where the configuration and the vector size corresponding to the pattern PD, the number of coefficients was able to be reduced (to about <NUM> million or slightly less) as compared with the pattern PA, and a better SDR was also achieved. Moreover, in the case where the configuration and the vector size correspond to the pattern PD, the number of coefficients was able to be reduced as compared with the pattern PC, and almost the same SDR was also achieved. As described above, both the patterns PC and PD are located at the upper left of the line connecting the patterns PA and PB, so that it has been verified that the patterns PC and PD achieve higher sound source separation performance with a reduction in the number of operations as compared with the conventional method.

From the above, it has been verified that the information processing device according to the present embodiment can reduce the number of operations as compared with the information processing device according to the typical configuration, and can not only prevent a deterioration in the sound source separation performance but also improve the sound source separation performance.

Moreover, from the results shown in <FIG>, it has been verified that the number of sub-neural network units is not limited to two, and the size of the vector input to each sub-neural network units may be different (may be unequally divided).

Next, a second embodiment will be described. Note that the matters described in the first embodiment and the like are applicable to the second embodiment unless otherwise specified.

<FIG> is a block diagram illustrating a configuration example of an information processing device (information processing device <NUM>) according to the second embodiment. Note that, in <FIG>, the configuration related to the DNN unit <NUM> is simplified as appropriate due to a space limitation on the drawing. The information processing device <NUM> has the configuration related to the encoder made for shared use in a configuration corresponding to a case where there is a plurality of sound sources to be separated (for example, in the configuration of the information processing device 1B illustrated in <FIG>).

In the information processing device 1B illustrated in <FIG>, the encoders <NUM> and <NUM> are separately provided as encoders, but are identical to each other in the details of process of reducing the vector size (the number of dimensions in this example) of the feature vector extracted from the mixed sound signal. Therefore, as illustrated in <FIG>, the information processing device <NUM> includes an encoder made for shared use among a plurality of DNN units (for example, the DNN units <NUM> and <NUM>). This allows a reduction in operation load on the information processing device <NUM>. The output from the encoder <NUM> is input to the sub-neural network unit <NUM> and the decoder <NUM> of the DNN unit <NUM> and to the sub-neural network unit <NUM> and the decoder <NUM> of the DNN unit <NUM>. The other processing is basically the same as the processing performed by the information processing device 1B, so that redundant description will be omitted.

When the processing is started, each module constituting the DNN unit <NUM> reads coefficients stored in the ROM or the like (not illustrated) in step ST21. Then, the processing proceeds to step ST22.

In step ST22, the mixed sound signal is input to the information processing device <NUM>. Then, the processing proceeds to step ST23.

In step ST23, the feature extraction unit <NUM> extracts a feature vector from the mixed sound signal. For example, a <NUM>-dimensional feature vector is input to the encoder <NUM> of the DNN unit <NUM>. Then, the processing proceeds to step ST24.

In step ST24, the encoder <NUM>, specifically, the first affine transformation unit 31A and the second affine transformation unit 31B, performs an encoding process. As a result of the process, for example, a vector having the number of dimensions reduced to <NUM> is output from the second affine transformation unit 31B. Such a vector is input to the sub-neural network unit <NUM> and the decoder <NUM> of the DNN unit <NUM> and to the sub-neural network unit <NUM> and the decoder <NUM> of the DNN unit <NUM>. Then, the processing proceeds to step ST25 and step ST29.

The processes related to steps ST25 to ST28 include the process performed by the sub-neural network unit <NUM>, the decoding process performed by the decoder <NUM>, the multiplication process performed by the multiplication unit <NUM>, and the separated sound source signal generation process performed by the separated sound source signal generation unit <NUM>. The separated sound source signal SA is generated as a result of the separated sound source signal generation process. Furthermore, the processes related to steps ST29 to ST32 include the process performed by the sub-neural network unit <NUM>, the decoding process performed by the decoder <NUM>, the multiplication process performed by the multiplication unit <NUM>, and the separated sound source signal generation process performed by the separated sound source signal generation unit <NUM>. The separated sound source signal SB is generated as a result of the separated sound source signal generation process. The details of each process have already been described, so that redundant description will be omitted as appropriate. For the processes related to steps ST28 and ST32, the process related to step ST33 is performed.

In step ST33, it is determined whether or not the input of the mixed sound signal is continuing. Such determination is performed, for example, by a CPU (not illustrated) that centrally controls how the information processing device <NUM> operates. In a case where there is no input of the mixed sound signal (in a case of No), the processing is brought to an end. In a case where the input of the mixed sound signal is continuing (in a case of Yes), the processing returns to step ST22, and the above-described processes are repeated.

Note that, in the information processing device <NUM>, the decoder and the decoder <NUM> may be replaced with a decoder made for shared use. Note that the decoders <NUM> and <NUM> each receive input via a sub-neural network unit having coefficients optimized for a corresponding sound source signal to be separated. It is therefore preferable that the coefficients of the decoder <NUM> be also optimized for a corresponding sound source signal to be separated from the viewpoint of preventing a deterioration in the sound source separation performance. It is therefore preferable that the decoder <NUM> and the decoder <NUM> be each provided for a corresponding sound source signal to be separated.

Next, a third embodiment will be described. Note that the matters described in the first and second embodiments and the like are applicable to the third embodiment unless otherwise specified. Roughly speaking, the third embodiment has a configuration obtained by combining the first and second embodiments.

<FIG> is a block diagram illustrating a configuration example of an information processing device (information processing device <NUM>) according to the third embodiment. In the information processing device <NUM>, the DNN unit <NUM> described in the first embodiment is used instead of the DNN unit <NUM> of the information processing device <NUM> described above. Furthermore, in the information processing device <NUM>, a DNN unit 6A is used instead of the DNN unit <NUM> of the information processing device <NUM> described above. The DNN unit 6A is different from the DNN unit <NUM> in the configuration of the sub-neural network unit. That is, the DNN unit 6A includes a plurality of sub-neural network units in a manner similar to the first embodiment. The DNN unit 6A includes, for example, a sub-neural network unit <NUM> and a sub-neural network unit <NUM>. The sub-neural network unit <NUM> includes a first RNN unit 65A, a second RNN unit 65B, and a third RNN unit 65C. Furthermore, the sub-neural network unit <NUM> includes a first RNN unit 66A, a second RNN unit 66B, and a third RNN unit 66C. The DNN unit 6A includes the decoder <NUM> in the same manner as the DNN unit <NUM>. The details of the processing performed by the information processing device <NUM> have been described in the first and second embodiments and the like, so that redundant description will be omitted. The third embodiment can obtain an effect similar to the effects obtained by the first and second embodiments.

<FIG> shows specific numerical examples of the number of coefficients used in the DNN unit in the first to third embodiments described above. As basic configurations, four patterns of a typical configuration (see <FIG>), a configuration including a plurality of sub-neural network units (see <FIG>), a configuration including an encoder made for shared use (see <FIG>), and a configuration including a plurality of sub-neural network units and an encoder made for shared use (see <FIG>) were prepared. The number of sound sources to be separated was two or ten, and sub-neural network units were provided so as to correspond to the number of sound sources to be separated.

As shown in <FIG>, with the typical configuration, in a case where the number of sound sources to be separated is two, the number of coefficients used in the DNN unit was approximately <NUM> million. Furthermore, with the typical configuration, in a case where the number of sound sources to be separated is ten, the number of coefficients used in the DNN unit was approximately <NUM> million. The number of coefficients used in the DNN unit in the other configurations is represented by a value relative to the number, taken as <NUM>%, of coefficients used in the DNN unit in the typical configuration and an approximate number of coefficients. For the configuration in which the GRU algorithm is applied to each RNN unit, and a plurality of sub-neural network units is provided, a value obtained by equally dividing the input/output vector size was used.

With the configuration including a plurality of sub-neural network units, the number of coefficients used in the DNN unit was approximately <NUM> million (about <NUM>%) in a case where the number of sound sources to be separated is two, and the number of coefficients used in the DNN unit was approximately <NUM> million (about <NUM>%) in a case where the number of sound sources to be separated is ten. That is, the number of coefficients was able to be reduced as compared with the typical configuration. In other words, the number of operations was able to be reduced.

With the configuration including an encoder made for shared use, as the number of sound sources increased, the number of coefficients used in the DNN unit was able to be reduced. (In the case of two sound sources, the number of coefficients used in the DNN unit was approximately <NUM> million (about <NUM>%), and in the case where the number of sound sources to be separated is ten, the number of coefficients used in the DNN unit was approximately <NUM> million (about <NUM>%).

With the configuration including a plurality of sub-neural network units and an encoder made for shared use, the number of coefficients used in the DNN unit was able to be further reduced. (In the case of two sound sources, the number of coefficients used in the DNN unit was approximately <NUM> million (about <NUM>%), and in the case where the number of sound sources to be separated is ten, the number of coefficients used in the DNN unit was approximately <NUM> million (about <NUM>%).

Although the embodiments of the present disclosure have been described above, the present disclosure is not limited to the above-described embodiments, and various modifications can be made.

As illustrated in <FIG>, the information processing device <NUM> may include a filter unit <NUM> (post filter) in a stage following the multiplication unit <NUM> and the multiplication unit <NUM>. The filter unit <NUM> separates a desired sound source signal with higher accuracy using a plurality of separated sound source (in the example illustrated in <FIG>, two sound source) signals. For example, it is assumed that a separated vocal signal is output from multiplication unit <NUM>, and a separated piano accompaniment sound signal is output from multiplication unit <NUM>. The filter unit <NUM> separates the vocal signal (an example of the separated sound source signal SA) with higher accuracy by removing a residual component (noise component) of the piano accompaniment sound signal from the vocal signal while referring to the piano accompaniment sound signal. As the filter unit <NUM>, a known filter such as a single-channel Wiener filter can be used.

For example, the present disclosure may be configured as cloud computing in which one function is shared by a plurality of devices over a network and processing is performed in cooperation. For example, the feature extraction unit may be provided in a server device, and the feature extraction process may be performed in the server device.

Claim 1:
A computer program comprising instructions which, when the program is executed by a computer causes the computer to execute an information processing method, the information processing method comprising:
generating, by a neural network unit, sound source separation information for separating a predetermined sound source signal from a mixed sound signal containing a plurality of sound source signals;
transforming, by an encoder included in the neural network unit, a feature extracted from the mixed sound signal;
inputting a process result from the encoder to each of a plurality of sub-neural network units included in the neural network unit; and
inputting the process result from the encoder and a process result from each of the plurality of sub-neural network units to a decoder included in the neural network unit,
wherein the encoder performs the transformation by reducing a size of the feature;
wherein the size of the feature is equally divided to correspond to a number of the plurality of sub-neural network units or unequally divided,
wherein, if the feature is equally divided, features with a size after the division are each input to a corresponding one of the sub-neural network units,
or if the feature is unequally divided, features with sizes after the division are each input to a corresponding one of the sub-neural network units.